Amazon web services 我可以在数据帧上应用AWS FindMatch转换吗?如果是,那怎么办
我想知道我是否可以在spark数据帧上应用AWS Glue中的FindMatch ml转换。目前我可以在dynamicframe上使用它。下面是我想在动态帧上使用findmatch转换的语法Amazon web services 我可以在数据帧上应用AWS FindMatch转换吗?如果是,那怎么办,amazon-web-services,machine-learning,aws-glue,Amazon Web Services,Machine Learning,Aws Glue,我想知道我是否可以在spark数据帧上应用AWS Glue中的FindMatch ml转换。目前我可以在dynamicframe上使用它。下面是我想在动态帧上使用findmatch转换的语法 <output DynamicFrame on which the ml transform has been applied> = FindMatches.apply(frame = <Input DynamicFrame>, transformId = <transfor
<output DynamicFrame on which the ml transform has been applied> =
FindMatches.apply(frame = <Input DynamicFrame>, transformId = <transformation
id of the findmatch ml transform created separately>)
下面是我尝试使用dynamicframe而不是dataframe时的代码,代码可以正常工作
import sys
from awsglue.transforms import *
from awsglue.utils import getResolvedOptions
from pyspark.context import SparkContext
from awsglue.context import GlueContext
from awsglue.job import Job
from awsglueml.transforms import FindMatches
args = getResolvedOptions(sys.argv, ['JOB_NAME'])
sc = SparkContext()
glueContext = GlueContext(sc)
spark = glueContext.spark_session
job = Job(glueContext)
job.init(args['JOB_NAME'], args)
datasource0 = glueContext.create_dynamic_frame.from_catalog(database = "hospitality", table_name =
"patientinputdata", transformation_ctx = "datasource0")
resolvechoice1 = ResolveChoice.apply(frame = datasource0, choice = "MATCH_CATALOG", database =
"hospitality", table_name = "patientinputdata", transformation_ctx = "resolvechoice1")
findmatches2 = FindMatches.apply(frame = resolvechoice1, transformId = "tfm-
0cadd1e6d2da40d7c18db7836e92be93833b6019", transformation_ctx = "findmatches2")
我尝试在线搜索是否可以找到FindMatch ml transform的代码,但在任何地方都找不到 FindMatch仅在动态帧上工作,正如您所知。。。 因此,您可以随时将spark df转换为动态帧
from awsglue.dynamicframe import DynamicFrame
glueContext = GlueContext(SparkContext.getOrCreate())
Dyf0 = DynamicFrame.fromDF(df0, glueContext, "anyname")
然后根据需要运行FindMatch
from awsglue.dynamicframe import DynamicFrame
glueContext = GlueContext(SparkContext.getOrCreate())
Dyf0 = DynamicFrame.fromDF(df0, glueContext, "anyname")