Warning: file_get_contents(/data/phpspider/zhask/data//catemap/2/python/280.json): failed to open stream: No such file or directory in /data/phpspider/zhask/libs/function.php on line 167

Warning: Invalid argument supplied for foreach() in /data/phpspider/zhask/libs/tag.function.php on line 1116

Notice: Undefined index: in /data/phpspider/zhask/libs/function.php on line 180

Warning: array_chunk() expects parameter 1 to be array, null given in /data/phpspider/zhask/libs/function.php on line 181
ommon.cache.LocalCache$LocalLoadingCache.get(LocalCache.java:4958) 位于org.apache.beam.runners.core.construction.RehydratedComponents.getCoder(RehydratedComponents.java:168) 位于org.apache.beam.runners.core.construction.PCollectionTranslation.fromProto(PCollectionTranslation.java:51) 位于org.apache.beam.runners.core.construction.RehydratedComponents$3.load(RehydratedComponents.java:108) 位于org.apache.beam.runners.core.construction.RehydratedComponents$3.load(RehydratedComponents.java:98) 位于org.apache.beam.vendor.guava.v26_0_jre.com.google.common.cache.LocalCache$LoadingValueReference.loadFuture(LocalCache.java:3528) 在org.apache.beam.vendor.guava.v26_0_jre.com.google.common.cache.LocalCache$Segment.loadSync(LocalCache.java:2277) 位于org.apache.beam.vendor.guava.v26_0_jre.com.google.common.cache.LocalCache$Segment.lockedGetOrLoad(LocalCache.java:2154) 位于org.apache.beam.vendor.guava.v26_0_jre.com.google.common.cache.LocalCache$Segment.get(LocalCache.java:2044) ... 还有27个 原因:java.lang.IllegalArgumentException:遇到不支持的逻辑类型URN:beam:logical:pythonsdk_any:v1 位于org.apache.beam.sdk.schemas.schematransation.fieldtypefromProtowithwithoutNullable(schematransation.java:316) 位于org.apache.beam.sdk.schemas.schematransation.fieldTypeFromProto(schematransation.java:232) 位于org.apache.beam.sdk.schemas.schematransation.fieldFromProto(schematransation.java:226) 位于org.apache.beam.sdk.schemas.schematransation.schemaFromProto(schematransation.java:212) 位于org.apache.beam.runners.core.construction.CoderTranslators$8.fromComponents(CoderTranslators.java:169) 位于org.apache.beam.runners.core.construction.CoderTranslators$8.fromComponents(CoderTranslators.java:151) 位于org.apache.beam.runners.core.construction.CoderTranslation.fromKnownCoder(CoderTranslation.java:170) 位于org.apache.beam.runners.core.construction.CoderTranslation.fromProto(CoderTranslation.java:145) 位于org.apache.beam.runners.core.construction.RehydratedComponents$2.load(RehydratedComponents.java:87) 位于org.apache.beam.runners.core.construction.RehydratedComponents$2.load(RehydratedComponents.java:82) 位于org.apache.beam.vendor.guava.v26_0_jre.com.google.common.cache.LocalCache$LoadingValueReference.loadFuture(LocalCache.java:3528) 在org.apache.beam.vendor.guava.v26_0_jre.com.google.common.cache.LocalCache$Segment.loadSync(LocalCache.java:2277) 位于org.apache.beam.vendor.guava.v26_0_jre.com.google.common.cache.LocalCache$Segment.lockedGetOrLoad(LocalCache.java:2154) 位于org.apache.beam.vendor.guava.v26_0_jre.com.google.common.cache.LocalCache$Segment.get(LocalCache.java:2044) ... 38多 _Python_Sql_Sdk_Apache Beam_Transformation - Fatal编程技术网

ommon.cache.LocalCache$LocalLoadingCache.get(LocalCache.java:4958) 位于org.apache.beam.runners.core.construction.RehydratedComponents.getCoder(RehydratedComponents.java:168) 位于org.apache.beam.runners.core.construction.PCollectionTranslation.fromProto(PCollectionTranslation.java:51) 位于org.apache.beam.runners.core.construction.RehydratedComponents$3.load(RehydratedComponents.java:108) 位于org.apache.beam.runners.core.construction.RehydratedComponents$3.load(RehydratedComponents.java:98) 位于org.apache.beam.vendor.guava.v26_0_jre.com.google.common.cache.LocalCache$LoadingValueReference.loadFuture(LocalCache.java:3528) 在org.apache.beam.vendor.guava.v26_0_jre.com.google.common.cache.LocalCache$Segment.loadSync(LocalCache.java:2277) 位于org.apache.beam.vendor.guava.v26_0_jre.com.google.common.cache.LocalCache$Segment.lockedGetOrLoad(LocalCache.java:2154) 位于org.apache.beam.vendor.guava.v26_0_jre.com.google.common.cache.LocalCache$Segment.get(LocalCache.java:2044) ... 还有27个 原因:java.lang.IllegalArgumentException:遇到不支持的逻辑类型URN:beam:logical:pythonsdk_any:v1 位于org.apache.beam.sdk.schemas.schematransation.fieldtypefromProtowithwithoutNullable(schematransation.java:316) 位于org.apache.beam.sdk.schemas.schematransation.fieldTypeFromProto(schematransation.java:232) 位于org.apache.beam.sdk.schemas.schematransation.fieldFromProto(schematransation.java:226) 位于org.apache.beam.sdk.schemas.schematransation.schemaFromProto(schematransation.java:212) 位于org.apache.beam.runners.core.construction.CoderTranslators$8.fromComponents(CoderTranslators.java:169) 位于org.apache.beam.runners.core.construction.CoderTranslators$8.fromComponents(CoderTranslators.java:151) 位于org.apache.beam.runners.core.construction.CoderTranslation.fromKnownCoder(CoderTranslation.java:170) 位于org.apache.beam.runners.core.construction.CoderTranslation.fromProto(CoderTranslation.java:145) 位于org.apache.beam.runners.core.construction.RehydratedComponents$2.load(RehydratedComponents.java:87) 位于org.apache.beam.runners.core.construction.RehydratedComponents$2.load(RehydratedComponents.java:82) 位于org.apache.beam.vendor.guava.v26_0_jre.com.google.common.cache.LocalCache$LoadingValueReference.loadFuture(LocalCache.java:3528) 在org.apache.beam.vendor.guava.v26_0_jre.com.google.common.cache.LocalCache$Segment.loadSync(LocalCache.java:2277) 位于org.apache.beam.vendor.guava.v26_0_jre.com.google.common.cache.LocalCache$Segment.lockedGetOrLoad(LocalCache.java:2154) 位于org.apache.beam.vendor.guava.v26_0_jre.com.google.common.cache.LocalCache$Segment.get(LocalCache.java:2044) ... 38多

ommon.cache.LocalCache$LocalLoadingCache.get(LocalCache.java:4958) 位于org.apache.beam.runners.core.construction.RehydratedComponents.getCoder(RehydratedComponents.java:168) 位于org.apache.beam.runners.core.construction.PCollectionTranslation.fromProto(PCollectionTranslation.java:51) 位于org.apache.beam.runners.core.construction.RehydratedComponents$3.load(RehydratedComponents.java:108) 位于org.apache.beam.runners.core.construction.RehydratedComponents$3.load(RehydratedComponents.java:98) 位于org.apache.beam.vendor.guava.v26_0_jre.com.google.common.cache.LocalCache$LoadingValueReference.loadFuture(LocalCache.java:3528) 在org.apache.beam.vendor.guava.v26_0_jre.com.google.common.cache.LocalCache$Segment.loadSync(LocalCache.java:2277) 位于org.apache.beam.vendor.guava.v26_0_jre.com.google.common.cache.LocalCache$Segment.lockedGetOrLoad(LocalCache.java:2154) 位于org.apache.beam.vendor.guava.v26_0_jre.com.google.common.cache.LocalCache$Segment.get(LocalCache.java:2044) ... 还有27个 原因:java.lang.IllegalArgumentException:遇到不支持的逻辑类型URN:beam:logical:pythonsdk_any:v1 位于org.apache.beam.sdk.schemas.schematransation.fieldtypefromProtowithwithoutNullable(schematransation.java:316) 位于org.apache.beam.sdk.schemas.schematransation.fieldTypeFromProto(schematransation.java:232) 位于org.apache.beam.sdk.schemas.schematransation.fieldFromProto(schematransation.java:226) 位于org.apache.beam.sdk.schemas.schematransation.schemaFromProto(schematransation.java:212) 位于org.apache.beam.runners.core.construction.CoderTranslators$8.fromComponents(CoderTranslators.java:169) 位于org.apache.beam.runners.core.construction.CoderTranslators$8.fromComponents(CoderTranslators.java:151) 位于org.apache.beam.runners.core.construction.CoderTranslation.fromKnownCoder(CoderTranslation.java:170) 位于org.apache.beam.runners.core.construction.CoderTranslation.fromProto(CoderTranslation.java:145) 位于org.apache.beam.runners.core.construction.RehydratedComponents$2.load(RehydratedComponents.java:87) 位于org.apache.beam.runners.core.construction.RehydratedComponents$2.load(RehydratedComponents.java:82) 位于org.apache.beam.vendor.guava.v26_0_jre.com.google.common.cache.LocalCache$LoadingValueReference.loadFuture(LocalCache.java:3528) 在org.apache.beam.vendor.guava.v26_0_jre.com.google.common.cache.LocalCache$Segment.loadSync(LocalCache.java:2277) 位于org.apache.beam.vendor.guava.v26_0_jre.com.google.common.cache.LocalCache$Segment.lockedGetOrLoad(LocalCache.java:2154) 位于org.apache.beam.vendor.guava.v26_0_jre.com.google.common.cache.LocalCache$Segment.get(LocalCache.java:2044) ... 38多 ,python,sql,sdk,apache-beam,transformation,Python,Sql,Sdk,Apache Beam,Transformation,tl;dr如果您在beam.Row调用中根据您期望的类型在每个字段周围添加强制转换,您应该能够避免此问题,例如: | beam.Map(lambda x: beam.Row(recipe = str(x[0]), fruit = str(x[1]), quantity = int(x[2]),

tl;dr如果您在
beam.Row
调用中根据您期望的类型在每个字段周围添加强制转换,您应该能够避免此问题,例如:

        | beam.Map(lambda x: beam.Row(recipe = str(x[0]),
                                 fruit = str(x[1]),
                                 quantity = int(x[2]),
                                 unit_cost = float(x[3]),
                                 is_berry = bool(x[1].endswith('berry')))))
详细说明 错误
java.lang.IllegalArgumentException:遇到不支持的逻辑类型URN:beam:logical:pythonsdk_any:v1
表示beam python无法确定要传递给SqlTransform的PCollection中某个字段的类型。通常这不是什么大问题,Beam Python只是使用一个回退(称为
Beam:logical:pythonsdk_any:v1
),用Python序列化(即pickle)对任何此类字段的值进行编码。这很好,因为下游Python转换完全能够读取pickle编码的数据。这可能会对性能产生一些影响,但不会破坏您的管道

然而,在
SqlTransform
的情况下,我们实际上使用了来自的实现,它不了解Python序列化。因此,当它遇到beam:logical:pythonsdk_any:v1时,它就会放弃

我在上面建议的解决方案是围绕每个值添加强制转换,确保Beam python将为每个字段推断出一种特定的类型,我们可以以可移植的方式进行编码


我申请了这个问题。在这种情况下,我们应该提出一个更有用的错误消息。谢谢你提出这个问题

tl;dr如果您在
beam.Row
调用中根据您期望的类型在每个字段周围添加强制转换,您应该能够避免此问题,例如:

        | beam.Map(lambda x: beam.Row(recipe = str(x[0]),
                                 fruit = str(x[1]),
                                 quantity = int(x[2]),
                                 unit_cost = float(x[3]),
                                 is_berry = bool(x[1].endswith('berry')))))
详细说明 错误
java.lang.IllegalArgumentException:遇到不支持的逻辑类型URN:beam:logical:pythonsdk_any:v1
表示beam python无法确定要传递给SqlTransform的PCollection中某个字段的类型。通常这不是什么大问题,Beam Python只是使用一个回退(称为
Beam:logical:pythonsdk_any:v1
),用Python序列化(即pickle)对任何此类字段的值进行编码。这很好,因为下游Python转换完全能够读取pickle编码的数据。这可能会对性能产生一些影响,但不会破坏您的管道

然而,在
SqlTransform
的情况下,我们实际上使用了来自的实现,它不了解Python序列化。因此,当它遇到beam:logical:pythonsdk_any:v1时,它就会放弃

我在上面建议的解决方案是围绕每个值添加强制转换,确保Beam python将为每个字段推断出一种特定的类型,我们可以以可移植的方式进行编码


我申请了这个问题。在这种情况下,我们应该提出一个更有用的错误消息。谢谢你提出这个问题

谢谢你的快速回答。我做了相应的更改,现在错误更改为:FileNotFoundError:[Errno 2]没有这样的文件或目录:“docker”嗯,您在使用DirectRunner吗?我想你需要安装docker(例如Ubuntu)来运行一个使用SqlTransformYes的管道,这就是我做的,现在正在工作。Hi@TheNeuralBit。我需要在docker上安装什么?我试图使用direct runner在apache beam中使用sql转换,但得到了“Valu”
        | beam.Map(lambda x: beam.Row(recipe = str(x[0]),
                                 fruit = str(x[1]),
                                 quantity = int(x[2]),
                                 unit_cost = float(x[3]),
                                 is_berry = bool(x[1].endswith('berry')))))