Google cloud dataflow 工作池缩小时出错:";无法在不丢失活动无序排列数据的情况下缩小尺寸”;

Google cloud dataflow 工作池缩小时出错:";无法在不丢失活动无序排列数据的情况下缩小尺寸”;,google-cloud-dataflow,Google Cloud Dataflow,更新至最新SDK版本0.3.150326,由于此错误,我们的作业失败: (d0f58ccaf368cf1f):工作流失败。原因:(539037ea87656484): 如果不丢失活动无序排列数据,则无法缩小大小。旧尺寸=10, 新尺寸=8 工作编号:2015-04-02_21_26_53-11930390736602232537 无法复制,但我想我应该问一下这是否是一个已知的问题 综上所述,自动缩放目前似乎只是“实验性的”,但我可以想象这是云数据流的核心功能,因此应该得到充分支持 1087 [m

更新至最新SDK版本0.3.150326,由于此错误,我们的作业失败:

(d0f58ccaf368cf1f):工作流失败。原因:(539037ea87656484): 如果不丢失活动无序排列数据,则无法缩小大小。旧尺寸=10, 新尺寸=8

工作编号:2015-04-02_21_26_53-11930390736602232537

无法复制,但我想我应该问一下这是否是一个已知的问题

综上所述,自动缩放目前似乎只是“实验性的”,但我可以想象这是云数据流的核心功能,因此应该得到充分支持

1087 [main] INFO  com.google.cloud.dataflow.sdk.runners.DataflowPipelineRunner  - Executing pipeline on the Dataflow Service, which will have billing implications related to Google Compute Engine usage and other Google Cloud Services.
1103 [main] INFO  com.google.cloud.dataflow.sdk.util.PackageUtil  - Uploading 79 files from PipelineOptions.filesToStage to GCS to prepare for execution in the cloud.
43086 [main] INFO  com.google.cloud.dataflow.sdk.util.PackageUtil  - Uploading PipelineOptions.filesToStage complete: 2 files newly uploaded, 77 files cached
Dataflow SDK version: 0.3.150326
57718 [main] INFO  com.google.cloud.dataflow.sdk.runners.DataflowPipelineRunner  - To access the Dataflow monitoring console, please navigate to https://console.developers.google.com/project/gdfp-7414/dataflow/job/2015-04-02_21_26_53-11930390736602232537
Submitted job: 2015-04-02_21_26_53-11930390736602232537
2015-04-03T04:26:54.710Z: (3a5437c7f9c6e33f): Expanding GroupByKey operations into optimizable parts.
2015-04-03T04:26:54.714Z: (3a5437c7f9c6e8dd): Annotating graph with Autotuner information.
2015-04-03T04:26:55.436Z: (3a5437c7f9c6e85b): Fusing adjacent ParDo, Read, Write, and Flatten operations
2015-04-03T04:26:55.453Z: (3a5437c7f9c6efad): Fusing consumer denormalized-write-to-BQ into events-denormalize
2015-04-03T04:26:55.455Z: (3a5437c7f9c6e54b): Fusing consumer events-denormalize into events-read-from-BQ
2015-04-03T04:26:55.457Z: (3a5437c7f9c6eae9): Fusing consumer unmapped-write-to-BQ into events-denormalize
2015-04-03T04:26:55.504Z: (3a5437c7f9c6e67d): Adding StepResource setup and teardown to workflow graph.
2015-04-03T04:26:55.525Z: (971aceaf96c03b86): Starting the input generators.
2015-04-03T04:26:55.546Z: (ea598353613cc1d3): Adding workflow start and stop steps.
2015-04-03T04:26:55.548Z: (ea598353613ccd39): Assigning stage ids.
2015-04-03T04:26:56.017Z: S07: (fb31ac3e5c3be05a): Executing operation WeightingFactor
2015-04-03T04:26:56.024Z: S09: (ee7049b2bfe3f48c): Executing operation Name_Community
2015-04-03T04:26:56.037Z: (3a5437c7f9c6e293): Starting worker pool setup.
2015-04-03T04:26:56.042Z: (3a5437c7f9c6edcf): Starting 5 workers...
2015-04-03T04:26:56.047Z: S01: (a25730bd9d25e5ed): Executing operation Browser_mapping
2015-04-03T04:26:56.049Z: S11: (fb31ac3e5c3beb06): Executing operation WebsiteVHH
2015-04-03T04:26:56.051Z: (30eb1307dfc8372f): Value "Name_Community.out" materialized.
2015-04-03T04:26:56.065Z: (52e655ceeab44257): Value "WeightingFactor.out" materialized.
2015-04-03T04:26:56.072Z: S03: (c024e27994951718): Executing operation OS_mapping
2015-04-03T04:26:56.076Z: S10: (a3947955b25f3830): Executing operation AsIterable3/CreatePCollectionView
2015-04-03T04:26:56.087Z: (4c9eb5a54721c4f7): Value "WebsiteVHH.out" materialized.
2015-04-03T04:26:56.094Z: S05: (52e655ceeab4458a): Executing operation SA1_Area_Metro
2015-04-03T04:26:56.103Z: S08: (c024e279949513f4): Executing operation AsIterable2/CreatePCollectionView
2015-04-03T04:26:56.106Z: (4c9eb5a54721cd78): Value "AsIterable3/CreatePCollectionView.out" materialized.
2015-04-03T04:26:56.107Z: (58b58f637f29b69a): Value "OS_mapping.out" materialized.
2015-04-03T04:26:56.115Z: (f0587ec8b1f9f69f): Value "Browser_mapping.out" materialized.
2015-04-03T04:26:56.126Z: (a277f34c719a133): Value "SA1_Area_Metro.out" materialized.
2015-04-03T04:26:56.127Z: S12: (c024e279949510d0): Executing operation AsIterable4/CreatePCollectionView
2015-04-03T04:26:56.132Z: S04: (52e655ceeab44adf): Executing operation AsIterable6/CreatePCollectionView
2015-04-03T04:26:56.136Z: (f0587ec8b1f9fd86): Value "AsIterable2/CreatePCollectionView.out" materialized.
2015-04-03T04:26:56.141Z: S02: (eb97fca639a2101b): Executing operation AsIterable5/CreatePCollectionView
2015-04-03T04:26:56.151Z: S06: (8cc6100045f0af9b): Executing operation AsIterable/CreatePCollectionView
2015-04-03T04:26:56.159Z: (6da6e59d099e8c60): Value "AsIterable4/CreatePCollectionView.out" materialized.
2015-04-03T04:26:56.163Z: (4c9eb5a54721c5f9): Value "AsIterable6/CreatePCollectionView.out" materialized.
2015-04-03T04:26:56.173Z: (a3947955b25f3701): Value "AsIterable5/CreatePCollectionView.out" materialized.
2015-04-03T04:26:56.178Z: (58b58f637f29b853): Value "AsIterable/CreatePCollectionView.out" materialized.
2015-04-03T04:26:56.199Z: S13: (8cc6100045f0ac67): Executing operation events-read-from-BQ+events-denormalize+denormalized-write-to-BQ+unmapped-write-to-BQ
2015-04-03T04:26:56.653Z: (6153d4cd276be2a0): Autoscaling: Enabled for job /workflows/wf-2015-04-02_21_26_53-11930390736602232537
2015-04-03T04:30:31.754Z: (a94b4f451005c934): Autoscaling: Resizing worker pool from 5 to 10.
2015-04-03T04:31:01.754Z: (a94b4f451005c38e): Autoscaling: Resizing worker pool from 10 to 8.
2015-04-03T04:31:02.363Z: (d0f58ccaf368cf1f): Workflow failed. Causes: (539037ea87656484): Cannot downsize without losing active shuffle data. old_size = 10, new_size = 8.
2015-04-03T04:31:02.396Z: (7f503ea3d5c37a55): Stopping the input generators.
2015-04-03T04:31:02.411Z: (58b58f637f29ba9f): Cleaning up.
2015-04-03T04:31:02.442Z: (58b58f637f29bc58): Tearing down pending resources...
2015-04-03T04:31:02.447Z: (58b58f637f29be11): Starting worker pool teardown.
2015-04-03T04:31:02.453Z: (58b58f637f29b05d): Stopping worker pool...
2015-04-03T04:31:03.062Z: (a1f260e16fea5b6): Workflow failed. Causes: (539037ea87656484): Cannot downsize without losing active shuffle data. old_size = 10, new_size = 8.
458752 [main] INFO  com.google.cloud.dataflow.sdk.runners.BlockingDataflowPipelineRunner  - Job finished with status FAILED
458755 [main] INFO  com.<removed>.cdf.job.AbstractCloudDataFlowJob  - com.google.cloud.dataflow.sdk.runners.BlockingDataflowPipelineRunner$PipelineJobState@27a7ef08
458755 [main] INFO  com.<removed>.cdf.job.AbstractCloudDataFlowJob  - Cleaning up after <removed> job. At the moment nothing to do.
Disconnected from the target VM, address: '127.0.0.1:57739', transport: 'socket'
1087[main]INFO com.google.cloud.dataflow.sdk.runners.DataflowPipelineRunner-在数据流服务上执行管道,这将对谷歌计算引擎使用和其他谷歌云服务产生计费影响。
1103[main]INFO com.google.cloud.dataflow.sdk.util.PackageUtil-将79个文件从PipelineOptions.filesToStage上载到GCS,以准备在云中执行。
43086[main]INFO com.google.cloud.dataflow.sdk.util.PackageUtil-上传PipelineOptions.filesToStage完成:新上传2个文件,缓存77个文件
数据流SDK版本:0.3.150326
57718[main]INFO com.google.cloud.dataflow.sdk.runners.DataflowPipelineRunner-要访问数据流监控控制台,请导航到https://console.developers.google.com/project/gdfp-7414/dataflow/job/2015-04-02_21_26_53-11930390736602232537
提交作业:2015-04-02_21_26_53-11930390736602232537
2015-04-03T04:26:54.710Z:(3a5437c7f9c6e33f):将GroupByKey操作扩展为可优化的部件。
2015-04-03T04:26:54.714Z:(3a5437c7f9c6e8dd):用自动调谐信息注释图形。
2015-04-03T04:26:55.436Z:(3a5437c7f9c6e85b):融合相邻ParDo、读取、写入和展平操作
2015-04-03T04:26:55.453Z:(3a5437c7f9c6efad):将消费者对BQ的非规范化写入融合为事件非规范化
2015-04-03T04:26:55.455Z:(3a5437c7f9c6e54b):将消费者事件非规范化融合为从BQ读取的事件
2015-04-03T04:26:55.457Z:(3a5437c7f9c6eae9):将消费者未映射到BQ的写入融合到事件非规范化
2015-04-03T04:26:55.504Z:(3a5437c7f9c6e67d):将步骤资源设置和拆卸添加到工作流图中。
2015-04-03T04:26:55.525Z:(971aceaf96c03b86):启动输入发电机。
2015-04-03T04:26:55.546Z:(ea598353613cc1d3):添加工作流开始和停止步骤。
2015-04-03T04:26:55.548Z:(ea598353613ccd39):分配阶段ID。
2015-04-03T04:26:56.017Z:S07:(fb31ac3e5c3be05a):执行操作权重因子
2015-04-03T04:26:56.024Z:S09:(ee7049b2bfe3f48c):正在执行的操作名称
2015-04-03T04:26:56.037Z:(3a5437c7f9c6e293):启动工作池设置。
2015-04-03T04:26:56.042Z:(3a5437c7f9c6edcf):开始5名工人。。。
2015-04-03T04:26:56.047Z:S01:(a25730bd9d25e5ed):执行操作浏览器映射
2015-04-03T04:26:56.049Z:S11:(fb31ac3e5c3beb06):执行操作网站VH
2015-04-03T04:26:56.051Z:(30eb1307dfc8372f):值“Name_Community.out”具体化。
2015-04-03T04:26:56.065Z:(52e655ceeab44257):值“权重因子.输出”具体化。
2015-04-03T04:26:56.072Z:S03:(c024e27994951718):执行操作OS_映射
2015-04-03T04:26:56.076Z:S10:(a3947955b25f3830):执行操作AsiteTable3/CreatePCollectionView
2015-04-03T04:26:56.087Z:(4c9eb5a54721c4f7):值“websiteVH.out”具体化。
2015-04-03T04:26:56.094Z:S05:(52e655ceeab4458a):执行SA1_地区地铁运营
2015-04-03T04:26:56.103Z:S08:(c024e279949513f4):执行操作AsiteTable2/CreatePCollectionView
2015-04-03T04:26:56.106Z:(4c9eb5a54721cd78):值“AsIterable3/CreatePCollectionView.out”具体化。
2015-04-03T04:26:56.107Z:(58b58f637f29b69a):值“OS_mapping.out”具体化。
2015-04-03T04:26:56.115Z:(f0587ec8b1f9f69f):值“Browser_mapping.out”具体化。
2015-04-03T04:26:56.126Z:(a277f34c719a133):值“SA1_区域_Metro.out”具体化。
2015-04-03T04:26:56.127Z:S12:(c024e279949510d0):执行操作AsiteLabel4/CreatePCollectionView
2015-04-03T04:26:56.132Z:S04:(52e655ceeab44adf):执行操作AsIterable6/CreatePCollectionView
2015-04-03T04:26:56.136Z:(f0587ec8b1f9fd86):值“AsIterable2/CreatePCollectionView.out”具体化。
2015-04-03T04:26:56.141Z:S02:(eb97fca639a2101b):执行操作AsiteTable5/CreatePCollectionView
2015-04-03T04:26:56.151Z:S06:(8cc6100045f0af9b):执行操作AsIterable/CreatePCollectionView
2015-04-03T04:26:56.159Z:(6da6e59d099e8c60):值“AsIterable4/CreatePCollectionView.out”具体化。
2015-04-03T04:26:56.163Z:(4c9eb5a54721c5f9):值“AsIterable6/CreatePCollectionView.out”具体化。
2015-04-03T04:26:56.173Z:(a3947955b25f3701):值“AsIterable5/CreatePCollectionView.out”具体化。
2015-04-03T04:26:56.178Z:(58b58f637f29b853):值“AsIterable/CreatePCollectionView.out”具体化。
2015-04-03T04:26:56.199Z:S13:(8cc6100045f0ac67):执行从BQ读取的操作事件+事件反规范化+反规范化写入BQ+未映射写入BQ
2015-04-03T04:26:56.653Z:(6153D34CD276BE2A0):自动缩放:为作业/工作流启用/wf-2015-04-02_21_26_53-11930390736602232537
2015-04-03T04:30:31.754Z:(A94B4F45105C934):自动缩放:将工作池大小从5调整为10。
2015-04-03T04:31:01.754Z:(A94B4F45105C38E):自动缩放:将工作池大小从10调整为8。
2015-04-03T04:31:02.363Z:(d0f58ccaf368cf1f):工作流失败。原因:(539037ea87656484):无法在不丢失活动随机数据的情况下缩小大小。旧尺寸=10,新尺寸=8。
2015-04-03T04:31:02.396Z:(7f503ea3d5c37a55):停止输入发生器。
2015-04-03T04:31:02.411Z:(58b58f637f29ba9f):清理。
2015-04-03T04:31:02.442Z:(58b58f637f29bc58):拆除未决资源。。。
2015-04-03T04:31:02.447Z:(58b58f637f29be11):开始工人池拆卸。
2015-04-03T04:31:02.453Z:(58b58f637f29b05d):停止w