Fatal编程技术网
  • python/
  • Python 无法再在requirements.txt文件中为谷歌云数据流作业安装“谷歌云数据存储”依赖项

Python 无法再在requirements.txt文件中为谷歌云数据流作业安装“谷歌云数据存储”依赖项

python google-cloud-platform google-cloud-dataflow

Python 无法再在requirements.txt文件中为谷歌云数据流作业安装“谷歌云数据存储”依赖项,python,google-cloud-platform,google-cloud-datastore,python-import,google-cloud-dataflow,Python,Google Cloud Platform,Google Cloud Datastore,Python Import,Google Cloud Dataflow,类似于上一篇文章的事情再次发生： 实际上，一夜之间，我们所有的云数据流工作似乎都无缘无故地中断了。没有新的部署，没有SDK更新，没有什么，我们的团队只是在StackDriver通知中醒来，说我们的映射作业一夜之间失败了 这是作业失败的堆栈跟踪 Traceback (most recent call last): File "/usr/local/lib/python2.7/dist-packages/dataflow_worker/batchworker.py", line 609, i

类似于上一篇文章的事情再次发生：

实际上，一夜之间，我们所有的云数据流工作似乎都无缘无故地中断了。没有新的部署，没有SDK更新，没有什么，我们的团队只是在StackDriver通知中醒来，说我们的映射作业一夜之间失败了

这是作业失败的堆栈跟踪

Traceback (most recent call last):
    File "/usr/local/lib/python2.7/dist-packages/dataflow_worker/batchworker.py", line 609, in do_work work_executor.execute()
    File "/usr/local/lib/python2.7/dist-packages/dataflow_worker/executor.py", line 167, in execute op.start()
    File "apache_beam/runners/worker/operations.py", line 340, in apache_beam.runners.worker.operations.DoOperation.start def start(self):
    File "apache_beam/runners/worker/operations.py", line 341, in apache_beam.runners.worker.operations.DoOperation.start with self.scoped_start_state:
    File "apache_beam/runners/worker/operations.py", line 346, in apache_beam.runners.worker.operations.DoOperation.start pickler.loads(self.spec.serialized_fn))
    File "/usr/local/lib/python2.7/dist-packages/apache_beam/internal/pickler.py", line 225, in loads return dill.loads(s)
    File "/usr/local/lib/python2.7/dist-packages/dill/dill.py", line 277, in loads return load(file)
    File "/usr/local/lib/python2.7/dist-packages/dill/dill.py", line 266, in load obj = pik.load()
    File "/usr/lib/python2.7/pickle.py", line 864, in load dispatch[key](self)
    File "/usr/lib/python2.7/pickle.py", line 1096, in load_global klass = self.find_class(module, name)
    File "/usr/local/lib/python2.7/dist-packages/dill/dill.py", line 423, in find_class return StockUnpickler.find_class(self, module, name)
    File "/usr/lib/python2.7/pickle.py", line 1130, in find_class __import__(module)
    File "/usr/local/lib/python2.7/dist-packages/dataflow_pipeline/invoice_overages.py", line 26,
in <module> from google.cloud.datastore.helpers import entity_from_protobuf ImportError: No module named datastore.helpers
我目前基本上无法在本地复制。我安装这些要求

httplib2==0.9.1
oauth2client==3.0.0
google-cloud-dataflow==2.5.0
我得到了上面显示的错误

$ python main.py
Traceback (most recent call last):
  File "main.py", line 25, in <module>
    import dataflow_pipeline.summarize_intervals as summarization_pipeline
  File "/Users/john/camio-mappers/box-counters-pipeline/dataflow_pipeline/summarize_intervals.py", line 31, in <module>
    from google.cloud.datastore.helpers import entity_from_protobuf
ImportError: No module named datastore.helpers
(venv)
我不知道这意味着什么。我确实似乎丢失了列出的
.proto
文件，但为什么会丢失它们呢

我的主要问题是，这是上次发生的情况：当我们不进行任何新的部署时，为什么作业可以中断？假设是，如果我们不更改任何代码，代码就不应该被破坏。如果我们不改变任何依赖关系，我们就不应该以管道破裂而告终，然后我们就不得不争先恐后地进行修复。云数据流已经过了测试版，应该是稳定的，如果它是在测试版，并且受到API更改的影响，这是意料之中的，但在这一点上应该是稳定的。到目前为止，我们的管道已经关闭，我们不知道需要多长时间

我对
googleclouddatastore==1.4.0
做了建议的更改，它部分修复了我的管道，但现在它在另一点上失败了。现在看来，
googlecloudpubsub
正在失败，出现了一个类似的错误，即缺少
.proto
文件

ERROR: (gcloud.app.deploy) Error Response: [9]
Application startup error:
    import dataflow_pipeline.tally_overages as overaging_pipeline
  File "/home/vmagent/app/dataflow_pipeline/tally_overages.py", line 29, in <module>
    from google.cloud import pubsub
  File "/env/local/lib/python2.7/site-packages/google/cloud/pubsub.py", line 17, in <module>
    from google.cloud.pubsub_v1 import PublisherClient
  File "/env/local/lib/python2.7/site-packages/google/cloud/pubsub_v1/__init__.py", line 17, in <module>
    from google.cloud.pubsub_v1 import types
  File "/env/local/lib/python2.7/site-packages/google/cloud/pubsub_v1/types.py", line 30, in <module>
    from google.cloud.pubsub_v1.proto import pubsub_pb2
  File "/env/local/lib/python2.7/site-packages/google/cloud/pubsub_v1/proto/pubsub_pb2.py", line 29, in <module>
    dependencies=[google_dot_api_dot_annotations__pb2.DESCRIPTOR,google_dot_protobuf_dot_duration__pb2.DESCRIPTOR,google_dot_protobuf_dot_empty__pb2.DESCRIPTOR,google_dot_protobuf_dot_field__mask__pb2.DESCRIPTOR,google_dot_protobuf_dot_timestamp__pb2.DESCRIPTOR,])
  File "/env/local/lib/python2.7/site-packages/google/protobuf/descriptor.py", line 878, in __new__
    return _message.default_pool.AddSerializedFile(serialized_pb)
TypeError: Couldn't build proto file into descriptor pool!
Invalid proto descriptor for file "google/cloud/pubsub_v1/proto/pubsub.proto":
  google.pubsub.v1.Topic.name: "google.pubsub.v1.Topic.name" is already defined in file "google/cloud/proto/pubsub/v1/pubsub.proto".
  google.pubsub.v1.Topic: "google.pubsub.v1.Topic" is already defined in file "google/cloud/proto/pubsub/v1/pubsub.proto".
  google.pubsub.v1.PubsubMessage.data: "google.pubsub.v1.PubsubMessage.data" is already defined in file "google/cloud/proto/pubsub/v1/pubsub.proto".
  google.pubsub.v1.PubsubMessage.attributes: "google.pubsub.v1.PubsubMessage.attributes" is already defined in file "google/cloud/proto/pubsub/v1/pubsub.proto".
  google.pubsub.v1.PubsubMessage.message_id: "google.pubsub.v1.PubsubMessage.message_id" is already defined in file "google/cloud/proto/pubsub/v1/pubsub.proto".
  google.pubsub.v1.PubsubMessage.publish_time: "google.pubsub.v1.PubsubMessage.publish_time" is already defined in file "google/cloud/proto/pubsub/v1/pubsub.proto".
  google.pubsub.v1.PubsubMessage.AttributesEntry.key: "google.pubsub.v1.PubsubMessage.AttributesEntry.key" is already defined in file "google/cloud/proto/pubsub/v1/pubsub.proto".
  google.pubsub.v1.PubsubMessage.AttributesEntry.value: "google.pubsub.v1.PubsubMessage.AttributesEntry.value" is already defined in file "google/cloud/proto/pubsub/v1/pubsub.proto".
  google.pubsub.v1.PubsubMessage.AttributesEntry: "google.pubsub.v1.PubsubMessage.AttributesEntry" is already defined in file "google/cloud/proto/pubsub/v1/pubsub.proto".
  google.pubsub.v1.PubsubMessage: "google.pubsub.v1.PubsubMessage" is already defined in file "google/cloud/proto/pubsub/v1/pubsub.proto".
  google.pubsub.v1.GetTopicRequest.topic: "google.pubsub.v1.GetTopicRequest.topic" is already defined in file "google/cloud/proto/pubsub/v1/pubsub.proto".
  google.pubsub.v1.GetTopicRequest: "google.pubsub.v1.GetTopicRequest" is already defined in file "google/cloud/proto/pubsub/v1/pubsub.proto".
  google.pubsub.v1.PublishRequest.topic: "google.pubsub.v1.PublishRequest.topic" is already defined in file "google/cloud/proto/pubsub/v1/pubsub.proto".
  google.pubsub.v1.PublishRequest.messages: "google.pubsub.v1.PublishRequest.messages" is already defined in file "google/cloud/proto/pubsub/v1/pubsub.proto".
  google.pubsub.v1.PublishRequest: "google.pubsub.v1.PublishRequest" is already defined in file "google/cloud/proto/pubsub/v1/pubsub.proto".
  google.pubsub.v1.PublishResponse.message_ids: "google.pubsub.v1.PublishResponse.message_ids" is already defined in file "google/cloud/proto/pubsub/v1/pubsub.proto".
  google.pubsub.v1.PublishResponse: "google.pubsub.v1.PublishResponse" is already defined in file "google/cloud/proto/pubsub/v1/pubsub.proto".
  google.pubsub.v1.ListTopicsRequest.project: "google.pubsub.v1.ListTopicsRequest.project" is already defined in file "google/cloud/proto/pubsub/v1/pubsub.proto".
  google.pubsub.v1.ListTopicsRequest.page_size: "google.pubsub.v1.ListTopicsRequest.page_size" is already defined in file "google/cloud/proto/pubsub/v1/pubsub.proto".
  google.pubsub.v1.ListTopicsRequest.page_token: "google.pubsub.v1.ListTopicsRequest.page_token" is already defined in file "google/cloud/proto/pubsub/v1/pubsub.proto".
  google.pubsub.v1.ListTopicsRequest: "google.pubsub.v1.ListTopicsRequest" is already defined in file "google/cloud/proto/pubsub/v1/pubsub.proto".
  google.pubsub.v1.ListTopicsResponse.topics: "google.pubsub.v1.ListTopicsResponse.topics" is already defined in file "google/cloud/proto/pubsub/v1/pubsub.proto".
  google.pubsub.v1.ListTopicsResponse.next_page_token: "google.pubsub.v1.ListTopicsResponse.next_page_token" is already defined in file "google/cloud/proto/pubsub/v1/pubsub.proto".
  google.pubsub.v1.ListTopicsResponse: "google.pubsub.v1.ListTopicsResponse" is already defined in file "google/cloud/proto/pubsub/v1/pubsub.proto".
  google.pubsub.v1.ListTopicSubscriptionsRequest.topic: "google.pubsub.v1.ListTopicSubscriptionsRequest.topic" is already defined in file "google/cloud/proto/pubsub/v1/pubsub.proto".
  google.pubsub.v1.ListTopicSubscriptionsRequest.page_size: "google.pubsub.v1.ListTopicSubscriptionsRequest.page_size" is already defined in file "google/cloud/proto/pubsub/v1/pubsub.proto".
  google.pubsub.v1.ListTopicSubscriptionsRequest.page_token: "google.pubsub.v1.ListTopicSubscriptionsRequest.page_token" is already defined in file "google/cloud/proto/pubsub/v1/pubsub.proto".
  google.pubsub.v1.ListTopicSubscriptionsRequest: "google.pubsub.v1.ListTopicSubscriptionsRequest" is already defined in file "google/cloud/proto/pubsub/v1/pubsub.proto".
  google.pubsub.v1.ListTopicSubscriptionsResponse.subscriptions: "google.pubsub.v1.ListTopicSubscriptionsResponse.subscriptions" is already defined in file "google/cloud/proto/pubsub/v1/pubsub.proto".
  google.pubsub.v1.ListTopicSubscriptionsResponse.next_page_token: "google.pubsub.v1.ListTopicSubscriptionsResponse.next_page_token" is already defined in file "google/cloud/proto/pubsub/v1/pubsub.proto".
  google.pubsub.v1.ListTopicSubscriptionsResponse: "google.pubsub.v1.ListTopicSubscriptionsResponse" is already defined in file "google/cloud/proto/pubsub/v1/pubsub.proto".
  google.pubsub.v1.DeleteTopicRequest.topic: "google.pubsub.v1.DeleteTopicRequest.topic" is already defined in file "google/cloud/proto/pubsub/v1/pubsub.proto".
  google.pubsub.v1.DeleteTopicRequest: "google.pubsub.v1.DeleteTopicRequest" is already defined in file "google/cloud/proto/pubsub/v1/pubsub.proto".
  google.pubsub.v1.Subscription.name: "google.pubsub.v1.Subscription.name" is already defined in file "google/cloud/proto/pubsub/v1/pubsub.proto".
  google.pubsub.v1.Subscription.topic: "google.pubsub.v1.Subscription.topic" is already defined in file "google/cloud/proto/pubsub/v1/pubsub.proto".
  google.pubsub.v1.Subscription.push_config: "google.pubsub.v1.Subscription.push_config" is already defined in file "google/cloud/proto/pubsub/v1/pubsub.proto".
  google.pubsub.v1.Subscription.ack_deadline_seconds: "google.pubsub.v1.Subscription.ack_deadline_seconds" is already defined in file "google/cloud/proto/pubsub/v1/pubsub.proto".
  google.pubsub.v1.Subscription.retain_acked_messages: "google.pubsub.v1.Subscription.retain_acked_messages" is already defined in 
  google.pubsub.v1.Subscriber.Seek: "google.pubsub.v1.SeekRequest" seems to be defined in "google/cloud/proto/pubsub/v1/pubsub.proto", which is not imported by "google/cloud/pubsub_v1/proto/pubsub.proto".  To use it here, please add the necessary import.
  google.pubsub.v1.Subscriber.Seek: "google.pubsub.v1.SeekResponse" seems to be defined in "google/cloud/proto/pubsub/v1/pubsub.proto", which is not imported by "google/cloud/pubsub_v1/proto/pubsub.proto".  To use it here, please add the necessary import.
  google.pubsub.v1.Publisher.CreateTopic: "google.pubsub.v1.Topic" seems to be defined in "google/cloud/proto/pubsub/v1/pubsub.proto", which is not imported by "google/cloud/pubsub_v1/proto/pubsub.proto".  To use it here, please add the necessary import.
  google.pubsub.v1.Publisher.CreateTopic: "google.pubsub.v1.Topic" seems to be defined in "google/cloud/proto/pubsub/v1/pubsub.proto", which is not imported by "google/cloud/pubsub_v1/proto/pubsub.proto".  To use it here, please add the necessary import.
  google.pubsub.v1.Publisher.UpdateTopic: "google.pubsub.v1.Topic" seems to be defined in "google/cloud/proto/pubsub/v1/pubsub.proto", which is not imported by "google/cloud/pubsub_v1/proto/pubsub.proto".  To use it here, please add the necessary import.
  google.pubsub.v1.Publisher.Publish: "google.pubsub.v1.PublishRequest" seems to be defined in "google/cloud/proto/pubsub/v1/pubsub.proto", which is not imported by "google/cloud/pubsub_v1/proto/pubsub.proto".  To use it here, please add the necessary import.
  google.pubsub.v1.Publisher.Publish: "google.pubsub.v1.PublishResponse" seems to be defined in "google/cloud/proto/pubsub/v1/pubsub.proto", which is not imported by "google/cloud/pubsub_v1/proto/pubsub.proto".  To use it here, please add the necessary import.
  google.pubsub.v1.Publisher.GetTopic: "google.pubsub.v1.GetTopicRequest" seems to be defined in "google/cloud/proto/pubsub/v1/pubsub.proto", which is not imported by "google/cloud/pubsub_v1/proto/pubsub.proto".  To use it here, please add the necessary import.
  google.pubsub.v1.Publisher.GetTopic: "google.pubsub.v1.Topic" seems to be defined in "google/cloud/proto/pubsub/v1/pubsub.proto", which is not imported by "google/cloud/pubsub_v1/proto/pubsub.proto".  To use it here, please add the necessary import.
  google.pubsub.v1.Publisher.ListTopics: "google.pubsub.v1.ListTopicsRequest" seems to be defined in "google/cloud/proto/pubsub/v1/pubsub.proto", which is not imported by "google/cloud/pubsub_v1/proto/pubsub.proto".  To use it here, please add the necessary import.
  google.pubsub.v1.Publisher.ListTopics: "google.pubsub.v1.ListTopicsResponse" seems to be defined in "google/cloud/proto/pubsub/v1/pubsub.proto", which is not imported by "google/cloud/pubsub_v1/proto/pubsub.proto".  To use it here, please add the necessary import.
  google.pubsub.v1.Publisher.ListTopicSubscriptions: "google.pubsub.v1.ListTopicSubscriptionsRequest" seems to be defined in "google/cloud/proto/pubsub/v1/pubsub.proto", which is not imported by "google/cloud/pubsub_v1/proto/pubsub.proto".  To use it here, please add the necessary import.
  google.pubsub.v1.Publisher.ListTopicSubscriptions: "google.pubsub.v1.ListTopicSubscriptionsResponse" seems to be defined in "google/cloud/proto/pubsub/v1/pubsub.proto", which is not imported by "google/cloud/pubsub_v1/proto/pubsub.proto".  To use it here, please add the necessary import.
  google.pubsub.v1.Publisher.DeleteTopic: "google.pubsub.v1.DeleteTopicRequest" seems to be defined in "google/cloud/proto/pubsub/v1/pubsub.proto", which is not imported by "google/cloud/pubsub_v1/proto/pubsub.proto".  To use it here, please add the necessary import.

import re
import os
import math
import datetime
import json
import logging
import base64
import traceback
import itertools
import pytz
import requests
from functools import reduce

# google cloud datastore imports
from google.cloud.proto.datastore.v1 import query_pb2
from googledatastore import helper as datastore_helper, PropertyFilter, CompositeFilter
from google.cloud.datastore.helpers import entity_from_protobuf
编辑-以下是代码的导入行

文件1

from __future__ import absolute_import

# standard imports
import datetime
import json
import logging
import base64
import collections
import traceback
import hashlib
from functools import reduce

# apache beam / dataflow imports
import apache_beam as beam
from apache_beam.io.gcp.datastore.v1.datastoreio import ReadFromDatastore
from apache_beam.options.pipeline_options import PipelineOptions

# google cloud datastore imports
from google.cloud.proto.datastore.v1 import query_pb2
from googledatastore import helper as datastore_helper, PropertyFilter, CompositeFilter
from google.cloud.datastore.helpers import entity_from_protobuf

# custom utility imports
from .util import *
文件2

from __future__ import absolute_import

import datetime
import math
import json
import logging
import traceback
import collections
import hashlib
from functools import reduce

import apache_beam as beam
from apache_beam.io.gcp.datastore.v1.datastoreio import ReadFromDatastore
from apache_beam.options.pipeline_options import PipelineOptions

from google.cloud.proto.datastore.v1 import query_pb2
from googledatastore import helper as datastore_helper, PropertyFilter, CompositeFilter
from google.cloud.datastore.helpers import entity_from_protobuf
from google.cloud import pubsub

from .util import *
文件3

from __future__ import absolute_import

import math
import json
import logging
import traceback
import uuid
import collections
import hashlib
import datetime
from functools import reduce

import apache_beam as beam
from apache_beam.io.gcp.datastore.v1.datastoreio import ReadFromDatastore
from apache_beam.options.pipeline_options import PipelineOptions

from google.cloud.proto.datastore.v1 import query_pb2
from googledatastore import helper as datastore_helper, PropertyFilter, CompositeFilter
from google.cloud.datastore.helpers import entity_from_protobuf
from google.cloud import pubsub

from .util import *
并从
util.*
文件

ERROR: (gcloud.app.deploy) Error Response: [9]
Application startup error:
    import dataflow_pipeline.tally_overages as overaging_pipeline
  File "/home/vmagent/app/dataflow_pipeline/tally_overages.py", line 29, in <module>
    from google.cloud import pubsub
  File "/env/local/lib/python2.7/site-packages/google/cloud/pubsub.py", line 17, in <module>
    from google.cloud.pubsub_v1 import PublisherClient
  File "/env/local/lib/python2.7/site-packages/google/cloud/pubsub_v1/__init__.py", line 17, in <module>
    from google.cloud.pubsub_v1 import types
  File "/env/local/lib/python2.7/site-packages/google/cloud/pubsub_v1/types.py", line 30, in <module>
    from google.cloud.pubsub_v1.proto import pubsub_pb2
  File "/env/local/lib/python2.7/site-packages/google/cloud/pubsub_v1/proto/pubsub_pb2.py", line 29, in <module>
    dependencies=[google_dot_api_dot_annotations__pb2.DESCRIPTOR,google_dot_protobuf_dot_duration__pb2.DESCRIPTOR,google_dot_protobuf_dot_empty__pb2.DESCRIPTOR,google_dot_protobuf_dot_field__mask__pb2.DESCRIPTOR,google_dot_protobuf_dot_timestamp__pb2.DESCRIPTOR,])
  File "/env/local/lib/python2.7/site-packages/google/protobuf/descriptor.py", line 878, in __new__
    return _message.default_pool.AddSerializedFile(serialized_pb)
TypeError: Couldn't build proto file into descriptor pool!
Invalid proto descriptor for file "google/cloud/pubsub_v1/proto/pubsub.proto":
  google.pubsub.v1.Topic.name: "google.pubsub.v1.Topic.name" is already defined in file "google/cloud/proto/pubsub/v1/pubsub.proto".
  google.pubsub.v1.Topic: "google.pubsub.v1.Topic" is already defined in file "google/cloud/proto/pubsub/v1/pubsub.proto".
  google.pubsub.v1.PubsubMessage.data: "google.pubsub.v1.PubsubMessage.data" is already defined in file "google/cloud/proto/pubsub/v1/pubsub.proto".
  google.pubsub.v1.PubsubMessage.attributes: "google.pubsub.v1.PubsubMessage.attributes" is already defined in file "google/cloud/proto/pubsub/v1/pubsub.proto".
  google.pubsub.v1.PubsubMessage.message_id: "google.pubsub.v1.PubsubMessage.message_id" is already defined in file "google/cloud/proto/pubsub/v1/pubsub.proto".
  google.pubsub.v1.PubsubMessage.publish_time: "google.pubsub.v1.PubsubMessage.publish_time" is already defined in file "google/cloud/proto/pubsub/v1/pubsub.proto".
  google.pubsub.v1.PubsubMessage.AttributesEntry.key: "google.pubsub.v1.PubsubMessage.AttributesEntry.key" is already defined in file "google/cloud/proto/pubsub/v1/pubsub.proto".
  google.pubsub.v1.PubsubMessage.AttributesEntry.value: "google.pubsub.v1.PubsubMessage.AttributesEntry.value" is already defined in file "google/cloud/proto/pubsub/v1/pubsub.proto".
  google.pubsub.v1.PubsubMessage.AttributesEntry: "google.pubsub.v1.PubsubMessage.AttributesEntry" is already defined in file "google/cloud/proto/pubsub/v1/pubsub.proto".
  google.pubsub.v1.PubsubMessage: "google.pubsub.v1.PubsubMessage" is already defined in file "google/cloud/proto/pubsub/v1/pubsub.proto".
  google.pubsub.v1.GetTopicRequest.topic: "google.pubsub.v1.GetTopicRequest.topic" is already defined in file "google/cloud/proto/pubsub/v1/pubsub.proto".
  google.pubsub.v1.GetTopicRequest: "google.pubsub.v1.GetTopicRequest" is already defined in file "google/cloud/proto/pubsub/v1/pubsub.proto".
  google.pubsub.v1.PublishRequest.topic: "google.pubsub.v1.PublishRequest.topic" is already defined in file "google/cloud/proto/pubsub/v1/pubsub.proto".
  google.pubsub.v1.PublishRequest.messages: "google.pubsub.v1.PublishRequest.messages" is already defined in file "google/cloud/proto/pubsub/v1/pubsub.proto".
  google.pubsub.v1.PublishRequest: "google.pubsub.v1.PublishRequest" is already defined in file "google/cloud/proto/pubsub/v1/pubsub.proto".
  google.pubsub.v1.PublishResponse.message_ids: "google.pubsub.v1.PublishResponse.message_ids" is already defined in file "google/cloud/proto/pubsub/v1/pubsub.proto".
  google.pubsub.v1.PublishResponse: "google.pubsub.v1.PublishResponse" is already defined in file "google/cloud/proto/pubsub/v1/pubsub.proto".
  google.pubsub.v1.ListTopicsRequest.project: "google.pubsub.v1.ListTopicsRequest.project" is already defined in file "google/cloud/proto/pubsub/v1/pubsub.proto".
  google.pubsub.v1.ListTopicsRequest.page_size: "google.pubsub.v1.ListTopicsRequest.page_size" is already defined in file "google/cloud/proto/pubsub/v1/pubsub.proto".
  google.pubsub.v1.ListTopicsRequest.page_token: "google.pubsub.v1.ListTopicsRequest.page_token" is already defined in file "google/cloud/proto/pubsub/v1/pubsub.proto".
  google.pubsub.v1.ListTopicsRequest: "google.pubsub.v1.ListTopicsRequest" is already defined in file "google/cloud/proto/pubsub/v1/pubsub.proto".
  google.pubsub.v1.ListTopicsResponse.topics: "google.pubsub.v1.ListTopicsResponse.topics" is already defined in file "google/cloud/proto/pubsub/v1/pubsub.proto".
  google.pubsub.v1.ListTopicsResponse.next_page_token: "google.pubsub.v1.ListTopicsResponse.next_page_token" is already defined in file "google/cloud/proto/pubsub/v1/pubsub.proto".
  google.pubsub.v1.ListTopicsResponse: "google.pubsub.v1.ListTopicsResponse" is already defined in file "google/cloud/proto/pubsub/v1/pubsub.proto".
  google.pubsub.v1.ListTopicSubscriptionsRequest.topic: "google.pubsub.v1.ListTopicSubscriptionsRequest.topic" is already defined in file "google/cloud/proto/pubsub/v1/pubsub.proto".
  google.pubsub.v1.ListTopicSubscriptionsRequest.page_size: "google.pubsub.v1.ListTopicSubscriptionsRequest.page_size" is already defined in file "google/cloud/proto/pubsub/v1/pubsub.proto".
  google.pubsub.v1.ListTopicSubscriptionsRequest.page_token: "google.pubsub.v1.ListTopicSubscriptionsRequest.page_token" is already defined in file "google/cloud/proto/pubsub/v1/pubsub.proto".
  google.pubsub.v1.ListTopicSubscriptionsRequest: "google.pubsub.v1.ListTopicSubscriptionsRequest" is already defined in file "google/cloud/proto/pubsub/v1/pubsub.proto".
  google.pubsub.v1.ListTopicSubscriptionsResponse.subscriptions: "google.pubsub.v1.ListTopicSubscriptionsResponse.subscriptions" is already defined in file "google/cloud/proto/pubsub/v1/pubsub.proto".
  google.pubsub.v1.ListTopicSubscriptionsResponse.next_page_token: "google.pubsub.v1.ListTopicSubscriptionsResponse.next_page_token" is already defined in file "google/cloud/proto/pubsub/v1/pubsub.proto".
  google.pubsub.v1.ListTopicSubscriptionsResponse: "google.pubsub.v1.ListTopicSubscriptionsResponse" is already defined in file "google/cloud/proto/pubsub/v1/pubsub.proto".
  google.pubsub.v1.DeleteTopicRequest.topic: "google.pubsub.v1.DeleteTopicRequest.topic" is already defined in file "google/cloud/proto/pubsub/v1/pubsub.proto".
  google.pubsub.v1.DeleteTopicRequest: "google.pubsub.v1.DeleteTopicRequest" is already defined in file "google/cloud/proto/pubsub/v1/pubsub.proto".
  google.pubsub.v1.Subscription.name: "google.pubsub.v1.Subscription.name" is already defined in file "google/cloud/proto/pubsub/v1/pubsub.proto".
  google.pubsub.v1.Subscription.topic: "google.pubsub.v1.Subscription.topic" is already defined in file "google/cloud/proto/pubsub/v1/pubsub.proto".
  google.pubsub.v1.Subscription.push_config: "google.pubsub.v1.Subscription.push_config" is already defined in file "google/cloud/proto/pubsub/v1/pubsub.proto".
  google.pubsub.v1.Subscription.ack_deadline_seconds: "google.pubsub.v1.Subscription.ack_deadline_seconds" is already defined in file "google/cloud/proto/pubsub/v1/pubsub.proto".
  google.pubsub.v1.Subscription.retain_acked_messages: "google.pubsub.v1.Subscription.retain_acked_messages" is already defined in 
  google.pubsub.v1.Subscriber.Seek: "google.pubsub.v1.SeekRequest" seems to be defined in "google/cloud/proto/pubsub/v1/pubsub.proto", which is not imported by "google/cloud/pubsub_v1/proto/pubsub.proto".  To use it here, please add the necessary import.
  google.pubsub.v1.Subscriber.Seek: "google.pubsub.v1.SeekResponse" seems to be defined in "google/cloud/proto/pubsub/v1/pubsub.proto", which is not imported by "google/cloud/pubsub_v1/proto/pubsub.proto".  To use it here, please add the necessary import.
  google.pubsub.v1.Publisher.CreateTopic: "google.pubsub.v1.Topic" seems to be defined in "google/cloud/proto/pubsub/v1/pubsub.proto", which is not imported by "google/cloud/pubsub_v1/proto/pubsub.proto".  To use it here, please add the necessary import.
  google.pubsub.v1.Publisher.CreateTopic: "google.pubsub.v1.Topic" seems to be defined in "google/cloud/proto/pubsub/v1/pubsub.proto", which is not imported by "google/cloud/pubsub_v1/proto/pubsub.proto".  To use it here, please add the necessary import.
  google.pubsub.v1.Publisher.UpdateTopic: "google.pubsub.v1.Topic" seems to be defined in "google/cloud/proto/pubsub/v1/pubsub.proto", which is not imported by "google/cloud/pubsub_v1/proto/pubsub.proto".  To use it here, please add the necessary import.
  google.pubsub.v1.Publisher.Publish: "google.pubsub.v1.PublishRequest" seems to be defined in "google/cloud/proto/pubsub/v1/pubsub.proto", which is not imported by "google/cloud/pubsub_v1/proto/pubsub.proto".  To use it here, please add the necessary import.
  google.pubsub.v1.Publisher.Publish: "google.pubsub.v1.PublishResponse" seems to be defined in "google/cloud/proto/pubsub/v1/pubsub.proto", which is not imported by "google/cloud/pubsub_v1/proto/pubsub.proto".  To use it here, please add the necessary import.
  google.pubsub.v1.Publisher.GetTopic: "google.pubsub.v1.GetTopicRequest" seems to be defined in "google/cloud/proto/pubsub/v1/pubsub.proto", which is not imported by "google/cloud/pubsub_v1/proto/pubsub.proto".  To use it here, please add the necessary import.
  google.pubsub.v1.Publisher.GetTopic: "google.pubsub.v1.Topic" seems to be defined in "google/cloud/proto/pubsub/v1/pubsub.proto", which is not imported by "google/cloud/pubsub_v1/proto/pubsub.proto".  To use it here, please add the necessary import.
  google.pubsub.v1.Publisher.ListTopics: "google.pubsub.v1.ListTopicsRequest" seems to be defined in "google/cloud/proto/pubsub/v1/pubsub.proto", which is not imported by "google/cloud/pubsub_v1/proto/pubsub.proto".  To use it here, please add the necessary import.
  google.pubsub.v1.Publisher.ListTopics: "google.pubsub.v1.ListTopicsResponse" seems to be defined in "google/cloud/proto/pubsub/v1/pubsub.proto", which is not imported by "google/cloud/pubsub_v1/proto/pubsub.proto".  To use it here, please add the necessary import.
  google.pubsub.v1.Publisher.ListTopicSubscriptions: "google.pubsub.v1.ListTopicSubscriptionsRequest" seems to be defined in "google/cloud/proto/pubsub/v1/pubsub.proto", which is not imported by "google/cloud/pubsub_v1/proto/pubsub.proto".  To use it here, please add the necessary import.
  google.pubsub.v1.Publisher.ListTopicSubscriptions: "google.pubsub.v1.ListTopicSubscriptionsResponse" seems to be defined in "google/cloud/proto/pubsub/v1/pubsub.proto", which is not imported by "google/cloud/pubsub_v1/proto/pubsub.proto".  To use it here, please add the necessary import.
  google.pubsub.v1.Publisher.DeleteTopic: "google.pubsub.v1.DeleteTopicRequest" seems to be defined in "google/cloud/proto/pubsub/v1/pubsub.proto", which is not imported by "google/cloud/pubsub_v1/proto/pubsub.proto".  To use it here, please add the necessary import.

import re
import os
import math
import datetime
import json
import logging
import base64
import traceback
import itertools
import pytz
import requests
from functools import reduce

# google cloud datastore imports
from google.cloud.proto.datastore.v1 import query_pb2
from googledatastore import helper as datastore_helper, PropertyFilter, CompositeFilter
from google.cloud.datastore.helpers import entity_from_protobuf
我最终通过创建一个最小的管道集合并逐个添加依赖项来实现它，直到找到了一些有效的组合。依赖项如下所示

Flask
gunicorn
apache-beam[gcp]==2.6.0
oauth2client==3.0.0
google-cloud-datastore==1.3.0
google-cloud-pubsub==0.28.0
google-cloud-core==0.27.0
google-cloud==0.34.0
我们仍然对这些东西在没有任何警告和行动的情况下破裂感到非常沮丧，我们的生产线被关闭了一周，但现在已经关闭了，希望这将有助于其他人

编辑-

我说得太快了。这种组合在本地工作，但在云中仍会中断>：-(

编辑2-

所以..谷歌似乎停止了从
requirements.txt
文件安装。我在
setup.py
文件中有这个

REQUIRED\u软件包=['google-cloud']

当我添加这段代码时，看起来这是唯一安装的依赖项

if os.path.exists('requirements.txt'):
    with open('requirements.txt') as fh:
        REQUIRED_PACKAGES=[line.strip() for line in fh.readlines()]
一切都成功了！现在我的管道开始运行了

因此，谷歌似乎改变了他们从
requirements.txt
停止安装的地方

什么！？！

我从google.cloud.datastore.helpers导入实体_从_protobuf中尝试了这个
，它对我来说工作顺利。我们之间的唯一区别是我的google cloud datastore版本是1.4.0，可能会尝试更新吗？我已经删除了google cloud datastore的版本，问题仍然存在。我将尝试指定版本n 1.4.0我真正想知道的是，为什么这些事情在我们没有做任何事情的情况下就发生了。我们会假设，当我们在生产中得到一个可工作的构建时，它将以稳定的方式运行，直到任何底层SDK/API被弃用，并且只有在合理的时间安排下才会发生。唤醒我们所有的批处理对于生产流水线来说，在我们自己没有改变的情况下，由于某些导入错误而导致的作业失败是不可接受的。我认为您应该在github上对此进行投诉。此外，我试图重现您的错误，但似乎发生了其他问题。如果有人感兴趣，可以进入讨论，而不是扩大评论列表。我正在尝试重新说明你能提供你的脚本正在使用的导入行吗？我还想指出：3中关于PyPI依赖项的第三点。-使用以下命令行选项运行管道：-requirements\u file requirements.txt我不认为Google改变了什么：@Temu我有点困惑..你发送的第二个链接声明它将在GAE Flex环境中运行时安装requirements.txt依赖项，但这正是这个问题的核心所在，它不再只是这样做。如果我部署到Flex环境中，除非在setup.py文件中枚举，否则不会安装这些需求。@Temu另外，关于你的第一点，我不会给出命令行选项当管道在云中运行时，我只需声明管道的入口点
entrypoint:gunicorn-b:$PORT main:app
，然后让她撕裂？@Temu，或者，你的意思是我应该在
beam.pipeline（选项=管道选项）中设置
需求文件
值
dictionary？我想这就留下了一个问题，为什么这在以前是有效的，但后来又随机停止了工作？您发送的链接已经说明GAE Flex environment是从requirements.txt文件安装的，所以这不是本地开发的冗余步骤吗？

if os.path.exists('requirements.txt'):
    with open('requirements.txt') as fh:
        REQUIRED_PACKAGES=[line.strip() for line in fh.readlines()]