Python 如果出现错误,如何重新启动tweepy脚本?
我有一个python脚本,它可以连续地将与跟踪的关键字相关的tweet存储到一个文件中。但是,由于下面附加的错误,脚本往往会反复崩溃。如何编辑脚本以使其自动重新启动?我见过很多解决方案,包括这个()但是我不知道如何在我的脚本中实现它Python 如果出现错误,如何重新启动tweepy脚本?,python,restart,tweepy,Python,Restart,Tweepy,我有一个python脚本,它可以连续地将与跟踪的关键字相关的tweet存储到一个文件中。但是,由于下面附加的错误,脚本往往会反复崩溃。如何编辑脚本以使其自动重新启动?我见过很多解决方案,包括这个()但是我不知道如何在我的脚本中实现它 import sys import tweepy import json import os consumer_key="" consumer_secret="" access_key = "" access_secret = "" auth = tweepy.
import sys
import tweepy
import json
import os
consumer_key=""
consumer_secret=""
access_key = ""
access_secret = ""
auth = tweepy.OAuthHandler(consumer_key, consumer_secret)
auth.set_access_token(access_key, access_secret)
api = tweepy.API(auth)
# directory that you want to save the json file
os.chdir("C:\Users\json_files")
# name of json file you want to create/open and append json to
save_file = open("12may.json", 'a')
class CustomStreamListener(tweepy.StreamListener):
def __init__(self, api):
self.api = api
super(tweepy.StreamListener, self).__init__()
# self.list_of_tweets = []
def on_data(self, tweet):
print tweet
save_file.write(str(tweet))
def on_error(self, status_code):
print >> sys.stderr, 'Encountered error with status code:', status_code
return True # Don't kill the stream
print "Stream restarted"
def on_timeout(self):
print >> sys.stderr, 'Timeout...'
return True # Don't kill the stream
print "Stream restarted"
sapi = tweepy.streaming.Stream(auth, CustomStreamListener(api))
sapi.filter(track=["test"])
===========================================================================
Traceback (most recent call last):
File "C:\Users\tweets_to_json.py", line 41, in <module>
sapi.filter(track=["test"])
File "C:\Python27\lib\site-packages\tweepy-2.3-py2.7.egg\tweepy\streaming.py", line 316, in filter
self._start(async)
File "C:\Python27\lib\site-packages\tweepy-2.3-py2.7.egg\tweepy\streaming.py", line 235, in _start
self._run()
File "C:\Python27\lib\site-packages\tweepy-2.3-py2.7.egg\tweepy\streaming.py", line 165, in _run
self._read_loop(resp)
File "C:\Python27\lib\site-packages\tweepy-2.3-py2.7.egg\tweepy\streaming.py", line 206, in _read_loop
for c in resp.iter_content():
File "C:\Python27\lib\site-packages\requests-1.2.3-py2.7.egg\requests\models.py", line 541, in generate
chunk = self.raw.read(chunk_size, decode_content=True)
File "C:\Python27\lib\site-packages\requests-1.2.3-py2.7.egg\requests\packages\urllib3\response.py", line 171, in read
data = self._fp.read(amt)
File "C:\Python27\lib\httplib.py", line 543, in read
return self._read_chunked(amt)
File "C:\Python27\lib\httplib.py", line 603, in _read_chunked
value.append(self._safe_read(amt))
File "C:\Python27\lib\httplib.py", line 660, in _safe_read
raise IncompleteRead(''.join(s), amt)
IncompleteRead: IncompleteRead(0 bytes read, 1 more expected)
回溯(最近一次呼叫最后一次):
文件“C:\Users\tweets_to_json.py”,第41行,在
过滤器(磁道=[“测试”])
文件“C:\Python27\lib\site packages\tweepy-2.3-py2.7.egg\tweepy\streaming.py”,第316行,在过滤器中
自启动(异步)
文件“C:\Python27\lib\site packages\tweepy-2.3-py2.7.egg\tweepy\streaming.py”,第235行,在_start中
self._run()
文件“C:\Python27\lib\site packages\tweepy-2.3-py2.7.egg\tweepy\streaming.py”,第165行,正在运行
自读循环(resp)
文件“C:\Python27\lib\site packages\tweepy-2.3-py2.7.egg\tweepy\streaming.py”,第206行,在\u read\u循环中
对于相应的iter_内容()中的c:
文件“C:\Python27\lib\site packages\requests-1.2.3-py2.7.egg\requests\models.py”,第541行,在generate中
chunk=self.raw.read(chunk\u size,decode\u content=True)
文件“C:\Python27\lib\site packages\requests-1.2.3-py2.7.egg\requests\packages\urllib3\response.py”,第171行,已读
数据=自计量读数(金额)
文件“C:\Python27\lib\httplib.py”,第543行,已读
返回自我。读取块(金额)
文件“C:\Python27\lib\httplib.py”,第603行,分块读取
附加值(自身安全读取(金额))
文件“C:\Python27\lib\httplib.py”,第660行,处于安全读取状态
提高未完成读数(“”.加入,金额)
不完全读取:不完全读取(读取0字节,预计还会有1字节)
通过为流编写新函数,了解了如何合并while/try循环:
def start_stream():
while True:
try:
sapi = tweepy.streaming.Stream(auth, CustomStreamListener(api))
sapi.filter(track=["Samsung", "s4", "s5", "note" "3", "HTC", "Sony", "Xperia", "Blackberry", "q5", "q10", "z10", "Nokia", "Lumia", "Nexus", "LG", "Huawei", "Motorola"])
except:
continue
start_stream()
我通过使用CMD+C手动中断程序来测试自动重启。尽管如此,我还是很高兴听到更好的方法来测试这种功能。我最近遇到了这个问题,希望与大家分享更多关于它的详细信息 导致此错误的原因是所选的流媒体筛选器太宽
test
。因此,您接收流的速度快于您可以接受的速度,这会导致IncompleRead
错误
这可以通过优化搜索或使用更具体的异常来解决:
from http.client import IncompleteRead
...
try:
sapi = tweepy.streaming.Stream(auth, CustomStreamListener(api))
sapi.filter(track=["test"])
except IncompleRead:
pass
一个选项是尝试
多处理
。我认为有两个原因
我已经用tweepy编写了一个2进程流媒体。它下载、压缩和转储数据到每小时旋转一次的文件中。 该程序每小时重新启动一次,它可以定期检查流媒体流程,查看是否下载了任何新的推文。 如果没有,则重新启动整个系统 代码可以找到。
请注意,对于压缩,它使用管道。如果不需要压缩,修改源代码很容易。最好使用递归调用,而不是无限while循环。查看下面的过滤器功能。e、 g
from tweepy import Stream
from service.twitter.listener.tweety_listener import TweetyStreamDataListener
from settings import twitter_config
class Tweety(object):
def __init__(self, listener=TweetyStreamDataListener()):
self.listener = listener
self.__auth__ = None
def __authenticate__(self):
from tweepy import OAuthHandler
if self.__auth__ is None:
self.__auth__ = OAuthHandler(twitter_config['consumer_key'], twitter_config['consumer_secret'])
self.__auth__.set_access_token(twitter_config['access_token'], twitter_config['access_token_secret'])
return self.__auth__ is not None
def __streamer__(self):
is_authenticated = self.__authenticate__()
if is_authenticated:
return Stream(self.__auth__, self.listener)
return None
def filter(self, keywords=None, async=True):
streamer = self.__streamer__()
try:
print "[STREAM] Started steam"
streamer.filter(track=keywords, async=async)
except Exception as ex:
print "[STREAM] Stream stopped! Reconnecting to twitter stream"
print ex.message, ex.args
self.filter(keywords=keywords, async=async)
当崩溃发生时,它是否会出现错误?我不这么认为,因为它不会打印状态代码遇到错误:“尝试将sapi=tweepy放入try and except..但这不是一个好方法。只需尝试一下('''.join(s),amt),您的程序中的这一行是这样的吗<代码>为True时:尝试:sapi=tweepy.streaming.Stream(auth,CustomStreamListener(api))sapi.filter(track=[“索尼”、“Xperia”、“三星”、“s4”、“s5”、“note”、“3”、“HTC”、“黑莓”、“q5”、“q10”、“z10”、“诺基亚”、“Lumia”、“Nexus”、“LG”、“华为”、“摩托罗拉”])除了:pass我必须捕获KeyboardInterrupt才能退出脚本除了KeyboardInterrupt:break我认为恰恰相反:使用递归而不是循环处理这种情况最终会导致内存问题,特别是对于设计为长时间生产的服务。
from tweepy import Stream
from service.twitter.listener.tweety_listener import TweetyStreamDataListener
from settings import twitter_config
class Tweety(object):
def __init__(self, listener=TweetyStreamDataListener()):
self.listener = listener
self.__auth__ = None
def __authenticate__(self):
from tweepy import OAuthHandler
if self.__auth__ is None:
self.__auth__ = OAuthHandler(twitter_config['consumer_key'], twitter_config['consumer_secret'])
self.__auth__.set_access_token(twitter_config['access_token'], twitter_config['access_token_secret'])
return self.__auth__ is not None
def __streamer__(self):
is_authenticated = self.__authenticate__()
if is_authenticated:
return Stream(self.__auth__, self.listener)
return None
def filter(self, keywords=None, async=True):
streamer = self.__streamer__()
try:
print "[STREAM] Started steam"
streamer.filter(track=keywords, async=async)
except Exception as ex:
print "[STREAM] Stream stopped! Reconnecting to twitter stream"
print ex.message, ex.args
self.filter(keywords=keywords, async=async)