Scrapy splash 飞溅冻结与“的;超时客户端:IPv4Address“;
我正在运行Scrapy splash 飞溅冻结与“的;超时客户端:IPv4Address“;,scrapy-splash,Scrapy Splash,我正在运行scrapy splash从一个网站上抓取数据 定期(随机)冻结下一个日志: [36msplash-service_1 |[0m 2020-07-16 08:49:35.119333 [-] "172.31.0.4" - - [16/Jul/2020:08:49:34 +0000] "POST /execute HTTP/1.1" 200 266018 "-" "Mozilla/5.0 (Windows
scrapy splash
从一个网站上抓取数据
定期(随机)冻结下一个日志:
[36msplash-service_1 |[0m 2020-07-16 08:49:35.119333 [-] "172.31.0.4" - - [16/Jul/2020:08:49:34 +0000] "POST /execute HTTP/1.1" 200 266018 "-" "Mozilla/5.0 (Windows NT 6.2; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/27.0.1453.93 Safari/537.36"
[36msplash-service_1 |[0m 2020-07-16 08:50:10.012973 [-] Timing out client: IPv4Address(type='TCP', host='172.31.0.4', port=51970)
[36msplash-service_1 |[0m 2020-07-16 08:50:10.858080 [-] Timing out client: IPv4Address(type='TCP', host='172.31.0.4', port=51978)
[36msplash-service_1 |[0m 2020-07-16 08:50:16.873014 [-] Timing out client: IPv4Address(type='TCP', host='172.31.0.4', port=51974)
[36msplash-service_1 |[0m 2020-07-16 08:50:17.547947 [-] Timing out client: IPv4Address(type='TCP', host='172.31.0.4', port=51966)
[36msplash-service_1 |[0m 2020-07-16 08:50:18.037436 [-] Timing out client: IPv4Address(type='TCP', host='172.31.0.4', port=51976)
[36msplash-service_1 |[0m 2020-07-16 08:50:29.064655 [-] Timing out client: IPv4Address(type='TCP', host='172.31.0.4', port=51932)
[36msplash-service_1 |[0m 2020-07-16 08:50:35.119997 [-] Timing out client: IPv4Address(type='TCP', host='172.31.0.4', port=51968)
我怎么才能知道原因呢?为什么会卡住
p.S我使用args={“lua\u source”:self.lua\u script\u navigate,“timeout”:60000}
请参阅参数的timeout
:
超时:浮动:可选
渲染的超时(秒)(默认为30)
默认情况下,超时允许的最大值为90秒。要覆盖它,请使用--max timeout命令行选项启动Splash。
例如,这里的Splash配置为允许最多5次超时
会议记录:
如果未使用--max timeout
启动splash,则即使在args
中设置了更高的超时,lua_脚本也会在30秒后中止。请参阅参数的timeout
:
超时:浮动:可选
渲染的超时(秒)(默认为30)
默认情况下,超时允许的最大值为90秒。要覆盖它,请使用--max timeout命令行选项启动Splash。
例如,这里的Splash配置为允许最多5次超时
会议记录:
如果您没有使用--max timeout
启动splash,则即使在args
中设置了更高的超时,lua_脚本也会在30秒后中止
$ docker run -it -p 8050:8050 scrapinghub/splash --max-timeout 300