使用python请求时接收不完整的数据_Python_Json_Django_Post_Request

使用python请求时接收不完整的数据

python json django post

使用python请求时接收不完整的数据,python,json,django,post,request,Python,Json,Django,Post,Request,我正在抓取一个网站并收集它的数据。我有一些爬虫机，他们把数据发送到中央服务器。爬虫程序中向服务器发送数据的部分代码如下： requests.post(url, json=data, timeout=timeout, cookies=cookies, headers=headers) 在使用django的中央服务器端，我有以下代码： def save_users_data(request): body = json.loads(request.body) // do someth

我正在抓取一个网站并收集它的数据。我有一些爬虫机，他们把数据发送到中央服务器。爬虫程序中向服务器发送数据的部分代码如下：

requests.post(url, json=data, timeout=timeout, cookies=cookies, headers=headers)

在使用django的中央服务器端，我有以下代码：

def save_users_data(request):
    body = json.loads(request.body)
    // do something on data received

有时服务器从爬虫程序接收不完整的数据，所以json包无法加载数据并引发错误。例如，服务器在request.body中接收到以下数据：

b'{"social_network": "some network", "text": "\\u0646\\u06cc\\u0633 \\u0628\\u0627\\u06cc\\u062f \\u0622\\u062a\\u06cc\\u0634 \\u062f\\u0631\\u0633\\u062a \\u06a9\\u0631\\u062f\\u0628\\u0631\\u06af\\u0634\\u062a\\u'

并引发以下错误：

json.decoder.JSONDecodeError: Invalid \uXXXX escape

问题在哪里

编辑

这是nginx error.log文件的一些行：

2018/07/25 12:54:39 [info] 29199#29199: *2520751 client 45.55.4.47 closed keepalive connection
2018/07/25 12:54:39 [info] 29199#29199: *2520753 client 188.166.71.114 closed keepalive connection
2018/07/25 12:55:35 [info] 29199#29199: *2520755 client 45.55.4.47 closed keepalive connection
2018/07/25 12:55:58 [info] 29199#29199: *2520757 client 45.55.4.47 closed keepalive connection
2018/07/25 12:55:59 [info] 29199#29199: *2520759 client 45.55.197.140 closed keepalive connection
2018/07/25 12:56:03 [info] 29199#29199: *2520761 client 188.166.71.114 closed keepalive connection
2018/07/25 12:56:04 [info] 29197#29197: *2520715 epoll_wait() reported that client prematurely closed connection, so upstream connection is closed too while sending request to upstream, client: 167.99.189.246, server: 91.208.165.33, request: "POST /crawler/save/users-data/ HTTP/1.1", upstream: "http://unix:/home/social/centralsystem/centralsystem.sock:/crawler/save/users-data/", host: "91.208.165.33"
2018/07/25 12:56:11 [info] 29197#29197: *2520723 epoll_wait() reported that client prematurely closed connection, so upstream connection is closed too while sending request to upstream, client: 159.89.20.103, server: 91.208.165.33, request: "POST /crawler/save/users-data/ HTTP/1.1", upstream: "http://unix:/home/social/centralsystem/centralsystem.sock:/crawler/save/users-data/", host: "91.208.165.33"
2018/07/25 12:56:12 [info] 29197#29197: *2520724 epoll_wait() reported that client prematurely closed connection, so upstream connection is closed too while sending request to upstream, client: 209.97.142.45, server: 91.208.165.33, request: "POST /crawler/save/users-data/ HTTP/1.1", upstream: "http://unix:/home/social/centralsystem/centralsystem.sock:/crawler/save/users-data/", host: "91.208.165.33"
2018/07/25 12:56:16 [info] 29199#29199: *2520765 client 67.207.92.190 closed keepalive connection
2018/07/25 12:56:17 [info] 29197#29197: *2520729 epoll_wait() reported that client prematurely closed connection, so upstream connection is closed too while sending request to upstream, client: 188.226.178.98, server: 91.208.165.33, request: "POST /crawler/save/users-data/ HTTP/1.1", upstream: "http://unix:/home/social/centralsystem/centralsystem.sock:/crawler/save/users-data/", host: "91.208.165.33"
2018/07/25 12:56:22 [info] 29199#29199: *2520770 client 188.166.71.114 closed keepalive connection
2018/07/25 12:56:26 [info] 29199#29199: *2520767 client 159.89.20.103 closed keepalive connection
2018/07/25 12:56:27 [info] 29197#29197: *2520777 client 159.89.20.103 closed keepalive connection
2018/07/25 12:56:28 [info] 29199#29199: *2520773 client 188.226.178.98 closed keepalive connection
2018/07/25 12:56:28 [info] 29197#29197: *2520779 client 45.55.197.140 closed keepalive connection
2018/07/25 12:56:29 [info] 29197#29197: *2520782 client 188.226.178.98 closed keepalive connection
2018/07/25 12:56:30 [info] 29199#29199: *2520768 client 209.97.142.45 closed keepalive connection
2018/07/25 12:56:30 [info] 29197#29197: *2520781 client 67.207.92.190 closed keepalive connection
2018/07/25 12:56:31 [info] 29197#29197: *2520786 client 209.97.142.45 closed keepalive connection
2018/07/25 12:56:36 [info] 29199#29199: *2520775 client 67.207.92.190 closed keepalive connection

编辑

您是否可以尝试使用

json.dumps

而不是

json.loads

来运行它

json.loads

将只接受unicode字符串，因此可能需要解码

body_unicode = request.body.decode('utf-8')
body = json.loads(body_unicode)
content = body['content']

请在此处阅读：

如评论中所述，问题是我的

请求。我的服务器响应的post

超时较低，客户端在服务器响应之前关闭了连接。

是否尝试

请求.json（）

获取json响应？我没有尝试过，但我认为我的http正文不完整。我错了吗？@jerinterGeorge

request.json（）

解析request.body本身的数据，因此它不会有帮助。OP能否检查服务器是否正确发送数据？比如记录正在发生的事情sent@ArpitSolanki如何记录最终在网络上发送的内容？我知道我的工作没有问题data@mohammad我要求您记录从服务器发送的内容，并检查从中央服务器接收的内容。这样你就会知道网络上发生了什么事我知道我的json数据不完整。我的问题是为什么？这有什么帮助？OP已经知道数据不完整