PostAPI请求如何在python中使用并行处理？requests.exceptions.ConnectionError：

我有这样的代码：

import requests
import multiprocessing as mp
import json
import time

BASE_URL = 'http://127.0.0.1:3000/employees'

with open('data.json', 'r') as f:
    array = json.load(f)

def internet_resource_getter(post_data):
    stuff_got = []
    response = requests.post(BASE_URL, json=post_data)
    stuff_got.append(response.json())
    print(stuff_got)
    time.sleep(1)
    return stuff_got

if __name__ == '__main__':
    # freeze_support() here if program needs to be frozen  
    start=time.time()
    with mp.Pool(mp.cpu_count()) as pool:
        pool.map(internet_resource_getter, array)
    elapsed = (time.time() - start)  
    print("\n","time elapsed is :", elapsed)

[
  {
    "id": 1,
    "first_name": "Sebastian",
    "last_name": "Eschweiler"
  },
  {
    "id": 2,
    "first_name": "Steve",
    "last_name": "Palmer"
  },
  {
    "id": 3,
    "first_name": "Ann",
    "last_name": "Smith"
  }
]

multiprocessing.pool.RemoteTraceback:
"""
Traceback (most recent call last):
  File "C:\Users\admin\AppData\Local\Programs\Python\Python38\lib\site-packages\urllib3\connectionpool.py", line 699, in urlopen
    httplib_response = self._make_request(
  File "C:\Users\admin\AppData\Local\Programs\Python\Python38\lib\site-packages\urllib3\connectionpool.py", line 445, in _make_request
    six.raise_from(e, None)
  File "<string>", line 3, in raise_from
  File "C:\Users\admin\AppData\Local\Programs\Python\Python38\lib\site-packages\urllib3\connectionpool.py", line 440, in _make_request
    httplib_response = conn.getresponse()
  File "C:\Users\admin\AppData\Local\Programs\Python\Python38\lib\http\client.py", line 1347, in getresponse
    response.begin()
  File "C:\Users\admin\AppData\Local\Programs\Python\Python38\lib\http\client.py", line 307, in begin
    version, status, reason = self._read_status()
  File "C:\Users\admin\AppData\Local\Programs\Python\Python38\lib\http\client.py", line 276, in _read_status
    raise RemoteDisconnected("Remote end closed connection without"
http.client.RemoteDisconnected: Remote end closed connection without response

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "C:\Users\admin\AppData\Local\Programs\Python\Python38\lib\site-packages\requests\adapters.py", line 439, in send
    resp = conn.urlopen(
  File "C:\Users\admin\AppData\Local\Programs\Python\Python38\lib\site-packages\urllib3\connectionpool.py", line 755, in urlopen
    retries = retries.increment(
  File "C:\Users\admin\AppData\Local\Programs\Python\Python38\lib\site-packages\urllib3\util\retry.py", line 531, in increment
    raise six.reraise(type(error), error, _stacktrace)
  File "C:\Users\admin\AppData\Local\Programs\Python\Python38\lib\site-packages\urllib3\packages\six.py", line 734, in reraise
    raise value.with_traceback(tb)
  File "C:\Users\admin\AppData\Local\Programs\Python\Python38\lib\site-packages\urllib3\connectionpool.py", line 699, in urlopen
    httplib_response = self._make_request(
  File "C:\Users\admin\AppData\Local\Programs\Python\Python38\lib\site-packages\urllib3\connectionpool.py", line 445, in _make_request
    six.raise_from(e, None)
  File "<string>", line 3, in raise_from
  File "C:\Users\admin\AppData\Local\Programs\Python\Python38\lib\site-packages\urllib3\connectionpool.py", line 440, in _make_request
    httplib_response = conn.getresponse()
  File "C:\Users\admin\AppData\Local\Programs\Python\Python38\lib\http\client.py", line 1347, in getresponse
    response.begin()
  File "C:\Users\admin\AppData\Local\Programs\Python\Python38\lib\http\client.py", line 307, in begin
    version, status, reason = self._read_status()
  File "C:\Users\admin\AppData\Local\Programs\Python\Python38\lib\http\client.py", line 276, in _read_status
    raise RemoteDisconnected("Remote end closed connection without"
urllib3.exceptions.ProtocolError: ('Connection aborted.', RemoteDisconnected('Remote end closed connection without response'))

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "paralel_pro.py", line 28, in <module>
    pool.map(internet_resource_getter, array)
  File "C:\Users\admin\AppData\Local\Programs\Python\Python38\lib\multiprocessing\pool.py", line 364, in map
    return self._map_async(func, iterable, mapstar, chunksize).get()
  File "C:\Users\admin\AppData\Local\Programs\Python\Python38\lib\multiprocessing\pool.py", line 771, in get
    raise self._value
requests.exceptions.ConnectionError: ('Connection aborted.', RemoteDisconnected('Remote end closed connection without response'))

文件data.json中包含500条记录，例如：

[{"first_name":"John","last_name":"Swen"},{"first_name":"Ricard","last_name":"Candra"}]

在BASE_URL中有如下数据：

import requests
import multiprocessing as mp
import json
import time

BASE_URL = 'http://127.0.0.1:3000/employees'

with open('data.json', 'r') as f:
    array = json.load(f)

def internet_resource_getter(post_data):
    stuff_got = []
    response = requests.post(BASE_URL, json=post_data)
    stuff_got.append(response.json())
    print(stuff_got)
    time.sleep(1)
    return stuff_got

if __name__ == '__main__':
    # freeze_support() here if program needs to be frozen  
    start=time.time()
    with mp.Pool(mp.cpu_count()) as pool:
        pool.map(internet_resource_getter, array)
    elapsed = (time.time() - start)  
    print("\n","time elapsed is :", elapsed)

[
  {
    "id": 1,
    "first_name": "Sebastian",
    "last_name": "Eschweiler"
  },
  {
    "id": 2,
    "first_name": "Steve",
    "last_name": "Palmer"
  },
  {
    "id": 3,
    "first_name": "Ann",
    "last_name": "Smith"
  }
]

multiprocessing.pool.RemoteTraceback:
"""
Traceback (most recent call last):
  File "C:\Users\admin\AppData\Local\Programs\Python\Python38\lib\site-packages\urllib3\connectionpool.py", line 699, in urlopen
    httplib_response = self._make_request(
  File "C:\Users\admin\AppData\Local\Programs\Python\Python38\lib\site-packages\urllib3\connectionpool.py", line 445, in _make_request
    six.raise_from(e, None)
  File "<string>", line 3, in raise_from
  File "C:\Users\admin\AppData\Local\Programs\Python\Python38\lib\site-packages\urllib3\connectionpool.py", line 440, in _make_request
    httplib_response = conn.getresponse()
  File "C:\Users\admin\AppData\Local\Programs\Python\Python38\lib\http\client.py", line 1347, in getresponse
    response.begin()
  File "C:\Users\admin\AppData\Local\Programs\Python\Python38\lib\http\client.py", line 307, in begin
    version, status, reason = self._read_status()
  File "C:\Users\admin\AppData\Local\Programs\Python\Python38\lib\http\client.py", line 276, in _read_status
    raise RemoteDisconnected("Remote end closed connection without"
http.client.RemoteDisconnected: Remote end closed connection without response

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "C:\Users\admin\AppData\Local\Programs\Python\Python38\lib\site-packages\requests\adapters.py", line 439, in send
    resp = conn.urlopen(
  File "C:\Users\admin\AppData\Local\Programs\Python\Python38\lib\site-packages\urllib3\connectionpool.py", line 755, in urlopen
    retries = retries.increment(
  File "C:\Users\admin\AppData\Local\Programs\Python\Python38\lib\site-packages\urllib3\util\retry.py", line 531, in increment
    raise six.reraise(type(error), error, _stacktrace)
  File "C:\Users\admin\AppData\Local\Programs\Python\Python38\lib\site-packages\urllib3\packages\six.py", line 734, in reraise
    raise value.with_traceback(tb)
  File "C:\Users\admin\AppData\Local\Programs\Python\Python38\lib\site-packages\urllib3\connectionpool.py", line 699, in urlopen
    httplib_response = self._make_request(
  File "C:\Users\admin\AppData\Local\Programs\Python\Python38\lib\site-packages\urllib3\connectionpool.py", line 445, in _make_request
    six.raise_from(e, None)
  File "<string>", line 3, in raise_from
  File "C:\Users\admin\AppData\Local\Programs\Python\Python38\lib\site-packages\urllib3\connectionpool.py", line 440, in _make_request
    httplib_response = conn.getresponse()
  File "C:\Users\admin\AppData\Local\Programs\Python\Python38\lib\http\client.py", line 1347, in getresponse
    response.begin()
  File "C:\Users\admin\AppData\Local\Programs\Python\Python38\lib\http\client.py", line 307, in begin
    version, status, reason = self._read_status()
  File "C:\Users\admin\AppData\Local\Programs\Python\Python38\lib\http\client.py", line 276, in _read_status
    raise RemoteDisconnected("Remote end closed connection without"
urllib3.exceptions.ProtocolError: ('Connection aborted.', RemoteDisconnected('Remote end closed connection without response'))

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "paralel_pro.py", line 28, in <module>
    pool.map(internet_resource_getter, array)
  File "C:\Users\admin\AppData\Local\Programs\Python\Python38\lib\multiprocessing\pool.py", line 364, in map
    return self._map_async(func, iterable, mapstar, chunksize).get()
  File "C:\Users\admin\AppData\Local\Programs\Python\Python38\lib\multiprocessing\pool.py", line 771, in get
    raise self._value
requests.exceptions.ConnectionError: ('Connection aborted.', RemoteDisconnected('Remote end closed connection without response'))

API发布后的预期输出：

[
  {
    "id": 1,
    "first_name": "Sebastian",
    "last_name": "Eschweiler"
  },
  {
    "id": 2,
    "first_name": "Steve",
    "last_name": "Palmer"
  },
  {
    "id": 3,
    "first_name": "Ann",
    "last_name": "Smith"
  },
 {
    "id": 4,
    "first_name": "John",
    "last_name": "Swen"
  },
{
    "id": 5,
    "first_name": "Ricard",
    "last_name": "Candra"
  },
]

在上面的代码中，输入url的数据只有420条记录，即使my data.json是500条记录。如何解决这个问题，以便将500条记录发布到url。我不知道为什么只处理了400个数据。 我有这样一个错误：

import requests
import multiprocessing as mp
import json
import time

BASE_URL = 'http://127.0.0.1:3000/employees'

with open('data.json', 'r') as f:
    array = json.load(f)

def internet_resource_getter(post_data):
    stuff_got = []
    response = requests.post(BASE_URL, json=post_data)
    stuff_got.append(response.json())
    print(stuff_got)
    time.sleep(1)
    return stuff_got

if __name__ == '__main__':
    # freeze_support() here if program needs to be frozen  
    start=time.time()
    with mp.Pool(mp.cpu_count()) as pool:
        pool.map(internet_resource_getter, array)
    elapsed = (time.time() - start)  
    print("\n","time elapsed is :", elapsed)

[
  {
    "id": 1,
    "first_name": "Sebastian",
    "last_name": "Eschweiler"
  },
  {
    "id": 2,
    "first_name": "Steve",
    "last_name": "Palmer"
  },
  {
    "id": 3,
    "first_name": "Ann",
    "last_name": "Smith"
  }
]

multiprocessing.pool.RemoteTraceback:
"""
Traceback (most recent call last):
  File "C:\Users\admin\AppData\Local\Programs\Python\Python38\lib\site-packages\urllib3\connectionpool.py", line 699, in urlopen
    httplib_response = self._make_request(
  File "C:\Users\admin\AppData\Local\Programs\Python\Python38\lib\site-packages\urllib3\connectionpool.py", line 445, in _make_request
    six.raise_from(e, None)
  File "<string>", line 3, in raise_from
  File "C:\Users\admin\AppData\Local\Programs\Python\Python38\lib\site-packages\urllib3\connectionpool.py", line 440, in _make_request
    httplib_response = conn.getresponse()
  File "C:\Users\admin\AppData\Local\Programs\Python\Python38\lib\http\client.py", line 1347, in getresponse
    response.begin()
  File "C:\Users\admin\AppData\Local\Programs\Python\Python38\lib\http\client.py", line 307, in begin
    version, status, reason = self._read_status()
  File "C:\Users\admin\AppData\Local\Programs\Python\Python38\lib\http\client.py", line 276, in _read_status
    raise RemoteDisconnected("Remote end closed connection without"
http.client.RemoteDisconnected: Remote end closed connection without response

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "C:\Users\admin\AppData\Local\Programs\Python\Python38\lib\site-packages\requests\adapters.py", line 439, in send
    resp = conn.urlopen(
  File "C:\Users\admin\AppData\Local\Programs\Python\Python38\lib\site-packages\urllib3\connectionpool.py", line 755, in urlopen
    retries = retries.increment(
  File "C:\Users\admin\AppData\Local\Programs\Python\Python38\lib\site-packages\urllib3\util\retry.py", line 531, in increment
    raise six.reraise(type(error), error, _stacktrace)
  File "C:\Users\admin\AppData\Local\Programs\Python\Python38\lib\site-packages\urllib3\packages\six.py", line 734, in reraise
    raise value.with_traceback(tb)
  File "C:\Users\admin\AppData\Local\Programs\Python\Python38\lib\site-packages\urllib3\connectionpool.py", line 699, in urlopen
    httplib_response = self._make_request(
  File "C:\Users\admin\AppData\Local\Programs\Python\Python38\lib\site-packages\urllib3\connectionpool.py", line 445, in _make_request
    six.raise_from(e, None)
  File "<string>", line 3, in raise_from
  File "C:\Users\admin\AppData\Local\Programs\Python\Python38\lib\site-packages\urllib3\connectionpool.py", line 440, in _make_request
    httplib_response = conn.getresponse()
  File "C:\Users\admin\AppData\Local\Programs\Python\Python38\lib\http\client.py", line 1347, in getresponse
    response.begin()
  File "C:\Users\admin\AppData\Local\Programs\Python\Python38\lib\http\client.py", line 307, in begin
    version, status, reason = self._read_status()
  File "C:\Users\admin\AppData\Local\Programs\Python\Python38\lib\http\client.py", line 276, in _read_status
    raise RemoteDisconnected("Remote end closed connection without"
urllib3.exceptions.ProtocolError: ('Connection aborted.', RemoteDisconnected('Remote end closed connection without response'))

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "paralel_pro.py", line 28, in <module>
    pool.map(internet_resource_getter, array)
  File "C:\Users\admin\AppData\Local\Programs\Python\Python38\lib\multiprocessing\pool.py", line 364, in map
    return self._map_async(func, iterable, mapstar, chunksize).get()
  File "C:\Users\admin\AppData\Local\Programs\Python\Python38\lib\multiprocessing\pool.py", line 771, in get
    raise self._value
requests.exceptions.ConnectionError: ('Connection aborted.', RemoteDisconnected('Remote end closed connection without response'))

multiprocessing.pool.RemoteTraceback：
"""
回溯（最近一次呼叫最后一次）：
文件“C:\Users\admin\AppData\Local\Programs\Python38\lib\site packages\urllib3\connectionpool.py”，第699行，在urlopen中
httplib\u response=self.\u发出请求(
文件“C:\Users\admin\AppData\Local\Programs\Python38\lib\site packages\urllib3\connectionpool.py”，第445行，在请求中
六、从（e，无）中提高
文件“”，第3行，从
文件“C:\Users\admin\AppData\Local\Programs\Python38\lib\site packages\urllib3\connectionpool.py”，第440行，在请求中
httplib_response=conn.getresponse（）
文件“C:\Users\admin\AppData\Local\Programs\Python38\lib\http\client.py”，第1347行，在getresponse中
response.begin（）
文件“C:\Users\admin\AppData\Local\Programs\Python38\lib\http\client.py”，第307行，在begin中
版本、状态、原因=self.\u读取\u状态（）
文件“C:\Users\admin\AppData\Local\Programs\Python38\lib\http\client.py”，第276行，处于读取状态
升起RemoteDisconnected（“远程端关闭连接，不带”
http.client.RemoteDisconnected:远程端关闭连接，无响应
在处理上述异常期间，发生了另一个异常：
回溯（最近一次呼叫最后一次）：
文件“C:\Users\admin\AppData\Local\Programs\Python38\lib\site packages\requests\adapters.py”，第439行，在send中
resp=conn.urlopen(
文件“C:\Users\admin\AppData\Local\Programs\Python38\lib\site packages\urllib3\connectionpool.py”，第755行，在urlopen中
重试次数=重试次数。增量(
文件“C:\Users\admin\AppData\Local\Programs\Python38\lib\site packages\urllib3\util\retry.py”，第531行，增量
升起六个。重新升起（类型（错误），错误，_stacktrace）
文件“C:\Users\admin\AppData\Local\Programs\Python38\lib\site packages\urllib3\packages\six.py”，第734行，在reraise中
通过_回溯（tb）提升值
文件“C:\Users\admin\AppData\Local\Programs\Python38\lib\site packages\urllib3\connectionpool.py”，第699行，在urlopen中
httplib\u response=self.\u发出请求(
文件“C:\Users\admin\AppData\Local\Programs\Python38\lib\site packages\urllib3\connectionpool.py”，第445行，在请求中
六、从（e，无）中提高
文件“”，第3行，从
文件“C:\Users\admin\AppData\Local\Programs\Python38\lib\site packages\urllib3\connectionpool.py”，第440行，在请求中
httplib_response=conn.getresponse（）
文件“C:\Users\admin\AppData\Local\Programs\Python38\lib\http\client.py”，第1347行，在getresponse中
response.begin（）
文件“C:\Users\admin\AppData\Local\Programs\Python38\lib\http\client.py”，第307行，在begin中
版本、状态、原因=self.\u读取\u状态（）
文件“C:\Users\admin\AppData\Local\Programs\Python38\lib\http\client.py”，第276行，处于读取状态
升起RemoteDisconnected（“远程端关闭连接，不带”
urllib3.exceptions.ProtocolError:（'Connection aborted'，RemoteDisconnected（'Remote end closed Connection With response'））
上述异常是以下异常的直接原因：
回溯（最近一次呼叫最后一次）：
文件“paralel_pro.py”，第28行，在
map（internet\u资源\u getter，数组）
文件“C:\Users\admin\AppData\Local\Programs\Python38\lib\multiprocessing\pool.py”，第364行，在map中
返回self.\u map\u async（func、iterable、mapstar、chunksize）.get（）
文件“C:\Users\admin\AppData\Local\Programs\Python38\lib\multiprocessing\pool.py”，第771行，在get中
提升自我价值
requests.exceptions.ConnectionError:（“连接中止”）、RemoteDisconnected（“远程端关闭连接，无响应”））

若要获得答案，请检查服务器日志。我怀疑是错误数据和/或服务器未妥善处理意外数据

客户端检查 查看80条记录是否总是相同的80条记录

服务器端检查

---

您能找到运行在

http://127.0.0.1:3000/employees

？它看起来像是在同一台机器上运行。服务器的应用程序或服务日志应该有确凿的证据。

我想知道数据集中的所有500个条目是否都重现了相同的结果。如果是这样，我将缩小数据集的范围，并与我们分享一些80失败的例子。如果我通过使用mp.Pool（3）更改代码

来减少进程数，那么坏数据可能会解释服务器突然断开连接（例如，如果它不能很好地处理由未初始化输入引起的NullPointerException）作为池：

则成功输入所有我的数据。但根据文件数据中的记录，我的数据不是连续的。json@irfan如果3个进程100%工作，但4个进程不工作（20%错误率），这对我来说非常有趣。在任何方面，这对我来说都像是服务器端问题，因为你甚至可以超过进程数（大约15-20岁，我认为它会影响我的macbook）。你的服务器会产生很多进程吗？那么@Irfan，根据你在这里的评论，你的问题就解决了吗？我使用json服务器，我只使用它来处理上面的代码。但是我想知道我是设置time.sleep（0.2）和m.pool（3）还是time.sleep（0.4）和m.pool（4），连接错误警告将像以前一样出现。但是如果time.sleep（0.4）和m.pool（3），则所有数据都将成功处理。因此，在我的代码中，最快的过程只有在time.sleep（0.4）和m.pool（3）时，才能成功处理整个数据。如果我想通过缩短时间来加快处理速度，请