Python 无法重建错误:对象丢失(用户收回或删除LRU),无法重建
我的代码如下所示:Python 无法重建错误:对象丢失(用户收回或删除LRU),无法重建,python,python-3.x,multiprocessing,ray,Python,Python 3.x,Multiprocessing,Ray,我的代码如下所示: # Features' construction - Multiprocessing # import pandas as pd import time import ray import multiprocessing ray.shutdown() num_cpus = multiprocessing.cpu_count() print(num_cpus) print() ray.init(num_cpus=num_cpus) start = time.time(
# Features' construction - Multiprocessing #
import pandas as pd
import time
import ray
import multiprocessing
ray.shutdown()
num_cpus = multiprocessing.cpu_count()
print(num_cpus)
print()
ray.init(num_cpus=num_cpus)
start = time.time()
@ray.remote
def features_construct(index, row):
dict_features ={}
...
return(dict_features)
data = []
for index, row in enumerate(raw_data):
data.append(features_construct.remote(index, row))
df_data = pd.DataFrame.from_records(ray.get(data))
end = time.time()
print(round(end - start, 2), 'seconds')
但是,我收到以下错误:
2019-10-21 11:30:06,751 WARNING worker.py:416 -- Local object store memory usage:
num clients with quota: 0
quota map size: 0
pinned quota map size: 0
allocated bytes: 5933914928
allocation limit: 7055781888
pinned bytes: 0
(global lru) capacity: 7055781888
(global lru) used: 84.1%
(global lru) num objects: 375838
(global lru) num evictions: 504699
(global lru) bytes evicted: 8467163584
---------------------------------------------------------------------------
UnreconstructableError Traceback (most recent call last)
<ipython-input-27-0f27da969dd5> in <module>
155 print()
156
--> 157 df_data = pd.DataFrame.from_records(ray.get(data))
158
159
/storage2/user/anaconda/lib/python3.7/site-packages/ray/worker.py in get(object_ids)
2347 if isinstance(value, ray.exceptions.UnreconstructableError):
2348 worker.dump_object_store_memory_usage()
-> 2349 raise value
2350
2351 # Run post processors.
UnreconstructableError: Object 54158c91583effffffff0100000000c001000000 is lost (either LRU evicted or deleted by user) and cannot be reconstructed. Try increasing the object store memory available with ray.init(object_store_memory=<bytes>) or setting object store limits with ray.remote(object_store_memory=<bytes>). See also: https://ray.readthedocs.io/en/latest/memory-management.html
将
32
作为输出。您是否找到了导致此不可恢复错误的原因?@mimoralea,老实说,我现在记不起来了:)谢谢您的跟进。出于追踪的目的,我在这里想办法:@mimoralea,当然,谢谢;)
num_cpus = multiprocessing.cpu_count()
print(num_cpus)