Memory 使用ShareDarray时内存使用量激增

我是并行计算新手，我对SharedArrays的内存使用有一个疑问。我正在进行并行计算，以便在每个worker中执行一个操作，并在数组的特定片上进行写入

如果我是正确的，

SharedArrays

的工作方式是将数组放在每个辅助进程中，但是如果我想要的东西只允许辅助进程访问我希望它写入新信息的数组的特定部分，该怎么办呢。我感觉我的数组很大，但不是很大，而且我的内存超载了

在我的例子中，我只希望每个工作线程在数组的特定行上进行写入

考虑一下

我得到以下错误

schedule: Task not runnable
error(::String) at .\error.jl:33
enq_work(::Task) at .\task.jl:411
schedule at .\task.jl:426 [inlined]
uv_writecb_task(::Ptr{Nothing}, ::Int32) at .\stream.jl:985
poptaskref(::Base.InvasiveLinkedListSynchronized{Task}) at .\task.jl:564
wait() at .\task.jl:591
uv_write(::Sockets.TCPSocket, ::Ptr{UInt8}, ::UInt64) at .\stream.jl:883
unsafe_write(::Sockets.TCPSocket, ::Ptr{UInt8}, ::UInt64) at .\stream.jl:941
unsafe_write at .\io.jl:522 [inlined]
macro expansion at .\gcutils.jl:87 [inlined]
write at .\io.jl:545 [inlined]
serialize_array_data at C:\cygwin\home\Administrator\buildbot\worker\package_win64\build\usr\share\julia\stdlib\v1.2\Serialization\src\Serialization.jl:246 [inlined]
serialize(::Distributed.ClusterSerializer{Sockets.TCPSocket}, ::Array{Float64,3}) at C:\cygwin\home\Administrator\buildbot\worker\package_win64\build\usr\share\julia\stdlib\v1.2\Serialization\src\Serialization.jl:263
serialize_any(::Distributed.ClusterSerializer{Sockets.TCPSocket}, ::Any) at C:\cygwin\home\Administrator\buildbot\worker\package_win64\build\usr\share\julia\stdlib\v1.2\Serialization\src\Serialization.jl:629
serialize(::Distributed.ClusterSerializer{Sockets.TCPSocket}, ::Any) at C:\cygwin\home\Administrator\buildbot\worker\package_win64\build\usr\share\julia\stdlib\v1.2\Serialization\src\Serialization.jl:608
serialize_any(::Distributed.ClusterSerializer{Sockets.TCPSocket}, ::Any) at C:\cygwin\home\Administrator\buildbot\worker\package_win64\build\usr\share\julia\stdlib\v1.2\Serialization\src\Serialization.jl:629
serialize at C:\cygwin\home\Administrator\buildbot\worker\package_win64\build\usr\share\julia\stdlib\v1.2\Serialization\src\Serialization.jl:608 [inlined]
serialize_msg(::Distributed.ClusterSerializer{Sockets.TCPSocket}, ::Distributed.CallMsg{:call}) at C:\cygwin\home\Administrator\buildbot\worker\package_win64\build\usr\share\julia\stdlib\v1.2\Distributed\src\messages.jl:90
#invokelatest#1 at .\essentials.jl:790 [inlined]
invokelatest at .\essentials.jl:789 [inlined]
send_msg_(::Distributed.Worker, ::Distributed.MsgHeader, ::Distributed.CallMsg{:call}, ::Bool) at C:\cygwin\home\Administrator\buildbot\worker\package_win64\build\usr\share\julia\stdlib\v1.2\Distributed\src\messages.jl:185
#remotecall#146 at C:\cygwin\home\Administrator\buildbot\worker\package_win64\build\usr\share\julia\stdlib\v1.2\Distributed\src\messages.jl:134 [inlined]
remotecall(::Function, ::Distributed.Worker) at C:\cygwin\home\Administrator\buildbot\worker\package_win64\build\usr\share\julia\stdlib\v1.2\Distributed\src\remotecall.jl:349
#remotecall#147(::Base.Iterators.Pairs{Union{},Union{},Tuple{},NamedTuple{(),Tuple{}}}, ::typeof(remotecall), ::Function, ::Int64) at C:\cygwin\home\Administrator\buildbot\worker\package_win64\build\usr\share\julia\stdlib\v1.2\Distributed\src\remotecall.jl:361
remotecall at C:\cygwin\home\Administrator\buildbot\worker\package_win64\build\usr\share\julia\stdlib\v1.2\Distributed\src\remotecall.jl:361 [inlined]
spawnat(::Int64, ::Function) at C:\cygwin\home\Administrator\buildbot\worker\package_win64\build\usr\share\julia\stdlib\v1.2\Distributed\src\macros.jl:15
spawn_somewhere(::Function) at C:\cygwin\home\Administrator\buildbot\worker\package_win64\build\usr\share\julia\stdlib\v1.2\Distributed\src\macros.jl:17
macro expansion at C:\cygwin\home\Administrator\buildbot\worker\package_win64\build\usr\share\julia\stdlib\v1.2\Distributed\src\macros.jl:46 [inlined]
(::getfield(Distributed, Symbol("##167#169")){getfield(Main, Symbol("##15#16")){Array{Float64,3},SharedArray{Float64,3}},UnitRange{Int64}})() at .\task.jl:253
sync_end(::Array{Any,1}) at task.jl:235
macro expansion at task.jl:254 [inlined]
innerloop(::Int64) at sa.jl:17
outerloop(::Int64, ::Int64) at sa.jl:26
top-level scope at sa.jl:32

在我的回复里写上你的名字和留言

WARNING: Workqueue inconsistency detected: popfirst!(Workqueue).state != :runnable
julia>       From worker 2:
      From worker 2:    Please submit a bug report with steps to reproduce this fault, and any error messages that follow (in their entirety). Thanks.
      From worker 2:    Exception: EXCEPTION_ACCESS_VIOLATION at 0x6b5d7b45 -- jl_assign_bits at /home/Administrator/buildbot/worker/package_win64/build/src/home/Administrator/buildbot/worker/package_win64/build/src\datatype.c:610 [inlined]
      From worker 2:    jl_set_nth_field at /home/Administrator/buildbot/worker/package_win64/build/src/home/Administrator/buildbot/worker/package_win64/build/src\datatype.c:977
      From worker 2:    in expression starting at none:0
      From worker 2:    jl_set_nth_field at /home/Administrator/buildbot/worker/package_win64/build/src/home/Administrator/buildbot/worker/package_win64/build/src\datatype.c:968
      From worker 2:    deserialize at C:\cygwin\home\Administrator\buildbot\worker\package_win64\build\usr\share\julia\stdlib\v1.2\Serialization\src\Serialization.jl:1292
      From worker 2:    jl_apply_generic at /home/Administrator/buildbot/worker/package_win64/build/src/home/Administrator/buildbot/worker/package_win64/build/src\gf.c:2191
      From worker 2:    handle_deserialize at C:\cygwin\home\Administrator\buildbot\worker\package_win64\build\usr\share\julia\stdlib\v1.2\Serialization\src\Serialization.jl:786
      From worker 2:    deserialize_msg at C:\cygwin\home\Administrator\buildbot\worker\package_win64\build\usr\share\julia\stdlib\v1.2\Serialization\src\Serialization.jl:722
      From worker 2:    jl_apply_generic at /home/Administrator/buildbot/worker/package_win64/build/src/home/Administrator/buildbot/worker/package_win64/build/src\gf.c:2191
      From worker 2:    jl_apply at /home/Administrator/buildbot/worker/package_win64/build/src/home/Administrator/buildbot/worker/package_win64/build/src\julia.h:1614 [inlined]
      From worker 2:    jl_f__apply at /home/Administrator/buildbot/worker/package_win64/build/src/home/Administrator/buildbot/worker/package_win64/build/src\builtins.c:563
      From worker 2:    jl_f__apply_latest at /home/Administrator/buildbot/worker/package_win64/build/src/home/Administrator/buildbot/worker/package_win64/build/src\builtins.c:601
      From worker 2:    #invokelatest#1 at .\essentials.jl:790 [inlined]
      From worker 2:    invokelatest at .\essentials.jl:789 [inlined]
      From worker 2:    message_handler_loop at C:\cygwin\home\Administrator\buildbot\worker\package_win64\build\usr\share\julia\stdlib\v1.2\Distributed\src\process_messages.jl:183
      From worker 2:    process_tcp_streams at C:\cygwin\home\Administrator\buildbot\worker\package_win64\build\usr\share\julia\stdlib\v1.2\Distributed\src\process_messages.jl:140
      From worker 2:    #105 at .\task.jl:268
      From worker 2:    jl_apply_generic at /home/Administrator/buildbot/worker/package_win64/build/src/home/Administrator/buildbot/worker/package_win64/build/src\gf.c:2197
      From worker 2:    jl_apply at /home/Administrator/buildbot/worker/package_win64/build/src/home/Administrator/buildbot/worker/package_win64/build/src\julia.h:1614 [inlined]
      From worker 2:    start_task at /home/Administrator/buildbot/worker/package_win64/build/src/home/Administrator/buildbot/worker/package_win64/build/src\task.c:596
      From worker 2:    Allocations: 4199815 (Pool: 4136163; Big: 63652); GC: 45

---

也发布了，但没有成功。

是你想要的吗？也许吧，但是DArrays是如何实现的？@phipsgabler当我使用

DistributedArray时

我应该事先指出工人的指数吗？哦，对不起，我自己并没有实际使用DArrays，我只是想它们可能是你想要的。据我所见，它将分配部件本身，但您可以指定使用哪些进程以及如何拆分维度。在多个节点上运行代码时使用

distributedarray

，在单个节点（服务器）上运行代码时使用

sharedarray

。我仍然认为您的问题是由于GC可能不收集

SharedArray

，另一方面，在每次函数调用时，您都会进行一次新的调用。我可以建议重新使用

SharedArray

。这是你想要的吗？也许吧，但是Darray是如何实现的？@phipsgabler当我使用

DistributedArray

时，我应该事先指出工人的索引吗？哦，对不起，我自己实际上没有使用Darray，我只是想它们可能是你想要的。据我所见，它将分配部件本身，但您可以指定使用哪些进程以及如何拆分维度。在多个节点上运行代码时使用

distributedarray

，在单个节点（服务器）上运行代码时使用

sharedarray

。我仍然认为您的问题是由于GC可能不收集

SharedArray

，另一方面，在每次函数调用时，您都会进行一次新的调用。我可以建议重新使用

SharedArray

。

WARNING: Workqueue inconsistency detected: popfirst!(Workqueue).state != :runnable
julia>       From worker 2:
      From worker 2:    Please submit a bug report with steps to reproduce this fault, and any error messages that follow (in their entirety). Thanks.
      From worker 2:    Exception: EXCEPTION_ACCESS_VIOLATION at 0x6b5d7b45 -- jl_assign_bits at /home/Administrator/buildbot/worker/package_win64/build/src/home/Administrator/buildbot/worker/package_win64/build/src\datatype.c:610 [inlined]
      From worker 2:    jl_set_nth_field at /home/Administrator/buildbot/worker/package_win64/build/src/home/Administrator/buildbot/worker/package_win64/build/src\datatype.c:977
      From worker 2:    in expression starting at none:0
      From worker 2:    jl_set_nth_field at /home/Administrator/buildbot/worker/package_win64/build/src/home/Administrator/buildbot/worker/package_win64/build/src\datatype.c:968
      From worker 2:    deserialize at C:\cygwin\home\Administrator\buildbot\worker\package_win64\build\usr\share\julia\stdlib\v1.2\Serialization\src\Serialization.jl:1292
      From worker 2:    jl_apply_generic at /home/Administrator/buildbot/worker/package_win64/build/src/home/Administrator/buildbot/worker/package_win64/build/src\gf.c:2191
      From worker 2:    handle_deserialize at C:\cygwin\home\Administrator\buildbot\worker\package_win64\build\usr\share\julia\stdlib\v1.2\Serialization\src\Serialization.jl:786
      From worker 2:    deserialize_msg at C:\cygwin\home\Administrator\buildbot\worker\package_win64\build\usr\share\julia\stdlib\v1.2\Serialization\src\Serialization.jl:722
      From worker 2:    jl_apply_generic at /home/Administrator/buildbot/worker/package_win64/build/src/home/Administrator/buildbot/worker/package_win64/build/src\gf.c:2191
      From worker 2:    jl_apply at /home/Administrator/buildbot/worker/package_win64/build/src/home/Administrator/buildbot/worker/package_win64/build/src\julia.h:1614 [inlined]
      From worker 2:    jl_f__apply at /home/Administrator/buildbot/worker/package_win64/build/src/home/Administrator/buildbot/worker/package_win64/build/src\builtins.c:563
      From worker 2:    jl_f__apply_latest at /home/Administrator/buildbot/worker/package_win64/build/src/home/Administrator/buildbot/worker/package_win64/build/src\builtins.c:601
      From worker 2:    #invokelatest#1 at .\essentials.jl:790 [inlined]
      From worker 2:    invokelatest at .\essentials.jl:789 [inlined]
      From worker 2:    message_handler_loop at C:\cygwin\home\Administrator\buildbot\worker\package_win64\build\usr\share\julia\stdlib\v1.2\Distributed\src\process_messages.jl:183
      From worker 2:    process_tcp_streams at C:\cygwin\home\Administrator\buildbot\worker\package_win64\build\usr\share\julia\stdlib\v1.2\Distributed\src\process_messages.jl:140
      From worker 2:    #105 at .\task.jl:268
      From worker 2:    jl_apply_generic at /home/Administrator/buildbot/worker/package_win64/build/src/home/Administrator/buildbot/worker/package_win64/build/src\gf.c:2197
      From worker 2:    jl_apply at /home/Administrator/buildbot/worker/package_win64/build/src/home/Administrator/buildbot/worker/package_win64/build/src\julia.h:1614 [inlined]
      From worker 2:    start_task at /home/Administrator/buildbot/worker/package_win64/build/src/home/Administrator/buildbot/worker/package_win64/build/src\task.c:596
      From worker 2:    Allocations: 4199815 (Pool: 4136163; Big: 63652); GC: 45