Neo4j批量进口“；neo4j管理导入&x201D；OutOfMemoryError:Java堆空间和OutOfMemoryError:超出了GC开销限制_Memory_Neo4j_Garbage Collection_Jvm

Neo4j批量进口“；neo4j管理导入&x201D；OutOfMemoryError:Java堆空间和OutOfMemoryError:超出了GC开销限制

memory neo4j jvm

Neo4j批量进口“；neo4j管理导入&x201D；OutOfMemoryError:Java堆空间和OutOfMemoryError:超出了GC开销限制,memory,neo4j,garbage-collection,jvm,Memory,Neo4j,Garbage Collection,Jvm,我的单机可用资源是： Total machine memory: 2.00 TB Free machine memory: 1.81 TB Max heap memory : 910.50 MB Processors: 192 Configured max memory: 1.63 TB 我的file1.csv文件大小为600GB 我的csv文件中的条目数=3 000 000 标题结构 attempt1 item_col1:ID(label),item_col2,item_col3:IGNO

我的单机可用资源是：

Total machine memory: 2.00 TB
Free machine memory: 1.81 TB
Max heap memory : 910.50 MB
Processors: 192
Configured max memory: 1.63 TB

我的file1.csv文件大小为600GB

我的csv文件中的条目数=3 000 000

标题结构

attempt1 
item_col1:ID(label),item_col2,item_col3:IGNORE,item_col4:IGNORE,item_col5,item_col6,item_col7,item_col8:IGNORE
Attempt2
item_col1:ID,item_col2,item_col3:IGNORE,item_col4:IGNORE,item_col5,item_col6,item_col7,item_col8:IGNORE
Attempt3
item_col1:ID,item_col2,item_col3:IGNORE,item_col4:IGNORE,item_col5:LABEL,item_col6,item_col7,item_col8:IGNORE`

Neo4j版本：3.2.1

已尝试配置组合1

 cat ../conf/neo4j.conf | grep "memory"
 dbms.memory.heap.initial_size=16000m
 dbms.memory.heap.max_size=16000m
 dbms.memory.pagecache.size=40g

尝试配置组合2

cat ../conf/neo4j.conf | grep "memory"
dbms.memory.heap.initial_size=900m
dbms.memory.heap.max_size=900m
dbms.memory.pagecache.size=4g

尝试配置组合3

dbms.memory.heap.initial_size=1000m
dbms.memory.heap.max_size=1000m
dbms.memory.pagecache.size=1g

尝试配置组合4

dbms.memory.heap.initial_size=10g
dbms.memory.heap.max_size=10g 
dbms.memory.pagecache.size=10g

尝试配置组合5（已注释）（无输出）

试过的命令

kaushik@machine1:/neo4j/import$ cl
kaushik@machine1:/neo4j/import$ rm -r ../data/databases/
kaushik@machine1:/neo4j/import$ mkdir ../data/databases/
kaushik@machine1:/neo4j/import$ cat ../conf/neo4j.conf | grep active
dbms.active_database=graph.db


kaushik@machine1:/neo4j/import$ ../bin/neo4j-admin import --mode csv --    database social.db --nodes head.csv,file1.csv
Neo4j version: 3.2.1
Importing the contents of these files into /neo4j/data/databases/social.db:
Nodes:
  /neo4j/import/head.csv
  /neo4j/import/file1.csv



Available resources:
Total machine memory: 2.00 TB
Free machine memory: 1.79 TB
Max heap memory : 910.50 MB
Processors: 192
Configured max memory: 1.61 TB

错误1

Nodes, started 2017-07-14 05:32:51.736+0000
[*NODE:7.63 MB---------------------------------------------------|PROPERTIE|LABEL SCAN--------]    0 ?    0
Done in 40s 439ms
Exception in thread "main" java.lang.OutOfMemoryError: GC overhead limit exceeded
at org.neo4j.csv.reader.Extractors$StringArrayExtractor.extract0(Extractors.java:739)
at org.neo4j.csv.reader.Extractors$ArrayExtractor.extract(Extractors.java:680)
at org.neo4j.csv.reader.BufferedCharSeeker.tryExtract(BufferedCharSeeker.java:239)
at org.neo4j.unsafe.impl.batchimport.input.csv.InputEntityDeserializer.deserializeNextFromSource(InputEntityDeserializer.java:138)
at org.neo4j.unsafe.impl.batchimport.input.csv.InputEntityDeserializer.fetchNextOrNull(InputEntityDeserializer.java:77)
at org.neo4j.unsafe.impl.batchimport.input.csv.InputEntityDeserializer.fetchNextOrNull(InputEntityDeserializer.java:41)
at org.neo4j.helpers.collection.PrefetchingIterator.peek(PrefetchingIterator.java:60)
at org.neo4j.helpers.collection.PrefetchingIterator.hasNext(PrefetchingIterator.java:46)
at org.neo4j.unsafe.impl.batchimport.input.csv.ParallelInputEntityDeserializer.lambda$new$0(ParallelInputEntityDeserializer.java:106)
at org.neo4j.unsafe.impl.batchimport.input.csv.ParallelInputEntityDeserializer$$Lambda$150/1372918763.apply(Unknown Source)
at org.neo4j.unsafe.impl.batchimport

错误2

Exception in thread "main" java.lang.OutOfMemoryError: GC overhead limit exceeded
at org.neo4j.csv.reader.Extractors$StringArrayExtractor.extract0(Extractors.java:739)
at org.neo4j.csv.reader.Extractors$ArrayExtractor.extract(Extractors.java:680)
at org.neo4j.csv.reader.BufferedCharSeeker.tryExtract(BufferedCharSeeker.java:239)
at org.neo4j.unsafe.impl.batchimport.input.csv.InputEntityDeserializer.deserializeNextFromSource(InputEntityDeserializer.java:138)
at org.neo4j.unsafe.impl.batchimport.input.csv.InputEntityDeserializer.fetchNextOrNull(InputEntityDeserializer.java:77)
at org.neo4j.unsafe.impl.batchimport.input.csv.InputEntityDeserializer.fetchNextOrNull(InputEntityDeserializer.java:41)
at org.neo4j.helpers.collection.PrefetchingIterator.peek(PrefetchingIterator.java:60)
at org.neo4j.helpers.collection.PrefetchingIterator.hasNext(PrefetchingIterator.java:46)
at org.neo4j.unsafe.impl.batchimport.input.csv.ParallelInputEntityDeserializer.lambda$new$0(ParallelInputEntityDeserializer.java:106)
at org.neo4j.unsafe.impl.batchimport.input.csv.ParallelInputEntityDeserializer$$Lambda$150/1372918763.apply(Unknown Source)
at org.neo4j.unsafe.impl.batchimport.staging.TicketedProcessing.lambda$submit$0(TicketedProcessing.java:110)
at org.neo4j.unsafe.impl.batchimport.staging.TicketedProcessing$$Lambda$154/1949503798.run(Unknown Source)
at org.neo4j.unsafe.impl.batchimport.executor.DynamicTaskExecutor$Processor.run(DynamicTaskExecutor.java:237)

错误3

Nodes, started 2017-07-14 05:39:48.602+0000
[NODE:7.63 MB-----------------------------------------------|PROPER|*LABEL SCAN---------------]    0 ?    0
Done in 42s 140ms
Exception in thread "main" java.lang.OutOfMemoryError: GC overhead limit exceeded
at java.util.Arrays.copyOfRange(Arrays.java:3664)
at java.lang.String.<init>(String.java:207)
at org.neo4j.csv.reader.Extractors$StringExtractor.extract0(Extractors.java:328)
at org.neo4j.csv.reader.Extractors$AbstractSingleValueExtractor.extract(Extractors.java:287)
at org.neo4j.csv.reader.BufferedCharSeeker.tryExtract(BufferedCharSeeker.java:239)
at org.neo4j.unsafe.impl.batchimport.input.csv.InputEntityDeserializer.deserializeNextFromSource(InputEntityDeserializer.java:138)
at org.neo4j.unsafe.impl.batchimport.input.csv.InputEntityDeserializer.fetchNextOrNull(InputEntityDeserializer.java:77)
at org.neo4j.unsafe.impl.batchimport.input.csv.InputEntityDeserializer.fetchNextOrNull(InputEntityDeserializer.java:41)
at org.neo4j.helpers.collection.PrefetchingIterator.peek(PrefetchingIterator.java:60)
at org.neo4j.helpers.collection.PrefetchingIterator.hasNext(PrefetchingIterator.java:46)
at org.neo4j.unsafe.impl.batchimport.input.csv.ParallelInputEntityDeserializer.lambda$new$0(ParallelInputEntityDeserializer.java:106)
at org.neo4j.unsafe.impl.batchimport.input.csv.ParallelInputEntityDeserializer$$Lambda$150/310855317.apply(Unknown Source)
at org.neo4j.unsafe.impl.batchimport.staging.TicketedProcessing.lambda$submit$0(TicketedProcessing.java:110)
at org.neo4j.unsafe.impl.batchimport.staging.TicketedProcessing$$Lambda$154/679112060.run(Unknown Source)
at org.neo4j.unsafe.impl.batchimport.executor.DynamicTaskExecutor$Processor.run(DynamicTaskExecutor.java:237)

错误5

Exception in thread "main" java.lang.OutOfMemoryError: Java heap space
at java.util.Arrays.copyOfRange(Arrays.java:3664)
at java.lang.String.<init>(String.java:207)
at org.neo4j.csv.reader.Extractors$StringExtractor.extract0(Extractors.java:328)
at org.neo4j.csv.reader.Extractors$AbstractSingleValueExtractor.extract(Extractors.java:287)
at org.neo4j.csv.reader.BufferedCharSeeker.tryExtract(BufferedCharSeeker.java:239)
at org.neo4j.unsafe.impl.batchimport.input.csv.InputEntityDeserializer.deserializeNextFromSource(InputEntityDeserializer.java:138)
at org.neo4j.unsafe.impl.batchimport.input.csv.InputEntityDeserializer.fetchNextOrNull(InputEntityDeserializer.java:77)
at org.neo4j.unsafe.impl.batchimport.input.csv.InputEntityDeserializer.fetchNextOrNull(InputEntityDeserializer.java:41)
at org.neo4j.helpers.collection.PrefetchingIterator.peek(PrefetchingIterator.java:60)
at org.neo4j.helpers.collection.PrefetchingIterator.hasNext(PrefetchingIterator.java:46)
at org.neo4j.unsafe.impl.batchimport.input.csv.ParallelInputEntityDeserializer.lambda$new$0(ParallelInputEntityDeserializer.java:106)
at org.neo4j.unsafe.impl.batchimport.input.csv.ParallelInputEntityDeserializer$$Lambda$118/950986004.apply(Unknown Source)
at org.neo4j.unsafe.impl.batchimport.staging.TicketedProcessing.lambda$submit$0(TicketedProcessing.java:110)
at org.neo4j.unsafe.impl.batchimport.staging.TicketedProcessing$$Lambda$122/151277029.run(Unknown Source)
at org.neo4j.unsafe.impl.batchimport.executor.DynamicTaskExecutor$Processor.run(DynamicTaskExecutor.java:237)

线程“main”java.lang.OutOfMemoryError中的异常：java堆空间位于java.util.Arrays.copyOfRange（Arrays.java:3664）位于java.lang.String。（String.java:207）位于org.neo4j.csv.reader.Extractors$StringExtractor.extractor0（Extractors.java:328）位于org.neo4j.csv.reader.Extractors$AbstractSingleValueExtractor.extract（Extractors.java:287）位于org.neo4j.csv.reader.BufferedCharSeeker.tryExtract（BufferedCharSeeker.java:239）位于org.neo4j.unsafe.impl.batchimport.input.csv.InputEntityDeserializer.deserializeNextFromSource（InputEntityDeserializer.java:138）位于org.neo4j.unsafe.impl.batchimport.input.csv.InputEntityDeserializer.fetchNextOrNull（InputEntityDeserializer.java:77）位于org.neo4j.unsafe.impl.batchimport.input.csv.InputEntityDeserializer.fetchNextOrNull（InputEntityDeserializer.java:41）位于org.neo4j.helpers.collection.PrefetchingIterator.peek（PrefetchingIterator.java:60）位于org.neo4j.helpers.collection.PrefetchingIterator.hasNext（PrefetchingIterator.java:46）位于org.neo4j.unsafe.impl.batchimport.input.csv.ParallelInputEntityDeserializer.lambda$new$0（ParallelInputEntityDeserializer.java:106）位于org.neo4j.unsafe.impl.batchimport.input.csv.ParallelInputEntityDeserializer$$Lambda$118/950986004.apply（未知源）位于org.neo4j.unsafe.impl.batchimport.staging.ticketdProcessing.lambda$submit$0（ticketdProcessing.java:110）位于org.neo4j.unsafe.impl.batchimport.staging.ticketdProcessing$$Lambda$122/151277029.run（未知源）位于org.neo4j.unsafe.impl.batchimport.executor.DynamicTaskExecutor$Processor.run（DynamicTaskExecutor.java:237）一般来说，如果你能解释一下第九章。绩效9.1。内存调整的一个例子，它将有助于很多初学者。

您能举一个例子来计算500 GB样本数据集的dbms.memory.heap.initial_size、dbms.memory.heap.max_size、dbms.memory.pagecache.size，其中30亿个条目在1TB RAM机器和100个处理器中有10列大小相等的数据

实际上，如果只进行节点计算，计算非常简单：

3 * 10^9 * 20 / 1024^3

因此，我将使用至少55Gb的堆大小。你能试试吗

问候,，

Tom

实际上，如果只进行节点计算，则计算非常简单：

3 * 10^9 * 20 / 1024^3

因此，我将使用至少55Gb的堆大小。你能试试吗

问候,，

汤姆

p.S.那是为了负载。我并没有说，对于数据库的实际操作运行，您需要使用dbms.memory.heap.initial_size=55g和dbms.memory.heap.max_size=55g，命令在节点处停止，开始于2017-07-14 17:21:20.940+0000[*>：？？-----------------------[*节点：7.63 MB--------------[*节点：7.63 MB--------------[节点：7.63 MB--------------[N |*属性-----]0∆ 0在5m 41s 141ms内完成。它停留在这个阶段。我看不到导入的节点数量，也看不到数据/数据库被填满。我应该设置dbms.memory.pagecache.size吗？P.s.这是用于加载的。我没有说在数据库的实际操作运行中需要它：-）使用dbms.memory.heap.initial_size=55g和dbms.memory.heap.max_size=55g，命令在节点处停止，开始于2017-07-14 17:21:20.940+0000[*>：？？-------------[*节点：7.63 MB--------------[*节点：7.63 MB--------------[节点：7.63 MB-----------[N |*属性-----]0∆ 0在5m 41s 141ms内完成此阶段已停滞。我看不到导入的节点数，也看不到数据/数据库已满。是否知道应该设置dbms.memory.pagecache.size？

3 * 10^9 * 20 / 1024^3