Apache spark 使用cassandra将10k记录提交到Janus graph时出错

我正在从一个oracle数据库中获取大约1000万条记录，并尝试将这些记录保存到Janus graph中，使用Cassandra作为存储后端[使用Spark框架]

当我尝试在循环中迭代记录并尝试每10k提交一次时，我得到以下错误

 ERROR StandardJanusGraph: Could not commit transaction [1] due to storage exception in commit
    org.janusgraph.core.JanusGraphException: Could not execute operation due to backend exception

当我试图从Oracle获得第一条1L记录并每1K提交一次时，它工作正常

有人能帮我解决这个错误吗？谢谢你的帮助。谢谢

更新：

WARN  [ReadStage-3] 2019-09-29 08:39:28,327 AbstractLocalAwareExecutorService.java:167 - Uncaught exception on thread Thread[ReadStage-3,5,main]: {}
WARN  [MemtableFlushWriter:17] 2019-09-29 09:09:40,843 NativeLibrary.java:304 - open(/var/lib/cassandra/data/circuit_equipment/system_properties-eeef4cb0e29711e9af61a34111381c19, O_RDONLY) failed, errno (2).
ERROR [MemtableFlushWriter:17] 2019-09-29 09:09:40,846 LogTransaction.java:272 - Transaction log [md_txn_flush_de900e80-e298-11e9-af61-a34111381c19.log in /var/lib/cassandra/data/circuit_equipment/system_properties-eeef4cb0e29711e9af61a34111381c19] indicates txn was not completed, trying to abort it now
ERROR [MemtableFlushWriter:17] 2019-09-29 09:09:40,847 LogTransaction.java:275 - Failed to abort transaction log [md_txn_flush_de900e80-e298-11e9-af61-a34111381c19.log in /var/lib/cassandra/data/circuit_equipment/system_properties-eeef4cb0e29711e9af61a34111381c19]
ERROR [MemtableFlushWriter:17] 2019-09-29 09:09:40,848 LogTransaction.java:222 - Unable to delete /var/lib/cassandra/data/circuit_equipment/system_properties-eeef4cb0e29711e9af61a34111381c19/md_txn_flush_de900e80-e298-11e9-af61-a34111381c19.log as it does not exist, see debug log file for stack trace
ERROR [MemtablePostFlush:9] 2019-09-29 09:09:40,849 CassandraDaemon.java:228 - Exception in thread Thread[MemtablePostFlush:9,5,main]
WARN  [StorageServiceShutdownHook] 2019-09-29 09:09:40,849 StorageService.java:4591 - Caught exception while waiting for memtable flushes during shutdown hook
ERROR [StorageServiceShutdownHook] 2019-09-29 09:09:40,931 AbstractCommitLogSegmentManager.java:308 - Failed to force-recycle all segments; at least one segment is still in use with dirty CFs.
WARN  [main] 2019-09-29 09:09:44,580 NativeLibrary.java:187 - Unable to lock JVM memory (ENOMEM). This can result in part of the JVM being swapped out, especially with mmapped I/O enabled. Increase RLIMIT_MEMLOCK or run Cassandra as root.
WARN  [main] 2019-09-29 09:09:44,581 StartupChecks.java:169 - JMX is not enabled to receive remote connections. Please see cassandra-env.sh for more info.
WARN  [main] 2019-09-29 09:09:44,591 SigarLibrary.java:174 - Cassandra server running in degraded mode. Is swap disabled? : false,  Address space adequate? : true,  nofile limit adequate? : true, nproc limit adequate? : true
WARN  [main] 2019-09-29 09:09:44,593 StartupChecks.java:311 - Maximum number of memory map areas per process (vm.max_map_count) 65530 is too low, recommended value: 1048575, you can change it with sysctl.
WARN  [Native-Transport-Requests-1] 2019-09-29 09:12:12,841 CompressionParams.java:383 - The sstable_compression option has been deprecated. You should use class instead
WARN  [Native-Transport-Requests-1] 2019-09-29 09:12:12,842 CompressionParams.java:334 - The chunk_length_kb option has been deprecated. You should use chunk_length_in_kb instead
WARN  [main] 2019-09-29 12:59:57,584 NativeLibrary.java:187 - Unable to lock JVM memory (ENOMEM). This can result in part of the JVM being swapped out, especially with mmapped I/O enabled. Increase RLIMIT_MEMLOCK or run Cassandra as root.
WARN  [main] 2019-09-29 12:59:57,585 StartupChecks.java:169 - JMX is not enabled to receive remote connections. Please see cassandra-env.sh for more info.
WARN  [main] 2019-09-29 12:59:57,599 SigarLibrary.java:174 - Cassandra server running in degraded mode. Is swap disabled? : false,  Address space adequate? : true,  nofile limit adequate? : true, nproc limit adequate? : true
WARN  [main] 2019-09-29 12:59:57,602 StartupChecks.java:311 - Maximum number of memory map areas per process (vm.max_map_count) 65530 is too low, recommended value: 1048575, you can change it with sysctl.
root@f451df425ca8:/var/log/cassandra#

---

从这些消息中，您应该禁用交换（这实际上是Cassandra中的主要建议之一）：

您还应该更改

max\u map\u count

。您可以使用设置生产环境的其他值。从该消息：

WARN  [main] 2019-09-29 12:59:57,602 StartupChecks.java:311 - Maximum number of memory map areas per process (vm.max_map_count) 65530 is too low, recommended value: 1048575, you can change it with sysctl.

---

C*或janus日志中是否有关于后端异常的详细信息？请运行以下

grep'ERROR\|WARN'/var/log/cassandra/system.log

，并将输出放入您的问题中。system.log的位置可能会有所不同，因此请更改为适当的值。@PedroGordo将问题中的cassandra错误/警告日志附加为更新。你能检查一下吗@谢谢你的回复。在/etc/sysctl.conf中设置vm.max_map_count=1048575就足够了吗？因为在下面的路径中找不到cassandra.conf文件-/etc/security/limits.d/cassandra.conf-，要禁用cassandra中的交换，可以清除/etc吗/fstab@PedroGordo->在中更改了cassandra.conf/etc/security/limits.d/cassandra.conf以及vm.max_map_count=1048575 in/etc/sysctl.conf。无法使用命令sudo sysctl-p saying->sysctl:setting key“vm.max_map_count”：只读文件系统并无法运行sudo swapoff--all（交换禁用）saying->swapoff:不是超级用户。感谢@Pedro Gordo的回复第1步：我们需要在/etc/sysctl.conf中添加vm.max\u map\u count=1048575。步骤2：在下面的文件夹中找不到cassandra.conf路径/etc/security/limits.d/cassandra.conf。为我的工作添加步骤1是否足够problem@Sruthi我认为

/etc/security/limits.d/cassandra.conf

会根据您的操作系统稍微改变位置，但我不确定。更重要的是，不要忘记关闭交换。

WARN  [main] 2019-09-29 12:59:57,602 StartupChecks.java:311 - Maximum number of memory map areas per process (vm.max_map_count) 65530 is too low, recommended value: 1048575, you can change it with sysctl.