Ignite异常，打开的文件太多，但ulimit为；打开文件（-n）1048576“；不行

我停止一个Ignite服务器，重新启动agagin，它抛出 除了打开的文件太多外，我已使用更改打开文件的ulimt

ulimit -n 1048576 

并检查数量变化，但点火仍然无法启动

# ulimit -a
core file size          (blocks, -c) 0
data seg size           (kbytes, -d) unlimited
scheduling priority             (-e) 0
file size               (blocks, -f) unlimited
pending signals                 (-i) 15083
max locked memory       (kbytes, -l) 64
max memory size         (kbytes, -m) unlimited
open files                      (-n) 1048576
pipe size            (512 bytes, -p) 8
POSIX message queues     (bytes, -q) 819200
real-time priority              (-r) 0
stack size              (kbytes, -s) 8192
cpu time               (seconds, -t) unlimited
max user processes              (-u) 15083
virtual memory          (kbytes, -v) unlimited
file locks                      (-x) unlimited

错误日志：

>>> VM name: 7493@test-server-node1
>>> Local node [ID=07F093E4-BBEF-471C-9046-4D1A50B84087, order=9, clientMode=false]
>>> Local node addresses: [san011.fr.alcatel-lucent.com/0:0:0:0:0:0:0:1%lo, 10.0.2.15/10.0.2.15, /127.0.0.1, /192.168.100.11]
>>> Local ports: TCP:10800 TCP:11090 TCP:11211 TCP:47100 UDP:47400 TCP:47500 

[19:13:57,052][INFO][main][GridDiscoveryManager] Topology snapshot [ver=9, servers=2, clients=1, CPUs=5, offheap=1.5GB, heap=4.0GB]
[19:13:57,052][INFO][main][GridDiscoveryManager] Data Regions Configured:
[19:13:57,052][INFO][main][GridDiscoveryManager]   ^-- default [initSize=256.0 MiB, maxSize=758.3 MiB, persistenceEnabled=true]
[19:13:57,255][INFO][sys-#59][GridDhtPartitionDemander] Completed (final) rebalancing [fromNode=918b2b4e-f98e-4faf-bffd-8f9d1dd97bf3, cacheOrGroup=TxCoinLatestInfoCache, topology=AffinityTopologyVersion [topVer=9, minorTopVer=0], time=307 ms]
[19:13:57,256][INFO][sys-#59][GridDhtPartitionDemander] Starting rebalancing [mode=ASYNC, fromNode=918b2b4e-f98e-4faf-bffd-8f9d1dd97bf3, partitionsCount=512, topology=AffinityTopologyVersion [topVer=9, minorTopVer=0], updateSeq=1]
[19:13:57,490][INFO][sys-#45][GridDhtPartitionDemander] Completed (final) rebalancing [fromNode=918b2b4e-f98e-4faf-bffd-8f9d1dd97bf3, cacheOrGroup=TxCoinMinInfoCache, topology=AffinityTopologyVersion [topVer=9, minorTopVer=0], time=232 ms]
[19:13:57,491][INFO][sys-#45][GridDhtPartitionDemander] Starting rebalancing [mode=ASYNC, fromNode=918b2b4e-f98e-4faf-bffd-8f9d1dd97bf3, partitionsCount=512, topology=AffinityTopologyVersion [topVer=9, minorTopVer=0], updateSeq=1]
[19:13:57,828][SEVERE][sys-#57][NodeInvalidator] Critical error with null is happened. All further operations will be failed and local node will be stopped.
class org.apache.ignite.internal.processors.cache.persistence.file.PersistentStorageIOException: Could not initialize file: part-347.bin
    at org.apache.ignite.internal.processors.cache.persistence.file.FilePageStore.init(FilePageStore.java:445)
    at org.apache.ignite.internal.processors.cache.persistence.file.FilePageStore.read(FilePageStore.java:332)
    at org.apache.ignite.internal.processors.cache.persistence.file.FilePageStoreManager.read(FilePageStoreManager.java:322)
    at org.apache.ignite.internal.processors.cache.persistence.file.FilePageStoreManager.read(FilePageStoreManager.java:306)
    at org.apache.ignite.internal.processors.cache.persistence.pagemem.PageMemoryImpl.acquirePage(PageMemoryImpl.java:655)
    at org.apache.ignite.internal.processors.cache.persistence.pagemem.PageMemoryImpl.acquirePage(PageMemoryImpl.java:575)
    at org.apache.ignite.internal.processors.cache.persistence.GridCacheOffheapManager$GridCacheDataStore.getOrAllocatePartitionMetas(GridCacheOffheapManager.java:1132)
    at org.apache.ignite.internal.processors.cache.persistence.GridCacheOffheapManager$GridCacheDataStore.init0(GridCacheOffheapManager.java:1030)
    at org.apache.ignite.internal.processors.cache.persistence.GridCacheOffheapManager$GridCacheDataStore.updateCounter(GridCacheOffheapManager.java:1265)
    at org.apache.ignite.internal.processors.cache.distributed.dht.GridDhtLocalPartition.updateCounter(GridDhtLocalPartition.java:849)
    at org.apache.ignite.internal.processors.cache.distributed.dht.preloader.GridDhtPartitionDemander.handleSupplyMessage(GridDhtPartitionDemander.java:697)
    at org.apache.ignite.internal.processors.cache.distributed.dht.preloader.GridDhtPreloader.handleSupplyMessage(GridDhtPreloader.java:375)
    at org.apache.ignite.internal.processors.cache.GridCachePartitionExchangeManager$5.apply(GridCachePartitionExchangeManager.java:364)
    at org.apache.ignite.internal.processors.cache.GridCachePartitionExchangeManager$5.apply(GridCachePartitionExchangeManager.java:354)
    at org.apache.ignite.internal.processors.cache.GridCacheIoManager.processMessage(GridCacheIoManager.java:1060)
    at org.apache.ignite.internal.processors.cache.GridCacheIoManager.onMessage0(GridCacheIoManager.java:579)
    at org.apache.ignite.internal.processors.cache.GridCacheIoManager.access$700(GridCacheIoManager.java:99)
    at org.apache.ignite.internal.processors.cache.GridCacheIoManager$OrderedMessageListener.onMessage(GridCacheIoManager.java:1609)
    at org.apache.ignite.internal.managers.communication.GridIoManager.invokeListener(GridIoManager.java:1555)
    at org.apache.ignite.internal.managers.communication.GridIoManager.access$4100(GridIoManager.java:126)
    at org.apache.ignite.internal.managers.communication.GridIoManager$GridCommunicationMessageSet.unwind(GridIoManager.java:2751)
    at org.apache.ignite.internal.managers.communication.GridIoManager.unwindMessageSet(GridIoManager.java:1515)
    at org.apache.ignite.internal.managers.communication.GridIoManager.access$4400(GridIoManager.java:126)
    at org.apache.ignite.internal.managers.communication.GridIoManager$10.run(GridIoManager.java:1484)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
    at java.lang.Thread.run(Thread.java:748)
Caused by: java.nio.file.FileSystemException: /usr/share/apache-ignite/work/db/node00-d2cb44e3-b649-4e2e-b6f9-f08f9ae1b3af/cache-TxCoinMinInfoToDbCache/part-347.bin: Too many open files
    at sun.nio.fs.UnixException.translateToIOException(UnixException.java:91)
    at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:102)
    at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:107)
    at sun.nio.fs.UnixFileSystemProvider.newAsynchronousFileChannel(UnixFileSystemProvider.java:196)
    at java.nio.channels.AsynchronousFileChannel.open(AsynchronousFileChannel.java:248)
    at java.nio.channels.AsynchronousFileChannel.open(AsynchronousFileChannel.java:301)
    at org.apache.ignite.internal.processors.cache.persistence.file.AsyncFileIO.<init>(AsyncFileIO.java:57)
    at org.apache.ignite.internal.processors.cache.persistence.file.AsyncFileIOFactory.create(AsyncFileIOFactory.java:53)
    at org.apache.ignite.internal.processors.cache.persistence.file.FilePageStore.init(FilePageStore.java:428)
    ... 26 more
[19:13:57,856][SEVERE][sys-#57][GridCacheIoManager] Failed processing message [senderId=918b2b4e-f98e-4faf-bffd-8f9d1dd97bf3, msg=GridDhtPartitionSupplyMessage [updateSeq=1, topVer=AffinityTopologyVersion [topVer=9, minorTopVer=0], missed=null, clean=[0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 25, 26, 27, 28, 29, 30, 31, 32, 33, 34, 35, 36, 37, 38, 39, 40, 41, 42, 43, 44, 45, 46, 47, 48, 49, 50, 51, 52, 53, 54, 55, 56, 57, 58, 59, 60, 61, 62, 63, 64, 65, 66, 67, 68, 69, 70, 71, 72, 73, 74, 75, 76, 77, 78, 79, 80, 81, 82, 83, 84, 85, 86, 87, 88, 89, 90, 91, 92, 93, 94, 95, 96, 97, 98, 99... and 412 more], msgSize=16500, estimatedKeysCnt=1, size=512, parts=[0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 25, 26, 27, 28, 29, 30, 31, 32, 33, 34, 35, 36, 37, 38, 39, 40, 41, 42, 43, 44, 45, 46, 47, 48, 49, 50, 51, 52, 53, 54, 55, 56, 57, 58, 59, 60, 61, 62, 63, 64, 65, 66, 67, 68, 69, 70, 71, 72, 73, 74, 75, 76, 77, 78, 79, 80, 81, 82, 83, 84, 85, 86, 87, 88, 89, 90, 91, 92, 93, 94, 95, 96, 97, 98, 99... and 412 more], super=GridCacheGroupIdMessage [grpId=-607232546]]]
class org.apache.ignite.IgniteException: Could not initialize file: part-347.bin
    at org.apache.ignite.internal.processors.cache.persistence.GridCacheOffheapManager$GridCacheDataStore.updateCounter(GridCacheOffheapManager.java:1271)
    at org.apache.ignite.internal.processors.cache.distributed.dht.GridDhtLocalPartition.updateCounter(GridDhtLocalPartition.java:849)
    at org.apache.ignite.internal.processors.cache.distributed.dht.preloader.GridDhtPartitionDemander.handleSupplyMessage(GridDhtPartitionDemander.java:697)
    at org.apache.ignite.internal.processors.cache.distributed.dht.preloader.GridDhtPreloader.handleSupplyMessage(GridDhtPreloader.java:375)
    at org.apache.ignite.internal.processors.cache.GridCachePartitionExchangeManager$5.apply(GridCachePartitionExchangeManager.java:364)
    at org.apache.ignite.internal.processors.cache.GridCachePartitionExchangeManager$5.apply(GridCachePartitionExchangeManager.java:354)
    at org.apache.ignite.internal.processors.cache.GridCacheIoManager.processMessage(GridCacheIoManager.java:1060)
    at org.apache.ignite.internal.processors.cache.GridCacheIoManager.onMessage0(GridCacheIoManager.java:579)
    at org.apache.ignite.internal.processors.cache.GridCacheIoManager.access$700(GridCacheIoManager.java:99)
    at org.apache.ignite.internal.processors.cache.GridCacheIoManager$OrderedMessageListener.onMessage(GridCacheIoManager.java:1609)
    at org.apache.ignite.internal.managers.communication.GridIoManager.invokeListener(GridIoManager.java:1555)
    at org.apache.ignite.internal.managers.communication.GridIoManager.access$4100(GridIoManager.java:126)
    at org.apache.ignite.internal.managers.communication.GridIoManager$GridCommunicationMessageSet.unwind(GridIoManager.java:2751)
    at org.apache.ignite.internal.managers.communication.GridIoManager.unwindMessageSet(GridIoManager.java:1515)
    at org.apache.ignite.internal.managers.communication.GridIoManager.access$4400(GridIoManager.java:126)
    at org.apache.ignite.internal.managers.communication.GridIoManager$10.run(GridIoManager.java:1484)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
    at java.lang.Thread.run(Thread.java:748)
Caused by: class org.apache.ignite.internal.processors.cache.persistence.file.PersistentStorageIOException: Could not initialize file: part-347.bin
    at org.apache.ignite.internal.processors.cache.persistence.file.FilePageStore.init(FilePageStore.java:445)
    at org.apache.ignite.internal.processors.cache.persistence.file.FilePageStore.read(FilePageStore.java:332)
    at org.apache.ignite.internal.processors.cache.persistence.file.FilePageStoreManager.read(FilePageStoreManager.java:322)
    at org.apache.ignite.internal.processors.cache.persistence.file.FilePageStoreManager.read(FilePageStoreManager.java:306)
    at org.apache.ignite.internal.processors.cache.persistence.pagemem.PageMemoryImpl.acquirePage(PageMemoryImpl.java:655)
    at org.apache.ignite.internal.processors.cache.persistence.pagemem.PageMemoryImpl.acquirePage(PageMemoryImpl.java:575)
    at org.apache.ignite.internal.processors.cache.persistence.GridCacheOffheapManager$GridCacheDataStore.getOrAllocatePartitionMetas(GridCacheOffheapManager.java:1132)
    at org.apache.ignite.internal.processors.cache.persistence.GridCacheOffheapManager$GridCacheDataStore.init0(GridCacheOffheapManager.java:1030)
    at org.apache.ignite.internal.processors.cache.persistence.GridCacheOffheapManager$GridCacheDataStore.updateCounter(GridCacheOffheapManager.java:1265)
    ... 18 more
Caused by: java.nio.file.FileSystemException: /usr/share/apache-ignite/work/db/node00-d2cb44e3-b649-4e2e-b6f9-f08f9ae1b3af/cache-TxCoinMinInfoToDbCache/part-347.bin: Too many open files
    at sun.nio.fs.UnixException.translateToIOException(UnixException.java:91)
    at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:102)
    at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:107)
    at sun.nio.fs.UnixFileSystemProvider.newAsynchronousFileChannel(UnixFileSystemProvider.java:196)
    at java.nio.channels.AsynchronousFileChannel.open(AsynchronousFileChannel.java:248)
    at java.nio.channels.AsynchronousFileChannel.open(AsynchronousFileChannel.java:301)
    at org.apache.ignite.internal.processors.cache.persistence.file.AsyncFileIO.<init>(AsyncFileIO.java:57)
    at org.apache.ignite.internal.processors.cache.persistence.file.AsyncFileIOFactory.create(AsyncFileIOFactory.java:53)
    at org.apache.ignite.internal.processors.cache.persistence.file.FilePageStore.init(FilePageStore.java:428)
    ... 26 more
[19:13:57,874][SEVERE][upd-ver-checker][GridUpdateNotifier] Runtime error caught during grid runnable execution: GridWorker [name=grid-version-checker, igniteInstanceName=null, finished=false, hashCode=73805044, interrupted=false, runner=upd-ver-checker]
java.lang.ExceptionInInitializerError
    at javax.crypto.JceSecurityManager.<clinit>(JceSecurityManager.java:65)
    at javax.crypto.Cipher.getConfiguredPermission(Cipher.java:2586)
    at javax.crypto.Cipher.getMaxAllowedKeyLength(Cipher.java:2610)
    at sun.security.ssl.CipherSuite$BulkCipher.isUnlimited(CipherSuite.java:535)
    at sun.security.ssl.CipherSuite$BulkCipher.<init>(CipherSuite.java:507)
    at sun.security.ssl.CipherSuite.<clinit>(CipherSuite.java:614)
    at sun.security.ssl.SSLContextImpl.getApplicableCipherSuiteList(SSLContextImpl.java:294)
    at sun.security.ssl.SSLContextImpl.access$100(SSLContextImpl.java:42)
    at sun.security.ssl.SSLContextImpl$AbstractTLSContext.<clinit>(SSLContextImpl.java:425)
    at java.lang.Class.forName0(Native Method)
    at java.lang.Class.forName(Class.java:264)
    at java.security.Provider$Service.getImplClass(Provider.java:1634)
    at java.security.Provider$Service.newInstance(Provider.java:1592)
    at sun.security.jca.GetInstance.getInstance(GetInstance.java:236)
    at sun.security.jca.GetInstance.getInstance(GetInstance.java:164)
    at javax.net.ssl.SSLContext.getInstance(SSLContext.java:156)
    at javax.net.ssl.SSLContext.getDefault(SSLContext.java:96)
    at javax.net.ssl.SSLSocketFactory.getDefault(SSLSocketFactory.java:122)
    at javax.net.ssl.HttpsURLConnection.getDefaultSSLSocketFactory(HttpsURLConnection.java:332)
    at javax.net.ssl.HttpsURLConnection.<init>(HttpsURLConnection.java:289)
    at sun.net.www.protocol.https.HttpsURLConnectionImpl.<init>(HttpsURLConnectionImpl.java:94)
    at sun.net.www.protocol.https.Handler.openConnection(Handler.java:62)
    at sun.net.www.protocol.https.Handler.openConnection(Handler.java:57)
    at java.net.URL.openConnection(URL.java:979)
    at org.apache.ignite.internal.processors.cluster.HttpIgniteUpdatesChecker.getUpdates(HttpIgniteUpdatesChecker.java:59)
    at org.apache.ignite.internal.processors.cluster.GridUpdateNotifier$UpdateChecker.body(GridUpdateNotifier.java:268)
    at org.apache.ignite.internal.util.worker.GridWorker.run(GridWorker.java:110)
    at org.apache.ignite.internal.processors.cluster.GridUpdateNotifier$1.run(GridUpdateNotifier.java:113)
    at java.lang.Thread.run(Thread.java:748)
Caused by: java.lang.SecurityException: Can not initialize cryptographic mechanism
    at javax.crypto.JceSecurity.<clinit>(JceSecurity.java:93)
    ... 29 more
Caused by: java.security.PrivilegedActionException: java.io.FileNotFoundException: /usr/lib/jvm/java-1.8.0-openjdk-1.8.0.171-7.b10.el7.x86_64/jre/lib/security/policy/unlimited/US_export_policy.jar (Too many open files)
    at java.security.AccessController.doPrivileged(Native Method)
    at javax.crypto.JceSecurity.<clinit>(JceSecurity.java:82)
    ... 29 more
Caused by: java.io.FileNotFoundException: /usr/lib/jvm/java-1.8.0-openjdk-1.8.0.171-7.b10.el7.x86_64/jre/lib/security/policy/unlimited/US_export_policy.jar (Too many open files)
    at java.util.zip.ZipFile.open(Native Method)
    at java.util.zip.ZipFile.<init>(ZipFile.java:225)
    at java.util.zip.ZipFile.<init>(ZipFile.java:155)
    at java.util.jar.JarFile.<init>(JarFile.java:166)
    at java.util.jar.JarFile.<init>(JarFile.java:130)
    at javax.crypto.JceSecurity.loadPolicies(JceSecurity.java:353)
    at javax.crypto.JceSecurity.setupJurisdictionPolicies(JceSecurity.java:323)
    at javax.crypto.JceSecurity.access$000(JceSecurity.java:50)
    at javax.crypto.JceSecurity$1.run(JceSecurity.java:85)
    ... 31 more
[19:13:57,897][INFO][node-stopper][GridTcpRestProtocol] Command protocol successfully stopped: TCP binary
[19:13:57,929][INFO][node-stopper][GridJettyRestProtocol] Command protocol successfully stopped: Jetty REST
[19:13:57,935][INFO][node-stopper][GridDhtPartitionDemander] Cancelled rebalancing from all nodes [topology=AffinityTopologyVersion [topVer=9, minorTopVer=0]]
[19:13:57,939][SEVERE][db-checkpoint-thread-#40][GridCacheDatabaseSharedManager] Runtime error caught during grid runnable execution: GridWorker [name=db-checkpoint-thread, igniteInstanceName=null, finished=false, hashCode=1713594100, interrupted=false, runner=db-checkpoint-thread-#40]
class org.apache.ignite.IgniteException: Failed to perform WAL operation (environment was invalidated by a previous error)
    at org.apache.ignite.internal.processors.cache.persistence.pagemem.PageMemoryImpl.beforeReleaseWrite(PageMemoryImpl.java:1490)
    at org.apache.ignite.internal.processors.cache.persistence.pagemem.PageMemoryImpl.writeUnlockPage(PageMemoryImpl.java:1349)
    at org.apache.ignite.internal.processors.cache.persistence.pagemem.PageMemoryImpl.writeUnlock(PageMemoryImpl.java:415)
    at org.apache.ignite.internal.processors.cache.persistence.pagemem.PageMemoryImpl.writeUnlock(PageMemoryImpl.java:409)
    at org.apache.ignite.internal.processors.cache.persistence.tree.util.PageHandler.writeUnlock(PageHandler.java:377)
    at org.apache.ignite.internal.processors.cache.persistence.DataStructure.writeUnlock(DataStructure.java:198)
    at org.apache.ignite.internal.processors.cache.persistence.freelist.PagesList.releaseAndClose(PagesList.java:359)
    at org.apache.ignite.internal.processors.cache.persistence.freelist.PagesList.saveMetadata(PagesList.java:318)
    at org.apache.ignite.internal.processors.cache.persistence.GridCacheOffheapManager.saveStoreMetadata(GridCacheOffheapManager.java:190)
    at org.apache.ignite.internal.processors.cache.persistence.GridCacheOffheapManager.onCheckpointBegin(GridCacheOffheapManager.java:167)
    at org.apache.ignite.internal.processors.cache.persistence.GridCacheDatabaseSharedManager$Checkpointer.markCheckpointBegin(GridCacheDatabaseSharedManager.java:2986)
    at org.apache.ignite.internal.processors.cache.persistence.GridCacheDatabaseSharedManager$Checkpointer.doCheckpoint(GridCacheDatabaseSharedManager.java:2754)
    at org.apache.ignite.internal.processors.cache.persistence.GridCacheDatabaseSharedManager$Checkpointer.body(GridCacheDatabaseSharedManager.java:2679)
    at org.apache.ignite.internal.util.worker.GridWorker.run(GridWorker.java:110)
    at java.lang.Thread.run(Thread.java:748)
Caused by: class org.apache.ignite.internal.pagemem.wal.StorageException: Failed to perform WAL operation (environment was invalidated by a previous error)
    at org.apache.ignite.internal.processors.cache.persistence.wal.FileWriteAheadLogManager.checkNode(FileWriteAheadLogManager.java:1354)
    at org.apache.ignite.internal.processors.cache.persistence.wal.FileWriteAheadLogManager.access$7700(FileWriteAheadLogManager.java:130)
    at org.apache.ignite.internal.processors.cache.persistence.wal.FileWriteAheadLogManager$FileWriteHandle.addRecord(FileWriteAheadLogManager.java:2509)
    at org.apache.ignite.internal.processors.cache.persistence.wal.FileWriteAheadLogManager$FileWriteHandle.access$1900(FileWriteAheadLogManager.java:2419)
    at org.apache.ignite.internal.processors.cache.persistence.wal.FileWriteAheadLogManager.log(FileWriteAheadLogManager.java:700)
    at org.apache.ignite.internal.processors.cache.persistence.pagemem.PageMemoryImpl.beforeReleaseWrite(PageMemoryImpl.java:1486)
    ... 14 more
[19:13:57,952][INFO][tcp-disco-sock-reader-#8][TcpDiscoverySpi] Finished serving remote node connection [rmtAddr=/192.168.100.13:32777, rmtPort=32777
[19:13:57,978][INFO][node-stopper][GridCacheProcessor] Stopped cache [cacheName=ignite-sys-cache]
[19:13:57,979][INFO][node-stopper][GridCacheProcessor] Stopped cache [cacheName=TxCoinMinInfoToDbCache]
[19:13:57,980][INFO][node-stopper][GridCacheProcessor] Stopped cache [cacheName=TxCoinMinInfoCache]
[19:13:57,981][INFO][node-stopper][GridCacheProcessor] Stopped cache [cacheName=TxCoinLatestInfoCache]
[19:13:57,983][INFO][node-stopper][GridCacheProcessor] Stopped cache [cacheName=datastructures_ATOMIC_PARTITIONED_1@default-ds-group, group=default-ds-group]
[19:13:57,985][INFO][node-stopper][GridCacheProcessor] Stopped cache [cacheName=ignite-sys-atomic-cache@default-ds-group, group=default-ds-group]
[19:13:57,986][INFO][node-stopper][GridCacheProcessor] Stopped cache [cacheName=TradeCoinInfoCache]
[19:13:57,986][INFO][node-stopper][GridCacheProcessor] Stopped cache [cacheName=LvOneTxCache]
[19:13:57,986][INFO][node-stopper][GridCacheProcessor] Stopped cache [cacheName=CoinTypeListCache]
[19:13:57,987][INFO][node-stopper][GridCacheProcessor] Stopped cache [cacheName=MatchResultRecordCache]
[19:14:02,540][INFO][node-stopper][GridDeploymentLocalStore] Removed undeployed class: GridDeployment [ts=1525259633995, depMode=SHARED, clsLdr=sun.misc.Launcher$AppClassLoader@764c12b6, clsLdrId=fa6be802361-07f093e4-bbef-471c-9046-4d1a50b84087, userVer=0, loc=true, sampleClsName=org.apache.ignite.internal.processors.cache.distributed.dht.preloader.GridDhtPartitionFullMap, pendingUndeploy=false, undeployed=true, usage=0]
[19:14:02,550][INFO][node-stopper][IgniteKernal] 

>>> +---------------------------------------------------------------------------------+
>>> Ignite ver. 2.4.0#20180305-sha1:aa342270b13cc1f4713382a8eb23b2eb7edaa3a5 stopped OK
>>> +---------------------------------------------------------------------------------+
>>> Grid uptime: 00:00:05.510

>>虚拟机名称：7493@test-服务器节点1
>>>本地节点[ID=07F093E4-BBEF-471C-9046-4D1A50B84087，顺序=9，客户端模式=false]
>>>本地节点地址：[san011.fr.alcatel-lucent.com/0:0:0:0:0:1%lo，10.0.2.15/10.0.2.15，/127.0.0.1，/192.168.100.11]
>>>本地端口：TCP:10800 TCP:11090 TCP:11211 TCP:47100 UDP:47400 TCP:47500
[19:13:57052][INFO][main][GridDiscoveryManager]拓扑快照[ver=9，server=2，clients=1，CPU=5，offheap=1.5GB，heap=4.0GB]
[19:13:57052][INFO][main][GridDiscoveryManager]配置的数据区域：
[19:13:57052][INFO][main][GridDiscoveryManager]^——默认值[initSize=256.0 MiB，maxSize=758.3 MiB，persistenceEnabled=true]
[19:13:57255][INFO][sys-#59][GridDhtPartitionDemander]完成（最终）再平衡[fromNode=918b2b4e-f98e-4faf-bffd-8f9d1dd97bf3，cacheOrGroup=txcoinatestinfocache，拓朴=AffinityTopologyVersion[topVer=9，minorTopVer=0]，时间=307 ms]
[19:13:57256][INFO][sys-#59][GridDhtPartitionDemander]开始重新平衡[mode=ASYNC，fromNode=918b2b4e-f98e-4faf-bffd-8f9d1dd97bf3，partitionsCount=512，拓扑=AffinityTopologyVersion[topVer=9，MinorTopPVer=0]，updateSeq=1]
[19:13:57490][INFO][sys-#45][GridDhtPartitionDemander]完成（最终）再平衡[fromNode=918b2b4e-f98e-4faf-bffd-8f9d1dd97bf3，cacheOrGroup=txcoinmininfo缓存，拓扑=AffinityTopologyVersion[toper=9，minorTopVer=0]，时间=232 ms]
[19:13:57491][INFO][sys-#45][GridDhtPartitionDemander]开始重新平衡[mode=ASYNC，fromNode=918b2b4e-f98e-4faf-bffd-8f9d1dd97bf3，partitionsCount=512，拓扑=AffinityTopologyVersion[topVer=9，minorTopVer=0]，updateSeq=1]
[19:13:57828][severy][sys-#57][NodeInvalidator]发生了null的严重错误。所有后续操作将失败，本地节点将停止。
类org.apache.ignite.internal.processors.cache.persistence.file.PersistentStorageIOException:无法初始化文件：part-347.bin
位于org.apache.ignite.internal.processors.cache.persistence.file.FilePageStore.init（FilePageStore.java:445）
位于org.apache.ignite.internal.processors.cache.persistence.file.FilePageStore.read（FilePageStore.java:332）
位于org.apache.ignite.internal.processors.cache.persistence.file.FilePageStoreManager.read（FilePageStoreManager.java:322）
位于org.apache.ignite.internal.processors.cache.persistence.file.FilePageStoreManager.read（FilePageStoreManager.java:306）
位于org.apache.ignite.internal.processors.cache.persistence.pagemem.PageMemoryImpl.acquirePage（PageMemoryImpl.java:655）
位于org.apache.ignite.internal.processors.cache.persistence.pagemem.PageMemoryImpl.acquirePage（PageMemoryImpl.java:575）
位于org.apache.ignite.internal.processors.cache.persistence.GridCacheOffheapManager$GridCacheDataStore.getOrAllocatePartitionMetas（GridCacheOffheapManager.java:1132）
位于org.apache.ignite.internal.processors.cache.persistence.GridCacheOffheapManager$GridCacheDataStore.init0（GridCacheOffheapManager.java:1030）
位于org.apache.ignite.internal.processors.cache.persistence.GridCacheOffheapManager$GridCacheDataStore.updateCounter（GridCacheOffheapManager.java:1265）
位于org.apache.ignite.internal.processors.cache.distributed.dht.GridDhtLocalPartition.updateCounter（GridDhtLocalPartition.java:849）
位于org.apache.ignite.internal.processors.cache.distributed.dht.preload.GridDhtPartitionDemander.handleSupplyMessage（GridDhtPartitionDemander.java:697）
位于org.apache.ignite.internal.processors.cache.distributed.dht.preload.GridDhtPreloader.handleSupplyMessage（GridDhtPreloader.java:375）
位于org.apache.ignite.internal.processors.cache.GridCachePartitionExchangeManager$5.apply（GridCachePartitionExchangeManager.java:364）
位于org.apache.ignite.internal.processors.cache.GridCachePartitionExchangeManager$5.apply（GridCachePartitionExchangeManager.java:354）
位于org.apache.ignite.internal.processors.cache.GridCacheIoManager.processMessage（GridCacheIoManager.java:1060）
位于org.apache.ignite.internal.processors.cache.GridCacheIoManager.onMessage0（GridCacheIoManager.java:579）
位于org.apache.ignite.internal.processors.cache.GridCacheIoManager.access$700（GridCacheIoManager.java:99）
位于org.apache.ignite.internal.processors.cache.GridCacheIoManager$OrderedMessageListener.onMessage（GridCacheIoManager.java:1609）
位于org.apache.ignite.internal.managers.communication.GridIoManager.invokeListener（GridIoManager.java:1555）
位于org.apache.ignite.internal.managers.communication.GridIoManager.access$4100（GridIoManager.java:126）
位于org.apache.ignite.internal.managers.communication.GridIoManager$GridCommunicationMessageSet.unwind（GridIoManager.java:2751）
位于org.apache.ignite.internal.managers.communication.GridIoManager.unwindmageset（GridIoManager.java:1515）
位于org.apache.ignite.internal.managers.communication.GridIoManager.access$4400（GridIoManager.java:126）
位于org.apache.ignite.internal.managers.communication.GridIoManager$10.run（GridIoManager.java:1484）
位于java.util.concurrent.ThreadPoolExecutor.runWorker（ThreadPoolExecutor.java:1149）
位于java.util.concurrent.ThreadPoolExecutor$Worker.run（ThreadPoolExecutor.java:624）
运行（Thread.java:748）
原因：java.nio.file.FileSystemException:/usr/share/apache ignite/work/db/node00-d2cb44e3-b649-4e2e-b6f9-f08f9ae1b3af/cache txcoinmininfo-todbcache/part-347.bin:打开的文件太多
位于sun.nio.fs.UnixException.translateToIOException（UnixException.java:91）
位于sun.nio.fs.UnixException.rethrowAsIOException（UnixException.java:102）
位于sun.nio.fs.UnixException.rethrowAsIOException（UnixException.java:107）
位于sun.nio.fs.UnixFileSystemProvider.newAsynchronousFileChannel（UnixFileSystemPro