C++ Boost:：进程间共享内存总线错误

我在使用OpenMPI 1.3.3的集群上使用CentOS 5.4 x86_64和Boost 1.42.0。我正在编写一个共享库，它使用共享内存存储大量数据，供多个进程使用。还有一个加载程序，它将从文件中读取数据并将其加载到共享内存中

当我运行加载器应用程序时，它会精确地确定存储数据所需的内存量，然后增加25%的开销。对于几乎每一个文件，其数据量将超过2 Gig。当我使用Boost的进程间库发出内存请求时，它表示已成功保留请求的内存量。但当我开始使用它时，我得到一个“总线错误”。据我所知，总线错误是访问内存段可用范围之外的内存的结果

因此，我开始研究Linux上的共享内存是如何配置的，以及要检查什么以确保我的系统正确配置为允许使用大量的共享内存

我查看了

/proc/sys/kernel/shm*

上的“文件”：

• shmall

  -4294967296（4 Gb）

• shmmax

  -68719476736（68 Gb）

• shmmni

  -4096

我调用了

ipcs-lm

命令： ------ Shared Memory Limits -------- max number of segments = 4096 max seg size (kbytes) = 67108864 max total shared memory (kbytes) = 17179869184 min seg size (bytes) = 1 就像我最初的应用程序一样，分配的共享内存大小上限为2 Gig。考虑到它“成功”分配了2352000000字节的内存，以Gb为单位（使用1024*1024*1024），它应该是2.19GB

当我运行实际程序以使用MPI加载数据时，我得到以下错误输出：

Requested: 2808771120 Recieved: 2808771120 [c1-master:13894] *** Process received signal *** [c1-master:13894] Signal: Bus error (7) [c1-master:13894] Signal code: (2) [c1-master:13894] Failing at address: 0x2b3190157000 [c1-master:13894] [ 0] /lib64/libpthread.so.0 [0x3a64e0e7c0] [c1-master:13894] [ 1] ../LookupPopulationLib/Release/libLookupPopulation.so(_ZN5boost12interprocess26uninitialized_copy_or_moveINS0_10offset_ptrIlEEPlEET0_T_S6_S5_PNS_10disable_ifINS0_11move_detail16is_move_iteratorIS6_EEvE4typeE+0x218) [0x2b310dcf3fb8] [c1-master:13894] [ 2] ../LookupPopulationLib/Release/libLookupPopulation.so(_ZN5boost9container6vectorIlNS_12interprocess9allocatorIlNS2_15segment_managerIcNS2_15rbtree_best_fitINS2_12mutex_familyENS2_10offset_ptrIvEELm0EEENS2_10iset_indexEEEEEE15priv_assign_auxINS7_IlEEEEvT_SG_St20forward_iterator_tag+0xa75) [0x2b310dd0a335] [c1-master:13894] [ 3] ../LookupPopulationLib/Release/libLookupPopulation.so(_ZN5boost9container17containers_detail25advanced_insert_aux_proxyINS0_6vectorIlNS_12interprocess9allocatorIlNS4_15segment_managerIcNS4_15rbtree_best_fitINS4_12mutex_familyENS4_10offset_ptrIvEELm0EEENS4_10iset_indexEEEEEEENS0_17constant_iteratorISF_lEEPSF_E25uninitialized_copy_all_toESI_+0x1d7) [0x2b310dd0b817] [c1-master:13894] [ 4] ../LookupPopulationLib/Release/libLookupPopulation.so(_ZN5boost9container6vectorINS1_IlNS_12interprocess9allocatorIlNS2_15segment_managerIcNS2_15rbtree_best_fitINS2_12mutex_familyENS2_10offset_ptrIvEELm0EEENS2_10iset_indexEEEEEEENS3_ISD_SB_EEE17priv_range_insertENS7_ISD_EEmRNS0_17containers_detail23advanced_insert_aux_intISD_PSD_EE+0x771) [0x2b310dd0d521] [c1-master:13894] [ 5] ../LookupPopulationLib/Release/libLookupPopulation.so(_ZN5boost12interprocess6detail8Ctor3ArgINS_9container6vectorINS4_IlNS0_9allocatorIlNS0_15segment_managerIcNS0_15rbtree_best_fitINS0_12mutex_familyENS0_10offset_ptrIvEELm0EEENS0_10iset_indexEEEEEEENS5_ISF_SD_EEEELb0EiSF_NS5_IvSD_EEE11construct_nEPvmRm+0x157) [0x2b310dd0d9a7] [c1-master:13894] [ 6] ../LookupPopulationLib/Release/libLookupPopulation.so(_ZN5boost12interprocess15segment_managerIcNS0_15rbtree_best_fitINS0_12mutex_familyENS0_10offset_ptrIvEELm0EEENS0_10iset_indexEE28priv_generic_named_constructIcEEPvmPKT_mbbRNS0_6detail18in_place_interfaceERNS7_INSE_12index_configISB_S6_EEEENSE_5bool_ILb1EEE+0x6fd) [0x2b310dd0c85d] [c1-master:13894] [ 7] ../LookupPopulationLib/Release/libLookupPopulation.so(_ZN5boost12interprocess15segment_managerIcNS0_15rbtree_best_fitINS0_12mutex_familyENS0_10offset_ptrIvEELm0EEENS0_10iset_indexEE22priv_generic_constructEPKcmbbRNS0_6detail18in_place_interfaceE+0xf8) [0x2b310dd0dd58] [c1-master:13894] [ 8] ../LookupPopulationLib/Release/libLookupPopulation.so(_ZN7POP_LTL16ExportPopulation22InitializeSharedMemoryEPKc+0x1609) [0x2b310dceea99] [c1-master:13894] [ 9] ../LookupPopulationLib/Release/libLookupPopulation.so(_ZN7POP_LTL10InitializeEPKc+0x349) [0x2b310dd0ebb9] [c1-master:13894] [10] MPI_Release/LookupPopulation.MpiLoader(main+0x372) [0x4205d2] [c1-master:13894] [11] /lib64/libc.so.6(__libc_start_main+0xf4) [0x3a6461d994] [c1-master:13894] [12] MPI_Release/LookupPopulation.MpiLoader(__gxx_personality_v0+0x239) [0x420009] [c1-master:13894] *** End of error message *** -------------------------------------------------------------------------- mpirun noticed that process rank 0 with PID 13894 on node c1-master exited on signal 7 (Bus error). -------------------------------------------------------------------------- 请求：2808771120 收到：2808771120 [c1主机：13894]***处理接收信号*** [c1主机：13894]信号：总线错误（7） [c1主机：13894]信号代码：（2） [c1主机：13894]在地址0x2b3190157000处失败 [c1 master:13894][0]/lib64/libpthread.so.0[0x3a64e0e7c0] [c1 master:13894][1]../LookupPopulationLib/Release/libLookupPopulation.so（_zn5boost12进程间未初始化的_copy_或_moveINS0_10offset_ptrIlEEPlEET0_S6_S5_PNS_10禁用_iIns0_11; move_11; detail16是_move_迭代器6_eeve4类型+0x218;）[0x2B310; 0x2B318] [c1 master:13894][2]../LookupPopulationLib/Release/libLookupPopulation.so（\u ZN5boost9container6vectorIlNS\u 12interprocess9allocatorin2\u 15segment2\u 15rbtree\u最佳匹配2\u 12mutex\u family\u 2\u 10 offset\u ptrive2\u indexeeee2\u分配\u auxin7\u ileeevet\u SG\u st20前向迭代器标签+0xA33105）[0xA335] [c1 master:13894][3]../LookupPopulationLib/Release/libLookupPopulation.so（\u zn5 boost9 container 17 containers\u detail25高级\u insert\u aux proxyINS0\u 6矢量\u 12进程间分配\u 15段\u manager\u rbtree\u最佳\u 12 mutex \u 10 family 4\u 10 offset\u ptriveelm4\u 10索引为常量\u迭代器\u lEEPSF\u 25未初始化的复制\u 10所有u-toESI+0x7）[0x3107] [c1 master:13894][4]../LookupPopulationLib/Release/libLookupPopulation.so（\uzn5boost9container6vectorins1\u2ilns\u12进程间分配2\u15segment2\u15rbtree\u15rbtree\u12mutex\u12family\u10offset\u10piveeens2\u10iset2\u10isd\u10isb\u10ees7专用范围\u10isd\u7\u10isd\u10isd\u10details\u10psd\u10details\u10advanced\u10advanced\u10e\u10isd\u10isd\u10details\u10x0\u10x0\u10details\u10details\u10e\u10x0\u10x0\u10x0\u [c1 master:13894][5]../LookupPopulationLib/Release/libLookupPopulation.so（\u zn5boost12进程间的细节因素、容器、向量、分类、分段、管理、最适合12个UTEX家庭、偏移、独立的ISF、NEPEEEEE5和IVEE5结构）[0x2b310dd0d9a7] [c1 master:13894][6]../LookupPopulationLib/Release/libLookupPopulation.so（\u zn5boost12进程间段\u管理\u树\u树\u最佳匹配\u 12家庭\u 10偏移\u pTriveelm0eens0 \u索引\u索引\u通用\u命名\u结构\u PKT\u mbbRNS0 \u详细信息\u位置\u接口工程师\u索引配置\u0索引\u6 \u6 \u6 \u6 \u6 \u6 \u6\u6\u6）[0x2b310dd0c85d] [c1 master:13894][7]../LookupPopulationLib/Release/libLookupPopulation.so（_zn5boost12进程间段_managerIcNS0 _15rbtree _最佳匹配度0 12; UTEX 10;家庭10;偏移10;驱动10;索引2驱动10;通用(构造)KCmbbrns0(细节)在(地方)接口对象+0x3108;[0xf8 0x2bdd0DD58] [c1 master:13894][8]../LookupPopulationLib/Release/libLookupPopulation.so（_ZN7POP_ltl16导出填充22初始化SharedMemoryEPKC+0x1609）[0x2b310dceea99] [c1 master:13894][9]../LookupPopulationLib/Release/libLookupPopulation.so（_ZN7POP_ltl10初始化eepkc+0x349）[0x2b310dd0ebb9] [c1 master:13894][10]MPI_发布/LookupPopulation.MpiLoader（main+0x372）[0x4205d2] [c1 master:13894][11]/lib64/libc.so.6（u libc_start_main+0xf4）[0x3a6461d994] [c1 master:13894][12]MPI_发布/LookupPopulation.MpiLoader（uuu gxx_personality_v0+0x239）[0x420009] [c1主机：13894]***错误消息结束*** -------------------------------------------------------------------------- mpirun注意到节点c1主节点上PID 13894为0的进程列组在信号7（总线错误）时退出。 -------------------------------------------------------------------------- 我真的不知道该怎么办。有人有什么建议吗

---

---

发布到Boost bug trac上：

好吧，如果你继续寻找答案足够长的时间

在Linux上，它使用的共享内存机制（

tmpfs

）默认将其限制为系统RAM的一半。因此，在my cluster上，它是2GB，因为我们有4GB的系统RAM。因此，当它尝试分配共享内存段时，它分配到

/dev/shm

上剩余的最大大小

但当Boost库无法分配请求的内存量时，它并没有指出错误，甚至没有报告正确的可用内存量。它只是很高兴地继续工作，直到到达段的末尾，然后出错

长期的解决办法是提高效率 g++ ShmTest.cpp -lboost_system -lrt Length: 280000000 sizeof(long): 8 Required: 2240000000 Requested: 2352000000 Allocated: 2352000000 Overhead: 224 Free: 2351999776 0: Vector Capacity: 0 Free: 2351999776 2800000: Vector Capacity: 3343205 Free: 2325254128 5600000: Vector Capacity: 8558607 Free: 2283530912 8400000: Vector Capacity: 8558607 Free: 2283530912 11200000: Vector Capacity: 13693771 Free: 2242449600 14000000: Vector Capacity: 21910035 Free: 2176719488 ... 19600000: Vector Capacity: 21910035 Free: 2176719488 22400000: Vector Capacity: 35056057 Free: 2071551312 ... 33600000: Vector Capacity: 35056057 Free: 2071551312 36400000: Vector Capacity: 56089691 Free: 1903282240 ... 56000000: Vector Capacity: 56089691 Free: 1903282240 58800000: Vector Capacity: 89743507 Free: 1634051712 ... 89600000: Vector Capacity: 89743507 Free: 1634051712 92400000: Vector Capacity: 143589611 Free: 1203282880 ... 142800000: Vector Capacity: 143589611 Free: 1203282880 145600000: Vector Capacity: 215384417 Free: 628924432 ... 212800000: Vector Capacity: 215384417 Free: 628924432 215600000: Vector Capacity: 293999969 Free: 16 ... 260400000: Vector Capacity: 293999969 Free: 16 Bus error total 2.0G 0 . 0 .. 2.0G ShmTest Requested: 2808771120 Recieved: 2808771120 [c1-master:13894] *** Process received signal *** [c1-master:13894] Signal: Bus error (7) [c1-master:13894] Signal code: (2) [c1-master:13894] Failing at address: 0x2b3190157000 [c1-master:13894] [ 0] /lib64/libpthread.so.0 [0x3a64e0e7c0] [c1-master:13894] [ 1] ../LookupPopulationLib/Release/libLookupPopulation.so(_ZN5boost12interprocess26uninitialized_copy_or_moveINS0_10offset_ptrIlEEPlEET0_T_S6_S5_PNS_10disable_ifINS0_11move_detail16is_move_iteratorIS6_EEvE4typeE+0x218) [0x2b310dcf3fb8] [c1-master:13894] [ 2] ../LookupPopulationLib/Release/libLookupPopulation.so(_ZN5boost9container6vectorIlNS_12interprocess9allocatorIlNS2_15segment_managerIcNS2_15rbtree_best_fitINS2_12mutex_familyENS2_10offset_ptrIvEELm0EEENS2_10iset_indexEEEEEE15priv_assign_auxINS7_IlEEEEvT_SG_St20forward_iterator_tag+0xa75) [0x2b310dd0a335] [c1-master:13894] [ 3] ../LookupPopulationLib/Release/libLookupPopulation.so(_ZN5boost9container17containers_detail25advanced_insert_aux_proxyINS0_6vectorIlNS_12interprocess9allocatorIlNS4_15segment_managerIcNS4_15rbtree_best_fitINS4_12mutex_familyENS4_10offset_ptrIvEELm0EEENS4_10iset_indexEEEEEEENS0_17constant_iteratorISF_lEEPSF_E25uninitialized_copy_all_toESI_+0x1d7) [0x2b310dd0b817] [c1-master:13894] [ 4] ../LookupPopulationLib/Release/libLookupPopulation.so(_ZN5boost9container6vectorINS1_IlNS_12interprocess9allocatorIlNS2_15segment_managerIcNS2_15rbtree_best_fitINS2_12mutex_familyENS2_10offset_ptrIvEELm0EEENS2_10iset_indexEEEEEEENS3_ISD_SB_EEE17priv_range_insertENS7_ISD_EEmRNS0_17containers_detail23advanced_insert_aux_intISD_PSD_EE+0x771) [0x2b310dd0d521] [c1-master:13894] [ 5] ../LookupPopulationLib/Release/libLookupPopulation.so(_ZN5boost12interprocess6detail8Ctor3ArgINS_9container6vectorINS4_IlNS0_9allocatorIlNS0_15segment_managerIcNS0_15rbtree_best_fitINS0_12mutex_familyENS0_10offset_ptrIvEELm0EEENS0_10iset_indexEEEEEEENS5_ISF_SD_EEEELb0EiSF_NS5_IvSD_EEE11construct_nEPvmRm+0x157) [0x2b310dd0d9a7] [c1-master:13894] [ 6] ../LookupPopulationLib/Release/libLookupPopulation.so(_ZN5boost12interprocess15segment_managerIcNS0_15rbtree_best_fitINS0_12mutex_familyENS0_10offset_ptrIvEELm0EEENS0_10iset_indexEE28priv_generic_named_constructIcEEPvmPKT_mbbRNS0_6detail18in_place_interfaceERNS7_INSE_12index_configISB_S6_EEEENSE_5bool_ILb1EEE+0x6fd) [0x2b310dd0c85d] [c1-master:13894] [ 7] ../LookupPopulationLib/Release/libLookupPopulation.so(_ZN5boost12interprocess15segment_managerIcNS0_15rbtree_best_fitINS0_12mutex_familyENS0_10offset_ptrIvEELm0EEENS0_10iset_indexEE22priv_generic_constructEPKcmbbRNS0_6detail18in_place_interfaceE+0xf8) [0x2b310dd0dd58] [c1-master:13894] [ 8] ../LookupPopulationLib/Release/libLookupPopulation.so(_ZN7POP_LTL16ExportPopulation22InitializeSharedMemoryEPKc+0x1609) [0x2b310dceea99] [c1-master:13894] [ 9] ../LookupPopulationLib/Release/libLookupPopulation.so(_ZN7POP_LTL10InitializeEPKc+0x349) [0x2b310dd0ebb9] [c1-master:13894] [10] MPI_Release/LookupPopulation.MpiLoader(main+0x372) [0x4205d2] [c1-master:13894] [11] /lib64/libc.so.6(__libc_start_main+0xf4) [0x3a6461d994] [c1-master:13894] [12] MPI_Release/LookupPopulation.MpiLoader(__gxx_personality_v0+0x239) [0x420009] [c1-master:13894] *** End of error message *** -------------------------------------------------------------------------- mpirun noticed that process rank 0 with PID 13894 on node c1-master exited on signal 7 (Bus error). --------------------------------------------------------------------------