Linux 在mod-mono-server4下运行的应用程序的设置和调试

Linux 在mod-mono-server4下运行的应用程序的设置和调试,linux,apache,mono,redhat,mod-mono,Linux,Apache,Mono,Redhat,Mod Mono,我有一个c#应用程序(servicebus),它运行在私有web服务器上。它的基本工作是接受一些web请求并创建其他进程来处理请求中描述的数据包。处理过程通常持续进行,可能需要数周时间 servicebus偶尔会开始消耗大量CPU。也就是说,它通常是空闲的,每天有1到2秒的CPU时间。当它进入这种奇怪的模式时,它会一直消耗100%以上的CPU。此时,如果出现新请求,apache将生成一个新的servicebus实例。此时,我将有两个servicebus副本正在运行(可能两个副本都在处理请求——我

我有一个c#应用程序(servicebus),它运行在私有web服务器上。它的基本工作是接受一些web请求并创建其他进程来处理请求中描述的数据包。处理过程通常持续进行,可能需要数周时间

servicebus偶尔会开始消耗大量CPU。也就是说,它通常是空闲的,每天有1到2秒的CPU时间。当它进入这种奇怪的模式时,它会一直消耗100%以上的CPU。此时,如果出现新请求,apache将生成一个新的servicebus实例。此时,我将有两个servicebus副本正在运行(可能两个副本都在处理请求——我不知道)

这是正常过程(通过ps-aef):

如您所见,该应用程序是一个C#程序(使用VS 2010 for.NET 4编译),通过mono下的mod-mono-server4运行。这是一个redhat linux enterprise 6.5系统

运行了一段时间后,该进程“变得疯狂”,开始消耗大量CPU,mod mono服务器创建了一个新实例。正如你所看到的,我直到周一早上才找到它,因为它已经使用了超过2天的CPU时间。以下是新的
ps-aef
输出:

UID        PID  PPID  C STIME TTY          TIME CMD
apache    8978     1 83 Sep19 ?        2-08:26:25 /opt/mono/bin/mono /opt/mono/lib/mono/4.0/mod-mono-server4.exe --filename /tmp/mod_mono_server_default --applications /:/opt/ov/vespa/servicebus --nonstop
apache   32538     1  0 Sep21 ?        00:00:00 /opt/mono/bin/mono /opt/mono/lib/mono/4.0/mod-mono-server4.exe --filename /tmp/mod_mono_server_default --applications /:/opt/ov/vespa/servicebus --nonstop
如果您需要查看应用程序是如何配置的,我有应用程序conf.d文件中的代码段:

# The user and group need to be set before mod_mono.conf is loaded.
User apache
Group apache
# Service Bus setup
Include /etc/httpd/conf/mod_mono.conf
Listen 8081
<VirtualHost *:8081>
    DocumentRoot /opt/ov/vespa/servicebus
    MonoServerPath default /opt/mono/bin/mod-mono-server4
    MonoApplications "/:/opt/ov/vespa/servicebus"
    <Location "/">
        SetHandler mono
        Allow from all
    </Location>
</VirtualHost>
因此,我需要在命令行中输入一个
--debug
,并确定实际要杀死哪个PID。然后我可以使用中的技术来调试核心文件

注意:我已经尝试将MonoMaxCPUTime和MonoAutoRestartTime指令放入apache conf文件中来解决这个问题。问题是,当一切都是名义上的,他们工作得很好。一旦进入这种不良状态(消耗大量CPU),重启就会失败。或者更确切地说,它成功地创建了一个新流程,但未能删除旧流程(基本上是我已经处于的状态)

到目前为止的调试:我看到我的
PID=8979的日志文件在9月21日03:27停止。考虑到它通常会产生200%或300%或更多的CPU,这很容易导致“崩溃”。在查看apache日志时,我发现了一个不寻常的事件。日志的转储如下所示:

...
[Sun Sep 21 03:28:01 2014] [notice] SIGHUP received.  Attempting to restart
mod-mono-server received a shutdown message
httpd: Could not reliably determine the server's fully qualified domain name, using localhost.localdomain for ServerName
Stacktrace:


Native stacktrace:

    /opt/mono/bin/mono() [0x48cc26]
    /lib64/libpthread.so.0() [0x32fca0f710]
    /lib64/libpthread.so.0(pthread_cond_wait+0xcc) [0x32fca0b5bc]
    /opt/mono/bin/mono() [0x5a6a9c]
    /opt/mono/bin/mono() [0x5ad4e9]
    /opt/mono/bin/mono() [0x5116d8]
    /opt/mono/bin/mono(mono_thread_manage+0x1ad) [0x5161cd]
    /opt/mono/bin/mono(mono_main+0x1401) [0x46a671]
    /lib64/libc.so.6(__libc_start_main+0xfd) [0x32fc21ed1d]
    /opt/mono/bin/mono() [0x4123a9]

Debug info from gdb:

warning: File "/opt/mono/bin/mono-gdb.py" auto-loading has been declined by your `auto-load safe-path' set to "/usr/share/gdb/auto-load:/usr/lib/debug:/usr/bin/mono-gdb.py".
To enable execution of this file add
    add-auto-load-safe-path /opt/mono/bin/mono-gdb.py
line to your configuration file "$HOME/.gdbinit".
To completely disable this security protection add
    set auto-load safe-path /
line to your configuration file "$HOME/.gdbinit".
For more information about this security protection see the
"Auto-loading safe path" section in the GDB manual.  E.g., run from the shell:
    info "(gdb)Auto-loading safe path"
[New LWP 9148]
[New LWP 9135]
[New LWP 9000]
[New LWP 8991]
[New LWP 8990]
[New LWP 8988]
[New LWP 8987]
[New LWP 8986]
[New LWP 8985]
[New LWP 8984]
[Thread debugging using libthread_db enabled]
0x00000032fca0e75d in read () from /lib64/libpthread.so.0
  11 Thread 0x7f0d8bcaf700 (LWP 8984)  0x00000032fca0b5bc in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
  10 Thread 0x7f0d8b2ae700 (LWP 8985)  0x00000032fca0b5bc in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
  9 Thread 0x7f0d8a8ad700 (LWP 8986)  0x00000032fca0b5bc in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
  8 Thread 0x7f0d89eac700 (LWP 8987)  0x00000032fca0b5bc in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
  7 Thread 0x7f0d894ab700 (LWP 8988)  0x00000032fca0b5bc in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
  6 Thread 0x7f0d88aaa700 (LWP 8990)  0x00000032fca0b5bc in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
  5 Thread 0x7f0d880a9700 (LWP 8991)  0x00000032fca0b5bc in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
  4 Thread 0x7f0d8713c700 (LWP 9000)  0x00000032fca0d930 in sem_wait () from /lib64/libpthread.so.0
  3 Thread 0x7f0d86157700 (LWP 9135)  0x00000032fc27a983 in malloc () from /lib64/libc.so.6
  2 Thread 0x7f0d8568b700 (LWP 9148)  0x00000032fc2792f0 in _int_malloc () from /lib64/libc.so.6
* 1 Thread 0x7f0d8bcb0740 (LWP 8978)  0x00000032fca0e75d in read () from /lib64/libpthread.so.0

Thread 11 (Thread 0x7f0d8bcaf700 (LWP 8984)):
#0  0x00000032fca0b5bc in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1  0x00000000005d59f7 in GC_wait_marker ()
#2  0x00000000005dbabd in GC_help_marker ()
#3  0x00000000005d4778 in GC_mark_thread ()
#4  0x00000032fca079d1 in start_thread () from /lib64/libpthread.so.0
#5  0x00000032fc2e8b5d in clone () from /lib64/libc.so.6

Thread 10 (Thread 0x7f0d8b2ae700 (LWP 8985)):
#0  0x00000032fca0b5bc in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1  0x00000000005d59f7 in GC_wait_marker ()
#2  0x00000000005dbabd in GC_help_marker ()
#3  0x00000000005d4778 in GC_mark_thread ()
#4  0x00000032fca079d1 in start_thread () from /lib64/libpthread.so.0
#5  0x00000032fc2e8b5d in clone () from /lib64/libc.so.6

Thread 9 (Thread 0x7f0d8a8ad700 (LWP 8986)):
#0  0x00000032fca0b5bc in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1  0x00000000005d59f7 in GC_wait_marker ()
#2  0x00000000005dbabd in GC_help_marker ()
#3  0x00000000005d4778 in GC_mark_thread ()
#4  0x00000032fca079d1 in start_thread () from /lib64/libpthread.so.0
#5  0x00000032fc2e8b5d in clone () from /lib64/libc.so.6

Thread 8 (Thread 0x7f0d89eac700 (LWP 8987)):
#0  0x00000032fca0b5bc in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1  0x00000000005d59f7 in GC_wait_marker ()
#2  0x00000000005dbabd in GC_help_marker ()
#3  0x00000000005d4778 in GC_mark_thread ()
#4  0x00000032fca079d1 in start_thread () from /lib64/libpthread.so.0
#5  0x00000032fc2e8b5d in clone () from /lib64/libc.so.6

Thread 7 (Thread 0x7f0d894ab700 (LWP 8988)):
#0  0x00000032fca0b5bc in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1  0x00000000005d59f7 in GC_wait_marker ()
#2  0x00000000005dbabd in GC_help_marker ()
#3  0x00000000005d4778 in GC_mark_thread ()
#4  0x00000032fca079d1 in start_thread () from /lib64/libpthread.so.0
#5  0x00000032fc2e8b5d in clone () from /lib64/libc.so.6

Thread 6 (Thread 0x7f0d88aaa700 (LWP 8990)):
#0  0x00000032fca0b5bc in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1  0x00000000005d59f7 in GC_wait_marker ()
#2  0x00000000005dbabd in GC_help_marker ()
#3  0x00000000005d4778 in GC_mark_thread ()
#4  0x00000032fca079d1 in start_thread () from /lib64/libpthread.so.0
#5  0x00000032fc2e8b5d in clone () from /lib64/libc.so.6

Thread 5 (Thread 0x7f0d880a9700 (LWP 8991)):
#0  0x00000032fca0b5bc in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1  0x00000000005d59f7 in GC_wait_marker ()
#2  0x00000000005dbabd in GC_help_marker ()
#3  0x00000000005d4778 in GC_mark_thread ()
#4  0x00000032fca079d1 in start_thread () from /lib64/libpthread.so.0
#5  0x00000032fc2e8b5d in clone () from /lib64/libc.so.6

Thread 4 (Thread 0x7f0d8713c700 (LWP 9000)):
#0  0x00000032fca0d930 in sem_wait () from /lib64/libpthread.so.0
#1  0x00000000005bea28 in mono_sem_wait ()
#2  0x000000000053b2bb in finalizer_thread ()
#3  0x000000000051375b in start_wrapper ()
#4  0x00000000005a8214 in thread_start_routine ()
#5  0x00000000005d565a in GC_start_routine ()
#6  0x00000032fca079d1 in start_thread () from /lib64/libpthread.so.0
#7  0x00000032fc2e8b5d in clone () from /lib64/libc.so.6

Thread 3 (Thread 0x7f0d86157700 (LWP 9135)):
#0  0x00000032fc27a983 in malloc () from /lib64/libc.so.6
#1  0x00000000005cd0e6 in monoeg_malloc ()
#2  0x00000000005cbef1 in monoeg_g_hash_table_insert_replace ()
#3  0x00000000005acff5 in WaitForMultipleObjectsEx ()
#4  0x0000000000512694 in ves_icall_System_Threading_WaitHandle_WaitAny_internal ()
#5  0x00000000417b0270 in ?? ()
#6  0x00007f0d68000c21 in ?? ()
#7  0x00007f0d847c4b40 in ?? ()
#8  0x00007f0d68003e00 in ?? ()
#9  0x000000004023e890 in ?? ()
#10 0x00007f0d68003e00 in ?? ()
#11 0x00007f0d86156940 in ?? ()
#12 0x00007f0d861568a0 in ?? ()
#13 0x00007f0d8767d000 in ?? ()
#14 0xffffffffffffffff in ?? ()
#15 0x00007f0d86156cc0 in ?? ()
#16 0x00007f0d847c4b40 in ?? ()
#17 0x000000004023e268 in ?? ()
#18 0x0000000000000000 in ?? ()

Thread 2 (Thread 0x7f0d8568b700 (LWP 9148)):
#0  0x00000032fc2792f0 in _int_malloc () from /lib64/libc.so.6
#1  0x00000032fc27a636 in calloc () from /lib64/libc.so.6
#2  0x00000000005cd148 in monoeg_malloc0 ()
#3  0x00000000005cbb94 in monoeg_g_hash_table_new ()
#4  0x00000000005acf94 in WaitForMultipleObjectsEx ()
#5  0x0000000000512694 in ves_icall_System_Threading_WaitHandle_WaitAny_internal ()
#6  0x00000000417b0270 in ?? ()
#7  0x00007f0d60000c21 in ?? ()
#8  0x00007f0d8767d000 in ?? ()
#9  0xffffffffffffffff in ?? ()
#10 0x000000004023e890 in ?? ()
#11 0x00007f0d68003e00 in ?? ()
#12 0x00007f0d8568a940 in ?? ()
#13 0x00007f0d8568a8a0 in ?? ()
#14 0x00007f0d8767d000 in ?? ()
#15 0xffffffffffffffff in ?? ()
#16 0x00007f0d8568acc0 in ?? ()
#17 0x00007f0d864e2990 in ?? ()
#18 0x000000004023e268 in ?? ()
#19 0x0000000000000000 in ?? ()

Thread 1 (Thread 0x7f0d8bcb0740 (LWP 8978)):
#0  0x00000032fca0e75d in read () from /lib64/libpthread.so.0
#1  0x000000000048cdb6 in mono_handle_native_sigsegv ()
#2  <signal handler called>
#3  0x00000032fca0b5bc in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#4  0x00000000005a6a9c in _wapi_handle_timedwait_signal_handle ()
#5  0x00000000005ad4e9 in WaitForMultipleObjectsEx ()
#6  0x00000000005116d8 in wait_for_tids ()
#7  0x00000000005161cd in mono_thread_manage ()
#8  0x000000000046a671 in mono_main ()
#9  0x00000032fc21ed1d in __libc_start_main () from /lib64/libc.so.6
#10 0x00000000004123a9 in _start ()

=================================================================
Got a SIGABRT while executing native code. This usually indicates
a fatal error in the mono runtime or one of the native libraries 
used by your application.
=================================================================
。。。
[太阳2014年9月21日03:28:01][通知]已收到通知。正在尝试重新启动
mod mono服务器收到关机消息
httpd:无法使用localhost.localdomain作为ServerName可靠地确定服务器的完全限定域名
堆栈跟踪:
本机堆栈跟踪:
/opt/mono/bin/mono()[0x48cc26]
/lib64/libpthread.so.0()[0x32fca0f710]
/lib64/libpthread.so.0(pthread_cond_wait+0xcc)[0x32fca0b5bc]
/opt/mono/bin/mono()[0x5a6a9c]
/opt/mono/bin/mono()[0x5ad4e9]
/opt/mono/bin/mono()[0x5116d8]
/opt/mono/bin/mono(单线程管理+0x1ad)[0x5161cd]
/opt/mono/bin/mono(mono_main+0x1401)[0x46a671]
/lib64/libc.so.6(u libc_start_main+0xfd)[0x32fc21ed1d]
/opt/mono/bin/mono()[0x4123a9]
来自gdb的调试信息:
警告:文件“/opt/mono/bin/mono-gdb.py”自动加载已被设置为“/usr/share/gdb/auto-load:/usr/lib/debug:/usr/bin/mono-gdb.py”的“自动加载安全路径”拒绝。
要启用此文件的执行,请添加
添加自动加载安全路径/opt/mono/bin/mono-gdb.py
配置文件“$HOME/.gdbinit”的行。
要完全禁用此安全保护,请添加
设置自动加载安全路径/
配置文件“$HOME/.gdbinit”的行。
有关此安全保护的更多信息,请参阅
GDB手册中的“自动加载安全路径”部分。例如,从外壳运行:
信息“(gdb)自动加载安全路径”
[新LWP 9148]
[新LWP 9135]
[新LWP 9000]
[新LWP 8991]
[新LWP 8990]
[新LWP 8988]
[新LWP 8987]
[新LWP 8986]
[新LWP 8985]
[新LWP 8984]
[已启用使用libthread_db的线程调试]
/lib64/libpthread.so.0中读取()的0x00000032fca0e75d
11/lib64/libpthread.so.0中pthread_cond_wait@@GLIBC_2.3.2()中的线程0x7f0d8bcaf700(LWP 8984)0x00000032fca0b5bc
在/lib64/libpthread.so.0的pthread_cond_wait@@GLIBC_2.3.2()中的10个线程0x7f0d8b2ae700(LWP 8985)0x00000032fca0b5bc
9线程0x7f0d8a8ad700(LWP 8986)0x00000032fca0b5bc位于/lib64/libpthread.so.0的pthread_cond_wait@@GLIBC_2.3.2()中
在/lib64/libpthread.so.0的pthread_cond_wait@@GLIBC_2.3.2()中的8个线程0x7f0d89eac700(LWP 8987)0x00000032fca0b5bc
在/lib64/libpthread.so.0的pthread_cond_wait@@GLIBC_2.3.2()中的7个线程0x7f0d894ab700(LWP 8988)0x00000032fca0b5bc
来自/lib64/libpthread.so.0的pthread_cond_wait@@GLIBC_2.3.2()中的6个线程0x7f0d88aaa700(LWP 8990)0x00000032fca0b5bc
来自/lib64/libpthread.so.0的pthread_cond_wait@@GLIBC_2.3.2()中的5个线程0x7f0d880a9700(LWP 8991)0x00000032fca0b5bc
来自/lib64/libpthread.so.0的sem_wait()中的4个线程0x7f0d8713c700(LWP 9000)0x00000032fca0d930
来自/lib64/libc.so.6的malloc()中的3个线程0x7f0d86157700(LWP 9135)0x00000032fc27a983
2线程0x7f0d8568b700(LWP 9148)0x00000032fc2792f0位于/lib64/libc.so.6中的_int_malloc()中
*从/lib64/libpthread.so.0读取()中的1个线程0x7f0d8bcb0740(LWP 8978)0x00000032fca0e75d
螺纹11(螺纹0x7f0d8bcaf700(LWP 8984)):
#在/lib64/libpthread.so.0的pthread_cond_wait@@GLIBC_2.3.2()中0 0x00000032fca0b5bc
#GC_wait_标记()中的1 0x00000000005d59f7
#GC_帮助_标记()中的2 0x00000000005DBBD
#3 0x00000000005d4778在GC_标记_螺纹中()
#4 0x00000032fca079d1,位于/lib64/libpthread.so.0的start_线程()中
#来自/lib64/libc.so.6的克隆()中的5 0x00000032fc2e8b5d
螺纹10(螺纹0x7f0d8b2ae700(LWP 8985)):
#在/lib64/libpthread.so.0的pthread_cond_wait@@GLIBC_2.3.2()中0 0x00000032fca0b5bc
#GC_wait_标记()中的1 0x00000000005d59f7
#GC_帮助_标记()中的2 0x00000000005DBBD
#3 0x00000000005d4778在GC_标记_螺纹中()
#4 0x00000032fca079d1,位于/lib64/libpthread.so.0的start_线程()中
#来自/lib64/libc.so.6的克隆()中的5 0x00000032fc2e8b5d
线程9(线程0x7
/opt/mono/bin/mono --debug /opt/test/testapp.exe
...
[Sun Sep 21 03:28:01 2014] [notice] SIGHUP received.  Attempting to restart
mod-mono-server received a shutdown message
httpd: Could not reliably determine the server's fully qualified domain name, using localhost.localdomain for ServerName
Stacktrace:


Native stacktrace:

    /opt/mono/bin/mono() [0x48cc26]
    /lib64/libpthread.so.0() [0x32fca0f710]
    /lib64/libpthread.so.0(pthread_cond_wait+0xcc) [0x32fca0b5bc]
    /opt/mono/bin/mono() [0x5a6a9c]
    /opt/mono/bin/mono() [0x5ad4e9]
    /opt/mono/bin/mono() [0x5116d8]
    /opt/mono/bin/mono(mono_thread_manage+0x1ad) [0x5161cd]
    /opt/mono/bin/mono(mono_main+0x1401) [0x46a671]
    /lib64/libc.so.6(__libc_start_main+0xfd) [0x32fc21ed1d]
    /opt/mono/bin/mono() [0x4123a9]

Debug info from gdb:

warning: File "/opt/mono/bin/mono-gdb.py" auto-loading has been declined by your `auto-load safe-path' set to "/usr/share/gdb/auto-load:/usr/lib/debug:/usr/bin/mono-gdb.py".
To enable execution of this file add
    add-auto-load-safe-path /opt/mono/bin/mono-gdb.py
line to your configuration file "$HOME/.gdbinit".
To completely disable this security protection add
    set auto-load safe-path /
line to your configuration file "$HOME/.gdbinit".
For more information about this security protection see the
"Auto-loading safe path" section in the GDB manual.  E.g., run from the shell:
    info "(gdb)Auto-loading safe path"
[New LWP 9148]
[New LWP 9135]
[New LWP 9000]
[New LWP 8991]
[New LWP 8990]
[New LWP 8988]
[New LWP 8987]
[New LWP 8986]
[New LWP 8985]
[New LWP 8984]
[Thread debugging using libthread_db enabled]
0x00000032fca0e75d in read () from /lib64/libpthread.so.0
  11 Thread 0x7f0d8bcaf700 (LWP 8984)  0x00000032fca0b5bc in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
  10 Thread 0x7f0d8b2ae700 (LWP 8985)  0x00000032fca0b5bc in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
  9 Thread 0x7f0d8a8ad700 (LWP 8986)  0x00000032fca0b5bc in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
  8 Thread 0x7f0d89eac700 (LWP 8987)  0x00000032fca0b5bc in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
  7 Thread 0x7f0d894ab700 (LWP 8988)  0x00000032fca0b5bc in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
  6 Thread 0x7f0d88aaa700 (LWP 8990)  0x00000032fca0b5bc in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
  5 Thread 0x7f0d880a9700 (LWP 8991)  0x00000032fca0b5bc in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
  4 Thread 0x7f0d8713c700 (LWP 9000)  0x00000032fca0d930 in sem_wait () from /lib64/libpthread.so.0
  3 Thread 0x7f0d86157700 (LWP 9135)  0x00000032fc27a983 in malloc () from /lib64/libc.so.6
  2 Thread 0x7f0d8568b700 (LWP 9148)  0x00000032fc2792f0 in _int_malloc () from /lib64/libc.so.6
* 1 Thread 0x7f0d8bcb0740 (LWP 8978)  0x00000032fca0e75d in read () from /lib64/libpthread.so.0

Thread 11 (Thread 0x7f0d8bcaf700 (LWP 8984)):
#0  0x00000032fca0b5bc in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1  0x00000000005d59f7 in GC_wait_marker ()
#2  0x00000000005dbabd in GC_help_marker ()
#3  0x00000000005d4778 in GC_mark_thread ()
#4  0x00000032fca079d1 in start_thread () from /lib64/libpthread.so.0
#5  0x00000032fc2e8b5d in clone () from /lib64/libc.so.6

Thread 10 (Thread 0x7f0d8b2ae700 (LWP 8985)):
#0  0x00000032fca0b5bc in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1  0x00000000005d59f7 in GC_wait_marker ()
#2  0x00000000005dbabd in GC_help_marker ()
#3  0x00000000005d4778 in GC_mark_thread ()
#4  0x00000032fca079d1 in start_thread () from /lib64/libpthread.so.0
#5  0x00000032fc2e8b5d in clone () from /lib64/libc.so.6

Thread 9 (Thread 0x7f0d8a8ad700 (LWP 8986)):
#0  0x00000032fca0b5bc in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1  0x00000000005d59f7 in GC_wait_marker ()
#2  0x00000000005dbabd in GC_help_marker ()
#3  0x00000000005d4778 in GC_mark_thread ()
#4  0x00000032fca079d1 in start_thread () from /lib64/libpthread.so.0
#5  0x00000032fc2e8b5d in clone () from /lib64/libc.so.6

Thread 8 (Thread 0x7f0d89eac700 (LWP 8987)):
#0  0x00000032fca0b5bc in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1  0x00000000005d59f7 in GC_wait_marker ()
#2  0x00000000005dbabd in GC_help_marker ()
#3  0x00000000005d4778 in GC_mark_thread ()
#4  0x00000032fca079d1 in start_thread () from /lib64/libpthread.so.0
#5  0x00000032fc2e8b5d in clone () from /lib64/libc.so.6

Thread 7 (Thread 0x7f0d894ab700 (LWP 8988)):
#0  0x00000032fca0b5bc in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1  0x00000000005d59f7 in GC_wait_marker ()
#2  0x00000000005dbabd in GC_help_marker ()
#3  0x00000000005d4778 in GC_mark_thread ()
#4  0x00000032fca079d1 in start_thread () from /lib64/libpthread.so.0
#5  0x00000032fc2e8b5d in clone () from /lib64/libc.so.6

Thread 6 (Thread 0x7f0d88aaa700 (LWP 8990)):
#0  0x00000032fca0b5bc in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1  0x00000000005d59f7 in GC_wait_marker ()
#2  0x00000000005dbabd in GC_help_marker ()
#3  0x00000000005d4778 in GC_mark_thread ()
#4  0x00000032fca079d1 in start_thread () from /lib64/libpthread.so.0
#5  0x00000032fc2e8b5d in clone () from /lib64/libc.so.6

Thread 5 (Thread 0x7f0d880a9700 (LWP 8991)):
#0  0x00000032fca0b5bc in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1  0x00000000005d59f7 in GC_wait_marker ()
#2  0x00000000005dbabd in GC_help_marker ()
#3  0x00000000005d4778 in GC_mark_thread ()
#4  0x00000032fca079d1 in start_thread () from /lib64/libpthread.so.0
#5  0x00000032fc2e8b5d in clone () from /lib64/libc.so.6

Thread 4 (Thread 0x7f0d8713c700 (LWP 9000)):
#0  0x00000032fca0d930 in sem_wait () from /lib64/libpthread.so.0
#1  0x00000000005bea28 in mono_sem_wait ()
#2  0x000000000053b2bb in finalizer_thread ()
#3  0x000000000051375b in start_wrapper ()
#4  0x00000000005a8214 in thread_start_routine ()
#5  0x00000000005d565a in GC_start_routine ()
#6  0x00000032fca079d1 in start_thread () from /lib64/libpthread.so.0
#7  0x00000032fc2e8b5d in clone () from /lib64/libc.so.6

Thread 3 (Thread 0x7f0d86157700 (LWP 9135)):
#0  0x00000032fc27a983 in malloc () from /lib64/libc.so.6
#1  0x00000000005cd0e6 in monoeg_malloc ()
#2  0x00000000005cbef1 in monoeg_g_hash_table_insert_replace ()
#3  0x00000000005acff5 in WaitForMultipleObjectsEx ()
#4  0x0000000000512694 in ves_icall_System_Threading_WaitHandle_WaitAny_internal ()
#5  0x00000000417b0270 in ?? ()
#6  0x00007f0d68000c21 in ?? ()
#7  0x00007f0d847c4b40 in ?? ()
#8  0x00007f0d68003e00 in ?? ()
#9  0x000000004023e890 in ?? ()
#10 0x00007f0d68003e00 in ?? ()
#11 0x00007f0d86156940 in ?? ()
#12 0x00007f0d861568a0 in ?? ()
#13 0x00007f0d8767d000 in ?? ()
#14 0xffffffffffffffff in ?? ()
#15 0x00007f0d86156cc0 in ?? ()
#16 0x00007f0d847c4b40 in ?? ()
#17 0x000000004023e268 in ?? ()
#18 0x0000000000000000 in ?? ()

Thread 2 (Thread 0x7f0d8568b700 (LWP 9148)):
#0  0x00000032fc2792f0 in _int_malloc () from /lib64/libc.so.6
#1  0x00000032fc27a636 in calloc () from /lib64/libc.so.6
#2  0x00000000005cd148 in monoeg_malloc0 ()
#3  0x00000000005cbb94 in monoeg_g_hash_table_new ()
#4  0x00000000005acf94 in WaitForMultipleObjectsEx ()
#5  0x0000000000512694 in ves_icall_System_Threading_WaitHandle_WaitAny_internal ()
#6  0x00000000417b0270 in ?? ()
#7  0x00007f0d60000c21 in ?? ()
#8  0x00007f0d8767d000 in ?? ()
#9  0xffffffffffffffff in ?? ()
#10 0x000000004023e890 in ?? ()
#11 0x00007f0d68003e00 in ?? ()
#12 0x00007f0d8568a940 in ?? ()
#13 0x00007f0d8568a8a0 in ?? ()
#14 0x00007f0d8767d000 in ?? ()
#15 0xffffffffffffffff in ?? ()
#16 0x00007f0d8568acc0 in ?? ()
#17 0x00007f0d864e2990 in ?? ()
#18 0x000000004023e268 in ?? ()
#19 0x0000000000000000 in ?? ()

Thread 1 (Thread 0x7f0d8bcb0740 (LWP 8978)):
#0  0x00000032fca0e75d in read () from /lib64/libpthread.so.0
#1  0x000000000048cdb6 in mono_handle_native_sigsegv ()
#2  <signal handler called>
#3  0x00000032fca0b5bc in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#4  0x00000000005a6a9c in _wapi_handle_timedwait_signal_handle ()
#5  0x00000000005ad4e9 in WaitForMultipleObjectsEx ()
#6  0x00000000005116d8 in wait_for_tids ()
#7  0x00000000005161cd in mono_thread_manage ()
#8  0x000000000046a671 in mono_main ()
#9  0x00000032fc21ed1d in __libc_start_main () from /lib64/libc.so.6
#10 0x00000000004123a9 in _start ()

=================================================================
Got a SIGABRT while executing native code. This usually indicates
a fatal error in the mono runtime or one of the native libraries 
used by your application.
=================================================================