Multithreading Perl线程有时会卡住(其中一些线程正在等待futex)
我正试图运行的多线程Perl应用程序(在Redhat7.4上使用Perl5.10.1)遇到了一个问题。该问题已通过以下设置重现(设置类似于原始Perl应用程序): 有两个文件:main.pl、module1.pm main.pl:Multithreading Perl线程有时会卡住(其中一些线程正在等待futex),multithreading,perl,unix,pipe,futex,Multithreading,Perl,Unix,Pipe,Futex,我正试图运行的多线程Perl应用程序(在Redhat7.4上使用Perl5.10.1)遇到了一个问题。该问题已通过以下设置重现(设置类似于原始Perl应用程序): 有两个文件:main.pl、module1.pm main.pl: 完成一组任务的顶级脚本,作为“main.pl”调用 我无法理解为什么会发生这种情况(可能与线程::队列或管道等有关),如果有人能帮我解决这个问题,那就太好了。我最终希望能够运行所有测试,而不出现任何futex等待/挂起问题。我试过: 使用IPC::Run的超时选
- 完成一组任务的顶级脚本,作为“main.pl”调用
- 使用IPC::Run的超时选项检查控件是否返回,但此处没有运气
- 使用open3()代替IPC::Run:-问题仍然存在,并且比IPC::Run更频繁
- 当我使用Perl5.16.3时,这个问题不会出现。但是我需要使用Perl 5.10.1使原始应用程序正常工作,因此如果有人能帮助我理解如何解决这个问题,那将是非常棒的:)
- 在module1::dummy()中,如果将$loCmd更改为“sleep”,将$loArgs更改为(比如)“10”,则问题不会出现(给人的印象是它可能与管道/IO缓冲区有关)
- 被卡住的线程数可能因不同的运行而有所不同,有时所有线程都会在没有任何问题的情况下完成(可能存在争用条件)
运行['bsub'、'-q regresse'、'-R ws70'、'-K'、“$loCmd$loArgs”]
此命令的作用是什么?其他命令也有同样的问题吗?例如,尝试替换一个伪命令,如sleep
实际上,我们需要一些有意义的替换来替换bsub
命令,或者一个指向如何安装它的指针,否则代码在测试设置中不容易使用。我必须使用bsub命令在远程LSF计算设施上运行lsfScript.pl脚本。我将编辑脚本以删除此命令,并在本地计算机上运行一个更简单的命令。我已删除bsub
命令,并将其更改为一个简单的echo
命令。
# main.pl
#!/apps/perl/5.10.1/bin/perl
use strict;
use warnings;
use v5.10.1;
use threads;
use threads::shared;
use module1;
use lib "<path to Thread::Queue module>";
use Thread::Queue;
my $glNumTests = shift(); # Number of tests(jobs) to run
my $glMaxThreads = shift(); # What is the max allowed number of workers (threads) for queue mode
my $q = Thread::Queue->new(); # A new empty queue
# Worker thread
sub worker
{
# Thread will loop until no more work
while (defined(my $item = $q->dequeue())) {
# Do work on $item
sleep(5); # dummy pre-work
$item->dummy(); # Actual task that needs to be run by each thread
}
}
sub start_threads
{
my $loNumThreads = shift();
$loNumThreads = $loNumThreads > $glMaxThreads ? $glMaxThreads : $loNumThreads;
print "Creating $loNumThreads threads...";
for ( 1..$loNumThreads )
{
threads->create(\&worker);
}
print "done\n";
}
sub initialize
{
my @lotests;
my $loNumTests = shift();
for my $i (1..$loNumTests)
{
push(@lotests,"Test_".$i);
}
return \@lotests;
}
sub launchjobs
{
my @lotests = @{shift()};
my $tests = {};
# Create objects
foreach my $lotest (@lotests)
{
$tests->{$lotest}->{"obj"} = module1->new($lotest);
}
# Start the threads before adding work to the queue
start_threads(scalar(@lotests));
# Adding work to queue
foreach my $lotest (@lotests)
{
$q->enqueue($tests->{$lotest}->{"obj"});
}
# No more work to be added
$q->end();
# Wait for threads to finish
foreach my $thr ( threads -> list() )
{
$thr->join();
}
}
launchjobs(initialize($glNumTests));
# module1.pm
package module1;
use lib "<InstallationPath>/IPC-Run-20200505.0/lib";
use IPC::Run qw( run );
sub new
{
my $class = shift();
my $test = shift();
my $self = {};
$self->{"testName"} = $test;
system("\\mkdir -p test_output/$self->{testName}");
# Create new file track.log for this object
open(my $OFH,">","test_output/$self->{testName}/track.log") || die "Cannot open track.log for writing in new\n";
close($OFH);
bless($self,$class);
return $self;
}
sub logTracker
{
# Writes out the message to track.log file
my $self = shift();
my $message = shift();
open(my $OFH,">>","test_output/$self->{testName}/track.log") || die"Cannot open track.log for writing\n";
print $OFH $message;
close($OFH);
}
sub dummy
{
my $self = shift();
print "running $self->{testName}\n";
my $loCmd = "echo"; # Command to be run
my $loArgs = "This is test $self->{testName}"; # Arguments to the above command
$self->logTracker("Calling run\n");
run [$loCmd,$loArgs],'>&',"test_output/$self->{testName}/output";
$self->logTracker("run completed\n");
}
1;
strace: Process 41187 attached with 8 threads
[pid 42343] read(22, <unfinished ...>
[pid 42292] read(20, <unfinished ...>
[pid 42291] read(25, <unfinished ...>
[pid 42282] read(24, <unfinished ...>
[pid 42234] read(43, <unfinished ...>
[pid 42212] read(18, <unfinished ...>
[pid 41187] futex(0x7ff2597939d0, FUTEX_WAIT, 42212, NULL <unfinished ...>
[pid 42338] read(16,
-perl,41187 main.pl 1000 128
|-perl,42614 main.pl 1000 128
|-perl,42615 main.pl 1000 128
|-perl,42616 main.pl 1000 128
|-perl,42617 main.pl 1000 128
|-perl,42618 main.pl 1000 128
|-perl,42620 main.pl 1000 128
|-perl,42621 main.pl 1000 128
|-{perl},42212
|-{perl},42234
|-{perl},42282
|-{perl},42291
|-{perl},42292
|-{perl},42338
`-{perl},42343