Python snakemake-工作流在执行多个作业后失败

我已经准备了一个Snakemake管道（Snakemake v5.2.2，Python v3.6.5），用于分析RRBS数据（），并将其放入Singularity（v2.5.2，由Ubuntu 18.04-）中。我已经用5个缩短的FASTQ文件测试了这个工作流，效果很好

当我将我的短测试FASTQ文件的数量增加到20（我的实验分析单元）时，问题就开始了。运行多个步骤后，管道失败，出现

错误127:command not found

。此错误不特定于任何文件或规则-有时在

trim galore

规则期间崩溃，有时在

bismark

期间崩溃

经过一些调查，我设法找到了一个解决方案：为每个规则添加内存使用限制（例如

资源：32000

），并运行带有标志

——resources mem_mb=100000

的Snakemake。遗憾的是，当我开始使用全尺寸文件进行实际分析时，返回了错误。我已经尝试过我脑海中最绝望的事情：

删除snakefile中的所有mv命令，并将每个规则只保留一行shell命令

使用各种标志运行snakemake（

--等待文件，
--延迟等待30--nolock--强制使用线程--重新启动时间2
）

将管道中使用的所有程序放入奇点路径

将资源限制增加到每个规则最多50GB的RAM

在一个工作流实例中运行较少数量的示例

该错误在使用Singularity容器时出现，在我的PC和工作中的服务器上都不会出现。失败的命令在通过命令行自行运行时也能正常工作。我不使用Snakemake的群集模式

我已经尝试并检查了所有我能想到的，我已经（愚蠢地）告诉我的主管分析正在运行（它是！T_T）。我将非常感谢你的帮助。你知道发生了什么事，我如何解决这个问题吗

下面是一个完整的示例性蛇击输出，带有错误：

以下是问题的实质：

[Thu Sep  6 22:05:40 2018]
rule Bismark:
    input: Diversity/B_Control-NM_86_S1_R1_001_val_1.fq_trimmed.fq.gz, Diversity/B_Control-NM_86_S1_R3_001_val_2.fq_trimmed.fq.gz
    output: Bismark_Raw/B_Control-NM_86_S1_R1_001_val_1.fq_trimmed_bismark_bt2_pe.bam, Reports/Bismark_Raw/B_Control-NM_86_S1_R1_001_val_1.fq_trimmed_bismark_bt2_PE_report.txt
    jobid: 5
    wildcards: b=B_Control-NM_86_S1
    threads: 10
    resources: mem_mb=32000

/Bismark*/bismark --bowtie2 --genome_folder ./Genome/ -1 Diversity/B_Control-NM_86_S1_R1_001_val_1.fq_trimmed.fq.gz -2 Diversity/B_Control-NM_86_S1_R3_001_val_2.fq_trimmed.fq.gz &>> Reports/Bismark_Raw/Bismark_Log.txt || error_exit && /Bismark*/bam2nuc --genome_folder ./Genome/ B_Control-NM_86_S1_R1_001_val_1.fq_trimmed_bismark_bt2_pe.bam &>> Reports/Bismark_Raw/Bam2nuc_Log.txt || error_exit && /Bismark*/bismark2summary B_Control-NM_86_S1_R1_001_val_1.fq_trimmed_bismark_bt2_pe.bam  &>> Reports/Bismark_Raw/Bismark2sum_Log.txt && /Bismark*/bismark2report B_Control-NM_86_S1_R1_001_val_1.fq_trimmed_bismark_bt2_pe.bam  &>> Reports/Bismark_Raw/Bismark2rep_Log.txt && /Bamqc/bamqc B_Control-NM_86_S1_R1_001_val_1.fq_trimmed_bismark_bt2_pe.bam  &>> Reports/Bismark_Raw/Bamqc_Raw_Log.txt && mv B_Control-NM_86_S1_R1_001_val_1.fq_trimmed_bismark_bt2_pe.bam ./Bismark_Raw/ && mv B_Control-NM_86_S1_R1_001_val_1.fq_trimmed_bismark_bt2_PE_report.txt B_Control-NM_86_S1_R1_001_val_1.fq_trimmed_bismark_bt2_pe.nucleotide_stats.txt B_Control-NM_86_S1_R1_001_val_1.fq_trimmed_bismark_bt2_PE_report.html B_Control-NM_86_S1_R1_001_val_1.fq_trimmed_bismark_bt2_pe_bamqc.html B_Control-NM_86_S1_R1_001_val_1.fq_trimmed_bismark_bt2_pe_bamqc.zip ./Reports/Bismark_Raw/
/bin/bash: error_exit: command not found
/bin/bash: error_exit: command not found
Full Traceback (most recent call last):
  File "/usr/local/lib/python3.6/dist-packages/snakemake/executors.py", line 1313, in run_wrapper
    singularity_args, use_singularity, None, jobid, is_shell)
  File "/Analysis/Snakefile", line 218, in __rule_Bismark
  File "/usr/local/lib/python3.6/dist-packages/snakemake/shell.py", line 133, in __new__
    raise sp.CalledProcessError(retcode, cmd)
subprocess.CalledProcessError: Command ' set -euo pipefail;  /Bismark*/bismark --bowtie2 --genome_folder ./Genome/ -1 Diversity/B_Control-NM_86_S1_R1_001_val_1.fq_trimmed.fq.gz -2 Diversity/B_Control-NM_86_S1_R3_001_val_2.fq_trimmed.fq.gz &>> Reports/Bismark_Raw/Bismark_Log.txt || error_exit && /Bismark*/bam2nuc --genome_folder ./Genome/ B_Control-NM_86_S1_R1_001_val_1.fq_trimmed_bismark_bt2_pe.bam &>> Reports/Bismark_Raw/Bam2nuc_Log.txt || error_exit && /Bismark*/bismark2summary B_Control-NM_86_S1_R1_001_val_1.fq_trimmed_bismark_bt2_pe.bam  &>> Reports/Bismark_Raw/Bismark2sum_Log.txt && /Bismark*/bismark2report B_Control-NM_86_S1_R1_001_val_1.fq_trimmed_bismark_bt2_pe.bam  &>> Reports/Bismark_Raw/Bismark2rep_Log.txt && /Bamqc/bamqc B_Control-NM_86_S1_R1_001_val_1.fq_trimmed_bismark_bt2_pe.bam  &>> Reports/Bismark_Raw/Bamqc_Raw_Log.txt && mv B_Control-NM_86_S1_R1_001_val_1.fq_trimmed_bismark_bt2_pe.bam ./Bismark_Raw/ && mv B_Control-NM_86_S1_R1_001_val_1.fq_trimmed_bismark_bt2_PE_report.txt B_Control-NM_86_S1_R1_001_val_1.fq_trimmed_bismark_bt2_pe.nucleotide_stats.txt B_Control-NM_86_S1_R1_001_val_1.fq_trimmed_bismark_bt2_PE_report.html B_Control-NM_86_S1_R1_001_val_1.fq_trimmed_bismark_bt2_pe_bamqc.html B_Control-NM_86_S1_R1_001_val_1.fq_trimmed_bismark_bt2_pe_bamqc.zip ./Reports/Bismark_Raw/ ' returned non-zero exit status 127.

    [Thu Sep  6 22:05:41 2018]
    Error in rule Bismark:
        jobid: 5
        output: Bismark_Raw/B_Control-NM_86_S1_R1_001_val_1.fq_trimmed_bismark_bt2_pe.bam, Reports/Bismark_Raw/B_Control-NM_86_S1_R1_001_val_1.fq_trimmed_bismark_bt2_PE_report.txt

Full Traceback (most recent call last):
  File "/usr/local/lib/python3.6/dist-packages/snakemake/executors.py", line 1313, in run_wrapper
    singularity_args, use_singularity, None, jobid, is_shell)
  File "/Analysis/Snakefile", line 218, in __rule_Bismark
  File "/usr/local/lib/python3.6/dist-packages/snakemake/shell.py", line 133, in __new__
    raise sp.CalledProcessError(retcode, cmd)
subprocess.CalledProcessError: Command ' set -euo pipefail;  /Bismark*/bismark --bowtie2 --genome_folder ./Genome/ -1 Diversity/B_Control-NM_86_S1_R1_001_val_1.fq_trimmed.fq.gz -2 Diversity/B_Control-NM_86_S1_R3_001_val_2.fq_trimmed.fq.gz &>> Reports/Bismark_Raw/Bismark_Log.txt || error_exit && /Bismark*/bam2nuc --genome_folder ./Genome/ B_Control-NM_86_S1_R1_001_val_1.fq_trimmed_bismark_bt2_pe.bam &>> Reports/Bismark_Raw/Bam2nuc_Log.txt || error_exit && /Bismark*/bismark2summary B_Control-NM_86_S1_R1_001_val_1.fq_trimmed_bismark_bt2_pe.bam  &>> Reports/Bismark_Raw/Bismark2sum_Log.txt && /Bismark*/bismark2report B_Control-NM_86_S1_R1_001_val_1.fq_trimmed_bismark_bt2_pe.bam  &>> Reports/Bismark_Raw/Bismark2rep_Log.txt && /Bamqc/bamqc B_Control-NM_86_S1_R1_001_val_1.fq_trimmed_bismark_bt2_pe.bam  &>> Reports/Bismark_Raw/Bamqc_Raw_Log.txt && mv B_Control-NM_86_S1_R1_001_val_1.fq_trimmed_bismark_bt2_pe.bam ./Bismark_Raw/ && mv B_Control-NM_86_S1_R1_001_val_1.fq_trimmed_bismark_bt2_PE_report.txt B_Control-NM_86_S1_R1_001_val_1.fq_trimmed_bismark_bt2_pe.nucleotide_stats.txt B_Control-NM_86_S1_R1_001_val_1.fq_trimmed_bismark_bt2_PE_report.html B_Control-NM_86_S1_R1_001_val_1.fq_trimmed_bismark_bt2_pe_bamqc.html B_Control-NM_86_S1_R1_001_val_1.fq_trimmed_bismark_bt2_pe_bamqc.zip ./Reports/Bismark_Raw/ ' returned non-zero exit status 127.

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/local/lib/python3.6/dist-packages/snakemake/executors.py", line 346, in _callback
    raise ex
  File "/usr/lib/python3.6/concurrent/futures/thread.py", line 56, in run
    result = self.fn(*self.args, **self.kwargs)
  File "/usr/local/lib/python3.6/dist-packages/snakemake/executors.py", line 1325, in run_wrapper
    show_traceback=True))
snakemake.exceptions.RuleException: CalledProcessError in line 119 of /Analysis/Snakefile:
Command ' set -euo pipefail;  /Bismark*/bismark --bowtie2 --genome_folder ./Genome/ -1 Diversity/B_Control-NM_86_S1_R1_001_val_1.fq_trimmed.fq.gz -2 Diversity/B_Control-NM_86_S1_R3_001_val_2.fq_trimmed.fq.gz &>> Reports/Bismark_Raw/Bismark_Log.txt || error_exit && /Bismark*/bam2nuc --genome_folder ./Genome/ B_Control-NM_86_S1_R1_001_val_1.fq_trimmed_bismark_bt2_pe.bam &>> Reports/Bismark_Raw/Bam2nuc_Log.txt || error_exit && /Bismark*/bismark2summary B_Control-NM_86_S1_R1_001_val_1.fq_trimmed_bismark_bt2_pe.bam  &>> Reports/Bismark_Raw/Bismark2sum_Log.txt && /Bismark*/bismark2report B_Control-NM_86_S1_R1_001_val_1.fq_trimmed_bismark_bt2_pe.bam  &>> Reports/Bismark_Raw/Bismark2rep_Log.txt && /Bamqc/bamqc B_Control-NM_86_S1_R1_001_val_1.fq_trimmed_bismark_bt2_pe.bam  &>> Reports/Bismark_Raw/Bamqc_Raw_Log.txt && mv B_Control-NM_86_S1_R1_001_val_1.fq_trimmed_bismark_bt2_pe.bam ./Bismark_Raw/ && mv B_Control-NM_86_S1_R1_001_val_1.fq_trimmed_bismark_bt2_PE_report.txt B_Control-NM_86_S1_R1_001_val_1.fq_trimmed_bismark_bt2_pe.nucleotide_stats.txt B_Control-NM_86_S1_R1_001_val_1.fq_trimmed_bismark_bt2_PE_report.html B_Control-NM_86_S1_R1_001_val_1.fq_trimmed_bismark_bt2_pe_bamqc.html B_Control-NM_86_S1_R1_001_val_1.fq_trimmed_bismark_bt2_pe_bamqc.zip ./Reports/Bismark_Raw/ ' returned non-zero exit status 127.
  File "/Analysis/Snakefile", line 119, in __rule_Bismark

RuleException:
CalledProcessError in line 119 of /Analysis/Snakefile:
Command ' set -euo pipefail;  /Bismark*/bismark --bowtie2 --genome_folder ./Genome/ -1 Diversity/B_Control-NM_86_S1_R1_001_val_1.fq_trimmed.fq.gz -2 Diversity/B_Control-NM_86_S1_R3_001_val_2.fq_trimmed.fq.gz &>> Reports/Bismark_Raw/Bismark_Log.txt || error_exit && /Bismark*/bam2nuc --genome_folder ./Genome/ B_Control-NM_86_S1_R1_001_val_1.fq_trimmed_bismark_bt2_pe.bam &>> Reports/Bismark_Raw/Bam2nuc_Log.txt || error_exit && /Bismark*/bismark2summary B_Control-NM_86_S1_R1_001_val_1.fq_trimmed_bismark_bt2_pe.bam  &>> Reports/Bismark_Raw/Bismark2sum_Log.txt && /Bismark*/bismark2report B_Control-NM_86_S1_R1_001_val_1.fq_trimmed_bismark_bt2_pe.bam  &>> Reports/Bismark_Raw/Bismark2rep_Log.txt && /Bamqc/bamqc B_Control-NM_86_S1_R1_001_val_1.fq_trimmed_bismark_bt2_pe.bam  &>> Reports/Bismark_Raw/Bamqc_Raw_Log.txt && mv B_Control-NM_86_S1_R1_001_val_1.fq_trimmed_bismark_bt2_pe.bam ./Bismark_Raw/ && mv B_Control-NM_86_S1_R1_001_val_1.fq_trimmed_bismark_bt2_PE_report.txt B_Control-NM_86_S1_R1_001_val_1.fq_trimmed_bismark_bt2_pe.nucleotide_stats.txt B_Control-NM_86_S1_R1_001_val_1.fq_trimmed_bismark_bt2_PE_report.html B_Control-NM_86_S1_R1_001_val_1.fq_trimmed_bismark_bt2_pe_bamqc.html B_Control-NM_86_S1_R1_001_val_1.fq_trimmed_bismark_bt2_pe_bamqc.zip ./Reports/Bismark_Raw/ ' returned non-zero exit status 127.
  File "/Analysis/Snakefile", line 119, in __rule_Bismark
  File "/usr/lib/python3.6/concurrent/futures/thread.py", line 56, in run
[Thu Sep  6 22:08:09 2018]
Finished job 6.
8 of 47 steps (17%) done
[Thu Sep  6 22:58:49 2018]
Finished job 42.
9 of 47 steps (19%) done
Shutting down, this might take some time.
Exiting because a job execution failed. Look above for error message
Complete log: /Analysis/.snakemake/log/2018-09-06T191518.240813.snakemake.log
unlocking
removing lock
removing lock
removed all locks

---

编辑

---

我想我已经找到了解决这个问题的方法。不幸的是，我仍然不知道是什么导致了最初的错误。尽管如此-请不要在这个问题上浪费时间，除非其他人在下面报告…

Snakemake由于

error\u exit

命令不可用而在这里失败

error\u exit

是一个自定义函数/命令，但从未在Snake文件中定义过。定义它们。但是，退一步说，为什么需要使用

error\u exit

？另一方面，bismark在这里似乎失败了，你需要找出原因。也就是说，您的主要问题不是由于蛇咬，而是由于

bismark

。很好，谢谢！我对程序进行了大量的测试和更改，以至于我忘了将error_exit作为一个函数添加到bash中。虽然这会导致错误的编号为-127-，但这无助于解决根本问题-error_exit函数仅在发生错误时执行。尽管如此，由于我已经通过使用不同的工作流管理器解决了我的问题，我认为在这个问题上浪费别人的时间是没有意义的。。。