Python 子流程'；当Popen出错时，s Popen关闭另一个线程中使用的stdout/stderr文件描述符

当我们从Python2.7.3升级到Python2.7.5时，一个大量使用subprocess.Popen（）的内部库的自动测试开始失败。此库在线程环境中使用。在调试了这个问题之后，我能够创建一个简短的Python脚本来演示失败测试中出现的错误

这是脚本（称为“threadedsubprocess.py”）：

注意：当从Python2.7.3运行时，此脚本不会以IOError退出。当从Python2.7.5（都在同一个Ubuntu12.04 64位虚拟机上）运行时，它至少有50%的失败率

Python 2.7.5中出现的错误如下：

/opt/python/2.7.5/bin/python ./threadedsubprocess.py 
main thread is: 139899583563520
failing command: [Errno 2] No such file or directory 139899583563520
Exception in thread Thread-1:
Traceback (most recent call last):
  File "/opt/python/2.7.5/lib/python2.7/threading.py", line 808, in __bootstrap_inner
    self.run()
  File "/opt/python/2.7.5/lib/python2.7/threading.py", line 761, in run
    self.__target(*self.__args, **self.__kwargs)
  File "./threadedsubprocess.py", line 13, in subprocesscall
    out, err = p.communicate()
  File "/opt/python/2.7.5/lib/python2.7/subprocess.py", line 806, in communicate
    return self._communicate(input)
  File "/opt/python/2.7.5/lib/python2.7/subprocess.py", line 1379, in _communicate
    self.stdin.close()
IOError: [Errno 9] Bad file descriptor

close failed in file object destructor:
IOError: [Errno 9] Bad file descriptor

在比较Python2.7.3和Python2.7.5的子流程模块时，我看到Popen（）的_init___;（）调用现在确实显式地关闭了stdin、stdout和stderr文件描述符，以防执行命令失败。这似乎是在Python2.7.4中应用的旨在防止泄漏文件描述符（）的修复程序

Python2.7.3和Python2.7.5之间似乎与此问题相关的区别在于Popen _init__;（）中：

我想我有三个问题：

1） 在线程化环境中，是否可以使用subprocess.Popen和stdin、stdout和stderr的管道

2） 当Popen（）在其中一个线程中失败时，如何防止stdin、stdout和stderr的文件描述符关闭

---

3） 我做错什么了吗

我想用以下方式回答您的问题：

对

你不应该这么做

没有

Python 2.7.4中也确实出现了此错误

我认为这是库代码中的一个bug。如果在程序中添加锁，并确保对

subprocess.Popen

的两个调用都是原子执行的，则不会发生错误

@@ -1,32 +1,40 @@
 import time
 import threading
 import subprocess

+lock = threading.Lock()
+
 def subprocesscall():
+    lock.acquire()
     p = subprocess.Popen(
         ['ls', '-l'],
         stdin=subprocess.PIPE,
         stdout=subprocess.PIPE,
         stderr=subprocess.PIPE,
         )
+    lock.release()
     time.sleep(2) # simulate the Popen call takes some time to complete.
     out, err = p.communicate()
     print 'succeeding command in thread:', threading.current_thread().ident

 def failingsubprocesscall():
     try:
+        lock.acquire()
         p = subprocess.Popen(
             ['thiscommandsurelydoesnotexist'],
             stdin=subprocess.PIPE,
             stdout=subprocess.PIPE,
             stderr=subprocess.PIPE,
             )
     except Exception as e:
         print 'failing command:', e, 'in thread:', threading.current_thread().ident
+    finally:
+        lock.release()
+

 print 'main thread is:', threading.current_thread().ident

 subprocesscall_thread = threading.Thread(target=subprocesscall)
 subprocesscall_thread.start()
 failingsubprocesscall()
 subprocesscall_thread.join()

这意味着这很可能是由于

Popen

实现中的一些数据竞争造成的。我冒着一个猜测的风险：这个bug可能在

pipe\u cloexec

的实现中，由

\u get\u handles

调用，它（在2.7.4中）是：

---

评论明确警告说它不是原子的。。。这肯定会导致数据竞争，但如果没有实验，我不知道这是否是导致问题的原因。

如果您没有处理打开的文件（例如，在构建API时），其他解决方案

我通过调用Windell API找到了解决问题的方法，将所有已打开的文件描述符标记为“不可继承”。这有点像黑客，这里有问答：

它将绕过Python2.7错误

---

另一个解决方案是使用Python 3.4+：）它已经被修复了

谢谢。如果您的评估确实正确，这意味着我将为Python2.7.5提交一份bugreport，并希望以某种方式获得另一个bugfixrelease。您认为我最初的问题是否足够提供这样一个bug报告的信息？这个bug报告已经被一些python核心开发人员收集到，因此我断定它确实是python中的一个bug。这就完成了对我的问题的回答。谢谢。很酷的一点是，这在Python3.4中是固定的+

@@ -671,12 +702,33 @@
          c2pread, c2pwrite,
          errread, errwrite) = self._get_handles(stdin, stdout, stderr)

-        self._execute_child(args, executable, preexec_fn, close_fds,
-                            cwd, env, universal_newlines,
-                            startupinfo, creationflags, shell,
-                            p2cread, p2cwrite,
-                            c2pread, c2pwrite,
-                            errread, errwrite)
+        try:
+            self._execute_child(args, executable, preexec_fn, close_fds,
+                                cwd, env, universal_newlines,
+                                startupinfo, creationflags, shell,
+                                p2cread, p2cwrite,
+                                c2pread, c2pwrite,
+                                errread, errwrite)
+        except Exception:
+            # Preserve original exception in case os.close raises.
+            exc_type, exc_value, exc_trace = sys.exc_info()
+
+            to_close = []
+            # Only close the pipes we created.
+            if stdin == PIPE:
+                to_close.extend((p2cread, p2cwrite))
+            if stdout == PIPE:
+                to_close.extend((c2pread, c2pwrite))
+            if stderr == PIPE:
+                to_close.extend((errread, errwrite))
+
+            for fd in to_close:
+                try:
+                    os.close(fd)
+                except EnvironmentError:
+                    pass
+
+            raise exc_type, exc_value, exc_trace

@@ -1,32 +1,40 @@
 import time
 import threading
 import subprocess

+lock = threading.Lock()
+
 def subprocesscall():
+    lock.acquire()
     p = subprocess.Popen(
         ['ls', '-l'],
         stdin=subprocess.PIPE,
         stdout=subprocess.PIPE,
         stderr=subprocess.PIPE,
         )
+    lock.release()
     time.sleep(2) # simulate the Popen call takes some time to complete.
     out, err = p.communicate()
     print 'succeeding command in thread:', threading.current_thread().ident

 def failingsubprocesscall():
     try:
+        lock.acquire()
         p = subprocess.Popen(
             ['thiscommandsurelydoesnotexist'],
             stdin=subprocess.PIPE,
             stdout=subprocess.PIPE,
             stderr=subprocess.PIPE,
             )
     except Exception as e:
         print 'failing command:', e, 'in thread:', threading.current_thread().ident
+    finally:
+        lock.release()
+

 print 'main thread is:', threading.current_thread().ident

 subprocesscall_thread = threading.Thread(target=subprocesscall)
 subprocesscall_thread.start()
 failingsubprocesscall()
 subprocesscall_thread.join()

def pipe_cloexec(self):
    """Create a pipe with FDs set CLOEXEC."""
    # Pipes' FDs are set CLOEXEC by default because we don't want them
    # to be inherited by other subprocesses: the CLOEXEC flag is removed
    # from the child's FDs by _dup2(), between fork() and exec().
    # This is not atomic: we would need the pipe2() syscall for that.
    r, w = os.pipe()
    self._set_cloexec_flag(r)
    self._set_cloexec_flag(w)
    return r, w