Python Django WSGI多线程和连接问题

我按照Graham的指示使用mod_wsgi:，将Apache与Django集成，但仍然遇到连接和响应时间方面的问题。因为它是随机发生的，并且Apache日志文件中没有任何错误，所以很难理解发生了什么

我的Apache是使用pre fork构建的，配置如下：

<IfModule prefork.c>
StartServers       8
MinSpareServers    5
MaxSpareServers   20
ServerLimit      256
MaxClients       256
MaxRequestsPerChild 0
</IfModule>

我注意到，尽管配置中最多有25个线程，但活动线程数永远不会超过4个，同时一些客户端可以等待新连接超过1分钟，而请求处理时间大约为2秒

如果一个请求到达服务器，它会得到快速处理，但有时（大约每100个请求中有1个）客户端只是等待连接，甚至会因为Apache的限制而偶尔超时：

Timeout 60 

我认为，这种行为在web应用程序世界中很容易被忽视，在web应用程序世界中，每100个请求中就有1个请求不起重要作用（用户可以重新加载页面），但在服务世界中，这确实是一个问题

我无法理解这一点——如果所有线程都忙于服务于其他客户机，为什么Django不产生另一个线程呢？如果不是关于线程，那么它可能是什么？格雷厄姆写的应用程序重新加载问题

以下是我的版本：

python26-2.6.8-3.30.amzn1.x86_64
Django-1.4.3
mod_wsgi-3.2-1.6.amzn1.x86_64
Server version: Apache/2.2.23 (Unix)
Server loaded:  APR 1.4.6, APR-Util 1.4.1
Compiled using: APR 1.4.6, APR-Util 1.4.1
Architecture:   64-bit
Server MPM:     Prefork
  threaded:     no
    forked:     yes (variable process count)

===========================第一次更新已实施Graham的建议================================

格雷厄姆等人

谢谢你的意见和建议。我已经检查了mod_wsgi版本，它是3.2（见上文）。 我的WSGI配置现在如下所示：

LogLevel info
LoadModule wsgi_module modules/mod_wsgi.so
WSGISocketPrefix run/wsgix
WSGIDaemonProcess somename user=apache group=apache threads=25
WSGIScriptAlias /  /wsgi-dir/script.wsgi process-group=somename application-group=%{GLOBAL}

<Directory /wsgi-dir>
   Order deny,allow
   Allow from all
</Directory>

注意：记录器也会记录时间

创建一个简单的stat接口（我不想分析所有客户端EC2上的日志）：

def日志（请求）：

客户机的工作方式如下：

@author: ogryb
'''
import requests
import datetime
import socket

from optparse import OptionParser
usage = "usage: %prog [options] init_url\n   init_url - http://<host>/init/ server's address"
parser = OptionParser(usage=usage)
parser.add_option("-i", "--id", dest="id",
                  help="instance ID", metavar="STRING")
parser.add_option("-p", "--phost", dest="phost",
                  help="public hostname", metavar="STRING")
parser.add_option("-l", "--lhost", dest="lhost",
                  help="local hostname", metavar="STRING")
parser.add_option("-t", "--type", dest="type",
                  help="instance type", metavar="STRING")
parser.add_option("-q", "--quiet",
                  action="store_true", dest="quiet", default=False,
                  help="Quiet mode")
(opt, args) = parser.parse_args()
ip = socket.gethostbyname(socket.gethostname())
if (not opt.quiet):
    print ("=== Getting metadata:\t{0} {1}".format(datetime.datetime.utcnow(), ip))
if not opt.id:
    r = requests.get(url='http://169.254.169.254/latest/meta-data/instance-id')
    opt.id = r.text
if not opt.phost:
    r = requests.get(url='http://169.254.169.254/latest/meta-data/public-hostname')
    opt.phost = r.text
if not opt.lhost:
    r = requests.get(url='http://169.254.169.254/latest/meta-data/local-hostname')
    opt.lhost = r.text
if not opt.type:
    r = requests.get(url='http://169.254.169.254/latest/meta-data/instance-type')
    opt.type = r.text
body = "id={0}&phost={1}&lhost={2}&type={3}".format(opt.id, opt.phost, opt.lhost, opt.type)
if (not opt.quiet):
    print ("=== Start sending:\t{0} {1} {2}".format(datetime.datetime.utcnow(), ip, opt.id))
r = requests.post(url=args[0], data=body, verify=False)
if (not opt.quiet):
    print ("=== End sending:\t{0} {1} {2}".format(datetime.datetime.utcnow(), ip, opt.id))
print r.text
if (not opt.quiet):
    print "Request Body={0} url={1}".format(body,args[0])
    print "Response: {0}\n{1}".format(r.status_code, r.text)

向“侦听器”URL发送两个请求，并计算客户端上的处理时间

将处理时间和EC2实例id一起发送到“日志”URL

如果我能够用这种简单的方法重现这个问题，它将变得可重现，我希望Django团队能够从中吸取教训

任何其他建议也将不胜感激。非常感谢所有回答的人

============================建议测试的第二次更新================================

我已经实现了建议的侦听器，可以重现这个问题，并希望其他人也能这样做-你只需要一个AWS帐户就可以启动大量EC2客户端-50个通常就足够了，但有时我需要到100个查看延迟

有趣的是，在这个测试中，活动线程的数量从1逐渐增加到8，这可能是因为服务器上的平均处理时间增加了，所以它确实可以工作，但仍然不足以防止延迟

我将客户机的脚本放到EC2的用户数据中，如下所示。如果您需要一个快速的指导，如何创建一个自动与所有这些客户群SCLAIN组，请让我知道

#!/bin/bash

do_send() {

        d1=`date +%s`
        res=`python ~ec2-user/client/fetch.py ${URL_ROOT}/init/`
        res=`echo $res | tr '\n' ' ' | tr ' ' +`
        d2=`date +%s`
        delta=`expr $d2 - $d1`
        echo $ami $ins $res $delta >>$LOG
        curl -s  "${URL_ROOT}/stat/?id=$ami&time=$delta&res=$ins:$res" >/dev/null 2>&1
}

URL_ROOT=<SERVICE-ROOT_URL>
LOG=~ec2-user/log.txt

ins=`curl -s http://169.254.169.254/latest/meta-data/instance-id 2>/dev/null`
ami=`curl -s http://169.254.169.254/latest/meta-data/ami-id 2>/dev/null`
echo "Instance=[$ins]" >$LOG

# First request
do_send

# Second request
do_send

完整的错误日志如下所示：

Wed Mar 20 06:29:45 2013] [info] Server built: Oct 21 2012 20:35:32
[Wed Mar 20 06:29:45 2013] [debug] prefork.c(1023): AcceptMutex: sysvsem (default: sysvsem)
[Wed Mar 20 06:29:45 2013] [info] mod_wsgi (pid=26891): Attach interpreter ''.
[Wed Mar 20 06:29:45 2013] [info] mod_wsgi (pid=26892): Attach interpreter ''.
[Wed Mar 20 06:29:45 2013] [info] mod_wsgi (pid=26893): Attach interpreter ''.
[Wed Mar 20 06:29:45 2013] [info] mod_wsgi (pid=26895): Attach interpreter ''.
[Wed Mar 20 06:29:45 2013] [info] mod_wsgi (pid=26894): Attach interpreter ''.
[Wed Mar 20 06:37:15 2013] [debug] proxy_util.c(1820): proxy: grabbed scoreboard slot 0 in child 27005 for worker proxy:reverse
[Wed Mar 20 06:37:15 2013] [debug] proxy_util.c(1839): proxy: worker proxy:reverse already initialized
[Wed Mar 20 06:37:15 2013] [debug] proxy_util.c(1936): proxy: initialized single connection worker 0 in child 27005 for (*)
[Wed Mar 20 06:37:15 2013] [info] mod_wsgi (pid=27005): Initializing Python.
[Wed Mar 20 06:37:15 2013] [info] mod_wsgi (pid=27005): Attach interpreter ''.
[Wed Mar 20 06:38:10 2013] [debug] proxy_util.c(1820): proxy: grabbed scoreboard slot 0 in child 27006 for worker proxy:reverse
[Wed Mar 20 06:38:10 2013] [debug] proxy_util.c(1839): proxy: worker proxy:reverse already initialized
[Wed Mar 20 06:38:10 2013] [debug] proxy_util.c(1936): proxy: initialized single connection worker 0 in child 27006 for (*)
[Wed Mar 20 06:38:10 2013] [info] mod_wsgi (pid=27006): Initializing Python.
[Wed Mar 20 06:38:10 2013] [info] mod_wsgi (pid=27006): Attach interpreter ''.
[Wed Mar 20 06:38:11 2013] [info] mod_wsgi (pid=26874): Destroying interpreters.
[Wed Mar 20 06:38:11 2013] [info] mod_wsgi (pid=26874): Cleanup interpreter ''.
[Wed Mar 20 06:38:11 2013] [info] mod_wsgi (pid=26874): Terminating Python.
[Wed Mar 20 06:38:11 2013] [error] Exception KeyError: KeyError(140627014572000,) in <module 'threading' from '/usr/lib64/python2.6/threading.pyc'> ignored
[Wed Mar 20 06:38:11 2013] [info] mod_wsgi (pid=26874): Python has shutdown.
[Wed Mar 20 06:38:44 2013] [debug] proxy_util.c(1820): proxy: grabbed scoreboard slot 0 in child 27007 for worker proxy:reverse
[Wed Mar 20 06:38:44 2013] [debug] proxy_util.c(1839): proxy: worker proxy:reverse already initialized
[Wed Mar 20 06:38:44 2013] [debug] proxy_util.c(1936): proxy: initialized single connection worker 0 in child 27007 for (*)
[Wed Mar 20 06:38:44 2013] [info] mod_wsgi (pid=27007): Initializing Python.
[Wed Mar 20 06:38:44 2013] [info] mod_wsgi (pid=27007): Attach interpreter ''.
[Wed Mar 20 06:38:45 2013] [info] mod_wsgi (pid=26880): Destroying interpreters.
[Wed Mar 20 06:38:45 2013] [info] mod_wsgi (pid=26880): Cleanup interpreter ''.
[Wed Mar 20 06:38:45 2013] [info] mod_wsgi (pid=26880): Terminating Python.
[Wed Mar 20 06:38:45 2013] [error] Exception KeyError: KeyError(140627014572000,) in <module 'threading' from '/usr/lib64/python2.6/threading.pyc'> ignored
[Wed Mar 20 06:38:45 2013] [info] mod_wsgi (pid=26880): Python has shutdown.

Wed Mar 20 06:29:45 2013][info]服务器构建时间：2012年10月21日20:35:32
[Wed Mar 20 06:29:45 2013][debug]prefork.c（1023）：AcceptMutex:sysvsem（默认值：sysvsem）
[Wed Mar 20 06:29:45 2013][info]mod_wsgi（pid=26891）：附加解释器“”。
[Wed Mar 20 06:29:45 2013][info]mod_wsgi（pid=26892）：附加解释器“”。
[Wed Mar 20 06:29:45 2013][info]mod_wsgi（pid=26893）：附加解释器“”。
[Wed Mar 20 06:29:45 2013][info]mod_wsgi（pid=26895）：附加解释器“”。
[Wed Mar 20 06:29:45 2013][info]mod_wsgi（pid=26894）：附加解释器“”。
[Wed Mar 20 06:37:15 2013][debug]proxy_util.c（1820）：proxy:为worker proxy在child 27005中抓取记分板槽0:反向
[Wed Mar 20 06:37:15 2013][debug]proxy_util.c（1839）：proxy:worker proxy:reverse已初始化
[Wed Mar 20 06:37:15 2013][debug]proxy_util.c（1936）：proxy:在子27005中为（*）初始化了单连接工作进程0
[Wed Mar 20 06:37:15 2013][info]mod_wsgi（pid=27005）：初始化Python。
[Wed Mar 20 06:37:15 2013][info]mod_wsgi（pid=27005）：附加解释器“”。
[Wed Mar 20 06:38:10 2013][debug]proxy_util.c（1820）：proxy:为worker proxy在child 27006中抓取记分板槽0:反向
[Wed Mar 20 06:38:10 2013][debug]proxy_util.c（1839）：proxy:worker proxy:reverse已初始化
[Wed Mar 20 06:38:10 2013][debug]proxy_util.c（1936）：代理：在子27006中为（*）初始化了单连接工作进程0
[Wed Mar 20 06:38:10 2013][info]mod_wsgi（pid=27006）：初始化Python。
[Wed Mar 20 06:38:10 2013][info]mod_wsgi（pid=27006）：附加解释器“”。
[Wed Mar 20 06:38:11 2013][info]mod_wsgi（pid=26874）：销毁口译员。
[Wed Mar 20 06:38:11 2013][info]mod_wsgi（pid=26874）：清理解释器“”。
[Wed Mar 20 06:38:11 2013][info]mod_wsgi（pid=26874）：终止Python。
[Wed Mar 20 06:38:11 2013][error]异常KeyError:KeyError（140627014572000，）被忽略
[Wed Mar 20 06:38:11 2013][info]mod_wsgi（pid=26874）：Python已关闭。
[Wed Mar 20 06:38:44 2013][debug]proxy_util.c（1820）：proxy:为worker proxy在子级27007中抓取记分板插槽0:反向
[Wed Mar 20 06:38:44 2013][debug]proxy_util.c（1839）：proxy:worker proxy:reverse已初始化
[Wed Mar 20 06:38:44 2013][debug]proxy_util.c（1936）：proxy:在子27007中为（*）初始化了单连接辅助进程0
[Wed Mar 20 06:38:44 2013][info]mod_wsgi（pid=27007）：初始化Python。
[Wed Mar 20 06:38:44 2013][info]mod_wsgi（pid=27007）：附加解释器“”。
[Wed Mar 20 06:38:45 2013][info]mod_wsgi（pid=26880）：销毁口译员。
[Wed Mar 20 06:38:45 2013][info]mod_wsgi（pid=26880）：清理解释器“”。
[Wed Mar 20 06:38:45 2013][info]mod_wsgi（pid=26880）：终止Python。
[Wed Mar 20 06:38:45 2013][error]异常键错误：忽略中的键错误（140627014572000，）
id = request.REQUEST['id']
time = request.REQUEST['time']
res = request.REQUEST['res']

if (id and time):
    logger.info("STAT Instance: {0}, Processing time: {1}, Response: {2}".format(id,time,res))

return HttpResponse('OK')

#!/bin/bash

do_send() {

        d1=`date +%s`
        res=`python ~ec2-user/client/fetch.py ${URL_ROOT}/init/`
        res=`echo $res | tr '\n' ' ' | tr ' ' +`
        d2=`date +%s`
        delta=`expr $d2 - $d1`
        echo $ami $ins $res $delta >>$LOG
        curl -s  "${URL_ROOT}/stat/?id=$ami&time=$delta&res=$ins:$res" >/dev/null 2>&1
}

URL_ROOT=<SERVICE-ROOT_URL>
LOG=~ec2-user/log.txt

ins=`curl -s http://169.254.169.254/latest/meta-data/instance-id 2>/dev/null`
ami=`curl -s http://169.254.169.254/latest/meta-data/ami-id 2>/dev/null`
echo "Instance=[$ins]" >$LOG

# First request
do_send

# Second request
do_send

@author: ogryb
'''
import requests
import datetime
import socket

from optparse import OptionParser
usage = "usage: %prog [options] init_url\n   init_url - http://<host>/init/ server's address"
parser = OptionParser(usage=usage)
parser.add_option("-i", "--id", dest="id",
                  help="instance ID", metavar="STRING")
parser.add_option("-p", "--phost", dest="phost",
                  help="public hostname", metavar="STRING")
parser.add_option("-l", "--lhost", dest="lhost",
                  help="local hostname", metavar="STRING")
parser.add_option("-t", "--type", dest="type",
                  help="instance type", metavar="STRING")
parser.add_option("-q", "--quiet",
                  action="store_true", dest="quiet", default=False,
                  help="Quiet mode")
(opt, args) = parser.parse_args()
ip = socket.gethostbyname(socket.gethostname())
if (not opt.quiet):
    print ("=== Getting metadata:\t{0} {1}".format(datetime.datetime.utcnow(), ip))
if not opt.id:
    r = requests.get(url='http://169.254.169.254/latest/meta-data/instance-id')
    opt.id = r.text
if not opt.phost:
    r = requests.get(url='http://169.254.169.254/latest/meta-data/public-hostname')
    opt.phost = r.text
if not opt.lhost:
    r = requests.get(url='http://169.254.169.254/latest/meta-data/local-hostname')
    opt.lhost = r.text
if not opt.type:
    r = requests.get(url='http://169.254.169.254/latest/meta-data/instance-type')
    opt.type = r.text
body = "id={0}&phost={1}&lhost={2}&type={3}".format(opt.id, opt.phost, opt.lhost, opt.type)
if (not opt.quiet):
    print ("=== Start sending:\t{0} {1} {2}".format(datetime.datetime.utcnow(), ip, opt.id))
r = requests.post(url=args[0], data=body, verify=False)
if (not opt.quiet):
    print ("=== End sending:\t{0} {1} {2}".format(datetime.datetime.utcnow(), ip, opt.id))
print r.text
if (not opt.quiet):
    print "Request Body={0} url={1}".format(body,args[0])
    print "Response: {0}\n{1}".format(r.status_code, r.text)

Wed Mar 20 06:37:15 2013] [info] mod_wsgi (pid=27005): Initializing Python.
[Wed Mar 20 06:37:15 2013] [info] mod_wsgi (pid=27005): Attach interpreter '

Wed Mar 20 06:29:45 2013] [info] Server built: Oct 21 2012 20:35:32
[Wed Mar 20 06:29:45 2013] [debug] prefork.c(1023): AcceptMutex: sysvsem (default: sysvsem)
[Wed Mar 20 06:29:45 2013] [info] mod_wsgi (pid=26891): Attach interpreter ''.
[Wed Mar 20 06:29:45 2013] [info] mod_wsgi (pid=26892): Attach interpreter ''.
[Wed Mar 20 06:29:45 2013] [info] mod_wsgi (pid=26893): Attach interpreter ''.
[Wed Mar 20 06:29:45 2013] [info] mod_wsgi (pid=26895): Attach interpreter ''.
[Wed Mar 20 06:29:45 2013] [info] mod_wsgi (pid=26894): Attach interpreter ''.
[Wed Mar 20 06:37:15 2013] [debug] proxy_util.c(1820): proxy: grabbed scoreboard slot 0 in child 27005 for worker proxy:reverse
[Wed Mar 20 06:37:15 2013] [debug] proxy_util.c(1839): proxy: worker proxy:reverse already initialized
[Wed Mar 20 06:37:15 2013] [debug] proxy_util.c(1936): proxy: initialized single connection worker 0 in child 27005 for (*)
[Wed Mar 20 06:37:15 2013] [info] mod_wsgi (pid=27005): Initializing Python.
[Wed Mar 20 06:37:15 2013] [info] mod_wsgi (pid=27005): Attach interpreter ''.
[Wed Mar 20 06:38:10 2013] [debug] proxy_util.c(1820): proxy: grabbed scoreboard slot 0 in child 27006 for worker proxy:reverse
[Wed Mar 20 06:38:10 2013] [debug] proxy_util.c(1839): proxy: worker proxy:reverse already initialized
[Wed Mar 20 06:38:10 2013] [debug] proxy_util.c(1936): proxy: initialized single connection worker 0 in child 27006 for (*)
[Wed Mar 20 06:38:10 2013] [info] mod_wsgi (pid=27006): Initializing Python.
[Wed Mar 20 06:38:10 2013] [info] mod_wsgi (pid=27006): Attach interpreter ''.
[Wed Mar 20 06:38:11 2013] [info] mod_wsgi (pid=26874): Destroying interpreters.
[Wed Mar 20 06:38:11 2013] [info] mod_wsgi (pid=26874): Cleanup interpreter ''.
[Wed Mar 20 06:38:11 2013] [info] mod_wsgi (pid=26874): Terminating Python.
[Wed Mar 20 06:38:11 2013] [error] Exception KeyError: KeyError(140627014572000,) in <module 'threading' from '/usr/lib64/python2.6/threading.pyc'> ignored
[Wed Mar 20 06:38:11 2013] [info] mod_wsgi (pid=26874): Python has shutdown.
[Wed Mar 20 06:38:44 2013] [debug] proxy_util.c(1820): proxy: grabbed scoreboard slot 0 in child 27007 for worker proxy:reverse
[Wed Mar 20 06:38:44 2013] [debug] proxy_util.c(1839): proxy: worker proxy:reverse already initialized
[Wed Mar 20 06:38:44 2013] [debug] proxy_util.c(1936): proxy: initialized single connection worker 0 in child 27007 for (*)
[Wed Mar 20 06:38:44 2013] [info] mod_wsgi (pid=27007): Initializing Python.
[Wed Mar 20 06:38:44 2013] [info] mod_wsgi (pid=27007): Attach interpreter ''.
[Wed Mar 20 06:38:45 2013] [info] mod_wsgi (pid=26880): Destroying interpreters.
[Wed Mar 20 06:38:45 2013] [info] mod_wsgi (pid=26880): Cleanup interpreter ''.
[Wed Mar 20 06:38:45 2013] [info] mod_wsgi (pid=26880): Terminating Python.
[Wed Mar 20 06:38:45 2013] [error] Exception KeyError: KeyError(140627014572000,) in <module 'threading' from '/usr/lib64/python2.6/threading.pyc'> ignored
[Wed Mar 20 06:38:45 2013] [info] mod_wsgi (pid=26880): Python has shutdown.

<Directory /wsgi-dir/script.wsgi>
   Order deny,allow
   Allow from all
   WSGIProcessGroup  somename
</Directory>

<Directory /wsgi-dir>
   Order deny,allow
   Allow from all
   WSGIProcessGroup  somename
   WSGIApplicationGroup %{GLOBAL}
</Directory>

WSGIScriptAlias / /wsgi-dir/script.wsgi process-group=somename application-group=%{GLOBAL}

<Directory /wsgi-dir>
   Order deny,allow
   Allow from all
</Directory>