C Eventloop具有高ksoftirqd负载；nginx不执行相同的系统调用，但执行相同的系统调用。为什么？

我编写了一些代码，其中包含一个epoll eventloop，接受新的连接，并假装是一个http服务器。 发布的代码是绝对最小值。。。我删除了所有内容（包括所有错误检查），使其尽可能简短和切中要害：

#include <stdlib.h>
#include <stdio.h>
#include <string.h>
#include <sys/epoll.h>
#include <sys/socket.h>
#include <netinet/ip.h>
#include <netinet/in.h>
#include <sys/uio.h>
#include <unistd.h>


int main () {
    int servFd = socket (AF_INET, SOCK_STREAM | SOCK_NONBLOCK | SOCK_CLOEXEC, IPPROTO_IP);
    int value = 1;
    setsockopt (servFd, SOL_SOCKET, SO_REUSEADDR, &value, sizeof (value));

    struct sockaddr_in servAddr;
    memset (&servAddr, 0, sizeof (servAddr));
    servAddr.sin_family = AF_INET;
    servAddr.sin_addr.s_addr = 0;
    servAddr.sin_port = htons (8081);
    bind (servFd, (struct sockaddr*)&servAddr, sizeof (servAddr));
    listen (servFd, 511);

    int efd = epoll_create1 (EPOLL_CLOEXEC);
    struct epoll_event epollEvt;
    epollEvt.events = EPOLLIN | EPOLLRDHUP;
    epollEvt.data.u32 = servFd;
    epoll_ctl (efd, EPOLL_CTL_ADD, servFd, &epollEvt);

    for (;;) {
        struct epoll_event pollEvent[512];
        int eventCount = epoll_wait (efd, pollEvent, 512, -1);
        for (int i = 0; i < eventCount; ++i) {
            struct epoll_event* curEvent = &pollEvent[i];
            if (curEvent->data.u32 == servFd) {
                int clientFd = accept4 (servFd, NULL, NULL, SOCK_NONBLOCK | SOCK_CLOEXEC);
                struct epoll_event epollEvt;
                epollEvt.events = EPOLLIN | EPOLLRDHUP | EPOLLET;
                epollEvt.data.u32 = clientFd;
                epoll_ctl (efd, EPOLL_CTL_ADD, clientFd, &epollEvt);
                continue;
            }

            int clientFd = curEvent->data.u32;
            char recvBuffer[2048];
            recvfrom (clientFd, recvBuffer, 2048, 0, NULL, NULL);
            char sndMsg[] = "HTTP/1.1 200 OK\r\nServer: nginx/1.12.2\r\nDate: Tue, 10 Apr 2018 17:41:57 GMT\r\nContent-Type: text/plain\r\nContent-Length: 0\r\nConnection: close\r\n\r\n";
            size_t sndMsgLength = sizeof (sndMsg) - 1;
            struct iovec sndBuffer;
            sndBuffer.iov_base = sndMsg;
            sndBuffer.iov_len = sndMsgLength;
            writev (clientFd, &sndBuffer, 1);
            close (clientFd);
        }
    }
    return 0;
}

---

---

我对这段代码进行了一些负载测试，并将其与nginx进行了比较，看看我是否做了一些错误的事情，或者是否有需要改进的地方。我希望这段代码是最快的实现，因为每一个“真正的”Web服务器都必须在用户空间中做更多的事情。但还是。。。在使用多个load generator线程时，nginx以每秒请求数的方式击败了它。（请注意，我将nginx配置为只使用一个worker来处理每个请求）

第一次测试有预期的结果。。。测试代码速度更快，因为它除了生成硬编码的响应之外什么都不做。但为什么nginx在多线程环境中速度更快？这怎么可能
我能给出的唯一解释是，内核空间中存在一些不同的东西，nginx使用一些sockopts（比如TCP\u FASTOPEN或TCP\u DEFER\u ACCEPT）或其他一些系统调用来完成它的工作。这就是为什么我做了一些strace，并让我的代码做与nginx完全相同的事情（从内核的角度来看）-->您可以看到下面附带的strace。还是。。。速度更快，我不明白为什么

//ab running 50 worker from remote machine 
localhost:~# ab -n 100000 -c 50 http://10.0.0.111:8081/index.html
Requests per second:    27507.60 [#/sec] (mean)  //[to nginx]
Requests per second:    24208.51 [#/sec] (mean)  //[to testcode]

这个测试cast有完全相同的结果，但是我注意到在CPU使用上有一些不同

• 我的测试代码以大约60%的CPU负载运行，ksoftirqd/0以大约80%的速度运行
• nginx的CPU负载约为99%，ksoftirqd/0的CPU负载仅为30%
• 在这两种情况下，ksoftirqd/0在本地主机设置中没有明显的CPU负载

---

nginx战略：

localhost:~# strace -tt -f /usr/sbin/nginx 
//Removed loading of libraries to stay under the 30k char body limit for this question
13:28:20.426369 open("/etc/localtime", O_RDONLY|O_NONBLOCK|O_CLOEXEC) = 3
13:28:20.426549 fstat(3, {st_mode=S_IFREG|0644, st_size=127, ...}) = 0
13:28:20.426723 mmap(NULL, 127, PROT_READ, MAP_SHARED, 3, 0) = 0x74ae1cf8c000
13:28:20.426847 close(3)                = 0
13:28:20.427023 getpid()                = 2186
13:28:20.427164 open("/var/lib/nginx/logs/error.log", O_WRONLY|O_CREAT|O_APPEND, 0644) = 3
13:28:20.427341 brk(0xd104603e000)      = 0xd104603e000
13:28:20.427503 open("/etc/ssl/openssl.cnf", O_RDONLY) = 4
13:28:20.427680 brk(0xd104603f000)      = 0xd104603f000
13:28:20.427819 readv(4, [{iov_base="", iov_len=0}, {iov_base="[ req ]\n#default_bits\t\t= 2048\n#d"..., iov_len=1024}], 2) = 745
13:28:20.428089 brk(0xd1046040000)      = 0xd1046040000
13:28:20.428243 readv(4, [{iov_base="", iov_len=0}, {iov_base="", iov_len=1024}], 2) = 0
13:28:20.428476 close(4)                = 0
13:28:20.428718 brk(0xd1046041000)      = 0xd1046041000
13:28:20.428880 brk(0xd1046042000)      = 0xd1046042000
13:28:20.429179 brk(0xd1046043000)      = 0xd1046043000
13:28:20.429319 brk(0xd1046044000)      = 0xd1046044000
13:28:20.429552 brk(0xd1046045000)      = 0xd1046045000
13:28:20.429775 brk(0xd1046046000)      = 0xd1046046000
13:28:20.429935 brk(0xd1046047000)      = 0xd1046047000
13:28:20.430220 brk(0xd1046048000)      = 0xd1046048000
13:28:20.430391 brk(0xd1046049000)      = 0xd1046049000
13:28:20.430515 brk(0xd104604b000)      = 0xd104604b000
13:28:20.430796 brk(0xd104604c000)      = 0xd104604c000
13:28:20.431042 brk(0xd104604d000)      = 0xd104604d000
13:28:20.431238 brk(0xd104604e000)      = 0xd104604e000
13:28:20.431396 brk(0xd104604f000)      = 0xd104604f000
13:28:20.431581 brk(0xd1046050000)      = 0xd1046050000
13:28:20.431820 brk(0xd1046051000)      = 0xd1046051000
13:28:20.432112 brk(0xd1046054000)      = 0xd1046054000
13:28:20.432374 brk(0xd1046055000)      = 0xd1046055000
13:28:20.432509 brk(0xd1046056000)      = 0xd1046056000
13:28:20.432666 brk(0xd1046057000)      = 0xd1046057000
13:28:20.432836 brk(0xd1046058000)      = 0xd1046058000
13:28:20.433004 brk(0xd1046059000)      = 0xd1046059000
13:28:20.433203 brk(0xd104605a000)      = 0xd104605a000
13:28:20.433400 brk(0xd104605b000)      = 0xd104605b000
13:28:20.433610 brk(0xd104605c000)      = 0xd104605c000
13:28:20.433740 brk(0xd104605d000)      = 0xd104605d000
13:28:20.433895 brk(0xd104605e000)      = 0xd104605e000
13:28:20.434020 brk(0xd104605f000)      = 0xd104605f000
13:28:20.434240 brk(0xd1046060000)      = 0xd1046060000
13:28:20.434419 brk(0xd1046061000)      = 0xd1046061000
13:28:20.434622 uname({sysname="Linux", nodename="localhost", ...}) = 0
13:28:20.434801 sched_getaffinity(0, 128, [0, 1, 2, 3, 4, 5]) = 32
13:28:20.435077 prlimit64(0, RLIMIT_NOFILE, NULL, {rlim_cur=1024, rlim_max=4*1024}) = 0
13:28:20.435260 brk(0xd1046066000)      = 0xd1046066000
13:28:20.435424 uname({sysname="Linux", nodename="localhost", ...}) = 0
13:28:20.435578 brk(0xd104606b000)      = 0xd104606b000
13:28:20.435700 open("/etc/nginx/nginx.conf", O_RDONLY) = 4
13:28:20.435912 fstat(4, {st_mode=S_IFREG|0644, st_size=2781, ...}) = 0
13:28:20.436115 pread64(4, "\n\n\nuser nginx;\npcre_jit on;\n#tim"..., 2781, 0) = 2781
13:28:20.436284 geteuid()               = 0
13:28:20.436440 open("/etc/passwd", O_RDONLY|O_CLOEXEC) = 5
13:28:20.436596 fcntl(5, F_SETFD, FD_CLOEXEC) = 0
13:28:20.436725 fcntl(5, F_SETFD, FD_CLOEXEC) = 0
13:28:20.436857 readv(5, [{iov_base="", iov_len=0}, {iov_base="root:x:0:0:root:/root:/bin/ash\nb"..., iov_len=1024}], 2) = 1024
13:28:20.437047 readv(5, [{iov_base="", iov_len=0}, {iov_base="sbin/nologin\nntp:x:123:123:NTP:/"..., iov_len=1024}], 2) = 397
13:28:20.437235 lseek(5, -43, SEEK_CUR) = 1378
13:28:20.437353 close(5)                = 0
13:28:20.437520 open("/etc/group", O_RDONLY|O_CLOEXEC) = 5
13:28:20.437684 fcntl(5, F_SETFD, FD_CLOEXEC) = 0
13:28:20.437800 fcntl(5, F_SETFD, FD_CLOEXEC) = 0
13:28:20.437933 readv(5, [{iov_base="", iov_len=0}, {iov_base="root:x:0:root\nbin:x:1:root,bin,d"..., iov_len=1024}], 2) = 776
13:28:20.438097 close(5)                = 0
13:28:20.438240 epoll_create1(0)        = 5
13:28:20.438429 close(5)                = 0
13:28:20.438681 brk(0xd1046070000)      = 0xd1046070000
13:28:20.438842 brk(0xd1046072000)      = 0xd1046072000
13:28:20.439053 brk(0xd1046074000)      = 0xd1046074000
13:28:20.439175 brk(0xd1046076000)      = 0xd1046076000
13:28:20.439418 brk(0xd104607b000)      = 0xd104607b000
13:28:20.439655 mmap(NULL, 1052672, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x74ae1ce8b000
13:28:20.439886 brk(0xd1046080000)      = 0xd1046080000
13:28:20.440020 brk(0xd1046085000)      = 0xd1046085000
13:28:20.440225 open("/etc/nginx/mime.types", O_RDONLY) = 5
13:28:20.440380 fstat(5, {st_mode=S_IFREG|0644, st_size=3957, ...}) = 0
13:28:20.440523 pread64(5, "\ntypes {\n    text/html          "..., 3957, 0) = 3957
13:28:20.440725 close(5)                = 0
13:28:20.440977 brk(0xd104608a000)      = 0xd104608a000
13:28:20.441297 brk(0xd104608c000)      = 0xd104608c000
13:28:20.441453 close(4)                = 0
13:28:20.441587 mkdir("/var/tmp/nginx/client_body", 0700) = -1 EEXIST (File exists)
13:28:20.441814 stat("/var/tmp/nginx/client_body", {st_mode=S_IFDIR|0700, st_size=4096, ...}) = 0
13:28:20.442022 mkdir("/var/tmp/nginx/proxy", 0700) = -1 EEXIST (File exists)
13:28:20.442149 stat("/var/tmp/nginx/proxy", {st_mode=S_IFDIR|0700, st_size=4096, ...}) = 0
13:28:20.442257 mkdir("/var/tmp/nginx/fastcgi", 0700) = -1 EEXIST (File exists)
13:28:20.442407 stat("/var/tmp/nginx/fastcgi", {st_mode=S_IFDIR|0700, st_size=4096, ...}) = 0
13:28:20.442568 mkdir("/var/tmp/nginx/uwsgi", 0700) = -1 EEXIST (File exists)
13:28:20.442732 stat("/var/tmp/nginx/uwsgi", {st_mode=S_IFDIR|0700, st_size=4096, ...}) = 0
13:28:20.442945 mkdir("/var/tmp/nginx/scgi", 0700) = -1 EEXIST (File exists)
13:28:20.443051 stat("/var/tmp/nginx/scgi", {st_mode=S_IFDIR|0700, st_size=4096, ...}) = 0
13:28:20.443229 open("/var/log/nginx/access.log", O_WRONLY|O_CREAT|O_APPEND, 0644) = 4
13:28:20.443417 fcntl(4, F_SETFD, FD_CLOEXEC) = 0
13:28:20.443586 open("/var/log/nginx/error.log", O_WRONLY|O_CREAT|O_APPEND, 0644) = 5
13:28:20.443750 fcntl(5, F_SETFD, FD_CLOEXEC) = 0
13:28:20.443889 open("/var/lib/nginx/logs/error.log", O_WRONLY|O_CREAT|O_APPEND, 0644) = 6
13:28:20.444040 fcntl(6, F_SETFD, FD_CLOEXEC) = 0
13:28:20.444197 mmap(NULL, 2097152, PROT_READ|PROT_WRITE, MAP_SHARED|MAP_ANONYMOUS, -1, 0) = 0x74ae1c0a0000
13:28:20.444382 socket(AF_INET, SOCK_STREAM, IPPROTO_IP) = 7
13:28:20.444515 setsockopt(7, SOL_SOCKET, SO_REUSEADDR, [1], 4) = 0
13:28:20.444680 ioctl(7, FIONBIO, [1])  = 0
13:28:20.444808 bind(7, {sa_family=AF_INET, sin_port=htons(8081), sin_addr=inet_addr("0.0.0.0")}, 16) = 0
13:28:20.445015 listen(7, 511)          = 0
13:28:20.445140 listen(7, 511)          = 0
13:28:20.445326 mmap(NULL, 65536, PROT_READ|PROT_WRITE|PROT_EXEC, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x74ae1ce7b000
13:28:20.445493 prlimit64(0, RLIMIT_NOFILE, NULL, {rlim_cur=1024, rlim_max=4*1024}) = 0
13:28:20.445671 mmap(NULL, 1280, PROT_READ|PROT_WRITE, MAP_SHARED|MAP_ANONYMOUS, -1, 0) = 0x74ae1ce7a000
13:28:20.445817 rt_sigprocmask(SIG_UNBLOCK, [RT_1 RT_2], NULL, 8) = 0
13:28:20.445977 rt_sigaction(SIGHUP, {sa_handler=0xd1041f1f3fc, sa_mask=[], sa_flags=SA_RESTORER, sa_restorer=0x74ae1cd4a6cf}, NULL, 8) = 0
13:28:20.446097 rt_sigaction(SIGUSR1, {sa_handler=0xd1041f1f3fc, sa_mask=[], sa_flags=SA_RESTORER, sa_restorer=0x74ae1cd4a6cf}, NULL, 8) = 0
13:28:20.446247 rt_sigaction(SIGWINCH, {sa_handler=0xd1041f1f3fc, sa_mask=[], sa_flags=SA_RESTORER, sa_restorer=0x74ae1cd4a6cf}, NULL, 8) = 0
13:28:20.446438 rt_sigaction(SIGTERM, {sa_handler=0xd1041f1f3fc, sa_mask=[], sa_flags=SA_RESTORER, sa_restorer=0x74ae1cd4a6cf}, NULL, 8) = 0
13:28:20.446635 rt_sigaction(SIGQUIT, {sa_handler=0xd1041f1f3fc, sa_mask=[], sa_flags=SA_RESTORER, sa_restorer=0x74ae1cd4a6cf}, NULL, 8) = 0
13:28:20.446886 rt_sigaction(SIGUSR2, {sa_handler=0xd1041f1f3fc, sa_mask=[], sa_flags=SA_RESTORER, sa_restorer=0x74ae1cd4a6cf}, NULL, 8) = 0
13:28:20.447093 rt_sigaction(SIGALRM, {sa_handler=0xd1041f1f3fc, sa_mask=[], sa_flags=SA_RESTORER, sa_restorer=0x74ae1cd4a6cf}, NULL, 8) = 0
13:28:20.447236 rt_sigaction(SIGINT, {sa_handler=0xd1041f1f3fc, sa_mask=[], sa_flags=SA_RESTORER, sa_restorer=0x74ae1cd4a6cf}, NULL, 8) = 0
13:28:20.447446 rt_sigaction(SIGIO, {sa_handler=0xd1041f1f3fc, sa_mask=[], sa_flags=SA_RESTORER, sa_restorer=0x74ae1cd4a6cf}, NULL, 8) = 0
13:28:20.447767 rt_sigaction(SIGCHLD, {sa_handler=0xd1041f1f3fc, sa_mask=[], sa_flags=SA_RESTORER, sa_restorer=0x74ae1cd4a6cf}, NULL, 8) = 0
13:28:20.447888 rt_sigaction(SIGSYS, {sa_handler=SIG_IGN, sa_mask=[], sa_flags=SA_RESTORER, sa_restorer=0x74ae1cd4a6cf}, NULL, 8) = 0
13:28:20.448094 rt_sigaction(SIGPIPE, {sa_handler=SIG_IGN, sa_mask=[], sa_flags=SA_RESTORER, sa_restorer=0x74ae1cd4a6cf}, NULL, 8) = 0
13:28:20.448253 rt_sigprocmask(SIG_BLOCK, ~[], [], 8) = 0
13:28:20.448396 fork(strace: Process 2187 attached
)                  = 2187
[pid  2187] 13:28:20.448594 gettid( <unfinished ...>
[pid  2186] 13:28:20.448643 rt_sigprocmask(SIG_SETMASK, [],  <unfinished ...>
[pid  2187] 13:28:20.448671 <... gettid resumed> ) = 2187
[pid  2186] 13:28:20.448700 <... rt_sigprocmask resumed> NULL, 8) = 0
[pid  2187] 13:28:20.448765 rt_sigprocmask(SIG_SETMASK, [],  <unfinished ...>
[pid  2186] 13:28:20.448792 exit_group(0 <unfinished ...>
[pid  2187] 13:28:20.448812 <... rt_sigprocmask resumed> NULL, 8) = 0
[pid  2186] 13:28:20.448836 <... exit_group resumed>) = ?
[pid  2187] 13:28:20.448854 getpid()    = 2187
[pid  2187] 13:28:20.448951 setsid( <unfinished ...>
[pid  2186] 13:28:20.449046 +++ exited with 0 +++
13:28:20.449055 <... setsid resumed> )  = 2187
13:28:20.449107 umask(000)              = 022
13:28:20.449212 open("/dev/null", O_RDWR) = 8
13:28:20.449309 dup2(8, 0)              = 0
13:28:20.449455 dup2(8, 1)              = 1
13:28:20.449573 close(8)                = 0
13:28:20.449692 open("/run/nginx/nginx.pid", O_RDWR|O_CREAT|O_TRUNC, 0644) = 8
13:28:20.449848 pwrite64(8, "2187\n", 5, 0) = 5
13:28:20.449978 close(8)                = 0
13:28:20.450111 dup2(6, 2)              = 2
13:28:20.450218 close(3)                = 0
13:28:20.450376 rt_sigprocmask(SIG_BLOCK, [HUP INT QUIT USR1 USR2 ALRM TERM CHLD WINCH IO], NULL, 8) = 0
13:28:20.450499 socketpair(AF_UNIX, SOCK_STREAM, 0, [3, 8]) = 0
13:28:20.450603 ioctl(3, FIONBIO, [1])  = 0
13:28:20.450696 ioctl(8, FIONBIO, [1])  = 0
13:28:20.450830 ioctl(3, FIOASYNC, [1]) = 0
13:28:20.450964 fcntl(3, F_SETOWN, 2187) = 0
13:28:20.451079 fcntl(3, F_SETFD, FD_CLOEXEC) = 0
13:28:20.451148 fcntl(8, F_SETFD, FD_CLOEXEC) = 0
13:28:20.451244 rt_sigprocmask(SIG_BLOCK, ~[], [HUP INT QUIT USR1 USR2 ALRM TERM CHLD WINCH IO], 8) = 0
13:28:20.451379 fork(strace: Process 2188 attached
 <unfinished ...>
[pid  2188] 13:28:20.451596 gettid( <unfinished ...>
[pid  2187] 13:28:20.451615 <... fork resumed> ) = 2188
[pid  2187] 13:28:20.451727 rt_sigprocmask(SIG_SETMASK, [HUP INT QUIT USR1 USR2 ALRM TERM CHLD WINCH IO],  <unfinished ...>
[pid  2188] 13:28:20.451754 <... gettid resumed> ) = 2188
[pid  2187] 13:28:20.451774 <... rt_sigprocmask resumed> NULL, 8) = 0
[pid  2188] 13:28:20.451942 rt_sigprocmask(SIG_SETMASK, [HUP INT QUIT USR1 USR2 ALRM TERM CHLD WINCH IO],  <unfinished ...>
[pid  2187] 13:28:20.451969 rt_sigsuspend([], 8 <unfinished ...>
[pid  2188] 13:28:20.451985 <... rt_sigprocmask resumed> NULL, 8) = 0
[pid  2188] 13:28:20.452053 getpid()    = 2188
[pid  2188] 13:28:20.452330 rt_sigprocmask(SIG_BLOCK, ~[RTMIN RT_1 RT_2], [HUP INT QUIT USR1 USR2 ALRM TERM CHLD WINCH IO], 8) = 0
[pid  2188] 13:28:20.452621 rt_sigprocmask(SIG_BLOCK, ~[], NULL, 8) = 0
[pid  2188] 13:28:20.452893 prlimit64(0, RLIMIT_NOFILE, {rlim_cur=8*1024, rlim_max=8*1024}, NULL) = 0
[pid  2188] 13:28:20.453075 futex(0x74ae1cf95064, FUTEX_WAKE_PRIVATE, 2147483647) = 0
[pid  2188] 13:28:20.453279 rt_sigprocmask(SIG_SETMASK, [HUP INT QUIT USR1 USR2 ALRM TERM CHLD WINCH IO], NULL, 8) = 0
[pid  2188] 13:28:20.453487 geteuid()   = 0
[pid  2188] 13:28:20.453667 rt_sigprocmask(SIG_BLOCK, ~[RTMIN RT_1 RT_2], [HUP INT QUIT USR1 USR2 ALRM TERM CHLD WINCH IO], 8) = 0
[pid  2188] 13:28:20.453861 rt_sigprocmask(SIG_BLOCK, ~[], NULL, 8) = 0
[pid  2188] 13:28:20.454091 setgid(103) = 0
[pid  2188] 13:28:20.454335 futex(0x74ae1cf95064, FUTEX_WAKE_PRIVATE, 2147483647) = 0
[pid  2188] 13:28:20.454583 rt_sigprocmask(SIG_SETMASK, [HUP INT QUIT USR1 USR2 ALRM TERM CHLD WINCH IO], NULL, 8) = 0
[pid  2188] 13:28:20.454822 socket(AF_UNIX, SOCK_STREAM|SOCK_CLOEXEC, 0) = 9
[pid  2188] 13:28:20.455183 connect(9, {sa_family=AF_UNIX, sun_path="/var/run/nscd/socket"}, 24) = -1 ENOENT (No such file or directory)
[pid  2188] 13:28:20.455537 close(9)    = 0
[pid  2188] 13:28:20.455800 open("/etc/group", O_RDONLY|O_CLOEXEC) = 9
[pid  2188] 13:28:20.456030 fcntl(9, F_SETFD, FD_CLOEXEC) = 0
[pid  2188] 13:28:20.456331 fcntl(9, F_SETFD, FD_CLOEXEC) = 0
[pid  2188] 13:28:20.456544 readv(9, [{iov_base="", iov_len=0}, {iov_base="root:x:0:root\nbin:x:1:root,bin,d"..., iov_len=1024}], 2) = 776
[pid  2188] 13:28:20.456799 readv(9, [{iov_base="", iov_len=0}, {iov_base="", iov_len=1024}], 2) = 0
[pid  2188] 13:28:20.456956 close(9)    = 0
[pid  2188] 13:28:20.457134 setgroups(3, [103, 82, 103]) = 0
[pid  2188] 13:28:20.457365 rt_sigprocmask(SIG_BLOCK, ~[RTMIN RT_1 RT_2], [HUP INT QUIT USR1 USR2 ALRM TERM CHLD WINCH IO], 8) = 0
[pid  2188] 13:28:20.457534 rt_sigprocmask(SIG_BLOCK, ~[], NULL, 8) = 0
[pid  2188] 13:28:20.457818 setuid(102) = 0
[pid  2188] 13:28:20.457990 futex(0x74ae1cf95064, FUTEX_WAKE_PRIVATE, 2147483647) = 0
[pid  2188] 13:28:20.458159 rt_sigprocmask(SIG_SETMASK, [HUP INT QUIT USR1 USR2 ALRM TERM CHLD WINCH IO], NULL, 8) = 0
[pid  2188] 13:28:20.458378 prctl(PR_SET_DUMPABLE, SUID_DUMP_USER) = 0
[pid  2188] 13:28:20.458598 chdir("/var/www") = 0
[pid  2188] 13:28:20.458868 rt_sigprocmask(SIG_SETMASK, [], NULL, 8) = 0
[pid  2188] 13:28:20.459703 epoll_create1(0) = 9
[pid  2188] 13:28:20.459994 eventfd2(0, 0) = 10
[pid  2188] 13:28:20.460340 epoll_ctl(9, EPOLL_CTL_ADD, 10, {EPOLLIN|EPOLLET, {u32=1109208384, u64=14363479846208}}) = 0
[pid  2188] 13:28:20.460600 eventfd2(0, 0) = 11
[pid  2188] 13:28:20.460878 ioctl(11, FIONBIO, [1]) = 0
[pid  2188] 13:28:20.461043 io_setup(32, [0x74ae1ce79000]) = 0
[pid  2188] 13:28:20.461389 epoll_ctl(9, EPOLL_CTL_ADD, 11, {EPOLLIN|EPOLLET, {u32=1109208032, u64=14363479845856}}) = 0
[pid  2188] 13:28:20.461729 socketpair(AF_UNIX, SOCK_STREAM, 0, [12, 13]) = 0
[pid  2188] 13:28:20.462043 epoll_ctl(9, EPOLL_CTL_ADD, 12, {EPOLLIN|EPOLLRDHUP|EPOLLET, {u32=1109208032, u64=14363479845856}}) = 0
[pid  2188] 13:28:20.462255 close(13)   = 0
[pid  2188] 13:28:20.462608 epoll_pwait(9, [{EPOLLIN|EPOLLHUP|EPOLLRDHUP, {u32=1109208032, u64=14363479845856}}], 1, 5000, NULL, 8) = 1
[pid  2188] 13:28:20.462969 close(12)   = 0
[pid  2188] 13:28:20.463325 mmap(NULL, 987136, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x74ae1bfaf000
[pid  2188] 13:28:20.463517 mmap(NULL, 397312, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x74ae1ce18000
[pid  2188] 13:28:20.464039 mmap(NULL, 397312, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x74ae1cdb7000
[pid  2188] 13:28:20.466039 epoll_ctl(9, EPOLL_CTL_ADD, 7, {EPOLLIN|EPOLLRDHUP, {u32=469430304, u64=128291142561824}}) = 0
[pid  2188] 13:28:20.466432 close(3)    = 0
[pid  2188] 13:28:20.466763 epoll_ctl(9, EPOLL_CTL_ADD, 8, {EPOLLIN|EPOLLRDHUP, {u32=469430544, u64=128291142562064}}) = 0
//Eventloop starts here
[pid  2188] 13:28:20.467046 epoll_pwait(9, [{EPOLLIN, {u32=469430304, u64=128291142561824}}], 512, -1, NULL, 8) = 1
[pid  2188] 13:28:34.390021 accept4(7, {sa_family=AF_INET, sin_port=htons(54280), sin_addr=inet_addr("10.0.0.15")}, [112->16], SOCK_NONBLOCK) = 3
[pid  2188] 13:28:34.390110 epoll_ctl(9, EPOLL_CTL_ADD, 3, {EPOLLIN|EPOLLRDHUP|EPOLLET, {u32=469430784, u64=128291142562304}}) = 0
[pid  2188] 13:28:34.390188 epoll_pwait(9, [{EPOLLIN, {u32=469430784, u64=128291142562304}}], 512, 30000, NULL, 8) = 1
[pid  2188] 13:28:34.390245 recvfrom(3, "GET /index.html HTTP/1.0\r\nHost: "..., 2048, 0, NULL, NULL) = 93
[pid  2188] 13:28:34.390462 writev(3, [{iov_base="HTTP/1.1 200 OK\r\nServer: nginx/1"..., iov_len=142}], 1) = 142
[pid  2188] 13:28:34.390602 close(3)    = 0

user nginx;
worker_processes 1;
working_directory /var/www;

events {
    accept_mutex off;
    worker_connections 4096;
}
http {
    include /etc/nginx/mime.types;
    server {
        listen 8081;
        server_name "10.0.0.110";
        return 200;
    }
}

编辑：
我做了更多的追踪。。。来自远程主机的400000请求。。。仍然不知道为什么会发生这种情况：

localhost:/~# strace -c -f /usr/sbin/nginx 
% time     seconds  usecs/call     calls    errors syscall
------ ----------- ----------- --------- --------- ----------------
 47.11    0.040309           0    400000           writev
 44.55    0.038115           0    400021           close
  3.11    0.002658           0    400002           accept4
  1.80    0.001538           0    400002           recvfrom
  1.74    0.001486           0    400007           epoll_ctl
  1.69    0.001450           0    400008           epoll_pwait

localhost:/~# strace -c -f ./test 
% time     seconds  usecs/call     calls    errors syscall
------ ----------- ----------- --------- --------- ----------------
 47.90    0.042760           0    400002           writev
 44.27    0.039518           0    400002           close
  3.13    0.002793           0    400002           accept4
  1.80    0.001610           0    400002           recvfrom
  1.57    0.001400           0    400005           epoll_pwait
  1.33    0.001183           0    400003           epoll_ctl

编辑：
正如在评论部分所建议的，我再次测试了内存对齐

localhost:~# strace -e raw=recvfrom,writev ./test 
recvfrom(0x5, 0x7c7364ae2ee0, 0x800, 0, 0, 0) = 0x5d
writev(0x5, 0x7c7364ae1650, 0x1)        = 0x69

localhost:~# strace -f -e raw=recvfrom,writev /usr/sbin/nginx 
[pid  2381] recvfrom(0x3, 0x2344ebbd6a0, 0x800, 0, 0, 0) = 0x5d
[pid  2381] writev(0x3, 0x7df116c39f90, 0x1) = 0x86

编辑：
根据要求，nginx的配置：

localhost:~# strace -tt -f /usr/sbin/nginx 
//Removed loading of libraries to stay under the 30k char body limit for this question
13:28:20.426369 open("/etc/localtime", O_RDONLY|O_NONBLOCK|O_CLOEXEC) = 3
13:28:20.426549 fstat(3, {st_mode=S_IFREG|0644, st_size=127, ...}) = 0
13:28:20.426723 mmap(NULL, 127, PROT_READ, MAP_SHARED, 3, 0) = 0x74ae1cf8c000
13:28:20.426847 close(3)                = 0
13:28:20.427023 getpid()                = 2186
13:28:20.427164 open("/var/lib/nginx/logs/error.log", O_WRONLY|O_CREAT|O_APPEND, 0644) = 3
13:28:20.427341 brk(0xd104603e000)      = 0xd104603e000
13:28:20.427503 open("/etc/ssl/openssl.cnf", O_RDONLY) = 4
13:28:20.427680 brk(0xd104603f000)      = 0xd104603f000
13:28:20.427819 readv(4, [{iov_base="", iov_len=0}, {iov_base="[ req ]\n#default_bits\t\t= 2048\n#d"..., iov_len=1024}], 2) = 745
13:28:20.428089 brk(0xd1046040000)      = 0xd1046040000
13:28:20.428243 readv(4, [{iov_base="", iov_len=0}, {iov_base="", iov_len=1024}], 2) = 0
13:28:20.428476 close(4)                = 0
13:28:20.428718 brk(0xd1046041000)      = 0xd1046041000
13:28:20.428880 brk(0xd1046042000)      = 0xd1046042000
13:28:20.429179 brk(0xd1046043000)      = 0xd1046043000
13:28:20.429319 brk(0xd1046044000)      = 0xd1046044000
13:28:20.429552 brk(0xd1046045000)      = 0xd1046045000
13:28:20.429775 brk(0xd1046046000)      = 0xd1046046000
13:28:20.429935 brk(0xd1046047000)      = 0xd1046047000
13:28:20.430220 brk(0xd1046048000)      = 0xd1046048000
13:28:20.430391 brk(0xd1046049000)      = 0xd1046049000
13:28:20.430515 brk(0xd104604b000)      = 0xd104604b000
13:28:20.430796 brk(0xd104604c000)      = 0xd104604c000
13:28:20.431042 brk(0xd104604d000)      = 0xd104604d000
13:28:20.431238 brk(0xd104604e000)      = 0xd104604e000
13:28:20.431396 brk(0xd104604f000)      = 0xd104604f000
13:28:20.431581 brk(0xd1046050000)      = 0xd1046050000
13:28:20.431820 brk(0xd1046051000)      = 0xd1046051000
13:28:20.432112 brk(0xd1046054000)      = 0xd1046054000
13:28:20.432374 brk(0xd1046055000)      = 0xd1046055000
13:28:20.432509 brk(0xd1046056000)      = 0xd1046056000
13:28:20.432666 brk(0xd1046057000)      = 0xd1046057000
13:28:20.432836 brk(0xd1046058000)      = 0xd1046058000
13:28:20.433004 brk(0xd1046059000)      = 0xd1046059000
13:28:20.433203 brk(0xd104605a000)      = 0xd104605a000
13:28:20.433400 brk(0xd104605b000)      = 0xd104605b000
13:28:20.433610 brk(0xd104605c000)      = 0xd104605c000
13:28:20.433740 brk(0xd104605d000)      = 0xd104605d000
13:28:20.433895 brk(0xd104605e000)      = 0xd104605e000
13:28:20.434020 brk(0xd104605f000)      = 0xd104605f000
13:28:20.434240 brk(0xd1046060000)      = 0xd1046060000
13:28:20.434419 brk(0xd1046061000)      = 0xd1046061000
13:28:20.434622 uname({sysname="Linux", nodename="localhost", ...}) = 0
13:28:20.434801 sched_getaffinity(0, 128, [0, 1, 2, 3, 4, 5]) = 32
13:28:20.435077 prlimit64(0, RLIMIT_NOFILE, NULL, {rlim_cur=1024, rlim_max=4*1024}) = 0
13:28:20.435260 brk(0xd1046066000)      = 0xd1046066000
13:28:20.435424 uname({sysname="Linux", nodename="localhost", ...}) = 0
13:28:20.435578 brk(0xd104606b000)      = 0xd104606b000
13:28:20.435700 open("/etc/nginx/nginx.conf", O_RDONLY) = 4
13:28:20.435912 fstat(4, {st_mode=S_IFREG|0644, st_size=2781, ...}) = 0
13:28:20.436115 pread64(4, "\n\n\nuser nginx;\npcre_jit on;\n#tim"..., 2781, 0) = 2781
13:28:20.436284 geteuid()               = 0
13:28:20.436440 open("/etc/passwd", O_RDONLY|O_CLOEXEC) = 5
13:28:20.436596 fcntl(5, F_SETFD, FD_CLOEXEC) = 0
13:28:20.436725 fcntl(5, F_SETFD, FD_CLOEXEC) = 0
13:28:20.436857 readv(5, [{iov_base="", iov_len=0}, {iov_base="root:x:0:0:root:/root:/bin/ash\nb"..., iov_len=1024}], 2) = 1024
13:28:20.437047 readv(5, [{iov_base="", iov_len=0}, {iov_base="sbin/nologin\nntp:x:123:123:NTP:/"..., iov_len=1024}], 2) = 397
13:28:20.437235 lseek(5, -43, SEEK_CUR) = 1378
13:28:20.437353 close(5)                = 0
13:28:20.437520 open("/etc/group", O_RDONLY|O_CLOEXEC) = 5
13:28:20.437684 fcntl(5, F_SETFD, FD_CLOEXEC) = 0
13:28:20.437800 fcntl(5, F_SETFD, FD_CLOEXEC) = 0
13:28:20.437933 readv(5, [{iov_base="", iov_len=0}, {iov_base="root:x:0:root\nbin:x:1:root,bin,d"..., iov_len=1024}], 2) = 776
13:28:20.438097 close(5)                = 0
13:28:20.438240 epoll_create1(0)        = 5
13:28:20.438429 close(5)                = 0
13:28:20.438681 brk(0xd1046070000)      = 0xd1046070000
13:28:20.438842 brk(0xd1046072000)      = 0xd1046072000
13:28:20.439053 brk(0xd1046074000)      = 0xd1046074000
13:28:20.439175 brk(0xd1046076000)      = 0xd1046076000
13:28:20.439418 brk(0xd104607b000)      = 0xd104607b000
13:28:20.439655 mmap(NULL, 1052672, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x74ae1ce8b000
13:28:20.439886 brk(0xd1046080000)      = 0xd1046080000
13:28:20.440020 brk(0xd1046085000)      = 0xd1046085000
13:28:20.440225 open("/etc/nginx/mime.types", O_RDONLY) = 5
13:28:20.440380 fstat(5, {st_mode=S_IFREG|0644, st_size=3957, ...}) = 0
13:28:20.440523 pread64(5, "\ntypes {\n    text/html          "..., 3957, 0) = 3957
13:28:20.440725 close(5)                = 0
13:28:20.440977 brk(0xd104608a000)      = 0xd104608a000
13:28:20.441297 brk(0xd104608c000)      = 0xd104608c000
13:28:20.441453 close(4)                = 0
13:28:20.441587 mkdir("/var/tmp/nginx/client_body", 0700) = -1 EEXIST (File exists)
13:28:20.441814 stat("/var/tmp/nginx/client_body", {st_mode=S_IFDIR|0700, st_size=4096, ...}) = 0
13:28:20.442022 mkdir("/var/tmp/nginx/proxy", 0700) = -1 EEXIST (File exists)
13:28:20.442149 stat("/var/tmp/nginx/proxy", {st_mode=S_IFDIR|0700, st_size=4096, ...}) = 0
13:28:20.442257 mkdir("/var/tmp/nginx/fastcgi", 0700) = -1 EEXIST (File exists)
13:28:20.442407 stat("/var/tmp/nginx/fastcgi", {st_mode=S_IFDIR|0700, st_size=4096, ...}) = 0
13:28:20.442568 mkdir("/var/tmp/nginx/uwsgi", 0700) = -1 EEXIST (File exists)
13:28:20.442732 stat("/var/tmp/nginx/uwsgi", {st_mode=S_IFDIR|0700, st_size=4096, ...}) = 0
13:28:20.442945 mkdir("/var/tmp/nginx/scgi", 0700) = -1 EEXIST (File exists)
13:28:20.443051 stat("/var/tmp/nginx/scgi", {st_mode=S_IFDIR|0700, st_size=4096, ...}) = 0
13:28:20.443229 open("/var/log/nginx/access.log", O_WRONLY|O_CREAT|O_APPEND, 0644) = 4
13:28:20.443417 fcntl(4, F_SETFD, FD_CLOEXEC) = 0
13:28:20.443586 open("/var/log/nginx/error.log", O_WRONLY|O_CREAT|O_APPEND, 0644) = 5
13:28:20.443750 fcntl(5, F_SETFD, FD_CLOEXEC) = 0
13:28:20.443889 open("/var/lib/nginx/logs/error.log", O_WRONLY|O_CREAT|O_APPEND, 0644) = 6
13:28:20.444040 fcntl(6, F_SETFD, FD_CLOEXEC) = 0
13:28:20.444197 mmap(NULL, 2097152, PROT_READ|PROT_WRITE, MAP_SHARED|MAP_ANONYMOUS, -1, 0) = 0x74ae1c0a0000
13:28:20.444382 socket(AF_INET, SOCK_STREAM, IPPROTO_IP) = 7
13:28:20.444515 setsockopt(7, SOL_SOCKET, SO_REUSEADDR, [1], 4) = 0
13:28:20.444680 ioctl(7, FIONBIO, [1])  = 0
13:28:20.444808 bind(7, {sa_family=AF_INET, sin_port=htons(8081), sin_addr=inet_addr("0.0.0.0")}, 16) = 0
13:28:20.445015 listen(7, 511)          = 0
13:28:20.445140 listen(7, 511)          = 0
13:28:20.445326 mmap(NULL, 65536, PROT_READ|PROT_WRITE|PROT_EXEC, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x74ae1ce7b000
13:28:20.445493 prlimit64(0, RLIMIT_NOFILE, NULL, {rlim_cur=1024, rlim_max=4*1024}) = 0
13:28:20.445671 mmap(NULL, 1280, PROT_READ|PROT_WRITE, MAP_SHARED|MAP_ANONYMOUS, -1, 0) = 0x74ae1ce7a000
13:28:20.445817 rt_sigprocmask(SIG_UNBLOCK, [RT_1 RT_2], NULL, 8) = 0
13:28:20.445977 rt_sigaction(SIGHUP, {sa_handler=0xd1041f1f3fc, sa_mask=[], sa_flags=SA_RESTORER, sa_restorer=0x74ae1cd4a6cf}, NULL, 8) = 0
13:28:20.446097 rt_sigaction(SIGUSR1, {sa_handler=0xd1041f1f3fc, sa_mask=[], sa_flags=SA_RESTORER, sa_restorer=0x74ae1cd4a6cf}, NULL, 8) = 0
13:28:20.446247 rt_sigaction(SIGWINCH, {sa_handler=0xd1041f1f3fc, sa_mask=[], sa_flags=SA_RESTORER, sa_restorer=0x74ae1cd4a6cf}, NULL, 8) = 0
13:28:20.446438 rt_sigaction(SIGTERM, {sa_handler=0xd1041f1f3fc, sa_mask=[], sa_flags=SA_RESTORER, sa_restorer=0x74ae1cd4a6cf}, NULL, 8) = 0
13:28:20.446635 rt_sigaction(SIGQUIT, {sa_handler=0xd1041f1f3fc, sa_mask=[], sa_flags=SA_RESTORER, sa_restorer=0x74ae1cd4a6cf}, NULL, 8) = 0
13:28:20.446886 rt_sigaction(SIGUSR2, {sa_handler=0xd1041f1f3fc, sa_mask=[], sa_flags=SA_RESTORER, sa_restorer=0x74ae1cd4a6cf}, NULL, 8) = 0
13:28:20.447093 rt_sigaction(SIGALRM, {sa_handler=0xd1041f1f3fc, sa_mask=[], sa_flags=SA_RESTORER, sa_restorer=0x74ae1cd4a6cf}, NULL, 8) = 0
13:28:20.447236 rt_sigaction(SIGINT, {sa_handler=0xd1041f1f3fc, sa_mask=[], sa_flags=SA_RESTORER, sa_restorer=0x74ae1cd4a6cf}, NULL, 8) = 0
13:28:20.447446 rt_sigaction(SIGIO, {sa_handler=0xd1041f1f3fc, sa_mask=[], sa_flags=SA_RESTORER, sa_restorer=0x74ae1cd4a6cf}, NULL, 8) = 0
13:28:20.447767 rt_sigaction(SIGCHLD, {sa_handler=0xd1041f1f3fc, sa_mask=[], sa_flags=SA_RESTORER, sa_restorer=0x74ae1cd4a6cf}, NULL, 8) = 0
13:28:20.447888 rt_sigaction(SIGSYS, {sa_handler=SIG_IGN, sa_mask=[], sa_flags=SA_RESTORER, sa_restorer=0x74ae1cd4a6cf}, NULL, 8) = 0
13:28:20.448094 rt_sigaction(SIGPIPE, {sa_handler=SIG_IGN, sa_mask=[], sa_flags=SA_RESTORER, sa_restorer=0x74ae1cd4a6cf}, NULL, 8) = 0
13:28:20.448253 rt_sigprocmask(SIG_BLOCK, ~[], [], 8) = 0
13:28:20.448396 fork(strace: Process 2187 attached
)                  = 2187
[pid  2187] 13:28:20.448594 gettid( <unfinished ...>
[pid  2186] 13:28:20.448643 rt_sigprocmask(SIG_SETMASK, [],  <unfinished ...>
[pid  2187] 13:28:20.448671 <... gettid resumed> ) = 2187
[pid  2186] 13:28:20.448700 <... rt_sigprocmask resumed> NULL, 8) = 0
[pid  2187] 13:28:20.448765 rt_sigprocmask(SIG_SETMASK, [],  <unfinished ...>
[pid  2186] 13:28:20.448792 exit_group(0 <unfinished ...>
[pid  2187] 13:28:20.448812 <... rt_sigprocmask resumed> NULL, 8) = 0
[pid  2186] 13:28:20.448836 <... exit_group resumed>) = ?
[pid  2187] 13:28:20.448854 getpid()    = 2187
[pid  2187] 13:28:20.448951 setsid( <unfinished ...>
[pid  2186] 13:28:20.449046 +++ exited with 0 +++
13:28:20.449055 <... setsid resumed> )  = 2187
13:28:20.449107 umask(000)              = 022
13:28:20.449212 open("/dev/null", O_RDWR) = 8
13:28:20.449309 dup2(8, 0)              = 0
13:28:20.449455 dup2(8, 1)              = 1
13:28:20.449573 close(8)                = 0
13:28:20.449692 open("/run/nginx/nginx.pid", O_RDWR|O_CREAT|O_TRUNC, 0644) = 8
13:28:20.449848 pwrite64(8, "2187\n", 5, 0) = 5
13:28:20.449978 close(8)                = 0
13:28:20.450111 dup2(6, 2)              = 2
13:28:20.450218 close(3)                = 0
13:28:20.450376 rt_sigprocmask(SIG_BLOCK, [HUP INT QUIT USR1 USR2 ALRM TERM CHLD WINCH IO], NULL, 8) = 0
13:28:20.450499 socketpair(AF_UNIX, SOCK_STREAM, 0, [3, 8]) = 0
13:28:20.450603 ioctl(3, FIONBIO, [1])  = 0
13:28:20.450696 ioctl(8, FIONBIO, [1])  = 0
13:28:20.450830 ioctl(3, FIOASYNC, [1]) = 0
13:28:20.450964 fcntl(3, F_SETOWN, 2187) = 0
13:28:20.451079 fcntl(3, F_SETFD, FD_CLOEXEC) = 0
13:28:20.451148 fcntl(8, F_SETFD, FD_CLOEXEC) = 0
13:28:20.451244 rt_sigprocmask(SIG_BLOCK, ~[], [HUP INT QUIT USR1 USR2 ALRM TERM CHLD WINCH IO], 8) = 0
13:28:20.451379 fork(strace: Process 2188 attached
 <unfinished ...>
[pid  2188] 13:28:20.451596 gettid( <unfinished ...>
[pid  2187] 13:28:20.451615 <... fork resumed> ) = 2188
[pid  2187] 13:28:20.451727 rt_sigprocmask(SIG_SETMASK, [HUP INT QUIT USR1 USR2 ALRM TERM CHLD WINCH IO],  <unfinished ...>
[pid  2188] 13:28:20.451754 <... gettid resumed> ) = 2188
[pid  2187] 13:28:20.451774 <... rt_sigprocmask resumed> NULL, 8) = 0
[pid  2188] 13:28:20.451942 rt_sigprocmask(SIG_SETMASK, [HUP INT QUIT USR1 USR2 ALRM TERM CHLD WINCH IO],  <unfinished ...>
[pid  2187] 13:28:20.451969 rt_sigsuspend([], 8 <unfinished ...>
[pid  2188] 13:28:20.451985 <... rt_sigprocmask resumed> NULL, 8) = 0
[pid  2188] 13:28:20.452053 getpid()    = 2188
[pid  2188] 13:28:20.452330 rt_sigprocmask(SIG_BLOCK, ~[RTMIN RT_1 RT_2], [HUP INT QUIT USR1 USR2 ALRM TERM CHLD WINCH IO], 8) = 0
[pid  2188] 13:28:20.452621 rt_sigprocmask(SIG_BLOCK, ~[], NULL, 8) = 0
[pid  2188] 13:28:20.452893 prlimit64(0, RLIMIT_NOFILE, {rlim_cur=8*1024, rlim_max=8*1024}, NULL) = 0
[pid  2188] 13:28:20.453075 futex(0x74ae1cf95064, FUTEX_WAKE_PRIVATE, 2147483647) = 0
[pid  2188] 13:28:20.453279 rt_sigprocmask(SIG_SETMASK, [HUP INT QUIT USR1 USR2 ALRM TERM CHLD WINCH IO], NULL, 8) = 0
[pid  2188] 13:28:20.453487 geteuid()   = 0
[pid  2188] 13:28:20.453667 rt_sigprocmask(SIG_BLOCK, ~[RTMIN RT_1 RT_2], [HUP INT QUIT USR1 USR2 ALRM TERM CHLD WINCH IO], 8) = 0
[pid  2188] 13:28:20.453861 rt_sigprocmask(SIG_BLOCK, ~[], NULL, 8) = 0
[pid  2188] 13:28:20.454091 setgid(103) = 0
[pid  2188] 13:28:20.454335 futex(0x74ae1cf95064, FUTEX_WAKE_PRIVATE, 2147483647) = 0
[pid  2188] 13:28:20.454583 rt_sigprocmask(SIG_SETMASK, [HUP INT QUIT USR1 USR2 ALRM TERM CHLD WINCH IO], NULL, 8) = 0
[pid  2188] 13:28:20.454822 socket(AF_UNIX, SOCK_STREAM|SOCK_CLOEXEC, 0) = 9
[pid  2188] 13:28:20.455183 connect(9, {sa_family=AF_UNIX, sun_path="/var/run/nscd/socket"}, 24) = -1 ENOENT (No such file or directory)
[pid  2188] 13:28:20.455537 close(9)    = 0
[pid  2188] 13:28:20.455800 open("/etc/group", O_RDONLY|O_CLOEXEC) = 9
[pid  2188] 13:28:20.456030 fcntl(9, F_SETFD, FD_CLOEXEC) = 0
[pid  2188] 13:28:20.456331 fcntl(9, F_SETFD, FD_CLOEXEC) = 0
[pid  2188] 13:28:20.456544 readv(9, [{iov_base="", iov_len=0}, {iov_base="root:x:0:root\nbin:x:1:root,bin,d"..., iov_len=1024}], 2) = 776
[pid  2188] 13:28:20.456799 readv(9, [{iov_base="", iov_len=0}, {iov_base="", iov_len=1024}], 2) = 0
[pid  2188] 13:28:20.456956 close(9)    = 0
[pid  2188] 13:28:20.457134 setgroups(3, [103, 82, 103]) = 0
[pid  2188] 13:28:20.457365 rt_sigprocmask(SIG_BLOCK, ~[RTMIN RT_1 RT_2], [HUP INT QUIT USR1 USR2 ALRM TERM CHLD WINCH IO], 8) = 0
[pid  2188] 13:28:20.457534 rt_sigprocmask(SIG_BLOCK, ~[], NULL, 8) = 0
[pid  2188] 13:28:20.457818 setuid(102) = 0
[pid  2188] 13:28:20.457990 futex(0x74ae1cf95064, FUTEX_WAKE_PRIVATE, 2147483647) = 0
[pid  2188] 13:28:20.458159 rt_sigprocmask(SIG_SETMASK, [HUP INT QUIT USR1 USR2 ALRM TERM CHLD WINCH IO], NULL, 8) = 0
[pid  2188] 13:28:20.458378 prctl(PR_SET_DUMPABLE, SUID_DUMP_USER) = 0
[pid  2188] 13:28:20.458598 chdir("/var/www") = 0
[pid  2188] 13:28:20.458868 rt_sigprocmask(SIG_SETMASK, [], NULL, 8) = 0
[pid  2188] 13:28:20.459703 epoll_create1(0) = 9
[pid  2188] 13:28:20.459994 eventfd2(0, 0) = 10
[pid  2188] 13:28:20.460340 epoll_ctl(9, EPOLL_CTL_ADD, 10, {EPOLLIN|EPOLLET, {u32=1109208384, u64=14363479846208}}) = 0
[pid  2188] 13:28:20.460600 eventfd2(0, 0) = 11
[pid  2188] 13:28:20.460878 ioctl(11, FIONBIO, [1]) = 0
[pid  2188] 13:28:20.461043 io_setup(32, [0x74ae1ce79000]) = 0
[pid  2188] 13:28:20.461389 epoll_ctl(9, EPOLL_CTL_ADD, 11, {EPOLLIN|EPOLLET, {u32=1109208032, u64=14363479845856}}) = 0
[pid  2188] 13:28:20.461729 socketpair(AF_UNIX, SOCK_STREAM, 0, [12, 13]) = 0
[pid  2188] 13:28:20.462043 epoll_ctl(9, EPOLL_CTL_ADD, 12, {EPOLLIN|EPOLLRDHUP|EPOLLET, {u32=1109208032, u64=14363479845856}}) = 0
[pid  2188] 13:28:20.462255 close(13)   = 0
[pid  2188] 13:28:20.462608 epoll_pwait(9, [{EPOLLIN|EPOLLHUP|EPOLLRDHUP, {u32=1109208032, u64=14363479845856}}], 1, 5000, NULL, 8) = 1
[pid  2188] 13:28:20.462969 close(12)   = 0
[pid  2188] 13:28:20.463325 mmap(NULL, 987136, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x74ae1bfaf000
[pid  2188] 13:28:20.463517 mmap(NULL, 397312, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x74ae1ce18000
[pid  2188] 13:28:20.464039 mmap(NULL, 397312, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x74ae1cdb7000
[pid  2188] 13:28:20.466039 epoll_ctl(9, EPOLL_CTL_ADD, 7, {EPOLLIN|EPOLLRDHUP, {u32=469430304, u64=128291142561824}}) = 0
[pid  2188] 13:28:20.466432 close(3)    = 0
[pid  2188] 13:28:20.466763 epoll_ctl(9, EPOLL_CTL_ADD, 8, {EPOLLIN|EPOLLRDHUP, {u32=469430544, u64=128291142562064}}) = 0
//Eventloop starts here
[pid  2188] 13:28:20.467046 epoll_pwait(9, [{EPOLLIN, {u32=469430304, u64=128291142561824}}], 512, -1, NULL, 8) = 1
[pid  2188] 13:28:34.390021 accept4(7, {sa_family=AF_INET, sin_port=htons(54280), sin_addr=inet_addr("10.0.0.15")}, [112->16], SOCK_NONBLOCK) = 3
[pid  2188] 13:28:34.390110 epoll_ctl(9, EPOLL_CTL_ADD, 3, {EPOLLIN|EPOLLRDHUP|EPOLLET, {u32=469430784, u64=128291142562304}}) = 0
[pid  2188] 13:28:34.390188 epoll_pwait(9, [{EPOLLIN, {u32=469430784, u64=128291142562304}}], 512, 30000, NULL, 8) = 1
[pid  2188] 13:28:34.390245 recvfrom(3, "GET /index.html HTTP/1.0\r\nHost: "..., 2048, 0, NULL, NULL) = 93
[pid  2188] 13:28:34.390462 writev(3, [{iov_base="HTTP/1.1 200 OK\r\nServer: nginx/1"..., iov_len=142}], 1) = 142
[pid  2188] 13:28:34.390602 close(3)    = 0

user nginx;
worker_processes 1;
working_directory /var/www;

events {
    accept_mutex off;
    worker_connections 4096;
}
http {
    include /etc/nginx/mime.types;
    server {
        listen 8081;
        server_name "10.0.0.110";
        return 200;
    }
}

简短的回答是：
有时更快意味着更慢

---

答案很长：

在单个请求设置中（一次一个请求）

epoll\u wait

始终阻止并强制进行上下文切换。客户端（

ab

在这种情况下）必须接收并处理响应并发送下一个请求，然后才能再次唤醒服务器并返回挂起的套接字。服务器生成响应的速度不会改变这种行为，因为客户端总是在发送下一个请求之前等待服务器响应

在多请求设置中，如果

epoll\u wait

强制进行上下文切换，则取决于服务器的性能。如果服务器应用程序速度较慢，则下一个请求已在等待的可能性更高；在这种情况下，不需要进行上下文切换，因为

epoll\u wait

可以立即返回。上下文切换代价高昂，可能比下一个请求到达所需的时间更长，这是浪费时间

当我通过将超时设置为零来消除执行上下文切换的机会时，我意识到了这一点。这将强制

epoll\u wait

立即返回，即使没有挂起的请求（因此它从不等待，因此没有强制上下文切换）。在这种情况下，即使在多请求设置中，测试应用程序的性能也优于nginx

我进一步证实了我的理论：

• 从nginx（禁用访问日志）中删除工作，这会使其速度变慢
• 向测试应用程序（自旋循环）添加工作使其更快

在多请求设置中

---

为了得到这个答案，我重新测量了所有的尺寸。所有的测试都是在一个更新的ubuntu机器上运行的，它有8个内核（超过5个），服务器和客户端同时在这台机器上运行。测试应用程序保持不变

• gcc（Ubuntu 7.3.0-16ubuntu3）7.3.0
• nginx版本：nginx/1.14.0（Ubuntu）

我写了一个小脚本，在下面列出了请求/秒数，它运行

ab

，60.000次请求，100次选择最快的结果

maxRequests=0
for ((i = 0; i < 100; ++i)); do
    requests=$(ab -n 60000 -c (1 or 4) http://127.0.0.1:8081/index.html 2>&1 | grep "Requests per second" | cut -d" " -f7)
    requests=${requests%.*}
    maxRequests=$(( maxRequests > requests ? maxRequests : requests ))
done
echo ${maxRequests}

为了使nginx更快（因此使用4c会更慢），我禁用了访问日志

Requests per second: 12028 (nginx, 1c, no access-log)
Requests per second: 28976 (nginx, 4c, no access-log)

如果测试应用程序通过将

epool\u wait

-时间设置为零而从不休眠，则结果如下。
有趣的是，1c设置中的差异显示了在新请求到达时唤醒测试应用程序的成本

Requests per second: 20079 (test-application, 1c, spinning)
Requests per second: 34522 (test-application, 4c, spinning)

对于这些最后的测量，我在测试应用程序中添加了一些工作，人为地降低了速度，以增加防止上下文切换的机会。我将下面的代码添加到内部for循环的末尾（在

关闭之后），并改变
j
的初始值以得到不同的结果

uint8_t j = 50;
while (--j != 0) {
    uint8_t i = 0;
    while (--i != 0)
        asm("");
}

添加了-tt的输出您是否尝试过使用
perf record
？这可能会很有帮助。如果我没记错的话，nginx使用一个主/从线程池。主线程侦听新的请求并将它们卸载给跟随者，因此它是一个非常轻量级的多线程模型。我能够复制OP的发现——对于单个客户端或少量客户端，这段代码比nginx快，但对于大量并发客户端，nginx——即使在前台运行——也比nginx快。上下文开关的数量有很大的不同-nginx只需要大约60%的上下文开关。我已经封锁了信号，增加了cpu亲和力，增加了更多的非封锁io提示，重新排列了代码顺序以尽可能多地脱离热路径，但没有能够重现nginx的低上下文开关号。还有一个有趣的结果是，nginx只产生了60%的自愿上下文开关
Requests per second: 12028 (nginx, 1c, no access-log)
Requests per second: 28976 (nginx, 4c, no access-log)

Requests per second: 20079 (test-application, 1c, spinning)
Requests per second: 34522 (test-application, 4c, spinning)

uint8_t j = 50;
while (--j != 0) {
    uint8_t i = 0;
    while (--i != 0)
        asm("");
}

Requests per second: 12910 (test-application, 1c, j=50)
Requests per second: 12126 (test-application, 1c, j=100)
Requests per second: 11634 (test-application, 1c, j=150)
Requests per second: 11020 (test-application, 1c, j=200)
Requests per second: 10235 (test-application, 1c, j=250)
Requests per second: 27447 (test-application, 4c, j=25)
Requests per second: 29464 (test-application, 4c, j=50)
Requests per second: 31334 (test-application, 4c, j=75)
Requests per second: 32079 (test-application, 4c, j=100)
Requests per second: 33510 (test-application, 4c, j=125)
Requests per second: 34241 (test-application, 4c, j=150)
Requests per second: 34189 (test-application, 4c, j=175)
Requests per second: 33855 (test-application, 4c, j=200)
Requests per second: 33328 (test-application, 4c, j=250)