Nginx代理高IO读取

我们有一个带有10 Gbps专用接口的大型SSD存储服务器，连接到10个小型服务器（内部网络），每个服务器有1 Gbps专用接口和1 Gbps公用接口，它们通过Nginx代理作为存储的客户端

存储的主要用途是将大文件下载到客户端；500MB~3GB文件

我们注意到，大型服务器具有约3.5 GB/s（~28 Gbps）的超高IO读取，而专用网络接口仅向小型服务器发送约6 Gbps的数据

blockdev --setra 8192 /dev/sda

我对代理缓冲区、代理缓冲区和输出缓冲区进行了更改，但没有效果

对于大型文件，nginx代理正常吗？ 是否有一种解决方案可以将磁盘IO降低到与客户端实际消耗的磁盘IO相同的水平

小型服务器上的Nginx代理配置：

proxy_cache                 off;
proxy_max_temp_file_size    0;
proxy_request_buffering     on;
proxy_buffering             on;
proxy_buffer_size           512k;
proxy_buffers               1   512k;

大型服务器上的Nginx配置（源）：

iotop：

Total DISK READ :       3.06 G/s | Total DISK WRITE :       0.00 B/s
Actual DISK READ:       3.052.82 | Actual DISK WRITE:    1006.26 K/s
   TID  PRIO  USER     DISK READ  DISK WRITE  SWAPIN     IO>    COMMAND
172187 be/4 nginx     354.04 M/s    0.00 B/s  0.00 % 16.68 % nginx: worker process
172057 be/4 nginx     169.40 M/s    0.00 B/s  0.00 % 12.48 % nginx: worker process
171642 be/4 nginx     216.52 M/s    0.00 B/s  0.00 % 11.94 % nginx: worker process
172139 be/4 nginx     126.93 M/s    0.00 B/s  0.00 %  7.67 % nginx: worker process
171599 be/4 nginx     175.99 M/s    0.00 B/s  0.00 %  6.95 % nginx: worker process
171407 be/4 nginx     124.73 M/s    0.00 B/s  0.00 %  5.28 % nginx: worker process
171385 be/4 nginx       2.18 M/s    0.00 B/s  0.00 %  3.94 % nginx: worker process
172603 be/4 nginx       3.65 M/s    0.00 B/s  0.00 %  2.34 % nginx: worker process
172659 be/4 nginx     898.32 K/s    0.00 B/s  0.00 %  2.21 % nginx: worker process
171602 be/4 nginx      24.44 M/s    0.00 B/s  0.00 %  2.17 % nginx: worker process
172874 be/4 nginx       3.35 M/s    0.00 B/s  0.00 %  2.05 % nginx: worker process
172801 be/4 nginx       3.50 M/s    0.00 B/s  0.00 %  1.78 % nginx: worker process
172309 be/4 nginx      14.12 M/s    0.00 B/s  0.00 %  1.67 % nginx: worker process
172326 be/4 nginx       3.74 M/s    0.00 B/s  0.00 %  1.65 % nginx: worker process
172870 be/4 nginx       2.14 M/s    0.00 B/s  0.00 %  1.58 % nginx: worker process
172173 be/4 nginx       6.53 M/s    0.00 B/s  0.00 %  1.54 % nginx: worker process
172360 be/4 nginx      22.98 M/s    0.00 B/s  0.00 %  1.50 % nginx: worker process
171219 be/4 nginx      45.75 M/s    0.00 B/s  0.00 %  1.49 % nginx: worker process
172794 be/4 nginx      12.21 M/s    0.00 B/s  0.00 %  1.45 % nginx: worker process
172127 be/4 nginx     449.16 K/s    0.00 B/s  0.00 %  1.42 % nginx: worker process
172511 be/4 nginx    1239.55 K/s    0.00 B/s  0.00 %  1.35 % nginx: worker process
172709 be/4 nginx       5.84 M/s    0.00 B/s  0.00 %  1.33 % nginx: worker process
172337 be/4 nginx       4.94 M/s    0.00 B/s  0.00 %  1.33 % nginx: worker process
172122 be/4 nginx       4.80 M/s    0.00 B/s  0.00 %  1.31 % nginx: worker process
172617 be/4 nginx       4.65 M/s    0.00 B/s  0.00 %  1.29 % nginx: worker process
172857 be/4 nginx      17.41 M/s    0.00 B/s  0.00 %  1.26 % nginx: worker process
172530 be/4 nginx     449.16 K/s    0.00 B/s  0.00 %  1.26 % nginx: worker process
172775 be/4 nginx       3.25 M/s    0.00 B/s  0.00 %  1.26 % nginx: worker process
172295 be/4 nginx     685.93 K/s    0.00 B/s  0.00 %  1.25 % nginx: worker process
172840 be/4 nginx       5.41 M/s    0.00 B/s  0.00 %  1.24 % nginx: worker process
172827 be/4 nginx      21.82 M/s    0.00 B/s  0.00 %  1.23 % nginx: worker process
172633 be/4 nginx      29.20 M/s    0.00 B/s  0.00 %  1.18 % nginx: worker process
172137 be/4 nginx     449.16 K/s    0.00 B/s  0.00 %  1.14 % nginx: worker process
172348 be/4 nginx       7.21 M/s    0.00 B/s  0.00 %  1.13 % nginx: worker process

iostat：

Device:         rrqm/s   wrqm/s     r/s     w/s    rkB/s    wkB/s avgrq-sz avgqu-sz   await r_await w_await  svctm  %util
md0               0.00     0.00 13177.50    0.00 3144810.00     0.00   477.30     0.00    0.00    0.00    0.00   0.00   0.00

---

尝试增加blockdev预读，以减少小型服务器上的IO等待

blockdev --setra 8192 /dev/sda