Performance linux内核模块及top输出的性能分析

我已经为一些数据包处理编写了一个内核模块。数据包通过10Gig卡进入系统。卡的所有中断都映射到最后8个CPU

当数据包进入模块时，根据某些逻辑，它使用队列\工作\将数据包推送到15个队列中的1个队列。然后，数据包上的工作线程由运行在CPU（1到15）上的工作线程进行处理，工作队列将排到该CPU。在我看来，从顶部输出来看，这15个CPU中有近80-85%是空闲的，但这些工作线程的CPU接近55-60%

queue_work_on 每CPU统计数据和在这些CPU上运行的工作线程的CPU利用率之间如何不匹配

此外，我还看到，尽管数据包到达机器和模块的速率相同，但每个CPU的利用率是不同的。所有数据包之间的处理也保持不变。我该如何分析这种行为

提前谢谢

请在下面找到上面提到的顶部输出

Cpu0 : 0.0%us, 13.6%sy, 0.0%ni, 86.4%id, 0.0%wa, 0.0%hi, 0.0%si, 0.0%st Cpu1 : 0.0%us, 14.6%sy, 0.0%ni, 83.5%id, 0.0%wa, 0.0%hi, 1.9%si, 0.0%st Cpu2 : 0.0%us, 13.6%sy, 0.0%ni, 86.4%id, 0.0%wa, 0.0%hi, 0.0%si, 0.0%st Cpu3 : 0.0%us, 13.9%sy, 0.0%ni, 86.1%id, 0.0%wa, 0.0%hi, 0.0%si, 0.0%st Cpu4 : 0.0%us, 13.7%sy, 0.0%ni, 86.3%id, 0.0%wa, 0.0%hi, 0.0%si, 0.0%st Cpu5 : 0.0%us, 14.7%sy, 0.0%ni, 84.3%id, 0.0%wa, 0.0%hi, 1.0%si, 0.0%st Cpu6 : 0.0%us, 13.6%sy, 0.0%ni, 86.4%id, 0.0%wa, 0.0%hi, 0.0%si, 0.0%st Cpu7 : 0.0%us, 14.6%sy, 0.0%ni, 83.5%id, 0.0%wa, 0.0%hi, 1.9%si, 0.0%st Cpu8 : 0.0%us, 12.5%sy, 0.0%ni, 86.5%id, 0.0%wa, 0.0%hi, 1.0%si, 0.0%st Cpu9 : 0.0%us, 12.6%sy, 0.0%ni, 85.4%id, 0.0%wa, 0.0%hi, 1.9%si, 0.0%st Cpu10 : 0.0%us, 12.6%sy, 0.0%ni, 86.4%id, 0.0%wa, 0.0%hi, 1.0%si, 0.0%st Cpu11 : 0.0%us, 15.4%sy, 0.0%ni, 82.7%id, 0.0%wa, 0.0%hi, 1.9%si, 0.0%st Cpu12 : 0.0%us, 13.6%sy, 0.0%ni, 86.4%id, 0.0%wa, 0.0%hi, 0.0%si, 0.0%st Cpu13 : 0.0%us, 14.6%sy, 0.0%ni, 83.5%id, 0.0%wa, 0.0%hi, 1.9%si, 0.0%st Cpu14 : 0.0%us, 13.7%sy, 0.0%ni, 86.3%id, 0.0%wa, 0.0%hi, 0.0%si, 0.0%st Cpu15 : 1.0%us, 1.0%sy, 0.0%ni, 98.1%id, 0.0%wa, 0.0%hi, 0.0%si, 0.0%st Cpu16 : 0.0%us, 1.0%sy, 0.0%ni, 89.4%id, 0.0%wa, 0.0%hi, 9.6%si, 0.0%st Cpu17 : 0.0%us, 0.0%sy, 0.0%ni, 87.3%id, 0.0%wa, 0.0%hi, 12.7%si, 0.0%st Cpu18 : 0.0%us, 0.0%sy, 0.0%ni, 90.3%id, 0.0%wa, 0.0%hi, 9.7%si, 0.0%st Cpu19 : 0.0%us, 1.0%sy, 0.0%ni, 68.6%id, 0.0%wa, 0.0%hi, 30.4%si, 0.0%st Cpu20 : 0.0%us, 0.0%sy, 0.0%ni, 92.2%id, 0.0%wa, 0.0%hi, 7.8%si, 0.0%st Cpu21 : 0.0%us, 0.0%sy, 0.0%ni, 91.2%id, 0.0%wa, 0.0%hi, 8.8%si, 0.0%st Cpu22 : 0.0%us, 0.0%sy, 0.0%ni, 92.2%id, 0.0%wa, 0.0%hi, 7.8%si, 0.0%st Cpu23 : 0.0%us, 0.0%sy, 0.0%ni, 73.1%id, 0.0%wa, 0.0%hi, 26.9%si, 0.0%st Mem: 49500248k total, 3662876k used, 45837372k free, 44612k buffers Swap: 1566332k total, 0k used, 1566332k free, 262376k cached PID USER PR NI VIRT RES SHR %CPU %MEM TIME+ RUSER P TIME nDRT WCHAN COMMAND 1451 root 20 0 0 0 0 55 0.0 21:21.77 root 12 21:21 0 worker_th [kworker/12:1] 20966 root 20 0 0 0 0 55 0.0 4:17.46 root 2 4:17 0 worker_th [kworker/2:2] 106 root 20 0 0 0 0 54 0.0 57:44.05 root 1 57:44 0 worker_th [kworker/1:1] 1450 root 20 0 0 0 0 54 0.0 19:15.19 root 13 19:15 0 worker_th [kworker/13:1] 22094 root 20 0 0 0 0 54 0.0 0:57.30 root 0 0:57 0 worker_th [kworker/0:0] 1449 root 20 0 0 0 0 53 0.0 21:14.16 root 14 21:14 0 worker_th [kworker/14:1] 100 root 20 0 0 0 0 51 0.0 20:35.85 root 7 20:35 0 worker_th [kworker/7:1] 103 root 20 0 0 0 0 51 0.0 18:32.88 root 4 18:32 0 worker_th [kworker/4:1] 96 root 20 0 0 0 0 50 0.0 18:25.24 root 11 18:25 0 worker_th [kworker/11:1] 97 root 20 0 0 0 0 50 0.0 20:14.37 root 10 20:14 0 worker_th [kworker/10:1] 98 root 20 0 0 0 0 50 0.0 20:24.93 root 9 20:24 0 worker_th [kworker/9:1] 101 root 20 0 0 0 0 50 0.0 18:33.93 root 6 18:33 0 worker_th [kworker/6:1] 102 root 20 0 0 0 0 50 0.0 18:40.52 root 5 18:40 0 worker_th [kworker/5:1] 99 root 20 0 0 0 0 50 0.0 20:10.48 root 8 20:10 0 worker_th [kworker/8:1] 104 root 20 0 0 0 0 49 0.0 17:33.27 root 3 17:33 0 worker_th [kworker/3:1] 1443 root 20 0 0 0 0 2 0.0 0:33.51 root 20 0:33 0 worker_th [kworker/20:1] 21979 root 20 0 2276 1076 756 2 0.0 0:04.07 root 19 0:04 0 - ./top -d 1 1440 root 20 0 0 0 0 1 0.0 0:29.94 root 23 0:29 0 worker_th [kworker/23:1] 1441 root 20 0 0 0 0 1 0.0 0:29.91 root 22 0:29 0 worker_th [kworker/22:1] 1444 root 20 0 0 0 0 1 0.0 0:40.17 root 19 0:40 0 worker_th [kworker/19:1] 1445 root 20 0 0 0 0 1 0.0 2:12.32 root 18 2:12 0 worker_th [kworker/18:1] Cpu0:0.0%us、13.6%sy、0.0%ni、86.4%id、0.0%wa、0.0%hi、0.0%si、0.0%st Cpu1:0.0%us、14.6%sy、0.0%ni、83.5%id、0.0%wa、0.0%hi、1.9%si、0.0%st Cpu2:0.0%us、13.6%sy、0.0%ni、86.4%id、0.0%wa、0.0%hi、0.0%si、0.0%st Cpu3:0.0%us、13.9%sy、0.0%ni、86.1%id、0.0%wa、0.0%hi、0.0%si、0.0%st Cpu4:0.0%us、13.7%sy、0.0%ni、86.3%id、0.0%wa、0.0%hi、0.0%si、0.0%st Cpu5:0.0%us、14.7%sy、0.0%ni、84.3%id、0.0%wa、0.0%hi、1.0%si、0.0%st Cpu6:0.0%us、13.6%sy、0.0%ni、86.4%id、0.0%wa、0.0%hi、0.0%si、0.0%st Cpu7:0.0%us、14.6%sy、0.0%ni、83.5%id、0.0%wa、0.0%hi、1.9%si、0.0%st Cpu8:0.0%us、12.5%sy、0.0%ni、86.5%id、0.0%wa、0.0%hi、1.0%si、0.0%st Cpu9:0.0%us、12.6%sy、0.0%ni、85.4%id、0.0%wa、0.0%hi、1.9%si、0.0%st Cpu10:0.0%us、12.6%sy、0.0%ni、86.4%id、0.0%wa、0.0%hi、1.0%si、0.0%st Cpu11:0.0%us、15.4%sy、0.0%ni、82.7%id、0.0%wa、0.0%hi、1.9%si、0.0%st Cpu12:0.0%us、13.6%sy、0.0%ni、86.4%id、0.0%wa、0.0%hi、0.0%si、0.0%st Cpu13:0.0%us、14.6%sy、0.0%ni、83.5%id、0.0%wa、0.0%hi、1.9%si、0.0%st Cpu14:0.0%us、13.7%sy、0.0%ni、86.3%id、0.0%wa、0.0%hi、0.0%si、0.0%st Cpu15:1.0%us、1.0%sy、0.0%ni、98.1%id、0.0%wa、0.0%hi、0.0%si、0.0%st Cpu16:0.0%us、1.0%sy、0.0%ni、89.4%id、0.0%wa、0.0%hi、9.6%si、0.0%st Cpu17:0.0%us、0.0%sy、0.0%ni、87.3%id、0.0%wa、0.0%hi、12.7%si、0.0%st Cpu18:0.0%us、0.0%sy、0.0%ni、90.3%id、0.0%wa、0.0%hi、9.7%si、0.0%st Cpu19:0.0%us、1.0%sy、0.0%ni、68.6%id、0.0%wa、0.0%hi、30.4%si、0.0%st Cpu20:0.0%us、0.0%sy、0.0%ni、92.2%id、0.0%wa、0.0%hi、7.8%si、0.0%st Cpu21:0.0%us、0.0%sy、0.0%ni、91.2%id、0.0%wa、0.0%hi、8.8%si、0.0%st Cpu22:0.0%us、0.0%sy、0.0%ni、92.2%id、0.0%wa、0.0%hi、7.8%si、0.0%st Cpu23:0.0%us、0.0%sy、0.0%ni、73.1%id、0.0%wa、0.0%hi、26.9%si、0.0%st 内存：总计49500248k，使用3662876k，可用45837372k，缓冲44612k 交换：总计1566332k，已使用0k，免费1566332k，缓存262376k PID用户PR NI VIRT RES SHR%CPU%MEM TIME+RUSER P TIME nDRT WCHAN命令 1451根20 0 0 0 0 55 0.0 21:21.77根12 21:21 0工人[kworker/12:1] 20966根20 0 0 0 0 0 55 0.0 4:17.46根2 4:17 0工人[kworker/2:2] 106根20 0 0 0 0 54 0.0 57:44.05根1 57:44 0工人[kworker/1:1] 1450根20 0 0 0 0 54 0.0 19:15.19根13 19:15 0工人[kworker/13:1] 22094根20 0 0 0 0 0 54 0.0 0 0:57.30根0 0:57 0工人[kworker/0:0] 1449根20 0 0 0 0 53 0.0 21:14.16根14 21:14 0工人[kworker/14:1] 100根20 0 0 0 51 0.0 20:35.85根7 20:35 0工人[kworker/7:1] 103根20 0 0 0 0 51 0.0 18:32.88根4 18:32 0工人[kworker/4:1] 96根20 0 0 0 50 0.0 18:25.24根11 18:25 0工人[kworker/11:1] 97根20 0 0 0 0 50 0.0 20:14.37根10 20:14 0工人[kworker/10:1] 98根20 0 0 0 0 50 0.0 20:24.93根9 20:24 0工人[kworker/9:1] 101根20 0 0 0 0 50 0.0 18:33.93根6 18:33 0工人[kworker/6:1] 102根20 0 0 0 50 0.0 18:40.52根5 18:40 0工人[kworker/5:1] 99根20 0 0 0