eBPF技术实践之virtio-net网卡队列可观测

在系统领域中，最具挑战性的问题通常是组件之间的边界定位。其中，virtio-net前后端的定界尤为困难。当网络报文从内核发送到virtio-net后端，或者从virtio-net后端发送到内核时，这一路径难以进行观测。一些复杂的网络抖动问题很可能是由于网卡队列不正常工作引起的。为了解决这类问题，我们基于eBPF技术扩展了网卡队列的可观测能力，使得virtio网卡前后端的定界问题不再困扰。

virtio-net 前后端驱动简介

virtio-net （后面称为 virtio 网卡）通常由两个组件组成：virtio driver（也称为virtio前端）和virtio device（也称为virtio后端）。virtio前端运行在客户机的内核中，而virtio后端可以由宿主机的内核承担。virtio网卡通常支持多队列，包括发送队列和接收队列。每个队列通过三个 ring 来实现，即avail ring、used ring和desc ring。现在我们将重点介绍 virtio 网卡前端的报文发送和接收流程，以更好地理解整个工作流程。

virtio 网卡前端发送报文

virto网卡前端发送报文主要流程包括：

a.start_xmit：virtio网卡驱动的报文发送入口函数会首先清理已发送的报文，即通过调用free_old_xmit_skbs函数来释放描述符中的报文，直到avail->idx等于used->idx为止；

b.xmit_skb：主要是为报文添加vnet_hdr头部信息，并将skb以scatter-gather形式显示，以记录报文数据的地址和长度信息；

c.virtqueue_add_outbuf：进行DMA映射，将scatter-gather记录的报文数据地址和长度信息添加到desc环中，并增加avail->idx的值；

d.virtqueue_notify：当发送队列存在数据，则通知后端。

virtio 网卡前端接收报文

virito网阿卡前端接收报文主要流程包括：

a.网卡硬中断：硬中断会将napi加入到CPU的处理队列，并启用中断抑制，以及触发软中断；

b.net_rx_action：网络软中断入口函数；

c.virtnet_poll：这个函数是virtio网卡的NAPI poll的回调函数。如果当前队列是发送队列，它将清理发送队列，也就是执行virtnet_poll_cleantx函数。如果当前队列是接收队列，它将进行报文的接收；

d.virtnet_receive：根据used->idx的值，从描述符环中读取报文数据，并更新last_used_idx。内核会为报文数据分配skb，并进入GRO流程，进行报文的合并；e.try_fill_recv：要给desc环添加空的内存区域，并增加avail->idx的值，以确保接收队列始终有可用的内存；

f.virtqueue_napi_complete：当接收的报文数量少于预定的budget（一般为64）时，表示没有更多的数据可以接收。这时，调用virtqueue_napi_complete来表示单次napi处理完毕。同时，通过virtqueue_enable_cb_prepare来关闭中断抑制。

网卡队列可观测

经过前面的分析，我们了解到virtio网卡队列中的几个重要参数，即avail->idx、used->idx和last_used_idx。使用这些参数，我们可以清晰地了解网卡队列当前包含的报文数量，并进一步得到以下可观测指标：

a.发送队列报文数：表示尚未被virtio网卡后端发送的报文数量。计算方法是avail->idx - used->idx；

b.接收队列报文数：表示尚未被virtio网卡前端接收的报文数量。计算方法是used->idx - last_used_idx；

c.网卡队列的last_used_idx：表示virtio网卡后端处理报文的进度；

d.队列饱和度：表示当前网卡队列使用量，计算方法是队列报文数/队列长度。

工作原理

我们将可观测的代码集成在了rtrace的工具里，rtrace是龙蜥社区推出的系统工具集SysAK的一个网络诊断分析工具，关于rtrace的具体原理，我们将在下回分析，eBPF 具体代码请参考代码：

https://gitee.com/anolis/sysak/blob/opensource_branch_sync/source/tools/detect/net/rtrace/src/bpf/virtio.bpf.c

virtio 网卡队列指标采集的主要流程如下：

a.rtrace挂载eBPF采集程序到内核dev_id_show和dev_port_show函数；

b.rtrace周期性读取/sys/class/net/[interface]/dev_id和/sys/class/net/[interface]/dev_port两个文件，其中dev_id文件用来表示采集发送队列信息，dev_port文件用来表示采集接收队列信息；

c.当读取文件时，会触发内核执行dev_id_show和dev_port_show两个函数。由于已经挂载了eBPF采集程序，内核会先执行eBPF采集程序；

d.eBPF采集程序通过解析dev_id_show和dev_port_show入参struct net_device获取网卡队列vring，然后从vring中解析出avail idx、used idx、队列长度和last_used_idx；

e.将数据发送给rtrace做进一步处理。

故障检测

下面是rtrace采集的网卡队列信息输出。

我们可以看到0926的1号发送队列的饱和度和last_used_idx分别是0.05%/3593，0928的1号发送队列的饱和度和last_used_idx分别是0.07%/3593，可以看到发送队列的饱和度在增加，但是last_used_idx在多个采集周期内保持不变。因此，可以确定1号发送队列出现了故障。

随后我们修复了1号发送队列故障，可以看见在0906的1号发送队列饱和度和last_used_idx分别是0.00%/3599，队列里面不再有驻留的报文，恢复了正常。

0924
SendQueue0.05%/35930.00%/8520.00%/45060.00%/16000.00%/4570.00%/5090.00%/31400.00%/13520.00%/3860.00%/4100.00%/17140.00%/17580.00%/16190.00%/4460.00%/35770.00%/24430.00%/460.00%/940.00%/2120.00%/2310.00%/1460.00%/1480.00%/2260.00%/640.00%/1090.00%/840.00%/780.00%/560.00%/870.00%/880.00%/850.00%/52
RecvQueue0.00%/28050.00%/132970.00%/4750.00%/3670.00%/123780.00%/1300.00%/2220.00%/111200.00%/3550.00%/30160.00%/1330.00%/1800.00%/129800.00%/103630.00%/28250.00%/6500.00%/1510.00%/5050.00%/51800.00%/2000.00%/266700.00%/1690.00%/10420.00%/98200.00%/95860.00%/33740.00%/2290.00%/14020.00%/87960.00%/1170.00%/3010.00%/275
0925
SendQueue0.05%/35930.00%/8520.00%/45060.00%/16000.00%/4570.00%/5090.00%/31400.00%/13520.00%/3860.00%/4100.00%/17140.00%/17580.00%/16190.00%/4460.00%/35770.00%/24440.00%/460.00%/940.00%/2120.00%/2310.00%/1460.00%/1480.00%/2260.00%/640.00%/1090.00%/840.00%/780.00%/560.00%/870.00%/890.00%/850.00%/52
RecvQueue0.00%/28050.00%/132970.00%/4750.00%/3670.00%/123780.00%/1300.00%/2220.00%/111200.00%/3550.00%/30160.00%/1330.00%/1800.00%/129800.00%/103630.00%/28250.00%/6500.00%/1510.00%/5050.00%/51800.00%/2000.00%/266700.00%/1690.00%/10420.00%/98200.00%/95860.00%/33740.00%/2290.00%/14020.00%/87960.00%/1170.00%/3030.00%/275
0926
SendQueue0.05%/35930.00%/8520.00%/45060.00%/16000.00%/4570.00%/5090.00%/31400.00%/13520.00%/3860.00%/4100.00%/17140.00%/17580.00%/16190.00%/4460.00%/35770.00%/24440.00%/460.00%/940.00%/2120.00%/2310.00%/1460.00%/1480.00%/2260.00%/640.00%/1090.00%/840.00%/780.00%/560.00%/870.00%/910.00%/850.00%/52
RecvQueue0.00%/28050.00%/132970.00%/4750.00%/3670.00%/123780.00%/1300.00%/2220.00%/111200.00%/3550.00%/30160.00%/1330.00%/1800.00%/129800.00%/103630.00%/28250.00%/6500.00%/1510.00%/5050.00%/51800.00%/2000.00%/266700.00%/1690.00%/10420.00%/98200.00%/95860.00%/33740.00%/2290.00%/14020.00%/87960.00%/1170.00%/3050.00%/275
0927
SendQueue0.07%/35930.00%/8520.00%/45060.00%/16000.00%/4570.00%/5090.00%/31400.00%/13520.00%/3860.00%/4100.00%/17140.00%/17580.00%/16190.00%/4460.00%/35770.00%/24440.00%/460.00%/940.00%/2120.00%/2310.00%/1460.00%/1480.00%/2260.00%/640.00%/1090.00%/840.00%/780.00%/560.00%/870.00%/930.00%/850.00%/52
RecvQueue0.00%/28050.00%/132980.00%/4750.00%/3670.00%/123780.00%/1300.00%/2220.00%/111200.00%/3550.00%/30160.00%/1330.00%/1800.00%/129800.00%/103630.00%/28250.00%/6500.00%/1510.00%/5050.00%/51800.00%/2000.00%/266700.00%/1690.00%/10420.00%/98200.00%/95860.00%/33740.00%/2290.00%/14020.00%/87960.00%/1170.00%/3070.00%/275
0928
SendQueue0.07%/35930.00%/8520.00%/45060.00%/16000.00%/4570.00%/5090.00%/31400.00%/13520.00%/3860.00%/4140.00%/17140.00%/17580.00%/16190.00%/4460.00%/35770.00%/24450.00%/460.00%/940.00%/2120.00%/2310.00%/1460.00%/1490.00%/2260.00%/640.00%/1090.00%/840.00%/780.00%/560.00%/870.00%/960.00%/870.00%/52
RecvQueue0.00%/28050.00%/132980.00%/4750.00%/3670.00%/123780.00%/1300.00%/2220.00%/111200.00%/3550.00%/30160.00%/1330.00%/1800.00%/129800.00%/103630.00%/28250.00%/6500.00%/1510.00%/5050.00%/51800.00%/2050.00%/266700.00%/1690.00%/10420.00%/98200.00%/95860.00%/33740.00%/2290.00%/14020.00%/87970.00%/1180.00%/3090.00%/275
0929
SendQueue0.07%/35930.00%/8520.00%/45060.00%/16000.00%/4570.00%/5090.00%/31400.00%/13520.00%/3860.00%/4140.00%/17140.00%/17580.00%/16190.00%/4460.00%/35770.00%/24450.00%/460.00%/940.00%/2120.00%/2310.00%/1460.00%/1490.00%/2260.00%/640.00%/1090.00%/840.00%/780.00%/560.00%/870.00%/980.00%/870.00%/52
RecvQueue0.00%/28050.00%/132980.00%/4750.00%/3670.00%/123780.00%/1300.00%/2220.00%/111200.00%/3550.00%/30160.00%/1330.00%/1800.00%/129800.00%/103630.00%/28250.00%/6500.00%/1510.00%/5050.00%/51800.00%/2050.00%/266700.00%/1690.00%/10420.00%/98200.00%/95860.00%/33740.00%/2290.00%/14020.00%/87970.00%/1180.00%/3110.00%/275
0930
SendQueue0.07%/35930.00%/8520.00%/45060.00%/16000.00%/4570.00%/5090.00%/31400.00%/13520.00%/3860.00%/4140.00%/17140.00%/17580.00%/16190.00%/4460.00%/35770.00%/24450.00%/460.00%/940.00%/2120.00%/2310.00%/1460.00%/1490.00%/2260.00%/640.00%/1090.00%/840.00%/780.00%/560.00%/870.00%/1000.00%/870.00%/52
RecvQueue0.00%/28050.00%/132980.00%/4750.00%/3670.00%/123780.00%/1300.00%/2220.00%/111200.00%/3550.00%/30160.00%/1330.00%/1800.00%/129800.00%/103630.00%/28250.00%/6500.00%/1510.00%/5050.00%/51800.00%/2050.00%/266700.00%/1690.00%/10420.00%/98200.00%/95860.00%/33740.00%/2290.00%/14020.00%/87970.00%/1180.00%/3130.00%/275
//...省略
0906
SendQueue0.00%/35990.00%/8560.00%/45110.00%/16020.00%/4650.00%/5100.00%/31400.00%/13520.00%/3860.00%/4200.00%/17160.00%/17660.00%/16190.00%/4480.00%/35780.00%/24510.00%/460.00%/940.00%/2120.00%/2310.00%/1480.00%/1490.00%/2260.00%/640.00%/1090.00%/850.00%/870.00%/560.00%/870.00%/1010.00%/1030.00%/52
RecvQueue0.00%/28070.00%/132990.00%/4770.00%/3690.00%/123780.00%/1400.00%/2230.00%/111200.00%/3550.00%/30320.00%/1420.00%/1800.00%/129800.00%/103630.00%/28250.00%/6520.00%/1510.00%/5050.00%/51800.00%/2050.00%/266700.00%/1700.00%/10570.00%/98200.00%/95860.00%/33740.00%/2300.00%/14140.00%/88000.00%/1180.00%/3270.00%/275

总结

在virtio网卡中，前端和后端之间通过共享的网卡队列进行通信。为了更好地理解和观测网卡队列的状态和性能指标，通过观测avail idx、used idx、last_used_idx等指标，我们可以对virtio网卡的性能进行评估和优化。同时，这些指标也为我们提供了对网卡队列状态的深入理解，有助于进行故障排查和性能调优。