Date: Fri, 31 Jan 2014 21:08:59 -0500 From: Mike Tancsa <mike@sentex.net> To: "freebsd-net@freebsd.org" <freebsd-net@freebsd.org> Subject: missing missing packets in igb stats ? Message-ID: <52EC573B.109@sentex.net>
next in thread | raw e-mail | index | archive | help
Hi Jack,
I was testing out forwarding and firewalling speeds of the igb driver on
RELENG_10 and noticed something odd.
I have 2 boxes connected to a FreeBSD box in the middle
FreeBSD-A(em1)-----------(igb1)Router-1(igb0)----------(em1)FreeBSD-B
So Box A generates packets as fast as it can to FreeBSD box B's em1 nic.
Router-1 is a FreeBSD box as releng10. Watching ifstat on Router-1 as
I execute the command on FreeBSD-A
# ./netblast 1.1.1.2 500 100 20
start: 1391219372.477992294
finish: 1391219392.496952108
send calls: 10877557
send errors: 0
approx send rate: 543877
approx error rate: 0
I see on the router-1 box
igb0 igb1
Kbps in Kbps out Kbps in Kbps out
0.00 0.00 0.00 0.00
1600.61 191639.1 280888.7 0.00
3669.84 434348.6 636134.9 0.00
3706.56 438636.7 596650.5 0.00
3755.10 444358.9 562814.3 0.00
3714.89 439478.5 562056.4 0.00
3796.79 449397.9 562042.9 0.00
3786.02 447957.2 577561.4 0.00
3629.18 429453.4 601285.7 0.00
3728.48 441312.7 597785.3 0.00
3806.67 450401.2 596247.0 0.00
3854.79 456150.2 597865.7 0.00
3690.11 436552.1 596695.8 0.00
3676.08 435002.6 596462.8 0.00
3730.35 441535.2 597132.1 0.00
3680.43 435518.2 596960.3 0.00
3741.41 442685.3 597750.8 0.00
3691.93 436870.6 596236.9 0.00
3627.31 429120.5 594116.8 0.00
3661.97 433492.7 595812.0 0.00
3693.86 437169.0 597826.9 0.00
2046.18 240635.3 331656.3 0.00
0.00 0.00 0.00 0.00
0.00 0.00 0.00 0.00
Notice the rate of traffic coming in on igb1 is higher than what is
going out on igb0. Box A thinks it sent traffic at some 536,616 pkts
per second or 590Mb/s. However, traffic going out is slower, and what
is seen at box B is less. It sees the traffic at 286Mb/s and 357,873 pps
Given the lost packets, should this not show up somewhere in the igb
statistics ?
dev.igb.0.%desc: Intel(R) PRO/1000 Network Connection version - 2.4.0
dev.igb.0.%driver: igb
dev.igb.0.%location: slot=0 function=0 handle=\_SB_.PCI0.PEG0.PEGP
dev.igb.0.%pnpinfo: vendor=0x8086 device=0x10c9 subvendor=0x8086
subdevice=0xa03c class=0x020000
dev.igb.0.%parent: pci1
dev.igb.0.nvm: -1
dev.igb.0.enable_aim: 1
dev.igb.0.fc: 3
dev.igb.0.rx_processing_limit: 100
dev.igb.0.link_irq: 2
dev.igb.0.dropped: 0
dev.igb.0.tx_dma_fail: 0
dev.igb.0.rx_overruns: 0
dev.igb.0.watchdog_timeouts: 0
dev.igb.0.device_control: 1488978497
dev.igb.0.rx_control: 67141634
dev.igb.0.interrupt_mask: 4
dev.igb.0.extended_int_mask: 2147483655
dev.igb.0.tx_buf_alloc: 0
dev.igb.0.rx_buf_alloc: 0
dev.igb.0.fc_high_water: 58976
dev.igb.0.fc_low_water: 58960
dev.igb.0.queue0.no_desc_avail: 19682298
dev.igb.0.queue0.tx_packets: 20962740
dev.igb.0.queue0.rx_packets: 1101622
dev.igb.0.queue0.rx_bytes: 66097424
dev.igb.0.queue0.lro_queued: 0
dev.igb.0.queue0.lro_flushed: 0
dev.igb.0.queue1.no_desc_avail: 32582207
dev.igb.0.queue1.tx_packets: 50082567
dev.igb.0.queue1.rx_packets: 6598
dev.igb.0.queue1.rx_bytes: 462728
dev.igb.0.queue1.lro_queued: 0
dev.igb.0.queue1.lro_flushed: 0
dev.igb.0.mac_stats.excess_coll: 0
dev.igb.0.mac_stats.single_coll: 0
dev.igb.0.mac_stats.multiple_coll: 0
dev.igb.0.mac_stats.late_coll: 0
dev.igb.0.mac_stats.collision_count: 0
dev.igb.0.mac_stats.symbol_errors: 0
dev.igb.0.mac_stats.sequence_errors: 0
dev.igb.0.mac_stats.defer_count: 138912
dev.igb.0.mac_stats.missed_packets: 0
dev.igb.0.mac_stats.recv_no_buff: 0
dev.igb.0.mac_stats.recv_undersize: 0
dev.igb.0.mac_stats.recv_fragmented: 0
dev.igb.0.mac_stats.recv_oversize: 0
dev.igb.0.mac_stats.recv_jabber: 0
dev.igb.0.mac_stats.recv_errs: 0
dev.igb.0.mac_stats.crc_errs: 0
dev.igb.0.mac_stats.alignment_errs: 0
dev.igb.0.mac_stats.coll_ext_errs: 0
dev.igb.0.mac_stats.xon_recvd: 550808
dev.igb.0.mac_stats.xon_txd: 0
dev.igb.0.mac_stats.xoff_recvd: 550808
dev.igb.0.mac_stats.xoff_txd: 0
dev.igb.0.mac_stats.total_pkts_recvd: 1108220
dev.igb.0.mac_stats.good_pkts_recvd: 6604
dev.igb.0.mac_stats.bcast_pkts_recvd: 0
dev.igb.0.mac_stats.mcast_pkts_recvd: 0
dev.igb.0.mac_stats.rx_frames_64: 1
dev.igb.0.mac_stats.rx_frames_65_127: 6603
dev.igb.0.mac_stats.rx_frames_128_255: 0
dev.igb.0.mac_stats.rx_frames_256_511: 0
dev.igb.0.mac_stats.rx_frames_512_1023: 0
dev.igb.0.mac_stats.rx_frames_1024_1522: 0
dev.igb.0.mac_stats.good_octets_recvd: 489608
dev.igb.0.mac_stats.good_octets_txd: 10120060648
dev.igb.0.mac_stats.total_pkts_txd: 71045307
dev.igb.0.mac_stats.good_pkts_txd: 71045307
dev.igb.0.mac_stats.bcast_pkts_txd: 2
dev.igb.0.mac_stats.mcast_pkts_txd: 0
dev.igb.0.mac_stats.tx_frames_64: 2
dev.igb.0.mac_stats.tx_frames_65_127: 5051081
dev.igb.0.mac_stats.tx_frames_128_255: 65994224
dev.igb.0.mac_stats.tx_frames_256_511: 0
dev.igb.0.mac_stats.tx_frames_512_1023: 0
dev.igb.0.mac_stats.tx_frames_1024_1522: 0
dev.igb.0.mac_stats.tso_txd: 0
dev.igb.0.mac_stats.tso_ctx_fail: 0
dev.igb.0.interrupts.asserts: 6564060
dev.igb.0.interrupts.rx_pkt_timer: 1108207
dev.igb.0.interrupts.rx_abs_timer: 0
dev.igb.0.interrupts.tx_pkt_timer: 0
dev.igb.0.interrupts.tx_abs_timer: 1108220
dev.igb.0.interrupts.tx_queue_empty: 71044772
dev.igb.0.interrupts.tx_queue_min_thresh: 0
dev.igb.0.interrupts.rx_desc_min_thresh: 0
dev.igb.0.interrupts.rx_overrun: 0
dev.igb.0.host.breaker_tx_pkt: 0
dev.igb.0.host.host_tx_pkt_discard: 0
dev.igb.0.host.rx_pkt: 13
dev.igb.0.host.breaker_rx_pkts: 0
dev.igb.0.host.breaker_rx_pkt_drop: 0
dev.igb.0.host.tx_good_pkt: 535
dev.igb.0.host.breaker_tx_pkt_drop: 0
dev.igb.0.host.rx_good_bytes: 70993032
dev.igb.0.host.tx_good_bytes: 10120060648
dev.igb.0.host.length_errors: 0
dev.igb.0.host.serdes_violation_pkt: 0
dev.igb.0.host.header_redir_missed: 0
dev.igb.0.wake: 0
dev.igb.1.%desc: Intel(R) PRO/1000 Network Connection version - 2.4.0
dev.igb.1.%driver: igb
dev.igb.1.%location: slot=0 function=1
dev.igb.1.%pnpinfo: vendor=0x8086 device=0x10c9 subvendor=0x8086
subdevice=0xa03c class=0x020000
dev.igb.1.%parent: pci1
dev.igb.1.nvm: -1
dev.igb.1.enable_aim: 1
dev.igb.1.fc: 3
dev.igb.1.rx_processing_limit: 100
dev.igb.1.link_irq: 2
dev.igb.1.dropped: 0
dev.igb.1.tx_dma_fail: 0
dev.igb.1.rx_overruns: 0
dev.igb.1.watchdog_timeouts: 0
dev.igb.1.device_control: 1488978497
dev.igb.1.rx_control: 67141634
dev.igb.1.interrupt_mask: 4
dev.igb.1.extended_int_mask: 2147483655
dev.igb.1.tx_buf_alloc: 0
dev.igb.1.rx_buf_alloc: 0
dev.igb.1.fc_high_water: 58976
dev.igb.1.fc_low_water: 58960
dev.igb.1.queue0.no_desc_avail: 0
dev.igb.1.queue0.tx_packets: 14
dev.igb.1.queue0.rx_packets: 27770289
dev.igb.1.queue0.rx_bytes: 3632804418
dev.igb.1.queue0.lro_queued: 0
dev.igb.1.queue0.lro_flushed: 0
dev.igb.1.queue1.no_desc_avail: 0
dev.igb.1.queue1.tx_packets: 6599
dev.igb.1.queue1.rx_packets: 58098597
dev.igb.1.queue1.rx_bytes: 8250006086
dev.igb.1.queue1.lro_queued: 0
dev.igb.1.queue1.lro_flushed: 0
dev.igb.1.mac_stats.excess_coll: 0
dev.igb.1.mac_stats.single_coll: 0
dev.igb.1.mac_stats.multiple_coll: 0
dev.igb.1.mac_stats.late_coll: 0
dev.igb.1.mac_stats.collision_count: 0
dev.igb.1.mac_stats.symbol_errors: 0
dev.igb.1.mac_stats.sequence_errors: 0
dev.igb.1.mac_stats.defer_count: 0
dev.igb.1.mac_stats.missed_packets: 0
dev.igb.1.mac_stats.recv_no_buff: 0
dev.igb.1.mac_stats.recv_undersize: 0
dev.igb.1.mac_stats.recv_fragmented: 0
dev.igb.1.mac_stats.recv_oversize: 0
dev.igb.1.mac_stats.recv_jabber: 0
dev.igb.1.mac_stats.recv_errs: 0
dev.igb.1.mac_stats.crc_errs: 0
dev.igb.1.mac_stats.alignment_errs: 0
dev.igb.1.mac_stats.coll_ext_errs: 0
dev.igb.1.mac_stats.xon_recvd: 0
dev.igb.1.mac_stats.xon_txd: 0
dev.igb.1.mac_stats.xoff_recvd: 0
dev.igb.1.mac_stats.xoff_txd: 0
dev.igb.1.mac_stats.total_pkts_recvd: 85868886
dev.igb.1.mac_stats.good_pkts_recvd: 85868886
dev.igb.1.mac_stats.bcast_pkts_recvd: 31
dev.igb.1.mac_stats.mcast_pkts_recvd: 0
dev.igb.1.mac_stats.rx_frames_64: 5
dev.igb.1.mac_stats.rx_frames_65_127: 6211527
dev.igb.1.mac_stats.rx_frames_128_255: 79657327
dev.igb.1.mac_stats.rx_frames_256_511: 27
dev.igb.1.mac_stats.rx_frames_512_1023: 0
dev.igb.1.mac_stats.rx_frames_1024_1522: 0
dev.igb.1.mac_stats.good_octets_recvd: 12226286048
dev.igb.1.mac_stats.good_octets_txd: 490260
dev.igb.1.mac_stats.total_pkts_txd: 6613
dev.igb.1.mac_stats.good_pkts_txd: 6613
dev.igb.1.mac_stats.bcast_pkts_txd: 4
dev.igb.1.mac_stats.mcast_pkts_txd: 0
dev.igb.1.mac_stats.tx_frames_64: 8
dev.igb.1.mac_stats.tx_frames_65_127: 6605
dev.igb.1.mac_stats.tx_frames_128_255: 0
dev.igb.1.mac_stats.tx_frames_256_511: 0
dev.igb.1.mac_stats.tx_frames_512_1023: 0
dev.igb.1.mac_stats.tx_frames_1024_1522: 0
dev.igb.1.mac_stats.tso_txd: 0
dev.igb.1.mac_stats.tso_ctx_fail: 0
dev.igb.1.interrupts.asserts: 8707927
dev.igb.1.interrupts.rx_pkt_timer: 85867976
dev.igb.1.interrupts.rx_abs_timer: 0
dev.igb.1.interrupts.tx_pkt_timer: 0
dev.igb.1.interrupts.tx_abs_timer: 85868886
dev.igb.1.interrupts.tx_queue_empty: 6613
dev.igb.1.interrupts.tx_queue_min_thresh: 0
dev.igb.1.interrupts.rx_desc_min_thresh: 0
dev.igb.1.interrupts.rx_overrun: 0
dev.igb.1.host.breaker_tx_pkt: 0
dev.igb.1.host.host_tx_pkt_discard: 0
dev.igb.1.host.rx_pkt: 910
dev.igb.1.host.breaker_rx_pkts: 0
dev.igb.1.host.breaker_rx_pkt_drop: 0
dev.igb.1.host.tx_good_pkt: 0
dev.igb.1.host.breaker_tx_pkt_drop: 0
dev.igb.1.host.rx_good_bytes: 12226288092
dev.igb.1.host.tx_good_bytes: 490260
dev.igb.1.host.length_errors: 0
dev.igb.1.host.serdes_violation_pkt: 0
dev.igb.1.host.header_redir_missed: 0
Motherboard is Intel
Base Board Information
Manufacturer: Intel Corporation
Product Name: DH87RL
Version: AAG74240-401
Serial Number: BQRL330000Q9
NIC is dual port
igb0@pci0:1:0:0: class=0x020000 card=0xa03c8086 chip=0x10c98086
rev=0x01 hdr=0x00
vendor = 'Intel Corporation'
device = '82576 Gigabit Network Connection'
class = network
subclass = ethernet
bar [10] = type Memory, range 32, base 0xf7c20000, size 131072,
enabled
bar [14] = type Memory, range 32, base 0xf7800000, size 4194304,
enabled
bar [18] = type I/O Port, range 32, base 0xe020, size 32, enabled
bar [1c] = type Memory, range 32, base 0xf7c44000, size 16384,
enabled
cap 01[40] = powerspec 3 supports D0 D3 current D0
cap 05[50] = MSI supports 1 message, 64 bit, vector masks
cap 11[70] = MSI-X supports 10 messages, enabled
Table in map 0x1c[0x0], PBA in map 0x1c[0x2000]
cap 10[a0] = PCI-Express 2 endpoint max data 128(512) FLR link x4(x4)
speed 2.5(2.5) ASPM disabled(L0s/L1)
ecap 0001[100] = AER 1 0 fatal 0 non-fatal 2 corrected
ecap 0003[140] = Serial 1 90e2baffff5eb48a
ecap 000e[150] = ARI 1
ecap 0010[160] = SRIOV 1
igb1@pci0:1:0:1: class=0x020000 card=0xa03c8086 chip=0x10c98086
rev=0x01 hdr=0x00
vendor = 'Intel Corporation'
device = '82576 Gigabit Network Connection'
class = network
subclass = ethernet
bar [10] = type Memory, range 32, base 0xf7c00000, size 131072,
enabled
bar [14] = type Memory, range 32, base 0xf7000000, size 4194304,
enabled
bar [18] = type I/O Port, range 32, base 0xe000, size 32, enabled
bar [1c] = type Memory, range 32, base 0xf7c40000, size 16384,
enabled
cap 01[40] = powerspec 3 supports D0 D3 current D0
cap 05[50] = MSI supports 1 message, 64 bit, vector masks
cap 11[70] = MSI-X supports 10 messages, enabled
Table in map 0x1c[0x0], PBA in map 0x1c[0x2000]
cap 10[a0] = PCI-Express 2 endpoint max data 128(512) FLR link x4(x4)
speed 2.5(2.5) ASPM disabled(L0s/L1)
ecap 0001[100] = AER 1 0 fatal 0 non-fatal 2 corrected
ecap 0003[140] = Serial 1 90e2baffff5eb48a
ecap 000e[150] = ARI 1
ecap 0010[160] = SRIOV 1
root@intel4gen-9:/usr/home/mdtancsa # netstat -m
6141/6489/12630 mbufs in use (current/cache/total)
6139/5871/12010/487416 mbuf clusters in use (current/cache/total/max)
6139/5861 mbuf+clusters out of packet secondary zone in use (current/cache)
0/5/5/243708 4k (page size) jumbo clusters in use (current/cache/total/max)
0/0/0/72209 9k jumbo clusters in use (current/cache/total/max)
0/0/0/40618 16k jumbo clusters in use (current/cache/total/max)
13813K/13384K/27197K bytes allocated to network (current/cache/total)
0/0/0 requests for mbufs denied (mbufs/clusters/mbuf+clusters)
0/0/0 requests for mbufs delayed (mbufs/clusters/mbuf+clusters)
0/0/0 requests for jumbo clusters delayed (4k/9k/16k)
0/0/0 requests for jumbo clusters denied (4k/9k/16k)
0 requests for sfbufs denied
0 requests for sfbufs delayed
0 requests for I/O initiated by sendfile
root@intel4gen-9:/usr/home/mdtancsa #
--
-------------------
Mike Tancsa, tel +1 519 651 3400
Sentex Communications, mike@sentex.net
Providing Internet services since 1994 www.sentex.net
Cambridge, Ontario Canada http://www.tancsa.com/
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?52EC573B.109>