Date: Fri, 17 Jan 2020 10:37:58 +0000 From: bugzilla-noreply@freebsd.org To: net@FreeBSD.org Subject: [Bug 243392] vmx driver input buffer corruption Message-ID: <bug-243392-7501-1Wb8PJ05RE@https.bugs.freebsd.org/bugzilla/> In-Reply-To: <bug-243392-7501@https.bugs.freebsd.org/bugzilla/> References: <bug-243392-7501@https.bugs.freebsd.org/bugzilla/>
next in thread | previous in thread | raw e-mail | index | archive | help
https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=243392 --- Comment #4 from alexandr.oleynikov@gmail.com --- I did some more tests. Hope this will provide some more information. First one with recompiled kernel with TSO patch. As a network load was a file coping to server using samba ifconfig vmx1 vmx1: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 9000 options=e403bb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,TSO4,TSO6,VLAN_HWTSO,RXCSUM_IPV6,TXCSUM_IPV6> ether 00:50:56:be:f0:13 inet 172.31.255.2 netmask 0xffffff00 broadcast 172.31.255.255 media: Ethernet autoselect status: active nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL> # tcpdump -i vmx1 icmp & # tcpdump: verbose output suppressed, use -v or -vv for full protocol decode listening on vmx1, link-type EN10MB (Ethernet), capture size 262144 bytes # ping -s 8000 172.31.255.3 PING 172.31.255.3 (172.31.255.3): 8000 data bytes 11:59:07.108253 IP 172.31.255.2 > 172.31.255.3: ICMP echo request, id 30548, seq 0, length 8008 11:59:07.108425 IP 172.31.255.3 > 172.31.255.2: ICMP echo reply, id 30548, seq 0, length 8008 8008 bytes from 172.31.255.3: icmp_seq=0 ttl=128 time=0.226 ms 11:59:08.126583 IP 172.31.255.2 > 172.31.255.3: ICMP echo request, id 30548, seq 1, length 8008 11:59:08.126754 IP 172.31.255.3 > 172.31.255.2: ICMP echo reply, id 30548, seq 1, length 8008 8008 bytes from 172.31.255.3: icmp_seq=1 ttl=128 time=0.213 ms --- skipped some lines --- 12:00:20.401492 IP 172.31.255.2 > 172.31.255.3: ICMP echo request, id 30548, seq 71, length 8008 8008 bytes from 172.31.255.3: icmp_seq=71 ttl=128 time=0.550 ms 12:00:20.402010 IP 172.31.255.3 > 172.31.255.2: ICMP echo reply, id 30548, seq 71, length 8008 12:00:21.408758 IP 172.31.255.2 > 172.31.255.3: ICMP echo request, id 30548, seq 72, length 8008 8008 bytes from 172.31.255.3: icmp_seq=72 ttl=128 time=2.303 ms 12:00:21.410995 IP 172.31.255.3 > 172.31.255.2: ICMP echo reply, id 30548, seq 72, length 8008 12:00:24.527165 IP 172.31.255.2 > 172.31.255.3: ICMP echo request, id 30548, seq 73, length 8008 8008 bytes from 172.31.255.3: icmp_seq=73 ttl=128 time=133.291 ms 12:00:24.592341 IP 172.31.255.3 > 172.31.255.2: ICMP echo reply, id 30548, seq 73, length 8008 12:00:25.569300 IP 172.31.255.2 > 172.31.255.3: ICMP echo request, id 30548, seq 74, length 8008 12:00:25.662953 IP 172.31.255.3 > 172.31.255.2: ICMP echo reply, id 30548, seq 74, length 8008 --- after seqnum 73 packets received by kernel and seen with tcpdump but not returned to ping process --- skipped some lines --- 12:01:27.114142 IP 172.31.255.2 > 172.31.255.3: ICMP echo request, id 30548, seq 134, length 8008 12:01:27.160943 IP 172.31.255.3 > 172.31.255.2: ICMP echo reply, id 30548, seq 134, length 8008 12:01:28.125972 IP 172.31.255.2 > 172.31.255.3: ICMP echo request, id 30548, seq 135, length 8008 12:01:28.126346 IP truncated-ip - 7982 bytes missing! 172.31.255.3 > 172.31.255.2: ICMP echo reply, id 30548, seq 135, length 8008 --- received malformed L2 frame from seqnum >= 135 12:01:29.198552 IP 172.31.255.2 > 172.31.255.3: ICMP echo request, id 30548, seq 136, length 8008 12:01:29.223302 IP truncated-ip - 7810 bytes missing! 172.31.255.3 > 172.31.255.2: ICMP echo reply, id 30548, seq 136, length 8008 12:01:30.214849 IP 172.31.255.2 > 172.31.255.3: ICMP echo request, id 30548, seq 137, length 8008 12:01:30.221687 IP truncated-ip - 7822 bytes missing! 172.31.255.3 > 172.31.255.2: ICMP echo reply, id 30548, seq 137, length 8008 12:01:31.246460 IP 172.31.255.2 > 172.31.255.3: ICMP echo request, id 30548, seq 138, length 8008 --- skip some lines 12:01:37.514942 IP 172.31.255.2 > 172.31.255.3: ICMP echo request, id 30548, seq 144, length 8008 12:01:37.517865 IP truncated-ip - 7808 bytes missing! 172.31.255.3 > 172.31.255.2: ICMP echo reply, id 30548, seq 144, length 8008 12:01:38.579626 IP 172.31.255.2 > 172.31.255.3: ICMP echo request, id 30548, seq 145, length 8008 12:01:38.615120 IP truncated-ip - 7928 bytes missing! 172.31.255.3 > 172.31.255.2: ICMP echo reply, id 30548, seq 145, length 8008 12:01:39.603253 IP 172.31.255.2 > 172.31.255.3: ICMP echo request, id 30548, seq 146, length 8008 12:01:40.614996 IP 172.31.255.2 > 172.31.255.3: ICMP echo request, id 30548, seq 147, length 8008 12:01:40.615183 IP 172.31.255.3 > 172.31.255.2: ICMP echo reply, id 30548, seq 146, length 8008 --- difference in 1 second between sending and receiveng reply from seqnum 146 12:01:40.615201 IP truncated-ip - 7928 bytes missing! 172.31.255.3 > 172.31.255.2: ICMP echo reply, id 30548, seq 147, length 8008 8008 bytes from 172.31.255.3: icmp_seq=146 ttl=128 time=1011.985 ms 12:01:41.657600 IP 172.31.255.2 > 172.31.255.3: ICMP echo request, id 30548, seq 148, length 8008 12:01:42.701072 IP 172.31.255.2 > 172.31.255.3: ICMP echo request, id 30548, seq 149, length 8008 12:01:42.701321 IP 172.31.255.3 > 172.31.255.2: ICMP echo reply, id 30548, seq 148, length 8008 8008 bytes from 172.31.255.3: icmp_seq=148 ttl=128 time=1043.763 ms 12:01:43.615120 IP truncated-ip - 7928 bytes missing! 172.31.255.3 > 172.31.255.2: ICMP echo reply, id 30548, seq 149, length 8008 12:01:43.714982 IP 172.31.255.2 > 172.31.255.3: ICMP echo request, id 30548, seq 150, length 8008 12:01:43.988367 IP truncated-ip - 7808 bytes missing! 172.31.255.3 > 172.31.255.2: ICMP echo reply, id 30548, seq 150, length 8008 12:01:44.787457 IP 172.31.255.2 > 172.31.255.3: ICMP echo request, id 30548, seq 151, length 8008 12:01:44.788966 IP truncated-ip - 7782 bytes missing! 172.31.255.3 > 172.31.255.2: ICMP echo reply, id 30548, seq 151, length 8008 12:01:45.815011 IP 172.31.255.2 > 172.31.255.3: ICMP echo request, id 30548, seq 152, length 8008 12:01:45.970727 IP truncated-ip - 7976 bytes missing! 172.31.255.3 > 172.31.255.2: ICMP echo reply, id 30548, seq 152, length 8008 12:01:46.834089 IP 172.31.255.2 > 172.31.255.3: ICMP echo request, id 30548, seq 153, length 8008 12:01:47.615212 IP truncated-ip - 7928 bytes missing! 172.31.255.3 > 172.31.255.2: ICMP echo reply, id 30548, seq 153, length 8008 12:01:47.897600 IP 172.31.255.2 > 172.31.255.3: ICMP echo request, id 30548, seq 154, length 8008 12:01:48.914981 IP 172.31.255.2 > 172.31.255.3: ICMP echo request, id 30548, seq 155, length 8008 12:01:48.915192 IP 172.31.255.3 > 172.31.255.2: ICMP echo reply, id 30548, seq 154, length 8008 8008 bytes from 172.31.255.3: icmp_seq=154 ttl=128 time=1017.638 ms --- some packet reveived undamaged but with delay in 1 second When i try using iperf as network load source in most cases was kernel panic as result: Fatal trap 12: page fault while in kernel mode cpuid = 1; apic id = 02 fault virtual address = 0x0 fault code = supervisor write data, page not present instruction pointer = 0x20:0xffffffff80cef252 stack pointer = 0x28:0xfffffe00753547c0 frame pointer = 0x28:0xfffffe00753548a0 code segment = base 0x0, limit 0xfffff, type 0x1b = DPL 0, pres 1, long 1, def32 0, gran 1 processor eflags = interrupt enabled, resume, IOPL = 0 current process = 0 (if_io_tqg_1) trap number = 12 panic: page fault cpuid = 1 time = 1579255990 KDB: stack backtrace: #0 0xffffffff80c1d297 at kdb_backtrace+0x67 #1 0xffffffff80bd05cd at vpanic+0x19d #2 0xffffffff80bd0423 at panic+0x43 #3 0xffffffff810a7d2c at trap_fatal+0x39c #4 0xffffffff810a7d79 at trap_pfault+0x49 #5 0xffffffff810a736f at trap+0x29f #6 0xffffffff81081a0c at calltrap+0x8 #7 0xffffffff80ce9be5 at _task_fn_rx+0x75 #8 0xffffffff80c1bb54 at gtaskqueue_run_locked+0x144 #9 0xffffffff80c1b7b8 at gtaskqueue_thread_loop+0x98 #10 0xffffffff80b90c23 at fork_exit+0x83 #11 0xffffffff81082a4e at fork_trampoline+0xe Uptime: 16m42s Then reverting to default kernel, disabling tso and reboot: # uname -a FreeBSD ******************* 12.1-RELEASE-p1 FreeBSD 12.1-RELEASE-p1 GENERIC amd64 # ifconfig vmx1 vmx1: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 9000 options=e400bb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,VLAN_HWTSO,RXCSUM_IPV6,TXCSUM_IPV6> ether 00:50:56:be:f0:13 inet 172.31.255.2 netmask 0xffffff00 broadcast 172.31.255.255 media: Ethernet autoselect status: active nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL> # iperf3 -c 172.31.255.2 -p 1234 Connecting to host 172.31.255.2, port 1234 [ 5] local 172.31.255.5 port 32466 connected to 172.31.255.2 port 1234 [ ID] Interval Transfer Bitrate Retr Cwnd [ 5] 0.00-1.03 sec 497 MBytes 4.05 Gbits/sec 11 8.74 KBytes [ 5] 1.03-2.07 sec 0.00 Bytes 0.00 bits/sec 3 8.74 KBytes [ 5] 2.07-3.06 sec 0.00 Bytes 0.00 bits/sec 1 8.74 KBytes [ 5] 3.06-4.02 sec 0.00 Bytes 0.00 bits/sec 1 8.74 KBytes [ 5] 4.02-5.01 sec 0.00 Bytes 0.00 bits/sec 0 8.74 KBytes [ 5] 5.01-6.03 sec 0.00 Bytes 0.00 bits/sec 1 8.74 KBytes [ 5] 6.03-7.06 sec 0.00 Bytes 0.00 bits/sec 0 8.74 KBytes [ 5] 7.06-8.04 sec 0.00 Bytes 0.00 bits/sec 0 8.74 KBytes [ 5] 8.04-9.07 sec 0.00 Bytes 0.00 bits/sec 0 8.74 KBytes [ 5] 9.07-10.01 sec 0.00 Bytes 0.00 bits/sec 1 8.74 KBytes - - - - - - - - - - - - - - - - - - - - - - - - - [ ID] Interval Transfer Bitrate Retr [ 5] 0.00-10.01 sec 497 MBytes 416 Mbits/sec 18 sender [ 5] 0.00-10.60 sec 496 MBytes 393 Mbits/sec receiver # ping -s 8000 172.31.255.5 PING 172.31.255.5 (172.31.255.5): 8000 data bytes 8008 bytes from 172.31.255.5: icmp_seq=0 ttl=64 time=0.322 ms 12:22:09.903151 IP 172.31.255.2 > 172.31.255.5: ICMP echo request, id 10122, seq 0, length 8008 12:22:09.903253 IP 172.31.255.5 > 172.31.255.2: ICMP echo reply, id 10122, seq 0, length 8008 12:22:10.922205 IP 172.31.255.2 > 172.31.255.5: ICMP echo request, id 10122, seq 1, length 8008 12:22:10.922300 IP 172.31.255.5 > 172.31.255.2: ICMP echo reply, id 10122, seq 1, length 8008 8008 bytes from 172.31.255.5: icmp_seq=1 ttl=64 time=0.147 ms 12:22:11.969930 IP 172.31.255.2 > 172.31.255.5: ICMP echo request, id 10122, seq 2, length 8008 12:22:11.970035 IP 172.31.255.5 > 172.31.255.2: ICMP echo reply, id 10122, seq 2, length 8008 8008 bytes from 172.31.255.5: icmp_seq=2 ttl=64 time=0.159 ms 12:22:12.997254 IP 172.31.255.2 > 172.31.255.5: ICMP echo request, id 10122, seq 3, length 8008 12:22:12.997386 IP 172.31.255.5 > 172.31.255.2: ICMP echo reply, id 10122, seq 3, length 8008 8008 bytes from 172.31.255.5: icmp_seq=3 ttl=64 time=0.175 ms 12:22:14.029823 IP 172.31.255.2 > 172.31.255.5: ICMP echo request, id 10122, seq 4, length 8008 12:22:14.030017 IP 172.31.255.5 > 172.31.255.2: ICMP echo reply, id 10122, seq 4, length 8008 8008 bytes from 172.31.255.5: icmp_seq=4 ttl=64 time=0.237 ms 12:22:15.058570 IP 172.31.255.2 > 172.31.255.5: ICMP echo request, id 10122, seq 5, length 8008 12:22:15.058769 IP 172.31.255.5 > 172.31.255.2: ICMP echo reply, id 10122, seq 5, length 8008 8008 bytes from 172.31.255.5: icmp_seq=5 ttl=64 time=0.241 ms 12:22:16.096803 IP 172.31.255.2 > 172.31.255.5: ICMP echo request, id 10122, seq 6, length 8008 12:22:16.096896 IP 172.31.255.5 > 172.31.255.2: ICMP echo reply, id 10122, seq 6, length 8008 8008 bytes from 172.31.255.5: icmp_seq=6 ttl=64 time=0.139 ms 12:22:17.136966 IP 172.31.255.2 > 172.31.255.5: ICMP echo request, id 10122, seq 7, length 8008 12:22:17.137224 IP 172.31.255.5 > 172.31.255.2: ICMP echo reply, id 10122, seq 7, length 8008 12:22:18.164014 IP 172.31.255.2 > 172.31.255.5: ICMP echo request, id 10122, seq 8, length 8008 12:22:18.164194 IP 172.31.255.5 > 172.31.255.2: ICMP echo reply, id 10122, seq 8, length 8008 --- packets stops sending to ping process --- skip some lines --- -- but after some time packets againg sending to ping process 12:26:15.636917 IP 172.31.255.2 > 172.31.255.5: ICMP echo request, id 10122, seq 238, length 8008 12:26:15.637147 IP 172.31.255.5 > 172.31.255.2: ICMP echo reply, id 10122, seq 238, length 8008 12:26:16.696907 IP 172.31.255.2 > 172.31.255.5: ICMP echo request, id 10122, seq 239, length 8008 12:26:16.697100 IP 172.31.255.5 > 172.31.255.2: ICMP echo reply, id 10122, seq 239, length 8008 8008 bytes from 172.31.255.5: icmp_seq=239 ttl=64 time=0.256 ms 12:26:17.756044 IP 172.31.255.2 > 172.31.255.5: ICMP echo request, id 10122, seq 240, length 8008 12:26:17.756178 IP 172.31.255.5 > 172.31.255.2: ICMP echo reply, id 10122, seq 240, length 8008 8008 bytes from 172.31.255.5: icmp_seq=240 ttl=64 time=0.190 ms 12:26:18.796861 IP 172.31.255.2 > 172.31.255.5: ICMP echo request, id 10122, seq 241, length 8008 12:26:18.796982 IP 172.31.255.5 > 172.31.255.2: ICMP echo reply, id 10122, seq 241, length 8008 8008 bytes from 172.31.255.5: icmp_seq=241 ttl=64 time=0.176 ms 12:26:19.836847 IP 172.31.255.2 > 172.31.255.5: ICMP echo request, id 10122, seq 242, length 8008 12:26:19.836981 IP 172.31.255.5 > 172.31.255.2: ICMP echo reply, id 10122, seq 242, length 8008 8008 bytes from 172.31.255.5: icmp_seq=242 ttl=64 time=0.192 ms -- You are receiving this mail because: You are the assignee for the bug.
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?bug-243392-7501-1Wb8PJ05RE>