Date: Fri, 17 Jan 2020 10:37:58 +0000 From: bugzilla-noreply@freebsd.org To: net@FreeBSD.org Subject: [Bug 243392] vmx driver input buffer corruption Message-ID: <bug-243392-7501-1Wb8PJ05RE@https.bugs.freebsd.org/bugzilla/> In-Reply-To: <bug-243392-7501@https.bugs.freebsd.org/bugzilla/> References: <bug-243392-7501@https.bugs.freebsd.org/bugzilla/>
next in thread | previous in thread | raw e-mail | index | archive | help
https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=3D243392 --- Comment #4 from alexandr.oleynikov@gmail.com --- I did some more tests. Hope this will provide some more information. First one with recompiled kernel with TSO patch. As a network load was a fi= le coping to server using samba=20 ifconfig vmx1 vmx1: flags=3D8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0= mtu 9000 =20=20=20=20=20=20=20 options=3De403bb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSU= M,TSO4,TSO6,VLAN_HWTSO,RXCSUM_IPV6,TXCSUM_IPV6> ether 00:50:56:be:f0:13 inet 172.31.255.2 netmask 0xffffff00 broadcast 172.31.255.255 media: Ethernet autoselect status: active nd6 options=3D29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL> # tcpdump -i vmx1 icmp & # tcpdump: verbose output suppressed, use -v or -vv for full protocol decode listening on vmx1, link-type EN10MB (Ethernet), capture size 262144 bytes # ping -s 8000 172.31.255.3 PING 172.31.255.3 (172.31.255.3): 8000 data bytes 11:59:07.108253 IP 172.31.255.2 > 172.31.255.3: ICMP echo request, id 30548, seq 0, length 8008 11:59:07.108425 IP 172.31.255.3 > 172.31.255.2: ICMP echo reply, id 30548, = seq 0, length 8008 8008 bytes from 172.31.255.3: icmp_seq=3D0 ttl=3D128 time=3D0.226 ms 11:59:08.126583 IP 172.31.255.2 > 172.31.255.3: ICMP echo request, id 30548, seq 1, length 8008 11:59:08.126754 IP 172.31.255.3 > 172.31.255.2: ICMP echo reply, id 30548, = seq 1, length 8008 8008 bytes from 172.31.255.3: icmp_seq=3D1 ttl=3D128 time=3D0.213 ms --- skipped some lines --- 12:00:20.401492 IP 172.31.255.2 > 172.31.255.3: ICMP echo request, id 30548, seq 71, length 8008 8008 bytes from 172.31.255.3: icmp_seq=3D71 ttl=3D128 time=3D0.550 ms 12:00:20.402010 IP 172.31.255.3 > 172.31.255.2: ICMP echo reply, id 30548, = seq 71, length 8008 12:00:21.408758 IP 172.31.255.2 > 172.31.255.3: ICMP echo request, id 30548, seq 72, length 8008 8008 bytes from 172.31.255.3: icmp_seq=3D72 ttl=3D128 time=3D2.303 ms 12:00:21.410995 IP 172.31.255.3 > 172.31.255.2: ICMP echo reply, id 30548, = seq 72, length 8008 12:00:24.527165 IP 172.31.255.2 > 172.31.255.3: ICMP echo request, id 30548, seq 73, length 8008 8008 bytes from 172.31.255.3: icmp_seq=3D73 ttl=3D128 time=3D133.291 ms 12:00:24.592341 IP 172.31.255.3 > 172.31.255.2: ICMP echo reply, id 30548, = seq 73, length 8008 12:00:25.569300 IP 172.31.255.2 > 172.31.255.3: ICMP echo request, id 30548, seq 74, length 8008 12:00:25.662953 IP 172.31.255.3 > 172.31.255.2: ICMP echo reply, id 30548, = seq 74, length 8008 --- after seqnum 73 packets received by kernel and seen with tcpdump but not returned to ping process --- skipped some lines ---=20 12:01:27.114142 IP 172.31.255.2 > 172.31.255.3: ICMP echo request, id 30548, seq 134, length 8008 12:01:27.160943 IP 172.31.255.3 > 172.31.255.2: ICMP echo reply, id 30548, = seq 134, length 8008 12:01:28.125972 IP 172.31.255.2 > 172.31.255.3: ICMP echo request, id 30548, seq 135, length 8008 12:01:28.126346 IP truncated-ip - 7982 bytes missing! 172.31.255.3 > 172.31.255.2: ICMP echo reply, id 30548, seq 135, length 8008 --- received malformed L2 frame from seqnum >=3D 135 12:01:29.198552 IP 172.31.255.2 > 172.31.255.3: ICMP echo request, id 30548, seq 136, length 8008 12:01:29.223302 IP truncated-ip - 7810 bytes missing! 172.31.255.3 > 172.31.255.2: ICMP echo reply, id 30548, seq 136, length 8008 12:01:30.214849 IP 172.31.255.2 > 172.31.255.3: ICMP echo request, id 30548, seq 137, length 8008 12:01:30.221687 IP truncated-ip - 7822 bytes missing! 172.31.255.3 > 172.31.255.2: ICMP echo reply, id 30548, seq 137, length 8008 12:01:31.246460 IP 172.31.255.2 > 172.31.255.3: ICMP echo request, id 30548, seq 138, length 8008 --- skip some lines 12:01:37.514942 IP 172.31.255.2 > 172.31.255.3: ICMP echo request, id 30548, seq 144, length 8008 12:01:37.517865 IP truncated-ip - 7808 bytes missing! 172.31.255.3 > 172.31.255.2: ICMP echo reply, id 30548, seq 144, length 8008 12:01:38.579626 IP 172.31.255.2 > 172.31.255.3: ICMP echo request, id 30548, seq 145, length 8008 12:01:38.615120 IP truncated-ip - 7928 bytes missing! 172.31.255.3 > 172.31.255.2: ICMP echo reply, id 30548, seq 145, length 8008 12:01:39.603253 IP 172.31.255.2 > 172.31.255.3: ICMP echo request, id 30548, seq 146, length 8008 12:01:40.614996 IP 172.31.255.2 > 172.31.255.3: ICMP echo request, id 30548, seq 147, length 8008 12:01:40.615183 IP 172.31.255.3 > 172.31.255.2: ICMP echo reply, id 30548, = seq 146, length 8008 --- difference in 1 second between sending and receiveng reply from seqnum = 146 12:01:40.615201 IP truncated-ip - 7928 bytes missing! 172.31.255.3 > 172.31.255.2: ICMP echo reply, id 30548, seq 147, length 8008 8008 bytes from 172.31.255.3: icmp_seq=3D146 ttl=3D128 time=3D1011.985 ms 12:01:41.657600 IP 172.31.255.2 > 172.31.255.3: ICMP echo request, id 30548, seq 148, length 8008 12:01:42.701072 IP 172.31.255.2 > 172.31.255.3: ICMP echo request, id 30548, seq 149, length 8008 12:01:42.701321 IP 172.31.255.3 > 172.31.255.2: ICMP echo reply, id 30548, = seq 148, length 8008 8008 bytes from 172.31.255.3: icmp_seq=3D148 ttl=3D128 time=3D1043.763 ms 12:01:43.615120 IP truncated-ip - 7928 bytes missing! 172.31.255.3 > 172.31.255.2: ICMP echo reply, id 30548, seq 149, length 8008 12:01:43.714982 IP 172.31.255.2 > 172.31.255.3: ICMP echo request, id 30548, seq 150, length 8008 12:01:43.988367 IP truncated-ip - 7808 bytes missing! 172.31.255.3 > 172.31.255.2: ICMP echo reply, id 30548, seq 150, length 8008 12:01:44.787457 IP 172.31.255.2 > 172.31.255.3: ICMP echo request, id 30548, seq 151, length 8008 12:01:44.788966 IP truncated-ip - 7782 bytes missing! 172.31.255.3 > 172.31.255.2: ICMP echo reply, id 30548, seq 151, length 8008 12:01:45.815011 IP 172.31.255.2 > 172.31.255.3: ICMP echo request, id 30548, seq 152, length 8008 12:01:45.970727 IP truncated-ip - 7976 bytes missing! 172.31.255.3 > 172.31.255.2: ICMP echo reply, id 30548, seq 152, length 8008 12:01:46.834089 IP 172.31.255.2 > 172.31.255.3: ICMP echo request, id 30548, seq 153, length 8008 12:01:47.615212 IP truncated-ip - 7928 bytes missing! 172.31.255.3 > 172.31.255.2: ICMP echo reply, id 30548, seq 153, length 8008 12:01:47.897600 IP 172.31.255.2 > 172.31.255.3: ICMP echo request, id 30548, seq 154, length 8008 12:01:48.914981 IP 172.31.255.2 > 172.31.255.3: ICMP echo request, id 30548, seq 155, length 8008 12:01:48.915192 IP 172.31.255.3 > 172.31.255.2: ICMP echo reply, id 30548, = seq 154, length 8008 8008 bytes from 172.31.255.3: icmp_seq=3D154 ttl=3D128 time=3D1017.638 ms --- some packet reveived undamaged but with delay in 1 second When i try using iperf as network load source in most cases was kernel pani= c as result: Fatal trap 12: page fault while in kernel mode cpuid =3D 1; apic id =3D 02 fault virtual address =3D 0x0 fault code =3D supervisor write data, page not present instruction pointer =3D 0x20:0xffffffff80cef252 stack pointer =3D 0x28:0xfffffe00753547c0 frame pointer =3D 0x28:0xfffffe00753548a0 code segment =3D base 0x0, limit 0xfffff, type 0x1b =3D DPL 0, pres 1, long 1, def32 0, gran 1 processor eflags =3D interrupt enabled, resume, IOPL =3D 0 current process =3D 0 (if_io_tqg_1) trap number =3D 12 panic: page fault cpuid =3D 1 time =3D 1579255990 KDB: stack backtrace: #0 0xffffffff80c1d297 at kdb_backtrace+0x67 #1 0xffffffff80bd05cd at vpanic+0x19d #2 0xffffffff80bd0423 at panic+0x43 #3 0xffffffff810a7d2c at trap_fatal+0x39c #4 0xffffffff810a7d79 at trap_pfault+0x49 #5 0xffffffff810a736f at trap+0x29f #6 0xffffffff81081a0c at calltrap+0x8 #7 0xffffffff80ce9be5 at _task_fn_rx+0x75 #8 0xffffffff80c1bb54 at gtaskqueue_run_locked+0x144 #9 0xffffffff80c1b7b8 at gtaskqueue_thread_loop+0x98 #10 0xffffffff80b90c23 at fork_exit+0x83 #11 0xffffffff81082a4e at fork_trampoline+0xe Uptime: 16m42s Then reverting to default kernel, disabling tso and reboot: # uname -a FreeBSD ******************* 12.1-RELEASE-p1 FreeBSD 12.1-RELEASE-p1 GENERI= C=20 amd64 # ifconfig vmx1 vmx1: flags=3D8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 9000 =20=20=20=20=20=20=20 options=3De400bb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSU= M,VLAN_HWTSO,RXCSUM_IPV6,TXCSUM_IPV6> ether 00:50:56:be:f0:13 inet 172.31.255.2 netmask 0xffffff00 broadcast 172.31.255.255 media: Ethernet autoselect status: active nd6 options=3D29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL> # iperf3 -c 172.31.255.2 -p 1234 Connecting to host 172.31.255.2, port 1234 [ 5] local 172.31.255.5 port 32466 connected to 172.31.255.2 port 1234 [ ID] Interval Transfer Bitrate Retr Cwnd [ 5] 0.00-1.03 sec 497 MBytes 4.05 Gbits/sec 11 8.74 KBytes [ 5] 1.03-2.07 sec 0.00 Bytes 0.00 bits/sec 3 8.74 KBytes [ 5] 2.07-3.06 sec 0.00 Bytes 0.00 bits/sec 1 8.74 KBytes [ 5] 3.06-4.02 sec 0.00 Bytes 0.00 bits/sec 1 8.74 KBytes [ 5] 4.02-5.01 sec 0.00 Bytes 0.00 bits/sec 0 8.74 KBytes [ 5] 5.01-6.03 sec 0.00 Bytes 0.00 bits/sec 1 8.74 KBytes [ 5] 6.03-7.06 sec 0.00 Bytes 0.00 bits/sec 0 8.74 KBytes [ 5] 7.06-8.04 sec 0.00 Bytes 0.00 bits/sec 0 8.74 KBytes [ 5] 8.04-9.07 sec 0.00 Bytes 0.00 bits/sec 0 8.74 KBytes [ 5] 9.07-10.01 sec 0.00 Bytes 0.00 bits/sec 1 8.74 KBytes - - - - - - - - - - - - - - - - - - - - - - - - - [ ID] Interval Transfer Bitrate Retr [ 5] 0.00-10.01 sec 497 MBytes 416 Mbits/sec 18 sender [ 5] 0.00-10.60 sec 496 MBytes 393 Mbits/sec recei= ver # ping -s 8000 172.31.255.5 PING 172.31.255.5 (172.31.255.5): 8000 data bytes 8008 bytes from 172.31.255.5: icmp_seq=3D0 ttl=3D64 time=3D0.322 ms 12:22:09.903151 IP 172.31.255.2 > 172.31.255.5: ICMP echo request, id 10122, seq 0, length 8008 12:22:09.903253 IP 172.31.255.5 > 172.31.255.2: ICMP echo reply, id 10122, = seq 0, length 8008 12:22:10.922205 IP 172.31.255.2 > 172.31.255.5: ICMP echo request, id 10122, seq 1, length 8008 12:22:10.922300 IP 172.31.255.5 > 172.31.255.2: ICMP echo reply, id 10122, = seq 1, length 8008 8008 bytes from 172.31.255.5: icmp_seq=3D1 ttl=3D64 time=3D0.147 ms 12:22:11.969930 IP 172.31.255.2 > 172.31.255.5: ICMP echo request, id 10122, seq 2, length 8008 12:22:11.970035 IP 172.31.255.5 > 172.31.255.2: ICMP echo reply, id 10122, = seq 2, length 8008 8008 bytes from 172.31.255.5: icmp_seq=3D2 ttl=3D64 time=3D0.159 ms 12:22:12.997254 IP 172.31.255.2 > 172.31.255.5: ICMP echo request, id 10122, seq 3, length 8008 12:22:12.997386 IP 172.31.255.5 > 172.31.255.2: ICMP echo reply, id 10122, = seq 3, length 8008 8008 bytes from 172.31.255.5: icmp_seq=3D3 ttl=3D64 time=3D0.175 ms 12:22:14.029823 IP 172.31.255.2 > 172.31.255.5: ICMP echo request, id 10122, seq 4, length 8008 12:22:14.030017 IP 172.31.255.5 > 172.31.255.2: ICMP echo reply, id 10122, = seq 4, length 8008 8008 bytes from 172.31.255.5: icmp_seq=3D4 ttl=3D64 time=3D0.237 ms 12:22:15.058570 IP 172.31.255.2 > 172.31.255.5: ICMP echo request, id 10122, seq 5, length 8008 12:22:15.058769 IP 172.31.255.5 > 172.31.255.2: ICMP echo reply, id 10122, = seq 5, length 8008 8008 bytes from 172.31.255.5: icmp_seq=3D5 ttl=3D64 time=3D0.241 ms 12:22:16.096803 IP 172.31.255.2 > 172.31.255.5: ICMP echo request, id 10122, seq 6, length 8008 12:22:16.096896 IP 172.31.255.5 > 172.31.255.2: ICMP echo reply, id 10122, = seq 6, length 8008 8008 bytes from 172.31.255.5: icmp_seq=3D6 ttl=3D64 time=3D0.139 ms 12:22:17.136966 IP 172.31.255.2 > 172.31.255.5: ICMP echo request, id 10122, seq 7, length 8008 12:22:17.137224 IP 172.31.255.5 > 172.31.255.2: ICMP echo reply, id 10122, = seq 7, length 8008 12:22:18.164014 IP 172.31.255.2 > 172.31.255.5: ICMP echo request, id 10122, seq 8, length 8008 12:22:18.164194 IP 172.31.255.5 > 172.31.255.2: ICMP echo reply, id 10122, = seq 8, length 8008 --- packets stops sending to ping process --- skip some lines --- -- but after some time packets againg sending to ping process 12:26:15.636917 IP 172.31.255.2 > 172.31.255.5: ICMP echo request, id 10122, seq 238, length 8008 12:26:15.637147 IP 172.31.255.5 > 172.31.255.2: ICMP echo reply, id 10122, = seq 238, length 8008 12:26:16.696907 IP 172.31.255.2 > 172.31.255.5: ICMP echo request, id 10122, seq 239, length 8008 12:26:16.697100 IP 172.31.255.5 > 172.31.255.2: ICMP echo reply, id 10122, = seq 239, length 8008 8008 bytes from 172.31.255.5: icmp_seq=3D239 ttl=3D64 time=3D0.256 ms 12:26:17.756044 IP 172.31.255.2 > 172.31.255.5: ICMP echo request, id 10122, seq 240, length 8008 12:26:17.756178 IP 172.31.255.5 > 172.31.255.2: ICMP echo reply, id 10122, = seq 240, length 8008 8008 bytes from 172.31.255.5: icmp_seq=3D240 ttl=3D64 time=3D0.190 ms 12:26:18.796861 IP 172.31.255.2 > 172.31.255.5: ICMP echo request, id 10122, seq 241, length 8008 12:26:18.796982 IP 172.31.255.5 > 172.31.255.2: ICMP echo reply, id 10122, = seq 241, length 8008 8008 bytes from 172.31.255.5: icmp_seq=3D241 ttl=3D64 time=3D0.176 ms 12:26:19.836847 IP 172.31.255.2 > 172.31.255.5: ICMP echo request, id 10122, seq 242, length 8008 12:26:19.836981 IP 172.31.255.5 > 172.31.255.2: ICMP echo reply, id 10122, = seq 242, length 8008 8008 bytes from 172.31.255.5: icmp_seq=3D242 ttl=3D64 time=3D0.192 ms --=20 You are receiving this mail because: You are the assignee for the bug.=
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?bug-243392-7501-1Wb8PJ05RE>