From owner-freebsd-bugs@freebsd.org Thu Jan 16 14:32:54 2020 Return-Path: Delivered-To: freebsd-bugs@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id 804D91E96EA for ; Thu, 16 Jan 2020 14:32:54 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from mailman.nyi.freebsd.org (unknown [127.0.1.3]) by mx1.freebsd.org (Postfix) with ESMTP id 47z69f2wFRz44vj for ; Thu, 16 Jan 2020 14:32:54 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: by mailman.nyi.freebsd.org (Postfix) id 640251E96E9; Thu, 16 Jan 2020 14:32:54 +0000 (UTC) Delivered-To: bugs@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id 62BBB1E96E8 for ; Thu, 16 Jan 2020 14:32:54 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from mxrelay.nyi.freebsd.org (mxrelay.nyi.freebsd.org [IPv6:2610:1c1:1:606c::19:3]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) server-signature RSA-PSS (4096 bits) client-signature RSA-PSS (4096 bits) client-digest SHA256) (Client CN "mxrelay.nyi.freebsd.org", Issuer "Let's Encrypt Authority X3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 47z69f1wRsz44vh for ; Thu, 16 Jan 2020 14:32:54 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from kenobi.freebsd.org (kenobi.freebsd.org [IPv6:2610:1c1:1:606c::50:1d]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mxrelay.nyi.freebsd.org (Postfix) with ESMTPS id 3D59120F07 for ; Thu, 16 Jan 2020 14:32:54 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from kenobi.freebsd.org ([127.0.1.5]) by kenobi.freebsd.org (8.15.2/8.15.2) with ESMTP id 00GEWs4e001794 for ; Thu, 16 Jan 2020 14:32:54 GMT (envelope-from bugzilla-noreply@freebsd.org) Received: (from www@localhost) by kenobi.freebsd.org (8.15.2/8.15.2/Submit) id 00GEWs0U001793 for bugs@FreeBSD.org; Thu, 16 Jan 2020 14:32:54 GMT (envelope-from bugzilla-noreply@freebsd.org) X-Authentication-Warning: kenobi.freebsd.org: www set sender to bugzilla-noreply@freebsd.org using -f From: bugzilla-noreply@freebsd.org To: bugs@FreeBSD.org Subject: [Bug 243392] vmx driver input buffer corruption Date: Thu, 16 Jan 2020 14:32:54 +0000 X-Bugzilla-Reason: AssignedTo X-Bugzilla-Type: new X-Bugzilla-Watch-Reason: None X-Bugzilla-Product: Base System X-Bugzilla-Component: kern X-Bugzilla-Version: 12.1-RELEASE X-Bugzilla-Keywords: X-Bugzilla-Severity: Affects Some People X-Bugzilla-Who: alexandr.oleynikov@gmail.com X-Bugzilla-Status: New X-Bugzilla-Resolution: X-Bugzilla-Priority: --- X-Bugzilla-Assigned-To: bugs@FreeBSD.org X-Bugzilla-Flags: X-Bugzilla-Changed-Fields: bug_id short_desc product version rep_platform op_sys bug_status bug_severity priority component assigned_to reporter Message-ID: Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Bugzilla-URL: https://bugs.freebsd.org/bugzilla/ Auto-Submitted: auto-generated MIME-Version: 1.0 X-BeenThere: freebsd-bugs@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Bug reports List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 16 Jan 2020 14:32:54 -0000 https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=3D243392 Bug ID: 243392 Summary: vmx driver input buffer corruption Product: Base System Version: 12.1-RELEASE Hardware: Any OS: Any Status: New Severity: Affects Some People Priority: --- Component: kern Assignee: bugs@FreeBSD.org Reporter: alexandr.oleynikov@gmail.com Tested on 12.1-Release, 12.1-Release-p1 and with patch regarding tso from https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=3D236999 All VM running on the same host with VMware ESXi, 6.7.0, 15160138=20 1 VM - FreeBSD 12.1 2 VM - Any OS (tested with different, second one is irellevant) After some network load packets in the interface queue being to get stuck Reproduce in differents enviroment with different netowrk load Easiest way to reproduce - copy large file from second VM to FreeBSD using samba or ftp. Some output examples: 172.31.255.2 - FreeBSD 172.31.255.3 - second VM Ping: 8008 bytes from 172.31.255.3: icmp_seq=3D321 ttl=3D128 time=3D0.375 ms 8008 bytes from 172.31.255.3: icmp_seq=3D322 ttl=3D128 time=3D0.229 ms 8008 bytes from 172.31.255.3: icmp_seq=3D323 ttl=3D128 time=3D0.179 ms 8008 bytes from 172.31.255.3: icmp_seq=3D324 ttl=3D128 time=3D0.203 ms 8008 bytes from 172.31.255.3: icmp_seq=3D325 ttl=3D128 time=3D0.278 ms 8008 bytes from 172.31.255.3: icmp_seq=3D326 ttl=3D128 time=3D0.178 ms 8008 bytes from 172.31.255.3: icmp_seq=3D327 ttl=3D128 time=3D2.204 ms 8008 bytes from 172.31.255.3: icmp_seq=3D328 ttl=3D128 time=3D0.226 ms 8008 bytes from 172.31.255.3: icmp_seq=3D329 ttl=3D128 time=3D0.221 ms 8008 bytes from 172.31.255.3: icmp_seq=3D330 ttl=3D128 time=3D0.466 ms 8008 bytes from 172.31.255.3: icmp_seq=3D331 ttl=3D128 time=3D0.253 ms=20= =20=20 8008 bytes from 172.31.255.3: icmp_seq=3D350 ttl=3D128 time=3D1020.494 ms = <-- start file copy 8008 bytes from 172.31.255.3: icmp_seq=3D362 ttl=3D128 time=3D1037.839 ms 8008 bytes from 172.31.255.3: icmp_seq=3D381 ttl=3D128 time=3D4162.085 ms 8008 bytes from 172.31.255.3: icmp_seq=3D382 ttl=3D128 time=3D5202.940 ms 8008 bytes from 172.31.255.3: icmp_seq=3D383 ttl=3D128 time=3D6200.121 ms 8008 bytes from 172.31.255.3: icmp_seq=3D385 ttl=3D128 time=3D7261.939 ms 8008 bytes from 172.31.255.3: icmp_seq=3D386 ttl=3D128 time=3D7200.745 ms 8008 bytes from 172.31.255.3: icmp_seq=3D390 ttl=3D128 time=3D4100.259 ms tcpdump after propblem appear: # tcpdump -i vmx1 icmp tcpdump: verbose output suppressed, use -v or -vv for full protocol decode listening on vmx1, link-type EN10MB (Ethernet), capture size 262144 bytes 16:09:41.700833 IP 172.31.255.2 > 172.31.255.3: ICMP echo request, id 23149, seq 0, length 8008 16:09:42.727307 IP 172.31.255.2 > 172.31.255.3: ICMP echo request, id 23149, seq 1, length 8008 16:09:43.761866 IP 172.31.255.2 > 172.31.255.3: ICMP echo request, id 23149, seq 2, length 8008 16:09:43.762117 IP 172.31.255.3 > 172.31.255.2: ICMP echo reply, id 23149, = seq 0, length 8008 16:09:44.771901 IP 172.31.255.2 > 172.31.255.3: ICMP echo request, id 23149, seq 3, length 8008 16:09:44.978178 IP truncated-ip - 7810 bytes missing! 172.31.255.3 > 172.31.255.2: ICMP echo reply, id 23149, seq 1, length 8008 16:09:45.788996 IP 172.31.255.2 > 172.31.255.3: ICMP echo request, id 23149, seq 4, length 8008 16:09:46.039983 IP truncated-ip - 7928 bytes missing! 172.31.255.3 > 172.31.255.2: ICMP echo reply, id 23149, seq 2, length 8008 16:09:46.290583 IP truncated-ip - 7928 bytes missing! 172.31.255.3 > 172.31.255.2: ICMP echo reply, id 23149, seq 3, length 8008 16:09:46.801911 IP 172.31.255.2 > 172.31.255.3: ICMP echo request, id 23149, seq 5, length 8008 16:09:47.861870 IP 172.31.255.2 > 172.31.255.3: ICMP echo request, id 23149, seq 6, length 8008 16:09:48.071886 IP truncated-ip - 7822 bytes missing! 172.31.255.3 > 172.31.255.2: ICMP echo reply, id 23149, seq 4, length 8008 16:09:48.871905 IP 172.31.255.2 > 172.31.255.3: ICMP echo request, id 23149, seq 7, length 8008 16:09:49.802033 IP truncated-ip - 7928 bytes missing! 172.31.255.3 > 172.31.255.2: ICMP echo reply, id 23149, seq 5, length 8008 16:09:49.891036 IP 172.31.255.2 > 172.31.255.3: ICMP echo request, id 23149, seq 8, length 8008 16:09:50.802001 IP truncated-ip - 7928 bytes missing! 172.31.255.3 > 172.31.255.2: ICMP echo reply, id 23149, seq 6, length 8008 16:09:50.901923 IP 172.31.255.2 > 172.31.255.3: ICMP echo request, id 23149, seq 9, length 8008 16:09:51.961848 IP 172.31.255.2 > 172.31.255.3: ICMP echo request, id 23149, seq 10, length 8008 Seems like new arrived packets pushed out old packets from the queue After some time even more than 40 seconds gap can exist between sending and receiving packet. tcpdump running on the same time on second VM shows sending a response immediately after receiving request. Disabling/enabling any driver features like tso, csum, etc, changing mtu irrelevant to the problem. --=20 You are receiving this mail because: You are the assignee for the bug.=