From owner-freebsd-net@FreeBSD.ORG Fri Oct 26 09:09:26 2012 Return-Path: Delivered-To: freebsd-net@FreeBSD.org Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id 2D33631A; Fri, 26 Oct 2012 09:09:26 +0000 (UTC) (envelope-from remi.pauchet@netasq.com) Received: from work.netasq.com (gwlille.netasq.com [91.212.116.1]) by mx1.freebsd.org (Postfix) with ESMTP id DAE5B8FC08; Fri, 26 Oct 2012 09:09:24 +0000 (UTC) Received: from [10.2.9.2] (unknown [91.212.116.2]) by work.netasq.com (Postfix) with ESMTPSA id 1250027056EE; Fri, 26 Oct 2012 11:09:23 +0200 (CEST) Subject: Re: panic ixgbevf / SMP under high network load Mime-Version: 1.0 (Apple Message framework v1283) Content-Type: multipart/signed; boundary="Apple-Mail=_9A3C1F97-28A1-4218-B71B-E277637F857D"; protocol="application/pkcs7-signature"; micalg=sha1 From: =?iso-8859-1?Q?R=E9mi_Pauchet?= In-Reply-To: <20121025164031.GA70741@FreeBSD.org> Date: Fri, 26 Oct 2012 11:09:20 +0200 Message-Id: References: <20121025164031.GA70741@FreeBSD.org> To: Gleb Smirnoff X-Mailer: Apple Mail (2.1283) X-Content-Filtered-By: Mailman/MimeDel 2.1.14 Cc: freebsd-net@FreeBSD.org X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 26 Oct 2012 09:09:26 -0000 --Apple-Mail=_9A3C1F97-28A1-4218-B71B-E277637F857D Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset=iso-8859-1 Hi I have the same crash with FreeBSD 10-current FreeBSD freebsd10 10.0-CURRENT FreeBSD 10.0-CURRENT #4 r241761: Sat Oct = 20 07:40:33 UTC 2012 = root@kaos.glenbarber.us:/usr/obj/usr/src/sys/GENERIC amd64 Sorry for the screenshots: panic doesn't dump the memory to the swap, I = can't figure out why I use udp frames (size 700) so this load is not supposed to produce ip = fragmentation. And again, the panic happens with 4 vcpus, no issue with 1 vcpu. Regards, R=E9mi Le 25 oct. 2012 =E0 18:40, Gleb Smirnoff a =E9crit : > On Thu, Oct 25, 2012 at 10:06:40AM +0200, R?mi Pauchet wrote: > R> I'm testing network performance of FreeBSD using vmware esxi 5.1 = with SR-IOV > R>=20 > R> I'm using FreeBSD 8.3 kernel GENERIC, 4 cpus, ixgbevf driver with = an Intel 82599EB dual 10 Gbps network interface > R>=20 > R> After a few seconds of udp ipv4 load (5Gbps x2, frame size 700), I = have the following panic : > R>=20 > R> (kgdb) bt > R> #0 doadump () at pcpu.h:224 > R> #1 0xffffffff8060ab90 in boot (howto=3D260) at = /usr/src/sys/kern/kern_shutdown.c:441 > R> #2 0xffffffff8060b031 in panic (fmt=3DVariable "fmt" is not = available. > R> ) at /usr/src/sys/kern/kern_shutdown.c:614 > R> #3 0xffffffff80900b80 in trap_fatal (frame=3D0xc, eva=3DVariable = "eva" is not available. > R> ) at /usr/src/sys/amd64/amd64/trap.c:825 > R> #4 0xffffffff80900ed1 in trap_pfault (frame=3D0xffffff800016a620, = usermode=3D0) at /usr/src/sys/amd64/amd64/trap.c:741 > R> #5 0xffffffff8090138f in trap (frame=3D0xffffff800016a620) at = /usr/src/sys/amd64/amd64/trap.c:478 > R> #6 0xffffffff808e88e4 in calltrap () at = /usr/src/sys/amd64/amd64/exception.S:228 > R> #7 0xffffffff80667ef7 in m_copym (m=3D0x0, off0=3D1500, len=3D1480, = wait=3D1) at /usr/src/sys/kern/uipc_mbuf.c:542 > R> #8 0xffffffff8071c8c2 in ip_fragment (ip=3D0xffffff0001a3700e, = m_frag=3D0xffffff800016a838, mtu=3DVariable "mtu" is not available. > R> ) at /usr/src/sys/netinet/ip_output.c:819 > R> #9 0xffffffff8071d93a in ip_output (m=3D0xffffff00019fd900, = opt=3DVariable "opt" is not available. > R> ) at /usr/src/sys/netinet/ip_output.c:650 > R> #10 0xffffffff8071a13a in ip_forward (m=3D0xffffff00019fd900, = srcrt=3DVariable "srcrt" is not available. > R> ) at /usr/src/sys/netinet/ip_input.c:1521 > R> #11 0xffffffff8071b77c in ip_input (m=3D0xffffff00019fd900) at = /usr/src/sys/netinet/ip_input.c:729 > R> #12 0xffffffff806c652e in netisr_dispatch_src (proto=3D1, = source=3DVariable "source" is not available. > R> ) at /usr/src/sys/net/netisr.c:859 > R> #13 0xffffffff806bc5cd in ether_demux (ifp=3D0xffffff000168e800, = m=3D0xffffff00019fd900) at /usr/src/sys/net/if_ethersubr.c:896 > R> #14 0xffffffff806bc9d7 in ether_input (ifp=3D0xffffff000168e800, = m=3D0xffffff00019fd900) at /usr/src/sys/net/if_ethersubr.c:755 > R> #15 0xffffffff803ee03e in ixv_rxeof (que=3D0xffffff0001643880, = count=3D117) at /usr/src/sys/dev/ixgbe/ixv.c:3256 > R> #16 0xffffffff803ef50b in ixv_handle_que (context=3DVariable = "context" is not available. >=20 > I have looked at several panics like this, and it appears that = ip_fragment() > is entered with incorrect byte order here. I failed to understand how = this happens, > and eventually had made the network stack in head to run consistently = in network > byte order, never modifying a forwarded packet. >=20 > If you can run recent 10-CURRENT under same tests, I'd like to know = the results. >=20 > --=20 > Totus tuus, Glebius. --Apple-Mail=_9A3C1F97-28A1-4218-B71B-E277637F857D--