From owner-freebsd-net@FreeBSD.ORG Thu Oct 25 16:40:35 2012 Return-Path: Delivered-To: freebsd-net@FreeBSD.org Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id 18AE659F for ; Thu, 25 Oct 2012 16:40:35 +0000 (UTC) (envelope-from glebius@FreeBSD.org) Received: from cell.glebius.int.ru (glebius.int.ru [81.19.64.117]) by mx1.freebsd.org (Postfix) with ESMTP id 6A94B8FC17 for ; Thu, 25 Oct 2012 16:40:34 +0000 (UTC) Received: from cell.glebius.int.ru (localhost [127.0.0.1]) by cell.glebius.int.ru (8.14.5/8.14.5) with ESMTP id q9PGeVTk005524; Thu, 25 Oct 2012 20:40:31 +0400 (MSK) (envelope-from glebius@FreeBSD.org) Received: (from glebius@localhost) by cell.glebius.int.ru (8.14.5/8.14.5/Submit) id q9PGeV2h005522; Thu, 25 Oct 2012 20:40:31 +0400 (MSK) (envelope-from glebius@FreeBSD.org) X-Authentication-Warning: cell.glebius.int.ru: glebius set sender to glebius@FreeBSD.org using -f Date: Thu, 25 Oct 2012 20:40:31 +0400 From: Gleb Smirnoff To: R?mi Pauchet Subject: Re: panic ixgbevf / SMP under high network load Message-ID: <20121025164031.GA70741@FreeBSD.org> References: MIME-Version: 1.0 Content-Type: text/plain; charset=koi8-r Content-Disposition: inline In-Reply-To: User-Agent: Mutt/1.5.21 (2010-09-15) Cc: freebsd-net@FreeBSD.org X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 25 Oct 2012 16:40:35 -0000 On Thu, Oct 25, 2012 at 10:06:40AM +0200, R?mi Pauchet wrote: R> I'm testing network performance of FreeBSD using vmware esxi 5.1 with SR-IOV R> R> I'm using FreeBSD 8.3 kernel GENERIC, 4 cpus, ixgbevf driver with an Intel 82599EB dual 10 Gbps network interface R> R> After a few seconds of udp ipv4 load (5Gbps x2, frame size 700), I have the following panic : R> R> (kgdb) bt R> #0 doadump () at pcpu.h:224 R> #1 0xffffffff8060ab90 in boot (howto=260) at /usr/src/sys/kern/kern_shutdown.c:441 R> #2 0xffffffff8060b031 in panic (fmt=Variable "fmt" is not available. R> ) at /usr/src/sys/kern/kern_shutdown.c:614 R> #3 0xffffffff80900b80 in trap_fatal (frame=0xc, eva=Variable "eva" is not available. R> ) at /usr/src/sys/amd64/amd64/trap.c:825 R> #4 0xffffffff80900ed1 in trap_pfault (frame=0xffffff800016a620, usermode=0) at /usr/src/sys/amd64/amd64/trap.c:741 R> #5 0xffffffff8090138f in trap (frame=0xffffff800016a620) at /usr/src/sys/amd64/amd64/trap.c:478 R> #6 0xffffffff808e88e4 in calltrap () at /usr/src/sys/amd64/amd64/exception.S:228 R> #7 0xffffffff80667ef7 in m_copym (m=0x0, off0=1500, len=1480, wait=1) at /usr/src/sys/kern/uipc_mbuf.c:542 R> #8 0xffffffff8071c8c2 in ip_fragment (ip=0xffffff0001a3700e, m_frag=0xffffff800016a838, mtu=Variable "mtu" is not available. R> ) at /usr/src/sys/netinet/ip_output.c:819 R> #9 0xffffffff8071d93a in ip_output (m=0xffffff00019fd900, opt=Variable "opt" is not available. R> ) at /usr/src/sys/netinet/ip_output.c:650 R> #10 0xffffffff8071a13a in ip_forward (m=0xffffff00019fd900, srcrt=Variable "srcrt" is not available. R> ) at /usr/src/sys/netinet/ip_input.c:1521 R> #11 0xffffffff8071b77c in ip_input (m=0xffffff00019fd900) at /usr/src/sys/netinet/ip_input.c:729 R> #12 0xffffffff806c652e in netisr_dispatch_src (proto=1, source=Variable "source" is not available. R> ) at /usr/src/sys/net/netisr.c:859 R> #13 0xffffffff806bc5cd in ether_demux (ifp=0xffffff000168e800, m=0xffffff00019fd900) at /usr/src/sys/net/if_ethersubr.c:896 R> #14 0xffffffff806bc9d7 in ether_input (ifp=0xffffff000168e800, m=0xffffff00019fd900) at /usr/src/sys/net/if_ethersubr.c:755 R> #15 0xffffffff803ee03e in ixv_rxeof (que=0xffffff0001643880, count=117) at /usr/src/sys/dev/ixgbe/ixv.c:3256 R> #16 0xffffffff803ef50b in ixv_handle_que (context=Variable "context" is not available. I have looked at several panics like this, and it appears that ip_fragment() is entered with incorrect byte order here. I failed to understand how this happens, and eventually had made the network stack in head to run consistently in network byte order, never modifying a forwarded packet. If you can run recent 10-CURRENT under same tests, I'd like to know the results. -- Totus tuus, Glebius.