From owner-freebsd-current@FreeBSD.ORG Thu Sep 15 22:51:52 2011 Return-Path: Delivered-To: freebsd-current@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id A517E106564A; Thu, 15 Sep 2011 22:51:52 +0000 (UTC) (envelope-from lacombar@gmail.com) Received: from mail-gy0-f182.google.com (mail-gy0-f182.google.com [209.85.160.182]) by mx1.freebsd.org (Postfix) with ESMTP id 4FEC68FC13; Thu, 15 Sep 2011 22:51:52 +0000 (UTC) Received: by gyf2 with SMTP id 2so3161686gyf.13 for ; Thu, 15 Sep 2011 15:51:51 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :cc:content-type:content-transfer-encoding; bh=KI4/Qh3GiBXs4aRUAEqpn/usJ+UNMdB5FvFcLtlHIbQ=; b=JZ1lvtFITmwNjhDzv039M1tOoS2MiYujMnVMUSmpYR00FNf62S2ZGz7RLJK5V5NMkN AFF4u4mXFDYSqcc4IJaYERTFSrQwoTvhSAVTHno//KMyhUXkfvskCAOuZBIoxJk5N/ss BthISnzHr6LrNjNxZQsXzWb6DNvYiQBbF56T4= MIME-Version: 1.0 Received: by 10.68.71.200 with SMTP id x8mr915581pbu.465.1316127111396; Thu, 15 Sep 2011 15:51:51 -0700 (PDT) Received: by 10.142.12.18 with HTTP; Thu, 15 Sep 2011 15:51:51 -0700 (PDT) In-Reply-To: References: Date: Thu, 15 Sep 2011 18:51:51 -0400 Message-ID: From: Arnaud Lacombe To: freebsd-net@freebsd.org Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: quoted-printable Cc: FreeBSD-Current , Jack Vogel Subject: Re: FreeBSD 7-STABLE mbuf corruption X-BeenThere: freebsd-current@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Discussions about the use of FreeBSD-current List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 15 Sep 2011 22:51:52 -0000 Hi, [added -current@ to the CC list, as the issue is still present in 9.0-BETA2= ] On Wed, Sep 7, 2011 at 7:19 PM, Arnaud Lacombe wrote: > Hi, > > On Mon, Sep 5, 2011 at 2:59 AM, Arnaud Lacombe wrote= : >> Hi folks, >> >> We have been trying to track down a bad mbuf management for about two >> weeks on a customized 7.1 base. I have finally been able to reproduce >> it with a stock FreeBSD 7-STABLE (kernel from r225276, userland from >> 7.4). >> >> With the help of the attached patches, I have just been able to >> trigger the following panic: >> >> panic: Corrupted unused flags, expected 0xffffffff00000000, got 0x0, fla= gs 0x3 >> cpuid =3D 1 >> Uptime: 3d10h5m3s >> Cannot dump. No dump device defined >> > General form of the crash is: > > panic: Corrupted unused flags, expected 0xffffffff00000000, got > 0xbabe0000000000, flags 0xbabe0000babe00 > cpuid =3D 0 > KDB: stack backtrace: > db_trace_self_wrapper(c0874e29,0,c0835757,f4574c48,0,...) at > db_trace_self_wrapper+0x26 > panic(c0835757,0,ffffffff,0,babe00,...) at panic+0x10b > igb_txeof(c6a25008,0,c0837083,5ea,17c,...) at igb_txeof+0x399 > igb_msix_que(c6a2b800,0,c084d367,4b6,c69dd068,...) at igb_msix_que+0x7b > ithread_loop(c6a29090,f4574d38,c084d0db,31c,c6a16828,...) at ithread_loop= +0xc3 > fork_exit(c061d520,c6a29090,f4574d38) at fork_exit+0xa6 > fork_trampoline() at fork_trampoline+0x8 > --- trap 0, eip =3D 0, esp =3D 0xf4574d70, ebp =3D 0 --- > Uptime: 1m42s > I converted igb(4) to use the legacy if_start() logic and triggered the following panic on the latest FreeBSD 9.0-BETA2: panic: Corrupted mbuf tainting, expected 0xffff, got 0xaabb, taint 0xaabb cpuid =3D 6 KDB: enter: panic [ thread pid 0 tid 100045 ] Stopped at kdb_enter+0x3b: movl $0,kdb_why db> bt Tracing pid 0 tid 100045 td 0xc6bd52e0 kdb_enter(c081831c,c081831c,c08026c1,c673ec28,6,...) at kdb_enter+0x3b panic(c08026c1,ffff,aabb,aabb,c6bd1400,...) at panic+0x103 igb_txeof(c6bd1408,0,c080411c,558,c6bd1408,...) at igb_txeof+0x318 igb_handle_que(c6bac400,1,c081e508,130,c673ecb0,...) at igb_handle_que+0xae taskqueue_run_locked(c6bdc400,c6bdc418,0,c080a966,0,...) at taskqueue_run_locked+0xa3 taskqueue_thread_loop(c6bac430,c673ed28,c0812d90,3f9,0,...) at taskqueue_thread_loop+0x4d fork_exit(c063ea10,c6bac430,c673ed28) at fork_exit+0xa4 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip =3D 0, esp =3D 0xc673ed60, ebp =3D 0 --- for those who have not followed the thread on -net, the same mbuf is queued twice in the interface queue, transmitted twice... and freed twice. Of course, after having been released first, it ends up eventually in a socket buffer, and when it gets released the second time, it triggers all kind of funny panic() and crashes. The 0xaabb pattern comes from memory tainting with INVARIANTS at the ends of m_free(). I can provide the patches I am testing with. - Arnaud > It happens particularly easily when the box receives wall of SYN > (about 1000 cnx attempts at once) every 5s or so. > > =A0- Arnaud > >> >> [cut stuff no one cares about...] >