From owner-freebsd-net@FreeBSD.ORG Mon May 10 18:13:10 2010 Return-Path: Delivered-To: freebsd-net@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id 1CBEE1065675; Mon, 10 May 2010 18:13:10 +0000 (UTC) (envelope-from ivoras@gmail.com) Received: from mail-wy0-f182.google.com (mail-wy0-f182.google.com [74.125.82.182]) by mx1.freebsd.org (Postfix) with ESMTP id 7371C8FC0A; Mon, 10 May 2010 18:13:09 +0000 (UTC) Received: by wyb32 with SMTP id 32so340557wyb.13 for ; Mon, 10 May 2010 11:13:08 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=domainkey-signature:received:mime-version:sender:received :in-reply-to:references:from:date:x-google-sender-auth:message-id :subject:to:cc:content-type:content-transfer-encoding; bh=B6dVSIcrdGPjT3xA6JyMnMlp7OCkWpMOcAz9r+SZfrQ=; b=nqhFkSLznRv90rJTGJ9VzgiKBRBnjokqp6JcCKso/l2gQltPNg08cCqDfskIVYlcmy kz7eynlE8Ju80DdBzBC+uxWqVciC1bAYDOdjErY3DIaHoKGTFBD7Zfg5Ggo5UVVKpmWE zab5deCUz+b4y0LD5XJy3cWipiae61QqvgeZE= DomainKey-Signature: a=rsa-sha1; c=nofws; d=gmail.com; s=gamma; h=mime-version:sender:in-reply-to:references:from:date :x-google-sender-auth:message-id:subject:to:cc:content-type :content-transfer-encoding; b=m4PAv6/SW09NE5F0KnrVLvaAm8fpFIr0cZzHFlQsr4EgkHMJkVDJvrZ2AQ9/Vkqd1N k3T428Xa7lAgyhRfQiN4pd7qz3X/ffdZqkz2Nkw+T8LBPz1TTjCmY5oi1eRT+h74YTqq gBs6/ayISI+XmPuxmmQ7bD9MWz5Ws8huZZGKE= Received: by 10.216.85.143 with SMTP id u15mr2679300wee.205.1273515188156; Mon, 10 May 2010 11:13:08 -0700 (PDT) MIME-Version: 1.0 Sender: ivoras@gmail.com Received: by 10.216.86.137 with HTTP; Mon, 10 May 2010 11:12:48 -0700 (PDT) In-Reply-To: <4BE82011.6050009@cs.duke.edu> References: <4BD885C6.10600@FreeBSD.org> <20100429204544.GC1286@arthur.nitro.dk> <1272998683.2406.38.camel@localhost.localdomain> <20100504190328.GC31196@valentine.liquidneon.com> <4BE80F07.8090309@cs.duke.edu> <4BE82011.6050009@cs.duke.edu> From: Ivan Voras Date: Mon, 10 May 2010 20:12:48 +0200 X-Google-Sender-Auth: EoIhzMjkOhcUL1qUaIXZRaX2j5c Message-ID: To: Andrew Gallatin Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: quoted-printable Cc: freebsd-net@freebsd.org, developers@freebsd.org Subject: Re: FreeBSD.org IPv6 issue - AAAA records disabled X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 10 May 2010 18:13:10 -0000 On 10 May 2010 17:02, Andrew Gallatin wrote: > Ivan Voras wrote: > In the case of 143046, he says that disabling IPv6 support in his > jails solved the problem. =C2=A0He also saw the panic:sbdrop, as well > as some others which makes it seem like something is holding a > pointer to a free mbuf and modifying it behind the back of the > new owner. =C2=A0He saw this most often: > > Tracing pid 12 tid 100063 td 0xffffff00092c7390 > mb_free_ext() at mb_free_ext+0x15 > m_freem() at m_freem+0x23 > ether_input() at ether_input+0x56 > mxge_intr() at mxge_intr+0x5b2 > intr_event_execute_handlers() at intr_event_execute_handlers+0x132 > ithread_loop() at ithread_loop+0x7d > fork_exit() at fork_exit+0x121 > fork_trampoline() at fork_trampoline+0xe > > This is coming from the "discard frame w/o packet header" > codepath in ether_input. =C2=A0With the way I stock my rings, > this is impossible unless something else is scribbling on > my mbufs. I have a different trace but the dmesg buffer on crashed machines usually contains the message "em0: discard frame w/o packet header" just before the panic so this still looks like it. Here are my most usual traces: #1 0xffffffff8058d179 in boot (howto=3D260) at /usr/src/sys/kern/kern_shutdown.c:416 #2 0xffffffff8058d5ac in panic ( fmt=3D0xffffffff80971c38 "%s: sockbuf %p and mbuf %p clashing") at /usr/src/sys/kern/kern_shutdown.c:579 #3 0xffffffff805ea6d4 in sbsndptr (sb=3DNA) at /usr/src/sys/kern/uipc_sockbuf.c:954 #4 0xffffffff806ff812 in tcp_output (tp=3D0xffffff0001ae46e0) at /usr/src/sys/netinet/tcp_output.c:814 #5 0xffffffff8070c274 in tcp_usr_send (so=3D0xffffff0001f7ed48, flags=3D0,= m=3DNA) at tcp_offload.h:282 #6 0xffffffff805f05b6 in sosend_generic (so=3D0xffffff0001f7ed48, addr=3D0= x0, uio=3D0x0, top=3D0xffffff0001909c00, control=3D0x0, flags=3DNA) at /usr/src/sys/kern/uipc_socket.c:1256 #7 0xffffffff8077c93f in svc_vc_reply (xprt=3D0xffffff00018ba000, msg=3DNA= ) at /usr/src/sys/rpc/svc_vc.c:732 #8 0xffffffff8077851a in svc_sendreply_common (rqstp=3D0xffffff00425f2800, rply=3D0xffffff8070811990, body=3D0xffffff006c70f000) at /usr/src/sys/rpc/svc.c:538 #9 0xffffffff807793b9 in svc_sendreply_mbuf (NA) at /usr/src/sys/rpc/svc.c:594 #10 0xffffffff80760669 in nfssvc_program (rqst=3D0xffffff00425f2800, xprt= =3DNA) at /usr/src/sys/nfsserver/nfs_srvkrpc.c:371 #11 0xffffffff80778f42 in svc_run_internal (pool=3D0xffffff00016cea00, ismaster=3D0) at /usr/src/sys/rpc/svc.c:893 #12 0xffffffff807792bb in svc_thread_start (arg=3DNA) at /usr/src/sys/rpc/svc.c:1198 #13 0xffffffff80564b78 in fork_exit ( callout=3D0xffffffff807792b0 , arg=3D0xffffff00016cea= 00, frame=3D0xffffff8070811c80) at /usr/src/sys/kern/kern_fork.c:843 #14 0xffffffff808594ae in fork_trampoline () at /usr/src/sys/amd64/amd64/exception.S:561 (Note: NFS, for what it's worth) And: #1 0xffffffff805a2a69 in boot (howto=3D260) at /usr/src/sys/kern/kern_shutdown.c:416 #2 0xffffffff805a2e9c in panic (fmt=3D0xffffffff8098f15f "sbdrop") at /usr/src/sys/kern/kern_shutdown.c:579 #3 0xffffffff80603a53 in sbdrop_internal (sb=3DNA) at /usr/src/sys/kern/uipc_sockbuf.c:858 #4 0xffffffff80710ae7 in tcp_do_segment (m=3D0xffffff0001941500, th=3D0xffffff000194157c, so=3D0xffffff0032efaaa0, tp=3D0xffffff00ddc7f0= 00, drop_hdrlen=3D52, tlen=3D14, iptos=3D0 '\0', ti_locked=3D2) at /usr/src/sys/netinet/tcp_input.c:2355 #5 0xffffffff807128d2 in tcp_input (m=3D0xffffff0001941500, off0=3DNA) at /usr/src/sys/netinet/tcp_input.c:1020 #6 0xffffffff806ae23a in ip_input (m=3D0xffffff0001941500) at /usr/src/sys/netinet/ip_input.c:804 #7 0xffffffff8065b30d in swi_net (arg=3DNA) at /usr/src/sys/net/netisr.c:7= 16 #8 0xffffffff8057bf8d in intr_event_execute_handlers (p=3DNA) at /usr/src/sys/kern/kern_intr.c:1220 #9 0xffffffff8057d63e in ithread_loop (arg=3D0xffffff00014b66c0) at /usr/src/sys/kern/kern_intr.c:1233 #10 0xffffffff80579f58 in fork_exit ( callout=3D0xffffffff8057d5b0 , arg=3D0xffffff00014b66c0, frame=3D0xffffff8000044c80) at /usr/src/sys/kern/kern_fork.c:843 #11 0xffffffff8086f3ae in fork_trampoline () at /usr/src/sys/amd64/amd64/exception.S:561 #12 0x0000000000000000 in ?? () (Looks like a regular input path) > I think something may be holding onto an mbuf after free, > then re-freeing it. =C2=A0But only after somebody else allocated > it. =C2=A0 I was hoping that the mbuf double free referenced > above was the smoking gun, but it turns out that there isn't > even a bge interface in my pr (just bce and mxge). Since I have an em it looks interface-agnostic. Also, moving this to freebsd-net@.