From owner-freebsd-net@FreeBSD.ORG Sun Jun 1 15:21:57 2014 Return-Path: Delivered-To: freebsd-net@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 50D45E3A; Sun, 1 Jun 2014 15:21:57 +0000 (UTC) Received: from mail-qg0-x236.google.com (mail-qg0-x236.google.com [IPv6:2607:f8b0:400d:c04::236]) (using TLSv1 with cipher ECDHE-RSA-RC4-SHA (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 0422E2531; Sun, 1 Jun 2014 15:21:56 +0000 (UTC) Received: by mail-qg0-f54.google.com with SMTP id q108so9045256qgd.41 for ; Sun, 01 Jun 2014 08:21:55 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:sender:date:message-id:subject:from:to:content-type; bh=mm0FbfX7Iq8bd9dkoPFO8NwZ+MHsXpC5vrNfjC2R/QA=; b=ePoxpmcG8Y5YokOrUBcgUPmDOFrYwd5eGtImVUxBW82eDVcIFqTOtqK0vzOyHtFpJi 1qBpY2iV1x1/Zi0HcKUa/k2k2Yln7HLVYMqVqkc32l4RFtK1eUC6a+A3duG0aom7pXNS pTB7meL2dstePdL8uLH6mMFC8K52VUSoWV99Mh+DCftETfkgjYBmYkKqMYbJMrOsyNrY 6Y6wBJ4YqhaKTDzoOIgCYdkchyR8gKw1O0jx9M2sXkChsv/3shHdYla2rwQLf9PUGooC b8kHamUGwgfw2fz+8DpIFMUqJBCUMSIcgUprfQ2lVX6YsnlcEhMv+wRa01vFcqDqEEHK 2NUg== MIME-Version: 1.0 X-Received: by 10.140.35.212 with SMTP id n78mr38284529qgn.87.1401636115836; Sun, 01 Jun 2014 08:21:55 -0700 (PDT) Sender: adrian.chadd@gmail.com Received: by 10.224.191.201 with HTTP; Sun, 1 Jun 2014 08:21:55 -0700 (PDT) Date: Sun, 1 Jun 2014 08:21:55 -0700 X-Google-Sender-Auth: 339m2wbqWM-d-pP9UDSMZlHRUuI Message-ID: Subject: [rfc] TCP timewait and credential handling - why would we get a TW with no credential? From: Adrian Chadd To: FreeBSD Net , "freebsd-arch@freebsd.org" Content-Type: text/plain; charset=UTF-8 X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 01 Jun 2014 15:21:57 -0000 Hi, I've been seeing this panic under high very short-term connection TCP throughput tests (30,000 connections a second) with SO_REUSEPORT: Current language: auto; currently minimal (kgdb) frame 11 #11 0xffffffff80a6bdf1 in tcp_twclose (tw=0xfffff801348b5780, reuse=0) at /usr/home/adrian/git/github/erikarn/freebsd/sys/netinet/tcp_timewait.c:644 644 crfree(tw->tw_cred); (kgdb) frame 11 #11 0xffffffff80a6bdf1 in tcp_twclose (tw=0xfffff801348b5780, reuse=0) at /usr/home/adrian/git/github/erikarn/freebsd/sys/netinet/tcp_timewait.c:644 644 crfree(tw->tw_cred); (kgdb) print tw $1 = (struct tcptw *) 0xfffff801348b5780 (kgdb) print tw->tw_cred $2 = (struct ucred *) 0x0 (kgdb) and: #10 0xffffffff808d017e in crfree (cr=0x0) at /usr/home/adrian/git/github/erikarn/freebsd/sys/kern/kern_prot.c:1837 #11 0xffffffff80a6bdf1 in tcp_twclose (tw=0xfffff801348b5780, reuse=0) at /usr/home/adrian/git/github/erikarn/freebsd/sys/netinet/tcp_timewait.c:644 #12 0xffffffff80a6bc53 in tcp_twcheck (inp=, to=, th=, m=, tlen=) at /usr/home/adrian/git/github/erikarn/freebsd/sys/netinet/tcp_timewait.c:425 #13 0xffffffff80a5cf3f in tcp_input (m=, off0=) at /usr/home/adrian/git/github/erikarn/freebsd/sys/netinet/tcp_input.c:949 #14 0xffffffff809fa4e7 in ip_input (m=0xfffff8004831e000) at /usr/home/adrian/git/github/erikarn/freebsd/sys/netinet/ip_input.c:729 #15 0xffffffff80998441 in netisr_dispatch_src (proto=, source=, m=0xffffffff) at /usr/home/adrian/git/github/erikarn/freebsd/sys/net/netisr.c:972 #16 0xffffffff80990173 in ether_demux (ifp=, m=0xfffff8004831e000) at /usr/home/adrian/git/github/erikarn/freebsd/sys/net/if_ethersubr.c:775 #17 0xffffffff80990dce in ether_nh_input (m=) at /usr/home/adrian/git/github/erikarn/freebsd/sys/net/if_ethersubr.c:582 #18 0xffffffff80998441 in netisr_dispatch_src (proto=, source=, m=0xffffffff) at /usr/home/adrian/git/github/erikarn/freebsd/sys/net/netisr.c:972 #19 0xffffffff809903c6 in ether_input (ifp=, m=0x0) at /usr/home/adrian/git/github/erikarn/freebsd/sys/net/if_ethersubr.c:683 #20 0xffffffff804e9d67 in igb_rxeof (count=99) at /usr/home/adrian/git/github/erikarn/freebsd/sys/dev/e1000/if_igb.c:4882 #21 0xffffffff804ea47f in igb_msix_que (arg=0xfffff8000d47e670) at /usr/home/adrian/git/github/erikarn/freebsd/sys/dev/e1000/if_igb.c:1626 #22 0xffffffff808aef03 in intr_event_execute_handlers (p=, ie=0xfffff8000f76cb00) at /usr/home/adrian/git/github/erikarn/freebsd/sys/kern/kern_intr.c:1263 #23 0xffffffff808af866 in ithread_loop (arg=0xfffff8000f908d20) at /usr/home/adrian/git/github/erikarn/freebsd/sys/kern/kern_intr.c:1276 #24 0xffffffff808acc41 in fork_exit (callout=0xffffffff808af7d0 , arg=0xfffff8000f908d20, frame=0xfffffe1046c2cac0) at /usr/home/adrian/git/github/erikarn/freebsd/sys/kern/kern_fork.c:977 #25 0xffffffff80c939ae in fork_trampoline () at /usr/home/adrian/git/github/erikarn/freebsd/sys/amd64/amd64/exception.S:605 #26 0x0000000000000000 in ?? () ... there's only one path to deleting the credentials and that's via tcp_twclose() -> tcp_tw_2msl_stop(). Has anyone seen this kind of problem before? -a