From owner-freebsd-net@FreeBSD.ORG Tue Sep 9 20:41:26 2014 Return-Path: Delivered-To: freebsd-net@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id B8F86C3E for ; Tue, 9 Sep 2014 20:41:26 +0000 (UTC) Received: from mail-qc0-x235.google.com (mail-qc0-x235.google.com [IPv6:2607:f8b0:400d:c01::235]) (using TLSv1 with cipher ECDHE-RSA-RC4-SHA (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 7A17A636 for ; Tue, 9 Sep 2014 20:41:26 +0000 (UTC) Received: by mail-qc0-f181.google.com with SMTP id i17so17896751qcy.40 for ; Tue, 09 Sep 2014 13:41:25 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:sender:date:message-id:subject:from:to:content-type; bh=BLsiPm47MK6joEJAYWS0jGurGoBKIw+YygchNxRQCcc=; b=kKhYqN0SVvauJ2P2apqvcy88RLHPS8V1ZGny7c2atnm4grQwPBYGj55Oik+HkI+EJR btEk2wx65NB/WhbunFyNbWeFV5Z1iUqvFBbHjRsFHqSjDMJSTXKDoNWj4lh5QUh5Egbu RnNCgyCdRR1xftckL7md/pVTJfxTVnoScQelVnq4Py1S2V0rTSKA6+l6e8QENs8xce6T ca/DUJ6dn/FpjYGxkliNHtf7jdRr68X0BhGMn7uqJ5AXXYesX9EzN2TC0tQL7Ix2F+7D 0OrFNtNCFXCGgJA9Ou27zbmyLdceq8v5GmTR1sIWvvxkhrAZAHo94tTUwQvK9JKiKTW7 GEzw== MIME-Version: 1.0 X-Received: by 10.224.151.69 with SMTP id b5mr55316755qaw.37.1410295285081; Tue, 09 Sep 2014 13:41:25 -0700 (PDT) Sender: adrian.chadd@gmail.com Received: by 10.224.39.139 with HTTP; Tue, 9 Sep 2014 13:41:25 -0700 (PDT) Date: Tue, 9 Sep 2014 13:41:25 -0700 X-Google-Sender-Auth: hu2P0zrKUarud81T5hQYE19l-oY Message-ID: Subject: inpcb lookups in udp_send -> inpcb laddr and flowtable From: Adrian Chadd To: FreeBSD Net Content-Type: text/plain; charset=UTF-8 X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.18-1 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 09 Sep 2014 20:41:26 -0000 Hi! I found this whilst doing UDP RSS testing. root@adrian-hackbox:/home/adrian/work/freebsd/head/src/sys/net # sysctl debug.lock.prof.stats | head -2 ; sysctl debug.lock.prof.stats | sort -nk4 | tail -5 debug.lock.prof.stats: max wait_max total wait_total count avg wait_avg cnt_hold cnt_lock name 40 11 114900 55276 359106 0 0 0 20962 /usr/home/adrian/work/freebsd/head/src/sys/dev/ixgbe/ixgbe.c:1550 (sleep mutex:ix0:tx(6)) 21 11 144253 71959 454169 0 0 0 27506 /usr/home/adrian/work/freebsd/head/src/sys/dev/ixgbe/ixgbe.c:1550 (sleep mutex:ix0:tx(1)) 9 10 3132066 85999 42942422 0 0 0 35943 /usr/home/adrian/work/freebsd/head/src/sys/netinet/udp_usrreq.c:369 (sleep mutex:so_rcv) 676 1713 805307 169654 7076283 0 0 0 69741 /usr/home/adrian/work/freebsd/head/src/sys/netinet/in_pcb.c:963 (sleep mutex:rtentry) 674 4426 2111324 6415012 7076284 0 0 0 489745 /usr/home/adrian/work/freebsd/head/src/sys/net/route.c:420 (sleep mutex:rtentry) then... root@adrian-hackbox:/home/adrian/git/github/erikarn/freebsd-rss/rss-udp-srv # dtrace -n 'fbt::rtalloc1_fib:entry { @[stack()] = count(); }' dtrace: description 'fbt::rtalloc1_fib:entry ' matched 1 probe ^[[A^C kernel`rtalloc_ign_fib+0x98 kernel`in_pcbladdr+0x7d kernel`in_pcbconnect_setup+0x1af kernel`udp_send+0x4af kernel`sosend_dgram+0x30b kernel`kern_sendit+0x20b kernel`sendit+0x129 kernel`sys_sendto+0x4d kernel`amd64_syscall+0x25a kernel`0xffffffff80cd178b 1848628 It turns out that if a local address isn't bound (ie, it's INADDR_ANY) then in_pcbladdr() will do a route table lookup to find the interface to transmit from. The v4/v6 laddr paths don't know about flowtable; they just do a routing table lookup. I'm currently experimenting with a kernel fix to use flowtable here for IPv4 UDP sockets but I'm not sure about the IPv6 path (the source interface/address selection is much crazier with all the various scope bits going on.) I'd like some help with the IPv6 source selection please! Thanks! -a