From owner-freebsd-net@FreeBSD.ORG Sun Jul 20 02:41:28 2014 Return-Path: Delivered-To: freebsd-net@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id EACEE2D6; Sun, 20 Jul 2014 02:41:28 +0000 (UTC) Received: from mail-qa0-x229.google.com (mail-qa0-x229.google.com [IPv6:2607:f8b0:400d:c00::229]) (using TLSv1 with cipher ECDHE-RSA-RC4-SHA (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 9CC1F2663; Sun, 20 Jul 2014 02:41:28 +0000 (UTC) Received: by mail-qa0-f41.google.com with SMTP id j7so4233306qaq.28 for ; Sat, 19 Jul 2014 19:41:26 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :cc:content-type; bh=aAyAvf5eo9vN5xTicqd5qDgV/RpCBDx3IBUIZPIJjJQ=; b=EH3RD/WQT3oGxoGMrnOh4bfGgNYtfc6bzeL6lFNZVz5xbMRlDvvIrGPbiRnTZ6bgZ1 f5Ou80opzM9NZehGWdoPCiWx7ghS9MXxQCqPmBfJly2PKaB7wBZPjUtadf3Kumall5oc BJ73Kcpu6EefJ22M7GclG9ZxaZ/uIJ35fbMFb8kN32rF6RtI5BUBFzKdpEpuOP5Z8EKd CizurdfJ44AG8/cCmTVfnMuqM1ZnTdcd4sUI2rkDRkWas/d2bwfS/xXCY6WaWDyAto2R I3f/3ijJeX5tn34ipsC4jR24EibBW199vXGuCyxu41ZFVhDmuCxErsvvmh9NcoGhexV8 FoPA== MIME-Version: 1.0 X-Received: by 10.224.43.77 with SMTP id v13mr24923077qae.34.1405824086789; Sat, 19 Jul 2014 19:41:26 -0700 (PDT) Received: by 10.96.73.39 with HTTP; Sat, 19 Jul 2014 19:41:26 -0700 (PDT) In-Reply-To: <53CA39BD.6050900@FreeBSD.org> References: <201407151132.53587.vegeta@tuxpowered.net> <53CA39BD.6050900@FreeBSD.org> Date: Sat, 19 Jul 2014 19:41:26 -0700 Message-ID: Subject: Re: Why is r250764 not in 9.3? From: hiren panchasara To: "Alexander V. Chernikov" Content-Type: text/plain; charset=UTF-8 Cc: Kajetan Staszkiewicz , "freebsd-net@freebsd.org" X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 20 Jul 2014 02:41:29 -0000 On Sat, Jul 19, 2014 at 2:26 AM, Alexander V. Chernikov wrote: > On 15.07.2014 21:03, hiren panchasara wrote: >> >> + Alexander >> >> On Tue, Jul 15, 2014 at 2:32 AM, Kajetan Staszkiewicz >> wrote: >>> >>> The time has come to upgrade my routers to FreeBSD 9.3. >>> >>> While going through list of patches I had on 9.1, I've noticed that >>> r248070 got >>> into 9.3 but r250764 did not. Why is that? >> >> Probably just missed it. > > Yes, I've missed it. > Unfortunately, I'm unable to merge it until 26July, feel free to do so if > you wish. Committed to stable/9 as r268906. cheers, Hiren From owner-freebsd-net@FreeBSD.ORG Sun Jul 20 09:04:14 2014 Return-Path: Delivered-To: freebsd-net@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 3C86C7F0; Sun, 20 Jul 2014 09:04:14 +0000 (UTC) Received: from mx.ipv6.elandsys.com (mx.ipv6.elandsys.com [IPv6:2001:470:f329:1::1]) by mx1.freebsd.org (Postfix) with ESMTP id E4496216F; Sun, 20 Jul 2014 09:04:13 +0000 (UTC) Received: from mx.elandsys.com (IDENT:logan@localhost [127.0.0.1]) by mx.elandsys.com (8.14.5/8.14.5) with ESMTP id s6K94B29029089 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-SHA bits=256 verify=NO); Sun, 20 Jul 2014 02:04:11 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=simple/simple; d=opendkim.org; s=mail2010; t=1405847052; x=1405933452; bh=brfb1nXTiwlY0ctuR3fSu6AAbS9nX4m6leTIiy/7Jsc=; h=Date:From:To:Cc:Subject; b=cXvFWSCVNYO+RgLccMnZ/IoUQtvvG8QP/ieVZOw9fyqbVC17n+mYJOGLe2N9eF9pP kHa0+DfCx2lSqkRZ82x7V60znUb3iVnUbfuq2tkL8jkgWxquI9/PqcD3c5fZ3Ah8U3 DNmqvK9T3XTfKmwwAOkQrBaPfLtE4a6LfELPXYlc= DKIM-Signature: v=1; a=rsa-sha256; c=simple/simple; d=elandsys.com; s=mail; t=1405847052; x=1405933452; i=@elandsys.com; bh=brfb1nXTiwlY0ctuR3fSu6AAbS9nX4m6leTIiy/7Jsc=; h=Date:From:To:Cc:Subject; b=mJ+CwQbuw0F7QdB7T5GVF4piIs8evmbk8+17KrMf3qysQJlWYscJabeSoCxqyc2SI LJfJXL25QTwGuTrudoyzNPLgF7WoTkmbM5yva+FXs/wZThOt2OGy113oQ1IyNLja2G xhossvg0Ko63qvY4jYy1f0Wo2L9LMpnCVmSV9gRo= Received: (from logan@localhost) by mx.elandsys.com (8.14.5/8.14.5/Submit) id s6K94APn007850; Sun, 20 Jul 2014 02:04:10 -0700 (PDT) X-Authentication-Warning: mx.elandsys.com: logan set sender to logan@elandsys.com using -f Date: Sun, 20 Jul 2014 02:04:10 -0700 From: Loganaden Velvindron To: freebsd-net@freebsd.org Subject: IPv6 nodeinfo default behaviour Message-ID: <20140720090410.GA7990@mx.elandsys.com> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline User-Agent: Mutt/1.5.21 (2010-09-15) Cc: gnn@freebsd.org, bz@freebsd.org X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 20 Jul 2014 09:04:14 -0000 Hi guys, OpenBSD recently removed support for RFC 4620 from their kernel completely. The default value is 3 in FreeBSD. According to the RFC: Security Considerations This protocol shares the security issues of ICMPv6 that are documented in the "Security Considerations" section of [5]. This protocol has the potential of revealing information useful to a would-be attacker. An implementation of this protocol MUST have a default configuration that refuses to answer queries from global- scope [3] addresses. I suggest that we switch to 0 by default to be more RFC compliant. Before I send the patch, I would like to get feedback. Kind regards, //Logan C-x-C-c From owner-freebsd-net@FreeBSD.ORG Sun Jul 20 14:31:03 2014 Return-Path: Delivered-To: freebsd-net@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 592A4693; Sun, 20 Jul 2014 14:31:03 +0000 (UTC) Received: from bigwig.baldwin.cx (bigwig.baldwin.cx [IPv6:2001:470:1f11:75::1]) (using TLSv1 with cipher DHE-RSA-CAMELLIA256-SHA (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 1C17C2895; Sun, 20 Jul 2014 14:31:03 +0000 (UTC) Received: from pippin.baldwin.cx (75-48-77-17.lightspeed.cncrca.sbcglobal.net [75.48.77.17]) by bigwig.baldwin.cx (Postfix) with ESMTPSA id 021D5B968; Sun, 20 Jul 2014 10:31:02 -0400 (EDT) From: John Baldwin To: Julian Elischer Subject: Re: NFS client READ performance on -current Date: Sat, 19 Jul 2014 13:28:19 -0400 Message-ID: <1780417.KfjTWjeQCU@pippin.baldwin.cx> User-Agent: KMail/4.12.5 (FreeBSD/11.0-CURRENT; KDE/4.12.5; amd64; ; ) In-Reply-To: <53C7B774.60304@freebsd.org> References: <2136988575.13956627.1405199640153.JavaMail.root@uoguelph.ca> <201407151034.54681.jhb@freebsd.org> <53C7B774.60304@freebsd.org> MIME-Version: 1.0 Content-Transfer-Encoding: 7Bit Content-Type: text/plain; charset="us-ascii" X-Greylist: Sender succeeded SMTP AUTH, not delayed by milter-greylist-4.2.7 (bigwig.baldwin.cx); Sun, 20 Jul 2014 10:31:02 -0400 (EDT) Cc: pyunyh@gmail.com, "Russell L. Carter" , Rick Macklem , freebsd-net@freebsd.org X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 20 Jul 2014 14:31:03 -0000 On Thursday 17 July 2014 19:45:56 Julian Elischer wrote: > On 7/15/14, 10:34 PM, John Baldwin wrote: > > On Saturday, July 12, 2014 5:14:00 pm Rick Macklem wrote: > >> Yonghyeon Pyun wrote: > >>> On Fri, Jul 11, 2014 at 09:54:23AM -0400, John Baldwin wrote: > >>>> On Thursday, July 10, 2014 6:31:43 pm Rick Macklem wrote: > >>>>> John Baldwin wrote: > >>>>>> On Thursday, July 03, 2014 8:51:01 pm Rick Macklem wrote: > >>>>>>> Russell L. Carter wrote: > >>>>>>>> On 07/02/14 19:09, Rick Macklem wrote: > >>>>>>>>> Could you please post the dmesg stuff for the network > >>>>>>>>> interface, > >>>>>>>>> so I can tell what driver is being used? I'll take a look > >>>>>>>>> at > >>>>>>>>> it, > >>>>>>>>> in case it needs to be changed to use m_defrag(). > >>>>>>>> > >>>>>>>> em0: port > >>>>>>>> 0xd020-0xd03f > >>>>>>>> mem > >>>>>>>> 0xfe4a0000-0xfe4bffff,0xfe480000-0xfe49ffff irq 44 at > >>>>>>>> device 0.0 > >>>>>>>> on > >>>>>>>> pci2 > >>>>>>>> em0: Using an MSI interrupt > >>>>>>>> em0: Ethernet address: 00:15:17:bc:29:ba > >>>>>>>> 001.000007 [2323] netmap_attach success for em0 > >>>>>>>> tx > >>>>>>>> 1/1024 > >>>>>>>> rx > >>>>>>>> 1/1024 queues/slots > >>>>>>>> > >>>>>>>> This is one of those dual nic cards, so there is em1 as > >>>>>>>> well... > >>>>>>> > >>>>>>> Well, I took a quick look at the driver and it does use > >>>>>>> m_defrag(), > >>>>>>> but > >>>>>>> I think that the "retry:" label it does a goto after doing so > >>>>>>> might > >>>>>>> be in > >>>>>>> the wrong place. > >>>>>>> > >>>>>>> The attached untested patch might fix this. > >>>>>>> > >>>>>>> Is it convenient to build a kernel with this patch applied > >>>>>>> and then > >>>>>>> try > >>>>>>> it with TSO enabled? > >>>>>>> > >>>>>>> rick > >>>>>>> ps: It does have the transmit segment limit set to 32. I have > >>>>>>> no > >>>>>>> idea if > >>>>>>> > >>>>>>> this is a hardware limitation. > >>>>>> > >>>>>> I think the retry is not in the wrong place, but the overhead > >>>>>> of all > >>>>>> those > >>>>>> pullups is apparently quite severe. > >>>>> > >>>>> The m_defrag() call after the first failure will just barely > >>>>> squeeze > >>>>> the just under 64K TSO segment into 32 mbuf clusters. Then I > >>>>> think any > >>>>> m_pullup() done during the retry will allocate an mbuf > >>>>> (at a glance it seems to always do this when the old mbuf is a > >>>>> cluster) > >>>>> and prepend that to the list. > >>>>> --> Now the list is > 32 mbufs again and the > >>>>> bus_dmammap_load_mbuf_sg() > >>>>> > >>>>> will fail again on the retry, this time fatally, I think? > >>>>> > >>>>> I can't see any reason to re-do all the stuff using m_pullup() > >>>>> and Russell > >>>>> reported that moving the "retry:" fixed his problem, from what I > >>>>> understood. > >>>> > >>>> Ah, I had assumed (incorrectly) that the m_pullup()s would all be > >>>> nops in this > >>>> case. It seems the NIC would really like to have all those things > >>>> in a single > >>>> segment, but it is not required, so I agree that your patch is > >>>> fine. > >>> > >>> I recall em(4) controllers have various limitation in TSO. Driver > >>> has to update IP header to make TSO work so driver has to get a > >>> writable mbufs. bpf(4) consumers will see IP packet length is 0 > >>> after this change. I think tcpdump has a compile time option to > >>> guess correct IP packet length. The firmware of controller also > >>> should be able to access complete IP/TCP header in a single buffer. > >>> I don't remember more details in TSO limitation but I guess you may > >>> be able to get more details TSO limitation from publicly available > >>> Intel data sheet. > >> > >> I think that the patch should handle this ok. All of the m_pullup() > >> stuff gets done the first time. Then, if the result is more than 32 > >> mbufs in the list, m_defrag() is called to copy the chain. This should > >> result in all the header stuff in the first mbuf cluster and the map > >> call is done again with this list of clusters. (Without the patch, > >> m_pullup() would allocate another prepended mbuf and make the chain > >> more than 32mbufs again.) > > > > Hmm, I am surprised by the m_pullup() behavior that it doesn't just > > notice that the first mbuf with a cluster has the desired data already > > and returns without doing anything. That is, I'm surprised the first > > > > statement in m_pullup() isn't just: > > if (n->m_len >= len) > > > > return (n); > > I seem to remember that the standard behaviour is for the caller to do > exactly that. Huh, the manpage doesn't really state that, and it does check in one case. However, I think that means that the code in em(4) is busted and should be checking m_len before all the calls to m_pullup(). I think this will fix the issue the same as Rick's change but it might also avoid unnecessary pullups in some cases when defrag isn't needed in the first place. Index: if_em.c =================================================================== --- if_em.c (revision 268570) +++ if_em.c (working copy) @@ -1857,32 +1857,41 @@ retry: * for IPv6 yet. */ ip_off = sizeof(struct ether_header); - m_head = m_pullup(m_head, ip_off); - if (m_head == NULL) { - *m_headp = NULL; - return (ENOBUFS); + if (m_head->m_len < ip_off) { + m_head = m_pullup(m_head, ip_off); + if (m_head == NULL) { + *m_headp = NULL; + return (ENOBUFS); + } } eh = mtod(m_head, struct ether_header *); if (eh->ether_type == htons(ETHERTYPE_VLAN)) { ip_off = sizeof(struct ether_vlan_header); - m_head = m_pullup(m_head, ip_off); + if (m_head->m_len < ip_off) { + m_head = m_pullup(m_head, ip_off); + if (m_head == NULL) { + *m_headp = NULL; + return (ENOBUFS); + } + } + } + if (m_head->m_len < ip_off + sizeof(struct ip)) { + m_head = m_pullup(m_head, ip_off + sizeof(struct ip)); if (m_head == NULL) { *m_headp = NULL; return (ENOBUFS); } } - m_head = m_pullup(m_head, ip_off + sizeof(struct ip)); - if (m_head == NULL) { - *m_headp = NULL; - return (ENOBUFS); - } ip = (struct ip *)(mtod(m_head, char *) + ip_off); poff = ip_off + (ip->ip_hl << 2); if (do_tso) { - m_head = m_pullup(m_head, poff + sizeof(struct tcphdr)); - if (m_head == NULL) { - *m_headp = NULL; - return (ENOBUFS); + if (m_head->m_len < poff + sizeof(struct tcphdr)) { + m_head = m_pullup(m_head, poff + + sizeof(struct tcphdr)); + if (m_head == NULL) { + *m_headp = NULL; + return (ENOBUFS); + } } tp = (struct tcphdr *)(mtod(m_head, char *) + poff); /* @@ -1889,10 +1898,13 @@ retry: * TSO workaround: * pull 4 more bytes of data into it. */ - m_head = m_pullup(m_head, poff + (tp->th_off << 2) + 4); - if (m_head == NULL) { - *m_headp = NULL; - return (ENOBUFS); + if (m_head->m_len < poff + (tp->th_off << 2) + 4) { + m_head = m_pullup(m_head, poff + + (tp->th_off << 2) + 4); + if (m_head == NULL) { + *m_headp = NULL; + return (ENOBUFS); + } } ip = (struct ip *)(mtod(m_head, char *) + ip_off); ip->ip_len = 0; @@ -1907,24 +1919,33 @@ retry: tp->th_sum = in_pseudo(ip->ip_src.s_addr, ip->ip_dst.s_addr, htons(IPPROTO_TCP)); } else if (m_head->m_pkthdr.csum_flags & CSUM_TCP) { - m_head = m_pullup(m_head, poff + sizeof(struct tcphdr)); - if (m_head == NULL) { - *m_headp = NULL; - return (ENOBUFS); + if (m_head->m_len < poff + sizeof(struct tcphdr)) { + m_head = m_pullup(m_head, poff + + sizeof(struct tcphdr)); + if (m_head == NULL) { + *m_headp = NULL; + return (ENOBUFS); + } } tp = (struct tcphdr *)(mtod(m_head, char *) + poff); - m_head = m_pullup(m_head, poff + (tp->th_off << 2)); - if (m_head == NULL) { - *m_headp = NULL; - return (ENOBUFS); + if (m_head->m_len < poff + (tp->th_off << 2)) { + m_head = m_pullup(m_head, poff + + (tp->th_off << 2)); + if (m_head == NULL) { + *m_headp = NULL; + return (ENOBUFS); + } } ip = (struct ip *)(mtod(m_head, char *) + ip_off); tp = (struct tcphdr *)(mtod(m_head, char *) + poff); } else if (m_head->m_pkthdr.csum_flags & CSUM_UDP) { - m_head = m_pullup(m_head, poff + sizeof(struct udphdr)); - if (m_head == NULL) { - *m_headp = NULL; - return (ENOBUFS); + if (m_head->m_len < poff + sizeof(struct udphdr)) { + m_head = m_pullup(m_head, poff + + sizeof(struct udphdr)); + if (m_head == NULL) { + *m_headp = NULL; + return (ENOBUFS); + } } ip = (struct ip *)(mtod(m_head, char *) + ip_off); } -- John Baldwin From owner-freebsd-net@FreeBSD.ORG Sun Jul 20 15:42:37 2014 Return-Path: Delivered-To: freebsd-net@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 6F90D750 for ; Sun, 20 Jul 2014 15:42:37 +0000 (UTC) Received: from mail-qg0-x235.google.com (mail-qg0-x235.google.com [IPv6:2607:f8b0:400d:c04::235]) (using TLSv1 with cipher ECDHE-RSA-RC4-SHA (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 3366A2F7A for ; Sun, 20 Jul 2014 15:42:37 +0000 (UTC) Received: by mail-qg0-f53.google.com with SMTP id q107so4695648qgd.12 for ; Sun, 20 Jul 2014 08:42:36 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:date:message-id:subject:from:to:content-type; bh=1JgcnOUtKggEdXHYPZIkc7G7gtlK3I+laZiD5/qdjQc=; b=JSmPvuRKUlVoadl9cz0rawAMm9/DskhzZhVhedWkaRLQEn0ug3QkfrHIrhkn61PQBo 2DEN9Xe40XQ7le7AKVTLSPg5pjwa0fzYiRNVYEZJ26+K9n56oGHH3T6+1erTzcj7j5Z0 XIav3NBNYZHQ8zC2T5NwLbOAsqnuL17oEjoCfn0FYRYEzLJ2coRd0zFY0oJJ2P5VuCdn PxjsKE/+tzeDSvXbJO5xD0Rt9vLiayQ6l+RJjOakH4DCag+b1reKTYDtwjvhnweFqFQo 6vlLIQmIX873EUDt4BLn7dXw1yyN9W+PNtgYWDgBd/WFI6L4JBNrh7TNH46nkH0+Cb5b Jrow== MIME-Version: 1.0 X-Received: by 10.224.167.136 with SMTP id q8mr30815424qay.35.1405870956102; Sun, 20 Jul 2014 08:42:36 -0700 (PDT) Received: by 10.224.8.136 with HTTP; Sun, 20 Jul 2014 08:42:36 -0700 (PDT) Date: Sun, 20 Jul 2014 23:42:36 +0800 Message-ID: Subject: A TCP problem on High Delay From: Niu Zhixiong To: freebsd-net@freebsd.org Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: quoted-printable X-Content-Filtered-By: Mailman/MimeDel 2.1.18 X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 20 Jul 2014 15:42:37 -0000 Dear All, I set up an FTP server in a high delay but fairly BW environment. When I try to download files from this FreeBSD FTP server, A FreeBSD client is always slower than Linux Client. (Linux Client is about 1MB/s, But FreeBSD client 300KB/s). I am sure that the two clients are in same networks and no firewalls are between server and clients. Then I try to use SCTP protocol rather than TCP protocol. The FreeBSD client is faster than itself in TCP. I try to increase the rbuff of the client. But, It always less than the linux one. I think the problem may be the smaller window of Freebsd than Linux. But, I changed some parameters. But, nothing changed. uname -a 10.0-RELEASE FreeBSD 10.0-RELEASE #0 r260789: Thu Jan 16 22:34:59 UTC 2014 root@snap.freebsd.org:/usr/obj/usr/src/sys/GENERIC amd64 According to http://www.freebsd.org/doc/en_US.ISO8859-1/books/handbook/configtuning-kern= el-limits.html I try to enable sysctl net.inet.tcp.inflight.enable or sysctl net.inet.tcp.inflight_enable , but there is no such option in FreeBSD 10. Then I follow this https://fasterdata.es.net/host-tuning/freebsd/ And make a large rbuffer. sysctl kern.ipc.maxsockbuf=3D16777216 sysctl net.inet.tcp.sendbuf_max=3D16777216 sysctl net.inet.tcp.recvbuf_max=3D16777216 sysctl net.inet.tcp.sendbuf_auto=3D1 sysctl net.inet.tcp.recvbuf_auto=3D1 sysctl net.inet.tcp.sendbuf_inc=3D16384 sysctl net.inet.tcp.recvbuf_inc=3D524288 sysctl net.inet.tcp.hostcache.expire=3D1 But, Nothing is happened. In the attachments, l_f is the linux packet capture and f_f is the freebsd packet capture. =E2=80=8B packet_capture.7z =E2=80=8B From owner-freebsd-net@FreeBSD.ORG Sun Jul 20 15:56:41 2014 Return-Path: Delivered-To: net@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 60383CBF; Sun, 20 Jul 2014 15:56:41 +0000 (UTC) Received: from hergotha.csail.mit.edu (wollman-1-pt.tunnel.tserv4.nyc4.ipv6.he.net [IPv6:2001:470:1f06:ccb::2]) (using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 1F8BC2093; Sun, 20 Jul 2014 15:56:40 +0000 (UTC) Received: from hergotha.csail.mit.edu (localhost [127.0.0.1]) by hergotha.csail.mit.edu (8.14.7/8.14.7) with ESMTP id s6KFuctF013782; Sun, 20 Jul 2014 11:56:38 -0400 (EDT) (envelope-from wollman@hergotha.csail.mit.edu) Received: (from wollman@localhost) by hergotha.csail.mit.edu (8.14.7/8.14.4/Submit) id s6KFuchL013781; Sun, 20 Jul 2014 11:56:38 -0400 (EDT) (envelope-from wollman) Date: Sun, 20 Jul 2014 11:56:38 -0400 (EDT) From: Garrett Wollman Message-Id: <201407201556.s6KFuchL013781@hergotha.csail.mit.edu> To: jhb@freebsd.org Subject: Re: NFS client READ performance on -current In-Reply-To: <201407151034.54681.jhb@freebsd.org> References: <2136988575.13956627.1405199640153.JavaMail.root@uoguelph.ca> Organization: none X-Greylist: Sender IP whitelisted, not delayed by milter-greylist-4.4.3 (hergotha.csail.mit.edu [127.0.0.1]); Sun, 20 Jul 2014 11:56:38 -0400 (EDT) X-Spam-Status: No, score=-1.0 required=5.0 tests=ALL_TRUSTED autolearn=disabled version=3.4.0 X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on hergotha.csail.mit.edu Cc: net@freebsd.org X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 20 Jul 2014 15:56:41 -0000 In article <201407151034.54681.jhb@freebsd.org>, jhb@freebsd.org writes: >Hmm, I am surprised by the m_pullup() behavior that it doesn't just >notice that the first mbuf with a cluster has the desired data already >and returns without doing anything. The specification of m_pullup() is that it returns a *writable* mbuf (and thus also that the "length" provided is less than MHLEN). Clusters are read-only. -GAWollman From owner-freebsd-net@FreeBSD.ORG Sun Jul 20 16:22:42 2014 Return-Path: Delivered-To: freebsd-net@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 23EA0A2D for ; Sun, 20 Jul 2014 16:22:42 +0000 (UTC) Received: from mail-pd0-x22b.google.com (mail-pd0-x22b.google.com [IPv6:2607:f8b0:400e:c02::22b]) (using TLSv1 with cipher ECDHE-RSA-RC4-SHA (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id ED1552331 for ; Sun, 20 Jul 2014 16:22:41 +0000 (UTC) Received: by mail-pd0-f171.google.com with SMTP id z10so7777103pdj.30 for ; Sun, 20 Jul 2014 09:22:41 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=from:to:references:in-reply-to:subject:date:message-id:mime-version :content-type:content-transfer-encoding:thread-index :content-language; bh=IkApidJUyjiy/CjEQnNr+z4zWz3rXbLqmEMDmaew8Rs=; b=fR9nNzrHRaM/ZOxIHYXjQQVrs/0uvLwjyJCFkxJONPRiYihaU3nSPix1E4N4GvrsrX vm6waZeCEBgIzS7zG8uB0Ms4fcUKEah+EUnSe3/cf7TNUXrGCdXWOyUmiE9Ur6RNhBCE alWt30GUrcF0qGraMst59bP1vDdVhONM3aNluB3g66eI3HfeZ3rVDpzBtltLbgYAajpY 1wXHR3N5OvM9Edds+GbO7Daa2v5kIvci3/Rf+yhs/FohH6jrhVDjCQJ5yIsEEOfP23mk 3pYQlf0Gku7vfNx7cvh3GjPtXabCsPIXTjZXW1rAYUq3ya9NkaMRYXH97sRU2SS0jVeV c9mA== X-Received: by 10.66.245.140 with SMTP id xo12mr11386732pac.38.1405873361522; Sun, 20 Jul 2014 09:22:41 -0700 (PDT) Received: from billwin7 (amx-tls2.starhub.net.sg. [203.116.164.12]) by mx.google.com with ESMTPSA id t12sm8880604pdj.12.2014.07.20.09.22.39 for (version=TLSv1 cipher=ECDHE-RSA-AES128-SHA bits=128/128); Sun, 20 Jul 2014 09:22:41 -0700 (PDT) From: "bycn82" To: "'Niu Zhixiong'" , References: In-Reply-To: Subject: RE: A TCP problem on High Delay Date: Mon, 21 Jul 2014 00:22:34 +0800 Message-ID: <00d201cfa436$d2936ce0$77ba46a0$@gmail.com> MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: quoted-printable X-Mailer: Microsoft Outlook 14.0 Thread-Index: AQKMo9OFtQeaVNjysT5QMLyH7y+hnpovJO9w Content-Language: en-us X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 20 Jul 2014 16:22:42 -0000 Hi, Yes, according to your pcap files,it is because of the window size. more information here http://www.ietf.org/rfc/rfc1323.txt=20 and share the result of "sysctl net.inet.tcp" Regards, bycn82 > -----Original Message----- > From: owner-freebsd-net@freebsd.org [mailto:owner-freebsd- > net@freebsd.org] On Behalf Of Niu Zhixiong > Sent: 20 July, 2014 23:43 > To: freebsd-net@freebsd.org > Subject: A TCP problem on High Delay >=20 > Dear All, > I set up an FTP server in a high delay but fairly BW environment. > When I try to download files from this FreeBSD FTP server, A = FreeBSD > client is always slower than Linux Client. (Linux Client is about = 1MB/s, > But FreeBSD client 300KB/s). I am sure that the two clients are in = same > networks and no firewalls are between server and clients. Then I try = to > use SCTP protocol rather than TCP protocol. The FreeBSD client is = faster > than itself in TCP. >=20 > I try to increase the rbuff of the client. But, It always less than = the > linux one. I think the problem may be the smaller window of Freebsd = than > Linux. But, I changed some parameters. But, nothing changed. >=20 > uname -a > 10.0-RELEASE FreeBSD 10.0-RELEASE #0 r260789: Thu Jan 16 22:34:59 UTC > 2014 > root@snap.freebsd.org:/usr/obj/usr/src/sys/GENERIC amd64 >=20 > According to > = http://www.freebsd.org/doc/en_US.ISO8859-1/books/handbook/configtuning- > kernel-limits.html >=20 > I try to enable sysctl net.inet.tcp.inflight.enable or sysctl > net.inet.tcp.inflight_enable , but there is no such option in FreeBSD = 10. >=20 > Then I follow this https://fasterdata.es.net/host-tuning/freebsd/ > And make a large rbuffer. >=20 >=20 > sysctl kern.ipc.maxsockbuf=3D16777216 > sysctl net.inet.tcp.sendbuf_max=3D16777216 > sysctl net.inet.tcp.recvbuf_max=3D16777216 > sysctl net.inet.tcp.sendbuf_auto=3D1 > sysctl net.inet.tcp.recvbuf_auto=3D1 > sysctl net.inet.tcp.sendbuf_inc=3D16384 > sysctl net.inet.tcp.recvbuf_inc=3D524288 > sysctl net.inet.tcp.hostcache.expire=3D1 >=20 > But, Nothing is happened. >=20 > In the attachments, l_f is the linux packet capture and f_f is the > freebsd packet capture. >=20 >=20 >=20 >=20 > =E2=80=8B > packet_capture.7z > = ive_web> > =E2=80=8B > _______________________________________________ > freebsd-net@freebsd.org mailing list > http://lists.freebsd.org/mailman/listinfo/freebsd-net > To unsubscribe, send any mail to "freebsd-net-unsubscribe@freebsd.org" From owner-freebsd-net@FreeBSD.ORG Sun Jul 20 18:21:09 2014 Return-Path: Delivered-To: freebsd-net@FreeBSD.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 3B7D73CC for ; Sun, 20 Jul 2014 18:21:09 +0000 (UTC) Received: from kenobi.freebsd.org (kenobi.freebsd.org [IPv6:2001:1900:2254:206a::16:76]) (using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 231492D76 for ; Sun, 20 Jul 2014 18:21:09 +0000 (UTC) Received: from bugs.freebsd.org ([127.0.1.118]) by kenobi.freebsd.org (8.14.8/8.14.8) with ESMTP id s6KIL9wS047985 for ; Sun, 20 Jul 2014 18:21:09 GMT (envelope-from bugzilla-noreply@freebsd.org) From: bugzilla-noreply@freebsd.org To: freebsd-net@FreeBSD.org Subject: [Bug 191975] [ng_iface] [regression] in 10.0: cannot contact local services Date: Sun, 20 Jul 2014 18:21:09 +0000 X-Bugzilla-Reason: AssignedTo X-Bugzilla-Type: changed X-Bugzilla-Watch-Reason: None X-Bugzilla-Product: Base System X-Bugzilla-Component: kern X-Bugzilla-Version: 10.0-RELEASE X-Bugzilla-Keywords: X-Bugzilla-Severity: Affects Some People X-Bugzilla-Who: linimon@FreeBSD.org X-Bugzilla-Status: Needs Triage X-Bugzilla-Priority: Normal X-Bugzilla-Assigned-To: freebsd-net@FreeBSD.org X-Bugzilla-Target-Milestone: --- X-Bugzilla-Flags: X-Bugzilla-Changed-Fields: assigned_to short_desc Message-ID: In-Reply-To: References: Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: 7bit X-Bugzilla-URL: https://bugs.freebsd.org/bugzilla/ Auto-Submitted: auto-generated MIME-Version: 1.0 X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 20 Jul 2014 18:21:09 -0000 https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=191975 Mark Linimon changed: What |Removed |Added ---------------------------------------------------------------------------- Assignee|freebsd-bugs@FreeBSD.org |freebsd-net@FreeBSD.org Summary|ng_iface regression in |[ng_iface] [regression] in |10.0: cannot contact local |10.0: cannot contact local |services |services --- Comment #1 from Mark Linimon --- Over to maintainers. -- You are receiving this mail because: You are the assignee for the bug. From owner-freebsd-net@FreeBSD.ORG Sun Jul 20 18:41:03 2014 Return-Path: Delivered-To: freebsd-net@FreeBSD.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 2230D9D7 for ; Sun, 20 Jul 2014 18:41:03 +0000 (UTC) Received: from kenobi.freebsd.org (kenobi.freebsd.org [IPv6:2001:1900:2254:206a::16:76]) (using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 09E212F0C for ; Sun, 20 Jul 2014 18:41:03 +0000 (UTC) Received: from bugs.freebsd.org ([127.0.1.118]) by kenobi.freebsd.org (8.14.8/8.14.8) with ESMTP id s6KIf2XN004101 for ; Sun, 20 Jul 2014 18:41:02 GMT (envelope-from bugzilla-noreply@freebsd.org) From: bugzilla-noreply@freebsd.org To: freebsd-net@FreeBSD.org Subject: [Bug 191786] [bce] [patch] bce link state changes to same state are ignored by lagg Date: Sun, 20 Jul 2014 18:41:03 +0000 X-Bugzilla-Reason: AssignedTo X-Bugzilla-Type: changed X-Bugzilla-Watch-Reason: None X-Bugzilla-Product: Base System X-Bugzilla-Component: kern X-Bugzilla-Version: 9.2-RELEASE X-Bugzilla-Keywords: X-Bugzilla-Severity: Affects Only Me X-Bugzilla-Who: linimon@FreeBSD.org X-Bugzilla-Status: Needs Triage X-Bugzilla-Priority: --- X-Bugzilla-Assigned-To: freebsd-net@FreeBSD.org X-Bugzilla-Target-Milestone: --- X-Bugzilla-Flags: X-Bugzilla-Changed-Fields: assigned_to short_desc Message-ID: In-Reply-To: References: Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: 7bit X-Bugzilla-URL: https://bugs.freebsd.org/bugzilla/ Auto-Submitted: auto-generated MIME-Version: 1.0 X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 20 Jul 2014 18:41:03 -0000 https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=191786 Mark Linimon changed: What |Removed |Added ---------------------------------------------------------------------------- Assignee|freebsd-bugs@FreeBSD.org |freebsd-net@FreeBSD.org Summary|bce link state changes to |[bce] [patch] bce link |same state are ignored by |state changes to same state |lagg |are ignored by lagg --- Comment #3 from Mark Linimon --- Over to maintainers. -- You are receiving this mail because: You are the assignee for the bug. From owner-freebsd-net@FreeBSD.ORG Sun Jul 20 19:51:31 2014 Return-Path: Delivered-To: net@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 9851B392; Sun, 20 Jul 2014 19:51:31 +0000 (UTC) Received: from esa-jnhn.mail.uoguelph.ca (esa-jnhn.mail.uoguelph.ca [131.104.91.44]) by mx1.freebsd.org (Postfix) with ESMTP id 4EB9F25B9; Sun, 20 Jul 2014 19:51:30 +0000 (UTC) X-IronPort-Anti-Spam-Filtered: true X-IronPort-Anti-Spam-Result: AqYEAIUczFODaFve/2dsb2JhbABZDoNSW4J0wh4KhnFTAYEcdoQEAQEEAQEBICsgCxsYAgINGQIpAQkmBggHBAEcBIghDak2lnQXgSyNTgEBGzQHgniBTgWYNoQ8kmKDAl4hLwEGgQU5 X-IronPort-AV: E=Sophos;i="5.01,696,1400040000"; d="scan'208";a="142171454" Received: from muskoka.cs.uoguelph.ca (HELO zcs3.mail.uoguelph.ca) ([131.104.91.222]) by esa-jnhn.mail.uoguelph.ca with ESMTP; 20 Jul 2014 15:51:23 -0400 Received: from zcs3.mail.uoguelph.ca (localhost.localdomain [127.0.0.1]) by zcs3.mail.uoguelph.ca (Postfix) with ESMTP id 8AEF2B3F17; Sun, 20 Jul 2014 15:51:23 -0400 (EDT) Date: Sun, 20 Jul 2014 15:51:23 -0400 (EDT) From: Rick Macklem To: Garrett Wollman Message-ID: <1479705503.1118601.1405885883555.JavaMail.root@uoguelph.ca> In-Reply-To: <201407201556.s6KFuchL013781@hergotha.csail.mit.edu> Subject: Re: NFS client READ performance on -current MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: 7bit X-Originating-IP: [172.17.91.209] X-Mailer: Zimbra 7.2.6_GA_2926 (ZimbraWebClient - FF3.0 (Win)/7.2.6_GA_2926) Cc: jhb@freebsd.org, net@freebsd.org X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 20 Jul 2014 19:51:31 -0000 Garrett Wollman wrote: > In article <201407151034.54681.jhb@freebsd.org>, jhb@freebsd.org > writes: > > >Hmm, I am surprised by the m_pullup() behavior that it doesn't just > >notice that the first mbuf with a cluster has the desired data > >already > >and returns without doing anything. > > The specification of m_pullup() is that it returns a *writable* mbuf > (and thus also that the "length" provided is less than MHLEN). > Clusters are read-only. > I suspect you already know this, but being nit-picky... I think the cluster starts out rw, but become M_RDONLY (not M_WRITABLE()) when copied by reference. Since that will happen in TCP before a segment gets handed to a device driver for transmission, I think it will always be M_RDONLY in the device driver's output function. Does that sound correct? rick > -GAWollman > > _______________________________________________ > freebsd-net@freebsd.org mailing list > http://lists.freebsd.org/mailman/listinfo/freebsd-net > To unsubscribe, send any mail to > "freebsd-net-unsubscribe@freebsd.org" > From owner-freebsd-net@FreeBSD.ORG Mon Jul 21 03:27:38 2014 Return-Path: Delivered-To: freebsd-net@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 17D05EDC for ; Mon, 21 Jul 2014 03:27:38 +0000 (UTC) Received: from mail-qg0-x22f.google.com (mail-qg0-x22f.google.com [IPv6:2607:f8b0:400d:c04::22f]) (using TLSv1 with cipher ECDHE-RSA-RC4-SHA (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id CA7342CB0 for ; Mon, 21 Jul 2014 03:27:37 +0000 (UTC) Received: by mail-qg0-f47.google.com with SMTP id i50so4972631qgf.34 for ; Sun, 20 Jul 2014 20:27:36 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :cc:content-type; bh=UsgFZLwzES2A+RQgDzxtj/Wi06r+mc36EZV6sNVO2M8=; b=dnDiipOrpnkG3F/2gwB0YYxWXWdq3NWJkp+vkqgL7556xu97wdCPWVUbd+SnythQPh QCUgbtGTgX43gC5a7aNMa1kBMIKJ5xe9ZdwKzQnEg1XzkiLLTYzi+gpjcInc9QpBfKmS A3vxNPmpEuN8ZXYT3UxKF6vY0a93TT5MCbOjdd6qDg9UREfsmWVkGEjlGd5o/Be9VFia N4mtyo/gCp+/f3ShI/eNJxpACRbhbkfKsIMRPolhUEN3vJ115oyVRT2jJo4rfylUs6QC qBhsaMnx3FK2ftzmrrbx6rmGliyrRB5i4cSat2FX3pT/vAgSCbhg5qQXHSTYqxLx4o6h ESkw== MIME-Version: 1.0 X-Received: by 10.224.160.134 with SMTP id n6mr18370993qax.84.1405913256775; Sun, 20 Jul 2014 20:27:36 -0700 (PDT) Received: by 10.224.8.136 with HTTP; Sun, 20 Jul 2014 20:27:36 -0700 (PDT) In-Reply-To: <00d201cfa436$d2936ce0$77ba46a0$@gmail.com> References: <00d201cfa436$d2936ce0$77ba46a0$@gmail.com> Date: Mon, 21 Jul 2014 11:27:36 +0800 Message-ID: Subject: Re: A TCP problem on High Delay From: Niu Zhixiong To: bycn82 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: quoted-printable X-Content-Filtered-By: Mailman/MimeDel 2.1.18 Cc: freebsd-net@freebsd.org X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 21 Jul 2014 03:27:38 -0000 Hi, Thank you so much. I am wondering that why FreeBSD default is less than Linux default in high delay * BW. Why, Why and Why. @FreeBSD_KVM:~ % sysctl net.inet.tcp net.inet.tcp.rfc1323: 1 net.inet.tcp.mssdflt: 536 net.inet.tcp.keepidle: 7200000 net.inet.tcp.keepintvl: 75000 net.inet.tcp.sendspace: 32768 net.inet.tcp.recvspace: 65536 net.inet.tcp.keepinit: 75000 net.inet.tcp.delacktime: 100 net.inet.tcp.v6mssdflt: 1220 net.inet.tcp.cc.algorithm: newreno net.inet.tcp.cc.available: newreno net.inet.tcp.hostcache.cachelimit: 15360 net.inet.tcp.hostcache.hashsize: 512 net.inet.tcp.hostcache.bucketlimit: 30 net.inet.tcp.hostcache.count: 0 net.inet.tcp.hostcache.expire: 1 net.inet.tcp.hostcache.prune: 300 net.inet.tcp.hostcache.purge: 0 net.inet.tcp.log_in_vain: 0 net.inet.tcp.blackhole: 0 net.inet.tcp.delayed_ack: 1 net.inet.tcp.drop_synfin: 0 net.inet.tcp.rfc3042: 1 net.inet.tcp.rfc3390: 1 net.inet.tcp.experimental.initcwnd10: 1 net.inet.tcp.rfc3465: 1 net.inet.tcp.abc_l_var: 2 net.inet.tcp.ecn.enable: 0 net.inet.tcp.ecn.maxretries: 1 net.inet.tcp.insecure_rst: 0 net.inet.tcp.recvbuf_auto: 1 net.inet.tcp.recvbuf_inc: 524288 net.inet.tcp.recvbuf_max: 16777216 net.inet.tcp.path_mtu_discovery: 1 net.inet.tcp.tso: 1 net.inet.tcp.sendbuf_auto: 1 net.inet.tcp.sendbuf_inc: 16384 net.inet.tcp.sendbuf_max: 16777216 net.inet.tcp.reass.maxsegments: 7900 net.inet.tcp.reass.cursegments: 0 net.inet.tcp.reass.overflows: 0 net.inet.tcp.sack.enable: 1 net.inet.tcp.sack.maxholes: 128 net.inet.tcp.sack.globalmaxholes: 65536 net.inet.tcp.sack.globalholes: 0 net.inet.tcp.minmss: 216 net.inet.tcp.log_debug: 0 net.inet.tcp.tcbhashsize: 16384 net.inet.tcp.do_tcpdrain: 1 net.inet.tcp.pcbcount: 4 net.inet.tcp.icmp_may_rst: 1 net.inet.tcp.isn_reseed_interval: 0 net.inet.tcp.soreceive_stream: 0 net.inet.tcp.syncookies: 1 net.inet.tcp.syncookies_only: 0 net.inet.tcp.syncache.bucketlimit: 30 net.inet.tcp.syncache.cachelimit: 15375 net.inet.tcp.syncache.count: 0 net.inet.tcp.syncache.hashsize: 512 net.inet.tcp.syncache.rexmtlimit: 3 net.inet.tcp.syncache.rst_on_sock_fail: 1 net.inet.tcp.msl: 30000 net.inet.tcp.rexmit_min: 30 net.inet.tcp.rexmit_slop: 200 net.inet.tcp.always_keepalive: 1 net.inet.tcp.fast_finwait2_recycle: 0 net.inet.tcp.finwait2_timeout: 60000 net.inet.tcp.keepcnt: 8 net.inet.tcp.rexmit_drop_options: 0 net.inet.tcp.per_cpu_timers: 0 net.inet.tcp.timer_race: 0 net.inet.tcp.maxtcptw: 12963 net.inet.tcp.nolocaltimewait: 0 Regards, Niu Zhixiong =EF=BC=8D=EF=BC=8D=EF=BC=8D=EF=BC=8D=EF=BC=8D=EF=BC=8D=EF=BC=8D=EF=BC=8D=EF= =BC=8D=EF=BC=8D=EF=BC=8D=EF=BC=8D=EF=BC=8D=EF=BC=8D=EF=BC=8D kaiaixi@gmail.com On Mon, Jul 21, 2014 at 12:22 AM, bycn82 wrote: > Hi, > Yes, according to your pcap files,it is because of the window size. > more information here http://www.ietf.org/rfc/rfc1323.txt > and share the result of "sysctl net.inet.tcp" > > Regards, > bycn82 > > > > > -----Original Message----- > > From: owner-freebsd-net@freebsd.org [mailto:owner-freebsd- > > net@freebsd.org] On Behalf Of Niu Zhixiong > > Sent: 20 July, 2014 23:43 > > To: freebsd-net@freebsd.org > > Subject: A TCP problem on High Delay > > > > Dear All, > > I set up an FTP server in a high delay but fairly BW environment. > > When I try to download files from this FreeBSD FTP server, A FreeBS= D > > client is always slower than Linux Client. (Linux Client is about 1MB/s= , > > But FreeBSD client 300KB/s). I am sure that the two clients are in same > > networks and no firewalls are between server and clients. Then I try to > > use SCTP protocol rather than TCP protocol. The FreeBSD client is faste= r > > than itself in TCP. > > > > I try to increase the rbuff of the client. But, It always less than the > > linux one. I think the problem may be the smaller window of Freebsd tha= n > > Linux. But, I changed some parameters. But, nothing changed. > > > > uname -a > > 10.0-RELEASE FreeBSD 10.0-RELEASE #0 r260789: Thu Jan 16 22:34:59 UTC > > 2014 > > root@snap.freebsd.org:/usr/obj/usr/src/sys/GENERIC amd64 > > > > According to > > http://www.freebsd.org/doc/en_US.ISO8859-1/books/handbook/configtuning- > > kernel-limits.html > > > > I try to enable sysctl net.inet.tcp.inflight.enable or sysctl > > net.inet.tcp.inflight_enable , but there is no such option in FreeBSD 1= 0. > > > > Then I follow this https://fasterdata.es.net/host-tuning/freebsd/ > > And make a large rbuffer. > > > > > > sysctl kern.ipc.maxsockbuf=3D16777216 > > sysctl net.inet.tcp.sendbuf_max=3D16777216 > > sysctl net.inet.tcp.recvbuf_max=3D16777216 > > sysctl net.inet.tcp.sendbuf_auto=3D1 > > sysctl net.inet.tcp.recvbuf_auto=3D1 > > sysctl net.inet.tcp.sendbuf_inc=3D16384 > > sysctl net.inet.tcp.recvbuf_inc=3D524288 > > sysctl net.inet.tcp.hostcache.expire=3D1 > > > > But, Nothing is happened. > > > > In the attachments, l_f is the linux packet capture and f_f is the > > freebsd packet capture. > > > > > > > > > > =E2=80=8B > > packet_capture.7z > > > ive_web> > > =E2=80=8B > > _______________________________________________ > > freebsd-net@freebsd.org mailing list > > http://lists.freebsd.org/mailman/listinfo/freebsd-net > > To unsubscribe, send any mail to "freebsd-net-unsubscribe@freebsd.org" > > From owner-freebsd-net@FreeBSD.ORG Mon Jul 21 08:00:13 2014 Return-Path: Delivered-To: freebsd-net@FreeBSD.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 47CBB557 for ; Mon, 21 Jul 2014 08:00:13 +0000 (UTC) Received: from kenobi.freebsd.org (kenobi.freebsd.org [IPv6:2001:1900:2254:206a::16:76]) (using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 1D3D422CB for ; Mon, 21 Jul 2014 08:00:13 +0000 (UTC) Received: from bugs.freebsd.org ([127.0.1.118]) by kenobi.freebsd.org (8.14.8/8.14.8) with ESMTP id s6L80CHd006499 for ; Mon, 21 Jul 2014 08:00:12 GMT (envelope-from bugzilla-noreply@freebsd.org) Message-Id: <201407210800.s6L80CHd006499@kenobi.freebsd.org> From: bugzilla-noreply@freebsd.org To: freebsd-net@FreeBSD.org Subject: [Bugzilla] Commit Needs MFC MIME-Version: 1.0 X-Bugzilla-Type: whine X-Bugzilla-URL: https://bugs.freebsd.org/bugzilla/ Auto-Submitted: auto-generated Date: Mon, 21 Jul 2014 08:00:12 +0000 Content-Type: text/plain X-Content-Filtered-By: Mailman/MimeDel 2.1.18 X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 21 Jul 2014 08:00:13 -0000 Hi, You have a bug in the "Needs MFC" state which has not been touched in 7 or more days. This email serves as a reminder that you may want to MFC this bug or marked it as completed. In the event you have a longer MFC timeout you may update this bug with a comment and I won't remind you again for 7 days. This reminder is only sent on Mondays. Please file a bug about concerns you may have. This search was scheduled by eadler@FreeBSD.org. (1 bugs) Bug 183659: https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=183659 Severity: Affects Only Me Priority: Normal Hardware: Any Assignee: freebsd-net@FreeBSD.org Status: Needs MFC Resolution: Summary: [tcp] TCP stack lock contention with short-lived connections From owner-freebsd-net@FreeBSD.ORG Mon Jul 21 14:18:53 2014 Return-Path: Delivered-To: freebsd-net@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 16AC747B for ; Mon, 21 Jul 2014 14:18:53 +0000 (UTC) Received: from a0i308.smtpcorp.com (a0i308.smtpcorp.com [216.22.15.140]) (using TLSv1.2 with cipher DHE-RSA-AES128-SHA (128/128 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id E294D2B18 for ; Mon, 21 Jul 2014 14:18:52 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=smtpcorp.com; s=a0_1; h=Content-Type:MIME-Version:Message-ID:In-Reply-To:Date:References:Subject:Cc:To:From; bh=MrOZW00VtDKbBu09dc7SLBLjoOEzDEyLsSdL/L5/yD8=; b=gLGZpIqKGepEPYt+NFmpns9dbETkUWDcvfkCeOoE5crmAZbK6Jjcb7eCh+7Wi0rYuShJvK8kW+MCmHgHq+qDcQGFvuxBbto8IVFbLiEUOg11Vy2MIBHaeI+IGP7SZGuwLY+yVTg986g6zFORD1hQO9OQiLQeFXqBiuuaH2F+CGs=; From: Daniel Corbe To: "Alexander V. Chernikov" Subject: Re: netmap, selective processing. References: <53CA3B4E.8080608@FreeBSD.org> Date: Mon, 21 Jul 2014 10:18:24 -0400 In-Reply-To: <53CA3B4E.8080608@FreeBSD.org> (Alexander V. Chernikov's message of "Sat, 19 Jul 2014 13:33:02 +0400") Message-ID: User-Agent: Gnus/5.13 (Gnus v5.13) Emacs/24.3 (berkeley-unix) MIME-Version: 1.0 Content-Type: text/plain X-Smtpcorp-Track: 1b9EQe4gfHYq05.zSFEyUPz0 Cc: freebsd-net@freebsd.org X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 21 Jul 2014 14:18:53 -0000 "Alexander V. Chernikov" writes: > On 16.07.2014 21:48, Daniel Corbe wrote: > Hm. What do you need from bird OSPF implementation? > IMHO it is much easier to improve and merge bird code instead of > writing another OSPF implementation from scratch. > I can't get an OSPF adjacency up between a bird box and a Cisco 3750. I've tried seeking help both in IRC and on their mailing list. If I can't get basic help using the software, I'm certainly not going to contribute to the project in other ways. And that's too bad, too because bird looks like a promising project. From owner-freebsd-net@FreeBSD.ORG Mon Jul 21 14:26:09 2014 Return-Path: Delivered-To: freebsd-net@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 1CC5B6C3 for ; Mon, 21 Jul 2014 14:26:09 +0000 (UTC) Received: from mail.ipfw.ru (mail.ipfw.ru [IPv6:2a01:4f8:120:6141::2]) (using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id D56B72BDE for ; Mon, 21 Jul 2014 14:26:08 +0000 (UTC) Received: from v6.mpls.in ([2a02:978:2::5] helo=ws.su29.net) by mail.ipfw.ru with esmtpsa (TLSv1:DHE-RSA-AES128-SHA:128) (Exim 4.82 (FreeBSD)) (envelope-from ) id 1X9Aax-000Jar-DO; Mon, 21 Jul 2014 14:13:23 +0400 Message-ID: <53CD22FB.3070706@FreeBSD.org> Date: Mon, 21 Jul 2014 18:26:03 +0400 From: "Alexander V. Chernikov" User-Agent: Mozilla/5.0 (X11; FreeBSD amd64; rv:24.0) Gecko/20100101 Thunderbird/24.6.0 MIME-Version: 1.0 To: Daniel Corbe Subject: Re: netmap, selective processing. References: <53CA3B4E.8080608@FreeBSD.org> In-Reply-To: X-Enigmail-Version: 1.6 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 7bit Cc: freebsd-net@freebsd.org X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 21 Jul 2014 14:26:09 -0000 On 21.07.2014 18:18, Daniel Corbe wrote: > "Alexander V. Chernikov" writes: > >> On 16.07.2014 21:48, Daniel Corbe wrote: >> Hm. What do you need from bird OSPF implementation? >> IMHO it is much easier to improve and merge bird code instead of >> writing another OSPF implementation from scratch. >> > > I can't get an OSPF adjacency up between a bird box and a Cisco 3750. > I've tried seeking help both in IRC and on their mailing list. If I Oh, sorry. I've missed you e-mail in ML. However it looks like Ondrej has already answered you. > can't get basic help using the software, I'm certainly not going to > contribute to the project in other ways. > > And that's too bad, too because bird looks like a promising project. It definitely is, and it is evolving. > From owner-freebsd-net@FreeBSD.ORG Mon Jul 21 15:34:43 2014 Return-Path: Delivered-To: freebsd-net@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 4488D567 for ; Mon, 21 Jul 2014 15:34:43 +0000 (UTC) Received: from mail-ig0-x232.google.com (mail-ig0-x232.google.com [IPv6:2607:f8b0:4001:c05::232]) (using TLSv1 with cipher ECDHE-RSA-RC4-SHA (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 1166522E0 for ; Mon, 21 Jul 2014 15:34:43 +0000 (UTC) Received: by mail-ig0-f178.google.com with SMTP id uq10so2945898igb.11 for ; Mon, 21 Jul 2014 08:34:42 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=message-id:date:from:user-agent:mime-version:to:subject :content-type:content-transfer-encoding; bh=crchBRbrynphQsfAr8xZLiGtoNDWz8yX+gnLOxCxGHs=; b=gYT0rRz9NjHg4ubgdFgmnL8oGV+/2dhfjt1uDC6Deo+vnZQYyPBb2xX5/byF/GkeW7 qDCO1X3XcFf+qekXEg5yuH6mhtr6h3RyUF3TXRNPDjB44RiC1IZ91AWwCnP5aecicL8b 4MO7iugjg8sEaomtvtV0nxgWWjExaQLHIFqLrQKuwMMvoNOce+fGZXEOJ4WBGgGPc/7s Ct2Bizy0e11UCszrBxFhn/IY2w5tQHjlIyybO6v/kJG7ef280O0QtgHhXHfmn/7P0v18 19O5XdSM/DWAqZ8qQc/zT+yeE6newmRZx+DY2Xka5z1ch3v9NLEsocp+gCScZd7nKw3F UjOQ== X-Received: by 10.50.33.16 with SMTP id n16mr6261087igi.15.1405956882412; Mon, 21 Jul 2014 08:34:42 -0700 (PDT) Received: from [10.1.68.187] (gs-sv-1-49-ac1.gsfc.nasa.gov. [198.119.56.43]) by mx.google.com with ESMTPSA id il3sm39674316igb.1.2014.07.21.08.34.40 for (version=TLSv1 cipher=ECDHE-RSA-RC4-SHA bits=128/128); Mon, 21 Jul 2014 08:34:41 -0700 (PDT) Message-ID: <53CD330E.6090407@gmail.com> Date: Mon, 21 Jul 2014 11:34:38 -0400 From: John Jasen User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:24.0) Gecko/20100101 Thunderbird/24.6.0 MIME-Version: 1.0 To: FreeBSD Net , Navdeep Parhar Subject: packet forwarding and possible mitigation of Intel QuickPath Interconnect ugliness in multi cpu systems X-Enigmail-Version: 1.5.2 Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: 7bit X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 21 Jul 2014 15:34:43 -0000 Executive Summary: Appropriate use of cpuset(1) can mitigate performance bottlenecks over the Intel QPI processor interconnection, and improve packets-per-second processing rate by over 100%. Test Environment: My test system is a Dell dual CPU R820, populated with evaluation cards graciously provided by Chelsio. Currently, each dual port chelsio card is populated in a 16x slot, one physically attached to each CPU. My load generators are 20 CentOS-based linux systems, using Mellanox VPI ConnectX-3 cards in ethernet mode. The test environment divides the load generators into 4 distinct subnets of 5 systems, with each one utilizing a Chelsio interface as its route to the other networks. I use iperf3 on the linux systems to generate packets. The test runs select two systems on each subnet to be a sender, and three on each to be receivers. The sending systems establish 4 UDP streams to each receiver. Results: netstat -w 1 -q 100 -d before each run I summarized results with the following. awk '{ipackets+=$1} {idrops+=$3} {opackets+=$5} {odrops+=$9} END {print "input " ipackets/NR, "idrops " idrops/NR, "opackets " opackets/NR, "odrops " odrops/NR}' Without any cpuset tuning at all: input 7.25464e+06 idrops 5.89939e+06 opackets 1.34888e+06 odrops 947.409 With cpuset assigning interrupts equally to each physical processor: input 1.10886e+07 idrops 9.85347e+06 opackets 1.22887e+06 odrops 3384.86 cpuset assigning interrupts across cores on the first physical processor: input 1.14046e+07 idrops 8.6674e+06 opackets 2.73365e+06 odrops 2420.75 cpuset assigning interrupts across cores on the second physical processor: input 1.16746e+07 idrops 8.96412e+06 opackets 2.70652e+06 odrops 3076.52 I will follow this up with both cards being in PCIE slots physically connected to the first CPU, but for a rule of thumb comparision, with cpuset'ing the interrupts appropriately, it was usually about 10-15% higher than cpuset-one-processor-low and cpuset-one-processor-high. Conclusion: The best solution for highest performance is still to avoid QPI as much as possible, by appropriate physical placement of the PCIe cards. However, in cases where that may not be possible or desirable, using cpuset to assign all the interrupt affinity to one processor will help mitigate performance loss. Credits: Thanks to Dell for the loan of the Dell R820 using for testing; Thanks to Chelsio for the loan of the two T580-CR cards; and thanks to the CXGBE maintainer, Navdeep Parhar, for his assistance and patience during debugging and testing. Feedback is always welcome. I can provide detailed results upon request. Scripts provided by a vendor, I need to get their permission to redistribute/publish, but I do not think thats a problem. From owner-freebsd-net@FreeBSD.ORG Mon Jul 21 15:40:29 2014 Return-Path: Delivered-To: freebsd-net@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 2E09C67B for ; Mon, 21 Jul 2014 15:40:29 +0000 (UTC) Received: from mail-qg0-x22f.google.com (mail-qg0-x22f.google.com [IPv6:2607:f8b0:400d:c04::22f]) (using TLSv1 with cipher ECDHE-RSA-RC4-SHA (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id E31E9239B for ; Mon, 21 Jul 2014 15:40:28 +0000 (UTC) Received: by mail-qg0-f47.google.com with SMTP id i50so5402745qgf.20 for ; Mon, 21 Jul 2014 08:40:28 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:sender:in-reply-to:references:date:message-id:subject :from:to:cc:content-type; bh=rlTD13N7SRgen9nHtaX+qAQgHtwRavUFTLzA2QZti14=; b=ZwpdsDYJgW6WnistVdapm4QMsxGAwR2xJXygcot/Z41VF3+WRzzIEbIZR1En6OVCUL km6Sc8bA9RoCPenxx5owXUWTfU0j3f7K7cYQoMVvwOtmVw1AVC10RCxQWU9D0e+St6Wb s4wYmsT0axsFj/VGsz9WogFGuPUWBv0Jq5WHMzpJa25qoEizNXBwykARSEJkzbziO/u7 +QBij2VLlp+pprkRB+BZFoPtqk2uWMzkI8/MkFa6dO7n2XO1vqEoj8UbYwfRGyMmNAYs lz1L6RrDA3nusLxLoksrVSx4ykAx3TEOrbZhrchEuWqttYGmjiQOPeXotqYkOStY+nI7 i54w== MIME-Version: 1.0 X-Received: by 10.140.41.133 with SMTP id z5mr31643908qgz.99.1405957228112; Mon, 21 Jul 2014 08:40:28 -0700 (PDT) Sender: adrian.chadd@gmail.com Received: by 10.224.1.6 with HTTP; Mon, 21 Jul 2014 08:40:28 -0700 (PDT) In-Reply-To: <53CD330E.6090407@gmail.com> References: <53CD330E.6090407@gmail.com> Date: Mon, 21 Jul 2014 08:40:28 -0700 X-Google-Sender-Auth: 84pycMoUlB37bbzZPFl2P1MMjAM Message-ID: Subject: Re: packet forwarding and possible mitigation of Intel QuickPath Interconnect ugliness in multi cpu systems From: Adrian Chadd To: John Jasen Content-Type: text/plain; charset=UTF-8 Cc: FreeBSD Net , Navdeep Parhar X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 21 Jul 2014 15:40:29 -0000 Hi! Yup. We need to extend the bus code to expose topology so we can have the drivers spawn their interrupt threads bound to local CPUs (and as an option not do this.) I'm _almost_ at the point where I'm about to take the long walk down this path. Nice work! -a On 21 July 2014 08:34, John Jasen wrote: > Executive Summary: > > Appropriate use of cpuset(1) can mitigate performance bottlenecks over > the Intel QPI processor interconnection, and improve packets-per-second > processing rate by over 100%. > > Test Environment: > > My test system is a Dell dual CPU R820, populated with evaluation cards > graciously provided by Chelsio. Currently, each dual port chelsio card > is populated in a 16x slot, one physically attached to each CPU. > > My load generators are 20 CentOS-based linux systems, using Mellanox VPI > ConnectX-3 cards in ethernet mode. > > The test environment divides the load generators into 4 distinct subnets > of 5 systems, with each one utilizing a Chelsio interface as its route > to the other networks. I use iperf3 on the linux systems to generate > packets. > > The test runs select two systems on each subnet to be a sender, and > three on each to be receivers. The sending systems establish 4 UDP > streams to each receiver. > > Results: > > netstat -w 1 -q 100 -d before each run > > I summarized results with the following. > awk '{ipackets+=$1} {idrops+=$3} {opackets+=$5} {odrops+=$9} END {print > "input " ipackets/NR, "idrops " idrops/NR, "opackets " opackets/NR, > "odrops " odrops/NR}' > > Without any cpuset tuning at all: > > input 7.25464e+06 idrops 5.89939e+06 opackets 1.34888e+06 odrops 947.409 > > With cpuset assigning interrupts equally to each physical processor: > > input 1.10886e+07 idrops 9.85347e+06 opackets 1.22887e+06 odrops 3384.86 > > cpuset assigning interrupts across cores on the first physical processor: > > input 1.14046e+07 idrops 8.6674e+06 opackets 2.73365e+06 odrops 2420.75 > > cpuset assigning interrupts across cores on the second physical processor: > > input 1.16746e+07 idrops 8.96412e+06 opackets 2.70652e+06 odrops 3076.52 > > I will follow this up with both cards being in PCIE slots physically > connected to the first CPU, but for a rule of thumb comparision, with > cpuset'ing the interrupts appropriately, it was usually about 10-15% > higher than cpuset-one-processor-low and cpuset-one-processor-high. > > Conclusion: > > The best solution for highest performance is still to avoid QPI as much > as possible, by appropriate physical placement of the PCIe cards. > However, in cases where that may not be possible or desirable, using > cpuset to assign all the interrupt affinity to one processor will help > mitigate performance loss. > > Credits: > > Thanks to Dell for the loan of the Dell R820 using for testing; Thanks > to Chelsio for the loan of the two T580-CR cards; and thanks to the > CXGBE maintainer, Navdeep Parhar, for his assistance and patience during > debugging and testing. > > Feedback is always welcome. > > I can provide detailed results upon request. > > Scripts provided by a vendor, I need to get their permission to > redistribute/publish, but I do not think thats a problem. > > > > > > > > > > > > > > > > > > _______________________________________________ > freebsd-net@freebsd.org mailing list > http://lists.freebsd.org/mailman/listinfo/freebsd-net > To unsubscribe, send any mail to "freebsd-net-unsubscribe@freebsd.org" From owner-freebsd-net@FreeBSD.ORG Mon Jul 21 17:03:37 2014 Return-Path: Delivered-To: net@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id C4076735 for ; Mon, 21 Jul 2014 17:03:37 +0000 (UTC) Received: from bigwig.baldwin.cx (bigwig.baldwin.cx [IPv6:2001:470:1f11:75::1]) (using TLSv1 with cipher DHE-RSA-CAMELLIA256-SHA (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 9C23F2D26 for ; Mon, 21 Jul 2014 17:03:37 +0000 (UTC) Received: from pippin.baldwin.cx (75-48-77-17.lightspeed.cncrca.sbcglobal.net [75.48.77.17]) by bigwig.baldwin.cx (Postfix) with ESMTPSA id 18C4BB91E; Mon, 21 Jul 2014 13:03:35 -0400 (EDT) From: John Baldwin To: Garrett Wollman Subject: Re: NFS client READ performance on -current Date: Mon, 21 Jul 2014 12:57:35 -0400 Message-ID: <2198372.PMBsJPH0RH@pippin.baldwin.cx> User-Agent: KMail/4.12.5 (FreeBSD/11.0-CURRENT; KDE/4.12.5; amd64; ; ) In-Reply-To: <201407201556.s6KFuchL013781@hergotha.csail.mit.edu> References: <2136988575.13956627.1405199640153.JavaMail.root@uoguelph.ca> <201407201556.s6KFuchL013781@hergotha.csail.mit.edu> MIME-Version: 1.0 Content-Transfer-Encoding: 7Bit Content-Type: text/plain; charset="us-ascii" X-Greylist: Sender succeeded SMTP AUTH, not delayed by milter-greylist-4.2.7 (bigwig.baldwin.cx); Mon, 21 Jul 2014 13:03:35 -0400 (EDT) Cc: net@freebsd.org X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 21 Jul 2014 17:03:37 -0000 On Sunday 20 July 2014 11:56:38 Garrett Wollman wrote: > In article <201407151034.54681.jhb@freebsd.org>, jhb@freebsd.org writes: > >Hmm, I am surprised by the m_pullup() behavior that it doesn't just > >notice that the first mbuf with a cluster has the desired data already > >and returns without doing anything. > > The specification of m_pullup() is that it returns a *writable* mbuf > (and thus also that the "length" provided is less than MHLEN). > Clusters are read-only. Well, my patch to if_em is definitely correct then as it doesn't want a writable mbuf, it just wants a certain portion of the mbuf chain to be stored in the first segment. Also, you should fix the manpage. It doesn't mention writable at all, just that the data is accessible so that mtod() will work: m_pullup(mbuf, len) Arrange that the first len bytes of an mbuf chain are contiguous and lay in the data area of mbuf, so they are accessible with mtod(mbuf, type). It is important to remember that this may involve reallocating some mbufs and moving data so all pointers referencing data within the old mbuf chain must be recalculated or made invalid. Return the new mbuf chain on success, NULL on fail- ure (the mbuf chain is freed in this case). Note: It does not allocate any mbuf clusters, so len must be less than or equal to MHLEN. -- John Baldwin From owner-freebsd-net@FreeBSD.ORG Mon Jul 21 17:16:09 2014 Return-Path: Delivered-To: freebsd-net@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id DEBFAADF for ; Mon, 21 Jul 2014 17:16:09 +0000 (UTC) Received: from mail-ie0-x22c.google.com (mail-ie0-x22c.google.com [IPv6:2607:f8b0:4001:c03::22c]) (using TLSv1 with cipher ECDHE-RSA-RC4-SHA (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id AB1602E29 for ; Mon, 21 Jul 2014 17:16:09 +0000 (UTC) Received: by mail-ie0-f172.google.com with SMTP id lx4so7071040iec.3 for ; Mon, 21 Jul 2014 10:16:09 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=message-id:date:from:user-agent:mime-version:to:subject:references :in-reply-to:content-type:content-transfer-encoding; bh=8bPi4PHy5BEaThWEyDrqOzGNdEZH+qzYQ4tIeXNF7Z4=; b=iUyujrM2UPVRxy+XHjAzBJSlw3Nfk1gY6Zkb+8MR4nY6rdd1JvUxYZMVz3m6wJTz15 ja8VBCxGtljItiQ8MqvatNC73egEssQuF45JylOLfPKC1HBxKaM9iaFb2f8/td7k8VaT VPsqitelPgybLk7kZu84rl++pf1NxuTazV9C3b2kldOs4n+X/maHLdro89OBVbeQlm3y iJxyXuS0tNd6UwdwBU5NVpJMltBeJ2EocjCfvSF0LMycKp8GhrxUR8pkT0yj3HemXIWO Ix4oO8+YtnNxj+ltz13vR95k3LrH133etkhVkcwhXazShZr4VUkKYOhXaoWaoUzZ2/ue uLwA== X-Received: by 10.42.130.133 with SMTP id v5mr14584414ics.57.1405962969157; Mon, 21 Jul 2014 10:16:09 -0700 (PDT) Received: from [10.1.68.187] (gs-sv-1-49-ac1.gsfc.nasa.gov. [198.119.56.43]) by mx.google.com with ESMTPSA id qp11sm33586401igb.21.2014.07.21.10.16.07 for (version=TLSv1 cipher=ECDHE-RSA-RC4-SHA bits=128/128); Mon, 21 Jul 2014 10:16:07 -0700 (PDT) Message-ID: <53CD4AD5.4030901@gmail.com> Date: Mon, 21 Jul 2014 13:16:05 -0400 From: John Jasen User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:24.0) Gecko/20100101 Thunderbird/24.6.0 MIME-Version: 1.0 To: FreeBSD Net Subject: Re: packet forwarding and possible mitigation of Intel QuickPath Interconnect ugliness in multi cpu systems References: <53CD330E.6090407@gmail.com> In-Reply-To: <53CD330E.6090407@gmail.com> X-Enigmail-Version: 1.5.2 Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: 7bit X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 21 Jul 2014 17:16:09 -0000 For completeness, the following are numbers for putting both dual port cards on PCIe lanes attached to physical CPU0. I'll note that the numbers for cpuset-1cpu-low are skewed. During testing, over 3.5 million packets per second was observed via netstat. I'll also note, untuned cpuset and cpuset across all available logical processors actually came in around 1 million and 887k packets-per-second forwarded, respectively. $ test3.cpuset-1cpu-low input 1.17679e+07 idrops 9.06871e+06 opackets 2.69356e+06 odrops 5577.45 $ test3.cpuset-1cpu-high input 1.34393e+07 idrops 1.05269e+07 opackets 2910347 odrops 1943.65 On 07/21/2014 11:34 AM, John Jasen wrote: > Executive Summary: > > Appropriate use of cpuset(1) can mitigate performance bottlenecks over > the Intel QPI processor interconnection, and improve packets-per-second > processing rate by over 100%. > > Test Environment: > > My test system is a Dell dual CPU R820, populated with evaluation cards > graciously provided by Chelsio. Currently, each dual port chelsio card > is populated in a 16x slot, one physically attached to each CPU. > > My load generators are 20 CentOS-based linux systems, using Mellanox VPI > ConnectX-3 cards in ethernet mode. > > The test environment divides the load generators into 4 distinct subnets > of 5 systems, with each one utilizing a Chelsio interface as its route > to the other networks. I use iperf3 on the linux systems to generate > packets. > > The test runs select two systems on each subnet to be a sender, and > three on each to be receivers. The sending systems establish 4 UDP > streams to each receiver. > > Results: > > netstat -w 1 -q 100 -d before each run > > I summarized results with the following. > awk '{ipackets+=$1} {idrops+=$3} {opackets+=$5} {odrops+=$9} END {print > "input " ipackets/NR, "idrops " idrops/NR, "opackets " opackets/NR, > "odrops " odrops/NR}' > > Without any cpuset tuning at all: > > input 7.25464e+06 idrops 5.89939e+06 opackets 1.34888e+06 odrops 947.409 > > With cpuset assigning interrupts equally to each physical processor: > > input 1.10886e+07 idrops 9.85347e+06 opackets 1.22887e+06 odrops 3384.86 > > cpuset assigning interrupts across cores on the first physical processor: > > input 1.14046e+07 idrops 8.6674e+06 opackets 2.73365e+06 odrops 2420.75 > > cpuset assigning interrupts across cores on the second physical processor: > > input 1.16746e+07 idrops 8.96412e+06 opackets 2.70652e+06 odrops 3076.52 > > I will follow this up with both cards being in PCIE slots physically > connected to the first CPU, but for a rule of thumb comparision, with > cpuset'ing the interrupts appropriately, it was usually about 10-15% > higher than cpuset-one-processor-low and cpuset-one-processor-high. > > Conclusion: > > The best solution for highest performance is still to avoid QPI as much > as possible, by appropriate physical placement of the PCIe cards. > However, in cases where that may not be possible or desirable, using > cpuset to assign all the interrupt affinity to one processor will help > mitigate performance loss. > > Credits: > > Thanks to Dell for the loan of the Dell R820 using for testing; Thanks > to Chelsio for the loan of the two T580-CR cards; and thanks to the > CXGBE maintainer, Navdeep Parhar, for his assistance and patience during > debugging and testing. > > Feedback is always welcome. > > I can provide detailed results upon request. > > Scripts provided by a vendor, I need to get their permission to > redistribute/publish, but I do not think thats a problem. > > > > > > > > > > > > > > > > > From owner-freebsd-net@FreeBSD.ORG Mon Jul 21 19:56:35 2014 Return-Path: Delivered-To: freebsd-net@FreeBSD.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 5EC153CD for ; Mon, 21 Jul 2014 19:56:35 +0000 (UTC) Received: from kenobi.freebsd.org (kenobi.freebsd.org [IPv6:2001:1900:2254:206a::16:76]) (using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 4651B2D3E for ; Mon, 21 Jul 2014 19:56:35 +0000 (UTC) Received: from bugs.freebsd.org ([127.0.1.118]) by kenobi.freebsd.org (8.14.8/8.14.8) with ESMTP id s6LJuZM7067828 for ; Mon, 21 Jul 2014 19:56:35 GMT (envelope-from bugzilla-noreply@freebsd.org) From: bugzilla-noreply@freebsd.org To: freebsd-net@FreeBSD.org Subject: [Bug 191975] [ng_iface] [regression] in 10.0: cannot contact local services Date: Mon, 21 Jul 2014 19:56:35 +0000 X-Bugzilla-Reason: AssignedTo X-Bugzilla-Type: changed X-Bugzilla-Watch-Reason: None X-Bugzilla-Product: Base System X-Bugzilla-Component: kern X-Bugzilla-Version: 10.0-RELEASE X-Bugzilla-Keywords: X-Bugzilla-Severity: Affects Some People X-Bugzilla-Who: dgilbert@eicat.ca X-Bugzilla-Status: Needs Triage X-Bugzilla-Priority: Normal X-Bugzilla-Assigned-To: freebsd-net@FreeBSD.org X-Bugzilla-Target-Milestone: --- X-Bugzilla-Flags: X-Bugzilla-Changed-Fields: Message-ID: In-Reply-To: References: Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: 7bit X-Bugzilla-URL: https://bugs.freebsd.org/bugzilla/ Auto-Submitted: auto-generated MIME-Version: 1.0 X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 21 Jul 2014 19:56:35 -0000 https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=191975 --- Comment #2 from dgilbert@eicat.ca --- I've done some additional work on the problem. One thing I have done is to enable the rc.conf variable ipv6_activate_all_interfaces="YES" ... which changes the ifconfig for the ngX interfaces to say: ng11: flags=88d1 metric 0 mtu 1436 inet 66.96.31.6 --> 66.96.16.50 netmask 0xffffffff inet6 fe80::219:b9ff:fef9:b9e7%ng11 prefixlen 64 scopeid 0x17 nd6 options=21 (this removes IFDISABLED ... which shouldn't matter for ipv4, but I'm working on ipv6 anyways). The 2nd thing I've done is compare the 9.1 and 10.0 versions if ng_iface.c. The main difference appears to be a rewrite of ng_iface_output() to make it's third argument a constant. I'm unsure reading this if the handling of the dst->sa_family could be causing my problem. -- You are receiving this mail because: You are the assignee for the bug. From owner-freebsd-net@FreeBSD.ORG Tue Jul 22 08:02:41 2014 Return-Path: Delivered-To: freebsd-net@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id DEB29F10; Tue, 22 Jul 2014 08:02:41 +0000 (UTC) Received: from mail-qg0-x22b.google.com (mail-qg0-x22b.google.com [IPv6:2607:f8b0:400d:c04::22b]) (using TLSv1 with cipher ECDHE-RSA-RC4-SHA (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 8FFFA28D1; Tue, 22 Jul 2014 08:02:41 +0000 (UTC) Received: by mail-qg0-f43.google.com with SMTP id a108so6648514qge.30 for ; Tue, 22 Jul 2014 01:02:40 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:sender:date:message-id:subject:from:to:content-type; bh=mPZNMhTTGxlm9J/V9d02a7W59eW07vX2g5HuQosyiMk=; b=sR7KVaSvwaTDgivftK9IUXkyQZOCXYLCdC66ta0s78MD8BntDwTkMAzhLhGF5VTtMI MJM/Yp/jhAPcpuBRYro94Z5plgTDdeRytdf/oi1nwRy/buOc3AKkqg5mgrcru4c9AzlD p2blMeeqxJu3ZQ0dzRUPPb4sDpALciKOq79O++97oIo159wK4WElUlGHx2v9fda8o6eX cU9fxuP/Oy3TuXYPyZUgzlGxkfh/b+Hx2fuXRpkDXWl/BZBAJkKIY+Z3zd+IQwTjJ7W/ p94V4sq9lzoEzQ6MUW2I7w4PRUcJo5wXhW1dOXiVmaaudGCCH2Se2K5SnwCQc9thSOLV Qseg== MIME-Version: 1.0 X-Received: by 10.224.156.145 with SMTP id x17mr34137783qaw.49.1406016160188; Tue, 22 Jul 2014 01:02:40 -0700 (PDT) Sender: adrian.chadd@gmail.com Received: by 10.224.1.6 with HTTP; Tue, 22 Jul 2014 01:02:40 -0700 (PDT) Date: Tue, 22 Jul 2014 01:02:40 -0700 X-Google-Sender-Auth: JvCqFbRyxP-Yi9KyslREGjFtYKU Message-ID: Subject: [rfc] UDP RSS transmit/receive message options; ip_output() inp flowid changes From: Adrian Chadd To: FreeBSD Net , "freebsd-arch@freebsd.org" Content-Type: text/plain; charset=UTF-8 X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 22 Jul 2014 08:02:42 -0000 Hi! I'd appreciate a review of this: http://people.freebsd.org/~adrian/rss/20140722-rss-udp-1.diff The overview: * Add a new flag to ip_output() to instruct it to not override the flowid with the inp cached value. Some forms of UDP transmit will break this. * Add new IP socket options to receive flowid and rss bucket information with the UDP datagram (via recvmsg.) * Add awareness in the send path of those same messages so the transmit path can avoid doing a hash calculation. The software hash calculations aren't done yet in the UDP transmit path - I'll work on those next. There are some comments which won't make it into the final committed patch. Bonus points if you get this far and see what they are. -a From owner-freebsd-net@FreeBSD.ORG Tue Jul 22 15:19:05 2014 Return-Path: Delivered-To: freebsd-net@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 27BAC585 for ; Tue, 22 Jul 2014 15:19:05 +0000 (UTC) Received: from mail-ie0-x230.google.com (mail-ie0-x230.google.com [IPv6:2607:f8b0:4001:c03::230]) (using TLSv1 with cipher ECDHE-RSA-RC4-SHA (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id EA20E212B for ; Tue, 22 Jul 2014 15:19:04 +0000 (UTC) Received: by mail-ie0-f176.google.com with SMTP id tr6so7917065ieb.7 for ; Tue, 22 Jul 2014 08:19:04 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=message-id:date:from:user-agent:mime-version:to:subject :content-type:content-transfer-encoding; bh=fepaNEJSsz8EgBfD7bUc+M9dM+AzrOll0RlF7eO+2FE=; b=eJUSJpcX6VdCItXqMPJdAsMqQSDAJH5muzsPlxSqUclgBEEAfUkBzjxc91QCKT0b+O Giekgii5jQsgsRllsE2GBfaFnJMvRk8Re/8UvXx1FipYAWIAw3BmLhngrHNmD2h2tqwZ SueeOCWUQKnBmmvLphGqJGYWEKVytfRnlQtncG8lmVwY6bG6HuMkYVgTpdyRsZF9QV7N XvUoXnOwZCnYuhRzavPzPn4UTrzi1BBiQY2Ien2DXuIYaNsjSWKDsMSVc0evSvVqBVJ2 fDQj2+8Vy0KkmRDflWstApOxgkmwvIqE3RSP3RabsjYA2Hmo9WJfx4GMax0FsVJv+jYO BHxQ== X-Received: by 10.43.63.15 with SMTP id xc15mr24564520icb.66.1406042344224; Tue, 22 Jul 2014 08:19:04 -0700 (PDT) Received: from [10.1.68.187] (gs-sv-1-49-ac1.gsfc.nasa.gov. [198.119.56.43]) by mx.google.com with ESMTPSA id n1sm49979975igp.1.2014.07.22.08.18.59 for (version=TLSv1 cipher=ECDHE-RSA-RC4-SHA bits=128/128); Tue, 22 Jul 2014 08:18:59 -0700 (PDT) Message-ID: <53CE80DD.9090109@gmail.com> Date: Tue, 22 Jul 2014 11:18:53 -0400 From: John Jasen User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:24.0) Gecko/20100101 Thunderbird/24.6.0 MIME-Version: 1.0 To: FreeBSD Net Subject: fastforward/routing: a 3 million packet-per-second system? X-Enigmail-Version: 1.5.2 Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: 7bit X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 22 Jul 2014 15:19:05 -0000 Feedback and/or tips and tricks more than welcome. Outstanding questions: Would increasing the number of processor cores help? Would a system where both processor QPI ports connect to each other mitigate QPI bottlenecks? Are there further performance optimizations I am missing? Server Description: The system in question is a Dell Poweredge R820, 16GB of RAM, and two Intel(R) Xeon(R) CPU E5-4610 0 @ 2.40GHz. Onboard, in a 16x PCIe slot, I have one Chelsio T-580-CR two-port 40GbE NIC, and in an 8x slot, another T-580-CR dual port. I am running FreeBSD 10.0-STABLE. BIOS tweaks: Hyperthreading (or Logical Processors) is turned off. Memory Node Interleaving is turned off, but did not appear to impact performance. /boot/loader.conf contents: #for CARP+PF testing carp_load="YES" #load cxgbe drivers. cxgbe_load="YES" #maxthreads appears to not exceed CPU. net.isr.maxthreads=12 #bindthreads may be indicated when using cpuset(1) on interrupts net.isr.bindthreads=1 #random guess based on googling net.isr.maxqlimit=60480 net.link.ifqmaxlen=90000 #discussions with cxgbe maintainer and list led me to trying this. Allows more interrupts #to be fixed to CPUs, which in some cases, improves interrupt balancing. hw.cxgbe.ntxq10g=16 hw.cxgbe.nrxq10g=16 /etc/sysctl.conf contents: #the following is also enabled by rc.conf gateway_enable. net.inet.ip.fastforwarding=1 #recommendations from BSD router project kern.random.sys.harvest.ethernet=0 kern.random.sys.harvest.point_to_point=0 kern.random.sys.harvest.interrupt=0 #probably should be removed, as cxgbe does not seem to affect/be affected by irq storm settings hw.intr_storm_threshold=25000000 #based on Calomel.Org performance suggestions. 4x40GbE, seemed reasonable to use 100GbE settings kern.ipc.maxsockbuf=1258291200 net.inet.tcp.recvbuf_max=1258291200 net.inet.tcp.sendbuf_max=1258291200 #attempting to play with ULE scheduler, making it serve packets versus netstat kern.sched.slice=1 kern.sched.interact=1 /etc/rc.conf contains: hostname="fbge1" #should remove, especially given below duplicate entry ifconfig_igb0="DHCP" sshd_enable="YES" #ntpd_enable="YES" # Set dumpdev to "AUTO" to enable crash dumps, "NO" to disable dumpdev="AUTO" # OpenBSD PF options to play with later. very bad for raw packet rates. #pf_enable="YES" #pflog_enable="YES" # enable packet forwarding # these enable forwarding and fastforwarding sysctls. inet6 does not have fastforward gateway_enable="YES" ipv6_gateway_enable="YES" # enable OpenBSD ftp-proxy # should comment out until actively playing with PF ftpproxy_enable="YES" #left in place, commented out from prior testing #ifconfig_mlxen1="inet 172.16.2.1 netmask 255.255.255.0 mtu 9000" #ifconfig_mlxen0="inet 172.16.1.1 netmask 255.255.255.0 mtu 9000" #ifconfig_mlxen3="inet 172.16.7.1 netmask 255.255.255.0 mtu 9000" #ifconfig_mlxen2="inet 172.16.8.1 netmask 255.255.255.0 mtu 9000" # -lro and -tso options added per mailing list suggestion from Bjoern A. Zeeb (bzeeb-lists at lists.zabbadoz.net) ifconfig_cxl0="inet 172.16.3.1 netmask 255.255.255.0 mtu 9000 -lro -tso up" ifconfig_cxl1="inet 172.16.4.1 netmask 255.255.255.0 mtu 9000 -lro -tso up" ifconfig_cxl2="inet 172.16.5.1 netmask 255.255.255.0 mtu 9000 -lro -tso up" ifconfig_cxl3="inet 172.16.6.1 netmask 255.255.255.0 mtu 9000 -lro -tso up" # aliases instead of reconfiguring test clients. See above commented out entries ifconfig_cxl0_alias0="172.16.7.1 netmask 255.255.255.0" ifconfig_cxl1_alias0="172.16.8.1 netmask 255.255.255.0" ifconfig_cxl2_alias0="172.16.1.1 netmask 255.255.255.0" ifconfig_cxl3_alias0="172.16.2.1 netmask 255.255.255.0" # for remote monitoring/admin of the test device ifconfig_igb0="inet 172.30.60.60 netmask 255.255.0.0" Additional configurations: cpuset-chelsio-6cpu-high # Original provided by Navdeep Parhar # takes vmstat -ai output into a list, and assigns interrupts in order to # the available CPU cores. # Modified: to assign only to the 'high CPUs', ie: on core1. # See: http://lists.freebsd.org/pipermail/freebsd-net/2014-July/039317.html #!/usr/local/bin/bash ncpu=12 irqlist=$(vmstat -ia | egrep 't4nex|t5nex|cxgbc' | cut -f1 -d: | cut -c4-) i=6 for irq in $irqlist; do cpuset -l $i -x $irq i=$((i+1)) [ $i -ge $ncpu ] && i=6 done Client Description: Two Intel(R) Xeon(R) CPU E5-2670 v2 @ 2.50GHz processors 64 GB ram Mellanox Technologies MT27500 Family [ConnectX-3] Centos 6.4 with updates iperf3 installed from yum repositories: iperf3-3.0.3-3.el6.x86_64 Test setup: I've found about 3 streams between Centos clients is about the best way to get the most out of them. Above certain points, the -b flag does not change results. -N is an artifact from using TCP -l is needed, as -M doesn't work for UDP. I usually use launch scripts similar to the following: for i in `seq 41 60`; do ssh loader$i "export TIME=120; export STREAMS=1; export PORT=52$i; export PKT=64; export RATE=2000m; /root/iperf-test-8port-udp" & done The scripts execute the following on each host. #!/bin/bash PORT1=$PORT PORT2=$(($PORT+1000)) PORT3=$(($PORT+2000)) iperf3 -c loader41-40gbe -u -b 10000m -i 0 -N -l $PKT -t$TIME -P$STREAMS -p$PORT1 & iperf3 -c loader42-40gbe -u -b 10000m -i 0 -N -l $PKT -t$TIME -P$STREAMS -p$PORT1 & iperf3 -c loader43-40gbe -u -b 10000m -i 0 -N -l $PKT -t$TIME -P$STREAMS -p$PORT1 & ... (through all clients and all three ports) ... iperf3 -c loader60-40gbe -u -b 10000m -i 0 -N -l $PKT -t$TIME -P$STREAMS -p$PORT3 & Results: Summarized, netstat -w 1 -q 240 -bd, run through: cat test4-tuning | egrep -v {'packets | input '} | awk '{ipackets+=$1} {idrops+=$3} {opackets+=$5} {odrops+=$9} END {print "input " ipackets/NR, "idrops " idrops/NR, "opackets " opackets/NR, "odrops " odrops/NR}' input 1.10662e+07 idrops 8.01783e+06 opackets 3.04516e+06 odrops 3152.4 Snapshot of raw output: input (Total) output packets errs idrops bytes packets errs bytes colls drops 11189148 0 7462453 1230805216 3725006 0 409750710 0 799 10527505 0 6746901 1158024978 3779096 0 415700708 0 127 10606163 0 6850760 1166676673 3751780 0 412695761 0 1535 10749324 0 7132014 1182425799 3617558 0 397930956 0 5972 10695667 0 7022717 1176521907 3669342 0 403627236 0 1461 10441173 0 6762134 1148528662 3675048 0 404255540 0 6021 10683773 0 7005635 1175215014 3676962 0 404465671 0 2606 10869859 0 7208696 1195683372 3658432 0 402427698 0 979 11948989 0 8310926 1314387881 3633773 0 399714986 0 725 12426195 0 8864415 1366877194 3562311 0 391853156 0 2762 13006059 0 9432389 1430661751 3570067 0 392706552 0 5158 12822243 0 9098871 1410443600 3715177 0 408668500 0 4064 13317864 0 9683602 1464961374 3632156 0 399536131 0 3684 13701905 0 10182562 1507207982 3523101 0 387540859 0 8690 13820227 0 10244870 1520221820 3562038 0 391823322 0 2426 14437060 0 10955483 1588073033 3480105 0 382810557 0 2619 14518471 0 11119573 1597028105 3397439 0 373717355 0 5691 14890287 0 11675003 1637926521 3199812 0 351978304 0 11007 14923610 0 11749091 1641594441 3171436 0 348857468 0 7389 14738704 0 11609730 1621254991 3117715 0 342948394 0 2597 14753975 0 11549735 1622935026 3207393 0 352812846 0 4798 From owner-freebsd-net@FreeBSD.ORG Tue Jul 22 16:53:18 2014 Return-Path: Delivered-To: freebsd-net@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 5229A167; Tue, 22 Jul 2014 16:53:18 +0000 (UTC) Received: from mail-we0-x230.google.com (mail-we0-x230.google.com [IPv6:2a00:1450:400c:c03::230]) (using TLSv1 with cipher ECDHE-RSA-RC4-SHA (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 9ADE42C0D; Tue, 22 Jul 2014 16:53:17 +0000 (UTC) Received: by mail-we0-f176.google.com with SMTP id q58so9392430wes.35 for ; Tue, 22 Jul 2014 09:53:13 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:sender:in-reply-to:references:date:message-id:subject :from:to:cc:content-type; bh=iyrqJZVAU9U17MIR/6LlCHMPJsMQ4Z4BzdfC0cjKC7M=; b=TJgUSAkILXgDOwFRt8JWfO9tmaKP29PfElSR/Cfa8xff7lx+8yGDU2greg6SINd+Ve 5UkpRg3QOW11WlZumD38M70hsWX2+1nakVTqtUZ4tJFuYyaykvf11q4DdfmV04UcFJ/X h2BqTb1U/yGBtBMHF3/08qQNSCcwceqh1GrDjrsw7z84iu3+I2OOiWBiwjfK3I4fH9vp ksusNpZmJGvlFbAjUZxxZ/jc/7RI7yBBFoNdnWhTtLr+AB2R1BtXgUSyOtMOHdo4n8sJ yB1PViL+HZQSB2KNIclGjjApRClXfzqhR9lZV/diuXby4hv3tBSGTrj7Jca0oO0156RT m6KA== MIME-Version: 1.0 X-Received: by 10.194.133.1 with SMTP id oy1mr35760051wjb.87.1406047993441; Tue, 22 Jul 2014 09:53:13 -0700 (PDT) Sender: jinmei.tatuya@gmail.com Received: by 10.194.108.166 with HTTP; Tue, 22 Jul 2014 09:53:13 -0700 (PDT) In-Reply-To: <20140720090410.GA7990@mx.elandsys.com> References: <20140720090410.GA7990@mx.elandsys.com> Date: Tue, 22 Jul 2014 09:53:13 -0700 X-Google-Sender-Auth: EeCFxD1LgnwH9Zpmv6k6wV8-bJE Message-ID: Subject: Re: IPv6 nodeinfo default behaviour From: =?UTF-8?B?56We5piO6YGU5ZOJ?= To: Loganaden Velvindron Content-Type: text/plain; charset=UTF-8 Cc: FreeBSD Net , bz@freebsd.org, gnn@freebsd.org X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 22 Jul 2014 16:53:18 -0000 At Sun, 20 Jul 2014 02:04:10 -0700, Loganaden Velvindron wrote: > Security Considerations > > This protocol shares the security issues of ICMPv6 that are > documented in the "Security Considerations" section of [5]. > > This protocol has the potential of revealing information useful to a > would-be attacker. An implementation of this protocol MUST have a > default configuration that refuses to answer queries from global- > scope [3] addresses. > > I suggest that we switch to 0 by default to be more RFC compliant. Are you referring to the value of '(V_)icmp6_nodeinfo'? If so, and to be compliant with the above MUST of the RFC, it doesn't seem to have to be 0; it only has to have the ICMP6_NODEINFO_GLOBALOK bit cleared: /* * Validate IPv6 source address. * The default configuration MUST be to refuse answering queries from * global-scope addresses according to RFC4602. * Notes: * - it's not very clear what "refuse" means; this implementation * simply drops it. * - it's not very easy to identify global-scope (unicast) addresses * since there are many prefixes for them. It should be safer * and in practice sufficient to check "all" but loopback and * link-local (note that site-local unicast was deprecated and * ULA is defined as global scope-wise) */ if ((V_icmp6_nodeinfo & ICMP6_NODEINFO_GLOBALOK) == 0 && !IN6_IS_ADDR_LOOPBACK(&ip6->ip6_src) && !IN6_IS_ADDR_LINKLOCAL(&ip6->ip6_src)) goto bad; and the default value already seems to meet this condition: VNET_DEFINE(int, icmp6_nodeinfo) = (ICMP6_NODEINFO_FQDNOK|ICMP6_NODEINFO_NODEADDROK); -- JINMEI, Tatuya From owner-freebsd-net@FreeBSD.ORG Tue Jul 22 17:01:53 2014 Return-Path: Delivered-To: freebsd-net@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id C9DE83F9; Tue, 22 Jul 2014 17:01:53 +0000 (UTC) Received: from mx.ipv6.elandsys.com (mx.ipv6.elandsys.com [IPv6:2001:470:f329:1::1]) by mx1.freebsd.org (Postfix) with ESMTP id 7ADF62CD9; Tue, 22 Jul 2014 17:01:53 +0000 (UTC) Received: from mx.elandsys.com (IDENT:logan@localhost [127.0.0.1]) by mx.elandsys.com (8.14.5/8.14.5) with ESMTP id s6MH1o9L005568 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-SHA bits=256 verify=NO); Tue, 22 Jul 2014 10:01:50 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=simple/simple; d=opendkim.org; s=mail2010; t=1406048511; x=1406134911; bh=cSmyrbB1jGz/kfDYG5r75mG7SbI/h62rhaBjwtcNS/o=; h=Date:From:To:Cc:Subject:References:In-Reply-To; b=v3qVoYUX4bIzw3dSD2wgYbfWjwTfcfayxzfMiL0fyx5dJxid/yXhUM3EMBh9zV+9C SKIE3NwwMQb74fVWEbOombmOzLfgrdZ3KYuRTeRzHfOSWBF8gNbUulOaNsRcsPk/Ma PKnSizp2z3viMYB3TbxdNxa1nPirnZl9G9qyYitE= DKIM-Signature: v=1; a=rsa-sha256; c=simple/simple; d=elandsys.com; s=mail; t=1406048511; x=1406134911; i=@elandsys.com; bh=cSmyrbB1jGz/kfDYG5r75mG7SbI/h62rhaBjwtcNS/o=; h=Date:From:To:Cc:Subject:References:In-Reply-To; b=Br8PeVBlCuFtxJsGF33wq0jvzRDqFEtE3FcP5//dqRcl/UqknY/Uve0Fujmq+Osb7 IuL9PhP7jde2IAj+4dnSOUnIFwPeDv0wpuHnGzPxRkGwWykgcP3il1c1Tx1PgQ+BRM gH4UtpC8N0mZEg3LXZnLCpTKIt1nARPzNSQ/BGlU= Received: (from logan@localhost) by mx.elandsys.com (8.14.5/8.14.5/Submit) id s6MH1oFo016517; Tue, 22 Jul 2014 10:01:50 -0700 (PDT) X-Authentication-Warning: mx.elandsys.com: logan set sender to logan@elandsys.com using -f Date: Tue, 22 Jul 2014 10:01:50 -0700 From: Loganaden Velvindron To: Jinmei Subject: Re: IPv6 nodeinfo default behaviour Message-ID: <20140722170150.GA971@mx.elandsys.com> References: <20140720090410.GA7990@mx.elandsys.com> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: User-Agent: Mutt/1.5.21 (2010-09-15) Cc: freebsd-net@freebsd.org, bz@freebsd.org, gnn@freebsd.org X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 22 Jul 2014 17:01:54 -0000 On Tue, Jul 22, 2014 at 09:53:13AM -0700, ???? wrote: > At Sun, 20 Jul 2014 02:04:10 -0700, > Loganaden Velvindron wrote: > > > Security Considerations > > > > This protocol shares the security issues of ICMPv6 that are > > documented in the "Security Considerations" section of [5]. > > > > This protocol has the potential of revealing information useful to a > > would-be attacker. An implementation of this protocol MUST have a > > default configuration that refuses to answer queries from global- > > scope [3] addresses. > > > > I suggest that we switch to 0 by default to be more RFC compliant. > > Are you referring to the value of '(V_)icmp6_nodeinfo'? I'm referring to the sysctl: net.inet6.icmp6.nodeinfo. In FreeBSD it's 3 by default. OpenBSD switched it to 0, then later removed it completely. I think that it's sensible to turn it to 0 by default, unless you need it. > > If so, and to be compliant with the above MUST of the RFC, it doesn't > seem to have to be 0; it only has to have the ICMP6_NODEINFO_GLOBALOK > bit cleared: > > /* > * Validate IPv6 source address. > * The default configuration MUST be to refuse answering queries from > * global-scope addresses according to RFC4602. > * Notes: > * - it's not very clear what "refuse" means; this implementation > * simply drops it. > * - it's not very easy to identify global-scope (unicast) addresses > * since there are many prefixes for them. It should be safer > * and in practice sufficient to check "all" but loopback and > * link-local (note that site-local unicast was deprecated and > * ULA is defined as global scope-wise) > */ > if ((V_icmp6_nodeinfo & ICMP6_NODEINFO_GLOBALOK) == 0 && > !IN6_IS_ADDR_LOOPBACK(&ip6->ip6_src) && > !IN6_IS_ADDR_LINKLOCAL(&ip6->ip6_src)) > goto bad; > > and the default value already seems to meet this condition: > > VNET_DEFINE(int, icmp6_nodeinfo) = > (ICMP6_NODEINFO_FQDNOK|ICMP6_NODEINFO_NODEADDROK); > > -- > JINMEI, Tatuya > _______________________________________________ > freebsd-net@freebsd.org mailing list > http://lists.freebsd.org/mailman/listinfo/freebsd-net > To unsubscribe, send any mail to "freebsd-net-unsubscribe@freebsd.org" From owner-freebsd-net@FreeBSD.ORG Tue Jul 22 17:41:25 2014 Return-Path: Delivered-To: freebsd-net@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 84057B9 for ; Tue, 22 Jul 2014 17:41:25 +0000 (UTC) Received: from h2.funkthat.com (gate2.funkthat.com [208.87.223.18]) (using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits)) (Client CN "funkthat.com", Issuer "funkthat.com" (not verified)) by mx1.freebsd.org (Postfix) with ESMTPS id 55DA82017 for ; Tue, 22 Jul 2014 17:41:24 +0000 (UTC) Received: from h2.funkthat.com (localhost [127.0.0.1]) by h2.funkthat.com (8.14.3/8.14.3) with ESMTP id s6MHfIcQ061541 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-SHA bits=256 verify=NO); Tue, 22 Jul 2014 10:41:18 -0700 (PDT) (envelope-from jmg@h2.funkthat.com) Received: (from jmg@localhost) by h2.funkthat.com (8.14.3/8.14.3/Submit) id s6MHfHhX061540; Tue, 22 Jul 2014 10:41:17 -0700 (PDT) (envelope-from jmg) Date: Tue, 22 Jul 2014 10:41:17 -0700 From: John-Mark Gurney To: John Jasen Subject: Re: fastforward/routing: a 3 million packet-per-second system? Message-ID: <20140722174117.GC43962@funkthat.com> Mail-Followup-To: John Jasen , FreeBSD Net References: <53CE80DD.9090109@gmail.com> Mime-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <53CE80DD.9090109@gmail.com> User-Agent: Mutt/1.4.2.3i X-Operating-System: FreeBSD 7.2-RELEASE i386 X-PGP-Fingerprint: 54BA 873B 6515 3F10 9E88 9322 9CB1 8F74 6D3F A396 X-Files: The truth is out there X-URL: http://resnet.uoregon.edu/~gurney_j/ X-Resume: http://resnet.uoregon.edu/~gurney_j/resume.html X-TipJar: bitcoin:13Qmb6AeTgQecazTWph4XasEsP7nGRbAPE X-to-the-FBI-CIA-and-NSA: HI! HOW YA DOIN? can i haz chizburger? X-Greylist: Sender IP whitelisted, not delayed by milter-greylist-4.2.2 (h2.funkthat.com [127.0.0.1]); Tue, 22 Jul 2014 10:41:18 -0700 (PDT) Cc: FreeBSD Net X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 22 Jul 2014 17:41:25 -0000 John Jasen wrote this message on Tue, Jul 22, 2014 at 11:18 -0400: > Feedback and/or tips and tricks more than welcome. You should look at netmap if you really want high PPS routing... >From the netmap paper: netmap has been implemented in FreeBSD and Linux for several 1 and 10 Gbit/s network adapters. In our pro- totype, a single core running at 900 MHz can send or receive 14.88 Mpps (the peak packet rate on 10 Gbit/s links) http://info.iet.unipi.it/~luigi/papers/20120503-netmap-atc12.pdf -- John-Mark Gurney Voice: +1 415 225 5579 "All that I will do, has been done, All that I have, has not." From owner-freebsd-net@FreeBSD.ORG Tue Jul 22 18:07:08 2014 Return-Path: Delivered-To: freebsd-net@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 7574F66B for ; Tue, 22 Jul 2014 18:07:08 +0000 (UTC) Received: from mail-qa0-x22a.google.com (mail-qa0-x22a.google.com [IPv6:2607:f8b0:400d:c00::22a]) (using TLSv1 with cipher ECDHE-RSA-RC4-SHA (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 368BD22E4 for ; Tue, 22 Jul 2014 18:07:08 +0000 (UTC) Received: by mail-qa0-f42.google.com with SMTP id j15so22821qaq.15 for ; Tue, 22 Jul 2014 11:07:07 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:sender:in-reply-to:references:date:message-id:subject :from:to:cc:content-type; bh=xl/CJ0LnuNhFvef3XQHJQ29AMGeoKuh1+yNC3YYwaf4=; b=Jnf+QvBw2ZPH3ClPGKV+KR6Scc5mRwE/th/lN2ZXSkzS+5KflH78jQD/LHxFJq7gDP x7CNgo3TPUKPU4P/RsyjgZu00bjAUw2Fl/P2uoktO22NfNh1y2RQINJfoOxAPGv8CBwm LWOZ/kry0Yvv1ecSRehoPSQN9mHnaGJ2/YAGOE9eX6JPdnITFZ10mp0KF3VUW8vgVCpt ctTKKx0ULep5FN2XfSsh6oDgMI9HfiX7AQijcrGPV6HoaJ5gEqyG89KEWOwG2DEW0KYp 0QhAQyoIWkfNml+qtPB1dIBPQRX+O8vzEGFXb874wwWebhAd+zDD+kx4nXpnIKZbTpWj ntHg== MIME-Version: 1.0 X-Received: by 10.224.171.197 with SMTP id i5mr59861178qaz.55.1406052427375; Tue, 22 Jul 2014 11:07:07 -0700 (PDT) Sender: adrian.chadd@gmail.com Received: by 10.224.1.6 with HTTP; Tue, 22 Jul 2014 11:07:07 -0700 (PDT) In-Reply-To: <53CE80DD.9090109@gmail.com> References: <53CE80DD.9090109@gmail.com> Date: Tue, 22 Jul 2014 11:07:07 -0700 X-Google-Sender-Auth: E7SYjpZqAxwBTuVBUC6dReQcWE8 Message-ID: Subject: Re: fastforward/routing: a 3 million packet-per-second system? From: Adrian Chadd To: John Jasen Content-Type: text/plain; charset=UTF-8 Cc: FreeBSD Net X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 22 Jul 2014 18:07:08 -0000 Hi! Well, what's missing is some dtrace/pmc/lockdebugging investigations into the system to see where it's currently maxing out at. I wonder if you're seeing contention on the transmit paths as drivers queue frames from one set of driver threads/queues to another potentially completely different set of driver transmit threads/queues. -a On 22 July 2014 08:18, John Jasen wrote: > Feedback and/or tips and tricks more than welcome. > > Outstanding questions: > > Would increasing the number of processor cores help? > > Would a system where both processor QPI ports connect to each other > mitigate QPI bottlenecks? > > Are there further performance optimizations I am missing? > > Server Description: > > The system in question is a Dell Poweredge R820, 16GB of RAM, and two > Intel(R) Xeon(R) CPU E5-4610 0 @ 2.40GHz. > > Onboard, in a 16x PCIe slot, I have one Chelsio T-580-CR two-port 40GbE > NIC, and in an 8x slot, another T-580-CR dual port. > > I am running FreeBSD 10.0-STABLE. > > BIOS tweaks: > > Hyperthreading (or Logical Processors) is turned off. > Memory Node Interleaving is turned off, but did not appear to impact > performance. > > /boot/loader.conf contents: > #for CARP+PF testing > carp_load="YES" > #load cxgbe drivers. > cxgbe_load="YES" > #maxthreads appears to not exceed CPU. > net.isr.maxthreads=12 > #bindthreads may be indicated when using cpuset(1) on interrupts > net.isr.bindthreads=1 > #random guess based on googling > net.isr.maxqlimit=60480 > net.link.ifqmaxlen=90000 > #discussions with cxgbe maintainer and list led me to trying this. > Allows more interrupts > #to be fixed to CPUs, which in some cases, improves interrupt balancing. > hw.cxgbe.ntxq10g=16 > hw.cxgbe.nrxq10g=16 > > /etc/sysctl.conf contents: > > #the following is also enabled by rc.conf gateway_enable. > net.inet.ip.fastforwarding=1 > #recommendations from BSD router project > kern.random.sys.harvest.ethernet=0 > kern.random.sys.harvest.point_to_point=0 > kern.random.sys.harvest.interrupt=0 > #probably should be removed, as cxgbe does not seem to affect/be > affected by irq storm settings > hw.intr_storm_threshold=25000000 > #based on Calomel.Org performance suggestions. 4x40GbE, seemed > reasonable to use 100GbE settings > kern.ipc.maxsockbuf=1258291200 > net.inet.tcp.recvbuf_max=1258291200 > net.inet.tcp.sendbuf_max=1258291200 > #attempting to play with ULE scheduler, making it serve packets versus > netstat > kern.sched.slice=1 > kern.sched.interact=1 > > /etc/rc.conf contains: > > hostname="fbge1" > #should remove, especially given below duplicate entry > ifconfig_igb0="DHCP" > sshd_enable="YES" > #ntpd_enable="YES" > # Set dumpdev to "AUTO" to enable crash dumps, "NO" to disable > dumpdev="AUTO" > # OpenBSD PF options to play with later. very bad for raw packet rates. > #pf_enable="YES" > #pflog_enable="YES" > # enable packet forwarding > # these enable forwarding and fastforwarding sysctls. inet6 does not > have fastforward > gateway_enable="YES" > ipv6_gateway_enable="YES" > # enable OpenBSD ftp-proxy > # should comment out until actively playing with PF > ftpproxy_enable="YES" > #left in place, commented out from prior testing > #ifconfig_mlxen1="inet 172.16.2.1 netmask 255.255.255.0 mtu 9000" > #ifconfig_mlxen0="inet 172.16.1.1 netmask 255.255.255.0 mtu 9000" > #ifconfig_mlxen3="inet 172.16.7.1 netmask 255.255.255.0 mtu 9000" > #ifconfig_mlxen2="inet 172.16.8.1 netmask 255.255.255.0 mtu 9000" > # -lro and -tso options added per mailing list suggestion from Bjoern A. > Zeeb (bzeeb-lists at lists.zabbadoz.net) > ifconfig_cxl0="inet 172.16.3.1 netmask 255.255.255.0 mtu 9000 -lro -tso up" > ifconfig_cxl1="inet 172.16.4.1 netmask 255.255.255.0 mtu 9000 -lro -tso up" > ifconfig_cxl2="inet 172.16.5.1 netmask 255.255.255.0 mtu 9000 -lro -tso up" > ifconfig_cxl3="inet 172.16.6.1 netmask 255.255.255.0 mtu 9000 -lro -tso up" > # aliases instead of reconfiguring test clients. See above commented out > entries > ifconfig_cxl0_alias0="172.16.7.1 netmask 255.255.255.0" > ifconfig_cxl1_alias0="172.16.8.1 netmask 255.255.255.0" > ifconfig_cxl2_alias0="172.16.1.1 netmask 255.255.255.0" > ifconfig_cxl3_alias0="172.16.2.1 netmask 255.255.255.0" > # for remote monitoring/admin of the test device > ifconfig_igb0="inet 172.30.60.60 netmask 255.255.0.0" > > Additional configurations: > cpuset-chelsio-6cpu-high > # Original provided by Navdeep Parhar > # takes vmstat -ai output into a list, and assigns interrupts in order to > # the available CPU cores. > # Modified: to assign only to the 'high CPUs', ie: on core1. > # See: http://lists.freebsd.org/pipermail/freebsd-net/2014-July/039317.html > #!/usr/local/bin/bash > ncpu=12 > irqlist=$(vmstat -ia | egrep 't4nex|t5nex|cxgbc' | cut -f1 -d: | cut -c4-) > i=6 > for irq in $irqlist; do > cpuset -l $i -x $irq > i=$((i+1)) > [ $i -ge $ncpu ] && i=6 > done > > Client Description: > > Two Intel(R) Xeon(R) CPU E5-2670 v2 @ 2.50GHz processors > 64 GB ram > Mellanox Technologies MT27500 Family [ConnectX-3] > Centos 6.4 with updates > iperf3 installed from yum repositories: iperf3-3.0.3-3.el6.x86_64 > > Test setup: > > I've found about 3 streams between Centos clients is about the best way > to get the most out of them. > Above certain points, the -b flag does not change results. > -N is an artifact from using TCP > -l is needed, as -M doesn't work for UDP. > > I usually use launch scripts similar to the following: > > for i in `seq 41 60`; do ssh loader$i "export TIME=120; export > STREAMS=1; export PORT=52$i; export PKT=64; export RATE=2000m; > /root/iperf-test-8port-udp" & done > > The scripts execute the following on each host. > > #!/bin/bash > PORT1=$PORT > PORT2=$(($PORT+1000)) > PORT3=$(($PORT+2000)) > iperf3 -c loader41-40gbe -u -b 10000m -i 0 -N -l $PKT -t$TIME > -P$STREAMS -p$PORT1 & > iperf3 -c loader42-40gbe -u -b 10000m -i 0 -N -l $PKT -t$TIME > -P$STREAMS -p$PORT1 & > iperf3 -c loader43-40gbe -u -b 10000m -i 0 -N -l $PKT -t$TIME > -P$STREAMS -p$PORT1 & > ... (through all clients and all three ports) ... > iperf3 -c loader60-40gbe -u -b 10000m -i 0 -N -l $PKT -t$TIME > -P$STREAMS -p$PORT3 & > > > Results: > > Summarized, netstat -w 1 -q 240 -bd, run through: > cat test4-tuning | egrep -v {'packets | input '} | awk '{ipackets+=$1} > {idrops+=$3} {opackets+=$5} {odrops+=$9} END {print "input " > ipackets/NR, "idrops " idrops/NR, "opackets " opackets/NR, "odrops " > odrops/NR}' > > input 1.10662e+07 idrops 8.01783e+06 opackets 3.04516e+06 odrops 3152.4 > > Snapshot of raw output: > > input (Total) output > packets errs idrops bytes packets errs bytes colls drops > 11189148 0 7462453 1230805216 3725006 0 409750710 0 799 > 10527505 0 6746901 1158024978 3779096 0 415700708 0 127 > 10606163 0 6850760 1166676673 3751780 0 412695761 0 1535 > 10749324 0 7132014 1182425799 3617558 0 397930956 0 5972 > 10695667 0 7022717 1176521907 3669342 0 403627236 0 1461 > 10441173 0 6762134 1148528662 3675048 0 404255540 0 6021 > 10683773 0 7005635 1175215014 3676962 0 404465671 0 2606 > 10869859 0 7208696 1195683372 3658432 0 402427698 0 979 > 11948989 0 8310926 1314387881 3633773 0 399714986 0 725 > 12426195 0 8864415 1366877194 3562311 0 391853156 0 2762 > 13006059 0 9432389 1430661751 3570067 0 392706552 0 5158 > 12822243 0 9098871 1410443600 3715177 0 408668500 0 4064 > 13317864 0 9683602 1464961374 3632156 0 399536131 0 3684 > 13701905 0 10182562 1507207982 3523101 0 387540859 0 > 8690 > 13820227 0 10244870 1520221820 3562038 0 391823322 0 > 2426 > 14437060 0 10955483 1588073033 3480105 0 382810557 0 > 2619 > 14518471 0 11119573 1597028105 3397439 0 373717355 0 > 5691 > 14890287 0 11675003 1637926521 3199812 0 351978304 0 > 11007 > 14923610 0 11749091 1641594441 3171436 0 348857468 0 > 7389 > 14738704 0 11609730 1621254991 3117715 0 342948394 0 > 2597 > 14753975 0 11549735 1622935026 3207393 0 352812846 0 > 4798 > > > > > > _______________________________________________ > freebsd-net@freebsd.org mailing list > http://lists.freebsd.org/mailman/listinfo/freebsd-net > To unsubscribe, send any mail to "freebsd-net-unsubscribe@freebsd.org" From owner-freebsd-net@FreeBSD.ORG Tue Jul 22 18:25:43 2014 Return-Path: Delivered-To: freebsd-net@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 127A1C59; Tue, 22 Jul 2014 18:25:43 +0000 (UTC) Received: from mail-wi0-x231.google.com (mail-wi0-x231.google.com [IPv6:2a00:1450:400c:c05::231]) (using TLSv1 with cipher ECDHE-RSA-RC4-SHA (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 57C3224ED; Tue, 22 Jul 2014 18:25:42 +0000 (UTC) Received: by mail-wi0-f177.google.com with SMTP id ho1so908268wib.16 for ; Tue, 22 Jul 2014 11:25:39 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:sender:in-reply-to:references:date:message-id:subject :from:to:cc:content-type; bh=mMzqum3sQh1daQ/RX+P+8IBCpDR1JwHfrCLZm6fKLJI=; b=RcPqf19eaXvbDYWiX5B8EAUUxlbP9ybJKOQkH533+QKQx5PybqZFannOVpvMXdUmkN 2sT14CDfqGQE8jYGFNmnLBfHrdM0K5P7vFGA7UnvAN3nfYr/Pe3eSF67Xr+O26mLf62c fdqZh0J9kIfBDoaDm/WM7RIE1Fsa5jQ1itQv8zw3P0RZL0mRaPgUp+7jWz3en5J09Ut8 pZG4kL74+6bKCNHttimict0fFm5ENd0K+xZq7C15PrjzhkFA/kn3LChlTUBkVJvDw+au IB7vC3Vbeko/rUIeYDM6qMfJ6NziF7eHfHpMWYirMf91Z4/YM+kduat3Blp34dpjyuu6 rsdA== MIME-Version: 1.0 X-Received: by 10.194.63.228 with SMTP id j4mr37150106wjs.7.1406053537979; Tue, 22 Jul 2014 11:25:37 -0700 (PDT) Sender: jinmei.tatuya@gmail.com Received: by 10.194.108.166 with HTTP; Tue, 22 Jul 2014 11:25:37 -0700 (PDT) In-Reply-To: <20140722170150.GA971@mx.elandsys.com> References: <20140720090410.GA7990@mx.elandsys.com> <20140722170150.GA971@mx.elandsys.com> Date: Tue, 22 Jul 2014 11:25:37 -0700 X-Google-Sender-Auth: hsPksJYTLMziCmxT9KtuGWXFrBI Message-ID: Subject: Re: IPv6 nodeinfo default behaviour From: =?UTF-8?B?56We5piO6YGU5ZOJ?= To: Loganaden Velvindron Content-Type: text/plain; charset=UTF-8 Cc: FreeBSD Net , bz@freebsd.org, gnn@freebsd.org X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 22 Jul 2014 18:25:43 -0000 At Tue, 22 Jul 2014 10:01:50 -0700, Loganaden Velvindron wrote: > > > Security Considerations > > > > > > This protocol has the potential of revealing information useful to a > > > would-be attacker. An implementation of this protocol MUST have a > > > default configuration that refuses to answer queries from global- > > > scope [3] addresses. > > > > > > I suggest that we switch to 0 by default to be more RFC compliant. > > > > Are you referring to the value of '(V_)icmp6_nodeinfo'? > > I'm referring to the sysctl: > > net.inet6.icmp6.nodeinfo. These two are essentially the same in this context: this sysctl is an interface to (V_)icmp6_nodeinfo. This variable is set to ICMP6_NODEINFO_FQDNOK|ICMP6_NODEINFO_NODEADDROK by default, and since ICMP6_NODEINFO_FQDNOK and ICMP6_NODEINFO_NODEADDROK are 0x1 and 0x2, respectively, the default value of the sysctl variable is 3 by default. In your original message, you said > > > I suggest that we switch to 0 by default to be more RFC compliant. and I tried to point out that it didn't make sense because "to be more RFC compliant" it doesn't have to switch to 0, it just needs to have the ICMP6_NODEINFO_GLOBALOK flag (0x8) cleared, and the current default meets the condition already. Now you're changing the reason: > I think that it's sensible to turn it to 0 by default, unless you need > it. Unlike being "RFC compliant", whether something is "sensible" is usually subjective, and different people may have different opinions. Personally, I often find "ping6 -w" quite useful for debugging purposes, and I think limiting its use to link-local by default gives a reasonable level of defense (and, disabling it by default would reduce the usability pretty much). So I'd rather prefer keeping the current default, but, again, other people may have a different preference. -- JINMEI, Tatuya From owner-freebsd-net@FreeBSD.ORG Tue Jul 22 18:36:52 2014 Return-Path: Delivered-To: freebsd-net@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 660DBF4A for ; Tue, 22 Jul 2014 18:36:52 +0000 (UTC) Received: from mail-ie0-x236.google.com (mail-ie0-x236.google.com [IPv6:2607:f8b0:4001:c03::236]) (using TLSv1 with cipher ECDHE-RSA-RC4-SHA (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 303F625F1 for ; Tue, 22 Jul 2014 18:36:52 +0000 (UTC) Received: by mail-ie0-f182.google.com with SMTP id y20so44968ier.41 for ; Tue, 22 Jul 2014 11:36:51 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=message-id:date:from:user-agent:mime-version:to:subject:references :in-reply-to:content-type:content-transfer-encoding; bh=gj5mLE2cJnxahRQsSBICCot/VGeuHBFlmnGaBQ5Fnaw=; b=lscaBEcpqq2QEmEEcqKzBASo+/ytAlSAMGqZu/zKduc+DVXZzVrEebw9/xy6VrnKXM 69PVY9SNNtyWb3TiOmMUFA74kz+jnBqNhGXpnsknABk8Xjr0QdJiW272XF1BCl+d+4Wv justQp3WsAZxON8ozbMKZ+qDlCGEAvnvLo+DRwujZWh3IUaagPG2uEc6h+C4Qgkc3dyC qYY4hX4RkWh/itb1LqGaZ97G5VFLBXfnZSFDA2WhRR1AluS5LxQnBtUTiOKHL2HZCect x7cSM61uJSLrS0j81c7DZyqB5QSrJpgECpVnt+GRUNPNWWSe0U39cQ8Vex9ahgQgHLX/ v6Yg== X-Received: by 10.43.104.132 with SMTP id dm4mr25336440icc.56.1406054211675; Tue, 22 Jul 2014 11:36:51 -0700 (PDT) Received: from [10.1.68.187] (gs-sv-1-49-ac1.gsfc.nasa.gov. [198.119.56.43]) by mx.google.com with ESMTPSA id hu5sm4533651igb.16.2014.07.22.11.36.49 for (version=TLSv1 cipher=ECDHE-RSA-RC4-SHA bits=128/128); Tue, 22 Jul 2014 11:36:50 -0700 (PDT) Message-ID: <53CEAF40.7030406@gmail.com> Date: Tue, 22 Jul 2014 14:36:48 -0400 From: John Jasen User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:24.0) Gecko/20100101 Thunderbird/24.6.0 MIME-Version: 1.0 To: FreeBSD Net Subject: Re: fastforward/routing: a 3 million packet-per-second system? References: <53CE80DD.9090109@gmail.com> <20140722174117.GC43962@funkthat.com> In-Reply-To: <20140722174117.GC43962@funkthat.com> X-Enigmail-Version: 1.5.2 Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: 7bit X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 22 Jul 2014 18:36:52 -0000 On 07/22/2014 01:41 PM, John-Mark Gurney wrote: > John Jasen wrote this message on Tue, Jul 22, 2014 at 11:18 -0400: >> Feedback and/or tips and tricks more than welcome. > You should look at netmap if you really want high PPS routing... Originally, I assumed an interface supporting netmap was required, but the manpage disabuses me of this. Besides, I think the Chelsio cards got netmap recently. I presume either the use of bridge in tools/netmap, or vale-ctl in the same location would start providing me sufficient clue on this? Unfortunately, both seem to be pretty short on verbosity ... More clue is always welcome! From owner-freebsd-net@FreeBSD.ORG Tue Jul 22 19:23:48 2014 Return-Path: Delivered-To: freebsd-net@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 469B9500 for ; Tue, 22 Jul 2014 19:23:48 +0000 (UTC) Received: from mail-qg0-x236.google.com (mail-qg0-x236.google.com [IPv6:2607:f8b0:400d:c04::236]) (using TLSv1 with cipher ECDHE-RSA-RC4-SHA (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 05FAE2B13 for ; Tue, 22 Jul 2014 19:23:47 +0000 (UTC) Received: by mail-qg0-f54.google.com with SMTP id z60so152976qgd.13 for ; Tue, 22 Jul 2014 12:23:47 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:in-reply-to:references:from:date:message-id:subject:to :cc:content-type; bh=qH3hMkjTR5rF5u8ad5tKNqJ+VOJt2PI9FKwK6GsT7Cw=; b=TvLOmzww9ePBFCeFSJZAJpTuyFnZtwpZj8wlr1KAofu2Lvh881zOZ+vVCMhmP/StxX fxp5F/6OcFClSZhG1lV2rlimEZ0AikrcNDor4KeM8GUDkFBivXuHFj11RFvTRB1XHDAA gCab259cwRYL4cVhJEIJyQQBWEzjmSglyuQNgUD1Ns5nlCUS0NqiWeGyug8Jw0dLuM6I nRcV/4RkP30Wk0n6I0GBrAOIVSNcm5LitGpCS0kAj1LM7ojrCYMrI/Kay2tyQt7fhKd0 FHjfjtZ0+QuyuWjuE2ofKsH/5yiPK/LdCYeXGK2sgGBGUvsoL4zvnG/FrJ1uC757u5DA utog== X-Received: by 10.224.131.8 with SMTP id v8mr22644244qas.31.1406057027048; Tue, 22 Jul 2014 12:23:47 -0700 (PDT) MIME-Version: 1.0 Received: by 10.96.74.69 with HTTP; Tue, 22 Jul 2014 12:23:06 -0700 (PDT) In-Reply-To: <53CEAF40.7030406@gmail.com> References: <53CE80DD.9090109@gmail.com> <20140722174117.GC43962@funkthat.com> <53CEAF40.7030406@gmail.com> From: Carlos Ferreira Date: Tue, 22 Jul 2014 20:23:06 +0100 Message-ID: Subject: Re: fastforward/routing: a 3 million packet-per-second system? To: John Jasen Content-Type: text/plain; charset=UTF-8 X-Content-Filtered-By: Mailman/MimeDel 2.1.18 Cc: FreeBSD Net X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 22 Jul 2014 19:23:48 -0000 I think the results presented at the paper are regarding one port sending or receiving at 14.88Mpps. Using several ports at the same time will surely give much lower results. But then again, if one wants 8, 16, 24 or even more ports at 10Gb/s, then it should look for FPGA implementations. On 22 July 2014 19:36, John Jasen wrote: > > On 07/22/2014 01:41 PM, John-Mark Gurney wrote: > > John Jasen wrote this message on Tue, Jul 22, 2014 at 11:18 -0400: > >> Feedback and/or tips and tricks more than welcome. > > You should look at netmap if you really want high PPS routing... > > Originally, I assumed an interface supporting netmap was required, but > the manpage disabuses me of this. Besides, I think the Chelsio cards got > netmap recently. > > I presume either the use of bridge in tools/netmap, or vale-ctl in the > same location would start providing me sufficient clue on this? > Unfortunately, both seem to be pretty short on verbosity ... More clue > is always welcome! > > > _______________________________________________ > freebsd-net@freebsd.org mailing list > http://lists.freebsd.org/mailman/listinfo/freebsd-net > To unsubscribe, send any mail to "freebsd-net-unsubscribe@freebsd.org" > -- Carlos Miguel Ferreira Researcher at Telecommunications Institute Aveiro - Portugal Work E-mail - cmf@av.it.pt Skype & GTalk -> carlosmf.pt@gmail.com LinkedIn -> http://www.linkedin.com/in/carlosmferreira From owner-freebsd-net@FreeBSD.ORG Tue Jul 22 19:30:28 2014 Return-Path: Delivered-To: freebsd-net@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 5FD9D789 for ; Tue, 22 Jul 2014 19:30:28 +0000 (UTC) Received: from mail-qg0-x22f.google.com (mail-qg0-x22f.google.com [IPv6:2607:f8b0:400d:c04::22f]) (using TLSv1 with cipher ECDHE-RSA-RC4-SHA (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 1C6BF2B81 for ; Tue, 22 Jul 2014 19:30:28 +0000 (UTC) Received: by mail-qg0-f47.google.com with SMTP id i50so172467qgf.6 for ; Tue, 22 Jul 2014 12:30:27 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:sender:in-reply-to:references:date:message-id:subject :from:to:cc:content-type; bh=/UU4xMkAlPg+PYZh0aeBYZtOKrs1fJtfseMHwEuOGdg=; b=gwWE/wccZaT1IdoN7pROcvjBkxVzuTGA4DmrUvtWay7uJdXTLLqKx7tQ7xOh5JwQkX enGQ3q8Ba9JpANKGSS0f5fUgykJUly5ELplWP0NVS0nTxtiWna7B6npkDrx+x7QmfDuY IeeV9TO7OmrKLwbt2VyJoXUbAjQudSttPc15ni/cU005QXPNmP87X6kE6aCVh/E/UuXW gXUczdlEQh8utYp0Dl120m+HqxyALDxFTsroPcYkLMDqJkxUtM/WQ9tdRIjUm0f2Zb6s 2Qi9vIpwRpi+Lh4IkuSGcFhp4iLZXAtKnyop2+Wnh8rsXX2V37mh1HDzqH8P+x7HRHiA yQ+g== MIME-Version: 1.0 X-Received: by 10.224.171.197 with SMTP id i5mr60738870qaz.55.1406057427260; Tue, 22 Jul 2014 12:30:27 -0700 (PDT) Sender: adrian.chadd@gmail.com Received: by 10.224.1.6 with HTTP; Tue, 22 Jul 2014 12:30:27 -0700 (PDT) In-Reply-To: <53CEB9B5.7020609@gmail.com> References: <53CE80DD.9090109@gmail.com> <53CEB090.7030701@gmail.com> <53CEB670.9060600@gmail.com> <53CEB9B5.7020609@gmail.com> Date: Tue, 22 Jul 2014 12:30:27 -0700 X-Google-Sender-Auth: 9vxgva6K-a6Gk-AJqW-ngAprZqk Message-ID: Subject: Re: fastforward/routing: a 3 million packet-per-second system? From: Adrian Chadd To: John Jasen , FreeBSD Net Content-Type: text/plain; charset=UTF-8 Cc: Navdeep Parhar X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 22 Jul 2014 19:30:28 -0000 hi! You can use 'pmcstat -S CPU_CLK_UNHALTED_CORE -O pmc.out' (then ctrl-C it after say 5 seconds), which will log the data to pmc.out; then 'pmcannotate -k /boot/kernel pmc.out /boot/kernel/kernel' to find out where the most cpu cycles are being spent. It should give us the location(s) inside the top CPU users. Hopefully it'll then be much more obvious! I'm glad you're digging into it! -a On 22 July 2014 12:21, John Jasen wrote: > Navdeep; > > I was struck by spending so much time in transmit as well. > > Adrian's suggestion on mining lock profiling gave me an excuse to up the > tx queues in /boot/loader.conf. Our prior conversations indicated that > up to 64 should be acceptable? > > > > > > On 07/22/2014 03:10 PM, Adrian Chadd wrote: >> Hi >> >> Right. Time to figure out why you're spending so much time in >> cxgbe_transmit() and t4_eth_tx(). Maybe ask Navdeep for some ideas? >> >> >> -a >> >> On 22 July 2014 12:07, John Jasen wrote: >>> The first is pretty easy to turn around. Reading on dtrace now. Thanks >>> for the pointers and help! >>> >>> PMC: [CPU_CLK_UNHALTED_CORE] Samples: 142654 (100.0%) , 124560 unresolved >>> >>> %SAMP IMAGE FUNCTION CALLERS >>> 34.0 if_cxgbe.k t4_eth_tx cxgbe_transmit:24.0 t4_tx_task:10.0 >>> 28.8 if_cxgbe.k cxgbe_transmit >>> 7.6 if_cxgbe.k service_iq t4_intr >>> 6.4 if_cxgbe.k get_scatter_segment service_iq >>> 4.9 if_cxgbe.k reclaim_tx_descs t4_eth_tx >>> 3.2 if_cxgbe.k write_sgl_to_txd t4_eth_tx >>> 2.8 hwpmc.ko pmclog_process_callc pmc_process_samples >>> 2.1 libc.so.7 bcopy pmclog_read >>> 1.9 if_cxgbe.k t4_eth_rx service_iq >>> 1.7 hwpmc.ko pmclog_reserve pmclog_process_callchain >>> 1.2 libpmc.so. pmclog_read >>> 1.0 if_cxgbe.k write_txpkts_wr t4_eth_tx >>> 0.8 kernel e1000_read_i2c_byte_ e1000_set_i2c_bb >>> 0.6 libc.so.7 memset >>> 0.5 if_cxgbe.k refill_fl service_iq >>> >>> >>> >>> >>> On 07/22/2014 02:45 PM, Adrian Chadd wrote: >>>> Hi, >>>> >>>> Well, start with CPU profiling with pmc: >>>> >>>> kldload hwpmc >>>> pmcstat -S CPU_CLK_UNHALTED_CORE -T -w 1 >>>> >>>> .. look at the freebsd dtrace onliners (google that) for lock >>>> contention and CPU usage. >>>> >>>> You could compile with LOCK_PROFILING (which will slow things down a >>>> little even when not in use) then enable it for a few seconds (which >>>> will definitely slow things down) to gather fine grained lock >>>> contention data. >>>> >>>> (sysctl debug.lock.prof when you have it compiled in. sysctl >>>> debug.lock.prof.enable=1; wait 10 seconds; sysctl >>>> debug.lock.prof.enable=0; sysctl debug.lock.prof.stats) >>>> >>>> >>>> -a >>>> >>>> >>>> On 22 July 2014 11:42, John Jasen wrote: >>>>> If you have ideas on what to instrument, I'll be happy to do it. >>>>> >>>>> I'm faintly familiar with dtrace, et al, so it might take me a few tries >>>>> to get it right -- or bludgeoning with the documentation. >>>>> >>>>> Thanks! >>>>> >>>>> >>>>> >>>>> >>>>> On 07/22/2014 02:07 PM, Adrian Chadd wrote: >>>>>> Hi! >>>>>> >>>>>> Well, what's missing is some dtrace/pmc/lockdebugging investigations >>>>>> into the system to see where it's currently maxing out at. >>>>>> >>>>>> I wonder if you're seeing contention on the transmit paths as drivers >>>>>> queue frames from one set of driver threads/queues to another >>>>>> potentially completely different set of driver transmit >>>>>> threads/queues. >>>>>> >>>>>> >>>>>> >>>>>> >>>>>> -a >>>>>> >>>>>> >>>>>> On 22 July 2014 08:18, John Jasen wrote: >>>>>>> Feedback and/or tips and tricks more than welcome. >>>>>>> >>>>>>> Outstanding questions: >>>>>>> >>>>>>> Would increasing the number of processor cores help? >>>>>>> >>>>>>> Would a system where both processor QPI ports connect to each other >>>>>>> mitigate QPI bottlenecks? >>>>>>> >>>>>>> Are there further performance optimizations I am missing? >>>>>>> >>>>>>> Server Description: >>>>>>> >>>>>>> The system in question is a Dell Poweredge R820, 16GB of RAM, and two >>>>>>> Intel(R) Xeon(R) CPU E5-4610 0 @ 2.40GHz. >>>>>>> >>>>>>> Onboard, in a 16x PCIe slot, I have one Chelsio T-580-CR two-port 40GbE >>>>>>> NIC, and in an 8x slot, another T-580-CR dual port. >>>>>>> >>>>>>> I am running FreeBSD 10.0-STABLE. >>>>>>> >>>>>>> BIOS tweaks: >>>>>>> >>>>>>> Hyperthreading (or Logical Processors) is turned off. >>>>>>> Memory Node Interleaving is turned off, but did not appear to impact >>>>>>> performance. >>>>>>> >>>>>>> /boot/loader.conf contents: >>>>>>> #for CARP+PF testing >>>>>>> carp_load="YES" >>>>>>> #load cxgbe drivers. >>>>>>> cxgbe_load="YES" >>>>>>> #maxthreads appears to not exceed CPU. >>>>>>> net.isr.maxthreads=12 >>>>>>> #bindthreads may be indicated when using cpuset(1) on interrupts >>>>>>> net.isr.bindthreads=1 >>>>>>> #random guess based on googling >>>>>>> net.isr.maxqlimit=60480 >>>>>>> net.link.ifqmaxlen=90000 >>>>>>> #discussions with cxgbe maintainer and list led me to trying this. >>>>>>> Allows more interrupts >>>>>>> #to be fixed to CPUs, which in some cases, improves interrupt balancing. >>>>>>> hw.cxgbe.ntxq10g=16 >>>>>>> hw.cxgbe.nrxq10g=16 >>>>>>> >>>>>>> /etc/sysctl.conf contents: >>>>>>> >>>>>>> #the following is also enabled by rc.conf gateway_enable. >>>>>>> net.inet.ip.fastforwarding=1 >>>>>>> #recommendations from BSD router project >>>>>>> kern.random.sys.harvest.ethernet=0 >>>>>>> kern.random.sys.harvest.point_to_point=0 >>>>>>> kern.random.sys.harvest.interrupt=0 >>>>>>> #probably should be removed, as cxgbe does not seem to affect/be >>>>>>> affected by irq storm settings >>>>>>> hw.intr_storm_threshold=25000000 >>>>>>> #based on Calomel.Org performance suggestions. 4x40GbE, seemed >>>>>>> reasonable to use 100GbE settings >>>>>>> kern.ipc.maxsockbuf=1258291200 >>>>>>> net.inet.tcp.recvbuf_max=1258291200 >>>>>>> net.inet.tcp.sendbuf_max=1258291200 >>>>>>> #attempting to play with ULE scheduler, making it serve packets versus >>>>>>> netstat >>>>>>> kern.sched.slice=1 >>>>>>> kern.sched.interact=1 >>>>>>> >>>>>>> /etc/rc.conf contains: >>>>>>> >>>>>>> hostname="fbge1" >>>>>>> #should remove, especially given below duplicate entry >>>>>>> ifconfig_igb0="DHCP" >>>>>>> sshd_enable="YES" >>>>>>> #ntpd_enable="YES" >>>>>>> # Set dumpdev to "AUTO" to enable crash dumps, "NO" to disable >>>>>>> dumpdev="AUTO" >>>>>>> # OpenBSD PF options to play with later. very bad for raw packet rates. >>>>>>> #pf_enable="YES" >>>>>>> #pflog_enable="YES" >>>>>>> # enable packet forwarding >>>>>>> # these enable forwarding and fastforwarding sysctls. inet6 does not >>>>>>> have fastforward >>>>>>> gateway_enable="YES" >>>>>>> ipv6_gateway_enable="YES" >>>>>>> # enable OpenBSD ftp-proxy >>>>>>> # should comment out until actively playing with PF >>>>>>> ftpproxy_enable="YES" >>>>>>> #left in place, commented out from prior testing >>>>>>> #ifconfig_mlxen1="inet 172.16.2.1 netmask 255.255.255.0 mtu 9000" >>>>>>> #ifconfig_mlxen0="inet 172.16.1.1 netmask 255.255.255.0 mtu 9000" >>>>>>> #ifconfig_mlxen3="inet 172.16.7.1 netmask 255.255.255.0 mtu 9000" >>>>>>> #ifconfig_mlxen2="inet 172.16.8.1 netmask 255.255.255.0 mtu 9000" >>>>>>> # -lro and -tso options added per mailing list suggestion from Bjoern A. >>>>>>> Zeeb (bzeeb-lists at lists.zabbadoz.net) >>>>>>> ifconfig_cxl0="inet 172.16.3.1 netmask 255.255.255.0 mtu 9000 -lro -tso up" >>>>>>> ifconfig_cxl1="inet 172.16.4.1 netmask 255.255.255.0 mtu 9000 -lro -tso up" >>>>>>> ifconfig_cxl2="inet 172.16.5.1 netmask 255.255.255.0 mtu 9000 -lro -tso up" >>>>>>> ifconfig_cxl3="inet 172.16.6.1 netmask 255.255.255.0 mtu 9000 -lro -tso up" >>>>>>> # aliases instead of reconfiguring test clients. See above commented out >>>>>>> entries >>>>>>> ifconfig_cxl0_alias0="172.16.7.1 netmask 255.255.255.0" >>>>>>> ifconfig_cxl1_alias0="172.16.8.1 netmask 255.255.255.0" >>>>>>> ifconfig_cxl2_alias0="172.16.1.1 netmask 255.255.255.0" >>>>>>> ifconfig_cxl3_alias0="172.16.2.1 netmask 255.255.255.0" >>>>>>> # for remote monitoring/admin of the test device >>>>>>> ifconfig_igb0="inet 172.30.60.60 netmask 255.255.0.0" >>>>>>> >>>>>>> Additional configurations: >>>>>>> cpuset-chelsio-6cpu-high >>>>>>> # Original provided by Navdeep Parhar >>>>>>> # takes vmstat -ai output into a list, and assigns interrupts in order to >>>>>>> # the available CPU cores. >>>>>>> # Modified: to assign only to the 'high CPUs', ie: on core1. >>>>>>> # See: http://lists.freebsd.org/pipermail/freebsd-net/2014-July/039317.html >>>>>>> #!/usr/local/bin/bash >>>>>>> ncpu=12 >>>>>>> irqlist=$(vmstat -ia | egrep 't4nex|t5nex|cxgbc' | cut -f1 -d: | cut -c4-) >>>>>>> i=6 >>>>>>> for irq in $irqlist; do >>>>>>> cpuset -l $i -x $irq >>>>>>> i=$((i+1)) >>>>>>> [ $i -ge $ncpu ] && i=6 >>>>>>> done >>>>>>> >>>>>>> Client Description: >>>>>>> >>>>>>> Two Intel(R) Xeon(R) CPU E5-2670 v2 @ 2.50GHz processors >>>>>>> 64 GB ram >>>>>>> Mellanox Technologies MT27500 Family [ConnectX-3] >>>>>>> Centos 6.4 with updates >>>>>>> iperf3 installed from yum repositories: iperf3-3.0.3-3.el6.x86_64 >>>>>>> >>>>>>> Test setup: >>>>>>> >>>>>>> I've found about 3 streams between Centos clients is about the best way >>>>>>> to get the most out of them. >>>>>>> Above certain points, the -b flag does not change results. >>>>>>> -N is an artifact from using TCP >>>>>>> -l is needed, as -M doesn't work for UDP. >>>>>>> >>>>>>> I usually use launch scripts similar to the following: >>>>>>> >>>>>>> for i in `seq 41 60`; do ssh loader$i "export TIME=120; export >>>>>>> STREAMS=1; export PORT=52$i; export PKT=64; export RATE=2000m; >>>>>>> /root/iperf-test-8port-udp" & done >>>>>>> >>>>>>> The scripts execute the following on each host. >>>>>>> >>>>>>> #!/bin/bash >>>>>>> PORT1=$PORT >>>>>>> PORT2=$(($PORT+1000)) >>>>>>> PORT3=$(($PORT+2000)) >>>>>>> iperf3 -c loader41-40gbe -u -b 10000m -i 0 -N -l $PKT -t$TIME >>>>>>> -P$STREAMS -p$PORT1 & >>>>>>> iperf3 -c loader42-40gbe -u -b 10000m -i 0 -N -l $PKT -t$TIME >>>>>>> -P$STREAMS -p$PORT1 & >>>>>>> iperf3 -c loader43-40gbe -u -b 10000m -i 0 -N -l $PKT -t$TIME >>>>>>> -P$STREAMS -p$PORT1 & >>>>>>> ... (through all clients and all three ports) ... >>>>>>> iperf3 -c loader60-40gbe -u -b 10000m -i 0 -N -l $PKT -t$TIME >>>>>>> -P$STREAMS -p$PORT3 & >>>>>>> >>>>>>> >>>>>>> Results: >>>>>>> >>>>>>> Summarized, netstat -w 1 -q 240 -bd, run through: >>>>>>> cat test4-tuning | egrep -v {'packets | input '} | awk '{ipackets+=$1} >>>>>>> {idrops+=$3} {opackets+=$5} {odrops+=$9} END {print "input " >>>>>>> ipackets/NR, "idrops " idrops/NR, "opackets " opackets/NR, "odrops " >>>>>>> odrops/NR}' >>>>>>> >>>>>>> input 1.10662e+07 idrops 8.01783e+06 opackets 3.04516e+06 odrops 3152.4 >>>>>>> >>>>>>> Snapshot of raw output: >>>>>>> >>>>>>> input (Total) output >>>>>>> packets errs idrops bytes packets errs bytes colls drops >>>>>>> 11189148 0 7462453 1230805216 3725006 0 409750710 0 799 >>>>>>> 10527505 0 6746901 1158024978 3779096 0 415700708 0 127 >>>>>>> 10606163 0 6850760 1166676673 3751780 0 412695761 0 1535 >>>>>>> 10749324 0 7132014 1182425799 3617558 0 397930956 0 5972 >>>>>>> 10695667 0 7022717 1176521907 3669342 0 403627236 0 1461 >>>>>>> 10441173 0 6762134 1148528662 3675048 0 404255540 0 6021 >>>>>>> 10683773 0 7005635 1175215014 3676962 0 404465671 0 2606 >>>>>>> 10869859 0 7208696 1195683372 3658432 0 402427698 0 979 >>>>>>> 11948989 0 8310926 1314387881 3633773 0 399714986 0 725 >>>>>>> 12426195 0 8864415 1366877194 3562311 0 391853156 0 2762 >>>>>>> 13006059 0 9432389 1430661751 3570067 0 392706552 0 5158 >>>>>>> 12822243 0 9098871 1410443600 3715177 0 408668500 0 4064 >>>>>>> 13317864 0 9683602 1464961374 3632156 0 399536131 0 3684 >>>>>>> 13701905 0 10182562 1507207982 3523101 0 387540859 0 >>>>>>> 8690 >>>>>>> 13820227 0 10244870 1520221820 3562038 0 391823322 0 >>>>>>> 2426 >>>>>>> 14437060 0 10955483 1588073033 3480105 0 382810557 0 >>>>>>> 2619 >>>>>>> 14518471 0 11119573 1597028105 3397439 0 373717355 0 >>>>>>> 5691 >>>>>>> 14890287 0 11675003 1637926521 3199812 0 351978304 0 >>>>>>> 11007 >>>>>>> 14923610 0 11749091 1641594441 3171436 0 348857468 0 >>>>>>> 7389 >>>>>>> 14738704 0 11609730 1621254991 3117715 0 342948394 0 >>>>>>> 2597 >>>>>>> 14753975 0 11549735 1622935026 3207393 0 352812846 0 >>>>>>> 4798 >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> _______________________________________________ >>>>>>> freebsd-net@freebsd.org mailing list >>>>>>> http://lists.freebsd.org/mailman/listinfo/freebsd-net >>>>>>> To unsubscribe, send any mail to "freebsd-net-unsubscribe@freebsd.org" > From owner-freebsd-net@FreeBSD.ORG Tue Jul 22 19:34:54 2014 Return-Path: Delivered-To: freebsd-net@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 13F71D46 for ; Tue, 22 Jul 2014 19:34:54 +0000 (UTC) Received: from mail-oa0-x233.google.com (mail-oa0-x233.google.com [IPv6:2607:f8b0:4003:c02::233]) (using TLSv1 with cipher ECDHE-RSA-RC4-SHA (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id CE3582C52 for ; Tue, 22 Jul 2014 19:34:53 +0000 (UTC) Received: by mail-oa0-f51.google.com with SMTP id o6so198711oag.24 for ; Tue, 22 Jul 2014 12:34:53 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :cc:content-type; bh=g+ZWoejw+QeAgAbscH/pDGReesRt+6vOBRsJkF3HCs4=; b=A+yDWr4qpH4hegjwV1LTB0n+uAlEzFSTUP0lUpLYNNERyuQVpUqwkeSyVLsa8NjFb6 +kGZbqSgXb3UY4PLCUwec9y13xoBiKxcY9JUa+aaSPqBFFUo5C2CngxPDJ0UIurujiQ7 bDVlXTBFkHIY8nauFmh4PosHK4iepgyjU59MQwbEnXLG+9WKPdqKqNHWkqmCZkn273+Q TisSYrbXwyoK1jUrPDuuW5os7eUp3QXdw+eBr+Wf0sSk/4HgvkoRCYmzgjKm0L6mZ5aQ rZpk70/6RXmzEJMJzh6AOYvPH28Ut4DS01URcEi0/zqoHcItByEvLZ/u0bjSlnsE0PMY fa4g== MIME-Version: 1.0 X-Received: by 10.60.176.36 with SMTP id cf4mr29396789oec.2.1406057693157; Tue, 22 Jul 2014 12:34:53 -0700 (PDT) Received: by 10.76.170.39 with HTTP; Tue, 22 Jul 2014 12:34:53 -0700 (PDT) In-Reply-To: References: <53CE80DD.9090109@gmail.com> <20140722174117.GC43962@funkthat.com> <53CEAF40.7030406@gmail.com> Date: Tue, 22 Jul 2014 21:34:53 +0200 Message-ID: Subject: Re: fastforward/routing: a 3 million packet-per-second system? From: Andreas Nilsson To: Carlos Ferreira Content-Type: text/plain; charset=UTF-8 X-Content-Filtered-By: Mailman/MimeDel 2.1.18 Cc: FreeBSD Net , John Jasen X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 22 Jul 2014 19:34:54 -0000 Well, it shows how easily one can saturate the link. Just use more ports, the will be saturated as well. The problem though is that netmap requires that one implements the forwarding "logic" I think. Best regards Andreas On Tue, Jul 22, 2014 at 9:23 PM, Carlos Ferreira wrote: > I think the results presented at the paper are regarding one port sending > or receiving at 14.88Mpps. Using several ports at the same time will surely > give much lower results. But then again, if one wants 8, 16, 24 or even > more ports at 10Gb/s, then it should look for FPGA implementations. > > > On 22 July 2014 19:36, John Jasen wrote: > > > > > On 07/22/2014 01:41 PM, John-Mark Gurney wrote: > > > John Jasen wrote this message on Tue, Jul 22, 2014 at 11:18 -0400: > > >> Feedback and/or tips and tricks more than welcome. > > > You should look at netmap if you really want high PPS routing... > > > > Originally, I assumed an interface supporting netmap was required, but > > the manpage disabuses me of this. Besides, I think the Chelsio cards got > > netmap recently. > > > > I presume either the use of bridge in tools/netmap, or vale-ctl in the > > same location would start providing me sufficient clue on this? > > Unfortunately, both seem to be pretty short on verbosity ... More clue > > is always welcome! > > > > > > _______________________________________________ > > freebsd-net@freebsd.org mailing list > > http://lists.freebsd.org/mailman/listinfo/freebsd-net > > To unsubscribe, send any mail to "freebsd-net-unsubscribe@freebsd.org" > > > > > > -- > > Carlos Miguel Ferreira > Researcher at Telecommunications Institute > Aveiro - Portugal > Work E-mail - cmf@av.it.pt > Skype & GTalk -> carlosmf.pt@gmail.com > LinkedIn -> http://www.linkedin.com/in/carlosmferreira > _______________________________________________ > freebsd-net@freebsd.org mailing list > http://lists.freebsd.org/mailman/listinfo/freebsd-net > To unsubscribe, send any mail to "freebsd-net-unsubscribe@freebsd.org" > From owner-freebsd-net@FreeBSD.ORG Tue Jul 22 19:35:25 2014 Return-Path: Delivered-To: freebsd-net@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id B34B1DDF; Tue, 22 Jul 2014 19:35:25 +0000 (UTC) Received: from mx.ipv6.elandsys.com (mx.ipv6.elandsys.com [IPv6:2001:470:f329:1::1]) by mx1.freebsd.org (Postfix) with ESMTP id 5FE3C2C62; Tue, 22 Jul 2014 19:35:25 +0000 (UTC) Received: from mx.elandsys.com (IDENT:logan@localhost [127.0.0.1]) by mx.elandsys.com (8.14.5/8.14.5) with ESMTP id s6MJZMHa014754 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-SHA bits=256 verify=NO); Tue, 22 Jul 2014 12:35:23 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=simple/simple; d=opendkim.org; s=mail2010; t=1406057724; x=1406144124; bh=kbc9KddkrpQzf30EEZWqRgfXlnNk+3U1usqYaSOjWWw=; h=Date:From:To:Cc:Subject:References:In-Reply-To; b=VnptWyffMNUl79TZlYJ85iDEBvkgODRfJ81DzaNfLStKh5ZUKSsOaZ6bCGVwkxfLI uEQwZK4ELbbuJPo3Q0vt0JoapPbWJHOi8bdDj8G5s0dfiAMQ2aGb4vpJ0JJNxATs/3 PP6s0Ximdc3Y/4qpp1uOS1zX+MlGz4K25w5ys96A= DKIM-Signature: v=1; a=rsa-sha256; c=simple/simple; d=elandsys.com; s=mail; t=1406057724; x=1406144124; i=@elandsys.com; bh=kbc9KddkrpQzf30EEZWqRgfXlnNk+3U1usqYaSOjWWw=; h=Date:From:To:Cc:Subject:References:In-Reply-To; b=OFqaHGIdB/HY7kZMVpkvD7TpkW5SSOsMC2g/98MkcZ13GEVvJqVX/woRlbw4dvJ+n qCX42QVImmUiJYbXbHUlnWX9RGMdzfMEPTMZyVVHkKr24TmGzpSwR15uvFBFVEHSjM zMO6sFTxoTFwN8r2zP6fm/k7JJavRAjpTcO1qiRY= Received: (from logan@localhost) by mx.elandsys.com (8.14.5/8.14.5/Submit) id s6MJZMde013237; Tue, 22 Jul 2014 12:35:22 -0700 (PDT) X-Authentication-Warning: mx.elandsys.com: logan set sender to logan@elandsys.com using -f Date: Tue, 22 Jul 2014 12:35:22 -0700 From: Loganaden Velvindron To: jinmei Subject: Re: IPv6 nodeinfo default behaviour Message-ID: <20140722193521.GA20775@mx.elandsys.com> References: <20140720090410.GA7990@mx.elandsys.com> <20140722170150.GA971@mx.elandsys.com> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: User-Agent: Mutt/1.5.21 (2010-09-15) Cc: FreeBSD Net , bz@freebsd.org, gnn@freebsd.org X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 22 Jul 2014 19:35:25 -0000 On Tue, Jul 22, 2014 at 11:25:37AM -0700, ???? wrote: > At Tue, 22 Jul 2014 10:01:50 -0700, > Loganaden Velvindron wrote: > > > > > Security Considerations > > > > > > > > This protocol has the potential of revealing information useful to a > > > > would-be attacker. An implementation of this protocol MUST have a > > > > default configuration that refuses to answer queries from global- > > > > scope [3] addresses. > > > > > > > > I suggest that we switch to 0 by default to be more RFC compliant. > > > > > > Are you referring to the value of '(V_)icmp6_nodeinfo'? > > > > I'm referring to the sysctl: > > > > net.inet6.icmp6.nodeinfo. > > These two are essentially the same in this context: this sysctl is an > interface to (V_)icmp6_nodeinfo. This variable is set to > ICMP6_NODEINFO_FQDNOK|ICMP6_NODEINFO_NODEADDROK by default, > and since ICMP6_NODEINFO_FQDNOK and ICMP6_NODEINFO_NODEADDROK are 0x1 > and 0x2, respectively, the default value of the sysctl variable is 3 > by default. > > In your original message, you said > > > > > I suggest that we switch to 0 by default to be more RFC compliant. > > and I tried to point out that it didn't make sense because "to be more > RFC compliant" it doesn't have to switch to 0, it just needs to have > the ICMP6_NODEINFO_GLOBALOK flag (0x8) cleared, and the current > default meets the condition already. > > Now you're changing the reason: > > > I think that it's sensible to turn it to 0 by default, unless you need > > it. > > Unlike being "RFC compliant", whether something is "sensible" is Sorry for the confusion I created. > usually subjective, and different people may have different opinions. > Personally, I often find "ping6 -w" quite useful for debugging > purposes, and I think limiting its use to link-local by default gives Agreed. Perhaps we should enable it only when we need to debug. > a reasonable level of defense (and, disabling it by default would > reduce the usability pretty much). So I'd rather prefer keeping the > current default, but, again, other people may have a different > preference. > > -- > JINMEI, Tatuya > _______________________________________________ > freebsd-net@freebsd.org mailing list > http://lists.freebsd.org/mailman/listinfo/freebsd-net > To unsubscribe, send any mail to "freebsd-net-unsubscribe@freebsd.org" From owner-freebsd-net@FreeBSD.ORG Tue Jul 22 20:38:23 2014 Return-Path: Delivered-To: freebsd-net@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 8B84CA0E; Tue, 22 Jul 2014 20:38:23 +0000 (UTC) Received: from mail-wg0-x232.google.com (mail-wg0-x232.google.com [IPv6:2a00:1450:400c:c00::232]) (using TLSv1 with cipher ECDHE-RSA-RC4-SHA (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id CF7B62381; Tue, 22 Jul 2014 20:38:22 +0000 (UTC) Received: by mail-wg0-f50.google.com with SMTP id n12so197404wgh.9 for ; Tue, 22 Jul 2014 13:38:21 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:sender:in-reply-to:references:date:message-id:subject :from:to:cc:content-type; bh=/WZYrawisDkGugwSNisWmpw/6zdV3Q/KZbIX1WlPkOM=; b=tcp88LCiNA8WgpGTBh44trqWJA0HHpDNOfD3Xk2IaLP6+tkbs+yyWfwchJ1N88tya5 c9PteLNg7CAgzCFkePhe9s2E/hyk+qOX5QEE3eeo/tkphvTFFKfPa8KPSanpZF4KXm6f 3e/1viK+FisCQeWWOqygvnSbRZi8nmV63rW2NGQhyAzr5ufUpQUB48JJOkuPDjbiH9yb ZJG6UnqikpyMlCehgnwR1Q/5PogfLxr+GBVnR+dcsasamcY6IpNCQN67mFy+zlG7/+T2 T5FHRDdzQUo6zlQJxQWUjRu9zMsNmwzEFf0WFSiyT2K5+48hLLZEAb+KnJCDuzdQON3G MstA== MIME-Version: 1.0 X-Received: by 10.194.20.230 with SMTP id q6mr38434223wje.43.1406061500998; Tue, 22 Jul 2014 13:38:20 -0700 (PDT) Sender: jinmei.tatuya@gmail.com Received: by 10.194.108.166 with HTTP; Tue, 22 Jul 2014 13:38:20 -0700 (PDT) In-Reply-To: <20140722193521.GA20775@mx.elandsys.com> References: <20140720090410.GA7990@mx.elandsys.com> <20140722170150.GA971@mx.elandsys.com> <20140722193521.GA20775@mx.elandsys.com> Date: Tue, 22 Jul 2014 13:38:20 -0700 X-Google-Sender-Auth: 9fWml7v_bzJmk3g0SDYGHJzbVxw Message-ID: Subject: Re: IPv6 nodeinfo default behaviour From: =?UTF-8?B?56We5piO6YGU5ZOJ?= To: Loganaden Velvindron Content-Type: text/plain; charset=UTF-8 Cc: FreeBSD Net , bz@freebsd.org, George Neville-Neil X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 22 Jul 2014 20:38:23 -0000 At Tue, 22 Jul 2014 12:35:22 -0700, Loganaden Velvindron wrote: > > usually subjective, and different people may have different opinions. > > Personally, I often find "ping6 -w" quite useful for debugging > > purposes, and I think limiting its use to link-local by default gives > > Agreed. Perhaps we should enable it only when we need to debug. > > > a reasonable level of defense (and, disabling it by default would > > reduce the usability pretty much). So I'd rather prefer keeping the > > current default, but, again, other people may have a different > > preference. To be clear, in case I wasn't: in my opinion it would become useless for debugging unless it's enabled by default, so I would like it to be (kept) enabled by default (note that it's already limited to link-local by default). But I understand YMMV. -- JINMEI, Tatuya From owner-freebsd-net@FreeBSD.ORG Wed Jul 23 03:38:06 2014 Return-Path: Delivered-To: freebsd-net@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 34CF9571 for ; Wed, 23 Jul 2014 03:38:06 +0000 (UTC) Received: from mail-pd0-x22f.google.com (mail-pd0-x22f.google.com [IPv6:2607:f8b0:400e:c02::22f]) (using TLSv1 with cipher ECDHE-RSA-RC4-SHA (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 0934127D9 for ; Wed, 23 Jul 2014 03:38:06 +0000 (UTC) Received: by mail-pd0-f175.google.com with SMTP id r10so775400pdi.6 for ; Tue, 22 Jul 2014 20:38:05 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=date:from:to:message-id:subject:mime-version:content-type; bh=mVKYS5RCQvlCvo+x+6cBTcho/o4bcjjHLS2XNUyzU8M=; b=VSjOl4fpBNHgXv3PMaU6GWOSAvJq6GsgzhwKcoeE7Gzkl9e3GCTiuQfrycprnlavd8 QYh44FkGwwIMJvH94Zd8myJWvjt4DbsNuznwMHJFWaIXkCO4QBlsEB2thIbe3a/Z54yV 1jMz0tRJhfSPdJontfkB3GO2pFKRfZ3xchsNy7nOCswML/jgDLBpyTXoDUu+zEWTuOjt f2ySwBHXS80ESRWCqgywppHlyAeS7oeJLsn+F6ZPBYuSUpdimqg3XbUKDLRqgnHyEScA XWb8DpAEK6QbTPkkM4bSmt5WvxBsEh/hMLrbL6oSem721PaP7hsmDxEcG04CFcXNAa6m VV8g== X-Received: by 10.66.250.161 with SMTP id zd1mr38374681pac.61.1406086685559; Tue, 22 Jul 2014 20:38:05 -0700 (PDT) Received: from [192.168.10.153] (114-34-221-39.HINET-IP.hinet.net. [114.34.221.39]) by mx.google.com with ESMTPSA id by7sm3676454pab.35.2014.07.22.20.38.04 for (version=TLSv1 cipher=RC4-SHA bits=128/128); Tue, 22 Jul 2014 20:38:04 -0700 (PDT) Date: Wed, 23 Jul 2014 11:38:01 +0800 From: Chen Wen To: freebsd-net@freebsd.org Message-ID: Subject: usr.sbin/ctld/login.c do not reply TargetPortalGroupTag in Login response X-Mailer: sparrow 1.6.3 (build 1173) MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: quoted-printable Content-Disposition: inline X-Content-Filtered-By: Mailman/MimeDel 2.1.18 X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 23 Jul 2014 03:38:06 -0000 Hi =20 I am a mac user, when I try to use xtendsan iSCSI initiator to connect na= tive iSCSI target, I found that the login response pdu do not have Target= PortalGroupTag key-pair. xtendsan told me TargetPortalGroupTag is missing and disconnected. I try to do a workaround to it and it works. Add some codes in usr.sbin/ctld/login.c:login=5Fnegotiate() > login=5Fnegotiate(struct connection *conn, struct pdu *request) > =7B > struct pdu *response; > struct iscsi=5Fbhs=5Flogin=5Fresponse *bhslr2; > struct keys *request=5Fkeys, *response=5Fkeys; > int i; > bool skipped=5Fsecurity; char *portal=5Fgroup=5Ftag; int rv; > =20 > if (request =3D=3D NULL) =7B > log=5Fdebugx(=22beginning parameter negotiation; =22 > =22waiting for Login PDU=22); > request =3D login=5Freceive(conn, false); > skipped=5Fsecurity =3D false; > =7D else > skipped=5Fsecurity =3D true; > =20 > request=5Fkeys =3D keys=5Fnew(); > keys=5Fload(request=5Fkeys, request); > =20 > response =3D login=5Fnew=5Fresponse(request); > bhslr2 =3D (struct iscsi=5Fbhs=5Flogin=5Fresponse *)response->p= du=5Fbhs; > bhslr2->bhslr=5Fflags =7C=3D BHSLR=5F=46LAGS=5FTRANSIT; > bhslr2->bhslr=5Ftsih =3D htons(0xbadd); > login=5Fset=5Fcsg(response, BHSLR=5FSTAGE=5FOPERATIONAL=5FNEGOT= IATION); > login=5Fset=5Fnsg(response, BHSLR=5FSTAGE=5F=46ULL=5F=46EATURE=5F= PHASE); > response=5Fkeys =3D keys=5Fnew(); if (conn->conn=5Fsession=5Ftype =3D=3D CONN=5FSESSION=5FTYPE=5FNO= RMAL) =7B if (conn->conn=5Ftarget->t=5Falias =21=3D NULL) keys=5Fadd(response=5Fkeys, =22TargetAlias=22, conn->conn=5Ftarget->t=5Fa= lias); rv =3D asprintf(&portal=5Fgroup=5Ftag, =22%d=22, conn->conn=5Fportal->p=5Fportal=5Fgroup->pg=5Ftag); if (rv <=3D 0) log=5Ferr(1, =22asprintf=22); keys=5Fadd(response=5Fkeys, =22TargetPortalGroupTag=22, portal=5Fgroup=5Ftag); free(portal=5Fgroup=5Ftag); =7D =20 > for (i =3D 0; i < KEYS=5FMAX; i++) =7B > if (request=5Fkeys->keys=5Fnames=5Bi=5D =3D=3D NULL) > break; > =20 > login=5Fnegotiate=5Fkey(request, request=5Fkeys->keys=5F= names=5Bi=5D, > request=5Fkeys->keys=5Fvalues=5Bi=5D, skipped=5Fsec= urity, > response=5Fkeys); > =7D I don=E2=80=99t read whole iSCSI R=46C, is this right to add missing Targ= etPortalGroupTag=3F Maybe you have better solution to fix this, please help me and thanks. -- =20 Chen Wen Sent with Sparrow (http://www.sparrowmailapp.com/=3Fsig) From owner-freebsd-net@FreeBSD.ORG Wed Jul 23 04:09:12 2014 Return-Path: Delivered-To: freebsd-net@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id E9B518FE; Wed, 23 Jul 2014 04:09:12 +0000 (UTC) Received: from h2.funkthat.com (gate2.funkthat.com [208.87.223.18]) (using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits)) (Client CN "funkthat.com", Issuer "funkthat.com" (not verified)) by mx1.freebsd.org (Postfix) with ESMTPS id C39EA2A48; Wed, 23 Jul 2014 04:09:12 +0000 (UTC) Received: from h2.funkthat.com (localhost [127.0.0.1]) by h2.funkthat.com (8.14.3/8.14.3) with ESMTP id s6N49BiL070024 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-SHA bits=256 verify=NO); Tue, 22 Jul 2014 21:09:11 -0700 (PDT) (envelope-from jmg@h2.funkthat.com) Received: (from jmg@localhost) by h2.funkthat.com (8.14.3/8.14.3/Submit) id s6N499Su070023; Tue, 22 Jul 2014 21:09:09 -0700 (PDT) (envelope-from jmg) Date: Tue, 22 Jul 2014 21:09:09 -0700 From: John-Mark Gurney To: Luigi Rizzo Subject: Re: netmap and other discussions on freebsd-net: please be open minded. Message-ID: <20140723040909.GJ43962@funkthat.com> Mail-Followup-To: Luigi Rizzo , Adrian Chadd , FreeBSD Net References: <201405181223.s4ICNTAf097378@fire.js.berklix.net> <20140518230242.GA28117@onelab2.iet.unipi.it> Mime-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: User-Agent: Mutt/1.4.2.3i X-Operating-System: FreeBSD 7.2-RELEASE i386 X-PGP-Fingerprint: 54BA 873B 6515 3F10 9E88 9322 9CB1 8F74 6D3F A396 X-Files: The truth is out there X-URL: http://resnet.uoregon.edu/~gurney_j/ X-Resume: http://resnet.uoregon.edu/~gurney_j/resume.html X-TipJar: bitcoin:13Qmb6AeTgQecazTWph4XasEsP7nGRbAPE X-to-the-FBI-CIA-and-NSA: HI! HOW YA DOIN? can i haz chizburger? X-Greylist: Sender IP whitelisted, not delayed by milter-greylist-4.2.2 (h2.funkthat.com [127.0.0.1]); Tue, 22 Jul 2014 21:09:11 -0700 (PDT) Cc: FreeBSD Net , Adrian Chadd X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 23 Jul 2014 04:09:13 -0000 Luigi Rizzo wrote this message on Mon, May 19, 2014 at 04:28 -0400: > On Sun, May 18, 2014 at 7:49 PM, Adrian Chadd wrote: > > > Is there a netmap list that these questions (regardless of OS) could go to? > > ???no there isn't one. At the moment there isn't enough traffic > to suggest that, and creating one is likely to generate > traffic to both the specific list and freebsd-net. What volume of traffic do you think is acceptable for creating a seperate list? The last few months I believe that there is enough traffic to justify a seperate list... and too often the questions are not at all useful to the FreeBSD community... If they were, I'd be more willing to tolerate their presence, but mostly they are, I'm running Linux and I can't be bothered to read the docs or try to figure things ous, can you set this up for me? type emails... > Note, the problem is not only for netmap: we have ipfw+dummynet > which is already cross platform, and as we bring pieces of the kernel > to userspace (see??? e.g. the various efforts related to bringing > up the network stack) we lose the tight bind to the os. Then maybe we need to create a special list for cross platform projects or something, but I don't notice any traffic wrt to ipfw or dummynet on -net... > > On 18 May 2014 16:02, Luigi Rizzo wrote: > > > Folks, i have two requests for you: > > > > > > 1. please do not complain about questions on this list related > > > to a core network-related FreeBSD subsystem (netmap, dummynet, > > > netgraph, tcp stack...) even if they are concerned with ports > > > to Linux or other OSes or to userspace. > > > > > > At least in principle, these topics _are_ relevant here precisely > > > because they relate to code whose main home is in the FreeBSD > > > source tree, and bugs or feature suggestions etc that may emerge > > > will directly benefit FreeBSD. > > > > > > 2. on the other hand, it would be good if people could avoid questions > > like > > > "give me step by step instructions on how to install/run/use X". > > > There are notes in README and Makefiles, something on the > > > web pages, and quick web searches should point you to previous > > > mailing list discussions on the same topic _before asking_. > > > > > > So in the specific case below i can understand a reply (perhaps a > > > private one) saying "have you done your homework (read documentation, > > > do a web search) ?", but "we only do FreeBSD here" does not sound > > > right to me. > > > > > > cheers > > > luigi > > > > > > On Sun, May 18, 2014 at 02:23:29PM +0200, XXX wrote: > > >> > I am using Fedora 18 with Intel 82599 NIC. > > >> .............^^^^^^^^^ > > >> > > >> Ask on a Linux/Fedora list then. This is a FreeBSD list. > > >> > > >> > I am new to netmap and want to experiment with it in the above > > environment. > > >> > Can somebody please advise what is the easiest way to go about the > > above > > >> > (installation etc.) so that I can concentrate on writing the userspace > > >> > programs using netmap quickly. -- John-Mark Gurney Voice: +1 415 225 5579 "All that I will do, has been done, All that I have, has not." From owner-freebsd-net@FreeBSD.ORG Wed Jul 23 04:14:46 2014 Return-Path: Delivered-To: freebsd-net@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 7F73C9E8; Wed, 23 Jul 2014 04:14:46 +0000 (UTC) Received: from mail-qg0-x235.google.com (mail-qg0-x235.google.com [IPv6:2607:f8b0:400d:c04::235]) (using TLSv1 with cipher ECDHE-RSA-RC4-SHA (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 271E32AE3; Wed, 23 Jul 2014 04:14:46 +0000 (UTC) Received: by mail-qg0-f53.google.com with SMTP id q107so765811qgd.12 for ; Tue, 22 Jul 2014 21:14:45 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:sender:in-reply-to:references:date:message-id:subject :from:to:content-type; bh=tpu4B2t2oDNnyb4j1JysCHy2t/bn+y/CG1JJSmoK62k=; b=iK+2n8yaWgALU9HJDW+ayeaPOdF0XuNGSLtdIGOKLBeU6XSfA1AtUzWzRGgsyQbZGu eJt2SkpL7uz09sTqoXuGMOxYXhIJStPyjsWU2lzUy5+YjFbjCXeE72zIq9oKr9i1p6kc 9PwZ8X2d5WG0FKJfzpWNvcvRcK5P40cOZ3evrsETn+S0oIX8RfnbMhfv2fSfivl9Qt2a IPPCr7KhgPQHnLkXaUXPFqm5HVHONm+93Ln3ScY59uv/BVInfnQht3OxbdcOPfjusvfP TaMwfAaSlNkdHwDV0EpEEn3G8iFnhD4iq3/TGaHSv85I0MIdvh+Lc3jlEnn29vz+TzmE sRuw== MIME-Version: 1.0 X-Received: by 10.224.97.65 with SMTP id k1mr64316404qan.28.1406088885048; Tue, 22 Jul 2014 21:14:45 -0700 (PDT) Sender: adrian.chadd@gmail.com Received: by 10.224.1.6 with HTTP; Tue, 22 Jul 2014 21:14:44 -0700 (PDT) In-Reply-To: <20140723040909.GJ43962@funkthat.com> References: <201405181223.s4ICNTAf097378@fire.js.berklix.net> <20140518230242.GA28117@onelab2.iet.unipi.it> <20140723040909.GJ43962@funkthat.com> Date: Tue, 22 Jul 2014 21:14:44 -0700 X-Google-Sender-Auth: eUTgAi-6wLhZ42SptKynmckoUDU Message-ID: Subject: Re: netmap and other discussions on freebsd-net: please be open minded. From: Adrian Chadd To: Luigi Rizzo , Adrian Chadd , FreeBSD Net Content-Type: text/plain; charset=UTF-8 X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 23 Jul 2014 04:14:46 -0000 We should just make netmap on freebsd .. much much nicer and easier to develop on. :) -a On 22 July 2014 21:09, John-Mark Gurney wrote: > Luigi Rizzo wrote this message on Mon, May 19, 2014 at 04:28 -0400: >> On Sun, May 18, 2014 at 7:49 PM, Adrian Chadd wrote: >> >> > Is there a netmap list that these questions (regardless of OS) could go to? >> >> ???no there isn't one. At the moment there isn't enough traffic >> to suggest that, and creating one is likely to generate >> traffic to both the specific list and freebsd-net. > > What volume of traffic do you think is acceptable for creating a > seperate list? > > The last few months I believe that there is enough traffic to justify > a seperate list... and too often the questions are not at all useful > to the FreeBSD community... If they were, I'd be more willing to > tolerate their presence, but mostly they are, I'm running Linux and > I can't be bothered to read the docs or try to figure things ous, can > you set this up for me? type emails... > >> Note, the problem is not only for netmap: we have ipfw+dummynet >> which is already cross platform, and as we bring pieces of the kernel >> to userspace (see??? e.g. the various efforts related to bringing >> up the network stack) we lose the tight bind to the os. > > Then maybe we need to create a special list for cross platform > projects or something, but I don't notice any traffic wrt to ipfw > or dummynet on -net... > >> > On 18 May 2014 16:02, Luigi Rizzo wrote: >> > > Folks, i have two requests for you: >> > > >> > > 1. please do not complain about questions on this list related >> > > to a core network-related FreeBSD subsystem (netmap, dummynet, >> > > netgraph, tcp stack...) even if they are concerned with ports >> > > to Linux or other OSes or to userspace. >> > > >> > > At least in principle, these topics _are_ relevant here precisely >> > > because they relate to code whose main home is in the FreeBSD >> > > source tree, and bugs or feature suggestions etc that may emerge >> > > will directly benefit FreeBSD. >> > > >> > > 2. on the other hand, it would be good if people could avoid questions >> > like >> > > "give me step by step instructions on how to install/run/use X". >> > > There are notes in README and Makefiles, something on the >> > > web pages, and quick web searches should point you to previous >> > > mailing list discussions on the same topic _before asking_. >> > > >> > > So in the specific case below i can understand a reply (perhaps a >> > > private one) saying "have you done your homework (read documentation, >> > > do a web search) ?", but "we only do FreeBSD here" does not sound >> > > right to me. >> > > >> > > cheers >> > > luigi >> > > >> > > On Sun, May 18, 2014 at 02:23:29PM +0200, XXX wrote: >> > >> > I am using Fedora 18 with Intel 82599 NIC. >> > >> .............^^^^^^^^^ >> > >> >> > >> Ask on a Linux/Fedora list then. This is a FreeBSD list. >> > >> >> > >> > I am new to netmap and want to experiment with it in the above >> > environment. >> > >> > Can somebody please advise what is the easiest way to go about the >> > above >> > >> > (installation etc.) so that I can concentrate on writing the userspace >> > >> > programs using netmap quickly. > > -- > John-Mark Gurney Voice: +1 415 225 5579 > > "All that I will do, has been done, All that I have, has not." From owner-freebsd-net@FreeBSD.ORG Wed Jul 23 12:37:41 2014 Return-Path: Delivered-To: freebsd-net@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id BD71F9E5 for ; Wed, 23 Jul 2014 12:37:41 +0000 (UTC) Received: from mail-pd0-x235.google.com (mail-pd0-x235.google.com [IPv6:2607:f8b0:400e:c02::235]) (using TLSv1 with cipher ECDHE-RSA-RC4-SHA (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 9571C2630 for ; Wed, 23 Jul 2014 12:37:41 +0000 (UTC) Received: by mail-pd0-f181.google.com with SMTP id g10so1555664pdj.40 for ; Wed, 23 Jul 2014 05:37:41 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=message-id:date:from:user-agent:mime-version:to:subject :content-type:content-transfer-encoding; bh=ph1QFoHosT/q9GqY5FI7CdsiDSlob+l8phGts95N8Xo=; b=vluhTwocryappYLUMkj2HiqzNzJ1oZjDX7MN5eavsZTJfoxlNf6xIRE/YWtNQ2V22B mPDNiuS+fW9v47dHNjQGnswecriX/WDEmg4mLsSCZRoiA+/KYMVYTG7TshuAWSsZ2Kc+ TMmP3st+KWBuloVAZlyC2Klz88Rp9BnAcNS4UyeouxUu/z7D5CqKJmz0+mPgy4oCThIA AjAOHE/u+8q5SJWPfbsPTi2vxylHSyGM3pUxp6oX2/MQ5RQQqoSfbpWWzHvC52drnRnK aae2nzGXMxGo8HNrxBug9KtbNSJMOntlRW0mLywx1wxngCWLcbzf4YwWXU7fOepYywRg xTDw== X-Received: by 10.68.125.226 with SMTP id mt2mr1428297pbb.6.1406119060975; Wed, 23 Jul 2014 05:37:40 -0700 (PDT) Received: from Peters-MacAir.local ([106.38.204.43]) by mx.google.com with ESMTPSA id b9sm3044867pdo.3.2014.07.23.05.37.39 for (version=TLSv1 cipher=ECDHE-RSA-RC4-SHA bits=128/128); Wed, 23 Jul 2014 05:37:40 -0700 (PDT) Message-ID: <53CFAC8F.8090404@gmail.com> Date: Wed, 23 Jul 2014 20:37:35 +0800 From: Xu Zhe User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10.9; rv:24.0) Gecko/20100101 Thunderbird/24.6.0 MIME-Version: 1.0 To: freebsd-net@freebsd.org Subject: Question on rx queue in ixgbe driver Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 7bit X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 23 Jul 2014 12:37:41 -0000 Hi, I am reading ixgbe driver of Freebsd and got some problems, which are described below. (1) Why rxd_tail does not equals to rxd_head? Here, rxd_tail/rxd_head is the value of dev.ix.0.queue0.rxd_tail/dev.ix.0.queue0.rxd_head from sysctl (take the first queue of ix0 as example). Actually, in most cases, rxd_head - rxd_tail == 1. Refers to the code, these values are actually next_to_refresh/next_to_check for each receive queue (though the sysctl implementation is read directly from the hardware registers I suppose). Why next_to_refresh is always one smaller than next_to_check (of course, when the latter is 0, the former is 2047 possibly, which is the size of rx ring - 1)? In my point of view, this means that we will always have one tiny mbuf (which is pointed by next_to_refresh) that is already checked (passed up to upper network stack) but not refreshed (not prepared for the next receive). It does not make sense? Or I missed anything important? (2) The init value of rxd_tail Even if (1) has no problem, when ixgbe device is inited, rxd_tail (or say, next_to_refresh) is set to zero (in function ixgbe_setup_receive_ring). I think it should be rxr->num_desc - 1. This should not matter much in the latest ixgbe driver, but it might cause old driver (ixgbe 2.5.8 at least) to double init the rx_ring descriptors (both in ixgbe_setup_receive_ring when ixgbe init up, and the first entry of ixgbe_refresh_mbufs of the first interrupt come). This is a case I met in my test environment. ========== Looking forward to any of your replies to help clarify my thoughts. Thanks in advance. Peter From owner-freebsd-net@FreeBSD.ORG Wed Jul 23 17:32:30 2014 Return-Path: Delivered-To: freebsd-net@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 92306235 for ; Wed, 23 Jul 2014 17:32:30 +0000 (UTC) Received: from mail-vc0-x235.google.com (mail-vc0-x235.google.com [IPv6:2607:f8b0:400c:c03::235]) (using TLSv1 with cipher ECDHE-RSA-RC4-SHA (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 50B19267C for ; Wed, 23 Jul 2014 17:32:30 +0000 (UTC) Received: by mail-vc0-f181.google.com with SMTP id lf12so2870865vcb.12 for ; Wed, 23 Jul 2014 10:32:29 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :cc:content-type; bh=bUcjd6PO8iN4JVZb0sM5t/GoY2IiefEgvfOL7nrVzG4=; b=tkfp6/p8IDHXyR2hSMgkDZodFzYO46M6bvGfZlM3KcKrwi6PPdIU+LwnIfr5PWPwQY tgeo+oJ2uQ47ax7VjWaOHDqUHs/PxQMHw5qUSfe8FJo2YnavXy44HEbJpJP6dhIQ9Wca e3kK17hBkEatYVUH7Uhlr9yFW53IGh7t6KGopR0YYqpMguOtHIgryO+K8BwLjkHEMw1j geweqikdqzBZiklBuQyl0T47ZL2UKKoIcu4QeBVvp9KIlB1xyp2WUH3oV0Q0PaPN61LN 2N5scj3hRxe7gFLAnRaxToz6vP2KWFMEpiq1Pk1CGL4Gl6bB2iiQGZY9huNyvbEQsFhg xIyA== MIME-Version: 1.0 X-Received: by 10.220.166.9 with SMTP id k9mr4549852vcy.20.1406136749307; Wed, 23 Jul 2014 10:32:29 -0700 (PDT) Received: by 10.220.194.129 with HTTP; Wed, 23 Jul 2014 10:32:29 -0700 (PDT) In-Reply-To: <53CFAC8F.8090404@gmail.com> References: <53CFAC8F.8090404@gmail.com> Date: Wed, 23 Jul 2014 10:32:29 -0700 Message-ID: Subject: Re: Question on rx queue in ixgbe driver From: Jack Vogel To: Xu Zhe Content-Type: text/plain; charset=ISO-8859-1 X-Content-Filtered-By: Mailman/MimeDel 2.1.18 Cc: FreeBSD Net X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 23 Jul 2014 17:32:30 -0000 HEAD and TAIL are actually hw registers, the driver as it is configured these days never (and cannot) modify HEAD, There is an option to use the HEAD register as a method of managing the RX side (called Head Writeback), but in ixgbe this is not used, rather we rely on the DD bit of a descriptor to be written back by the hardware to indicate an operation is complete. The span between HEAD and TAIL is the extent of usable buffers for the hw. In the starting state having them equal indicates a clean slate, however once operating TAIL will trail HEAD, the range between them being "uncleaned" The next_to_check and next_to_refresh indices are used by the driver to manage the ring. I designed them to allow a seperated operation of cleaning and of refreshing the buffers. In old drivers these two were always in lock step, and you could never send a packet to the stack until you FIRST did the mbuf allocation, then if that failed for any reason the packet was actually dropped in order to keep that precious mbuf cluster. Now refresh can be deferred as many as 8 while cleaning. These two indices are each exclusively controlled by those two operations, and its only refresh, by writing the TAIL register, that actually limits what the hw engine can do, since it will not go beyond TAIL. There is no "should be" on those indices, they are set up so things operate correctly, and they do :) Hope this clarifies things somewhat? Jack On Wed, Jul 23, 2014 at 5:37 AM, Xu Zhe wrote: > Hi, > > I am reading ixgbe driver of Freebsd and got some problems, which are > described below. > > (1) Why rxd_tail does not equals to rxd_head? > > Here, rxd_tail/rxd_head is the value of > dev.ix.0.queue0.rxd_tail/dev.ix.0.queue0.rxd_head from sysctl (take the > first > queue of ix0 as example). > > Actually, in most cases, rxd_head - rxd_tail == 1. > > Refers to the code, these values are actually next_to_refresh/next_to_check > for each receive queue (though the sysctl implementation is read directly > from > the hardware registers I suppose). Why next_to_refresh is always one > smaller > than next_to_check (of course, when the latter is 0, the former is 2047 > possibly, which is the size of rx ring - 1)? In my point of view, this > means > that we will always have one tiny mbuf (which is pointed by > next_to_refresh) > that is already checked (passed up to upper network stack) but not > refreshed > (not prepared for the next receive). It does not make sense? Or I missed > anything important? > > (2) The init value of rxd_tail > > Even if (1) has no problem, when ixgbe device is inited, rxd_tail (or say, > next_to_refresh) is set to zero (in function ixgbe_setup_receive_ring). I > think it should be rxr->num_desc - 1. > > This should not matter much in the latest ixgbe driver, but it might cause > old > driver (ixgbe 2.5.8 at least) to double init the rx_ring descriptors (both > in > ixgbe_setup_receive_ring when ixgbe init up, and the first entry of > ixgbe_refresh_mbufs of the first interrupt come). This is a case I met in > my > test environment. > > ========== > > Looking forward to any of your replies to help clarify my thoughts. Thanks > in > advance. > > Peter > _______________________________________________ > freebsd-net@freebsd.org mailing list > http://lists.freebsd.org/mailman/listinfo/freebsd-net > To unsubscribe, send any mail to "freebsd-net-unsubscribe@freebsd.org" > From owner-freebsd-net@FreeBSD.ORG Wed Jul 23 23:31:42 2014 Return-Path: Delivered-To: freebsd-net@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id EE22F4F3; Wed, 23 Jul 2014 23:31:42 +0000 (UTC) Received: from h2.funkthat.com (gate2.funkthat.com [208.87.223.18]) (using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits)) (Client CN "funkthat.com", Issuer "funkthat.com" (not verified)) by mx1.freebsd.org (Postfix) with ESMTPS id ADF6C2767; Wed, 23 Jul 2014 23:31:42 +0000 (UTC) Received: from h2.funkthat.com (localhost [127.0.0.1]) by h2.funkthat.com (8.14.3/8.14.3) with ESMTP id s6NNVfkE085893 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-SHA bits=256 verify=NO); Wed, 23 Jul 2014 16:31:41 -0700 (PDT) (envelope-from jmg@h2.funkthat.com) Received: (from jmg@localhost) by h2.funkthat.com (8.14.3/8.14.3/Submit) id s6NNVeVf085892; Wed, 23 Jul 2014 16:31:40 -0700 (PDT) (envelope-from jmg) Date: Wed, 23 Jul 2014 16:31:40 -0700 From: John-Mark Gurney To: Rick Macklem Subject: Re: [RFC] Allow m_dup() to use JUMBO clusters Message-ID: <20140723233139.GR43962@funkthat.com> Mail-Followup-To: Rick Macklem , Hans Petter Selasky , freebsd-net@freebsd.org, freebsd-current@freebsd.org References: <20140708005234.GJ45513@funkthat.com> <1065824414.8871880.1404850207148.JavaMail.root@uoguelph.ca> Mime-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <1065824414.8871880.1404850207148.JavaMail.root@uoguelph.ca> User-Agent: Mutt/1.4.2.3i X-Operating-System: FreeBSD 7.2-RELEASE i386 X-PGP-Fingerprint: 54BA 873B 6515 3F10 9E88 9322 9CB1 8F74 6D3F A396 X-Files: The truth is out there X-URL: http://resnet.uoregon.edu/~gurney_j/ X-Resume: http://resnet.uoregon.edu/~gurney_j/resume.html X-TipJar: bitcoin:13Qmb6AeTgQecazTWph4XasEsP7nGRbAPE X-to-the-FBI-CIA-and-NSA: HI! HOW YA DOIN? can i haz chizburger? X-Greylist: Sender IP whitelisted, not delayed by milter-greylist-4.2.2 (h2.funkthat.com [127.0.0.1]); Wed, 23 Jul 2014 16:31:41 -0700 (PDT) Cc: Hans Petter Selasky , freebsd-net@freebsd.org, freebsd-current@freebsd.org X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 23 Jul 2014 23:31:43 -0000 Rick Macklem wrote this message on Tue, Jul 08, 2014 at 16:10 -0400: > I tried: > m = m_getjcl(M_NOWAIT..M_JUMPAGESIZE); > if (m == NULL) > m = getjcl(M_WAITOK..MCLBYTES); > when I was experimenting with MJUMPAGESIZE clusters for NFS and what happened > was the thread looped in the first m_getjcl() instead of returning NULL. > It is about 12 layers of function calls deep and most fail/return NULL, but > somewhere one of them decides to "try again". I didn't locate the location > of that and don't know if it would be safe to change it so that m_getjcl() > returns NULL for this case. So, I took a quick look at this, and I see a weird issue... In mb_ctor_mbuf in kern_mbuf.c, the how argument is passed to m_init instead of flags... how and flags SHOULD be the same, but it could be that uma could change how to something else and we are loosing _NOWAIT.. Also, m_getjcl is calling two different uma_zalloc_arg's, know which one would be useful... You should be able to verify this w/ dtrace pretty easily in your case.. Brief call path: m_getjcl -> uma_zalloc_arg -> mb_ctor_mbuf -> m_init -> m_pkthdr_init -> mac_mbuf_init -> m_tag_get or mac_mbuf_tag_init... I didn't trace it down beyond mac_mbuf_init.. Verifying that mac_mbuf_init still has M_NOWAIT would be good... -- John-Mark Gurney Voice: +1 415 225 5579 "All that I will do, has been done, All that I have, has not." From owner-freebsd-net@FreeBSD.ORG Thu Jul 24 03:28:02 2014 Return-Path: Delivered-To: freebsd-net@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 1ECF09E0 for ; Thu, 24 Jul 2014 03:28:02 +0000 (UTC) Received: from ns.kevlo.org (220-135-115-6.HINET-IP.hinet.net [220.135.115.6]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client CN "ns.kevlo.org", Issuer "ns.kevlo.org" (not verified)) by mx1.freebsd.org (Postfix) with ESMTPS id 93F742C1B for ; Thu, 24 Jul 2014 03:28:00 +0000 (UTC) Received: from ns.kevlo.org (localhost [127.0.0.1]) by ns.kevlo.org (8.14.8/8.14.8) with ESMTP id s6O3Qt6A080410 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-GCM-SHA384 bits=256 verify=NO) for ; Thu, 24 Jul 2014 11:26:56 +0800 (CST) (envelope-from kevlo@ns.kevlo.org) Received: (from kevlo@localhost) by ns.kevlo.org (8.14.8/8.14.8/Submit) id s6O3QtXX080409 for freebsd-net@freebsd.org; Thu, 24 Jul 2014 11:26:55 +0800 (CST) (envelope-from kevlo) Date: Thu, 24 Jul 2014 11:26:55 +0800 From: Kevin Lo To: freebsd-net@freebsd.org Subject: [PATCH] merge 'struct ip6protosw' and 'struct protosw' into one Message-ID: <20140724032655.GA80400@ns.kevlo.org> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline User-Agent: Mutt/1.5.22 (2013-10-16) X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 24 Jul 2014 03:28:02 -0000 Hi, The diff [1] merges 'struct ip6protosw' and 'struct protosw' into one, since we don't need a separate structure which is shared between ipv4 and ipv6. The key difference between the two is the definition of pr_input function. [1] https://phabric.freebsd.org/D476 Kevin From owner-freebsd-net@FreeBSD.ORG Thu Jul 24 07:25:45 2014 Return-Path: Delivered-To: freebsd-net@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 2B6DC401 for ; Thu, 24 Jul 2014 07:25:45 +0000 (UTC) Received: from mail-pd0-x231.google.com (mail-pd0-x231.google.com [IPv6:2607:f8b0:400e:c02::231]) (using TLSv1 with cipher ECDHE-RSA-RC4-SHA (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id F16FC2E12 for ; Thu, 24 Jul 2014 07:25:44 +0000 (UTC) Received: by mail-pd0-f177.google.com with SMTP id p10so3138148pdj.22 for ; Thu, 24 Jul 2014 00:25:44 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=message-id:date:from:user-agent:mime-version:to:cc:subject :references:in-reply-to:content-type; bh=z0ZKyVZDp4QfHRxBIOGmJQ7b0MqMb7ieX4FO+TzMivI=; b=wTIDi3XcqXDqDmCTEjeOZRvyLJJRWZZS6nJesPqnR8q9pO4yeMXLyT4SsZ/A2jh4ZK +oWBe78tS/EXu05zKHgJmhqiTxnFo5KIdmP8geHJNgR1+b8whDjWBYzxkbOSOYwKrOR6 kcrYUtP3PqJlbvr7/p8EXtJKBf4LZksWXavepwINEFyomFnVuNeRefyTUGC/PFX6PwqA JIr12c51GcHtD6rtiaiB6ai/dqgP5/cuF9WO/0EXigSmUf9GBhgFpwrP+H28+YC5y/c0 sV8yAUGN52bYP8m3tzi6hVJNlDNmMl/jj5gzKH/rJqRWX9XUax/v2kv2HhtNJa36Wpbt ofNA== X-Received: by 10.70.140.35 with SMTP id rd3mr7961123pdb.67.1406186744309; Thu, 24 Jul 2014 00:25:44 -0700 (PDT) Received: from Peters-MacAir.local ([106.38.204.43]) by mx.google.com with ESMTPSA id nt15sm6136332pdb.63.2014.07.24.00.25.41 for (version=TLSv1 cipher=ECDHE-RSA-RC4-SHA bits=128/128); Thu, 24 Jul 2014 00:25:43 -0700 (PDT) Message-ID: <53D0B4F1.2030408@gmail.com> Date: Thu, 24 Jul 2014 15:25:37 +0800 From: Xu Zhe User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10.9; rv:24.0) Gecko/20100101 Thunderbird/24.6.0 MIME-Version: 1.0 To: Jack Vogel Subject: Re: Question on rx queue in ixgbe driver References: <53CFAC8F.8090404@gmail.com> In-Reply-To: Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit X-Content-Filtered-By: Mailman/MimeDel 2.1.18 Cc: FreeBSD Net X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 24 Jul 2014 07:25:45 -0000 Hi, Jack, Thanks for the quick response and detailed explaination. :) I think now I understand why tail is always one less than head, if it is the rule to play from the hardware datasheet. As for the init value, I still not quite clear why their starting state should be all zeros. I had traced in real test with ixgbe driver version 2.5.8 that, when the specific ix0 inited (set the first IP), ixgbe_msix_que would be triggered once (even no packet received I suppose, don't know why for now), and rx_ring descriptors are re-inited since ixgbe_rx_unrefreshed would assume there are unrefreshed descriptors (though there is not, since rx_head == rx_tail == 0). Then during refresh in ixgbe_refresh_mbufs(), the loop will go over 2048 times to re-init all the descriptors. IMHO, another way to change this situation might be in ixgbe_rx_unrefreshed: when head equals to tail, we could return zero since it's during start phase. After all, this should not a big problem though, since it only happens when interface first init (or down -> up), and it should not exist in the latest ixgbe 2.5.15 driver (in the latest driver, I found that ixgbe_msix_que would return directly when failed to find DRV_RUNNING flag). Peter 于 14-7-24 1:32, Jack Vogel 写道: > HEAD and TAIL are actually hw registers, the driver as it is configured these > days never (and cannot) modify HEAD, There is an option to use the HEAD > register as a method of managing the RX side (called Head Writeback), but > in ixgbe this is not used, rather we rely on the DD bit of a descriptor to be > written back by the hardware to indicate an operation is complete. > > The span between HEAD and TAIL is the extent of usable buffers for the hw. > In the starting state having them equal indicates a clean slate, however once > operating TAIL will trail HEAD, the range between them being "uncleaned" > > The next_to_check and next_to_refresh indices are used by the driver to manage > the ring. I designed them to allow a seperated operation of cleaning and of > refreshing > the buffers. In old drivers these two were always in lock step, and you > could never > send a packet to the stack until you FIRST did the mbuf allocation, then if that > failed for any reason the packet was actually dropped in order to keep that > precious > mbuf cluster. Now refresh can be deferred as many as 8 while cleaning. > > These two indices are each exclusively controlled by those two operations, > and its > only refresh, by writing the TAIL register, that actually limits what the hw > engine > can do, since it will not go beyond TAIL. > > There is no "should be" on those indices, they are set up so things operate > correctly, and they do :) > > Hope this clarifies things somewhat? > > Jack > > > > > On Wed, Jul 23, 2014 at 5:37 AM, Xu Zhe > wrote: > > Hi, > > I am reading ixgbe driver of Freebsd and got some problems, which are > described below. > > (1) Why rxd_tail does not equals to rxd_head? > > Here, rxd_tail/rxd_head is the value of > dev.ix.0.queue0.rxd_tail/dev.ix.0.queue0.rxd_head from sysctl (take the > first > queue of ix0 as example). > > Actually, in most cases, rxd_head - rxd_tail == 1. > > Refers to the code, these values are actually next_to_refresh/next_to_check > for each receive queue (though the sysctl implementation is read > directly from > the hardware registers I suppose). Why next_to_refresh is always one smaller > than next_to_check (of course, when the latter is 0, the former is 2047 > possibly, which is the size of rx ring - 1)? In my point of view, this means > that we will always have one tiny mbuf (which is pointed by next_to_refresh) > that is already checked (passed up to upper network stack) but not refreshed > (not prepared for the next receive). It does not make sense? Or I missed > anything important? > > (2) The init value of rxd_tail > > Even if (1) has no problem, when ixgbe device is inited, rxd_tail (or say, > next_to_refresh) is set to zero (in function ixgbe_setup_receive_ring). I > think it should be rxr->num_desc - 1. > > This should not matter much in the latest ixgbe driver, but it might > cause old > driver (ixgbe 2.5.8 at least) to double init the rx_ring descriptors > (both in > ixgbe_setup_receive_ring when ixgbe init up, and the first entry of > ixgbe_refresh_mbufs of the first interrupt come). This is a case I met in my > test environment. > > ========== > > Looking forward to any of your replies to help clarify my thoughts. > Thanks in > advance. > > Peter > _______________________________________________ > freebsd-net@freebsd.org mailing list > http://lists.freebsd.org/mailman/listinfo/freebsd-net > To unsubscribe, send any mail to "freebsd-net-unsubscribe@freebsd.org > " > > From owner-freebsd-net@FreeBSD.ORG Thu Jul 24 09:24:26 2014 Return-Path: Delivered-To: freebsd-net@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 9334F2B4; Thu, 24 Jul 2014 09:24:26 +0000 (UTC) Received: from forward9l.mail.yandex.net (forward9l.mail.yandex.net [IPv6:2a02:6b8:0:1819::9]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client CN "forwards.mail.yandex.net", Issuer "Certum Level IV CA" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4B8AD2966; Thu, 24 Jul 2014 09:24:26 +0000 (UTC) Received: from smtp18.mail.yandex.net (smtp18.mail.yandex.net [95.108.252.18]) by forward9l.mail.yandex.net (Yandex) with ESMTP id CEC62E60DA6; Thu, 24 Jul 2014 13:24:14 +0400 (MSK) Received: from smtp18.mail.yandex.net (localhost [127.0.0.1]) by smtp18.mail.yandex.net (Yandex) with ESMTP id 5C5F618A0141; Thu, 24 Jul 2014 13:24:14 +0400 (MSK) Received: from 84.201.164.118-vpn.dhcp.yndx.net (84.201.164.118-vpn.dhcp.yndx.net [84.201.164.118]) by smtp18.mail.yandex.net (nwsmtp/Yandex) with ESMTPSA id qvfdzBKTH2-OEeqfbJO; Thu, 24 Jul 2014 13:24:14 +0400 (using TLSv1 with cipher AES128-SHA (128/128 bits)) (Client certificate not present) X-Yandex-Uniq: 0a5c922e-ed72-45fa-b924-76f322b676d6 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=yandex.ru; s=mail; t=1406193854; bh=d0Rto8DAfEAHge4qI6PRo1ATtA2rYfAen3Mj+vABqoo=; h=Message-ID:Date:From:User-Agent:MIME-Version:To:Subject: References:In-Reply-To:X-Enigmail-Version:Content-Type: Content-Transfer-Encoding; b=LfYxtgpOrEeuSKRW5h/aH3r0cU5kBLTE4+Q+PkoflQz+gsZMBniDRjMWpEr07COjO R8xB3xhGcxHvN7ylxD/55fUDfWb1CMUsedLvLE04W7bY8FF10gZTglkUGtshUUetXB Yd//tWz2StgEVnOrsYgy+zPOy8DOwYdnbxvfPRJY= Authentication-Results: smtp18.mail.yandex.net; dkim=pass header.i=@yandex.ru Message-ID: <53D0D0B2.6080600@yandex.ru> Date: Thu, 24 Jul 2014 13:24:02 +0400 From: "Andrey V. Elsukov" User-Agent: Mozilla/5.0 (X11; FreeBSD amd64; rv:24.0) Gecko/20100101 Thunderbird/24.5.0 MIME-Version: 1.0 To: John Jasen , FreeBSD Net , "Alexander V. Chernikov" Subject: Re: fastforward/routing: a 3 million packet-per-second system? References: <53CE80DD.9090109@gmail.com> In-Reply-To: <53CE80DD.9090109@gmail.com> X-Enigmail-Version: 1.6 Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: 7bit X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 24 Jul 2014 09:24:26 -0000 On 22.07.2014 19:18, John Jasen wrote: > Feedback and/or tips and tricks more than welcome. > > Outstanding questions: > > Would increasing the number of processor cores help? AFAIR, increasing the number of cores will lead to worse results. With patched and tuned FreeBSD we able to route (with fastforwarding) about 7 Mpps IPv4 and 2.5Mpps IPv6. But the stock system is far from even half of this results. -- WBR, Andrey V. Elsukov From owner-freebsd-net@FreeBSD.ORG Thu Jul 24 09:50:16 2014 Return-Path: Delivered-To: freebsd-net@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id ECBE0E89; Thu, 24 Jul 2014 09:50:15 +0000 (UTC) Received: from mail-pd0-x22c.google.com (mail-pd0-x22c.google.com [IPv6:2607:f8b0:400e:c02::22c]) (using TLSv1 with cipher ECDHE-RSA-RC4-SHA (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id BD34D2BCF; Thu, 24 Jul 2014 09:50:15 +0000 (UTC) Received: by mail-pd0-f172.google.com with SMTP id ft15so3346484pdb.17 for ; Thu, 24 Jul 2014 02:50:14 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :cc:content-type; bh=vbKJ8o11JuqDrYaecDf/cKEnVPLUOOJA3/WJTBiJumo=; b=K+dz4NNgOYahETWh5OV6u9EnnnHm9laqdFfDDMkIqEfd8b1+5QS/QYMLTUzntqyBg6 iYpFHdTkVx6CY3dfuk9TGifgRCvpsT2GByREMKwOHyWTjMp1IKLaXKosPF3I64MaHa4F 3UzUrJRhMIapp3R1LgXWF8dXHI62PnhdbOkNVMPbYp7oiPK4B49qBs7OBoxdm2w2VrJW C57mAWDZyw7jIkOkZNu5W/IOS1WOBbuwqT3IrWo1c2Jphp1nl+a6sXuaKDwDUWLHQRS0 fX7E9evMJwTYpLB1+VJ1vMI5vRtPuvQO549iyPa53BDnvDDJsRy8UjZG03QkdnGUSo83 oYXw== MIME-Version: 1.0 X-Received: by 10.68.139.225 with SMTP id rb1mr8675582pbb.153.1406194948796; Thu, 24 Jul 2014 02:42:28 -0700 (PDT) Received: by 10.70.74.170 with HTTP; Thu, 24 Jul 2014 02:42:28 -0700 (PDT) Received: by 10.70.74.170 with HTTP; Thu, 24 Jul 2014 02:42:28 -0700 (PDT) In-Reply-To: <53D0D0B2.6080600@yandex.ru> References: <53CE80DD.9090109@gmail.com> <53D0D0B2.6080600@yandex.ru> Date: Thu, 24 Jul 2014 12:42:28 +0300 Message-ID: Subject: Re: fastforward/routing: a 3 million packet-per-second system? From: Sami Halabi To: "Andrey V. Elsukov" Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: quoted-printable X-Content-Filtered-By: Mailman/MimeDel 2.1.18 Cc: freebsd-net@freebsd.org, "Alexander V. Chernikov" , John Jasen X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 24 Jul 2014 09:50:16 -0000 Hi, Can you share tununings and patches? Thanks in advance, Sami =D7=91=D7=AA=D7=90=D7=A8=D7=99=D7=9A 24 =D7=91=D7=99=D7=95=D7=9C 2014 12:24= , "Andrey V. Elsukov" =D7=9B=D7=AA=D7=91: > On 22.07.2014 19:18, John Jasen wrote: > > Feedback and/or tips and tricks more than welcome. > > > > Outstanding questions: > > > > Would increasing the number of processor cores help? > > AFAIR, increasing the number of cores will lead to worse results. > With patched and tuned FreeBSD we able to route (with fastforwarding) > about 7 Mpps IPv4 and 2.5Mpps IPv6. But the stock system is far from > even half of this results. > > -- > WBR, Andrey V. Elsukov > _______________________________________________ > freebsd-net@freebsd.org mailing list > http://lists.freebsd.org/mailman/listinfo/freebsd-net > To unsubscribe, send any mail to "freebsd-net-unsubscribe@freebsd.org" > From owner-freebsd-net@FreeBSD.ORG Thu Jul 24 12:47:23 2014 Return-Path: Delivered-To: freebsd-net@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 98510B39; Thu, 24 Jul 2014 12:47:23 +0000 (UTC) Received: from mail-ig0-x230.google.com (mail-ig0-x230.google.com [IPv6:2607:f8b0:4001:c05::230]) (using TLSv1 with cipher ECDHE-RSA-RC4-SHA (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 5AEBA2C4A; Thu, 24 Jul 2014 12:47:23 +0000 (UTC) Received: by mail-ig0-f176.google.com with SMTP id hn18so6410056igb.15 for ; Thu, 24 Jul 2014 05:47:22 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=message-id:date:from:user-agent:mime-version:to:subject:references :in-reply-to:content-type:content-transfer-encoding; bh=gpfxwxhRjaefu/+T2wa7GMhOmui7U++OJ8sIdyCnW+c=; b=zww+/kZcGfMvIgrzEkpWVzBGoARGCyWwjEZQZcRfdM+X9nsH40LVkTAR6l3xwRGyuv Av26R69khmPtzixHYS71hYDi86oyXi2t3DEhNcQbR4db/zUNMjj3cV9vJBGeTBAEN5S+ KDmrYMLOyruG/GQDDSRfCUX5EYetliWyFjLZHSmErAZSqTZcRi+wv5sb7VQwGBRaWCkI Eq+yaYlz0mI4Y3H/elrQ+IoOYK8mNoYwqAGcWkKECL1QmM2spmfXva05fzH2RjpqBvQh BKDFIlTbDlwhjX96XXEfI9R48qWmNdVnzzakdjMJgXmxp0csqTWaEqtavMjrXRm8LHFc dsZw== X-Received: by 10.50.136.167 with SMTP id qb7mr5669418igb.31.1406206042800; Thu, 24 Jul 2014 05:47:22 -0700 (PDT) Received: from [10.0.0.215] ([96.234.167.12]) by mx.google.com with ESMTPSA id ik1sm69831023igb.18.2014.07.24.05.47.18 for (version=TLSv1 cipher=ECDHE-RSA-RC4-SHA bits=128/128); Thu, 24 Jul 2014 05:47:19 -0700 (PDT) Message-ID: <53D10055.1050304@gmail.com> Date: Thu, 24 Jul 2014 08:47:17 -0400 From: John Jasen User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:24.0) Gecko/20100101 Thunderbird/24.6.0 MIME-Version: 1.0 To: "Andrey V. Elsukov" , FreeBSD Net , "Alexander V. Chernikov" Subject: Re: fastforward/routing: a 3 million packet-per-second system? References: <53CE80DD.9090109@gmail.com> <53D0D0B2.6080600@yandex.ru> In-Reply-To: <53D0D0B2.6080600@yandex.ru> X-Enigmail-Version: 1.5.2 Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: 7bit X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 24 Jul 2014 12:47:23 -0000 On 07/24/2014 05:24 AM, Andrey V. Elsukov wrote: > On 22.07.2014 19:18, John Jasen wrote: >> Feedback and/or tips and tricks more than welcome. >> >> Outstanding questions: >> >> Would increasing the number of processor cores help? > AFAIR, increasing the number of cores will lead to worse results. > With patched and tuned FreeBSD we able to route (with fastforwarding) > about 7 Mpps IPv4 and 2.5Mpps IPv6. But the stock system is far from > even half of this results. > Increasing the physical CPU count can (and probably will) result in performance degradation. However, from what I've seen, balancing IRQs across the cores on a single physical CPU seems to help. I am curious as well, as to how you achieved 7 Mpps. Can you share the system specs, the patches and tuning? Thanks! -- John Jasen (jjasen@gmail.com) From owner-freebsd-net@FreeBSD.ORG Thu Jul 24 13:07:00 2014 Return-Path: Delivered-To: freebsd-net@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 76FA6C6 for ; Thu, 24 Jul 2014 13:07:00 +0000 (UTC) Received: from mail-ig0-x22c.google.com (mail-ig0-x22c.google.com [IPv6:2607:f8b0:4001:c05::22c]) (using TLSv1 with cipher ECDHE-RSA-RC4-SHA (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 424F12E27 for ; Thu, 24 Jul 2014 13:07:00 +0000 (UTC) Received: by mail-ig0-f172.google.com with SMTP id h15so6417386igd.17 for ; Thu, 24 Jul 2014 06:06:59 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=message-id:date:from:user-agent:mime-version:cc:subject:references :in-reply-to:content-type:content-transfer-encoding; bh=dnXr8UfeGbEg1LfHO5cDD9Ou19N2YILj4Cffe0lVhho=; b=FGP/QoCxPgZco/gbV7+yPI3weIuNLY4kYUGDN/YSRmB5aNA9n4ZRljE8+mAYov0Z1j PUhj1WZzOk281+3e1Faaql5ngxXkWKInv+2m725oZXjJTdUkW+Hv7/xoeS5KOTFhFPuT dgCOx/KJe644DhWZAKh+jPAq5hTiMbCaxVt49vQHD3OXiJlQA275W1A8XtxE9iqnFFCH oIJFtBiZJUaetRm6osVsioLu5CbD3jbcIES25LnoY5cY5CFRQvYkNG3WHIV1xyYjCdVV hGRx0WnKTtQLWZmbg4/wjQoFEWhijhF7P1YtE9gcxjHJZPXimy1ezAk8heM0uPhHAfoX Z4DA== X-Received: by 10.43.156.77 with SMTP id ll13mr12275021icc.81.1406207219566; Thu, 24 Jul 2014 06:06:59 -0700 (PDT) Received: from [10.0.0.215] ([96.234.167.12]) by mx.google.com with ESMTPSA id d4sm23192418igc.5.2014.07.24.06.06.57 for (version=TLSv1 cipher=ECDHE-RSA-RC4-SHA bits=128/128); Thu, 24 Jul 2014 06:06:58 -0700 (PDT) Message-ID: <53D104F0.6010103@gmail.com> Date: Thu, 24 Jul 2014 09:06:56 -0400 From: John Jasen User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:24.0) Gecko/20100101 Thunderbird/24.6.0 MIME-Version: 1.0 CC: FreeBSD Net Subject: Re: fastforward/routing: a 3 million packet-per-second system? References: <53CE80DD.9090109@gmail.com> In-Reply-To: X-Enigmail-Version: 1.5.2 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 7bit X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 24 Jul 2014 13:07:00 -0000 As a follow-up, from man cxgbe(4), I changed the following /boot/loader.conf settings. # in discussions with cxgbe maintainer and testing. higher than 48 seems to cause kernel panics. hw.cxgbe.ntxq10g=48 # note, to go above 30-some odd tx queues, you need to set the following to turn off toe queues hw.cxgbe.toecaps_allowed=0 # testing hw.cxgbe.cong_drop options hw.cxgbe.cong_drop=1 Changing ntxq10g and toecaps improved pmc metrics, which previously had shown a lot of time in transmit states, Changing cong_drop is ... probably not a good idea for production environments. However, testing with 1, which allows drop as opposed to tx_pause, I saw two things: a) packet per second rate reliably crossed over 4 million packets per second, as measured via netstat -db 1. b) there were several occasions where the test system became completely unresponsive over the ssh sessions I had established to run netstat, top, et al. On 07/22/2014 02:07 PM, Adrian Chadd wrote: > Hi! > > Well, what's missing is some dtrace/pmc/lockdebugging investigations > into the system to see where it's currently maxing out at. > > I wonder if you're seeing contention on the transmit paths as drivers > queue frames from one set of driver threads/queues to another > potentially completely different set of driver transmit > threads/queues. > > > > > -a > > > On 22 July 2014 08:18, John Jasen wrote: >> Feedback and/or tips and tricks more than welcome. >> >> Outstanding questions: >> >> Would increasing the number of processor cores help? >> >> Would a system where both processor QPI ports connect to each other >> mitigate QPI bottlenecks? >> >> Are there further performance optimizations I am missing? >> >> Server Description: >> >> The system in question is a Dell Poweredge R820, 16GB of RAM, and two >> Intel(R) Xeon(R) CPU E5-4610 0 @ 2.40GHz. >> >> Onboard, in a 16x PCIe slot, I have one Chelsio T-580-CR two-port 40GbE >> NIC, and in an 8x slot, another T-580-CR dual port. >> >> I am running FreeBSD 10.0-STABLE. >> >> BIOS tweaks: >> >> Hyperthreading (or Logical Processors) is turned off. >> Memory Node Interleaving is turned off, but did not appear to impact >> performance. >> >> /boot/loader.conf contents: >> #for CARP+PF testing >> carp_load="YES" >> #load cxgbe drivers. >> cxgbe_load="YES" >> #maxthreads appears to not exceed CPU. >> net.isr.maxthreads=12 >> #bindthreads may be indicated when using cpuset(1) on interrupts >> net.isr.bindthreads=1 >> #random guess based on googling >> net.isr.maxqlimit=60480 >> net.link.ifqmaxlen=90000 >> #discussions with cxgbe maintainer and list led me to trying this. >> Allows more interrupts >> #to be fixed to CPUs, which in some cases, improves interrupt balancing. >> hw.cxgbe.ntxq10g=16 >> hw.cxgbe.nrxq10g=16 >> >> /etc/sysctl.conf contents: >> >> #the following is also enabled by rc.conf gateway_enable. >> net.inet.ip.fastforwarding=1 >> #recommendations from BSD router project >> kern.random.sys.harvest.ethernet=0 >> kern.random.sys.harvest.point_to_point=0 >> kern.random.sys.harvest.interrupt=0 >> #probably should be removed, as cxgbe does not seem to affect/be >> affected by irq storm settings >> hw.intr_storm_threshold=25000000 >> #based on Calomel.Org performance suggestions. 4x40GbE, seemed >> reasonable to use 100GbE settings >> kern.ipc.maxsockbuf=1258291200 >> net.inet.tcp.recvbuf_max=1258291200 >> net.inet.tcp.sendbuf_max=1258291200 >> #attempting to play with ULE scheduler, making it serve packets versus >> netstat >> kern.sched.slice=1 >> kern.sched.interact=1 >> >> /etc/rc.conf contains: >> >> hostname="fbge1" >> #should remove, especially given below duplicate entry >> ifconfig_igb0="DHCP" >> sshd_enable="YES" >> #ntpd_enable="YES" >> # Set dumpdev to "AUTO" to enable crash dumps, "NO" to disable >> dumpdev="AUTO" >> # OpenBSD PF options to play with later. very bad for raw packet rates. >> #pf_enable="YES" >> #pflog_enable="YES" >> # enable packet forwarding >> # these enable forwarding and fastforwarding sysctls. inet6 does not >> have fastforward >> gateway_enable="YES" >> ipv6_gateway_enable="YES" >> # enable OpenBSD ftp-proxy >> # should comment out until actively playing with PF >> ftpproxy_enable="YES" >> #left in place, commented out from prior testing >> #ifconfig_mlxen1="inet 172.16.2.1 netmask 255.255.255.0 mtu 9000" >> #ifconfig_mlxen0="inet 172.16.1.1 netmask 255.255.255.0 mtu 9000" >> #ifconfig_mlxen3="inet 172.16.7.1 netmask 255.255.255.0 mtu 9000" >> #ifconfig_mlxen2="inet 172.16.8.1 netmask 255.255.255.0 mtu 9000" >> # -lro and -tso options added per mailing list suggestion from Bjoern A. >> Zeeb (bzeeb-lists at lists.zabbadoz.net) >> ifconfig_cxl0="inet 172.16.3.1 netmask 255.255.255.0 mtu 9000 -lro -tso up" >> ifconfig_cxl1="inet 172.16.4.1 netmask 255.255.255.0 mtu 9000 -lro -tso up" >> ifconfig_cxl2="inet 172.16.5.1 netmask 255.255.255.0 mtu 9000 -lro -tso up" >> ifconfig_cxl3="inet 172.16.6.1 netmask 255.255.255.0 mtu 9000 -lro -tso up" >> # aliases instead of reconfiguring test clients. See above commented out >> entries >> ifconfig_cxl0_alias0="172.16.7.1 netmask 255.255.255.0" >> ifconfig_cxl1_alias0="172.16.8.1 netmask 255.255.255.0" >> ifconfig_cxl2_alias0="172.16.1.1 netmask 255.255.255.0" >> ifconfig_cxl3_alias0="172.16.2.1 netmask 255.255.255.0" >> # for remote monitoring/admin of the test device >> ifconfig_igb0="inet 172.30.60.60 netmask 255.255.0.0" >> >> Additional configurations: >> cpuset-chelsio-6cpu-high >> # Original provided by Navdeep Parhar >> # takes vmstat -ai output into a list, and assigns interrupts in order to >> # the available CPU cores. >> # Modified: to assign only to the 'high CPUs', ie: on core1. >> # See: http://lists.freebsd.org/pipermail/freebsd-net/2014-July/039317.html >> #!/usr/local/bin/bash >> ncpu=12 >> irqlist=$(vmstat -ia | egrep 't4nex|t5nex|cxgbc' | cut -f1 -d: | cut -c4-) >> i=6 >> for irq in $irqlist; do >> cpuset -l $i -x $irq >> i=$((i+1)) >> [ $i -ge $ncpu ] && i=6 >> done >> >> Client Description: >> >> Two Intel(R) Xeon(R) CPU E5-2670 v2 @ 2.50GHz processors >> 64 GB ram >> Mellanox Technologies MT27500 Family [ConnectX-3] >> Centos 6.4 with updates >> iperf3 installed from yum repositories: iperf3-3.0.3-3.el6.x86_64 >> >> Test setup: >> >> I've found about 3 streams between Centos clients is about the best way >> to get the most out of them. >> Above certain points, the -b flag does not change results. >> -N is an artifact from using TCP >> -l is needed, as -M doesn't work for UDP. >> >> I usually use launch scripts similar to the following: >> >> for i in `seq 41 60`; do ssh loader$i "export TIME=120; export >> STREAMS=1; export PORT=52$i; export PKT=64; export RATE=2000m; >> /root/iperf-test-8port-udp" & done >> >> The scripts execute the following on each host. >> >> #!/bin/bash >> PORT1=$PORT >> PORT2=$(($PORT+1000)) >> PORT3=$(($PORT+2000)) >> iperf3 -c loader41-40gbe -u -b 10000m -i 0 -N -l $PKT -t$TIME >> -P$STREAMS -p$PORT1 & >> iperf3 -c loader42-40gbe -u -b 10000m -i 0 -N -l $PKT -t$TIME >> -P$STREAMS -p$PORT1 & >> iperf3 -c loader43-40gbe -u -b 10000m -i 0 -N -l $PKT -t$TIME >> -P$STREAMS -p$PORT1 & >> ... (through all clients and all three ports) ... >> iperf3 -c loader60-40gbe -u -b 10000m -i 0 -N -l $PKT -t$TIME >> -P$STREAMS -p$PORT3 & >> >> >> Results: >> >> Summarized, netstat -w 1 -q 240 -bd, run through: >> cat test4-tuning | egrep -v {'packets | input '} | awk '{ipackets+=$1} >> {idrops+=$3} {opackets+=$5} {odrops+=$9} END {print "input " >> ipackets/NR, "idrops " idrops/NR, "opackets " opackets/NR, "odrops " >> odrops/NR}' >> >> input 1.10662e+07 idrops 8.01783e+06 opackets 3.04516e+06 odrops 3152.4 >> >> Snapshot of raw output: >> >> input (Total) output >> packets errs idrops bytes packets errs bytes colls drops >> 11189148 0 7462453 1230805216 3725006 0 409750710 0 799 >> 10527505 0 6746901 1158024978 3779096 0 415700708 0 127 >> 10606163 0 6850760 1166676673 3751780 0 412695761 0 1535 >> 10749324 0 7132014 1182425799 3617558 0 397930956 0 5972 >> 10695667 0 7022717 1176521907 3669342 0 403627236 0 1461 >> 10441173 0 6762134 1148528662 3675048 0 404255540 0 6021 >> 10683773 0 7005635 1175215014 3676962 0 404465671 0 2606 >> 10869859 0 7208696 1195683372 3658432 0 402427698 0 979 >> 11948989 0 8310926 1314387881 3633773 0 399714986 0 725 >> 12426195 0 8864415 1366877194 3562311 0 391853156 0 2762 >> 13006059 0 9432389 1430661751 3570067 0 392706552 0 5158 >> 12822243 0 9098871 1410443600 3715177 0 408668500 0 4064 >> 13317864 0 9683602 1464961374 3632156 0 399536131 0 3684 >> 13701905 0 10182562 1507207982 3523101 0 387540859 0 >> 8690 >> 13820227 0 10244870 1520221820 3562038 0 391823322 0 >> 2426 >> 14437060 0 10955483 1588073033 3480105 0 382810557 0 >> 2619 >> 14518471 0 11119573 1597028105 3397439 0 373717355 0 >> 5691 >> 14890287 0 11675003 1637926521 3199812 0 351978304 0 >> 11007 >> 14923610 0 11749091 1641594441 3171436 0 348857468 0 >> 7389 >> 14738704 0 11609730 1621254991 3117715 0 342948394 0 >> 2597 >> 14753975 0 11549735 1622935026 3207393 0 352812846 0 >> 4798 >> >> >> >> >> >> _______________________________________________ >> freebsd-net@freebsd.org mailing list >> http://lists.freebsd.org/mailman/listinfo/freebsd-net >> To unsubscribe, send any mail to "freebsd-net-unsubscribe@freebsd.org" From owner-freebsd-net@FreeBSD.ORG Thu Jul 24 13:11:40 2014 Return-Path: Delivered-To: freebsd-net@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id B45C31A2 for ; Thu, 24 Jul 2014 13:11:40 +0000 (UTC) Received: from mail.ijs.si (mail.ijs.si [IPv6:2001:1470:ff80::25]) by mx1.freebsd.org (Postfix) with ESMTP id 3F81A2ECD for ; Thu, 24 Jul 2014 13:11:40 +0000 (UTC) Received: from amavis-proxy-ori.ijs.si (localhost [IPv6:::1]) by mail.ijs.si (Postfix) with ESMTP id 3hJv7t46Txz1Ff for ; Thu, 24 Jul 2014 15:11:38 +0200 (CEST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=ijs.si; h= user-agent:message-id:organization:subject:subject:from:from :date:date:content-transfer-encoding:content-type:content-type :mime-version:received:received:received:received; s=jakla2; t= 1406207491; x=1408799492; bh=RrG4pjMDx5PN8CKCrMO3lFi4NlyKQGeCTmg MQy2XWRw=; b=Qsz2LL9Oo4+uEbs3BwfYdNoYiKS5aoai6VTbD0vn0bfjVEcRyEj TzXaGsvVJUjI/ZpGlEtKKLPSDWjK4xEA6KvnNS8AR2WFWV63RrRvkf7D+WxJQCQT lZ2byE3lNb+EvVPzFzkZweZ2ul7t1uBIluJ68tNTRZAhHRoBmZ7GR72A= X-Virus-Scanned: amavisd-new at ijs.si Received: from mail.ijs.si ([IPv6:::1]) by amavis-proxy-ori.ijs.si (mail.ijs.si [IPv6:::1]) (amavisd-new, port 10012) with ESMTP id rKTpe7YUraWE for ; Thu, 24 Jul 2014 15:11:31 +0200 (CEST) Received: from mildred.ijs.si (mailbox.ijs.si [IPv6:2001:1470:ff80::143:1]) by mail.ijs.si (Postfix) with ESMTP for ; Thu, 24 Jul 2014 15:11:31 +0200 (CEST) Received: from neli.ijs.si (neli.ijs.si [IPv6:2001:1470:ff80:88:21c:c0ff:feb1:8c91]) by mildred.ijs.si (Postfix) with ESMTP id 3hJv7k6klzzgJ for ; Thu, 24 Jul 2014 15:11:30 +0200 (CEST) Received: from neli.ijs.si ([2001:1470:ff80:88:21c:c0ff:feb1:8c91]) by neli.ijs.si with HTTP (HTTP/1.1 POST); Thu, 24 Jul 2014 15:11:30 +0200 MIME-Version: 1.0 Content-Type: text/plain; charset=US-ASCII; format=flowed Content-Transfer-Encoding: 7bit Date: Thu, 24 Jul 2014 15:11:30 +0200 From: Mark Martinec To: freebsd-net@freebsd.org Subject: Bumping up a default net.graph.maxdata to avoid "Write failed: Cannot allocate memory" Organization: J. Stefan Institute Message-ID: <17db180e1fa763f41dd0051780741b9f@mailbox.ijs.si> X-Sender: Mark.Martinec+freebsd@ijs.si User-Agent: Roundcube Webmail/1.0.1 X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 24 Jul 2014 13:11:40 -0000 Syncing zfs snapshots across the net using 'zfs send' over ssh started failing one day with ssh reporting "Write failed: Cannot allocate memory" after transferring about 15 to 25 GB of data (as it turned out this snapshot was larger than usual). Neither of the two hosts were particularly low on memory, the receiving end was running 10.0-STABLE amd64, network between the two was a gigabit ethernet, interface em. The problem was repeatable at will. Simplifying the experiment to a: ssh zfs send | wc -c also ended up with the same "Write failed: Cannot allocate memory" on the receiving side after transferring about 20 GB. Doing some other experiments ruled out a potential blame from zfs send. As it turned out (luckily for me, after banging my had over it) I'm not the only one with this problem - it was reported three years ago: http://lists.freebsd.org/pipermail/freebsd-emulation/2011-July/008971.html http://lists.freebsd.org/pipermail/freebsd-stable/2011-July/063322.html And yes, I too had a smallish virtual host running under VirtualBox on this receiving machine, which was mostly idling. That virtual host was not involved in any of these experiments. So the problem is that NetGraph was running out of space for data queue and net.graph.maxdata needed to be bumped up from a default of 512. My current setting is net.graph.maxdata=2048 and this suffices for reliable transfer of huge files (> 60 GB) over ssh. After one such transfer the vmstat -z reports: ITEM SIZE LIMIT USED FREE REQ FAIL SLEEP [...] NetGraph items: 72, 4123, 0, 527, 28, 0, 0 NetGraph data items: 72, 2077, 0, 1023,65650892, 0, 0 while previously the FAIL count was in the hundreds, just as in the problem report from July 2011. And the issue is not limited to ssh, others have reported the same over ftp. Btw, the 'ngctl list' shows: There are 5 total nodes: Name: ngctl78040 Type: socket ID: 00000010 Num hooks: 0 Name: em0 Type: ether ID: 00000001 Num hooks: 2 Name: em1 Type: ether ID: 00000002 Num hooks: 0 Name: vboxnet0 Type: ether ID: 00000003 Num hooks: 0 Name: vboxnetflt_em0 Type: vboxnetflt ID: 00000004 Num hooks: 2 Unfortunately the 2011 thread remained suspended in the air, with no action, neither in documentation nor bumping up the default queue limit. So to save more people from bumping into the same problem, puzzled over a mystery "Write failed: Cannot allocate memory" failure, I'd like to suggest bumping up the default value of net.graph.maxdata, or at least documenting the fact in the handbook. Mark From owner-freebsd-net@FreeBSD.ORG Thu Jul 24 17:22:33 2014 Return-Path: Delivered-To: freebsd-net@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 5B104C12 for ; Thu, 24 Jul 2014 17:22:33 +0000 (UTC) Received: from mail-lb0-x22e.google.com (mail-lb0-x22e.google.com [IPv6:2a00:1450:4010:c04::22e]) (using TLSv1 with cipher ECDHE-RSA-RC4-SHA (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id C34B02753 for ; Thu, 24 Jul 2014 17:22:32 +0000 (UTC) Received: by mail-lb0-f174.google.com with SMTP id c11so2488645lbj.5 for ; Thu, 24 Jul 2014 10:22:30 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=clockworksquid.com; s=google; h=mime-version:sender:in-reply-to:references:from:date:message-id :subject:to:cc:content-type; bh=4VUWXB6784D73KiWlfny/MN6JyUrGXJm55PudRjPtQc=; b=lQOO4rmk0SwUP76qrIMtD8FvIRC4K4j9nMPU/YMjN6ljLDoWw8VdBXDau1qjSdrVPm YjXrjTc6Lt1AtoJNAn4v8I0tgFF6oEROrSudOWPsBpm84zRxbSpW+VAiVWn7C2S+lIvu XBgVcx5yg07D8Pg+tUzup8c/fEysHfCKgfBaU= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20130820; h=x-gm-message-state:mime-version:sender:in-reply-to:references:from :date:message-id:subject:to:cc:content-type; bh=4VUWXB6784D73KiWlfny/MN6JyUrGXJm55PudRjPtQc=; b=LFoqmvp+LEEJf+Or2+ZKTBFKL05jtFAFEOKDvYPnYhXivgFIixSWK54HrzmkDM4L7K pf23kYpPaSWKV1yNR6lKcUKbURQIi9MzFWdAgrNXiBNtj0XEMVAiKjrRKAqpk+8TnbZo nR4JrBUwryAsY+O5UtC+/9D+b5mpaaSGalzTMLyD2YZDz0B6oTSSnwxe6d/0Lx9Xu3vF aJgQcjT53OXKvXjAUtwofxq0XKRJMz5S/jiH9q5QNJY82F2kHkLChcuKbAd7VHAeE0EM UODttA/GAbPJ1vzAIa3/g6jeMjiwdj0ZRrTND+9HCoM0f+WmDzQ1OvvbYivnjpZRc7xX Q/tQ== X-Gm-Message-State: ALoCoQnRKrtsCXFqHBrKtPtmz7XqdPGcIzXXgvH5z9gyRcwx0kG1XPiHkkmZrywF1tDpWT8oMDsN X-Received: by 10.112.16.199 with SMTP id i7mr10947697lbd.5.1406222550570; Thu, 24 Jul 2014 10:22:30 -0700 (PDT) MIME-Version: 1.0 Sender: juli@clockworksquid.com Received: by 10.152.229.225 with HTTP; Thu, 24 Jul 2014 10:22:10 -0700 (PDT) In-Reply-To: References: <53CE80DD.9090109@gmail.com> <20140722174117.GC43962@funkthat.com> <53CEAF40.7030406@gmail.com> From: Juli Mallett Date: Thu, 24 Jul 2014 10:22:10 -0700 X-Google-Sender-Auth: ajajZbuxv9ypVizlFbPJ01jedgs Message-ID: Subject: Re: fastforward/routing: a 3 million packet-per-second system? To: Carlos Ferreira Content-Type: text/plain; charset=UTF-8 X-Content-Filtered-By: Mailman/MimeDel 2.1.18 Cc: FreeBSD Net , John Jasen X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 24 Jul 2014 17:22:33 -0000 On Tue, Jul 22, 2014 at 12:23 PM, Carlos Ferreira wrote: > I think the results presented at the paper are regarding one port sending > or receiving at 14.88Mpps. Using several ports at the same time will surely > give much lower results. But then again, if one wants 8, 16, 24 or even > more ports at 10Gb/s, then it should look for FPGA implementations. This is incorrect and misleading speculation, assuming your hardware is not set up badly in some way. With netmap it is quite easy to have additional threads handling other ports, so long as you have enough real cores to run those threads on (ideally pinned, and with nothing else running.) If you've got the RAM and the CPU, 8 ports of 10GbE with netmap on FreeBSD is not at all infeasible. The question is always how much work you need to be able to do per-packet and whether you're able to do that work in an efficient way that fits with the model. So something with a slightly-specific or at least sneaky architecture is very important here, so that you can just swap buffers rather than having to do a gratuitous copy. And a very careful design even just for parsing the packets, let alone the process of doing the routing. Using several ports at the same time on a single CPU absolutely gives lower results, but someone with 8 10GbE ports can hopefully pretty well handle providing 16 cores to get the job done and still have a meaningful control plane. The idea that netmap or similar solutions would be bottlenecked otherwise is wrong, unless there's something wrong with your hardware (not so uncommon as one might like.) (For cache performance, etc., you really don't want to use hyperthreads for something as timing-sensitive and data-hungry as this kind of packet processing; your results may vary.) Juli. On 22 July 2014 19:36, John Jasen wrote: > > > > > On 07/22/2014 01:41 PM, John-Mark Gurney wrote: > > > John Jasen wrote this message on Tue, Jul 22, 2014 at 11:18 -0400: > > >> Feedback and/or tips and tricks more than welcome. > > > You should look at netmap if you really want high PPS routing... > > > > Originally, I assumed an interface supporting netmap was required, but > > the manpage disabuses me of this. Besides, I think the Chelsio cards got > > netmap recently. > > > > I presume either the use of bridge in tools/netmap, or vale-ctl in the > > same location would start providing me sufficient clue on this? > > Unfortunately, both seem to be pretty short on verbosity ... More clue > > is always welcome! > > > > > > _______________________________________________ > > freebsd-net@freebsd.org mailing list > > http://lists.freebsd.org/mailman/listinfo/freebsd-net > > To unsubscribe, send any mail to "freebsd-net-unsubscribe@freebsd.org" > > > > > > -- > > Carlos Miguel Ferreira > Researcher at Telecommunications Institute > Aveiro - Portugal > Work E-mail - cmf@av.it.pt > Skype & GTalk -> carlosmf.pt@gmail.com > LinkedIn -> http://www.linkedin.com/in/carlosmferreira > _______________________________________________ > freebsd-net@freebsd.org mailing list > http://lists.freebsd.org/mailman/listinfo/freebsd-net > To unsubscribe, send any mail to "freebsd-net-unsubscribe@freebsd.org" > From owner-freebsd-net@FreeBSD.ORG Thu Jul 24 18:00:03 2014 Return-Path: Delivered-To: freebsd-net@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 3313AACC; Thu, 24 Jul 2014 18:00:03 +0000 (UTC) Received: from mail-qg0-x22f.google.com (mail-qg0-x22f.google.com [IPv6:2607:f8b0:400d:c04::22f]) (using TLSv1 with cipher ECDHE-RSA-RC4-SHA (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id D3A942A92; Thu, 24 Jul 2014 18:00:02 +0000 (UTC) Received: by mail-qg0-f47.google.com with SMTP id i50so3649779qgf.20 for ; Thu, 24 Jul 2014 11:00:02 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:in-reply-to:references:from:date:message-id:subject:to :cc:content-type; bh=4mBDh1lstxnfdIt6gVPxuD9UPj3+Dx3CjcsTOFDnsto=; b=okSK4oZJXkqhw4/M4pYBRia5mLNgxwsIBW4iscEF7ABGnjUV5ciqtz9baWBr990TuE DkY/cDyUuK/tTZCJHNYP+ezaJH/evkoon+E0Ku24gR09WU6xEV2e75kSQoVOpXO9E7+s 22uASXsfIlc0xQ7q1/aRILXYKz5Km3+FIP/uP79lzL2rywGeUpGCUGX8GTgaoI8YHOX8 JX1+ld9Ou7VPJM0VWuGg+em4wOhXGeJucTMcecvv5hDcsjgdOwztXwrdPtDrtt01iq/j 4s5zhrOlvnj3hD7d6aWmxayyfhW9JAGJJvSvfxdnOrqASMDEAsU3lJFyDkb6cTkEpn2i 1+vA== X-Received: by 10.140.109.164 with SMTP id l33mr17028514qgf.72.1406224801952; Thu, 24 Jul 2014 11:00:01 -0700 (PDT) MIME-Version: 1.0 Received: by 10.96.74.69 with HTTP; Thu, 24 Jul 2014 10:59:21 -0700 (PDT) In-Reply-To: References: <53CE80DD.9090109@gmail.com> <20140722174117.GC43962@funkthat.com> <53CEAF40.7030406@gmail.com> From: Carlos Ferreira Date: Thu, 24 Jul 2014 18:59:21 +0100 Message-ID: Subject: Re: fastforward/routing: a 3 million packet-per-second system? To: Juli Mallett Content-Type: text/plain; charset=UTF-8 X-Content-Filtered-By: Mailman/MimeDel 2.1.18 Cc: FreeBSD Net , John Jasen X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 24 Jul 2014 18:00:03 -0000 Juliet, a network equipment with eight SFP 10Gb Ethernet port is a small solution. Network operators usually have solutions with 24, 48 or even more ports mounted in racks. Check out the OLT solutions that Cisco has to offer. Thinking that more cores always solves the problem is incorrect because you will end up strangling the bus at the motherboard. Also, there's always a limit to how many generic cpu cores you can add to a single system and the power they need to operate. There's a good reason why the optical network equipments cards use FPGA's. It's not just deterministic execution, power consumption, reliability and maintenance cost but also that dedicated digital circuit implementation will always be faster than a generic cpu implementation. I simply cannot imagine someone installing a network solution with eight 10GbE ports with a generic processor with 16 cores. But then again, I'm humble enough to accept and study a solution of that kind, and the reasons why someone would want to implement such solution in an operator network. If you have seen such solution, please tell me where I can find it. On 24 July 2014 18:22, Juli Mallett wrote: > On Tue, Jul 22, 2014 at 12:23 PM, Carlos Ferreira > wrote: > >> I think the results presented at the paper are regarding one port sending >> or receiving at 14.88Mpps. Using several ports at the same time will >> surely >> give much lower results. But then again, if one wants 8, 16, 24 or even >> more ports at 10Gb/s, then it should look for FPGA implementations. > > > This is incorrect and misleading speculation, assuming your hardware is > not set up badly in some way. With netmap it is quite easy to have > additional threads handling other ports, so long as you have enough real > cores to run those threads on (ideally pinned, and with nothing else > running.) If you've got the RAM and the CPU, 8 ports of 10GbE with netmap > on FreeBSD is not at all infeasible. The question is always how much work > you need to be able to do per-packet and whether you're able to do that > work in an efficient way that fits with the model. So something with a > slightly-specific or at least sneaky architecture is very important here, > so that you can just swap buffers rather than having to do a gratuitous > copy. And a very careful design even just for parsing the packets, let > alone the process of doing the routing. > > Using several ports at the same time on a single CPU absolutely gives > lower results, but someone with 8 10GbE ports can hopefully pretty well > handle providing 16 cores to get the job done and still have a meaningful > control plane. The idea that netmap or similar solutions would be > bottlenecked otherwise is wrong, unless there's something wrong with your > hardware (not so uncommon as one might like.) (For cache performance, > etc., you really don't want to use hyperthreads for something as > timing-sensitive and data-hungry as this kind of packet processing; your > results may vary.) > > Juli. > > On 22 July 2014 19:36, John Jasen wrote: >> >> > >> > On 07/22/2014 01:41 PM, John-Mark Gurney wrote: >> > > John Jasen wrote this message on Tue, Jul 22, 2014 at 11:18 -0400: >> > >> Feedback and/or tips and tricks more than welcome. >> > > You should look at netmap if you really want high PPS routing... >> > >> > Originally, I assumed an interface supporting netmap was required, but >> > the manpage disabuses me of this. Besides, I think the Chelsio cards got >> > netmap recently. >> > >> > I presume either the use of bridge in tools/netmap, or vale-ctl in the >> > same location would start providing me sufficient clue on this? >> > Unfortunately, both seem to be pretty short on verbosity ... More clue >> > is always welcome! >> > >> > >> > _______________________________________________ >> > freebsd-net@freebsd.org mailing list >> > http://lists.freebsd.org/mailman/listinfo/freebsd-net >> > To unsubscribe, send any mail to "freebsd-net-unsubscribe@freebsd.org" >> > >> >> >> >> -- >> >> Carlos Miguel Ferreira >> Researcher at Telecommunications Institute >> Aveiro - Portugal >> Work E-mail - cmf@av.it.pt >> Skype & GTalk -> carlosmf.pt@gmail.com >> LinkedIn -> http://www.linkedin.com/in/carlosmferreira >> _______________________________________________ >> freebsd-net@freebsd.org mailing list >> http://lists.freebsd.org/mailman/listinfo/freebsd-net >> To unsubscribe, send any mail to "freebsd-net-unsubscribe@freebsd.org" >> > > -- Carlos Miguel Ferreira Researcher at Telecommunications Institute Aveiro - Portugal Work E-mail - cmf@av.it.pt Skype & GTalk -> carlosmf.pt@gmail.com LinkedIn -> http://www.linkedin.com/in/carlosmferreira From owner-freebsd-net@FreeBSD.ORG Fri Jul 25 16:20:41 2014 Return-Path: Delivered-To: freebsd-net@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id D30E6FE4 for ; Fri, 25 Jul 2014 16:20:41 +0000 (UTC) Received: from mail-lb0-x236.google.com (mail-lb0-x236.google.com [IPv6:2a00:1450:4010:c04::236]) (using TLSv1 with cipher ECDHE-RSA-RC4-SHA (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4995622A7 for ; Fri, 25 Jul 2014 16:20:41 +0000 (UTC) Received: by mail-lb0-f182.google.com with SMTP id z11so3642422lbi.13 for ; Fri, 25 Jul 2014 09:20:39 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :content-type; bh=QtsUebttApqMJwZhzChJnTwvYmrdrsiM5Kn7L8uOXS4=; b=k+Tr7ZJPweoabsHAkANPiwhd5iTvSjtripAO/rfYkIqFtdSjAKTYr+SjlZG/xCDeJ7 1KBe/Zm9SsJue2RbyrsAcY4Nu7er//a0ZU7KM82DokPVb3SLHPuxxU1GLXSXhUblqKBw 9sciXBxFWeUe4bEfRE3wcbzJwRr3x2lysr5v7NUfkay4+rudM6whJIjm2g2j+8RZ/7XY 7RkPVjAzlYRF/5vE3SB2Vh51ZmHZWx94krKsP6zrPz9Mv4w6A0dioBwPyM3x9RT3I0Gu 60HdnRFUPaW61/+xr6pSX1rW+IWEyS/yJPPFDIkCpmjC6dvP2ksXL0M6hiRD968LvOjF TzSg== MIME-Version: 1.0 X-Received: by 10.152.5.167 with SMTP id t7mr17848216lat.54.1406305239138; Fri, 25 Jul 2014 09:20:39 -0700 (PDT) Received: by 10.114.176.106 with HTTP; Fri, 25 Jul 2014 09:20:38 -0700 (PDT) In-Reply-To: <53CE80DD.9090109@gmail.com> References: <53CE80DD.9090109@gmail.com> Date: Fri, 25 Jul 2014 12:20:38 -0400 Message-ID: Subject: Re: fastforward/routing: a 3 million packet-per-second system? From: John Jasen To: FreeBSD Net Content-Type: text/plain; charset=UTF-8 X-Content-Filtered-By: Mailman/MimeDel 2.1.18 X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 25 Jul 2014 16:20:41 -0000 Based on advice I received, I've collected lock profile debugging output, and pmcannotate'd data from the system while it was processing about 3 million packets/second. Combined, the files are about 325k in size, so I'll submit highlights here. I can provide the raw files to interested parties privately. pmcannotate summary output: grep ^Profile pmcannotate.20140725 Profile trace for function: __rw_rlock() [17.04%] Profile trace for function: __mtx_unlock_flags() [9.10%] Profile trace for function: _rw_runlock_cookie() [7.67%] Profile trace for function: sched_idletd() [5.73%] Profile trace for function: memcpy() [5.64%] Profile trace for function: bcopy() [5.04%] Profile trace for function: bcmp() [5.01%] Profile trace for function: __mtx_lock_flags() [3.66%] Profile trace for function: t4_eth_tx() [3.25%] Profile trace for function: lock_profile_release_lock() [2.73%] Profile trace for function: ip_fastforward() [2.68%] Profile trace for function: ether_output() [2.50%] Profile trace for function: get_scatter_segment() [1.75%] Profile trace for function: rn_match() [1.74%] Profile trace for function: _mtx_lock_spin_cookie() [1.53%] Profile trace for function: lock_profile_obtain_lock_success() [1.49%] Profile trace for function: cxgbe_transmit() [1.37%] Profile trace for function: uma_zalloc_arg() [1.31%] Profile trace for function: bzero() [1.30%] Profile trace for function: service_iq() [1.26%] Profile trace for function: ether_nh_input() [1.23%] Profile trace for function: __mtx_lock_sleep() [1.19%] Profile trace for function: arpresolve() [1.07%] Profile trace for function: uma_zfree_arg() [0.95%] Profile trace for function: reclaim_tx_descs() [0.87%] Profile trace for function: _mtx_trylock_flags_() [0.80%] Profile trace for function: bounce_bus_dmamap_load_buffer() [0.72%] Profile trace for function: ether_demux() [0.64%] Profile trace for function: mb_ctor_mbuf() [0.63%] Profile trace for function: rtalloc1_fib() [0.54%] sysctl debug.lock.prof.stats summary: (some of the highest hit counts, especially in cnt_hold: 7 17 3271389 2258698 63952832 0 0 0 6761237 /usr/src/sys/netinet/in_rmx.c:114 (sleep mutex:rtentry) 7 13 11543138 470545 63952832 0 0 0 815777 /usr/src/sys/net/route.c:439 (sleep mutex:rtentry) 6 15 3943582 1545195 63952779 0 0 0 3439254 /usr/src/sys/netinet/ip_fastfwd.c:593 (sleep mutex:rtentry On Tue, Jul 22, 2014 at 11:18 AM, John Jasen wrote: > Feedback and/or tips and tricks more than welcome. > From owner-freebsd-net@FreeBSD.ORG Fri Jul 25 17:14:01 2014 Return-Path: Delivered-To: freebsd-net@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 95832E4E for ; Fri, 25 Jul 2014 17:14:01 +0000 (UTC) Received: from vps1.elischer.org (vps1.elischer.org [204.109.63.16]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client CN "vps1.elischer.org", Issuer "CA Cert Signing Authority" (not verified)) by mx1.freebsd.org (Postfix) with ESMTPS id 741AE2805 for ; Fri, 25 Jul 2014 17:14:01 +0000 (UTC) Received: from Julian-MBP3.local (ppp121-45-236-203.lns20.per1.internode.on.net [121.45.236.203]) (authenticated bits=0) by vps1.elischer.org (8.14.9/8.14.9) with ESMTP id s6PHDt2A004654 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES128-SHA bits=128 verify=NO); Fri, 25 Jul 2014 10:13:58 -0700 (PDT) (envelope-from julian@freebsd.org) Message-ID: <53D2904E.7090509@freebsd.org> Date: Sat, 26 Jul 2014 01:13:50 +0800 From: Julian Elischer User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10.9; rv:24.0) Gecko/20100101 Thunderbird/24.6.0 MIME-Version: 1.0 To: John Jasen , FreeBSD Net Subject: Re: fastforward/routing: a 3 million packet-per-second system? References: <53CE80DD.9090109@gmail.com> In-Reply-To: <53CE80DD.9090109@gmail.com> Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 25 Jul 2014 17:14:01 -0000 On 7/22/14, 11:18 PM, John Jasen wrote: > Feedback and/or tips and tricks more than welcome. > > Outstanding questions: > > Would increasing the number of processor cores help? > > Would a system where both processor QPI ports connect to each other > mitigate QPI bottlenecks? > > Are there further performance optimizations I am missing? > > Server Description: > > The system in question is a Dell Poweredge R820, 16GB of RAM, and two > Intel(R) Xeon(R) CPU E5-4610 0 @ 2.40GHz. > > Onboard, in a 16x PCIe slot, I have one Chelsio T-580-CR two-port 40GbE > NIC, and in an 8x slot, another T-580-CR dual port. > > I am running FreeBSD 10.0-STABLE. > > BIOS tweaks: > > Hyperthreading (or Logical Processors) is turned off. while this used to be a win the newer processors have got this (more) right so that logical processors can be a real win now. Make sure you KNOW you need this turned off by doing tests. > Memory Node Interleaving is turned off, but did not appear to impact > performance. > > /boot/loader.conf contents: > #for CARP+PF testing > carp_load="YES" > #load cxgbe drivers. > cxgbe_load="YES" > #maxthreads appears to not exceed CPU. > net.isr.maxthreads=12 > #bindthreads may be indicated when using cpuset(1) on interrupts > net.isr.bindthreads=1 > #random guess based on googling > net.isr.maxqlimit=60480 > net.link.ifqmaxlen=90000 > #discussions with cxgbe maintainer and list led me to trying this. > Allows more interrupts > #to be fixed to CPUs, which in some cases, improves interrupt balancing. > hw.cxgbe.ntxq10g=16 > hw.cxgbe.nrxq10g=16 > > /etc/sysctl.conf contents: > > #the following is also enabled by rc.conf gateway_enable. > net.inet.ip.fastforwarding=1 > #recommendations from BSD router project > kern.random.sys.harvest.ethernet=0 > kern.random.sys.harvest.point_to_point=0 > kern.random.sys.harvest.interrupt=0 > #probably should be removed, as cxgbe does not seem to affect/be > affected by irq storm settings > hw.intr_storm_threshold=25000000 > #based on Calomel.Org performance suggestions. 4x40GbE, seemed > reasonable to use 100GbE settings > kern.ipc.maxsockbuf=1258291200 > net.inet.tcp.recvbuf_max=1258291200 > net.inet.tcp.sendbuf_max=1258291200 > #attempting to play with ULE scheduler, making it serve packets versus > netstat > kern.sched.slice=1 > kern.sched.interact=1 > > /etc/rc.conf contains: > > hostname="fbge1" > #should remove, especially given below duplicate entry > ifconfig_igb0="DHCP" > sshd_enable="YES" > #ntpd_enable="YES" > # Set dumpdev to "AUTO" to enable crash dumps, "NO" to disable > dumpdev="AUTO" > # OpenBSD PF options to play with later. very bad for raw packet rates. > #pf_enable="YES" > #pflog_enable="YES" > # enable packet forwarding > # these enable forwarding and fastforwarding sysctls. inet6 does not > have fastforward > gateway_enable="YES" > ipv6_gateway_enable="YES" > # enable OpenBSD ftp-proxy > # should comment out until actively playing with PF > ftpproxy_enable="YES" > #left in place, commented out from prior testing > #ifconfig_mlxen1="inet 172.16.2.1 netmask 255.255.255.0 mtu 9000" > #ifconfig_mlxen0="inet 172.16.1.1 netmask 255.255.255.0 mtu 9000" > #ifconfig_mlxen3="inet 172.16.7.1 netmask 255.255.255.0 mtu 9000" > #ifconfig_mlxen2="inet 172.16.8.1 netmask 255.255.255.0 mtu 9000" > # -lro and -tso options added per mailing list suggestion from Bjoern A. > Zeeb (bzeeb-lists at lists.zabbadoz.net) > ifconfig_cxl0="inet 172.16.3.1 netmask 255.255.255.0 mtu 9000 -lro -tso up" > ifconfig_cxl1="inet 172.16.4.1 netmask 255.255.255.0 mtu 9000 -lro -tso up" > ifconfig_cxl2="inet 172.16.5.1 netmask 255.255.255.0 mtu 9000 -lro -tso up" > ifconfig_cxl3="inet 172.16.6.1 netmask 255.255.255.0 mtu 9000 -lro -tso up" > # aliases instead of reconfiguring test clients. See above commented out > entries > ifconfig_cxl0_alias0="172.16.7.1 netmask 255.255.255.0" > ifconfig_cxl1_alias0="172.16.8.1 netmask 255.255.255.0" > ifconfig_cxl2_alias0="172.16.1.1 netmask 255.255.255.0" > ifconfig_cxl3_alias0="172.16.2.1 netmask 255.255.255.0" > # for remote monitoring/admin of the test device > ifconfig_igb0="inet 172.30.60.60 netmask 255.255.0.0" > > Additional configurations: > cpuset-chelsio-6cpu-high > # Original provided by Navdeep Parhar > # takes vmstat -ai output into a list, and assigns interrupts in order to > # the available CPU cores. > # Modified: to assign only to the 'high CPUs', ie: on core1. > # See: http://lists.freebsd.org/pipermail/freebsd-net/2014-July/039317.html > #!/usr/local/bin/bash > ncpu=12 > irqlist=$(vmstat -ia | egrep 't4nex|t5nex|cxgbc' | cut -f1 -d: | cut -c4-) > i=6 > for irq in $irqlist; do > cpuset -l $i -x $irq > i=$((i+1)) > [ $i -ge $ncpu ] && i=6 > done > > Client Description: > > Two Intel(R) Xeon(R) CPU E5-2670 v2 @ 2.50GHz processors > 64 GB ram > Mellanox Technologies MT27500 Family [ConnectX-3] > Centos 6.4 with updates > iperf3 installed from yum repositories: iperf3-3.0.3-3.el6.x86_64 > > Test setup: > > I've found about 3 streams between Centos clients is about the best way > to get the most out of them. > Above certain points, the -b flag does not change results. > -N is an artifact from using TCP > -l is needed, as -M doesn't work for UDP. > > I usually use launch scripts similar to the following: > > for i in `seq 41 60`; do ssh loader$i "export TIME=120; export > STREAMS=1; export PORT=52$i; export PKT=64; export RATE=2000m; > /root/iperf-test-8port-udp" & done > > The scripts execute the following on each host. > > #!/bin/bash > PORT1=$PORT > PORT2=$(($PORT+1000)) > PORT3=$(($PORT+2000)) > iperf3 -c loader41-40gbe -u -b 10000m -i 0 -N -l $PKT -t$TIME > -P$STREAMS -p$PORT1 & > iperf3 -c loader42-40gbe -u -b 10000m -i 0 -N -l $PKT -t$TIME > -P$STREAMS -p$PORT1 & > iperf3 -c loader43-40gbe -u -b 10000m -i 0 -N -l $PKT -t$TIME > -P$STREAMS -p$PORT1 & > ... (through all clients and all three ports) ... > iperf3 -c loader60-40gbe -u -b 10000m -i 0 -N -l $PKT -t$TIME > -P$STREAMS -p$PORT3 & > > > Results: > > Summarized, netstat -w 1 -q 240 -bd, run through: > cat test4-tuning | egrep -v {'packets | input '} | awk '{ipackets+=$1} > {idrops+=$3} {opackets+=$5} {odrops+=$9} END {print "input " > ipackets/NR, "idrops " idrops/NR, "opackets " opackets/NR, "odrops " > odrops/NR}' > > input 1.10662e+07 idrops 8.01783e+06 opackets 3.04516e+06 odrops 3152.4 > > Snapshot of raw output: > > input (Total) output > packets errs idrops bytes packets errs bytes colls drops > 11189148 0 7462453 1230805216 3725006 0 409750710 0 799 > 10527505 0 6746901 1158024978 3779096 0 415700708 0 127 > 10606163 0 6850760 1166676673 3751780 0 412695761 0 1535 > 10749324 0 7132014 1182425799 3617558 0 397930956 0 5972 > 10695667 0 7022717 1176521907 3669342 0 403627236 0 1461 > 10441173 0 6762134 1148528662 3675048 0 404255540 0 6021 > 10683773 0 7005635 1175215014 3676962 0 404465671 0 2606 > 10869859 0 7208696 1195683372 3658432 0 402427698 0 979 > 11948989 0 8310926 1314387881 3633773 0 399714986 0 725 > 12426195 0 8864415 1366877194 3562311 0 391853156 0 2762 > 13006059 0 9432389 1430661751 3570067 0 392706552 0 5158 > 12822243 0 9098871 1410443600 3715177 0 408668500 0 4064 > 13317864 0 9683602 1464961374 3632156 0 399536131 0 3684 > 13701905 0 10182562 1507207982 3523101 0 387540859 0 > 8690 > 13820227 0 10244870 1520221820 3562038 0 391823322 0 > 2426 > 14437060 0 10955483 1588073033 3480105 0 382810557 0 > 2619 > 14518471 0 11119573 1597028105 3397439 0 373717355 0 > 5691 > 14890287 0 11675003 1637926521 3199812 0 351978304 0 > 11007 > 14923610 0 11749091 1641594441 3171436 0 348857468 0 > 7389 > 14738704 0 11609730 1621254991 3117715 0 342948394 0 > 2597 > 14753975 0 11549735 1622935026 3207393 0 352812846 0 > 4798 > > > > > > _______________________________________________ > freebsd-net@freebsd.org mailing list > http://lists.freebsd.org/mailman/listinfo/freebsd-net > To unsubscribe, send any mail to "freebsd-net-unsubscribe@freebsd.org" > > From owner-freebsd-net@FreeBSD.ORG Fri Jul 25 20:51:05 2014 Return-Path: Delivered-To: freebsd-net@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 80590EF8 for ; Fri, 25 Jul 2014 20:51:05 +0000 (UTC) Received: from mail-qg0-x22d.google.com (mail-qg0-x22d.google.com [IPv6:2607:f8b0:400d:c04::22d]) (using TLSv1 with cipher ECDHE-RSA-RC4-SHA (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 416F32BFA for ; Fri, 25 Jul 2014 20:51:05 +0000 (UTC) Received: by mail-qg0-f45.google.com with SMTP id f51so5713598qge.18 for ; Fri, 25 Jul 2014 13:51:04 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:sender:in-reply-to:references:date:message-id:subject :from:to:cc:content-type; bh=t9gyCaDCB0TNkTyHftn2kOYHJBNEhj4MA9mOdvNcGtM=; b=O326rufww3jxr0ugaxIju8uYACdj41YzNZIZmlykQgulD/Ku7Nu8M8nnJRLsjx1LUg ajGfnDMeu3ntDnsT/m0yiBqbAKljzaEpuMZK6/R4LBgDTl9PbPvs0VIJqgPaS/XKrA6x IqlFk2xvGlWqSfgog3ag9x4uBxr403GF4UGLYhTgn3I7q6/2tmPl3l27sA/Zqph+9oeG nayejvWYiFuOssB0DwcNAHF/LsewYagM+omDNDE8yl3Xg3x7GkY9XKbIxqcmb58s24Qf EXn9VlN5QTlxhXHdv3eY60ALOoFNGEgY8YaIDp5S9hqf6NSmUMG1tbilly6kYq5oBGIv pZTw== MIME-Version: 1.0 X-Received: by 10.224.112.1 with SMTP id u1mr30758615qap.7.1406321464216; Fri, 25 Jul 2014 13:51:04 -0700 (PDT) Sender: adrian.chadd@gmail.com Received: by 10.224.1.6 with HTTP; Fri, 25 Jul 2014 13:51:03 -0700 (PDT) In-Reply-To: References: <53CE80DD.9090109@gmail.com> Date: Fri, 25 Jul 2014 13:51:03 -0700 X-Google-Sender-Auth: uFJqQuQ_1t92F0DNooaDS4vH5MM Message-ID: Subject: Re: fastforward/routing: a 3 million packet-per-second system? From: Adrian Chadd To: John Jasen Content-Type: text/plain; charset=UTF-8 Cc: FreeBSD Net X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 25 Jul 2014 20:51:05 -0000 Ugh, the forwarding table stupidity. Try enabling FLOWTABLE as an option. I really dislike how the rtentry locking works. But that isn't a rwlock - i'll have to look at your full lock profiling output to see. -a On 25 July 2014 09:20, John Jasen wrote: > Based on advice I received, I've collected lock profile debugging output, > and pmcannotate'd data from the system while it was processing about 3 > million packets/second. > > Combined, the files are about 325k in size, so I'll submit highlights here. > I can provide the raw files to interested parties privately. > > pmcannotate summary output: > > grep ^Profile pmcannotate.20140725 > Profile trace for function: __rw_rlock() [17.04%] > Profile trace for function: __mtx_unlock_flags() [9.10%] > Profile trace for function: _rw_runlock_cookie() [7.67%] > Profile trace for function: sched_idletd() [5.73%] > Profile trace for function: memcpy() [5.64%] > Profile trace for function: bcopy() [5.04%] > Profile trace for function: bcmp() [5.01%] > Profile trace for function: __mtx_lock_flags() [3.66%] > Profile trace for function: t4_eth_tx() [3.25%] > Profile trace for function: lock_profile_release_lock() [2.73%] > Profile trace for function: ip_fastforward() [2.68%] > Profile trace for function: ether_output() [2.50%] > Profile trace for function: get_scatter_segment() [1.75%] > Profile trace for function: rn_match() [1.74%] > Profile trace for function: _mtx_lock_spin_cookie() [1.53%] > Profile trace for function: lock_profile_obtain_lock_success() [1.49%] > Profile trace for function: cxgbe_transmit() [1.37%] > Profile trace for function: uma_zalloc_arg() [1.31%] > Profile trace for function: bzero() [1.30%] > Profile trace for function: service_iq() [1.26%] > Profile trace for function: ether_nh_input() [1.23%] > Profile trace for function: __mtx_lock_sleep() [1.19%] > Profile trace for function: arpresolve() [1.07%] > Profile trace for function: uma_zfree_arg() [0.95%] > Profile trace for function: reclaim_tx_descs() [0.87%] > Profile trace for function: _mtx_trylock_flags_() [0.80%] > Profile trace for function: bounce_bus_dmamap_load_buffer() [0.72%] > Profile trace for function: ether_demux() [0.64%] > Profile trace for function: mb_ctor_mbuf() [0.63%] > Profile trace for function: rtalloc1_fib() [0.54%] > > sysctl debug.lock.prof.stats summary: (some of the highest hit counts, > especially in cnt_hold: > > 7 17 3271389 2258698 63952832 0 0 0 > 6761237 /usr/src/sys/netinet/in_rmx.c:114 (sleep mutex:rtentry) > > 7 13 11543138 470545 63952832 0 0 0 > 815777 /usr/src/sys/net/route.c:439 (sleep mutex:rtentry) > > 6 15 3943582 1545195 63952779 0 0 0 > 3439254 /usr/src/sys/netinet/ip_fastfwd.c:593 (sleep mutex:rtentry > > > > > > > > > > > > On Tue, Jul 22, 2014 at 11:18 AM, John Jasen wrote: > >> Feedback and/or tips and tricks more than welcome. >> > > > _______________________________________________ > freebsd-net@freebsd.org mailing list > http://lists.freebsd.org/mailman/listinfo/freebsd-net > To unsubscribe, send any mail to "freebsd-net-unsubscribe@freebsd.org" From owner-freebsd-net@FreeBSD.ORG Fri Jul 25 20:53:32 2014 Return-Path: Delivered-To: freebsd-net@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 00AE7B6 for ; Fri, 25 Jul 2014 20:53:31 +0000 (UTC) Received: from mail-qg0-x234.google.com (mail-qg0-x234.google.com [IPv6:2607:f8b0:400d:c04::234]) (using TLSv1 with cipher ECDHE-RSA-RC4-SHA (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id B52BB2C99 for ; Fri, 25 Jul 2014 20:53:31 +0000 (UTC) Received: by mail-qg0-f52.google.com with SMTP id f51so5555160qge.25 for ; Fri, 25 Jul 2014 13:53:30 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:sender:in-reply-to:references:date:message-id:subject :from:to:cc:content-type; bh=2l+3oLLC2oQkuw60fOpC3f4Qo4fJIJJanzUvkEHNpFI=; b=PHrC4U/BtVOB4v9aEqv9vxkYG7jJDR8D4A6Qjz7FsUtEdIDRaBGj2iuEZnvK8jWwoL jDs1bwoY4aCCNgM4OmOFIkZ1FVO3le87Mn4Nm8RT4XTjDuPXA06GQ/IDFvN44C2oYGse 5Odh7ZN00fJL09B1fWH+taMzGvu78FhKvAL9qx+mFOFd+eEk/uibQLphleOJv3/iH2nf CUefwc/ufzdBt/MduUbZ9EXKs5h0xU7P6d8WjnT1zbiOSce0sMC2iISNzI1tLTca544n NpePKk4S40icvA+UtASReYbHt4mAiuMdOsiwomq74J4VMRtqoJZGtTGxiLAD3X92YCWb zsCA== MIME-Version: 1.0 X-Received: by 10.224.55.131 with SMTP id u3mr31361011qag.98.1406321610672; Fri, 25 Jul 2014 13:53:30 -0700 (PDT) Sender: adrian.chadd@gmail.com Received: by 10.224.1.6 with HTTP; Fri, 25 Jul 2014 13:53:30 -0700 (PDT) In-Reply-To: References: <53CE80DD.9090109@gmail.com> Date: Fri, 25 Jul 2014 13:53:30 -0700 X-Google-Sender-Auth: yqgjqWjp5jco-hQdL024_cVe0Q4 Message-ID: Subject: Re: fastforward/routing: a 3 million packet-per-second system? From: Adrian Chadd To: John Jasen Content-Type: text/plain; charset=UTF-8 Cc: FreeBSD Net X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 25 Jul 2014 20:53:32 -0000 Yeah: Adrians-MacBook-Pro:Downloads adrian$ head -2 debug.lock.prof.stats.out.20140725 ; cat debug.lock.prof.stats.out.20140725 | sort -nk4 | tail -10 debug.lock.prof.stats: max wait_max total wait_total count avg wait_avg cnt_hold cnt_lock name 6 3 419 145 160 2 0 0 63 /usr/src/sys/kern/kern_condvar.c:145 (sleep mutex:Giant) 282 133 991 215 8 123 26 0 2 /usr/src/sys/modules/cxgbe/if_cxgbe/../../../dev/cxgbe/t4_main.c:6657 (sleep mutex:cxl3 txq26) 69 72 71 250 5 14 50 0 4 /usr/src/sys/modules/cxgbe/if_cxgbe/../../../dev/cxgbe/t4_main.c:6657 (sleep mutex:cxl1 txq37) 281 197 1638 286 13 126 22 0 2 /usr/src/sys/modules/cxgbe/if_cxgbe/../../../dev/cxgbe/t4_main.c:6657 (sleep mutex:cxl1 txq46) 351 182 2416 499 38 63 13 0 10 /usr/src/sys/modules/cxgbe/if_cxgbe/../../../dev/cxgbe/t4_main.c:6657 (sleep mutex:cxl3 txq17) 276 193 802 643 10 80 64 0 5 /usr/src/sys/modules/cxgbe/if_cxgbe/../../../dev/cxgbe/t4_main.c:6657 (sleep mutex:cxl3 txq27) 0 1 98578 1341 482441 0 0 0 3767 /usr/src/sys/kern/subr_turnstile.c:552 (spin mutex:turnstile chain) 7 13 11543138 470545 63952832 0 0 0 815777 /usr/src/sys/net/route.c:439 (sleep mutex:rtentry) 6 15 3943582 1545195 63952779 0 0 0 3439254 /usr/src/sys/netinet/ip_fastfwd.c:593 (sleep mutex:rtentry) 7 17 3271389 2258698 63952832 0 0 0 6761237 /usr/src/sys/netinet/in_rmx.c:114 (sleep mutex:rtentry) .. try FLOWTABLE. The in_rmx.c is the hook to check for temporary routes installed by redirect ICMP messages. It's .. not very pretty. Just use FLOWTABLE for now and see if it improves things. (Yes, we likely can do better on the rtentry locking..) -a On 25 July 2014 13:51, Adrian Chadd wrote: > Ugh, the forwarding table stupidity. Try enabling FLOWTABLE as an option. > > I really dislike how the rtentry locking works. But that isn't a > rwlock - i'll have to look at your full lock profiling output to see. > > > -a > > > On 25 July 2014 09:20, John Jasen wrote: >> Based on advice I received, I've collected lock profile debugging output, >> and pmcannotate'd data from the system while it was processing about 3 >> million packets/second. >> >> Combined, the files are about 325k in size, so I'll submit highlights here. >> I can provide the raw files to interested parties privately. >> >> pmcannotate summary output: >> >> grep ^Profile pmcannotate.20140725 >> Profile trace for function: __rw_rlock() [17.04%] >> Profile trace for function: __mtx_unlock_flags() [9.10%] >> Profile trace for function: _rw_runlock_cookie() [7.67%] >> Profile trace for function: sched_idletd() [5.73%] >> Profile trace for function: memcpy() [5.64%] >> Profile trace for function: bcopy() [5.04%] >> Profile trace for function: bcmp() [5.01%] >> Profile trace for function: __mtx_lock_flags() [3.66%] >> Profile trace for function: t4_eth_tx() [3.25%] >> Profile trace for function: lock_profile_release_lock() [2.73%] >> Profile trace for function: ip_fastforward() [2.68%] >> Profile trace for function: ether_output() [2.50%] >> Profile trace for function: get_scatter_segment() [1.75%] >> Profile trace for function: rn_match() [1.74%] >> Profile trace for function: _mtx_lock_spin_cookie() [1.53%] >> Profile trace for function: lock_profile_obtain_lock_success() [1.49%] >> Profile trace for function: cxgbe_transmit() [1.37%] >> Profile trace for function: uma_zalloc_arg() [1.31%] >> Profile trace for function: bzero() [1.30%] >> Profile trace for function: service_iq() [1.26%] >> Profile trace for function: ether_nh_input() [1.23%] >> Profile trace for function: __mtx_lock_sleep() [1.19%] >> Profile trace for function: arpresolve() [1.07%] >> Profile trace for function: uma_zfree_arg() [0.95%] >> Profile trace for function: reclaim_tx_descs() [0.87%] >> Profile trace for function: _mtx_trylock_flags_() [0.80%] >> Profile trace for function: bounce_bus_dmamap_load_buffer() [0.72%] >> Profile trace for function: ether_demux() [0.64%] >> Profile trace for function: mb_ctor_mbuf() [0.63%] >> Profile trace for function: rtalloc1_fib() [0.54%] >> >> sysctl debug.lock.prof.stats summary: (some of the highest hit counts, >> especially in cnt_hold: >> >> 7 17 3271389 2258698 63952832 0 0 0 >> 6761237 /usr/src/sys/netinet/in_rmx.c:114 (sleep mutex:rtentry) >> >> 7 13 11543138 470545 63952832 0 0 0 >> 815777 /usr/src/sys/net/route.c:439 (sleep mutex:rtentry) >> >> 6 15 3943582 1545195 63952779 0 0 0 >> 3439254 /usr/src/sys/netinet/ip_fastfwd.c:593 (sleep mutex:rtentry >> >> >> >> >> >> >> >> >> >> >> >> On Tue, Jul 22, 2014 at 11:18 AM, John Jasen wrote: >> >>> Feedback and/or tips and tricks more than welcome. >>> >> >> >> _______________________________________________ >> freebsd-net@freebsd.org mailing list >> http://lists.freebsd.org/mailman/listinfo/freebsd-net >> To unsubscribe, send any mail to "freebsd-net-unsubscribe@freebsd.org" From owner-freebsd-net@FreeBSD.ORG Fri Jul 25 21:52:45 2014 Return-Path: Delivered-To: freebsd-net@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id D2906343 for ; Fri, 25 Jul 2014 21:52:45 +0000 (UTC) Received: from mail-qa0-x235.google.com (mail-qa0-x235.google.com [IPv6:2607:f8b0:400d:c00::235]) (using TLSv1 with cipher ECDHE-RSA-RC4-SHA (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 9668521BD for ; Fri, 25 Jul 2014 21:52:45 +0000 (UTC) Received: by mail-qa0-f53.google.com with SMTP id v10so5097511qac.26 for ; Fri, 25 Jul 2014 14:52:44 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:from:date:message-id:subject:to:content-type; bh=IDPkyzEAMhkmddTjd2DjmeTWDQK3nfzlbCkef+OdCps=; b=PfHT7HsDHlqWtnBtyAVFLfVJCf2G9hs1YLKGucZMNAkwXQbc7iA6O+CCl+SuK0ym5Z sPHO+8CrI4cTIGkMYAwBdd2SYNt7G2Rpzu/x887xl/ypQ1giQqONqpD2B8+xpvLH0zPp A5u4sj2o6/DHx94gh3U0UskM7sOudEaPTsefCNuv8K5ks8PqJuzFRbKsw8iQ6WahvYIC xAv9HsJ1uAjhSa3MzPfwyk87INqdp7RVkVuZDWU2AkrlEV8O7oPukDuZIXuMU9xcvKxG +Kh1EtPeKXBk4cWtR8QNQ0JHCkCcqOIxeT6yX6SrQq6pxVAuRM3kBmerUw++uvJ4M5cO ARXg== X-Received: by 10.224.46.8 with SMTP id h8mr24752694qaf.6.1406325164640; Fri, 25 Jul 2014 14:52:44 -0700 (PDT) MIME-Version: 1.0 Received: by 10.140.97.228 with HTTP; Fri, 25 Jul 2014 14:52:24 -0700 (PDT) From: Sridhar Iyer Date: Fri, 25 Jul 2014 14:52:24 -0700 Message-ID: Subject: Using lagg To: freebsd-net@freebsd.org Content-Type: text/plain; charset=UTF-8 X-Content-Filtered-By: Mailman/MimeDel 2.1.18 X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 25 Jul 2014 21:52:45 -0000 Hi all, I have been looking at apis in sys/net/if_lagg.c and usage in ifconfig/iflagg.c. They both seem to be different. I'm new to freebsd and I need to use the apis to create a lagg of interfaces. What would be the best way to get up to speed? Is there some example out there, which just creates a lagg interface and then just attaches the interfaces to it? Any pointers would be really helpful. Regards, Sridhar From owner-freebsd-net@FreeBSD.ORG Fri Jul 25 21:58:36 2014 Return-Path: Delivered-To: freebsd-net@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 6CB71587 for ; Fri, 25 Jul 2014 21:58:36 +0000 (UTC) Received: from smtp1.multiplay.co.uk (smtp1.multiplay.co.uk [85.236.96.35]) by mx1.freebsd.org (Postfix) with ESMTP id 31A642225 for ; Fri, 25 Jul 2014 21:58:35 +0000 (UTC) Received: by smtp1.multiplay.co.uk (Postfix, from userid 65534) id D2A0520E70890; Fri, 25 Jul 2014 21:58:26 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on smtp1.multiplay.co.uk X-Spam-Level: ** X-Spam-Status: No, score=2.2 required=8.0 tests=AWL,BAYES_00,DOS_OE_TO_MX, FSL_HELO_NON_FQDN_1,HELO_NO_DOMAIN,RDNS_DYNAMIC,STOX_REPLY_TYPE autolearn=no version=3.3.1 Received: from r2d2 (82-69-141-170.dsl.in-addr.zen.co.uk [82.69.141.170]) by smtp1.multiplay.co.uk (Postfix) with ESMTPS id 8164E20E7088C; Fri, 25 Jul 2014 21:58:21 +0000 (UTC) Message-ID: <371D9ED786FD4379859A0993E8D12F15@multiplay.co.uk> From: "Steven Hartland" To: "Sridhar Iyer" , References: Subject: Re: Using lagg Date: Fri, 25 Jul 2014 22:58:17 +0100 MIME-Version: 1.0 Content-Type: text/plain; format=flowed; charset="iso-8859-1"; reply-type=original Content-Transfer-Encoding: 7bit X-Priority: 3 X-MSMail-Priority: Normal X-Mailer: Microsoft Outlook Express 6.00.2900.5931 X-MimeOLE: Produced By Microsoft MimeOLE V6.00.2900.6157 X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 25 Jul 2014 21:58:36 -0000 Have a look at the man page using: man lagg It gives you all the details including examples. Regards Steve ----- Original Message ----- From: "Sridhar Iyer" > Hi all, > > I have been looking at apis in sys/net/if_lagg.c and usage in > ifconfig/iflagg.c. They both seem to be different. I'm new to freebsd and I > need to use the apis to create a lagg of interfaces. What would be the best > way to get up to speed? Is there some example out there, which just creates > a lagg interface and then just attaches the interfaces to it? > > Any pointers would be really helpful. > Regards, > > Sridhar > _______________________________________________ > freebsd-net@freebsd.org mailing list > http://lists.freebsd.org/mailman/listinfo/freebsd-net > To unsubscribe, send any mail to "freebsd-net-unsubscribe@freebsd.org" > From owner-freebsd-net@FreeBSD.ORG Fri Jul 25 22:05:22 2014 Return-Path: Delivered-To: freebsd-net@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 0D3C46D3 for ; Fri, 25 Jul 2014 22:05:22 +0000 (UTC) Received: from mail-qg0-x22d.google.com (mail-qg0-x22d.google.com [IPv6:2607:f8b0:400d:c04::22d]) (using TLSv1 with cipher ECDHE-RSA-RC4-SHA (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id BFD8822EC for ; Fri, 25 Jul 2014 22:05:21 +0000 (UTC) Received: by mail-qg0-f45.google.com with SMTP id f51so5802098qge.18 for ; Fri, 25 Jul 2014 15:05:21 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:in-reply-to:references:from:date:message-id:subject:to :cc:content-type; bh=PABWcTvMqR50yJhQa8DAVf2OvPgoruiVYPwQqe0lvbc=; b=gw/oU2jLJloBopAovNQj1DGTSib1Il7dDxBF6TtNP0k15Lk0oR5KaugJBdA1asdDfQ QmPT4SR3hs9LS2SylQrNJY/bq3F9nNG+EM3pVPbugyQGJNgfJj1WhVWQ9dyhyuuXkynp sT05BqWQkU3Pji6/BZLiYcTrha3pxhWjLWqY9RHOPEz3vloh1KhdXXqVwc8//Jcw26EJ nT4u1FqEHDt36b4qlvUVqJ5gYY54CuLP10HTkxgJYbcgu/OZH4h15nv8XL7vFag4QX7S ORZYl8C+ykoB+I1l4S5fCHlG7DMfZHFVtEXSfxfesk8+URh8rqL1ATi8qKZDT3w1usge +iIw== X-Received: by 10.224.87.130 with SMTP id w2mr32135229qal.5.1406325920952; Fri, 25 Jul 2014 15:05:20 -0700 (PDT) MIME-Version: 1.0 Received: by 10.140.97.228 with HTTP; Fri, 25 Jul 2014 15:05:00 -0700 (PDT) In-Reply-To: <371D9ED786FD4379859A0993E8D12F15@multiplay.co.uk> References: <371D9ED786FD4379859A0993E8D12F15@multiplay.co.uk> From: Sridhar Iyer Date: Fri, 25 Jul 2014 15:05:00 -0700 Message-ID: Subject: Re: Using lagg To: Steven Hartland Content-Type: text/plain; charset=UTF-8 X-Content-Filtered-By: Mailman/MimeDel 2.1.18 Cc: freebsd-net X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 25 Jul 2014 22:05:22 -0000 I didn't find the api examples http://www.freebsd.org/cgi/man.cgi?query=lagg&sektion=4&apropos=0&manpath=FreeBSD+9.2-RELEASE+and+Ports This only shows the usage via cli. Regards, Sridhar On Fri, Jul 25, 2014 at 2:58 PM, Steven Hartland wrote: > Have a look at the man page using: > man lagg > > It gives you all the details including examples. > > Regards > Steve > > ----- Original Message ----- From: "Sridhar Iyer" < > sridhar.v.iyer@gmail.com> > > > Hi all, >> >> I have been looking at apis in sys/net/if_lagg.c and usage in >> ifconfig/iflagg.c. They both seem to be different. I'm new to freebsd and >> I >> need to use the apis to create a lagg of interfaces. What would be the >> best >> way to get up to speed? Is there some example out there, which just >> creates >> a lagg interface and then just attaches the interfaces to it? >> >> Any pointers would be really helpful. >> Regards, >> >> Sridhar >> _______________________________________________ >> freebsd-net@freebsd.org mailing list >> http://lists.freebsd.org/mailman/listinfo/freebsd-net >> To unsubscribe, send any mail to "freebsd-net-unsubscribe@freebsd.org" >> >> From owner-freebsd-net@FreeBSD.ORG Fri Jul 25 22:27:02 2014 Return-Path: Delivered-To: freebsd-net@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 9475EA46 for ; Fri, 25 Jul 2014 22:27:02 +0000 (UTC) Received: from smtp1.multiplay.co.uk (smtp1.multiplay.co.uk [85.236.96.35]) by mx1.freebsd.org (Postfix) with ESMTP id 57CE82480 for ; Fri, 25 Jul 2014 22:27:01 +0000 (UTC) Received: by smtp1.multiplay.co.uk (Postfix, from userid 65534) id 6164E20E70890; Fri, 25 Jul 2014 22:27:01 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on smtp1.multiplay.co.uk X-Spam-Level: ** X-Spam-Status: No, score=2.2 required=8.0 tests=AWL,BAYES_00,DOS_OE_TO_MX, FSL_HELO_NON_FQDN_1,HELO_NO_DOMAIN,RDNS_DYNAMIC,STOX_REPLY_TYPE autolearn=no version=3.3.1 Received: from r2d2 (82-69-141-170.dsl.in-addr.zen.co.uk [82.69.141.170]) by smtp1.multiplay.co.uk (Postfix) with ESMTPS id 161BC20E7088C; Fri, 25 Jul 2014 22:26:56 +0000 (UTC) Message-ID: From: "Steven Hartland" To: "Sridhar Iyer" References: <371D9ED786FD4379859A0993E8D12F15@multiplay.co.uk> Subject: Re: Using lagg Date: Fri, 25 Jul 2014 23:26:52 +0100 MIME-Version: 1.0 Content-Type: text/plain; format=flowed; charset="iso-8859-1"; reply-type=original Content-Transfer-Encoding: 7bit X-Priority: 3 X-MSMail-Priority: Normal X-Mailer: Microsoft Outlook Express 6.00.2900.5931 X-MimeOLE: Produced By Microsoft MimeOLE V6.00.2900.6157 Cc: freebsd-net X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 25 Jul 2014 22:27:02 -0000 Not sure what you mean by API examples? The usual use of this would be to configure it at startup in /etc/rc.conf Regards Steve ----- Original Message ----- From: "Sridhar Iyer" To: "Steven Hartland" Cc: "freebsd-net" Sent: Friday, July 25, 2014 11:05 PM Subject: Re: Using lagg >I didn't find the api examples > http://www.freebsd.org/cgi/man.cgi?query=lagg&sektion=4&apropos=0&manpath=FreeBSD+9.2-RELEASE+and+Ports > This only shows the usage via cli. > > > Regards, > > Sridhar > > > On Fri, Jul 25, 2014 at 2:58 PM, Steven Hartland > wrote: > >> Have a look at the man page using: >> man lagg >> >> It gives you all the details including examples. >> >> Regards >> Steve >> >> ----- Original Message ----- From: "Sridhar Iyer" < >> sridhar.v.iyer@gmail.com> >> >> >> Hi all, >>> >>> I have been looking at apis in sys/net/if_lagg.c and usage in >>> ifconfig/iflagg.c. They both seem to be different. I'm new to freebsd and >>> I >>> need to use the apis to create a lagg of interfaces. What would be the >>> best >>> way to get up to speed? Is there some example out there, which just >>> creates >>> a lagg interface and then just attaches the interfaces to it? >>> >>> Any pointers would be really helpful. >>> Regards, >>> >>> Sridhar >>> _______________________________________________ >>> freebsd-net@freebsd.org mailing list >>> http://lists.freebsd.org/mailman/listinfo/freebsd-net >>> To unsubscribe, send any mail to "freebsd-net-unsubscribe@freebsd.org" >>> >>> > _______________________________________________ > freebsd-net@freebsd.org mailing list > http://lists.freebsd.org/mailman/listinfo/freebsd-net > To unsubscribe, send any mail to "freebsd-net-unsubscribe@freebsd.org" > From owner-freebsd-net@FreeBSD.ORG Fri Jul 25 22:28:29 2014 Return-Path: Delivered-To: freebsd-net@freebsd.org Received: from hub.FreeBSD.org (hub.freebsd.org [IPv6:2001:1900:2254:206c::16:88]) (using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 8CF44AEF; Fri, 25 Jul 2014 22:28:28 +0000 (UTC) Date: Fri, 25 Jul 2014 18:28:24 -0400 From: Glen Barber To: Sridhar Iyer Subject: Re: Using lagg Message-ID: <20140725222824.GN1065@hub.FreeBSD.org> References: <371D9ED786FD4379859A0993E8D12F15@multiplay.co.uk> MIME-Version: 1.0 Content-Type: multipart/signed; micalg=pgp-sha256; protocol="application/pgp-signature"; boundary="qDmf+FkDYU/jKsh2" Content-Disposition: inline In-Reply-To: X-Operating-System: FreeBSD 11.0-CURRENT amd64 X-SCUD-Definition: Sudden Completely Unexpected Dataloss X-SULE-Definition: Sudden Unexpected Learning Event User-Agent: Mutt/1.5.23 (2014-03-12) Cc: freebsd-net , Steven Hartland X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 25 Jul 2014 22:28:29 -0000 --qDmf+FkDYU/jKsh2 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline Content-Transfer-Encoding: quoted-printable On Fri, Jul 25, 2014 at 03:05:00PM -0700, Sridhar Iyer wrote: > I didn't find the api examples > http://www.freebsd.org/cgi/man.cgi?query=3Dlagg&sektion=3D4&apropos=3D0&m= anpath=3DFreeBSD+9.2-RELEASE+and+Ports > This only shows the usage via cli. >=20 See the rc.conf(5) manual. Glen --qDmf+FkDYU/jKsh2 Content-Type: application/pgp-signature -----BEGIN PGP SIGNATURE----- Version: GnuPG v2 iQIcBAEBCAAGBQJT0toIAAoJELls3eqvi17QRxYP/0okHToEHKNQJ+qjQpecWMWu /wqZML7bz2bQKSMmL26yjRNn+obk++03M984yLUhQhyaGwpqSkHI8CbFPHX+xjjb r/wB7DtS6yqsu85Te49AH5idxFZmi81C/kIEwu9b093Mi06AKDIg9N69NEd4Ol6o 0mL6DRFrYCDqgS9+vLOwC9Y2dBmUHFfhfaVSnIKBDG8utCxgkB/DRM00sM2JVcEH vgsYQ/GkWGZ0mmTKHewSyEXzOsvtIBxmgU01y9rAYZnqd8Gu5dFt2BF3NaKO5Jq4 qcqOCkvLWOtO/cjBSDmQgYxTBXy7GfmRPjwCXkgEHzbdzvXH3qeUJH0k6fzFn090 LoFPonYhmAWUJAofwF+G+XGv0gfvdR46dXp13uMQ+g01Rvcw5fInPIPFx+fDpjS4 4PiTmtxlcHBvD7V+xZIrcIuoek5EpMHA1qv7JxUBjlabzm4XSYMo3T9CmefRmZfi TQkm6NMslt32knEIhvMeOwZ9YOnyCedFE6IWJNc6HkYWwFxvdj/Avee8j+nA0l+W eofVyEu2iTyjxEB5L9UOQKAYXMuZFzhKRPZhQjuGmzf/U5n5Lb69apX/Bf4OI2g2 PBG03vEaLiwIm8Bl9j5ScgLUk1z8mlidcTGIGGoWnh+p2XV9pRv5FgpqAYpAg6H9 4iEC3xdL2yM0xbQ7wwC6 =fXSJ -----END PGP SIGNATURE----- --qDmf+FkDYU/jKsh2-- From owner-freebsd-net@FreeBSD.ORG Fri Jul 25 22:33:05 2014 Return-Path: Delivered-To: freebsd-net@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 305B6CAE; Fri, 25 Jul 2014 22:33:05 +0000 (UTC) Received: from mail-qa0-x236.google.com (mail-qa0-x236.google.com [IPv6:2607:f8b0:400d:c00::236]) (using TLSv1 with cipher ECDHE-RSA-RC4-SHA (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id D4143254F; Fri, 25 Jul 2014 22:33:04 +0000 (UTC) Received: by mail-qa0-f54.google.com with SMTP id k15so5353918qaq.27 for ; Fri, 25 Jul 2014 15:33:04 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:in-reply-to:references:from:date:message-id:subject:to :cc:content-type; bh=/zhtzmDL7yNh0fmIAJ7Rq0Is+BR1lV3xlHHuLCsQb1w=; b=qRj5twjUU8/A1hYZW5tBZoztNgodsEB6cXJH4qi+gfe0pLmIJ8MLUTxDG2jcNsdf5j UshGS/5EKYYAXZHPb713Qgw8fXGlhTgQoZ6aBrTqrpj6PUBBoqBjYIJL8zC4iyuxh2UQ 6J37CYQknhnzjGFGL0oWs8hHJpNBJ8lHPh6TQdo4wUrIshrG9PWfZNAGaCUs3JrFALbF NQA9+1Cx6JeYxqN+8UygAnFStmtG2QGbLqP2ul96faGDSNTiNh4kxLn/fXRyOx6t6nie I3ZcV6H8bUhma+JfAaOmnUVLpG8O0odAJcuPZMJB5heMqt2abu/pIBbPgHf6n/bn8fIs Fpug== X-Received: by 10.224.47.8 with SMTP id l8mr32862420qaf.30.1406327583977; Fri, 25 Jul 2014 15:33:03 -0700 (PDT) MIME-Version: 1.0 Received: by 10.140.97.228 with HTTP; Fri, 25 Jul 2014 15:32:43 -0700 (PDT) In-Reply-To: <20140725222824.GN1065@hub.FreeBSD.org> References: <371D9ED786FD4379859A0993E8D12F15@multiplay.co.uk> <20140725222824.GN1065@hub.FreeBSD.org> From: Sridhar Iyer Date: Fri, 25 Jul 2014 15:32:43 -0700 Message-ID: Subject: Re: Using lagg To: Glen Barber Content-Type: text/plain; charset=UTF-8 X-Content-Filtered-By: Mailman/MimeDel 2.1.18 Cc: freebsd-net , Steven Hartland X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 25 Jul 2014 22:33:05 -0000 I was looking at sys/net/if_lagg.c, and wanted to know the usage of functions such as lagg_ioctl(), lagg_input() etc. Regards, Sridhar On Fri, Jul 25, 2014 at 3:28 PM, Glen Barber wrote: > On Fri, Jul 25, 2014 at 03:05:00PM -0700, Sridhar Iyer wrote: > > I didn't find the api examples > > > http://www.freebsd.org/cgi/man.cgi?query=lagg&sektion=4&apropos=0&manpath=FreeBSD+9.2-RELEASE+and+Ports > > This only shows the usage via cli. > > > > See the rc.conf(5) manual. > > Glen > > From owner-freebsd-net@FreeBSD.ORG Fri Jul 25 23:48:18 2014 Return-Path: Delivered-To: freebsd-net@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 4049690C for ; Fri, 25 Jul 2014 23:48:18 +0000 (UTC) Received: from mail.bsdinfo.com.br (mail.bsdinfo.com.br [67.212.89.78]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 112562BE4 for ; Fri, 25 Jul 2014 23:48:17 +0000 (UTC) Received: from mail.bsdinfo.com.br (mail.bsdinfo.com.br [127.0.0.1]) by mail.bsdinfo.com.br (Postfix) with ESMTP id 03DE8139CB for ; Fri, 25 Jul 2014 20:39:43 -0300 (BRT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=bsdinfo.com.br; h=content-transfer-encoding:content-type:content-type:subject :subject:to:mime-version:user-agent:from:from:date:date :message-id; s=dkim; t=1406331582; x=1407195583; bh=zUlfQkTSFWhi eNOsTGU68iyVziGjti/kMMZLc8BqKOY=; b=fuHXjSIG/ln03mviMH1QImS4rcpg pYROnn9lJz/uHutyp4O4hq0NxXWBhbuXflut+taSfa2+DpNOlqkxGwlRFV4Rn9cF LEZMuYMKzM5UwpkgROsLLp5Sb5bwuPa0kLYRf4fZ43m3WOQisqg5u05To7RpP/OR TnKqx9v9SO8Ixlg= X-Virus-Scanned: amavisd-new at mail.bsdinfo.com.br Received: from mail.bsdinfo.com.br ([127.0.0.1]) by mail.bsdinfo.com.br (mail.bsdinfo.com.br [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id Z3RvVYuR9vsw for ; Fri, 25 Jul 2014 20:39:42 -0300 (BRT) Received: from [192.168.10.208] (unknown [186.193.54.69]) by mail.bsdinfo.com.br (Postfix) with ESMTPSA id ECF00139CA for ; Fri, 25 Jul 2014 20:39:41 -0300 (BRT) Message-ID: <53D2EAAF.8070707@bsdinfo.com.br> Date: Fri, 25 Jul 2014 20:39:27 -0300 From: Marcelo Gondim User-Agent: Mozilla/5.0 (Windows NT 6.3; WOW64; rv:24.0) Gecko/20100101 Thunderbird/24.6.0 MIME-Version: 1.0 To: FreeBSD Net Subject: IPv6 alias do not respond from outside Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 25 Jul 2014 23:48:18 -0000 Dear, I'm having the following problem: Host A (my network): =================== em0: flags=8843 metric 0 mtu 1500 options=4219b ether 00:1e:67:30:ea:27 inet 186.xxx.48.3 netmask 0xffffffe0 broadcast 186.xxx.48.31 inet6 fe80::21e:67ff:fe30:ea27%em0 prefixlen 64 scopeid 0x1 inet6 2805:xxxx:dead::3 prefixlen 64 inet 186.xxx.48.4 netmask 0xffffffff broadcast 186.xxx.48.4 inet 186.xxx.48.5 netmask 0xffffffff broadcast 186.xxx.48.5 inet6 2805:xxxx:dead::5 prefixlen 64 inet6 2805:xxxx:dead::4 prefixlen 64 nd6 options=21 media: Ethernet autoselect (1000baseT ) status: active Router (Internet): =============== vlan0: flags=8843 metric 0 mtu 1500 options=303 ether 00:1b:21:89:25:32 inet 186.xxx.48.1 netmask 0xffffffe0 broadcast 186.xxx.48.31 inet6 fe80::21b:21ff:fe89:2532%vlan0 prefixlen 64 scopeid 0x10 inet6 2805:xxxx:dead::1 prefixlen 64 nd6 options=21 media: Ethernet autoselect (10Gbase-SR ) status: active vlan: 3081 parent interface: ix0 Host B (Datacenter OVH): ====================== eth0 Link encap:Ethernet HWaddr 00:27:0e:36:67:ae inet addr:91.xxx.115.215 Bcast:91.xxx.115.255 Mask:255.255.255.0 inet6 addr: fe80::227:eff:fe36:67ae/64 Scope:Link inet6 addr: 2001:xxxx:1:aad7::1/64 Scope:Global UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1 RX packets:680611577 errors:0 dropped:115467 overruns:0 frame:0 TX packets:410997111 errors:0 dropped:0 overruns:0 carrier:0 collisions:0 txqueuelen:1000 RX bytes:929626301412 (865.7 GiB) TX bytes:28858569748 (26.8 GiB) If I try to ping the origin Host Bto destination 2805:xxxx:dead::3, I get an answer correctly: (hostb)# ping6 2805:xxxx:dead::3 PING 2805:xxxx:dead::3(2805:xxxx:dead::3) 56 data bytes 64 bytes from 2805:xxxx:dead::3: icmp_seq=1 ttl=55 time=219 ms 64 bytes from 2805:xxxx:dead::3: icmp_seq=2 ttl=55 time=218 ms 64 bytes from 2805:xxxx:dead::3: icmp_seq=3 ttl=55 time=219 ms 64 bytes from 2805:xxxx:dead::3: icmp_seq=4 ttl=55 time=218 ms But it does not ping 2805:xxxx:dead::4 and 2805:xxxx:dead::5: (hostb)# ping6 2805:xxxx:dead::4 PING 2805:xxxx:dead::4(2805:xxxx:dead::4) 56 data bytes From 2001:xxxx:2001:1001::96 icmp_seq=2 Destination unreachable: Address unreachable From 2001:xxxx:2001:1001::96 icmp_seq=8 Destination unreachable: Address unreachable Now if I do a ping from the router to Host A, it goes to work: (router)# ping6 2805:xxxx:dead::4 PING6(56=40+8+8 bytes) 2805:xxxx:dead::1 --> 2805:xxxx:dead::4 16 bytes from 2805:xxxx:dead::4, icmp_seq=0 hlim=64 time=13.550 ms 16 bytes from 2805:xxxx:dead::4, icmp_seq=1 hlim=64 time=7.776 ms (hostb)# ping6 2805:xxxx:dead::4 PING 2805:xxxx:dead::4(2805:xxxx:dead::4) 56 data bytes 64 bytes from 2805:xxxx:dead::4: icmp_seq=1 ttl=56 time=219 ms 64 bytes from 2805:xxxx:dead::4: icmp_seq=2 ttl=56 time=218 ms 64 bytes from 2805:xxxx:dead::4: icmp_seq=3 ttl=56 time=219 ms Why work now? FreeBSD xxxxx 10.0-STABLE FreeBSD 10.0-STABLE #10 r267839: Thu Jul 10 15:35:04 BRT 2014 root@xxxxx:/usr/obj/usr/src/sys/GONDIM amd64 Thanks and best regards, Gondim From owner-freebsd-net@FreeBSD.ORG Sat Jul 26 07:24:09 2014 Return-Path: Delivered-To: freebsd-net@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 1F3F434A for ; Sat, 26 Jul 2014 07:24:09 +0000 (UTC) Received: from mail.shmtech.biz (unknown [IPv6:2001:41c8:10:8c::4:3]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client CN "mail.shmtech.biz", Issuer "Go Daddy Secure Certification Authority" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id AE0E92F4A for ; Sat, 26 Jul 2014 07:24:08 +0000 (UTC) Received: from fleabag.domlan.talk2dom.com (92.40.4.124.threembb.co.uk [92.40.4.124]) (authenticated bits=0) by mail.shmtech.biz (8.14.8/8.14.5) with ESMTP id s6Q7O5aw059105 (version=TLSv1/SSLv3 cipher=DHE-RSA-CAMELLIA256-SHA bits=256 verify=NO) for ; Sat, 26 Jul 2014 08:24:06 +0100 (BST) (envelope-from dom@talk2dom.com) DKIM-Signature: v=1; a=rsa-sha256; c=simple/simple; d=talk2dom.com; s=shmtech1; t=1406359446; bh=/p4wY/pbCcGIsFzcVDtLiVemG11qnmnz3P19SyKPtY0=; h=Date:From:To:Subject:References:In-Reply-To; b=loWUi0+gbrf0SulKFcOiYhcNjv9788UA2osUKm3UWlQWbC6cTj3iZK/uaGd7aLWFF paW49JxEhIjKp/ITRsZbNvW2cM2VbhoExgt8pPTT8LplxYdABxCUCxtvdnH1gULsnf RPWmjq8I2v+HI5osEvgcFOFvEYWAbGR1Uh7HFMtY= X-Authentication-Warning: sendmail: Host 92.40.4.124.threembb.co.uk [92.40.4.124] claimed to be fleabag.domlan.talk2dom.com Message-ID: <53D35794.4030305@talk2dom.com> Date: Sat, 26 Jul 2014 08:24:04 +0100 From: Dominic Froud User-Agent: Mozilla/5.0 (X11; FreeBSD amd64; rv:24.0) Gecko/20100101 Thunderbird/24.2.0 MIME-Version: 1.0 To: freebsd-net@freebsd.org Subject: Re: Using lagg References: <371D9ED786FD4379859A0993E8D12F15@multiplay.co.uk> <20140725222824.GN1065@hub.FreeBSD.org> In-Reply-To: Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 26 Jul 2014 07:24:09 -0000 On 25/07/2014 23:32, Sridhar Iyer wrote: > I was looking at sys/net/if_lagg.c, and wanted to know the usage of > functions such as lagg_ioctl(), lagg_input() etc. > > Regards, > > Sridhar > Those functions are part of the implementation of the lagg driver, internal to the kernel. They generally mirror similarly-named functions for all network drivers. xxx_ioctl() is for processing ioctl() calls from userland xxx_input() is for processing packets that arrive at the interface xxx_start() is for processing packets that are outbound from the interface xxx_create() for creating a new lagg interface xxx_attach() for attaching a laggport to a lagg interface and so on... If you want to programmatically attach, configure, modify, destroy lagg interfaces then you have choices like: shell scripts C code that performs the desired functions in the same way as ifconfig (userland) I can't see any immediate reason why you'd call functions like lagg_input() directly and you would have to be kernel-side to do so. What are you trying to achieve? Hope this helps! Dominic From owner-freebsd-net@FreeBSD.ORG Sat Jul 26 12:22:55 2014 Return-Path: Delivered-To: freebsd-net@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 9C9A51CB for ; Sat, 26 Jul 2014 12:22:55 +0000 (UTC) Received: from mail.unitedinsong.com.au (mail.unitedinsong.com.au [150.101.178.33]) (using TLSv1 with cipher DHE-RSA-CAMELLIA256-SHA (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 566B126EC for ; Sat, 26 Jul 2014 12:22:55 +0000 (UTC) Received: from laptop3.herveybayaustralia.com.au (laptop3.herveybayaustralia.com.au [192.168.0.185]) (using TLSv1 with cipher DHE-RSA-AES128-SHA (128/128 bits)) (No client certificate requested) by mail.unitedinsong.com.au (Postfix) with ESMTPSA id C9DE427376 for ; Sat, 26 Jul 2014 22:22:52 +1000 (EST) Message-ID: <53D39D9C.8040407@herveybayaustralia.com.au> Date: Sat, 26 Jul 2014 22:22:52 +1000 From: Da Rock User-Agent: Mozilla/5.0 (X11; FreeBSD amd64; rv:24.0) Gecko/20100101 Thunderbird/24.3.0 MIME-Version: 1.0 To: "freebsd-net@freebsd.org" Subject: systat -ifstat with missing statistics Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 26 Jul 2014 12:22:55 -0000 I came across this in my late night searching that this was not new, but I can't seem to find it again. May not even have been on this list. I just wanted to say "me too". New install of FreeBSD 10, trying out the new wlan drivers and I wanted to see how fast it was actually going to compare with the old. Ran systat -ifstat and I only get half the results! I had some heavy network going, and it was going quite fast I know, just not how fast exactly. I could see the incoming but not the outgoing. Unfortunately my traffic was outgoing... I tried incoming after and got some good results, but still this is not a good position to be in. Anyone on this one? Or is a pr required? Cheers From owner-freebsd-net@FreeBSD.ORG Sat Jul 26 16:02:40 2014 Return-Path: Delivered-To: freebsd-net@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 634393E4 for ; Sat, 26 Jul 2014 16:02:40 +0000 (UTC) Received: from mail.bsdinfo.com.br (mail.bsdinfo.com.br [67.212.89.78]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 326CA283C for ; Sat, 26 Jul 2014 16:02:39 +0000 (UTC) Received: from mail.bsdinfo.com.br (mail.bsdinfo.com.br [127.0.0.1]) by mail.bsdinfo.com.br (Postfix) with ESMTP id BFCD0139C8 for ; Sat, 26 Jul 2014 13:02:47 -0300 (BRT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=bsdinfo.com.br; h=content-transfer-encoding:content-type:content-type :in-reply-to:references:subject:subject:to:mime-version :user-agent:from:from:date:date:message-id; s=dkim; t= 1406390565; x=1407254566; bh=r/9kw+U3WSJJK8acs5zNrfSVbqO7BH/5Ijz v6P6fSq4=; b=BUF5bmMQFag7I7zBb9HkwdBAqd0fuFy5TiF1RkNfgcrdqAOpNhg kDOVfXdMAO0bd0O2Ow9r97bHmJripUmTtxF1QHJyZbVsWZ9J1nhxLqJZvYKJepqI ZzcITRnH0x93G8gxxjuNDbz2Dd0KVTdcUq7522fGnsOq1ZJXUstQ2v1Y= X-Virus-Scanned: amavisd-new at mail.bsdinfo.com.br Received: from mail.bsdinfo.com.br ([127.0.0.1]) by mail.bsdinfo.com.br (mail.bsdinfo.com.br [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id 8mKuMt4dmVmu for ; Sat, 26 Jul 2014 13:02:45 -0300 (BRT) Received: from [192.168.88.24] (unknown [186.193.48.8]) by mail.bsdinfo.com.br (Postfix) with ESMTPSA id A1D1A139C4 for ; Sat, 26 Jul 2014 13:02:45 -0300 (BRT) Message-ID: <53D3D118.9060307@bsdinfo.com.br> Date: Sat, 26 Jul 2014 13:02:32 -0300 From: Marcelo Gondim User-Agent: Mozilla/5.0 (Windows NT 6.3; WOW64; rv:24.0) Gecko/20100101 Thunderbird/24.6.0 MIME-Version: 1.0 To: FreeBSD Net Subject: Re: IPv6 alias do not respond from outside - resolved References: <53D2EAAF.8070707@bsdinfo.com.br> In-Reply-To: <53D2EAAF.8070707@bsdinfo.com.br> Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 26 Jul 2014 16:02:40 -0000 Em 25/07/2014 20:39, Marcelo Gondim escreveu: > Dear, > > I'm having the following problem: > > Host A (my network): > =================== > em0: flags=8843 metric 0 mtu 1500 > options=4219b > > ether 00:1e:67:30:ea:27 > inet 186.xxx.48.3 netmask 0xffffffe0 broadcast 186.xxx.48.31 > inet6 fe80::21e:67ff:fe30:ea27%em0 prefixlen 64 scopeid 0x1 > inet6 2805:xxxx:dead::3 prefixlen 64 > inet 186.xxx.48.4 netmask 0xffffffff broadcast 186.xxx.48.4 > inet 186.xxx.48.5 netmask 0xffffffff broadcast 186.xxx.48.5 > inet6 2805:xxxx:dead::5 prefixlen 64 > inet6 2805:xxxx:dead::4 prefixlen 64 > nd6 options=21 > media: Ethernet autoselect (1000baseT ) > status: active > > Router (Internet): > =============== > vlan0: flags=8843 metric 0 mtu > 1500 > options=303 > ether 00:1b:21:89:25:32 > inet 186.xxx.48.1 netmask 0xffffffe0 broadcast 186.xxx.48.31 > inet6 fe80::21b:21ff:fe89:2532%vlan0 prefixlen 64 scopeid 0x10 > inet6 2805:xxxx:dead::1 prefixlen 64 > nd6 options=21 > media: Ethernet autoselect (10Gbase-SR ) > status: active > vlan: 3081 parent interface: ix0 > > Host B (Datacenter OVH): > ====================== > eth0 Link encap:Ethernet HWaddr 00:27:0e:36:67:ae > inet addr:91.xxx.115.215 Bcast:91.xxx.115.255 > Mask:255.255.255.0 > inet6 addr: fe80::227:eff:fe36:67ae/64 Scope:Link > inet6 addr: 2001:xxxx:1:aad7::1/64 Scope:Global > UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1 > RX packets:680611577 errors:0 dropped:115467 overruns:0 frame:0 > TX packets:410997111 errors:0 dropped:0 overruns:0 carrier:0 > collisions:0 txqueuelen:1000 > RX bytes:929626301412 (865.7 GiB) TX bytes:28858569748 > (26.8 GiB) > > > > If I try to ping the origin Host Bto destination 2805:xxxx:dead::3, I > get an answer correctly: > > (hostb)# ping6 2805:xxxx:dead::3 > PING 2805:xxxx:dead::3(2805:xxxx:dead::3) 56 data bytes > 64 bytes from 2805:xxxx:dead::3: icmp_seq=1 ttl=55 time=219 ms > 64 bytes from 2805:xxxx:dead::3: icmp_seq=2 ttl=55 time=218 ms > 64 bytes from 2805:xxxx:dead::3: icmp_seq=3 ttl=55 time=219 ms > 64 bytes from 2805:xxxx:dead::3: icmp_seq=4 ttl=55 time=218 ms > > But it does not ping 2805:xxxx:dead::4 and 2805:xxxx:dead::5: > > (hostb)# ping6 2805:xxxx:dead::4 > PING 2805:xxxx:dead::4(2805:xxxx:dead::4) 56 data bytes > From 2001:xxxx:2001:1001::96 icmp_seq=2 Destination unreachable: > Address unreachable > From 2001:xxxx:2001:1001::96 icmp_seq=8 Destination unreachable: > Address unreachable > > Now if I do a ping from the router to Host A, it goes to work: > > (router)# ping6 2805:xxxx:dead::4 > PING6(56=40+8+8 bytes) 2805:xxxx:dead::1 --> 2805:xxxx:dead::4 > 16 bytes from 2805:xxxx:dead::4, icmp_seq=0 hlim=64 time=13.550 ms > 16 bytes from 2805:xxxx:dead::4, icmp_seq=1 hlim=64 time=7.776 ms > > (hostb)# ping6 2805:xxxx:dead::4 > PING 2805:xxxx:dead::4(2805:xxxx:dead::4) 56 data bytes > 64 bytes from 2805:xxxx:dead::4: icmp_seq=1 ttl=56 time=219 ms > 64 bytes from 2805:xxxx:dead::4: icmp_seq=2 ttl=56 time=218 ms > 64 bytes from 2805:xxxx:dead::4: icmp_seq=3 ttl=56 time=219 ms > > Why work now? > > FreeBSD xxxxx 10.0-STABLE FreeBSD 10.0-STABLE #10 r267839: Thu Jul 10 > 15:35:04 BRT 2014 root@xxxxx:/usr/obj/usr/src/sys/GONDIM amd64 > > Thanks and best regards, > Gondim > > _______________________________________________ > freebsd-net@freebsd.org mailing list > http://lists.freebsd.org/mailman/listinfo/freebsd-net > To unsubscribe, send any mail to "freebsd-net-unsubscribe@freebsd.org" > Dear, The problem was only one Firewall rule. Sorry. Gondim From owner-freebsd-net@FreeBSD.ORG Sat Jul 26 21:40:15 2014 Return-Path: Delivered-To: freebsd-net@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id E9EDC466 for ; Sat, 26 Jul 2014 21:40:15 +0000 (UTC) Received: from mail-qa0-x22f.google.com (mail-qa0-x22f.google.com [IPv6:2607:f8b0:400d:c00::22f]) (using TLSv1 with cipher ECDHE-RSA-RC4-SHA (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id AC9012408 for ; Sat, 26 Jul 2014 21:40:15 +0000 (UTC) Received: by mail-qa0-f47.google.com with SMTP id i13so6061475qae.20 for ; Sat, 26 Jul 2014 14:40:14 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:sender:in-reply-to:references:date:message-id:subject :from:to:cc:content-type; bh=PX35QV4L498XJUyzQvlI0+R48E8TpW3lcUQPYLtQxFg=; b=0WdG0/FOSPh88SaKPQDex4t8qZksBpnUIidhWgoYiH+lpg+42tzaaVCauCqgSY0qfF QO3mvzMGMJMJ8lhvcn/b4B++5G1jPrhVTPkitEHSbU+y77vQFAamLp9/rGJOQRKgIn/I 91Ca+Wl6GoBberFQnjfCO+smQFZABEN+t2UhNZ5v7VCEvSHSfhIB0oZZxmWsNOaIju70 zG0THSADLVgKsey8zH2rrhk62ZVNByo1Vgp0bezTqvYqCPTuIB3Qyi8XoTGAQTken1mL 5SQ0v8uGtX9WwGE7oYBRokUm7t3x2akxDeLjge25IVxgpKHihGNo5eS4egcnBMmG138r BJrQ== MIME-Version: 1.0 X-Received: by 10.140.38.169 with SMTP id t38mr41131757qgt.3.1406410814895; Sat, 26 Jul 2014 14:40:14 -0700 (PDT) Sender: adrian.chadd@gmail.com Received: by 10.224.1.6 with HTTP; Sat, 26 Jul 2014 14:40:14 -0700 (PDT) In-Reply-To: <53D39D9C.8040407@herveybayaustralia.com.au> References: <53D39D9C.8040407@herveybayaustralia.com.au> Date: Sat, 26 Jul 2014 14:40:14 -0700 X-Google-Sender-Auth: UlRFLiClKaEon2IsrTap7CdXefc Message-ID: Subject: Re: systat -ifstat with missing statistics From: Adrian Chadd To: Da Rock Content-Type: text/plain; charset=UTF-8 Cc: "freebsd-net@freebsd.org" X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 26 Jul 2014 21:40:16 -0000 Hi, Someone just needs to go through and audit which place(s) the physical interface (ath0, iwn0, etc) and the VAPs (wlan0, wlan1, etc) should have their interface statistics updated. I recall finding that this isn't consistently done by all NICs but then I kinda got burnt out doing wifi stuff. I'd appreciate some help. Thanks, -a On 26 July 2014 05:22, Da Rock wrote: > I came across this in my late night searching that this was not new, but I > can't seem to find it again. May not even have been on this list. I just > wanted to say "me too". > > New install of FreeBSD 10, trying out the new wlan drivers and I wanted to > see how fast it was actually going to compare with the old. Ran systat > -ifstat and I only get half the results! > > I had some heavy network going, and it was going quite fast I know, just not > how fast exactly. I could see the incoming but not the outgoing. > Unfortunately my traffic was outgoing... I tried incoming after and got some > good results, but still this is not a good position to be in. > > Anyone on this one? Or is a pr required? > > Cheers > _______________________________________________ > freebsd-net@freebsd.org mailing list > http://lists.freebsd.org/mailman/listinfo/freebsd-net > To unsubscribe, send any mail to "freebsd-net-unsubscribe@freebsd.org"