Date: Sat, 31 Aug 2013 05:41:53 -0700 (PDT) From: Barney Cordoba <barney_cordoba@yahoo.com> To: Luigi Rizzo <rizzo@iet.unipi.it>, Alan Somers <asomers@freebsd.org> Cc: Jack F Vogel <jfv@freebsd.org>, "T.C. Gubatayao" <tgubatayao@barracuda.com>, "Justin T. Gibbs" <gibbs@freebsd.org>, Andre Oppermann <andre@freebsd.org>, "net@freebsd.org" <net@freebsd.org> Subject: Re: Flow ID, LACP, and igb Message-ID: <1377952913.44129.YahooMailNeo@web121605.mail.ne1.yahoo.com> In-Reply-To: <CA%2BhQ2%2BhgTaK1ZCOLGVFjSPY8nyNPHK4waSecyRQxR1gQcyjztg@mail.gmail.com> References: <D01A0CB2-B1E3-4F4B-97FA-4C821C0E3FD2@FreeBSD.org> <521BBD21.4070304@freebsd.org> <CAOtMX2jvKGY==t9i-a_8RtMAPH2p1VDj950nMHHouryoz3nbsA@mail.gmail.com> <521EE8DA.3060107@freebsd.org> <BCC2C62D4FE171479E2F1C2593FE508B0BE24383@BN-SCL-MBX03.Cudanet.local> <CAOtMX2h5SGh5eYV50y%2BQB_s367V9iattGU862wwXcONDV%2BTG8g@mail.gmail.com> <CA%2BhQ2%2BhgTaK1ZCOLGVFjSPY8nyNPHK4waSecyRQxR1gQcyjztg@mail.gmail.com>
next in thread | previous in thread | raw e-mail | index | archive | help
May I express my glee and astonishment that you're debating the use of complicated hash functions for something that's likely to have from 2-8 slots? Also, the *most* important thing is distribution with realistic data. The goal should be to use the most trivial function that gives the most balanced distribution with real numbers. Faster is not better if the result is an unbalanced distribution. Many of your ports will be 80 and 53, and if you're going through a router your ethernets may not be very unique, so why even bother to include them? Does getting a good distribution require that you hash every element individually, or can you get the same distribution with a faster, simpler way of creating the seed? There's also the other consideration of packet size. Packets on port 53 are likely to be smaller than packets on port 80. What you want is equal distribution PER PORT on the ports that will carry that vast majority of your traffic. When designing efficient systems, you must not assume that ports and IPs are random, because they're not. 99% of your load will be on a small number of destination ports and a limited range of source ports. For a web server application, geting a perfect distribution on the http ports is most crucial. The hash function in if_lagg.c looks like more of a classroom exercise than a practical implementation. If you're going to consider 100M iterations; consider that much of the time is wasted parsing the packet (again). Why not add a simple sysctl that enables a hash that is created in the ip parser, when all of the pieces are available without having to re-parse the mbuf? Or better yet, use the same number of queues on igb as you have LAGG ports, and use the queue id (or RSS) as the hash, so that your traffic is sync'd between the ethernet adapter queues and the LAGG ports. The card has already done the work for you. BC ________________________________ From: Luigi Rizzo <rizzo@iet.unipi.it> To: Alan Somers <asomers@freebsd.org> Cc: Jack F Vogel <jfv@freebsd.org>; "net@freebsd.org" <net@freebsd.org>; Justin T. Gibbs <gibbs@freebsd.org>; Andre Oppermann <andre@freebsd.org>; T.C. Gubatayao <tgubatayao@barracuda.com> Sent: Friday, August 30, 2013 8:04 PM Subject: Re: Flow ID, LACP, and igb Alan, On Thu, Aug 29, 2013 at 6:45 PM, Alan Somers <asomers@freebsd.org> wrote: > > > ... > I pulled all four hash functions out into userland and microbenchmarked > them. The upshot is that hash32 and fnv_hash are the fastest, jenkins_hash > is slower, and siphash24 is the slowest. Also, Clang resulted in much > faster code than gcc. > > i missed this part of your message, but if i read your code well, you are running 100M iterations and the numbers below are in seconds, so if you multiply the numbers by 10 you have the cost per hash in nanoseconds. What CPU did you use for your tests ? Also some of the numbers (FNV and hash32) are suspiciously low. I believe that the compiler (both of them) have figure out that everything is constant in these functions, and fnv_32_buf() and hash32_buf() are inline, hence they can be optimized to just return a constant. This does not happen for siphash and jenkins because they are defined externally. Can you please re-run the tests in a way that defeats the optimization ? (e.g. pass a non constant argument to the the hashes so you actually need to run the code). cheers luigi http://people.freebsd.org/~asomers/lagg_hash/ > > [root@sm4u-4 /usr/home/alans/ctest/lagg_hash]# ./lagg_hash-gcc-4.8 > FNV: 0.76 > hash32: 1.18 > SipHash24: 44.39 > Jenkins: 6.20 > [root@sm4u-4 /usr/home/alans/ctest/lagg_hash]# ./lagg_hash-gcc-4.2.1 > FNV: 0.74 > hash32: 1.35 > SipHash24: 55.25 > Jenkins: 7.37 > [root@sm4u-4 /usr/home/alans/ctest/lagg_hash]# ./lagg_hash.clang-3.3 > FNV: 0.30 > hash32: 0.30 > SipHash24: 55.97 > Jenkins: 6.45 > [root@sm4u-4 /usr/home/alans/ctest/lagg_hash]# ./lagg_hash.clang-3.2 > FNV: 0.30 > hash32: 0.30 > SipHash24: 44.52 > Jenkins: 6.48 > > > > > T.C. > > > > [1] > > > http://svnweb.freebsd.org/base/head/sys/libkern/jenkins_hash.c?view=markup > _______________________________________________ > freebsd-net@freebsd.org mailing list > http://lists.freebsd.org/mailman/listinfo/freebsd-net > To unsubscribe, send any mail to "freebsd-net-unsubscribe@freebsd.org" > _______________________________________________ freebsd-net@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-net To unsubscribe, send any mail to "freebsd-net-unsubscribe@freebsd.org" From owner-freebsd-net@FreeBSD.ORG Sat Aug 31 14:50:37 2013 Return-Path: <owner-freebsd-net@FreeBSD.ORG> Delivered-To: net@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTP id 6E9D6602 for <net@freebsd.org>; Sat, 31 Aug 2013 14:50:37 +0000 (UTC) (envelope-from barney_cordoba@yahoo.com) Received: from nm13-vm5.bullet.mail.ne1.yahoo.com (nm13-vm5.bullet.mail.ne1.yahoo.com [98.138.91.235]) (using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits)) (No client certificate requested) by mx1.freebsd.org (Postfix) with ESMTPS id 2F39F25BB for <net@freebsd.org>; Sat, 31 Aug 2013 14:50:36 +0000 (UTC) Received: from [98.138.226.179] by nm13.bullet.mail.ne1.yahoo.com with NNFMP; 31 Aug 2013 14:47:48 -0000 Received: from [98.138.88.239] by tm14.bullet.mail.ne1.yahoo.com with NNFMP; 31 Aug 2013 14:47:48 -0000 Received: from [127.0.0.1] by omp1039.mail.ne1.yahoo.com with NNFMP; 31 Aug 2013 14:47:48 -0000 X-Yahoo-Newman-Property: ymail-3 X-Yahoo-Newman-Id: 11055.63437.bm@omp1039.mail.ne1.yahoo.com Received: (qmail 1521 invoked by uid 60001); 31 Aug 2013 14:47:47 -0000 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=yahoo.com; s=s1024; t77960467; bh‘z46d+R+3onCaet/dXTIaEVGu76b3VAiwEAhziMStI=; h=X-YMail-OSG:Received:X-Rocket-MIMEInfo:X-Mailer:References:Message-ID:Date:From:Reply-To:Subject:To:Cc:In-Reply-To:MIME-Version:Content-Type; b=0vc/X0MuzCRm0Pe3apvKdA2M40v+jeAPakoMKmmcezHv4FpA418BfWX8f+0wN38qOI4EqLPw6/oV8V4zpwR/xgR3okQxQmu0/px1FIKnh1I/2LFr0QOf9xMW7oVgY798D0dIeej4QDaNblx3fUGULgR08Scm7NbHlplxEz7/aTwDomainKey-Signature: a=rsa-sha1; q=dns; c=nofws; s=s1024; d=yahoo.com; h=X-YMail-OSG:Received:X-Rocket-MIMEInfo:X-Mailer:References:Message-ID:Date:From:Reply-To:Subject:To:Cc:In-Reply-To:MIME-Version:Content-Type; b=eO+dtiWX9Ukfsx7CmNIEay2MFKeHjU2zVSxXsYMoM+dRwhtXLC3+tNi2x4AsW1gNjhx2wOnj4B2clfyjNVCI4mQcTdra1bGkWlQNeDvxahig6Ewwyq7/exkb4McIoRBnaRsPr13xRiMsB4sA4walhmCv5fX5gBLI94AZgsxMarQ=; X-YMail-OSG: upHBLR4VM1kga1rhHiwyEXJI5MEDL17tFUKdbT7XFWh6rKD dTA_nZG9a2OdVZW7dGnNzW64q0AiCITb7GjphParmIq7qWUnw8irC60HvmYV Pja3Q1CbuYY1C_hNWkYeO.D1dHsFgxyWyYG7Znx5WHP7Sbqk9TsPnulk3CzX 8CznhUBzvNYg7iq4Dr.gLK3DX9NOBv8mmBS48EMApkNdpV6aJx91DwG1zn_N WMOIN7gnheymnx357Cv1suudwm8In0UzSJM07f9URCEGXwTrE4YTi.5cY4m6 .QVarhLcpZ556bfIQpHW8jq_HKi6ybnKdjP3_14MyeQpquQI.oVasgGsfleK toRbHI9rHfwNwq9IcZoWpYUv2A3KVU07gWLQSDZgxQA5zF95LcyZGGIvNTTi bCuXu9lHalBSwptou_BZl5jtTtG_VErF.ESYkmIi5gkEZqCuEejxLbQNZKd6 R77O5LtTtitak5EkfNc7rBaKPJl2Lqx0XQUCrT8LD1RV5JX5JrSaCNbJ7x_h S0x8h9.Jt6iWCaiV5VX86AXe4dr.U6Hx1GCg0bQaNVBko29oNALkTyyPU_E0 - Received: from [98.203.118.124] by web121603.mail.ne1.yahoo.com via HTTP; Sat, 31 Aug 2013 07:47:47 PDT X-Rocket-MIMEInfo: 002.001, QW5kIGFub3RoZXIgdGhpbmc7IHRoZSB1c2Ugb2YgbW9kdWxvIGlzIHZlcnkgZXhwZW5zaXZlIHdoZW4gdGhlIG51bWJlciBvZiBwb3J0cwp1c2VkIGluIExBR0cgaXMgKnVzdWFsbHkqIGEgcG93ZXIgb2YgMi4gZm9vJihTTE9UUy0xKSBpcyBhIGxvdCBmYXN0ZXIgdGhhbiAoZm9vJVNMT1RTKS7CoAoKaWYgKFNMT1RTID09IDIgfHwgU0xPVFMgPT0gNCB8fCBTTE9UUyA9PSA4KQrCoCDCoCBoYXNoID0gaGFzaCYoU0xPVFMtMSk7CmVsc2UKwqAgwqAgaGFzaCA9IGhhc2ggJSBTTE9UUzsKCmlzIG1vcmUgdGhhbiABMAEBAQE- X-Mailer: YahooMailWebService/0.8.156.576 References: <D01A0CB2-B1E3-4F4B-97FA-4C821C0E3FD2@FreeBSD.org> <521BBD21.4070304@freebsd.org> <CAOtMX2jvKGY==t9i-a_8RtMAPH2p1VDj950nMHHouryoz3nbsA@mail.gmail.com> <521EE8DA.3060107@freebsd.org> <BCC2C62D4FE171479E2F1C2593FE508B0BE24383@BN-SCL-MBX03.Cudanet.local> <CAOtMX2h5SGh5eYV50y+QB_s367V9iattGU862wwXcONDV+TG8g@mail.gmail.com> <CA+hQ2+hgTaK1ZCOLGVFjSPY8nyNPHK4waSecyRQxR1gQcyjztg@mail.gmail.com> Message-ID: <1377960467.93173.YahooMailNeo@web121603.mail.ne1.yahoo.com> Date: Sat, 31 Aug 2013 07:47:47 -0700 (PDT) From: Barney Cordoba <barney_cordoba@yahoo.com> Subject: Re: Flow ID, LACP, and igb To: Luigi Rizzo <rizzo@iet.unipi.it>, Alan Somers <asomers@freebsd.org> In-Reply-To: <CA+hQ2+hgTaK1ZCOLGVFjSPY8nyNPHK4waSecyRQxR1gQcyjztg@mail.gmail.com> MIME-Version: 1.0 Content-Type: text/plain; charset=iso-8859-1 Content-Transfer-Encoding: quoted-printable X-Content-Filtered-By: Mailman/MimeDel 2.1.14 Cc: Jack F Vogel <jfv@freebsd.org>, "T.C. Gubatayao" <tgubatayao@barracuda.com>, "Justin T. Gibbs" <gibbs@freebsd.org>, Andre Oppermann <andre@freebsd.org>, "net@freebsd.org" <net@freebsd.org> X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list Reply-To: Barney Cordoba <barney_cordoba@yahoo.com> List-Id: Networking and TCP/IP with FreeBSD <freebsd-net.freebsd.org> List-Unsubscribe: <http://lists.freebsd.org/mailman/options/freebsd-net>, <mailto:freebsd-net-request@freebsd.org?subject=unsubscribe> List-Archive: <http://lists.freebsd.org/pipermail/freebsd-net>; List-Post: <mailto:freebsd-net@freebsd.org> List-Help: <mailto:freebsd-net-request@freebsd.org?subject=help> List-Subscribe: <http://lists.freebsd.org/mailman/listinfo/freebsd-net>, <mailto:freebsd-net-request@freebsd.org?subject=subscribe> X-List-Received-Date: Sat, 31 Aug 2013 14:50:37 -0000 And another thing; the use of modulo is very expensive when the number of ports used in LAGG is *usually* a power of 2. foo&(SLOTS-1) is a lot faster than (foo%SLOTS). if (SLOTS == 2 || SLOTS == 4 || SLOTS == 8) hash = hash&(SLOTS-1); else hash = hash % SLOTS; is more than twice as fast as hash % SLOTS; BC ________________________________ From: Luigi Rizzo <rizzo@iet.unipi.it> To: Alan Somers <asomers@freebsd.org> Cc: Jack F Vogel <jfv@freebsd.org>; "net@freebsd.org" <net@freebsd.org>; Justin T. Gibbs <gibbs@freebsd.org>; Andre Oppermann <andre@freebsd.org>; T.C. Gubatayao <tgubatayao@barracuda.com> Sent: Friday, August 30, 2013 8:04 PM Subject: Re: Flow ID, LACP, and igb Alan, On Thu, Aug 29, 2013 at 6:45 PM, Alan Somers <asomers@freebsd.org> wrote: > > > ... > I pulled all four hash functions out into userland and microbenchmarked > them. The upshot is that hash32 and fnv_hash are the fastest, jenkins_hash > is slower, and siphash24 is the slowest. Also, Clang resulted in much > faster code than gcc. > > i missed this part of your message, but if i read your code well, you are running 100M iterations and the numbers below are in seconds, so if you multiply the numbers by 10 you have the cost per hash in nanoseconds. What CPU did you use for your tests ? Also some of the numbers (FNV and hash32) are suspiciously low. I believe that the compiler (both of them) have figure out that everything is constant in these functions, and fnv_32_buf() and hash32_buf() are inline, hence they can be optimized to just return a constant. This does not happen for siphash and jenkins because they are defined externally. Can you please re-run the tests in a way that defeats the optimization ? (e.g. pass a non constant argument to the the hashes so you actually need to run the code). cheers luigi http://people.freebsd.org/~asomers/lagg_hash/ > > [root@sm4u-4 /usr/home/alans/ctest/lagg_hash]# ./lagg_hash-gcc-4.8 > FNV: 0.76 > hash32: 1.18 > SipHash24: 44.39 > Jenkins: 6.20 > [root@sm4u-4 /usr/home/alans/ctest/lagg_hash]# ./lagg_hash-gcc-4.2.1 > FNV: 0.74 > hash32: 1.35 > SipHash24: 55.25 > Jenkins: 7.37 > [root@sm4u-4 /usr/home/alans/ctest/lagg_hash]# ./lagg_hash.clang-3.3 > FNV: 0.30 > hash32: 0.30 > SipHash24: 55.97 > Jenkins: 6.45 > [root@sm4u-4 /usr/home/alans/ctest/lagg_hash]# ./lagg_hash.clang-3.2 > FNV: 0.30 > hash32: 0.30 > SipHash24: 44.52 > Jenkins: 6.48 > > > > > T.C. > > > > [1] > > > http://svnweb.freebsd.org/base/head/sys/libkern/jenkins_hash.c?view=markup > _______________________________________________ > freebsd-net@freebsd.org mailing list > http://lists.freebsd.org/mailman/listinfo/freebsd-net > To unsubscribe, send any mail to "freebsd-net-unsubscribe@freebsd.org" > _______________________________________________ freebsd-net@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-net To unsubscribe, send any mail to "freebsd-net-unsubscribe@freebsd.org" From owner-freebsd-net@FreeBSD.ORG Sat Aug 31 15:05:23 2013 Return-Path: <owner-freebsd-net@FreeBSD.ORG> Delivered-To: freebsd-net@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTP id 701BDB81 for <freebsd-net@freebsd.org>; Sat, 31 Aug 2013 15:05:23 +0000 (UTC) (envelope-from barney_cordoba@yahoo.com) Received: from nm21-vm4.bullet.mail.ne1.yahoo.com (nm21-vm4.bullet.mail.ne1.yahoo.com [98.138.91.181]) (using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits)) (No client certificate requested) by mx1.freebsd.org (Postfix) with ESMTPS id 1EA6D2656 for <freebsd-net@freebsd.org>; Sat, 31 Aug 2013 15:05:22 +0000 (UTC) Received: from [98.138.101.128] by nm21.bullet.mail.ne1.yahoo.com with NNFMP; 31 Aug 2013 15:03:08 -0000 Received: from [98.138.89.248] by tm16.bullet.mail.ne1.yahoo.com with NNFMP; 31 Aug 2013 15:03:08 -0000 Received: from [127.0.0.1] by omp1040.mail.ne1.yahoo.com with NNFMP; 31 Aug 2013 15:03:08 -0000 X-Yahoo-Newman-Property: ymail-3 X-Yahoo-Newman-Id: 307911.32973.bm@omp1040.mail.ne1.yahoo.com Received: (qmail 38024 invoked by uid 60001); 31 Aug 2013 15:03:08 -0000 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=yahoo.com; s=s1024; t77961388; bh=Y7fnNQoRaB9Mi4aMrvBp20u0Jchwx0rrwWWy0Jz3lMw=; h=X-YMail-OSG:Received:X-Rocket-MIMEInfo:X-Mailer:References:Message-ID:Date:From:Reply-To:Subject:To:In-Reply-To:MIME-Version:Content-Type; b=mYfGjAfYimzkoy0S2YCm47jExxgCxB9/sGt60fEAmtyGc88ohT79tDls79W6OI59UNLaSGxQDL2h89St8qo4G4b//1wlq9zAvjlPQjyvTkdhS45pNJmGLMLgyNBRdfXaq4b4MHK9bNZ9gwjavxPmO+jM6DQB8+cbgEdSuV2En9YDomainKey-Signature: a=rsa-sha1; q=dns; c=nofws; s=s1024; d=yahoo.com; h=X-YMail-OSG:Received:X-Rocket-MIMEInfo:X-Mailer:References:Message-ID:Date:From:Reply-To:Subject:To:In-Reply-To:MIME-Version:Content-Type; b=ExHGsnoQn35p3cmBt8J5dbDx+EClbaAS6h12mTmHDbzdPQgbYqYkn0v6paHO16ot1K+rn9e5lK+y5iGaa/ieFrTdJ8aocOSVSWIt1eHfXGqCYGI/HY3VKDu4GK/wNsN+i6G5SoW/tuofW8Tf2kvmYFq3Jz28Xb/hltoSgd7qjfY=; X-YMail-OSG: ob_reFgVM1lDffF6LE7hWbGzHbMGfKgIk0e6GWQyyC3FBT9 noVa6TSxl70hWFu9H3ep9GCTM0XrY0v.r8xdn0a3f4Dtv_9LvAI9UWM2rd8g o2TGdzGUYgdULWWfaxS_OgR1YU0XUcYRQRcCYViVK.V_89x6EWMqmJxUCvUk xp_l1ocXjkdwEp.6G5huw8624Wv0jchbKkoOXTPytM63KMLpwPV4aE_quPtS 94zfqrMowraoCynekyrfSnoW5xugQvykV4VKgpXP3VKlgC3in0uhOwLvEXCI .FhKTLbDrMeI.Wlv5PIypHX6TgnPJ_h91Lr4vByOHCEAF2jjIp4FoXtOQvoV hRmDmAS_1V9mezXJazZPjFEpeH63Wdemqi1HZnHzQY3hftGqvB5tIJ8V1Cw4 KN2EVxQIbhceL.pJN5kV_hxBtvzpBfwU_BKC7evZqMA5IeKjrnq7b1r07IVJ zripu_ZMp0Bi7QyunKQoaBSV5y_fxYi0EEHZIG4PFsNa.79N8Ck1ijCWvLrV Nu34TTEk7KdtQtx6e6gS41Zuy.fcwVtbtnF69Wcb0tyTAOYHVez2BVHNwokC L.VsnOocD1zqLDPIvlyQnT6ZAKENkgfWce8Via6y8n99ibWbQj1RG559YcJY lvoDJhCWhgjc- Received: from [98.203.118.124] by web121601.mail.ne1.yahoo.com via HTTP; Sat, 31 Aug 2013 08:03:08 PDT X-Rocket-MIMEInfo: 002.001, VGhhdCdzIHdheSB0b28gaGlnaC4gWW91ciBiYXNlIHJ4IHJlcXVpcmVtZW50IGlzwqAKClBvcnRzICogcXVldWVzICogcnhkwqAKCldpdGggYSBxdWFkIGNhcmQgeW91IHNob3VsZG4ndCBiZSB1c2luZyBtb3JlIHRoYW4gMiBxdWV1ZXMsIHNvIHlvdXIgcmVxdWlyZW1lbnQKd2l0aCA1IHBvcnRzIGlzIDEwLDI0MCBqdXN0IGZvciB0aGUgcmVjZWl2ZSBzZXR1cC4gSWYgeW91J3JlIHVzaW5nIDQgcXVldWVzIHRoYXQKbnVtYmVyIGRvdWJsZXMsIHdoaWNoIHdvdWxkIG1ha2UgMjUsNjAwIG5vdCBlbm91Z2guwqABMAEBAQE- X-Mailer: YahooMailWebService/0.8.156.576 References: <B966242F-A52D-43F7-A001-99942D53339E@ebureau.com> <CAFMmRNwAuwaGLSQ4P-y=Vzh63jpGXoDRCOXbxeWPoVb3ucy0kQ@mail.gmail.com> <71042F7C-5CBB-4494-B53A-EF4CE45B41BE@ebureau.com> Message-ID: <1377961388.28903.YahooMailNeo@web121601.mail.ne1.yahoo.com> Date: Sat, 31 Aug 2013 08:03:08 -0700 (PDT) From: Barney Cordoba <barney_cordoba@yahoo.com> Subject: Re: Intel 4-port ethernet adaptor link aggregation issue To: Joe Moog <joemoog@ebureau.com>, freebsd-net <freebsd-net@freebsd.org> In-Reply-To: <71042F7C-5CBB-4494-B53A-EF4CE45B41BE@ebureau.com> MIME-Version: 1.0 Content-Type: text/plain; charset=iso-8859-1 Content-Transfer-Encoding: quoted-printable X-Content-Filtered-By: Mailman/MimeDel 2.1.14 X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list Reply-To: Barney Cordoba <barney_cordoba@yahoo.com> List-Id: Networking and TCP/IP with FreeBSD <freebsd-net.freebsd.org> List-Unsubscribe: <http://lists.freebsd.org/mailman/options/freebsd-net>, <mailto:freebsd-net-request@freebsd.org?subject=unsubscribe> List-Archive: <http://lists.freebsd.org/pipermail/freebsd-net>; List-Post: <mailto:freebsd-net@freebsd.org> List-Help: <mailto:freebsd-net-request@freebsd.org?subject=help> List-Subscribe: <http://lists.freebsd.org/mailman/listinfo/freebsd-net>, <mailto:freebsd-net-request@freebsd.org?subject=subscribe> X-List-Received-Date: Sat, 31 Aug 2013 15:05:23 -0000 That's way too high. Your base rx requirement is Ports * queues * rxd With a quad card you shouldn't be using more than 2 queues, so your requirement with 5 ports is 10,240 just for the receive setup. If you're using 4 queues that number doubles, which would make 25,600 not enough. Note that setting mbufs to a huge number doesn't allocate the buffers; they'll be allocated as needed. It's a ceiling. The reason for the ceiling is so that you don't blow up your memory. If your system is using 2 million mbuf clusters then you have much bigger problems than LAGG. Anyone who recommends 2 million clearly has no idea what they're doing. BC ________________________________ From: Joe Moog <joemoog@ebureau.com> To: freebsd-net <freebsd-net@freebsd.org> Sent: Wednesday, August 28, 2013 9:36 AM Subject: Re: Intel 4-port ethernet adaptor link aggregation issue All: Thanks again to everybody for the responses and suggestions to our 4-port lagg issue. The solution (for those that may find the information of some value) was to set the value for kern.ipc.nmbclusters to a higher value than we had initially. Our previous tuning had this value set at 25600, but following a recommendation from the good folks at iXSystems we bumped this to a value closer to 2000000, and the 4-port lagg is functioning as expected now. Thank you all. Joe _______________________________________________ freebsd-net@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-net To unsubscribe, send any mail to "freebsd-net-unsubscribe@freebsd.org"
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?1377952913.44129.YahooMailNeo>