From owner-freebsd-net@FreeBSD.ORG Sat Aug 31 12:42:01 2013 Return-Path: Delivered-To: net@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTP id 8CD4D135 for ; Sat, 31 Aug 2013 12:42:01 +0000 (UTC) (envelope-from barney_cordoba@yahoo.com) Received: from nm25-vm5.bullet.mail.ne1.yahoo.com (nm25-vm5.bullet.mail.ne1.yahoo.com [98.138.91.247]) (using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits)) (No client certificate requested) by mx1.freebsd.org (Postfix) with ESMTPS id 2D9DE2F49 for ; Sat, 31 Aug 2013 12:42:00 +0000 (UTC) Received: from [98.138.90.54] by nm25.bullet.mail.ne1.yahoo.com with NNFMP; 31 Aug 2013 12:41:54 -0000 Received: from [98.138.101.163] by tm7.bullet.mail.ne1.yahoo.com with NNFMP; 31 Aug 2013 12:41:54 -0000 Received: from [127.0.0.1] by omp1074.mail.ne1.yahoo.com with NNFMP; 31 Aug 2013 12:41:54 -0000 X-Yahoo-Newman-Property: ymail-3 X-Yahoo-Newman-Id: 318374.35467.bm@omp1074.mail.ne1.yahoo.com Received: (qmail 59789 invoked by uid 60001); 31 Aug 2013 12:41:54 -0000 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=yahoo.com; s=s1024; t=1377952914; bh=1EHxvoH8V8rYNzrqDND6z3u4ABlyi9RltR2y16rhNjM=; h=X-YMail-OSG:Received:X-Rocket-MIMEInfo:X-Mailer:References:Message-ID:Date:From:Reply-To:Subject:To:Cc:In-Reply-To:MIME-Version:Content-Type; b=IuPYlnq9for77UvA044H65tVACn8J8rrGkaggtYRk9oBEYPRNGaocWhiNl+55t4O0k+T+meCEoqsMPrStPnMFE/6kfXnavzYSEQ4g4ktl9p2vnI17KwKwdsJcS7SKGIOwiWXgRklGDllgETEE572LKAjDTG7B50xoyTWdDCuN30= DomainKey-Signature: a=rsa-sha1; q=dns; c=nofws; s=s1024; d=yahoo.com; h=X-YMail-OSG:Received:X-Rocket-MIMEInfo:X-Mailer:References:Message-ID:Date:From:Reply-To:Subject:To:Cc:In-Reply-To:MIME-Version:Content-Type; b=pZgO0B3Gp6x2NoRhrJNcEX//Z6ONaceILTU++jbpF7/md/0aDMGiF6IwdrlMDX02kOdTMTUv2Zg7VtKITj+waD0VJzr0qdGWa6SM5YNYmG1it6BiOUBjaHyV3DdWgM77+sl5mDM4MBaeNJyH+IUpv8mSkYSFHmgBqCYsgnTaOvE=; X-YMail-OSG: QfW5D34VM1l93hf7uMM_4bP0UYKih.R14jWrEjISQwicC47 cL3J4dHwNrXXlPpFwOofeUQAMx_oEjX2uuPpbDw92i6y.hslxlTZrJrUZ553 Z5GSp8PKTg2QfbttpiUokXmJFsMobRb2fZIVaeSiRNv4jpXsaYwpl_t6GFNu oVFGoFLBMjClwlqerC8Kds4njsyCUml3mSrsywxSM1D3ZjTR.qZEtVLSRepU O3UDJb0hRI5HYGpoP8K0Ei.iO_295YeE7YLmkXmeq5FUsmHU3rECLwL8lEuU KtLhRZCoaQogYuu10ub8QSvzKarUgieId4r80GJtFBWBwWikjdYuD.dtFKZY H5R8NYh9W_uPDYMU7up6PBip8_B9ZmjopyTvqeUSiIrg.qRLes9fTCG1me40 NHRqYs.2x34Od3wYiIO36dO32geXiN0QaHCbCHQzErb3j2lUAdX4naFJcR9Y D3bH1nhNnvggYk52rKciAqKFSu.5xANkElzE9NUEDs60PsUdediAa_IAfrPa vgjFFiGD8n7RJO3NbY0uDDRwj3zFt3lpSI84R1Sp_S8ZRZThZ_WVJ.zjzKDi cQAPvq6.lggDfg7hDuwEe4Q-- Received: from [98.203.118.124] by web121605.mail.ne1.yahoo.com via HTTP; Sat, 31 Aug 2013 05:41:53 PDT X-Rocket-MIMEInfo: 002.001, TWF5IEkgZXhwcmVzcyBteSBnbGVlIGFuZCBhc3RvbmlzaG1lbnQgdGhhdCDCoHlvdSdyZSBkZWJhdGluZyB0aGUgdXNlIG9mIGNvbXBsaWNhdGVkIGhhc2ggZnVuY3Rpb25zCmZvciBzb21ldGhpbmcgdGhhdCdzIGxpa2VseSB0byBoYXZlIGZyb20gMi04IHNsb3RzPwoKQWxzbywgdGhlICptb3N0KiBpbXBvcnRhbnQgdGhpbmcgaXMgZGlzdHJpYnV0aW9uIHdpdGggcmVhbGlzdGljIGRhdGEuIFRoZSBnb2FsIHNob3VsZCBiZSB0byB1c2UgdGhlCm1vc3QgdHJpdmlhbCBmdW5jdGlvbiB0aGF0IGdpdmVzIHRoZSABMAEBAQE- X-Mailer: YahooMailWebService/0.8.156.576 References: <521BBD21.4070304@freebsd.org> <521EE8DA.3060107@freebsd.org> Message-ID: <1377952913.44129.YahooMailNeo@web121605.mail.ne1.yahoo.com> Date: Sat, 31 Aug 2013 05:41:53 -0700 (PDT) From: Barney Cordoba Subject: Re: Flow ID, LACP, and igb To: Luigi Rizzo , Alan Somers In-Reply-To: MIME-Version: 1.0 Content-Type: text/plain; charset=iso-8859-1 Content-Transfer-Encoding: quoted-printable X-Content-Filtered-By: Mailman/MimeDel 2.1.14 Cc: Jack F Vogel , "T.C. Gubatayao" , "Justin T. Gibbs" , Andre Oppermann , "net@freebsd.org" X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list Reply-To: Barney Cordoba List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 31 Aug 2013 12:42:01 -0000 May I express my glee and astonishment that =A0you're debating the use of c= omplicated hash functions=0Afor something that's likely to have from 2-8 sl= ots?=0A=0AAlso, the *most* important thing is distribution with realistic d= ata. The goal should be to use the=0Amost trivial function that gives the m= ost balanced distribution with real numbers. Faster is=0Anot better if the = result is an unbalanced distribution.=0A=0AMany of your ports will be 80 an= d 53, and if you're going through a router your ethernets=0Amay not be very= unique, so why even bother to include them? Does getting a good distributi= on=0Arequire that you hash every element individually, or can you get the s= ame distribution with=0Aa faster, simpler way of creating the seed?=0A=0ATh= ere's also the other consideration of packet size. Packets on port 53 are l= ikely to be smaller=0Athan packets on port 80. What you want is equal distr= ibution PER PORT on the ports that will=0Acarry that vast majority of your = traffic.=0A=0AWhen designing efficient systems, you must not assume that po= rts and IPs are random, because they're=0Anot. 99% of your load will be on = a small number of destination ports and a limited range of source ports.=0A= =0AFor a web server application, geting a perfect distribution on the http = ports is most crucial.=0A=0AThe hash function in if_lagg.c looks like more = of a classroom exercise than a practical implementation.=A0=0AIf you're goi= ng to consider 100M iterations; consider that much of the time is wasted pa= rsing the=0Apacket (again). Why not add a simple sysctl that enables a hash= that is created in the ip parser,=0Awhen all of the pieces are available w= ithout having to re-parse the mbuf?=0A=0AOr better yet, use the same number= of queues on igb as you have LAGG ports, and use the queue id (or RSS)=0Aa= s the hash, so that=A0your traffic is sync'd between the ethernet adapter q= ueues and the LAGG ports. The card=0Ahas already done the work for you.=0A= =0ABC=0A=0A=0A=0A=0A=0A________________________________=0A From: Luigi Rizz= o =0ATo: Alan Somers =0ACc: Jack = F Vogel ; "net@freebsd.org" ; Justin T. G= ibbs ; Andre Oppermann ; T.C. Gubatay= ao =0ASent: Friday, August 30, 2013 8:04 PM=0ASu= bject: Re: Flow ID, LACP, and igb=0A =0A=0AAlan,=0A=0A=0AOn Thu, Aug 29, 20= 13 at 6:45 PM, Alan Somers wrote:=0A>=0A>=0A> ...=0A>= I pulled all four hash functions out into userland and microbenchmarked=0A= > them.=A0 The upshot is that hash32 and fnv_hash are the fastest, jenkins_= hash=0A> is slower, and siphash24 is the slowest.=A0 Also, Clang resulted i= n much=0A> faster code than gcc.=0A>=0A>=0Ai missed this part of your messa= ge, but if i read your code well,=0Ayou are running 100M iterations and the= numbers below are in seconds,=0Aso if you multiply the numbers by 10 you h= ave the cost per hash in=0Ananoseconds.=0A=0AWhat CPU did you use for your = tests ?=0A=0AAlso some of the numbers (FNV and hash32) are suspiciously low= .=0A=0AI believe that the compiler (both of them) have figure out that ever= ything=0Ais constant in these functions, and fnv_32_buf() and hash32_buf() = are=0Ainline,=0Ahence they can be optimized to just return a constant.=0ATh= is does not happen for siphash and jenkins because they are defined=0Aexter= nally.=0A=0ACan you please re-run the tests in a way that defeats the optim= ization ?=0A(e.g. pass a non constant argument to the the hashes so you act= ually need=0Ato run the code).=0A=0Acheers=0Aluigi=0A=0A=0Ahttp://people.fr= eebsd.org/~asomers/lagg_hash/=0A>=0A> [root@sm4u-4 /usr/home/alans/ctest/la= gg_hash]# ./lagg_hash-gcc-4.8=0A> FNV: 0.76=0A> hash32: 1.18=0A> SipHash24:= 44.39=0A> Jenkins: 6.20=0A> [root@sm4u-4 /usr/home/alans/ctest/lagg_hash]#= ./lagg_hash-gcc-4.2.1=0A> FNV: 0.74=0A> hash32: 1.35=0A> SipHash24: 55.25= =0A> Jenkins: 7.37=0A> [root@sm4u-4 /usr/home/alans/ctest/lagg_hash]# ./lag= g_hash.clang-3.3=0A> FNV: 0.30=0A> hash32: 0.30=0A> SipHash24: 55.97=0A> Je= nkins: 6.45=0A> [root@sm4u-4 /usr/home/alans/ctest/lagg_hash]# ./lagg_hash.= clang-3.2=0A> FNV: 0.30=0A> hash32: 0.30=0A> SipHash24: 44.52=0A> Jenkins: = 6.48=0A>=0A>=0A>=0A> > T.C.=0A> >=0A> > [1]=0A> >=0A> http://svnweb.freebsd= .org/base/head/sys/libkern/jenkins_hash.c?view=3Dmarkup=0A> _______________= ________________________________=0A> freebsd-net@freebsd.org mailing list= =0A> http://lists.freebsd.org/mailman/listinfo/freebsd-net=0A> To unsubscri= be, send any mail to "freebsd-net-unsubscribe@freebsd.org"=0A>=0A__________= _____________________________________=0Afreebsd-net@freebsd.org mailing lis= t=0Ahttp://lists.freebsd.org/mailman/listinfo/freebsd-net=0ATo unsubscribe,= send any mail to "freebsd-net-unsubscribe@freebsd.org" From owner-freebsd-net@FreeBSD.ORG Sat Aug 31 14:50:37 2013 Return-Path: Delivered-To: net@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTP id 6E9D6602 for ; Sat, 31 Aug 2013 14:50:37 +0000 (UTC) (envelope-from barney_cordoba@yahoo.com) Received: from nm13-vm5.bullet.mail.ne1.yahoo.com (nm13-vm5.bullet.mail.ne1.yahoo.com [98.138.91.235]) (using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits)) (No client certificate requested) by mx1.freebsd.org (Postfix) with ESMTPS id 2F39F25BB for ; Sat, 31 Aug 2013 14:50:36 +0000 (UTC) Received: from [98.138.226.179] by nm13.bullet.mail.ne1.yahoo.com with NNFMP; 31 Aug 2013 14:47:48 -0000 Received: from [98.138.88.239] by tm14.bullet.mail.ne1.yahoo.com with NNFMP; 31 Aug 2013 14:47:48 -0000 Received: from [127.0.0.1] by omp1039.mail.ne1.yahoo.com with NNFMP; 31 Aug 2013 14:47:48 -0000 X-Yahoo-Newman-Property: ymail-3 X-Yahoo-Newman-Id: 11055.63437.bm@omp1039.mail.ne1.yahoo.com Received: (qmail 1521 invoked by uid 60001); 31 Aug 2013 14:47:47 -0000 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=yahoo.com; s=s1024; t=1377960467; bh=91z46d+R+3onCaet/dXTIaEVGu76b3VAiwEAhziMStI=; h=X-YMail-OSG:Received:X-Rocket-MIMEInfo:X-Mailer:References:Message-ID:Date:From:Reply-To:Subject:To:Cc:In-Reply-To:MIME-Version:Content-Type; b=0vc/X0MuzCRm0Pe3apvKdA2M40v+jeAPakoMKmmcezHv4FpA418BfWX8f+0wN38qOI4EqLPw6/oV8V4zpwR/xgR3okQxQmu0/px1FIKnh1I/2LFr0QOf9xMW7oVgY798D0dIeej4QDaNblx3fUGULgR08Scm7NbHlplxEz7/aTw= DomainKey-Signature: a=rsa-sha1; q=dns; c=nofws; s=s1024; d=yahoo.com; h=X-YMail-OSG:Received:X-Rocket-MIMEInfo:X-Mailer:References:Message-ID:Date:From:Reply-To:Subject:To:Cc:In-Reply-To:MIME-Version:Content-Type; b=eO+dtiWX9Ukfsx7CmNIEay2MFKeHjU2zVSxXsYMoM+dRwhtXLC3+tNi2x4AsW1gNjhx2wOnj4B2clfyjNVCI4mQcTdra1bGkWlQNeDvxahig6Ewwyq7/exkb4McIoRBnaRsPr13xRiMsB4sA4walhmCv5fX5gBLI94AZgsxMarQ=; X-YMail-OSG: upHBLR4VM1kga1rhHiwyEXJI5MEDL17tFUKdbT7XFWh6rKD dTA_nZG9a2OdVZW7dGnNzW64q0AiCITb7GjphParmIq7qWUnw8irC60HvmYV Pja3Q1CbuYY1C_hNWkYeO.D1dHsFgxyWyYG7Znx5WHP7Sbqk9TsPnulk3CzX 8CznhUBzvNYg7iq4Dr.gLK3DX9NOBv8mmBS48EMApkNdpV6aJx91DwG1zn_N WMOIN7gnheymnx357Cv1suudwm8In0UzSJM07f9URCEGXwTrE4YTi.5cY4m6 .QVarhLcpZ556bfIQpHW8jq_HKi6ybnKdjP3_14MyeQpquQI.oVasgGsfleK toRbHI9rHfwNwq9IcZoWpYUv2A3KVU07gWLQSDZgxQA5zF95LcyZGGIvNTTi bCuXu9lHalBSwptou_BZl5jtTtG_VErF.ESYkmIi5gkEZqCuEejxLbQNZKd6 R77O5LtTtitak5EkfNc7rBaKPJl2Lqx0XQUCrT8LD1RV5JX5JrSaCNbJ7x_h S0x8h9.Jt6iWCaiV5VX86AXe4dr.U6Hx1GCg0bQaNVBko29oNALkTyyPU_E0 - Received: from [98.203.118.124] by web121603.mail.ne1.yahoo.com via HTTP; Sat, 31 Aug 2013 07:47:47 PDT X-Rocket-MIMEInfo: 002.001, QW5kIGFub3RoZXIgdGhpbmc7IHRoZSB1c2Ugb2YgbW9kdWxvIGlzIHZlcnkgZXhwZW5zaXZlIHdoZW4gdGhlIG51bWJlciBvZiBwb3J0cwp1c2VkIGluIExBR0cgaXMgKnVzdWFsbHkqIGEgcG93ZXIgb2YgMi4gZm9vJihTTE9UUy0xKSBpcyBhIGxvdCBmYXN0ZXIgdGhhbiAoZm9vJVNMT1RTKS7CoAoKaWYgKFNMT1RTID09IDIgfHwgU0xPVFMgPT0gNCB8fCBTTE9UUyA9PSA4KQrCoCDCoCBoYXNoID0gaGFzaCYoU0xPVFMtMSk7CmVsc2UKwqAgwqAgaGFzaCA9IGhhc2ggJSBTTE9UUzsKCmlzIG1vcmUgdGhhbiABMAEBAQE- X-Mailer: YahooMailWebService/0.8.156.576 References: <521BBD21.4070304@freebsd.org> <521EE8DA.3060107@freebsd.org> Message-ID: <1377960467.93173.YahooMailNeo@web121603.mail.ne1.yahoo.com> Date: Sat, 31 Aug 2013 07:47:47 -0700 (PDT) From: Barney Cordoba Subject: Re: Flow ID, LACP, and igb To: Luigi Rizzo , Alan Somers In-Reply-To: MIME-Version: 1.0 Content-Type: text/plain; charset=iso-8859-1 Content-Transfer-Encoding: quoted-printable X-Content-Filtered-By: Mailman/MimeDel 2.1.14 Cc: Jack F Vogel , "T.C. Gubatayao" , "Justin T. Gibbs" , Andre Oppermann , "net@freebsd.org" X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list Reply-To: Barney Cordoba List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 31 Aug 2013 14:50:37 -0000 And another thing; the use of modulo is very expensive when the number of p= orts=0Aused in LAGG is *usually* a power of 2. foo&(SLOTS-1) is a lot faste= r than (foo%SLOTS).=A0=0A=0Aif (SLOTS =3D=3D 2 || SLOTS =3D=3D 4 || SLOTS = =3D=3D 8)=0A=A0 =A0 hash =3D hash&(SLOTS-1);=0Aelse=0A=A0 =A0 hash =3D hash= % SLOTS;=0A=0Ais more than twice as fast as=A0=0A=0Ahash % SLOTS;=0A=0ABC= =0A=0A=0A________________________________=0A From: Luigi Rizzo =0ATo: Alan Somers =0ACc: Jack F Vogel ; "net@freebsd.org" ; Justin T. Gibbs ; Andre Oppermann ; T.C. Gubatayao =0ASent: Friday, August 30, 2013 8:04 PM=0ASubject: Re: Flo= w ID, LACP, and igb=0A =0A=0AAlan,=0A=0A=0AOn Thu, Aug 29, 2013 at 6:45 PM,= Alan Somers wrote:=0A>=0A>=0A> ...=0A> I pulled all = four hash functions out into userland and microbenchmarked=0A> them.=A0 The= upshot is that hash32 and fnv_hash are the fastest, jenkins_hash=0A> is sl= ower, and siphash24 is the slowest.=A0 Also, Clang resulted in much=0A> fas= ter code than gcc.=0A>=0A>=0Ai missed this part of your message, but if i r= ead your code well,=0Ayou are running 100M iterations and the numbers below= are in seconds,=0Aso if you multiply the numbers by 10 you have the cost p= er hash in=0Ananoseconds.=0A=0AWhat CPU did you use for your tests ?=0A=0AA= lso some of the numbers (FNV and hash32) are suspiciously low.=0A=0AI=0A be= lieve that the compiler (both of them) have figure out that everything=0Ais= constant in these functions, and fnv_32_buf() and hash32_buf() are=0Ainlin= e,=0Ahence they can be optimized to just return a constant.=0AThis does not= happen for siphash and jenkins because they are defined=0Aexternally.=0A= =0ACan you please re-run the tests in a way that defeats the optimization ?= =0A(e.g. pass a non constant argument to the the hashes so you actually nee= d=0Ato run the code).=0A=0Acheers=0Aluigi=0A=0A=0Ahttp://people.freebsd.org= /~asomers/lagg_hash/=0A>=0A> [root@sm4u-4 /usr/home/alans/ctest/lagg_hash]#= ./lagg_hash-gcc-4.8=0A> FNV: 0.76=0A> hash32: 1.18=0A> SipHash24: 44.39=0A= > Jenkins: 6.20=0A> [root@sm4u-4 /usr/home/alans/ctest/lagg_hash]# ./lagg_h= ash-gcc-4.2.1=0A> FNV: 0.74=0A> hash32: 1.35=0A> SipHash24: 55.25=0A> Jenki= ns: 7.37=0A> [root@sm4u-4 /usr/home/alans/ctest/lagg_hash]# ./lagg_hash.cla= ng-3.3=0A> FNV: 0.30=0A> hash32: 0.30=0A> SipHash24: 55.97=0A> Jenkins: 6.4= 5=0A> [root@sm4u-4 /usr/home/alans/ctest/lagg_hash]# ./lagg_hash.clang-3.2= =0A> FNV: 0.30=0A> hash32: 0.30=0A> SipHash24: 44.52=0A> Jenkins: 6.48=0A>= =0A>=0A>=0A> > T.C.=0A> >=0A> > [1]=0A> >=0A> http://svnweb.freebsd.org/bas= e/head/sys/libkern/jenkins_hash.c?view=3Dmarkup=0A> _______________________= ________________________=0A> freebsd-net@freebsd.org mailing list=0A> http:= //lists.freebsd.org/mailman/listinfo/freebsd-net=0A> To unsubscribe, send a= ny mail to "freebsd-net-unsubscribe@freebsd.org"=0A>=0A____________________= ___________________________=0Afreebsd-net@freebsd.org mailing list=0Ahttp:/= /lists.freebsd.org/mailman/listinfo/freebsd-net=0ATo unsubscribe, send any = mail to "freebsd-net-unsubscribe@freebsd.org" From owner-freebsd-net@FreeBSD.ORG Sat Aug 31 15:05:23 2013 Return-Path: Delivered-To: freebsd-net@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTP id 701BDB81 for ; Sat, 31 Aug 2013 15:05:23 +0000 (UTC) (envelope-from barney_cordoba@yahoo.com) Received: from nm21-vm4.bullet.mail.ne1.yahoo.com (nm21-vm4.bullet.mail.ne1.yahoo.com [98.138.91.181]) (using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits)) (No client certificate requested) by mx1.freebsd.org (Postfix) with ESMTPS id 1EA6D2656 for ; Sat, 31 Aug 2013 15:05:22 +0000 (UTC) Received: from [98.138.101.128] by nm21.bullet.mail.ne1.yahoo.com with NNFMP; 31 Aug 2013 15:03:08 -0000 Received: from [98.138.89.248] by tm16.bullet.mail.ne1.yahoo.com with NNFMP; 31 Aug 2013 15:03:08 -0000 Received: from [127.0.0.1] by omp1040.mail.ne1.yahoo.com with NNFMP; 31 Aug 2013 15:03:08 -0000 X-Yahoo-Newman-Property: ymail-3 X-Yahoo-Newman-Id: 307911.32973.bm@omp1040.mail.ne1.yahoo.com Received: (qmail 38024 invoked by uid 60001); 31 Aug 2013 15:03:08 -0000 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=yahoo.com; s=s1024; t=1377961388; bh=Y7fnNQoRaB9Mi4aMrvBp20u0Jchwx0rrwWWy0Jz3lMw=; h=X-YMail-OSG:Received:X-Rocket-MIMEInfo:X-Mailer:References:Message-ID:Date:From:Reply-To:Subject:To:In-Reply-To:MIME-Version:Content-Type; b=mYfGjAfYimzkoy0S2YCm47jExxgCxB9/sGt60fEAmtyGc88ohT79tDls79W6OI59UNLaSGxQDL2h89St8qo4G4b//1wlq9zAvjlPQjyvTkdhS45pNJmGLMLgyNBRdfXaq4b4MHK9bNZ9gwjavxPmO+jM6DQB8+cbgEdSuV2En9Y= DomainKey-Signature: a=rsa-sha1; q=dns; c=nofws; s=s1024; d=yahoo.com; h=X-YMail-OSG:Received:X-Rocket-MIMEInfo:X-Mailer:References:Message-ID:Date:From:Reply-To:Subject:To:In-Reply-To:MIME-Version:Content-Type; b=ExHGsnoQn35p3cmBt8J5dbDx+EClbaAS6h12mTmHDbzdPQgbYqYkn0v6paHO16ot1K+rn9e5lK+y5iGaa/ieFrTdJ8aocOSVSWIt1eHfXGqCYGI/HY3VKDu4GK/wNsN+i6G5SoW/tuofW8Tf2kvmYFq3Jz28Xb/hltoSgd7qjfY=; X-YMail-OSG: ob_reFgVM1lDffF6LE7hWbGzHbMGfKgIk0e6GWQyyC3FBT9 noVa6TSxl70hWFu9H3ep9GCTM0XrY0v.r8xdn0a3f4Dtv_9LvAI9UWM2rd8g o2TGdzGUYgdULWWfaxS_OgR1YU0XUcYRQRcCYViVK.V_89x6EWMqmJxUCvUk xp_l1ocXjkdwEp.6G5huw8624Wv0jchbKkoOXTPytM63KMLpwPV4aE_quPtS 94zfqrMowraoCynekyrfSnoW5xugQvykV4VKgpXP3VKlgC3in0uhOwLvEXCI .FhKTLbDrMeI.Wlv5PIypHX6TgnPJ_h91Lr4vByOHCEAF2jjIp4FoXtOQvoV hRmDmAS_1V9mezXJazZPjFEpeH63Wdemqi1HZnHzQY3hftGqvB5tIJ8V1Cw4 KN2EVxQIbhceL.pJN5kV_hxBtvzpBfwU_BKC7evZqMA5IeKjrnq7b1r07IVJ zripu_ZMp0Bi7QyunKQoaBSV5y_fxYi0EEHZIG4PFsNa.79N8Ck1ijCWvLrV Nu34TTEk7KdtQtx6e6gS41Zuy.fcwVtbtnF69Wcb0tyTAOYHVez2BVHNwokC L.VsnOocD1zqLDPIvlyQnT6ZAKENkgfWce8Via6y8n99ibWbQj1RG559YcJY lvoDJhCWhgjc- Received: from [98.203.118.124] by web121601.mail.ne1.yahoo.com via HTTP; Sat, 31 Aug 2013 08:03:08 PDT X-Rocket-MIMEInfo: 002.001, VGhhdCdzIHdheSB0b28gaGlnaC4gWW91ciBiYXNlIHJ4IHJlcXVpcmVtZW50IGlzwqAKClBvcnRzICogcXVldWVzICogcnhkwqAKCldpdGggYSBxdWFkIGNhcmQgeW91IHNob3VsZG4ndCBiZSB1c2luZyBtb3JlIHRoYW4gMiBxdWV1ZXMsIHNvIHlvdXIgcmVxdWlyZW1lbnQKd2l0aCA1IHBvcnRzIGlzIDEwLDI0MCBqdXN0IGZvciB0aGUgcmVjZWl2ZSBzZXR1cC4gSWYgeW91J3JlIHVzaW5nIDQgcXVldWVzIHRoYXQKbnVtYmVyIGRvdWJsZXMsIHdoaWNoIHdvdWxkIG1ha2UgMjUsNjAwIG5vdCBlbm91Z2guwqABMAEBAQE- X-Mailer: YahooMailWebService/0.8.156.576 References: <71042F7C-5CBB-4494-B53A-EF4CE45B41BE@ebureau.com> Message-ID: <1377961388.28903.YahooMailNeo@web121601.mail.ne1.yahoo.com> Date: Sat, 31 Aug 2013 08:03:08 -0700 (PDT) From: Barney Cordoba Subject: Re: Intel 4-port ethernet adaptor link aggregation issue To: Joe Moog , freebsd-net In-Reply-To: <71042F7C-5CBB-4494-B53A-EF4CE45B41BE@ebureau.com> MIME-Version: 1.0 Content-Type: text/plain; charset=iso-8859-1 Content-Transfer-Encoding: quoted-printable X-Content-Filtered-By: Mailman/MimeDel 2.1.14 X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list Reply-To: Barney Cordoba List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 31 Aug 2013 15:05:23 -0000 That's way too high. Your base rx requirement is=A0=0A=0APorts * queues * r= xd=A0=0A=0AWith a quad card you shouldn't be using more than 2 queues, so y= our requirement=0Awith 5 ports is 10,240 just for the receive setup. If you= 're using 4 queues that=0Anumber doubles, which would make 25,600 not enoug= h.=A0=0A=0ANote that setting mbufs to a huge number doesn't allocate the bu= ffers; they'll be=0Aallocated as needed. It's a ceiling. The reason for the= ceiling is so that you don't=0Ablow up your memory. If your system is usin= g 2 million mbuf clusters then you=0Ahave much bigger problems than LAGG.= =0A=0AAnyone who recommends 2 million clearly has no idea what they're doin= g.=0A=0ABC=0A=0A=0A________________________________=0A From: Joe Moog =0ATo: freebsd-net =0ASent: Wedne= sday, August 28, 2013 9:36 AM=0ASubject: Re: Intel 4-port ethernet adaptor = link aggregation issue=0A =0A=0AAll:=0A=0AThanks again to everybody for the= responses and suggestions to our 4-port lagg issue. The solution (for thos= e that may find the information of some value) was to set the value for ker= n.ipc.nmbclusters to a higher value than we had initially. Our previous tun= ing had this value set at 25600, but following a recommendation from the go= od folks at iXSystems we bumped this to a value closer to 2000000, and the = 4-port lagg is functioning as expected now.=0A=0AThank you all.=0A=0AJoe=0A= =0A_______________________________________________=0Afreebsd-net@freebsd.or= g mailing list=0Ahttp://lists.freebsd.org/mailman/listinfo/freebsd-net=0ATo= unsubscribe, send any mail to "freebsd-net-unsubscribe@freebsd.org"