From owner-freebsd-net@FreeBSD.ORG Thu Jun 20 15:33:50 2013 Return-Path: Delivered-To: freebsd-net@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by hub.freebsd.org (Postfix) with ESMTP id 27956B8C for ; Thu, 20 Jun 2013 15:33:50 +0000 (UTC) (envelope-from barney_cordoba@yahoo.com) Received: from nm19-vm1.bullet.mail.ne1.yahoo.com (nm19-vm1.bullet.mail.ne1.yahoo.com [98.138.91.56]) by mx1.freebsd.org (Postfix) with ESMTP id CF5961124 for ; Thu, 20 Jun 2013 15:33:49 +0000 (UTC) Received: from [98.138.90.48] by nm19.bullet.mail.ne1.yahoo.com with NNFMP; 20 Jun 2013 15:33:43 -0000 Received: from [98.138.226.160] by tm1.bullet.mail.ne1.yahoo.com with NNFMP; 20 Jun 2013 15:33:43 -0000 Received: from [127.0.0.1] by omp1061.mail.ne1.yahoo.com with NNFMP; 20 Jun 2013 15:33:43 -0000 X-Yahoo-Newman-Property: ymail-3 X-Yahoo-Newman-Id: 197182.83548.bm@omp1061.mail.ne1.yahoo.com Received: (qmail 52702 invoked by uid 60001); 20 Jun 2013 15:33:42 -0000 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=yahoo.com; s=s1024; t=1371742422; bh=yEuGygb6PbwhCAjYcSPTPeWgxct9uGBrmRJxvGPIeic=; h=X-YMail-OSG:Received:X-Rocket-MIMEInfo:X-Mailer:Message-ID:Date:From:Subject:To:Cc:In-Reply-To:MIME-Version:Content-Type:Content-Transfer-Encoding; b=1NdyDmAIlaactKeDQaKep7Ektwtvk5nKC5EXUYmEm87r6QLYSUvYTj/fydUxSz0Y3M9tolmhcRofkEUSRhOyr+ukbmSC7BRYosO/+1KHz3391SCHn4+7dKT7X3j6YEkgkJS0YZ/3S7mgjk1nuXygD5TT/+Kos1NiZ911obv8XyU= DomainKey-Signature: a=rsa-sha1; q=dns; c=nofws; s=s1024; d=yahoo.com; h=X-YMail-OSG:Received:X-Rocket-MIMEInfo:X-Mailer:Message-ID:Date:From:Subject:To:Cc:In-Reply-To:MIME-Version:Content-Type:Content-Transfer-Encoding; b=jQh3VoIZV6/38pGqpzhQ+I8j+Q9OWHeWAClTm39FefOy592z8yFzydnoJ0YUt3Qo8iH/uZG/WX/HDb07UIy1eHbeYad1iy28LZdajFny8P43/OPWu/uELk5XYzjs7B3yxBSer02KRj74yA1/iwZhwqCBWERILGZYIfdSoQWbpp4=; X-YMail-OSG: G._LySsVM1lTaZyt9oLBq3BegEXRF0nwMu0L0L5xdyFZ1Wh AF0h49AckPgVa.geUNXUTxwGB_7BzOuP0YVW4caE4AOJpxJ1nkHUfCcM8n2f M0Gwb6tJlbsbCJv87TVlu3hSGQdxbGN6Oby2LVJihpQtPmFcPtpzpfM.htGL rM3ihsW5UtcCo1OjydTE9S.2nfFySq5GlpDy3tz5NaSt85q._8QDOkqYTIuT 3gI3HEVjaclbVCgMkcyD5iCZGcJAG2MSKHqwopr0maNqY.6Ctqc7gnWn7gmI KfUqodOzmXE8BKTHR4r6Ne1TG3DxKISfyAbQnb4VbDNSTd0H8hZIOjKrIobn laPX3YJwsGnZ0PN55mgn6nQfG3Yac8PaFUWZkG1TpDSHaGO6e9XR8dYKGiVJ MRPyACr5tJ081Z7lFekXStSmw17mNreLhr60xBOI15r8WZZco2I6SqO0IaHX 0QyZ6HHX_M5dJNc5Mh0oGtqEn.oJev.b6.R2ibj0NvYXn8CAX0Po4zAtMS.B u7ge_j88GdM_WU.8ikx6pmasblfmI3BezZ0VHKWUEZ9VW6oGu3fMVweTm62f uc4AfcgR2Xndy Received: from [98.203.118.124] by web121606.mail.ne1.yahoo.com via HTTP; Thu, 20 Jun 2013 08:33:42 PDT X-Rocket-MIMEInfo: 002.001, LS0tIE9uIFRodSwgNi8yMC8xMywgQW5kcmUgT3BwZXJtYW5uICB3cm90ZToNCg0KJmd0OyBGcm9tOiBBbmRyZSBPcHBlcm1hbm4gDQomZ3Q7IFN1YmplY3Q6IFJlOiBody5pZ2IubnVtX3F1ZXVlcyBkZWZhdWx0DQomZ3Q7IFRvOiAmcXVvdDtFdWdlbmUgR3Jvc2JlaW4mcXVvdDsgDQomZ3Q7IENjOiAmcXVvdDtmcmVlYnNkLW5ldEBmcmVlYnNkLm9yZyZxdW90OyAsICZxdW90O0VnZ2VydCwgTGFycyZxdW90OyAsICZxdW90O0phY2sgVm9nZWwmcXVvdDsgDQomZ3Q7IERhdGU6IFRodXJzZGF5LCBKdW5lIDIwLCABMAEBAQE- X-Mailer: YahooMailClassic/184 YahooMailWebService/0.8.148.554 Message-ID: <1371742422.50315.YahooMailBasic@web121606.mail.ne1.yahoo.com> Date: Thu, 20 Jun 2013 08:33:42 -0700 (PDT) From: Barney Cordoba Subject: Re: hw.igb.num_queues default To: Eugene Grosbein , Andre Oppermann In-Reply-To: <51C311D6.5090801@freebsd.org> MIME-Version: 1.0 Content-Type: text/plain; charset=iso-8859-1 Content-Transfer-Encoding: quoted-printable Cc: "freebsd-net@freebsd.org" , LarsEggert , Jack Vogel X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 20 Jun 2013 15:33:50 -0000 --- On Thu, 6/20/13, Andre Oppermann wrote: > From: Andre Oppermann=20 > Subject: Re: hw.igb.num_queues default > To: "Eugene Grosbein"=20 > Cc: "freebsd-net@freebsd.org" , "Eggert, Lars" , &= quot;Jack Vogel"=20 > Date: Thursday, June 20, 2013, 10:29 AM > On 20.06.2013 15:37, Eugene Grosbein > wrote: > > On 20.06.2013 17:34, Eggert, Lars wrote: > > > >> real memory=A0 =3D 8589934592 (8192 MB) > >> avail memory =3D 8239513600 (7857 MB) > > > >> By default, the igb driver seems to set up one > queue per detected CPU. Googling around, people seemed to > suggest that limiting the number of queues makes things work > better. I can confirm that setting hw.igb.num_queues=3D2 seems > to have fixed the issue. (Two was the first value I tried, > maybe other values other than 0 would work, too.) > >> > >> In order to uphold POLA, should the igb driver > maybe default to a conservative value for hw.igb.num_queues > that may not deliver optimal performance, but at least works > out of the box? > > > > Or, better, make nmbclusters auto-tuning smarter, if > any. > > I mean, use more nmbclusters for machines with large > amounts of memory. >=20 > That has already been done in HEAD. >=20 > The other problem is the pre-filling of the large rings for > all queues > stranding large amounts of mbuf clusters.=A0 OpenBSD > starts with a small > number of filled mbufs in the RX ring and then dynamically > adjusts the > number upwards if there is enough traffic to maintain deep > buffers.=A0 I > don't know if it always quickly scales in practice though. You're probably not running with 512MB these days, so pre-filling isn&#= 39;t much of an issue. 4 queues is only 8MB of ram with 1024 descriptors per queue, and 4MB with 5= 12. Think about the # of queues issue. In order to have acceptable latency, you= need to do 6k-10k=20 interrupts per second per queue. So with 4 queues you have to process 40K i= nts/second and with 2 you only process 20k. For a gig link 2 queues is much more effi= cient. "Spreading" for the sake of spreading is more about Intel marketi= ng than it is about practical computing. BC BC