From owner-freebsd-net@FreeBSD.ORG Thu Jun 20 14:30:05 2013 Return-Path: Delivered-To: freebsd-net@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) by hub.freebsd.org (Postfix) with ESMTP id 7E492E3F for ; Thu, 20 Jun 2013 14:30:05 +0000 (UTC) (envelope-from andre@freebsd.org) Received: from c00l3r.networx.ch (c00l3r.networx.ch [62.48.2.2]) by mx1.freebsd.org (Postfix) with ESMTP id E91E51CDE for ; Thu, 20 Jun 2013 14:30:04 +0000 (UTC) Received: (qmail 65271 invoked from network); 20 Jun 2013 15:24:21 -0000 Received: from c00l3r.networx.ch (HELO [127.0.0.1]) ([62.48.2.2]) (envelope-sender ) by c00l3r.networx.ch (qmail-ldap-1.03) with SMTP for ; 20 Jun 2013 15:24:21 -0000 Message-ID: <51C311D6.5090801@freebsd.org> Date: Thu, 20 Jun 2013 16:29:42 +0200 From: Andre Oppermann User-Agent: Mozilla/5.0 (Windows NT 6.1; WOW64; rv:17.0) Gecko/20130509 Thunderbird/17.0.6 MIME-Version: 1.0 To: Eugene Grosbein Subject: Re: hw.igb.num_queues default References: <843F7891-FD87-4F16-A279-B45D4A674F4E@netapp.com> <51C305B3.7050703@grosbein.net> In-Reply-To: <51C305B3.7050703@grosbein.net> Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit Cc: "freebsd-net@freebsd.org" , "Eggert, Lars" , Jack Vogel X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 20 Jun 2013 14:30:05 -0000 On 20.06.2013 15:37, Eugene Grosbein wrote: > On 20.06.2013 17:34, Eggert, Lars wrote: > >> real memory = 8589934592 (8192 MB) >> avail memory = 8239513600 (7857 MB) > >> By default, the igb driver seems to set up one queue per detected CPU. Googling around, people seemed to suggest that limiting the number of queues makes things work better. I can confirm that setting hw.igb.num_queues=2 seems to have fixed the issue. (Two was the first value I tried, maybe other values other than 0 would work, too.) >> >> In order to uphold POLA, should the igb driver maybe default to a conservative value for hw.igb.num_queues that may not deliver optimal performance, but at least works out of the box? > > Or, better, make nmbclusters auto-tuning smarter, if any. > I mean, use more nmbclusters for machines with large amounts of memory. That has already been done in HEAD. The other problem is the pre-filling of the large rings for all queues stranding large amounts of mbuf clusters. OpenBSD starts with a small number of filled mbufs in the RX ring and then dynamically adjusts the number upwards if there is enough traffic to maintain deep buffers. I don't know if it always quickly scales in practice though. -- Andre