From owner-freebsd-stable@FreeBSD.ORG Thu Apr 14 20:28:17 2011 Return-Path: Delivered-To: freebsd-stable@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id F3F2C106566C for ; Thu, 14 Apr 2011 20:28:16 +0000 (UTC) (envelope-from jfvogel@gmail.com) Received: from mail-vx0-f182.google.com (mail-vx0-f182.google.com [209.85.220.182]) by mx1.freebsd.org (Postfix) with ESMTP id A448B8FC0A for ; Thu, 14 Apr 2011 20:28:16 +0000 (UTC) Received: by vxc34 with SMTP id 34so2089258vxc.13 for ; Thu, 14 Apr 2011 13:28:16 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=domainkey-signature:mime-version:in-reply-to:references:date :message-id:subject:from:to:content-type; bh=fjCLbW/9j74R1NRdgsH0ayyhUpHqhiyAZJF5d3NH+X4=; b=hG7uxue+KAjpYkpau0BAol4HhdKvob2jhs0BOFf7oAmvzLN28V9nJqAL2yOJ1LWOb5 AntD12LxUGSvHyBTV1jyIm2A6P/0Z0eJugSXH9O4y63W989VbxysV5MssiEMHrFnzJVV JD3C8tmfrTicczGvKJQ8Vuq2ZHChrasR8x4MA= DomainKey-Signature: a=rsa-sha1; c=nofws; d=gmail.com; s=gamma; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :content-type; b=jTRwRwXQWhAuAV/hbhon/zoO3ejaBniR5U0q8B94woVYUwxIAuGiJc9Llk9VWqPTuF y8P/fNbxkGJvbtINmlyHsnjQa2Z6urNZmUuCwYLUNcbKp9xWFZOWUUfIuiV+Xiypavgk fDTrBHmhIFJ87BwgTUWt7PeCuMLsxtHvHuhw4= MIME-Version: 1.0 Received: by 10.52.89.72 with SMTP id bm8mr1234157vdb.281.1302812895954; Thu, 14 Apr 2011 13:28:15 -0700 (PDT) Received: by 10.52.161.168 with HTTP; Thu, 14 Apr 2011 13:28:15 -0700 (PDT) In-Reply-To: <20110414194448.GG9116@emmi.physik-pool.tu-berlin.de> References: <20110414130540.GD9116@emmi.physik-pool.tu-berlin.de> <20110414194448.GG9116@emmi.physik-pool.tu-berlin.de> Date: Thu, 14 Apr 2011 13:28:15 -0700 Message-ID: From: Jack Vogel To: "K. Macy" , Jack Vogel , =?ISO-8859-1?Q?Leon_Me=DFner?= , freebsd-stable@freebsd.org Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: quoted-printable X-Content-Filtered-By: Mailman/MimeDel 2.1.5 Cc: Subject: Re: ixgbe(4) and "Could not setup receive structures" X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 14 Apr 2011 20:28:17 -0000 If you are using the latest code, then the RX ring size is set to be 2K descriptors, so you will use that many 9k jumbos per queue to initialize things. Having a spare amount free to use as you clean/refresh is needed also. I upped the ring size for performance reasons on 10G, its possible to try dropping it to 1K. But, for 10G, I don't think its unreasonable to have enough memory around t= o handle this. Cheers, Jack On Thu, Apr 14, 2011 at 12:44 PM, Leon Me=DFner wrote: > On Thu, Apr 14, 2011 at 08:55:17PM +0200, K. Macy wrote: > > That isn't guaranteed to work if he is KVA limited. > > > > On Thu, Apr 14, 2011 at 6:44 PM, Jack Vogel wrote: > > > If you get this message its only for one reason, you don't have enoug= h > mbufs > > > to > > > fill your rings. You must do one of two things, either reduce the > number of > > > queues, > > > or increase the relevant mbuf pool. > > > > > > Increase the 9K mbuf cluster pool. > > I did set it to twice the default, and now it works and netstat -m > shows: > > 8192/391/8583/12800 9k jumbo clusters in use (current/cache/total/max) > > Whats a reasonable amount to set kern.ipc.nmbjumbo9 to and is there > any > form of auto-tuning (i have absolutely no load on this machine and > mbufs > are higher than default pool size). > > Thanks to all, > Leon > > > > On Thu, Apr 14, 2011 at 6:05 AM, Leon Me=DFner > > > wrote: > > > > > >> Hi, > > >> > > >> i tried setting the mtu on one of my ixgbe(4) intel NICs to support > > >> jumbo frames. This is on a box with RELENG_8 from today. > > >> > > >> # ifconfig ix0 mtu 9198 > > >> > > >> I then get the following error: > > >> > > >> # tail -n 1 /var/log/messages > > >> Apr 14 12:48:43 siloneu kernel: ix0: Could not setup receive > structures > > >> > > >> I already tried the following patch because of Jack Vogel's advice > given > > >> in the following thread on -stable in Oct. last year, which still > > >> produces the same error message and leaves the box unpingable: > > >> > > >> > http://lists.freebsd.org/pipermail/freebsd-stable/2010-October/059541.htm= l > > >> > > >> # cat ~/patches/ixgbe.num_queues_to_4.patch > > >> --- /root/.vimbackup/ixgbe.c~ 2011-04-12 22:14:27.000000000 +0000 > > >> +++ sys/dev/ixgbe/ixgbe.c 2011-04-12 22:14:27.000000000 +0000 > > >> @@ -273,7 +273,7 @@ TUNABLE_INT("hw.ixgbe.hdr_split", &ixgbe > > >> * number of cpus. Each queue is a pair > > >> * of RX and TX rings with a msix vector > > >> */ > > >> -static int ixgbe_num_queues =3D 0; > > >> +static int ixgbe_num_queues =3D 4; > > >> TUNABLE_INT("hw.ixgbe.num_queues", &ixgbe_num_queues); > > >> > > >> /* > > >> > > >> _______________________________________________ > > >> freebsd-stable@freebsd.org mailing list > > >> http://lists.freebsd.org/mailman/listinfo/freebsd-stable > > >> To unsubscribe, send any mail to " > freebsd-stable-unsubscribe@freebsd.org" > > >> > > > _______________________________________________ > > > freebsd-stable@freebsd.org mailing list > > > http://lists.freebsd.org/mailman/listinfo/freebsd-stable > > > To unsubscribe, send any mail to " > freebsd-stable-unsubscribe@freebsd.org" > > > > > _______________________________________________ > > freebsd-stable@freebsd.org mailing list > > http://lists.freebsd.org/mailman/listinfo/freebsd-stable > > To unsubscribe, send any mail to "freebsd-stable-unsubscribe@freebsd.or= g > " >