Date: Thu, 20 Mar 2014 19:32:17 -0300 From: Christopher Forgeron <csforgeron@gmail.com> To: Jack Vogel <jfvogel@gmail.com> Cc: FreeBSD Net <freebsd-net@freebsd.org>, Garrett Wollman <wollman@hergotha.csail.mit.edu> Subject: Re: 9.2 ixgbe tx queue hang Message-ID: <CAB2_NwAHTGq8cpTfy-t5OyCr4pN9nd3wsvT2JdOYX73grAM0tA@mail.gmail.com> In-Reply-To: <CAFOYbcn3%2BxwgK0q842Fvo-V5hOi-CeEzvv=7XFtZCJG1%2BYj7Tw@mail.gmail.com> References: <CAB2_NwDG=gB1WCJ7JKTHpkJCrvPuAhipkn%2BvPyT%2BxXzOBrTGkg@mail.gmail.com> <1159309884.25490921.1395282576806.JavaMail.root@uoguelph.ca> <CAB2_NwAOmPtZjB03pdDiTK2OvQgqk-tYf83Jq4Ukt9jnZA8CNA@mail.gmail.com> <201403202113.s2KLD7GB085085@hergotha.csail.mit.edu> <CAB2_NwCjQ6qtx0SbEERVGs2Y_5pae-g=UbFEwkzmWYGoTuoP%2BQ@mail.gmail.com> <CAFOYbckqw1wT41a-_3FMs6-KNVMV319nODtQ2F09eDRZavFPTg@mail.gmail.com> <CAB2_NwB4ZO7HYBvvuJ-NDSqo=HvbCz34RUnW_YmGagQs1guaFw@mail.gmail.com> <CAFOYbcn3%2BxwgK0q842Fvo-V5hOi-CeEzvv=7XFtZCJG1%2BYj7Tw@mail.gmail.com>
next in thread | previous in thread | raw e-mail | index | archive | help
I agree, performance is noticeably worse with TSO off, but I thought it
would be a good step in troubleshooting. I'm glad you're a regular reader
of the list, so I don't have to settle for slow performance. :-)
I'm applying your patch now, I think it will fix it - but I'll report in
after it's run iometer for the night regardless.
On another note: What's so different about memory allocation in 10 that is
making this an issue?
On Thu, Mar 20, 2014 at 7:24 PM, Jack Vogel <jfvogel@gmail.com> wrote:
> I strongly discourage anyone from disabling TSO on 10G, its necessary to
> get the
> performance one wants to see on the hardware.
>
> Here is a patch to do what i'm talking about:
>
> *** ixgbe.c Fri Jan 10 18:12:20 2014
> --- ixgbe.jfv.c Thu Mar 20 23:04:15 2014
> *************** ixgbe_init_locked(struct adapter *adapte
> *** 1140,1151 ****
> */
> if (adapter->max_frame_size <= 2048)
> adapter->rx_mbuf_sz = MCLBYTES;
> - else if (adapter->max_frame_size <= 4096)
> - adapter->rx_mbuf_sz = MJUMPAGESIZE;
> - else if (adapter->max_frame_size <= 9216)
> - adapter->rx_mbuf_sz = MJUM9BYTES;
> else
> ! adapter->rx_mbuf_sz = MJUM16BYTES;
>
> /* Prepare receive descriptors and buffers */
> if (ixgbe_setup_receive_structures(adapter)) {
> --- 1140,1147 ----
> */
> if (adapter->max_frame_size <= 2048)
> adapter->rx_mbuf_sz = MCLBYTES;
> else
> ! adapter->rx_mbuf_sz = MJUMPAGESIZE;
>
> /* Prepare receive descriptors and buffers */
> if (ixgbe_setup_receive_structures(adapter)) {
>
>
>
>
>
>
> On Thu, Mar 20, 2014 at 3:12 PM, Christopher Forgeron <
> csforgeron@gmail.com> wrote:
>
>> Hi Jack,
>>
>> I'm on ixgbe 2.5.15
>>
>> I see a few other threads about using MJUMPAGESIZE instead of MJUM9BYTES.
>>
>> If you have a patch you'd like me to test, I'll compile it in and let
>> you know. I was just looking at Garrett's if_em.c patch and thinking about
>> applying it to ixgbe..
>>
>> As it stands I seem to not be having the problem now that I have
>> disabled TSO on ix0, but I still need more test runs to confirm - Which is
>> also in line (i think) with what you are all saying.
>>
>>
>>
>>
>> On Thu, Mar 20, 2014 at 7:00 PM, Jack Vogel <jfvogel@gmail.com> wrote:
>>
>>> What he's saying is that the driver should not be using 9K mbuf
>>> clusters, I thought
>>> this had been changed but I see the code in HEAD is still using the
>>> larger clusters
>>> when you up the mtu. I will put it on my list to change with the next
>>> update to HEAD.
>>>
>>>
>>> What version of ixgbe are you using?
>>>
>>> Jack
>>>
>>>
>>>
>>> On Thu, Mar 20, 2014 at 2:34 PM, Christopher Forgeron <
>>> csforgeron@gmail.com> wrote:
>>>
>>>> I have found this:
>>>>
>>>> http://lists.freebsd.org/pipermail/freebsd-net/2013-October/036955.html
>>>>
>>>> I think what you're saying is that;
>>>> - a MTU of 9000 doesn't need to equal a 9k mbuf / jumbo cluster
>>>> - modern NIC drivers can gather 9000 bytes of data from various memory
>>>> locations
>>>> - The fact that I'm seeing 9k jumbo clusters is showing me that my
>>>> driver
>>>> is trying to allocate 9k of contiguous space, and it's failing.
>>>>
>>>> Please correct me if I'm off here, I'd love to understand more.
>>>>
>>>>
>>>> On Thu, Mar 20, 2014 at 6:13 PM, Garrett Wollman <
>>>> wollman@hergotha.csail.mit.edu> wrote:
>>>>
>>>> > In article
>>>> > <CAB2_NwAOmPtZjB03pdDiTK2OvQgqk-tYf83Jq4Ukt9jnZA8CNA@mail.gmail.com>,
>>>> > csforgeron@gmail.com writes:
>>>> >
>>>> > >50/27433/0 requests for jumbo clusters denied (4k/9k/16k)
>>>> >
>>>> > This is going to screw you. You need to make sure that no NIC driver
>>>> > ever allocates 9k jumbo pages -- unless you are using one of those
>>>> > mythical drivers that can't do scatter/gather DMA on receive, which
>>>> > you don't appear to be.
>>>> >
>>>> > These failures occur when the driver is trying to replenish its
>>>> > receive queue, but is unable to allocate three *physically* contiguous
>>>> > pages of RAM to construct the 9k jumbo cluster (of which the remaining
>>>> > 3k is simply wasted). This happens on any moderately active server,
>>>> > once physical memory gets checkerboarded with active single pages,
>>>> > particularly with ZFS where those pages are wired in kernel memory and
>>>> > so can't be evicted.
>>>> >
>>>> > -GAWollman
>>>> >
>>>> _______________________________________________
>>>> freebsd-net@freebsd.org mailing list
>>>> http://lists.freebsd.org/mailman/listinfo/freebsd-net
>>>> To unsubscribe, send any mail to "freebsd-net-unsubscribe@freebsd.org"
>>>>
>>>
>>>
>>
>
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?CAB2_NwAHTGq8cpTfy-t5OyCr4pN9nd3wsvT2JdOYX73grAM0tA>