Date: Sat, 4 May 2013 13:54:56 -0700 From: Richard Sharpe <realrichardsharpe@gmail.com> To: Jack Vogel <jfvogel@gmail.com> Cc: FreeBSD Net <freebsd-net@freebsd.org>, Adrian Chadd <adrian@freebsd.org> Subject: Re: Is there any way to limit the amount of data in an mbuf chain submitted to a driver? Message-ID: <CACyXjPy=ErKS_A3Nmwa8PnMc_D=2LByd4VJPFdTj9sqgSaCTfw@mail.gmail.com> In-Reply-To: <CAFOYbcmZMW=-7Mwz9mwJLeM3Ju%2BF8_AsXFAPqCa8%2BuuRWq3xsg@mail.gmail.com> References: <CACyXjPwC5LRb7DT82n6PMbawceER3_nHko9c9tvrdQqceLiPww@mail.gmail.com> <CAJ-Vmon_5eyXMP5UOsVVBP8UgKQLw5HLMO1NgswoGb-zF=2wtg@mail.gmail.com> <CACyXjPzu3fXpo0i5YcdVBFye%2BRFTPUye=fgZ%2BycTkkiEmcRh%2BQ@mail.gmail.com> <CAFOYbcmZMW=-7Mwz9mwJLeM3Ju%2BF8_AsXFAPqCa8%2BuuRWq3xsg@mail.gmail.com>
next in thread | previous in thread | raw e-mail | index | archive | help
On Sat, May 4, 2013 at 1:41 PM, Jack Vogel <jfvogel@gmail.com> wrote:
> If you don't use TSO you will hurt your TX performance significantly from
> the tests that I've run. What exactly is the device you are using, I don'=
t
> have the source in front of me now, but I'm almost sure that the limit is
> not 64K but 256K, or are you using some ancient version of the driver?
ix0 pnpinfo vendor=3D0x8086 device=3D0x1528 subvendor=3D0x8086
subdevice=3D0x0001 class=3D0x020000 at slot=3D0 function=3D0
ix1 pnpinfo vendor=3D0x8086 device=3D0x1528 subvendor=3D0x8086
subdevice=3D0x0001 class=3D0x020000 at slot=3D0 function=3D1
The version calls itself ixgbe-2.4.4 ...
Hmmm, copyright is 2001-2010 ... so perhaps a bit old.
> Jack
>
>
>
> On Sat, May 4, 2013 at 1:30 PM, Richard Sharpe <realrichardsharpe@gmail.c=
om>
> wrote:
>>
>> On Sat, May 4, 2013 at 10:39 AM, Adrian Chadd <adrian@freebsd.org> wrote=
:
>> > On 4 May 2013 06:52, Richard Sharpe <realrichardsharpe@gmail.com> wrot=
e:
>> >> Hi folks,
>> >>
>> >> I understand better why I am seeing EINVAL intermittently when sendin=
g
>> >> data from Samba via SMB2.
>> >>
>> >> The ixgbe driver, for TSO reasons, limits the amount of data that can
>> >> be DMA'd to 65535 bytes. It returns EINVAL for any mbuf chain larger
>> >> than that.
>> >>
>> >> The SO_SNDBUF for that socket is set to 131972. Mostly there is less
>> >> than 64kiB of space available, so that is all TCP etc can put into th=
e
>> >> socket in one chain of mbufs. However, every now and then there is
>> >> more than 65535 bytes available in the socket buffers, and we have an
>> >> SMB packet that is larger than 65535 bytes, and we get hit.
>> >>
>> >> To confirm this I am going to set SO_SNDBUF back to the default of
>> >> 65536 and test again. My repros are very reliable.
>> >>
>> >> However, I wondered if my only way around this if I want to continue
>> >> to use SO_SNDBUF sizes larger than 65536 is to fragment large mbuf
>> >> chains in the driver?
>> >
>> > Hm, is this is a problem without TSO?
>>
>> We are using the card without TSO, so I am thinking of changing that
>> limit to 131072 and retesting.
>>
>> I am currently testing with SO_SNDBUF=3D32768 and have not hit the probl=
em.
>>
>> > Is the problem that the NIC can't handle a frame that big, or a buffer
>> > that big?
>> > Ie - if you handed the hardware two descriptors of 64k each, for the
>> > same IP datagram, will it complain?
>>
>> I can't find any documentation, but it seems that with TSO it cannot
>> handle a frame that big. Actually, since we are not using TSO, there
>> really should not be a problem with larger frames.
>>
>> > Or do you need to break it up into two separate IP datagrams, facing
>> > the driver, with a maximum size of 64k each?
>>
>> Not sure, but it looks like we need to do that.
>>
>>
>> --
>> Regards,
>> Richard Sharpe
>> (=E4=BD=95=E4=BB=A5=E8=A7=A3=E6=86=82=EF=BC=9F=E5=94=AF=E6=9C=89=E6=9D=
=9C=E5=BA=B7=E3=80=82--=E6=9B=B9=E6=93=8D)
>> _______________________________________________
>> freebsd-net@freebsd.org mailing list
>> http://lists.freebsd.org/mailman/listinfo/freebsd-net
>> To unsubscribe, send any mail to "freebsd-net-unsubscribe@freebsd.org"
>
>
--=20
Regards,
Richard Sharpe
(=E4=BD=95=E4=BB=A5=E8=A7=A3=E6=86=82=EF=BC=9F=E5=94=AF=E6=9C=89=E6=9D=9C=
=E5=BA=B7=E3=80=82--=E6=9B=B9=E6=93=8D)
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?CACyXjPy=ErKS_A3Nmwa8PnMc_D=2LByd4VJPFdTj9sqgSaCTfw>