Date: Fri, 31 Oct 2014 13:19:01 +0100 From: Julien Charbon <jch@freebsd.org> To: freebsd-net <freebsd-net@freebsd.org> Cc: Oleg Bulyzhin <oleg@freebsd.org>, Ryan Stone <rysto32@gmail.com>, "K. Macy" <kmacy@freebsd.org>, "De La Gueronniere, Marc" <mdelagueronniere@verisign.com> Subject: Re: buf_ring in HEAD is racy Message-ID: <54537E35.2060108@freebsd.org> In-Reply-To: <CAHM0Q_PCr=8e26E%2BsTM_9GSZe0uOQyEw_UeHyhpbF-J81-AXyw@mail.gmail.com> References: <CAFMmRNyJpvZ0AewWr62w16=qKer%2BFNXUJJy0Qc=EBqMnUV3OyQ@mail.gmail.com> <20131226175410.GA15332@lath.rinet.ru> <D06D9912.1507F%mdelagueronniere@verisign.com> <CAHM0Q_PCr=8e26E%2BsTM_9GSZe0uOQyEw_UeHyhpbF-J81-AXyw@mail.gmail.com>
next in thread | previous in thread | raw e-mail | index | archive | help
[-- Attachment #1 --]
Hi,
On 30/10/14 20:39, K. Macy wrote:
>> I also suspect there are further problems with buf_ring. A full wrap
>> around of the atomically swapped value is possible. I.e. the code thinks
>> it just atomically updated a head/tail index when in fact a full wrap
>> around occurred leading to undefined land. A relatively simple way to
>> avoid this is to only mask on ring array access, and to let the
>> head/tail/prod/cons indices overflow the array.
>
> Up until Rui Paulo complained to me of packet drops with buf_ring a
> couple of days ago I had thought that this patch had been committed.
> This patch (now 273866) fixes the problem for him. Without further
> scrutiny and testing I won't provide the UL guarantee for
> buf_ring_enqueue, but this is a clear improvement.
I have tested r273866 fix against our traffic and it seems it fixes the
ENOBUFS issue for us too. Below the Dtrace script used for checking who
was returning ENOBUFS:
$ cat enobufs.d
fbt::sendit:return
/arg1 == ENOBUFS/
{
@senditENOBUFS[execname,pid] = count();
}
fbt::ixgbe_mq_start:return
/arg1 == ENOBUFS/
{
@ixgbeStartENOBUFS[execname,pid] = count();
}
fbt::ip_output:return
/arg1 != 0/
{
@ipOutputErr[execname,pid] = count();
}
END
{
printf("\nsendit ENOBUFS:\n");
printa(@senditENOBUFS);
printf("\nixgbe_mq_start ENOBUFS:\n");
printa(@ixgbeStartENOBUFS);
printf("\nip_output Errors:\n");
printa(@ipOutputErr);
}
The result without the fix:
$ dtrace -s enobufs.d
dtrace: script 'enobufs.d' matched 4 probes
^C
CPU ID FUNCTION:NAME
0 2 :END
sendit ENOBUFS:
server 1779 47
ixgbe_mq_start ENOBUFS:
server 1779 47
ip_output Errors:
server 1779 47
Thus 47 ENOBUFS errors, all returned by ixgbe_mq_start() which calls
drbr_enqueue().
And result with the fix applied:
$ dtrace -s enobufs.d
dtrace: script 'enobufs.d' matched 4 probes
^C
CPU ID FUNCTION:NAME
3 2 :END
sendit ENOBUFS:
ixgbe_mq_start ENOBUFS:
ip_output Errors:
Thus 0 ENOBUFS errors. I will let Marc review this fix further to
check his full wrap case.
Thanks.
--
Julien
[-- Attachment #2 --]
-----BEGIN PGP SIGNATURE-----
Version: GnuPG/MacGPG2 v2.0.22 (Darwin)
Comment: GPGTools - https://gpgtools.org
iQEcBAEBCgAGBQJUU348AAoJEKVlQ5Je6dhxYGwIALJihOjnir20oD5/J8sDK7Vc
7ypJEga3zc296bM5v5F4r3wRdRH/3iEmVI1f+JI+IU4o40JqhwDNsMBz2DZ41oTO
iTtK7KD1v9BxOq8YD/WfMKZmC6Y0yGjd8LRYXZ8YiZLVnbhHioKMvkqn3LMBGOfE
68cJqPgYqEGSURk/lynHAaspS6l1qX0bUjBXDO3Gv79YmO7AcQuFEsc3+tkhfTPE
649ecEgNPDh3uVKT9D9hUdxslw5Rhe97KAj35cEbyQoczMZCvEpoL3VIOhaIYifN
djf5J4VsJdcanE3vWkDm1MFxDEsVRJ+CntsMZ7PXSscGjgbgO4pu2Bc3m5UQ9EE=
=bm38
-----END PGP SIGNATURE-----
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?54537E35.2060108>