From owner-freebsd-net@FreeBSD.ORG Fri Oct 31 12:19:20 2014 Return-Path: Delivered-To: freebsd-net@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 671BD44F for ; Fri, 31 Oct 2014 12:19:20 +0000 (UTC) Received: from mail-la0-f49.google.com (mail-la0-f49.google.com [209.85.215.49]) (using TLSv1 with cipher ECDHE-RSA-RC4-SHA (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id DA629991 for ; Fri, 31 Oct 2014 12:19:19 +0000 (UTC) Received: by mail-la0-f49.google.com with SMTP id ge10so5903102lab.8 for ; Fri, 31 Oct 2014 05:19:12 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20130820; h=x-gm-message-state:message-id:date:from:user-agent:mime-version:to :cc:subject:references:in-reply-to:content-type; bh=R3mYbLgaKRGTjp5A12gNES7kH491KJuqQ+Xu9KA3YUc=; b=jhFL3SVnRkGCIXDVIzrjNo6quI/9p1JpOwTGFk1HuMdR7n3WlhsyHF3w3lyCTNW/Tx tU1cX34gedl/9n4SMfDqazEgAVnu1Mn7hlpA5Df0uv28FIF19xGwwoVLvm5Hx7FzrMfq UbfAUpRVjObLQePg5NIx7I4Lg65c8DAuVAwyRmZds3QM7Ymo+x/1rPD1zt5YXc4FGj22 Oh+LAbhlwxxBjbdpWY8iE+ZfsWztCQ+UWd3RUFFELJ8rBKc8sElcXYQaMOB0t0JuV7eB Cv961MbovyFqtm7uhf6vc8f6ds6IKAluKfRpsk5upCZpitaIWTaRLQe/gWq8Jipd0mrk c1GA== X-Gm-Message-State: ALoCoQlRi4THoZY3bC8mWLmkzZsmNvxuQsVA1FB9yX13i0yF9mfPbDZaQSNi/gVrTsyH6EINkV9T X-Received: by 10.112.12.35 with SMTP id v3mr25768830lbb.80.1414757952344; Fri, 31 Oct 2014 05:19:12 -0700 (PDT) Received: from FRI2JCHARBON-M1.local ([217.30.88.7]) by mx.google.com with ESMTPSA id lf5sm4390605lac.16.2014.10.31.05.19.10 for (version=TLSv1 cipher=ECDHE-RSA-RC4-SHA bits=128/128); Fri, 31 Oct 2014 05:19:11 -0700 (PDT) Message-ID: <54537E35.2060108@freebsd.org> Date: Fri, 31 Oct 2014 13:19:01 +0100 From: Julien Charbon User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10.9; rv:24.0) Gecko/20100101 Thunderbird/24.6.0 MIME-Version: 1.0 To: freebsd-net Subject: Re: buf_ring in HEAD is racy References: <20131226175410.GA15332@lath.rinet.ru> In-Reply-To: Content-Type: multipart/signed; micalg=pgp-sha512; protocol="application/pgp-signature"; boundary="ue48W3K8oNOB8hpL4MxaQn4mP7sKkFQfn" Cc: Oleg Bulyzhin , Ryan Stone , "K. Macy" , "De La Gueronniere, Marc" X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.18-1 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 31 Oct 2014 12:19:20 -0000 This is an OpenPGP/MIME signed message (RFC 4880 and 3156) --ue48W3K8oNOB8hpL4MxaQn4mP7sKkFQfn Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: quoted-printable Hi, On 30/10/14 20:39, K. Macy wrote: >> I also suspect there are further problems with buf_ring. A full wrap >> around of the atomically swapped value is possible. I.e. the code thin= ks >> it just atomically updated a head/tail index when in fact a full wrap >> around occurred leading to undefined land. A relatively simple way to >> avoid this is to only mask on ring array access, and to let the >> head/tail/prod/cons indices overflow the array. >=20 > Up until Rui Paulo complained to me of packet drops with buf_ring a > couple of days ago I had thought that this patch had been committed. > This patch (now 273866) fixes the problem for him. Without further > scrutiny and testing I won't provide the UL guarantee for > buf_ring_enqueue, but this is a clear improvement. I have tested r273866 fix against our traffic and it seems it fixes the ENOBUFS issue for us too. Below the Dtrace script used for checking who was returning ENOBUFS: $ cat enobufs.d fbt::sendit:return /arg1 =3D=3D ENOBUFS/ { @senditENOBUFS[execname,pid] =3D count(); } fbt::ixgbe_mq_start:return /arg1 =3D=3D ENOBUFS/ { @ixgbeStartENOBUFS[execname,pid] =3D count(); } fbt::ip_output:return /arg1 !=3D 0/ { @ipOutputErr[execname,pid] =3D count(); } END { printf("\nsendit ENOBUFS:\n"); printa(@senditENOBUFS); printf("\nixgbe_mq_start ENOBUFS:\n"); printa(@ixgbeStartENOBUFS); printf("\nip_output Errors:\n"); printa(@ipOutputErr); } The result without the fix: $ dtrace -s enobufs.d dtrace: script 'enobufs.d' matched 4 probes ^C CPU ID FUNCTION:NAME 0 2 :END sendit ENOBUFS: server 1779 47 ixgbe_mq_start ENOBUFS: server 1779 47 ip_output Errors: server 1779 47 Thus 47 ENOBUFS errors, all returned by ixgbe_mq_start() which calls drbr_enqueue(). And result with the fix applied: $ dtrace -s enobufs.d dtrace: script 'enobufs.d' matched 4 probes ^C CPU ID FUNCTION:NAME 3 2 :END sendit ENOBUFS: ixgbe_mq_start ENOBUFS: ip_output Errors: Thus 0 ENOBUFS errors. I will let Marc review this fix further to check his full wrap case. Thanks. -- Julien --ue48W3K8oNOB8hpL4MxaQn4mP7sKkFQfn Content-Type: application/pgp-signature; name="signature.asc" Content-Description: OpenPGP digital signature Content-Disposition: attachment; filename="signature.asc" -----BEGIN PGP SIGNATURE----- Version: GnuPG/MacGPG2 v2.0.22 (Darwin) Comment: GPGTools - https://gpgtools.org iQEcBAEBCgAGBQJUU348AAoJEKVlQ5Je6dhxYGwIALJihOjnir20oD5/J8sDK7Vc 7ypJEga3zc296bM5v5F4r3wRdRH/3iEmVI1f+JI+IU4o40JqhwDNsMBz2DZ41oTO iTtK7KD1v9BxOq8YD/WfMKZmC6Y0yGjd8LRYXZ8YiZLVnbhHioKMvkqn3LMBGOfE 68cJqPgYqEGSURk/lynHAaspS6l1qX0bUjBXDO3Gv79YmO7AcQuFEsc3+tkhfTPE 649ecEgNPDh3uVKT9D9hUdxslw5Rhe97KAj35cEbyQoczMZCvEpoL3VIOhaIYifN djf5J4VsJdcanE3vWkDm1MFxDEsVRJ+CntsMZ7PXSscGjgbgO4pu2Bc3m5UQ9EE= =bm38 -----END PGP SIGNATURE----- --ue48W3K8oNOB8hpL4MxaQn4mP7sKkFQfn--