From owner-freebsd-net@freebsd.org Thu Mar 16 19:50:44 2017 Return-Path: Delivered-To: freebsd-net@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 37055D0F583 for ; Thu, 16 Mar 2017 19:50:44 +0000 (UTC) (envelope-from jjasen@gmail.com) Received: from mail-ot0-x236.google.com (mail-ot0-x236.google.com [IPv6:2607:f8b0:4003:c0f::236]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id E1F4F104E for ; Thu, 16 Mar 2017 19:50:43 +0000 (UTC) (envelope-from jjasen@gmail.com) Received: by mail-ot0-x236.google.com with SMTP id i1so68602563ota.3 for ; Thu, 16 Mar 2017 12:50:43 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=mime-version:in-reply-to:references:from:date:message-id:subject:to :cc; bh=DyWFFE4DcaHgYn5SbWnmUUZ0orkWaMo+MxEn/vZmJBU=; b=vW/ALZ/uDeVSk85VCejx8ep+KMqW6YYyLxCJj38bUoTdND0/hwaVRZvIBx5SSMY07P xpu0KA1wQn9hwjFmIz7ypzel18HxtgAmioB2mt1gYj+n/0AyxXYMHXiMHtozzjPx3q5I wv2ah67O06QvkwNYzB/co1WmuY3d7tn1uVJPtjllTYgOfKEJOuV7xjHfUEgvsac8LO2I X1CmzrWkxEZHpFzBf3aNJWvFTKREkuf5neeZVNW0J2qiEkrHQ5jZH7wNfQsXNl5pmdkq mOXF9gX6JRjwVhHM91l96M6VyyQRQqgmay4GDlAXwA0mn4MKCeo3CBP3frcnJu+aAgWN IHsw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:in-reply-to:references:from:date :message-id:subject:to:cc; bh=DyWFFE4DcaHgYn5SbWnmUUZ0orkWaMo+MxEn/vZmJBU=; b=X1+O2N7v3jBklVcGURRCeWY1S1eMXfvEKG6mymsE38GJ8rGu/TRExHUn5U4VfuUFYn NlPS4VtREvbk8ETWkjQgCFgw0JFbGojK4DeX6Fb1vh0qs6gH3M4t2hiqHso+6RNfGtK0 62796GJdblpqISRuvv55CdSnFFzPzvFVBCtSLJbwm6YrzINfk+WzZs6dXEKl/PMJjrUd KKZQP60wf1ZuzcbK8lIcEWLlqK8rbhUnHWf/DinwBtjFlQ0NrRuztTICcwcCBhwYaEfs 2vDBf9GDJmuIs+FlPdstpejNSqoQEv2sKjAPfFqSfX6AgbNrw/TI+qvG9ifIn89QF2bH lhlw== X-Gm-Message-State: AFeK/H0qwWssmpatXzvzPWYWBks45s96p6YIuPi0n5vWltn8oNhK0nzsqT/g+7uQt1ZTkIKf1OI8qGGIhBe/6g== X-Received: by 10.202.51.10 with SMTP id z10mr5783741oiz.214.1489693843051; Thu, 16 Mar 2017 12:50:43 -0700 (PDT) MIME-Version: 1.0 Received: by 10.157.47.4 with HTTP; Thu, 16 Mar 2017 12:50:42 -0700 (PDT) In-Reply-To: References: <40a413f3-2c44-ee9d-9961-67114d8dffca@gmail.com> <20170205175531.GA20287@dwarf> <7d349edd-0c81-2e3f-d3b9-27af232de76d@gmail.com> <20170209153409.GG41673@dwarf> <6ad029e0-86c6-af3d-8fc3-694d4bcdc683@gmail.com> <20170312231826.GV15630@zxy.spb.ru> <74654520-b8b6-6118-2e46-902a8ea107ac@gmail.com> <173fffac-7ae2-786a-66c0-e9cd7ab78f44@gmail.com> From: John Jasen Date: Thu, 16 Mar 2017 15:50:42 -0400 Message-ID: Subject: Re: bad throughput performance on multiple systems: Re: Fwd: Re: Disappointing packets-per-second performance results on a Dell,PE R530 To: Navdeep Parhar Cc: Slawa Olhovchenkov , "freebsd-net@freebsd.org" , "Caraballo-vega, Jordan A." Content-Type: text/plain; charset=UTF-8 X-Content-Filtered-By: Mailman/MimeDel 2.1.23 X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 16 Mar 2017 19:50:44 -0000 As a few points of note, partial resolution, and curiosity: Following down leads that 11-STABLE had tryforward improvements over 11-RELENG, I upgraded. The same tests (24 client streams over UDP with small packets), the system went from passing 1.7m pps to about 2.5m. Following indications from Navdeep Parhar that UDP queue hashing is not as efficient as it could be, we started running the tests with various powers of 2 streams (2,4,8,16,32) -- and were able to push the system up to 5m pps. We are currently seeing in the tests approximately 10-11m pps on the outside interface, around 5-6m dropped, and 5 million passed. On Mon, Mar 13, 2017 at 1:31 PM, Navdeep Parhar wrote: > On Mon, Mar 13, 2017 at 10:13 AM, John Jasen wrote: > > On 03/13/2017 01:03 PM, Navdeep Parhar wrote: > > > >> On Sun, Mar 12, 2017 at 5:35 PM, John Jasen wrote: > >>> UDP traffic. dmesg reports 16 txq, 8 rxq -- which is the default for > >>> Chelsio. > >>> > >> I don't recall offhand, but UDP might be using 2-tuple hashing by > >> default and that might affect the distribution of flows across queues. > >> Are there senders generating IP fragments by any chance (that'll > >> depend on the "send size" that your UDP application is using)? > > > > No, they're not fragmenting. > > > >> Have you tried limiting the adapter's rx ithreads to the CPU that the > >> PCIe slot with the adapter is wired to? > > > > Above and beyond the use of cpuset, you mean? > > I meant cpuset. > > If possible, try your experiments on a single socket system. > > Regards, > Navdeep >