From owner-freebsd-net@freebsd.org Tue Jan 24 15:31:19 2017 Return-Path: Delivered-To: freebsd-net@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id A15D6CC0043; Tue, 24 Jan 2017 15:31:19 +0000 (UTC) (envelope-from sbruno@freebsd.org) Received: from mail.ignoranthack.me (ignoranthack.me [199.102.79.106]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 66157F6F; Tue, 24 Jan 2017 15:31:18 +0000 (UTC) (envelope-from sbruno@freebsd.org) Received: from [192.168.0.6] (67-0-248-244.albq.qwest.net [67.0.248.244]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) (Authenticated sender: sbruno@ignoranthack.me) by mail.ignoranthack.me (Postfix) with ESMTPSA id 025E31928BA; Tue, 24 Jan 2017 15:31:17 +0000 (UTC) Subject: Re: HEADS-UP: IFLIB implementations of sys/dev/e1000 em, lem, igb pending To: =?UTF-8?Q?Olivier_Cochard-Labb=c3=a9?= References: <30f21c75-d3a2-edcd-1999-d5ed9f970c06@freebsd.org> <1598d97bf2a.c6bcb76838987.6501340920645175463@nextbsd.org> <574a7ac7-4842-9518-8286-a4d89a9f7a27@freebsd.org> <6c6cb534-73c7-464b-8af1-7445a9c0188c@freebsd.org> <1598f29d379.ea6360351471.8752933472741761813@nextbsd.org> <1598f3f8588.d20017893749.339651164872952258@nextbsd.org> <1598f42ad77.eeec05be4113.9201780237587761460@nextbsd.org> <159902b73ed.10775291e21533.7488368455500235608@nextbsd.org> <18abdd64-08a6-50ca-fb6b-9c01a3d7b60c@freebsd.org> Cc: Matthew Macy , "freebsd-net@freebsd.org" , "freebsd-current@freebsd.org" From: Sean Bruno Message-ID: Date: Tue, 24 Jan 2017 08:31:13 -0700 User-Agent: Mozilla/5.0 (X11; FreeBSD amd64; rv:45.0) Gecko/20100101 Thunderbird/45.6.0 MIME-Version: 1.0 In-Reply-To: Content-Type: multipart/signed; micalg=pgp-sha512; protocol="application/pgp-signature"; boundary="Gjupok5Xwh92XDQejKlekdOSOrwjfXdic" X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 24 Jan 2017 15:31:19 -0000 This is an OpenPGP/MIME signed message (RFC 4880 and 3156) --Gjupok5Xwh92XDQejKlekdOSOrwjfXdic Content-Type: multipart/mixed; boundary="xDI9f4ipfWJNETqafQe9wabhuoRSaXFIG"; protected-headers="v1" From: Sean Bruno To: =?UTF-8?Q?Olivier_Cochard-Labb=c3=a9?= Cc: Matthew Macy , "freebsd-net@freebsd.org" , "freebsd-current@freebsd.org" Message-ID: Subject: Re: HEADS-UP: IFLIB implementations of sys/dev/e1000 em, lem, igb pending References: <30f21c75-d3a2-edcd-1999-d5ed9f970c06@freebsd.org> <1598d97bf2a.c6bcb76838987.6501340920645175463@nextbsd.org> <574a7ac7-4842-9518-8286-a4d89a9f7a27@freebsd.org> <6c6cb534-73c7-464b-8af1-7445a9c0188c@freebsd.org> <1598f29d379.ea6360351471.8752933472741761813@nextbsd.org> <1598f3f8588.d20017893749.339651164872952258@nextbsd.org> <1598f42ad77.eeec05be4113.9201780237587761460@nextbsd.org> <159902b73ed.10775291e21533.7488368455500235608@nextbsd.org> <18abdd64-08a6-50ca-fb6b-9c01a3d7b60c@freebsd.org> In-Reply-To: --xDI9f4ipfWJNETqafQe9wabhuoRSaXFIG Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable On 01/24/17 08:27, Olivier Cochard-Labb=C3=A9 wrote: > On Tue, Jan 24, 2017 at 3:17 PM, Sean Bruno > wrote: >=20 >=20 >=20 > Did you increase the number of rx/tx rings to 8 and the number of > descriptors to 4k in your tests or just the defaults? >=20 >=20 > Tuning are same as described in my previous email (rxd|txd=3D2048, rx|t= x > process_limit=3D-1, max_interrupt_rate=3D16000). > [root@apu2]~# sysctl hw.igb. > hw.igb.tx_process_limit: -1 > hw.igb.rx_process_limit: -1 > hw.igb.num_queues: 0 > hw.igb.header_split: 0 > hw.igb.max_interrupt_rate: 16000 > hw.igb.enable_msix: 1 > hw.igb.enable_aim: 1 > hw.igb.txd: 2048 > hw.igb.rxd: 2048 >=20 >=20 Oh, I think you missed my note on these. In order to adjust txd/rxd you need to tweak the iflib version of these numbers. nrxds/ntxds should be adjust upwards to your value of 2048. nrxqs/ntxqs should be adjust upwards to 8, I think, so you can test equivalent settings to the legacy driver. Specifically, you may want to adjust these: dev.em.0.iflib.override_nrxds: 0 dev.em.0.iflib.override_ntxds: 0 dev.em.0.iflib.override_nrxqs: 0 dev.em.0.iflib.override_ntxqs: 0 sean > But I've did a new benchs with default setting, and the performance dro= p > is now about -25% : >=20 > x head r311848 packets-per-second (default settings) > + head r311849 packets-per-second (default settings) > +----------------------------------------------------------------------= ----+ > |+ = | > |+ = x | > |+ = xx| > |++ = xx| > | = A|| > |A| = | > +----------------------------------------------------------------------= ----+ > N Min Max Median Avg St= ddev > x 5 618711 621135 619930.5 619840.8 951.8= 3787 > + 5 467389 468740 467778 467864.8 550.4= 0322 > Difference at 95.0% confidence > -151976 +/- 1133.9 > -24.5186% +/- 0.150581% > (Student's t, pooled s =3D 777.476) >=20 --xDI9f4ipfWJNETqafQe9wabhuoRSaXFIG-- --Gjupok5Xwh92XDQejKlekdOSOrwjfXdic Content-Type: application/pgp-signature; name="signature.asc" Content-Description: OpenPGP digital signature Content-Disposition: attachment; filename="signature.asc" -----BEGIN PGP SIGNATURE----- iQGTBAEBCgB9FiEEuq1GMucSHejSCZfdEgHvyh5yfmQFAliHc0FfFIAAAAAALgAo aXNzdWVyLWZwckBub3RhdGlvbnMub3BlbnBncC5maWZ0aGhvcnNlbWFuLm5ldEJB QUQ0NjMyRTcxMjFERThEMjA5OTdERDEyMDFFRkNBMUU3MjdFNjQACgkQEgHvyh5y fmRKDAgAlleMvp13X8F/56RMAHj+zK2gnL3R1BRAzjeG3fGgPTiz/SnqaR9ngadl d3kYkAxMUyFz5zSmWW3d9BB49No+syppWdBIyH1Pv4wqNsSQOUP5pXl1ot143H84 Y9jEDgN4/QluuQXUaj1YvG7s6n1Bw8wE7SSCcjqvDg0O1L5xTOSQF+atwDd3+o6r zUOzhdYwKuc0f98vsGTspCu5JQ/FfS/p9Y5F3xvTo9Gj6HtbbBXHuJ3k5eBeHJ0f aKUhBSpxVrDBLUBpaj7ClO0/166PnBBUU1bRaty5iDJbMmYgnWDtQtvaoCOS2/U7 XM9vSI6D6kuZAY6k7MidzZjFCLP3sQ== =GH+B -----END PGP SIGNATURE----- --Gjupok5Xwh92XDQejKlekdOSOrwjfXdic--