Date: Thu, 1 Dec 2016 20:57:44 -0500 From: Allan Jude <allanjude@freebsd.org> To: freebsd-performance@freebsd.org Subject: Re: netmap, netmap-fwd, and how many M packets-per-second? Message-ID: <d4ffde3f-07e1-6e75-33c0-9a00eab6dc68@freebsd.org> In-Reply-To: <596e5c15-1a85-1fdc-ca88-11a0ccd5e125@gmail.com> References: <596e5c15-1a85-1fdc-ca88-11a0ccd5e125@gmail.com>
next in thread | previous in thread | raw e-mail | index | archive | help
This is an OpenPGP/MIME signed message (RFC 4880 and 3156) --A5NBumTpSC0H133TEWGpOh5wh2R2wnvC3 Content-Type: multipart/mixed; boundary="dlN5p3kIwoAhIs69TnRdLn95HeM226HHo"; protected-headers="v1" From: Allan Jude <allanjude@freebsd.org> To: freebsd-performance@freebsd.org Message-ID: <d4ffde3f-07e1-6e75-33c0-9a00eab6dc68@freebsd.org> Subject: Re: netmap, netmap-fwd, and how many M packets-per-second? References: <596e5c15-1a85-1fdc-ca88-11a0ccd5e125@gmail.com> In-Reply-To: <596e5c15-1a85-1fdc-ca88-11a0ccd5e125@gmail.com> --dlN5p3kIwoAhIs69TnRdLn95HeM226HHo Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable On 2016-12-01 18:55, Jordan Caraballo wrote: > Feedback and/or tips and tricks more than welcome. >=20 > We are trying to process huge amounts of small (64 bytes) pps through a= > router. So far results have not been as we expected. We have tested > FreeBSD 10.3, 11.0, 11.0-STABLE, and 12.0-CURRENT with and without > netmap. Based on netmap documentation we were expecting about 5.0M pps;= > alongside with the routing improvements from the freebsd routing > proposal a total of 12.0M. >=20 > Server Description: >=20 > Dell PowerEdge R530 with 2 Intel(R) Xeon(R) E5=C2=AD2695 CPU's, 18 core= s per > cpu. Equipped with a Chelsio T-580-CR dual port in an 8x slot. >=20 > BIOS tweaks: >=20 > Hyperthreading (or Logical Processors) is turned off. >=20 > Current results are shown below. Additional configurations can be given= > upon request. >=20 > Test Environment: > 5 clients and 5 servers - 4 Dell C6100 and 2 Dell R420; each one > equipped with 10G NICS (4 intel 8259X and 6 with mellanox connectx2). >=20 > Script that execute the following on each host. >=20 > #!/usr/local/bin/bash > # Iterate through ports and start tests > for ((i=3D1;i<=3DPORTS;i++)); do > PORT=3D$(($PORT+1)) > iperf3 -c 172.16.2.$IP -u -b10000m -i 0 -N -l$PKT -t$TIME -P$STREAM= S > -p$PORT & > #iperf3 -c 172.16.2.$IP -i0 -N -l$PKT -t$TIME -P$STREAMS -p$PORT & > done >=20 > # FreeBSD 10.3 - 4 streams to 80 ports from each client (5) >=20 > input (Total) output > packets errs idrops bytes packets errs bytes colls dr= ops > 1.9M 0 1.3M 194M 540k 0 57M 0 0 > 2.1M 0 1.5M 216M 556k 0 58M 0 0 > 1.8M 0 1.3M 192M 553k 0 58M 0 0 > 1.7M 0 1.1M 174M 542k 0 57M 0 0 > 1.9M 0 1.4M 204M 537k 0 56M 0 0 > 1.6M 0 1.1M 171M 550k 0 58M 0 0 > 1.6M 0 1.1M 173M 546k 0 57M 0 0 > 1.7M 0 1.1M 176M 564k 0 59M 0 0 > 2.0M 0 1.5M 212M 543k 0 57M 0 0 > 2.1M 0 1.5M 219M 557k 0 58M 0 0 > 1.9M 0 1.4M 205M 547k 0 57M 0 0 > 1.7M 0 1.2M 179M 553k 0 58M 0 0 >=20 > # FreeBSD 11.0 - 4 streams to 80 ports from each client (5) >=20 > input (Total) output > packets errs idrops bytes packets errs bytes colls dr= ops > 3.1M 0 1.8M 326M 1.3M 0 134M 0 0 > 2.6M 0 1.5M 269M 1.1M 0 116M 0 0 > 2.7M 0 1.5M 285M 1.2M 0 127M 0 0 > 2.4M 0 1.3M 257M 1.1M 0 119M 0 0 > 2.7M 0 1.5M 287M 1.3M 0 134M 0 0 > 2.5M 0 1.3M 262M 1.2M 0 127M 0 0 > 2.1M 0 1.1M 224M 1.0M 0 108M 0 0 > 2.7M 0 1.4M 285M 1.4M 0 143M 0 0 > 2.6M 0 1.3M 272M 1.3M 0 136M 0 0 > 2.5M 0 1.4M 265M 1.1M 0 120M 0 0 > # FreeBSD 11.0-STABLE - 4 streams to 80 ports from each client (5) >=20 > input (Total) output > packets errs idrops bytes packets errs bytes colls dr= ops > 1.9M 0 849k 195M 1.0M 0 107M 0 0 > 1.9M 0 854k 196M 1.0M 0 106M 0 0 > 1.9M 0 851k 196M 1.0M 0 107M 0 0 > 1.9M 0 851k 196M 1.0M 0 107M 0 0 > 1.9M 0 851k 196M 1.0M 0 107M 0 0 > 1.9M 0 852k 196M 1.0M 0 107M 0 0 > 1.9M 0 847k 195M 1.0M 0 107M 0 0 > 1.9M 0 836k 195M 1.0M 0 107M 0 0 > 1.9M 0 843k 195M 1.0M 0 107M 0 0 > # FreeBSD 12.0-CURRENT - 4 streams to 80 ports from each client (5) >=20 > input (Total) output > packets errs idrops bytes packets errs bytes colls dr= ops > 1.1M 259 0 115M 1.1M 0 115M 0 0 > 1.2M 273 0 124M 1.2M 0 124M 0 0 > 1.1M 200 0 112M 1.1M 0 112M 0 0 > 1.2M 290 0 122M 1.2M 0 122M 0 0 > 1.0M 132 0 107M 1.0M 0 107M 0 0 > 1.1M 303 0 118M 1.1M 0 118M 0 0 > 1.1M 278 0 112M 1.1M 0 112M 0 0 > 1.2M 243 0 122M 1.2M 0 122M 0 0 > 1.1M 168 0 112M 1.1M 0 112M 0 0 > 1.1M 161 0 112M 1.1M 0 112M 0 0 > # FreeBSD 12.0-CURRENT + Netmap - 4 streams to 80 ports from each clien= t > (5) >=20 > input (Total) output > packets errs idrops bytes packets errs bytes colls dr= ops > 1.4M 10 0 144M 1.4M 0 144M 0 0 > 1.5M 6 0 159M 1.5M 0 159M 0 0 > 1.4M 5 0 144M 1.4M 0 144M 0 0 > 1.5M 14 0 158M 1.5M 0 158M 0 0 > 1.4M 5 0 151M 1.4M 0 151M 0 0 > 1.4M 10 0 152M 1.4M 0 152M 0 0 > 1.4M 12 0 148M 1.4M 0 148M 0 0 > 1.5M 9 0 155M 1.5M 0 155M 0 0 > 1.4M 23 0 151M 1.4M 0 151M 0 0 > 1.4M 11 0 151M 1.4M 0 151M 0 0 > # FreeBSD 12.0-CURRENT + Netmap + Tuning - 4 streams to 80 ports from > each client (5) >=20 > input (Total) output > packets errs idrops bytes packets errs bytes colls dr= ops > 1.4M 15 0 145M 1.4M 0 145M 0 0 > 1.5M 18 0 157M 1.5M 0 157M 0 0 > 1.5M 10 0 156M 1.5M 0 156M 0 0 > 1.5M 15 0 154M 1.5M 0 154M 0 0 > 1.4M 13 0 146M 1.4M 0 146M 0 0 > 1.5M 15 0 156M 1.5M 0 156M 0 0 > 1.5M 9 0 155M 1.5M 0 155M 0 0 > 1.5M 13 0 153M 1.5M 0 153M 0 0 > 1.4M 14 0 145M 1.4M 0 145M 0 0 > 1.4M 17 0 151M 1.4M 0 151M 0 0 >=20 > _______________________________________________ > freebsd-performance@freebsd.org mailing list > https://lists.freebsd.org/mailman/listinfo/freebsd-performance > To unsubscribe, send any mail to > "freebsd-performance-unsubscribe@freebsd.org" You might try pktgen, the netmap tool for generating packets, rather than iperf, as your source of traffic. Olivier Cochard-Labb=C3=A9 has a series of benchmarks on github in his 'netbenches' project https://github.com/ocochard/netbenches/blob/master/Xeon_E5-2650-8Cores-Ch= elsio_T540-CR/fastforwarding-pf-ipfw/results/fbsd12-head.r299288.D8526/RE= ADME.md This shows FreeBSD 12 @r299288 doing ~ 2.5 M pps of forwarding (so recv and then resend) in a stock configuration, and reaching almost 5 million pps after a patch that has since been committed to -CURRENT. --=20 Allan Jude --dlN5p3kIwoAhIs69TnRdLn95HeM226HHo-- --A5NBumTpSC0H133TEWGpOh5wh2R2wnvC3 Content-Type: application/pgp-signature; name="signature.asc" Content-Description: OpenPGP digital signature Content-Disposition: attachment; filename="signature.asc" -----BEGIN PGP SIGNATURE----- Version: GnuPG v2.0.22 (MingW32) iQIcBAEBAgAGBQJYQNUdAAoJEBmVNT4SmAt+FUAQAMuDmX1M0z5Z+s3EgYRBMsEt Tygqe4BvlzYVNJTnreIDZR3gJt28dMhdxoT1EHvpEi5L/1bcbMM5f0CDQhf4E/XF M1mpSgXlx9CsNrNDZlVEKOLDwZapRBoAn8qxw46vfVrZlwO8DYqW+vQBPBCNiPQb 1LcjBE4H+cAamx9Q2PE8CjZOhXunkbiqh1jAAho6kSbmg0jG7sxkytcGWsYn5s/f dAUVvb5XgJtU6In8s74E3PG2qsi7c6EDUu5zWIbczkf+ecEjxQ4uXlEsB0K0yRYp oD/KI/SMMJ7P9yu0f5v1DQeDuKm38k6hetxkbNIW2nDfF1/KzdYsRMnytSjqt1GT pbjR1vWCFahUE2XpG2KUHBw3fxdXBk1BvOAV7Z3kupSXRHPXPuiS1bFEV+aMYdIZ FW8JzhhyCOMOpPfLkmlcA5qKpfL4+mkytaRiWNudJFrx0gbErTNVDgv5Hnx/Z9Tp h+OnX1qNimxhylYSXyYgp63RGy/lPFQ1mxoJZe/GT19j4aaOZKG/LakPDdakwccy 25zSy3lGekDLKgdIJX3/Wisqm9Fo5VHlplqWvHt7uLV38WS3hNS9G87FqKzZdfw+ vJB8IbG+ZxR/y93pStHzG3gKbwZ/sqL+FR0zAZI+0IRc4PhfHb5oqsp29NZc1b1R OiF0v8e964tNkWoBuySx =IiJE -----END PGP SIGNATURE----- --A5NBumTpSC0H133TEWGpOh5wh2R2wnvC3--
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?d4ffde3f-07e1-6e75-33c0-9a00eab6dc68>