From owner-freebsd-hpc@freebsd.org Thu Dec 1 23:56:00 2016 Return-Path: Delivered-To: freebsd-hpc@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id BBE92C6246E; Thu, 1 Dec 2016 23:56:00 +0000 (UTC) (envelope-from jordancaraballo87@gmail.com) Received: from mail-qk0-x241.google.com (mail-qk0-x241.google.com [IPv6:2607:f8b0:400d:c09::241]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 6E23C175B; Thu, 1 Dec 2016 23:55:57 +0000 (UTC) (envelope-from jordancaraballo87@gmail.com) Received: by mail-qk0-x241.google.com with SMTP id x190so28498601qkb.0; Thu, 01 Dec 2016 15:55:57 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=to:from:subject:message-id:date:user-agent:mime-version :content-transfer-encoding; bh=qjaNJl1tNUlvHHPfgb6DniaPFb0SeNfUaHLVG4mocFM=; b=W7MVe7/JGx3+Ww+vElPNIOBEa+1q4XsNRUkfBUtMYKI2AybK9sKlYCwiG2q5m8+Nnq 6HrVYfXd1TcMtrr9cZLWHX8LY2doKmH+55FqAYeswqdsiwlk29GSeB0VOd0f0Dp6KFuB qcYF4Xa19L+ZD8hfjc0zEMK6tPXNtKSpksTFytTm6qGU0f4cpebYpaYwNkY9kCFep6le iqEzlXa7P0bUyRGVPqJd9/ajSGf6qp9J/U3kbIgGnpSIXiOz6bjpDVxdxSWpvzM3caN9 Ujth4gBevxtGBZsvDJpirmFrohsY4td3BXECRxRysP1yAay52EW5EecIkNO7KPvTzzaX BZGQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20130820; h=x-gm-message-state:to:from:subject:message-id:date:user-agent :mime-version:content-transfer-encoding; bh=qjaNJl1tNUlvHHPfgb6DniaPFb0SeNfUaHLVG4mocFM=; b=NeFm0S2xBNw/4DcIxLgdXzJbH0rzt/A1yxzSIskJQ3fMjp+byiIS3QiB0ogncwMi38 abgxjaX5+yGfUYSIZbWM7PBUA2/xB83JT7hkiiGK5BirxUusxq8oEox+ZghfjYn5pB+G 8IVtdn7qeezuI2IlCtzvU+RiU1fqs4CmX1XPaO32zwBvFe823nwVlcJHrCjSD/Umsmdv MCcgot0BqxbnirzY+EMn1q9UTJfGEJ1G00v5VZWiSGyAC/hehJJZ/ZIS0rdS8yk5cI8f 8dDLkURGaoi36PHKAF+ozusFTKRb0WkDj0DVJlC5XJ0JIhJXoNxi8URWLktBRUkJm+aZ bkYg== X-Gm-Message-State: AKaTC00Q3yjMss+vj364rtdOalsl4KR26aC5cWc16UaiHO/pmNSZiTDhNEADrDfqM+C32Q== X-Received: by 10.55.116.135 with SMTP id p129mr40035180qkc.54.1480636556636; Thu, 01 Dec 2016 15:55:56 -0800 (PST) Received: from [192.168.1.5] (pool-96-241-162-208.washdc.fios.verizon.net. [96.241.162.208]) by smtp.gmail.com with ESMTPSA id 30sm1323246qth.14.2016.12.01.15.55.55 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Thu, 01 Dec 2016 15:55:55 -0800 (PST) To: freebsd-net@freebsd.org, freebsd-hpc@freebsd.org, freebsd-performance@freebsd.org From: Jordan Caraballo Subject: netmap, netmap-fwd, and how many M packets-per-second? Message-ID: <596e5c15-1a85-1fdc-ca88-11a0ccd5e125@gmail.com> Date: Thu, 1 Dec 2016 19:55:55 -0400 User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:45.0) Gecko/20100101 Thunderbird/45.3.0 MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8; format=flowed Content-Transfer-Encoding: 8bit X-BeenThere: freebsd-hpc@freebsd.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: "FreeBSD in High Performance Computing environments." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 01 Dec 2016 23:56:00 -0000 Feedback and/or tips and tricks more than welcome. We are trying to process huge amounts of small (64 bytes) pps through a router. So far results have not been as we expected. We have tested FreeBSD 10.3, 11.0, 11.0-STABLE, and 12.0-CURRENT with and without netmap. Based on netmap documentation we were expecting about 5.0M pps; alongside with the routing improvements from the freebsd routing proposal a total of 12.0M. Server Description: Dell PowerEdge R530 with 2 Intel(R) Xeon(R) E5­2695 CPU's, 18 cores per cpu. Equipped with a Chelsio T-580-CR dual port in an 8x slot. BIOS tweaks: Hyperthreading (or Logical Processors) is turned off. Current results are shown below. Additional configurations can be given upon request. Test Environment: 5 clients and 5 servers - 4 Dell C6100 and 2 Dell R420; each one equipped with 10G NICS (4 intel 8259X and 6 with mellanox connectx2). Script that execute the following on each host. #!/usr/local/bin/bash # Iterate through ports and start tests for ((i=1;i<=PORTS;i++)); do PORT=$(($PORT+1)) iperf3 -c 172.16.2.$IP -u -b10000m -i 0 -N -l$PKT -t$TIME -P$STREAMS -p$PORT & #iperf3 -c 172.16.2.$IP -i0 -N -l$PKT -t$TIME -P$STREAMS -p$PORT & done # FreeBSD 10.3 - 4 streams to 80 ports from each client (5) input (Total) output packets errs idrops bytes packets errs bytes colls drops 1.9M 0 1.3M 194M 540k 0 57M 0 0 2.1M 0 1.5M 216M 556k 0 58M 0 0 1.8M 0 1.3M 192M 553k 0 58M 0 0 1.7M 0 1.1M 174M 542k 0 57M 0 0 1.9M 0 1.4M 204M 537k 0 56M 0 0 1.6M 0 1.1M 171M 550k 0 58M 0 0 1.6M 0 1.1M 173M 546k 0 57M 0 0 1.7M 0 1.1M 176M 564k 0 59M 0 0 2.0M 0 1.5M 212M 543k 0 57M 0 0 2.1M 0 1.5M 219M 557k 0 58M 0 0 1.9M 0 1.4M 205M 547k 0 57M 0 0 1.7M 0 1.2M 179M 553k 0 58M 0 0 # FreeBSD 11.0 - 4 streams to 80 ports from each client (5) input (Total) output packets errs idrops bytes packets errs bytes colls drops 3.1M 0 1.8M 326M 1.3M 0 134M 0 0 2.6M 0 1.5M 269M 1.1M 0 116M 0 0 2.7M 0 1.5M 285M 1.2M 0 127M 0 0 2.4M 0 1.3M 257M 1.1M 0 119M 0 0 2.7M 0 1.5M 287M 1.3M 0 134M 0 0 2.5M 0 1.3M 262M 1.2M 0 127M 0 0 2.1M 0 1.1M 224M 1.0M 0 108M 0 0 2.7M 0 1.4M 285M 1.4M 0 143M 0 0 2.6M 0 1.3M 272M 1.3M 0 136M 0 0 2.5M 0 1.4M 265M 1.1M 0 120M 0 0 # FreeBSD 11.0-STABLE - 4 streams to 80 ports from each client (5) input (Total) output packets errs idrops bytes packets errs bytes colls drops 1.9M 0 849k 195M 1.0M 0 107M 0 0 1.9M 0 854k 196M 1.0M 0 106M 0 0 1.9M 0 851k 196M 1.0M 0 107M 0 0 1.9M 0 851k 196M 1.0M 0 107M 0 0 1.9M 0 851k 196M 1.0M 0 107M 0 0 1.9M 0 852k 196M 1.0M 0 107M 0 0 1.9M 0 847k 195M 1.0M 0 107M 0 0 1.9M 0 836k 195M 1.0M 0 107M 0 0 1.9M 0 843k 195M 1.0M 0 107M 0 0 # FreeBSD 12.0-CURRENT - 4 streams to 80 ports from each client (5) input (Total) output packets errs idrops bytes packets errs bytes colls drops 1.1M 259 0 115M 1.1M 0 115M 0 0 1.2M 273 0 124M 1.2M 0 124M 0 0 1.1M 200 0 112M 1.1M 0 112M 0 0 1.2M 290 0 122M 1.2M 0 122M 0 0 1.0M 132 0 107M 1.0M 0 107M 0 0 1.1M 303 0 118M 1.1M 0 118M 0 0 1.1M 278 0 112M 1.1M 0 112M 0 0 1.2M 243 0 122M 1.2M 0 122M 0 0 1.1M 168 0 112M 1.1M 0 112M 0 0 1.1M 161 0 112M 1.1M 0 112M 0 0 # FreeBSD 12.0-CURRENT + Netmap - 4 streams to 80 ports from each client (5) input (Total) output packets errs idrops bytes packets errs bytes colls drops 1.4M 10 0 144M 1.4M 0 144M 0 0 1.5M 6 0 159M 1.5M 0 159M 0 0 1.4M 5 0 144M 1.4M 0 144M 0 0 1.5M 14 0 158M 1.5M 0 158M 0 0 1.4M 5 0 151M 1.4M 0 151M 0 0 1.4M 10 0 152M 1.4M 0 152M 0 0 1.4M 12 0 148M 1.4M 0 148M 0 0 1.5M 9 0 155M 1.5M 0 155M 0 0 1.4M 23 0 151M 1.4M 0 151M 0 0 1.4M 11 0 151M 1.4M 0 151M 0 0 # FreeBSD 12.0-CURRENT + Netmap + Tuning - 4 streams to 80 ports from each client (5) input (Total) output packets errs idrops bytes packets errs bytes colls drops 1.4M 15 0 145M 1.4M 0 145M 0 0 1.5M 18 0 157M 1.5M 0 157M 0 0 1.5M 10 0 156M 1.5M 0 156M 0 0 1.5M 15 0 154M 1.5M 0 154M 0 0 1.4M 13 0 146M 1.4M 0 146M 0 0 1.5M 15 0 156M 1.5M 0 156M 0 0 1.5M 9 0 155M 1.5M 0 155M 0 0 1.5M 13 0 153M 1.5M 0 153M 0 0 1.4M 14 0 145M 1.4M 0 145M 0 0 1.4M 17 0 151M 1.4M 0 151M 0 0 From owner-freebsd-hpc@freebsd.org Fri Dec 2 01:00:08 2016 Return-Path: Delivered-To: freebsd-hpc@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id A6D9DC607E8; Fri, 2 Dec 2016 01:00:08 +0000 (UTC) (envelope-from nparhar@gmail.com) Received: from mail-lf0-x243.google.com (mail-lf0-x243.google.com [IPv6:2a00:1450:4010:c07::243]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 1CF651649; Fri, 2 Dec 2016 01:00:08 +0000 (UTC) (envelope-from nparhar@gmail.com) Received: by mail-lf0-x243.google.com with SMTP id p100so20624877lfg.2; Thu, 01 Dec 2016 17:00:08 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:in-reply-to:references:from:date:message-id:subject:to :cc:content-transfer-encoding; bh=MDxyQiSYQFdmFFoNBZfre7b611ahJL++XZefXqGXG5A=; b=DGkYmLGy5yWW+mBF1+M6g4UB2dfrVCloRXfdWSLws26fVfGJjjW1x2c96I3Ub5nqw9 Hh5YdFzBjZZ7y2u+amwlvlS6KrscvP2KUlfo8+gkTHLv+oYM5pzFbJjyJYTKehsq+5yV reHVsdvlbIkXc/1wFkRp5+aUErX1rzLprwbhrHACStLjm2xqo23uh+wI5L1o9HEUFV+y 3jnDmktUSKQw6a7Jcd9enBcnu3HICebB4ci9xc+cHHAJ1zCjyC3DMm67UQPx/NuQu+ti w5dGdUcIcKDqkL0YPt7qHZy5AWkDQCYnfubNkDLY+9ijFLUzAGITXzT8YFfHyQZ/e4U7 7ZXA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20130820; h=x-gm-message-state:mime-version:in-reply-to:references:from:date :message-id:subject:to:cc:content-transfer-encoding; bh=MDxyQiSYQFdmFFoNBZfre7b611ahJL++XZefXqGXG5A=; b=Io8ravib8V2naRdjb3j+FP99uuthJI68oiBYNWQudV1Tr1lgn7upiiTHUdq1/H6uSX A7tVX1Vz6OLsETW57KMD4BJpWeI1PytocwKkVbhKmCIcgxgxsUtCeuwFbxD2BhnT3OFa a8pQgudAJvKOHcOMPVCfpyFf7qsLxvNJuqwDoPs6n7AWdauA6ylcRC4A2zHETeIvLGo1 Bth2jYLtuQdf4cM+qtci9uTFBR/54G5WBsmvE4RQUBZViPQSnZgTXbNCSNUu5TFdE3v4 qnxXUeivPgEOUf5u9ECjJtahX3HsPdIiEYhduS7mLTVA3jY7OSclF50WLgWA2QsrJTo6 Uq5w== X-Gm-Message-State: AKaTC03t1eSeBex/QyXI5/CIkxIBnsYLzhYwyea7vdz2gyUwzQ761hH/ms/x8Tp3pdN/4OucV49/v1eOmtCmow== X-Received: by 10.25.130.149 with SMTP id e143mr15864736lfd.87.1480640406254; Thu, 01 Dec 2016 17:00:06 -0800 (PST) MIME-Version: 1.0 Received: by 10.25.208.141 with HTTP; Thu, 1 Dec 2016 17:00:05 -0800 (PST) In-Reply-To: <596e5c15-1a85-1fdc-ca88-11a0ccd5e125@gmail.com> References: <596e5c15-1a85-1fdc-ca88-11a0ccd5e125@gmail.com> From: Navdeep Parhar Date: Thu, 1 Dec 2016 17:00:05 -0800 Message-ID: Subject: Re: netmap, netmap-fwd, and how many M packets-per-second? To: Jordan Caraballo Cc: "freebsd-net@freebsd.org" , freebsd-hpc@freebsd.org, freebsd-performance@freebsd.org Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: quoted-printable X-BeenThere: freebsd-hpc@freebsd.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: "FreeBSD in High Performance Computing environments." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 02 Dec 2016 01:00:08 -0000 How have you configured netmap-fwd? If you provide details on how the router or firewall is setup I can try similar experiments here. On Thu, Dec 1, 2016 at 3:55 PM, Jordan Caraballo wrote: > Feedback and/or tips and tricks more than welcome. > > We are trying to process huge amounts of small (64 bytes) pps through a > router. So far results have not been as we expected. We have tested > FreeBSD 10.3, 11.0, 11.0-STABLE, and 12.0-CURRENT with and without > netmap. Based on netmap documentation we were expecting about 5.0M pps; > alongside with the routing improvements from the freebsd routing > proposal a total of 12.0M. > > Server Description: > > Dell PowerEdge R530 with 2 Intel(R) Xeon(R) E5=C2=AD2695 CPU's, 18 cores = per > cpu. Equipped with a Chelsio T-580-CR dual port in an 8x slot. > > BIOS tweaks: > > Hyperthreading (or Logical Processors) is turned off. > > Current results are shown below. Additional configurations can be given > upon request. > > Test Environment: > 5 clients and 5 servers - 4 Dell C6100 and 2 Dell R420; each one > equipped with 10G NICS (4 intel 8259X and 6 with mellanox connectx2). > > Script that execute the following on each host. > > #!/usr/local/bin/bash > # Iterate through ports and start tests > for ((i=3D1;i<=3DPORTS;i++)); do > PORT=3D$(($PORT+1)) > iperf3 -c 172.16.2.$IP -u -b10000m -i 0 -N -l$PKT -t$TIME -P$STREAMS > -p$PORT & > #iperf3 -c 172.16.2.$IP -i0 -N -l$PKT -t$TIME -P$STREAMS -p$PORT & > done > > # FreeBSD 10.3 - 4 streams to 80 ports from each client (5) > > input (Total) output > packets errs idrops bytes packets errs bytes colls drop= s > 1.9M 0 1.3M 194M 540k 0 57M 0 0 > 2.1M 0 1.5M 216M 556k 0 58M 0 0 > 1.8M 0 1.3M 192M 553k 0 58M 0 0 > 1.7M 0 1.1M 174M 542k 0 57M 0 0 > 1.9M 0 1.4M 204M 537k 0 56M 0 0 > 1.6M 0 1.1M 171M 550k 0 58M 0 0 > 1.6M 0 1.1M 173M 546k 0 57M 0 0 > 1.7M 0 1.1M 176M 564k 0 59M 0 0 > 2.0M 0 1.5M 212M 543k 0 57M 0 0 > 2.1M 0 1.5M 219M 557k 0 58M 0 0 > 1.9M 0 1.4M 205M 547k 0 57M 0 0 > 1.7M 0 1.2M 179M 553k 0 58M 0 0 > > # FreeBSD 11.0 - 4 streams to 80 ports from each client (5) > > input (Total) output > packets errs idrops bytes packets errs bytes colls drop= s > 3.1M 0 1.8M 326M 1.3M 0 134M 0 0 > 2.6M 0 1.5M 269M 1.1M 0 116M 0 0 > 2.7M 0 1.5M 285M 1.2M 0 127M 0 0 > 2.4M 0 1.3M 257M 1.1M 0 119M 0 0 > 2.7M 0 1.5M 287M 1.3M 0 134M 0 0 > 2.5M 0 1.3M 262M 1.2M 0 127M 0 0 > 2.1M 0 1.1M 224M 1.0M 0 108M 0 0 > 2.7M 0 1.4M 285M 1.4M 0 143M 0 0 > 2.6M 0 1.3M 272M 1.3M 0 136M 0 0 > 2.5M 0 1.4M 265M 1.1M 0 120M 0 0 > # FreeBSD 11.0-STABLE - 4 streams to 80 ports from each client (5) > > input (Total) output > packets errs idrops bytes packets errs bytes colls drop= s > 1.9M 0 849k 195M 1.0M 0 107M 0 0 > 1.9M 0 854k 196M 1.0M 0 106M 0 0 > 1.9M 0 851k 196M 1.0M 0 107M 0 0 > 1.9M 0 851k 196M 1.0M 0 107M 0 0 > 1.9M 0 851k 196M 1.0M 0 107M 0 0 > 1.9M 0 852k 196M 1.0M 0 107M 0 0 > 1.9M 0 847k 195M 1.0M 0 107M 0 0 > 1.9M 0 836k 195M 1.0M 0 107M 0 0 > 1.9M 0 843k 195M 1.0M 0 107M 0 0 > # FreeBSD 12.0-CURRENT - 4 streams to 80 ports from each client (5) > > input (Total) output > packets errs idrops bytes packets errs bytes colls drop= s > 1.1M 259 0 115M 1.1M 0 115M 0 0 > 1.2M 273 0 124M 1.2M 0 124M 0 0 > 1.1M 200 0 112M 1.1M 0 112M 0 0 > 1.2M 290 0 122M 1.2M 0 122M 0 0 > 1.0M 132 0 107M 1.0M 0 107M 0 0 > 1.1M 303 0 118M 1.1M 0 118M 0 0 > 1.1M 278 0 112M 1.1M 0 112M 0 0 > 1.2M 243 0 122M 1.2M 0 122M 0 0 > 1.1M 168 0 112M 1.1M 0 112M 0 0 > 1.1M 161 0 112M 1.1M 0 112M 0 0 > # FreeBSD 12.0-CURRENT + Netmap - 4 streams to 80 ports from each client = (5) > > input (Total) output > packets errs idrops bytes packets errs bytes colls drop= s > 1.4M 10 0 144M 1.4M 0 144M 0 0 > 1.5M 6 0 159M 1.5M 0 159M 0 0 > 1.4M 5 0 144M 1.4M 0 144M 0 0 > 1.5M 14 0 158M 1.5M 0 158M 0 0 > 1.4M 5 0 151M 1.4M 0 151M 0 0 > 1.4M 10 0 152M 1.4M 0 152M 0 0 > 1.4M 12 0 148M 1.4M 0 148M 0 0 > 1.5M 9 0 155M 1.5M 0 155M 0 0 > 1.4M 23 0 151M 1.4M 0 151M 0 0 > 1.4M 11 0 151M 1.4M 0 151M 0 0 > # FreeBSD 12.0-CURRENT + Netmap + Tuning - 4 streams to 80 ports from > each client (5) > > input (Total) output > packets errs idrops bytes packets errs bytes colls drop= s > 1.4M 15 0 145M 1.4M 0 145M 0 0 > 1.5M 18 0 157M 1.5M 0 157M 0 0 > 1.5M 10 0 156M 1.5M 0 156M 0 0 > 1.5M 15 0 154M 1.5M 0 154M 0 0 > 1.4M 13 0 146M 1.4M 0 146M 0 0 > 1.5M 15 0 156M 1.5M 0 156M 0 0 > 1.5M 9 0 155M 1.5M 0 155M 0 0 > 1.5M 13 0 153M 1.5M 0 153M 0 0 > 1.4M 14 0 145M 1.4M 0 145M 0 0 > 1.4M 17 0 151M 1.4M 0 151M 0 0 > > _______________________________________________ > freebsd-net@freebsd.org mailing list > https://lists.freebsd.org/mailman/listinfo/freebsd-net > To unsubscribe, send any mail to "freebsd-net-unsubscribe@freebsd.org" From owner-freebsd-hpc@freebsd.org Fri Dec 2 08:55:35 2016 Return-Path: Delivered-To: freebsd-hpc@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 98D51C62B60 for ; Fri, 2 Dec 2016 08:55:35 +0000 (UTC) (envelope-from jpa@semihalf.com) Received: from mail-qt0-x22f.google.com (mail-qt0-x22f.google.com [IPv6:2607:f8b0:400d:c0d::22f]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 673D91849 for ; Fri, 2 Dec 2016 08:55:35 +0000 (UTC) (envelope-from jpa@semihalf.com) Received: by mail-qt0-x22f.google.com with SMTP id c47so245866787qtc.2 for ; Fri, 02 Dec 2016 00:55:35 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=semihalf-com.20150623.gappssmtp.com; s=20150623; h=mime-version:in-reply-to:references:from:date:message-id:subject:to :cc; bh=KUl3vjzHYvhF7XhxZQNMQ7k6orHwQ/2vuHIUD5DWSWQ=; b=BYwWzBo4oYO4EVXkSL1j74QYZqRu43/DxfDLxGhG8TbJeWDmT59+xi6tvN8DFL3ZMM nio+T1rJub6bvlaWOqQ5MuzR1WEtM9hMfeGCCuPrl8iIAM1L9L+lfSEnwUQ3NCMeyUfa QLnexIbQv54gOrFJi6Nb82oP8Es4CvQuSJ5wCBHr088QzCe0tZE1SF5ccxlBb4Z1m1ia 7Gkoo4FJK3Al43gFCl+TyG9cbvPD0d44kZts/ZldmSofYIeklnFz+1WPMD4dMbwfIUMH Ah+Y4ubDTjbMMO5n7SlaTSB/WhCD+kmuBqdYb2EHwnm74DKaQWoABbv64lmckF7IOBF7 dV0g== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20130820; h=x-gm-message-state:mime-version:in-reply-to:references:from:date :message-id:subject:to:cc; bh=KUl3vjzHYvhF7XhxZQNMQ7k6orHwQ/2vuHIUD5DWSWQ=; b=CUrCpWsdEM8kkxhg61FM31dW0xZWDHOM5A+YdWrSfZjKc5brkjnkU++WxVnnR6+WI+ a4eH0lccI+TtrDHkzDsuj4TSmCBe/QBby+DgbOZua3+paEVmx7OZZF4rELkYnpjsuc6r uHNdRYuZqCak3aiJetjGO5TDcqjUJtnpYO0+L3MF6CFKliNpP6xrseNlwR+VKKx7ovLq PupzbnryNeZ5+B2cPj99usmepTkkeanXMYDCC/3bw/iuOh2P0N74cV5jY2+AbqPuZ0bM cTXJ9WIPzA6tt/EhGnK/32yyqWXbB7qBdy5Ajs3S88S3qIqyCpZJwx47OFxrSc6LVpkD TwAw== X-Gm-Message-State: AKaTC03ByS2ktRubXNLxdOzFFtE8e6zNe/8cbr+SWU8LSHJ7OEdzxqD8n84Cf/U7lb36irB4qUWxi+do9VzIKA== X-Received: by 10.200.56.20 with SMTP id q20mr37023654qtb.178.1480668934251; Fri, 02 Dec 2016 00:55:34 -0800 (PST) MIME-Version: 1.0 Received: by 10.55.38.15 with HTTP; Fri, 2 Dec 2016 00:55:03 -0800 (PST) In-Reply-To: <596e5c15-1a85-1fdc-ca88-11a0ccd5e125@gmail.com> References: <596e5c15-1a85-1fdc-ca88-11a0ccd5e125@gmail.com> From: Jakub Palider Date: Fri, 2 Dec 2016 09:55:03 +0100 Message-ID: Subject: Re: netmap, netmap-fwd, and how many M packets-per-second? To: Jordan Caraballo Cc: freebsd-net@freebsd.org, freebsd-hpc@freebsd.org, freebsd-performance@freebsd.org Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: quoted-printable X-Content-Filtered-By: Mailman/MimeDel 2.1.23 X-BeenThere: freebsd-hpc@freebsd.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: "FreeBSD in High Performance Computing environments." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 02 Dec 2016 08:55:35 -0000 Hi, I have run some performance tests but not as router, which is your case, so some information might not be relevant: 0. validate your hw stats with what you get from the output 1. Try using FLOWTABLE option in your kernel config - this may kill your machine in -CURRENT, but it's worth checking 2. MTU set to some reasonably small value (~120) shall prevent from consolidating small packets into big frame 3. disable all kernel tracing (INVARIANTS, WITNESS, etc) 4. I read you tuned the system, but make sure there are enough resources 5. Check that context switches are not killing it Regardless on the above you might want profile the system with dtrace and export results to flamegraph, this will give you an idea what going on in the system and possibly locate the cause of the problem. It's definitely worth the effort. Jakub On Fri, Dec 2, 2016 at 12:55 AM, Jordan Caraballo < jordancaraballo87@gmail.com> wrote: > Feedback and/or tips and tricks more than welcome. > > We are trying to process huge amounts of small (64 bytes) pps through a > router. So far results have not been as we expected. We have tested > FreeBSD 10.3, 11.0, 11.0-STABLE, and 12.0-CURRENT with and without > netmap. Based on netmap documentation we were expecting about 5.0M pps; > alongside with the routing improvements from the freebsd routing > proposal a total of 12.0M. > > Server Description: > > Dell PowerEdge R530 with 2 Intel(R) Xeon(R) E5=C2=AD2695 CPU's, 18 cores = per > cpu. Equipped with a Chelsio T-580-CR dual port in an 8x slot. > > BIOS tweaks: > > Hyperthreading (or Logical Processors) is turned off. > > Current results are shown below. Additional configurations can be given > upon request. > > Test Environment: > 5 clients and 5 servers - 4 Dell C6100 and 2 Dell R420; each one > equipped with 10G NICS (4 intel 8259X and 6 with mellanox connectx2). > > Script that execute the following on each host. > > #!/usr/local/bin/bash > # Iterate through ports and start tests > for ((i=3D1;i<=3DPORTS;i++)); do > PORT=3D$(($PORT+1)) > iperf3 -c 172.16.2.$IP -u -b10000m -i 0 -N -l$PKT -t$TIME -P$STREAMS > -p$PORT & > #iperf3 -c 172.16.2.$IP -i0 -N -l$PKT -t$TIME -P$STREAMS -p$PORT & > done > > # FreeBSD 10.3 - 4 streams to 80 ports from each client (5) > > input (Total) output > packets errs idrops bytes packets errs bytes colls drop= s > 1.9M 0 1.3M 194M 540k 0 57M 0 0 > 2.1M 0 1.5M 216M 556k 0 58M 0 0 > 1.8M 0 1.3M 192M 553k 0 58M 0 0 > 1.7M 0 1.1M 174M 542k 0 57M 0 0 > 1.9M 0 1.4M 204M 537k 0 56M 0 0 > 1.6M 0 1.1M 171M 550k 0 58M 0 0 > 1.6M 0 1.1M 173M 546k 0 57M 0 0 > 1.7M 0 1.1M 176M 564k 0 59M 0 0 > 2.0M 0 1.5M 212M 543k 0 57M 0 0 > 2.1M 0 1.5M 219M 557k 0 58M 0 0 > 1.9M 0 1.4M 205M 547k 0 57M 0 0 > 1.7M 0 1.2M 179M 553k 0 58M 0 0 > > # FreeBSD 11.0 - 4 streams to 80 ports from each client (5) > > input (Total) output > packets errs idrops bytes packets errs bytes colls drop= s > 3.1M 0 1.8M 326M 1.3M 0 134M 0 0 > 2.6M 0 1.5M 269M 1.1M 0 116M 0 0 > 2.7M 0 1.5M 285M 1.2M 0 127M 0 0 > 2.4M 0 1.3M 257M 1.1M 0 119M 0 0 > 2.7M 0 1.5M 287M 1.3M 0 134M 0 0 > 2.5M 0 1.3M 262M 1.2M 0 127M 0 0 > 2.1M 0 1.1M 224M 1.0M 0 108M 0 0 > 2.7M 0 1.4M 285M 1.4M 0 143M 0 0 > 2.6M 0 1.3M 272M 1.3M 0 136M 0 0 > 2.5M 0 1.4M 265M 1.1M 0 120M 0 0 > # FreeBSD 11.0-STABLE - 4 streams to 80 ports from each client (5) > > input (Total) output > packets errs idrops bytes packets errs bytes colls drop= s > 1.9M 0 849k 195M 1.0M 0 107M 0 0 > 1.9M 0 854k 196M 1.0M 0 106M 0 0 > 1.9M 0 851k 196M 1.0M 0 107M 0 0 > 1.9M 0 851k 196M 1.0M 0 107M 0 0 > 1.9M 0 851k 196M 1.0M 0 107M 0 0 > 1.9M 0 852k 196M 1.0M 0 107M 0 0 > 1.9M 0 847k 195M 1.0M 0 107M 0 0 > 1.9M 0 836k 195M 1.0M 0 107M 0 0 > 1.9M 0 843k 195M 1.0M 0 107M 0 0 > # FreeBSD 12.0-CURRENT - 4 streams to 80 ports from each client (5) > > input (Total) output > packets errs idrops bytes packets errs bytes colls drop= s > 1.1M 259 0 115M 1.1M 0 115M 0 0 > 1.2M 273 0 124M 1.2M 0 124M 0 0 > 1.1M 200 0 112M 1.1M 0 112M 0 0 > 1.2M 290 0 122M 1.2M 0 122M 0 0 > 1.0M 132 0 107M 1.0M 0 107M 0 0 > 1.1M 303 0 118M 1.1M 0 118M 0 0 > 1.1M 278 0 112M 1.1M 0 112M 0 0 > 1.2M 243 0 122M 1.2M 0 122M 0 0 > 1.1M 168 0 112M 1.1M 0 112M 0 0 > 1.1M 161 0 112M 1.1M 0 112M 0 0 > # FreeBSD 12.0-CURRENT + Netmap - 4 streams to 80 ports from each client > (5) > > input (Total) output > packets errs idrops bytes packets errs bytes colls drop= s > 1.4M 10 0 144M 1.4M 0 144M 0 0 > 1.5M 6 0 159M 1.5M 0 159M 0 0 > 1.4M 5 0 144M 1.4M 0 144M 0 0 > 1.5M 14 0 158M 1.5M 0 158M 0 0 > 1.4M 5 0 151M 1.4M 0 151M 0 0 > 1.4M 10 0 152M 1.4M 0 152M 0 0 > 1.4M 12 0 148M 1.4M 0 148M 0 0 > 1.5M 9 0 155M 1.5M 0 155M 0 0 > 1.4M 23 0 151M 1.4M 0 151M 0 0 > 1.4M 11 0 151M 1.4M 0 151M 0 0 > # FreeBSD 12.0-CURRENT + Netmap + Tuning - 4 streams to 80 ports from > each client (5) > > input (Total) output > packets errs idrops bytes packets errs bytes colls drop= s > 1.4M 15 0 145M 1.4M 0 145M 0 0 > 1.5M 18 0 157M 1.5M 0 157M 0 0 > 1.5M 10 0 156M 1.5M 0 156M 0 0 > 1.5M 15 0 154M 1.5M 0 154M 0 0 > 1.4M 13 0 146M 1.4M 0 146M 0 0 > 1.5M 15 0 156M 1.5M 0 156M 0 0 > 1.5M 9 0 155M 1.5M 0 155M 0 0 > 1.5M 13 0 153M 1.5M 0 153M 0 0 > 1.4M 14 0 145M 1.4M 0 145M 0 0 > 1.4M 17 0 151M 1.4M 0 151M 0 0 > > _______________________________________________ > freebsd-net@freebsd.org mailing list > https://lists.freebsd.org/mailman/listinfo/freebsd-net > To unsubscribe, send any mail to "freebsd-net-unsubscribe@freebsd.org" From owner-freebsd-hpc@freebsd.org Fri Dec 2 12:16:14 2016 Return-Path: Delivered-To: freebsd-hpc@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 168F3C6237B; Fri, 2 Dec 2016 12:16:14 +0000 (UTC) (envelope-from v.maffione@gmail.com) Received: from mail-oi0-x241.google.com (mail-oi0-x241.google.com [IPv6:2607:f8b0:4003:c06::241]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id CE33A15BD; Fri, 2 Dec 2016 12:16:13 +0000 (UTC) (envelope-from v.maffione@gmail.com) Received: by mail-oi0-x241.google.com with SMTP id f201so27070665oib.0; Fri, 02 Dec 2016 04:16:13 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:in-reply-to:references:from:date:message-id:subject:to :cc; bh=JS3Jt6b7QflLiHAOqQRJjf+Ov20cCEY4J0XCR7cTdYI=; b=sgS9gIj5owIAjuYj6XtvoEpZQ9dKYpQS7kUngpl18uzSe2TWI1K1np3oWgyp6yzzCf aFGpGDJRp/cxu1kQDeI9kG1u9wmrFoTDUzBt0ZHJoD2JZhQvXEiK+Ca4kd0GFs3F+cPi 0EWIISyvHmJJD8YkHy4dU5F9paImRfulTT1ghBd4QhwPyEyE94XmJIbpWcdVaFTpCW15 TvpOfcwG1/3K2j5oXiFgyPPRZ8Lg03HCSGTjhh1T9TASythhuQudHLDnMpQ897iYiKgK 2/w/5u5RKdd9jUJgprifck1yHVhxyX45h0Pwej5Z255y4JaOtmPPNxkerKWGi/zdz2MU YmuA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20130820; h=x-gm-message-state:mime-version:in-reply-to:references:from:date :message-id:subject:to:cc; bh=JS3Jt6b7QflLiHAOqQRJjf+Ov20cCEY4J0XCR7cTdYI=; b=BRXagHNKQWSmWMpR84dClwevX06ylbjs5qSZNUjZKSG2BmSPeK8jlzb3meL6BTwLUi XUxmRd1hqrtK8kP8OOftpGBqiTkH69+QPIbU9rm/Rv7QNoGoSwgtKrs+y5Z2+ynTFyFK zAn8mW3myTXcrdqvdSsZxJW27LPaw2eHtz5IrnO9eFcToDFNVTgZ+kQibxfzlTyVi5ki t/rxAdQEb7lYdCASJqdcZ20ID4POWON4ThhCCuHsdFr1iFbLOgElRFAzBga/WQGhzb8m G0WoouPjqXecl8JlbY2CehHlDCjvsVqvx1uwWNqVZDzcHlWSi0sOH3dUwfDp8fJJpxt9 2gCA== X-Gm-Message-State: AKaTC00IPK3QDtAX1wAJg+vOe1K/RYiyEGj+OTLkTcl2ASitBGEKIYrjLu4o1FBb/zp2eV09nnoFd05iUF5AuA== X-Received: by 10.202.235.78 with SMTP id j75mr22518632oih.74.1480680973110; Fri, 02 Dec 2016 04:16:13 -0800 (PST) MIME-Version: 1.0 Received: by 10.157.45.200 with HTTP; Fri, 2 Dec 2016 04:16:12 -0800 (PST) In-Reply-To: <596e5c15-1a85-1fdc-ca88-11a0ccd5e125@gmail.com> References: <596e5c15-1a85-1fdc-ca88-11a0ccd5e125@gmail.com> From: Vincenzo Maffione Date: Fri, 2 Dec 2016 13:16:12 +0100 Message-ID: Subject: Re: netmap, netmap-fwd, and how many M packets-per-second? To: Jordan Caraballo Cc: "freebsd-net@freebsd.org" , freebsd-hpc@freebsd.org, freebsd-performance@freebsd.org Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: quoted-printable X-Content-Filtered-By: Mailman/MimeDel 2.1.23 X-BeenThere: freebsd-hpc@freebsd.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: "FreeBSD in High Performance Computing environments." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 02 Dec 2016 12:16:14 -0000 Hi, If I'm getting it right, you are doing a test with 10 machines directly connected on the same LAN, using iperf to test the performance of the TCP/IP stack with your 10Gbit cards. But as far as I can see, netmap is not involved in your setup, you are not using it, aren't you? IOW , I don't understand what does that "+ Netmap" mean you our test description banner. Cheers, Vincenzo 2016-12-02 0:55 GMT+01:00 Jordan Caraballo : > Feedback and/or tips and tricks more than welcome. > > We are trying to process huge amounts of small (64 bytes) pps through a > router. So far results have not been as we expected. We have tested > FreeBSD 10.3, 11.0, 11.0-STABLE, and 12.0-CURRENT with and without > netmap. Based on netmap documentation we were expecting about 5.0M pps; > alongside with the routing improvements from the freebsd routing > proposal a total of 12.0M. > > Server Description: > > Dell PowerEdge R530 with 2 Intel(R) Xeon(R) E5=C2=AD2695 CPU's, 18 cores = per > cpu. Equipped with a Chelsio T-580-CR dual port in an 8x slot. > > BIOS tweaks: > > Hyperthreading (or Logical Processors) is turned off. > > Current results are shown below. Additional configurations can be given > upon request. > > Test Environment: > 5 clients and 5 servers - 4 Dell C6100 and 2 Dell R420; each one > equipped with 10G NICS (4 intel 8259X and 6 with mellanox connectx2). > > Script that execute the following on each host. > > #!/usr/local/bin/bash > # Iterate through ports and start tests > for ((i=3D1;i<=3DPORTS;i++)); do > PORT=3D$(($PORT+1)) > iperf3 -c 172.16.2.$IP -u -b10000m -i 0 -N -l$PKT -t$TIME -P$STREAMS > -p$PORT & > #iperf3 -c 172.16.2.$IP -i0 -N -l$PKT -t$TIME -P$STREAMS -p$PORT & > done > > # FreeBSD 10.3 - 4 streams to 80 ports from each client (5) > > input (Total) output > packets errs idrops bytes packets errs bytes colls drop= s > 1.9M 0 1.3M 194M 540k 0 57M 0 0 > 2.1M 0 1.5M 216M 556k 0 58M 0 0 > 1.8M 0 1.3M 192M 553k 0 58M 0 0 > 1.7M 0 1.1M 174M 542k 0 57M 0 0 > 1.9M 0 1.4M 204M 537k 0 56M 0 0 > 1.6M 0 1.1M 171M 550k 0 58M 0 0 > 1.6M 0 1.1M 173M 546k 0 57M 0 0 > 1.7M 0 1.1M 176M 564k 0 59M 0 0 > 2.0M 0 1.5M 212M 543k 0 57M 0 0 > 2.1M 0 1.5M 219M 557k 0 58M 0 0 > 1.9M 0 1.4M 205M 547k 0 57M 0 0 > 1.7M 0 1.2M 179M 553k 0 58M 0 0 > > # FreeBSD 11.0 - 4 streams to 80 ports from each client (5) > > input (Total) output > packets errs idrops bytes packets errs bytes colls drop= s > 3.1M 0 1.8M 326M 1.3M 0 134M 0 0 > 2.6M 0 1.5M 269M 1.1M 0 116M 0 0 > 2.7M 0 1.5M 285M 1.2M 0 127M 0 0 > 2.4M 0 1.3M 257M 1.1M 0 119M 0 0 > 2.7M 0 1.5M 287M 1.3M 0 134M 0 0 > 2.5M 0 1.3M 262M 1.2M 0 127M 0 0 > 2.1M 0 1.1M 224M 1.0M 0 108M 0 0 > 2.7M 0 1.4M 285M 1.4M 0 143M 0 0 > 2.6M 0 1.3M 272M 1.3M 0 136M 0 0 > 2.5M 0 1.4M 265M 1.1M 0 120M 0 0 > # FreeBSD 11.0-STABLE - 4 streams to 80 ports from each client (5) > > input (Total) output > packets errs idrops bytes packets errs bytes colls drop= s > 1.9M 0 849k 195M 1.0M 0 107M 0 0 > 1.9M 0 854k 196M 1.0M 0 106M 0 0 > 1.9M 0 851k 196M 1.0M 0 107M 0 0 > 1.9M 0 851k 196M 1.0M 0 107M 0 0 > 1.9M 0 851k 196M 1.0M 0 107M 0 0 > 1.9M 0 852k 196M 1.0M 0 107M 0 0 > 1.9M 0 847k 195M 1.0M 0 107M 0 0 > 1.9M 0 836k 195M 1.0M 0 107M 0 0 > 1.9M 0 843k 195M 1.0M 0 107M 0 0 > # FreeBSD 12.0-CURRENT - 4 streams to 80 ports from each client (5) > > input (Total) output > packets errs idrops bytes packets errs bytes colls drop= s > 1.1M 259 0 115M 1.1M 0 115M 0 0 > 1.2M 273 0 124M 1.2M 0 124M 0 0 > 1.1M 200 0 112M 1.1M 0 112M 0 0 > 1.2M 290 0 122M 1.2M 0 122M 0 0 > 1.0M 132 0 107M 1.0M 0 107M 0 0 > 1.1M 303 0 118M 1.1M 0 118M 0 0 > 1.1M 278 0 112M 1.1M 0 112M 0 0 > 1.2M 243 0 122M 1.2M 0 122M 0 0 > 1.1M 168 0 112M 1.1M 0 112M 0 0 > 1.1M 161 0 112M 1.1M 0 112M 0 0 > # FreeBSD 12.0-CURRENT + Netmap - 4 streams to 80 ports from each client > (5) > > input (Total) output > packets errs idrops bytes packets errs bytes colls drop= s > 1.4M 10 0 144M 1.4M 0 144M 0 0 > 1.5M 6 0 159M 1.5M 0 159M 0 0 > 1.4M 5 0 144M 1.4M 0 144M 0 0 > 1.5M 14 0 158M 1.5M 0 158M 0 0 > 1.4M 5 0 151M 1.4M 0 151M 0 0 > 1.4M 10 0 152M 1.4M 0 152M 0 0 > 1.4M 12 0 148M 1.4M 0 148M 0 0 > 1.5M 9 0 155M 1.5M 0 155M 0 0 > 1.4M 23 0 151M 1.4M 0 151M 0 0 > 1.4M 11 0 151M 1.4M 0 151M 0 0 > # FreeBSD 12.0-CURRENT + Netmap + Tuning - 4 streams to 80 ports from > each client (5) > > input (Total) output > packets errs idrops bytes packets errs bytes colls drop= s > 1.4M 15 0 145M 1.4M 0 145M 0 0 > 1.5M 18 0 157M 1.5M 0 157M 0 0 > 1.5M 10 0 156M 1.5M 0 156M 0 0 > 1.5M 15 0 154M 1.5M 0 154M 0 0 > 1.4M 13 0 146M 1.4M 0 146M 0 0 > 1.5M 15 0 156M 1.5M 0 156M 0 0 > 1.5M 9 0 155M 1.5M 0 155M 0 0 > 1.5M 13 0 153M 1.5M 0 153M 0 0 > 1.4M 14 0 145M 1.4M 0 145M 0 0 > 1.4M 17 0 151M 1.4M 0 151M 0 0 > > _______________________________________________ > freebsd-net@freebsd.org mailing list > https://lists.freebsd.org/mailman/listinfo/freebsd-net > To unsubscribe, send any mail to "freebsd-net-unsubscribe@freebsd.org" --=20 Vincenzo Maffione