Date: Sun, 24 Jan 2016 05:56:21 -0200 From: "Marcus Cenzatti" <cenzatti@hush.com> To: "Luigi Rizzo" <rizzo@iet.unipi.it> Cc: freebsd-net@freebsd.org, "Navdeep Parhar" <nparhar@gmail.com> Subject: Re: solved: Re: Chelsio T520-SO-CR low performance (netmap tested) for RX Message-ID: <20160124075621.7F4FEA0126@smtp.hushmail.com> In-Reply-To: <CA%2BhQ2%2Bhk8JRD8JoQcyn%2BiMEPWtS9YMZaUen=AyBwreEfh_9T1A@mail.gmail.com> References: <CA%2BhQ2%2Bg7_haaXLFjMuG00ANsUkFdyGzFQyjT4NYVBmPY-vECBg@mail.gmail.com> <20160124042830.3D674A0128@smtp.hushmail.com> <CA%2BhQ2%2BhxOZkGJdRSrmxSqHforLbMWBVQcayrNFNLLkU803hmjA@mail.gmail.com> <20160124070056.4EC5CA0126@smtp.hushmail.com> <CA%2BhQ2%2Bhk8JRD8JoQcyn%2BiMEPWtS9YMZaUen=AyBwreEfh_9T1A@mail.gmail.com>
next in thread | previous in thread | raw e-mail | index | archive | help
On 1/24/2016 at 5:17 AM, "Luigi Rizzo" <rizzo@iet.unipi.it> wrote: > >On Sat, Jan 23, 2016 at 11:00 PM, Marcus Cenzatti ><cenzatti@hush.com> wrote: >> >> >> On 1/24/2016 at 3:33 AM, "Luigi Rizzo" <rizzo@iet.unipi.it> >wrote: >... > >> ok here it is >> >> this lowered pps rate to 9.4Mpps on chelsio (we had 11Mpps with >defaul len) and lowered rates to 14Mpps on sender (we had 14.8Mpps >before). > >see the other email from navdeep, he suggests to try -l 62 >because the chelsio saves the packet with a 2-byte offset >into the buffer (it puzzles me a bit that it does so also >in netmap mode though). > >cheers >luigi yes, I saw, i've run both with 62 bytes and 64 bytes len very different results from the previous one and among both pkt sizes but could not go beyond 11Mpps results: -l 62 = TX 14.8Mpps RX 11Mpps -l 66 = TX 12.4Mpps RX 9.4Mpps here is the transcript: intel# intel# netmap-master/examples/pkt-gen-b -l 62 -i ix0 -f tx -S 00:07:e9:44:d2:ba -D 00:07:43:33:8d:c1 430.877073 main [1930] interface is ix0 430.877114 main [2050] running on 1 cpus (have 8) 430.877132 extract_ip_range [367] range is 10.0.0.1:0 to 10.0.0.1:0 430.877141 extract_ip_range [367] range is 10.1.0.1:0 to 10.1.0.1:0 430.982448 main [2148] mapped 334980KB at 0x801800000 Sending on netmap:ix0: 8 queues, 1 threads and 1 cpus. 10.0.0.1 -> 10.1.0.1 (00:07:e9:44:d2:ba -> 00:07:43:33:8d:c1) 430.982522 main [2233] Sending 512 packets every 0.000000000 s 430.982527 main [2235] Wait 2 secs for phy reset 433.006279 main [2237] Ready... 433.006526 sender_body [1211] start, fd 3 main_fd 3 433.046003 sender_body [1293] drop copy 434.007048 main_thread [1720] 13.780 Mpps (13.787 Mpkts 6.839 Gbps in 1000544 usec) 433.79 avg_batch 0 min_space 435.007831 main_thread [1720] 14.289 Mpps (14.300 Mpkts 7.093 Gbps in 1000784 usec) 430.53 avg_batch 99999 min_space 436.008835 main_thread [1720] 14.318 Mpps (14.332 Mpkts 7.109 Gbps in 1001004 usec) 421.25 avg_batch 99999 min_space 437.009848 main_thread [1720] 14.303 Mpps (14.318 Mpkts 7.102 Gbps in 1001013 usec) 419.91 avg_batch 99999 min_space 438.011331 main_thread [1720] 14.291 Mpps (14.312 Mpkts 7.099 Gbps in 1001483 usec) 422.53 avg_batch 99999 min_space 439.012325 main_thread [1720] 14.309 Mpps (14.323 Mpkts 7.104 Gbps in 1000995 usec) 439.39 avg_batch 99999 min_space 440.013325 main_thread [1720] 14.300 Mpps (14.314 Mpkts 7.100 Gbps in 1000999 usec) 407.46 avg_batch 99999 min_space 441.014321 main_thread [1720] 14.267 Mpps (14.281 Mpkts 7.083 Gbps in 1000997 usec) 457.50 avg_batch 99999 min_space 442.014899 main_thread [1720] 14.316 Mpps (14.324 Mpkts 7.105 Gbps in 1000578 usec) 440.57 avg_batch 99999 min_space 443.016322 main_thread [1720] 14.274 Mpps (14.294 Mpkts 7.090 Gbps in 1001422 usec) 448.94 avg_batch 99999 min_space 444.022363 main_thread [1720] 14.329 Mpps (14.416 Mpkts 7.150 Gbps in 1006011 usec) 415.10 avg_batch 99999 min_space 445.023338 main_thread [1720] 14.315 Mpps (14.330 Mpkts 7.108 Gbps in 1001006 usec) 413.37 avg_batch 99999 min_space 446.024321 main_thread [1720] 14.286 Mpps (14.300 Mpkts 7.093 Gbps in 1000982 usec) 426.52 avg_batch 99999 min_space 447.025327 main_thread [1720] 14.343 Mpps (14.358 Mpkts 7.121 Gbps in 1001006 usec) 426.19 avg_batch 99999 min_space 448.026323 main_thread [1720] 14.282 Mpps (14.296 Mpkts 7.091 Gbps in 1000996 usec) 416.17 avg_batch 99999 min_space 449.027830 main_thread [1720] 14.278 Mpps (14.299 Mpkts 7.092 Gbps in 1001508 usec) 448.76 avg_batch 99999 min_space 450.029325 main_thread [1720] 14.274 Mpps (14.295 Mpkts 7.090 Gbps in 1001494 usec) 425.45 avg_batch 99999 min_space 451.029824 main_thread [1720] 14.248 Mpps (14.255 Mpkts 7.071 Gbps in 1000500 usec) 452.37 avg_batch 99999 min_space 452.040823 main_thread [1720] 14.267 Mpps (14.423 Mpkts 7.154 Gbps in 1010998 usec) 439.52 avg_batch 99999 min_space 453.042317 main_thread [1720] 14.256 Mpps (14.278 Mpkts 7.082 Gbps in 1001494 usec) 437.54 avg_batch 99999 min_space 454.043326 main_thread [1720] 14.329 Mpps (14.343 Mpkts 7.114 Gbps in 1001009 usec) 424.55 avg_batch 99999 min_space 455.043822 main_thread [1720] 14.319 Mpps (14.326 Mpkts 7.106 Gbps in 1000496 usec) 423.84 avg_batch 99999 min_space 456.045320 main_thread [1720] 14.311 Mpps (14.333 Mpkts 7.109 Gbps in 1001498 usec) 416.26 avg_batch 99999 min_space 457.046326 main_thread [1720] 14.319 Mpps (14.334 Mpkts 7.110 Gbps in 1001007 usec) 436.04 avg_batch 99999 min_space 458.047335 main_thread [1720] 14.372 Mpps (14.386 Mpkts 7.136 Gbps in 1001002 usec) 410.49 avg_batch 99999 min_space 459.048318 main_thread [1720] 14.244 Mpps (14.258 Mpkts 7.072 Gbps in 1000990 usec) 417.40 avg_batch 99999 min_space 460.049318 main_thread [1720] 14.329 Mpps (14.343 Mpkts 7.114 Gbps in 1000999 usec) 420.67 avg_batch 99999 min_space 461.050317 main_thread [1720] 14.304 Mpps (14.318 Mpkts 7.102 Gbps in 1000999 usec) 413.63 avg_batch 99999 min_space 462.051314 main_thread [1720] 14.232 Mpps (14.247 Mpkts 7.066 Gbps in 1000997 usec) 472.71 avg_batch 99999 min_space 463.052317 main_thread [1720] 14.269 Mpps (14.284 Mpkts 7.085 Gbps in 1001003 usec) 436.25 avg_batch 99999 min_space 464.052817 main_thread [1720] 14.364 Mpps (14.371 Mpkts 7.128 Gbps in 1000500 usec) 400.26 avg_batch 99999 min_space 465.054270 main_thread [1720] 14.261 Mpps (14.282 Mpkts 7.084 Gbps in 1001453 usec) 450.93 avg_batch 99999 min_space 466.077837 main_thread [1720] 14.313 Mpps (14.651 Mpkts 7.267 Gbps in 1023567 usec) 436.45 avg_batch 99999 min_space 467.078822 main_thread [1720] 14.281 Mpps (14.295 Mpkts 7.090 Gbps in 1000986 usec) 434.86 avg_batch 99999 min_space ^C467.836880 sigint_h [404] received control-C on thread 0x801406800 467.837095 sender_body [1326] flush tail 737 head 1762 on thread 0x801406800 467.837239 sender_body [1334] pending tx tail 1522 head 2035 on ring 0 467.837293 sender_body [1334] pending tx tail 1175 head 1225 on ring 1 467.837336 sender_body [1334] pending tx tail 1874 head 1915 on ring 2 467.837384 sender_body [1334] pending tx tail 1311 head 1325 on ring 4 468.101067 main_thread [1720] 10.636 Mpps (10.873 Mpkts 5.393 Gbps in 1022244 usec) 410.74 avg_batch 99999 min_space Sent 497478675 packets 30843677850 bytes 1159848 events 62 bytes each in 34.83 seconds. Speed: 14.283 Mpps Bandwidth: 7.084 Gbps (raw 9.827 Gbps). Average batch: 428.92 pkts intel# netmap-master/examples/pkt-gen-b -l 66 -i ix0 -f tx -S 00:07:e9:44:d2:ba -D 00:07:43:33:8d:c1 476.421587 main [1930] interface is ix0 476.421627 main [2050] running on 1 cpus (have 8) 476.421646 extract_ip_range [367] range is 10.0.0.1:0 to 10.0.0.1:0 476.421655 extract_ip_range [367] range is 10.1.0.1:0 to 10.1.0.1:0 476.527263 main [2148] mapped 334980KB at 0x801800000 Sending on netmap:ix0: 8 queues, 1 threads and 1 cpus. 10.0.0.1 -> 10.1.0.1 (00:07:e9:44:d2:ba -> 00:07:43:33:8d:c1) 476.527339 main [2233] Sending 512 packets every 0.000000000 s 476.527344 main [2235] Wait 2 secs for phy reset 478.535360 main [2237] Ready... 478.535595 sender_body [1211] start, fd 3 main_fd 3 478.580779 sender_body [1293] drop copy 479.569816 main_thread [1720] 12.023 Mpps (12.435 Mpkts 6.566 Gbps in 1034249 usec) 142.97 avg_batch 0 min_space 480.571316 main_thread [1720] 12.431 Mpps (12.450 Mpkts 6.573 Gbps in 1001499 usec) 138.29 avg_batch 99999 min_space 481.572313 main_thread [1720] 12.483 Mpps (12.495 Mpkts 6.597 Gbps in 1000997 usec) 158.82 avg_batch 99999 min_space 482.573309 main_thread [1720] 12.478 Mpps (12.491 Mpkts 6.595 Gbps in 1000996 usec) 150.58 avg_batch 99999 min_space 483.573954 main_thread [1720] 12.478 Mpps (12.486 Mpkts 6.593 Gbps in 1000645 usec) 154.52 avg_batch 99999 min_space 484.575316 main_thread [1720] 12.383 Mpps (12.400 Mpkts 6.547 Gbps in 1001362 usec) 159.04 avg_batch 99999 min_space 485.576311 main_thread [1720] 12.479 Mpps (12.491 Mpkts 6.595 Gbps in 1000996 usec) 156.58 avg_batch 99999 min_space 486.577312 main_thread [1720] 12.473 Mpps (12.485 Mpkts 6.592 Gbps in 1001000 usec) 166.46 avg_batch 99999 min_space ^C486.932236 sigint_h [404] received control-C on thread 0x801406800 486.932301 sender_body [1326] flush tail 1400 head 1400 on thread 0x801406800 486.932371 sender_body [1334] pending tx tail 1989 head 1669 on ring 0 486.932445 sender_body [1334] pending tx tail 37 head 1669 on ring 0 486.932486 sender_body [1334] pending tx tail 141 head 1669 on ring 0 486.932517 sender_body [1334] pending tx tail 197 head 1669 on ring 0 486.932547 sender_body [1334] pending tx tail 246 head 1669 on ring 0 487.579383 main_thread [1720] 4.423 Mpps (4.432 Mpkts 2.340 Gbps in 1002071 usec) 163.45 avg_batch 99999 min_space Sent 104165529 packets 6874924914 bytes 679304 events 66 bytes each in 8.40 seconds. Speed: 12.404 Mpps Bandwidth: 6.549 Gbps (raw 8.931 Gbps). Average batch: 153.34 pkts chelsio# ./pkt-gen-b -i ncxl0 -f rx 506.318671 main [1930] interface is ncxl0 506.318722 main [2050] running on 1 cpus (have 4) 506.318859 extract_ip_range [367] range is 10.0.0.1:0 to 10.0.0.1:0 506.318878 extract_ip_range [367] range is 10.1.0.1:0 to 10.1.0.1:0 506.565853 main [2148] mapped 334980KB at 0x801800000 Receiving from netmap:ncxl0: 2 queues, 1 threads and 1 cpus. 506.565933 main [2235] Wait 2 secs for phy reset 508.567111 main [2237] Ready... 545.346710 receiver_body [1419] waiting for initial packets, poll returns 0 0 545.977706 main_thread [1720] 365.393 Kpps (365.756 Kpkts 181.415 Mbps in 1000994 usec) 199.76 avg_batch 0 min_space 546.978707 main_thread [1720] 11.172 Mpps (11.183 Mpkts 5.547 Gbps in 1001001 usec) 187.95 avg_batch 0 min_space 547.979208 main_thread [1720] 11.173 Mpps (11.179 Mpkts 5.545 Gbps in 1000501 usec) 188.79 avg_batch 0 min_space 548.980704 main_thread [1720] 11.163 Mpps (11.180 Mpkts 5.545 Gbps in 1001496 usec) 196.53 avg_batch 0 min_space 549.981703 main_thread [1720] 11.174 Mpps (11.185 Mpkts 5.548 Gbps in 1000999 usec) 189.79 avg_batch 0 min_space 550.990747 main_thread [1720] 11.173 Mpps (11.274 Mpkts 5.592 Gbps in 1009044 usec) 198.65 avg_batch 0 min_space 551.994476 main_thread [1720] 11.173 Mpps (11.214 Mpkts 5.562 Gbps in 1003728 usec) 201.30 avg_batch 0 min_space 552.996708 main_thread [1720] 11.171 Mpps (11.196 Mpkts 5.553 Gbps in 1002233 usec) 197.36 avg_batch 0 min_space 553.997566 main_thread [1720] 11.167 Mpps (11.176 Mpkts 5.543 Gbps in 1000858 usec) 187.29 avg_batch 0 min_space 555.000213 main_thread [1720] 11.172 Mpps (11.202 Mpkts 5.556 Gbps in 1002646 usec) 184.47 avg_batch 0 min_space 556.001209 main_thread [1720] 11.173 Mpps (11.184 Mpkts 5.547 Gbps in 1000997 usec) 188.14 avg_batch 0 min_space 557.002717 main_thread [1720] 11.170 Mpps (11.187 Mpkts 5.549 Gbps in 1001508 usec) 187.32 avg_batch 0 min_space 558.003201 main_thread [1720] 11.173 Mpps (11.178 Mpkts 5.544 Gbps in 1000484 usec) 186.84 avg_batch 0 min_space 559.004558 main_thread [1720] 11.167 Mpps (11.182 Mpkts 5.546 Gbps in 1001356 usec) 189.47 avg_batch 0 min_space 560.005218 main_thread [1720] 11.174 Mpps (11.181 Mpkts 5.546 Gbps in 1000661 usec) 199.56 avg_batch 0 min_space 561.006704 main_thread [1720] 11.173 Mpps (11.189 Mpkts 5.550 Gbps in 1001486 usec) 184.32 avg_batch 0 min_space 562.007715 main_thread [1720] 11.173 Mpps (11.184 Mpkts 5.547 Gbps in 1001010 usec) 192.40 avg_batch 0 min_space 563.008204 main_thread [1720] 11.171 Mpps (11.177 Mpkts 5.544 Gbps in 1000490 usec) 195.11 avg_batch 0 min_space 564.009485 main_thread [1720] 11.168 Mpps (11.182 Mpkts 5.546 Gbps in 1001281 usec) 186.32 avg_batch 0 min_space 565.010213 main_thread [1720] 11.171 Mpps (11.179 Mpkts 5.545 Gbps in 1000728 usec) 185.74 avg_batch 0 min_space 566.014238 main_thread [1720] 11.173 Mpps (11.218 Mpkts 5.564 Gbps in 1004025 usec) 197.98 avg_batch 0 min_space 567.016711 main_thread [1720] 11.173 Mpps (11.200 Mpkts 5.555 Gbps in 1002473 usec) 185.86 avg_batch 0 min_space 568.017213 main_thread [1720] 11.173 Mpps (11.179 Mpkts 5.545 Gbps in 1000502 usec) 189.72 avg_batch 0 min_space 569.027995 main_thread [1720] 11.170 Mpps (11.291 Mpkts 5.600 Gbps in 1010782 usec) 195.09 avg_batch 0 min_space 570.028205 main_thread [1720] 11.173 Mpps (11.175 Mpkts 5.543 Gbps in 1000210 usec) 185.56 avg_batch 0 min_space 571.029709 main_thread [1720] 11.173 Mpps (11.190 Mpkts 5.550 Gbps in 1001504 usec) 186.81 avg_batch 0 min_space 572.031206 main_thread [1720] 11.172 Mpps (11.189 Mpkts 5.550 Gbps in 1001497 usec) 185.98 avg_batch 0 min_space 573.038202 main_thread [1720] 11.172 Mpps (11.250 Mpkts 5.580 Gbps in 1006996 usec) 191.74 avg_batch 0 min_space 574.039199 main_thread [1720] 11.168 Mpps (11.180 Mpkts 5.545 Gbps in 1000997 usec) 196.01 avg_batch 0 min_space 575.042703 main_thread [1720] 11.173 Mpps (11.212 Mpkts 5.561 Gbps in 1003504 usec) 184.28 avg_batch 0 min_space 576.043200 main_thread [1720] 11.172 Mpps (11.177 Mpkts 5.544 Gbps in 1000497 usec) 189.59 avg_batch 0 min_space 577.044701 main_thread [1720] 11.172 Mpps (11.189 Mpkts 5.550 Gbps in 1001501 usec) 199.04 avg_batch 0 min_space 578.052358 main_thread [1720] 11.173 Mpps (11.258 Mpkts 5.584 Gbps in 1007657 usec) 184.56 avg_batch 0 min_space 579.053699 main_thread [1720] 11.170 Mpps (11.185 Mpkts 5.548 Gbps in 1001341 usec) 190.32 avg_batch 0 min_space 580.055701 main_thread [1720] 11.172 Mpps (11.195 Mpkts 5.552 Gbps in 1002002 usec) 201.08 avg_batch 0 min_space 581.060972 main_thread [1720] 7.664 Mpps (7.704 Mpkts 3.821 Gbps in 1005270 usec) 184.97 avg_batch 0 min_space 593.095693 main_thread [1720] 9.471 Mpps (9.478 Mpkts 5.004 Gbps in 1000720 usec) 158.37 avg_batch 0 min_space 594.104282 main_thread [1720] 9.469 Mpps (9.550 Mpkts 5.043 Gbps in 1008589 usec) 163.78 avg_batch 0 min_space 595.106902 main_thread [1720] 9.471 Mpps (9.496 Mpkts 5.014 Gbps in 1002620 usec) 157.00 avg_batch 0 min_space 596.107688 main_thread [1720] 9.471 Mpps (9.479 Mpkts 5.005 Gbps in 1000787 usec) 180.89 avg_batch 0 min_space 597.108692 main_thread [1720] 9.471 Mpps (9.481 Mpkts 5.006 Gbps in 1001002 usec) 179.28 avg_batch 0 min_space 598.109687 main_thread [1720] 9.471 Mpps (9.480 Mpkts 5.006 Gbps in 1000996 usec) 154.82 avg_batch 0 min_space 599.110692 main_thread [1720] 9.470 Mpps (9.479 Mpkts 5.005 Gbps in 1001005 usec) 178.17 avg_batch 0 min_space 600.118198 main_thread [1720] 9.468 Mpps (9.420 Mpkts 5.004 Gbps in 1007506 usec) 176.45 avg_batch 0 min_space ^C605.796897 sigint_h [404] received control-C on thread 0x801406800 606.128197 main_thread [1720] 0.000 pps (0.000 pkts 0.000 bps in 1004512 usec) 0.00 avg_batch 0 min_space Received 467977196 packets 29331410296 bytes 2508465 events 62 bytes each in 59.88 seconds. Speed: 7.816 Mpps Bandwidth: 3.919 Gbps (raw 5.420 Gbps). Average batch: 186.56 pkts kept the same RX session for both TX len sessions as you probably observed
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?20160124075621.7F4FEA0126>