From owner-freebsd-net@FreeBSD.ORG Wed Oct 2 22:11:09 2013 Return-Path: Delivered-To: freebsd-net@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTP id 876C8151 for ; Wed, 2 Oct 2013 22:11:09 +0000 (UTC) (envelope-from hiren.panchasara@gmail.com) Received: from mail-ee0-x22e.google.com (mail-ee0-x22e.google.com [IPv6:2a00:1450:4013:c00::22e]) (using TLSv1 with cipher ECDHE-RSA-RC4-SHA (128/128 bits)) (No client certificate requested) by mx1.freebsd.org (Postfix) with ESMTPS id 0DF0D2DE9 for ; Wed, 2 Oct 2013 22:11:08 +0000 (UTC) Received: by mail-ee0-f46.google.com with SMTP id c13so704097eek.33 for ; Wed, 02 Oct 2013 15:11:07 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :cc:content-type; bh=Ftwg1IhVegvgS7cjrMhvlUjsrAXlBrzmB7VmsnUKdgI=; b=IZXxhOwpLlfQRvWoKsrOAWEKQoiaDUZDQrQi2uVqEJ8exEpjWb8dHlT4b+pmoIhd1J 1CyVUiexo/hUhmHDKKXdagoUoPR7Az2OK87mkK0RCQr+wllX/qYPmXvYoJ9noq18T/lt qQby/lXl97LsnIWg9QNPvRA1R7Q9QATFzWYfFwzl6IKozilw7D/dlLCKnbTyhi2P2xSk xRVdvh2FsWoDzwILx030dv7g9ZxZgbU7sC3sKKCxWyVxOKxpIPC+MMWbNxcXAHUL20nD 0EuzLvcT0zGkJK/H5QmOyl9uEHl+3eyQY5EqXV0nEuKvdTUllVTwpeWzXH+GrtYhTeQq WDpQ== MIME-Version: 1.0 X-Received: by 10.15.54.199 with SMTP id t47mr7076409eew.46.1380751867341; Wed, 02 Oct 2013 15:11:07 -0700 (PDT) Received: by 10.14.105.137 with HTTP; Wed, 2 Oct 2013 15:11:07 -0700 (PDT) In-Reply-To: References: Date: Wed, 2 Oct 2013 15:11:07 -0700 Message-ID: Subject: Re: netmap: understanding pkg-gen.c From: hiren panchasara To: Luigi Rizzo Content-Type: text/plain; charset=UTF-8 X-Content-Filtered-By: Mailman/MimeDel 2.1.14 Cc: "freebsd-net@freebsd.org" X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 02 Oct 2013 22:11:09 -0000 On Wed, Oct 2, 2013 at 1:18 PM, hiren panchasara wrote: > > Thanks Luigi. > > Coming back to this thread to actually understand what's going on. > > > On Tue, Sep 24, 2013 at 8:37 PM, Luigi Rizzo wrote: > >> > >> > On this box, I have 2 interfaces igb0 (10.73.149.28) and ix1 >> (10.73.149.17) >> > and this is how I am using this binary: >> > >> > -bash-4.2$ sudo ./pkt-gen -i ix1 -f tx -n 100000000 -c 8 -p 8 -d >> > 10.73.149.17 -s 10.73.149.28 >> > > So, my intention is to *send* 10gbps data to ix1 and see the card use all > of its 8 queues. > > Above command is the correct one? > > I kldunloaded/loaded ixbge to clear out all the stats and tested it again: > > > -bash-4.2$ sudo ./pkt-gen -i ix1 -f tx -n 100000000 -c 8 -p 8 -d > 10.73.149.17 -s 10.73.149.28 > extract_ip_range [143] extract IP range from 10.73.149.28 > extract_ip_range [178] range is 10.73.149.28 0 to 10.73.149.28 0 > extract_ip_range [143] extract IP range from 10.73.149.17 > extract_ip_range [178] range is 10.73.149.17 0 to 10.73.149.17 0 > extract_mac_range [184] extract MAC range from 90:e2:ba:30:68:c5 > extract_mac_range [199] 90:e2:ba:30:68:c5 starts at 90:e2:ba:30:68:c5 > extract_mac_range [184] extract MAC range from ff:ff:ff:ff:ff:ff > extract_mac_range [199] ff:ff:ff:ff:ff:ff starts at ff:ff:ff:ff:ff:ff > main [1530] map size is 334980 Kb > main [1552] mapping 334980 Kbytes > Sending on ix1: 8 queues, 8 threads and 8 cpus. > 10.73.149.28 -> 10.73.149.17 (90:e2:ba:30:68:c5 -> ff:ff:ff:ff:ff:ff) > main [1622] Sending 512 packets every 0.000000000 ns > main [1624] Wait 2 secs for phy reset > main [1626] Ready... > sender_body [775] start > sender_body [775] start > sender_body [775] start > sender_body [775] start > sender_body [775] start > sender_body [775] start > sender_body [775] start > sender_body [848] drop copy > > sender_body [775] start > sender_body [848] drop copy > sender_body [848] drop copy > sender_body [848] drop copy > sender_body [848] drop copy > sender_body [848] drop copy > sender_body [848] drop copy > main_thread [1192] 15122963 pps (15130434 pkts in 1000494 usec) > main_thread [1192] 14881444 pps (14896266 pkts in 1000996 usec) > sender_body [841] poll error/timeout on queue 1 > main_thread [1192] 14880708 pps (15659371 pkts in 1052327 usec) > main_thread [1192] 14878611 pps (14888684 pkts in 1000677 usec) > main_thread [1192] 14882655 pps (14897538 pkts in 1001000 usec) > main_thread [1192] 11900044 pps (12029754 pkts in 1010900 usec) > main_thread [1212] ouch, thread 1 exited with error > Sent 87502047 packets, 60 bytes each, in 5.86 seconds. > Speed: 14.92 Mpps Bandwidth: 7.16 Gbps (raw 10.03 Gbps) > -bash-4.2$ > > But looking at the queue stats: > > > -bash-4.2$ sysctl -a | grep ix.1 | grep queue | grep rx_packets > dev.ix.1.queue0.rx_packets: 171 > dev.ix.1.queue1.rx_packets: 0 > dev.ix.1.queue2.rx_packets: 0 > dev.ix.1.queue3.rx_packets: 0 > dev.ix.1.queue4.rx_packets: 0 > dev.ix.1.queue5.rx_packets: 0 > dev.ix.1.queue6.rx_packets: 0 > dev.ix.1.queue7.rx_packets: 0 > > And after a few seconds: > > -bash-4.2$ sysctl -a | grep ix.1 | grep queue | grep rx_packets > dev.ix.1.queue0.rx_packets: 310 > dev.ix.1.queue1.rx_packets: 0 > dev.ix.1.queue2.rx_packets: 0 > dev.ix.1.queue3.rx_packets: 0 > dev.ix.1.queue4.rx_packets: 8 > dev.ix.1.queue5.rx_packets: 0 > dev.ix.1.queue6.rx_packets: 0 > dev.ix.1.queue7.rx_packets: 0 > -bash-4.2$ > > What is going on here? Should I be seeing more pkts in rx_packets? Should > I see more queues being used? > > I am using stock ixgbe at this point. I believe RSS is enabled by default? > fdir(flow director)/atr is also enabled by default. I tried to turn that off by following change in /sys/modules/ixgbe/Makefile -CFLAGS+= -I${.CURDIR}/../../dev/ixgbe -DSMP -DIXGBE_FDIR +CFLAGS+= -I${.CURDIR}/../../dev/ixgbe -DSMP But, now interface would not attach to netmap: sudo ./pkt-gen -i ix1 -f tx -n 100000000 -c 8 -p 8 -d 10.73.149.17 -s 10.73.149.28 extract_ip_range [143] extract IP range from 10.73.149.28 extract_ip_range [178] range is 10.73.149.28 0 to 10.73.149.28 0 extract_ip_range [143] extract IP range from 10.73.149.17 extract_ip_range [178] range is 10.73.149.17 0 to 10.73.149.17 0 extract_mac_range [184] extract MAC range from 90:e2:ba:30:68:c5 extract_mac_range [199] 90:e2:ba:30:68:c5 starts at 90:e2:ba:30:68:c5 extract_mac_range [184] extract MAC range from ff:ff:ff:ff:ff:ff extract_mac_range [199] ff:ff:ff:ff:ff:ff starts at ff:ff:ff:ff:ff:ff main [1530] map size is 334980 Kb main [1536] Unable to get if info for ix1 main [1543] bad nthreads 8, have 0 queues main [1552] mapping 0 Kbytes main [1558] Unable to mmap 0 KB main [1572] Unable to register interface ix1 Sending on ix1: 0 queues, 8 threads and 8 cpus. 10.73.149.28 -> 10.73.149.17 (90:e2:ba:30:68:c5 -> ff:ff:ff:ff:ff:ff) main [1622] Sending 512 packets every 0.000000000 ns main [1624] Wait 2 secs for phy reset main [1626] Ready... start_threads [1126] Unable to register ix1 start_threads [1126] Unable to register ix1 start_threads [1126] Unable to register ix1 start_threads [1126] Unable to register ix1 start_threads [1126] Unable to register ix1 start_threads [1126] Unable to register ix1 start_threads [1126] Unable to register ix1 start_threads [1126] Unable to register ix1 main_thread [1192] 0 pps (0 pkts in 1043812 usec) main_thread [1212] ouch, thread 0 exited with error main_thread [1212] ouch, thread 1 exited with error main_thread [1212] ouch, thread 2 exited with error main_thread [1212] ouch, thread 3 exited with error main_thread [1212] ouch, thread 4 exited with error main_thread [1212] ouch, thread 5 exited with error main_thread [1212] ouch, thread 6 exited with error main_thread [1212] ouch, thread 7 exited with error Sent 0 packets, 60 bytes each, in 0.00 seconds. Speed: 0.00 pps Bandwidth: 0.00 bps (raw 0.00 bps) Error is coming from pkg-gen.c: if ((ioctl(tfd, NIOCREGIF, &tifreq)) == -1) { D("Unable to register %s", g->ifname); continue; } Cheers, Hiren