From owner-freebsd-net@FreeBSD.ORG Wed Oct 2 23:15:23 2013 Return-Path: Delivered-To: freebsd-net@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTP id 43D11B81 for ; Wed, 2 Oct 2013 23:15:23 +0000 (UTC) (envelope-from hiren.panchasara@gmail.com) Received: from mail-ea0-x22c.google.com (mail-ea0-x22c.google.com [IPv6:2a00:1450:4013:c01::22c]) (using TLSv1 with cipher ECDHE-RSA-RC4-SHA (128/128 bits)) (No client certificate requested) by mx1.freebsd.org (Postfix) with ESMTPS id BCC6C2196 for ; Wed, 2 Oct 2013 23:15:22 +0000 (UTC) Received: by mail-ea0-f172.google.com with SMTP id r16so723080ead.31 for ; Wed, 02 Oct 2013 16:15:21 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :cc:content-type; bh=V0dfqGBXH/pkgVtPjO+N65z/z29JYmi5glcuGHiL8VI=; b=J9m2ld2yyDvLzIte/e4hUBT7Xf4hXJ2b35nrts1FM5/ZMwMZ3Um2S2ufKLqrzeWWAs RbGgXb+6TYXqQFCqDt/90mQBA2ePg9BYFi/xmrwAkffDJcHblk5Lp8B+Emv1PauZM1yC vq8BodYhqz16WDudWOQecH0A6ADsRfwS/5d6HsaGiaDt04QXfOL5pKVATLhh5iHduoy8 tPzv565JpdSsUYEP2mIfZxJKjlUctF7Hlu3hsRdwvJMj3eRfR+T3GzM160F81A1mLVj6 06cV3PhypreEGxxkJ4P0Q5xTK83ydTgYF0zXYYptKrhy76yGsEzxfeWa7fe/M4gKazcm N84A== MIME-Version: 1.0 X-Received: by 10.14.223.1 with SMTP id u1mr7529492eep.10.1380755721046; Wed, 02 Oct 2013 16:15:21 -0700 (PDT) Received: by 10.14.105.137 with HTTP; Wed, 2 Oct 2013 16:15:20 -0700 (PDT) In-Reply-To: References: Date: Wed, 2 Oct 2013 16:15:20 -0700 Message-ID: Subject: Re: netmap: understanding pkg-gen.c From: hiren panchasara To: Luigi Rizzo Content-Type: text/plain; charset=UTF-8 X-Content-Filtered-By: Mailman/MimeDel 2.1.14 Cc: "freebsd-net@freebsd.org" X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 02 Oct 2013 23:15:23 -0000 On Wed, Oct 2, 2013 at 3:11 PM, hiren panchasara wrote: > > > > On Wed, Oct 2, 2013 at 1:18 PM, hiren panchasara < > hiren.panchasara@gmail.com> wrote: > >> >> Thanks Luigi. >> >> Coming back to this thread to actually understand what's going on. >> >> >> On Tue, Sep 24, 2013 at 8:37 PM, Luigi Rizzo wrote: >> >>> > >>> > On this box, I have 2 interfaces igb0 (10.73.149.28) and ix1 >>> (10.73.149.17) >>> > and this is how I am using this binary: >>> > >>> > -bash-4.2$ sudo ./pkt-gen -i ix1 -f tx -n 100000000 -c 8 -p 8 -d >>> > 10.73.149.17 -s 10.73.149.28 >>> >> >> So, my intention is to *send* 10gbps data to ix1 and see the card use all >> of its 8 queues. >> >> Above command is the correct one? >> >> I kldunloaded/loaded ixbge to clear out all the stats and tested it again: >> >> >> -bash-4.2$ sudo ./pkt-gen -i ix1 -f tx -n 100000000 -c 8 -p 8 -d >> 10.73.149.17 -s 10.73.149.28 >> extract_ip_range [143] extract IP range from 10.73.149.28 >> extract_ip_range [178] range is 10.73.149.28 0 to 10.73.149.28 0 >> extract_ip_range [143] extract IP range from 10.73.149.17 >> extract_ip_range [178] range is 10.73.149.17 0 to 10.73.149.17 0 >> extract_mac_range [184] extract MAC range from 90:e2:ba:30:68:c5 >> extract_mac_range [199] 90:e2:ba:30:68:c5 starts at 90:e2:ba:30:68:c5 >> extract_mac_range [184] extract MAC range from ff:ff:ff:ff:ff:ff >> extract_mac_range [199] ff:ff:ff:ff:ff:ff starts at ff:ff:ff:ff:ff:ff >> main [1530] map size is 334980 Kb >> main [1552] mapping 334980 Kbytes >> Sending on ix1: 8 queues, 8 threads and 8 cpus. >> 10.73.149.28 -> 10.73.149.17 (90:e2:ba:30:68:c5 -> ff:ff:ff:ff:ff:ff) >> main [1622] Sending 512 packets every 0.000000000 ns >> main [1624] Wait 2 secs for phy reset >> main [1626] Ready... >> sender_body [775] start >> sender_body [775] start >> sender_body [775] start >> sender_body [775] start >> sender_body [775] start >> sender_body [775] start >> sender_body [775] start >> sender_body [848] drop copy >> >> sender_body [775] start >> sender_body [848] drop copy >> sender_body [848] drop copy >> sender_body [848] drop copy >> sender_body [848] drop copy >> sender_body [848] drop copy >> sender_body [848] drop copy >> main_thread [1192] 15122963 pps (15130434 pkts in 1000494 usec) >> main_thread [1192] 14881444 pps (14896266 pkts in 1000996 usec) >> sender_body [841] poll error/timeout on queue 1 >> main_thread [1192] 14880708 pps (15659371 pkts in 1052327 usec) >> main_thread [1192] 14878611 pps (14888684 pkts in 1000677 usec) >> main_thread [1192] 14882655 pps (14897538 pkts in 1001000 usec) >> main_thread [1192] 11900044 pps (12029754 pkts in 1010900 usec) >> main_thread [1212] ouch, thread 1 exited with error >> Sent 87502047 packets, 60 bytes each, in 5.86 seconds. >> Speed: 14.92 Mpps Bandwidth: 7.16 Gbps (raw 10.03 Gbps) >> -bash-4.2$ >> >> But looking at the queue stats: >> >> >> -bash-4.2$ sysctl -a | grep ix.1 | grep queue | grep rx_packets >> dev.ix.1.queue0.rx_packets: 171 >> dev.ix.1.queue1.rx_packets: 0 >> dev.ix.1.queue2.rx_packets: 0 >> dev.ix.1.queue3.rx_packets: 0 >> dev.ix.1.queue4.rx_packets: 0 >> dev.ix.1.queue5.rx_packets: 0 >> dev.ix.1.queue6.rx_packets: 0 >> dev.ix.1.queue7.rx_packets: 0 >> >> And after a few seconds: >> >> -bash-4.2$ sysctl -a | grep ix.1 | grep queue | grep rx_packets >> dev.ix.1.queue0.rx_packets: 310 >> dev.ix.1.queue1.rx_packets: 0 >> dev.ix.1.queue2.rx_packets: 0 >> dev.ix.1.queue3.rx_packets: 0 >> dev.ix.1.queue4.rx_packets: 8 >> dev.ix.1.queue5.rx_packets: 0 >> dev.ix.1.queue6.rx_packets: 0 >> dev.ix.1.queue7.rx_packets: 0 >> -bash-4.2$ >> >> What is going on here? Should I be seeing more pkts in rx_packets? Should >> I see more queues being used? >> >> I am using stock ixgbe at this point. I believe RSS is enabled by >> default? >> > > fdir(flow director)/atr is also enabled by default. > > I tried to turn that off by following change in /sys/modules/ixgbe/Makefile > > -CFLAGS+= -I${.CURDIR}/../../dev/ixgbe -DSMP -DIXGBE_FDIR > +CFLAGS+= -I${.CURDIR}/../../dev/ixgbe -DSMP > > But, now interface would not attach to netmap: > > > sudo ./pkt-gen -i ix1 -f tx -n 100000000 -c 8 -p 8 -d 10.73.149.17 -s > 10.73.149.28 > extract_ip_range [143] extract IP range from 10.73.149.28 > extract_ip_range [178] range is 10.73.149.28 0 to 10.73.149.28 0 > extract_ip_range [143] extract IP range from 10.73.149.17 > extract_ip_range [178] range is 10.73.149.17 0 to 10.73.149.17 0 > extract_mac_range [184] extract MAC range from 90:e2:ba:30:68:c5 > extract_mac_range [199] 90:e2:ba:30:68:c5 starts at 90:e2:ba:30:68:c5 > extract_mac_range [184] extract MAC range from ff:ff:ff:ff:ff:ff > extract_mac_range [199] ff:ff:ff:ff:ff:ff starts at ff:ff:ff:ff:ff:ff > main [1530] map size is 334980 Kb > main [1536] Unable to get if info for ix1 > main [1543] bad nthreads 8, have 0 queues > main [1552] mapping 0 Kbytes > main [1558] Unable to mmap 0 KB > main [1572] Unable to register interface ix1 > > Sending on ix1: 0 queues, 8 threads and 8 cpus. > 10.73.149.28 -> 10.73.149.17 (90:e2:ba:30:68:c5 -> ff:ff:ff:ff:ff:ff) > main [1622] Sending 512 packets every 0.000000000 ns > main [1624] Wait 2 secs for phy reset > main [1626] Ready... > start_threads [1126] Unable to register ix1 > start_threads [1126] Unable to register ix1 > start_threads [1126] Unable to register ix1 > start_threads [1126] Unable to register ix1 > start_threads [1126] Unable to register ix1 > start_threads [1126] Unable to register ix1 > start_threads [1126] Unable to register ix1 > start_threads [1126] Unable to register ix1 > main_thread [1192] 0 pps (0 pkts in 1043812 usec) > main_thread [1212] ouch, thread 0 exited with error > > main_thread [1212] ouch, thread 1 exited with error > main_thread [1212] ouch, thread 2 exited with error > main_thread [1212] ouch, thread 3 exited with error > main_thread [1212] ouch, thread 4 exited with error > main_thread [1212] ouch, thread 5 exited with error > main_thread [1212] ouch, thread 6 exited with error > main_thread [1212] ouch, thread 7 exited with error > Sent 0 packets, 60 bytes each, in 0.00 seconds. > Speed: 0.00 pps Bandwidth: 0.00 bps (raw 0.00 bps) > > Error is coming from pkg-gen.c: > > if ((ioctl(tfd, NIOCREGIF, &tifreq)) == -1) { > D("Unable to register %s", g->ifname); > continue; > } > So somehow I am always seeing this error now. No matter what changes I have made to ixgbe. I've tried a reboot too. I've removed "device ixgbe" from kernconf and building it as a module. Should I not do that? Let me try putting it back in the kernel and see if I am still seeing the issue. Really weird. cheers, Hiren