From owner-freebsd-net@FreeBSD.ORG Wed Oct 2 23:39:36 2013 Return-Path: Delivered-To: freebsd-net@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTP id 166D8330 for ; Wed, 2 Oct 2013 23:39:36 +0000 (UTC) (envelope-from hiren.panchasara@gmail.com) Received: from mail-ee0-x22b.google.com (mail-ee0-x22b.google.com [IPv6:2a00:1450:4013:c00::22b]) (using TLSv1 with cipher ECDHE-RSA-RC4-SHA (128/128 bits)) (No client certificate requested) by mx1.freebsd.org (Postfix) with ESMTPS id 908102313 for ; Wed, 2 Oct 2013 23:39:35 +0000 (UTC) Received: by mail-ee0-f43.google.com with SMTP id e52so725379eek.2 for ; Wed, 02 Oct 2013 16:39:34 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :cc:content-type; bh=gBPFKp3r5RcGtRewrTqrQSZB+LSvKTQUspxKJqsTCYc=; b=p1xb2XlYWrDHGu8Qwd4Er5HzLwE0cR4IClOWgo057V1HPVIRmBewvSkIZT9UKUl9IE x3lJwKUR9CsC2ChMm6kax0X0sG9X1TkIrogjRTrRT+9oKgnNOlgJkSKyPIKqbdbdQQ6k cV4BwYk+TsI+ZetLR2Mo4edFtPZOl4CYeMI0i8DAeHECmN0Ie88C+ChHzMBrhox4cGhx 6ZcBn3xD3uaYeJdfJcrQmVJsqA2q4BTfygnxpKuGBwzSMsofUyNi1Jg+jjiYmtVHDPR8 0fQFyEw2Cot/Xi5SQSq0txTKQdWKRP+PkYwtOF2MW4ACMGlRgPSD9GP8NFMx6w4DY1fJ d7sA== MIME-Version: 1.0 X-Received: by 10.14.246.11 with SMTP id p11mr7642445eer.9.1380757174002; Wed, 02 Oct 2013 16:39:34 -0700 (PDT) Received: by 10.14.105.137 with HTTP; Wed, 2 Oct 2013 16:39:33 -0700 (PDT) In-Reply-To: References: Date: Wed, 2 Oct 2013 16:39:33 -0700 Message-ID: Subject: Re: netmap: understanding pkg-gen.c From: hiren panchasara To: Luigi Rizzo Content-Type: text/plain; charset=UTF-8 X-Content-Filtered-By: Mailman/MimeDel 2.1.14 Cc: "freebsd-net@freebsd.org" X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 02 Oct 2013 23:39:36 -0000 On Wed, Oct 2, 2013 at 4:15 PM, hiren panchasara wrote: > > > > On Wed, Oct 2, 2013 at 3:11 PM, hiren panchasara < > hiren.panchasara@gmail.com> wrote: > >> >> >> >> On Wed, Oct 2, 2013 at 1:18 PM, hiren panchasara < >> hiren.panchasara@gmail.com> wrote: >> >>> >>> Thanks Luigi. >>> >>> Coming back to this thread to actually understand what's going on. >>> >>> >>> On Tue, Sep 24, 2013 at 8:37 PM, Luigi Rizzo wrote: >>> >>>> > >>>> > On this box, I have 2 interfaces igb0 (10.73.149.28) and ix1 >>>> (10.73.149.17) >>>> > and this is how I am using this binary: >>>> > >>>> > -bash-4.2$ sudo ./pkt-gen -i ix1 -f tx -n 100000000 -c 8 -p 8 -d >>>> > 10.73.149.17 -s 10.73.149.28 >>>> >>> >>> So, my intention is to *send* 10gbps data to ix1 and see the card use >>> all of its 8 queues. >>> >>> Above command is the correct one? >>> >>> I kldunloaded/loaded ixbge to clear out all the stats and tested it >>> again: >>> >>> >>> -bash-4.2$ sudo ./pkt-gen -i ix1 -f tx -n 100000000 -c 8 -p 8 -d >>> 10.73.149.17 -s 10.73.149.28 >>> extract_ip_range [143] extract IP range from 10.73.149.28 >>> extract_ip_range [178] range is 10.73.149.28 0 to 10.73.149.28 0 >>> extract_ip_range [143] extract IP range from 10.73.149.17 >>> extract_ip_range [178] range is 10.73.149.17 0 to 10.73.149.17 0 >>> extract_mac_range [184] extract MAC range from 90:e2:ba:30:68:c5 >>> extract_mac_range [199] 90:e2:ba:30:68:c5 starts at 90:e2:ba:30:68:c5 >>> extract_mac_range [184] extract MAC range from ff:ff:ff:ff:ff:ff >>> extract_mac_range [199] ff:ff:ff:ff:ff:ff starts at ff:ff:ff:ff:ff:ff >>> main [1530] map size is 334980 Kb >>> main [1552] mapping 334980 Kbytes >>> Sending on ix1: 8 queues, 8 threads and 8 cpus. >>> 10.73.149.28 -> 10.73.149.17 (90:e2:ba:30:68:c5 -> ff:ff:ff:ff:ff:ff) >>> main [1622] Sending 512 packets every 0.000000000 ns >>> main [1624] Wait 2 secs for phy reset >>> main [1626] Ready... >>> sender_body [775] start >>> sender_body [775] start >>> sender_body [775] start >>> sender_body [775] start >>> sender_body [775] start >>> sender_body [775] start >>> sender_body [775] start >>> sender_body [848] drop copy >>> >>> sender_body [775] start >>> sender_body [848] drop copy >>> sender_body [848] drop copy >>> sender_body [848] drop copy >>> sender_body [848] drop copy >>> sender_body [848] drop copy >>> sender_body [848] drop copy >>> main_thread [1192] 15122963 pps (15130434 pkts in 1000494 usec) >>> main_thread [1192] 14881444 pps (14896266 pkts in 1000996 usec) >>> sender_body [841] poll error/timeout on queue 1 >>> main_thread [1192] 14880708 pps (15659371 pkts in 1052327 usec) >>> main_thread [1192] 14878611 pps (14888684 pkts in 1000677 usec) >>> main_thread [1192] 14882655 pps (14897538 pkts in 1001000 usec) >>> main_thread [1192] 11900044 pps (12029754 pkts in 1010900 usec) >>> main_thread [1212] ouch, thread 1 exited with error >>> Sent 87502047 packets, 60 bytes each, in 5.86 seconds. >>> Speed: 14.92 Mpps Bandwidth: 7.16 Gbps (raw 10.03 Gbps) >>> -bash-4.2$ >>> >>> But looking at the queue stats: >>> >>> >>> -bash-4.2$ sysctl -a | grep ix.1 | grep queue | grep rx_packets >>> dev.ix.1.queue0.rx_packets: 171 >>> dev.ix.1.queue1.rx_packets: 0 >>> dev.ix.1.queue2.rx_packets: 0 >>> dev.ix.1.queue3.rx_packets: 0 >>> dev.ix.1.queue4.rx_packets: 0 >>> dev.ix.1.queue5.rx_packets: 0 >>> dev.ix.1.queue6.rx_packets: 0 >>> dev.ix.1.queue7.rx_packets: 0 >>> >>> And after a few seconds: >>> >>> -bash-4.2$ sysctl -a | grep ix.1 | grep queue | grep rx_packets >>> dev.ix.1.queue0.rx_packets: 310 >>> dev.ix.1.queue1.rx_packets: 0 >>> dev.ix.1.queue2.rx_packets: 0 >>> dev.ix.1.queue3.rx_packets: 0 >>> dev.ix.1.queue4.rx_packets: 8 >>> dev.ix.1.queue5.rx_packets: 0 >>> dev.ix.1.queue6.rx_packets: 0 >>> dev.ix.1.queue7.rx_packets: 0 >>> -bash-4.2$ >>> >>> What is going on here? Should I be seeing more pkts in rx_packets? >>> Should I see more queues being used? >>> >>> I am using stock ixgbe at this point. I believe RSS is enabled by >>> default? >>> >> >> fdir(flow director)/atr is also enabled by default. >> >> I tried to turn that off by following change in >> /sys/modules/ixgbe/Makefile >> >> -CFLAGS+= -I${.CURDIR}/../../dev/ixgbe -DSMP -DIXGBE_FDIR >> +CFLAGS+= -I${.CURDIR}/../../dev/ixgbe -DSMP >> >> But, now interface would not attach to netmap: >> >> >> sudo ./pkt-gen -i ix1 -f tx -n 100000000 -c 8 -p 8 -d 10.73.149.17 -s >> 10.73.149.28 >> extract_ip_range [143] extract IP range from 10.73.149.28 >> extract_ip_range [178] range is 10.73.149.28 0 to 10.73.149.28 0 >> extract_ip_range [143] extract IP range from 10.73.149.17 >> extract_ip_range [178] range is 10.73.149.17 0 to 10.73.149.17 0 >> extract_mac_range [184] extract MAC range from 90:e2:ba:30:68:c5 >> extract_mac_range [199] 90:e2:ba:30:68:c5 starts at 90:e2:ba:30:68:c5 >> extract_mac_range [184] extract MAC range from ff:ff:ff:ff:ff:ff >> extract_mac_range [199] ff:ff:ff:ff:ff:ff starts at ff:ff:ff:ff:ff:ff >> main [1530] map size is 334980 Kb >> main [1536] Unable to get if info for ix1 >> main [1543] bad nthreads 8, have 0 queues >> main [1552] mapping 0 Kbytes >> main [1558] Unable to mmap 0 KB >> main [1572] Unable to register interface ix1 >> >> Sending on ix1: 0 queues, 8 threads and 8 cpus. >> 10.73.149.28 -> 10.73.149.17 (90:e2:ba:30:68:c5 -> ff:ff:ff:ff:ff:ff) >> main [1622] Sending 512 packets every 0.000000000 ns >> main [1624] Wait 2 secs for phy reset >> main [1626] Ready... >> start_threads [1126] Unable to register ix1 >> start_threads [1126] Unable to register ix1 >> start_threads [1126] Unable to register ix1 >> start_threads [1126] Unable to register ix1 >> start_threads [1126] Unable to register ix1 >> start_threads [1126] Unable to register ix1 >> start_threads [1126] Unable to register ix1 >> start_threads [1126] Unable to register ix1 >> main_thread [1192] 0 pps (0 pkts in 1043812 usec) >> main_thread [1212] ouch, thread 0 exited with error >> >> main_thread [1212] ouch, thread 1 exited with error >> main_thread [1212] ouch, thread 2 exited with error >> main_thread [1212] ouch, thread 3 exited with error >> main_thread [1212] ouch, thread 4 exited with error >> main_thread [1212] ouch, thread 5 exited with error >> main_thread [1212] ouch, thread 6 exited with error >> main_thread [1212] ouch, thread 7 exited with error >> Sent 0 packets, 60 bytes each, in 0.00 seconds. >> Speed: 0.00 pps Bandwidth: 0.00 bps (raw 0.00 bps) >> >> Error is coming from pkg-gen.c: >> >> if ((ioctl(tfd, NIOCREGIF, &tifreq)) == -1) { >> D("Unable to register %s", g->ifname); >> continue; >> } >> > > So somehow I am always seeing this error now. No matter what changes I > have made to ixgbe. I've tried a reboot too. > > I've removed "device ixgbe" from kernconf and building it as a module. > Should I not do that? > > Let me try putting it back in the kernel and see if I am still seeing the > issue. > Yep. Works now. So, ixgbe cannot be built/used as a module if I want to make fdir related changes. Alright back to the original issue of all 8 queues not being used. I removed -DIXGBE_FDIR to disable flow director. Just to see how it behaves: And I still see the same results on sudo ./pkt-gen -i ix1 -f tx -n 100000000 -c 8 -p 8 -d 10.73.149.17 -s 10.73.149.28 -bash-4.2$ sysctl -a | grep dev.ix.1 | grep queue | grep rx_packets dev.ix.1.queue0.rx_packets: 922 dev.ix.1.queue1.rx_packets: 48 dev.ix.1.queue2.rx_packets: 0 dev.ix.1.queue3.rx_packets: 0 dev.ix.1.queue4.rx_packets: 0 dev.ix.1.queue5.rx_packets: 0 dev.ix.1.queue6.rx_packets: 0 dev.ix.1.queue7.rx_packets: 0 Any help is appreciated. cheers, Hiren