Date: Wed, 2 Oct 2013 16:39:33 -0700 From: hiren panchasara <hiren.panchasara@gmail.com> To: Luigi Rizzo <rizzo@iet.unipi.it> Cc: "freebsd-net@freebsd.org" <freebsd-net@freebsd.org> Subject: Re: netmap: understanding pkg-gen.c Message-ID: <CALCpEUFJu_S2Q-EPOV1dm7q5aDYe%2BZj_MgkVoAwQP047KV8suw@mail.gmail.com> In-Reply-To: <CALCpEUFA6Z3ASgKFKPsWjt1HfUCFNM1iAhSte5ChNHLpi6oJLw@mail.gmail.com> References: <CALCpEUHj_Mke_njM0NwzCUa1f_pXZdCy3_aaqJtKz5NUJUU8Mg@mail.gmail.com> <CA%2BhQ2%2BjYyoxpLEoz0kHm8Hikfgf8Fke-o_ctkO3fKBdGqEz7wQ@mail.gmail.com> <CALCpEUG1dM9Z6OG1BqRAPAfHciANXxaYVpezDEOUNug22QN6HQ@mail.gmail.com> <CALCpEUGi%2Boaq9JftSWX%2BR-UL%2BCVHaYwpmjvzksKANjhLcn-niA@mail.gmail.com> <CALCpEUFA6Z3ASgKFKPsWjt1HfUCFNM1iAhSte5ChNHLpi6oJLw@mail.gmail.com>
next in thread | previous in thread | raw e-mail | index | archive | help
On Wed, Oct 2, 2013 at 4:15 PM, hiren panchasara <hiren.panchasara@gmail.com
> wrote:
>
>
>
> On Wed, Oct 2, 2013 at 3:11 PM, hiren panchasara <
> hiren.panchasara@gmail.com> wrote:
>
>>
>>
>>
>> On Wed, Oct 2, 2013 at 1:18 PM, hiren panchasara <
>> hiren.panchasara@gmail.com> wrote:
>>
>>>
>>> Thanks Luigi.
>>>
>>> Coming back to this thread to actually understand what's going on.
>>>
>>>
>>> On Tue, Sep 24, 2013 at 8:37 PM, Luigi Rizzo <rizzo@iet.unipi.it> wrote:
>>>
>>>> >
>>>> > On this box, I have 2 interfaces igb0 (10.73.149.28) and ix1
>>>> (10.73.149.17)
>>>> > and this is how I am using this binary:
>>>> >
>>>> > -bash-4.2$ sudo ./pkt-gen -i ix1 -f tx -n 100000000 -c 8 -p 8 -d
>>>> > 10.73.149.17 -s 10.73.149.28
>>>>
>>>
>>> So, my intention is to *send* 10gbps data to ix1 and see the card use
>>> all of its 8 queues.
>>>
>>> Above command is the correct one?
>>>
>>> I kldunloaded/loaded ixbge to clear out all the stats and tested it
>>> again:
>>>
>>>
>>> -bash-4.2$ sudo ./pkt-gen -i ix1 -f tx -n 100000000 -c 8 -p 8 -d
>>> 10.73.149.17 -s 10.73.149.28
>>> extract_ip_range [143] extract IP range from 10.73.149.28
>>> extract_ip_range [178] range is 10.73.149.28 0 to 10.73.149.28 0
>>> extract_ip_range [143] extract IP range from 10.73.149.17
>>> extract_ip_range [178] range is 10.73.149.17 0 to 10.73.149.17 0
>>> extract_mac_range [184] extract MAC range from 90:e2:ba:30:68:c5
>>> extract_mac_range [199] 90:e2:ba:30:68:c5 starts at 90:e2:ba:30:68:c5
>>> extract_mac_range [184] extract MAC range from ff:ff:ff:ff:ff:ff
>>> extract_mac_range [199] ff:ff:ff:ff:ff:ff starts at ff:ff:ff:ff:ff:ff
>>> main [1530] map size is 334980 Kb
>>> main [1552] mapping 334980 Kbytes
>>> Sending on ix1: 8 queues, 8 threads and 8 cpus.
>>> 10.73.149.28 -> 10.73.149.17 (90:e2:ba:30:68:c5 -> ff:ff:ff:ff:ff:ff)
>>> main [1622] Sending 512 packets every 0.000000000 ns
>>> main [1624] Wait 2 secs for phy reset
>>> main [1626] Ready...
>>> sender_body [775] start
>>> sender_body [775] start
>>> sender_body [775] start
>>> sender_body [775] start
>>> sender_body [775] start
>>> sender_body [775] start
>>> sender_body [775] start
>>> sender_body [848] drop copy
>>>
>>> sender_body [775] start
>>> sender_body [848] drop copy
>>> sender_body [848] drop copy
>>> sender_body [848] drop copy
>>> sender_body [848] drop copy
>>> sender_body [848] drop copy
>>> sender_body [848] drop copy
>>> main_thread [1192] 15122963 pps (15130434 pkts in 1000494 usec)
>>> main_thread [1192] 14881444 pps (14896266 pkts in 1000996 usec)
>>> sender_body [841] poll error/timeout on queue 1
>>> main_thread [1192] 14880708 pps (15659371 pkts in 1052327 usec)
>>> main_thread [1192] 14878611 pps (14888684 pkts in 1000677 usec)
>>> main_thread [1192] 14882655 pps (14897538 pkts in 1001000 usec)
>>> main_thread [1192] 11900044 pps (12029754 pkts in 1010900 usec)
>>> main_thread [1212] ouch, thread 1 exited with error
>>> Sent 87502047 packets, 60 bytes each, in 5.86 seconds.
>>> Speed: 14.92 Mpps Bandwidth: 7.16 Gbps (raw 10.03 Gbps)
>>> -bash-4.2$
>>>
>>> But looking at the queue stats:
>>>
>>>
>>> -bash-4.2$ sysctl -a | grep ix.1 | grep queue | grep rx_packets
>>> dev.ix.1.queue0.rx_packets: 171
>>> dev.ix.1.queue1.rx_packets: 0
>>> dev.ix.1.queue2.rx_packets: 0
>>> dev.ix.1.queue3.rx_packets: 0
>>> dev.ix.1.queue4.rx_packets: 0
>>> dev.ix.1.queue5.rx_packets: 0
>>> dev.ix.1.queue6.rx_packets: 0
>>> dev.ix.1.queue7.rx_packets: 0
>>>
>>> And after a few seconds:
>>>
>>> -bash-4.2$ sysctl -a | grep ix.1 | grep queue | grep rx_packets
>>> dev.ix.1.queue0.rx_packets: 310
>>> dev.ix.1.queue1.rx_packets: 0
>>> dev.ix.1.queue2.rx_packets: 0
>>> dev.ix.1.queue3.rx_packets: 0
>>> dev.ix.1.queue4.rx_packets: 8
>>> dev.ix.1.queue5.rx_packets: 0
>>> dev.ix.1.queue6.rx_packets: 0
>>> dev.ix.1.queue7.rx_packets: 0
>>> -bash-4.2$
>>>
>>> What is going on here? Should I be seeing more pkts in rx_packets?
>>> Should I see more queues being used?
>>>
>>> I am using stock ixgbe at this point. I believe RSS is enabled by
>>> default?
>>>
>>
>> fdir(flow director)/atr is also enabled by default.
>>
>> I tried to turn that off by following change in
>> /sys/modules/ixgbe/Makefile
>>
>> -CFLAGS+= -I${.CURDIR}/../../dev/ixgbe -DSMP -DIXGBE_FDIR
>> +CFLAGS+= -I${.CURDIR}/../../dev/ixgbe -DSMP
>>
>> But, now interface would not attach to netmap:
>>
>>
>> sudo ./pkt-gen -i ix1 -f tx -n 100000000 -c 8 -p 8 -d 10.73.149.17 -s
>> 10.73.149.28
>> extract_ip_range [143] extract IP range from 10.73.149.28
>> extract_ip_range [178] range is 10.73.149.28 0 to 10.73.149.28 0
>> extract_ip_range [143] extract IP range from 10.73.149.17
>> extract_ip_range [178] range is 10.73.149.17 0 to 10.73.149.17 0
>> extract_mac_range [184] extract MAC range from 90:e2:ba:30:68:c5
>> extract_mac_range [199] 90:e2:ba:30:68:c5 starts at 90:e2:ba:30:68:c5
>> extract_mac_range [184] extract MAC range from ff:ff:ff:ff:ff:ff
>> extract_mac_range [199] ff:ff:ff:ff:ff:ff starts at ff:ff:ff:ff:ff:ff
>> main [1530] map size is 334980 Kb
>> main [1536] Unable to get if info for ix1
>> main [1543] bad nthreads 8, have 0 queues
>> main [1552] mapping 0 Kbytes
>> main [1558] Unable to mmap 0 KB
>> main [1572] Unable to register interface ix1
>>
>> Sending on ix1: 0 queues, 8 threads and 8 cpus.
>> 10.73.149.28 -> 10.73.149.17 (90:e2:ba:30:68:c5 -> ff:ff:ff:ff:ff:ff)
>> main [1622] Sending 512 packets every 0.000000000 ns
>> main [1624] Wait 2 secs for phy reset
>> main [1626] Ready...
>> start_threads [1126] Unable to register ix1
>> start_threads [1126] Unable to register ix1
>> start_threads [1126] Unable to register ix1
>> start_threads [1126] Unable to register ix1
>> start_threads [1126] Unable to register ix1
>> start_threads [1126] Unable to register ix1
>> start_threads [1126] Unable to register ix1
>> start_threads [1126] Unable to register ix1
>> main_thread [1192] 0 pps (0 pkts in 1043812 usec)
>> main_thread [1212] ouch, thread 0 exited with error
>>
>> main_thread [1212] ouch, thread 1 exited with error
>> main_thread [1212] ouch, thread 2 exited with error
>> main_thread [1212] ouch, thread 3 exited with error
>> main_thread [1212] ouch, thread 4 exited with error
>> main_thread [1212] ouch, thread 5 exited with error
>> main_thread [1212] ouch, thread 6 exited with error
>> main_thread [1212] ouch, thread 7 exited with error
>> Sent 0 packets, 60 bytes each, in 0.00 seconds.
>> Speed: 0.00 pps Bandwidth: 0.00 bps (raw 0.00 bps)
>>
>> Error is coming from pkg-gen.c:
>>
>> if ((ioctl(tfd, NIOCREGIF, &tifreq)) == -1) {
>> D("Unable to register %s", g->ifname);
>> continue;
>> }
>>
>
> So somehow I am always seeing this error now. No matter what changes I
> have made to ixgbe. I've tried a reboot too.
>
> I've removed "device ixgbe" from kernconf and building it as a module.
> Should I not do that?
>
> Let me try putting it back in the kernel and see if I am still seeing the
> issue.
>
Yep. Works now. So, ixgbe cannot be built/used as a module if I want to
make fdir related changes.
Alright back to the original issue of all 8 queues not being used.
I removed -DIXGBE_FDIR to disable flow director. Just to see how it behaves:
And I still see the same results on
sudo ./pkt-gen -i ix1 -f tx -n 100000000 -c 8 -p 8 -d 10.73.149.17 -s
10.73.149.28
-bash-4.2$ sysctl -a | grep dev.ix.1 | grep queue | grep rx_packets
dev.ix.1.queue0.rx_packets: 922
dev.ix.1.queue1.rx_packets: 48
dev.ix.1.queue2.rx_packets: 0
dev.ix.1.queue3.rx_packets: 0
dev.ix.1.queue4.rx_packets: 0
dev.ix.1.queue5.rx_packets: 0
dev.ix.1.queue6.rx_packets: 0
dev.ix.1.queue7.rx_packets: 0
Any help is appreciated.
cheers,
Hiren
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?CALCpEUFJu_S2Q-EPOV1dm7q5aDYe%2BZj_MgkVoAwQP047KV8suw>