From owner-freebsd-net@FreeBSD.ORG Wed Oct 2 20:18:41 2013 Return-Path: Delivered-To: freebsd-net@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTP id 0ABE8E00 for ; Wed, 2 Oct 2013 20:18:41 +0000 (UTC) (envelope-from hiren.panchasara@gmail.com) Received: from mail-ea0-x22e.google.com (mail-ea0-x22e.google.com [IPv6:2a00:1450:4013:c01::22e]) (using TLSv1 with cipher ECDHE-RSA-RC4-SHA (128/128 bits)) (No client certificate requested) by mx1.freebsd.org (Postfix) with ESMTPS id 82E5525D4 for ; Wed, 2 Oct 2013 20:18:40 +0000 (UTC) Received: by mail-ea0-f174.google.com with SMTP id z15so650778ead.5 for ; Wed, 02 Oct 2013 13:18:38 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :cc:content-type; bh=Qx5od+Z2YIm8y7zM9j4qhvOWp6K2rOD+2jpt3IbobnI=; b=fz45oaO5wnf1GkiWdtsv0TRwUfOZt4azXrnldUD5yMnmySn3OKwQRgD9lH+bOPopDn zSggFgyM/e1QWCEiVBBHc0FwlJbc7/w/ord91kck/pDwJOdkqkOxPt5qaD2KlH7tjUXm mKeuW5zsZYfnPRgRNuqCdiWGoxot6nqTU0+/FAxKKvVpxAS1M04JZZUSbiMX+YcB/m5x xF0UVqhuwtEqLpoU03ekMXvED2HOVg76Nfgrj8RVrV5eci9CsqqMBAcL3IPX4p6vD4bO RhH3V32oMLBVcAuq7jktHsZH4eM58IhMsoR7K47FLqS/9PdNxQcrTy2h2MvoTRvAX6is a8vg== MIME-Version: 1.0 X-Received: by 10.15.98.194 with SMTP id bj42mr6481611eeb.12.1380745118795; Wed, 02 Oct 2013 13:18:38 -0700 (PDT) Received: by 10.14.105.137 with HTTP; Wed, 2 Oct 2013 13:18:38 -0700 (PDT) In-Reply-To: References: Date: Wed, 2 Oct 2013 13:18:38 -0700 Message-ID: Subject: Re: netmap: understanding pkg-gen.c From: hiren panchasara To: Luigi Rizzo Content-Type: text/plain; charset=UTF-8 X-Content-Filtered-By: Mailman/MimeDel 2.1.14 Cc: "freebsd-net@freebsd.org" X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 02 Oct 2013 20:18:41 -0000 Thanks Luigi. Coming back to this thread to actually understand what's going on. On Tue, Sep 24, 2013 at 8:37 PM, Luigi Rizzo wrote: > > > > On this box, I have 2 interfaces igb0 (10.73.149.28) and ix1 > (10.73.149.17) > > and this is how I am using this binary: > > > > -bash-4.2$ sudo ./pkt-gen -i ix1 -f tx -n 100000000 -c 8 -p 8 -d > > 10.73.149.17 -s 10.73.149.28 > So, my intention is to *send* 10gbps data to ix1 and see the card use all of its 8 queues. Above command is the correct one? I kldunloaded/loaded ixbge to clear out all the stats and tested it again: -bash-4.2$ sudo ./pkt-gen -i ix1 -f tx -n 100000000 -c 8 -p 8 -d 10.73.149.17 -s 10.73.149.28 extract_ip_range [143] extract IP range from 10.73.149.28 extract_ip_range [178] range is 10.73.149.28 0 to 10.73.149.28 0 extract_ip_range [143] extract IP range from 10.73.149.17 extract_ip_range [178] range is 10.73.149.17 0 to 10.73.149.17 0 extract_mac_range [184] extract MAC range from 90:e2:ba:30:68:c5 extract_mac_range [199] 90:e2:ba:30:68:c5 starts at 90:e2:ba:30:68:c5 extract_mac_range [184] extract MAC range from ff:ff:ff:ff:ff:ff extract_mac_range [199] ff:ff:ff:ff:ff:ff starts at ff:ff:ff:ff:ff:ff main [1530] map size is 334980 Kb main [1552] mapping 334980 Kbytes Sending on ix1: 8 queues, 8 threads and 8 cpus. 10.73.149.28 -> 10.73.149.17 (90:e2:ba:30:68:c5 -> ff:ff:ff:ff:ff:ff) main [1622] Sending 512 packets every 0.000000000 ns main [1624] Wait 2 secs for phy reset main [1626] Ready... sender_body [775] start sender_body [775] start sender_body [775] start sender_body [775] start sender_body [775] start sender_body [775] start sender_body [775] start sender_body [848] drop copy sender_body [775] start sender_body [848] drop copy sender_body [848] drop copy sender_body [848] drop copy sender_body [848] drop copy sender_body [848] drop copy sender_body [848] drop copy main_thread [1192] 15122963 pps (15130434 pkts in 1000494 usec) main_thread [1192] 14881444 pps (14896266 pkts in 1000996 usec) sender_body [841] poll error/timeout on queue 1 main_thread [1192] 14880708 pps (15659371 pkts in 1052327 usec) main_thread [1192] 14878611 pps (14888684 pkts in 1000677 usec) main_thread [1192] 14882655 pps (14897538 pkts in 1001000 usec) main_thread [1192] 11900044 pps (12029754 pkts in 1010900 usec) main_thread [1212] ouch, thread 1 exited with error Sent 87502047 packets, 60 bytes each, in 5.86 seconds. Speed: 14.92 Mpps Bandwidth: 7.16 Gbps (raw 10.03 Gbps) -bash-4.2$ But looking at the queue stats: -bash-4.2$ sysctl -a | grep ix.1 | grep queue | grep rx_packets dev.ix.1.queue0.rx_packets: 171 dev.ix.1.queue1.rx_packets: 0 dev.ix.1.queue2.rx_packets: 0 dev.ix.1.queue3.rx_packets: 0 dev.ix.1.queue4.rx_packets: 0 dev.ix.1.queue5.rx_packets: 0 dev.ix.1.queue6.rx_packets: 0 dev.ix.1.queue7.rx_packets: 0 And after a few seconds: -bash-4.2$ sysctl -a | grep ix.1 | grep queue | grep rx_packets dev.ix.1.queue0.rx_packets: 310 dev.ix.1.queue1.rx_packets: 0 dev.ix.1.queue2.rx_packets: 0 dev.ix.1.queue3.rx_packets: 0 dev.ix.1.queue4.rx_packets: 8 dev.ix.1.queue5.rx_packets: 0 dev.ix.1.queue6.rx_packets: 0 dev.ix.1.queue7.rx_packets: 0 -bash-4.2$ What is going on here? Should I be seeing more pkts in rx_packets? Should I see more queues being used? I am using stock ixgbe at this point. I believe RSS is enabled by default? I apologize if I am asking obvious/answered questions here. cheers, Hiren