From owner-freebsd-net@FreeBSD.ORG Tue Aug 16 09:39:34 2011 Return-Path: Delivered-To: net@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 392101065676 for ; Tue, 16 Aug 2011 09:39:34 +0000 (UTC) (envelope-from syuu@dokukino.com) Received: from mail-qw0-f54.google.com (mail-qw0-f54.google.com [209.85.216.54]) by mx1.freebsd.org (Postfix) with ESMTP id F367D8FC14 for ; Tue, 16 Aug 2011 09:39:33 +0000 (UTC) Received: by qwc9 with SMTP id 9so3777366qwc.13 for ; Tue, 16 Aug 2011 02:39:33 -0700 (PDT) Received: by 10.229.119.151 with SMTP id z23mr3175135qcq.2.1313486041240; Tue, 16 Aug 2011 02:14:01 -0700 (PDT) MIME-Version: 1.0 Received: by 10.229.100.84 with HTTP; Tue, 16 Aug 2011 02:13:21 -0700 (PDT) From: Takuya ASADA Date: Tue, 16 Aug 2011 18:13:21 +0900 Message-ID: To: net@freebsd.org Content-Type: text/plain; charset=UTF-8 Cc: Subject: Multiqueue support for bpf X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 16 Aug 2011 09:39:34 -0000 Hi all, I implemented multiqueue support for bpf, I'd like to present for review. This is a Google Summer of Code project, the project goal is to support multiqueue network interface on BPF, and provide interfaces for multithreaded packet processing using BPF. Modern high performance NICs have multiple receive/send queues and RSS feature, this allows to process packet concurrently on multiple processors. Main purpose of the project is to support these hardware and get benefit of parallelism. This provides following new APIs: - queue filter for each bpf descriptor (bpf ioctl) - BIOCENAQMASK Enables multiqueue filter on the descriptor - BIOCDISQMASK Disables multiqueue filter on the descriptor - BIOCSTRXQMASK Set mask bit on specified RX queue - BIOCCRRXQMASK Clear mask bit on specified RX queue - BIOCGTRXQMASK Get mask bit on specified RX queue - BIOCSTTXQMASK Set mask bit on specified TX queue - BIOCCRTXQMASK Clear mask bit on specified TX queue - BIOCGTTXQMASK Get mask bit on specified TX queue - BIOCSTOTHERMASK Set mask bit for the packets which not tied with any queues - BIOCCROTHERMASK Clear mask bit for the packets which not tied with any queues - BIOCGTOTHERMASK Get mask bit for the packets which not tied with any queues - generic interface for getting hardware queue information from NIC driver (socket ioctl) - SIOCGIFQLEN Get interface RX/TX queue length - SIOCGIFRXQAFFINITY Get interface RX queue affinity - SIOCGIFTXQAFFINITY Get interface TX queue affinity Patch for -CURRENT is here, right now it only supports igb(4), ixgbe(4), mxge(4): http://www.dokukino.com/mq_bpf_20110813.diff And below is performance benchmark: ==== I implemented benchmark programs based on bpfnull(//depot/projects/zcopybpf/utils/bpfnull/), test_sqbpf measures bpf throughput on one thread, without using multiqueue APIs. http://p4db.freebsd.org/fileViewer.cgi?FSPC=//depot/projects/soc2011/mq_bpf/src/tools/regression/bpf/mq_bpf/test_sqbpf/test_sqbpf.c test_mqbpf is multithreaded version of test_sqbpf, using multiqueue APIs. http://p4db.freebsd.org/fileViewer.cgi?FSPC=//depot/projects/soc2011/mq_bpf/src/tools/regression/bpf/mq_bpf/test_mqbpf/test_mqbpf.c I benchmarked with six conditions: - benchmark1 only reads bpf, doesn't write packet anywhere - benchmark2 writes packet on memory(mfs) - benchmark3 writes packet on hdd(zfs) - benchmark4 only reads bpf, doesn't write packet anywhere, with zerocopy - benchmark5 writes packet on memory(mfs), with zerocopy - benchmark6 writes packet on hdd(zfs), with zerocopy >From benchmark result, I can say the performance is increased using mq_bpf on 10GbE, but not on GbE. * Throughput benchmark - Test environment - FreeBSD node CPU: Core i7 X980 (12 threads) MB: ASUS P6X58D Premium(Intel X58) NIC1: Intel Gigabit ET Dual Port Server Adapter(82576) NIC2: Intel Ethernet X520-DA2 Server Adapter(82599) - Linux node CPU: Core 2 Quad (4 threads) MB: GIGABYTE GA-G33-DS3R(Intel G33) NIC1: Intel Gigabit ET Dual Port Server Adapter(82576) NIC2: Intel Ethernet X520-DA2 Server Adapter(82599) iperf used for generate network traffic, with following argument options - Linux node: iperf -c [IP] -i 10 -t 100000 -P12 - FreeBSD node: iperf -s # 12 threads, TCP following sysctl parameter is changed sysctl -w net.bpf.maxbufsize=1048576 - Benchmark1 Benchmark1 doesn't write packet anywhere using following commands ./test_sqbpf -i [interface] -b 1048576 ./test_mqbpf -i [interface] -b 1048576 - ixgbe test_mqbpf: 5303.09007533333 Mbps test_sqbpf: 3959.83021733333 Mbps - igb test_mqbpf: 916.752133333333 Mbps test_sqbpf: 917.597079 Mbps - Benchmark2 Benchmark2 write packet on mfs using following commands mdmfs -s 10G md /mnt ./test_sqbpf -i [interface] -b 1048576 -w -f /mnt/test ./test_mqbpf -i [interface] -b 1048576 -w -f /mnt/test - ixgbe test_mqbpf: 1061.24890333333 Mbps test_sqbpf: 204.779881 Mbps - igb test_mqbpf: 916.656664666667 Mbps test_sqbpf: 914.378636 Mbps - Benchmark3 Benchmark3 write packet on zfs(on HDD) using following commands ./test_sqbpf -i [interface] -b 1048576 -w -f test ./test_mqbpf -i [interface] -b 1048576 -w -f test - ixgbe test_mqbpf: 119.912253333333 Mbps test_sqbpf: 101.195918 Mbps - igb test_mqbpf: 228.910355333333 Mbps test_sqbpf: 199.639093666667 Mbps - Benchmark4 Benchmark4 doesn't write packet anywhere using following commands, with zerocopy ./test_sqbpf -i [interface] -b 1048576 ./test_mqbpf -i [interface] -b 1048576 - ixgbe test_mqbpf: 4772.924974 Mbps test_sqbpf: 3173.19967133333 Mbps - igb test_mqbpf: 931.217345 Mbps test_sqbpf: 925.965270666667 Mbps - Benchmark5 Benchmark5 write packet on mfs using following commands, with zerocopy mdmfs -s 10G md /mnt ./test_sqbpf -i [interface] -b 1048576 -w -f /mnt/test ./test_mqbpf -i [interface] -b 1048576 -w -f /mnt/test - ixgbe test_mqbpf: 306.902822333333 Mbps test_sqbpf: 317.605016666667 Mbps - igb test_mqbpf: 729.075349666667 Mbps test_sqbpf: 708.987822666667 Mbps - Benchmark6 Benchmark6 write packet on zfs(on HDD) using following commands, with zerocopy ./test_sqbpf -i [interface] -b 1048576 -w -f test ./test_mqbpf -i [interface] -b 1048576 -w -f test - ixgbe test_mqbpf: 174.016136666667 Mbps test_sqbpf: 138.068732666667 Mbps - igb test_mqbpf: 228.794880333333 Mbps test_sqbpf: 229.367386333333 Mbps