From owner-freebsd-net@FreeBSD.ORG Tue Jun 2 16:23:24 2015 Return-Path: Delivered-To: freebsd-net@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 6349D6DE; Tue, 2 Jun 2015 16:23:24 +0000 (UTC) (envelope-from helpdesk@sealnet.de) Received: from mxout.sealnet.de (mxout.sealnet.de [194.77.96.6]) by mx1.freebsd.org (Postfix) with ESMTP id CB4E318ED; Tue, 2 Jun 2015 16:23:23 +0000 (UTC) (envelope-from helpdesk@sealnet.de) Received: from sealnet.de ([10.13.2.41]) by mxout.sealnet.de (8.14.3/8.14.3/Debian-5+lenny1) with ESMTP id t52GLKMS010062; Tue, 2 Jun 2015 18:21:21 +0200 Received: from DOM_RZ-MTA by sealnet.de with Novell_GroupWise; Tue, 02 Jun 2015 18:20:37 +0200 Message-Id: <556DF3D9020000D100008E2A@sealnet.de> X-Mailer: Novell GroupWise Internet Agent 14.0.1 Date: Tue, 02 Jun 2015 18:20:40 +0200 From: "HelpDesk SEALNET" To: Cc: , , , , Subject: Re: netmap and mlx4 driver status (linux) Mime-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Content-Disposition: inline X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 02 Jun 2015 16:23:24 -0000 -- von unterwegs gesendet. > Am 02.06.2015 um 17:40 schrieb Adrian Chadd : > > Hi, > > You'll likely want to poke the linux mellanox driver maintainer for some help. > > > > -adrian > > >> On 1 June 2015 at 17:08, Blake Caldwell wrote: >> Wondering if those experienced with other netmap drivers might be able to comment what is limiting performance of mlx4. It seems that the reason pkt-gen is only getting 2.4Mpps with mlx4 40G is that pkt-gen is saturating a core. This clearly shouldn’t be the case as evidenced by netmap papers (14.8Mpps at 900Mz core). As would be expected, the output from ‘perf top’ shows that sender_body and poll() are the largest userspace CPU hogs (measured in % of samples—over 24 cpus) >> >> 29.65% [netmap] [k] netmap_poll >> 12.47% [mlx4_en] [k] mlx4_netmap_txsync >> 8.69% libc-2.19.so [.] poll >> 6.15% pkt-gen [.] sender_body >> 2.26% [kernel] [k] local_clock >> 2.12% [kernel] [k] context_tracking_user_exit >> 1.87% [kernel] [k] select_estimate_accuracy >> 1.81% [kernel] [k] system_call >> …. >> 1.24% [netmap] [k] nm_txsync_prologue >> …. >> 0.63% [mlx4_en] [k] mlx4_en_arm_cq >> 0.61% [kernel] [k] account_user_time >> >> >> Furthermore it appears from annotating the code in pkt-gen.c with utilization, about 50% of sender_body is spent on this line while iterating through the rings: >> https://github.com/caldweba/netmap/blob/master/examples/pkt-gen.c#L1091 >> if (nm_ring_empty(txring)) >> >> Does this mean it is waiting for free slots most of the time and increasing from 8 rings might help? >> >> Here are the current module parameters in case they shed light on the issue. Also, netmap config kernel messages are shown below. >> >> Thanks in advance. >> >> /sys/module/netmap/parameters/adaptive_io: 0 >> /sys/module/netmap/parameters/admode: 0 >> /sys/module/netmap/parameters/bridge_batch: 1024 >> /sys/module/netmap/parameters/buf_curr_num: 163840 >> /sys/module/netmap/parameters/buf_curr_size: 2048 >> /sys/module/netmap/parameters/buf_num: 163840 >> /sys/module/netmap/parameters/buf_size: 2048 >> /sys/module/netmap/parameters/default_