From owner-freebsd-net@FreeBSD.ORG Mon May 23 17:19:30 2011 Return-Path: Delivered-To: net@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 5397E106564A for ; Mon, 23 May 2011 17:19:30 +0000 (UTC) (envelope-from luigi@onelab2.iet.unipi.it) Received: from onelab2.iet.unipi.it (onelab2.iet.unipi.it [131.114.59.238]) by mx1.freebsd.org (Postfix) with ESMTP id 2050D8FC14 for ; Mon, 23 May 2011 17:19:29 +0000 (UTC) Received: by onelab2.iet.unipi.it (Postfix, from userid 275) id 5AA937300A; Mon, 23 May 2011 19:20:10 +0200 (CEST) Date: Mon, 23 May 2011 19:20:10 +0200 From: Luigi Rizzo To: net@freebsd.org Message-ID: <20110523172010.GA540@onelab2.iet.unipi.it> Mime-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline User-Agent: Mutt/1.4.2.3i Cc: Subject: strange performance receiving with ixgbe/82599 X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 23 May 2011 17:19:30 -0000 Jack, i am doing some experiments with a modified ixgbe driver, and i am seeing some strange performance numbers which I'd like to explain. My card uses the 82599 chip, monted on an x16 slot on an Asus motherboard. CPU is i7-870 @ 2.93GHz. Both -head and RELENG_8 exhibit the same behaviour. I am using a modified driver which essentially accesses the rings from userspace, sending or receiving frames as soon as there is room. On the transmit side, using 64 byte packets (60+4 CRC) i can saturate the link (measured 14.87-14.88 Mpps) with no problem with just one core at 1.33 GHz and 2 or 4 queues (one queue tops at 12Mpps, but that's not a big issue). On the receive side, i can sustain 14.2Mpps (on 2-4 queues) if the sender generates 68-byte packets (i.e. 64+4CRC), but as soon as i go to 63+4 or less, the receive rate drops to 11.5-11.6Mpps. CPU cycles abound, because the speed is unchanged down to 1.45GHz Any reason to explain that suddend drop with short (but legal) packet sizes ? The stats counters report a lot of "missed" packets, though i never see the receive rings become full. I was wondering if there is any minimum inter-packet time that the receiver side needs, but could not find any register that controls that. The rings have 2048 slots each, so even something slow happening when the rings wrap don't really explain such a huge (25% or so) loss level. cheers luigi