From owner-freebsd-net@FreeBSD.ORG Mon Jan 19 19:36:57 2015 Return-Path: Delivered-To: net@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 9CEE12F5 for ; Mon, 19 Jan 2015 19:36:57 +0000 (UTC) Received: from mail-wi0-f170.google.com (mail-wi0-f170.google.com [209.85.212.170]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 3884FCD2 for ; Mon, 19 Jan 2015 19:36:56 +0000 (UTC) Received: by mail-wi0-f170.google.com with SMTP id em10so327467wid.1 for ; Mon, 19 Jan 2015 11:36:49 -0800 (PST) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20130820; h=x-gm-message-state:message-id:date:from:user-agent:mime-version:to :subject:content-type:content-transfer-encoding; bh=jDtN+ncIJ1qq+X57DkaGcKagoPofofMgLHgkDqCFhYg=; b=X52oVj471Cs2mtaBH3TiNe8Vrlzsrj2cV9ECqu45i8OkHbcPPSc67y8v3kSF1kgJWL LU14iDPmyJLCRFBfZnjFnP9TSFBq36WzEbCc9LeF8pF3yvRPfAfV9AOU5GzYo0g2v/HY vx+rEvQtyJCgTcq1W1aWtVow/+v3cZzcIYCTdcxddP7I9z9nSWPcqhl4Z81qSaUNDMnn LEY5ElRxH8Kkmj6VVdBLAI+qvADGdANrVeRY8wiwM4aLN5oVEK0ZwCYJtKTOY2EoEn5E w33s/TH1CgWirE62H8NgTU+m4caItXj3M3ibpB+BBhhqAHkKU05SSkjVJ27/yKPxNq7/ hXuA== X-Gm-Message-State: ALoCoQnlq2q0zdfCwskdkcN5pvcM7m5dM0QAtv51zN22JTCuCrW+7L8xRVp/BPPbmAGJWw/RkZ9O X-Received: by 10.180.74.229 with SMTP id x5mr38603328wiv.1.1421696209130; Mon, 19 Jan 2015 11:36:49 -0800 (PST) Received: from [192.168.0.101] ([90.152.119.35]) by mx.google.com with ESMTPSA id da2sm8267704wjb.21.2015.01.19.11.36.47 for (version=TLSv1.2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Mon, 19 Jan 2015 11:36:48 -0800 (PST) Message-ID: <54BD5CCF.8080300@linaro.org> Date: Mon, 19 Jan 2015 19:36:47 +0000 From: Zoltan Kiss User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:31.0) Gecko/20100101 Thunderbird/31.3.0 MIME-Version: 1.0 To: net@freebsd.org Subject: ixgbe TX desc prefetch Content-Type: text/plain; charset=utf-8; format=flowed Content-Transfer-Encoding: 7bit X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.18-1 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 19 Jan 2015 19:36:57 -0000 Hi, I'm using netmap on Ubuntu 14.04 (3.13.0-44 kernel, ixgbe 3.15.1-k), and I can't max out a 10G link with pktgen : Sent 1068122462 packets, 64 bytes each, in 84.29 seconds. Speed: 12.67 Mpps Bandwidth: 6.49 Gbps (raw 8.92 Gbps) The README says "ixgbe tops at about 12.5 Mpps unless the driver prefetches tx descriptors". I've checked the actual driver code, it does a prefetch(tx_desc) in the TX completion interrupt handler, is that what you mean? Top shows ksoftirqd eats up one core while the pktgen process is around 45% My problem gets even worse when I want to use the another port on this same dual port card to do receive back the traffic (I'm sending my packets through a device I want to test for switching performance). The sending performance drops down to 9.39 Mpps (6.61 Gbps), and the receiving goes this much as well. I'm trying to bind the threads to cores with "-a 3" and so, but they don't seem to obey based on top. The TX now uses ca 50% CPU while RX is %20, but they don't seem to run on their assigned CPU. My card is an Intel 82599ES, the CPU is i5-4570 @ 3.2GHz (no HT). Maybe the fact it is a workstation CPU contributes to this problem? All suggestions welcome! Regards, Zoltan Kiss