From owner-freebsd-net@FreeBSD.ORG Mon Jan 19 19:37:12 2015 Return-Path: Delivered-To: net@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 2964B381 for ; Mon, 19 Jan 2015 19:37:12 +0000 (UTC) Received: from mail-wi0-f174.google.com (mail-wi0-f174.google.com [209.85.212.174]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id B6E5FCE0 for ; Mon, 19 Jan 2015 19:37:11 +0000 (UTC) Received: by mail-wi0-f174.google.com with SMTP id n3so1619272wiv.1 for ; Mon, 19 Jan 2015 11:37:04 -0800 (PST) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20130820; h=x-gm-message-state:message-id:date:from:user-agent:mime-version:to :cc:subject:content-type:content-transfer-encoding; bh=jDtN+ncIJ1qq+X57DkaGcKagoPofofMgLHgkDqCFhYg=; b=d5bo/taQhjW24iWXZyTyJ9sAtFd+pTRuUNMA1mS3zR8cfX5YIQ7sWsTnXUuF/rt/8M 9wbiA9KHHZZLm8+J/QFHK/pdhfxv3kS+ryjRLLSZV38i5nvhGuBIiVXucmJ3PlSYM0oT sOee0ubIvxxCIfAN+g03frbxWU26fdCWVjTNP4SfgiTgoiGMgNFe8G+Rx090/JlqZzBV jNPma05E0RN/d2tqSgM7cJU2SXPzq/tEKhsKMH8DNu6n5NZRZH2o4z9DIb0ySSi08k3Y ZnH/inoApfrJS5fNZ/+/B/8cYuD25PT0nDuEBXDQru6JmIlkv0u3fDBnfxlYXRobQdVU LbDg== X-Gm-Message-State: ALoCoQkXY1a/imcE6CY66ka/+bJioGuxDfL2IPziR/OemEA7UnsReT01yXagBQXTyUfjyuV/S/Eq X-Received: by 10.180.95.9 with SMTP id dg9mr8829757wib.1.1421696224473; Mon, 19 Jan 2015 11:37:04 -0800 (PST) Received: from [192.168.0.101] ([90.152.119.35]) by mx.google.com with ESMTPSA id uo6sm18412901wjc.49.2015.01.19.11.37.03 (version=TLSv1.2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Mon, 19 Jan 2015 11:37:03 -0800 (PST) Message-ID: <54BD5CDF.20602@linaro.org> Date: Mon, 19 Jan 2015 19:37:03 +0000 From: Zoltan Kiss User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:31.0) Gecko/20100101 Thunderbird/31.3.0 MIME-Version: 1.0 To: net@freebsd.org Subject: ixgbe TX desc prefetch Content-Type: text/plain; charset=utf-8; format=flowed Content-Transfer-Encoding: 7bit Cc: Mike Holmes , Ciprian Barbu X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.18-1 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 19 Jan 2015 19:37:12 -0000 Hi, I'm using netmap on Ubuntu 14.04 (3.13.0-44 kernel, ixgbe 3.15.1-k), and I can't max out a 10G link with pktgen : Sent 1068122462 packets, 64 bytes each, in 84.29 seconds. Speed: 12.67 Mpps Bandwidth: 6.49 Gbps (raw 8.92 Gbps) The README says "ixgbe tops at about 12.5 Mpps unless the driver prefetches tx descriptors". I've checked the actual driver code, it does a prefetch(tx_desc) in the TX completion interrupt handler, is that what you mean? Top shows ksoftirqd eats up one core while the pktgen process is around 45% My problem gets even worse when I want to use the another port on this same dual port card to do receive back the traffic (I'm sending my packets through a device I want to test for switching performance). The sending performance drops down to 9.39 Mpps (6.61 Gbps), and the receiving goes this much as well. I'm trying to bind the threads to cores with "-a 3" and so, but they don't seem to obey based on top. The TX now uses ca 50% CPU while RX is %20, but they don't seem to run on their assigned CPU. My card is an Intel 82599ES, the CPU is i5-4570 @ 3.2GHz (no HT). Maybe the fact it is a workstation CPU contributes to this problem? All suggestions welcome! Regards, Zoltan Kiss