From owner-freebsd-net@freebsd.org Wed Aug 12 02:05:18 2015 Return-Path: Delivered-To: freebsd-net@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 014E999EECD for ; Wed, 12 Aug 2015 02:05:18 +0000 (UTC) (envelope-from sobomax@sippysoft.com) Received: from mail-wi0-f174.google.com (mail-wi0-f174.google.com [209.85.212.174]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 946FB6AA for ; Wed, 12 Aug 2015 02:05:17 +0000 (UTC) (envelope-from sobomax@sippysoft.com) Received: by wijp15 with SMTP id p15so198945495wij.0 for ; Tue, 11 Aug 2015 19:05:15 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20130820; h=x-gm-message-state:mime-version:sender:in-reply-to:references:date :message-id:subject:from:to:content-type; bh=XfDNmOeptl6UJkEU7WV7YKyjpOcBs2jLYR7prUlCw7Y=; b=CEp2L8Gh2H+G/Ngd8VqnTpK0Of0h9K2wEZeqorj/kzfQF2kqgyCnlb+wMaCTXmJZYg X+SJ1xD8K0jDXl23MpJuPii8N/I1SKOjKTXqAnLdChltRYQOzA6UEPwd2CiFXfg7Qv48 Jb4nR28wSXXTSfQngOYsDF2GtghVp2ctUByxGu7d4k+o2E4LUPcw+VWnUiAIDKwoukW+ rfOO9e/Q4WgizGdq47gQQkCFrnuyNvfDcC4IGzI0/cN85dZ8C6hMo8TOFLGiKJSygpZ2 nmNEr7HmXDJOyXaOSnGbAvplpMYudQ0rqZQa5Lld7oRGhdCw8ng+9jz2k/0gMNbki0xw zxTg== X-Gm-Message-State: ALoCoQmS1WEEHzK0/pCzkYXJUmFmj1ce9MXO5UbogZ+PTyOET4aSpfPS7YbKtdP10kLL2v53vxK6 MIME-Version: 1.0 X-Received: by 10.194.78.164 with SMTP id c4mr59995431wjx.65.1439345115412; Tue, 11 Aug 2015 19:05:15 -0700 (PDT) Sender: sobomax@sippysoft.com Received: by 10.27.143.15 with HTTP; Tue, 11 Aug 2015 19:05:15 -0700 (PDT) In-Reply-To: References: Date: Tue, 11 Aug 2015 19:05:15 -0700 X-Google-Sender-Auth: vBPMlHX7RrNBcy8dEZclZvM-w50 Message-ID: Subject: Re: Poor high-PPS performance of the 10G ixgbe(9) NIC/driver in FreeBSD 10.1 From: Maxim Sobolev To: Luigi Rizzo , FreeBSD Net , =?UTF-8?Q?Jev_Bj=C3=B6rsell?= Content-Type: text/plain; charset=UTF-8 X-Content-Filtered-By: Mailman/MimeDel 2.1.20 X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 12 Aug 2015 02:05:18 -0000 OK, so following Luigi's suggestion we've re-enabled AIM, set max_interrupt_rate to 8000 (matching igb), and reduced number of queues to 6. We'll have next peak in about 14 hours, I'll try to capture and record history of the per-queue interrupt rate. It still remains somewhat puzzling why something that worked with old-gen hardware/driver almost out of the box now needs some extra tuning. Thanks everyone for useful suggestions in any case! On Tue, Aug 11, 2015 at 4:04 PM, Luigi Rizzo wrote: > On Wed, Aug 12, 2015 at 12:46 AM, Maxim Sobolev > wrote: > > Hi Luigi, thanks for the input. Are you sure about the number of queues? > > Keep in mind we are not using netmap or anything as advanced, so we need > lot > > of parallel pipes just to push packet through the whole UDP/IP kernel > stack. > > To make things worse, our app cannot use connected sockets at the > moment, so > > we are having worst case scenario wrt kernel overhead. Single thread > sender > > on that hardware is limited to only 225K PPS until it hits CPU limit. We > use > > 8 queues on other older servers with I350 NICs. My send-only testing > shows > > that the hardware peaks at about 5 concurrent threads for connected > sockets > > and 7 concurrent threads for unconnected. This is with default 8 hardware > > queues. The test is somewhat crippled since it only tests outgoing > traffic. > > > > thing is, you are using 8 queues per NIC so the total number of different > interrupts is huge and will likely eat all of your cores. > This is why i'd try to start low and ramp up later. > > > http://bit.ly/1MiLFHg > > > > With regards to the moderation, what effect it has on the traffic? Is NIC > > going to drop packets once it starts moderating interrupts? > > moderation only means you get larger batches on each interrupt, > which overall makes the system more efficient. The NIC will only drop > packets when the CPU is not fast enough to drain them,. > > (this said, it might be a good idea to try netmap, passing packets > that you do not know how to handle to the host stack, and instead > intercepting those that you can deal with in netmap. those > unconnected udp sockets are the ideal case for netmap.) > > cheers > luigi >