From owner-freebsd-net@freebsd.org Tue Aug 11 21:18:28 2015 Return-Path: Delivered-To: freebsd-net@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 296EF99F4EE for ; Tue, 11 Aug 2015 21:18:28 +0000 (UTC) (envelope-from sobomax@sippysoft.com) Received: from mail-wi0-f180.google.com (mail-wi0-f180.google.com [209.85.212.180]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id BEBFC75D for ; Tue, 11 Aug 2015 21:18:27 +0000 (UTC) (envelope-from sobomax@sippysoft.com) Received: by wicja10 with SMTP id ja10so90440321wic.1 for ; Tue, 11 Aug 2015 14:18:20 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20130820; h=x-gm-message-state:mime-version:sender:date:message-id:subject:from :to:cc:content-type; bh=kkry+rP7kmNzgmCcCR7YBt9YOOvw520ZWisdlFGvHyA=; b=BKiKKocFQ+4UL5tkdehDYZlpgHL6hrjMOua8XD8HHeWTKFwta+ozwUQCEsnlEqAb/0 HApjSEwLeV93ft73VxpAkyndTUvvvVmCeYJJuQtk0TR7EXvb39oY4XS29y8CllkuQf0k 6NqLNgGUhE9/TkTMq3rjsT+jHVatf2AUjAIqM01ckJJKhDYLqxgv8ZPeENVtZKqU9xCm 4NDyVNZWfI0uRMCVkstw/eWTojlEJMBtsdryqMk4GOQswwvwtp+KpqJBxZ6OSrmTYjuY hroBD1DsLaXtFPwj9YBjqaWYkLFf0y+noIKLxWBz5k51BZbv9nugTYxmQWn3DUelivrK fUVA== X-Gm-Message-State: ALoCoQnWBb60cVobrxlCDaORM2LXsB5gNfnsIBFSiACRPt3xKgmMYDEty5Va5Lqvuk5upMZUY6Pm MIME-Version: 1.0 X-Received: by 10.194.78.164 with SMTP id c4mr57929916wjx.65.1439327899891; Tue, 11 Aug 2015 14:18:19 -0700 (PDT) Sender: sobomax@sippysoft.com Received: by 10.27.143.15 with HTTP; Tue, 11 Aug 2015 14:18:19 -0700 (PDT) Date: Tue, 11 Aug 2015 14:18:19 -0700 X-Google-Sender-Auth: R71as7e1Cojoyot60-kr12bxWwE Message-ID: Subject: Poor high-PPS performance of the 10G ixgbe(9) NIC/driver in FreeBSD 10.1 From: Maxim Sobolev To: FreeBSD Net , freebsd@intel.com Cc: =?UTF-8?Q?Jev_Bj=C3=B6rsell?= Content-Type: text/plain; charset=UTF-8 X-Content-Filtered-By: Mailman/MimeDel 2.1.20 X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 11 Aug 2015 21:18:28 -0000 Hi folks, We've trying to migrate some of our high-PPS systems to a new hardware that has four X540-AT2 10G NICs and observed that interrupt time goes through roof after we cross around 200K PPS in and 200K out (two ports in LACP). The previous hardware was stable up to about 350K PPS in and 350K out. I believe the old one was equipped with the I350 and had the identical LACP configuration. The new box also has better CPU with more cores (i.e. 24 cores vs. 16 cores before). CPU itself is 2 x E5-2690 v3. After hitting this limit with the default settings, I've tried to tweak the following settings: hw.ix.rx_process_limit="-1" hw.ix.tx_process_limit="-1" hw.ix.enable_aim="0" hw.ix.max_interrupt_rate="-1" hw.ix.rxd="4096" hw.ix.txd="4096" dev.ix.0.fc=0 dev.ix.1.fc=0 dev.ix.2.fc=0 dev.ix.3.fc=0 hw.intr_storm_threshold=0 But there is little or no effect on the performance. The workload is just lot of small UDP packets being relayed between bunch of hosts. The symptoms are always the same - the box runs nice and cool until it his the said PPS threshold, with kernel spending just few percent in the interrupts and then it jumps straight to 100% interrupt time, thereby scaring some traffic away due to packet loss and such, so that the load drops and the system goes into the "cool" state again. It looks very much like some contention in the driver or in the hardware. Linked are some monitoring screenshots displaying the issue unfolding as well as systat -vm screenshots from the "cool" state. http://sobomax.sippysoft.com/ScreenShot387.png <- CPU utilization right before the "bang event" http://sobomax.sippysoft.com/ScreenShot382.png <- issue itself http://sobomax.sippysoft.com/ScreenShot385.png <- systat -vm few minutes after traffic declined somewhat We are now trying to get customer install 1Gig NIC so that we can run it and compare performance with the rest of the hardware and software being essentially the same. Any ideas on how to improve/resolve this problem are welcome. Thanks!