From owner-freebsd-performance@freebsd.org Sat Feb 6 06:50:26 2016 Return-Path: Delivered-To: freebsd-performance@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id E5AB4A9E8BF for ; Sat, 6 Feb 2016 06:50:26 +0000 (UTC) (envelope-from savetherbtz@gmail.com) Received: from mail-pa0-x234.google.com (mail-pa0-x234.google.com [IPv6:2607:f8b0:400e:c03::234]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id B16D7C15; Sat, 6 Feb 2016 06:50:26 +0000 (UTC) (envelope-from savetherbtz@gmail.com) Received: by mail-pa0-x234.google.com with SMTP id cy9so45831268pac.0; Fri, 05 Feb 2016 22:50:26 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=subject:mime-version:content-type:from:in-reply-to:date:cc :message-id:references:to; bh=nhC3vZJ05KvbVDICzhuP1nIST4Yxur++HrJYJmQOAT0=; b=z9kZipfhiuJzOR5sLOKw9UPGaEVsPWo8Vjy4+wB9PsWw2o21BY5zSH3NT4IiU7TqTd kZHKbJHrRYFCi2fmdD7F8llPsi5dhmz912C+2iTO6mvrgRIi0bEDMnxvIQmhM3EMp4NW xIoMDvQ90uDkix98BAb+hA5LAwRVEMoCZ/VjPDgXJKGps7gSqewklhBZPCSNR/zrD9QW 45sLY8NnioUxZQkDiP+8Rm3xwlwTmffkcraC2iu8re2h8OvBwocQwteMwH9gtmCXo/OS pXDwavLAwypQtKV6slmilJM0vJ8HdYExNhpbf2vPjVLEV3nlSKbFuc/FoG65LAMlR4p2 Lh0Q== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20130820; h=x-gm-message-state:subject:mime-version:content-type:from :in-reply-to:date:cc:message-id:references:to; bh=nhC3vZJ05KvbVDICzhuP1nIST4Yxur++HrJYJmQOAT0=; b=XTvWWQTJBoKTFmQ6o/sqh30OX4FGcepjdk3BJ8yZlaK+7zVui5dVz9oWn6WDii58pC h55hgxMNWTn2r+nJ6KHccsyNsglVTKBZbA5Hzu+BBKuIE5cm4VKS8cM1pWTOCWwNacIO KxHpharqMTBAbNePiDCVejEorQoMQ5OKJunIFlIc/jSspZfmBAXfwr4fAqoUmKCieWlZ Vdyob9yTyWIkZJQkQFOPi54w6zpgVOqLgFaKpYIfqqP5XUn7W2m4BTph+W2PETGX0wsz PkoXdwzL4nKK8IIYbSqm4Z0B7V4E0xpnITOS7mazeHBCxRc/7ljruYDWw3DZt8dfVfL1 chrg== X-Gm-Message-State: AG10YOTKsImxaqd14zv0b05laS3ewrqOI63UhrnoEiYRj7h9dxGsPUKfjlWk47k7PwYzfA== X-Received: by 10.66.242.17 with SMTP id wm17mr25486321pac.155.1454741426095; Fri, 05 Feb 2016 22:50:26 -0800 (PST) Received: from [10.0.1.12] (c-98-248-221-187.hsd1.ca.comcast.net. [98.248.221.187]) by smtp.gmail.com with ESMTPSA id fa3sm28625923pab.45.2016.02.05.22.50.24 (version=TLS1 cipher=ECDHE-RSA-AES128-SHA bits=128/128); Fri, 05 Feb 2016 22:50:25 -0800 (PST) Subject: Re: ixgbe: Network performance tuning (#TCP connections) Mime-Version: 1.0 (Mac OS X Mail 9.1 \(3096.5\)) Content-Type: multipart/signed; boundary="Apple-Mail=_ACB17E14-EDA7-44C2-8FD0-6F36EBE3B85B"; protocol="application/pgp-signature"; micalg=pgp-sha512 X-Pgp-Agent: GPGMail 2.6b2 From: Alexey Ivanov In-Reply-To: Date: Fri, 5 Feb 2016 22:50:20 -0800 Cc: "freebsd-performance@freebsd.org" , Adrian Chadd , Allan Jude Message-Id: References: <56B275EC.2060604@freebsd.org> To: "Meyer, Wolfgang" X-Mailer: Apple Mail (2.3096.5) X-BeenThere: freebsd-performance@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: Performance/tuning List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 06 Feb 2016 06:50:27 -0000 --Apple-Mail=_ACB17E14-EDA7-44C2-8FD0-6F36EBE3B85B Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset=us-ascii In Linux/Intel I would use following methodology for performance = analysis: Hardware: * turbostat Look for C/P states for cores, frequencies, number of SMIs. [1] * cpufreq-info Look for current driver, frequencies, and governor. * atop Look for interrupt distribution across cores Look for context switches, interrupts. * ethtool -S for stats, look for errors, drops, overruns, missed interrupts, etc -k for offloads, enable GRO/GSO, rss/rps/rfs/xps[0] -g for ring sizes, increase -c for interrupt coalescing Kernel: * /proc/net/softirq [2] and /proc/interrupts [3] Again, distribution, missed, delayed interrupts (optional) NUMA-affinity * perf top Look where kernel spends its time. * iptables Look if there are rules (if any) that may affect performance. * netstat -s, netstat -m Look for error counters and buffer counts * sysctl / grub So much to tweak here. Try increasing hashtable sizes, playing with = memory buffers and other limits. BSD has alternatives to most of these, e.g. perf -> hwpmc, ethtool -> = ifconfig, softirq -> netisr, menu.lst -> loader.conf In your case I suppose turning on interrupt coalescing along with = enabling hardware offloads may help. [0] Comparing mutiqueue support Linux vs FreeBSD = https://wiki.freebsd.org/201305DevSummit/NetworkReceivePerformance/Compari= ngMutiqueueSupportLinuxvsFreeBSD [1] You can pin your processor to a specific C-state: https://gist.github.com/SaveTheRbtz/f5e8d1ca7b55b6a7897b [2] You can analyze that data with: https://gist.github.com/SaveTheRbtz/172b2e2eb3cbd96b598d [3] You can set affinity with: https://gist.github.com/SaveTheRbtz/8875474 PS. Sorry for so many Linuxisms on freebsd-performance@ > On Feb 5, 2016, at 2:57 PM, Adrian Chadd = wrote: >=20 > On 5 February 2016 at 10:15, Meyer, Wolfgang = wrote: >>=20 >>=20 >>> -----Original Message----- >>> From: owner-freebsd-performance@freebsd.org [mailto:owner-freebsd- >>> performance@freebsd.org] On Behalf Of Allan Jude >>> Sent: Mittwoch, 3. Februar 2016 22:50 >>> To: freebsd-performance@freebsd.org >>> Subject: Re: ixgbe: Network performance tuning (#TCP connections) >>>=20 >>> On 2016-02-03 16:34, Adrian Chadd wrote: >>>> hi, >>>>=20 >>>> can you share your testing program source? >>>>=20 >>>>=20 >>>> -a >>>>=20 >>>=20 >>> I have a Dual E5-2630 v3 (2x8x 2.40ghz (+HT)) with the Intel = X540-AT2 that I >>> can try to replicate this one to help debug it. >>>=20 >>> -- >>> Allan Jude >>=20 >> I'll try to do some polishing and removal of cruft next week than I = hope I will feel comfortable putting it to the public :-) >>=20 >> Not that they are overly sophisticated programs, just some test = set-up created in the past and over time one gets used to use it as a = sort of private "benchmark". >=20 > Please do - it'd be good to see what you're doing and figure out > what's causing the poor behaviour. >=20 > Also having more public benchmarks for testing and reproducibility is > always good. :) >=20 >=20 >=20 > -adrian >=20 >>=20 >> Regards, >> Wolfgang Meyer >>=20 >>=20 >> ________________________________ >>=20 >> Follow HOB: >>=20 >> - HOB: http://www.hob.de/redirect/hob.html >> - Xing: http://www.hob.de/redirect/xing.html >> - LinkedIn: http://www.hob.de/redirect/linkedin.html >> - HOBLink Mobile: http://www.hob.de/redirect/hoblinkmobile.html >> - Facebook: http://www.hob.de/redirect/facebook.html >> - Twitter: http://www.hob.de/redirect/twitter.html >> - YouTube: http://www.hob.de/redirect/youtube.html >> - E-Mail: http://www.hob.de/redirect/mail.html >>=20 >>=20 >> HOB GmbH & Co. KG >> Schwadermuehlstr. 3 >> D-90556 Cadolzburg >>=20 >> Geschaeftsfuehrung: Klaus Brandstaetter, Zoran Adamovic >>=20 >> AG Fuerth, HRA 5180 >> Steuer-Nr. 218/163/00107 >> USt-ID-Nr. DE 132747002 >>=20 >> Komplementaerin HOB electronic Beteiligungs GmbH >> AG Fuerth, HRB 3416 >> _______________________________________________ >> freebsd-performance@freebsd.org mailing list >> https://lists.freebsd.org/mailman/listinfo/freebsd-performance >> To unsubscribe, send any mail to = "freebsd-performance-unsubscribe@freebsd.org" > _______________________________________________ > freebsd-performance@freebsd.org mailing list > https://lists.freebsd.org/mailman/listinfo/freebsd-performance > To unsubscribe, send any mail to = "freebsd-performance-unsubscribe@freebsd.org" --Apple-Mail=_ACB17E14-EDA7-44C2-8FD0-6F36EBE3B85B Content-Transfer-Encoding: 7bit Content-Disposition: attachment; filename=signature.asc Content-Type: application/pgp-signature; name=signature.asc Content-Description: Message signed with OpenPGP using GPGMail -----BEGIN PGP SIGNATURE----- Comment: GPGTools - https://gpgtools.org iQIcBAEBCgAGBQJWtZevAAoJECvXQw+IBr2adcsQAJhR1eC1WOal8W+iCO0g+l/6 3SL+8yjbu1CnBkqXO+lHSKpilUbkb66TNTliynfzB/GVAmODClJN+WtJfK9P5hrV i0MUoMtF1jg5GxEZ8gXn5bZvVtZbaZjDsEiGPOPxy17b/AYF73k741/aOl6GA7BS sIMzBopImBCsS5avLm872EqKLHaXvHhElA9ivSqG5X8hLiubE0RDN0bJgP1IZTHV 9WEaef+PCVJnkCpN5RayRO0Cxj+GxEEsq6T/qFqmzZ94m3hT7ZpuXG7k8VENJXpK Wos4ntzzOud+1rc7+2nkAZ16PMhWJKYv1XF/3owRnuCWGEEy9Vsl9/crLkkuTK2C t4wds2QVOqyAQKJ7KngBcSBfpJs873rw3qFL7FQK8ccNCqCSyR5ZFQu9mH0NlH5g F2FuWYzCfi4cqDPjhxSoOobL5oovcYNcZbqGIAECXvZIrrXAmTwv43oloKD4Xh0N QBfLkz7BLiYi6tIDwehE+nJfpBREpGhnzflqwGdHs0ZMcEe8QDiudmZeZEjrTaf4 XUaCpPa2OIiM6qLHitQCRga8LCTe9cI8g17Lg6MfwT/8uJjKwcpIlWIcnLDOEuWK MgCi+LpluSfMTlPK+4OLLMvBW0CcVT/OtJd+UY/8GQLCWfxb2XbNBhJc7IbvT4M3 t0nD8C/ZZDg5yEwkjzbV =4pta -----END PGP SIGNATURE----- --Apple-Mail=_ACB17E14-EDA7-44C2-8FD0-6F36EBE3B85B--