Date: Fri, 5 Feb 2016 22:50:20 -0800 From: Alexey Ivanov <savetherbtz@gmail.com> To: "Meyer, Wolfgang" <wolfgang.meyer@hob.de> Cc: "freebsd-performance@freebsd.org" <freebsd-performance@freebsd.org>, Adrian Chadd <adrian.chadd@gmail.com>, Allan Jude <allanjude@freebsd.org> Subject: Re: ixgbe: Network performance tuning (#TCP connections) Message-ID: <F7C49408-4B1C-47BD-9145-467EF7B3774A@gmail.com> In-Reply-To: <CAJ-VmoniMZd6zkaU8Z_xh=UrUYGpS%2BB1GCm1b51=A4crR5punQ@mail.gmail.com> References: <EC88118611AE564AB0B10C6A4569004D0137D57AEB@HOBEX11.hob.de> <CAJ-Vmo=Jy1wDqb-7cPt=9DC8Rfb9o8d0APwqpG-NH-vBKg8prQ@mail.gmail.com> <56B275EC.2060604@freebsd.org> <EC88118611AE564AB0B10C6A4569004D0137D58065@HOBEX11.hob.de> <CAJ-VmoniMZd6zkaU8Z_xh=UrUYGpS%2BB1GCm1b51=A4crR5punQ@mail.gmail.com>
next in thread | previous in thread | raw e-mail | index | archive | help
[-- Attachment #1 --] In Linux/Intel I would use following methodology for performance analysis: Hardware: * turbostat Look for C/P states for cores, frequencies, number of SMIs. [1] * cpufreq-info Look for current driver, frequencies, and governor. * atop Look for interrupt distribution across cores Look for context switches, interrupts. * ethtool -S for stats, look for errors, drops, overruns, missed interrupts, etc -k for offloads, enable GRO/GSO, rss/rps/rfs/xps[0] -g for ring sizes, increase -c for interrupt coalescing Kernel: * /proc/net/softirq [2] and /proc/interrupts [3] Again, distribution, missed, delayed interrupts (optional) NUMA-affinity * perf top Look where kernel spends its time. * iptables Look if there are rules (if any) that may affect performance. * netstat -s, netstat -m Look for error counters and buffer counts * sysctl / grub So much to tweak here. Try increasing hashtable sizes, playing with memory buffers and other limits. BSD has alternatives to most of these, e.g. perf -> hwpmc, ethtool -> ifconfig, softirq -> netisr, menu.lst -> loader.conf In your case I suppose turning on interrupt coalescing along with enabling hardware offloads may help. [0] Comparing mutiqueue support Linux vs FreeBSD https://wiki.freebsd.org/201305DevSummit/NetworkReceivePerformance/ComparingMutiqueueSupportLinuxvsFreeBSD [1] You can pin your processor to a specific C-state: https://gist.github.com/SaveTheRbtz/f5e8d1ca7b55b6a7897b [2] You can analyze that data with: https://gist.github.com/SaveTheRbtz/172b2e2eb3cbd96b598d [3] You can set affinity with: https://gist.github.com/SaveTheRbtz/8875474 PS. Sorry for so many Linuxisms on freebsd-performance@ > On Feb 5, 2016, at 2:57 PM, Adrian Chadd <adrian.chadd@gmail.com> wrote: > > On 5 February 2016 at 10:15, Meyer, Wolfgang <wolfgang.meyer@hob.de> wrote: >> >> >>> -----Original Message----- >>> From: owner-freebsd-performance@freebsd.org [mailto:owner-freebsd- >>> performance@freebsd.org] On Behalf Of Allan Jude >>> Sent: Mittwoch, 3. Februar 2016 22:50 >>> To: freebsd-performance@freebsd.org >>> Subject: Re: ixgbe: Network performance tuning (#TCP connections) >>> >>> On 2016-02-03 16:34, Adrian Chadd wrote: >>>> hi, >>>> >>>> can you share your testing program source? >>>> >>>> >>>> -a >>>> >>> >>> I have a Dual E5-2630 v3 (2x8x 2.40ghz (+HT)) with the Intel X540-AT2 that I >>> can try to replicate this one to help debug it. >>> >>> -- >>> Allan Jude >> >> I'll try to do some polishing and removal of cruft next week than I hope I will feel comfortable putting it to the public :-) >> >> Not that they are overly sophisticated programs, just some test set-up created in the past and over time one gets used to use it as a sort of private "benchmark". > > Please do - it'd be good to see what you're doing and figure out > what's causing the poor behaviour. > > Also having more public benchmarks for testing and reproducibility is > always good. :) > > > > -adrian > >> >> Regards, >> Wolfgang Meyer >> >> >> ________________________________ >> >> Follow HOB: >> >> - HOB: http://www.hob.de/redirect/hob.html >> - Xing: http://www.hob.de/redirect/xing.html >> - LinkedIn: http://www.hob.de/redirect/linkedin.html >> - HOBLink Mobile: http://www.hob.de/redirect/hoblinkmobile.html >> - Facebook: http://www.hob.de/redirect/facebook.html >> - Twitter: http://www.hob.de/redirect/twitter.html >> - YouTube: http://www.hob.de/redirect/youtube.html >> - E-Mail: http://www.hob.de/redirect/mail.html >> >> >> HOB GmbH & Co. KG >> Schwadermuehlstr. 3 >> D-90556 Cadolzburg >> >> Geschaeftsfuehrung: Klaus Brandstaetter, Zoran Adamovic >> >> AG Fuerth, HRA 5180 >> Steuer-Nr. 218/163/00107 >> USt-ID-Nr. DE 132747002 >> >> Komplementaerin HOB electronic Beteiligungs GmbH >> AG Fuerth, HRB 3416 >> _______________________________________________ >> freebsd-performance@freebsd.org mailing list >> https://lists.freebsd.org/mailman/listinfo/freebsd-performance >> To unsubscribe, send any mail to "freebsd-performance-unsubscribe@freebsd.org" > _______________________________________________ > freebsd-performance@freebsd.org mailing list > https://lists.freebsd.org/mailman/listinfo/freebsd-performance > To unsubscribe, send any mail to "freebsd-performance-unsubscribe@freebsd.org" [-- Attachment #2 --] -----BEGIN PGP SIGNATURE----- Comment: GPGTools - https://gpgtools.org iQIcBAEBCgAGBQJWtZevAAoJECvXQw+IBr2adcsQAJhR1eC1WOal8W+iCO0g+l/6 3SL+8yjbu1CnBkqXO+lHSKpilUbkb66TNTliynfzB/GVAmODClJN+WtJfK9P5hrV i0MUoMtF1jg5GxEZ8gXn5bZvVtZbaZjDsEiGPOPxy17b/AYF73k741/aOl6GA7BS sIMzBopImBCsS5avLm872EqKLHaXvHhElA9ivSqG5X8hLiubE0RDN0bJgP1IZTHV 9WEaef+PCVJnkCpN5RayRO0Cxj+GxEEsq6T/qFqmzZ94m3hT7ZpuXG7k8VENJXpK Wos4ntzzOud+1rc7+2nkAZ16PMhWJKYv1XF/3owRnuCWGEEy9Vsl9/crLkkuTK2C t4wds2QVOqyAQKJ7KngBcSBfpJs873rw3qFL7FQK8ccNCqCSyR5ZFQu9mH0NlH5g F2FuWYzCfi4cqDPjhxSoOobL5oovcYNcZbqGIAECXvZIrrXAmTwv43oloKD4Xh0N QBfLkz7BLiYi6tIDwehE+nJfpBREpGhnzflqwGdHs0ZMcEe8QDiudmZeZEjrTaf4 XUaCpPa2OIiM6qLHitQCRga8LCTe9cI8g17Lg6MfwT/8uJjKwcpIlWIcnLDOEuWK MgCi+LpluSfMTlPK+4OLLMvBW0CcVT/OtJd+UY/8GQLCWfxb2XbNBhJc7IbvT4M3 t0nD8C/ZZDg5yEwkjzbV =4pta -----END PGP SIGNATURE-----
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?F7C49408-4B1C-47BD-9145-467EF7B3774A>