From owner-svn-src-head@freebsd.org Fri Aug 26 14:49:36 2016 Return-Path: Delivered-To: svn-src-head@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 8BC41B71DD0; Fri, 26 Aug 2016 14:49:36 +0000 (UTC) (envelope-from slw@zxy.spb.ru) Received: from zxy.spb.ru (zxy.spb.ru [195.70.199.98]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 4A6EE8B; Fri, 26 Aug 2016 14:49:36 +0000 (UTC) (envelope-from slw@zxy.spb.ru) Received: from slw by zxy.spb.ru with local (Exim 4.86 (FreeBSD)) (envelope-from ) id 1bdIRi-0006mC-4t; Fri, 26 Aug 2016 17:49:26 +0300 Date: Fri, 26 Aug 2016 17:49:26 +0300 From: Slawa Olhovchenkov To: Bruce Simpson Cc: Ryan Stone , "svn-src-head@freebsd.org" , Ryan Stone , "src-committers@freebsd.org" , "svn-src-all@freebsd.org" , Adrian Chadd Subject: Re: svn commit: r304436 - in head: . sys/netinet Message-ID: <20160826144926.GE88122@zxy.spb.ru> References: <20160820184506.GV8192@zxy.spb.ru> <0f42c5fb-f930-c6e3-75d6-df97f67c201d@fastmail.net> <20160820204106.GW8192@zxy.spb.ru> <0acba141-4701-d9c2-0ddb-46d1f60ff55b@fastmail.net> <20160820220510.GX8192@zxy.spb.ru> <8ac23bd1-dcb3-7c64-f195-5039f9af0eaf@fastmail.net> <20160821000400.GY8192@zxy.spb.ru> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <20160821000400.GY8192@zxy.spb.ru> User-Agent: Mutt/1.5.24 (2015-08-30) X-SA-Exim-Connect-IP: X-SA-Exim-Mail-From: slw@zxy.spb.ru X-SA-Exim-Scanned: No (on zxy.spb.ru); SAEximRunCond expanded to false X-BeenThere: svn-src-head@freebsd.org X-Mailman-Version: 2.1.22 Precedence: list List-Id: SVN commit messages for the src tree for head/-current List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 26 Aug 2016 14:49:36 -0000 On Sun, Aug 21, 2016 at 03:04:00AM +0300, Slawa Olhovchenkov wrote: > On Sun, Aug 21, 2016 at 12:25:46AM +0100, Bruce Simpson wrote: > > > On 20/08/16 23:05, Slawa Olhovchenkov wrote: > > > I am think this substitution is very bad idea (by design). > > > Also, on transmit side this is must be irrelevant on received L2 > > > header (and this in many cases this is will be L2 unicast packet). For > > > other cases packet will be created on host and don't have any received > > > information. > > > > > > > Whilst I agree with your concerns about multipoint, I support the > > motivation behind Ryan's original change: optimize the common case. > > Oh, common case... > I am have pmc profiling for TCP output and see on this SVG picture and > don't find any simple way. > You want to watch too? At time peak network traffic (more then 25K connections, about 20Gbit total traffic) half of cores fully utilised by network stack. This is flamegraph from one core: http://zxy.spb.ru/cpu10.svg This is same, but stack cut of at ixgbe_rxeof for more unified tcp/ip stack view http://zxy.spb.ru/cpu10u.svg Top 3 used lines is: 7036 0xffffffff804bf02d atomic_cmpset_long /usr/obj/usr/src/sys/VSTREAM/./machine/atomic.h:163 static __inline int atomic_cmpset_long(volatile u_long *dst, u_long expect, u_long src) { u_char res; __asm __volatile( " " MPLOCKED " " > " cmpxchgq %3,%1 ; " " sete %0 ; " "# atomic_cmpset_long" : "=q" (res), /* 0 */ "+m" (*dst), /* 1 */ "+a" (expect) /* 2 */ : "r" (src) /* 3 */ : "memory", "cc"); return (res); } 6099 0xffffffff81171963 ?? ??:0 0xffffffff81171940 : mov 0x10(%r15),%rax 0xffffffff81171944 : add $0x8,%rax 0xffffffff81171948 : mov -0x4c(%rbp),%ecx 0xffffffff8117194b : test %cx,%cx 0xffffffff8117194e : mov %rax,0x10(%r15) 0xffffffff81171952 : je 0xffffffff8117198d 0xffffffff81171954 : mov 0x10(%rdi),%rcx 0xffffffff81171958 : mov -0x4c(%rbp),%edx 0xffffffff8117195b : nopl 0x0(%rax,%rax,1) 0xffffffff81171960 : mov (%rcx),%rsi 0xffffffff81171963 : mov %rsi,(%rax) 0xffffffff81171966 : mov 0x8(%rcx),%rsi 0xffffffff8117196a : mov %rsi,0x8(%rax) 0xffffffff8117196e : mov 0x10(%rcx),%rsi 0xffffffff81171972 : mov %rsi,0x10(%rax) 0xffffffff81171976 : mov 0x18(%rcx),%rsi 0xffffffff8117197a : mov %rsi,0x18(%rax) 0xffffffff8117197e : add $0xffffffffffffffe0,%edx 0xffffffff81171981 : add $0x20,%rcx 0xffffffff81171985 : add $0x20,%rax 0xffffffff81171989 : test %edx,%edx 5594 0xffffffff8053395a mb_free_ext /usr/src/sys/kern/uipc_mbuf.c:301 if (*(m->m_ext.ref_cnt) == 1 || I am able collect and process more measure for help to improve FreeBSD network stack. Do you have some idea about this? I am don't see evident and simple points of optimisation :(