From owner-freebsd-net@FreeBSD.ORG Fri Dec 23 19:00:52 2011 Return-Path: Delivered-To: net@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id AE487106566B; Fri, 23 Dec 2011 19:00:52 +0000 (UTC) (envelope-from jhb@freebsd.org) Received: from cyrus.watson.org (cyrus.watson.org [65.122.17.42]) by mx1.freebsd.org (Postfix) with ESMTP id 8147E8FC21; Fri, 23 Dec 2011 19:00:52 +0000 (UTC) Received: from bigwig.baldwin.cx (bigwig.baldwin.cx [96.47.65.170]) by cyrus.watson.org (Postfix) with ESMTPSA id 0E17D46B09; Fri, 23 Dec 2011 14:00:52 -0500 (EST) Received: from jhbbsd.localnet (unknown [209.249.190.124]) by bigwig.baldwin.cx (Postfix) with ESMTPSA id 90CD8B969; Fri, 23 Dec 2011 14:00:51 -0500 (EST) From: John Baldwin To: Arnaud Lacombe Date: Fri, 23 Dec 2011 13:56:04 -0500 User-Agent: KMail/1.13.5 (FreeBSD/8.2-CBSD-20110714-p8; KDE/4.5.5; amd64; ; ) References: <201112221130.01823.jhb@freebsd.org> In-Reply-To: MIME-Version: 1.0 Content-Type: Text/Plain; charset="iso-8859-1" Content-Transfer-Encoding: 7bit Message-Id: <201112231356.04668.jhb@freebsd.org> X-Greylist: Sender succeeded SMTP AUTH, not delayed by milter-greylist-4.2.7 (bigwig.baldwin.cx); Fri, 23 Dec 2011 14:00:51 -0500 (EST) Cc: Robert Watson , net@freebsd.org Subject: Re: Transitioning if_addr_lock to an rwlock X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 23 Dec 2011 19:00:52 -0000 On Friday, December 23, 2011 12:13:44 pm Arnaud Lacombe wrote: > Hi, > > On Thu, Dec 22, 2011 at 11:30 AM, John Baldwin wrote: > > Another bit of lock contention I ran into between a device driver doing slow > > MAC filter updates and the receive path is IF_ADDR_LOCK(). NIC device drivers > > typically hold this lock while iterating the list of multicast addresses to > > program their MAC filter. OTOH, ip_input() uses this lock to check to see if > > an incoming packet is a broadcast packet or not. So even with the pcbinfo > > contention from my previous patch addressed, I still ran into a problem with > > IF_ADDR_LOCK(). We already have some partial support for making this lock be > > an rwlock (the APIs that drivers now use implies an rlock), so I finished the > > transition and checked various non-driver users to see which ones could use a > > read lock (most uses can). The current patch I have for this is on 8, but if > > folks think this is a good idea I can work on porting it to HEAD. For HEAD > > the strategy I would use would be to split this up into 2 phases: > > > > 1) Add IF_ADDR locking macros to differentiate read locks vs write locks along > > with appropriate assertion macros. Update current users of the locking > > macros to use either read or write locks explicitly. To preserve KPI, > > the existing LOCK/UNLOCK macros would map to write lock operations. In > > the initial patch, the locking would still be implemented via a mtx. > > > > 2) Replace the mutex with an rwlock and update the locking macros as > > appropriate. > > > out of curiosity, what do you expect from the conversion ? performance > improvement ? latency improvement ? > > Does this particular lock show up in any significant way in lock > profiling that make the change noticeable ? More of a latency / don't drop packets under load improvement. Specifically, I have a latency-sensitive workload where packets are being dropped on the card because the ISR is contending on this lock with another thread on another CPU that is joining a multicast group on a socket causing the MAC filters on the same NIC to be updated. (The packets are dropped because the ISR can't drain the packets fast enough while it is stuck on the lock, this causes the receive ring to fill up and eventually the card runs out of room and drops incoming traffic.) -- John Baldwin