From owner-freebsd-current@FreeBSD.ORG Sat Feb 24 11:47:35 2007 Return-Path: X-Original-To: current@FreeBSD.org Delivered-To: freebsd-current@FreeBSD.ORG Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id DBB1816A400 for ; Sat, 24 Feb 2007 11:47:34 +0000 (UTC) (envelope-from rwatson@FreeBSD.org) Received: from cyrus.watson.org (cyrus.watson.org [209.31.154.42]) by mx1.freebsd.org (Postfix) with ESMTP id B30E913C471 for ; Sat, 24 Feb 2007 11:47:34 +0000 (UTC) (envelope-from rwatson@FreeBSD.org) Received: from fledge.watson.org (fledge.watson.org [209.31.154.41]) by cyrus.watson.org (Postfix) with ESMTP id 4935446FBA; Sat, 24 Feb 2007 06:47:34 -0500 (EST) Date: Sat, 24 Feb 2007 11:47:34 +0000 (GMT) From: Robert Watson X-X-Sender: robert@fledge.watson.org To: current@FreeBSD.org Message-ID: <20070224095324.K36322@fledge.watson.org> MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII; format=flowed Cc: performance@FreeBSD.org Subject: Fine-grained locking for UNIX domain sockets: patch updated X-BeenThere: freebsd-current@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Discussions about the use of FreeBSD-current List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 24 Feb 2007 11:47:35 -0000 As part of Kris and Jeff's recent work on improving MySQL scalability on FreeBSD, I've updated my fine-grained locking patch for UNIX domain sockets to a more recent 7-CURRENT: http://www.watson.org/~robert/freebsd/netperf/20070224-uds-fine-grain.diff This patch replaces the global mutex with a global rwlock, and introduces per-unpcb locks to protect the state of each socket at the UNIX domain socket layer. Depending on the protocol operation, either the per-unpcb mutex is sufficient, or global read or write locks may also be required. The role of the global lock is to protect the topology of the connections between sockets, protect vnode v_socket pointers, and to prevent lock order issues in the event the per-unpcb lock of more than one socket must be acquired at a time. For example, exclusive global locking is required during bind() and connect() to protect v_socket and change the topology of connected sockets, whereas recv() requires only the unpcb lock of the receiving socket. send() may require either just an unpcb lock or a global lock if send() is implicitly connecting (sendto()) or disconnecting (PRUS_EOF) a socket. This is basically the same patch that I have been maintaining since 2005 and posted at intermittent intervals when there have been reports of negative performance due to high contention on the UNIX domain socket subsystem lock. I continue to look for performance feedback, especially from workloads that are heavy consumers of UNIX domain sockets (such as MySQL). My experience has been that it gives a small performance hit on workloads and in configurations without much contention/scalability, but a measurable performance benefit with anything more substantial. Decrease in lower end workloads and configurations is at least in part due to the overhead of some additional locking vs. the benefits of reduced contention. Something that I am particularly interested in looking for is little or no performance hit on UP. This is a fairly heavily tested and refined patch, so the issue in deciding whether or not to commit this is its real-world effect on performance. Robert N M Watson Computer Laboratory University of Cambridge