From owner-freebsd-hackers@FreeBSD.ORG Fri Aug 16 08:16:11 2013 Return-Path: Delivered-To: freebsd-hackers@FreeBSD.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTP id 22AC7F0F; Fri, 16 Aug 2013 08:16:11 +0000 (UTC) (envelope-from jlh@FreeBSD.org) Received: from caravan.chchile.org (caravan.chchile.org [178.32.125.136]) (using TLSv1 with cipher ADH-CAMELLIA256-SHA (256/256 bits)) (No client certificate requested) by mx1.freebsd.org (Postfix) with ESMTPS id A7B752743; Fri, 16 Aug 2013 08:16:10 +0000 (UTC) Received: by caravan.chchile.org (Postfix, from userid 1000) id 504B7B70D2; Fri, 16 Aug 2013 08:16:03 +0000 (UTC) Date: Fri, 16 Aug 2013 10:16:03 +0200 From: Jeremie Le Hen To: Alexander Motin Subject: Re: New CAM locking preview Message-ID: <20130816081603.GA4984@caravan.chchile.org> Mail-Followup-To: Alexander Motin , FreeBSD SCSI , freebsd-hackers@FreeBSD.org, "Justin T. Gibbs" , Scott Long , ken , Jeff Roberson , Steven Hartland References: <520D4ADB.50209@FreeBSD.org> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <520D4ADB.50209@FreeBSD.org> User-Agent: Mutt/1.5.21 (2010-09-15) Cc: Scott Long , Jeff Roberson , ken , freebsd-hackers@FreeBSD.org, FreeBSD SCSI , Steven Hartland , "Justin T. Gibbs" X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 16 Aug 2013 08:16:11 -0000 On Fri, Aug 16, 2013 at 12:40:43AM +0300, Alexander Motin wrote: > Hi. > > Last weeks I've made substantial progress on my CAM locking work. In > fact, at this moment I think I've tied all loose ends good enough to > consider the new design viable and implementation worth further testing > and bug fixing. So I would like to ask for review of my work from > everybody who interested in CAM internals. > > In short, my idea was to split single per-SIM lock, that creates huge > congestion under high IOPS, into several smaller ones. So design I've > finally chosen includes such locks: > 1) New per-device (per-LUN) locks to protect state of the devices and > respective periphs. In most cases peripheral drivers just use that lock > instead of SIM lock used before, so code modification is minimal and > straightforward. > 2) New per-target lock to protect list of LUNs fetched from the device. > 3) Old single per-SIM lock to protect SIM driver internals, but only > that. No parts of CAM itself use that lock. Keeping it for SIMs allows > to keep API and hopefully ABI compatibility. Reducing its scope allows > to reduce congestion. > 4) New per-SIM lock to protect SIM and device command queues. That > allows execute queued commands from any context unrelated to other > locks. Also this lock serializes accesses to sim_action() method for the > most of commands, this allows to mostly avoid busy spilling on SIM lock > collision. > 5) New per-bus locks to protect target, device and periphs reference > counters. It allows to create and destroy paths unrelated to other locks > in any possible context. > > Numbers above also define supposed lock ordering: while holding > per-device lock 1) is allowed to request SIM lock 3), but not backward. > Cases where opposite is required (command completions and async events) > are handled via queuing events via several completion threads. The rest > of locks are self-contained and does not really suppose cascading. > > All these changes combined with GEOM direct dispatch (it will be next > separate project) allow to double system performance in disk I/O > microbenchmarks, comparing to present head, same as it was announced on > 2013-05 DevSummit: http://people.freebsd.org/~mav/camlock.pdf . Tests > without GEOM changes also show performance improvement, but limited by > heavy bottleneck at the GEOM g_up/g_down threads at the level of 5-20%. > > Project sources could be found at SVN projects/camlock branch: > http://svnweb.freebsd.org/base/projects/camlock/ . Many early changes > from that branch are already integrated to head, so to simplify review > the rest patches for changes before r254059 were manually remade and > could be found here: http://people.freebsd.org/~mav/camlock_patches/ . > > These changes do not require controller driver modifications, keeping > KPIs and hopefully KBIs intact, but create base for later work to use > multiqueue capabilities of new controllers. > > This work is sponsored by iXsystems, Inc. Excellent, thanks to both you and iXsystems. I'm eager to see everything merged to -CURRENT before the code slush ;). -- Jeremie Le Hen Scientists say the world is made up of Protons, Neutrons and Electrons. They forgot to mention Morons.