Date: Fri, 16 Aug 2013 09:51:20 +0300 From: Alexander Motin <mav@FreeBSD.org> To: Adrian Chadd <adrian@freebsd.org> Cc: Scott Long <scottl@freebsd.org>, Jeff Roberson <jeff@freebsd.org>, ken <ken@freebsd.org>, "freebsd-hackers@freebsd.org" <freebsd-hackers@freebsd.org>, FreeBSD SCSI <freebsd-scsi@freebsd.org>, Steven Hartland <smh@freebsd.org>, "Justin T. Gibbs" <gibbs@freebsd.org> Subject: Re: New CAM locking preview Message-ID: <520DCBE8.9050703@FreeBSD.org> In-Reply-To: <CAJ-VmonD_MJSdc6M_H14k-nWotg1cN1GG1=U7B2OtOnhGDn02g@mail.gmail.com> References: <520D4ADB.50209@FreeBSD.org> <CAJ-VmonD_MJSdc6M_H14k-nWotg1cN1GG1=U7B2OtOnhGDn02g@mail.gmail.com>
next in thread | previous in thread | raw e-mail | index | archive | help
On 16.08.2013 04:12, Adrian Chadd wrote: > Cool! > > I assume you've run this with full witness debugging enabled, to catch > lock ordering issues? Of course! I've endless times switched between debug and normal builds to test both correctness and performance after each change. But more external tests are welcome. > This is great. I look forward to per-CPU, pinned, completion threads > that I can do interesting things with (like schedule aio-sendfile > completions..) > > On 15 August 2013 14:40, Alexander Motin <mav@freebsd.org > <mailto:mav@freebsd.org>> wrote: > > Hi. > > Last weeks I've made substantial progress on my CAM locking work. In > fact, at this moment I think I've tied all loose ends good enough to > consider the new design viable and implementation worth further > testing and bug fixing. So I would like to ask for review of my work > from everybody who interested in CAM internals. > > In short, my idea was to split single per-SIM lock, that creates > huge congestion under high IOPS, into several smaller ones. So > design I've finally chosen includes such locks: > 1) New per-device (per-LUN) locks to protect state of the devices > and respective periphs. In most cases peripheral drivers just use > that lock instead of SIM lock used before, so code modification is > minimal and straightforward. > 2) New per-target lock to protect list of LUNs fetched from the > device. > 3) Old single per-SIM lock to protect SIM driver internals, but > only that. No parts of CAM itself use that lock. Keeping it for SIMs > allows to keep API and hopefully ABI compatibility. Reducing its > scope allows to reduce congestion. > 4) New per-SIM lock to protect SIM and device command queues. That > allows execute queued commands from any context unrelated to other > locks. Also this lock serializes accesses to sim_action() method for > the most of commands, this allows to mostly avoid busy spilling on > SIM lock collision. > 5) New per-bus locks to protect target, device and periphs > reference counters. It allows to create and destroy paths unrelated > to other locks in any possible context. > > Numbers above also define supposed lock ordering: while holding > per-device lock 1) is allowed to request SIM lock 3), but not > backward. Cases where opposite is required (command completions and > async events) are handled via queuing events via several completion > threads. The rest of locks are self-contained and does not really > suppose cascading. > > All these changes combined with GEOM direct dispatch (it will be > next separate project) allow to double system performance in disk > I/O microbenchmarks, comparing to present head, same as it was > announced on 2013-05 DevSummit: > http://people.freebsd.org/~__mav/camlock.pdf > <http://people.freebsd.org/~mav/camlock.pdf>; . Tests without GEOM > changes also show performance improvement, but limited by heavy > bottleneck at the GEOM g_up/g_down threads at the level of 5-20%. > > Project sources could be found at SVN projects/camlock branch: > http://svnweb.freebsd.org/__base/projects/camlock/ > <http://svnweb.freebsd.org/base/projects/camlock/>; . Many early > changes from that branch are already integrated to head, so to > simplify review the rest patches for changes before r254059 were > manually remade and could be found here: > http://people.freebsd.org/~__mav/camlock_patches/ > <http://people.freebsd.org/~mav/camlock_patches/>; . > > These changes do not require controller driver modifications, > keeping KPIs and hopefully KBIs intact, but create base for later > work to use multiqueue capabilities of new controllers. > > This work is sponsored by iXsystems, Inc. -- Alexander Motin
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?520DCBE8.9050703>