From owner-freebsd-hackers@FreeBSD.ORG Fri Aug 16 06:51:27 2013 Return-Path: Delivered-To: freebsd-hackers@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTP id 2930AD5; Fri, 16 Aug 2013 06:51:27 +0000 (UTC) (envelope-from mavbsd@gmail.com) Received: from mail-ee0-x232.google.com (mail-ee0-x232.google.com [IPv6:2a00:1450:4013:c00::232]) (using TLSv1 with cipher ECDHE-RSA-RC4-SHA (128/128 bits)) (No client certificate requested) by mx1.freebsd.org (Postfix) with ESMTPS id 8910F23BE; Fri, 16 Aug 2013 06:51:25 +0000 (UTC) Received: by mail-ee0-f50.google.com with SMTP id d51so768042eek.37 for ; Thu, 15 Aug 2013 23:51:23 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=sender:message-id:date:from:user-agent:mime-version:to:cc:subject :references:in-reply-to:content-type:content-transfer-encoding; bh=KnwRBAjKnIuR3HOQY/riLwATfVN+0cHyD8Je4BbaQb0=; b=tEL4dRivnVqSd9ABdiJRI1703tNB6meeYFHwaCdRD29PhK9s1paMWLyOj212wIHCXf 150+zMDUtfu8kshqajvT8CDAVXTvNsWbKKeoBgA+3X9N2jFoMj2IILyFpDAvk/Bxp9g5 NzpXVJgQUFTRtsMNDxb0qtW/6Rk8g0VdPgyggszmDYIUaLwRdR/aQbWWqrx7AP4D6njM cE5RG9EYIl+0i+T4rEIlWycQLd/BgJlrL2l7dcZ5fb0jPLE6vjaw7VMy5gK5WDyseoiw /X4AJGraTCpRujGgcdO/fAJzxMRbr0S0G87iqoGnRDARCvEvkYjA3QzmLnvtyYdMooxr 9GLQ== X-Received: by 10.15.76.71 with SMTP id m47mr26109eey.85.1376635883909; Thu, 15 Aug 2013 23:51:23 -0700 (PDT) Received: from mavbook.mavhome.dp.ua ([37.229.21.195]) by mx.google.com with ESMTPSA id bq1sm210635eeb.9.1969.12.31.16.00.00 (version=TLSv1 cipher=ECDHE-RSA-RC4-SHA bits=128/128); Thu, 15 Aug 2013 23:51:22 -0700 (PDT) Sender: Alexander Motin Message-ID: <520DCBE8.9050703@FreeBSD.org> Date: Fri, 16 Aug 2013 09:51:20 +0300 From: Alexander Motin User-Agent: Mozilla/5.0 (X11; FreeBSD amd64; rv:17.0) Gecko/20130616 Thunderbird/17.0.6 MIME-Version: 1.0 To: Adrian Chadd Subject: Re: New CAM locking preview References: <520D4ADB.50209@FreeBSD.org> In-Reply-To: Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit Cc: Scott Long , Jeff Roberson , ken , "freebsd-hackers@freebsd.org" , FreeBSD SCSI , Steven Hartland , "Justin T. Gibbs" X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 16 Aug 2013 06:51:27 -0000 On 16.08.2013 04:12, Adrian Chadd wrote: > Cool! > > I assume you've run this with full witness debugging enabled, to catch > lock ordering issues? Of course! I've endless times switched between debug and normal builds to test both correctness and performance after each change. But more external tests are welcome. > This is great. I look forward to per-CPU, pinned, completion threads > that I can do interesting things with (like schedule aio-sendfile > completions..) > > On 15 August 2013 14:40, Alexander Motin > wrote: > > Hi. > > Last weeks I've made substantial progress on my CAM locking work. In > fact, at this moment I think I've tied all loose ends good enough to > consider the new design viable and implementation worth further > testing and bug fixing. So I would like to ask for review of my work > from everybody who interested in CAM internals. > > In short, my idea was to split single per-SIM lock, that creates > huge congestion under high IOPS, into several smaller ones. So > design I've finally chosen includes such locks: > 1) New per-device (per-LUN) locks to protect state of the devices > and respective periphs. In most cases peripheral drivers just use > that lock instead of SIM lock used before, so code modification is > minimal and straightforward. > 2) New per-target lock to protect list of LUNs fetched from the > device. > 3) Old single per-SIM lock to protect SIM driver internals, but > only that. No parts of CAM itself use that lock. Keeping it for SIMs > allows to keep API and hopefully ABI compatibility. Reducing its > scope allows to reduce congestion. > 4) New per-SIM lock to protect SIM and device command queues. That > allows execute queued commands from any context unrelated to other > locks. Also this lock serializes accesses to sim_action() method for > the most of commands, this allows to mostly avoid busy spilling on > SIM lock collision. > 5) New per-bus locks to protect target, device and periphs > reference counters. It allows to create and destroy paths unrelated > to other locks in any possible context. > > Numbers above also define supposed lock ordering: while holding > per-device lock 1) is allowed to request SIM lock 3), but not > backward. Cases where opposite is required (command completions and > async events) are handled via queuing events via several completion > threads. The rest of locks are self-contained and does not really > suppose cascading. > > All these changes combined with GEOM direct dispatch (it will be > next separate project) allow to double system performance in disk > I/O microbenchmarks, comparing to present head, same as it was > announced on 2013-05 DevSummit: > http://people.freebsd.org/~__mav/camlock.pdf > . Tests without GEOM > changes also show performance improvement, but limited by heavy > bottleneck at the GEOM g_up/g_down threads at the level of 5-20%. > > Project sources could be found at SVN projects/camlock branch: > http://svnweb.freebsd.org/__base/projects/camlock/ > . Many early > changes from that branch are already integrated to head, so to > simplify review the rest patches for changes before r254059 were > manually remade and could be found here: > http://people.freebsd.org/~__mav/camlock_patches/ > . > > These changes do not require controller driver modifications, > keeping KPIs and hopefully KBIs intact, but create base for later > work to use multiqueue capabilities of new controllers. > > This work is sponsored by iXsystems, Inc. -- Alexander Motin