From owner-freebsd-hackers@FreeBSD.ORG Thu Aug 15 21:40:49 2013 Return-Path: Delivered-To: freebsd-hackers@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTP id 82F16BCE; Thu, 15 Aug 2013 21:40:49 +0000 (UTC) (envelope-from mavbsd@gmail.com) Received: from mail-ea0-x22a.google.com (mail-ea0-x22a.google.com [IPv6:2a00:1450:4013:c01::22a]) (using TLSv1 with cipher ECDHE-RSA-RC4-SHA (128/128 bits)) (No client certificate requested) by mx1.freebsd.org (Postfix) with ESMTPS id 2C3882BA0; Thu, 15 Aug 2013 21:40:48 +0000 (UTC) Received: by mail-ea0-f170.google.com with SMTP id h14so634130eak.29 for ; Thu, 15 Aug 2013 14:40:46 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=sender:message-id:date:from:user-agent:mime-version:to:subject :content-type:content-transfer-encoding; bh=LjxekYHvC+iIKiI94sZqU16MN2ruOFa5AWPUBnQ+vtM=; b=js3WYVdRQFzSNF1xwd4hhesO53tCk9qjFYpYjKhwb1goSQcXysQkN3N9J4smm1lKrQ HkO4bXrlDb9QHLPBFhBU3ZJ5yoyOdrFipGVbG19IsNvy5hLZQxEbBwEJ1YmacLHNlJ87 2kDneupclCz583g3VVxSiuzg9q+UgNQSUBwJPeRPnBgJlSMjBYGFyVbnwlpNw+WzTqGe a0uAwObk3zBDRI5IWNV7yBTZD3tnY+B6BUj+u4WrVk9ZfjPJyMRmr+lSQhB8rP68Th9j z1U8iFDFEmRJCtaZpiFXvkFXQRPcAAuQqB3M96gvGsDG+dsY1r4dtQ/NVxe3WvZbQIpP xb+w== X-Received: by 10.15.90.132 with SMTP id q4mr180193eez.98.1376602846545; Thu, 15 Aug 2013 14:40:46 -0700 (PDT) Received: from mavbook.mavhome.dp.ua ([37.229.21.195]) by mx.google.com with ESMTPSA id k7sm1995775eeg.13.2013.08.15.14.40.44 for (version=TLSv1 cipher=ECDHE-RSA-RC4-SHA bits=128/128); Thu, 15 Aug 2013 14:40:45 -0700 (PDT) Sender: Alexander Motin Message-ID: <520D4ADB.50209@FreeBSD.org> Date: Fri, 16 Aug 2013 00:40:43 +0300 From: Alexander Motin User-Agent: Mozilla/5.0 (X11; FreeBSD amd64; rv:17.0) Gecko/20130616 Thunderbird/17.0.6 MIME-Version: 1.0 To: FreeBSD SCSI , freebsd-hackers@FreeBSD.org, "Justin T. Gibbs" , Scott Long , ken , Jeff Roberson , Steven Hartland Subject: New CAM locking preview Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 15 Aug 2013 21:40:49 -0000 Hi. Last weeks I've made substantial progress on my CAM locking work. In fact, at this moment I think I've tied all loose ends good enough to consider the new design viable and implementation worth further testing and bug fixing. So I would like to ask for review of my work from everybody who interested in CAM internals. In short, my idea was to split single per-SIM lock, that creates huge congestion under high IOPS, into several smaller ones. So design I've finally chosen includes such locks: 1) New per-device (per-LUN) locks to protect state of the devices and respective periphs. In most cases peripheral drivers just use that lock instead of SIM lock used before, so code modification is minimal and straightforward. 2) New per-target lock to protect list of LUNs fetched from the device. 3) Old single per-SIM lock to protect SIM driver internals, but only that. No parts of CAM itself use that lock. Keeping it for SIMs allows to keep API and hopefully ABI compatibility. Reducing its scope allows to reduce congestion. 4) New per-SIM lock to protect SIM and device command queues. That allows execute queued commands from any context unrelated to other locks. Also this lock serializes accesses to sim_action() method for the most of commands, this allows to mostly avoid busy spilling on SIM lock collision. 5) New per-bus locks to protect target, device and periphs reference counters. It allows to create and destroy paths unrelated to other locks in any possible context. Numbers above also define supposed lock ordering: while holding per-device lock 1) is allowed to request SIM lock 3), but not backward. Cases where opposite is required (command completions and async events) are handled via queuing events via several completion threads. The rest of locks are self-contained and does not really suppose cascading. All these changes combined with GEOM direct dispatch (it will be next separate project) allow to double system performance in disk I/O microbenchmarks, comparing to present head, same as it was announced on 2013-05 DevSummit: http://people.freebsd.org/~mav/camlock.pdf . Tests without GEOM changes also show performance improvement, but limited by heavy bottleneck at the GEOM g_up/g_down threads at the level of 5-20%. Project sources could be found at SVN projects/camlock branch: http://svnweb.freebsd.org/base/projects/camlock/ . Many early changes from that branch are already integrated to head, so to simplify review the rest patches for changes before r254059 were manually remade and could be found here: http://people.freebsd.org/~mav/camlock_patches/ . These changes do not require controller driver modifications, keeping KPIs and hopefully KBIs intact, but create base for later work to use multiqueue capabilities of new controllers. This work is sponsored by iXsystems, Inc. -- Alexander Motin