From owner-freebsd-hackers@FreeBSD.ORG Mon Sep 2 08:49:39 2013 Return-Path: Delivered-To: freebsd-hackers@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTP id D212FC1E; Mon, 2 Sep 2013 08:49:39 +0000 (UTC) (envelope-from mavbsd@gmail.com) Received: from mail-bk0-x22b.google.com (mail-bk0-x22b.google.com [IPv6:2a00:1450:4008:c01::22b]) (using TLSv1 with cipher ECDHE-RSA-RC4-SHA (128/128 bits)) (No client certificate requested) by mx1.freebsd.org (Postfix) with ESMTPS id D14CA283A; Mon, 2 Sep 2013 08:49:38 +0000 (UTC) Received: by mail-bk0-f43.google.com with SMTP id mz13so1423112bkb.16 for ; Mon, 02 Sep 2013 01:49:37 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=sender:message-id:date:from:user-agent:mime-version:to:subject :references:in-reply-to:content-type:content-transfer-encoding; bh=2NdJTP8xy/tghqvhT0eqh0KJKUr9tGy7D+crGKM1GIY=; b=dKSlxcmStpiRbVyDOmmOlKsD6ruXFiUKuXOJqnBTxRrBK3pUDKyYHW0OncjxVyavQE ymt70Y5gpagtjMNX2k0bD8zdteYVASadji67T46/14NnA7LZjzBS1HJdWin8BKTnrZOv qANR7zVGc+q1uBxDGohlm5OAt2V5VMTuCEqZVuP5/CILpislCQwEvRmVT6ZwNsCbzgGh wSdA5WRfuSSVEJ+OL8DilROTVVxA9HZ8zqqrtGz74YoAx8d4Fo1U1WtUXPo3H6oi1dFP N7ImHW63gKhaOXr22ie4ttVcGxNDdNLhqO/aeHzbfss57P8SCzlU4IffaTOZjlEYaw7D eCtw== X-Received: by 10.204.199.132 with SMTP id es4mr623140bkb.28.1378111776996; Mon, 02 Sep 2013 01:49:36 -0700 (PDT) Received: from mavbook.mavhome.dp.ua ([37.229.21.195]) by mx.google.com with ESMTPSA id jt14sm1918024bkb.0.1969.12.31.16.00.00 (version=TLSv1 cipher=ECDHE-RSA-RC4-SHA bits=128/128); Mon, 02 Sep 2013 01:49:36 -0700 (PDT) Sender: Alexander Motin Message-ID: <5224511D.4090503@FreeBSD.org> Date: Mon, 02 Sep 2013 11:49:33 +0300 From: Alexander Motin User-Agent: Mozilla/5.0 (X11; FreeBSD amd64; rv:17.0) Gecko/20130616 Thunderbird/17.0.6 MIME-Version: 1.0 To: FreeBSD SCSI , freebsd-hackers@FreeBSD.org, freebsd-geom@FreeBSD.org, "freebsd-current@freebsd.org" Subject: [RFC][CFT] GEOM direct dispatch and fine-grained CAM locking References: <520D4ADB.50209@FreeBSD.org> In-Reply-To: <520D4ADB.50209@FreeBSD.org> Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 02 Sep 2013 08:49:39 -0000 Hi. I would like to invite more people to review and test my patches for improving CAM and GEOM scalability, that for last six months you could see developing in project/camlock SVN branch. Full diff of that branch against present head (r255131) can be found here: http://people.freebsd.org/~mav/camlock_patches/camlock_20130902.patch Heavy CAM changes there were focused on reducing scope of SIM lock to only protecting SIM internals, but not CAM core. That allows many times reduce lock congestion, especially on heavily parallel request submission with GEOM changes below. More detailed description of changes you could see here earlier: http://docs.freebsd.org/cgi/mid.cgi?520D4ADB.50209 GEOM changes were focused on avoiding switching to GEOM up/down threads in relatively simple setups where respective classes don't require it (and were explicitly marked so). That allows save on context switches and on systems with several HBAs and disks talk to them concurrently (that is where CAM locking changes are handy). Such classes were modified to support it: DEV, DISK, LABEL, MULTIPATH, NOP, PART, RAID (partially), STRIPE, ZERO, VFS, ZFS::VDEV, ZFS::ZVOL and some others. Requests to/from other classes will be queued to GEOM threads same as before. Together that allows to double block subsystem performance on high (at least 100-200K) IOPS benchmarks, allowing to reach up to a million total IOPS, while keeping full compatibility with all major ABIs/KBIs. Since we are already in 10.0 release process and changes are quite big, my plan is to wait and commit them to head branch after the freeze end, and then merge to stable/10. I hope the release process will go on schedule to not delay this work for another six months. This work is sponsored by iXsystems, Inc. -- Alexander Motin