From owner-freebsd-current@FreeBSD.ORG Wed Sep 4 09:00:45 2013 Return-Path: Delivered-To: freebsd-current@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTP id 2DA83A76 for ; Wed, 4 Sep 2013 09:00:45 +0000 (UTC) (envelope-from mavbsd@gmail.com) Received: from mail-bk0-x233.google.com (mail-bk0-x233.google.com [IPv6:2a00:1450:4008:c01::233]) (using TLSv1 with cipher ECDHE-RSA-RC4-SHA (128/128 bits)) (No client certificate requested) by mx1.freebsd.org (Postfix) with ESMTPS id 9A3AC2059 for ; Wed, 4 Sep 2013 09:00:44 +0000 (UTC) Received: by mail-bk0-f51.google.com with SMTP id mx10so16168bkb.10 for ; Wed, 04 Sep 2013 02:00:42 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=sender:message-id:date:from:user-agent:mime-version:to:cc:subject :references:in-reply-to:content-type:content-transfer-encoding; bh=P3vjJjPnft2RivgGO98WZ//OlQ0CsLDgEs7ECue7+38=; b=wyTv85tQuKfqfvzQBI8gHk39YZUSd1Dkv6k04P0sCHV0QGbZsUkiNVTAucPp7ILLXs 64+ux4YXE5+kIvN6gMJeqoq8TX/TgIx3Xcp3VmiVzZhXp8flPvhUQVRbgUGkpShfbTFl ugDn4ZjPaGBBBC+9GajL2wmAM8nIj6+O860AGwb04QEzKI07fOXTZM2JGVrseS06Svlh W1c2tyCO7diYTp3Pl0cuwqzHBpLgnK7CTP24bTj/oCraJJMgnNb/DCdSnTBO95CsRI82 TqMzK5c6FuICH5heAmkn25qKJbmmp0ZvSGauaL2AR+NIUB7ig2YXo60E2hEFQxaWL9yX 3pyA== X-Received: by 10.204.123.199 with SMTP id q7mr1446436bkr.10.1378285242846; Wed, 04 Sep 2013 02:00:42 -0700 (PDT) Received: from mavbook.mavhome.dp.ua ([37.229.21.195]) by mx.google.com with ESMTPSA id a4sm244817bko.11.1969.12.31.16.00.00 (version=TLSv1 cipher=ECDHE-RSA-RC4-SHA bits=128/128); Wed, 04 Sep 2013 02:00:41 -0700 (PDT) Sender: Alexander Motin Message-ID: <5226F6B7.3070706@FreeBSD.org> Date: Wed, 04 Sep 2013 12:00:39 +0300 From: Alexander Motin User-Agent: Mozilla/5.0 (X11; FreeBSD amd64; rv:17.0) Gecko/20130616 Thunderbird/17.0.6 MIME-Version: 1.0 To: Johan Hendriks Subject: Re: [RFC][CFT] GEOM direct dispatch and fine-grained CAM locking References: <520D4ADB.50209@FreeBSD.org> <5224511D.4090503@FreeBSD.org> <5226ED39.3070200@gmail.com> In-Reply-To: <5226ED39.3070200@gmail.com> Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit Cc: FreeBSD X-BeenThere: freebsd-current@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Discussions about the use of FreeBSD-current List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 04 Sep 2013 09:00:45 -0000 On 04.09.2013 11:20, Johan Hendriks wrote: > Alexander Motin wrote: >> I would like to invite more people to review and test my patches for >> improving CAM and GEOM scalability, that for last six months you could >> see developing in project/camlock SVN branch. Full diff of that branch >> against present head (r255131) can be found here: >> http://people.freebsd.org/~mav/camlock_patches/camlock_20130902.patch >> >> Heavy CAM changes there were focused on reducing scope of SIM lock to >> only protecting SIM internals, but not CAM core. That allows many >> times reduce lock congestion, especially on heavily parallel request >> submission with GEOM changes below. More detailed description of >> changes you could see here earlier: >> http://docs.freebsd.org/cgi/mid.cgi?520D4ADB.50209 >> >> GEOM changes were focused on avoiding switching to GEOM up/down >> threads in relatively simple setups where respective classes don't >> require it (and were explicitly marked so). That allows save on >> context switches and on systems with several HBAs and disks talk to >> them concurrently (that is where CAM locking changes are handy). Such >> classes were modified to support it: DEV, DISK, LABEL, MULTIPATH, NOP, >> PART, RAID (partially), STRIPE, ZERO, VFS, ZFS::VDEV, ZFS::ZVOL and >> some others. Requests to/from other classes will be queued to GEOM >> threads same as before. >> >> Together that allows to double block subsystem performance on high (at >> least 100-200K) IOPS benchmarks, allowing to reach up to a million >> total IOPS, while keeping full compatibility with all major ABIs/KBIs. >> >> Since we are already in 10.0 release process and changes are quite >> big, my plan is to wait and commit them to head branch after the >> freeze end, and then merge to stable/10. I hope the release process >> will go on schedule to not delay this work for another six months. >> >> This work is sponsored by iXsystems, Inc. >> > Hello i would like to test this patch set. > But how can i stress the machine, do you have a script or something i > can use to make the system do heavy I/O on the disks! For testing IOPS over RAW disks or different GEOM providers (to exclude FS influence) I am using such a trivial synthetic test: #!/bin/sh for disk in da0 da1 da2 da3 da4 da5 da6 da7 da8 da9 da10 da11 da12 da13 da14 da15 do for i in `jot 16` do dd if=/dev/$disk of=/dev/null bs=512 & done done iostat -zxw3 -c12 | grep total |tail -n 10 |cut -c 6-18 killall dd For total IOPS measurement in above script I am using dirtily hacked iostat tool that prints summary values in addition to per-disk ones: http://people.freebsd.org/~mav/camlock_patches/iostat.patch BTW I've uploaded new patch with some more minor CAM changes: http://people.freebsd.org/~mav/camlock_patches/camlock_20130904.patch -- Alexander Motin