From owner-freebsd-geom@FreeBSD.ORG Mon Jan 16 01:41:56 2012 Return-Path: Delivered-To: freebsd-geom@hub.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id BA927106566B; Mon, 16 Jan 2012 01:41:56 +0000 (UTC) (envelope-from linimon@FreeBSD.org) Received: from freefall.freebsd.org (freefall.freebsd.org [IPv6:2001:4f8:fff6::28]) by mx1.freebsd.org (Postfix) with ESMTP id 8C9C28FC0C; Mon, 16 Jan 2012 01:41:56 +0000 (UTC) Received: from freefall.freebsd.org (localhost [127.0.0.1]) by freefall.freebsd.org (8.14.5/8.14.5) with ESMTP id q0G1fuBw009212; Mon, 16 Jan 2012 01:41:56 GMT (envelope-from linimon@freefall.freebsd.org) Received: (from linimon@localhost) by freefall.freebsd.org (8.14.5/8.14.5/Submit) id q0G1fuSc009208; Mon, 16 Jan 2012 01:41:56 GMT (envelope-from linimon) Date: Mon, 16 Jan 2012 01:41:56 GMT Message-Id: <201201160141.q0G1fuSc009208@freefall.freebsd.org> To: linimon@FreeBSD.org, freebsd-bugs@FreeBSD.org, freebsd-geom@FreeBSD.org From: linimon@FreeBSD.org Cc: Subject: Re: kern/164143: [geom] Partition table not recognized after upgrade R8.3 -> 9.0 [regression] X-BeenThere: freebsd-geom@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: GEOM-specific discussions and implementations List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 16 Jan 2012 01:41:56 -0000 Old Synopsis: Partition table not recognized after upgrade R8.3 -> 9.0 New Synopsis: [geom] Partition table not recognized after upgrade R8.3 -> 9.0 [regression] Responsible-Changed-From-To: freebsd-bugs->freebsd-geom Responsible-Changed-By: linimon Responsible-Changed-When: Mon Jan 16 01:41:09 UTC 2012 Responsible-Changed-Why: reclassify. http://www.freebsd.org/cgi/query-pr.cgi?pr=164143 From owner-freebsd-geom@FreeBSD.ORG Mon Jan 16 11:07:02 2012 Return-Path: Delivered-To: freebsd-geom@FreeBSD.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 73156106564A for ; Mon, 16 Jan 2012 11:07:02 +0000 (UTC) (envelope-from owner-bugmaster@FreeBSD.org) Received: from freefall.freebsd.org (freefall.freebsd.org [IPv6:2001:4f8:fff6::28]) by mx1.freebsd.org (Postfix) with ESMTP id 18BD58FC20 for ; Mon, 16 Jan 2012 11:07:02 +0000 (UTC) Received: from freefall.freebsd.org (localhost [127.0.0.1]) by freefall.freebsd.org (8.14.5/8.14.5) with ESMTP id q0GB71ZJ057629 for ; Mon, 16 Jan 2012 11:07:01 GMT (envelope-from owner-bugmaster@FreeBSD.org) Received: (from gnats@localhost) by freefall.freebsd.org (8.14.5/8.14.5/Submit) id q0GB714N057627 for freebsd-geom@FreeBSD.org; Mon, 16 Jan 2012 11:07:01 GMT (envelope-from owner-bugmaster@FreeBSD.org) Date: Mon, 16 Jan 2012 11:07:01 GMT Message-Id: <201201161107.q0GB714N057627@freefall.freebsd.org> X-Authentication-Warning: freefall.freebsd.org: gnats set sender to owner-bugmaster@FreeBSD.org using -f From: FreeBSD bugmaster To: freebsd-geom@FreeBSD.org Cc: Subject: Current problem reports assigned to freebsd-geom@FreeBSD.org X-BeenThere: freebsd-geom@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: GEOM-specific discussions and implementations List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 16 Jan 2012 11:07:02 -0000 Note: to view an individual PR, use: http://www.freebsd.org/cgi/query-pr.cgi?pr=(number). The following is a listing of current problems submitted by FreeBSD users. These represent problem reports covering all versions including experimental development code and obsolete releases. S Tracker Resp. Description -------------------------------------------------------------------------------- o kern/164143 geom [geom] Partition table not recognized after upgrade R8 o kern/163020 geom [geli] [patch] enable the Camellia-XTS on GEOM ELI o kern/162010 geom [geli] panic: Provider's error should be set (error=0) o kern/161979 geom [geom] glabel doesn't update after newfs, and glabel s o kern/161752 geom [geom] glabel(8) doesn't get gpt label change o bin/161677 geom gpart(8) Probably bug in gptboot o kern/160562 geom [geom][patch] Allow to insert new component to geom_ra o kern/160409 geom [geli] failed to attach provider f kern/159595 geom [geom] [panic] panic on gmirror unload in vbox [regres p kern/158398 geom [headers] [patch] includes o kern/158197 geom [geom] geom_cache with size>1000 leads to panics o kern/157879 geom [libgeom] [regression] ABI change without version bump o kern/157863 geom [geli] kbdmux prevents geli passwords from being enter o kern/157739 geom [geom] GPT labels with geom_multipath o kern/157724 geom [geom] gpart(8) 'add' command must preserve gap for sc o kern/157723 geom [geom] GEOM should not process 'c' (raw) partitions fo o kern/157108 geom [gjournal] dumpon(8) fails on gjournal providers o kern/155994 geom [geom] Long "Suspend time" when reading large files fr o kern/154226 geom [geom] GEOM label does not change when you modify them o kern/150858 geom [geom] [geom_label] [patch] glabel(8) is not compatibl o kern/150626 geom [geom] [gjournal] gjournal(8) destroys label o kern/150555 geom [geom] gjournal unusable on GPT partitions o kern/150334 geom [geom] [udf] [patch] geom label does not support UDF o kern/149762 geom volume labels with rogue characters o bin/149215 geom [panic] [geom_part] gpart(8): Delete linux's slice via o kern/147667 geom [gmirror] Booting with one component of a gmirror, the o kern/145818 geom [geom] geom_stat_open showing cached information for n o kern/145042 geom [geom] System stops booting after printing message "GE o kern/143455 geom gstripe(8) in RELENG_8 (31st Jan 2010) broken o kern/142563 geom [geom] [hang] ioctl freeze in zpool o kern/141740 geom [geom] gjournal(8): g_journal_destroy concurrent error o kern/140352 geom [geom] gjournal + glabel not working o kern/135898 geom [geom] Severe filesystem corruption - large files or l o kern/134922 geom [gmirror] [panic] kernel panic when use fdisk on disk o kern/134113 geom [geli] Problem setting secondary GELI key o kern/133931 geom [geli] [request] intentionally wrong password to destr o bin/132845 geom [geom] [patch] ggated(8) does not close files opened a o bin/131415 geom [geli] keystrokes are unregulary sent to Geli when typ o kern/131353 geom [geom] gjournal(8) kernel lock o kern/129674 geom [geom] gjournal root did not mount on boot o kern/129645 geom gjournal(8): GEOM_JOURNAL causes system to fail to boo o kern/129245 geom [geom] gcache is more suitable for suffix based provid f kern/128276 geom [gmirror] machine lock up when gmirror module is used o kern/127420 geom [geom] [gjournal] [panic] Journal overflow on gmirrore o kern/124973 geom [gjournal] [patch] boot order affects geom_journal con o kern/124969 geom gvinum(8): gvinum raid5 plex does not detect missing s o kern/123962 geom [panic] [gjournal] gjournal (455Gb data, 8Gb journal), o kern/123122 geom [geom] GEOM / gjournal kernel lock o kern/122738 geom [geom] gmirror list "losts consumers" after gmirror de o kern/122067 geom [geom] [panic] Geom crashed during boot o kern/121364 geom [gmirror] Removing all providers create a "zombie" mir o kern/120091 geom [geom] [geli] [gjournal] geli does not prompt for pass o kern/115856 geom [geli] ZFS thought it was degraded when it should have o kern/115547 geom [geom] [patch] [request] let GEOM Eli get password fro o kern/114532 geom [geom] GEOM_MIRROR shows up in kldstat even if compile f kern/113957 geom [gmirror] gmirror is intermittently reporting a degrad o kern/113837 geom [geom] unable to access 1024 sector size storage o kern/113419 geom [geom] geom fox multipathing not failing back o kern/107707 geom [geom] [patch] [request] add new class geom_xbox360 to o kern/94632 geom [geom] Kernel output resets input while GELI asks for o kern/90582 geom [geom] [panic] Restore cause panic string (ffs_blkfree o bin/90093 geom fdisk(8) incapable of altering in-core geometry o kern/87544 geom [gbde] mmaping large files on a gbde filesystem deadlo o bin/86388 geom [geom] [geom_part] periodic(8) daily should backup gpa o kern/84556 geom [geom] [panic] GBDE-encrypted swap causes panic at shu o kern/79251 geom [2TB] newfs fails on 2.6TB gbde device o kern/79035 geom [vinum] gvinum unable to create a striped set of mirro o bin/78131 geom gbde(8) "destroy" not working. 68 problems total. From owner-freebsd-geom@FreeBSD.ORG Wed Jan 18 01:04:00 2012 Return-Path: Delivered-To: freebsd-geom@hub.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id D44D8106564A; Wed, 18 Jan 2012 01:04:00 +0000 (UTC) (envelope-from linimon@FreeBSD.org) Received: from freefall.freebsd.org (freefall.freebsd.org [IPv6:2001:4f8:fff6::28]) by mx1.freebsd.org (Postfix) with ESMTP id AA2D48FC0C; Wed, 18 Jan 2012 01:04:00 +0000 (UTC) Received: from freefall.freebsd.org (localhost [127.0.0.1]) by freefall.freebsd.org (8.14.5/8.14.5) with ESMTP id q0I1405i010235; Wed, 18 Jan 2012 01:04:00 GMT (envelope-from linimon@freefall.freebsd.org) Received: (from linimon@localhost) by freefall.freebsd.org (8.14.5/8.14.5/Submit) id q0I140PJ010231; Wed, 18 Jan 2012 01:04:00 GMT (envelope-from linimon) Date: Wed, 18 Jan 2012 01:04:00 GMT Message-Id: <201201180104.q0I140PJ010231@freefall.freebsd.org> To: linimon@FreeBSD.org, freebsd-bugs@FreeBSD.org, freebsd-geom@FreeBSD.org From: linimon@FreeBSD.org Cc: Subject: Re: kern/164252: [geom] gjournal overflow X-BeenThere: freebsd-geom@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: GEOM-specific discussions and implementations List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 18 Jan 2012 01:04:00 -0000 Old Synopsis: Journal Overflow New Synopsis: [geom] gjournal overflow Responsible-Changed-From-To: freebsd-bugs->freebsd-geom Responsible-Changed-By: linimon Responsible-Changed-When: Wed Jan 18 01:02:55 UTC 2012 Responsible-Changed-Why: Over to maintainer(s). http://www.freebsd.org/cgi/query-pr.cgi?pr=164252 From owner-freebsd-geom@FreeBSD.ORG Wed Jan 18 01:05:07 2012 Return-Path: Delivered-To: freebsd-geom@hub.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 997F91065691; Wed, 18 Jan 2012 01:05:07 +0000 (UTC) (envelope-from linimon@FreeBSD.org) Received: from freefall.freebsd.org (freefall.freebsd.org [IPv6:2001:4f8:fff6::28]) by mx1.freebsd.org (Postfix) with ESMTP id 6EBC38FC12; Wed, 18 Jan 2012 01:05:07 +0000 (UTC) Received: from freefall.freebsd.org (localhost [127.0.0.1]) by freefall.freebsd.org (8.14.5/8.14.5) with ESMTP id q0I157qT010443; Wed, 18 Jan 2012 01:05:07 GMT (envelope-from linimon@freefall.freebsd.org) Received: (from linimon@localhost) by freefall.freebsd.org (8.14.5/8.14.5/Submit) id q0I157wF010439; Wed, 18 Jan 2012 01:05:07 GMT (envelope-from linimon) Date: Wed, 18 Jan 2012 01:05:07 GMT Message-Id: <201201180105.q0I157wF010439@freefall.freebsd.org> To: linimon@FreeBSD.org, freebsd-bugs@FreeBSD.org, freebsd-geom@FreeBSD.org From: linimon@FreeBSD.org Cc: Subject: Re: kern/164254: [geom] gjournal not stopping on GPT partitions X-BeenThere: freebsd-geom@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: GEOM-specific discussions and implementations List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 18 Jan 2012 01:05:07 -0000 Old Synopsis: GJournal not stopping on GPT partitions New Synopsis: [geom] gjournal not stopping on GPT partitions Responsible-Changed-From-To: freebsd-bugs->freebsd-geom Responsible-Changed-By: linimon Responsible-Changed-When: Wed Jan 18 01:04:10 UTC 2012 Responsible-Changed-Why: Over to maintainer(s). http://www.freebsd.org/cgi/query-pr.cgi?pr=164254 From owner-freebsd-geom@FreeBSD.ORG Wed Jan 18 11:50:09 2012 Return-Path: Delivered-To: freebsd-geom@hub.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id BB153106564A for ; Wed, 18 Jan 2012 11:50:09 +0000 (UTC) (envelope-from gnats@FreeBSD.org) Received: from freefall.freebsd.org (freefall.freebsd.org [IPv6:2001:4f8:fff6::28]) by mx1.freebsd.org (Postfix) with ESMTP id 853798FC13 for ; Wed, 18 Jan 2012 11:50:09 +0000 (UTC) Received: from freefall.freebsd.org (localhost [127.0.0.1]) by freefall.freebsd.org (8.14.5/8.14.5) with ESMTP id q0IBo900044167 for ; Wed, 18 Jan 2012 11:50:09 GMT (envelope-from gnats@freefall.freebsd.org) Received: (from gnats@localhost) by freefall.freebsd.org (8.14.5/8.14.5/Submit) id q0IBo9cL044166; Wed, 18 Jan 2012 11:50:09 GMT (envelope-from gnats) Date: Wed, 18 Jan 2012 11:50:09 GMT Message-Id: <201201181150.q0IBo9cL044166@freefall.freebsd.org> To: freebsd-geom@FreeBSD.org From: Vincent Hoffman-Kazlauskas Cc: Subject: Re: kern/164254: [geom] gjournal not stopping on GPT partitions X-BeenThere: freebsd-geom@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list Reply-To: Vincent Hoffman-Kazlauskas List-Id: GEOM-specific discussions and implementations List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 18 Jan 2012 11:50:09 -0000 The following reply was made to PR kern/164254; it has been noted by GNATS. From: Vincent Hoffman-Kazlauskas To: bug-followup@FreeBSD.org, grimeton@gmx.net Cc: Subject: Re: kern/164254: [geom] gjournal not stopping on GPT partitions Date: Wed, 18 Jan 2012 09:42:05 +0000 I'm not a maintainer or even a coder but I got bitten by this before. The solution in the sort term is to disable gtpid labels sysctl kern.geom.label.gptid.enable=0 Its somewhat annoying though and while I dont use gptid labels, some people must. an entry in one (all?) of the appropriate geom manpages might be an idea. Vince From owner-freebsd-geom@FreeBSD.ORG Fri Jan 20 08:40:59 2012 Return-Path: Delivered-To: freebsd-geom@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 705D9106566C; Fri, 20 Jan 2012 08:40:59 +0000 (UTC) (envelope-from ndenev@gmail.com) Received: from mail-ey0-f182.google.com (mail-ey0-f182.google.com [209.85.215.182]) by mx1.freebsd.org (Postfix) with ESMTP id 96E138FC12; Fri, 20 Jan 2012 08:40:55 +0000 (UTC) Received: by eaai10 with SMTP id i10so110403eaa.13 for ; Fri, 20 Jan 2012 00:40:54 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=subject:mime-version:content-type:from:in-reply-to:date:cc :content-transfer-encoding:message-id:references:to:x-mailer; bh=A4fVTjcC6Hz5l30uWQkJhmm3DMsTMvfkpXNwyp7+WxI=; b=AzgWpxXE1qn0eB88os0hlheFByhmMPxp+MfxiXNrxstWh7RhkFbT1OZhNwMe1F1FrC B7WWzL35iDj/ngQsfQjhwd13rD+p4JpCVzipExenn5KKJYGM2zBXiGdsxbbPLB5Mg3n0 eiKgEmsJN1b1PMEUVNDHUDnNUWdj5+5vZEn+U= Received: by 10.213.35.12 with SMTP id n12mr7671455ebd.68.1327046998419; Fri, 20 Jan 2012 00:09:58 -0800 (PST) Received: from ndenevsa.sf.moneybookers.net (g1.moneybookers.com. [217.18.249.148]) by mx.google.com with ESMTPS id t59sm8383473eeh.10.2012.01.20.00.09.55 (version=TLSv1/SSLv3 cipher=OTHER); Fri, 20 Jan 2012 00:09:56 -0800 (PST) Mime-Version: 1.0 (Apple Message framework v1251.1) Content-Type: text/plain; charset=us-ascii From: Nikolay Denev In-Reply-To: <20111114210957.GA68559@in-addr.com> Date: Fri, 20 Jan 2012 10:09:56 +0200 Content-Transfer-Encoding: quoted-printable Message-Id: <059C17DB-3A7B-41AA-BF91-2F8EBAF17D01@gmail.com> References: <4EAF00A6.5060903@FreeBSD.org> <05E0E64F-5EC4-425A-81E4-B6C35320608B@neveragain.de> <4EB05566.3060700@FreeBSD.org> <20111114210957.GA68559@in-addr.com> To: Gary Palmer X-Mailer: Apple Mail (2.1251.1) Cc: Alexander Motin , FreeBSD-Current , Dennis K?gel , freebsd-geom@freebsd.org Subject: Re: RFC: GEOM MULTIPATH rewrite X-BeenThere: freebsd-geom@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: GEOM-specific discussions and implementations List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 20 Jan 2012 08:40:59 -0000 On Nov 14, 2011, at 11:09 PM, Gary Palmer wrote: > On Tue, Nov 01, 2011 at 10:24:06PM +0200, Alexander Motin wrote: >> On 01.11.2011 19:50, Dennis K?gel wrote: >>> Not sure if replying on-list or off-list makes more sense... >>=20 >> Replying on-list could share experience to other users. >>=20 >>> Anyway, some first impressions, on stable/9: >>>=20 >>> The lab environment here is a EMC VNX / Clariion SAN, which has two = Storage Processors, connected to different switches, connected to two = isp(4)s on the test machine. So at any time, the machine sees four = paths, but only two are available (depending on which SP owns the LUN). >>>=20 >>> 580# camcontrol devlist >>> at scbus0 target 0 lun 0 = (da0,pass0) >>> at scbus0 target 1 lun 0 = (da1,pass1) >>> at scbus1 target 0 lun 0 = (da2,pass2) >>> at scbus1 target 1 lun 0 = (da3,pass3) >>> at scbus2 target 0 lun 0 = (da4,pass4) >>> at scbus2 target 1 lun 0 = (da5,pass5) >>> at scbus4 target 0 lun 0 = (cd0,pass6) >>>=20 >>> I miss the ability to "add" disks to automatic mode multipaths, but = I (just now) realized this only makes sense when gmultipath has some = kind of path checking facility (like periodically trying to read sector = 0 of each configured device, this is was Linux' devicemapper-multipathd = does). >>=20 >> In automatic mode other paths supposed to be detected via metadata >> reading. If in your case some paths are not readable, automatic mode >> can't work as expected. By the way, could you describe how your >> configuration supposed to work, like when other paths will start >> working?=20 >=20 > Without knowledge of the particular Clariion SAN Dennis is working = with, > I've seen some so-called active/active RAID controllers force a LUN=20 > fail over from one controller to another (taking it offline for 3 = seconds > in the process) because the LUN received an I/O down a path to the = controller > that was formerly taking the standby role for that LUN (and it was = per-LUN, > so some would be owned by one controller and some by the other). = During > the controller switch, all I/O to the LUN would fail. Thankfully that > particular RAID model where I observed this behaviour hasn't been sold = in > several years, but I would tend to expect such behaviour at the lower > end of the storage market with the higher end units doing true = active/active > configurations. (and no, I won't name the manufacturer on a public = list) >=20 > This is exactly why Linux ships with a multipath configuration file, = so > it can describe exactly what form of brain damage the controller in > question implements so it can work around it, and maybe even=20 > document some vendor-specific extensions so that the host can detect > which controller is taking which role for a particular path. >=20 > Even some controllers that don't have pathological behaviour when > they receive I/O down the wrong path have sub-optimal behaviour unless > you choose the right path. NetApp SANs in particular typically have = two > independant controllers with a high-speed internal interconnect, = however > there is a measurable and not-insignificant penalty for sending the = I/O > to the "partner" controller for a LUN, across the internal = interconnect > (called a "VTIC" I believe) to the "owner" controller. I've been = told, > although I have not measured this myself, that it can add several ms = to > a transaction, which when talking about SAN storage is potentially = several > times what it takes to do the same I/O directly to the controller that > owns it. There's probably a way to make the "partner" controller not > advertise the LUN until it takes over in a failover scenario, but = every > NetApp I've worked with is set (by default I believe) to advertise the > LUN out both controllers. >=20 > Gary > _______________________________________________ > freebsd-current@freebsd.org mailing list > http://lists.freebsd.org/mailman/listinfo/freebsd-current > To unsubscribe, send any mail to = "freebsd-current-unsubscribe@freebsd.org" Another thing I've observed is that active/active probably only makes = sense if you are accessing single LUN. In my tests where I have 24 LUNS that form 4 vdevs in a single zpool, = the highest performance was achieved when I split the active paths among the controllers installed in the = server importing the pool. (basically "gmultipath rotate $LUN" in = rc.local for half of the paths) Using active/active in this situation resulted in fluctuating = performance. From owner-freebsd-geom@FreeBSD.ORG Fri Jan 20 11:08:35 2012 Return-Path: Delivered-To: freebsd-geom@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id AA3AB1065670; Fri, 20 Jan 2012 11:08:35 +0000 (UTC) (envelope-from ndenev@gmail.com) Received: from mail-qy0-f182.google.com (mail-qy0-f182.google.com [209.85.216.182]) by mx1.freebsd.org (Postfix) with ESMTP id 1C3688FC1A; Fri, 20 Jan 2012 11:08:34 +0000 (UTC) Received: by qcse1 with SMTP id e1so337998qcs.13 for ; Fri, 20 Jan 2012 03:08:34 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=references:in-reply-to:mime-version:from:date:message-id:subject:to :cc:content-type; bh=FqNOiVXEjai+218PgP9JVrNzGwFtDdlsLvLolgirAOw=; b=bqeNzA8411NwzBL9J28lVxLMwxvtAe/QBqvIjNJKzglm7KwPkplTvzv5v3qAyCkB0D TFanT3DIkl5jxsmpo9tiBc2rxJlesMLi+Pc4M8X2aMaM1nom4AWwvML8ZpdJt+bbhJXM 2rsOuPd/6aA5KfGjH3qSJbXLBbhgCdpG3/8Zw= Received: by 10.229.136.82 with SMTP id q18mr11435710qct.139.1327057714417; Fri, 20 Jan 2012 03:08:34 -0800 (PST) References: <4EAF00A6.5060903@FreeBSD.org> <05E0E64F-5EC4-425A-81E4-B6C35320608B@neveragain.de> <4EB05566.3060700@FreeBSD.org> <20111114210957.GA68559@in-addr.com> <059C17DB-3A7B-41AA-BF91-2F8EBAF17D01@gmail.com> <4F19474A.9020600@FreeBSD.org> In-Reply-To: <4F19474A.9020600@FreeBSD.org> Mime-Version: 1.0 (1.0) From: Nikolay Denev Date: Fri, 20 Jan 2012 13:08:29 +0200 Message-ID: <-2439788735531654851@unknownmsgid> To: Alexander Motin Content-Type: text/plain; charset=ISO-8859-1 Cc: Gary Palmer , FreeBSD-Current , Dennis K?gel , "freebsd-geom@freebsd.org" Subject: Re: RFC: GEOM MULTIPATH rewrite X-BeenThere: freebsd-geom@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: GEOM-specific discussions and implementations List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 20 Jan 2012 11:08:35 -0000 On 20.01.2012, at 12:51, Alexander Motin wrote: > On 01/20/12 10:09, Nikolay Denev wrote: >> Another thing I've observed is that active/active probably only makes sense if you are accessing single LUN. >> In my tests where I have 24 LUNS that form 4 vdevs in a single zpool, the highest performance was achieved >> when I split the active paths among the controllers installed in the server importing the pool. (basically "gmultipath rotate $LUN" in rc.local for half of the paths) >> Using active/active in this situation resulted in fluctuating performance. > > How big was fluctuation? Between speed of one and all paths? > > Several active/active devices without knowledge about each other with some probability will send part of requests via the same links, while ZFS itself already does some balancing between vdevs. > > -- > Alexander Motin I will test in a bit and post results. P.S.: Is there a way to enable/disable active-active on the fly? I'm currently re-labeling to achieve that. From owner-freebsd-geom@FreeBSD.ORG Fri Jan 20 11:15:46 2012 Return-Path: Delivered-To: freebsd-geom@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id EDB3C106566C; Fri, 20 Jan 2012 11:15:46 +0000 (UTC) (envelope-from mavbsd@gmail.com) Received: from mail-ee0-f54.google.com (mail-ee0-f54.google.com [74.125.83.54]) by mx1.freebsd.org (Postfix) with ESMTP id 435408FC12; Fri, 20 Jan 2012 11:15:45 +0000 (UTC) Received: by eekb47 with SMTP id b47so163419eek.13 for ; Fri, 20 Jan 2012 03:15:45 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=sender:message-id:date:from:user-agent:mime-version:to:cc:subject :references:in-reply-to:content-type:content-transfer-encoding; bh=izzUevYsVgK40PGTzr/YLDf3pDzCwUFqoB5BwvZuRwc=; b=VV5Sv8KCF04AtIOXM52Lz6eMhhxsD1Xo3565oHoQdoy0S2800m5B52rL6TDvCHnMVo 2aj3PLUHyoFULwtyQR6paV7/HIeHP47S/7CD/VdhYwR9vMjcpb8r/LvzHtIfRWf40KN2 99fCLPRxLprl0fM7MSaliIUJZCWdOQkIQiIto= Received: by 10.213.10.82 with SMTP id o18mr5273314ebo.142.1327056719024; Fri, 20 Jan 2012 02:51:59 -0800 (PST) Received: from mavbook2.mavhome.dp.ua (pc.mavhome.dp.ua. [212.86.226.226]) by mx.google.com with ESMTPS id t59sm9905189eeh.10.2012.01.20.02.51.56 (version=SSLv3 cipher=OTHER); Fri, 20 Jan 2012 02:51:57 -0800 (PST) Sender: Alexander Motin Message-ID: <4F19474A.9020600@FreeBSD.org> Date: Fri, 20 Jan 2012 12:51:54 +0200 From: Alexander Motin User-Agent: Mozilla/5.0 (X11; FreeBSD amd64; rv:9.0) Gecko/20111227 Thunderbird/9.0 MIME-Version: 1.0 To: Nikolay Denev References: <4EAF00A6.5060903@FreeBSD.org> <05E0E64F-5EC4-425A-81E4-B6C35320608B@neveragain.de> <4EB05566.3060700@FreeBSD.org> <20111114210957.GA68559@in-addr.com> <059C17DB-3A7B-41AA-BF91-2F8EBAF17D01@gmail.com> In-Reply-To: <059C17DB-3A7B-41AA-BF91-2F8EBAF17D01@gmail.com> Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit Cc: Gary Palmer , FreeBSD-Current , Dennis K?gel , freebsd-geom@freebsd.org Subject: Re: RFC: GEOM MULTIPATH rewrite X-BeenThere: freebsd-geom@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: GEOM-specific discussions and implementations List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 20 Jan 2012 11:15:47 -0000 On 01/20/12 10:09, Nikolay Denev wrote: > Another thing I've observed is that active/active probably only makes sense if you are accessing single LUN. > In my tests where I have 24 LUNS that form 4 vdevs in a single zpool, the highest performance was achieved > when I split the active paths among the controllers installed in the server importing the pool. (basically "gmultipath rotate $LUN" in rc.local for half of the paths) > Using active/active in this situation resulted in fluctuating performance. How big was fluctuation? Between speed of one and all paths? Several active/active devices without knowledge about each other with some probability will send part of requests via the same links, while ZFS itself already does some balancing between vdevs. -- Alexander Motin From owner-freebsd-geom@FreeBSD.ORG Fri Jan 20 11:30:17 2012 Return-Path: Delivered-To: freebsd-geom@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 6901510656D3; Fri, 20 Jan 2012 11:30:10 +0000 (UTC) (envelope-from mavbsd@gmail.com) Received: from mail-ee0-f54.google.com (mail-ee0-f54.google.com [74.125.83.54]) by mx1.freebsd.org (Postfix) with ESMTP id 6B3DB8FC16; Fri, 20 Jan 2012 11:30:09 +0000 (UTC) Received: by eekb47 with SMTP id b47so168602eek.13 for ; Fri, 20 Jan 2012 03:30:08 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=sender:message-id:date:from:user-agent:mime-version:to:cc:subject :references:in-reply-to:content-type:content-transfer-encoding; bh=UyA9G7fRiIQWUHDV9A5OZdRRSDSBdH1ggsxuB6+0rLE=; b=N1jqEeXVmGB/98dylboUEg83gkYEUpLTtIlvEemBAjU5nwD1Xh3+XNGmG0m+awTD2y TGsJHoXZh0oWo15OkMmXUVdaDnjvlxJodIorDADzhDmBlCOHyjgBntd8U0VkKRMFWJGH B8bAEMfX+PaheFnS8wEeqvaT/W8NhkPfZXLQU= Received: by 10.213.110.2 with SMTP id l2mr7678555ebp.22.1327059008234; Fri, 20 Jan 2012 03:30:08 -0800 (PST) Received: from mavbook2.mavhome.dp.ua (pc.mavhome.dp.ua. [212.86.226.226]) by mx.google.com with ESMTPS id n17sm10385781eei.3.2012.01.20.03.30.05 (version=SSLv3 cipher=OTHER); Fri, 20 Jan 2012 03:30:06 -0800 (PST) Sender: Alexander Motin Message-ID: <4F19503B.2090200@FreeBSD.org> Date: Fri, 20 Jan 2012 13:30:03 +0200 From: Alexander Motin User-Agent: Mozilla/5.0 (X11; FreeBSD amd64; rv:9.0) Gecko/20111227 Thunderbird/9.0 MIME-Version: 1.0 To: Nikolay Denev References: <4EAF00A6.5060903@FreeBSD.org> <05E0E64F-5EC4-425A-81E4-B6C35320608B@neveragain.de> <4EB05566.3060700@FreeBSD.org> <20111114210957.GA68559@in-addr.com> <059C17DB-3A7B-41AA-BF91-2F8EBAF17D01@gmail.com> <4F19474A.9020600@FreeBSD.org> <-2439788735531654851@unknownmsgid> In-Reply-To: <-2439788735531654851@unknownmsgid> Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit Cc: Gary Palmer , FreeBSD-Current , Dennis K?gel , "freebsd-geom@freebsd.org" Subject: Re: RFC: GEOM MULTIPATH rewrite X-BeenThere: freebsd-geom@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: GEOM-specific discussions and implementations List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 20 Jan 2012 11:30:17 -0000 On 01/20/12 13:08, Nikolay Denev wrote: > On 20.01.2012, at 12:51, Alexander Motin wrote: > >> On 01/20/12 10:09, Nikolay Denev wrote: >>> Another thing I've observed is that active/active probably only makes sense if you are accessing single LUN. >>> In my tests where I have 24 LUNS that form 4 vdevs in a single zpool, the highest performance was achieved >>> when I split the active paths among the controllers installed in the server importing the pool. (basically "gmultipath rotate $LUN" in rc.local for half of the paths) >>> Using active/active in this situation resulted in fluctuating performance. >> >> How big was fluctuation? Between speed of one and all paths? >> >> Several active/active devices without knowledge about each other with some probability will send part of requests via the same links, while ZFS itself already does some balancing between vdevs. >> >> -- >> Alexander Motin > > I will test in a bit and post results. > > P.S.: Is there a way to enable/disable active-active on the fly? I'm > currently re-labeling to achieve that. No, there is not now. But for experiments you may achieve the same results by manually marking as failed all paths except one. It is not dangerous, as if that link fail, all other will resurrect automatically. -- Alexander Motin From owner-freebsd-geom@FreeBSD.ORG Fri Jan 20 12:13:26 2012 Return-Path: Delivered-To: freebsd-geom@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id C0743106564A; Fri, 20 Jan 2012 12:13:26 +0000 (UTC) (envelope-from ndenev@gmail.com) Received: from mail-ee0-f54.google.com (mail-ee0-f54.google.com [74.125.83.54]) by mx1.freebsd.org (Postfix) with ESMTP id A19168FC2B; Fri, 20 Jan 2012 12:13:25 +0000 (UTC) Received: by eekb47 with SMTP id b47so184817eek.13 for ; Fri, 20 Jan 2012 04:13:24 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=subject:mime-version:content-type:from:in-reply-to:date:cc :content-transfer-encoding:message-id:references:to:x-mailer; bh=k4IjwMtd3EQAnyBacvRokdyKanGqfS4kAnMo2sE3wgg=; b=P6vCUqccZDNSY06DyrqvkQS9ijMHiANQnWYIILgvQCnDqPm000OGBPpJ+Qgjo98QLf ++thc4TIqcEiNBpebn9Zhe+tARSK9v4VKEGCDXKQbYUes+s4On08ErLzDMM7WQx3FJ6M Qvi5+zYdgfaGIQAY9IFSXG4WHf7hHlHbj8XJY= Received: by 10.14.9.38 with SMTP id 38mr3201665ees.101.1327061604619; Fri, 20 Jan 2012 04:13:24 -0800 (PST) Received: from ndenevsa.sf.moneybookers.net (g1.moneybookers.com. [217.18.249.148]) by mx.google.com with ESMTPS id t59sm10698354eeh.10.2012.01.20.04.13.18 (version=TLSv1/SSLv3 cipher=OTHER); Fri, 20 Jan 2012 04:13:21 -0800 (PST) Mime-Version: 1.0 (Apple Message framework v1251.1) Content-Type: text/plain; charset=iso-8859-1 From: Nikolay Denev In-Reply-To: <4F19503B.2090200@FreeBSD.org> Date: Fri, 20 Jan 2012 14:13:16 +0200 Content-Transfer-Encoding: quoted-printable Message-Id: <25C45DA0-4B52-42E4-A1A3-DD5168451423@gmail.com> References: <4EAF00A6.5060903@FreeBSD.org> <05E0E64F-5EC4-425A-81E4-B6C35320608B@neveragain.de> <4EB05566.3060700@FreeBSD.org> <20111114210957.GA68559@in-addr.com> <059C17DB-3A7B-41AA-BF91-2F8EBAF17D01@gmail.com> <4F19474A.9020600@FreeBSD.org> <-2439788735531654851@unknownmsgid> <4F19503B.2090200@FreeBSD.org> To: Alexander Motin X-Mailer: Apple Mail (2.1251.1) Cc: Gary Palmer , FreeBSD-Current , Dennis K?gel , "freebsd-geom@freebsd.org" Subject: Re: RFC: GEOM MULTIPATH rewrite X-BeenThere: freebsd-geom@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: GEOM-specific discussions and implementations List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 20 Jan 2012 12:13:26 -0000 On Jan 20, 2012, at 1:30 PM, Alexander Motin wrote: > On 01/20/12 13:08, Nikolay Denev wrote: >> On 20.01.2012, at 12:51, Alexander Motin wrote: >>=20 >>> On 01/20/12 10:09, Nikolay Denev wrote: >>>> Another thing I've observed is that active/active probably only = makes sense if you are accessing single LUN. >>>> In my tests where I have 24 LUNS that form 4 vdevs in a single = zpool, the highest performance was achieved >>>> when I split the active paths among the controllers installed in = the server importing the pool. (basically "gmultipath rotate $LUN" in = rc.local for half of the paths) >>>> Using active/active in this situation resulted in fluctuating = performance. >>>=20 >>> How big was fluctuation? Between speed of one and all paths? >>>=20 >>> Several active/active devices without knowledge about each other = with some probability will send part of requests via the same links, = while ZFS itself already does some balancing between vdevs. >>>=20 >>> -- >>> Alexander Motin >>=20 >> I will test in a bit and post results. >>=20 >> P.S.: Is there a way to enable/disable active-active on the fly? I'm >> currently re-labeling to achieve that. >=20 > No, there is not now. But for experiments you may achieve the same = results by manually marking as failed all paths except one. It is not = dangerous, as if that link fail, all other will resurrect automatically. >=20 > --=20 > Alexander Motin I had to destroy and relabel anyways, since I was not using = active-active currently. Here's what I did (maybe a little too verbose): gmultipath label -A -v LD_0 /dev/da0 /dev/da24=20 gmultipath label -A -v LD_1 /dev/da1 /dev/da25=20 gmultipath label -A -v LD_2 /dev/da2 /dev/da26=20 gmultipath label -A -v LD_3 /dev/da3 /dev/da27=20 gmultipath label -A -v LD_4 /dev/da4 /dev/da28=20 gmultipath label -A -v LD_5 /dev/da5 /dev/da29=20 gmultipath label -A -v LD_6 /dev/da6 /dev/da30=20 gmultipath label -A -v LD_7 /dev/da7 /dev/da31=20 gmultipath label -A -v LD_8 /dev/da8 /dev/da32=20 gmultipath label -A -v LD_9 /dev/da9 /dev/da33=20 gmultipath label -A -v LD_10 /dev/da10 /dev/da34=20 gmultipath label -A -v LD_11 /dev/da11 /dev/da35=20 gmultipath label -A -v LD_12 /dev/da12 /dev/da36=20 gmultipath label -A -v LD_13 /dev/da13 /dev/da37=20 gmultipath label -A -v LD_14 /dev/da14 /dev/da38=20 gmultipath label -A -v LD_15 /dev/da15 /dev/da39=20 gmultipath label -A -v LD_16 /dev/da16 /dev/da40=20 gmultipath label -A -v LD_17 /dev/da17 /dev/da41=20 gmultipath label -A -v LD_18 /dev/da18 /dev/da42=20 gmultipath label -A -v LD_19 /dev/da19 /dev/da43=20 gmultipath label -A -v LD_20 /dev/da20 /dev/da44=20 gmultipath label -A -v LD_21 /dev/da21 /dev/da45=20 gmultipath label -A -v LD_22 /dev/da22 /dev/da46=20 gmultipath label -A -v LD_23 /dev/da23 /dev/da47=20 :~# gmultipath status Name Status Components multipath/LD_0 OPTIMAL da0 (ACTIVE) da24 (ACTIVE) multipath/LD_1 OPTIMAL da1 (ACTIVE) da25 (ACTIVE) multipath/LD_2 OPTIMAL da2 (ACTIVE) da26 (ACTIVE) multipath/LD_3 OPTIMAL da3 (ACTIVE) da27 (ACTIVE) multipath/LD_4 OPTIMAL da4 (ACTIVE) da28 (ACTIVE) multipath/LD_5 OPTIMAL da5 (ACTIVE) da29 (ACTIVE) multipath/LD_6 OPTIMAL da6 (ACTIVE) da30 (ACTIVE) multipath/LD_7 OPTIMAL da7 (ACTIVE) da31 (ACTIVE) multipath/LD_8 OPTIMAL da8 (ACTIVE) da32 (ACTIVE) multipath/LD_9 OPTIMAL da9 (ACTIVE) da33 (ACTIVE) multipath/LD_10 OPTIMAL da10 (ACTIVE) da34 (ACTIVE) multipath/LD_11 OPTIMAL da11 (ACTIVE) da35 (ACTIVE) multipath/LD_12 OPTIMAL da12 (ACTIVE) da36 (ACTIVE) multipath/LD_13 OPTIMAL da13 (ACTIVE) da37 (ACTIVE) multipath/LD_14 OPTIMAL da14 (ACTIVE) da38 (ACTIVE) multipath/LD_15 OPTIMAL da15 (ACTIVE) da39 (ACTIVE) multipath/LD_16 OPTIMAL da16 (ACTIVE) da40 (ACTIVE) multipath/LD_17 OPTIMAL da17 (ACTIVE) da41 (ACTIVE) multipath/LD_18 OPTIMAL da18 (ACTIVE) da42 (ACTIVE) multipath/LD_19 OPTIMAL da19 (ACTIVE) da43 (ACTIVE) multipath/LD_20 OPTIMAL da20 (ACTIVE) da44 (ACTIVE) multipath/LD_21 OPTIMAL da21 (ACTIVE) da45 (ACTIVE) multipath/LD_22 OPTIMAL da22 (ACTIVE) da46 (ACTIVE) multipath/LD_23 OPTIMAL da23 (ACTIVE) da47 (ACTIVE) :~# zpool import tank :~# zpool status pool: tank state: ONLINE scan: none requested config: NAME STATE READ WRITE CKSUM tank ONLINE 0 0 0 raidz2-0 ONLINE 0 0 0 multipath/LD_0 ONLINE 0 0 0 multipath/LD_1 ONLINE 0 0 0 multipath/LD_2 ONLINE 0 0 0 multipath/LD_3 ONLINE 0 0 0 multipath/LD_4 ONLINE 0 0 0 multipath/LD_5 ONLINE 0 0 0 raidz2-1 ONLINE 0 0 0 multipath/LD_6 ONLINE 0 0 0 multipath/LD_7 ONLINE 0 0 0 multipath/LD_8 ONLINE 0 0 0 multipath/LD_9 ONLINE 0 0 0 multipath/LD_10 ONLINE 0 0 0 multipath/LD_11 ONLINE 0 0 0 raidz2-2 ONLINE 0 0 0 multipath/LD_12 ONLINE 0 0 0 multipath/LD_13 ONLINE 0 0 0 multipath/LD_14 ONLINE 0 0 0 multipath/LD_15 ONLINE 0 0 0 multipath/LD_16 ONLINE 0 0 0 multipath/LD_17 ONLINE 0 0 0 raidz2-3 ONLINE 0 0 0 multipath/LD_18 ONLINE 0 0 0 multipath/LD_19 ONLINE 0 0 0 multipath/LD_20 ONLINE 0 0 0 multipath/LD_21 ONLINE 0 0 0 multipath/LD_22 ONLINE 0 0 0 multipath/LD_23 ONLINE 0 0 0 errors: No known data errors And now a very naive benchmark : :~# dd if=3D/dev/zero of=3D/tank/TEST bs=3D1M count=3D512 =20 512+0 records in 512+0 records out 536870912 bytes transferred in 7.282780 secs (73717855 bytes/sec) :~# dd if=3D/dev/zero of=3D/tank/TEST bs=3D1M count=3D512 512+0 records in 512+0 records out 536870912 bytes transferred in 38.422724 secs (13972745 bytes/sec) :~# dd if=3D/dev/zero of=3D/tank/TEST bs=3D1M count=3D512 512+0 records in 512+0 records out 536870912 bytes transferred in 10.810989 secs (49659740 bytes/sec) Now deactivate the alternative paths : /sbin/gmultipath fail LD_0 da24 /sbin/gmultipath fail LD_1 da25 /sbin/gmultipath fail LD_2 da26 /sbin/gmultipath fail LD_3 da27 /sbin/gmultipath fail LD_4 da28 /sbin/gmultipath fail LD_5 da29 /sbin/gmultipath fail LD_6 da6 /sbin/gmultipath fail LD_7 da7 /sbin/gmultipath fail LD_8 da8 /sbin/gmultipath fail LD_9 da9 /sbin/gmultipath fail LD_10 da10 /sbin/gmultipath fail LD_11 da11 /sbin/gmultipath fail LD_12 da36 /sbin/gmultipath fail LD_13 da37 /sbin/gmultipath fail LD_14 da38 /sbin/gmultipath fail LD_15 da39 /sbin/gmultipath fail LD_16 da40 /sbin/gmultipath fail LD_17 da41 /sbin/gmultipath fail LD_18 da18 /sbin/gmultipath fail LD_19 da19 /sbin/gmultipath fail LD_20 da20 /sbin/gmultipath fail LD_21 da21 /sbin/gmultipath fail LD_22 da22 /sbin/gmultipath fail LD_23 da23 :~# gmultipath status Name Status Components multipath/LD_0 DEGRADED da0 (ACTIVE) da24 (FAIL) multipath/LD_1 DEGRADED da1 (ACTIVE) da25 (FAIL) multipath/LD_2 DEGRADED da2 (ACTIVE) da26 (FAIL) multipath/LD_3 DEGRADED da3 (ACTIVE) da27 (FAIL) multipath/LD_4 DEGRADED da4 (ACTIVE) da28 (FAIL) multipath/LD_5 DEGRADED da5 (ACTIVE) da29 (FAIL) multipath/LD_6 DEGRADED da6 (FAIL) da30 (ACTIVE) multipath/LD_7 DEGRADED da7 (FAIL) da31 (ACTIVE) multipath/LD_8 DEGRADED da8 (FAIL) da32 (ACTIVE) multipath/LD_9 DEGRADED da9 (FAIL) da33 (ACTIVE) multipath/LD_10 DEGRADED da10 (FAIL) da34 (ACTIVE) multipath/LD_11 DEGRADED da11 (FAIL) da35 (ACTIVE) multipath/LD_12 DEGRADED da12 (ACTIVE) da36 (FAIL) multipath/LD_13 DEGRADED da13 (ACTIVE) da37 (FAIL) multipath/LD_14 DEGRADED da14 (ACTIVE) da38 (FAIL) multipath/LD_15 DEGRADED da15 (ACTIVE) da39 (FAIL) multipath/LD_16 DEGRADED da16 (ACTIVE) da40 (FAIL) multipath/LD_17 DEGRADED da17 (ACTIVE) da41 (FAIL) multipath/LD_18 DEGRADED da18 (FAIL) da42 (ACTIVE) multipath/LD_19 DEGRADED da19 (FAIL) da43 (ACTIVE) multipath/LD_20 DEGRADED da20 (FAIL) da44 (ACTIVE) multipath/LD_21 DEGRADED da21 (FAIL) da45 (ACTIVE) multipath/LD_22 DEGRADED da22 (FAIL) da46 (ACTIVE) multipath/LD_23 DEGRADED da23 (FAIL) da47 (ACTIVE) And the benchmark again: :~# dd if=3D/dev/zero of=3D/tank/TEST bs=3D1M count=3D512 512+0 records in 512+0 records out 536870912 bytes transferred in 1.083226 secs (495622270 bytes/sec) :~# dd if=3D/dev/zero of=3D/tank/TEST bs=3D1M count=3D512 512+0 records in 512+0 records out 536870912 bytes transferred in 1.409975 secs (380766249 bytes/sec) :~# dd if=3D/dev/zero of=3D/tank/TEST bs=3D1M count=3D512 512+0 records in 512+0 records out 536870912 bytes transferred in 1.136110 secs (472551848 bytes/sec) P.S.: The server is running 8.2-STABLE, dual port isp(4) card, and is = directly connected to a 4Gbps Xyratex dual-controller (active-active) = storage array. All the 24 SAS drives are setup as single disk RAID0 LUNs.= From owner-freebsd-geom@FreeBSD.ORG Fri Jan 20 12:31:06 2012 Return-Path: Delivered-To: freebsd-geom@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id EEE18106566B; Fri, 20 Jan 2012 12:31:06 +0000 (UTC) (envelope-from mavbsd@gmail.com) Received: from mail-ee0-f54.google.com (mail-ee0-f54.google.com [74.125.83.54]) by mx1.freebsd.org (Postfix) with ESMTP id 235748FC16; Fri, 20 Jan 2012 12:31:05 +0000 (UTC) Received: by eekb47 with SMTP id b47so192272eek.13 for ; Fri, 20 Jan 2012 04:31:05 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=sender:message-id:date:from:user-agent:mime-version:to:cc:subject :references:in-reply-to:content-type:content-transfer-encoding; bh=ZlK0Z/o6ziCtaCoIm/Npq2fsoxrU6gzsK77Zcv7asY8=; b=naSsIC48MnbvSrt64xq8xy5gs7nz+lGcriDhXB7DCnyjZtqdB0Jgd0pbuxkhfdN0a3 Nee2uZ77VOWTuSbsOKMdc1XdhbiZu/B0CliwI0YVU7ZazzSsxm+USyIsv6avob91llsr 9QqpyLVgADvs458W6I7+kI0hEKP0qFwAML2EQ= Received: by 10.14.16.98 with SMTP id g74mr29210eeg.77.1327062664928; Fri, 20 Jan 2012 04:31:04 -0800 (PST) Received: from mavbook2.mavhome.dp.ua (pc.mavhome.dp.ua. [212.86.226.226]) by mx.google.com with ESMTPS id y12sm10860897eeb.11.2012.01.20.04.31.02 (version=SSLv3 cipher=OTHER); Fri, 20 Jan 2012 04:31:03 -0800 (PST) Sender: Alexander Motin Message-ID: <4F195E85.4010708@FreeBSD.org> Date: Fri, 20 Jan 2012 14:31:01 +0200 From: Alexander Motin User-Agent: Mozilla/5.0 (X11; FreeBSD amd64; rv:9.0) Gecko/20111227 Thunderbird/9.0 MIME-Version: 1.0 To: Nikolay Denev References: <4EAF00A6.5060903@FreeBSD.org> <05E0E64F-5EC4-425A-81E4-B6C35320608B@neveragain.de> <4EB05566.3060700@FreeBSD.org> <20111114210957.GA68559@in-addr.com> <059C17DB-3A7B-41AA-BF91-2F8EBAF17D01@gmail.com> <4F19474A.9020600@FreeBSD.org> <-2439788735531654851@unknownmsgid> <4F19503B.2090200@FreeBSD.org> <25C45DA0-4B52-42E4-A1A3-DD5168451423@gmail.com> In-Reply-To: <25C45DA0-4B52-42E4-A1A3-DD5168451423@gmail.com> Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit Cc: Gary Palmer , FreeBSD-Current , Dennis K?gel , "freebsd-geom@freebsd.org" Subject: Re: RFC: GEOM MULTIPATH rewrite X-BeenThere: freebsd-geom@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: GEOM-specific discussions and implementations List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 20 Jan 2012 12:31:07 -0000 On 01/20/12 14:13, Nikolay Denev wrote: > On Jan 20, 2012, at 1:30 PM, Alexander Motin wrote: >> On 01/20/12 13:08, Nikolay Denev wrote: >>> On 20.01.2012, at 12:51, Alexander Motin wrote: >>> >>>> On 01/20/12 10:09, Nikolay Denev wrote: >>>>> Another thing I've observed is that active/active probably only makes sense if you are accessing single LUN. >>>>> In my tests where I have 24 LUNS that form 4 vdevs in a single zpool, the highest performance was achieved >>>>> when I split the active paths among the controllers installed in the server importing the pool. (basically "gmultipath rotate $LUN" in rc.local for half of the paths) >>>>> Using active/active in this situation resulted in fluctuating performance. >>>> >>>> How big was fluctuation? Between speed of one and all paths? >>>> >>>> Several active/active devices without knowledge about each other with some probability will send part of requests via the same links, while ZFS itself already does some balancing between vdevs. >>> >>> I will test in a bit and post results. >>> >>> P.S.: Is there a way to enable/disable active-active on the fly? I'm >>> currently re-labeling to achieve that. >> >> No, there is not now. But for experiments you may achieve the same results by manually marking as failed all paths except one. It is not dangerous, as if that link fail, all other will resurrect automatically. > > I had to destroy and relabel anyways, since I was not using active-active currently. Here's what I did (maybe a little too verbose): > > And now a very naive benchmark : > > :~# dd if=/dev/zero of=/tank/TEST bs=1M count=512 > 512+0 records in > 512+0 records out > 536870912 bytes transferred in 7.282780 secs (73717855 bytes/sec) > :~# dd if=/dev/zero of=/tank/TEST bs=1M count=512 > 512+0 records in > 512+0 records out > 536870912 bytes transferred in 38.422724 secs (13972745 bytes/sec) > :~# dd if=/dev/zero of=/tank/TEST bs=1M count=512 > 512+0 records in > 512+0 records out > 536870912 bytes transferred in 10.810989 secs (49659740 bytes/sec) > > Now deactivate the alternative paths : > And the benchmark again: > > :~# dd if=/dev/zero of=/tank/TEST bs=1M count=512 > 512+0 records in > 512+0 records out > 536870912 bytes transferred in 1.083226 secs (495622270 bytes/sec) > :~# dd if=/dev/zero of=/tank/TEST bs=1M count=512 > 512+0 records in > 512+0 records out > 536870912 bytes transferred in 1.409975 secs (380766249 bytes/sec) > :~# dd if=/dev/zero of=/tank/TEST bs=1M count=512 > 512+0 records in > 512+0 records out > 536870912 bytes transferred in 1.136110 secs (472551848 bytes/sec) > > P.S.: The server is running 8.2-STABLE, dual port isp(4) card, and is directly connected to a 4Gbps Xyratex dual-controller (active-active) storage array. > All the 24 SAS drives are setup as single disk RAID0 LUNs. This difference is too huge to explain it with ineffective paths utilization. Can't this storage have some per-LUN port/controller affinity that may penalize concurrent access to the same LUN from different paths? Can't it be active/active on port level, but active/passive for each specific LUN? If there are really two controllers inside, they may need to synchronize their caches or bounce requests, that may be expensive. -- Alexander Motin From owner-freebsd-geom@FreeBSD.ORG Fri Jan 20 13:27:09 2012 Return-Path: Delivered-To: freebsd-geom@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 4ABF41065670; Fri, 20 Jan 2012 13:27:09 +0000 (UTC) (envelope-from ndenev@gmail.com) Received: from mail-ee0-f54.google.com (mail-ee0-f54.google.com [74.125.83.54]) by mx1.freebsd.org (Postfix) with ESMTP id 021668FC1A; Fri, 20 Jan 2012 13:27:07 +0000 (UTC) Received: by eekb47 with SMTP id b47so216608eek.13 for ; Fri, 20 Jan 2012 05:27:07 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=subject:mime-version:content-type:from:in-reply-to:date:cc :content-transfer-encoding:message-id:references:to:x-mailer; bh=fd3MQ3MYtmhgMNcJ2XLEefOaHPXVp8VWsmQVQo/OhFo=; b=wZdSmprI9yu3IpkbiGojtSnpXLEAV3rgLNs1jwVJHeeczDV6ASOqIssDU9/ioMDEs8 EPTMxR7TnZ8lrAzLkSGtMCH8d1SN6VZ9TGyicUFuw0hLdwGakh5f8VJ9vvBQZRMsRFug qP9+dxUQClUshnnOCf1kcCydAvqegJtNCXoJ0= Received: by 10.14.3.154 with SMTP id 26mr3244979eeh.40.1327066027057; Fri, 20 Jan 2012 05:27:07 -0800 (PST) Received: from ndenevsa.sf.moneybookers.net (g1.moneybookers.com. [217.18.249.148]) by mx.google.com with ESMTPS id c16sm11558558eei.1.2012.01.20.05.27.04 (version=TLSv1/SSLv3 cipher=OTHER); Fri, 20 Jan 2012 05:27:05 -0800 (PST) Mime-Version: 1.0 (Apple Message framework v1251.1) Content-Type: text/plain; charset=iso-8859-1 From: Nikolay Denev In-Reply-To: <4F195E85.4010708@FreeBSD.org> Date: Fri, 20 Jan 2012 15:27:03 +0200 Content-Transfer-Encoding: quoted-printable Message-Id: References: <4EAF00A6.5060903@FreeBSD.org> <05E0E64F-5EC4-425A-81E4-B6C35320608B@neveragain.de> <4EB05566.3060700@FreeBSD.org> <20111114210957.GA68559@in-addr.com> <059C17DB-3A7B-41AA-BF91-2F8EBAF17D01@gmail.com> <4F19474A.9020600@FreeBSD.org> <-2439788735531654851@unknownmsgid> <4F19503B.2090200@FreeBSD.org> <25C45DA0-4B52-42E4-A1A3-DD5168451423@gmail.com> <4F195E85.4010708@FreeBSD.org> To: Alexander Motin X-Mailer: Apple Mail (2.1251.1) Cc: Gary Palmer , FreeBSD-Current , Dennis K?gel , "freebsd-geom@freebsd.org" Subject: Re: RFC: GEOM MULTIPATH rewrite X-BeenThere: freebsd-geom@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: GEOM-specific discussions and implementations List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 20 Jan 2012 13:27:09 -0000 On Jan 20, 2012, at 2:31 PM, Alexander Motin wrote: > On 01/20/12 14:13, Nikolay Denev wrote: >> On Jan 20, 2012, at 1:30 PM, Alexander Motin wrote: >>> On 01/20/12 13:08, Nikolay Denev wrote: >>>> On 20.01.2012, at 12:51, Alexander Motin wrote: >>>>=20 >>>>> On 01/20/12 10:09, Nikolay Denev wrote: >>>>>> Another thing I've observed is that active/active probably only = makes sense if you are accessing single LUN. >>>>>> In my tests where I have 24 LUNS that form 4 vdevs in a single = zpool, the highest performance was achieved >>>>>> when I split the active paths among the controllers installed in = the server importing the pool. (basically "gmultipath rotate $LUN" in = rc.local for half of the paths) >>>>>> Using active/active in this situation resulted in fluctuating = performance. >>>>>=20 >>>>> How big was fluctuation? Between speed of one and all paths? >>>>>=20 >>>>> Several active/active devices without knowledge about each other = with some probability will send part of requests via the same links, = while ZFS itself already does some balancing between vdevs. >>>>=20 >>>> I will test in a bit and post results. >>>>=20 >>>> P.S.: Is there a way to enable/disable active-active on the fly? = I'm >>>> currently re-labeling to achieve that. >>>=20 >>> No, there is not now. But for experiments you may achieve the same = results by manually marking as failed all paths except one. It is not = dangerous, as if that link fail, all other will resurrect automatically. >>=20 >> I had to destroy and relabel anyways, since I was not using = active-active currently. Here's what I did (maybe a little too verbose): >>=20 >> And now a very naive benchmark : >>=20 >> :~# dd if=3D/dev/zero of=3D/tank/TEST bs=3D1M count=3D512 >> 512+0 records in >> 512+0 records out >> 536870912 bytes transferred in 7.282780 secs (73717855 bytes/sec) >> :~# dd if=3D/dev/zero of=3D/tank/TEST bs=3D1M count=3D512 >> 512+0 records in >> 512+0 records out >> 536870912 bytes transferred in 38.422724 secs (13972745 bytes/sec) >> :~# dd if=3D/dev/zero of=3D/tank/TEST bs=3D1M count=3D512 >> 512+0 records in >> 512+0 records out >> 536870912 bytes transferred in 10.810989 secs (49659740 bytes/sec) >>=20 >> Now deactivate the alternative paths : >> And the benchmark again: >>=20 >> :~# dd if=3D/dev/zero of=3D/tank/TEST bs=3D1M count=3D512 >> 512+0 records in >> 512+0 records out >> 536870912 bytes transferred in 1.083226 secs (495622270 bytes/sec) >> :~# dd if=3D/dev/zero of=3D/tank/TEST bs=3D1M count=3D512 >> 512+0 records in >> 512+0 records out >> 536870912 bytes transferred in 1.409975 secs (380766249 bytes/sec) >> :~# dd if=3D/dev/zero of=3D/tank/TEST bs=3D1M count=3D512 >> 512+0 records in >> 512+0 records out >> 536870912 bytes transferred in 1.136110 secs (472551848 bytes/sec) >>=20 >> P.S.: The server is running 8.2-STABLE, dual port isp(4) card, and is = directly connected to a 4Gbps Xyratex dual-controller (active-active) = storage array. >> All the 24 SAS drives are setup as single disk RAID0 LUNs. >=20 > This difference is too huge to explain it with ineffective paths = utilization. Can't this storage have some per-LUN port/controller = affinity that may penalize concurrent access to the same LUN from = different paths? Can't it be active/active on port level, but = active/passive for each specific LUN? If there are really two = controllers inside, they may need to synchronize their caches or bounce = requests, that may be expensive. >=20 > --=20 > Alexander Motin Yes, I think that's what's happening. There are two controllers each = with it's own CPU and cache and have cache synchronization enabled. I will try to test multipath if both paths are connected to the same = controller (there are two ports on each controller). But that will = require remote hands and take some time. In the mean time I've now disabled the writeback cache on the array = (this disables also the cache synchronization) and here are the results = : ACTIVE-ACTIVE: :~# dd if=3D/dev/zero of=3D/tank/TEST bs=3D1M count=3D512 512+0 records in 512+0 records out 536870912 bytes transferred in 2.497415 secs (214970639 bytes/sec) :~# dd if=3D/dev/zero of=3D/tank/TEST bs=3D1M count=3D512 512+0 records in 512+0 records out 536870912 bytes transferred in 1.076070 secs (498918172 bytes/sec) :~# dd if=3D/dev/zero of=3D/tank/TEST bs=3D1M count=3D512 512+0 records in 512+0 records out 536870912 bytes transferred in 1.908101 secs (281363979 bytes/sec) ACTIVE-PASSIVE (half of the paths failed the same way as in the previous = email): :~# dd if=3D/dev/zero of=3D/tank/TEST bs=3D1M count=3D512 512+0 records in 512+0 records out 536870912 bytes transferred in 0.324483 secs (1654542913 bytes/sec) :~# dd if=3D/dev/zero of=3D/tank/TEST bs=3D1M count=3D512 512+0 records in 512+0 records out 536870912 bytes transferred in 0.795685 secs (674727909 bytes/sec) :~# dd if=3D/dev/zero of=3D/tank/TEST bs=3D1M count=3D512 512+0 records in 512+0 records out 536870912 bytes transferred in 0.233859 secs (2295702835 bytes/sec) This increased the performance for both cases, probably because = writeback caching does nothing for large sequential writes. Anyways, here ACTIVE-ACTIVE is still slower, but not by that much. From owner-freebsd-geom@FreeBSD.ORG Fri Jan 20 13:39:03 2012 Return-Path: Delivered-To: freebsd-geom@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id DDA80106566B; Fri, 20 Jan 2012 13:39:03 +0000 (UTC) (envelope-from mavbsd@gmail.com) Received: from mail-ee0-f54.google.com (mail-ee0-f54.google.com [74.125.83.54]) by mx1.freebsd.org (Postfix) with ESMTP id 0928E8FC0C; Fri, 20 Jan 2012 13:39:02 +0000 (UTC) Received: by eekb47 with SMTP id b47so221778eek.13 for ; Fri, 20 Jan 2012 05:39:01 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=sender:message-id:date:from:user-agent:mime-version:to:cc:subject :references:in-reply-to:content-type:content-transfer-encoding; bh=A9XzUwUZFcs0Xs66OHmjxgZwuoB3PgAm7gliTmbgpcU=; b=hD2lHQC7fwVrZfPosmNYVkRWM+MdMCIWk5gSVPEQlyVNeJt9REmfGZ3gyUKraYtAj4 v4LKj4mP2/WnjU0BEUYnH3AVKspjJYpXg8MbU4H/HM7GprQHPideZ98FIlO8nZ1CFyeP lAbHruHgNJpVzS5Avm+PMgJZKbLpW7PESmOWg= Received: by 10.14.14.7 with SMTP id c7mr3066953eec.89.1327066741830; Fri, 20 Jan 2012 05:39:01 -0800 (PST) Received: from mavbook2.mavhome.dp.ua (pc.mavhome.dp.ua. [212.86.226.226]) by mx.google.com with ESMTPS id x43sm11563793eef.8.2012.01.20.05.38.59 (version=SSLv3 cipher=OTHER); Fri, 20 Jan 2012 05:39:00 -0800 (PST) Sender: Alexander Motin Message-ID: <4F196E72.40903@FreeBSD.org> Date: Fri, 20 Jan 2012 15:38:58 +0200 From: Alexander Motin User-Agent: Mozilla/5.0 (X11; FreeBSD amd64; rv:9.0) Gecko/20111227 Thunderbird/9.0 MIME-Version: 1.0 To: Nikolay Denev References: <4EAF00A6.5060903@FreeBSD.org> <05E0E64F-5EC4-425A-81E4-B6C35320608B@neveragain.de> <4EB05566.3060700@FreeBSD.org> <20111114210957.GA68559@in-addr.com> <059C17DB-3A7B-41AA-BF91-2F8EBAF17D01@gmail.com> <4F19474A.9020600@FreeBSD.org> <-2439788735531654851@unknownmsgid> <4F19503B.2090200@FreeBSD.org> <25C45DA0-4B52-42E4-A1A3-DD5168451423@gmail.com> <4F195E85.4010708@FreeBSD.org> In-Reply-To: Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit Cc: Gary Palmer , FreeBSD-Current , Dennis K?gel , "freebsd-geom@freebsd.org" Subject: Re: RFC: GEOM MULTIPATH rewrite X-BeenThere: freebsd-geom@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: GEOM-specific discussions and implementations List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 20 Jan 2012 13:39:04 -0000 On 01/20/12 15:27, Nikolay Denev wrote: > > On Jan 20, 2012, at 2:31 PM, Alexander Motin wrote: > >> On 01/20/12 14:13, Nikolay Denev wrote: >>> On Jan 20, 2012, at 1:30 PM, Alexander Motin wrote: >>>> On 01/20/12 13:08, Nikolay Denev wrote: >>>>> On 20.01.2012, at 12:51, Alexander Motin wrote: >>>>> >>>>>> On 01/20/12 10:09, Nikolay Denev wrote: >>>>>>> Another thing I've observed is that active/active probably only makes sense if you are accessing single LUN. >>>>>>> In my tests where I have 24 LUNS that form 4 vdevs in a single zpool, the highest performance was achieved >>>>>>> when I split the active paths among the controllers installed in the server importing the pool. (basically "gmultipath rotate $LUN" in rc.local for half of the paths) >>>>>>> Using active/active in this situation resulted in fluctuating performance. >>>>>> >>>>>> How big was fluctuation? Between speed of one and all paths? >>>>>> >>>>>> Several active/active devices without knowledge about each other with some probability will send part of requests via the same links, while ZFS itself already does some balancing between vdevs. >>>>> >>>>> I will test in a bit and post results. >>>>> >>>>> P.S.: Is there a way to enable/disable active-active on the fly? I'm >>>>> currently re-labeling to achieve that. >>>> >>>> No, there is not now. But for experiments you may achieve the same results by manually marking as failed all paths except one. It is not dangerous, as if that link fail, all other will resurrect automatically. >>> >>> I had to destroy and relabel anyways, since I was not using active-active currently. Here's what I did (maybe a little too verbose): >>> >>> And now a very naive benchmark : >>> >>> :~# dd if=/dev/zero of=/tank/TEST bs=1M count=512 >>> 512+0 records in >>> 512+0 records out >>> 536870912 bytes transferred in 7.282780 secs (73717855 bytes/sec) >>> :~# dd if=/dev/zero of=/tank/TEST bs=1M count=512 >>> 512+0 records in >>> 512+0 records out >>> 536870912 bytes transferred in 38.422724 secs (13972745 bytes/sec) >>> :~# dd if=/dev/zero of=/tank/TEST bs=1M count=512 >>> 512+0 records in >>> 512+0 records out >>> 536870912 bytes transferred in 10.810989 secs (49659740 bytes/sec) >>> >>> Now deactivate the alternative paths : >>> And the benchmark again: >>> >>> :~# dd if=/dev/zero of=/tank/TEST bs=1M count=512 >>> 512+0 records in >>> 512+0 records out >>> 536870912 bytes transferred in 1.083226 secs (495622270 bytes/sec) >>> :~# dd if=/dev/zero of=/tank/TEST bs=1M count=512 >>> 512+0 records in >>> 512+0 records out >>> 536870912 bytes transferred in 1.409975 secs (380766249 bytes/sec) >>> :~# dd if=/dev/zero of=/tank/TEST bs=1M count=512 >>> 512+0 records in >>> 512+0 records out >>> 536870912 bytes transferred in 1.136110 secs (472551848 bytes/sec) >>> >>> P.S.: The server is running 8.2-STABLE, dual port isp(4) card, and is directly connected to a 4Gbps Xyratex dual-controller (active-active) storage array. >>> All the 24 SAS drives are setup as single disk RAID0 LUNs. >> >> This difference is too huge to explain it with ineffective paths utilization. Can't this storage have some per-LUN port/controller affinity that may penalize concurrent access to the same LUN from different paths? Can't it be active/active on port level, but active/passive for each specific LUN? If there are really two controllers inside, they may need to synchronize their caches or bounce requests, that may be expensive. >> >> -- >> Alexander Motin > > Yes, I think that's what's happening. There are two controllers each with it's own CPU and cache and have cache synchronization enabled. > I will try to test multipath if both paths are connected to the same controller (there are two ports on each controller). But that will require remote hands and take some time. > > In the mean time I've now disabled the writeback cache on the array (this disables also the cache synchronization) and here are the results : > > ACTIVE-ACTIVE: > > :~# dd if=/dev/zero of=/tank/TEST bs=1M count=512 > 512+0 records in > 512+0 records out > 536870912 bytes transferred in 2.497415 secs (214970639 bytes/sec) > :~# dd if=/dev/zero of=/tank/TEST bs=1M count=512 > 512+0 records in > 512+0 records out > 536870912 bytes transferred in 1.076070 secs (498918172 bytes/sec) > :~# dd if=/dev/zero of=/tank/TEST bs=1M count=512 > 512+0 records in > 512+0 records out > 536870912 bytes transferred in 1.908101 secs (281363979 bytes/sec) > > ACTIVE-PASSIVE (half of the paths failed the same way as in the previous email): > > :~# dd if=/dev/zero of=/tank/TEST bs=1M count=512 > 512+0 records in > 512+0 records out > 536870912 bytes transferred in 0.324483 secs (1654542913 bytes/sec) > :~# dd if=/dev/zero of=/tank/TEST bs=1M count=512 > 512+0 records in > 512+0 records out > 536870912 bytes transferred in 0.795685 secs (674727909 bytes/sec) > :~# dd if=/dev/zero of=/tank/TEST bs=1M count=512 > 512+0 records in > 512+0 records out > 536870912 bytes transferred in 0.233859 secs (2295702835 bytes/sec) > > This increased the performance for both cases, probably because writeback caching does nothing for large sequential writes. > Anyways, here ACTIVE-ACTIVE is still slower, but not by that much. Thank you for numbers, but I have some doubts about them. 2295702835 bytes/sec is about 18Gbps. If you have 4Gbps links, that would need more then 4 of them, I think. -- Alexander Motin From owner-freebsd-geom@FreeBSD.ORG Fri Jan 20 14:08:00 2012 Return-Path: Delivered-To: freebsd-geom@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 3E1E3106566B; Fri, 20 Jan 2012 14:08:00 +0000 (UTC) (envelope-from ndenev@gmail.com) Received: from mail-ee0-f54.google.com (mail-ee0-f54.google.com [74.125.83.54]) by mx1.freebsd.org (Postfix) with ESMTP id EA58D8FC1B; Fri, 20 Jan 2012 14:07:58 +0000 (UTC) Received: by eekb47 with SMTP id b47so234789eek.13 for ; Fri, 20 Jan 2012 06:07:58 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=subject:mime-version:content-type:from:in-reply-to:date:cc :content-transfer-encoding:message-id:references:to:x-mailer; bh=glra8vDqgqnZKiOFKIYZmjjpW+Uny1sMZ48AMhtP4uM=; b=ggjcd8vsSxT3434Yl8EZjrXEDdRFFt7C0R2Ax9BE1Wnequ6h0cfb5+zzDRSaOh2fbd Z8GmPqJcGMGTFE7yvmsZdb/z0OnA1NW3q+JEakI0A/vq1rReCJNXFH16FuBPofKUmm27 RcIFvA6zxnUlFTNYx61iQjpDbgn2bQ6Efyx0c= Received: by 10.14.10.35 with SMTP id 35mr3398477eeu.121.1327068478045; Fri, 20 Jan 2012 06:07:58 -0800 (PST) Received: from ndenevsa.sf.moneybookers.net (g1.moneybookers.com. [217.18.249.148]) by mx.google.com with ESMTPS id 28sm11972338eed.0.2012.01.20.06.07.56 (version=TLSv1/SSLv3 cipher=OTHER); Fri, 20 Jan 2012 06:07:56 -0800 (PST) Mime-Version: 1.0 (Apple Message framework v1251.1) Content-Type: text/plain; charset=iso-8859-1 From: Nikolay Denev In-Reply-To: <4F196E72.40903@FreeBSD.org> Date: Fri, 20 Jan 2012 16:07:32 +0200 Content-Transfer-Encoding: quoted-printable Message-Id: <19A6D02C-C42C-4C9A-A8D0-C076E901F98F@gmail.com> References: <4EAF00A6.5060903@FreeBSD.org> <05E0E64F-5EC4-425A-81E4-B6C35320608B@neveragain.de> <4EB05566.3060700@FreeBSD.org> <20111114210957.GA68559@in-addr.com> <059C17DB-3A7B-41AA-BF91-2F8EBAF17D01@gmail.com> <4F19474A.9020600@FreeBSD.org> <-2439788735531654851@unknownmsgid> <4F19503B.2090200@FreeBSD.org> <25C45DA0-4B52-42E4-A1A3-DD5168451423@gmail.com> <4F195E85.4010708@FreeBSD.org> <4F196E72.40903@FreeBSD.org> To: Alexander Motin X-Mailer: Apple Mail (2.1251.1) Cc: Gary Palmer , FreeBSD-Current , Dennis K?gel , "freebsd-geom@freebsd.org" Subject: Re: RFC: GEOM MULTIPATH rewrite X-BeenThere: freebsd-geom@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: GEOM-specific discussions and implementations List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 20 Jan 2012 14:08:00 -0000 On Jan 20, 2012, at 3:38 PM, Alexander Motin wrote: > On 01/20/12 15:27, Nikolay Denev wrote: >>=20 >> On Jan 20, 2012, at 2:31 PM, Alexander Motin wrote: >>=20 >>> On 01/20/12 14:13, Nikolay Denev wrote: >>>> On Jan 20, 2012, at 1:30 PM, Alexander Motin wrote: >>>>> On 01/20/12 13:08, Nikolay Denev wrote: >>>>>> On 20.01.2012, at 12:51, Alexander Motin = wrote: >>>>>>=20 >>>>>>> On 01/20/12 10:09, Nikolay Denev wrote: >>>>>>>> Another thing I've observed is that active/active probably only = makes sense if you are accessing single LUN. >>>>>>>> In my tests where I have 24 LUNS that form 4 vdevs in a single = zpool, the highest performance was achieved >>>>>>>> when I split the active paths among the controllers installed = in the server importing the pool. (basically "gmultipath rotate $LUN" in = rc.local for half of the paths) >>>>>>>> Using active/active in this situation resulted in fluctuating = performance. >>>>>>>=20 >>>>>>> How big was fluctuation? Between speed of one and all paths? >>>>>>>=20 >>>>>>> Several active/active devices without knowledge about each other = with some probability will send part of requests via the same links, = while ZFS itself already does some balancing between vdevs. >>>>>>=20 >>>>>> I will test in a bit and post results. >>>>>>=20 >>>>>> P.S.: Is there a way to enable/disable active-active on the fly? = I'm >>>>>> currently re-labeling to achieve that. >>>>>=20 >>>>> No, there is not now. But for experiments you may achieve the same = results by manually marking as failed all paths except one. It is not = dangerous, as if that link fail, all other will resurrect automatically. >>>>=20 >>>> I had to destroy and relabel anyways, since I was not using = active-active currently. Here's what I did (maybe a little too verbose): >>>>=20 >>>> And now a very naive benchmark : >>>>=20 >>>> :~# dd if=3D/dev/zero of=3D/tank/TEST bs=3D1M count=3D512 >>>> 512+0 records in >>>> 512+0 records out >>>> 536870912 bytes transferred in 7.282780 secs (73717855 bytes/sec) >>>> :~# dd if=3D/dev/zero of=3D/tank/TEST bs=3D1M count=3D512 >>>> 512+0 records in >>>> 512+0 records out >>>> 536870912 bytes transferred in 38.422724 secs (13972745 bytes/sec) >>>> :~# dd if=3D/dev/zero of=3D/tank/TEST bs=3D1M count=3D512 >>>> 512+0 records in >>>> 512+0 records out >>>> 536870912 bytes transferred in 10.810989 secs (49659740 bytes/sec) >>>>=20 >>>> Now deactivate the alternative paths : >>>> And the benchmark again: >>>>=20 >>>> :~# dd if=3D/dev/zero of=3D/tank/TEST bs=3D1M count=3D512 >>>> 512+0 records in >>>> 512+0 records out >>>> 536870912 bytes transferred in 1.083226 secs (495622270 bytes/sec) >>>> :~# dd if=3D/dev/zero of=3D/tank/TEST bs=3D1M count=3D512 >>>> 512+0 records in >>>> 512+0 records out >>>> 536870912 bytes transferred in 1.409975 secs (380766249 bytes/sec) >>>> :~# dd if=3D/dev/zero of=3D/tank/TEST bs=3D1M count=3D512 >>>> 512+0 records in >>>> 512+0 records out >>>> 536870912 bytes transferred in 1.136110 secs (472551848 bytes/sec) >>>>=20 >>>> P.S.: The server is running 8.2-STABLE, dual port isp(4) card, and = is directly connected to a 4Gbps Xyratex dual-controller (active-active) = storage array. >>>> All the 24 SAS drives are setup as single disk RAID0 LUNs. >>>=20 >>> This difference is too huge to explain it with ineffective paths = utilization. Can't this storage have some per-LUN port/controller = affinity that may penalize concurrent access to the same LUN from = different paths? Can't it be active/active on port level, but = active/passive for each specific LUN? If there are really two = controllers inside, they may need to synchronize their caches or bounce = requests, that may be expensive. >>>=20 >>> -- >>> Alexander Motin >>=20 >> Yes, I think that's what's happening. There are two controllers each = with it's own CPU and cache and have cache synchronization enabled. >> I will try to test multipath if both paths are connected to the same = controller (there are two ports on each controller). But that will = require remote hands and take some time. >>=20 >> In the mean time I've now disabled the writeback cache on the array = (this disables also the cache synchronization) and here are the results = : >>=20 >> ACTIVE-ACTIVE: >>=20 >> :~# dd if=3D/dev/zero of=3D/tank/TEST bs=3D1M count=3D512 >> 512+0 records in >> 512+0 records out >> 536870912 bytes transferred in 2.497415 secs (214970639 bytes/sec) >> :~# dd if=3D/dev/zero of=3D/tank/TEST bs=3D1M count=3D512 >> 512+0 records in >> 512+0 records out >> 536870912 bytes transferred in 1.076070 secs (498918172 bytes/sec) >> :~# dd if=3D/dev/zero of=3D/tank/TEST bs=3D1M count=3D512 >> 512+0 records in >> 512+0 records out >> 536870912 bytes transferred in 1.908101 secs (281363979 bytes/sec) >>=20 >> ACTIVE-PASSIVE (half of the paths failed the same way as in the = previous email): >>=20 >> :~# dd if=3D/dev/zero of=3D/tank/TEST bs=3D1M count=3D512 >> 512+0 records in >> 512+0 records out >> 536870912 bytes transferred in 0.324483 secs (1654542913 bytes/sec) >> :~# dd if=3D/dev/zero of=3D/tank/TEST bs=3D1M count=3D512 >> 512+0 records in >> 512+0 records out >> 536870912 bytes transferred in 0.795685 secs (674727909 bytes/sec) >> :~# dd if=3D/dev/zero of=3D/tank/TEST bs=3D1M count=3D512 >> 512+0 records in >> 512+0 records out >> 536870912 bytes transferred in 0.233859 secs (2295702835 bytes/sec) >>=20 >> This increased the performance for both cases, probably because = writeback caching does nothing for large sequential writes. >> Anyways, here ACTIVE-ACTIVE is still slower, but not by that much. >=20 > Thank you for numbers, but I have some doubts about them. 2295702835 = bytes/sec is about 18Gbps. If you have 4Gbps links, that would need more = then 4 of them, I think. >=20 > --=20 > Alexander Motin Hmm, thats silly of me. 512M is just too small, and probably I've = benched the ZFS cache. (I have only two 4Gbps links to the array). Here's run with 8G file: ACTIVE-ACTIVE: # dd if=3D/dev/zero of=3D/tank/TEST bs=3D1M count=3D8096 8096+0 records in 8096+0 records out 8489271296 bytes transferred in 62.120919 secs (136657207 bytes/sec) # dd if=3D/dev/zero of=3D/tank/TEST bs=3D1M count=3D8096 8096+0 records in 8096+0 records out 8489271296 bytes transferred in 65.066861 secs (130469969 bytes/sec) # dd if=3D/dev/zero of=3D/tank/TEST bs=3D1M count=3D8096 8096+0 records in 8096+0 records out 8489271296 bytes transferred in 64.011907 secs (132620190 bytes/sec) ACTIVE-PASSIVE: # dd if=3D/dev/zero of=3D/tank/TEST bs=3D1M count=3D8096 8096+0 records in 8096+0 records out 8489271296 bytes transferred in 34.297121 secs (247521398 bytes/sec) # dd if=3D/dev/zero of=3D/tank/TEST bs=3D1M count=3D8096 8096+0 records in 8096+0 records out 8489271296 bytes transferred in 31.709855 secs (267717127 bytes/sec) # dd if=3D/dev/zero of=3D/tank/TEST bs=3D1M count=3D8096 8096+0 records in 8096+0 records out 8489271296 bytes transferred in 34.111564 secs (248867840 bytes/sec)