From owner-freebsd-questions@FreeBSD.ORG Wed Apr 17 23:10:38 2013 Return-Path: Delivered-To: freebsd-questions@freebsd.org Received: from mx1.freebsd.org (mx1.FreeBSD.org [8.8.178.115]) by hub.freebsd.org (Postfix) with ESMTP id 99052A8D; Wed, 17 Apr 2013 23:10:38 +0000 (UTC) (envelope-from outbackdingo@gmail.com) Received: from mail-oa0-f48.google.com (mail-oa0-f48.google.com [209.85.219.48]) by mx1.freebsd.org (Postfix) with ESMTP id 57201CF8; Wed, 17 Apr 2013 23:10:37 +0000 (UTC) Received: by mail-oa0-f48.google.com with SMTP id f4so402945oah.7 for ; Wed, 17 Apr 2013 16:10:37 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:x-received:in-reply-to:references:date:message-id :subject:from:to:cc:content-type; bh=RVKuCvPpvRAr0jhlZjUvBbLbsQTAEJItDCFn4jq8TTU=; b=H9wcNY/asZZBavexBU/VcPGb2VMbVj3oFJOsAResJ4gEw6gzf0gxDAdCHgJbVnMki1 MxPJLY4njJswWTDPuisP+IYzh7Lzd3N6lZZZSrO5GqoalOg/2o24Zppw7XNfT3cBCGH3 7z/I4bW4u5ja5KXdcqFKYvi6R8qUi1vniBSqYpUEjgVKpznNs7RfJLe1F/bgGqgd/Y/v dMLIV1/KjF+Vx1x6W2FP6x4D359DgnPeNfMEp4c0hGsIXqi6+Or6zDdtcG/pD3cGEdcn Ocs3JK+Otvr++DvdwGgaX36UyB3VElHs/o5976PS1LVdDPiMpkyyp3UlxAvHMZp5z+bz VTPA== MIME-Version: 1.0 X-Received: by 10.182.19.168 with SMTP id g8mr3821415obe.21.1366240237439; Wed, 17 Apr 2013 16:10:37 -0700 (PDT) Received: by 10.76.135.194 with HTTP; Wed, 17 Apr 2013 16:10:37 -0700 (PDT) In-Reply-To: <13CA24D6AB415D428143D44749F57D7201F05E0A@ltcfiswmsgmb21> References: <13CA24D6AB415D428143D44749F57D7201F05E0A@ltcfiswmsgmb21> Date: Wed, 17 Apr 2013 19:10:37 -0400 Message-ID: Subject: Re: gmultipath, ses and shared disks / cant seem to share between local nodes From: Outback Dingo To: Devin Teske Content-Type: text/plain; charset=windows-1252 Content-Transfer-Encoding: quoted-printable X-Content-Filtered-By: Mailman/MimeDel 2.1.14 Cc: FreeBSD Questions X-BeenThere: freebsd-questions@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: User questions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 17 Apr 2013 23:10:38 -0000 On Wed, Apr 17, 2013 at 6:39 PM, Teske, Devin wr= ote: > > On Apr 17, 2013, at 3:26 PM, Outback Dingo wrote: > > > Ok, maybe im at a loss here in the way my brain is viewing this > > > > we have a box, its got 2 nodes in the chassis, and 32 sata drives > > attached to a SATA/SAS backplane via 4 (2 per node) LSI MPT SAS2 cards > > should i not logically be seeing 4 controllers X #drive count ?? > > > > camcontrol devlist shows 32 devices, daX,passX and sesX,passX > > > > at scbus0 target 9 lun 0 (da0,pass0) > > at scbus0 target 10 lun 0 (ses0,pass1) > > at scbus0 target 11 lun 0 (da1,pass2= ) > > at scbus0 target 12 lun 0 (ses1,pass3) > > at scbus0 target 13 lun 0 (da2,pass4= ) > > at scbus0 target 14 lun 0 (ses2,pass5) > > at scbus0 target 15 lun 0 (da3,pass6= ) > > at scbus0 target 16 lun 0 (ses3,pass7) > > at scbus0 target 17 lun 0 (da4,pass8= ) > > at scbus0 target 18 lun 0 (ses4,pass9) > > at scbus0 target 19 lun 0 (da5,pass1= 0) > > at scbus0 target 20 lun 0 (ses5,pass11) > > at scbus0 target 21 lun 0 (da6,pass1= 2) > > at scbus0 target 22 lun 0 (ses6,pass13) > > at scbus0 target 23 lun 0 (da7,pass1= 4) > > at scbus0 target 24 lun 0 (ses7,pass15) > > at scbus1 target 0 lun 0 (da8,pass16= ) > > at scbus1 target 1 lun 0 (da9,pass17= ) > > at scbus8 target 10 lun 0 (ses8,pass19) > > at scbus8 target 11 lun 0 > (da11,pass20) > > at scbus8 target 12 lun 0 (ses9,pass21) > > at scbus8 target 13 lun 0 > (da12,pass22) > > at scbus8 target 14 lun 0 (ses10,pass23) > > at scbus8 target 15 lun 0 > (da13,pass24) > > at scbus8 target 16 lun 0 (ses11,pass25) > > at scbus8 target 17 lun 0 > (da14,pass26) > > at scbus8 target 18 lun 0 (ses12,pass27) > > at scbus8 target 19 lun 0 > (da15,pass28) > > at scbus8 target 20 lun 0 (ses13,pass29) > > at scbus8 target 21 lun 0 > (da16,pass30) > > at scbus8 target 22 lun 0 (ses14,pass31) > > at scbus8 target 23 lun 0 > (da17,pass32) > > at scbus8 target 24 lun 0 (ses15,pass33) > > at scbus9 target 0 lun 0 (da18,pass3= 4) > > > > > > we would like to create a zpool from all the devices, that in theory if > > nodeA failed > > then nodeB could force import the pool, > > gmultipath (which you mention in the subject) is the appropriate tool for > this, but there's no need for an import of the pool if you build the pool > out of multipath devices. In our experience, we can pull a cable and zfs > continues working just fine. > > In other words, don't build the pool out of the devices, put a gmultipath > label on each device and then use /dev/multipath/LABEL for the zpool > devices. > > > > nodeA and NodeB are attached through > > dual LSI controllers, to the SATA/SAS backplane. but i cant seem to > create > > a zpool from sesX or passX devices, i can however create a 16 drive zp0= ol > > on either node, from any daX device. what did i miss? ive looked at > > gmirror, and also ses documents. Any insight is appreciated, thanks in > > advance. > > gmirror is the wrong tool, gmultipath is what you want. The basic task is > to use "gmultipath label FOO da#" to write a cookie on the disk (used to > identify new/existing paths during GOEM "taste" events for example). > > After you've labeled the da# devices with gmultipath you say "gmultipath > status" to see the components of each label and you use "multipath/LABEL" > as your disk name when creating the zpool (these correspond directly to > /dev/multipath/LABEL, but "zpool create =85" or "zpool add =85" allow you= to > omit the leading "/dev"). > sanity check me on node A i did zpool destroy master gmultipath label FOO da0 gmultipath status Name Status Components multipath/FOO DEGRADED da0 (ACTIVE) multipath/FOO-619648737 DEGRADED da1 (ACTIVE) multipath/FOO-191725652 DEGRADED da2 (ACTIVE) multipath/FOO-1539342315 DEGRADED da3 (ACTIVE) multipath/FOO-1276041606 DEGRADED da4 (ACTIVE) multipath/FOO-2000832198 DEGRADED da5 (ACTIVE) multipath/FOO-1285640577 DEGRADED da6 (ACTIVE) multipath/FOO-1816092574 DEGRADED da7 (ACTIVE) multipath/FOO-1102254444 DEGRADED da8 (ACTIVE) multipath/FOO-330300690 DEGRADED da9 (ACTIVE) multipath/FOO-92140635 DEGRADED da10 (ACTIVE) multipath/FOO-855257672 DEGRADED da11 (ACTIVE) multipath/FOO-1003634134 DEGRADED da12 (ACTIVE) multipath/FOO-2449862 DEGRADED da13 (ACTIVE) multipath/FOO-1137080233 DEGRADED da14 (ACTIVE) multipath/FOO-1696804371 DEGRADED da15 (ACTIVE) multipath/FOO-1304457562 DEGRADED da16 (ACTIVE) multipath/FOO-912159854 DEGRADED da17 (ACTIVE) now on node B i should do the same? reboot both nodes and i should be able "see" 32 multipath/FOO deices to create a pool from ? > -- > Devin > > _____________ > The information contained in this message is proprietary and/or > confidential. If you are not the intended recipient, please: (i) delete t= he > message and all copies; (ii) do not disclose, distribute or use the messa= ge > in any manner; and (iii) notify the sender immediately. In addition, plea= se > be aware that any message addressed to our domain is subject to archiving > and review by persons other than the intended recipient. Thank you. >