From owner-freebsd-fs@freebsd.org Tue Jan 22 12:04:57 2019 Return-Path: Delivered-To: freebsd-fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 116E414BA75B for ; Tue, 22 Jan 2019 12:04:57 +0000 (UTC) (envelope-from borjam@sarenet.es) Received: from cu1176c.smtpx.saremail.com (cu1176c.smtpx.saremail.com [195.16.148.151]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 31F3289091 for ; Tue, 22 Jan 2019 12:04:56 +0000 (UTC) (envelope-from borjam@sarenet.es) Received: from [172.16.8.5] (unknown [192.148.167.11]) by proxypop02.sare.net (Postfix) with ESMTPA id 5852A9DC715; Tue, 22 Jan 2019 13:04:46 +0100 (CET) Content-Type: text/plain; charset=utf-8 Mime-Version: 1.0 (Mac OS X Mail 12.2 \(3445.102.3\)) Subject: Re: ZFS on Hardware RAID From: Borja Marcos In-Reply-To: Date: Tue, 22 Jan 2019 13:04:45 +0100 Cc: Ireneusz Pluta , freebsd-fs Content-Transfer-Encoding: quoted-printable Message-Id: References: <1180280695.63420.1547910313494.JavaMail.zimbra@gausus.net> <92646202.63422.1547910433715.JavaMail.zimbra@gausus.net> To: andy thomas X-Mailer: Apple Mail (2.3445.102.3) X-Rspamd-Queue-Id: 31F3289091 X-Spamd-Bar: - Authentication-Results: mx1.freebsd.org; spf=pass (mx1.freebsd.org: domain of borjam@sarenet.es designates 195.16.148.151 as permitted sender) smtp.mailfrom=borjam@sarenet.es X-Spamd-Result: default: False [-1.19 / 15.00]; ARC_NA(0.00)[]; RCVD_VIA_SMTP_AUTH(0.00)[]; NEURAL_HAM_MEDIUM(-0.32)[-0.316,0]; FROM_HAS_DN(0.00)[]; RCPT_COUNT_THREE(0.00)[3]; R_SPF_ALLOW(-0.20)[+ip4:195.16.148.0/24]; MV_CASE(0.50)[]; MIME_GOOD(-0.10)[text/plain]; RCVD_TLS_LAST(0.00)[]; DMARC_NA(0.00)[sarenet.es]; NEURAL_HAM_LONG(-0.98)[-0.982,0]; TO_MATCH_ENVRCPT_SOME(0.00)[]; TO_DN_ALL(0.00)[]; MX_GOOD(-0.01)[cached: smtp.sarenet.es]; NEURAL_HAM_SHORT(-0.09)[-0.093,0]; RCVD_IN_DNSWL_NONE(0.00)[151.148.16.195.list.dnswl.org : 127.0.10.0]; IP_SCORE(0.01)[country: ES(0.05)]; FROM_EQ_ENVFROM(0.00)[]; R_DKIM_NA(0.00)[]; MIME_TRACE(0.00)[0:+]; ASN(0.00)[asn:3262, ipnet:195.16.128.0/19, country:ES]; FREEMAIL_CC(0.00)[wp.pl]; MID_RHS_MATCH_FROM(0.00)[]; RCVD_COUNT_TWO(0.00)[2] X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 22 Jan 2019 12:04:57 -0000 > On 22 Jan 2019, at 12:15, andy thomas wrote: >=20 >=20 > Yesterday I set up a spare Dell 2950 with Perc 5/i Integrated HBA and = six 73 GB SAS disks, with the first two disks configured as a RAID 1 = system disk (/dev/mfid0) and the remaining 4 disks as RAID 0 (mfid1- = mfid4). After adding a freebsd-zfs GPT partition to each of these 4 = disks I then created a RAIDz1 pool using mfid1p1, mfid2p1 and mfid3p1 = with mfid4p1 as a spare, followed by creating a simple ZFS filesystem. >=20 > After copying a few hundred MB of files to the ZFS filesystem, I = yanked /dev/mfid3 out to simulate a disk failure. I was then able to = manually detach the failed disk and replace it with the spare. Later, = after pushing /dev/mfid3 back in followed by a reboot and scrubbing the = pool, mfid4 automatically replaced the former mfid3 that was pulled out = and mfid3 became the new spare. You shouldn=E2=80=99t require a reboot. If the actual targets are = exposed to the CAM layer your disks will appear as =E2=80=9Cda=E2=80=9D = (SAS backplane) and you can offline a device,=20 hot plug a new one, at most do a =E2=80=9Ccamcontrol rescan=E2=80=9D to = detect it and run a zfs replace (or whatever) without stopping the = system.=20 If your drives are =E2=80=9Cmfid=E2=80=9D devices you may need either a = reboot or some magic rituals using =E2=80=9Csas2ircu=E2=80=9D or = =E2=80=9Csas3ircu=E2=80=9D to have the controller recognize the new = drive and accept it as a valid volume. > So a spare disk replacing a failed disk seems to be semi-automatic in = FreeBSD (this was version 10.3) although I have seen fully automatic = replacement on a Solaris parc platform. There are several stages at play here: 1- Starting up and recognizing a SAS or SATA drive.=20 2- Having it recognized as a volume by a RAID card. With LSI cards and = single disk RAID0 volumes it may require a reboot or using the = sas2ircu/sas3ircu utility. 3- ZFS replacement, which on Solaris can be automatic and on FreeBSD is = done manually (I haven=E2=80=99t tried zfsd yet). Borja.=