Date: Thu, 12 May 2022 01:18:03 +0300 From: Christos Chatzaras <chris@cretaforce.gr> To: FreeBSD <freebsd-questions@freebsd.org> Subject: Re: ZFS replace a mirrored disk Message-ID: <B911CBE8-F384-4E94-99B4-A2D558CB7754@cretaforce.gr> In-Reply-To: <20220511121422.baddv74m65xsbwmj@x1> References: <C46DD702-BED7-466F-986D-289A4A56DC4F@cretaforce.gr> <20220511121422.baddv74m65xsbwmj@x1>
next in thread | previous in thread | raw e-mail | index | archive | help
[-- Attachment #1 --]
> First please define "without success", what doesn't work?
>
> please paste output of:
>
> $> gpart show nvd1
>
> also, is it an UEFI system or classicla BIOS with GPT? What FreeBSD
> version?
>
> zpool replace zroot nvd0 is invalid, you should use:
>
> $> zpool replace zroot nvd1 nvd0 (but it uses the entire disk, which is
> probably incorrect too)
It's legacy BIOS with GPT.
What I want to do is "simulate" a disk failure and rebuild the RAID-1.
First I run these commands from the main OS:
------------------------
$> gpart show
=> 40 7501476448 nvd0 GPT (3.5T)
40 1024 1 freebsd-boot (512K)
1064 984 - free - (492K)
2048 33554432 2 freebsd-swap (16G)
33556480 7467919360 3 freebsd-zfs (3.5T)
7501475840 648 - free - (324K)
=> 40 7501476448 nvd1 GPT (3.5T)
40 1024 1 freebsd-boot (512K)
1064 984 - free - (492K)
2048 33554432 2 freebsd-swap (16G)
33556480 7467919360 3 freebsd-zfs (3.5T)
7501475840 648 - free - (324K)
$> zpool status
pool: zroot
state: ONLINE
config:
NAME STATE READ WRITE CKSUM
zroot ONLINE 0 0 0
mirror-0 ONLINE 0 0 0
nvd0p3 ONLINE 0 0 0
nvd1p3 ONLINE 0 0 0
errors: No known data errors
------------------------------
Then I boot with mfsBSD and run this command to "simulate" a disk failure:
$> gpart destroy -F nvd0
------------------------------
Then I boot again in main OS and I run these commands:
$> zpool status
pool: zroot
state: DEGRADED
status: One or more devices could not be used because the label is missing or
invalid. Sufficient replicas exist for the pool to continue
functioning in a degraded state.
action: Replace the device using 'zpool replace'.
see: https://openzfs.github.io/openzfs-docs/msg/ZFS-8000-4J
config:
NAME STATE READ WRITE CKSUM
zroot DEGRADED 0 0 0
mirror-0 DEGRADED 0 0 0
nvd0 UNAVAIL 0 0 0 invalid label
nvd1p3 ONLINE 0 0 0
errors: No known data errors
$> gmirror status
Name Status Components
mirror/swap DEGRADED nvd1p2 (ACTIVE)
------------------------------
Then I backup / restore the partitions:
$> gpart backup nvd1 | gpart restore -F nvd0
$> gpart show
=> 40 7501476448 nvd1 GPT (3.5T)
40 1024 1 freebsd-boot (512K)
1064 984 - free - (492K)
2048 33554432 2 freebsd-swap (16G)
33556480 7467919360 3 freebsd-zfs (3.5T)
7501475840 648 - free - (324K)
=> 40 7501476448 nvd0 GPT (3.5T)
40 1024 1 freebsd-boot (512K)
1064 984 - free - (492K)
2048 33554432 2 freebsd-swap (16G)
33556480 7467919360 3 freebsd-zfs (3.5T)
7501475840 648 - free - (324K)
------------------------------
Without doing a "gmirror forget swap" and "gmirror insert swap /dev/nvd0p2" I see that swap is already mirrored:
$> gmirror status
Name Status Components
mirror/swap COMPLETE nvd1p2 (ACTIVE)
nvd0p2 (ACTIVE)
So first question is if the swap is mirrored automatically because nvd0 is the same disk (not replaced by a new disk).
-------------------------------
Then I write the bootloader:
$> gpart bootcode -b /boot/pmbr -p /boot/gptzfsboot -i 1 nvd0
--------------------------------
Then I want to add this disk to zpool but these commands don't do it:
$> zpool replace zroot nvd0
invalid vdev specification
use '-f' to override the following errors:
/dev/nvd0 is part of active pool 'zroot'
$> zpool replace -f zroot nvd0
invalid vdev specification
the following errors must be manually repaired:
/dev/nvd0 is part of active pool 'zroot'
-----------------------------------
Also these commands don't work:
$> zpool replace zroot nvd1 nvd0
invalid vdev specification
use '-f' to override the following errors:
/dev/nvd0 is part of active pool 'zroot'
$> zpool replace -f zroot nvd1 nvd0
invalid vdev specification
the following errors must be manually repaired:
/dev/nvd0 is part of active pool 'zroot'
-----------------------------------
Instead these commands work:
$> zpool offline zroot nvd0
zpool status
pool: zroot
state: DEGRADED
status: One or more devices has been taken offline by the administrator.
Sufficient replicas exist for the pool to continue functioning in a
degraded state.
action: Online the device using 'zpool online' or replace the device with
'zpool replace'.
config:
NAME STATE READ WRITE CKSUM
zroot DEGRADED 0 0 0
mirror-0 DEGRADED 0 0 0
nvd0 OFFLINE 0 0 0
nvd1p3 ONLINE 0 0 0
errors: No known data errors
$> zpool online zroot nvd0
$> zpool status
pool: zroot
state: ONLINE
scan: resilvered 5.55M in 00:00:00 with 0 errors on Thu May 12 00:22:13 2022
config:
NAME STATE READ WRITE CKSUM
zroot ONLINE 0 0 0
mirror-0 ONLINE 0 0 0
nvd0p3 ONLINE 0 0 0
nvd1p3 ONLINE 0 0 0
errors: No known data errors
------------------------------------
The second question is if instead of "zpool replace zroot nvd0" I had to use "zpool offline zroot nvd0" and "zpool online zroot nvd0" because nvd0 is the same disk (not replaced by a new disk).
Also I notice that if I don't do "zpool offline zroot nvd0" and "zpool online zroot nvd0" , but do a server reboot instead then zpool automatically puts nvd0 online:
$> zpool status
pool: zroot
state: ONLINE
status: One or more devices has experienced an unrecoverable error. An
attempt was made to correct the error. Applications are unaffected.
action: Determine if the device needs to be replaced, and clear the errors
using 'zpool clear' or replace the device with 'zpool replace'.
see: https://openzfs.github.io/openzfs-docs/msg/ZFS-8000-9P
scan: resilvered 3.50M in 00:00:00 with 0 errors on Thu May 12 01:04:09 2022
config:
NAME STATE READ WRITE CKSUM
zroot ONLINE 0 0 0
mirror-0 ONLINE 0 0 0
nvd0p3 ONLINE 0 0 2
nvd1p3 ONLINE 0 0 0
errors: No known data errors
$> zpool clear zroot
$> zpool status
pool: zroot
state: ONLINE
scan: resilvered 3.50M in 00:00:00 with 0 errors on Thu May 12 01:04:09 2022
config:
NAME STATE READ WRITE CKSUM
zroot ONLINE 0 0 0
mirror-0 ONLINE 0 0 0
nvd0p3 ONLINE 0 0 0
nvd1p3 ONLINE 0 0 0
errors: No known data errors
[-- Attachment #2 --]
<html><head><meta http-equiv="Content-Type" content="text/html; charset=us-ascii"></head><body style="word-wrap: break-word; -webkit-nbsp-mode: space; line-break: after-white-space;" class=""><div><br class=""><blockquote type="cite" class=""><div class=""><span style="caret-color: rgb(0, 0, 0); font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: 400; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none; float: none; display: inline !important;" class="">First please define "without success", what doesn't work?</span><br style="caret-color: rgb(0, 0, 0); font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: 400; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none;" class=""><br style="caret-color: rgb(0, 0, 0); font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: 400; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none;" class=""><span style="caret-color: rgb(0, 0, 0); font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: 400; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none; float: none; display: inline !important;" class="">please paste output of:</span><br style="caret-color: rgb(0, 0, 0); font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: 400; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none;" class=""><br style="caret-color: rgb(0, 0, 0); font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: 400; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none;" class=""><span style="caret-color: rgb(0, 0, 0); font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: 400; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none; float: none; display: inline !important;" class="">$> gpart show nvd1</span><br style="caret-color: rgb(0, 0, 0); font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: 400; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none;" class=""><br style="caret-color: rgb(0, 0, 0); font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: 400; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none;" class=""><span style="caret-color: rgb(0, 0, 0); font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: 400; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none; float: none; display: inline !important;" class="">also, is it an UEFI system or classicla BIOS with GPT? What FreeBSD</span><br style="caret-color: rgb(0, 0, 0); font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: 400; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none;" class=""><span style="caret-color: rgb(0, 0, 0); font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: 400; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none; float: none; display: inline !important;" class="">version?</span><br style="caret-color: rgb(0, 0, 0); font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: 400; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none;" class=""><br style="caret-color: rgb(0, 0, 0); font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: 400; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none;" class=""><span style="caret-color: rgb(0, 0, 0); font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: 400; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none; float: none; display: inline !important;" class="">zpool replace zroot nvd0 is invalid, you should use:</span><br style="caret-color: rgb(0, 0, 0); font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: 400; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none;" class=""><br style="caret-color: rgb(0, 0, 0); font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: 400; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none;" class=""><span style="caret-color: rgb(0, 0, 0); font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: 400; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none; float: none; display: inline !important;" class="">$> zpool replace zroot nvd1 nvd0 (but it uses the entire disk, which is</span><br style="caret-color: rgb(0, 0, 0); font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: 400; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none;" class=""><span style="caret-color: rgb(0, 0, 0); font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: 400; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none; float: none; display: inline !important;" class="">probably incorrect too)</span><br style="caret-color: rgb(0, 0, 0); font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: 400; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none;" class=""></div></blockquote></div><br class=""><div class=""><br class=""></div><div class=""><br class="">It's legacy BIOS with GPT.<br class=""><br class="">What I want to do is "simulate" a disk failure and rebuild the RAID-1.<br class=""><br class="">First I run these commands from the main OS:<br class=""><br class="">------------------------<br class=""><br class="">$> gpart show<br class="">=> 40 7501476448 nvd0 GPT (3.5T)<br class=""> 40 1024 1 freebsd-boot (512K)<br class=""> 1064 984 - free - (492K)<br class=""> 2048 33554432 2 freebsd-swap (16G)<br class=""> 33556480 7467919360 3 freebsd-zfs (3.5T)<br class=""> 7501475840 648 - free - (324K)<br class=""><br class="">=> 40 7501476448 nvd1 GPT (3.5T)<br class=""> 40 1024 1 freebsd-boot (512K)<br class=""> 1064 984 - free - (492K)<br class=""> 2048 33554432 2 freebsd-swap (16G)<br class=""> 33556480 7467919360 3 freebsd-zfs (3.5T)<br class=""> 7501475840 648 - free - (324K)<br class=""><br class=""><br class="">$> zpool status<br class=""> pool: zroot<br class=""> state: ONLINE<br class="">config:<br class=""><br class=""> NAME STATE READ WRITE CKSUM<br class=""> zroot ONLINE 0 0 0<br class=""> mirror-0 ONLINE 0 0 0<br class=""> nvd0p3 ONLINE 0 0 0<br class=""> nvd1p3 ONLINE 0 0 0<br class=""><br class="">errors: No known data errors<br class=""><br class=""><br class="">------------------------------<br class=""><br class=""><br class="">Then I boot with mfsBSD and run this command to "simulate" a disk failure:<br class=""><br class="">$> gpart destroy -F nvd0<br class=""><br class=""><br class="">------------------------------<br class=""><br class=""><br class="">Then I boot again in main OS and I run these commands:<br class=""><br class="">$> zpool status<br class=""> pool: zroot<br class=""> state: DEGRADED<br class="">status: One or more devices could not be used because the label is missing or<br class=""> invalid. Sufficient replicas exist for the pool to continue<br class=""> functioning in a degraded state.<br class="">action: Replace the device using 'zpool replace'.<br class=""> see: <a href="https://openzfs.github.io/openzfs-docs/msg/ZFS-8000-4J" class="">https://openzfs.github.io/openzfs-docs/msg/ZFS-8000-4J</a><br class="">config:<br class=""><br class=""> NAME STATE READ WRITE CKSUM<br class=""> zroot DEGRADED 0 0 0<br class=""> mirror-0 DEGRADED 0 0 0<br class=""> nvd0 UNAVAIL 0 0 0 invalid label<br class=""> nvd1p3 ONLINE 0 0 0<br class=""><br class="">errors: No known data errors<br class=""><br class=""><br class="">$> gmirror status<br class=""> Name Status Components<br class="">mirror/swap DEGRADED nvd1p2 (ACTIVE)<br class=""><br class=""><br class="">------------------------------<br class=""><br class=""><br class="">Then I backup / restore the partitions:<br class=""><br class="">$> gpart backup nvd1 | gpart restore -F nvd0<br class=""><br class="">$> gpart show<br class="">=> 40 7501476448 nvd1 GPT (3.5T)<br class=""> 40 1024 1 freebsd-boot (512K)<br class=""> 1064 984 - free - (492K)<br class=""> 2048 33554432 2 freebsd-swap (16G)<br class=""> 33556480 7467919360 3 freebsd-zfs (3.5T)<br class=""> 7501475840 648 - free - (324K)<br class=""><br class="">=> 40 7501476448 nvd0 GPT (3.5T)<br class=""> 40 1024 1 freebsd-boot (512K)<br class=""> 1064 984 - free - (492K)<br class=""> 2048 33554432 2 freebsd-swap (16G)<br class=""> 33556480 7467919360 3 freebsd-zfs (3.5T)<br class=""> 7501475840 648 - free - (324K)<br class=""><br class=""><br class="">------------------------------<br class=""><br class=""><br class="">Without doing a "gmirror forget swap" and "gmirror insert swap /dev/nvd0p2" I see that swap is already mirrored:<br class=""><br class="">$> gmirror status<br class=""> Name Status Components<br class="">mirror/swap COMPLETE nvd1p2 (ACTIVE)<br class=""> nvd0p2 (ACTIVE)<br class=""><br class="">So first question is if the swap is mirrored automatically because nvd0 is the same disk (not replaced by a new disk).<br class=""><br class=""><br class="">-------------------------------<br class=""><br class=""><br class="">Then I write the bootloader:<br class=""><br class="">$> gpart bootcode -b /boot/pmbr -p /boot/gptzfsboot -i 1 nvd0 <br class=""><br class="">--------------------------------<br class=""><br class=""><br class="">Then I want to add this disk to zpool but these commands don't do it:<br class=""><br class=""><br class="">$> zpool replace zroot nvd0<br class="">invalid vdev specification<br class="">use '-f' to override the following errors:<br class="">/dev/nvd0 is part of active pool 'zroot'<br class=""><br class=""><br class="">$> zpool replace -f zroot nvd0<br class="">invalid vdev specification<br class="">the following errors must be manually repaired:<br class="">/dev/nvd0 is part of active pool 'zroot'<br class=""><br class=""><br class="">-----------------------------------<br class=""><br class="">Also these commands don't work:<br class=""><br class="">$> zpool replace zroot nvd1 nvd0<br class="">invalid vdev specification<br class="">use '-f' to override the following errors:<br class="">/dev/nvd0 is part of active pool 'zroot'<br class=""><br class="">$> zpool replace -f zroot nvd1 nvd0<br class="">invalid vdev specification<br class="">the following errors must be manually repaired:<br class="">/dev/nvd0 is part of active pool 'zroot'<br class=""><br class=""><br class="">-----------------------------------<br class=""><br class=""><br class="">Instead these commands work:<br class=""><br class="">$> zpool offline zroot nvd0<br class=""><br class="">zpool status<br class=""> pool: zroot<br class=""> state: DEGRADED<br class="">status: One or more devices has been taken offline by the administrator.<br class=""> Sufficient replicas exist for the pool to continue functioning in a<br class=""> degraded state.<br class="">action: Online the device using 'zpool online' or replace the device with<br class=""> 'zpool replace'.<br class="">config:<br class=""><br class=""> NAME STATE READ WRITE CKSUM<br class=""> zroot DEGRADED 0 0 0<br class=""> mirror-0 DEGRADED 0 0 0<br class=""> nvd0 OFFLINE 0 0 0<br class=""> nvd1p3 ONLINE 0 0 0<br class=""><br class="">errors: No known data errors<br class=""><br class=""><br class="">$> zpool online zroot nvd0<br class=""><br class=""><br class="">$> zpool status<br class=""> pool: zroot<br class=""> state: ONLINE<br class=""> scan: resilvered 5.55M in 00:00:00 with 0 errors on Thu May 12 00:22:13 2022<br class="">config:<br class=""><br class=""> NAME STATE READ WRITE CKSUM<br class=""> zroot ONLINE 0 0 0<br class=""> mirror-0 ONLINE 0 0 0<br class=""> nvd0p3 ONLINE 0 0 0<br class=""> nvd1p3 ONLINE 0 0 0<br class=""><br class="">errors: No known data errors<br class=""><br class=""><br class="">------------------------------------</div><div class=""><br class=""><br class="">The second question is if instead of "zpool replace zroot nvd0" I had to use "zpool offline zroot nvd0" and "zpool online zroot nvd0" because nvd0 is the same disk (not replaced by a new disk).<br class=""><br class="">Also I notice that if I don't do "zpool offline zroot nvd0" and "zpool online zroot nvd0" , but do a server reboot instead then zpool automatically puts nvd0 online:<br class=""><br class="">$> zpool status<br class=""> pool: zroot<br class=""> state: ONLINE<br class="">status: One or more devices has experienced an unrecoverable error. An<br class=""> attempt was made to correct the error. Applications are unaffected.<br class="">action: Determine if the device needs to be replaced, and clear the errors<br class=""> using 'zpool clear' or replace the device with 'zpool replace'.<br class=""> see: <a href="https://openzfs.github.io/openzfs-docs/msg/ZFS-8000-9P" class="">https://openzfs.github.io/openzfs-docs/msg/ZFS-8000-9P</a><br class=""> scan: resilvered 3.50M in 00:00:00 with 0 errors on Thu May 12 01:04:09 2022<br class="">config:<br class=""><br class=""> NAME STATE READ WRITE CKSUM<br class=""> zroot ONLINE 0 0 0<br class=""> mirror-0 ONLINE 0 0 0<br class=""> nvd0p3 ONLINE 0 0 2<br class=""> nvd1p3 ONLINE 0 0 0<br class=""><br class="">errors: No known data errors<br class=""><br class=""><br class="">$> zpool clear zroot<br class=""><br class="">$> zpool status<br class=""> pool: zroot<br class=""> state: ONLINE<br class=""> scan: resilvered 3.50M in 00:00:00 with 0 errors on Thu May 12 01:04:09 2022<br class="">config:<br class=""><br class=""> NAME STATE READ WRITE CKSUM<br class=""> zroot ONLINE 0 0 0<br class=""> mirror-0 ONLINE 0 0 0<br class=""> nvd0p3 ONLINE 0 0 0<br class=""> nvd1p3 ONLINE 0 0 0<br class=""><br class="">errors: No known data errors<br class=""><br class=""><br class=""></div></body></html>
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?B911CBE8-F384-4E94-99B4-A2D558CB7754>