Date: Wed, 6 Apr 2022 13:22:01 +0000 From: "Michael Jung (USER)" <mikej@paymentallianceintl.com> To: freebsd-fs <freebsd-fs@freebsd.org> Subject: DRAID - Expansion and other issues Message-ID: <f42afbe959104522952f3eb57fa5f865@MAIL-HUB.pai.local>
next in thread | raw e-mail | index | archive | help
--_000_f42afbe959104522952f3eb57fa5f865MAILHUBpailocal_
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: quoted-printable
Hi!
I started playing with draid some months ago and I have a problem I cannot =
figure out.
I started out with a single draid2:2d:10c:0s and life was good. Ran some t=
ests, added
a special device and life was better. Then I expanded the pool by adding an=
other
draid2:2d:10c:0s. Now the problem: the pool says it has ~18TB worth of spa=
ce, the
file system only shows ~9TB. It did not auto-expand.
I'm currently on main-n253875-8e72f458c6d: but I initially spun this up jus=
t after
DRAID was brought Into the tree, at least when I became aware of it. I have=
tried
individually on lining devices in the pool with "-e", exporting/importing t=
he pool etc.
and basically every suggestion my "google foo" would lead me to.
I have no useful data on this pool and I can destroy and re-create it but I=
would
like to save some time and get opinions or fact on:
Can a draid pool be expanded after a special device has been added? It wou=
ld seem so
but the filesystem does not reflect that.?
What at is the correct way to wire down the following - I had problems afte=
r not doing
this since the 2.x days and would really want to wire down my draid devices=
now
that I have other pools.
I have pool construction all scripted but of course I want physical disk da=
<x> to always be
consistent. Then I easily test any construct someone comes up with.
root@draid:/home/mikej # camcontrol devlist -b
scbus0 on ata0 bus 0
scbus1 on ata1 bus 0
scbus2 on mpt0 bus 0
scbus3 on mps0 bus 0 <--
scbus4 on camsim0 bus 0
scbus-1 on xpt0 bus 0
root@draid:/home/mikej #
<dmesg>
mps0: <Avago Technologies (LSI) SAS2008> port 0x5000-0x50ff mem 0xfd4fc000-=
0xfd4fffff,0xfd480000-0xfd4bffff irq 19 at device 0.0 on pci4
mps0: Firmware: 20.00.04.00, Driver: 21.02.00.00-fbsd
mps0: IOCCapabilities: 1285c<ScsiTaskFull,DiagTrace,SnapBuf,EEDP,TransRetry=
,EventReplay,HostDisc>
<camcontrol>
<SEAGATE ST91000640SS AS09> at scbus3 target 63 lun 0 (da42,pass35)
<SEAGATE ST91000640SS AS09> at scbus3 target 67 lun 0 (da46,pass39)
<SEAGATE ST91000640SS AS08> at scbus3 target 68 lun 0 (da47,pass40)
EXAMPLE: Is this correct?
hint.scbus.3.at=3D"mps0"
hint.da.42.at=3D"scbus3"
hint.da.42.target=3D"63"
hint.da.42.unit=3D"0"
hint.da.46.at=3D"scbus3"
hint.da.46.target=3D"67"
hint.da.46.unit=3D"0"
hint.da.47.at=3D"scbus3"
hint.da.47.target=3D"68"
hint.da.47.unit=3D"0"
I will try again, but this did not seem to work for me. This is my home la=
b but it's
now painful to spin down this host as it's a iSCSI target for several machi=
nes for a
ESXi project I'm working on so I would like to put wired down SCSI devices =
to rest.
And yes, I have scheduled VM backups for my project to another data store t=
hat is not
this box ;-)
It's also odd, that I did not get all GPT labels in the zpool. All disks ha=
ve GPT labels.
I have added for the next reboot to loader.conf.
kern.geom.label.disk_ident.enable=3D"0"
kern.geom.label.gptid.enable=3D"1
This is an old Dell MD-1000 shelf I had laying around with a bunch of 1TB d=
rives
which I thought would be perfect for playing around with draid.
mikej@draid:~ $ zpool get all tank | grep expand
tank autoexpand on local
tank expandsize - -
mikej@draid:~ $ zpool list -v tank
NAME SIZE ALLOC FREE CKPOINT EXPANDSZ FRAG CAP D=
EDUP HEALTH ALTROOT
tank 18.5T 8.02T 10.4T - - 0% 43% 1=
.00x ONLINE -
draid2:2d:10c:0s-0 9.03T 4.06T 4.97T - - 0% 44.9% =
- ONLINE
gpt/da0p1 - - - - - - - =
- ONLINE
gpt/d10p1 - - - - - - - =
- ONLINE
da39p1 - - - - - - - =
- ONLINE
da40p1 - - - - - - - =
- ONLINE
da41p1 - - - - - - - =
- ONLINE
da43p1 - - - - - - - =
- ONLINE
da44p1 - - - - - - - =
- ONLINE
da42p1 - - - - - - - =
- ONLINE
da45p1 - - - - - - - =
- ONLINE
gpt/d19p1 - - - - - - - =
- ONLINE
draid2:2d:10c:0s-1 9.03T 3.96T 5.07T - - 0% 43.8% =
- ONLINE
da46p1 - - - - - - - =
- ONLINE
gpt/d11p1 - - - - - - - =
- ONLINE
gpt/d12p1 - - - - - - - =
- ONLINE
gpt/d13p1 - - - - - - - =
- ONLINE
gpt/d14p1 - - - - - - - =
- ONLINE
gpt/d15p1 - - - - - - - =
- ONLINE
gpt/d16p1 - - - - - - - =
- ONLINE
gpt/d17p1 - - - - - - - =
- ONLINE
gpt/d18p1 - - - - - - - =
- ONLINE
da18p1 - - - - - - - =
- ONLINE
special - - - - - - - =
- -
mirror-3 398G 2.86G 395G - - 0% 0.71% =
- ONLINE
gpt/special0 - - - - - - - =
- ONLINE
gpt/special1 - - - - - - - =
- ONLINE
logs - - - - - - - =
- -
mirror-2 15.5G 256K 15.5G - - 0% 0.00% =
- ONLINE
gpt/slog0 - - - - - - - =
- ONLINE
gpt/slog1 - - - - - - - =
- ONLINE
mikej@draid:~ $ zfs get mountpoint tank
NAME PROPERTY VALUE SOURCE
tank mountpoint /tank default
mikej@draid:~ $ df /tank
Filesystem 1K-blocks Used Avail Capacity Mounted on
tank 9558892004 4301310224 5257581780 45% /tank <-- ~9TB not =
18TB
mikej@draid:~ $
mikej@draid:~ $ zpool list
NAME SIZE ALLOC FREE CKPOINT EXPANDSZ FRAG CAP DEDUP =
HEALTH ALTROOT
ccache 9.50G 9.10G 406M - - 89% 95% 1.00x =
ONLINE -
raid-5400-1 6.28T 1.19T 5.09T - - 6% 19% 1.00x =
ONLINE -
tank 18.5T 8.02T 10.4T - - 0% 43% 1.00x =
ONLINE - <--
zfsroot 103G 32.6G 70.4G - - 25% 31% 1.00x =
ONLINE -
mikej@draid:~ $
Thanks in advance for any comments or sugesstions.
--mikej
CONFIDENTIALITY NOTE: This message is intended only for the use
of the individual or entity to whom it is addressed and may
contain information that is privileged, confidential, and
exempt from disclosure under applicable law. If the reader
of this message is not the intended recipient, you are hereby
notified that any dissemination, distribution or copying
of this communication is strictly prohibited. If you have
received this transmission in error, please notify us by
telephone at (502) 212-4000 or notify us at: PAI, Dept. 99,
2101 High Wickham Place, Suite 101, Louisville, KY 40245
Disclaimer
The information contained in this communication from the sender is confiden=
tial. It is intended solely for use by the recipient and others authorized =
to receive it. If you are not the recipient, you are hereby notified that a=
ny disclosure, copying, distribution or taking action in relation of the co=
ntents of this information is strictly prohibited and may be unlawful.
This email has been scanned for viruses and malware, and may have been auto=
matically archived by Mimecast, a leader in email security and cyber resili=
ence. Mimecast integrates email defenses with brand protection, security aw=
areness training, web security, compliance and other essential capabilities=
. Mimecast helps protect large and small organizations from malicious activ=
ity, human error and technology failure; and to lead the movement toward bu=
ilding a more resilient world. To find out more, visit our website.
--_000_f42afbe959104522952f3eb57fa5f865MAILHUBpailocal_
Content-Type: text/html; charset=UTF-8
Content-Transfer-Encoding: quoted-printable
<html><head>
<meta http-equiv=3D"Content-Type" content=3D"text/html; charset=3Dus-ascii"=
>
<meta name=3D"Generator" content=3D"Microsoft Word 15 (filtered medium)">
<style><!--
/* Font Definitions */
@font-face
=09{font-family:Wingdings;
=09panose-1:5 0 0 0 0 0 0 0 0 0;}
@font-face
=09{font-family:"Cambria Math";
=09panose-1:2 4 5 3 5 4 6 3 2 4;}
@font-face
=09{font-family:Calibri;
=09panose-1:2 15 5 2 2 2 4 3 2 4;}
/* Style Definitions */
p.MsoNormal, li.MsoNormal, div.MsoNormal
=09{margin:0in;
=09margin-bottom:.0001pt;
=09font-size:11.0pt;
=09font-family:"Calibri",sans-serif;}
a:link, span.MsoHyperlink
=09{mso-style-priority:99;
=09color:#0563C1;
=09text-decoration:underline;}
a:visited, span.MsoHyperlinkFollowed
=09{mso-style-priority:99;
=09color:#954F72;
=09text-decoration:underline;}
p.MsoListParagraph, li.MsoListParagraph, div.MsoListParagraph
=09{mso-style-priority:34;
=09margin-top:0in;
=09margin-right:0in;
=09margin-bottom:0in;
=09margin-left:.5in;
=09margin-bottom:.0001pt;
=09font-size:11.0pt;
=09font-family:"Calibri",sans-serif;}
span.EmailStyle17
=09{mso-style-type:personal-compose;
=09font-family:"Calibri",sans-serif;
=09color:windowtext;}
.MsoChpDefault
=09{mso-style-type:export-only;
=09font-family:"Calibri",sans-serif;}
@page WordSection1
=09{size:8.5in 11.0in;
=09margin:1.0in 1.0in 1.0in 1.0in;}
div.WordSection1
=09{page:WordSection1;}
/* List Definitions */
@list l0
=09{mso-list-id:207111470;
=09mso-list-type:hybrid;
=09mso-list-template-ids:682635594 -1484850496 67698691 67698693 67698689 6=
7698691 67698693 67698689 67698691 67698693;}
@list l0:level1
=09{mso-level-start-at:502;
=09mso-level-number-format:bullet;
=09mso-level-text:\F0B7;
=09mso-level-tab-stop:none;
=09mso-level-number-position:left;
=09margin-left:.75in;
=09text-indent:-.25in;
=09font-family:Symbol;
=09mso-fareast-font-family:Calibri;
=09mso-bidi-font-family:"Times New Roman";}
@list l0:level2
=09{mso-level-number-format:bullet;
=09mso-level-text:o;
=09mso-level-tab-stop:none;
=09mso-level-number-position:left;
=09margin-left:1.25in;
=09text-indent:-.25in;
=09font-family:"Courier New";}
@list l0:level3
=09{mso-level-number-format:bullet;
=09mso-level-text:\F0A7;
=09mso-level-tab-stop:none;
=09mso-level-number-position:left;
=09margin-left:1.75in;
=09text-indent:-.25in;
=09font-family:Wingdings;}
@list l0:level4
=09{mso-level-number-format:bullet;
=09mso-level-text:\F0B7;
=09mso-level-tab-stop:none;
=09mso-level-number-position:left;
=09margin-left:2.25in;
=09text-indent:-.25in;
=09font-family:Symbol;}
@list l0:level5
=09{mso-level-number-format:bullet;
=09mso-level-text:o;
=09mso-level-tab-stop:none;
=09mso-level-number-position:left;
=09margin-left:2.75in;
=09text-indent:-.25in;
=09font-family:"Courier New";}
@list l0:level6
=09{mso-level-number-format:bullet;
=09mso-level-text:\F0A7;
=09mso-level-tab-stop:none;
=09mso-level-number-position:left;
=09margin-left:3.25in;
=09text-indent:-.25in;
=09font-family:Wingdings;}
@list l0:level7
=09{mso-level-number-format:bullet;
=09mso-level-text:\F0B7;
=09mso-level-tab-stop:none;
=09mso-level-number-position:left;
=09margin-left:3.75in;
=09text-indent:-.25in;
=09font-family:Symbol;}
@list l0:level8
=09{mso-level-number-format:bullet;
=09mso-level-text:o;
=09mso-level-tab-stop:none;
=09mso-level-number-position:left;
=09margin-left:4.25in;
=09text-indent:-.25in;
=09font-family:"Courier New";}
@list l0:level9
=09{mso-level-number-format:bullet;
=09mso-level-text:\F0A7;
=09mso-level-tab-stop:none;
=09mso-level-number-position:left;
=09margin-left:4.75in;
=09text-indent:-.25in;
=09font-family:Wingdings;}
ol
=09{margin-bottom:0in;}
ul
=09{margin-bottom:0in;}
--></style><!--[if gte mso 9]><xml>
<o:shapedefaults v:ext=3D"edit" spidmax=3D"1026" />
</xml><![endif]--><!--[if gte mso 9]><xml>
<o:shapelayout v:ext=3D"edit">
<o:idmap v:ext=3D"edit" data=3D"1" />
</o:shapelayout></xml><![endif]-->
<style type=3D"text/css">.style1 {font-family: "Times New Roman";}</style><=
/head><body lang=3D"EN-US" link=3D"#0563C1" vlink=3D"#954F72">
<div class=3D"WordSection1">
<p class=3D"MsoNormal"><span style=3D"font-size:8.0pt;font-family:"Cou=
rier New"">Hi!<o:p></o:p></span></p>
<p class=3D"MsoNormal"><span style=3D"font-size:8.0pt;font-family:"Cou=
rier New""><o:p> </o:p></span></p>
<p class=3D"MsoNormal"><span style=3D"font-size:8.0pt;font-family:"Cou=
rier New"">I started playing with draid some months ago and I have a p=
roblem I cannot figure out.<o:p></o:p></span></p>
<p class=3D"MsoNormal"><span style=3D"font-size:8.0pt;font-family:"Cou=
rier New""><o:p> </o:p></span></p>
<p class=3D"MsoNormal"><span style=3D"font-size:8.0pt;font-family:"Cou=
rier New"">I started out with a single draid2:2d:10c:0s and life was g=
ood. Ran some tests, added
<o:p></o:p></span></p>
<p class=3D"MsoNormal"><span style=3D"font-size:8.0pt;font-family:"Cou=
rier New"">a special device and life was better. Then I expanded the p=
ool by adding another
<o:p></o:p></span></p>
<p class=3D"MsoNormal"><span style=3D"font-size:8.0pt;font-family:"Cou=
rier New"">draid2:2d:10c:0s. Now the problem: the pool says it h=
as ~18TB worth of space, the
<o:p></o:p></span></p>
<p class=3D"MsoNormal"><span style=3D"font-size:8.0pt;font-family:"Cou=
rier New"">file system only shows ~9TB. It did not auto-expand.<o:p></=
o:p></span></p>
<p class=3D"MsoNormal"><span style=3D"font-size:8.0pt;font-family:"Cou=
rier New""><o:p> </o:p></span></p>
<p class=3D"MsoNormal"><span style=3D"font-size:8.0pt;font-family:"Cou=
rier New"">I’m currently on main-n253875-8e72f458c6d: but I init=
ially spun this up just after
<o:p></o:p></span></p>
<p class=3D"MsoNormal"><span style=3D"font-size:8.0pt;font-family:"Cou=
rier New"">DRAID was brought Into the tree, at least when I became awa=
re of it. I have tried
<o:p></o:p></span></p>
<p class=3D"MsoNormal"><span style=3D"font-size:8.0pt;font-family:"Cou=
rier New"">individually on lining devices in the pool with “-e&#=
8221;, exporting/importing the pool etc.<o:p></o:p></span></p>
<p class=3D"MsoNormal"><span style=3D"font-size:8.0pt;font-family:"Cou=
rier New"">and basically every suggestion my “google foo” =
would lead me to.<o:p></o:p></span></p>
<p class=3D"MsoNormal"><span style=3D"font-size:8.0pt;font-family:"Cou=
rier New""><o:p> </o:p></span></p>
<p class=3D"MsoNormal"><span style=3D"font-size:8.0pt;font-family:"Cou=
rier New"">I have no useful data on this pool and I can destroy and re=
-create it but I would
<o:p></o:p></span></p>
<p class=3D"MsoNormal"><span style=3D"font-size:8.0pt;font-family:"Cou=
rier New"">like to save some time and get opinions or fact on:<o:p></o=
:p></span></p>
<p class=3D"MsoNormal"><span style=3D"font-size:8.0pt;font-family:"Cou=
rier New""><o:p> </o:p></span></p>
<p class=3D"MsoNormal"><span style=3D"font-size:8.0pt;font-family:"Cou=
rier New"">Can a draid pool be expanded after a special device has bee=
n added? It would seem so<o:p></o:p></span></p>
<p class=3D"MsoNormal"><span style=3D"font-size:8.0pt;font-family:"Cou=
rier New"">but the filesystem does not reflect that.?<o:p></o:p></span=
></p>
<p class=3D"MsoNormal"><span style=3D"font-size:8.0pt;font-family:"Cou=
rier New""><o:p> </o:p></span></p>
<p class=3D"MsoNormal"><span style=3D"font-size:8.0pt;font-family:"Cou=
rier New"">What at is the correct way to wire down the following ̵=
1; I had problems after not doing<o:p></o:p></span></p>
<p class=3D"MsoNormal"><span style=3D"font-size:8.0pt;font-family:"Cou=
rier New"">this since the 2.x days and would really want to wire down =
my draid devices now
<o:p></o:p></span></p>
<p class=3D"MsoNormal"><span style=3D"font-size:8.0pt;font-family:"Cou=
rier New"">that I have other pools.
<o:p></o:p></span></p>
<p class=3D"MsoNormal"><span style=3D"font-size:8.0pt;font-family:"Cou=
rier New""><o:p> </o:p></span></p>
<p class=3D"MsoNormal"><span style=3D"font-size:8.0pt;font-family:"Cou=
rier New"">I have pool construction all scripted but of course I want =
physical disk da<x> to always be
<o:p></o:p></span></p>
<p class=3D"MsoNormal"><span style=3D"font-size:8.0pt;font-family:"Cou=
rier New"">consistent. Then I easily test any construct someone comes =
up with.<o:p></o:p></span></p>
<p class=3D"MsoNormal"><span style=3D"font-size:8.0pt;font-family:"Cou=
rier New""><o:p> </o:p></span></p>
<p class=3D"MsoNormal"><span style=3D"font-size:8.0pt;font-family:"Cou=
rier New"">root@draid:/home/mikej # camcontrol devlist -b<o:p></o:p></=
span></p>
<p class=3D"MsoNormal"><span style=3D"font-size:8.0pt;font-family:"Cou=
rier New"">scbus0 on ata0 bus 0<o:p></o:p></span></p>
<p class=3D"MsoNormal"><span style=3D"font-size:8.0pt;font-family:"Cou=
rier New"">scbus1 on ata1 bus 0<o:p></o:p></span></p>
<p class=3D"MsoNormal"><span style=3D"font-size:8.0pt;font-family:"Cou=
rier New"">scbus2 on mpt0 bus 0<o:p></o:p></span></p>
<p class=3D"MsoNormal"><span style=3D"font-size:8.0pt;font-family:"Cou=
rier New"">scbus3 on mps0 bus 0
</span><span style=3D"font-size:8.0pt;font-family:Wingdings">ß</span>=
<span style=3D"font-size:8.0pt;font-family:"Courier New""><o:p></=
o:p></span></p>
<p class=3D"MsoNormal"><span style=3D"font-size:8.0pt;font-family:"Cou=
rier New"">scbus4 on camsim0 bus 0<o:p></o:p></span></p>
<p class=3D"MsoNormal"><span style=3D"font-size:8.0pt;font-family:"Cou=
rier New"">scbus-1 on xpt0 bus 0<o:p></o:p></span></p>
<p class=3D"MsoNormal"><span style=3D"font-size:8.0pt;font-family:"Cou=
rier New"">root@draid:/home/mikej #<o:p></o:p></span></p>
<p class=3D"MsoNormal" style=3D"margin-left:.5in"><span style=3D"font-size:=
8.0pt;font-family:"Courier New""><o:p> </o:p></span></p>
<p class=3D"MsoNormal" style=3D"margin-left:.5in"><span style=3D"font-size:=
8.0pt;font-family:"Courier New""><o:p> </o:p></span></p>
<p class=3D"MsoNormal"><span style=3D"font-size:8.0pt;font-family:"Cou=
rier New""><dmesg><o:p></o:p></span></p>
<p class=3D"MsoNormal"><span style=3D"font-size:8.0pt;font-family:"Cou=
rier New"">mps0: <Avago Technologies (LSI) SAS2008> port 0x5000-=
0x50ff mem 0xfd4fc000-0xfd4fffff,0xfd480000-0xfd4bffff irq 19 at device 0.0=
on pci4<o:p></o:p></span></p>
<p class=3D"MsoNormal"><span style=3D"font-size:8.0pt;font-family:"Cou=
rier New"">mps0: Firmware: 20.00.04.00, Driver: 21.02.00.00-fbsd<o:p><=
/o:p></span></p>
<p class=3D"MsoNormal"><span style=3D"font-size:8.0pt;font-family:"Cou=
rier New"">mps0: IOCCapabilities: 1285c<ScsiTaskFull,DiagTrace,Snap=
Buf,EEDP,TransRetry,EventReplay,HostDisc><o:p></o:p></span></p>
<p class=3D"MsoNormal"><span style=3D"font-size:8.0pt;font-family:"Cou=
rier New""><o:p> </o:p></span></p>
<p class=3D"MsoNormal"><span style=3D"font-size:8.0pt;font-family:"Cou=
rier New""><camcontrol><o:p></o:p></span></p>
<p class=3D"MsoNormal"><span style=3D"font-size:8.0pt;font-family:"Cou=
rier New""><SEAGATE ST91000640SS AS09> &n=
bsp; at scbus3 target 63 lun 0 (da42,pass35)<o:p></o:p></span><=
/p>
<p class=3D"MsoNormal"><span style=3D"font-size:8.0pt;font-family:"Cou=
rier New""><SEAGATE ST91000640SS AS09> &n=
bsp; at scbus3 target 67 lun 0 (da46,pass39)<o:p></o:p></span><=
/p>
<p class=3D"MsoNormal"><span style=3D"font-size:8.0pt;font-family:"Cou=
rier New""><SEAGATE ST91000640SS AS08> &n=
bsp; at scbus3 target 68 lun 0 (da47,pass40)<o:p></o:p></span><=
/p>
<p class=3D"MsoNormal"><span style=3D"font-size:8.0pt;font-family:"Cou=
rier New""><o:p> </o:p></span></p>
<p class=3D"MsoNormal"><span style=3D"font-size:8.0pt;font-family:"Cou=
rier New"">EXAMPLE: Is this correct?<o:p></o:p></span></p>
<p class=3D"MsoNormal"><span style=3D"font-size:8.0pt;font-family:"Cou=
rier New""><o:p> </o:p></span></p>
<p class=3D"MsoNormal"><span style=3D"font-size:8.0pt;font-family:"Cou=
rier New"">hint.scbus.3.at=3D”mps0”<o:p></o:p></span></p>
<p class=3D"MsoNormal"><span style=3D"font-size:8.0pt;font-family:"Cou=
rier New""><o:p> </o:p></span></p>
<p class=3D"MsoNormal"><span style=3D"font-size:8.0pt;font-family:"Cou=
rier New"">hint.da.42.at=3D”scbus3”<o:p></o:p></span></p>
<p class=3D"MsoNormal"><span style=3D"font-size:8.0pt;font-family:"Cou=
rier New"">hint.da.42.target=3D”63”<o:p></o:p></span></p>
<p class=3D"MsoNormal"><span style=3D"font-size:8.0pt;font-family:"Cou=
rier New"">hint.da.42.unit=3D”0”<o:p></o:p></span></p>
<p class=3D"MsoNormal"><span style=3D"font-size:8.0pt;font-family:"Cou=
rier New""><o:p> </o:p></span></p>
<p class=3D"MsoNormal"><span style=3D"font-size:8.0pt;font-family:"Cou=
rier New"">hint.da.46.at=3D”scbus3”<o:p></o:p></span></p>
<p class=3D"MsoNormal"><span style=3D"font-size:8.0pt;font-family:"Cou=
rier New"">hint.da.46.target=3D”67”<o:p></o:p></span></p>
<p class=3D"MsoNormal"><span style=3D"font-size:8.0pt;font-family:"Cou=
rier New"">hint.da.46.unit=3D”0”<o:p></o:p></span></p>
<p class=3D"MsoNormal"><span style=3D"font-size:8.0pt;font-family:"Cou=
rier New""><o:p> </o:p></span></p>
<p class=3D"MsoNormal"><span style=3D"font-size:8.0pt;font-family:"Cou=
rier New"">hint.da.47.at=3D”scbus3”<o:p></o:p></span></p>
<p class=3D"MsoNormal"><span style=3D"font-size:8.0pt;font-family:"Cou=
rier New"">hint.da.47.target=3D”68”<o:p></o:p></span></p>
<p class=3D"MsoNormal"><span style=3D"font-size:8.0pt;font-family:"Cou=
rier New"">hint.da.47.unit=3D”0"<o:p></o:p></span></p>
<p class=3D"MsoNormal"><span style=3D"font-size:8.0pt;font-family:"Cou=
rier New""><o:p> </o:p></span></p>
<p class=3D"MsoNormal"><span style=3D"font-size:8.0pt;font-family:"Cou=
rier New"">I will try again, but this did not seem to work for me.&nbs=
p; This is my home lab but it’s<o:p></o:p></span></p>
<p class=3D"MsoNormal"><span style=3D"font-size:8.0pt;font-family:"Cou=
rier New"">now painful to spin down this host as it’s a iSCSI ta=
rget for several machines for a<o:p></o:p></span></p>
<p class=3D"MsoNormal"><span style=3D"font-size:8.0pt;font-family:"Cou=
rier New"">ESXi project I’m working on so I would like to put wi=
red down SCSI devices to rest.<o:p></o:p></span></p>
<p class=3D"MsoNormal"><span style=3D"font-size:8.0pt;font-family:"Cou=
rier New""><o:p> </o:p></span></p>
<p class=3D"MsoNormal"><span style=3D"font-size:8.0pt;font-family:"Cou=
rier New"">And yes, I have scheduled VM backups for my project to anot=
her data store that is not<o:p></o:p></span></p>
<p class=3D"MsoNormal"><span style=3D"font-size:8.0pt;font-family:"Cou=
rier New"">this box ;-)<o:p></o:p></span></p>
<p class=3D"MsoNormal"><span style=3D"font-size:8.0pt;font-family:"Cou=
rier New""><o:p> </o:p></span></p>
<p class=3D"MsoNormal"><span style=3D"font-size:8.0pt;font-family:"Cou=
rier New"">It’s also odd, that I did not get all GPT labels in t=
he zpool. All disks have GPT labels.
<o:p></o:p></span></p>
<p class=3D"MsoNormal"><span style=3D"font-size:8.0pt;font-family:"Cou=
rier New""><o:p> </o:p></span></p>
<p class=3D"MsoNormal"><span style=3D"font-size:8.0pt;font-family:"Cou=
rier New"">I have added for the next reboot to loader.conf.
<o:p></o:p></span></p>
<p class=3D"MsoNormal"><span style=3D"font-size:8.0pt;font-family:"Cou=
rier New""><o:p> </o:p></span></p>
<p class=3D"MsoNormal"><span style=3D"font-size:8.0pt;font-family:"Cou=
rier New"">kern.geom.label.disk_ident.enable=3D"0"<o:p></o:p=
></span></p>
<p class=3D"MsoNormal"><span style=3D"font-size:8.0pt;font-family:"Cou=
rier New"">kern.geom.label.gptid.enable=3D"1<o:p></o:p></span></p=
>
<p class=3D"MsoNormal"><span style=3D"font-size:8.0pt;font-family:"Cou=
rier New""><o:p> </o:p></span></p>
<p class=3D"MsoNormal"><span style=3D"font-size:8.0pt;font-family:"Cou=
rier New"">This is an old Dell MD-1000 shelf I had laying around with =
a bunch of 1TB drives
<o:p></o:p></span></p>
<p class=3D"MsoNormal"><span style=3D"font-size:8.0pt;font-family:"Cou=
rier New"">which I thought would be perfect for playing around with dr=
aid.<o:p></o:p></span></p>
<p class=3D"MsoNormal"><span style=3D"font-size:8.0pt;font-family:"Cou=
rier New""><o:p> </o:p></span></p>
<p class=3D"MsoNormal"><span style=3D"font-size:8.0pt;font-family:"Cou=
rier New"">mikej@draid:~ $ zpool get all tank | grep expand<o:p></o:p>=
</span></p>
<p class=3D"MsoNormal"><span style=3D"font-size:8.0pt;font-family:"Cou=
rier New"">tank autoexpand &n=
bsp;  =
; on  =
; &n=
bsp; local<o:p></o:p></span></p>
<p class=3D"MsoNormal"><span style=3D"font-size:8.0pt;font-family:"Cou=
rier New"">tank expandsize &n=
bsp;  =
; - =
&nb=
sp; -<o:p></o:p></span></p>
<p class=3D"MsoNormal"><span style=3D"font-size:8.0pt;font-family:"Cou=
rier New"">mikej@draid:~ $ zpool list -v tank<o:p></o:p></span></p>
<p class=3D"MsoNormal"><span style=3D"font-size:8.0pt;font-family:"Cou=
rier New"">NAME &=
nbsp; SIZE ALLOC FREE=
CKPOINT EXPANDSZ FRAG CAP =
DEDUP HEALTH ALTROOT<o:p></o:p></span></p>
<p class=3D"MsoNormal"><span style=3D"font-size:8.0pt;font-family:"Cou=
rier New"">tank &=
nbsp; 18.5T 8.02T 10.4T &nbs=
p; -  =
; - 0% 43% 1.00x =
; ONLINE -<o:p></o:p></span></p>
<p class=3D"MsoNormal"><span style=3D"font-size:8.0pt;font-family:"Cou=
rier New""> draid2:2d:10c:0s-0 9.03T 4.06T 4.9=
7T -  =
; - 0% 44.9% &nb=
sp; - ONLINE<o:p></o:p></span></p>
<p class=3D"MsoNormal"><span style=3D"font-size:8.0pt;font-family:"Cou=
rier New""> gpt/da0p1 &=
nbsp; - - =
- - &nbs=
p; - -&nbs=
p; - - &nb=
sp; ONLINE<o:p></o:p></span></p>
<p class=3D"MsoNormal"><span style=3D"font-size:8.0pt;font-family:"Cou=
rier New""> gpt/d10p1 &=
nbsp; - - =
- -  =
; - -&nbs=
p; - - &nb=
sp; ONLINE<o:p></o:p></span></p>
<p class=3D"MsoNormal"><span style=3D"font-size:8.0pt;font-family:"Cou=
rier New""> da39p1 &nbs=
p; -  =
; - - &nbs=
p; - - &nb=
sp; - - -=
ONLINE<o:p></o:p></span></p>
<p class=3D"MsoNormal"><span style=3D"font-size:8.0pt;font-family:"Cou=
rier New""> da40p1 &nbs=
p; -  =
; - - &nbs=
p; - - &nb=
sp; - - -=
ONLINE<o:p></o:p></span></p>
<p class=3D"MsoNormal"><span style=3D"font-size:8.0pt;font-family:"Cou=
rier New""> da41p1 &nbs=
p; -  =
; - - &nbs=
p; - - &nb=
sp; - - -=
ONLINE<o:p></o:p></span></p>
<p class=3D"MsoNormal"><span style=3D"font-size:8.0pt;font-family:"Cou=
rier New""> da43p1 &nbs=
p; -  =
; - - &nbs=
p; - - &nb=
sp; - - -=
ONLINE<o:p></o:p></span></p>
<p class=3D"MsoNormal"><span style=3D"font-size:8.0pt;font-family:"Cou=
rier New""> da44p1 &nbs=
p; -  =
; - - &nbs=
p; - - &nb=
sp; - - -=
ONLINE<o:p></o:p></span></p>
<p class=3D"MsoNormal"><span style=3D"font-size:8.0pt;font-family:"Cou=
rier New""> da42p1 &nbs=
p; -  =
; - - &nbs=
p; - - &nb=
sp; - - -=
ONLINE<o:p></o:p></span></p>
<p class=3D"MsoNormal"><span style=3D"font-size:8.0pt;font-family:"Cou=
rier New""> da45p1 &nbs=
p; -  =
; - - &nbs=
p; - - &n=
bsp; - - -=
ONLINE<o:p></o:p></span></p>
<p class=3D"MsoNormal"><span style=3D"font-size:8.0pt;font-family:"Cou=
rier New""> gpt/d19p1 &=
nbsp; - - =
- -  =
; - -&nbs=
p; - - &nb=
sp; ONLINE<o:p></o:p></span></p>
<p class=3D"MsoNormal"><span style=3D"font-size:8.0pt;font-family:"Cou=
rier New""> draid2:2d:10c:0s-1 9.03T 3.96T 5.0=
7T -  =
; - 0% 43.8% &nb=
sp; - ONLINE<o:p></o:p></span></p>
<p class=3D"MsoNormal"><span style=3D"font-size:8.0pt;font-family:"Cou=
rier New""> da46p1 &nbs=
p; -  =
; - - &nb=
sp;- - &nb=
sp; - - -=
ONLINE<o:p></o:p></span></p>
<p class=3D"MsoNormal"><span style=3D"font-size:8.0pt;font-family:"Cou=
rier New""> gpt/d11p1 &=
nbsp; - - =
- -  =
; - -&nbs=
p; - - &nb=
sp; ONLINE<o:p></o:p></span></p>
<p class=3D"MsoNormal"><span style=3D"font-size:8.0pt;font-family:"Cou=
rier New""> gpt/d12p1 &=
nbsp; - - =
- -  =
; - -&nbs=
p; - - &nb=
sp; ONLINE<o:p></o:p></span></p>
<p class=3D"MsoNormal"><span style=3D"font-size:8.0pt;font-family:"Cou=
rier New""> gpt/d13p1 &=
nbsp; - - =
- -  =
; - -&nbs=
p; - - &nb=
sp; ONLINE<o:p></o:p></span></p>
<p class=3D"MsoNormal"><span style=3D"font-size:8.0pt;font-family:"Cou=
rier New""> gpt/d14p1 &=
nbsp; - - =
- -  =
; - -&nbs=
p; - - &nb=
sp; ONLINE<o:p></o:p></span></p>
<p class=3D"MsoNormal"><span style=3D"font-size:8.0pt;font-family:"Cou=
rier New""> gpt/d15p1 &=
nbsp; - - =
- -  =
; - -&nbs=
p; - - &nb=
sp; ONLINE<o:p></o:p></span></p>
<p class=3D"MsoNormal"><span style=3D"font-size:8.0pt;font-family:"Cou=
rier New""> gpt/d16p1 =
- - =
- -  =
; - -&nbs=
p; - - &nb=
sp; ONLINE<o:p></o:p></span></p>
<p class=3D"MsoNormal"><span style=3D"font-size:8.0pt;font-family:"Cou=
rier New""> gpt/d17p1 &=
nbsp; - - =
- -  =
; - -&nbs=
p; - - &nb=
sp; ONLINE<o:p></o:p></span></p>
<p class=3D"MsoNormal"><span style=3D"font-size:8.0pt;font-family:"Cou=
rier New""> gpt/d18p1 &=
nbsp; - - =
- -  =
; - -&nbs=
p; - - &nb=
sp; ONLINE<o:p></o:p></span></p>
<p class=3D"MsoNormal"><span style=3D"font-size:8.0pt;font-family:"Cou=
rier New""> da18p1 &nbs=
p; -  =
; - - &nbs=
p; - - &nb=
sp; - - -=
ONLINE<o:p></o:p></span></p>
<p class=3D"MsoNormal"><span style=3D"font-size:8.0pt;font-family:"Cou=
rier New"">special &nbs=
p; -  =
; - - &nbs=
p; - - &nb=
sp; - - -=
-<o:p></o:p></span></p>
<p class=3D"MsoNormal"><span style=3D"font-size:8.0pt;font-family:"Cou=
rier New""> mirror-3 &n=
bsp; 398G 2.86G 395G &=
nbsp; - - =
0% 0.71% - &nb=
sp; ONLINE<o:p></o:p></span></p>
<p class=3D"MsoNormal"><span style=3D"font-size:8.0pt;font-family:"Cou=
rier New""> gpt/special0 &nbs=
p; - - &nb=
sp; - - &n=
bsp; - - &=
nbsp; - - ONLINE<o:p>=
</o:p></span></p>
<p class=3D"MsoNormal"><span style=3D"font-size:8.0pt;font-family:"Cou=
rier New""> gpt/special1 &nbs=
p; - - &nb=
sp; - - &n=
bsp; - - &=
nbsp; - - ONLINE<o:p>=
</o:p></span></p>
<p class=3D"MsoNormal"><span style=3D"font-size:8.0pt;font-family:"Cou=
rier New"">logs &=
nbsp; - &n=
bsp; - - &=
nbsp; - - =
- -  =
; - -<o:p></o:p=
></span></p>
<p class=3D"MsoNormal"><span style=3D"font-size:8.0pt;font-family:"Cou=
rier New""> mirror-2 &n=
bsp; 15.5G 256K 15.5G =
- -  =
; 0% 0.00% - &nb=
sp; ONLINE<o:p></o:p></span></p>
<p class=3D"MsoNormal"><span style=3D"font-size:8.0pt;font-family:"Cou=
rier New""> gpt/slog0 &=
nbsp; - - =
- -  =
; - -&nbs=
p; - - &nb=
sp; ONLINE<o:p></o:p></span></p>
<p class=3D"MsoNormal"><span style=3D"font-size:8.0pt;font-family:"Cou=
rier New""> gpt/slog1 &=
nbsp; - - =
- -  =
; - -&nbs=
p; - - &nb=
sp; ONLINE<o:p></o:p></span></p>
<p class=3D"MsoNormal"><span style=3D"font-size:8.0pt;font-family:"Cou=
rier New""><o:p> </o:p></span></p>
<p class=3D"MsoNormal"><span style=3D"font-size:8.0pt;font-family:"Cou=
rier New"">mikej@draid:~ $ zfs get mountpoint tank<o:p></o:p></span></=
p>
<p class=3D"MsoNormal"><span style=3D"font-size:8.0pt;font-family:"Cou=
rier New"">NAME PROPERTY VALUE &nbs=
p; SOURCE<o:p></o:p></span></p>
<p class=3D"MsoNormal"><span style=3D"font-size:8.0pt;font-family:"Cou=
rier New"">tank mountpoint /tank &n=
bsp; default<o:p></o:p></span></p>
<p class=3D"MsoNormal"><span style=3D"font-size:8.0pt;font-family:"Cou=
rier New""><o:p> </o:p></span></p>
<p class=3D"MsoNormal"><span style=3D"font-size:8.0pt;font-family:"Cou=
rier New""><o:p> </o:p></span></p>
<p class=3D"MsoNormal"><span style=3D"font-size:8.0pt;font-family:"Cou=
rier New"">mikej@draid:~ $ df /tank<o:p></o:p></span></p>
<p class=3D"MsoNormal"><span style=3D"font-size:8.0pt;font-family:"Cou=
rier New"">Filesystem 1K-blocks &nb=
sp; Used Avail Capacity Mounted on<o:p>=
</o:p></span></p>
<p class=3D"MsoNormal"><span style=3D"font-size:8.0pt;font-family:"Cou=
rier New"">tank 9558892004 4301310=
224 5257581780 45% /tank
</span><span style=3D"font-size:8.0pt;font-family:Wingdings">ß</span>=
<span style=3D"font-size:8.0pt;font-family:"Courier New""> ~9TB n=
ot 18TB<o:p></o:p></span></p>
<p class=3D"MsoNormal"><span style=3D"font-size:8.0pt;font-family:"Cou=
rier New"">mikej@draid:~ $<o:p></o:p></span></p>
<p class=3D"MsoNormal"><span style=3D"font-size:8.0pt;font-family:"Cou=
rier New""><o:p> </o:p></span></p>
<p class=3D"MsoNormal"><span style=3D"font-size:8.0pt;font-family:"Cou=
rier New"">mikej@draid:~ $ zpool list<o:p></o:p></span></p>
<p class=3D"MsoNormal"><span style=3D"font-size:8.0pt;font-family:"Cou=
rier New"">NAME =
SIZE ALLOC FREE CKPOINT EXPANDSZ =
FRAG CAP DEDUP HEALTH ALTRO=
OT<o:p></o:p></span></p>
<p class=3D"MsoNormal"><span style=3D"font-size:8.0pt;font-family:"Cou=
rier New"">ccache 9.50G 9.10=
G 406M - &=
nbsp; - 89% &nbs=
p; 95% 1.00x ONLINE -<o:p></o:p></span></p>
<p class=3D"MsoNormal"><span style=3D"font-size:8.0pt;font-family:"Cou=
rier New"">raid-5400-1 6.28T 1.19T 5.09T =
- &=
nbsp; - 6% 19% 1.00x &=
nbsp; ONLINE -<o:p></o:p></span></p>
<p class=3D"MsoNormal"><span style=3D"font-size:8.0pt;font-family:"Cou=
rier New"">tank 18.5T&=
nbsp; 8.02T 10.4T - &n=
bsp; - 0% =
43% 1.00x ONLINE - &n=
bsp;
</span><span style=3D"font-size:8.0pt;font-family:Wingdings">ß</span>=
<span style=3D"font-size:8.0pt;font-family:"Courier New"">
<o:p></o:p></span></p>
<p class=3D"MsoNormal"><span style=3D"font-size:8.0pt;font-family:"Cou=
rier New"">zfsroot 103G 32.6=
G 70.4G - =
- 25% 31=
% 1.00x ONLINE -<o:p></o:p></span></p>
<p class=3D"MsoNormal"><span style=3D"font-size:8.0pt;font-family:"Cou=
rier New"">mikej@draid:~ $<o:p></o:p></span></p>
<p class=3D"MsoNormal"><span style=3D"font-size:8.0pt;font-family:"Cou=
rier New""><o:p> </o:p></span></p>
<p class=3D"MsoNormal"><span style=3D"font-size:8.0pt;font-family:"Cou=
rier New"">Thanks in advance for any comments or sugesstions.<o:p></o:=
p></span></p>
<p class=3D"MsoNormal"><span style=3D"font-size:8.0pt;font-family:"Cou=
rier New""><o:p> </o:p></span></p>
<p class=3D"MsoNormal"><span style=3D"font-size:8.0pt;font-family:"Cou=
rier New"">--mikej<o:p></o:p></span></p>
<p class=3D"MsoNormal"><o:p> </o:p></p>
<p class=3D"MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p style=3D"FONT-SIZE: 10pt; FONT-FAMILY: Arial"> </p>
<div style=3D"FONT-SIZE: 10px; BORDER-TOP: #666666 1px solid; FONT-FAMILY: =
Verdana; WIDTH: 410px; BORDER-BOTTOM: #666666 1px solid; PADDING-BOTTOM: 5p=
x; PADDING-TOP: 5px; PADDING-LEFT: 5px; PADDING-RIGHT: 5px">
CONFIDENTIALITY NOTE: This message is intended only for the use<br>
of the individual or entity to whom it is addressed and may <br>
contain information that is privileged, confidential, and <br>
exempt from disclosure under applicable law. If the reader <br>
of this message is not the intended recipient, you are hereby <br>
notified that any dissemination, distribution or copying <br>
of this communication is strictly prohibited. If you have <br>
received this transmission in error, please notify us by <br>
telephone at (502) 212-4000 or notify us at: PAI, Dept. 99, <br>
2101 High Wickham Place, Suite 101, Louisville, KY 40245<br>
<br>
<br>
<br>
<p style=3D"FONT-SIZE: 10px; FONT-FAMILY: Verdana"></p>
<p style=3D"FONT-SIZE: 10px; FONT-FAMILY: Verdana"></p>
<p style=3D"FONT-SIZE: 10px; FONT-FAMILY: Verdana"></p>
<p style=3D"FONT-SIZE: 10px; FONT-FAMILY: Verdana"></p>
<p style=3D"FONT-SIZE: 10px; FONT-FAMILY: Verdana"></p>
<p style=3D"FONT-SIZE: 10px; FONT-FAMILY: Verdana"></p>
<p style=3D"FONT-SIZE: 10px; FONT-FAMILY: Verdana"></p>
<p style=3D"FONT-SIZE: 10px; FONT-FAMILY: Verdana"></p>
<p style=3D"FONT-SIZE: 10px; FONT-FAMILY: Verdana"></p>
<p style=3D"FONT-SIZE: 10px; FONT-FAMILY: Verdana"></p>
<p style=3D"FONT-SIZE: 10px; FONT-FAMILY: Verdana"></p>
<p style=3D"FONT-SIZE: 10px; FONT-FAMILY: Verdana"></p>
<p style=3D"FONT-SIZE: 10px; FONT-FAMILY: Verdana"></p>
<p style=3D"FONT-SIZE: 10px; FONT-FAMILY: Verdana"></p>
<p style=3D"FONT-SIZE: 10px; FONT-FAMILY: Verdana"></p>
<p style=3D"FONT-SIZE: 10px; FONT-FAMILY: Verdana"></p>
<p style=3D"FONT-SIZE: 10px; FONT-FAMILY: Verdana"></p>
<p style=3D"FONT-SIZE: 10px; FONT-FAMILY: Verdana"></p>
<p style=3D"FONT-SIZE: 10px; FONT-FAMILY: Verdana"></p>
<p style=3D"FONT-SIZE: 10px; FONT-FAMILY: Verdana"></p>
<p style=3D"FONT-SIZE: 10px; FONT-FAMILY: Verdana"></p>
<p style=3D"FONT-SIZE: 10px; FONT-FAMILY: Verdana"></p>
<p style=3D"FONT-SIZE: 10px; FONT-FAMILY: Verdana"></p>
<p style=3D"FONT-SIZE: 10px; FONT-FAMILY: Verdana"></p>
<p style=3D"FONT-SIZE: 10px; FONT-FAMILY: Verdana"></p>
<p style=3D"FONT-SIZE: 10px; FONT-FAMILY: Verdana"></p>
<p style=3D"FONT-SIZE: 10px; FONT-FAMILY: Verdana"></p>
<p style=3D"FONT-SIZE: 10px; FONT-FAMILY: Verdana"></p>
<p style=3D"FONT-SIZE: 10px; FONT-FAMILY: Verdana"></p>
<p style=3D"FONT-SIZE: 10px; FONT-FAMILY: Verdana"></p>
<p style=3D"FONT-SIZE: 10px; FONT-FAMILY: Verdana"></p>
<p style=3D"FONT-SIZE: 10px; FONT-FAMILY: Verdana"></p>
<p style=3D"FONT-SIZE: 10px; FONT-FAMILY: Verdana"></p>
<p style=3D"FONT-SIZE: 10px; FONT-FAMILY: Verdana"></p>
<p style=3D"FONT-SIZE: 10px; FONT-FAMILY: Verdana"></p>
<p style=3D"FONT-SIZE: 10px; FONT-FAMILY: Verdana"></p>
<p style=3D"FONT-SIZE: 10px; FONT-FAMILY: Verdana"></p>
<p style=3D"FONT-SIZE: 10px; FONT-FAMILY: Verdana"></p>
<p style=3D"FONT-SIZE: 10px; FONT-FAMILY: Verdana"></p>
<p style=3D"FONT-SIZE: 10px; FONT-FAMILY: Verdana"></p>
<p style=3D"FONT-SIZE: 10px; FONT-FAMILY: Verdana"></p>
<p style=3D"FONT-SIZE: 10px; FONT-FAMILY: Verdana"></p>
<p style=3D"FONT-SIZE: 10px; FONT-FAMILY: Verdana"></p>
<p style=3D"FONT-SIZE: 10px; FONT-FAMILY: Verdana"></p>
<p style=3D"FONT-SIZE: 10px; FONT-FAMILY: Verdana"></p>
<p style=3D"FONT-SIZE: 10px; FONT-FAMILY: Verdana"></p>
<p style=3D"FONT-SIZE: 10px; FONT-FAMILY: Verdana"></p>
<p style=3D"FONT-SIZE: 10px; FONT-FAMILY: Verdana"></p>
<p style=3D"FONT-SIZE: 10px; FONT-FAMILY: Verdana"></p>
<p style=3D"FONT-SIZE: 10px; FONT-FAMILY: Verdana"></p>
<p style=3D"FONT-SIZE: 10px; FONT-FAMILY: Verdana"></p>
<p style=3D"FONT-SIZE: 10px; FONT-FAMILY: Verdana"></p>
<p style=3D"FONT-SIZE: 10px; FONT-FAMILY: Verdana"></p>
<p style=3D"FONT-SIZE: 10px; FONT-FAMILY: Verdana"></p>
<p style=3D"FONT-SIZE: 10px; FONT-FAMILY: Verdana"></p>
<p style=3D"FONT-SIZE: 10px; FONT-FAMILY: Verdana"></p>
<p style=3D"FONT-SIZE: 10px; FONT-FAMILY: Verdana"></p>
<p style=3D"FONT-SIZE: 10px; FONT-FAMILY: Verdana"></p>
<p style=3D"FONT-SIZE: 10px; FONT-FAMILY: Verdana"></p>
<p style=3D"FONT-SIZE: 10px; FONT-FAMILY: Verdana"></p>
<p style=3D"FONT-SIZE: 10px; FONT-FAMILY: Verdana"></p>
<p style=3D"FONT-SIZE: 10px; FONT-FAMILY: Verdana"></p>
<p style=3D"FONT-SIZE: 10px; FONT-FAMILY: Verdana"></p>
<p style=3D"FONT-SIZE: 10px; FONT-FAMILY: Verdana"></p>
<p style=3D"FONT-SIZE: 10px; FONT-FAMILY: Verdana"></p>
<p style=3D"FONT-SIZE: 10px; FONT-FAMILY: Verdana"></p>
<p style=3D"FONT-SIZE: 10px; FONT-FAMILY: Verdana"></p>
<p style=3D"FONT-SIZE: 10px; FONT-FAMILY: Verdana"></p>
<p style=3D"FONT-SIZE: 10px; FONT-FAMILY: Verdana"></p>
<p style=3D"FONT-SIZE: 10px; FONT-FAMILY: Verdana"></p>
<p style=3D"FONT-SIZE: 10px; FONT-FAMILY: Verdana"></p>
<p style=3D"FONT-SIZE: 10px; FONT-FAMILY: Verdana"></p>
<p style=3D"FONT-SIZE: 10px; FONT-FAMILY: Verdana"></p>
<p style=3D"FONT-SIZE: 10px; FONT-FAMILY: Verdana"></p>
<p style=3D"FONT-SIZE: 10px; FONT-FAMILY: Verdana"></p>
<p style=3D"FONT-SIZE: 10px; FONT-FAMILY: Verdana"></p>
<p style=3D"FONT-SIZE: 10px; FONT-FAMILY: Verdana"></p>
<p style=3D"FONT-SIZE: 10px; FONT-FAMILY: Verdana"></p>
<p style=3D"FONT-SIZE: 10px; FONT-FAMILY: Verdana"></p>
<p style=3D"FONT-SIZE: 10px; FONT-FAMILY: Verdana"></p>
<p style=3D"FONT-SIZE: 10px; FONT-FAMILY: Verdana"></p>
<p style=3D"FONT-SIZE: 10px; FONT-FAMILY: Verdana"></p>
<p style=3D"FONT-SIZE: 10px; FONT-FAMILY: Verdana"></p>
<p style=3D"FONT-SIZE: 10px; FONT-FAMILY: Verdana"></p>
<p style=3D"FONT-SIZE: 10px; FONT-FAMILY: Verdana"></p>
<p style=3D"FONT-SIZE: 10px; FONT-FAMILY: Verdana"></p>
<p style=3D"FONT-SIZE: 10px; FONT-FAMILY: Verdana"></p>
<p style=3D"FONT-SIZE: 10px; FONT-FAMILY: Verdana"></p>
<p style=3D"FONT-SIZE: 10px; FONT-FAMILY: Verdana"></p>
<p style=3D"FONT-SIZE: 10px; FONT-FAMILY: Verdana"></p>
<p style=3D"FONT-SIZE: 10px; FONT-FAMILY: Verdana"></p>
<p style=3D"FONT-SIZE: 10px; FONT-FAMILY: Verdana"></p>
<p style=3D"FONT-SIZE: 10px; FONT-FAMILY: Verdana"></p>
<p style=3D"FONT-SIZE: 10px; FONT-FAMILY: Verdana"></p>
<p style=3D"FONT-SIZE: 10px; FONT-FAMILY: Verdana"></p>
<p style=3D"FONT-SIZE: 10px; FONT-FAMILY: Verdana"></p>
<p style=3D"FONT-SIZE: 10px; FONT-FAMILY: Verdana"></p>
<p style=3D"FONT-SIZE: 10px; FONT-FAMILY: Verdana"></p>
<p style=3D"FONT-SIZE: 10px; FONT-FAMILY: Verdana"></p>
<p style=3D"FONT-SIZE: 10px; FONT-FAMILY: Verdana"></p>
<p style=3D"FONT-SIZE: 10px; FONT-FAMILY: Verdana"></p>
<p style=3D"FONT-SIZE: 10px; FONT-FAMILY: Verdana"></p>
<p style=3D"FONT-SIZE: 10px; FONT-FAMILY: Verdana"></p>
<p style=3D"FONT-SIZE: 10px; FONT-FAMILY: Verdana"></p>
<p style=3D"FONT-SIZE: 10px; FONT-FAMILY: Verdana"></p>
<p style=3D"FONT-SIZE: 10px; FONT-FAMILY: Verdana"></p>
<p style=3D"FONT-SIZE: 10px; FONT-FAMILY: Verdana"></p>
<p style=3D"FONT-SIZE: 10px; FONT-FAMILY: Verdana"></p>
<p style=3D"FONT-SIZE: 10px; FONT-FAMILY: Verdana"></p>
<p style=3D"FONT-SIZE: 10px; FONT-FAMILY: Verdana"></p>
<p style=3D"FONT-SIZE: 10px; FONT-FAMILY: Verdana"></p>
<p style=3D"FONT-SIZE: 10px; FONT-FAMILY: Verdana"></p>
<p style=3D"FONT-SIZE: 10px; FONT-FAMILY: Verdana"></p>
<p style=3D"FONT-SIZE: 10px; FONT-FAMILY: Verdana"></p>
<p style=3D"FONT-SIZE: 10px; FONT-FAMILY: Verdana"></p>
<p style=3D"FONT-SIZE: 10px; FONT-FAMILY: Verdana"></p>
<p style=3D"FONT-SIZE: 10px; FONT-FAMILY: Verdana"></p>
<p style=3D"FONT-SIZE: 10px; FONT-FAMILY: Verdana"></p>
<p style=3D"FONT-SIZE: 10px; FONT-FAMILY: Verdana"></p>
<p style=3D"FONT-SIZE: 10px; FONT-FAMILY: Verdana"></p>
<p style=3D"FONT-SIZE: 10px; FONT-FAMILY: Verdana"></p>
<p style=3D"FONT-SIZE: 10px; FONT-FAMILY: Verdana"></p>
<p style=3D"FONT-SIZE: 10px; FONT-FAMILY: Verdana"></p>
<p style=3D"FONT-SIZE: 10px; FONT-FAMILY: Verdana"></p>
<p style=3D"FONT-SIZE: 10px; FONT-FAMILY: Verdana"></p>
<p style=3D"FONT-SIZE: 10px; FONT-FAMILY: Verdana"></p>
<p style=3D"FONT-SIZE: 10px; FONT-FAMILY: Verdana"></p>
<p style=3D"FONT-SIZE: 10px; FONT-FAMILY: Verdana"></p>
<p style=3D"FONT-SIZE: 10px; FONT-FAMILY: Verdana"></p>
<p style=3D"FONT-SIZE: 10px; FONT-FAMILY: Verdana"></p>
<p style=3D"FONT-SIZE: 10px; FONT-FAMILY: Verdana"></p>
<p style=3D"FONT-SIZE: 10px; FONT-FAMILY: Verdana"></p>
<p style=3D"FONT-SIZE: 10px; FONT-FAMILY: Verdana"></p>
<p style=3D"FONT-SIZE: 10px; FONT-FAMILY: Verdana"></p>
<p style=3D"FONT-SIZE: 10px; FONT-FAMILY: Verdana"></p>
<p style=3D"FONT-SIZE: 10px; FONT-FAMILY: Verdana"></p>
<p style=3D"FONT-SIZE: 10px; FONT-FAMILY: Verdana"></p>
<p style=3D"FONT-SIZE: 10px; FONT-FAMILY: Verdana"></p>
<p style=3D"FONT-SIZE: 10px; FONT-FAMILY: Verdana"></p>
<p style=3D"FONT-SIZE: 10px; FONT-FAMILY: Verdana"></p>
</div>
</div>
<br><br><p style=3D"font-family: Verdana; font-size:10pt; color:#666666;"><=
b>Disclaimer</b></p><p style=3D"font-family: Verdana; font-size:8pt; color:=
#666666;">The information contained in this communication from the sender i=
s confidential. It is intended solely for use by the recipient and others a=
uthorized to receive it. If you are not the recipient, you are hereby notif=
ied that any disclosure, copying, distribution or taking action in relation=
of the contents of this information is strictly prohibited and may be unla=
wful.<br><br>This email has been scanned for viruses and malware, and may h=
ave been automatically archived by Mimecast, a leader in email security and=
cyber resilience. Mimecast integrates email defenses with brand protection=
, security awareness training, web security, compliance and other essential=
capabilities. Mimecast helps protect large and small organizations from ma=
licious activity, human error and technology failure; and to lead the movem=
ent toward building a more resilient world. To find out more, visit our web=
site.</p></body></html>
--_000_f42afbe959104522952f3eb57fa5f865MAILHUBpailocal_--
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?f42afbe959104522952f3eb57fa5f865>