Date: Wed, 6 Apr 2022 14:05:22 +0000 From: "Michael Jung (USER)" <mikej@paymentallianceintl.com> To: Rich <rincebrain@gmail.com>, freebsd-fs <freebsd-fs@freebsd.org> Subject: RE: DRAID - Expansion and other issues Message-ID: <fe01e1eebc4f46c58e6732b89781221b@MAIL-HUB.pai.local> In-Reply-To: <CAOeNLuqvuAdeTr54XXnN_uL%2BoTShihTf6xOiqvS%2B4RU4HBG7Dw@mail.gmail.com> References: <f42afbe959104522952f3eb57fa5f865@MAIL-HUB.pai.local> <CAOeNLuqvuAdeTr54XXnN_uL%2BoTShihTf6xOiqvS%2B4RU4HBG7Dw@mail.gmail.com>
next in thread | previous in thread | raw e-mail | index | archive | help
[-- Attachment #1 --] Thanks Rich. I hear what you are saying, but I added them like you would add a second vdev to a raidz1/z2 pool. None of the online talks nor documentation mention draid expansion, and I was under the impression that you could not expand a draid2:2d:10c:0s to draid2:2d:20c:0s for example. I absolutely could be wrong in my assumption but do not see how to do the expansion other than how I did it. --mikej -------------------------------- I'm not an expert on draid, but you appear to have defined two draid vdevs with a ratio of 2 data disks to 2 parity disks for the internal "redundancy groups", so 50% overhead seems par for the course, much like if you had defined 4-disk raidz2 vdevs. - Rich On Wed, Apr 6, 2022 at 9:22 AM Michael Jung (USER) <mikej@paymentallianceintl.com<mailto:mikej@paymentallianceintl.com>> wrote: Hi! I started playing with draid some months ago and I have a problem I cannot figure out. I started out with a single draid2:2d:10c:0s and life was good. Ran some tests, added a special device and life was better. Then I expanded the pool by adding another draid2:2d:10c:0s. Now the problem: the pool says it has ~18TB worth of space, the file system only shows ~9TB. It did not auto-expand. I’m currently on main-n253875-8e72f458c6d: but I initially spun this up just after DRAID was brought Into the tree, at least when I became aware of it. I have tried individually on lining devices in the pool with “-e”, exporting/importing the pool etc. and basically every suggestion my “google foo” would lead me to. I have no useful data on this pool and I can destroy and re-create it but I would like to save some time and get opinions or fact on: Can a draid pool be expanded after a special device has been added? It would seem so but the filesystem does not reflect that.? What at is the correct way to wire down the following – I had problems after not doing this since the 2.x days and would really want to wire down my draid devices now that I have other pools. I have pool construction all scripted but of course I want physical disk da<x> to always be consistent. Then I easily test any construct someone comes up with. root@draid:/home/mikej # camcontrol devlist -b scbus0 on ata0 bus 0 scbus1 on ata1 bus 0 scbus2 on mpt0 bus 0 scbus3 on mps0 bus 0 <-- scbus4 on camsim0 bus 0 scbus-1 on xpt0 bus 0 root@draid:/home/mikej # <dmesg> mps0: <Avago Technologies (LSI) SAS2008> port 0x5000-0x50ff mem 0xfd4fc000-0xfd4fffff,0xfd480000-0xfd4bffff irq 19 at device 0.0 on pci4 mps0: Firmware: 20.00.04.00, Driver: 21.02.00.00-fbsd mps0: IOCCapabilities: 1285c<ScsiTaskFull,DiagTrace,SnapBuf,EEDP,TransRetry,EventReplay,HostDisc> <camcontrol> <SEAGATE ST91000640SS AS09> at scbus3 target 63 lun 0 (da42,pass35) <SEAGATE ST91000640SS AS09> at scbus3 target 67 lun 0 (da46,pass39) <SEAGATE ST91000640SS AS08> at scbus3 target 68 lun 0 (da47,pass40) EXAMPLE: Is this correct? hint.scbus.3.at<http://hint.scbus.3.at>=”mps0 hint.da.42.at<http://hint.da.42.at>=”scbus3 hint.da.42.target=”63” hint.da.42.unit=”0” hint.da.46.at<http://hint.da.46.at>=”scbus3 hint.da.46.target=”67” hint.da.46.unit=”0” hint.da.47.at<http://hint.da.47.at>=”scbus3 hint.da.47.target=”68” hint.da.47.unit=”0" I will try again, but this did not seem to work for me. This is my home lab but it’s now painful to spin down this host as it’s a iSCSI target for several machines for a ESXi project I’m working on so I would like to put wired down SCSI devices to rest. And yes, I have scheduled VM backups for my project to another data store that is not this box ;-) It’s also odd, that I did not get all GPT labels in the zpool. All disks have GPT labels. I have added for the next reboot to loader.conf. kern.geom.label.disk_ident.enable="0" kern.geom.label.gptid.enable="1 This is an old Dell MD-1000 shelf I had laying around with a bunch of 1TB drives which I thought would be perfect for playing around with draid. mikej@draid:~ $ zpool get all tank | grep expand tank autoexpand on local tank expandsize - - mikej@draid:~ $ zpool list -v tank NAME SIZE ALLOC FREE CKPOINT EXPANDSZ FRAG CAP DEDUP HEALTH ALTROOT tank 18.5T 8.02T 10.4T - - 0% 43% 1.00x ONLINE - draid2:2d:10c:0s-0 9.03T 4.06T 4.97T - - 0% 44.9% - ONLINE gpt/da0p1 - - - - - - - - ONLINE gpt/d10p1 - - - - - - - - ONLINE da39p1 - - - - - - - - ONLINE da40p1 - - - - - - - - ONLINE da41p1 - - - - - - - - ONLINE da43p1 - - - - - - - - ONLINE da44p1 - - - - - - - - ONLINE da42p1 - - - - - - - - ONLINE da45p1 - - - - - - - - ONLINE gpt/d19p1 - - - - - - - - ONLINE draid2:2d:10c:0s-1 9.03T 3.96T 5.07T - - 0% 43.8% - ONLINE da46p1 - - - - - - - - ONLINE gpt/d11p1 - - - - - - - - ONLINE gpt/d12p1 - - - - - - - - ONLINE gpt/d13p1 - - - - - - - - ONLINE gpt/d14p1 - - - - - - - - ONLINE gpt/d15p1 - - - - - - - - ONLINE gpt/d16p1 - - - - - - - - ONLINE gpt/d17p1 - - - - - - - - ONLINE gpt/d18p1 - - - - - - - - ONLINE da18p1 - - - - - - - - ONLINE special - - - - - - - - - mirror-3 398G 2.86G 395G - - 0% 0.71% - ONLINE gpt/special0 - - - - - - - - ONLINE gpt/special1 - - - - - - - - ONLINE logs - - - - - - - - - mirror-2 15.5G 256K 15.5G - - 0% 0.00% - ONLINE gpt/slog0 - - - - - - - - ONLINE gpt/slog1 - - - - - - - - ONLINE mikej@draid:~ $ zfs get mountpoint tank NAME PROPERTY VALUE SOURCE tank mountpoint /tank default mikej@draid:~ $ df /tank Filesystem 1K-blocks Used Avail Capacity Mounted on tank 9558892004 4301310224 5257581780 45% /tank <-- ~9TB not 18TB mikej@draid:~ $ mikej@draid:~ $ zpool list NAME SIZE ALLOC FREE CKPOINT EXPANDSZ FRAG CAP DEDUP HEALTH ALTROOT ccache 9.50G 9.10G 406M - - 89% 95% 1.00x ONLINE - raid-5400-1 6.28T 1.19T 5.09T - - 6% 19% 1.00x ONLINE - tank 18.5T 8.02T 10.4T - - 0% 43% 1.00x ONLINE - <-- zfsroot 103G 32.6G 70.4G - - 25% 31% 1.00x ONLINE - mikej@draid:~ $ Thanks in advance for any comments or sugesstions. --mikej CONFIDENTIALITY NOTE: This message is intended only for the use of the individual or entity to whom it is addressed and may contain information that is privileged, confidential, and exempt from disclosure under applicable law. If the reader of this message is not the intended recipient, you are hereby notified that any dissemination, distribution or copying of this communication is strictly prohibited. If you have received this transmission in error, please notify us by telephone at (502) 212-4000 or notify us at: PAI, Dept. 99, 2101 High Wickham Place, Suite 101, Louisville, KY 40245 Disclaimer The information contained in this communication from the sender is confidential. It is intended solely for use by the recipient and others authorized to receive it. If you are not the recipient, you are hereby notified that any disclosure, copying, distribution or taking action in relation of the contents of this information is strictly prohibited and may be unlawful. This email has been scanned for viruses and malware, and may have been automatically archived by Mimecast, a leader in email security and cyber resilience. Mimecast integrates email defenses with brand protection, security awareness training, web security, compliance and other essential capabilities. Mimecast helps protect large and small organizations from malicious activity, human error and technology failure; and to lead the movement toward building a more resilient world. To find out more, visit our website. Disclaimer The information contained in this communication from the sender is confidential. It is intended solely for use by the recipient and others authorized to receive it. If you are not the recipient, you are hereby notified that any disclosure, copying, distribution or taking action in relation of the contents of this information is strictly prohibited and may be unlawful. This email has been scanned for viruses and malware, and may have been automatically archived by Mimecast, a leader in email security and cyber resilience. Mimecast integrates email defenses with brand protection, security awareness training, web security, compliance and other essential capabilities. Mimecast helps protect large and small organizations from malicious activity, human error and technology failure; and to lead the movement toward building a more resilient world. To find out more, visit our website. [-- Attachment #2 --] <html><head> <meta http-equiv="Content-Type" content="text/html; charset=utf-8"> <meta name="Generator" content="Microsoft Word 15 (filtered medium)"> <style><!-- /* Font Definitions */ @font-face {font-family:Wingdings; panose-1:5 0 0 0 0 0 0 0 0 0;} @font-face {font-family:"Cambria Math"; panose-1:2 4 5 3 5 4 6 3 2 4;} @font-face {font-family:Calibri; panose-1:2 15 5 2 2 2 4 3 2 4;} @font-face {font-family:Verdana; panose-1:2 11 6 4 3 5 4 4 2 4;} /* Style Definitions */ p.MsoNormal, li.MsoNormal, div.MsoNormal {margin:0in; margin-bottom:.0001pt; font-size:12.0pt; font-family:"Times New Roman",serif;} a:link, span.MsoHyperlink {mso-style-priority:99; color:blue; text-decoration:underline;} a:visited, span.MsoHyperlinkFollowed {mso-style-priority:99; color:purple; text-decoration:underline;} p {mso-style-priority:99; mso-margin-top-alt:auto; margin-right:0in; mso-margin-bottom-alt:auto; margin-left:0in; font-size:12.0pt; font-family:"Times New Roman",serif;} span.EmailStyle18 {mso-style-type:personal; font-family:"Calibri",sans-serif; color:#1F497D;} span.EmailStyle19 {mso-style-type:personal-compose; font-family:"Calibri",sans-serif; color:windowtext;} .MsoChpDefault {mso-style-type:export-only; font-family:"Calibri",sans-serif;} @page WordSection1 {size:8.5in 11.0in; margin:1.0in 1.0in 1.0in 1.0in;} div.WordSection1 {page:WordSection1;} --></style><!--[if gte mso 9]><xml> <o:shapedefaults v:ext="edit" spidmax="1026" /> </xml><![endif]--><!--[if gte mso 9]><xml> <o:shapelayout v:ext="edit"> <o:idmap v:ext="edit" data="1" /> </o:shapelayout></xml><![endif]--> <style type="text/css">.style1 {font-family: "Times New Roman";}</style></head><body lang="EN-US" link="blue" vlink="purple"> <div class="WordSection1"> <p class="MsoNormal"><span style="font-size:8.0pt;font-family:"Courier New"">Thanks Rich.<o:p></o:p></span></p> <p class="MsoNormal"><span style="font-size:8.0pt;font-family:"Courier New""><o:p> </o:p></span></p> <p class="MsoNormal"><span style="font-size:8.0pt;font-family:"Courier New"">I hear what you are saying, but I added them like you would add a second vdev to a raidz1/z2 pool. None of the online talks nor documentation mention draid expansion, and I was under the impression that you could not expand a draid2:2d:10c:0s to draid2:2d:20c:0s for example.<o:p></o:p></span></p> <p class="MsoNormal"><span style="font-size:8.0pt;font-family:"Courier New""><o:p> </o:p></span></p> <p class="MsoNormal"><span style="font-size:8.0pt;font-family:"Courier New"">I absolutely could be wrong in my assumption but do not see how to do the expansion other than how I did it.<o:p></o:p></span></p> <p class="MsoNormal"><span style="font-size:8.0pt;font-family:"Courier New""><o:p> </o:p></span></p> <p class="MsoNormal"><span style="font-size:8.0pt;font-family:"Courier New"">--mikej<o:p></o:p></span></p> <p class="MsoNormal"><span style="font-size:8.0pt;font-family:"Courier New";color:#1F497D"><o:p> </o:p></span></p> <p class="MsoNormal"><span style="font-size:8.0pt;font-family:"Courier New";color:#1F497D">--------------------------------<o:p></o:p></span></p> <p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D"><o:p> </o:p></span></p> <div> <p class="MsoNormal">I'm not an expert on draid, but you appear to have defined two draid vdevs with a ratio of 2 data disks to 2 parity disks for the internal "redundancy groups", so 50% overhead seems par for the course, much like if you had defined 4-disk raidz2 vdevs.<o:p></o:p></p> <div> <p class="MsoNormal"><o:p> </o:p></p> </div> <div> <p class="MsoNormal">- Rich<o:p></o:p></p> </div> </div> <p class="MsoNormal"><o:p> </o:p></p> <div> <div> <p class="MsoNormal">On Wed, Apr 6, 2022 at 9:22 AM Michael Jung (USER) <<a href="mailto:mikej@paymentallianceintl.com">mikej@paymentallianceintl.com</a>> wrote:<o:p></o:p></p> </div> <blockquote style="border:none;border-left:solid #CCCCCC 1.0pt;padding:0in 0in 0in 6.0pt;margin-left:4.8pt;margin-right:0in"> <div> <div> <p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span style="font-size:8.0pt;font-family:"Courier New"">Hi!</span><o:p></o:p></p> <p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span style="font-size:8.0pt;font-family:"Courier New""> </span><o:p></o:p></p> <p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span style="font-size:8.0pt;font-family:"Courier New"">I started playing with draid some months ago and I have a problem I cannot figure out.</span><o:p></o:p></p> <p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span style="font-size:8.0pt;font-family:"Courier New""> </span><o:p></o:p></p> <p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span style="font-size:8.0pt;font-family:"Courier New"">I started out with a single draid2:2d:10c:0s and life was good. Ran some tests, added </span><o:p></o:p></p> <p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span style="font-size:8.0pt;font-family:"Courier New"">a special device and life was better. Then I expanded the pool by adding another </span><o:p></o:p></p> <p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span style="font-size:8.0pt;font-family:"Courier New"">draid2:2d:10c:0s. Now the problem: the pool says it has ~18TB worth of space, the </span><o:p></o:p></p> <p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span style="font-size:8.0pt;font-family:"Courier New"">file system only shows ~9TB. It did not auto-expand.</span><o:p></o:p></p> <p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span style="font-size:8.0pt;font-family:"Courier New""> </span><o:p></o:p></p> <p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span style="font-size:8.0pt;font-family:"Courier New"">I’m currently on main-n253875-8e72f458c6d: but I initially spun this up just after </span><o:p></o:p></p> <p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span style="font-size:8.0pt;font-family:"Courier New"">DRAID was brought Into the tree, at least when I became aware of it. I have tried </span><o:p></o:p></p> <p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span style="font-size:8.0pt;font-family:"Courier New"">individually on lining devices in the pool with “-e”, exporting/importing the pool etc.</span><o:p></o:p></p> <p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span style="font-size:8.0pt;font-family:"Courier New"">and basically every suggestion my “google foo” would lead me to.</span><o:p></o:p></p> <p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span style="font-size:8.0pt;font-family:"Courier New""> </span><o:p></o:p></p> <p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span style="font-size:8.0pt;font-family:"Courier New"">I have no useful data on this pool and I can destroy and re-create it but I would </span><o:p></o:p></p> <p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span style="font-size:8.0pt;font-family:"Courier New"">like to save some time and get opinions or fact on:</span><o:p></o:p></p> <p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span style="font-size:8.0pt;font-family:"Courier New""> </span><o:p></o:p></p> <p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span style="font-size:8.0pt;font-family:"Courier New"">Can a draid pool be expanded after a special device has been added? It would seem so</span><o:p></o:p></p> <p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span style="font-size:8.0pt;font-family:"Courier New"">but the filesystem does not reflect that.?</span><o:p></o:p></p> <p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span style="font-size:8.0pt;font-family:"Courier New""> </span><o:p></o:p></p> <p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span style="font-size:8.0pt;font-family:"Courier New"">What at is the correct way to wire down the following – I had problems after not doing</span><o:p></o:p></p> <p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span style="font-size:8.0pt;font-family:"Courier New"">this since the 2.x days and would really want to wire down my draid devices now </span><o:p></o:p></p> <p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span style="font-size:8.0pt;font-family:"Courier New"">that I have other pools. </span><o:p></o:p></p> <p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span style="font-size:8.0pt;font-family:"Courier New""> </span><o:p></o:p></p> <p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span style="font-size:8.0pt;font-family:"Courier New"">I have pool construction all scripted but of course I want physical disk da<x> to always be </span><o:p></o:p></p> <p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span style="font-size:8.0pt;font-family:"Courier New"">consistent. Then I easily test any construct someone comes up with.</span><o:p></o:p></p> <p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span style="font-size:8.0pt;font-family:"Courier New""> </span><o:p></o:p></p> <p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span style="font-size:8.0pt;font-family:"Courier New"">root@draid:/home/mikej # camcontrol devlist -b</span><o:p></o:p></p> <p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span style="font-size:8.0pt;font-family:"Courier New"">scbus0 on ata0 bus 0</span><o:p></o:p></p> <p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span style="font-size:8.0pt;font-family:"Courier New"">scbus1 on ata1 bus 0</span><o:p></o:p></p> <p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span style="font-size:8.0pt;font-family:"Courier New"">scbus2 on mpt0 bus 0</span><o:p></o:p></p> <p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span style="font-size:8.0pt;font-family:"Courier New"">scbus3 on mps0 bus 0 </span><span style="font-size:8.0pt;font-family:Wingdings">ß</span><o:p></o:p></p> <p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span style="font-size:8.0pt;font-family:"Courier New"">scbus4 on camsim0 bus 0</span><o:p></o:p></p> <p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span style="font-size:8.0pt;font-family:"Courier New"">scbus-1 on xpt0 bus 0</span><o:p></o:p></p> <p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span style="font-size:8.0pt;font-family:"Courier New"">root@draid:/home/mikej #</span><o:p></o:p></p> <p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto;margin-left:.5in"> <span style="font-size:8.0pt;font-family:"Courier New""> </span><o:p></o:p></p> <p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto;margin-left:.5in"> <span style="font-size:8.0pt;font-family:"Courier New""> </span><o:p></o:p></p> <p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span style="font-size:8.0pt;font-family:"Courier New""><dmesg></span><o:p></o:p></p> <p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span style="font-size:8.0pt;font-family:"Courier New"">mps0: <Avago Technologies (LSI) SAS2008> port 0x5000-0x50ff mem 0xfd4fc000-0xfd4fffff,0xfd480000-0xfd4bffff irq 19 at device 0.0 on pci4</span><o:p></o:p></p> <p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span style="font-size:8.0pt;font-family:"Courier New"">mps0: Firmware: 20.00.04.00, Driver: 21.02.00.00-fbsd</span><o:p></o:p></p> <p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span style="font-size:8.0pt;font-family:"Courier New"">mps0: IOCCapabilities: 1285c<ScsiTaskFull,DiagTrace,SnapBuf,EEDP,TransRetry,EventReplay,HostDisc></span><o:p></o:p></p> <p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span style="font-size:8.0pt;font-family:"Courier New""> </span><o:p></o:p></p> <p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span style="font-size:8.0pt;font-family:"Courier New""><camcontrol></span><o:p></o:p></p> <p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span style="font-size:8.0pt;font-family:"Courier New""><SEAGATE ST91000640SS AS09> at scbus3 target 63 lun 0 (da42,pass35)</span><o:p></o:p></p> <p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span style="font-size:8.0pt;font-family:"Courier New""><SEAGATE ST91000640SS AS09> at scbus3 target 67 lun 0 (da46,pass39)</span><o:p></o:p></p> <p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span style="font-size:8.0pt;font-family:"Courier New""><SEAGATE ST91000640SS AS08> at scbus3 target 68 lun 0 (da47,pass40)</span><o:p></o:p></p> <p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span style="font-size:8.0pt;font-family:"Courier New""> </span><o:p></o:p></p> <p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span style="font-size:8.0pt;font-family:"Courier New"">EXAMPLE: Is this correct?</span><o:p></o:p></p> <p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span style="font-size:8.0pt;font-family:"Courier New""> </span><o:p></o:p></p> <p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span style="font-size:8.0pt;font-family:"Courier New""><a href="http://hint.scbus.3.at" target="_blank">hint.scbus.3.at</a>=”mps0”</span><o:p></o:p></p> <p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span style="font-size:8.0pt;font-family:"Courier New""> </span><o:p></o:p></p> <p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span style="font-size:8.0pt;font-family:"Courier New""><a href="http://hint.da.42.at" target="_blank">hint.da.42.at</a>=”scbus3”</span><o:p></o:p></p> <p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span style="font-size:8.0pt;font-family:"Courier New"">hint.da.42.target=”63”</span><o:p></o:p></p> <p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span style="font-size:8.0pt;font-family:"Courier New"">hint.da.42.unit=”0”</span><o:p></o:p></p> <p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span style="font-size:8.0pt;font-family:"Courier New""> </span><o:p></o:p></p> <p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span style="font-size:8.0pt;font-family:"Courier New""><a href="http://hint.da.46.at" target="_blank">hint.da.46.at</a>=”scbus3”</span><o:p></o:p></p> <p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span style="font-size:8.0pt;font-family:"Courier New"">hint.da.46.target=”67”</span><o:p></o:p></p> <p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span style="font-size:8.0pt;font-family:"Courier New"">hint.da.46.unit=”0”</span><o:p></o:p></p> <p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span style="font-size:8.0pt;font-family:"Courier New""> </span><o:p></o:p></p> <p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span style="font-size:8.0pt;font-family:"Courier New""><a href="http://hint.da.47.at" target="_blank">hint.da.47.at</a>=”scbus3”</span><o:p></o:p></p> <p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span style="font-size:8.0pt;font-family:"Courier New"">hint.da.47.target=”68”</span><o:p></o:p></p> <p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span style="font-size:8.0pt;font-family:"Courier New"">hint.da.47.unit=”0"</span><o:p></o:p></p> <p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span style="font-size:8.0pt;font-family:"Courier New""> </span><o:p></o:p></p> <p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span style="font-size:8.0pt;font-family:"Courier New"">I will try again, but this did not seem to work for me. This is my home lab but it’s</span><o:p></o:p></p> <p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span style="font-size:8.0pt;font-family:"Courier New"">now painful to spin down this host as it’s a iSCSI target for several machines for a</span><o:p></o:p></p> <p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span style="font-size:8.0pt;font-family:"Courier New"">ESXi project I’m working on so I would like to put wired down SCSI devices to rest.</span><o:p></o:p></p> <p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span style="font-size:8.0pt;font-family:"Courier New""> </span><o:p></o:p></p> <p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span style="font-size:8.0pt;font-family:"Courier New"">And yes, I have scheduled VM backups for my project to another data store that is not</span><o:p></o:p></p> <p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span style="font-size:8.0pt;font-family:"Courier New"">this box ;-)</span><o:p></o:p></p> <p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span style="font-size:8.0pt;font-family:"Courier New""> </span><o:p></o:p></p> <p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span style="font-size:8.0pt;font-family:"Courier New"">It’s also odd, that I did not get all GPT labels in the zpool. All disks have GPT labels. </span><o:p></o:p></p> <p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span style="font-size:8.0pt;font-family:"Courier New""> </span><o:p></o:p></p> <p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span style="font-size:8.0pt;font-family:"Courier New"">I have added for the next reboot to loader.conf. </span><o:p></o:p></p> <p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span style="font-size:8.0pt;font-family:"Courier New""> </span><o:p></o:p></p> <p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span style="font-size:8.0pt;font-family:"Courier New"">kern.geom.label.disk_ident.enable="0"</span><o:p></o:p></p> <p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span style="font-size:8.0pt;font-family:"Courier New"">kern.geom.label.gptid.enable="1</span><o:p></o:p></p> <p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span style="font-size:8.0pt;font-family:"Courier New""> </span><o:p></o:p></p> <p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span style="font-size:8.0pt;font-family:"Courier New"">This is an old Dell MD-1000 shelf I had laying around with a bunch of 1TB drives </span><o:p></o:p></p> <p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span style="font-size:8.0pt;font-family:"Courier New"">which I thought would be perfect for playing around with draid.</span><o:p></o:p></p> <p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span style="font-size:8.0pt;font-family:"Courier New""> </span><o:p></o:p></p> <p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span style="font-size:8.0pt;font-family:"Courier New"">mikej@draid:~ $ zpool get all tank | grep expand</span><o:p></o:p></p> <p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span style="font-size:8.0pt;font-family:"Courier New"">tank autoexpand on local</span><o:p></o:p></p> <p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span style="font-size:8.0pt;font-family:"Courier New"">tank expandsize - -</span><o:p></o:p></p> <p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span style="font-size:8.0pt;font-family:"Courier New"">mikej@draid:~ $ zpool list -v tank</span><o:p></o:p></p> <p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span style="font-size:8.0pt;font-family:"Courier New"">NAME SIZE ALLOC FREE CKPOINT EXPANDSZ FRAG CAP DEDUP HEALTH ALTROOT</span><o:p></o:p></p> <p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span style="font-size:8.0pt;font-family:"Courier New"">tank 18.5T 8.02T 10.4T - - 0% 43% 1.00x ONLINE -</span><o:p></o:p></p> <p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span style="font-size:8.0pt;font-family:"Courier New""> draid2:2d:10c:0s-0 9.03T 4.06T 4.97T - - 0% 44.9% - ONLINE</span><o:p></o:p></p> <p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span style="font-size:8.0pt;font-family:"Courier New""> gpt/da0p1 - - - - - - - - ONLINE</span><o:p></o:p></p> <p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span style="font-size:8.0pt;font-family:"Courier New""> gpt/d10p1 - - - - - - - - ONLINE</span><o:p></o:p></p> <p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span style="font-size:8.0pt;font-family:"Courier New""> da39p1 - - - - - - - - ONLINE</span><o:p></o:p></p> <p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span style="font-size:8.0pt;font-family:"Courier New""> da40p1 - - - - - - - - ONLINE</span><o:p></o:p></p> <p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span style="font-size:8.0pt;font-family:"Courier New""> da41p1 - - - - - - - - ONLINE</span><o:p></o:p></p> <p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span style="font-size:8.0pt;font-family:"Courier New""> da43p1 - - - - - - - - ONLINE</span><o:p></o:p></p> <p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span style="font-size:8.0pt;font-family:"Courier New""> da44p1 - - - - - - - - ONLINE</span><o:p></o:p></p> <p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span style="font-size:8.0pt;font-family:"Courier New""> da42p1 - - - - - - - - ONLINE</span><o:p></o:p></p> <p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span style="font-size:8.0pt;font-family:"Courier New""> da45p1 - - - - - - - - ONLINE</span><o:p></o:p></p> <p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span style="font-size:8.0pt;font-family:"Courier New""> gpt/d19p1 - - - - - - - - ONLINE</span><o:p></o:p></p> <p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span style="font-size:8.0pt;font-family:"Courier New""> draid2:2d:10c:0s-1 9.03T 3.96T 5.07T - - 0% 43.8% - ONLINE</span><o:p></o:p></p> <p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span style="font-size:8.0pt;font-family:"Courier New""> da46p1 - - - - - - - - ONLINE</span><o:p></o:p></p> <p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span style="font-size:8.0pt;font-family:"Courier New""> gpt/d11p1 - - - - - - - - ONLINE</span><o:p></o:p></p> <p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span style="font-size:8.0pt;font-family:"Courier New""> gpt/d12p1 - - - - - - - - ONLINE</span><o:p></o:p></p> <p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span style="font-size:8.0pt;font-family:"Courier New""> gpt/d13p1 - - - - - - - - ONLINE</span><o:p></o:p></p> <p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span style="font-size:8.0pt;font-family:"Courier New""> gpt/d14p1 - - - - - - - - ONLINE</span><o:p></o:p></p> <p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span style="font-size:8.0pt;font-family:"Courier New""> gpt/d15p1 - - - - - - - - ONLINE</span><o:p></o:p></p> <p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span style="font-size:8.0pt;font-family:"Courier New""> gpt/d16p1 - - - - - - - - ONLINE</span><o:p></o:p></p> <p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span style="font-size:8.0pt;font-family:"Courier New""> gpt/d17p1 - - - - - - - - ONLINE</span><o:p></o:p></p> <p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span style="font-size:8.0pt;font-family:"Courier New""> gpt/d18p1 - - - - - - - - ONLINE</span><o:p></o:p></p> <p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span style="font-size:8.0pt;font-family:"Courier New""> da18p1 - - - - - - - - ONLINE</span><o:p></o:p></p> <p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span style="font-size:8.0pt;font-family:"Courier New"">special - - - - - - - - -</span><o:p></o:p></p> <p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span style="font-size:8.0pt;font-family:"Courier New""> mirror-3 398G 2.86G 395G - - 0% 0.71% - ONLINE</span><o:p></o:p></p> <p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span style="font-size:8.0pt;font-family:"Courier New""> gpt/special0 - - - - - - - - ONLINE</span><o:p></o:p></p> <p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span style="font-size:8.0pt;font-family:"Courier New""> gpt/special1 - - - - - - - - ONLINE</span><o:p></o:p></p> <p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span style="font-size:8.0pt;font-family:"Courier New"">logs - - - - - - - - -</span><o:p></o:p></p> <p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span style="font-size:8.0pt;font-family:"Courier New""> mirror-2 15.5G 256K 15.5G - - 0% 0.00% - ONLINE</span><o:p></o:p></p> <p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span style="font-size:8.0pt;font-family:"Courier New""> gpt/slog0 - - - - - - - - ONLINE</span><o:p></o:p></p> <p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span style="font-size:8.0pt;font-family:"Courier New""> gpt/slog1 - - - - - - - - ONLINE</span><o:p></o:p></p> <p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span style="font-size:8.0pt;font-family:"Courier New""> </span><o:p></o:p></p> <p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span style="font-size:8.0pt;font-family:"Courier New"">mikej@draid:~ $ zfs get mountpoint tank</span><o:p></o:p></p> <p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span style="font-size:8.0pt;font-family:"Courier New"">NAME PROPERTY VALUE SOURCE</span><o:p></o:p></p> <p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span style="font-size:8.0pt;font-family:"Courier New"">tank mountpoint /tank default</span><o:p></o:p></p> <p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span style="font-size:8.0pt;font-family:"Courier New""> </span><o:p></o:p></p> <p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span style="font-size:8.0pt;font-family:"Courier New""> </span><o:p></o:p></p> <p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span style="font-size:8.0pt;font-family:"Courier New"">mikej@draid:~ $ df /tank</span><o:p></o:p></p> <p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span style="font-size:8.0pt;font-family:"Courier New"">Filesystem 1K-blocks Used Avail Capacity Mounted on</span><o:p></o:p></p> <p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span style="font-size:8.0pt;font-family:"Courier New"">tank 9558892004 4301310224 5257581780 45% /tank </span><span style="font-size:8.0pt;font-family:Wingdings">ß</span><span style="font-size:8.0pt;font-family:"Courier New""> ~9TB not 18TB</span><o:p></o:p></p> <p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span style="font-size:8.0pt;font-family:"Courier New"">mikej@draid:~ $</span><o:p></o:p></p> <p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span style="font-size:8.0pt;font-family:"Courier New""> </span><o:p></o:p></p> <p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span style="font-size:8.0pt;font-family:"Courier New"">mikej@draid:~ $ zpool list</span><o:p></o:p></p> <p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span style="font-size:8.0pt;font-family:"Courier New"">NAME SIZE ALLOC FREE CKPOINT EXPANDSZ FRAG CAP DEDUP HEALTH ALTROOT</span><o:p></o:p></p> <p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span style="font-size:8.0pt;font-family:"Courier New"">ccache 9.50G 9.10G 406M - - 89% 95% 1.00x ONLINE -</span><o:p></o:p></p> <p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span style="font-size:8.0pt;font-family:"Courier New"">raid-5400-1 6.28T 1.19T 5.09T - - 6% 19% 1.00x ONLINE -</span><o:p></o:p></p> <p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span style="font-size:8.0pt;font-family:"Courier New"">tank 18.5T 8.02T 10.4T - - 0% 43% 1.00x ONLINE - </span><span style="font-size:8.0pt;font-family:Wingdings">ß</span><span style="font-size:8.0pt;font-family:"Courier New""> </span><o:p></o:p></p> <p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span style="font-size:8.0pt;font-family:"Courier New"">zfsroot 103G 32.6G 70.4G - - 25% 31% 1.00x ONLINE -</span><o:p></o:p></p> <p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span style="font-size:8.0pt;font-family:"Courier New"">mikej@draid:~ $</span><o:p></o:p></p> <p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span style="font-size:8.0pt;font-family:"Courier New""> </span><o:p></o:p></p> <p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span style="font-size:8.0pt;font-family:"Courier New"">Thanks in advance for any comments or sugesstions.</span><o:p></o:p></p> <p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span style="font-size:8.0pt;font-family:"Courier New""> </span><o:p></o:p></p> <p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span style="font-size:8.0pt;font-family:"Courier New"">--mikej</span><o:p></o:p></p> <p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"> <o:p></o:p></p> <p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"> <o:p></o:p></p> </div> <div> <p><span style="font-size:10.0pt;font-family:"Arial",sans-serif"> <o:p></o:p></span></p> <div style="border-top:solid #666666 1.0pt;border-left:none;border-bottom:solid #666666 1.0pt;border-right:none;padding:4.0pt 0in 4.0pt 0in"> <p class="MsoNormal" style="margin-bottom:12.0pt"><span style="font-size:7.5pt;font-family:"Verdana",sans-serif">CONFIDENTIALITY NOTE: This message is intended only for the use<br> of the individual or entity to whom it is addressed and may <br> contain information that is privileged, confidential, and <br> exempt from disclosure under applicable law. If the reader <br> of this message is not the intended recipient, you are hereby <br> notified that any dissemination, distribution or copying <br> of this communication is strictly prohibited. If you have <br> received this transmission in error, please notify us by <br> telephone at (502) 212-4000 or notify us at: PAI, Dept. 99, <br> 2101 High Wickham Place, Suite 101, Louisville, KY 40245<br> <br> <br> <o:p></o:p></span></p> </div> </div> <p class="MsoNormal" style="margin-bottom:12.0pt"><o:p> </o:p></p> <p><b><span style="font-size:10.0pt;font-family:"Verdana",sans-serif;color:#666666">Disclaimer</span></b><span style="font-size:10.0pt;font-family:"Verdana",sans-serif;color:#666666"><o:p></o:p></span></p> <p><span style="font-size:8.0pt;font-family:"Verdana",sans-serif;color:#666666">The information contained in this communication from the sender is confidential. It is intended solely for use by the recipient and others authorized to receive it. If you are not the recipient, you are hereby notified that any disclosure, copying, distribution or taking action in relation of the contents of this information is strictly prohibited and may be unlawful.<br> <br> This email has been scanned for viruses and malware, and may have been automatically archived by Mimecast, a leader in email security and cyber resilience. Mimecast integrates email defenses with brand protection, security awareness training, web security, compliance and other essential capabilities. Mimecast helps protect large and small organizations from malicious activity, human error and technology failure; and to lead the movement toward building a more resilient world. To find out more, visit our website.<o:p></o:p></span></p> </div> </blockquote> </div> </div> <br><br><p style="font-family: Verdana; font-size:10pt; color:#666666;"><b>Disclaimer</b></p><p style="font-family: Verdana; font-size:8pt; color:#666666;">The information contained in this communication from the sender is confidential. It is intended solely for use by the recipient and others authorized to receive it. If you are not the recipient, you are hereby notified that any disclosure, copying, distribution or taking action in relation of the contents of this information is strictly prohibited and may be unlawful.<br><br>This email has been scanned for viruses and malware, and may have been automatically archived by Mimecast, a leader in email security and cyber resilience. Mimecast integrates email defenses with brand protection, security awareness training, web security, compliance and other essential capabilities. Mimecast helps protect large and small organizations from malicious activity, human error and technology failure; and to lead the movement toward building a more resilient world. To find out more, visit our website.</p></body></html>
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?fe01e1eebc4f46c58e6732b89781221b>