Date: Fri, 2 Oct 2015 10:24:18 +0200 From: Willem Jan Withagen <wjw@digiware.nl> To: Tom Curry <thomasrcurry@gmail.com> Cc: "freebsd-fs@freebsd.org" <freebsd-fs@freebsd.org> Subject: Re: ZFS, Zvol, iSCSI and windows Message-ID: <560E3F32.5060600@digiware.nl> In-Reply-To: <5606EBEA.2090103@digiware.nl> References: <56050EFC.1000303@digiware.nl> <1443179614.5271.42.camel@data-b104.adm.slu.se> <CAGtEZUD-e-9uH1H3hzL2Twj%2BvpMQJtw0neQ4TtQTJK392EKUJw@mail.gmail.com> <5606EBEA.2090103@digiware.nl>
next in thread | previous in thread | raw e-mail | index | archive | help
On 26-9-2015 21:03, Willem Jan Withagen wrote: > On 25-9-2015 16:05, Tom Curry wrote: >> On Fri, Sep 25, 2015 at 7:13 AM, Karli Sjöberg <karli.sjoberg@slu.se >> <mailto:karli.sjoberg@slu.se>> wrote: >> >> fre 2015-09-25 klockan 11:08 +0200 skrev Willem Jan Withagen: >> > Hi, >> > >> > Because of the Network Video Recorder (on windows) we use only likes >> > "real" disks, and not SMB disks. We started using ZVOLs which are >> > exported thru iSCSI/ctld.... >> > >> > And that works really well, so there all thumbs up for this combo. >> > >> > However..... (you knew that was coming) >> > I do have some questions, and hope somebody can share some insights. >> > >> > 1) >> > This is a ZFS question. >> > >> > I have created a ZVOL with: >> > zfs create -s -V 5T zfsraid/nvr2 >> > >> > Looking at the disk usage in 'zfs get all': >> > NAME PROPERTY VALUE SOURCE >> > zfsraid/nvr2 used 9.97T - >> > zfsraid/nvr2 available 438G - >> > zfsraid/nvr2 referenced 9.97T - >> > zfsraid/nvr2 compressratio 1.00x - >> > zfsraid/nvr2 reservation none default >> > zfsraid/nvr2 volsize 5T local >> > zfsraid/nvr2 volblocksize 8K - >> > zfsraid/nvr2 checksum on default >> > zfsraid/nvr2 compression lz4 default >> > zfsraid/nvr2 primarycache all default >> > zfsraid/nvr2 secondarycache all default >> > zfsraid/nvr2 usedbysnapshots 0 - >> > zfsraid/nvr2 usedbydataset 9.97T - >> > zfsraid/nvr2 usedbychildren 0 - >> > zfsraid/nvr2 usedbyrefreservation 0 - >> > zfsraid/nvr2 sync standard default >> > zfsraid/nvr2 written 9.97T - >> > zfsraid/nvr2 logicalused 4.97T - >> > zfsraid/nvr2 logicalreferenced 4.97T - >> > zfsraid/nvr2 volmode default default >> > >> > And what sort of "worries" me is that it seems that this volume is >> using >> > twice the amount of diskspace it is offering as ZVOL? >> > >> > a) Is this really true? >> > b) Should I have done something different to not waste so much >> overhead? >> > >> > Note that the compression rate is 1.00x, which is of course due to >> > writing h264 media streams that do not compress at all. But is the >> > parents default, and I haven't turned it off. >> > >> > 2) >> > The second one might be more a Windows question, but anyways. >> > >> > The export is that same 5T ZVOL, plain create with >> > zfs create -s -V 5T zfsraid/nvr2 >> > Under Windows I used the regular stuff to format the dis with GPT and >> > NTFS with 8k segments. (matching the ZVOL blocksize) >> > >> > Upon reboot FreeBSD notices the following: >> > GEOM_PART: partition 1 on (zvol/zfsraid/nvr2, GPT) is not aligned on >> > 8192 bytes >> > By itself not technical problem. But the warning does hint that the >> > alignment is thus, that performance might suffer. the backing >> disks are >> > WD REDs which are 4K sectors.... So the misalignment could cause too >> > many read/writes for writing. >> > >> > Does anybody have a clue as how to get Windows to do the aligning >> correctly? >> >> >> Is it this effect you are seeing: >> https://forums.freebsd.org/threads/mitigating-4k-disk-issues-on-raidz.37365/ > > The article attributes certain issues to using 4k blocks zvols in zraid. > It could very well be on the money. But it seems testing is the best way > to find out. > >> >> I ran into the same problem, the solution was to create the zvol with a >> larger volblocksize like 32k. If space efficiency is the primary concern >> I believe you need to choose a blocksize that is greater than or equal >> to the number of non parity drives multiplied by their sector size. >> Judging by your overhead I'm going to assume you have a 6 disk raidz2, >> so 4 * 4k = 16k at least. Try creating another zvol with "-o >> volblocksize=16k" and see how that treats you. > > Right guess, > > Actually the it is a 2* 6 vol raidz2 > > THe article seems to show that the overhead gets less and less when the > blocksize increases. > In which case I start worrying about the performance, because writting > smaller blocks will require a read/wite cycle. > > In my specific case I'm writing large video files (5Gb) in streaming > mode, so the read/write effect will be mitigated due to streaming. > > I'll go and make a few ZVOLS and see what happens with the overhead when > they start filling up. But it looks like the windows blocksize should > certainly not match zvol blocksize. > >> As to your second question, modern versions of windows should align >> partitions correctly, what version are you using? You may wish to open >> an elevated command prompt and sanity check the output of "fsutil fsinfo >> ntfsinfo x:" > > I'm just loading the disk in explorer, rightclick it, and then format. > Just as basic as that. > Not sure what to make of it, but this gives: > > C:\WINDOWS\system32>fsutil fsinfo ntfsinfo m: > NTFS Volume Serial Number : 0x2aaa50b8aa50826d > NTFS Version : 3.1 > LFS Version : 2.0 > Number Sectors : 0x000000027ffbefff > Total Clusters : 0x0000000027ffbeff > Free Clusters : 0x00000000008497fc > Total Reserved : 0x00000000000a8ec0 > Bytes Per Sector : 512 > Bytes Per Physical Sector : 4096 > Bytes Per Cluster : 8192 > Bytes Per FileRecord Segment : 1024 > Clusters Per FileRecord Segment : 0 > Mft Valid Data Length : 0x0000000000140000 > Mft Start Lcn : 0x0000000000060000 > Mft2 Start Lcn : 0x0000000000000001 > Mft Zone Start : 0x00000000000600a0 > Mft Zone End : 0x0000000000066420 > Max Device Trim Extent Count : 4294967295 > Max Device Trim Byte Count : 0xffffffff > Max Volume Trim Extent Count : 62 > Max Volume Trim Byte Count : 0x40000000 > Resource Manager Identifier : FD836E1F-1372-11E5-825F-000272131F3C > > The gpart info in the zvol is reported as unaligned when booting > FreeBSD, but that would be because windows put it in there in an > unaligned postition. Something like at 63 * 512 bytes...?? > > Any commandline incarnation that allows me to put it a a different offset? > >> > >> Finally, from my own experience with using iscsi under windows I >> found I had a random 4k iops increase from ~10,000 to ~90,000 (which >> is close to my pools real speed) by adjusting the following registry >> key > > >> Create this DWORD key with a value of 1: >> HKLM\SYSTEM\CurrentControlSet\Control\Class\{4D36E97B-E325-11CE-BFC1-08002BE10318}\<Instance >> Number>\Parameters\iSCSIDisableNagle > > Very usefull hint, thanx for that. And as a followup, I created a new ZVOL: zfs create -V 2T -b 128k -o compression=off zfsraid/nvr That gives: NAME PROPERTY VALUE SOURCE zfsraid/nvr used 2.00T - zfsraid/nvr referenced 1.86T - zfsraid/nvr compressratio 1.00x - zfsraid/nvr reservation none default zfsraid/nvr volsize 2T local zfsraid/nvr volblocksize 128K - zfsraid/nvr checksum on default zfsraid/nvr compression off local zfsraid/nvr refreservation 2.00T local zfsraid/nvr usedbysnapshots 0 - zfsraid/nvr usedbydataset 1.86T - zfsraid/nvr usedbychildren 0 - zfsraid/nvr usedbyrefreservation 150G - zfsraid/nvr written 1.86T - zfsraid/nvr logicalused 1.86T - zfsraid/nvr logicalreferenced 1.86T - And if there is any overhead, it does not really show here. So the moral of the story: It is not always wise to match NTFS blocksize with the ZVOL blocksize. As for possible performance penalties on the Windows side.... It is hard to find a niche and sensible disk tester. Using the error scanner from HDtune, performance has dropped a tiny bit. I'm seeing a 1000 IOPS and 30Mbyte/sec of throuput. This is on a 2* vdev 6* disk raidz2 volume. --WjW
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?560E3F32.5060600>