Skip site navigation (1)Skip section navigation (2)
Date:      Sat, 26 Sep 2015 21:03:06 +0200
From:      Willem Jan Withagen <wjw@digiware.nl>
To:        Tom Curry <thomasrcurry@gmail.com>
Cc:        "freebsd-fs@freebsd.org" <freebsd-fs@freebsd.org>
Subject:   Re: ZFS, Zvol, iSCSI and windows
Message-ID:  <5606EBEA.2090103@digiware.nl>
In-Reply-To: <CAGtEZUD-e-9uH1H3hzL2Twj%2BvpMQJtw0neQ4TtQTJK392EKUJw@mail.gmail.com>
References:  <56050EFC.1000303@digiware.nl> <1443179614.5271.42.camel@data-b104.adm.slu.se> <CAGtEZUD-e-9uH1H3hzL2Twj%2BvpMQJtw0neQ4TtQTJK392EKUJw@mail.gmail.com>

next in thread | previous in thread | raw e-mail | index | archive | help
On 25-9-2015 16:05, Tom Curry wrote:
> On Fri, Sep 25, 2015 at 7:13 AM, Karli Sjöberg <karli.sjoberg@slu.se
> <mailto:karli.sjoberg@slu.se>> wrote:
> 
>     fre 2015-09-25 klockan 11:08 +0200 skrev Willem Jan Withagen:
>     > Hi,
>     >
>     > Because of the Network Video Recorder (on windows) we use only likes
>     > "real" disks, and not SMB disks. We started using ZVOLs which are
>     > exported thru iSCSI/ctld....
>     >
>     > And that works really well, so there all thumbs up for this combo.
>     >
>     > However..... (you knew that was coming)
>     > I do have some questions, and hope somebody can share some insights.
>     >
>     > 1)
>     > This is a ZFS question.
>     >
>     > I have created a ZVOL with:
>     >       zfs create -s -V 5T zfsraid/nvr2
>     >
>     > Looking at the disk usage in 'zfs get all':
>     > NAME          PROPERTY              VALUE                  SOURCE
>     > zfsraid/nvr2  used                  9.97T                  -
>     > zfsraid/nvr2  available             438G                   -
>     > zfsraid/nvr2  referenced            9.97T                  -
>     > zfsraid/nvr2  compressratio         1.00x                  -
>     > zfsraid/nvr2  reservation           none                   default
>     > zfsraid/nvr2  volsize               5T                     local
>     > zfsraid/nvr2  volblocksize          8K                     -
>     > zfsraid/nvr2  checksum              on                     default
>     > zfsraid/nvr2  compression           lz4                    default
>     > zfsraid/nvr2  primarycache          all                    default
>     > zfsraid/nvr2  secondarycache        all                    default
>     > zfsraid/nvr2  usedbysnapshots       0                      -
>     > zfsraid/nvr2  usedbydataset         9.97T                  -
>     > zfsraid/nvr2  usedbychildren        0                      -
>     > zfsraid/nvr2  usedbyrefreservation  0                      -
>     > zfsraid/nvr2  sync                  standard               default
>     > zfsraid/nvr2  written               9.97T                  -
>     > zfsraid/nvr2  logicalused           4.97T                  -
>     > zfsraid/nvr2  logicalreferenced     4.97T                  -
>     > zfsraid/nvr2  volmode               default                default
>     >
>     > And what sort of "worries" me is that it seems that this volume is
>     using
>     > twice the amount of diskspace it is offering as ZVOL?
>     >
>     > a) Is this really true?
>     > b) Should I have done something different to not waste so much
>     overhead?
>     >
>     > Note that the compression rate is 1.00x, which is of course due to
>     > writing h264 media streams that do not compress at all. But is the
>     > parents default, and I haven't turned it off.
>     >
>     > 2)
>     > The second one might be more a Windows question, but anyways.
>     >
>     > The export is that same 5T ZVOL, plain create with
>     >       zfs create -s -V 5T zfsraid/nvr2
>     > Under Windows I used the regular stuff to format the dis with GPT and
>     > NTFS with 8k segments. (matching the ZVOL blocksize)
>     >
>     > Upon reboot FreeBSD notices the following:
>     > GEOM_PART: partition 1 on (zvol/zfsraid/nvr2, GPT) is not aligned on
>     >       8192 bytes
>     > By itself not technical problem. But the warning does hint that the
>     > alignment is thus, that performance might suffer. the backing
>     disks are
>     > WD REDs which are 4K sectors.... So the misalignment could cause too
>     > many read/writes for writing.
>     >
>     > Does anybody have a clue as how to get Windows to do the aligning
>     correctly?
> 
> 
>     Is it this effect you are seeing:
>     https://forums.freebsd.org/threads/mitigating-4k-disk-issues-on-raidz.37365/

The article attributes certain issues to using 4k blocks zvols in zraid.
It could very well be on the money. But it seems testing is the best way
to find out.

> 
> I ran into the same problem, the solution was to create the zvol with a
> larger volblocksize like 32k. If space efficiency is the primary concern
> I believe you need to choose a blocksize that is greater than or equal
> to the number of non parity drives multiplied by their sector size.
> Judging by your overhead I'm going to assume you have a 6 disk raidz2,
> so 4 * 4k = 16k at least. Try creating another zvol with "-o
> volblocksize=16k" and see how that treats you.

Right guess,

Actually the it is a 2* 6 vol raidz2

THe article seems to show that the overhead gets less and less when the
blocksize increases.
In which case I start worrying about the performance, because writting
smaller blocks will require a read/wite cycle.

In my specific case I'm writing large video files (5Gb) in streaming
mode, so the read/write effect will be mitigated due to streaming.

I'll go and make a few ZVOLS and see what happens with the overhead when
they start filling up. But it looks like the windows blocksize should
certainly not match zvol blocksize.

> As to your second question, modern versions of windows should align
> partitions correctly, what version are you using? You may wish to open
> an elevated command prompt and sanity check the output of "fsutil fsinfo
> ntfsinfo x:"

I'm just loading the disk in explorer, rightclick it, and then format.
Just as basic as that.
Not sure what to make of it, but this gives:

C:\WINDOWS\system32>fsutil fsinfo ntfsinfo m:
NTFS Volume Serial Number :        0x2aaa50b8aa50826d
NTFS Version   :                   3.1
LFS Version    :                   2.0
Number Sectors :                   0x000000027ffbefff
Total Clusters :                   0x0000000027ffbeff
Free Clusters  :                   0x00000000008497fc
Total Reserved :                   0x00000000000a8ec0
Bytes Per Sector  :                512
Bytes Per Physical Sector :        4096
Bytes Per Cluster :                8192
Bytes Per FileRecord Segment    :  1024
Clusters Per FileRecord Segment :  0
Mft Valid Data Length :            0x0000000000140000
Mft Start Lcn  :                   0x0000000000060000
Mft2 Start Lcn :                   0x0000000000000001
Mft Zone Start :                   0x00000000000600a0
Mft Zone End   :                   0x0000000000066420
Max Device Trim Extent Count :     4294967295
Max Device Trim Byte Count :       0xffffffff
Max Volume Trim Extent Count :     62
Max Volume Trim Byte Count :       0x40000000
Resource Manager Identifier :     FD836E1F-1372-11E5-825F-000272131F3C

The gpart info in the zvol is reported as unaligned when booting
FreeBSD, but that would be because windows put it in there in an
unaligned postition. Something like at 63 * 512 bytes...??

Any commandline incarnation that allows me to put it a a different offset?

> 

> Finally, from my own experience with using iscsi under windows I
> found I had a random 4k iops increase from ~10,000 to ~90,000 (which
> is close to my pools real speed) by adjusting the following registry
> key


> Create this DWORD key with a value of 1:
> HKLM\SYSTEM\CurrentControlSet\Control\Class\{4D36E97B-E325-11CE-BFC1-08002BE10318}\<Instance
> Number>\Parameters\iSCSIDisableNagle

Very usefull hint, thanx for that.

--WjW




Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?5606EBEA.2090103>