Date: Fri, 22 Dec 2023 10:59:26 +0100 (CET) From: Ronald Klop <ronald-lists@klop.ws> To: freebsd-fs <freebsd-fs@freebsd.org>, void <void@f-m.fm> Subject: Re: measuring swap partition speed Message-ID: <478987228.2760.1703239166835@localhost> In-Reply-To: <1137397689.3080.1703237862787@localhost> References: <D9C150E2-8453-463B-9A23-B0474D0AE390.ref@yahoo.com> <D9C150E2-8453-463B-9A23-B0474D0AE390@yahoo.com> <ZYRe3GFrf1fSIPjk@int21h> <1439231787.9333.1703184981914@localhost> <1dbf8804-f1ac-4578-a538-889744d7de9a@app.fastmail.com> <1137397689.3080.1703237862787@localhost>
next in thread | previous in thread | raw e-mail | index | archive | help
------=_Part_2759_327864946.1703239166820
Content-Type: text/plain; charset=us-ascii; format=flowed
Content-Transfer-Encoding: 7bit
Van: Ronald Klop <ronald-lists@klop.ws>
Datum: vrijdag, 22 december 2023 10:37
Aan: void <void@f-m.fm>
CC: freebsd-fs <freebsd-fs@freebsd.org>
Onderwerp: Re: measuring swap partition speed
>
>
> Van: void <void@f-m.fm>
> Datum: vrijdag, 22 december 2023 01:05
> Aan: freebsd-fs <freebsd-fs@freebsd.org>
> Onderwerp: Re: measuring swap partition speed
>>
>> On Thu, 21 Dec 2023, at 18:56, Ronald Klop wrote:
>> > A bit weird that a simple dd is so slow.
>> >
>> > Just a quick thought.
>> > Are your partitions aligned properly?
>>
>> no idea. How would I check? The system when installed,
>> the auto-zfs option was selected, block size set to 4k. The swap
>> was changed from the default 2gb to 12gb. geli encryption is
>> active for the zfs filesystem but not for swap. It's only the swap
>> that shows performance issues.
>>
>> > Is other IO going on on the same time?
>>
>> not really, the tests were done with cron jobs deactivated, almost completely idle,
>> swap was not being used at all.
>>
>> > What does gstat say about %util, queue length and all the other stats
>> > while running the dd? Or "iostat -x -d 1".
>>
>> I'll try that next, thanks
>>
>> > Could you try if another disk has the same issues?
>>
>> no spare or equivalent disk for this arch.
>>
>> > How is your disk connected? USB-to-SATA-adapter? Any output of dmesg?
>>
>> Regular usb3 connection.
>>
>> da0: 400.000MB/s transfers
>> da0: 953869MB (1953525168 512 byte sectors)
>> da0: quirks=0x2<NO_6_BYTE>
>> da0: Delete methods: <NONE(*),ZERO>
>> --
>>
>>
>>
>>
>
>
> Can you provide more concrete information? Like:
>
> # usbconfig list
> ugen0.1: <(0x1106) XHCI root HUB> at usbus0, cfg=0 md=HOST spd=SUPER (5.0Gbps) pwr=SAVE (0mA)
> ugen0.2: <vendor 0x2109 USB2.0 Hub> at usbus0, cfg=0 md=HOST spd=HIGH (480Mbps) pwr=SAVE (100mA)
> ugen0.3: <Silicon Labs CP2102 USB to UART Bridge Controller> at usbus0, cfg=0 md=HOST spd=FULL (12Mbps) pwr=ON (100mA)
> ugen0.4: <USB 3.0 Device USB 3.0 Device> at usbus0, cfg=0 md=HOST spd=SUPER (5.0Gbps) pwr=ON (224mA)
> ugen0.5: <Western Digital Elements 25A2> at usbus0, cfg=0 md=HOST spd=SUPER (5.0Gbps) pwr=ON (224mA)
>
> And the output of "devinfo -v". That will give people much more information to work with instead of "anecdotal evidence" about your hardware being usb3.
> And the complete output of dmesg instead of only the "da0:" lines. There is more information in there which can be valuable for people to help you.
>
> About the alignment. You wrote in your first mail:
> "# gpart show
>
> => 40 1953525088 da0 GPT (932G)
> 40 532480 1 efi (260M)
> 532520 2008 - free - (1.0M)
> 534528 4194304 2 freebsd-swap (2.0G)
> 4728832 4194304 4 freebsd-swap (2.0G)
> 8923136 4194304 5 freebsd-swap (2.0G)
> 13117440 4194304 6 freebsd-swap (2.0G)
> 17311744 4194304 7 freebsd-swap (2.0G)
> 21506048 4194304 8 freebsd-swap (2.0G)
> 25700352 1927823360 3 freebsd-zfs (920G)
> 1953523712 1416 - free - (708K)
> "
>
> Your dd tests were on da0p4 which starts on block 4728832.
> Assuming "gpart list" shows "Sectorsize: 512".
> (4728832 * 512) mod 4096 = 0, so the partitiion is aligned on a 4KB page size. And the calculation is also 0 for 8KB (is this the page size of arm64?) which is good.
> AFAIK: this is properly aligned such that the hardware does not need to double write or read-before-write when paging out.
>
> Up until now I haven't seen anything which would explain the slow dd speed on of=/dev/da0p4 vs the quick dd on of=/mnt/test8k.bin.
>
> Regards,
> Ronald.
>
Just did the same test on my RPI4 running FreeBSD 15.0-CURRENT of Dec 14.
Used an unused partition on spinning disk. My used swap is on an ssd nowadays.
[root@rpi4 ~]# dd if=/dev/urandom of=/dev/da1p2 bs=8k count=25000 conv=sync status=progress
193200128 bytes (193 MB, 184 MiB) transferred 10.024s, 19 MB/s
25000+0 records in
25000+0 records out
204800000 bytes transferred in 10.592984 secs (19333551 bytes/sec)
NB: using bs=128k makes it go to 69MB/s.
# gpart show
=> 40 1953458096 da1 GPT (931G)
40 102400 1 efi (50M)
102440 8388608 2 freebsd-swap (4.0G)
8491048 1944967088 3 freebsd-zfs (927G)
ugen0.5: <Western Digital Elements 25A2> at usbus0
umass1 on uhub0
umass1: <Western Digital Elements 25A2, class 0/0, rev 3.00/10.04, addr 4> on usbus0
da1 at umass-sim1 bus 1 scbus1 target 0 lun 0
da1: <WD Elements 25A2 1004> Fixed Direct Access SPC-4 SCSI device
da1: Serial Number 575843314538354834414536
da1: 400.000MB/s transfers
da1: 953837MB (1953458176 512 byte sectors)
da1: quirks=0x2<NO_6_BYTE>
# usbconfig list
ugen0.1: <(0x1106) XHCI root HUB> at usbus0, cfg=0 md=HOST spd=SUPER (5.0Gbps) pwr=SAVE (0mA)
ugen0.2: <vendor 0x2109 USB2.0 Hub> at usbus0, cfg=0 md=HOST spd=HIGH (480Mbps) pwr=SAVE (100mA)
ugen0.3: <Silicon Labs CP2102 USB to UART Bridge Controller> at usbus0, cfg=0 md=HOST spd=FULL (12Mbps) pwr=ON (100mA)
ugen0.4: <USB 3.0 Device USB 3.0 Device> at usbus0, cfg=0 md=HOST spd=SUPER (5.0Gbps) pwr=ON (224mA)
ugen0.5: <Western Digital Elements 25A2> at usbus0, cfg=0 md=HOST spd=SUPER (5.0Gbps) pwr=ON (224mA)
Found some more things to check:
Do these settings have something non-default?
# sysctl kern.geom.disk
# sysctl kern.cam
And your disk prints this in dmesg. How does that influence things? I haven't seen this before anywhere.
da0: Delete methods: <NONE(*),ZERO>
Because of this I'm particularly interested in:
sysctl kern.cam.da.0.delete_method
and
sysctl kern.cam.da.0.trim_count
I might be totally of in my thinking here, but you never know with such unpredictable problems.
Regards,
Ronald.
------=_Part_2759_327864946.1703239166820
Content-Type: text/html; charset=us-ascii
Content-Transfer-Encoding: 7bit
<html><head></head><body><br>
<p><strong>Van:</strong> Ronald Klop <ronald-lists@klop.ws><br>
<strong>Datum:</strong> vrijdag, 22 december 2023 10:37<br>
<strong>Aan:</strong> void <void@f-m.fm><br>
<strong>CC:</strong> freebsd-fs <freebsd-fs@freebsd.org><br>
<strong>Onderwerp:</strong> Re: measuring swap partition speed</p>
<blockquote style="padding-right: 0px; padding-left: 5px; margin-left: 5px; border-left: #000000 2px solid; margin-right: 0px">
<div class="MessageRFC822Viewer" id="P">
<div class="MultipartAlternativeViewer">
<div class="TextHTMLViewer" id="P.P.P">
<p><strong>Van:</strong> void <void@f-m.fm><br>
<strong>Datum:</strong> vrijdag, 22 december 2023 01:05<br>
<strong>Aan:</strong> freebsd-fs <freebsd-fs@freebsd.org><br>
<strong>Onderwerp:</strong> Re: measuring swap partition speed</p>
<blockquote style="padding-right: 0px; padding-left: 5px; margin-left: 5px; border-left: #000000 2px solid; margin-right: 0px">
<div class="MessageRFC822Viewer" id="P">
<div class="TextPlainViewer" id="P.P">On Thu, 21 Dec 2023, at 18:56, Ronald Klop wrote:<br>
> A bit weird that a simple dd is so slow.<br>
><br>
> Just a quick thought.<br>
> Are your partitions aligned properly?<br>
<br>
no idea. How would I check? The system when installed,<br>
the auto-zfs option was selected, block size set to 4k. The swap<br>
was changed from the default 2gb to 12gb. geli encryption is<br>
active for the zfs filesystem but not for swap. It's only the swap<br>
that shows performance issues.<br>
<br>
> Is other IO going on on the same time?<br>
<br>
not really, the tests were done with cron jobs deactivated, almost completely idle,<br>
swap was not being used at all.<br>
<br>
> What does gstat say about %util, queue length and all the other stats<br>
> while running the dd? Or "iostat -x -d 1".<br>
<br>
I'll try that next, thanks<br>
<br>
> Could you try if another disk has the same issues?<br>
<br>
no spare or equivalent disk for this arch.<br>
<br>
> How is your disk connected? USB-to-SATA-adapter? Any output of dmesg?<br>
<br>
Regular usb3 connection.<br>
<br>
da0: 400.000MB/s transfers<br>
da0: 953869MB (1953525168 512 byte sectors)<br>
da0: quirks=0x2<NO_6_BYTE><br>
da0: Delete methods: <NONE(*),ZERO><br>
-- <br>
</div>
<hr></div>
</blockquote>
<br>
<br>
Can you provide more concrete information? Like:<br>
<br>
# usbconfig list<br>
ugen0.1: <(0x1106) XHCI root HUB> at usbus0, cfg=0 md=HOST spd=SUPER (5.0Gbps) pwr=SAVE (0mA)<br>
ugen0.2: <vendor 0x2109 USB2.0 Hub> at usbus0, cfg=0 md=HOST spd=HIGH (480Mbps) pwr=SAVE (100mA)<br>
ugen0.3: <Silicon Labs CP2102 USB to UART Bridge Controller> at usbus0, cfg=0 md=HOST spd=FULL (12Mbps) pwr=ON (100mA)<br>
ugen0.4: <USB 3.0 Device USB 3.0 Device> at usbus0, cfg=0 md=HOST spd=SUPER (5.0Gbps) pwr=ON (224mA)<br>
ugen0.5: <Western Digital Elements 25A2> at usbus0, cfg=0 md=HOST spd=SUPER (5.0Gbps) pwr=ON (224mA)<br>
<br>
And the output of "devinfo -v". That will give people much more information to work with instead of "anecdotal evidence" about your hardware being usb3.<br>
And the complete output of dmesg instead of only the "da0:" lines. There is more information in there which can be valuable for people to help you.<br>
<br>
About the alignment. You wrote in your first mail:<br>
"# gpart show<br>
<br>
=> 40 1953525088 da0 GPT (932G)<br>
40 532480 1 efi (260M)<br>
532520 2008 - free - (1.0M)<br>
534528 4194304 2 freebsd-swap (2.0G)<br>
4728832 4194304 4 freebsd-swap (2.0G)<br>
8923136 4194304 5 freebsd-swap (2.0G)<br>
13117440 4194304 6 freebsd-swap (2.0G)<br>
17311744 4194304 7 freebsd-swap (2.0G)<br>
21506048 4194304 8 freebsd-swap (2.0G)<br>
25700352 1927823360 3 freebsd-zfs (920G)<br>
1953523712 1416 - free - (708K)<br>
"<br>
<br>
Your dd tests were on da0p4 which starts on block 4728832.<br>
Assuming "gpart list" shows "Sectorsize: 512".<br>
(4728832 * 512) mod 4096 = 0, so the partitiion is aligned on a 4KB page size. And the calculation is also 0 for 8KB (is this the page size of arm64?) which is good.<br>
AFAIK: this is properly aligned such that the hardware does not need to double write or read-before-write when paging out.<br>
<br>
Up until now I haven't seen anything which would explain the slow dd speed on of=/dev/da0p4 vs the quick dd on of=/mnt/test8k.bin.<br>
<br>
Regards,<br>
Ronald.<br>
</div>
</div>
</div>
</blockquote>
<br>
Just did the same test on my RPI4 running FreeBSD 15.0-CURRENT of Dec 14.<br>
Used an unused partition on spinning disk. My used swap is on an ssd nowadays.<br>
<br>
[root@rpi4 ~]# dd if=/dev/urandom of=/dev/da1p2 bs=8k count=25000 conv=sync status=progress<br>
193200128 bytes (193 MB, 184 MiB) transferred 10.024s, 19 MB/s<br>
25000+0 records in<br>
25000+0 records out<br>
204800000 bytes transferred in 10.592984 secs (19333551 bytes/sec)<br>
<br>
NB: using bs=128k makes it go to 69MB/s.<br>
<br>
# gpart show<br>
=> 40 1953458096 da1 GPT (931G)<br>
40 102400 1 efi (50M)<br>
102440 8388608 2 freebsd-swap (4.0G)<br>
8491048 1944967088 3 freebsd-zfs (927G)<br>
<br>
ugen0.5: <Western Digital Elements 25A2> at usbus0<br>
umass1 on uhub0<br>
umass1: <Western Digital Elements 25A2, class 0/0, rev 3.00/10.04, addr 4> on usbus0<br>
da1 at umass-sim1 bus 1 scbus1 target 0 lun 0<br>
da1: <WD Elements 25A2 1004> Fixed Direct Access SPC-4 SCSI device<br>
da1: Serial Number 575843314538354834414536<br>
da1: 400.000MB/s transfers<br>
da1: 953837MB (1953458176 512 byte sectors)<br>
da1: quirks=0x2<NO_6_BYTE><br>
<br>
# usbconfig list<br>
ugen0.1: <(0x1106) XHCI root HUB> at usbus0, cfg=0 md=HOST spd=SUPER (5.0Gbps) pwr=SAVE (0mA)<br>
ugen0.2: <vendor 0x2109 USB2.0 Hub> at usbus0, cfg=0 md=HOST spd=HIGH (480Mbps) pwr=SAVE (100mA)<br>
ugen0.3: <Silicon Labs CP2102 USB to UART Bridge Controller> at usbus0, cfg=0 md=HOST spd=FULL (12Mbps) pwr=ON (100mA)<br>
ugen0.4: <USB 3.0 Device USB 3.0 Device> at usbus0, cfg=0 md=HOST spd=SUPER (5.0Gbps) pwr=ON (224mA)<br>
ugen0.5: <Western Digital Elements 25A2> at usbus0, cfg=0 md=HOST spd=SUPER (5.0Gbps) pwr=ON (224mA)<br>
<br>
<br>
Found some more things to check:<br>
<br>
Do these settings have something non-default?<br>
# sysctl kern.geom.disk<br>
# sysctl kern.cam<br>
<br>
And your disk prints this in dmesg. How does that influence things? I haven't seen this before anywhere.<br>
da0: Delete methods: <NONE(*),ZERO><br>
<br>
Because of this I'm particularly interested in:<br>
sysctl kern.cam.da.0.delete_method<br>
and<br>
sysctl kern.cam.da.0.trim_count<br>
<br>
I might be totally of in my thinking here, but you never know with such unpredictable problems.<br>
<br>
Regards,<br>
Ronald.<br>
</body></html>
------=_Part_2759_327864946.1703239166820--
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?478987228.2760.1703239166835>