Skip site navigation (1)Skip section navigation (2)
Date:      Tue, 3 Nov 2015 10:12:02 +0100
From:      Nicolas Gilles <nicolas.gilles@gmail.com>
To:        freebsd-stable@freebsd.org
Subject:   Re: ZFS, SSDs, and TRIM performance
Message-ID:  <CANwv7WuiDapMGfsjKHDY-ZHngKN%2BoMsxVf9HLhXyQAT%2BV_gYqQ@mail.gmail.com>
In-Reply-To: <563263ED.1070402@multiplay.co.uk>
References:  <449F8F4D-425D-46B5-BB9C-BE5A0CD11C55@smkelly.org> <563263ED.1070402@multiplay.co.uk>

next in thread | previous in thread | raw e-mail | index | archive | help
Not sure about the Samsung XS1715, but lots of SSDs seem to suck at
large amounts of TRIM in general leading a "let me pause everything
for a while" symptom. In fact I think there is work in ZFS to make
TRIMs work better, and to throttle them in case large amounts are
freed to avoid this kind of starvation.

-- Nicolas


On Thu, Oct 29, 2015 at 7:22 PM, Steven Hartland
<killing@multiplay.co.uk> wrote:
> If you running NVMe, are you running a version which has this:
> https://svnweb.freebsd.org/base?view=3Drevision&revision=3D285767
>
> I'm pretty sure 10.2 does have that, so you should be good, but best to
> check.
>
> Other questions:
> 1. What does "gstat -d -p" show during the stalls?
> 2. Do you have any other zfs tuning in place?
>
> On 29/10/2015 16:54, Sean Kelly wrote:
>>
>> Me again. I have a new issue and I=E2=80=99m not sure if it is hardware =
or
>> software. I have nine servers running 10.2-RELEASE-p5 with Dell OEM=E2=
=80=99d
>> Samsung XS1715 NVMe SSDs. They are paired up in a single mirrored zpool =
on
>> each server. They perform great most of the time. However, I have a prob=
lem
>> when ZFS fires off TRIMs. Not during vdev creation, but like if I delete=
 a
>> 20GB snapshot.
>>
>> If I destroy a 20GB snapshot or delete large files, ZFS fires off tons o=
f
>> TRIMs to the disks. I can see the kstat.zfs.misc.zio_trim.success and
>> kstat.zfs.misc.zio_trim.bytes sysctls skyrocket. While this is happening=
,
>> any synchronous writes seem to block. For example, we=E2=80=99re running=
 PostgreSQL
>> which does fsync()s all the time. While these TRIMs happen, Postgres jus=
t
>> hangs on writes. This causes reads to block due to lock contention as we=
ll.
>>
>> If I change sync=3Ddisabled on my tank/pgsql dataset while this is
>> happening, it unblocks for the most part. But obviously this is not an i=
deal
>> way to run PostgreSQL.
>>
>> I=E2=80=99m working with my vendor to get some Intel SSDs to test, but a=
ny ideas
>> if this could somehow be a software issue? Or does the Samsung XS1715 ju=
st
>> suck at TRIM and SYNC?
>>
>> We=E2=80=99re thinking of just setting the vfs.zfs.trim.enabled=3D0 tuna=
ble for now
>> since WAL segment turnover actually causes TRIM operations a lot, but
>> unfortunately this is a reboot. But disabling TRIM does seem to fix the
>> issue on other servers I=E2=80=99ve tested with the same hardware config=
.
>>
>
> _______________________________________________
> freebsd-stable@freebsd.org mailing list
> https://lists.freebsd.org/mailman/listinfo/freebsd-stable
> To unsubscribe, send any mail to "freebsd-stable-unsubscribe@freebsd.org"



Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?CANwv7WuiDapMGfsjKHDY-ZHngKN%2BoMsxVf9HLhXyQAT%2BV_gYqQ>