Skip site navigation (1)Skip section navigation (2)
Date:      Thu, 29 Oct 2015 11:54:38 -0500
From:      Sean Kelly <smkelly@smkelly.org>
To:        FreeBSD-STABLE Mailing List <freebsd-stable@freebsd.org>
Subject:   ZFS, SSDs, and TRIM performance
Message-ID:  <449F8F4D-425D-46B5-BB9C-BE5A0CD11C55@smkelly.org>

next in thread | raw e-mail | index | archive | help
Me again. I have a new issue and I=E2=80=99m not sure if it is hardware =
or software. I have nine servers running 10.2-RELEASE-p5 with Dell =
OEM=E2=80=99d Samsung XS1715 NVMe SSDs. They are paired up in a single =
mirrored zpool on each server. They perform great most of the time. =
However, I have a problem when ZFS fires off TRIMs. Not during vdev =
creation, but like if I delete a 20GB snapshot.

If I destroy a 20GB snapshot or delete large files, ZFS fires off tons =
of TRIMs to the disks. I can see the kstat.zfs.misc.zio_trim.success and =
kstat.zfs.misc.zio_trim.bytes sysctls skyrocket. While this is =
happening, any synchronous writes seem to block. For example, we=E2=80=99r=
e running PostgreSQL which does fsync()s all the time. While these TRIMs =
happen, Postgres just hangs on writes. This causes reads to block due to =
lock contention as well.

If I change sync=3Ddisabled on my tank/pgsql dataset while this is =
happening, it unblocks for the most part. But obviously this is not an =
ideal way to run PostgreSQL.

I=E2=80=99m working with my vendor to get some Intel SSDs to test, but =
any ideas if this could somehow be a software issue? Or does the Samsung =
XS1715 just suck at TRIM and SYNC?

We=E2=80=99re thinking of just setting the vfs.zfs.trim.enabled=3D0 =
tunable for now since WAL segment turnover actually causes TRIM =
operations a lot, but unfortunately this is a reboot. But disabling TRIM =
does seem to fix the issue on other servers I=E2=80=99ve tested with the =
same hardware config.

--=20
Sean Kelly
smkelly@smkelly.org
http://smkelly.org




Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?449F8F4D-425D-46B5-BB9C-BE5A0CD11C55>