From owner-freebsd-stable@freebsd.org Thu Oct 29 16:54:41 2015 Return-Path: Delivered-To: freebsd-stable@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 2C262A202A9 for ; Thu, 29 Oct 2015 16:54:41 +0000 (UTC) (envelope-from smkelly@smkelly.org) Received: from out4-smtp.messagingengine.com (out4-smtp.messagingengine.com [66.111.4.28]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id EF8101CFD for ; Thu, 29 Oct 2015 16:54:40 +0000 (UTC) (envelope-from smkelly@smkelly.org) Received: from compute6.internal (compute6.nyi.internal [10.202.2.46]) by mailout.nyi.internal (Postfix) with ESMTP id 4CA7E208AF for ; Thu, 29 Oct 2015 12:54:39 -0400 (EDT) Received: from frontend1 ([10.202.2.160]) by compute6.internal (MEProxy); Thu, 29 Oct 2015 12:54:39 -0400 DKIM-Signature: v=1; a=rsa-sha1; c=relaxed/relaxed; d=smkelly.org; h= content-type:date:from:message-id:mime-version:subject:to :x-sasl-enc:x-sasl-enc; s=mesmtp; bh=d1dkIn2oevSuCr9FZca2sVS9GJQ =; b=mLpAF1WujJV01TacWosWgmvrdCd4a1xIGVmBWIgxFjwphdYdPGdqY6F23Q0 lLx1SGMg5EpvJ79Fz/8uEUmf2zMO/aDY4/+gx2dwAOkrmoh6ISFRb8mPr/81Oe/J gIbmnPcCTWSpu9Ztw+qROkg2JRPMUYIzcVnt9ASaAJnw0VD4= DKIM-Signature: v=1; a=rsa-sha1; c=relaxed/relaxed; d= messagingengine.com; h=content-type:date:from:message-id :mime-version:subject:to:x-sasl-enc:x-sasl-enc; s=smtpout; bh=d1 dkIn2oevSuCr9FZca2sVS9GJQ=; b=mgH77TEA2nYRCP6em1+eaUbEp8/eruUg+G JChH9+fx6HAoMbNYG8iEFlG6HKbMLx2qaXFv6ULaZKROnLeWzluJT1D3JVxknpA+ peMklX+EiN7ffbtli+wZhauwisH/i99PXCGJrPqehhshMYu8G2cifREhZ3nC/t+7 dQD7Jg+zc= X-Sasl-enc: yfi/stil/bMFiI+4y1SR73VK24w82Q69fo9Ql3ivM+do 1446137678 Received: from tardis.gwp.corp.flightaware.com (unknown [38.100.147.146]) by mail.messagingengine.com (Postfix) with ESMTPA id EA90CC00014 for ; Thu, 29 Oct 2015 12:54:38 -0400 (EDT) From: Sean Kelly Subject: ZFS, SSDs, and TRIM performance Message-Id: <449F8F4D-425D-46B5-BB9C-BE5A0CD11C55@smkelly.org> Date: Thu, 29 Oct 2015 11:54:38 -0500 To: FreeBSD-STABLE Mailing List Mime-Version: 1.0 (Mac OS X Mail 9.2 \(3106\)) X-Mailer: Apple Mail (2.3106) Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable X-Content-Filtered-By: Mailman/MimeDel 2.1.20 X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 29 Oct 2015 16:54:41 -0000 Me again. I have a new issue and I=E2=80=99m not sure if it is hardware = or software. I have nine servers running 10.2-RELEASE-p5 with Dell = OEM=E2=80=99d Samsung XS1715 NVMe SSDs. They are paired up in a single = mirrored zpool on each server. They perform great most of the time. = However, I have a problem when ZFS fires off TRIMs. Not during vdev = creation, but like if I delete a 20GB snapshot. If I destroy a 20GB snapshot or delete large files, ZFS fires off tons = of TRIMs to the disks. I can see the kstat.zfs.misc.zio_trim.success and = kstat.zfs.misc.zio_trim.bytes sysctls skyrocket. While this is = happening, any synchronous writes seem to block. For example, we=E2=80=99r= e running PostgreSQL which does fsync()s all the time. While these TRIMs = happen, Postgres just hangs on writes. This causes reads to block due to = lock contention as well. If I change sync=3Ddisabled on my tank/pgsql dataset while this is = happening, it unblocks for the most part. But obviously this is not an = ideal way to run PostgreSQL. I=E2=80=99m working with my vendor to get some Intel SSDs to test, but = any ideas if this could somehow be a software issue? Or does the Samsung = XS1715 just suck at TRIM and SYNC? We=E2=80=99re thinking of just setting the vfs.zfs.trim.enabled=3D0 = tunable for now since WAL segment turnover actually causes TRIM = operations a lot, but unfortunately this is a reboot. But disabling TRIM = does seem to fix the issue on other servers I=E2=80=99ve tested with the = same hardware config. --=20 Sean Kelly smkelly@smkelly.org http://smkelly.org