From owner-freebsd-stable@freebsd.org Thu Oct 29 18:22:32 2015 Return-Path: Delivered-To: freebsd-stable@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 3D62EA21438 for ; Thu, 29 Oct 2015 18:22:32 +0000 (UTC) (envelope-from killing@multiplay.co.uk) Received: from mail-wm0-x232.google.com (mail-wm0-x232.google.com [IPv6:2a00:1450:400c:c09::232]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id E448D1C4B for ; Thu, 29 Oct 2015 18:22:31 +0000 (UTC) (envelope-from killing@multiplay.co.uk) Received: by wmff134 with SMTP id f134so30000898wmf.0 for ; Thu, 29 Oct 2015 11:22:30 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=multiplay_co_uk.20150623.gappssmtp.com; s=20150623; h=subject:to:references:from:message-id:date:user-agent:mime-version :in-reply-to:content-type:content-transfer-encoding; bh=P5xsC5WiZPkuEqzrjU8Gl/DghjQ2hRRtoGeIkyJIxww=; b=FPRDsA2SQfFE55VMr7yOI/s0DGQoAKMm1rvLxpJL04HAzW+BHAg7YHYdoxza+vQ0PJ DLcoqI5grIwrnNK0Al5KcSE2C5KnYTL1zYLlZCr3UzFYCk0y+F5jAGlrEox7MrmuEP5E FVBh3ByY8d62UL94/i6//M3nENWpc3cWTGzB14nw5lWopX6tZSvGGbmzDZV4sjKG0t/3 amYwxJN4sypReNUgZIzGqSlwAZ/UmhdOGJSp2mu+YmxK2oDJtzAQOo6sDuAs8yjmyHvW 8WVBNdJoHx7OA0nohMPK+ogM9ui5lUHYJtM2MRycbkxm4GZ6h/eo6eiR2gAB1b+Usi5Y hl1Q== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20130820; h=x-gm-message-state:subject:to:references:from:message-id:date :user-agent:mime-version:in-reply-to:content-type :content-transfer-encoding; bh=P5xsC5WiZPkuEqzrjU8Gl/DghjQ2hRRtoGeIkyJIxww=; b=Z8d57kC6b3WBhx0GYHefjHVAqWi/jSHqRaX18Y9md0CMq5HNKrdrKhXwW4qe59Rwb0 W4bnFM5EOvKx31Jdg1QN44raAW5aVtiZiDoUdNA6WI1fmLDuAruYvMpvRm+rKpjFHypP YgLe+bqUuJaFLJ/5VxQ6bMLaQ34GLn+gmDQvi5NBxaO0eHbf3L9ffPVh76MuFl+sIn5r ryUhrLe7y93Btvr5NKlsSDPfL8qyfiMbzQG84oMixuAslzRJZv+84GZYVQexhwN+yakl POL6XeL2DE/WnfpI1Guih+LifeB3gxyfqMIK5Z3whey7o9SiE0akxxpsqKfX5on5U5eG t6kA== X-Gm-Message-State: ALoCoQnkWi844/Gkk71T2kemlqDwK1YNemK1LVhuQiSiAjvVNS8s2hHH4aqkafKgUiuk0COqk7tz X-Received: by 10.28.16.132 with SMTP id 126mr8028300wmq.86.1446142950122; Thu, 29 Oct 2015 11:22:30 -0700 (PDT) Received: from [10.10.1.58] (liv3d.labs.multiplay.co.uk. [82.69.141.171]) by smtp.gmail.com with ESMTPSA id b12sm10399280wma.6.2015.10.29.11.22.29 for (version=TLSv1.2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Thu, 29 Oct 2015 11:22:29 -0700 (PDT) Subject: Re: ZFS, SSDs, and TRIM performance To: freebsd-stable@freebsd.org References: <449F8F4D-425D-46B5-BB9C-BE5A0CD11C55@smkelly.org> From: Steven Hartland Message-ID: <563263ED.1070402@multiplay.co.uk> Date: Thu, 29 Oct 2015 18:22:37 +0000 User-Agent: Mozilla/5.0 (Windows NT 10.0; WOW64; rv:38.0) Gecko/20100101 Thunderbird/38.3.0 MIME-Version: 1.0 In-Reply-To: <449F8F4D-425D-46B5-BB9C-BE5A0CD11C55@smkelly.org> Content-Type: text/plain; charset=utf-8; format=flowed Content-Transfer-Encoding: 8bit X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 29 Oct 2015 18:22:32 -0000 If you running NVMe, are you running a version which has this: https://svnweb.freebsd.org/base?view=revision&revision=285767 I'm pretty sure 10.2 does have that, so you should be good, but best to check. Other questions: 1. What does "gstat -d -p" show during the stalls? 2. Do you have any other zfs tuning in place? On 29/10/2015 16:54, Sean Kelly wrote: > Me again. I have a new issue and I’m not sure if it is hardware or software. I have nine servers running 10.2-RELEASE-p5 with Dell OEM’d Samsung XS1715 NVMe SSDs. They are paired up in a single mirrored zpool on each server. They perform great most of the time. However, I have a problem when ZFS fires off TRIMs. Not during vdev creation, but like if I delete a 20GB snapshot. > > If I destroy a 20GB snapshot or delete large files, ZFS fires off tons of TRIMs to the disks. I can see the kstat.zfs.misc.zio_trim.success and kstat.zfs.misc.zio_trim.bytes sysctls skyrocket. While this is happening, any synchronous writes seem to block. For example, we’re running PostgreSQL which does fsync()s all the time. While these TRIMs happen, Postgres just hangs on writes. This causes reads to block due to lock contention as well. > > If I change sync=disabled on my tank/pgsql dataset while this is happening, it unblocks for the most part. But obviously this is not an ideal way to run PostgreSQL. > > I’m working with my vendor to get some Intel SSDs to test, but any ideas if this could somehow be a software issue? Or does the Samsung XS1715 just suck at TRIM and SYNC? > > We’re thinking of just setting the vfs.zfs.trim.enabled=0 tunable for now since WAL segment turnover actually causes TRIM operations a lot, but unfortunately this is a reboot. But disabling TRIM does seem to fix the issue on other servers I’ve tested with the same hardware config. >