From owner-freebsd-stable@freebsd.org Tue Nov 3 09:57:25 2015 Return-Path: Delivered-To: freebsd-stable@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id E38C1A23262 for ; Tue, 3 Nov 2015 09:57:25 +0000 (UTC) (envelope-from killing@multiplay.co.uk) Received: from mail-wi0-x22d.google.com (mail-wi0-x22d.google.com [IPv6:2a00:1450:400c:c05::22d]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 76168175A for ; Tue, 3 Nov 2015 09:57:24 +0000 (UTC) (envelope-from killing@multiplay.co.uk) Received: by wicll6 with SMTP id ll6so7781655wic.1 for ; Tue, 03 Nov 2015 01:57:23 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=multiplay_co_uk.20150623.gappssmtp.com; s=20150623; h=subject:to:references:from:message-id:date:user-agent:mime-version :in-reply-to:content-type:content-transfer-encoding; bh=ZYUoe6kMa+qCyNVlT8LBkZPAUGgwAq5Iv/PYM0k8FQw=; b=st8d4nbxl/JOE9mXhKUdU/WsxwxIxCxfbpXzui3rR8a/NIbQxLnVWWw5XFyLJkVHTj YjqjGxMxuNTGxxpE1aLu+dmURm2FoYxeea412WJ4iV/LrtyMtqRh4O+b5M/DaHVTko3I IntPJ6zyukCx5n8iHAD8kMLH+zQSSi8dAJ8aVjGzmp8M5rzK1dVjk72iBNgv3PBtSHhM 7JVAAzO5IXa2sMRLqun6jYovyifUKyPQhoh+r5W5U+uH7JFvYNSsdHXkF4411Oq9fpfE 41tDFOivMhqQriJsnBFO1fRjA6m5zEF0oJYFwe9zPzqXW3DTmlAWA1LsLXvvCadyoY8m EttA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20130820; h=x-gm-message-state:subject:to:references:from:message-id:date :user-agent:mime-version:in-reply-to:content-type :content-transfer-encoding; bh=ZYUoe6kMa+qCyNVlT8LBkZPAUGgwAq5Iv/PYM0k8FQw=; b=g3wNZUQHMfYOvSrv82iIqV6IjKYFVZrrHlcXEvdZ6/SWDaabKPfMEEzcaCCugdlcNR XYIROdSZLy4OVtDazkH7XmvBAGSxIfYjBoxvbHFTcc4o3/jWGT3n9u0gl4zC9xaTi65h 0vimmUqrAD/ca1iVweSQjfamnRjOfMSfGvvFZfuyn9RgrxiVjPdum3+VxX6/uKre47EU QZJ8f1j9hGs18JRx1vXk/qD14rJNBnCPlBN3VqnTcvn/SYoVAmI5Xu7DmOF5kJ47Rth4 R6AjgAEIcZdODUYK96/QP84OyXnh91P34+YBmAhn4iqjFym0XRQM542c4G5Qz35/jMhG EHmw== X-Gm-Message-State: ALoCoQl6WMLa8pJYx9xKZj9XnX17fqUPGsLwQDnn1+wwvFU5Cl6FAC+okHbkHYXicSsaupZlEZIp X-Received: by 10.194.142.166 with SMTP id rx6mr28418171wjb.54.1446544643123; Tue, 03 Nov 2015 01:57:23 -0800 (PST) Received: from [10.10.1.58] (liv3d.labs.multiplay.co.uk. [82.69.141.171]) by smtp.gmail.com with ESMTPSA id 199sm22420556wml.22.2015.11.03.01.57.21 for (version=TLSv1.2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Tue, 03 Nov 2015 01:57:21 -0800 (PST) Subject: Re: ZFS, SSDs, and TRIM performance To: freebsd-stable@freebsd.org References: <449F8F4D-425D-46B5-BB9C-BE5A0CD11C55@smkelly.org> <563263ED.1070402@multiplay.co.uk> From: Steven Hartland Message-ID: <56388502.5080809@multiplay.co.uk> Date: Tue, 3 Nov 2015 09:57:22 +0000 User-Agent: Mozilla/5.0 (Windows NT 10.0; WOW64; rv:38.0) Gecko/20100101 Thunderbird/38.3.0 MIME-Version: 1.0 In-Reply-To: Content-Type: text/plain; charset=utf-8; format=flowed Content-Transfer-Encoding: 8bit X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 03 Nov 2015 09:57:26 -0000 This is something we've already done in FreeBSD, both myself and a others have iterated a few times on this very thing. There's currently nothing outstanding that I'm aware so its important to capture the details as people experience them to see if there is any more work to do in this area. On 03/11/2015 09:12, Nicolas Gilles wrote: > Not sure about the Samsung XS1715, but lots of SSDs seem to suck at > large amounts of TRIM in general leading a "let me pause everything > for a while" symptom. In fact I think there is work in ZFS to make > TRIMs work better, and to throttle them in case large amounts are > freed to avoid this kind of starvation. > > -- Nicolas > > > On Thu, Oct 29, 2015 at 7:22 PM, Steven Hartland > wrote: >> If you running NVMe, are you running a version which has this: >> https://svnweb.freebsd.org/base?view=revision&revision=285767 >> >> I'm pretty sure 10.2 does have that, so you should be good, but best to >> check. >> >> Other questions: >> 1. What does "gstat -d -p" show during the stalls? >> 2. Do you have any other zfs tuning in place? >> >> On 29/10/2015 16:54, Sean Kelly wrote: >>> Me again. I have a new issue and I’m not sure if it is hardware or >>> software. I have nine servers running 10.2-RELEASE-p5 with Dell OEM’d >>> Samsung XS1715 NVMe SSDs. They are paired up in a single mirrored zpool on >>> each server. They perform great most of the time. However, I have a problem >>> when ZFS fires off TRIMs. Not during vdev creation, but like if I delete a >>> 20GB snapshot. >>> >>> If I destroy a 20GB snapshot or delete large files, ZFS fires off tons of >>> TRIMs to the disks. I can see the kstat.zfs.misc.zio_trim.success and >>> kstat.zfs.misc.zio_trim.bytes sysctls skyrocket. While this is happening, >>> any synchronous writes seem to block. For example, we’re running PostgreSQL >>> which does fsync()s all the time. While these TRIMs happen, Postgres just >>> hangs on writes. This causes reads to block due to lock contention as well. >>> >>> If I change sync=disabled on my tank/pgsql dataset while this is >>> happening, it unblocks for the most part. But obviously this is not an ideal >>> way to run PostgreSQL. >>> >>> I’m working with my vendor to get some Intel SSDs to test, but any ideas >>> if this could somehow be a software issue? Or does the Samsung XS1715 just >>> suck at TRIM and SYNC? >>> >>> We’re thinking of just setting the vfs.zfs.trim.enabled=0 tunable for now >>> since WAL segment turnover actually causes TRIM operations a lot, but >>> unfortunately this is a reboot. But disabling TRIM does seem to fix the >>> issue on other servers I’ve tested with the same hardware config. >>> >> _______________________________________________ >> freebsd-stable@freebsd.org mailing list >> https://lists.freebsd.org/mailman/listinfo/freebsd-stable >> To unsubscribe, send any mail to "freebsd-stable-unsubscribe@freebsd.org" > _______________________________________________ > freebsd-stable@freebsd.org mailing list > https://lists.freebsd.org/mailman/listinfo/freebsd-stable > To unsubscribe, send any mail to "freebsd-stable-unsubscribe@freebsd.org"