From owner-freebsd-stable@freebsd.org Tue Nov 3 09:12:03 2015 Return-Path: Delivered-To: freebsd-stable@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 84FC6A1E2FD for ; Tue, 3 Nov 2015 09:12:03 +0000 (UTC) (envelope-from nicolas.gilles@gmail.com) Received: from mail-vk0-x22a.google.com (mail-vk0-x22a.google.com [IPv6:2607:f8b0:400c:c05::22a]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 3B5801E71 for ; Tue, 3 Nov 2015 09:12:03 +0000 (UTC) (envelope-from nicolas.gilles@gmail.com) Received: by vkex70 with SMTP id x70so5511271vke.3 for ; Tue, 03 Nov 2015 01:12:02 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :content-type:content-transfer-encoding; bh=7Cftp/DnY7mcJwJfYC7vfMxs6e+hEqLn5EILhlQ+CQw=; b=RW1LHm4AQjenjzJcsXm37CaOqAIklPcYo8dacduDDOI3HiBWspb9kYHH/xMPHwxo64 0s52xUNnnepTQ3fF2ZKGmiZayuH515/fBcQsVt8AvmjNHr3+MSoDfqFMKm/6AdY3FOTS jGQBP+cU5mmkDOH/hG5DHMZXdTE1uMFZ0NDKNvCTkJ7vB1b5EDKWCyoZgCHg2qZwWlv7 tu+XpdRxxarPZKSKi2Ffe0zk/pPBzmLjeetQiiULPFY3GqkGMB0HerfRt6B0zPcmh2Sy I5eCxxKlAykKgpgUTBUogSsrtqERkoz9ZHOgGAUqhBm1IYFGRJDGrRHnDAtEte5wBNb1 RstQ== MIME-Version: 1.0 X-Received: by 10.31.3.75 with SMTP id 72mr18255400vkd.3.1446541922319; Tue, 03 Nov 2015 01:12:02 -0800 (PST) Received: by 10.31.106.199 with HTTP; Tue, 3 Nov 2015 01:12:02 -0800 (PST) In-Reply-To: <563263ED.1070402@multiplay.co.uk> References: <449F8F4D-425D-46B5-BB9C-BE5A0CD11C55@smkelly.org> <563263ED.1070402@multiplay.co.uk> Date: Tue, 3 Nov 2015 10:12:02 +0100 Message-ID: Subject: Re: ZFS, SSDs, and TRIM performance From: Nicolas Gilles To: freebsd-stable@freebsd.org Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: quoted-printable X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 03 Nov 2015 09:12:03 -0000 Not sure about the Samsung XS1715, but lots of SSDs seem to suck at large amounts of TRIM in general leading a "let me pause everything for a while" symptom. In fact I think there is work in ZFS to make TRIMs work better, and to throttle them in case large amounts are freed to avoid this kind of starvation. -- Nicolas On Thu, Oct 29, 2015 at 7:22 PM, Steven Hartland wrote: > If you running NVMe, are you running a version which has this: > https://svnweb.freebsd.org/base?view=3Drevision&revision=3D285767 > > I'm pretty sure 10.2 does have that, so you should be good, but best to > check. > > Other questions: > 1. What does "gstat -d -p" show during the stalls? > 2. Do you have any other zfs tuning in place? > > On 29/10/2015 16:54, Sean Kelly wrote: >> >> Me again. I have a new issue and I=E2=80=99m not sure if it is hardware = or >> software. I have nine servers running 10.2-RELEASE-p5 with Dell OEM=E2= =80=99d >> Samsung XS1715 NVMe SSDs. They are paired up in a single mirrored zpool = on >> each server. They perform great most of the time. However, I have a prob= lem >> when ZFS fires off TRIMs. Not during vdev creation, but like if I delete= a >> 20GB snapshot. >> >> If I destroy a 20GB snapshot or delete large files, ZFS fires off tons o= f >> TRIMs to the disks. I can see the kstat.zfs.misc.zio_trim.success and >> kstat.zfs.misc.zio_trim.bytes sysctls skyrocket. While this is happening= , >> any synchronous writes seem to block. For example, we=E2=80=99re running= PostgreSQL >> which does fsync()s all the time. While these TRIMs happen, Postgres jus= t >> hangs on writes. This causes reads to block due to lock contention as we= ll. >> >> If I change sync=3Ddisabled on my tank/pgsql dataset while this is >> happening, it unblocks for the most part. But obviously this is not an i= deal >> way to run PostgreSQL. >> >> I=E2=80=99m working with my vendor to get some Intel SSDs to test, but a= ny ideas >> if this could somehow be a software issue? Or does the Samsung XS1715 ju= st >> suck at TRIM and SYNC? >> >> We=E2=80=99re thinking of just setting the vfs.zfs.trim.enabled=3D0 tuna= ble for now >> since WAL segment turnover actually causes TRIM operations a lot, but >> unfortunately this is a reboot. But disabling TRIM does seem to fix the >> issue on other servers I=E2=80=99ve tested with the same hardware config= . >> > > _______________________________________________ > freebsd-stable@freebsd.org mailing list > https://lists.freebsd.org/mailman/listinfo/freebsd-stable > To unsubscribe, send any mail to "freebsd-stable-unsubscribe@freebsd.org"