From owner-freebsd-stable@freebsd.org Tue May 17 09:09:08 2016 Return-Path: Delivered-To: freebsd-stable@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 1D600B3DC13 for ; Tue, 17 May 2016 09:09:08 +0000 (UTC) (envelope-from killing@multiplay.co.uk) Received: from mail-wm0-x236.google.com (mail-wm0-x236.google.com [IPv6:2a00:1450:400c:c09::236]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id C98881493 for ; Tue, 17 May 2016 09:09:07 +0000 (UTC) (envelope-from killing@multiplay.co.uk) Received: by mail-wm0-x236.google.com with SMTP id g17so19030088wme.1 for ; Tue, 17 May 2016 02:09:07 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=multiplay-co-uk.20150623.gappssmtp.com; s=20150623; h=subject:to:references:from:message-id:date:user-agent:mime-version :in-reply-to; bh=bxxIbhmNWdVotwqh2sQtBm5XbFeJzTvm0o8zriI91FY=; b=ZOuEbf0uHyfkNibh5v8OJQstSpNkw+Fuw0iK1Vbe9EpUyoUvB7ykBrqRj5BBrlHWXW dOEDHLg1S23MYzBnOzRxT1ONPcoBiZ5dxJchlUKCLHJszIUbFlUovu9C38E7yFcs7+PJ TLzqnazH9FAKVJnS92m/ytaOMgRqD6l8j9UrVHZOdaBWwJvOWgn1fUJNv9DDghcihDub DX1LX0x/932cZV9zKgu9zy5qDjn5anD+gy58z0PmWEpkdfYAFUAqtH6DUTUUa09snPTp UKZmuTHGqn1B7DRyKzIkoOyuhFKv9boCtv2OtsdQ3rdSRRdU4RGy3febh+xk3eem7E1B tqpQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20130820; h=x-gm-message-state:subject:to:references:from:message-id:date :user-agent:mime-version:in-reply-to; bh=bxxIbhmNWdVotwqh2sQtBm5XbFeJzTvm0o8zriI91FY=; b=QXFXVqthiPwk24S9TcbYY9gMx1DTZl4TDKxaIe0eFVxw6r5U6fG42nTHT4R5CXtP3l tqPtHkBdSJk9bO4QEA55hFn4lvZSMzTOAyo/x0QUglwrGi/aRzCyeqI/eW78XCmyy7K/ pzF/X1EAsOJImMOKxTSV2XjH4s6nz4pbDwMZpWyomnP3sdpXHCVm2I/AjbKgehQ83xxI bITgJeBumvFBfZKrYH7+7RZrrGplbI4qgoH+CDNTnA18Helw0lwJ3DdMKQsreLqOx7aq XgsguuwOU6kBczJEUcfpTtqjAFBn2ivscMQSyXedV+Nvvn2mK/JBGoL2Xw1J4LjCev7/ 2e0w== X-Gm-Message-State: AOPr4FUC6fhHBUJlsb4Lk2qqzi0axXr249VTjKa5cJIftiwKJUjsu4hEDaX+C+z+B1z92luI X-Received: by 10.194.6.65 with SMTP id y1mr251723wjy.12.1463476145726; Tue, 17 May 2016 02:09:05 -0700 (PDT) Received: from [10.10.1.58] (liv3d.labs.multiplay.co.uk. [82.69.141.171]) by smtp.gmail.com with ESMTPSA id v143sm22760309wmv.4.2016.05.17.02.09.04 for (version=TLSv1/SSLv3 cipher=OTHER); Tue, 17 May 2016 02:09:04 -0700 (PDT) Subject: Re: ZFS and NVMe, trim caused stalling To: freebsd-stable@freebsd.org References: <5E710EA5-C9B0-4521-85F1-3FE87555B0AF@bsdimp.com> From: Steven Hartland Message-ID: <20a155fd-8695-ca42-6a72-32cb78864a22@multiplay.co.uk> Date: Tue, 17 May 2016 10:09:10 +0100 User-Agent: Mozilla/5.0 (Windows NT 10.0; WOW64; rv:45.0) Gecko/20100101 Thunderbird/45.1.0 MIME-Version: 1.0 In-Reply-To: Content-Type: text/plain; charset=utf-8; format=flowed Content-Transfer-Encoding: 8bit X-Content-Filtered-By: Mailman/MimeDel 2.1.22 X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.22 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 17 May 2016 09:09:08 -0000 On 17/05/2016 08:49, Borja Marcos wrote: >> On 05 May 2016, at 16:39, Warner Losh wrote: >> >>> What do you think? In some cases it’s clear that TRIM can do more harm than good. >> I think it’s best we not overreact. > I agree. But with this issue the system is almost unusable for now. > >> This particular case is cause by the nvd driver, not the Intel P3500 NVME drive. You need >> a solution (3): Fix the driver. >> >> Specifically, ZFS is pushing down a boatload of BIO_DELETE requests. In ata/da land, these >> requests are queued up, then collapsed together as much as makes sense (or is possible). >> This vastly helps performance (even with the extra sorting that I forced to be in there that I >> need to fix before 11). The nvd driver needs to do the same thing. > I understand that, but I don’t think it’s a good that ZFS depends blindly on a driver feature such > as that. Of course, it’s great to exploit it. > > I have also noticed that ZFS has a good throttling mechanism for write operations. A similar > mechanism should throttle trim requests so that trim requests don’t clog the whole system. It already does. > >> I’d be extremely hesitant to tossing away TRIMs. They are actually quite important for >> the FTL in the drive’s firmware to proper manage the NAND wear. More free space always >> reduces write amplification. It tends to go as 1 / freespace, so simply dropping them on >> the floor should be done with great reluctance. > I understand. I was wondering about choosing the lesser between two evils. A 15 minute > I/O stall (I deleted 2 TB of data, that’s a lot, but not so unrealistic) or settings trims aside > during the peak activity. > > I see that I was wrong on that, as a throttling mechanism would be more than enough probably, > unless the system is close to running out of space. > > I’ve filed a bug report anyway. And copying to -stable. > > > https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=209571 > TBH it sounds like you may have badly behaved HW, we've used ZFS + TRIM and for years on large production boxes and while we're seen slow down we haven't experienced the total lockups you're describing. The graphs on you're ticket seem to indicate peak throughput of 250MB/s which is extremely slow for standard SSD's let alone NVMe ones and when you add in the fact you have 10 well it seems like something is VERY wrong. I just did a quick test on our DB box here creating and then deleting a 2G file as you describe and I couldn't even spot the delete in the general noise it was so quick to process and that's a 6 disk machine with P3700's. Regards Steve