From owner-freebsd-fs@FreeBSD.ORG Mon Apr 29 10:59:17 2013 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.FreeBSD.org [8.8.178.115]) by hub.freebsd.org (Postfix) with ESMTP id F20E2D9E for ; Mon, 29 Apr 2013 10:59:17 +0000 (UTC) (envelope-from jdc@koitsu.org) Received: from qmta12.emeryville.ca.mail.comcast.net (qmta12.emeryville.ca.mail.comcast.net [IPv6:2001:558:fe2d:44:76:96:27:227]) by mx1.freebsd.org (Postfix) with ESMTP id D693C1829 for ; Mon, 29 Apr 2013 10:59:17 +0000 (UTC) Received: from omta13.emeryville.ca.mail.comcast.net ([76.96.30.52]) by qmta12.emeryville.ca.mail.comcast.net with comcast id Vmh21l00417UAYkACmzHBR; Mon, 29 Apr 2013 10:59:17 +0000 Received: from koitsu.strangled.net ([67.180.84.87]) by omta13.emeryville.ca.mail.comcast.net with comcast id VmzG1l00R1t3BNj8ZmzGMj; Mon, 29 Apr 2013 10:59:17 +0000 Received: by icarus.home.lan (Postfix, from userid 1000) id 8EAFB73A1C; Mon, 29 Apr 2013 03:59:16 -0700 (PDT) Date: Mon, 29 Apr 2013 03:59:16 -0700 From: Jeremy Chadwick To: Steven Hartland Subject: Re: seeing data corruption with zfs trim functionality Message-ID: <20130429105916.GA1584@icarus.home.lan> References: <60316751643743738AB83DABC6A5934B@multiplay.co.uk> <20130429105143.GA1492@icarus.home.lan> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <20130429105143.GA1492@icarus.home.lan> User-Agent: Mutt/1.5.21 (2010-09-15) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=comcast.net; s=q20121106; t=1367233157; bh=fx+MnpWjfiwQrKzuoXXo/7JM6U1Vvh8OHHrVaSQx7JE=; h=Received:Received:Received:Date:From:To:Subject:Message-ID: MIME-Version:Content-Type; b=XRRYy08/Q/Nk9JzqRkAUIMFWvvNcnsT6C6MDeo7fxSHhBR92rtqXwRLi4krydOjKM Nnjqp5ifbdjlkC1Fw9/3TQU7H+v+rS6PIRgf4ptML84+x9esGYg4PYoNR1P/ksvs9l QvdhOb94xwMVTjPdsVjaMbVKK86JONLyAqKxfS57vfEPH0f4LWhvnr0KJ99mHzv3oK DGHz6jvCN7dMyOVxZfqm6Xqm/kEC0P0mcSgNrcbK5EPyZztwfFY3zKKfK18g2dC7Mg Z78OKZUIAYMehZR+drX/+ew+eZIokHBz0uVd0DVFb/XrSNGCPH/3NZ4/agge6xAy97 KUDoFin70zHyA== Cc: freebsd-fs@freebsd.org, Alexander Motin X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 29 Apr 2013 10:59:18 -0000 On Mon, Apr 29, 2013 at 03:51:43AM -0700, Jeremy Chadwick wrote: > On Mon, Apr 29, 2013 at 09:22:06AM +0100, Steven Hartland wrote: > > ----- Original Message ----- From: "Ajit Jain" > > > > > > >I am running zfs with trim functionality (ported from head). Seeing data > > >corruption when running iotest* with multiple threads (never saw data > > >corruption with single thread). > > > > > >The patches merged to add trim support are as follows: > > >1. 240868 (zfs trim patch) > > >2. 230053 and 245252 (block device driver trim support) > > >3. 239655 (fix an issue in patch 230053) > > > > > >I am "NOT" seeing data corruption in the following cases: > > >1. Running iotest with single thread (Trim is enabled at entire io stack). > > >2. Trim is enabled at zfs layer but disable at driver layer i.e. delete > > >method is set to NONE (even with multiple threads). > > > > > > > > >Since patch 240868 alone was not working as I pulled in additional zfs trim > > >patches 244155, 244187, 244188, 248572 (however I am not using separate > > >L2arc device), 248573, 248574, 248575 and 248576. Still I am seeing the > > >same issue. > > > > > >Issue: After some time running with multiple thread write system call > > >return sometimes with EIO or 122 (checksum error) error code. > > > > > >I looked at GEOM code a bit I think it already has the trim (DELETE) > > >command support. Still I am doubtful if I have pulled in all required > > >patches in the entire I/O stack. > > > > > >I am using a LSI SAS HBA card to connect to the SSD, firmware seems to > > >claim the support for trim. > > > > > >*iotest: non standard freebsd FreeBSD utility, which creates files and does > > >I/O on the files and can be invoked in single/multithread mode to do the > > >I/O. > > > > What version are you porting the changes to? > > > > What SSD are you using? > > > > What LSI controller are you using? > > I'd also like to see "zpool status" (for every pool that involves this > SSD) and "gpart show" against the disk itself. Also, the controller involved is an mps(4) controller, which to the underlying subsystem is SCSI. TRIM (as it's called; the actual name per ATA standard is DATA SET MANAGEMENT) is purely an ATA specification thing. The SCSI equivalent is called UNMAP, or alternately WRITE SAME. (This is not the case here, but just mentioning it: even in the cases of SCSI controllers that have SATA disks attached, the OS ends up submitting UNMAP/WRITE SAME, which the controller has to convert into the relevant ATA DATA SET MANAGEMENT command. If the controller firmware screws this up there not much we can do about it) References for FreeBSD: http://lists.freebsd.org/pipermail/freebsd-current/2011-December/030714.html PLEASE READ THE LAST PARAGRAPH OF THAT POST. This brings into question whether or not relevant subsystems (ranging from mps(4) to GEOM(4) to CAM(4)) actually have working UNMAP/WRITE SAME or not, or if the controller itself is doing something stupid with them. I'm CC'ing mav@ for what should be obvious reasons. -- | Jeremy Chadwick jdc@koitsu.org | | UNIX Systems Administrator http://jdc.koitsu.org/ | | Mountain View, CA, US | | Making life hard for others since 1977. PGP 4BD6C0CB |