From owner-freebsd-scsi@freebsd.org Mon Aug 7 09:16:06 2017 Return-Path: Delivered-To: freebsd-scsi@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 47AD7DC6298 for ; Mon, 7 Aug 2017 09:16:06 +0000 (UTC) (envelope-from janm@transactionware.com) Received: from mail3.transactionware.com (mail.transactionware.com [203.14.245.7]) by mx1.freebsd.org (Postfix) with SMTP id A33BE2ADC for ; Mon, 7 Aug 2017 09:16:03 +0000 (UTC) (envelope-from janm@transactionware.com) Received: (qmail 99899 invoked by uid 907); 7 Aug 2017 09:09:19 -0000 Received: from Unknown (HELO jmmacpro.tmst.com.au) (203.14.245.130) (smtp-auth username janm, mechanism plain) by mail3.transactionware.com (qpsmtpd/0.84) with (ECDHE-RSA-AES256-GCM-SHA384 encrypted) ESMTPSA; Mon, 07 Aug 2017 19:09:19 +1000 From: Jan Mikkelsen Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable Mime-Version: 1.0 (Mac OS X Mail 10.3 \(3273\)) Subject: BIO_FLUSH, drive write cache: Which drivers do the Right Thing? Message-Id: <6E659EED-746D-4936-9C53-702D72DA84B8@transactionware.com> Date: Mon, 7 Aug 2017 19:11:18 +1000 To: freebsd-scsi@freebsd.org X-Mailer: Apple Mail (2.3273) X-BeenThere: freebsd-scsi@freebsd.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: SCSI subsystem List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 07 Aug 2017 09:16:06 -0000 Hi, I'm looking at how ZFS interacts with drive-based write caches, so that = I can work out what is =E2=80=9Csafe=E2=80=9D. The premise is that drive write caching is safe because ZFS can flush = the drive write cache when necessary to ensure updates get to the = physical media in the correct order. For this to work, the actual = command has to get to the drive. This seems to work for ata(4), da(4) and amr(4). Other non-CAM RAID controllers, eg. mfi(4) don=E2=80=99t seem to support = this at all, including for JBOD drives like mfisyspd. For CAM-based controllers (eg. mpr(4) or mrsas(4)), it seems that = controllers like mpr(4) pass the SCSI command on as expected. RAID = controllers implementing CAM have to do the right thing to make it work; = mrsas(4) seems to find SYNCHRONIZE_CACHE and do something but I=E2=80=99m = not sure that it really sending a cache flush command to any underlying = drives. =46rom all of this, this is what I see: - ata(4) and amr(4) pass the flush command to the drive, allowing the = use of drive write cache. - da(4) on =E2=80=9Cplain SCSI=E2=80=9D controllers pass the command to = the drive, allowing the use of drive write cache. - da(4) for RAID controllers may or may not get the command to the = drives. - Other RAID controllers will not currently get the command to the = underlying drives, making drive write cache dangerous. Some questions: Does mrsas(4) really get a write cache flush to the drive? Does it vary = for the JBOD and RAID volume case? Is all this correct? What have I missed? Thanks, Jan Mikkesen