Date: Mon, 18 Feb 2013 23:39:23 +0100 From: Pawel Jakub Dawidek <pjd@FreeBSD.org> To: John-Mark Gurney <jmg@funkthat.com> Cc: freebsd-fs@FreeBSD.org Subject: Re: ZFS on 9.1 doesn't see errors on geli volumes... Message-ID: <20130218223923.GB1375@garage.freebsd.pl> In-Reply-To: <20130218200121.GJ55866@funkthat.com> References: <20130218191242.GI55866@funkthat.com> <20130218200121.GJ55866@funkthat.com>
next in thread | previous in thread | raw e-mail | index | archive | help
--rS8CxjVDS/+yyDmU
Content-Type: text/plain; charset=us-ascii
Content-Disposition: inline
Content-Transfer-Encoding: quoted-printable
On Mon, Feb 18, 2013 at 12:01:21PM -0800, John-Mark Gurney wrote:
> John-Mark Gurney wrote this message on Mon, Feb 18, 2013 at 11:12 -0800:
> > I'm running 9.1:
> > FreeBSD gold.funkthat.com 9.1-PRERELEASE FreeBSD 9.1-PRERELEASE #26 r24=
1041M: Wed Dec 12 23:02:31 PST 2012 jmg@gold.funkthat.com:/usr/src.9sta=
ble/sys/amd64/compile/gold amd64
> >=20
> > The modifications are limited to improving AES-NI performance.
> >=20
> > On a box, and decided to go full ZFS w/ geli encrypted volumes (includi=
ng
> > root fs)... One of the hard drives started going bad, so I started
> > seeing:
> > hpt27xx: Device error information 0x1000000
> > hpt27xx: Task file error, StatusReg=3D0x51, ErrReg=3D0x40, LBA[0-3]=3D0=
xf495e928,LBA[4-7]=3D0x0.
> > (da3:hpt27xx0:0:3:0): READ(10). CDB: 28 0 f4 95 e8 f8 0 0 80 0=20
> > (da3:hpt27xx0:0:3:0): CAM status: Auto-Sense Retrieval Failed
> > (da3:hpt27xx0:0:3:0): Error 5, Unretryable error
> > GEOM_ELI: g_eli_read_done() failed label/toby.eli[READ(offset=3D2100974=
186496, length=3D90112)]
> >=20
> > and:
> > (da3:hpt27xx0:0:3:0): WRITE(10). CDB: 2a 0 ef cc 10 90 0 0 8 0=20
> > (da3:hpt27xx0:0:3:0): CAM status: Auto-Sense Retrieval Failed
> > (da3:hpt27xx0:0:3:0): Error 5, Unretryable error
> > GEOM_ELI: Crypto WRITE request failed (error=3D5). label/toby.eli[WRITE=
(offset=3D2059841654784, length=3D4096)]
> >=20
> > So we can see that geli is failing, but zpool status command doesn't sh=
ow
> > any errors at all... The READ and WRITE columns both show 0 for the de=
vice..
> >=20
> > Now I do know that the WRITEs are not retried, because if I do a scrub
> > afterward, it detects cksum errors, and does properly increases the
> > count in the CKSUM column...
> >=20
> > Now if I pull a device, it will see that the device is lost, but no
> > matter how many read or write errors get returned by geli, zfs doesn't
> > seem to count them...
> >=20
> > Has anyone else seen this w/ ZFS? Is it possible that it's a problem w/
> > geli, and not ZFS?
> >=20
> > I haven't tried to run a test w/ gnop to fail some read/writes on -curr=
ent..
> >=20
> > P.S. Please keep me cc'd, as I'm not on the list.
>=20
> Well, after some digging w/ help from smh@, it looks like the write
> case in geli is broken... in g_eli_write_done, we have the code:
> if (pbp->bio_error !=3D 0) {
> G_ELI_LOGREQ(0, pbp, "Crypto WRITE request failed (error=
=3D%d).",
> pbp->bio_error);
> pbp->bio_completed =3D 0;
> }
> /*
> * Write is finished, send it up.
> */
> pbp->bio_completed =3D pbp->bio_length;
> sc =3D pbp->bio_to->geom->softc;
> g_io_deliver(pbp, pbp->bio_error);
> atomic_subtract_int(&sc->sc_inflight, 1);
>=20
> so, we just end up overwriting the bio_completed error...
>=20
> pjd, should we just put the bio_completed =3D line under an else?
>=20
> something like:
> if (pbp->bio_error !=3D 0) {
> G_ELI_LOGREQ(0, pbp, "Crypto WRITE request failed (error=3D%d).",
> pbp->bio_error);
> pbp->bio_completed =3D 0;
> } else
> pbp->bio_completed =3D pbp->bio_length;
>=20
> /* Write is finished, send it up. */
> g_io_deliver(pbp, pbp->bio_error);
> sc =3D pbp->bio_to->geom->softc;
> atomic_subtract_int(&sc->sc_inflight, 1);
>=20
> But doesn't explain why read's aren't being counted though...
Your patch looks correct (but add { } around else content before
committing).
The logic in vdev_geom.c should also be modified to treat other errors
just like EIOs.
This all still doesn't explain what you are seeing, as you did have
EIOs. Experimenting with gnop may provide more info.
--=20
Pawel Jakub Dawidek http://www.wheelsystems.com
FreeBSD committer http://www.FreeBSD.org
Am I Evil? Yes, I Am! http://tupytaj.pl
--rS8CxjVDS/+yyDmU
Content-Type: application/pgp-signature
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v2.0.19 (FreeBSD)
iEYEARECAAYFAlEirZsACgkQForvXbEpPzT9/ACgz5v6IEAct2VE77NxS6TBo+YP
IWoAn3EAyT2rnsSKSIGacppcyFc193iM
=oeJ/
-----END PGP SIGNATURE-----
--rS8CxjVDS/+yyDmU--
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?20130218223923.GB1375>