From owner-freebsd-current@FreeBSD.ORG Wed Dec 12 23:17:29 2007 Return-Path: Delivered-To: freebsd-current@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id E3F4F16A41B for ; Wed, 12 Dec 2007 23:17:29 +0000 (UTC) (envelope-from Peter_Losher@isc.org) Received: from mx.isc.org (mx.isc.org [IPv6:2001:4f8:0:2::1c]) by mx1.freebsd.org (Postfix) with ESMTP id D934813C468 for ; Wed, 12 Dec 2007 23:17:29 +0000 (UTC) (envelope-from Peter_Losher@isc.org) Received: from farside.isc.org (farside.isc.org [IPv6:2001:4f8:3:bb::5]) (using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits)) (Client CN "farside.isc.org", Issuer "ISC CA" (verified OK)) by mx.isc.org (Postfix) with ESMTP id A9F5111401E for ; Wed, 12 Dec 2007 23:17:28 +0000 (UTC) (envelope-from Peter_Losher@isc.org) Received: from manx.isc.org (manx.isc.org [IPv6:2001:4f8:3:bb::37]) (using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits)) (Client did not present a certificate) by farside.isc.org (Postfix) with ESMTP id 5FFE9E6056 for ; Wed, 12 Dec 2007 23:17:29 +0000 (UTC) (envelope-from Peter_Losher@isc.org) Message-ID: <47606C09.2070209@isc.org> Date: Wed, 12 Dec 2007 15:17:29 -0800 From: Peter Losher Organization: ISC User-Agent: Thunderbird 2.0.0.9 (Macintosh/20071031) MIME-Version: 1.0 To: FreeBSD Current X-Enigmail-Version: 0.95.5 Content-Type: multipart/signed; micalg=pgp-sha1; protocol="application/pgp-signature"; boundary="------------enigAB71B231D3E7261D9266AEBE" Subject: ZFS melting under postgres... X-BeenThere: freebsd-current@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Discussions about the use of FreeBSD-current List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 12 Dec 2007 23:17:30 -0000 This is an OpenPGP/MIME signed message (RFC 2440 and 3156) --------------enigAB71B231D3E7261D9266AEBE Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: quoted-printable Hi, As part of our testing 7.0/ZFS we tried putting it thru it's paces having ZFS act as our storage medium for some test pgsql db's (like for sqlgrey, etc) and in both BETA2 and BETA4 (amd64) we get the same results with a RAIDZ2 container: -=3D- Dec 12 14:24:12 nsa sqlgrey: fatal: setconfig error at /usr/local/sbin/sqlgrey line 186. Dec 12 16:49:53 nsa root: ZFS: checksum mismatch, zpool=3Dvault path=3D/dev/ad4 offset=3D3665128448 size=3D22016 Dec 12 16:49:53 nsa root: ZFS: checksum mismatch, zpool=3Dvault path=3D/dev/ad6 offset=3D3665128448 size=3D22016 Dec 12 16:49:53 nsa root: ZFS: checksum mismatch, zpool=3Dvault path=3D/dev/ad8 offset=3D3665128448 size=3D22016 Dec 12 16:49:53 nsa root: ZFS: checksum mismatch, zpool=3Dvault path=3D/dev/ad10 offset=3D3665128448 size=3D22016 Dec 12 16:49:53 nsa root: ZFS: checksum mismatch, zpool=3Dvault path=3D/dev/ad12 offset=3D3665128448 size=3D22016 Dec 12 16:49:53 nsa root: ZFS: checksum mismatch, zpool=3Dvault path=3D/dev/ad14 offset=3D3665128448 size=3D22016 Dec 12 16:49:53 nsa root: ZFS: checksum mismatch, zpool=3Dvault path=3D/dev/ad16 offset=3D3665128448 size=3D21504 Dec 12 16:49:53 nsa root: ZFS: checksum mismatch, zpool=3Dvault path=3D/dev/ad18 offset=3D3665128448 size=3D21504 Dec 12 16:49:53 nsa root: ZFS: checksum mismatch, zpool=3Dvault path=3D/dev/ad4 offset=3D3665128448 size=3D22016 Dec 12 16:49:53 nsa root: ZFS: checksum mismatch, zpool=3Dvault path=3D/dev/ad6 offset=3D3665128448 size=3D22016 Dec 12 16:49:53 nsa root: ZFS: checksum mismatch, zpool=3Dvault path=3D/dev/ad8 offset=3D3665128448 size=3D22016 Dec 12 16:49:53 nsa root: ZFS: checksum mismatch, zpool=3Dvault path=3D/dev/ad10 offset=3D3665128448 size=3D22016 Dec 12 16:49:53 nsa root: ZFS: checksum mismatch, zpool=3Dvault path=3D/dev/ad12 offset=3D3665128448 size=3D22016 Dec 12 16:49:53 nsa root: ZFS: checksum mismatch, zpool=3Dvault path=3D/dev/ad14 offset=3D3665128448 size=3D22016 Dec 12 16:49:53 nsa root: ZFS: checksum mismatch, zpool=3Dvault path=3D/dev/ad16 offset=3D3665128448 size=3D21504 Dec 12 16:49:53 nsa root: ZFS: checksum mismatch, zpool=3Dvault path=3D/dev/ad18 offset=3D3665128448 size=3D21504 Dec 12 16:49:53 nsa root: ZFS: zpool I/O failure, zpool=3Dvault error=3D8= 6 Dec 12 16:49:53 nsa root: ZFS: checksum mismatch, zpool=3Dvault path=3D/dev/ad4 offset=3D3665128448 size=3D22016 Dec 12 16:49:53 nsa root: ZFS: checksum mismatch, zpool=3Dvault path=3D/dev/ad6 offset=3D3665128448 size=3D22016 Dec 12 16:49:53 nsa root: ZFS: checksum mismatch, zpool=3Dvault path=3D/dev/ad8 offset=3D3665128448 size=3D22016 Dec 12 16:49:53 nsa root: ZFS: checksum mismatch, zpool=3Dvault path=3D/dev/ad10 offset=3D3665128448 size=3D22016 Dec 12 16:49:53 nsa root: ZFS: checksum mismatch, zpool=3Dvault path=3D/dev/ad12 offset=3D3665128448 size=3D22016 Dec 12 16:49:53 nsa root: ZFS: checksum mismatch, zpool=3Dvault path=3D/dev/ad14 offset=3D3665128448 size=3D22016 Dec 12 16:49:53 nsa root: ZFS: checksum mismatch, zpool=3Dvault path=3D/dev/ad16 offset=3D3665128448 size=3D21504 Dec 12 16:49:53 nsa root: ZFS: checksum mismatch, zpool=3Dvault path=3D/dev/ad18 offset=3D3665128448 size=3D21504 Dec 12 16:49:53 nsa postgres[50527]: [5-1] PANIC: could not write to log file 2, segment 53 at offset 7864320, length 8192: Input/output error= Dec 12 16:49:53 nsa root: ZFS: checksum mismatch, zpool=3Dvault path=3D/dev/ad4 offset=3D3665128448 size=3D22016 Dec 12 16:49:53 nsa root: ZFS: checksum mismatch, zpool=3Dvault path=3D/dev/ad6 offset=3D3665128448 size=3D22016 Dec 12 16:49:53 nsa root: ZFS: checksum mismatch, zpool=3Dvault path=3D/dev/ad8 offset=3D3665128448 size=3D22016 Dec 12 16:49:53 nsa root: ZFS: checksum mismatch, zpool=3Dvault path=3D/dev/ad10 offset=3D3665128448 size=3D22016 Dec 12 16:49:53 nsa root: ZFS: checksum mismatch, zpool=3Dvault path=3D/dev/ad12 offset=3D3665128448 size=3D22016 Dec 12 16:49:53 nsa root: ZFS: checksum mismatch, zpool=3Dvault path=3D/dev/ad14 offset=3D3665128448 size=3D22016 Dec 12 16:49:53 nsa root: ZFS: checksum mismatch, zpool=3Dvault path=3D/dev/ad16 offset=3D3665128448 size=3D21504 Dec 12 16:49:53 nsa root: ZFS: checksum mismatch, zpool=3Dvault path=3D/dev/ad18 offset=3D3665128448 size=3D21504 Dec 12 16:49:53 nsa root: ZFS: zpool I/O failure, zpool=3Dvault error=3D8= 6 Dec 12 16:49:53 nsa postgres[50596]: [1-1] FATAL: the database system is starting up Dec 12 16:49:53 nsa kernel: pid 50527 (postgres), uid 70: exited on signal 6 (core dumped) -=3D- It basically corrupts the container from the inside until it fails completely (usually withing 24-48 hours depending on how busy the db is) I had thought it was a bad SATA replicator/controller, but we had that replaced w/ one from Supermicro. So it's either the disks, or something in ZFS. Anyone used ZFS to backend any db's (mysql or pgsql?) If you need more info, let me know... --=20 Peter_Losher@isc.org | ISC | OpenPGP 0xE8048D08 | "The bits must flow" --------------enigAB71B231D3E7261D9266AEBE Content-Type: application/pgp-signature; name="signature.asc" Content-Description: OpenPGP digital signature Content-Disposition: attachment; filename="signature.asc" -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.7 (Darwin) iD8DBQFHYGwJPtVx9OgEjQgRAsPuAKCUOz2/Jj9V/+mnK6sdTatoFb9nSwCgsJws PbXTr1C+vOsKWqfdfYZpe9Q= =H4Ak -----END PGP SIGNATURE----- --------------enigAB71B231D3E7261D9266AEBE--