From owner-freebsd-current@FreeBSD.ORG Thu Dec 13 03:00:50 2007 Return-Path: Delivered-To: freebsd-current@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 43B6216A419 for ; Thu, 13 Dec 2007 03:00:50 +0000 (UTC) (envelope-from kip.macy@gmail.com) Received: from wa-out-1112.google.com (wa-out-1112.google.com [209.85.146.180]) by mx1.freebsd.org (Postfix) with ESMTP id 2719813C43E for ; Thu, 13 Dec 2007 03:00:49 +0000 (UTC) (envelope-from kip.macy@gmail.com) Received: by wa-out-1112.google.com with SMTP id k17so815729waf.3 for ; Wed, 12 Dec 2007 19:00:49 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=domainkey-signature:received:received:message-id:date:from:to:subject:cc:in-reply-to:mime-version:content-type:content-transfer-encoding:content-disposition:references; bh=Z0mL4D9QB11RYjGmCKHAIfbJ/QHviX3LADvgiAZy0Ok=; b=ML3lN6qA1rb6QRfM/T+mhvx+pFsioSGVobCCOejWwgxPW8EA8GZPyiBdFrSQDhToxOPSdkLxxblHqo4708GTKnrEtrPiqIxi3ER1+0W57j/SwzDP1gjsnj3XOpH+Wo6ynXgvAPrgcuzbEYDqTk0UAesB4bNtZ7wtFJsMe7UIlCw= DomainKey-Signature: a=rsa-sha1; c=nofws; d=gmail.com; s=gamma; h=message-id:date:from:to:subject:cc:in-reply-to:mime-version:content-type:content-transfer-encoding:content-disposition:references; b=AxzLF8tgoLcj7uWoBzqvG0pdDz8ABfKbbLiU5V8Ub6V7n6YZCcLJC29KxL3iL7o0f7Dn6BZiTMAl+dRIY7S/KiteED68M3t2eMTMh6xU4RRoIginglVw1DSN1anCzGcL4MpLBRoBPxEQZ9f87WAVHW9Z+egHQVNJ/Sx3AIZnkyY= Received: by 10.114.26.1 with SMTP id 1mr1640728waz.80.1197514846848; Wed, 12 Dec 2007 19:00:46 -0800 (PST) Received: by 10.114.255.11 with HTTP; Wed, 12 Dec 2007 19:00:46 -0800 (PST) Message-ID: Date: Wed, 12 Dec 2007 19:00:46 -0800 From: "Kip Macy" To: "Hugo Silva" In-Reply-To: <47609FE3.8040606@barafranca.com> MIME-Version: 1.0 Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: 7bit Content-Disposition: inline References: <47606C09.2070209@isc.org> <47609F0A.7010805@clearchain.com> <47609FE3.8040606@barafranca.com> Cc: freebsd-current@freebsd.org, Benjamin Close Subject: Re: ZFS melting under postgres... X-BeenThere: freebsd-current@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Discussions about the use of FreeBSD-current List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 13 Dec 2007 03:00:50 -0000 On Dec 12, 2007 6:58 PM, Hugo Silva wrote: > > Benjamin Close wrote: > > Peter Losher wrote: > >> Hi, > >> > >> As part of our testing 7.0/ZFS we tried putting it thru it's paces > >> having ZFS act as our storage medium for some test pgsql db's (like for > >> sqlgrey, etc) and in both BETA2 and BETA4 (amd64) we get the same > >> results with a RAIDZ2 container: > >> > >> -=- > >> Dec 12 14:24:12 nsa sqlgrey: fatal: setconfig error at > >> /usr/local/sbin/sqlgrey line 186. > >> Dec 12 16:49:53 nsa root: ZFS: checksum mismatch, zpool=vault > >> path=/dev/ad4 offset=3665128448 size=22016 > >> Dec 12 16:49:53 nsa root: ZFS: checksum mismatch, zpool=vault > >> path=/dev/ad6 offset=3665128448 size=22016 > >> Dec 12 16:49:53 nsa root: ZFS: checksum mismatch, zpool=vault > >> path=/dev/ad8 offset=3665128448 size=22016 > >> Dec 12 16:49:53 nsa root: ZFS: checksum mismatch, zpool=vault > >> path=/dev/ad10 offset=3665128448 size=22016 > >> Dec 12 16:49:53 nsa root: ZFS: checksum mismatch, zpool=vault > >> path=/dev/ad12 offset=3665128448 size=22016 > >> Dec 12 16:49:53 nsa root: ZFS: checksum mismatch, zpool=vault > >> path=/dev/ad14 offset=3665128448 size=22016 > >> Dec 12 16:49:53 nsa root: ZFS: checksum mismatch, zpool=vault > >> path=/dev/ad16 offset=3665128448 size=21504 > >> Dec 12 16:49:53 nsa root: ZFS: checksum mismatch, zpool=vault > >> path=/dev/ad18 offset=3665128448 size=21504 > >> Dec 12 16:49:53 nsa root: ZFS: checksum mismatch, zpool=vault > >> path=/dev/ad4 offset=3665128448 size=22016 > >> Dec 12 16:49:53 nsa root: ZFS: checksum mismatch, zpool=vault > >> path=/dev/ad6 offset=3665128448 size=22016 > >> Dec 12 16:49:53 nsa root: ZFS: checksum mismatch, zpool=vault > >> path=/dev/ad8 offset=3665128448 size=22016 > >> Dec 12 16:49:53 nsa root: ZFS: checksum mismatch, zpool=vault > >> path=/dev/ad10 offset=3665128448 size=22016 > >> Dec 12 16:49:53 nsa root: ZFS: checksum mismatch, zpool=vault > >> path=/dev/ad12 offset=3665128448 size=22016 > >> Dec 12 16:49:53 nsa root: ZFS: checksum mismatch, zpool=vault > >> path=/dev/ad14 offset=3665128448 size=22016 > >> Dec 12 16:49:53 nsa root: ZFS: checksum mismatch, zpool=vault > >> path=/dev/ad16 offset=3665128448 size=21504 > >> Dec 12 16:49:53 nsa root: ZFS: checksum mismatch, zpool=vault > >> path=/dev/ad18 offset=3665128448 size=21504 > >> Dec 12 16:49:53 nsa root: ZFS: zpool I/O failure, zpool=vault error=86 > >> Dec 12 16:49:53 nsa root: ZFS: checksum mismatch, zpool=vault > >> path=/dev/ad4 offset=3665128448 size=22016 > >> Dec 12 16:49:53 nsa root: ZFS: checksum mismatch, zpool=vault > >> path=/dev/ad6 offset=3665128448 size=22016 > >> Dec 12 16:49:53 nsa root: ZFS: checksum mismatch, zpool=vault > >> path=/dev/ad8 offset=3665128448 size=22016 > >> Dec 12 16:49:53 nsa root: ZFS: checksum mismatch, zpool=vault > >> path=/dev/ad10 offset=3665128448 size=22016 > >> Dec 12 16:49:53 nsa root: ZFS: checksum mismatch, zpool=vault > >> path=/dev/ad12 offset=3665128448 size=22016 > >> Dec 12 16:49:53 nsa root: ZFS: checksum mismatch, zpool=vault > >> path=/dev/ad14 offset=3665128448 size=22016 > >> Dec 12 16:49:53 nsa root: ZFS: checksum mismatch, zpool=vault > >> path=/dev/ad16 offset=3665128448 size=21504 > >> Dec 12 16:49:53 nsa root: ZFS: checksum mismatch, zpool=vault > >> path=/dev/ad18 offset=3665128448 size=21504 > >> Dec 12 16:49:53 nsa postgres[50527]: [5-1] PANIC: could not write to > >> log file 2, segment 53 at offset 7864320, length 8192: Input/output > >> error > >> Dec 12 16:49:53 nsa root: ZFS: checksum mismatch, zpool=vault > >> path=/dev/ad4 offset=3665128448 size=22016 > >> Dec 12 16:49:53 nsa root: ZFS: checksum mismatch, zpool=vault > >> path=/dev/ad6 offset=3665128448 size=22016 > >> Dec 12 16:49:53 nsa root: ZFS: checksum mismatch, zpool=vault > >> path=/dev/ad8 offset=3665128448 size=22016 > >> Dec 12 16:49:53 nsa root: ZFS: checksum mismatch, zpool=vault > >> path=/dev/ad10 offset=3665128448 size=22016 > >> Dec 12 16:49:53 nsa root: ZFS: checksum mismatch, zpool=vault > >> path=/dev/ad12 offset=3665128448 size=22016 > >> Dec 12 16:49:53 nsa root: ZFS: checksum mismatch, zpool=vault > >> path=/dev/ad14 offset=3665128448 size=22016 > >> Dec 12 16:49:53 nsa root: ZFS: checksum mismatch, zpool=vault > >> path=/dev/ad16 offset=3665128448 size=21504 > >> Dec 12 16:49:53 nsa root: ZFS: checksum mismatch, zpool=vault > >> path=/dev/ad18 offset=3665128448 size=21504 > >> Dec 12 16:49:53 nsa root: ZFS: zpool I/O failure, zpool=vault error=86 > >> Dec 12 16:49:53 nsa postgres[50596]: [1-1] FATAL: the database system > >> is starting up > >> Dec 12 16:49:53 nsa kernel: pid 50527 (postgres), uid 70: exited on > >> signal 6 (core dumped) > >> -=- > >> > >> It basically corrupts the container from the inside until it fails > >> completely (usually withing 24-48 hours depending on how busy the db is) > >> > >> I had thought it was a bad SATA replicator/controller, but we had that > >> replaced w/ one from Supermicro. So it's either the disks, or something > >> in ZFS. Anyone used ZFS to backend any db's (mysql or pgsql?) > >> > >> If you need more info, let me know... > >> > >> > > Try turning of zil, whilst I don't use a db, I have zfs under high > > load. I've found without zil turned off I see checksum corruption as > > well: > > > > /boot/loader.conf > > > > vfs.zfs.zil_disable=1 > > > > Cheers, > > Benjamin > > Wouldn't it be a bad idea to disable ZIL ? > > http://www.solarisinternals.com/wiki/index.php/ZFS_Evil_Tuning_Guide#Disabling_the_ZIL_.28Don.27t.29 Yes. However, FreeBSD suffers from deadlocks under load if ZIL is enabled. -Kip