From owner-freebsd-hardware Fri Mar 14 7:52: 3 2003 Delivered-To: freebsd-hardware@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id BD3D437B404; Fri, 14 Mar 2003 07:51:58 -0800 (PST) Received: from ecserv7.uwaterloo.ca (ecserv7.uwaterloo.ca [129.97.50.127]) by mx1.FreeBSD.org (Postfix) with ESMTP id C9E0A43F93; Fri, 14 Mar 2003 07:51:57 -0800 (PST) (envelope-from bruce@engmail.uwaterloo.ca) Received: from ecserv7.uwaterloo.ca (localhost.uwaterloo.ca [127.0.0.1]) by ecserv7.uwaterloo.ca (8.12.6/8.12.6) with ESMTP id h2EFpupc097257; Fri, 14 Mar 2003 10:51:56 -0500 (EST) (envelope-from bruce@engmail.uwaterloo.ca) Received: (from www@localhost) by ecserv7.uwaterloo.ca (8.12.6/8.12.6/Submit) id h2EFpuN0097256; Fri, 14 Mar 2003 10:51:56 -0500 (EST) X-Authentication-Warning: ecserv7.uwaterloo.ca: www set sender to bruce@engmail.uwaterloo.ca using -f Received: from 129.97.50.50 ( [129.97.50.50]) as user bruce@engmail.uwaterloo.ca by www.nexusmail.uwaterloo.ca with HTTP; Fri, 14 Mar 2003 10:51:56 -0500 Message-ID: <1047657116.3e71fa9c28266@www.nexusmail.uwaterloo.ca> Date: Fri, 14 Mar 2003 10:51:56 -0500 From: Bruce Campbell To: freebsd-questions@freebsd.org, freebsd-hardware@freebsd.org Subject: Re: problem on 1TB filesystem RAID 5 3ware References: <1047519493.3e6fe105a4f15@www.nexusmail.uwaterloo.ca> In-Reply-To: <1047519493.3e6fe105a4f15@www.nexusmail.uwaterloo.ca> MIME-Version: 1.0 Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: 8bit User-Agent: Internet Messaging Program (IMP) 3.1 / FreeBSD-4.6.2 X-Originating-IP: 129.97.50.50 Sender: owner-freebsd-hardware@FreeBSD.ORG Precedence: bulk List-ID: List-Archive: (Web Archive) List-Help: (List Instructions) List-Subscribe: List-Unsubscribe: X-Loop: FreeBSD.org Not solved this yet, but I have determined a few things that the problem isn't. Info at: http://www.freebsd.uwaterloo.ca/twiki/bin/view/Freebsd/BackupServerProblem Tested with soft updates off and on, fails in either case, so that isn't it. Seems like the problem is either: - 3ware card or driver - something to do with the large filesystem Quoting Bruce Campbell : > > File corruption on 2 identical systems, designed to be backup > servers to contain dumps of other systems: > > FreeBSD ecserv18.uwaterloo.ca 4.7-RELEASE FreeBSD 4.7-RELEASE #0: Wed Oct 9 > > 15:08:34 GMT 2002 root@builder.freebsdmall.com:/usr/obj/usr/src/sys/GENERIC > > i386 > > with 1TB /backup partition, on a 3ware 7500-8 ATA RAID card, RAID 5: > > Filesystem 1K-blocks Used Avail Capacity Mounted on > /dev/twed0s1a 20644846 906552 18086708 5% / > procfs 4 4 0 100% /proc > /dev/twed0s1e 938819776 279031856 584682338 32% /backup > > disks are 6 x Western Digital 2000JB (200GB) > > I ran tests on /backup for 10 days on each system (fill disk with > 50GB files of pseudo random data, then reading them all back and > verify contents, then erase, then start over). Tests ran perfectly. > > details on hardware config at: > > http://www.freebsd.uwaterloo.ca/twiki/bin/view/Freebsd/BackupServerHardware > > Then, I was ready to put the systems into production, so I copied > data from my 2 older backup servers (which have 360GB vinum partitions) > and after copying the data (approx 250GB in 325 files) about a dozen > files were corrupt after the copy. I copied via an NFS mount. > > All corruption started on a 64K boundary, except one which was on a 16K > boundary. Recopied the dozen corrupt files, and then only 6 were corrupt. > Same problem on both systems, each which copied from a different source > server. > > File seems corrupt to the end after first corruption starts, I have > not looked for a pattern to see if it is another files contents, > or misplaced contents from the same file. > > fsck shows no problems > > Restarted my test filling with 50GB files again, has run perfectly. > > I plan to try: > > - turn off soft updates > - RAID 10 instead of 5 > - different file system parameters, for example I don't need > 100 million inodes. > - rcp'ing the files > - staring at computer screen > > By the way, 3ware has not officially approved the WD 200GB drive last > time I checked. > > Lots of good experience with the motherboard (ASUS P4S533) and > network card (Intel Pro/100). Lots of good experience with > vinum striped partitions of smaller size (360GB) > > Does anyone have any suggestions ? > > -- > Bruce Campbell > Engineering Computing > CPH-2374B > University of Waterloo > (519)888-4567 ext 5889 > > ---------------------------------------- > This mail sent through www.mywaterloo.ca > -- Bruce Campbell Engineering Computing CPH-2374B University of Waterloo (519)888-4567 ext 5889 ---------------------------------------- This mail sent through www.mywaterloo.ca To Unsubscribe: send mail to majordomo@FreeBSD.org with "unsubscribe freebsd-hardware" in the body of the message