From owner-freebsd-bugs Mon Apr 19 15:12:28 1999 Delivered-To: freebsd-bugs@freebsd.org Received: from freefall.freebsd.org (freefall.FreeBSD.ORG [204.216.27.21]) by hub.freebsd.org (Postfix) with ESMTP id 1585515678 for ; Mon, 19 Apr 1999 15:12:26 -0700 (PDT) (envelope-from gnats@FreeBSD.org) Received: (from gnats@localhost) by freefall.freebsd.org (8.9.2/8.9.2) id PAA25181; Mon, 19 Apr 1999 15:10:00 -0700 (PDT) (envelope-from gnats@FreeBSD.org) Received: from moth.vicor-nb.com (moth.vicor-nb.com [208.206.78.46]) by hub.freebsd.org (Postfix) with ESMTP id 4D90515577 for ; Mon, 19 Apr 1999 15:09:11 -0700 (PDT) (envelope-from milt@moth.vicor-nb.com) Received: (from milt@localhost) by moth.vicor-nb.com (8.8.5/8.8.5) id PAA05235; Mon, 19 Apr 1999 15:08:24 -0700 (PDT) Message-Id: <199904192208.PAA05235@moth.vicor-nb.com> Date: Mon, 19 Apr 1999 15:08:24 -0700 (PDT) From: milt To: FreeBSD-gnats-submit@freebsd.org Cc: daver@moth.vicor-nb.com, jpl@moth.vicor-nb.com Subject: kern/11226: Invalid files on disk after fsync Sender: owner-freebsd-bugs@FreeBSD.ORG Precedence: bulk X-Loop: FreeBSD.org >Number: 11226 >Category: kern >Synopsis: Invalid files on disk after fsync >Confidential: no >Severity: critical >Priority: high >Responsible: freebsd-bugs >State: open >Quarter: >Keywords: >Date-Required: >Class: sw-bug >Submitter-Id: current-users >Arrival-Date: Mon Apr 19 15:10:00 PDT 1999 >Closed-Date: >Last-Modified: >Originator: milt@vicor-nb.com >Release: FreeBSD 2.2.6-STABLE >Organization: vicor >Environment: We are running a very busy server which includes scsi drives and lots of nfs mounts. df and dmesg are: 52. ? df Filesystem 1K-blocks Used Avail Capacity Mounted on /dev/sd0s1a 59471 21463 33251 39% / /dev/sd0s1f 8202004 2540061 5005783 34% /usr /dev/sd0s1e 98479 4681 85920 5% /var /dev/sd1c 8621381 1 7931670 0% /disk2 procfs 4 4 0 100% /proc opshome:/usr/opshome 8241012 4715979 2865753 62% /usr/opshome oos0a:/raid1 60821938 10291873 45664310 18% /raid1 oos0a:/raid2 60821938 11946475 44009708 21% /raid2 oos0a:/raid3 60821938 73528 55882655 0% /raid3 oos0a:/usr/env1/data/lb 1843066 847170 848451 50% /usr/env1/data/lb oos0a:/usr/env5/data/lb 1843066 847170 848451 50% /usr/env5/data/lb oos0a:/usr/env3/data/lb 1843066 847170 848451 50% /usr/env3/data/lb oos0a:/usr/env4/data/lb 1843066 847170 848451 50% /usr/env4/data/lb Copyright (c) 1992-1998 FreeBSD Inc. Copyright (c) 1982, 1986, 1989, 1991, 1993 The Regents of the University of California. All rights reserved. FreeBSD 2.2.6-STABLE #0: Tue Apr 28 15:08:30 PDT 1998 root@ipm0.lbxrich.vicor-nb.com:/disk2/src.patched/sys/compile/WSS CPU: Pentium (199.43-MHz 586-class CPU) Origin = "GenuineIntel" Id = 0x544 Stepping=4 Features=0x8001bf real memory = 134217728 (131072K bytes) avail memory = 129515520 (126480K bytes) Probing for devices on PCI bus 0: chip0 rev 1 on pci0:0:0 chip1 rev 1 on pci0:7:0 chip2 rev 1 on pci0:7:1 chip3 rev 1 int d irq 11 on pci0:7:2 chip4 rev 1 on pci0:7:3 ahc0 rev 0 int a irq 10 on pci0:9:0 ahc0: aic7870 Single Channel, SCSI Id=7, 16 SCBs ahc0 waiting for scsi devices to settle ahc0: target 0 Tagged Queuing Device (ahc0:0:0): "SEAGATE ST19171N 0023" type 0 fixed SCSI 2 sd0(ahc0:0:0): Direct-Access 8683MB (17783112 512 byte sectors) ahc0: target 1 Tagged Queuing Device (ahc0:1:0): "SEAGATE ST19171N 0023" type 0 fixed SCSI 2 sd1(ahc0:1:0): Direct-Access 8683MB (17783112 512 byte sectors) (ahc0:3:0): "HP C1533A A708" type 1 removable SCSI 2 st0(ahc0:3:0): Sequential-Access density code 0x24, variable blocks, write-enabled fxp0 rev 1 int a irq 9 on pci0:10:0 fxp0: Ethernet address 00:a0:c9:27:93:fd vga0 rev 0 int a irq 12 on pci0:11:0 Probing for devices on the ISA bus: sc0 at 0x60-0x6f irq 1 on motherboard sc0: VGA color <16 virtual consoles, flags=0x0> sio0 at 0x3f8-0x3ff irq 4 on isa sio0: type 16550A sio1 at 0x2f8-0x2ff irq 3 on isa sio1: type 16550A psm0 not found at 0x60 fdc0 at 0x3f0-0x3f7 irq 6 drq 2 on isa fdc0: FIFO enabled, 8 bytes threshold fd0: 1.44MB 3.5in npx0 on motherboard npx0: INT 16 interface Intel Pentium F00F detected, installing workaround >Description: What we notice is invalid file contents. The sequence is: 1. Our Qft utility writes a file by using a open,write/write/...,fsync,close sequence: int outFd = open (inPath, O_WRONLY | O_CREAT | O_TRUNC, 0664); if (outFd <= 0) ReportError ("Unable to open " << inPath << " for writing") while (... ... receive 8192 bytes from tcp if (write (fd, fileBuffer, sizeof (fileBuffer)) != sizeof (fileBuffer)) ReportError ("Error Writing File " << inPath); ... the last write may be less than 8192 bytes. if (fsync (outFd)) ReportError ("fsync failure on file " << inPath) close (outFd); 2. Some program (maybe a backup utility, maybe Qft) reads the file and receives valid contents. 3. Some program reads the file and receives invalid contents. *. At this point, the data on disk is permanently invalid. *. The problem consists of 8192 wrong bytes which start on a page boundary in the middle of the file. The first time this came up, the bad data consisted of two short files which had existed some weeks prior to the problem. The second time this came up, the bad data looked like part of an executeable and could have been old or new. *. The file date/time modified (from ls -l) remains the time of the initial file write from step 1. THAT IS: we write a file, we read it and find it good, we read it and find it invalid even though the date hasn't changed. Probably you will think I am making this up, confused, or incompetent. Sounds pretty phenomenal. Tell me how one could accidentally change the file contents without changing the access date? Also, the valid read is happening within 5 to 20 minutes of the initial file write and may well be seeing data which is in cache but never makes it to disk. (The invalid read is happening hours or days later.) >How-To-Repeat: I wish I could. I spent all week end trying. The cause is going to have to include some unusual interaction or other or I would have duplicated it by now. We have seen two incidents one month apart on two different hosts at the same site (our busiest site). >Fix: I wish I knew. Fortunately it doesn't come up often. >Release-Note: >Audit-Trail: >Unformatted: Reply-To: milt@vicor-nb.com X-send-pr-version: 3.2 To Unsubscribe: send mail to majordomo@FreeBSD.org with "unsubscribe freebsd-bugs" in the body of the message