From owner-freebsd-stable@FreeBSD.ORG Sat May 17 12:45:32 2014 Return-Path: Delivered-To: freebsd-stable@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id F13025F8 for ; Sat, 17 May 2014 12:45:31 +0000 (UTC) Received: from esa-jnhn.mail.uoguelph.ca (esa-jnhn.mail.uoguelph.ca [131.104.91.44]) by mx1.freebsd.org (Postfix) with ESMTP id 8A3EB2DF9 for ; Sat, 17 May 2014 12:45:30 +0000 (UTC) X-IronPort-Anti-Spam-Filtered: true X-IronPort-Anti-Spam-Result: AqYEAM7aa1ODaFve/2dsb2JhbABZg1VYgme7PoZrUQGBKXSCJQEBAQMBAQEBIAQnIAsFFhgCAg0ZAikBCSYGCAcEARwEiBgIDat2pDoXgSqMPRoBAQIZNAeCdIFLBJZbhBmRP4NSITV8CBci X-IronPort-AV: E=Sophos;i="4.98,859,1392181200"; d="scan'208";a="123077432" Received: from muskoka.cs.uoguelph.ca (HELO zcs3.mail.uoguelph.ca) ([131.104.91.222]) by esa-jnhn.mail.uoguelph.ca with ESMTP; 17 May 2014 08:45:29 -0400 Received: from zcs3.mail.uoguelph.ca (localhost.localdomain [127.0.0.1]) by zcs3.mail.uoguelph.ca (Postfix) with ESMTP id 0C927B4125; Sat, 17 May 2014 08:45:29 -0400 (EDT) Date: Sat, 17 May 2014 08:45:29 -0400 (EDT) From: Rick Macklem To: Damian Danielecki Message-ID: <933053021.3570827.1400330729023.JavaMail.root@uoguelph.ca> In-Reply-To: Subject: Re: [9.3 PRE] filesystem full even if there are free descriptors and disk space MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: 7bit X-Originating-IP: [172.17.91.201] X-Mailer: Zimbra 7.2.6_GA_2926 (ZimbraWebClient - FF3.0 (Win)/7.2.6_GA_2926) Cc: freebsd-stable@freebsd.org X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 17 May 2014 12:45:32 -0000 Damian Danielecki wrote: > I am receiving errors like this on heavily loaded FreeBSD > 9.3-PRERELEASE nfs server: > May 17 05:01:02 nfsd kernel: pid 4173 (nfsd), uid 0 inumber 185391682 > on /exports: filesystem full > Do you have -maproot=root on the line for /exports in /etc/exports? If not, it might be running into a quota for "nobody", which is what "uid 0" is mapped to by default. rick > Filesystem is newly created and server newly installed with custom > minimalist kernel&world. I am experienced user. > There are many free inodes and much disk space: > > # df -i > Filesystem 1K-blocks Used Avail Capacity iused > ifree %iused Mounted on > /dev/mirror/gm0s1a 473048844 3449532 431755408 1% 306673 > 60848397 1% / > devfs 1 1 0 100% 0 > 0 100% /dev > /dev/da0p1 3783567612 1772595572 1708286632 51% 20964037 > 468035769 4% /exports > > # df -h > Filesystem Size Used Avail Capacity Mounted on > /dev/mirror/gm0s1a 451G 3.3G 411G 1% / > devfs 1.0k 1.0k 0B 100% /dev > /dev/da0p1 3.5T 1.7T 1.6T 51% /exports > > # uname -a > FreeBSD nfsd.xxx.pl 9.3-PRERELEASE FreeBSD 9.3-PRERELEASE #5: Fri May > 16 15:41:36 CEST 2014 > root@nfsd.xxx.pl:/usr/obj/usr/src/sys/FREEBSD9 amd64 > > Filesystem is clean and not fragmented. > For sure fsck has been done on unmounted filesystem. > I see these errors also after fsck. > > # fsck -t ufs -y /dev/da0p1 > ** /dev/da0p1 > ** Last Mounted on /exports > ** Phase 1 - Check Blocks and Sizes > ** Phase 2 - Check Pathnames > ** Phase 3 - Check Connectivity > ** Phase 4 - Check Reference Counts > ** Phase 5 - Check Cyl groups > 20964037 files, 443148893 used, 502743010 free (396842 frags, > 62793271 > blocks, 0.0% fragmentation) > > These are my filesystem params. > > # tunefs -p /exports > tunefs: POSIX.1e ACLs: (-a) disabled > tunefs: NFSv4 ACLs: (-N) disabled > tunefs: MAC multilabel: (-l) disabled > tunefs: soft updates: (-n) enabled > tunefs: soft update journaling: (-j) disabled > tunefs: gjournal: (-J) disabled > tunefs: trim: (-t) disabled > tunefs: maximum blocks per file in a cylinder group: (-e) 4096 > tunefs: average file size: (-f) 16384 > tunefs: average number of files in a directory: (-s) 64 > tunefs: minimum percentage of free space: (-m) 8% > tunefs: space to hold for metadata blocks: (-k) 6408 > tunefs: optimization preference: (-o) time > tunefs: volume label: (-L) > > My mouting flags are: > /dev/da0p1 on /exports (ufs, NFS exported, local, noatime, nosuid, > with quotas, soft-updates) > > I found file releated to sample inode, it's users's backup. > > File: "/exports/(...)/xxx.tar.gz" > Size: 53706752 FileType: Regular File > Mode: (0640/-rw-r-----) Uid: ( 6225/ (6225)) Gid: ( 1676/ > (1676)) > Device: 0,78 Inode: 185391682 Links: 1 > Access: Sat May 17 05:00:57 2014 > Modify: Sat May 17 05:01:02 2014 > Change: Sat May 17 05:01:48 2014 > > I am unable to decompress it, it's actually damaged: > # cp xxx.tar.gz /otherfilesystem/ > # cd /otherfilesystem && gzip -d xxx.tar.gz > gzip: xxx.tar.gz: unexpected end of file > gzip: xxx.tar.gz: uncompress failed > > I've tried to rerun this user's backup and I am able to reproduce > problem just now. The same backup file of the same user now has > different inode but problem is the same: > pid 4173 (nfsd), uid 0 inumber 185391652 on /exports: filesystem full > > # stat -x xxx.tar.gz > File: "xxx.tar.gz" > Size: 55902208 FileType: Regular File > Mode: (0640/-rw-r-----) Uid: ( 6225/ (6225)) Gid: ( 1676/ > (1676)) > Device: 0,78 Inode: 185391652 Links: 1 > Access: Sat May 17 12:51:47 2014 > Modify: Sat May 17 12:51:51 2014 > Change: Sat May 17 12:52:52 2014 > > Of course I am able to create any new big file from /dev/random, > there > is free space. I've easily created single 30GB file via nfs. > I've also tried to count md5 sum of newly generated 1GB file many > times via nfs and it's still the same. I checked this to be sure nfs > transmission is valid and da0 device is working properly. > > I've added some primitive debug to kernel sources but this is > production env so I will be able to reboot server only at night. Just > now it's impossible. > > /usr/src/sys/ufs # grep -R 'filesystem full' * > ffs/ffs_balloc.c: ffs_fserr(fs, > ip->i_number, "filesystem full line 320"); > ffs/ffs_balloc.c: ffs_fserr(fs, > ip->i_number, "filesystem full line 397"); > ffs/ffs_balloc.c: ffs_fserr(fs, > ip->i_number, "filesystem full line 882"); > ffs/ffs_balloc.c: ffs_fserr(fs, > ip->i_number, "filesystem full line 960"); > ffs/ffs_alloc.c: ffs_fserr(fs, ip->i_number, > "filesystem full line 227"); > ffs/ffs_alloc.c: ffs_fserr(fs, ip->i_number, > "filesystem full line 438"); > > Any help will be appreciated! Should be corrected before 9.3-RELEASE. > > DD > _______________________________________________ > freebsd-stable@freebsd.org mailing list > http://lists.freebsd.org/mailman/listinfo/freebsd-stable > To unsubscribe, send any mail to > "freebsd-stable-unsubscribe@freebsd.org" >