From owner-freebsd-hackers@FreeBSD.ORG Mon Dec 13 18:57:42 2004 Return-Path: Delivered-To: freebsd-hackers@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 4E63B16A4CE for ; Mon, 13 Dec 2004 18:57:42 +0000 (GMT) Received: from mail15.syd.optusnet.com.au (mail15.syd.optusnet.com.au [211.29.132.196]) by mx1.FreeBSD.org (Postfix) with ESMTP id 8513C43D1D for ; Mon, 13 Dec 2004 18:57:41 +0000 (GMT) (envelope-from PeterJeremy@optushome.com.au) Received: from cirb503493.alcatel.com.au (c211-30-75-229.belrs2.nsw.optusnet.com.au [211.30.75.229]) iBDIvcWX018178 (version=TLSv1/SSLv3 cipher=EDH-RSA-DES-CBC3-SHA bits=168 verify=NO); Tue, 14 Dec 2004 05:57:39 +1100 Received: from cirb503493.alcatel.com.au (localhost.alcatel.com.au [127.0.0.1])iBDIvcxP088081; Tue, 14 Dec 2004 05:57:38 +1100 (EST) (envelope-from pjeremy@cirb503493.alcatel.com.au) Received: (from pjeremy@localhost)iBDIvcYr088080; Tue, 14 Dec 2004 05:57:38 +1100 (EST) (envelope-from pjeremy) Date: Tue, 14 Dec 2004 05:57:38 +1100 From: Peter Jeremy To: henry Message-ID: <20041213185738.GD79646@cirb503493.alcatel.com.au> References: <200412131142.iBDBgMp4082138@mail.squidge.com> Mime-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <200412131142.iBDBgMp4082138@mail.squidge.com> User-Agent: Mutt/1.4.2i cc: freebsd-hackers@freebsd.org Subject: Re: du(1)/fts(3) integer overflow X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.1 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 13 Dec 2004 18:57:42 -0000 On Mon, 2004-Dec-13 11:42:17 -0000, henry wrote: >I have noticed a problem with the fts(3) library or the way du(1) interacts >with it. > >A 3.2TiB file gives the following output: >> du -cs /fs/file >3408720016 /fs/file >-886247279 total > >This is because while stat(2) reports blocks as a 64bit number du(1) uses >the 32bit value FTSENT.fts_number to store the result: I think the large filesystem support is insufficiently exercised and there will probably be more of these sort of things lurking around. >The simplest change appears to be to make fts_number 64bit however this >changes the fts(3) abi, so I am not sure if this is acceptable. For 6.x, the ABI isn't fixed so fts_number can be changed to int64_t. 4.x doesn't really support large filesystems due to integer overflow issues in UFS/FFS code so it's not really a problem there. The 5.x ABI is fixed so there's no simple solution there. Possible hacks would be: - Add a new 'fts_number64' at the end of FTSENT. Since FTSENT is always managed by fts(3) and the documentation allows for undocumented fields, this should be permitted, though a "new" du(1) with an "old" libc would break badly. - Move fts_number to the end of FTSENT and leave a 'long' hole where the existing fts_number is. This changes the ABI but old programs remain compatible with the new fts. (Though new programs break with the old fts). - Have du(1) treat fts_pointer as an integer that it can concatenate to fts_number on 32-bit architectures: int64_t x = (int64_t)(ulong)p->fts_parent->fts_number | ((int64_t)(ulong)p->fts_parent->fts_pointer) << 32; x += p->fts_statp->st_blocks; p->fts_parent->fts_number = (long)x; p->fts_parent->fts_pointer = (void *)(long)(x >> 32); etc. -- Peter Jeremy