From owner-freebsd-arch@FreeBSD.ORG Sun Apr 15 09:31:51 2007 Return-Path: X-Original-To: arch@freebsd.org Delivered-To: freebsd-arch@FreeBSD.ORG Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id 369CA16A401 for ; Sun, 15 Apr 2007 09:31:51 +0000 (UTC) (envelope-from dds@aueb.gr) Received: from blue.servers.aueb.gr (blue.servers.aueb.gr [195.251.255.132]) by mx1.freebsd.org (Postfix) with ESMTP id B6D6D13C44B for ; Sun, 15 Apr 2007 09:31:50 +0000 (UTC) (envelope-from dds@aueb.gr) Received: from [127.0.0.1] ([::ffff:195.251.249.147]) by blue.servers.aueb.gr with esmtp; Sun, 15 Apr 2007 12:31:45 +0300 id 000D52FA.4621F101.00001A82 Message-ID: <4621E826.6050306@aueb.gr> Date: Sun, 15 Apr 2007 11:53:58 +0300 From: Diomidis Spinellis User-Agent: Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.8.0.9) Gecko/20061211 SeaMonkey/1.0.7 MIME-Version: 1.0 To: Robert Watson References: <461958CC.4040804@aueb.gr> <20070414170218.M76326@fledge.watson.org> In-Reply-To: <20070414170218.M76326@fledge.watson.org> Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit Cc: arch@freebsd.org, re@freebsd.org Subject: Re: Accounting changes X-BeenThere: freebsd-arch@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Discussion related to FreeBSD architecture List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 15 Apr 2007 09:31:51 -0000 Robert Watson wrote: > > On Mon, 9 Apr 2007, Diomidis Spinellis wrote: > >> I'd like your permission to perform a change during the slush period. >> >> Following some gentle prodding by bde, I'm working on an overhaul of >> the accounting system. The reason for the change is that time values >> with AHZ == 64 (1/64 second) are very often zero on modern machines. >> The change involves storing acct(5) time values as proper IEEE-487 >> floats with AHZ being 1000000 (i.e storing \mu s). For the change >> I'll touch the following files/directories: >> >> sys/sys/acct.h >> sys/kern/kern_acct.c >> usr.sbin/sa >> usr.bin/lastcomm >> share/man/man5/acct.5 >> UPDATING >> >> The change will render old daily accounting files incompatible with >> the new commands and formats. Because the summary databases don't >> store any version information, the old versions of these will also >> become unusable. Therefore users will need to clear >> /var/account{acct*,savacct,usracct}. Apparently, this problem has >> occurred multiple times in the past, when we changed the size of >> elements like uid_t and dev_t. > > What do you think of the idea of changing the file format a little to > include a short file header at the front, and that the first field of > that head is zero-filled u_int32_t, and the second a version number? > Right now, the first field of the acct structure is the name of the > command, which will always be a non-nul string, so always have a first > character non-nul. If we see non-nul data in the file header's first > field, we use the old structure layout, and otherwise we check the > version number and use the new layout? This would provide backwards > compatibility for reading old accounting data, which I would think would > generally be desirable, and allow us to explicitly version the file in > the future. There are three types of accounting files: acct(5), which contains records written by the kernel, and two others containing sa(8) aggregated results of acct per user and per command. I'd prefer to keep acct(5) free of version data. This 1) simplifies the kernel, which would otherwise have to write version data each time the file is turned over, 2) is consistent with current practice for storing such files, and 3) simplifies the processing programs, which can process data at arbritrary locations in the file. (lastcomm reads records backwards, and recently I updated it to also work on the output of tail -f). Typically, these files are aggregated and turned over daily; IMO doing this also over an OS upgrade isn't a big deal. The aggregated files are different. They are dbm-based, and their structure is opaque. I could therefore easily add a special "file version" record to them. My concern is how to handle multiple file versions over a long time period. Hard-coding legacy file formats in the code is inelegant. I'd prefer to add a text-based import/export capability, so that administrators can migrate data by exporting it before an OS upgrade and importing it afterwords. Even that could be an overkill. Based on the problems Bruce and I have seen in the accounting system, my impression is that not many people use it nowadays. Once/if we get dtrace in FreeBSD, I'd prever to scrap t existing accounting infrastructure and re-implement it on top of that. Diomidis - http://www.spinellis.gr