From owner-svn-src-all@freebsd.org Thu Feb 11 12:42:13 2016 Return-Path: Delivered-To: svn-src-all@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id D265FAA4F67 for ; Thu, 11 Feb 2016 12:42:13 +0000 (UTC) (envelope-from se@freebsd.org) Received: from mailout05.t-online.de (mailout05.t-online.de [194.25.134.82]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client CN "mailout00.t-online.de", Issuer "TeleSec ServerPass DE-1" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 636FC3B5; Thu, 11 Feb 2016 12:42:13 +0000 (UTC) (envelope-from se@freebsd.org) Received: from fwd08.aul.t-online.de (fwd08.aul.t-online.de [172.20.26.151]) by mailout05.t-online.de (Postfix) with SMTP id E7490D3767; Thu, 11 Feb 2016 13:33:26 +0100 (CET) Received: from [192.168.119.17] (ZGWexkZcQhriTiz5WEu+MWjVwusWtpQFx+Mf3qnhNqm3kmt0JjKbk0-5-zTpooZwZp@[87.151.208.3]) by fwd08.t-online.de with (TLSv1.2:ECDHE-RSA-AES256-SHA encrypted) esmtp id 1aTqR2-38BHGK0; Thu, 11 Feb 2016 13:33:24 +0100 Subject: Re: svn commit: r295465 - head/usr.sbin/services_mkdb To: svn-src-all@freebsd.org References: <201602100940.u1A9ejb6086175@repo.freebsd.org> <20160211091516.GA37735@walton.maths.tcd.ie> From: Stefan Esser X-Enigmail-Draft-Status: N1110 Message-ID: <56BC7F91.6060207@freebsd.org> Date: Thu, 11 Feb 2016 13:33:21 +0100 User-Agent: Mozilla/5.0 (Windows NT 6.1; WOW64; rv:38.0) Gecko/20100101 Thunderbird/38.5.1 MIME-Version: 1.0 In-Reply-To: <20160211091516.GA37735@walton.maths.tcd.ie> Content-Type: text/plain; charset=windows-1252 Content-Transfer-Encoding: 7bit X-ID: ZGWexkZcQhriTiz5WEu+MWjVwusWtpQFx+Mf3qnhNqm3kmt0JjKbk0-5-zTpooZwZp X-TOI-MSGID: dd33142c-2ea2-45ef-b22f-7d625b076222 X-BeenThere: svn-src-all@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: "SVN commit messages for the entire src tree \(except for " user" and " projects" \)" List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 11 Feb 2016 12:42:14 -0000 Am 11.02.2016 um 10:15 schrieb David Malone: > On Wed, Feb 10, 2016 at 09:40:45AM +0000, Stefan Esser wrote: >> Author: se >> Date: Wed Feb 10 09:40:45 2016 >> New Revision: 295465 >> URL: https://svnweb.freebsd.org/changeset/base/295465 >> >> Log: >> Remove O_SYNC from the options passed to dbmopen(). >> >> The services db is created as a temporary file that is moved over the >> existing file after completion. Thus there is no need to immediately >> flush all created db records to the temporary file. > > This was changed to fix a problem with fsync not being called, which > can leave the db files empty after a sudden reboot. O_SYNC is not > the right fix, but I think I've found the real problem and there > is an open phabricator report on this, which fixes the problem in > a better way: > > https://reviews.freebsd.org/D5186 I added the following information as a comment to D5168: - I think that buffers are flushed in hdestroy() in hash.c, at least there is a comment that seems to indicate this. - Tests showed that the _mdb commands create bitwise identical files with or without O_SYNC (and without the _fsync() calls you suggest in D5168). - The cap_mkdb command does also use dbopen(O_SYNC) and completes in less than 1 second instead of more than 25 with O_SYNC removed. BTW there is one further problem in services_mkdb: There is a check for the existence of the temporary output file and if a second instance of services_mkdb is started, it will terminate, but only *after unlinking the other instance's temp output file*. Due to the speed-up without O_SYNC the time window is much smaller now, but I can easily provoke this even with the faster version: # services_mkdb -qo /tmp/xx.db & services_mkdb -qo /tmp/xx.db [2] 71869 services_mkdb: Error opening temporary database `/tmp/xx.db.tmp': File exists # services_mkdb-SYNC: Cannot rename `/tmp/xx.db.tmp' to `/tmp/xx.db': No such file or directory [2]- Exit 1 services_mkdb -qo /tmp/xx.db The second process detects that the temp output file exists complains and deletes it. When the first process completed the creation of the db, it tries to rename the file (that has already been removed by the second command). A better approach might be to have the second process unlink the temp file, but then to continue after creation of its own temp file. Then the first process will still find that it cannot rename the temp file to its final name, but the second one will complete its temp file and then rename it to its final name. Regards, STefan