From owner-freebsd-questions@freebsd.org Tue Jun 15 18:50:01 2021 Return-Path: Delivered-To: freebsd-questions@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id 7A8E16504FF for ; Tue, 15 Jun 2021 18:50:01 +0000 (UTC) (envelope-from kh@panix.com) Received: from mailbackend.panix.com (mailbackend.panix.com [166.84.1.89]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 4G4HS86TzPz4rjm for ; Tue, 15 Jun 2021 18:50:00 +0000 (UTC) (envelope-from kh@panix.com) Received: from rain.home (pool-96-230-243-2.bstnma.fios.verizon.net [96.230.243.2]) by mailbackend.panix.com (Postfix) with ESMTPSA id 4G4HS74Qc5z3jFn for ; Tue, 15 Jun 2021 14:49:59 -0400 (EDT) DKIM-Signature: v=1; a=rsa-sha256; c=simple/simple; d=panix.com; s=panix; t=1623782999; bh=8DL0L6kR1WB2FIqEVgiUUArL7yqIHheoKYJq3k/qL7U=; h=Subject:To:References:From:Date:In-Reply-To; b=MC/cYKqnaUrewxkpSzaoX5vT+VJdQRpKr/QyQZpYhivX4xykibl4C8fzva1PJMlzn FKkKDiApV974MkJkdXYQwHYwOF17ilj2BytJcm/bpA7wwy3wLLJskRCWCTIYg9lhlN b6WbJWLV1yNh6aFw5FnIz8rP72oio2YZ+HdgWOjc= Subject: Re: Is a successful call to write(2) atomic? To: freebsd-questions@freebsd.org References: <22440.1623740785@segfault.tristatelogic.com> From: Kurt Hackenberg Message-ID: <44e15917-0c92-08f2-462e-a1b3705f9afb@panix.com> Date: Tue, 15 Jun 2021 14:49:57 -0400 User-Agent: Mozilla/5.0 (X11; FreeBSD amd64; rv:78.0) Gecko/20100101 Thunderbird/78.6.0 MIME-Version: 1.0 In-Reply-To: <22440.1623740785@segfault.tristatelogic.com> Content-Type: text/plain; charset=utf-8; format=flowed Content-Language: en-US Content-Transfer-Encoding: 7bit X-Rspamd-Queue-Id: 4G4HS86TzPz4rjm X-Spamd-Bar: ---- Authentication-Results: mx1.freebsd.org; dkim=pass header.d=panix.com header.s=panix header.b=MC/cYKqn; dmarc=none; spf=pass (mx1.freebsd.org: domain of kh@panix.com designates 166.84.1.89 as permitted sender) smtp.mailfrom=kh@panix.com X-Spamd-Result: default: False [-4.20 / 15.00]; RCVD_VIA_SMTP_AUTH(0.00)[]; RWL_MAILSPIKE_GOOD(0.00)[166.84.1.89:from]; R_SPF_ALLOW(-0.20)[+ip4:166.84.0.0/16]; TO_DN_NONE(0.00)[]; RCVD_DKIM_ARC_DNSWL_MED(-0.50)[]; RCVD_IN_DNSWL_MED(-0.20)[166.84.1.89:from]; DKIM_TRACE(0.00)[panix.com:+]; NEURAL_HAM_SHORT(-1.00)[-1.000]; FROM_EQ_ENVFROM(0.00)[]; MIME_TRACE(0.00)[0:+]; SUBJECT_ENDS_QUESTION(1.00)[]; ASN(0.00)[asn:2033, ipnet:166.84.0.0/16, country:US]; MID_RHS_MATCH_FROM(0.00)[]; RECEIVED_SPAMHAUS_PBL(0.00)[96.230.243.2:received]; ARC_NA(0.00)[]; NEURAL_HAM_MEDIUM(-1.00)[-1.000]; R_DKIM_ALLOW(-0.20)[panix.com:s=panix]; FROM_HAS_DN(0.00)[]; TO_MATCH_ENVRCPT_ALL(0.00)[]; NEURAL_HAM_LONG(-1.00)[-1.000]; MIME_GOOD(-0.10)[text/plain]; PREVIOUSLY_DELIVERED(0.00)[freebsd-questions@freebsd.org]; DMARC_NA(0.00)[panix.com]; RCPT_COUNT_ONE(0.00)[1]; DWL_DNSWL_LOW(-1.00)[panix.com:dkim]; RCVD_COUNT_TWO(0.00)[2]; RCVD_TLS_ALL(0.00)[]; MAILMAN_DEST(0.00)[freebsd-questions] X-BeenThere: freebsd-questions@freebsd.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: User questions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 15 Jun 2021 18:50:01 -0000 On 2021/06/15 03:06, Ronald F. Guilmette wrote: > This is deeply distrubing. I never knew about this. I am furiously reading > the FreeBSD & Linux man pages for open(2). (It appears that both support > that flags, O_DIRECT and O_SYNC, aqnd Linux also adds the O_DSYNC flag. No, write(2) is not guaranteed atomic, and that's not obvious. Probably a lot of people have learned that the hard way. All I know about those flags is what I just read in the man page, but I think they don't guarantee atomic writes either -- at least, they're not intended to. The sync stuff means the system call won't return until the data has been written to disk (as opposed to queuing the write to be executed sometime later). That doesn't say anything about interleaving with other processes. Direct -- non-cached -- sounds like sort of the same thing, but at a different level, and it isn't even guaranteed to do that. The man page doesn't say much. Either of those might slow things down a lot, without necessarily solving your problem. Either might make the problem less likely to occur, and so harder to debug. I suggest that you don't use those flags for this purpose. This is an old, general problem: concurrent access to a shared resource. There are two common solutions. Paul suggested one of them: serialize access through a single process. The other is to serialize it through some kind of lock, that can only be held by one process at a time. Each process acquires the lock, uses the shared resource briefly, and immediately releases the lock. Semaphores are that kind of lock, invented for that purpose. Unix file systems have file locking, which does that for all or part of a file; see fcntl(2). Note that Unix file locking is advisory -- voluntary -- not compulsory. It only works if all the processes agree to use it. Also, in the past, it has not always worked for files accessed across the network, through NFS. I don't know whether it works through modern NFS. It's best to use it only for local files. Both approaches work fine, if they're done correctly; they're both a little complicated to implement. Getting it right requires a clear understanding of the problem and the solution. Sounds like you have the idea now.