From owner-freebsd-questions@freebsd.org Thu Jun 17 19:12:04 2021 Return-Path: Delivered-To: freebsd-questions@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id EE55E659CEF for ; Thu, 17 Jun 2021 19:12:04 +0000 (UTC) (envelope-from rfg@tristatelogic.com) Received: from outgoing.tristatelogic.com (segfault.tristatelogic.com [69.62.255.118]) by mx1.freebsd.org (Postfix) with ESMTP id 4G5Wrg6JBPz3jLP for ; Thu, 17 Jun 2021 19:12:03 +0000 (UTC) (envelope-from rfg@tristatelogic.com) Received: by segfault.tristatelogic.com (Postfix, from userid 1237) id D1C924E680; Thu, 17 Jun 2021 12:12:01 -0700 (PDT) From: "Ronald F. Guilmette" To: freeBSD Mailing List Subject: Re: Is a successful call to write(2) atomic? In-Reply-To: <7801a6e8-2428-619e-5722-7317841595a1@qeng-ho.org> MIME-Version: 1.0 Content-Type: text/plain; charset="us-ascii" Content-ID: <36326.1623957121.1@segfault.tristatelogic.com> Date: Thu, 17 Jun 2021 12:12:01 -0700 Message-ID: <36327.1623957121@segfault.tristatelogic.com> X-Rspamd-Queue-Id: 4G5Wrg6JBPz3jLP X-Spamd-Bar: / Authentication-Results: mx1.freebsd.org; dkim=none; dmarc=none; spf=pass (mx1.freebsd.org: domain of rfg@tristatelogic.com designates 69.62.255.118 as permitted sender) smtp.mailfrom=rfg@tristatelogic.com X-Spamd-Result: default: False [-0.51 / 15.00]; ARC_NA(0.00)[]; RBL_DBL_DONT_QUERY_IPS(0.00)[69.62.255.118:from]; NEURAL_HAM_MEDIUM(-0.99)[-0.992]; FROM_HAS_DN(0.00)[]; TO_MATCH_ENVRCPT_ALL(0.00)[]; R_SPF_ALLOW(-0.20)[+mx]; MIME_GOOD(-0.10)[text/plain]; DMARC_NA(0.00)[tristatelogic.com]; NEURAL_HAM_LONG(-1.00)[-1.000]; RCPT_COUNT_ONE(0.00)[1]; SPAMHAUS_ZRD(0.00)[69.62.255.118:from:127.0.2.255]; TO_DN_ALL(0.00)[]; NEURAL_SPAM_SHORT(0.78)[0.777]; RCVD_COUNT_ZERO(0.00)[0]; FROM_EQ_ENVFROM(0.00)[]; R_DKIM_NA(0.00)[]; MIME_TRACE(0.00)[0:+]; ASN(0.00)[asn:14051, ipnet:69.62.128.0/17, country:US]; SUBJECT_ENDS_QUESTION(1.00)[]; MAILMAN_DEST(0.00)[freebsd-questions] X-BeenThere: freebsd-questions@freebsd.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: User questions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 17 Jun 2021 19:12:05 -0000 Update: I need to say "Thank you all!" to all of the folks who chimed in and who have graciously tried to help me to get this problem sorted out. Although I didn't try the idea of passing buffers/lines back up from the "worker bee" child processes to the parent (which would then write those lines to FD 1 in a nice sequentialized fashion) I did try all of the other suggesions that have been made here, specifically (a) setting O_APPEND on the FD, and also (b) wrapping my calls to write() in calls to flock() -and- also (c) embedding in that flock section an additional call to lseek() to position next following write, reliably, at the current actual EOF of the output file. Every step along the way, the "garbled output lines" problem seemed to get smaller, but at the end, it was still present. So I added some more code to do more & better checking of the integrity of the data lines before I even tried to write them to FD #1. And sure enough, once I had that additional "input sanitizing" code in place, the problem went away entirely. Now I'm re-running the whole process again, but with all of the extra stuff that people suggested to me, i.e. the code that sets O_APPEND on the FD, the calls to flock() and the call to lseek() all #ifdef'd out. The results aren't final yet, but for the first 18,000+ lines of output, the "garbled output lines" problem does indeed seem to be gone entirely. So, bottom line, it now appears that I owe everyone an apology for taking up your time. My bad! It now appears that the -actual- problem was most probably related to "logic errors" in my own code, and more specifically my failure to adequately sanitize input data (which is being acquired from remote network hosts that I do not control) before even calling write(). I'm suitably embarassed to admit this, but I feel morally compelled to confess my sins and to take responsibility for my own screw ups. In any case, this has all been quite educational, and for that I thank you all. Regards, rfg