Date: Thu, 17 Jun 2021 12:12:01 -0700 From: "Ronald F. Guilmette" <rfg@tristatelogic.com> To: freeBSD Mailing List <freebsd-questions@freebsd.org> Subject: Re: Is a successful call to write(2) atomic? Message-ID: <36327.1623957121@segfault.tristatelogic.com> In-Reply-To: <7801a6e8-2428-619e-5722-7317841595a1@qeng-ho.org>
next in thread | previous in thread | raw e-mail | index | archive | help
Update: I need to say "Thank you all!" to all of the folks who chimed in and who have graciously tried to help me to get this problem sorted out. Although I didn't try the idea of passing buffers/lines back up from the "worker bee" child processes to the parent (which would then write those lines to FD 1 in a nice sequentialized fashion) I did try all of the other suggesions that have been made here, specifically (a) setting O_APPEND on the FD, and also (b) wrapping my calls to write() in calls to flock() -and- also (c) embedding in that flock section an additional call to lseek() to position next following write, reliably, at the current actual EOF of the output file. Every step along the way, the "garbled output lines" problem seemed to get smaller, but at the end, it was still present. So I added some more code to do more & better checking of the integrity of the data lines before I even tried to write them to FD #1. And sure enough, once I had that additional "input sanitizing" code in place, the problem went away entirely. Now I'm re-running the whole process again, but with all of the extra stuff that people suggested to me, i.e. the code that sets O_APPEND on the FD, the calls to flock() and the call to lseek() all #ifdef'd out. The results aren't final yet, but for the first 18,000+ lines of output, the "garbled output lines" problem does indeed seem to be gone entirely. So, bottom line, it now appears that I owe everyone an apology for taking up your time. My bad! It now appears that the -actual- problem was most probably related to "logic errors" in my own code, and more specifically my failure to adequately sanitize input data (which is being acquired from remote network hosts that I do not control) before even calling write(). I'm suitably embarassed to admit this, but I feel morally compelled to confess my sins and to take responsibility for my own screw ups. In any case, this has all been quite educational, and for that I thank you all. Regards, rfg
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?36327.1623957121>