From owner-freebsd-current@freebsd.org Thu Mar 9 00:09:51 2017 Return-Path: Delivered-To: freebsd-current@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 2303DD02DD5 for ; Thu, 9 Mar 2017 00:09:51 +0000 (UTC) (envelope-from devin@shxd.cx) Received: from shxd.cx (mail.shxd.cx [64.201.244.140]) (using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 14D6015CF; Thu, 9 Mar 2017 00:09:51 +0000 (UTC) (envelope-from devin@shxd.cx) Received: from [74.217.198.10] (port=34966 helo=[10.1.4.66]) by shxd.cx with esmtpsa (TLSv1:AES256-SHA:256) (Exim 4.77 (FreeBSD)) (envelope-from ) id 1cl3iK-000J1f-1l; Tue, 07 Mar 2017 01:14:56 +0000 Content-Type: text/plain; charset=windows-1252 Mime-Version: 1.0 (Mac OS X Mail 9.3 \(3124\)) Subject: Re: I/O semantics of pipe and FIFO. From: Devin Teske In-Reply-To: <17381773-019a-2181-f00f-1908d04b8d22@freebsd.org> Date: Wed, 8 Mar 2017 16:09:48 -0800 Cc: Devin Teske , Julian Elischer , freebsd-current Content-Transfer-Encoding: quoted-printable Message-Id: References: <20170304214812.GA16845@chaz.gmail.com> <8efdc961-1768-0bc0-715f-4a1e103359d4@elischer.org> <17381773-019a-2181-f00f-1908d04b8d22@freebsd.org> To: Alfred Perlstein X-Mailer: Apple Mail (2.3124) Sender: devin@shxd.cx X-BeenThere: freebsd-current@freebsd.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: Discussions about the use of FreeBSD-current List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 09 Mar 2017 00:09:51 -0000 Problem we had found was: Executing dd with a closed stdout and stderr would cause the summary = messages printed at the end to go into the destination output file. For example, dd if=3D/dev/zero of=3D/tmp/foo bs=3D1m count=3D1 Works fine, but the following: dd if=3D/dev/zero of=3D/tmp/foo bs=3D1m count=3D1 >&- 2>&- Will cause the summary statistics of dd to appear in /tmp/foo instead of = on the console. The issue is that the summary statistics are send to fd1, which if you = close down stdout and stdin, fd1 is actually the output file since it = got the lowest file descriptor available when open(2) was called on the = output file. This was never fixed because it was deemed =93silly developer, don=92t = close stdout and stderr before invoking dd=94. The argument has been made by Jilles T. that it is generally a bad idea = to close down any of the standard file descriptors because it cannot be = predicted how a particular UNIX utility will react (e.g., in the case of = dd, causing a simple printf(3) to go to an unexpected location). =97=20 Devin > On Mar 4, 2017, at 8:12 PM, Alfred Perlstein = wrote: >=20 > Devin and I found this when we worked together. I think it was due to = some situation in dd(1) where short reads would exit pre-maturely, = however I may be mis-remembering. Devin, do you recall the specifics? >=20 >=20 > On 3/4/17 7:44 PM, Julian Elischer wrote: >>=20 >> an interesting point to discuss? is our behaviour in this test right? >> from: "austin-group mailng list (posix standard discussion)" >>=20 >> ------ rest of email is quoted ------- >> On 5/3/17 5:48 am, Stephane Chazelas wrote: >>=20 >> 2017-03-04 13:14:08 +0000, Danny Niu: >>> Hi all. >>>=20 >>> I couldn't remember where I saw it saying, that when reading >>> from a pipe or a FIFO, the read syscall returns the content of >>> at most one write call. It's a bit similar to the >>> message-nondiscard semantics of dear old STREAM. >>>=20 >>> Currently, I'm reading through the text to find out a bit >>> more, and I appreciate a bit of pointer on this. >> [...] >>=20 >> (echo x; echo y) | (sleep 1; dd count=3D1 2> /dev/null) >>=20 >> outputs both x and y in all of Linux, FreeBSD and Solaris in my >> tests. >>=20 >> That a read wouldn't read what's currently in the pipe would be >> quite surprising. >>=20 >> I also wouldn't expect pipes to store the writes as individual >> separate message but use one buffer. >>=20 >> In: >>=20 >> ( >> dd bs=3D40000 count=3D1 if=3D/dev/zero 2> /dev/null >> echo first through >&2 >> dd bs=3D40000 count=3D1 if=3D/dev/zero 2> /dev/null >> echo second through >&2 >> ) | (sleep 1; dd bs=3D100000 count=3D1 2> /dev/null) | wc -c >>=20 >> That is where the second write blocks because the pipe is full, >> the reading dd still reads both writes in Linux and Solaris in >> my tests (on Solaris (10 on amd64 at least), reduce to 20000 >> instead of 40000 or both writes would block). >>=20 >> On FreeBSD, I get only the first write (using 8000 followed by >> 10000 for instance). >>=20 >> FreeBSD is also the only one of the three where >>=20 >> dd bs=3D1000000 count=3D1 if=3D/dev/zero | dd bs=3D1000000 count=3D1 = | wc -c >>=20 >> Doesn't output 1000000. The others schedule both processes back >> and forth during their write() and read() system call while the >> pipe is being filled and emptied several times. >>=20 >=20