Date: Wed, 8 Mar 2017 16:09:48 -0800 From: Devin Teske <dteske@freebsd.org> To: Alfred Perlstein <alfred@freebsd.org> Cc: Devin Teske <dteske@freebsd.org>, Julian Elischer <julian@elischer.org>, freebsd-current <freebsd-current@freebsd.org> Subject: Re: I/O semantics of pipe and FIFO. Message-ID: <B71629F9-A171-4E01-BA0A-39F74C77724B@freebsd.org> In-Reply-To: <17381773-019a-2181-f00f-1908d04b8d22@freebsd.org> References: <20170304214812.GA16845@chaz.gmail.com> <8efdc961-1768-0bc0-715f-4a1e103359d4@elischer.org> <17381773-019a-2181-f00f-1908d04b8d22@freebsd.org>
next in thread | previous in thread | raw e-mail | index | archive | help
Problem we had found was: Executing dd with a closed stdout and stderr would cause the summary = messages printed at the end to go into the destination output file. For example, dd if=3D/dev/zero of=3D/tmp/foo bs=3D1m count=3D1 Works fine, but the following: dd if=3D/dev/zero of=3D/tmp/foo bs=3D1m count=3D1 >&- 2>&- Will cause the summary statistics of dd to appear in /tmp/foo instead of = on the console. The issue is that the summary statistics are send to fd1, which if you = close down stdout and stdin, fd1 is actually the output file since it = got the lowest file descriptor available when open(2) was called on the = output file. This was never fixed because it was deemed =93silly developer, don=92t = close stdout and stderr before invoking dd=94. The argument has been made by Jilles T. that it is generally a bad idea = to close down any of the standard file descriptors because it cannot be = predicted how a particular UNIX utility will react (e.g., in the case of = dd, causing a simple printf(3) to go to an unexpected location). =97=20 Devin > On Mar 4, 2017, at 8:12 PM, Alfred Perlstein <alfred@freebsd.org> = wrote: >=20 > Devin and I found this when we worked together. I think it was due to = some situation in dd(1) where short reads would exit pre-maturely, = however I may be mis-remembering. Devin, do you recall the specifics? >=20 >=20 > On 3/4/17 7:44 PM, Julian Elischer wrote: >>=20 >> an interesting point to discuss? is our behaviour in this test right? >> from: "austin-group mailng list (posix standard discussion)" >>=20 >> ------ rest of email is quoted ------- >> On 5/3/17 5:48 am, Stephane Chazelas wrote: >>=20 >> 2017-03-04 13:14:08 +0000, Danny Niu: >>> Hi all. >>>=20 >>> I couldn't remember where I saw it saying, that when reading >>> from a pipe or a FIFO, the read syscall returns the content of >>> at most one write call. It's a bit similar to the >>> message-nondiscard semantics of dear old STREAM. >>>=20 >>> Currently, I'm reading through the text to find out a bit >>> more, and I appreciate a bit of pointer on this. >> [...] >>=20 >> (echo x; echo y) | (sleep 1; dd count=3D1 2> /dev/null) >>=20 >> outputs both x and y in all of Linux, FreeBSD and Solaris in my >> tests. >>=20 >> That a read wouldn't read what's currently in the pipe would be >> quite surprising. >>=20 >> I also wouldn't expect pipes to store the writes as individual >> separate message but use one buffer. >>=20 >> In: >>=20 >> ( >> dd bs=3D40000 count=3D1 if=3D/dev/zero 2> /dev/null >> echo first through >&2 >> dd bs=3D40000 count=3D1 if=3D/dev/zero 2> /dev/null >> echo second through >&2 >> ) | (sleep 1; dd bs=3D100000 count=3D1 2> /dev/null) | wc -c >>=20 >> That is where the second write blocks because the pipe is full, >> the reading dd still reads both writes in Linux and Solaris in >> my tests (on Solaris (10 on amd64 at least), reduce to 20000 >> instead of 40000 or both writes would block). >>=20 >> On FreeBSD, I get only the first write (using 8000 followed by >> 10000 for instance). >>=20 >> FreeBSD is also the only one of the three where >>=20 >> dd bs=3D1000000 count=3D1 if=3D/dev/zero | dd bs=3D1000000 count=3D1 = | wc -c >>=20 >> Doesn't output 1000000. The others schedule both processes back >> and forth during their write() and read() system call while the >> pipe is being filled and emptied several times. >>=20 >=20
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?B71629F9-A171-4E01-BA0A-39F74C77724B>