From owner-freebsd-current@freebsd.org Thu Mar 9 02:47:31 2017 Return-Path: Delivered-To: freebsd-current@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id C9F4DD03B4C for ; Thu, 9 Mar 2017 02:47:31 +0000 (UTC) (envelope-from alfred@freebsd.org) Received: from elvis.mu.org (elvis.mu.org [192.203.228.196]) by mx1.freebsd.org (Postfix) with ESMTP id AD057D49; Thu, 9 Mar 2017 02:47:31 +0000 (UTC) (envelope-from alfred@freebsd.org) Received: from [IPv6:2601:645:8003:a4d6:1cb9:f89b:6c3a:dc8f] (unknown [IPv6:2601:645:8003:a4d6:1cb9:f89b:6c3a:dc8f]) by elvis.mu.org (Postfix) with ESMTPSA id 2D288346DDF5; Wed, 8 Mar 2017 18:47:25 -0800 (PST) Content-Type: text/plain; charset=utf-8 Mime-Version: 1.0 (1.0) Subject: Re: I/O semantics of pipe and FIFO. From: Alfred Perlstein X-Mailer: iPhone Mail (14D27) In-Reply-To: Date: Wed, 8 Mar 2017 18:47:24 -0800 Cc: Julian Elischer , freebsd-current Content-Transfer-Encoding: quoted-printable Message-Id: <819671BE-4700-4E73-A697-095BBDE287D4@freebsd.org> References: <20170304214812.GA16845@chaz.gmail.com> <8efdc961-1768-0bc0-715f-4a1e103359d4@elischer.org> <17381773-019a-2181-f00f-1908d04b8d22@freebsd.org> To: Devin Teske X-BeenThere: freebsd-current@freebsd.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: Discussions about the use of FreeBSD-current List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 09 Mar 2017 02:47:31 -0000 I've seen that bug, but I think our bug was more that dd would exit if it go= t back a short read from input. So if you did something like this (maybe not= exactly, but close): dd bs=3D1m | dd bs=3D10m > f.out You might get only 1mb in f.out. This had to do with skipping disklabels and= other such schenanigans.=20 Sent from my iPhone > On Mar 8, 2017, at 4:09 PM, Devin Teske wrote: >=20 > Problem we had found was: >=20 > Executing dd with a closed stdout and stderr would cause the summary messa= ges printed at the end to go into the destination output file. >=20 > For example, >=20 > dd if=3D/dev/zero of=3D/tmp/foo bs=3D1m count=3D1 >=20 > Works fine, but the following: >=20 > dd if=3D/dev/zero of=3D/tmp/foo bs=3D1m count=3D1 >&- 2>&- >=20 > Will cause the summary statistics of dd to appear in /tmp/foo instead of o= n the console. >=20 > The issue is that the summary statistics are send to fd1, which if you clo= se down stdout and stdin, fd1 is actually the output file since it got the l= owest file descriptor available when open(2) was called on the output file. >=20 > This was never fixed because it was deemed =E2=80=9Csilly developer, don=E2= =80=99t close stdout and stderr before invoking dd=E2=80=9D. >=20 > The argument has been made by Jilles T. that it is generally a bad idea to= close down any of the standard file descriptors because it cannot be predic= ted how a particular UNIX utility will react (e.g., in the case of dd, causi= ng a simple printf(3) to go to an unexpected location). > =E2=80=94=20 > Devin >=20 >=20 >> On Mar 4, 2017, at 8:12 PM, Alfred Perlstein wrote: >>=20 >> Devin and I found this when we worked together. I think it was due to so= me situation in dd(1) where short reads would exit pre-maturely, however I m= ay be mis-remembering. Devin, do you recall the specifics? >>=20 >>=20 >>> On 3/4/17 7:44 PM, Julian Elischer wrote: >>>=20 >>> an interesting point to discuss? is our behaviour in this test right? >>> from: "austin-group mailng list (posix standard discussion)" >>>=20 >>> ------ rest of email is quoted ------- >>> On 5/3/17 5:48 am, Stephane Chazelas wrote: >>>=20 >>> 2017-03-04 13:14:08 +0000, Danny Niu: >>>> Hi all. >>>>=20 >>>> I couldn't remember where I saw it saying, that when reading >>>> from a pipe or a FIFO, the read syscall returns the content of >>>> at most one write call. It's a bit similar to the >>>> message-nondiscard semantics of dear old STREAM. >>>>=20 >>>> Currently, I'm reading through the text to find out a bit >>>> more, and I appreciate a bit of pointer on this. >>> [...] >>>=20 >>> (echo x; echo y) | (sleep 1; dd count=3D1 2> /dev/null) >>>=20 >>> outputs both x and y in all of Linux, FreeBSD and Solaris in my >>> tests. >>>=20 >>> That a read wouldn't read what's currently in the pipe would be >>> quite surprising. >>>=20 >>> I also wouldn't expect pipes to store the writes as individual >>> separate message but use one buffer. >>>=20 >>> In: >>>=20 >>> ( >>> dd bs=3D40000 count=3D1 if=3D/dev/zero 2> /dev/null >>> echo first through >&2 >>> dd bs=3D40000 count=3D1 if=3D/dev/zero 2> /dev/null >>> echo second through >&2 >>> ) | (sleep 1; dd bs=3D100000 count=3D1 2> /dev/null) | wc -c >>>=20 >>> That is where the second write blocks because the pipe is full, >>> the reading dd still reads both writes in Linux and Solaris in >>> my tests (on Solaris (10 on amd64 at least), reduce to 20000 >>> instead of 40000 or both writes would block). >>>=20 >>> On FreeBSD, I get only the first write (using 8000 followed by >>> 10000 for instance). >>>=20 >>> FreeBSD is also the only one of the three where >>>=20 >>> dd bs=3D1000000 count=3D1 if=3D/dev/zero | dd bs=3D1000000 count=3D1 | w= c -c >>>=20 >>> Doesn't output 1000000. The others schedule both processes back >>> and forth during their write() and read() system call while the >>> pipe is being filled and emptied several times. >>>=20 >>=20 >=20