From owner-freebsd-hackers@FreeBSD.ORG Sun Mar 2 11:44:02 2008 Return-Path: Delivered-To: freebsd-hackers@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id F3EF2106566B for ; Sun, 2 Mar 2008 11:44:01 +0000 (UTC) (envelope-from rwatson@FreeBSD.org) Received: from cyrus.watson.org (cyrus.watson.org [209.31.154.42]) by mx1.freebsd.org (Postfix) with ESMTP id D22EF8FC25 for ; Sun, 2 Mar 2008 11:44:01 +0000 (UTC) (envelope-from rwatson@FreeBSD.org) Received: from fledge.watson.org (fledge.watson.org [209.31.154.41]) by cyrus.watson.org (Postfix) with ESMTP id 09DDD46B10; Sun, 2 Mar 2008 06:44:01 -0500 (EST) Date: Sun, 2 Mar 2008 11:44:01 +0000 (GMT) From: Robert Watson X-X-Sender: robert@fledge.watson.org To: Martin Laabs In-Reply-To: Message-ID: <20080302113704.B5198@fledge.watson.org> References: MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII; format=flowed Cc: "freebsd-hackers@freebsd.org" Subject: Re: multi volume dump with gzip/crypt - solved X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 02 Mar 2008 11:44:02 -0000 On Sat, 1 Mar 2008, Martin Laabs wrote: > I solved the dump problem in a not very clean way. First I changed dumps > behaviour to ignore the SIGPIPE signal when the -P and -a option is given. > The problem with occured than was the loss of data because of the race > condition between "write date to pipe" and "receive SIGPIPE". > > I solved that problem with a script that takes the data from the pipe of > dump, compress it and sends a dedicated SIGPIPE to the three dump processes > when the output volume is nealy full. After that it flushs the rest out of > the pipe to avoid data loss. SIGPIPE occurs when a sending process sends data that the recipient doesn't expect, as the recipient has (presumably) closed the socket because it doesn't expect further data. Stream sockets are intended to provided buffered, reliable communication only when open, so generally protocols running over stream sockets (be it TCP, UNIX domain sockets, etc) have an explicit protocol close sequence so that the two endpoints reach mutual agreement that it's time to close the sockets and avoid data loss associated with unexpected assymetric close. If the recipient closed the socket while the sender is still writing to the socket, you may "lose" up to the size of the receive socket buffer, which from the perspective of the sender may have been reported as successfully delivered. The receive socket buffer, FYI, is effectively the amount of data that the recipient has agreed can be locally buffered before causing the sender to block. Robert N M Watson Computer Laboratory University of Cambridge