From owner-freebsd-ports@freebsd.org Tue May 10 12:35:50 2016 Return-Path: Delivered-To: freebsd-ports@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 6B86FB35B57 for ; Tue, 10 May 2016 12:35:50 +0000 (UTC) (envelope-from mad@madpilot.net) Received: from mail.madpilot.net (grunt.madpilot.net [78.47.145.38]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 2E8B61C8B for ; Tue, 10 May 2016 12:35:49 +0000 (UTC) (envelope-from mad@madpilot.net) Received: from mail (mail [192.168.254.3]) by mail.madpilot.net (Postfix) with ESMTP id 3r3zHd1ZXbzZxV for ; Tue, 10 May 2016 14:35:41 +0200 (CEST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=madpilot.net; h= content-transfer-encoding:content-type:content-type:in-reply-to :mime-version:user-agent:date:date:message-id:from:from :references:subject:subject:received:received; s=mail; t= 1462883735; x=1464698136; bh=RWgUdBGVrsE3M1ypfmCbXEmMLfiuxehupYc CZGwBuSE=; b=axazCR8RSMI2M95fKB6gHkXlIepsojr7jsLVAIzw0uhHQPm663S HmzqCOPt9ADppBOdrApU9/itDjp259C75mMXlvFZT/uG0tCL5/A0kWRg73q+3b5S LY57MiGASHH8DWolCKwGSyD+5uJWRBFNV9nff/lHmTNGnb6y07rxqo04= Received: from mail.madpilot.net ([192.168.254.3]) by mail (mail.madpilot.net [192.168.254.3]) (amavisd-new, port 10024) with ESMTP id 82mAfN13k_So for ; Tue, 10 May 2016 14:35:35 +0200 (CEST) Received: from tommy.madpilot.net (micro.madpilot.net [88.149.173.206]) by mail.madpilot.net (Postfix) with ESMTPSA for ; Tue, 10 May 2016 14:35:35 +0200 (CEST) Subject: Re: Poudriere question To: freebsd-ports@freebsd.org References: <3557cbcd-3992-5db5-c5dc-7912508e1956@madpilot.net> <20160510123517.2107653b@gumby.homeunix.com> From: Guido Falsi Message-ID: Date: Tue, 10 May 2016 14:35:35 +0200 User-Agent: Mozilla/5.0 (X11; FreeBSD amd64; rv:45.0) Gecko/20100101 Thunderbird/45.1.0 MIME-Version: 1.0 In-Reply-To: <20160510123517.2107653b@gumby.homeunix.com> Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: 8bit X-BeenThere: freebsd-ports@freebsd.org X-Mailman-Version: 2.1.22 Precedence: list List-Id: Porting software to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 10 May 2016 12:35:50 -0000 On 05/10/16 13:35, RW via freebsd-ports wrote: > On Mon, 9 May 2016 20:15:12 +0200 > Guido Falsi wrote: > >> On 05/09/16 19:52, Fernando Apesteguía wrote: >>> Hi all, >>> >>> Is it safe to use different invocations of poudriere concurrently >>> for different jails but using the same ports collection? >>> >> >> Yes it is, or at least should be. >> >> The ports trees are mounted read only in the jails, the wrkdir is >> defined at a different path. > > What about the distfiles directory? > > Having two "make checksums" running on the same file used to work > fairly well, but not any more because the target now deletes an > incomplete file rather than trying to resume it. > > This wont damage packages, but it can cause two "make checksums" to get > locked in a cycle of deleting each other's files and end with one > getting a failed checksum. Yes it happens, I even have used the same disfiles over NFS with more than one machine/poudriere accessing it. The various instances do overwrite each other and checksums do fail but usually in the end one of them "wins" and the correct file ends up being completed, with other instances reading that one. I agree this happens just by chance and not due to good design. As far as I understand Unix Filesystem semantics each download actually creates a new file, with only the last one to start referencing the actual file visible on the filesystem. So the last one starting to download is the one which will "win" creating the correct file on the FS, then checksumming it and going on. The other files have actuay been deleted and are simply removed from disk as soon as the download ends, if at that point the "winning" one has finished the download, they will checksum that file. There is a chance of the loosing download to end before the winning one ends and overwriting it again, but in my experience with at most 3-4 instances over NFS it usually fixes itself in the long run. IMHO best solution is to make sure you already have distfiles on disk for what you are going to build. -- Guido Falsi