From owner-freebsd-ports@freebsd.org Tue May 10 14:25:44 2016 Return-Path: Delivered-To: freebsd-ports@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id D5F3EB35939 for ; Tue, 10 May 2016 14:25:44 +0000 (UTC) (envelope-from rwmaillists@googlemail.com) Received: from mailman.ysv.freebsd.org (unknown [127.0.1.3]) by mx1.freebsd.org (Postfix) with ESMTP id B57B01387 for ; Tue, 10 May 2016 14:25:44 +0000 (UTC) (envelope-from rwmaillists@googlemail.com) Received: by mailman.ysv.freebsd.org (Postfix) id B1134B35938; Tue, 10 May 2016 14:25:44 +0000 (UTC) Delivered-To: ports@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id B0B49B35937 for ; Tue, 10 May 2016 14:25:44 +0000 (UTC) (envelope-from rwmaillists@googlemail.com) Received: from mail-wm0-x22d.google.com (mail-wm0-x22d.google.com [IPv6:2a00:1450:400c:c09::22d]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 481811386 for ; Tue, 10 May 2016 14:25:44 +0000 (UTC) (envelope-from rwmaillists@googlemail.com) Received: by mail-wm0-x22d.google.com with SMTP id g17so31111206wme.1 for ; Tue, 10 May 2016 07:25:44 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=googlemail.com; s=20120113; h=date:from:to:subject:message-id:in-reply-to:references:mime-version :content-transfer-encoding; bh=yn5CFEKk2++L8L0D8gbv4h/hOglBUYccvikFvURPOeE=; b=SlJv/K2ssIu+DY96YtUVJtnzE3jxOZhF2hdPC5MKaGB+937SM7l4ECBnRFJDMZ4z6s P05wNPnp+ikDy9/haCQc2d8geMeFh47A0tBSAToz+WBwZEwjmMpcLopZ7iYiU923j9x7 fY3K5juDxT02BXqGIMZrCz21SX2QHxS75Sn2LZw61GQnjrA1orKvwaL3cfbHZGttdDhc B4tu6AAaqC4sv5nYwD6bavmAnCvOXzVqGgsA0TtIQErDmKM/M/nF4n5IuIpiGV7b3oxN aQhyRirrrMgxWc1RKThYnActz6w7KFJ/ZoKegNUo3zH9uxamgHb9FIaZ7uR2o3t1VeHR TfjQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20130820; h=x-gm-message-state:date:from:to:subject:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=yn5CFEKk2++L8L0D8gbv4h/hOglBUYccvikFvURPOeE=; b=Ke6Xpex4eo1SEzyALJjGktUSrTg8a/zCQdxEuMX01+G/jDHAAofmmGT7wRArg4j+0B JfZj+fVGue8dlMGdYqbpW9Jam4uk+Z/Yoel32MQkaG4pCq/sd8YqVbqA5QsEgVHP/ooz Be2oup1TY318YZijkVuaOWzPJvbYsQgheQsgUBo4OCHbNBTSk2AG7iK8Qq6C89TNgusI awjtZCEI9huxG6Lk/gxCbrn02PH3GrcBHh38MBH5uUerQyNRDMFT1n1hzTEFM5kbgdk/ b/MHSM9+/79fP2lfRupAVCZbue4BYe/RnPnZPlkp9iuBCPL0p3HEN1dGQsCvNbfgyJmM Q6xA== X-Gm-Message-State: AOPr4FX7by8qCXpqcjQRc/GuvgSme57gJdihWEZNyHriapN1GybA9drZxJOCX8Vs287K8A== X-Received: by 10.28.30.148 with SMTP id e142mr16613257wme.69.1462890342892; Tue, 10 May 2016 07:25:42 -0700 (PDT) Received: from gumby.homeunix.com ([2.124.245.61]) by smtp.gmail.com with ESMTPSA id e8sm3281175wma.2.2016.05.10.07.25.41 for (version=TLSv1/SSLv3 cipher=OTHER); Tue, 10 May 2016 07:25:41 -0700 (PDT) Date: Tue, 10 May 2016 15:25:40 +0100 From: RW To: ports@freebsd.org Subject: Re: Poudriere question Message-ID: <20160510152540.31793420@gumby.homeunix.com> In-Reply-To: References: <3557cbcd-3992-5db5-c5dc-7912508e1956@madpilot.net> <20160510123517.2107653b@gumby.homeunix.com> X-Mailer: Claws Mail 3.13.2 (GTK+ 2.24.29; amd64-portbld-freebsd10.2) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: quoted-printable X-BeenThere: freebsd-ports@freebsd.org X-Mailman-Version: 2.1.22 Precedence: list List-Id: Porting software to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 10 May 2016 14:25:44 -0000 On Tue, 10 May 2016 14:35:35 +0200 Guido Falsi wrote: > On 05/10/16 13:35, RW via freebsd-ports wrote: > > On Mon, 9 May 2016 20:15:12 +0200 > > Guido Falsi wrote: > > =20 > >> On 05/09/16 19:52, Fernando Apestegu=C3=ADa wrote: =20 > >>> Hi all, > >>> > >>> Is it safe to use different invocations of poudriere concurrently > >>> for different jails but using the same ports collection? > >>> =20 > >> > >> Yes it is, or at least should be. > >> > >> The ports trees are mounted read only in the jails, the wrkdir is > >> defined at a different path. =20 > >=20 > > What about the distfiles directory?=20 > >=20 > > Having two "make checksums" running on the same file used to work > > fairly well, but not any more because the target now deletes an > > incomplete file rather than trying to resume it. > >=20 > > This wont damage packages, but it can cause two "make checksums" to > > get locked in a cycle of deleting each other's files and end w > > one getting a failed checksum. =20 >=20 > Yes it happens, I even have used the same disfiles over NFS with more > than one machine/poudriere accessing it. >=20 > The various instances do overwrite each other and checksums do fail > but usually in the end one of them "wins" and the correct file ends > up being completed, with other instances reading that one. I agree > this happens just by chance and not due to good design. Only the last process will terminate with a complete file and without error, when another process runs out of retries, the file with the directory entry is a download in progress which will fail the checksum. If it commonly ends-up working in poudriere that's probably a property of how poudriere orders things. But you still have the problem of wasted time and bandwidth. This problem is most likely with large distfiles and there's at least one that's 1 GB. The way this used to work is that the second process would try to resume the download which presumably involved getting a lock on the file. For smaller files it would just work. Worst case was that the second process would fail after a timeout. I think the change came in to delete possible re-rolled distfiles automatically (a relatively minor problem), but in the process it created this problem and also broke resuming downloads.=20 I don't see the reason for checking and deleting the file before attempting to resume it. > As far as I understand Unix Filesystem semantics each download > actually creates a new file, with only the last one to start > referencing the actual file visible on the filesystem. So the last > one starting to download is the one which will "win" creating the > correct file on the FS, then checksumming it and going on. The other > files have actuay been deleted and are simply removed from disk as > soon as the download ends, if at that point the "winning" one has > finished the download, they will checksum that file. >=20 > There is a chance of the loosing download to end before the winning > one ends and overwriting it again, but in my experience with at most > 3-4 instances over NFS it usually fixes itself in the long run. >=20 > IMHO best solution is to make sure you already have distfiles on disk > for what you are going to build. >=20