From owner-freebsd-pkg@freebsd.org Fri Sep 27 17:24:17 2019 Return-Path: Delivered-To: freebsd-pkg@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id E5B58128627 for ; Fri, 27 Sep 2019 17:24:17 +0000 (UTC) (envelope-from dan@langille.org) Received: from out2-smtp.messagingengine.com (out2-smtp.messagingengine.com [66.111.4.26]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 46fzDb6Cwpz44l0 for ; Fri, 27 Sep 2019 17:24:15 +0000 (UTC) (envelope-from dan@langille.org) Received: from compute4.internal (compute4.nyi.internal [10.202.2.44]) by mailout.nyi.internal (Postfix) with ESMTP id 2F852222D5 for ; Fri, 27 Sep 2019 13:24:15 -0400 (EDT) Received: from imap36 ([10.202.2.86]) by compute4.internal (MEProxy); Fri, 27 Sep 2019 13:24:15 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=langille.org; h= mime-version:message-id:in-reply-to:references:date:from:to :subject:content-type; s=fm2; bh=+jkbKW+6V/ARGGeeKZ4Yn3MvPoHaK3U Qp9T+Z2hX5mM=; b=k4dptFevYTlucGx/TuU3fx6Bsiziyr8LCHbNTf0BSVE5k/S NDA6eMIlG+GamFuo0wnf0mRBfPLrBLHdQfmFpY3vV3eIslTXQAFdkXaSafR9LGph 3OxdjFvjM/94a6QEOw3iqaD75btFjMnWao0BMPktyUpELad/ES4TotPibiHB7t2B kBN26xgK94HFSQdCyKH08+TUf2DJSzyT1/RLmIbEJK/QL7KjyNqytGcwYrtLDK4i YDEJkDqiEhPS5oF1FxOeHIxzZsEX5tWPrsA1eTHD9qNZelNU5kPJ1zfmbe5BgWI3 g9DAjPh2cNM9o9SJfe1Cjbc4RzT7bV5A/0gDVIg== DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d= messagingengine.com; h=content-type:date:from:in-reply-to :message-id:mime-version:references:subject:to:x-me-proxy :x-me-proxy:x-me-sender:x-me-sender:x-sasl-enc; s=fm3; bh=+jkbKW +6V/ARGGeeKZ4Yn3MvPoHaK3UQp9T+Z2hX5mM=; b=VSDt1/62poWycbIgVcbW+e OcE4vDx2mllrH4wjRilMa2wULM/+tOLVKAz8LCVvO/KkRGtAobCPc2jsBLTeyXD1 4z0jP1Y15+VrSe887QRFYeMSwTvWtntC+KGEq84D3XKHhu7wxvZ704tYhTR23OJm PYFDpyuAJktD1B9d1HZHj5FeF1pAlf7YvEnxQMcu6DXMXsbKOTJjENI2CYGD9mgm D/RjyJRmJ4XOsrXiqrZJ2jZ2gFCrH1lsKitg9ZUSi6JTvZebh0Vtu98WyfL4wyRC FgFNysa+W4R/JUYUjFqq2i2iKrIGehI9XNihhegeJO6jDaBng90akbLFt2r2KInQ == X-ME-Sender: X-ME-Proxy-Cause: gggruggvucftvghtrhhoucdtuddrgedufedrfeeigdduuddtucetufdoteggodetrfdotf fvucfrrhhofhhilhgvmecuhfgrshhtofgrihhlpdfqfgfvpdfurfetoffkrfgpnffqhgen uceurghilhhouhhtmecufedttdenucenucfjughrpefofgggkfgjfhffhffvufgtsehttd ertderredtnecuhfhrohhmpedfffgrnhcunfgrnhhgihhllhgvfdcuoegurghnsehlrghn ghhilhhlvgdrohhrgheqnecurfgrrhgrmhepmhgrihhlfhhrohhmpegurghnsehlrghngh hilhhlvgdrohhrghenucevlhhushhtvghrufhiiigvpedt X-ME-Proxy: Received: by mailuser.nyi.internal (Postfix, from userid 501) id 9FDA2188005C; Fri, 27 Sep 2019 13:24:14 -0400 (EDT) X-Mailer: MessagingEngine.com Webmail Interface User-Agent: Cyrus-JMAP/3.1.7-305-g4111847-fmstable-20190924v1 Mime-Version: 1.0 Message-Id: <33729b1d-4a6e-430f-8edc-2f98810ab3b0@www.fastmail.com> In-Reply-To: <5404b363-e887-728b-112d-a125aa8f8d5b@FreeBSD.org> References: <6c130e1c-57df-40da-b9ae-169f8b20238d@www.fastmail.com> <5404b363-e887-728b-112d-a125aa8f8d5b@FreeBSD.org> Date: Fri, 27 Sep 2019 13:23:54 -0400 From: "Dan Langille" To: freebsd-pkg@freebsd.org Subject: Re: poudriere: build on on server, rsync to another Content-Type: text/plain X-Rspamd-Queue-Id: 46fzDb6Cwpz44l0 X-Spamd-Bar: ----- Authentication-Results: mx1.freebsd.org; dkim=pass header.d=langille.org header.s=fm2 header.b=k4dptFev; dkim=pass header.d=messagingengine.com header.s=fm3 header.b=VSDt1/62; dmarc=pass (policy=none) header.from=langille.org; spf=pass (mx1.freebsd.org: domain of dan@langille.org designates 66.111.4.26 as permitted sender) smtp.mailfrom=dan@langille.org X-Spamd-Result: default: False [-5.55 / 15.00]; ARC_NA(0.00)[]; NEURAL_HAM_MEDIUM(-1.00)[-1.000,0]; R_DKIM_ALLOW(-0.20)[langille.org:s=fm2,messagingengine.com:s=fm3]; XM_UA_NO_VERSION(0.01)[]; FROM_HAS_DN(0.00)[]; R_SPF_ALLOW(-0.20)[+ip4:66.111.4.26]; TO_MATCH_ENVRCPT_ALL(0.00)[]; MIME_GOOD(-0.10)[text/plain]; PREVIOUSLY_DELIVERED(0.00)[freebsd-pkg@freebsd.org]; TO_DN_NONE(0.00)[]; RCPT_COUNT_ONE(0.00)[1]; NEURAL_HAM_LONG(-1.00)[-1.000,0]; RCVD_COUNT_THREE(0.00)[4]; IP_SCORE(-3.46)[ip: (-9.73), ipnet: 66.111.4.0/24(-4.85), asn: 11403(-2.68), country: US(-0.05)]; MV_CASE(0.50)[]; DKIM_TRACE(0.00)[langille.org:+,messagingengine.com:+]; DMARC_POLICY_ALLOW(-0.50)[langille.org,none]; RCVD_IN_DNSWL_LOW(-0.10)[26.4.111.66.list.dnswl.org : 127.0.5.1]; FROM_EQ_ENVFROM(0.00)[]; MIME_TRACE(0.00)[0:+]; ASN(0.00)[asn:11403, ipnet:66.111.4.0/24, country:US]; RCVD_TLS_LAST(0.00)[]; MID_RHS_WWW(0.50)[] X-BeenThere: freebsd-pkg@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Binary package management and package tools discussion List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 27 Sep 2019 17:24:18 -0000 On Fri, Sep 27, 2019, at 10:44 AM, Matthew Seaman wrote: > On 27/09/2019 15:26, Dan Langille wrote: > > Hello, > > > > I am working on putting a new poudriere build server into production at work. > > > > At present, we build packages on one server & deploy them from another server. > > > > The package build server mounts an NFS share from /usr/local/poudriere/data/packages > > > > The package distribution server mounts that same NFS share in RO-mode. All hosts install from this server. > > > > I want to change this. The goal: > > > > * poudriere builds to local disk > > * /usr/local/poudriere/data/packages will be rsync'd later to NFS > > * the build & rsync process will not be serial/linked (i.e. a build will not trigger an rsync) > > > > My concern: will this work? Will the package distribution server get incomplete sets of packages? > > > > Thank you. > > > > If you enable the ATOMIC_PACKAGE_REPOSITORY setting in poudriere.conf, > then you won't see inconsistencies because you're rysnc'ing while > poudriere is in the middle of re-building the repository. > > You might see inconsistencies because someone is trying to install > packages while you're in the middle of rsyncing an updated repository. > However, the atomic package repository mechanism works by juggling > symbolic links at the top of the /usr/local/poudriere/data/packages > directory tree -- if you arrange things to rsync any subdirectories > first, and then finally synch over the symbolic link bits, you should be > golden. This is pretty much what we came up with on Twitter, but not so quickly. ;) > Although, just sticking a webserver onto your poudriere build box would > likely be easier overall and less effort to maintain. Yes, I think so. We do have about 1TB of free disk space on this host. Thinking further on this: one reason we install from another host is avoid blocking package maintenance on other hosts when the build server undergoes maintenance. This is something we can think about. Thank you. -- Dan Langille dan@langille.org