From owner-freebsd-cloud@freebsd.org Wed Mar 13 16:43:46 2019 Return-Path: Delivered-To: freebsd-cloud@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 5352815396E7 for ; Wed, 13 Mar 2019 16:43:46 +0000 (UTC) (envelope-from raf@rafal.net) Received: from smtp-out-2.mxes.net (smtp-out-2.mxes.net [205.237.194.127]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 5286B891E9 for ; Wed, 13 Mar 2019 16:43:45 +0000 (UTC) (envelope-from raf@rafal.net) Received: from Customer-MUA (mua.mxes.net [10.0.0.1]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.mxes.net (Postfix) with ESMTPSA id B29FB2753E; Wed, 13 Mar 2019 12:43:34 -0400 (EDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=rafal.net; s=tm; t=1552495416; bh=VaqWeW0kRiGwkIKRhbH+uYzVZuMVwBJ8O//VcOGX8FE=; h=Subject:From:In-Reply-To:Date:Cc:References:To:From; b=OnLMUOgizbMIcm3MbiDdf+RoM33fySPIdz2PfKMFJMq3RfXGUwWq2+A62NW1HNZTC SP44aiKUFGKUvIRTCt0g3DZ2ei9DUDxHMSz/OHnMtBCUkfH36BJWn397DEY4G1n6cm jhFSzlzCc+BytiHKZsXunisYrwbIwc2ya2APW+bw= Content-Type: text/plain; charset=utf-8 Mime-Version: 1.0 (Mac OS X Mail 12.2 \(3445.102.3\)) Subject: Re: How to base64 encode and upload userdata for EC2 Launch Template From: Rafal Lukawiecki In-Reply-To: <00238cea-75dc-57e2-a304-671a8dc5f5b5@freebsd.org> Date: Wed, 13 Mar 2019 16:43:32 +0000 Cc: freebsd-cloud@freebsd.org Content-Transfer-Encoding: quoted-printable Message-Id: <8601F0A6-9099-4C7C-8889-38D789A81EFD@rafal.net> References: <16C73F7C-3673-44CC-B59E-DA247A0C2DF7@rafal.net> <00238cea-75dc-57e2-a304-671a8dc5f5b5@freebsd.org> To: Colin Percival , "Philip M. Gollucci" X-Mailer: Apple Mail (2.3445.102.3) X-Sent-To: X-Sender: rafal.net X-Rspamd-Queue-Id: 5286B891E9 X-Spamd-Bar: ------ Authentication-Results: mx1.freebsd.org; dkim=pass header.d=rafal.net header.s=tm header.b=OnLMUOgi; dmarc=pass (policy=none) header.from=rafal.net; spf=pass (mx1.freebsd.org: domain of raf@rafal.net designates 205.237.194.127 as permitted sender) smtp.mailfrom=raf@rafal.net X-Spamd-Result: default: False [-6.26 / 15.00]; ARC_NA(0.00)[]; SUBJECT_ENDS_SPACES(0.50)[]; R_DKIM_ALLOW(-0.20)[rafal.net:s=tm]; RCVD_VIA_SMTP_AUTH(0.00)[]; FROM_HAS_DN(0.00)[]; RCPT_COUNT_THREE(0.00)[3]; R_SPF_ALLOW(-0.20)[+ip4:205.237.194.0/25]; MV_CASE(0.50)[]; MIME_GOOD(-0.10)[text/plain]; NEURAL_HAM_LONG(-1.00)[-1.000,0]; TO_DN_SOME(0.00)[]; RCVD_IN_DNSWL_LOW(-0.10)[127.194.237.205.list.dnswl.org : 127.0.5.1]; TO_MATCH_ENVRCPT_SOME(0.00)[]; DKIM_TRACE(0.00)[rafal.net:+]; DMARC_POLICY_ALLOW(-0.50)[rafal.net,none]; MX_GOOD(-0.01)[mxin.mxes.net,mxin.mxes.net]; NEURAL_HAM_MEDIUM(-1.00)[-1.000,0]; NEURAL_HAM_SHORT(-0.99)[-0.985,0]; FROM_EQ_ENVFROM(0.00)[]; MIME_TRACE(0.00)[0:+]; IP_SCORE(-3.16)[ip: (-8.29), ipnet: 205.237.192.0/20(-4.14), asn: 10607(-3.32), country: US(-0.07)]; ASN(0.00)[asn:10607, ipnet:205.237.192.0/20, country:US]; MID_RHS_MATCH_FROM(0.00)[]; RCVD_TLS_ALL(0.00)[]; RCVD_COUNT_TWO(0.00)[2] X-BeenThere: freebsd-cloud@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: "FreeBSD on cloud platforms \(EC2, GCE, Azure, etc.\)" List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 13 Mar 2019 16:43:46 -0000 Colin, Philip, Thank you both for your suggestions. My set of scripts is a nice 5kB = when compressed, but tar without any compression is 20kB, which means it = does not get accepted as UserData by aws. Since I already use EFS for = distribution of some config data (mainly rc.init scripts), I will = simplify the configinit stage, repurpose the logic of it that processes = files in a loop seeing if they are >, >> or #!, but run it against the = mounted EFS, rather than from data that was passed in UserData. Philip: my current set-up already caters for different versions of = configs, as in, older but in production, current testing, and future. I = like your idea of colours. :) In end end, I will combine the best of both of your advice, for which I = would like to thank you. I wish, however, that AWS LT and LC worked with = encoded data correctly, and preferably, in the same way. Colin: have you = managed to pass any userdata to your cloudconfig in a compressed tar = format in a Launch Template? I have never succeeded with that, yet. Regards from Ireland, Rafal > On 11 Mar 2019, at 19:37, Colin Percival wrote: >=20 > Try it without the 'y' flag. It's possible that this will produce = something > too big to pass as a user-data file; but if it's small enough this = would be > a useful indication of where the problem lies. >=20 > Colin Percival >=20 > On 3/11/19 12:24 PM, Rafal Lukawiecki wrote: >> I have been using =E2=80=9Ctar cvyf fff ddd=E2=80=9D, which I think = is a compressed tarball using bzip. Should I use something else? >>=20 >> I have also noticed odd behaviour in the Console for LCs, but = different behaviour in the Console for LCs, and yet different for CLI=E2=80= =A6 >>=20 >> Thank you, Colin. >> Rafal >>=20 >>> On 11 Mar 2019, at 19:21, Colin Percival = wrote: >>>=20 >>> On 3/11/19 2:42 AM, Rafal Lukawiecki wrote: >>>> I hope someone can explain, as I have been lost in this for over a = week. I would like to pass a small (4-5kB) tarball of files for the = built-in cloud init script of the official FreeBSD 12.0 EC2 AMI to get = untarred and processed (>, >>, or #!) upon instance launch. I have = successfully done it for over 2 years using Launch Configurations but I = cannot make it work using the new Launch Templates. >>>=20 >>> Is this a tarball, or a compressed tarball? >>>=20 >>> There's a longstanding bug (which I'm told is going to get fixed = eventually!) >>> in the AWS Console whereby any non-7-bit-clean user-data files get = mangled >>> and come out with each byte UTF-8 encoded. I was only aware of this = as an >>> issue with the Console but it's entirely possible that someone at = Amazon wrote >>> the same bug in multiple places. >>>=20 >>> --=20 >>> Colin Percival >>> Security Officer Emeritus, FreeBSD | The power to serve >>> Founder, Tarsnap | www.tarsnap.com | Online backups for the truly = paranoid >>=20 >>=20 >>=20 >=20 > --=20 > Colin Percival > Security Officer Emeritus, FreeBSD | The power to serve > Founder, Tarsnap | www.tarsnap.com | Online backups for the truly = paranoid