From owner-freebsd-cloud@FreeBSD.ORG Mon Nov 24 09:43:14 2014 Return-Path: Delivered-To: freebsd-cloud@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 3087E859 for ; Mon, 24 Nov 2014 09:43:14 +0000 (UTC) Received: from o3.shared.sendgrid.net (o3.shared.sendgrid.net [208.117.48.85]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id ED35097C for ; Mon, 24 Nov 2014 09:43:13 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha1; c=relaxed; d=sendgrid.info; h=from:mime-version:to:subject:content-type:content-transfer-encoding; s=smtpapi; bh=pE8cs3O5wr0QP4LU8qsrDQtk32M=; b=r3Ffp8OgxKrAzn5k3N NKXBkcno5G4/blKaRpkbA0zDhxMtrOYmZ8hqM0xnMRUMzSPJFy7pi2zCupeMBfaV 2V/JosWzYCSX+M4Kn7WzFyx/PRLtr/Fiu2TYXHRY+dcNGnUpngVVb8MJeOI+inIW aFXVsUIfGmifjklsww1xvNmT8= Received: by filter0161p1mdw1.sendgrid.net with SMTP id filter0161p1mdw1.23412.5472FDA97 2014-11-24 09:43:06.506762496 +0000 UTC Received: from mail.tarsnap.com (unknown [10.100.60.108]) by ismtpd-001 (SG) with ESMTP id 149e12ee1a8.2f42.b93ee for ; Mon, 24 Nov 2014 09:43:06 +0000 (UTC) Received: (qmail 54815 invoked from network); 24 Nov 2014 09:43:05 -0000 Received: from unknown (HELO clamshell.daemonology.net) (127.0.0.1) by ec2-107-20-205-189.compute-1.amazonaws.com with ESMTP; 24 Nov 2014 09:43:05 -0000 Received: (qmail 36886 invoked from network); 24 Nov 2014 09:41:45 -0000 Received: from unknown (HELO clamshell.daemonology.net) (127.0.0.1) by clamshell.daemonology.net with SMTP; 24 Nov 2014 09:41:45 -0000 Message-ID: <5472FD59.4040906@freebsd.org> Date: Mon, 24 Nov 2014 01:41:45 -0800 From: Colin Percival User-Agent: Mozilla/5.0 (X11; FreeBSD amd64; rv:31.0) Gecko/20100101 Thunderbird/31.2.0 MIME-Version: 1.0 To: freebsd-cloud@freebsd.org Subject: Patches to support EC2 AMI building in base system Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: 7bit X-SG-EID: XhyBwObMhraAR+zdwMupjd7LSrZ36RouVZZW3oDOpB0bMbwNkKvPutZRROApdD0zpUfRGuwjjMb523 621zNp2M0y3Dk2iNE0w6xBi8pal7inhNjMPRYjfUuRKx5bBYvyDndxSAwAzTN2bDzL63yn7L7WmrPA HTUvSCghDb8EcdrIsoaiyT35APIb7JjyJXft X-BeenThere: freebsd-cloud@freebsd.org X-Mailman-Version: 2.1.18-1 Precedence: list List-Id: "FreeBSD on cloud platforms \(EC2, GCE, Azure, etc.\)" List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 24 Nov 2014 09:43:14 -0000 Hi everyone, I have some patches to support building EC2 AMIs from the FreeBSD base system. I hope that these will be merged soon, but in the mean time you can apply them to your tree. My patches are based off gjb's release-vmimage branch, but they build images which take advantage of the 'growfs' rc.d script in HEAD (which hasn't been merged to gjb's branch) so you'll want to check out HEAD and then get gjb's release bits on top of that: svnlite co https://svn0.us-west.FreeBSD.org/base/head /usr/src svnlite co \ https://svn0.us-west.FreeBSD.org/base/projects/release-vmimage/release \ /usr/src/release-vmimage Then download and apply my patches to the release-vmimage bits: cd /usr/src/release-vmimage fetch http://people.freebsd.org/~cperciva/0-pkg-infrastructure.patch fetch http://people.freebsd.org/~cperciva/1-ec2-conf.patch fetch http://people.freebsd.org/~cperciva/2-ec2.ami.patch patch < 0-pkg-infrastructure.patch patch < 1-ec2-conf.patch patch < 2-ec2-ami.patch You'll also need my bsdec2-image-upload code: make -C /usr/ports/net/bsdec2-image-upload install clean Next you'll need to set up your AWS account to support importing a disk image and creating an AMI from it: 1. Create an S3 bucket in the region you want to use (in my case, us-west-2 aka. Oregon). I called mine "freebsd-release-staging". 2. Create a Lifecycle Rule on the bucket to Permanently Delete objects 1 day after the object's creation date. 3. Create an IAM user named "freebsd-release-upload". 4. Attach a Policy to the IAM user granting permission for: * s3:DeleteObject, s3:GetObject, and s3:PutObject on resource arn:aws:s3:::YOURBUCKETNAME/* * ec2:CopyImage, ec2:CreateSnapshot, ec2:DeleteVolume, ec2:DescribeConversionTasks, ec2:DescribeImages, ec2:DescribeSnapshots, ec2:ImportVolume, ec2:ModifyImageAttribute and ec2:RegisterImage on resource * 5. Create AWS Access Keys for the IAM user, and create a file in the format ACCESS_KEY_ID=AKIEXAMPLEEXAMPLE ACCESS_KEY_SECRET=EXAMPLEEXAMPLEEXAMPLEEXAMPLEEXAMPLE Finally, build world and kernel, and then the "ec2ami" target (with the appropriate parameters based on your AWS setup above): cd /usr/src && make buildkernel buildworld cd /usr/src/release-vmimage && make WITH_CLOUDWARE=YES \ AWSKEYFILE=/root/aws.key AWSREGION=us-west-2 \ AWSBUCKET=freebsd-release-staging EC2PUBLIC=YES \ ec2ami This will construct a disk image, import it into EC2, create a snapshot from it, register an AMI from that, copy the AMI to all the (publicly available) regions, and mark the AMIs as public. To skip the copying and marking of the AMIs as public, remove the "EC2PUBLIC=YES" part of the ec2ami make command. -- Colin Percival Security Officer Emeritus, FreeBSD | The power to serve Founder, Tarsnap | www.tarsnap.com | Online backups for the truly paranoid From owner-freebsd-cloud@FreeBSD.ORG Mon Nov 24 11:30:17 2014 Return-Path: Delivered-To: freebsd-cloud@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 8AE57121 for ; Mon, 24 Nov 2014 11:30:17 +0000 (UTC) Received: from mail-wi0-x229.google.com (mail-wi0-x229.google.com [IPv6:2a00:1450:400c:c05::229]) (using TLSv1 with cipher ECDHE-RSA-RC4-SHA (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 1BD7F69B for ; Mon, 24 Nov 2014 11:30:17 +0000 (UTC) Received: by mail-wi0-f169.google.com with SMTP id r20so8667637wiv.0 for ; Mon, 24 Nov 2014 03:30:15 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=qxnitro.org; s=google; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :cc:content-type; bh=wpDbc0ooL5RBghbhk90x9PxrujG/1i1NJ0VgwcqV/iA=; b=g9HkzBmB9Dgy1e6pyHMFRnRK5yNgthy85u+zROF46cQSDEK4Cad5775aoTMaO//pU9 3WN8z1a07r2dWoIJkP8gWNZchAnQ8VMQPkZAxG/V9tgfLhGZQ0dLNIMp/lR43RcMzZQi DWN9jlF1gQQQ7JkvK9gbwFnP+ZpGF5hSdS4gc= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20130820; h=x-gm-message-state:mime-version:in-reply-to:references:date :message-id:subject:from:to:cc:content-type; bh=wpDbc0ooL5RBghbhk90x9PxrujG/1i1NJ0VgwcqV/iA=; b=Itp424YSRpdB9tBeAxerosfeb77hwhbUSA+7ylPjYMKBd/o1/2yLgLDL02oYbgFAhh yxlXPXke5yyY8uYSqfy0LhfHqXy+i/kpL/h5+lNTHWzKMIoN8nTUNdKoE66vCu+jHby7 Jmb9U1UioRUcMC0Jyh5KQXMTvgHHK63mWlkjcumWdyPPT8qazwRhoiZ2JFbik6wB1VIZ 4ncq4lie98+/AuSrZoI5cbCrhMZkpG4lb6VhxfaUxhVh5ms7FuMr9LlaIk2gpCEVLe8j sjo8anyyfgJFboqVRuGSKoWurGST+Jhl0Z/SlHcUfw564WW2uQXEjH6Nt9kut7rVJ7p4 I2qQ== X-Gm-Message-State: ALoCoQm5fpgudq+BY1GCS543dM020qs6ZimGKX8x5mvxLdxQZmQCZLfZteMN3c+XhFkOe11EKr4I MIME-Version: 1.0 X-Received: by 10.194.122.10 with SMTP id lo10mr31562251wjb.19.1416828615502; Mon, 24 Nov 2014 03:30:15 -0800 (PST) Received: by 10.27.176.88 with HTTP; Mon, 24 Nov 2014 03:30:15 -0800 (PST) In-Reply-To: <5472FD59.4040906@freebsd.org> References: <5472FD59.4040906@freebsd.org> Date: Mon, 24 Nov 2014 11:30:15 +0000 Message-ID: Subject: Re: Patches to support EC2 AMI building in base system From: "Simon L. B. Nielsen" To: Colin Percival Content-Type: text/plain; charset=UTF-8 Cc: freebsd-cloud@freebsd.org X-BeenThere: freebsd-cloud@freebsd.org X-Mailman-Version: 2.1.18-1 Precedence: list List-Id: "FreeBSD on cloud platforms \(EC2, GCE, Azure, etc.\)" List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 24 Nov 2014 11:30:17 -0000 On 24 November 2014 at 09:41, Colin Percival wrote: > Hi everyone, > > I have some patches to support building EC2 AMIs from the FreeBSD base > system. I hope that these will be merged soon, but in the mean time > you can apply them to your tree. > > My patches are based off gjb's release-vmimage branch, but they build > images which take advantage of the 'growfs' rc.d script in HEAD (which > hasn't been merged to gjb's branch) so you'll want to check out HEAD and > then get gjb's release bits on top of that: Are there any reason not just to get them committed to head? They look mostly straight forward. I started playing with GCE (Google Compute Engine), so time permitting I'm going to try and make a similar config for GCE, therefor less manual patching the better :-). -- Simon L. B. Nielsen