From owner-freebsd-cloud@freebsd.org Wed Aug 30 14:14:25 2017 Return-Path: Delivered-To: freebsd-cloud@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id B9763E03122 for ; Wed, 30 Aug 2017 14:14:25 +0000 (UTC) (envelope-from raf@rafal.net) Received: from fallback-in2.mxes.net (fallback-out2.mxes.net [216.86.168.191]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 957326EDCF for ; Wed, 30 Aug 2017 14:14:24 +0000 (UTC) (envelope-from raf@rafal.net) Received: from mxout-08.mxes.net (mxout-08.mxes.net [216.86.168.183]) by fallback-in2.mxes.net (Postfix) with ESMTP id 2C3BA2FDC36 for ; Wed, 30 Aug 2017 10:08:08 -0400 (EDT) Received: from [192.168.40.37] (unknown [86.40.118.125]) (using TLSv1.2 with cipher DHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.mxes.net (Postfix) with ESMTPSA id 025D3509B8 for ; Wed, 30 Aug 2017 10:08:00 -0400 (EDT) From: Rafal Lukawiecki Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: quoted-printable Mime-Version: 1.0 (Mac OS X Mail 10.3 \(3273\)) Subject: AWS CloudWatch Logs Agent Message-Id: <4FFC6F3B-E8A1-49EE-A05A-C6B9B0A615FF@rafal.net> Date: Wed, 30 Aug 2017 15:07:59 +0100 To: freebsd-cloud@freebsd.org X-Mailer: Apple Mail (2.3273) X-BeenThere: freebsd-cloud@freebsd.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: "FreeBSD on cloud platforms \(EC2, GCE, Azure, etc.\)" List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 30 Aug 2017 14:14:25 -0000 First of all: thank you Colin, and thank you all, for making FreeBSD = available in AWS and the other clouds. This is my first email here, so = let me briefly introduce myself and the context. I am evaluating FreeBSD = in AWS as a replacement for our existing (small) AWS fleet of CentOS 7 = and Amazon Linux machines that serve my company (Project Botticelli Ltd) = online training business, which focuses on analytics and what is now = called data science (my main profession) but used to be called data = mining and just plain-old statistics. Colin Percival has already been very helpful in giving me some guidance = with regards to his configinit scripts, which I am happy to say are = working well and making machine launch and teardown easy. While my three = devs have been working hard on the new web front and back ends, and our = app API, all of which we hope to ship in a month, it is down to me to = focus on the sysadmin part of moving the server infrastructure to = FreeBSD. So far, working with FreeBSD has been a pleasure and a = wonderful connection to my old Unix-days' roots. One nice AWS component is its CloudWatch Logs engine that collects = streams of syslog-generated data and centralises them, with a simple but = usable UI. AWS provides an automated set-up for the awslogs daemon = agent, but from a cursory look at the code, although parts of it are = plain Python, much of it is very Linux-focused shell script, with much = regard for runlevels, logrotate etc and not immediately usable as an = rc.d script. I was wondering if anyone has hacked that script yet to run on FreeBSD = or if you have any other suggestions how to help me get AWS CloudWatch = Logs integrated into FreeBSD running on AWS. Many thanks for your help and my best regards from Ireland PS. I have posted a request on the forum for opinions regarding FreeBSD = in the cloud, in case anyone is interested: = https://forums.freebsd.org/threads/62190/ -- Rafal Lukawiecki Data Scientist and Director=20 Project Botticelli Ltd= From owner-freebsd-cloud@freebsd.org Wed Aug 30 23:06:12 2017 Return-Path: Delivered-To: freebsd-cloud@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 389DAE0C4DD for ; Wed, 30 Aug 2017 23:06:12 +0000 (UTC) (envelope-from 0100015e3565112c-406dc3cb-8399-4597-b0eb-8528bb6319d5-000000@amazonses.com) Received: from a8-56.smtp-out.amazonses.com (a8-56.smtp-out.amazonses.com [54.240.8.56]) (using TLSv1 with cipher ECDHE-RSA-AES128-SHA (128/128 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id F30DC83ECA for ; Wed, 30 Aug 2017 23:06:11 +0000 (UTC) (envelope-from 0100015e3565112c-406dc3cb-8399-4597-b0eb-8528bb6319d5-000000@amazonses.com) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/simple; s=ae7m2yrxjw65l2cqdpjxuucyrvy564tn; d=tarsnap.com; t=1504134369; h=Subject:To:References:From:Message-ID:Date:MIME-Version:In-Reply-To:Content-Type:Content-Transfer-Encoding; bh=7qRT5PnwAW/T+BvS9ux0LrIp+qk2K5um6Z/4RnjrGO8=; b=AVL4fECtiuUNQAfoWsYXL+KPH1L7rdnV8c17aJ+x/LqBSC++AsigZYnesHciSxMG kCtfGYHcjm/WrchCPzeBdoIVApKoQcC7YH+MiSfLBWArf+nuLW4TikIMet5t/hvHCUq jVeG4kTcYns0FI2mBCPTc8VOPXXhqBx2/vlpvYQY= DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/simple; s=224i4yxa5dv7c2xz3womw6peuasteono; d=amazonses.com; t=1504134369; h=Subject:To:References:From:Message-ID:Date:MIME-Version:In-Reply-To:Content-Type:Content-Transfer-Encoding:Feedback-ID; bh=7qRT5PnwAW/T+BvS9ux0LrIp+qk2K5um6Z/4RnjrGO8=; b=FpwmzYYjAlBjnfhHPIf4pnWxR81HGrT5lSRCWhSxXY+aYJRWcjmpWiCy7oaD44+m SQtSvsrpkPVdADjDMyr0r8NNu8iHNZ/tjZIHmEAlIUtPppusLDInZjEYvyGyZ6LOpcG i/DVcDq/r/CWv5EyccWKQvx2gHwpsSu66IFyTTUc= Subject: Re: AWS CloudWatch Logs Agent To: Rafal Lukawiecki , freebsd-cloud@freebsd.org References: <4FFC6F3B-E8A1-49EE-A05A-C6B9B0A615FF@rafal.net> From: Colin Percival Message-ID: <0100015e3565112c-406dc3cb-8399-4597-b0eb-8528bb6319d5-000000@email.amazonses.com> Date: Wed, 30 Aug 2017 23:06:09 +0000 User-Agent: Mozilla/5.0 (X11; FreeBSD amd64; rv:45.0) Gecko/20100101 Thunderbird/45.5.1 MIME-Version: 1.0 In-Reply-To: <4FFC6F3B-E8A1-49EE-A05A-C6B9B0A615FF@rafal.net> Content-Type: text/plain; charset=windows-1252 Content-Transfer-Encoding: 7bit X-SES-Outgoing: 2017.08.30-54.240.8.56 Feedback-ID: 1.us-east-1.Lv9FVjaNvvR5llaqfLoOVbo2VxOELl7cjN0AOyXnPlk=:AmazonSES X-BeenThere: freebsd-cloud@freebsd.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: "FreeBSD on cloud platforms \(EC2, GCE, Azure, etc.\)" List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 30 Aug 2017 23:06:12 -0000 On 08/30/17 07:07, Rafal Lukawiecki wrote: > One nice AWS component is its CloudWatch Logs engine that collects streams > of syslog-generated data and centralises them, with a simple but usable UI. > AWS provides an automated set-up for the awslogs daemon agent, but from a > cursory look at the code, although parts of it are plain Python, much of it > is very Linux-focused shell script, with much regard for runlevels, > logrotate etc and not immediately usable as an rc.d script. > > I was wondering if anyone has hacked that script yet to run on FreeBSD or > if you have any other suggestions how to help me get AWS CloudWatch Logs > integrated into FreeBSD running on AWS. Huh, somehow I never noticed CloudWatch Logs; the aspect of CloudWatch I had been thinking could be useful for FreeBSD was to create an agent which would record memory/swap/disk usage to CloudWatch. Unfortunately I don't have time to work on either right now; but for a very simple proof of principle I'd suggest running the Linux CloudWatch Logs agent via the Linux emulation system. -- Colin Percival Security Officer Emeritus, FreeBSD | The power to serve Founder, Tarsnap | www.tarsnap.com | Online backups for the truly paranoid From owner-freebsd-cloud@freebsd.org Wed Aug 30 23:19:51 2017 Return-Path: Delivered-To: freebsd-cloud@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 5162EE0C74F for ; Wed, 30 Aug 2017 23:19:51 +0000 (UTC) (envelope-from raf@rafal.net) Received: from mxout-08.mxes.net (mxout-08.mxes.net [216.86.168.183]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 0B29084396 for ; Wed, 30 Aug 2017 23:19:50 +0000 (UTC) (envelope-from raf@rafal.net) Received: from [192.168.40.37] (unknown [86.40.118.125]) (using TLSv1.2 with cipher DHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.mxes.net (Postfix) with ESMTPSA id 215AE509B6; Wed, 30 Aug 2017 19:19:48 -0400 (EDT) Content-Type: text/plain; charset=utf-8 Mime-Version: 1.0 (Mac OS X Mail 10.3 \(3273\)) Subject: Re: AWS CloudWatch Logs Agent From: Rafal Lukawiecki In-Reply-To: <0100015e3565144c-18dc3e52-fcba-4b30-bf2f-58b99b55562d-000000@email.amazonses.com> Date: Thu, 31 Aug 2017 00:19:47 +0100 Cc: freebsd-cloud@freebsd.org Content-Transfer-Encoding: quoted-printable Message-Id: <456A3074-08D2-4141-81EF-5CD6B1B53899@rafal.net> References: <4FFC6F3B-E8A1-49EE-A05A-C6B9B0A615FF@rafal.net> <0100015e3565144c-18dc3e52-fcba-4b30-bf2f-58b99b55562d-000000@email.amazonses.com> To: Colin Percival X-Mailer: Apple Mail (2.3273) X-BeenThere: freebsd-cloud@freebsd.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: "FreeBSD on cloud platforms \(EC2, GCE, Azure, etc.\)" List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 30 Aug 2017 23:19:51 -0000 Thank you, Colin, for your suggestion. I have never run the Linux = emulation system in FreeBSD yet, I suppose this is a good time to try = it. However, I did not realise that it would be able to cope with setup = of rc/runlevels/deamons. I thought it was only able to run more mundane = utilities, rather than assist in system/boot init set-up. Regarding your comment about CloudWatch metrics, I am *also* using = CloudWatch to report in-machine memory and CPU utilisation in my = existing Linux machines. I have followed this guide: = http://docs.aws.amazon.com/AWSEC2/latest/UserGuide/mon-scripts.html It was easy. Essentially, I have installed the script from = http://aws-cloudwatch.s3.amazonaws.com/downloads/CloudWatchMonitoringScrip= ts-1.2.1.zip and in my crontab I have something along the lines of: */5 * * * * /opt/aws-scripts-mon/mon-put-instance-data.pl --mem-util = --mem-used --mem-avail --swap-util --swap-used --disk-path=3D/ = --disk-space-util --disk-space-used --disk-space-avail --from-cron The metrics magically and reliably appear in CloudWatch so that I can = set-up alarms etc. It works really well in Linux, I hope to replicate = the success with FreeBSD. As for the CW Logs, unfortunately the script at curl = https://s3.amazonaws.com/aws-cloudwatch/downloads/latest/awslogs-agent-set= up.py is a bit too daunting for me to try rewriting using my brand-new = rc.d scripting skills, with only a hundred lines working (!) so far = behind my belt. Thank you for all your help, which is much appreciated. I hope to be = able to give back somehow one day=E2=80=A6if I manage to persist in this = adventure. Fingers crossed. Rafal -- Rafal Lukawiecki Data Scientist and Director=20 Project Botticelli Ltd > On 31 Aug 2017, at 00:06, Colin Percival wrote: >=20 > On 08/30/17 07:07, Rafal Lukawiecki wrote: >> One nice AWS component is its CloudWatch Logs engine that collects = streams >> of syslog-generated data and centralises them, with a simple but = usable UI. >> AWS provides an automated set-up for the awslogs daemon agent, but = from a >> cursory look at the code, although parts of it are plain Python, much = of it >> is very Linux-focused shell script, with much regard for runlevels, >> logrotate etc and not immediately usable as an rc.d script. >>=20 >> I was wondering if anyone has hacked that script yet to run on = FreeBSD or >> if you have any other suggestions how to help me get AWS CloudWatch = Logs >> integrated into FreeBSD running on AWS. >=20 > Huh, somehow I never noticed CloudWatch Logs; the aspect of CloudWatch = I had > been thinking could be useful for FreeBSD was to create an agent which = would > record memory/swap/disk usage to CloudWatch. >=20 > Unfortunately I don't have time to work on either right now; but for a = very > simple proof of principle I'd suggest running the Linux CloudWatch = Logs agent > via the Linux emulation system. >=20 > --=20 > Colin Percival > Security Officer Emeritus, FreeBSD | The power to serve > Founder, Tarsnap | www.tarsnap.com | Online backups for the truly = paranoid From owner-freebsd-cloud@freebsd.org Wed Aug 30 23:28:16 2017 Return-Path: Delivered-To: freebsd-cloud@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 356A2E0C965 for ; Wed, 30 Aug 2017 23:28:16 +0000 (UTC) (envelope-from pete@nomadlogic.org) Received: from vps-mail.nomadlogic.org (unknown [IPv6:2607:f2f8:a098::2]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 114B88476A for ; Wed, 30 Aug 2017 23:28:15 +0000 (UTC) (envelope-from pete@nomadlogic.org) Received: from [192.168.1.26] (cpe-23-242-94-236.socal.res.rr.com [23.242.94.236]) by vps-mail.nomadlogic.org (OpenSMTPD) with ESMTPSA id 137705d6 TLS version=TLSv1.2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128 verify=NO for ; Wed, 30 Aug 2017 16:28:14 -0700 (PDT) Subject: Re: AWS CloudWatch Logs Agent To: freebsd-cloud@freebsd.org References: <4FFC6F3B-E8A1-49EE-A05A-C6B9B0A615FF@rafal.net> <0100015e3565144c-18dc3e52-fcba-4b30-bf2f-58b99b55562d-000000@email.amazonses.com> <456A3074-08D2-4141-81EF-5CD6B1B53899@rafal.net> From: Pete Wright Message-ID: <084bd6ad-9ffd-c097-1d46-c8cb41d745ee@nomadlogic.org> Date: Wed, 30 Aug 2017 16:28:13 -0700 User-Agent: Mozilla/5.0 (X11; FreeBSD amd64; rv:52.0) Gecko/20100101 Thunderbird/52.3.0 MIME-Version: 1.0 In-Reply-To: <456A3074-08D2-4141-81EF-5CD6B1B53899@rafal.net> Content-Type: text/plain; charset=utf-8; format=flowed Content-Transfer-Encoding: 8bit Content-Language: en-US X-BeenThere: freebsd-cloud@freebsd.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: "FreeBSD on cloud platforms \(EC2, GCE, Azure, etc.\)" List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 30 Aug 2017 23:28:16 -0000 On 08/30/2017 16:19, Rafal Lukawiecki wrote: > > As for the CW Logs, unfortunately the script at curl https://s3.amazonaws.com/aws-cloudwatch/downloads/latest/awslogs-agent-setup.py is a bit too daunting for me to try rewriting using my brand-new rc.d scripting skills, with only a hundred lines working (!) so far behind my belt. hi rafal - i also took a look at that python setup script. my initial impression was that is was full of linux'isms (the first line being #!/usr/bin/python being the first thing i noticed).  and while a lot of the setup code is linux specific, i think there are some bits in there which could be ported to freebsd.  do you happen to know if there is a git or other open-source repository available with the awslogs-agent code out there?  i didn't find anything after a quick google search. cheers, -pete -- Pete Wright pete@nomadlogic.org @nomadlogicLA From owner-freebsd-cloud@freebsd.org Wed Aug 30 23:39:30 2017 Return-Path: Delivered-To: freebsd-cloud@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 10BD3E0CC4B for ; Wed, 30 Aug 2017 23:39:30 +0000 (UTC) (envelope-from raf@rafal.net) Received: from mxout-08.mxes.net (mxout-08.mxes.net [216.86.168.183]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id DFCCA84BC4 for ; Wed, 30 Aug 2017 23:39:29 +0000 (UTC) (envelope-from raf@rafal.net) Received: from [192.168.40.37] (unknown [86.40.118.125]) (using TLSv1.2 with cipher DHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.mxes.net (Postfix) with ESMTPSA id C75C4509B8; Wed, 30 Aug 2017 19:39:27 -0400 (EDT) Content-Type: text/plain; charset=utf-8 Mime-Version: 1.0 (Mac OS X Mail 10.3 \(3273\)) Subject: Re: AWS CloudWatch Logs Agent From: Rafal Lukawiecki In-Reply-To: <084bd6ad-9ffd-c097-1d46-c8cb41d745ee@nomadlogic.org> Date: Thu, 31 Aug 2017 00:39:26 +0100 Cc: freebsd-cloud@freebsd.org Content-Transfer-Encoding: quoted-printable Message-Id: <0A93F7FC-5547-4A67-B3D9-BB59027FDAD6@rafal.net> References: <4FFC6F3B-E8A1-49EE-A05A-C6B9B0A615FF@rafal.net> <0100015e3565144c-18dc3e52-fcba-4b30-bf2f-58b99b55562d-000000@email.amazonses.com> <456A3074-08D2-4141-81EF-5CD6B1B53899@rafal.net> <084bd6ad-9ffd-c097-1d46-c8cb41d745ee@nomadlogic.org> To: Pete Wright X-Mailer: Apple Mail (2.3273) X-BeenThere: freebsd-cloud@freebsd.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: "FreeBSD on cloud platforms \(EC2, GCE, Azure, etc.\)" List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 30 Aug 2017 23:39:30 -0000 Dear Pete, that was my feeling too=E2=80=94I suspect that for someone = more skilled than me in FreeBSD rc.d it should be realistically possible = to port it, but the spaghetti nature of the awslogs script would also be = something to contend with. I did a bit of searching, but the only = reference to awslogs that I found on GH is this one: https://github.com/jorgebastida/awslogs While this looks like a good utility for CloudWatch Logs querying, it = does not seem to deal with the agent and actual log collection. The AWS = GitHub account does not seem to list anything related amongst their 70+ = repositories... If you embark on such a port, I would be happy to test, help, and give = feedback. On the other hand, please let me know if you find anything = else that could be of use. Many thanks. Rafal -- Rafal Lukawiecki Data Scientist and Director=20 Project Botticelli Ltd > On 31 Aug 2017, at 00:28, Pete Wright wrote: >=20 >=20 >=20 > On 08/30/2017 16:19, Rafal Lukawiecki wrote: >>=20 >> As for the CW Logs, unfortunately the script at curl = https://s3.amazonaws.com/aws-cloudwatch/downloads/latest/awslogs-agent-set= up.py is a bit too daunting for me to try rewriting using my brand-new = rc.d scripting skills, with only a hundred lines working (!) so far = behind my belt. >=20 > hi rafal - i also took a look at that python setup script. >=20 > my initial impression was that is was full of linux'isms (the first = line being #!/usr/bin/python being the first thing i noticed). and = while a lot of the setup code is linux specific, i think there are some = bits in there which could be ported to freebsd. do you happen to know = if there is a git or other open-source repository available with the = awslogs-agent code out there? i didn't find anything after a quick = google search. >=20 > cheers, > -pete >=20 > --=20 > Pete Wright > pete@nomadlogic.org > @nomadlogicLA >=20 > _______________________________________________ > freebsd-cloud@freebsd.org mailing list > https://lists.freebsd.org/mailman/listinfo/freebsd-cloud > To unsubscribe, send any mail to = "freebsd-cloud-unsubscribe@freebsd.org" From owner-freebsd-cloud@freebsd.org Wed Aug 30 23:45:49 2017 Return-Path: Delivered-To: freebsd-cloud@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 8782DE0CE86 for ; Wed, 30 Aug 2017 23:45:49 +0000 (UTC) (envelope-from pete@nomadlogic.org) Received: from vps-mail.nomadlogic.org (unknown [IPv6:2607:f2f8:a098::2]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 69F9184FF4 for ; Wed, 30 Aug 2017 23:45:49 +0000 (UTC) (envelope-from pete@nomadlogic.org) Received: from [192.168.1.26] (cpe-23-242-94-236.socal.res.rr.com [23.242.94.236]) by vps-mail.nomadlogic.org (OpenSMTPD) with ESMTPSA id da4dfd39 TLS version=TLSv1.2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128 verify=NO; Wed, 30 Aug 2017 16:45:48 -0700 (PDT) Subject: Re: AWS CloudWatch Logs Agent To: Rafal Lukawiecki Cc: freebsd-cloud@freebsd.org References: <4FFC6F3B-E8A1-49EE-A05A-C6B9B0A615FF@rafal.net> <0100015e3565144c-18dc3e52-fcba-4b30-bf2f-58b99b55562d-000000@email.amazonses.com> <456A3074-08D2-4141-81EF-5CD6B1B53899@rafal.net> <084bd6ad-9ffd-c097-1d46-c8cb41d745ee@nomadlogic.org> <0A93F7FC-5547-4A67-B3D9-BB59027FDAD6@rafal.net> From: Pete Wright Message-ID: Date: Wed, 30 Aug 2017 16:45:47 -0700 User-Agent: Mozilla/5.0 (X11; FreeBSD amd64; rv:52.0) Gecko/20100101 Thunderbird/52.3.0 MIME-Version: 1.0 In-Reply-To: <0A93F7FC-5547-4A67-B3D9-BB59027FDAD6@rafal.net> Content-Type: text/plain; charset=utf-8; format=flowed Content-Transfer-Encoding: 8bit Content-Language: en-US X-BeenThere: freebsd-cloud@freebsd.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: "FreeBSD on cloud platforms \(EC2, GCE, Azure, etc.\)" List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 30 Aug 2017 23:45:49 -0000 On 08/30/2017 16:39, Rafal Lukawiecki wrote: > Dear Pete, that was my feeling too—I suspect that for someone more skilled than me in FreeBSD rc.d it should be realistically possible to port it, but the spaghetti nature of the awslogs script would also be something to contend with. I did a bit of searching, but the only reference to awslogs that I found on GH is this one: > > https://github.com/jorgebastida/awslogs > > While this looks like a good utility for CloudWatch Logs querying, it does not seem to deal with the agent and actual log collection. The AWS GitHub account does not seem to list anything related amongst their 70+ repositories... > > If you embark on such a port, I would be happy to test, help, and give feedback. On the other hand, please let me know if you find anything else that could be of use. Many thanks. hrm yea - so i did a bit of reading through that setup script and if i'm not mistaken it is actually installing the python package: https://pypi.python.org/pypi/awscli-cwlogs a majority of the code located at https://s3.amazonaws.com//aws-cloudwatch/downloads/latest/awslogs-agent-setup.py seems to configure the behaviour of awscli-cwlogs. i may poke at this later this week and see if i can get awscli-cwlogs working in my environment, if it looks good it may be a good starting point for a freebsd port of this functionality. also since it seems to be mostly python  code doing the actual log-shipping we might be in good shape. -p -- Pete Wright pete@nomadlogic.org @nomadlogicLA From owner-freebsd-cloud@freebsd.org Wed Aug 30 23:48:37 2017 Return-Path: Delivered-To: freebsd-cloud@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id EE484E0CECF for ; Wed, 30 Aug 2017 23:48:37 +0000 (UTC) (envelope-from raf@rafal.net) Received: from mxout-08.mxes.net (mxout-08.mxes.net [216.86.168.183]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id C813A89 for ; Wed, 30 Aug 2017 23:48:37 +0000 (UTC) (envelope-from raf@rafal.net) Received: from [192.168.40.37] (unknown [86.40.118.125]) (using TLSv1.2 with cipher DHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.mxes.net (Postfix) with ESMTPSA id DB713509B8; Wed, 30 Aug 2017 19:48:35 -0400 (EDT) Content-Type: text/plain; charset=us-ascii Mime-Version: 1.0 (Mac OS X Mail 10.3 \(3273\)) Subject: Re: AWS CloudWatch Logs Agent From: Rafal Lukawiecki In-Reply-To: Date: Thu, 31 Aug 2017 00:48:34 +0100 Cc: freebsd-cloud@freebsd.org Content-Transfer-Encoding: quoted-printable Message-Id: <38BA8CBD-8A74-4705-BB7B-5737C809540D@rafal.net> References: <4FFC6F3B-E8A1-49EE-A05A-C6B9B0A615FF@rafal.net> <0100015e3565144c-18dc3e52-fcba-4b30-bf2f-58b99b55562d-000000@email.amazonses.com> <456A3074-08D2-4141-81EF-5CD6B1B53899@rafal.net> <084bd6ad-9ffd-c097-1d46-c8cb41d745ee@nomadlogic.org> <0A93F7FC-5547-4A67-B3D9-BB59027FDAD6@rafal.net> To: Pete Wright X-Mailer: Apple Mail (2.3273) X-BeenThere: freebsd-cloud@freebsd.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: "FreeBSD on cloud platforms \(EC2, GCE, Azure, etc.\)" List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 30 Aug 2017 23:48:38 -0000 Pete, that sounds wonderful. Let me know when I can assist, and if it = works out, I will owe you a drink at the very least. Thank you, Rafal -- Rafal Lukawiecki Data Scientist and Director=20 Project Botticelli Ltd > On 31 Aug 2017, at 00:45, Pete Wright wrote: >=20 > hrm yea - so i did a bit of reading through that setup script and if = i'm not mistaken it is actually installing the python package: > https://pypi.python.org/pypi/awscli-cwlogs >=20 > a majority of the code located at = https://s3.amazonaws.com//aws-cloudwatch/downloads/latest/awslogs-agent-se= tup.py seems to configure the behaviour of awscli-cwlogs. >=20 > i may poke at this later this week and see if i can get awscli-cwlogs = working in my environment, if it looks good it may be a good starting = point for a freebsd port of this functionality. also since it seems to = be mostly python code doing the actual log-shipping we might be in good = shape. >=20 >=20 > -p >=20 > --=20 > Pete Wright > pete@nomadlogic.org > @nomadlogicLA >=20 From owner-freebsd-cloud@freebsd.org Wed Aug 30 23:50:50 2017 Return-Path: Delivered-To: freebsd-cloud@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id A9D64E0CFED for ; Wed, 30 Aug 2017 23:50:50 +0000 (UTC) (envelope-from pete@nomadlogic.org) Received: from vps-mail.nomadlogic.org (unknown [IPv6:2607:f2f8:a098::2]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 914175FA for ; Wed, 30 Aug 2017 23:50:50 +0000 (UTC) (envelope-from pete@nomadlogic.org) Received: from [192.168.1.26] (cpe-23-242-94-236.socal.res.rr.com [23.242.94.236]) by vps-mail.nomadlogic.org (OpenSMTPD) with ESMTPSA id d34985b2 TLS version=TLSv1.2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128 verify=NO; Wed, 30 Aug 2017 16:50:48 -0700 (PDT) Subject: Re: AWS CloudWatch Logs Agent To: Rafal Lukawiecki Cc: freebsd-cloud@freebsd.org References: <4FFC6F3B-E8A1-49EE-A05A-C6B9B0A615FF@rafal.net> <0100015e3565144c-18dc3e52-fcba-4b30-bf2f-58b99b55562d-000000@email.amazonses.com> <456A3074-08D2-4141-81EF-5CD6B1B53899@rafal.net> <084bd6ad-9ffd-c097-1d46-c8cb41d745ee@nomadlogic.org> <0A93F7FC-5547-4A67-B3D9-BB59027FDAD6@rafal.net> <38BA8CBD-8A74-4705-BB7B-5737C809540D@rafal.net> From: Pete Wright Message-ID: <2a296862-8364-9814-d307-971fa9cf62fd@nomadlogic.org> Date: Wed, 30 Aug 2017 16:50:47 -0700 User-Agent: Mozilla/5.0 (X11; FreeBSD amd64; rv:52.0) Gecko/20100101 Thunderbird/52.3.0 MIME-Version: 1.0 In-Reply-To: <38BA8CBD-8A74-4705-BB7B-5737C809540D@rafal.net> Content-Type: text/plain; charset=utf-8; format=flowed Content-Transfer-Encoding: 8bit Content-Language: en-US X-BeenThere: freebsd-cloud@freebsd.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: "FreeBSD on cloud platforms \(EC2, GCE, Azure, etc.\)" List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 30 Aug 2017 23:50:50 -0000 On 08/30/2017 16:48, Rafal Lukawiecki wrote: > Pete, that sounds wonderful. Let me know when I can assist, and if it works out, I will owe you a drink at the very least. lol no prob - although i promise nothing ;^)  but hopefully i can cobble together a proof of concept - i'll keep ya in the loop. -p -- Pete Wright pete@nomadlogic.org @nomadlogicLA From owner-freebsd-cloud@freebsd.org Thu Aug 31 16:47:57 2017 Return-Path: Delivered-To: freebsd-cloud@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id A7C7EE01C8F for ; Thu, 31 Aug 2017 16:47:57 +0000 (UTC) (envelope-from raf@rafal.net) Received: from mxout-08.mxes.net (mxout-08.mxes.net [216.86.168.183]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 84E8E83E9B for ; Thu, 31 Aug 2017 16:47:57 +0000 (UTC) (envelope-from raf@rafal.net) Received: from edina.glencottage.net (unknown [86.40.118.125]) (using TLSv1.2 with cipher DHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.mxes.net (Postfix) with ESMTPSA id 76A42509B8 for ; Thu, 31 Aug 2017 12:47:55 -0400 (EDT) From: Rafal Lukawiecki Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable Mime-Version: 1.0 (Mac OS X Mail 10.3 \(3273\)) Subject: Why pkg offers different versions of packages on two machines? Message-Id: <73266CFC-4040-4A48-8FAB-974439F7A690@rafal.net> Date: Thu, 31 Aug 2017 17:47:53 +0100 To: freebsd-cloud@freebsd.org X-Mailer: Apple Mail (2.3273) X-BeenThere: freebsd-cloud@freebsd.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: "FreeBSD on cloud platforms \(EC2, GCE, Azure, etc.\)" List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 31 Aug 2017 16:47:57 -0000 Apologies for another newbie question. Trying to get my head around = package management. If I run "pkg search certbot" using the official AWS FreeBSD = 11.1-RELEASE image it finds: py27-certbot-0.16.0_1,1 Let's Encrypt client but if run the same search using my locally installed FreeBSD, which = happens to be a slightly older, 11.0-STABLE, I find the newer (which = also happens to be the current) version of that package: py27-certbot-0.17.0,1 Let's Encrypt client In both cases pkg.conf and the repo config are the same, with all the = settings at their unchanged by me defaults (url: = "pkg+http://pkg.FreeBSD.org/${ABI}/quarterly=E2=80=9D). Both versions of = pkg itself are the same. As far as I can see, I am resolving to the same = mirror in both cases, as I see the same IP (213.138.116.73) when I ping = pkg.freebsd.org. When I manually navigate to = http://pkg.freebsd.org/FreeBSD:11:amd64/quarterly/All/ I can see the = older version of the package (py27-certbot-0.16.0_1,1) on the server. = When I navigate to the latest tree = (http://pkg.freebsd.org/FreeBSD:11:amd64/latest/All/) I can see the = newer package in there. Why does my locally installed pkg find the newer version if both configs = point to the same, quarterly, repo URL? Many thanks, Rafal -- Rafal Lukawiecki Data Scientist=20 Project Botticelli Ltd From owner-freebsd-cloud@freebsd.org Thu Aug 31 16:56:02 2017 Return-Path: Delivered-To: freebsd-cloud@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 49585E01E05 for ; Thu, 31 Aug 2017 16:56:02 +0000 (UTC) (envelope-from raf@rafal.net) Received: from mxout-08.mxes.net (mxout-08.mxes.net [216.86.168.183]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 2599C282 for ; Thu, 31 Aug 2017 16:56:01 +0000 (UTC) (envelope-from raf@rafal.net) Received: from edina.glencottage.net (unknown [86.40.118.125]) (using TLSv1.2 with cipher DHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.mxes.net (Postfix) with ESMTPSA id B36FF509BB for ; Thu, 31 Aug 2017 12:55:59 -0400 (EDT) From: Rafal Lukawiecki Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: quoted-printable Mime-Version: 1.0 (Mac OS X Mail 10.3 \(3273\)) Subject: Re: Why pkg offers different versions of packages on two machines? Date: Thu, 31 Aug 2017 17:55:58 +0100 References: <73266CFC-4040-4A48-8FAB-974439F7A690@rafal.net> To: freebsd-cloud@freebsd.org In-Reply-To: <73266CFC-4040-4A48-8FAB-974439F7A690@rafal.net> Message-Id: <687F0692-A1D6-4B90-89A9-34A9CB8C7EC4@rafal.net> X-Mailer: Apple Mail (2.3273) X-BeenThere: freebsd-cloud@freebsd.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: "FreeBSD on cloud platforms \(EC2, GCE, Azure, etc.\)" List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 31 Aug 2017 16:56:02 -0000 Oops, apologies for posting to freebsd-cloud, this should have gone to = freebsd-questions. I will repost there immediately. I will be more = careful the next time, so sorry. Rafal -- Rafal Lukawiecki Data Scientist and Director=20 Project Botticelli Ltd From owner-freebsd-cloud@freebsd.org Fri Sep 1 21:24:58 2017 Return-Path: Delivered-To: freebsd-cloud@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 59734E07EAE for ; Fri, 1 Sep 2017 21:24:58 +0000 (UTC) (envelope-from raf@rafal.net) Received: from mxout-08.mxes.net (mxout-08.mxes.net [216.86.168.183]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 2A2AB64545 for ; Fri, 1 Sep 2017 21:24:57 +0000 (UTC) (envelope-from raf@rafal.net) Received: from [192.168.40.37] (unknown [86.40.118.125]) (using TLSv1.2 with cipher DHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.mxes.net (Postfix) with ESMTPSA id 13243509B6 for ; Fri, 1 Sep 2017 17:24:49 -0400 (EDT) From: Rafal Lukawiecki Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable Mime-Version: 1.0 (Mac OS X Mail 10.3 \(3273\)) Subject: Launch and boot times of AWS FreeBSD AMI Message-Id: Date: Fri, 1 Sep 2017 22:24:46 +0100 To: freebsd-cloud@freebsd.org X-Mailer: Apple Mail (2.3273) X-BeenThere: freebsd-cloud@freebsd.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: "FreeBSD on cloud platforms \(EC2, GCE, Azure, etc.\)" List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 01 Sep 2017 21:24:58 -0000 I wonder if there is any possibility to shorten the launch and reboot = times of the FreeBSD AMI. I understand that it is necessary to have the = first post-install and update reboot, after the initial launch, but it = seems to take a longish time compared to Amazon Linux. Without any solid = measurements, it seems, from the =E2=80=9CVGA=E2=80=9D console, that it = does pause a bit after printing the SSH fingerprint and starting cron, = sendmail, configuring vt, scheduling a background fsck in 60 secs, and = printing these odd messages: [: =3D: unexpected operator [: =3D: unexpected operator mount: /dev/gpt/rootfs: Device busy [: =3D: unexpected operator ...all before rebooting itself for that first time. I wonder if it could be optimised, somehow. It would be nice to shave a = minute or more from the waits, if possible. I am also curious what those = errors are. Rafal -- Rafal Lukawiecki Data Scientist=20 Project Botticelli Ltd From owner-freebsd-cloud@freebsd.org Fri Sep 1 21:33:10 2017 Return-Path: Delivered-To: freebsd-cloud@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 18A99E0853E for ; Fri, 1 Sep 2017 21:33:10 +0000 (UTC) (envelope-from 0100015e3f5c9ca6-5867960d-56cb-4292-965a-eb57b007fe4b-000000@amazonses.com) Received: from a8-56.smtp-out.amazonses.com (a8-56.smtp-out.amazonses.com [54.240.8.56]) (using TLSv1 with cipher ECDHE-RSA-AES128-SHA (128/128 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id D279F649D3 for ; Fri, 1 Sep 2017 21:33:09 +0000 (UTC) (envelope-from 0100015e3f5c9ca6-5867960d-56cb-4292-965a-eb57b007fe4b-000000@amazonses.com) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/simple; s=ae7m2yrxjw65l2cqdpjxuucyrvy564tn; d=tarsnap.com; t=1504301587; h=Subject:To:References:From:Message-ID:Date:MIME-Version:In-Reply-To:Content-Type:Content-Transfer-Encoding; bh=YJ+vET1SaqaP8sb7tkJTitRYRtWjDrjK4XJ65NcS1i0=; b=YFZ9kPPHExsyu/HjmCX44Zw9GF+EpQFewTjvW6e7baDXY+p+CKlbIahAQfoKIEFQ WD7R0YHXEOWPMfx2FKCkgEWF5rcySpZCgNUuNr//VsE01ciugO3pJBMETiyndwASeOW lUGhbLZ1U8dFCw07iLRTgHBY4eyjxw5QAkW0unYw= DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/simple; s=224i4yxa5dv7c2xz3womw6peuasteono; d=amazonses.com; t=1504301587; h=Subject:To:References:From:Message-ID:Date:MIME-Version:In-Reply-To:Content-Type:Content-Transfer-Encoding:Feedback-ID; bh=YJ+vET1SaqaP8sb7tkJTitRYRtWjDrjK4XJ65NcS1i0=; b=Gfz0ayYQ62rfETH+FOffeZGG+fLqhbsgsCCwtWEF5Y6byfHKvi1e7NxjTKGvgcGx vseeJpAOE+ppWvnRLx87CH4TTMwOsnojnvV/dUmgfBcLG6DvVsIM3yWhg9va4EwMlz7 sVGEtdZXKCvAGgZHczPugNDSogkuSoweiRLKLbNg= Subject: Re: Launch and boot times of AWS FreeBSD AMI To: Rafal Lukawiecki , freebsd-cloud@freebsd.org References: From: Colin Percival Message-ID: <0100015e3f5c9ca6-5867960d-56cb-4292-965a-eb57b007fe4b-000000@email.amazonses.com> Date: Fri, 1 Sep 2017 21:33:07 +0000 User-Agent: Mozilla/5.0 (X11; FreeBSD amd64; rv:45.0) Gecko/20100101 Thunderbird/45.5.1 MIME-Version: 1.0 In-Reply-To: Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: 8bit X-SES-Outgoing: 2017.09.01-54.240.8.56 Feedback-ID: 1.us-east-1.Lv9FVjaNvvR5llaqfLoOVbo2VxOELl7cjN0AOyXnPlk=:AmazonSES X-BeenThere: freebsd-cloud@freebsd.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: "FreeBSD on cloud platforms \(EC2, GCE, Azure, etc.\)" List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 01 Sep 2017 21:33:10 -0000 On 09/01/17 14:24, Rafal Lukawiecki wrote: > I wonder if there is any possibility to shorten the launch and reboot times > of the FreeBSD AMI. I understand that it is necessary to have the first > post-install and update reboot, after the initial launch, but it seems to > take a longish time compared to Amazon Linux. Without any solid > measurements, it seems, from the “VGA” console, that it does pause a bit > after printing the SSH fingerprint and starting cron, sendmail, configuring > vt, scheduling a background fsck in 60 secs, and printing these odd > messages: Yes, it is annoyingly slow. Part of this is the initial freebsd-update pulling down updates, but there are definitely things which should be faster. > [: =: unexpected operator > [: =: unexpected operator > mount: /dev/gpt/rootfs: Device busy > [: =: unexpected operator I've been meaning to track down where these are coming from. Setting rc_debug=YES in /etc/rc.conf might help... > I wonder if it could be optimised, somehow. It would be nice to shave a > minute or more from the waits, if possible. I am also curious what those > errors are. One issue I *have* tracked down is that on large systems we spend a crazy amount of time initializing VM -- around 45 seconds on a x1.32xlarge. I tested a patch from markj last week but I don't think it has gone into the tree yet. -- Colin Percival Security Officer Emeritus, FreeBSD | The power to serve Founder, Tarsnap | www.tarsnap.com | Online backups for the truly paranoid