Skip site navigation (1)Skip section navigation (2)
Date:      Sun, 13 Oct 2019 11:01:06 +0200
From:      "Dave Cottlehuber" <dch@skunkwerks.at>
To:        freebsd-questions <freebsd-questions@freebsd.org>
Subject:   Re: Ansible for FreeBSD - use cases?
Message-ID:  <b9b6d85d-d59c-4b31-a396-24d8c806c7a6@www.fastmail.com>
In-Reply-To: <20191005141507.GA1223@admin.sibptus.ru>
References:  <20191005141507.GA1223@admin.sibptus.ru>

next in thread | previous in thread | raw e-mail | index | archive | help
On Sat, 5 Oct 2019, at 16:15, Victor Sudakov wrote:
> Dear Colleagues,
> 
> Is anyone using ansible on FreeBSD for anything useful in production? 

I look after a number of customers with reasonably complex
requirements, bare metal, network provisioning, jails & custom
apps.

Typically we use bare metal systems to boot up FreeBSD, and then
ansible-ify everything else from that.

> I've experimented with the pkgng module, it does install packages :-)
> Most modules are meant for Linux however.
> 
> What FreeBSD tasks do you automate with ansible, and with what modules?

I highly recommend sysutils/py-mitogen (for massive speed up) & https://github.com/austinhyde/ansible-sshjail (for remote jail access,
works like you expect it to.

The core functionality driven by ansible is roughly:

- node bootstrap
    - sshd, users, ntp
    - sudo, sysctls, unbound
    - tarsnaps, pf, zfs
    - freebsd-update & IDS
- set up zerotier mesh network to link jails across servers
- jails (currently with iocell but soon "simple jails")
- haproxy, pf, bgp networking, spiped, TLS certs
- collectd, rsyslog, smtp
- everything else lives in jails
    - graylog, riemann, vault
    - clustered rabbitmq & couchdb
    - a few bespoke apps in erlang & elixir
- build server
    - setup poudriere
    - kick off builds from a custom ports repo

Updating apps is as simple as:

- use BGP to offline a node from incoming data
- shut down jails
- zfs destroy zroot/jails
- re-run ansible to deploy all the things
- let haproxy figure out things are up again

We have datasets like zroot/jailed/app which contains all the 
permanent data, and the app & config lives in zroot/jails/app
which is both transient and deployed by ansible.


Some of the stuff never changes, so I've started using
pkg-create(8) to wrap up a lot of the stuff that I used
to have as configs, and simply deploy that as a package
via ansible.

> I use net/rdist6 to update some configs on remote hosts, so I don't
> think I'll benefit much from file copying modules of ansible.

Ansible shines for the case where you need more variable
injection than rdist or simple shell scripts can provide,
but there are a few things that I *don't* enjoy about
ansible.

1. ansible suffers from its own success. GitHub is a
terrible solution for managing 1000s of ever-changing
PRs for modules, and I don't think they've yet found
an appropriate approach for engaging with the community
to keep PRs flowing. This is a generic issue for any
FLOSS project of this level of activity.

2. hiera - if you've used puppet, you'll miss this.
While ansible provides a similar hash table to merge
inventory data, none of the community provided roles
can use this so everything is "namespaced" via
snake_case:  postgres_config_thing instead of some
postgres.config.thing (where each level is an actual
hash). Where this matters is when you want a group
of nodes to be able to override a particular
setting, but inherit elsewhere.

3. DAG - if you're doing DevOps with tools that have
no concept of dependencies, you're doing it wrong.
For example, if 90% of the play is already set up,
there is no need to re-run it all from scratch every
time. I work around this with keeping plays in small
task-focused sets, and using conditional clauses to
skip large chunks of plays if the final condition
of the play is already set.

4. versions, the core is changing continuously. You
basically need to keep up with the leading edge to
remain in support. This means updating your plays
every quarter, and patching up anything that's out
of whack. There's always something I use that's
being deprecated, but only once has this been
problematic.

That aside, generally I'm pretty happy with ansible,
especially for FreeBSD. Most roles comprise the same
steps:

- make a new jail
- install some packages
- setup a few directories & zfs datasets
- ensure daemon is running
- ensure data is synced across cluster nodes/jails
- tweak settings for jails & proxies for a home
  grown "service mesh"

I guess it takes around 1-2 hours to make an ansible
role for any given application we wish to run in
this way, across multiple jails.

It would be great if FreeBSD were more programmable
in some ways - more libxo for plumbing orchestration
tools into the system, things like iflib or a libfw
to programmatically manage networks instead of
wrapping ansible line-in-file and block functions,
but right now it's good enough.

Jail and zfs support makes isolating things really
easy and the performance is more than sufficient
for my needs. The zfs books & the jails book are
excellent companions.

Happy to expand on this if something above is of
interest.

A+
Dave



Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?b9b6d85d-d59c-4b31-a396-24d8c806c7a6>