In this post I’ll show you how I used Pallet to configure the server hosting this blog.
What and why?
I’m setting up a web server to host a bunch of static HTML, CSS and image files. That’s a pretty simple task, so what is Pallet and why am I using it?
I want to have reproducible server configuration. If I ever need to move quanttype.net to a new server, I do not want to figure out which packages to install and which hand-edited configuration files I need to copy over. Various configuration management tools such as this problem by programmatically applying my configuration to a server.
I’ve earlier dabbled with some of more popular configuration management tools such as Puppet and Chef. This time I chose to use Pallet because it’s simpler and hence suits my simple needs better.
Pallet: cloud automation with Clojure
Pallet is a Clojure library that can be used to describe a server configuration and then apply it to a server. It’s simple to use: all you need is a Clojure REPL on your local computer and SSH connection to your servers. Pallet also suppert automatically spinning up nodes with cloud providers such as Amazon or Rackspace.
Pallet’s abstraction level is somewhere between those of Puppet and Fabric. You can easily use pre-made modules (“crates” in Pallet parlance) for installing and configuring common software like nginx, but you do not need to set up any central servers or repositories.
Acquiring a server
Quanttype is hosted on DigitalOcean, mainly because their prices are cheap. Pallet does not support their API, so I manually created a Ubuntu VM, or a droplet as DigitalOcean calls them. Before using Pallet I did some setup on the server:
-
Create a non-root user with passwordless sudo access. This is not mandatory: you could use Pallet as root as well, but this the way I like to organize the things.
# as root adduser arcatan visudo -
Install your SSH key. Pallet uses SSH to connect to the server and you don’t want to store your password in your Pallet configuration.
# on your own computer ssh-copy-id quanttype.net
A new Pallet project
Pallet’s getting started guide
suggests to use Leiningen to create a new project with pallet template. I did
so, but I didn’t know what to do with all the stuff Leiningen put there, so I
revereted back to a plain Clojure project.
lein init quanttype-ops
Frankly, I had a hard time following Pallet’s documentation on this. It covers okay how you do various things, but it’s lacking on how to structure your Pallet project.
The stable release of Pallet is 0.7, but 0.8 is going to be released soon. The first release candidate is out there already, so it’s probably a good idea to base new projects on that.
Pallet is distributed as a Clojure library, so I added [com.palletops/pallet "0.8.0-RC.1"] to project.clj. I’m also going to use nginx-crate, so I’ll add
that, too.
(defproject quanttype-ops "0.1.0-SNAPSHOT"
:description "Pallet for quanttype.net"
:dependencies [[org.clojure/clojure "1.5.1"]
[com.palletops/pallet "0.8.0-RC.1"]
[org.clojars.strad/nginx-crate "0.8.3"]])
Main configuration
Now we can start configuring the server. To properly understand this walkthrough, keep Pallet documentation handy.
I put my configuration in the
namespace net.quanttype.ops.core. Here’s the ns form:
(ns net.quanttype.ops.core
(:use
pallet.actions
pallet.api
pallet.compute
pallet.crate.nginx)
(:require
[pallet.crate :refer [defplan]]))
Pallet does not support DigitalOcean’s API, but once you’ve acquired a server,
you can just give Pallet a list of IPs of your server and it will happily
connect there. To do so, you need to define a node-list compute
service:
(def digital-ocean
(instantiate-provider
"node-list"
:node-list
;; A list of nodes: [name group IP operating-system]
[["quanttype.net" "web" "37.139.12.210" :ubuntu]]))
Pallet also needs to know about the user account it should use. As I already set up an account that uses my default SSH key and has passwordless sudo, all I need is to tell Pallet the username.
(def my-user
(make-user "arcatan"))
First, I’m going to want some packages for easier usage of the server. I
created a server specification, which install my packages. In
Pallet parlance phase is a sequnce of actions to be executed. Different phases
are applied at different times - :bootstrap is applied when a new node is
started. I also set my shell to be zsh.
(def with-my-packages
(server-spec
:phases {:bootstrap
(plan-fn
(packages :aptitude ["git" "zsh" "vim"])
(user (:username my-user) :shell :zsh)}))
Next, I will set up nginx. This is easiest to with nginx-crate. I used Ryan Stradling’s fork which has been updated to Pallet 0.8. The configuration keys match those of the real nginx configuration files, so it’s best to see nginx documentation to figure out what you need.
(def http-server-config
{:install-strategy :packages
:user "www-data"
:group "www-data"
:sites [{:action :enable
;; Name of the configuration file must be something.site,
;; because nginx's main configuration files includes *.site.
:name "quanttype.site"
:servers [{:server-name "quanttype.net www.quanttype.net \"\""
:listen "80"
:index "index.html"
:root "/var/www/quanttype.net"}]}]})
With this configuration, nginx serves quanttype.net from
/var/www/quanttype.net. Here’s a plan function that makes sure that the
directory exists and has the proper rights for me to rsync files there.
(defplan quanttype-directories
[]
(group "www-data")
(user "www-data"
:system true
:home "/var/www"
:create-home false
:shell false
:group "www-data")
(exec-script ("usermod" "-a" "-G" "www-data" (:username my-user))
(directory "/var/www/quanttype.net"
:owner (:username my-user)
:group "www-data"
:mode "0755"))
We need another server specification to encapsulate the nginx configuration.
The key here is that it extends (nginx http-server-config), where nginx is
a function provided by nginx-crate.
(def quanttype-server
(server-spec
:extends [(nginx http-server-config)]
:phases {:configure (plan-fn (quanttype-directories))}))
Finally, to pull everything together, I define the "web" group. In the
node-list we defined quanttype.net’s group as web and this group
specification maps it to the server specifications we saw above.
(def web-group
(group-spec
"web"
:extends [with-my-config quanttype-server]))
Applying the configuration
The configuration is applied to the servers with lift and
converge. I have only one server, so lift is what I need.
Here’s a helper function for executing the config.
(defn execute
[& args]
(apply lift
web-group
:user my-user
:compute digital-ocean
args))
Now, open a REPL and use (execute). Because I didn’t use Pallet to provision
the nodes, I need to run :bootstrap myself.
user> (use 'net.quanttype.ops.core)
nil
user> (execute :bootstrap)
user> (execute :install) # custom phase by nginx-crate to install nginx
user> (execute) # :configure is executed by default
user> (execute :restart) # custom phase by nginx-crate to restart nginx
Ta-da, everything is ready and the only thing left is to rsync the content to the server.
Conclusion
Now you’ve seen how to automatically install and configure nginx with Pallet. Hopefully sharing this helps people to get started with Pallet. I found that it’s pretty easy to use after a while, but figuring out what to do at first is hard.
Here’s a to-do list for the future:
- Testing the configuration locally in VirtualBox
- Automating the pre-Pallet steps
- Support for DigitalOcean in Pallet
Another thing I’ve been looking at is Docker. It packages applications in isolated containers that can be easily deployed anywhere (as long as you’re running modern-enough Linux). If it catches on, it might a very good idea for future-proofing the setup. I’m not yet quite sure how it fits in this picture, but I’m keeping an eye on it.