Setting up nginx for static content with Pallet

In this post I’ll show you how I used Pallet to configure the server hosting this blog.

What and why?

I’m setting up a web server to host a bunch of static HTML, CSS and image files. That’s a pretty simple task, so what is Pallet and why am I using it?

I want to have reproducible server configuration. If I ever need to move quanttype.net to a new server, I do not want to figure out which packages to install and which hand-edited configuration files I need to copy over. Various configuration management tools such as this problem by programmatically applying my configuration to a server.

I’ve earlier dabbled with some of more popular configuration management tools such as Puppet and Chef. This time I chose to use Pallet because it’s simpler and hence suits my simple needs better.

Pallet: cloud automation with Clojure

Pallet is a Clojure library that can be used to describe a server configuration and then apply it to a server. It’s simple to use: all you need is a Clojure REPL on your local computer and SSH connection to your servers. Pallet also suppert automatically spinning up nodes with cloud providers such as Amazon or Rackspace.

Pallet’s abstraction level is somewhere between those of Puppet and Fabric. You can easily use pre-made modules (“crates” in Pallet parlance) for installing and configuring common software like nginx, but you do not need to set up any central servers or repositories.

Acquiring a server

Quanttype is hosted on DigitalOcean, mainly because their prices are cheap. Pallet does not support their API, so I manually created a Ubuntu VM, or a droplet as DigitalOcean calls them. Before using Pallet I did some setup on the server:

  1. Create a non-root user with passwordless sudo access. This is not mandatory: you could use Pallet as root as well, but this the way I like to organize the things.

    # as root
    adduser arcatan
    visudo
  2. Install your SSH key. Pallet uses SSH to connect to the server and you don’t want to store your password in your Pallet configuration.

    # on your own computer
    ssh-copy-id quanttype.net

A new Pallet project

Pallet’s getting started guide suggests to use Leiningen to create a new project with pallet template. I did so, but I didn’t know what to do with all the stuff Leiningen put there, so I revereted back to a plain Clojure project.

lein init quanttype-ops

Frankly, I had a hard time following Pallet’s documentation on this. It covers okay how you do various things, but it’s lacking on how to structure your Pallet project.

The stable release of Pallet is 0.7, but 0.8 is going to be released soon. The first release candidate is out there already, so it’s probably a good idea to base new projects on that.

Pallet is distributed as a Clojure library, so I added [com.palletops/pallet "0.8.0-RC.1"] to project.clj. I’m also going to use nginx-crate, so I’ll add that, too.

(defproject quanttype-ops "0.1.0-SNAPSHOT"
  :description "Pallet for quanttype.net"
  :dependencies [[org.clojure/clojure "1.5.1"]
                 [com.palletops/pallet "0.8.0-RC.1"]
                 [org.clojars.strad/nginx-crate "0.8.3"]])

Main configuration

Now we can start configuring the server. To properly understand this walkthrough, keep Pallet documentation handy.

I put my configuration in the namespace net.quanttype.ops.core. Here’s the ns form:

(ns net.quanttype.ops.core
  (:use
   pallet.actions
   pallet.api
   pallet.compute
   pallet.crate.nginx)
  (:require
   [pallet.crate :refer [defplan]]))

Pallet does not support DigitalOcean’s API, but once you’ve acquired a server, you can just give Pallet a list of IPs of your server and it will happily connect there. To do so, you need to define a node-list compute service:

(def digital-ocean
  (instantiate-provider
    "node-list"
    :node-list
    ;; A list of nodes: [name group IP operating-system]
    [["quanttype.net" "web" "37.139.12.210" :ubuntu]]))

Pallet also needs to know about the user account it should use. As I already set up an account that uses my default SSH key and has passwordless sudo, all I need is to tell Pallet the username.

(def my-user
  (make-user "arcatan"))

First, I’m going to want some packages for easier usage of the server. I created a server specification, which install my packages. In Pallet parlance phase is a sequnce of actions to be executed. Different phases are applied at different times - :bootstrap is applied when a new node is started. I also set my shell to be zsh.

(def with-my-packages
  (server-spec
   :phases {:bootstrap
            (plan-fn
             (packages :aptitude ["git" "zsh" "vim"])
             (user (:username my-user) :shell :zsh)}))

Next, I will set up nginx. This is easiest to with nginx-crate. I used Ryan Stradling’s fork which has been updated to Pallet 0.8. The configuration keys match those of the real nginx configuration files, so it’s best to see nginx documentation to figure out what you need.

(def http-server-config
  {:install-strategy :packages
   :user "www-data"
   :group "www-data"
   :sites [{:action :enable
            ;; Name of the configuration file must be something.site,
            ;; because nginx's main configuration files includes *.site.
            :name "quanttype.site"
            :servers [{:server-name "quanttype.net www.quanttype.net \"\""
                       :listen "80"
                       :index "index.html"
                       :root "/var/www/quanttype.net"}]}]})

With this configuration, nginx serves quanttype.net from /var/www/quanttype.net. Here’s a plan function that makes sure that the directory exists and has the proper rights for me to rsync files there.

(defplan quanttype-directories
  []
  (group "www-data")
  (user "www-data"
        :system true
        :home "/var/www"
        :create-home false
        :shell false
        :group "www-data")
  (exec-script ("usermod" "-a" "-G" "www-data" (:username my-user))
  (directory "/var/www/quanttype.net"
             :owner (:username my-user)
             :group "www-data"
             :mode "0755"))

We need another server specification to encapsulate the nginx configuration. The key here is that it extends (nginx http-server-config), where nginx is a function provided by nginx-crate.

(def quanttype-server
  (server-spec
   :extends [(nginx http-server-config)]
   :phases {:configure (plan-fn (quanttype-directories))}))

Finally, to pull everything together, I define the "web" group. In the node-list we defined quanttype.net’s group as web and this group specification maps it to the server specifications we saw above.

(def web-group
  (group-spec
   "web"
   :extends [with-my-config quanttype-server]))

Applying the configuration

The configuration is applied to the servers with lift and converge. I have only one server, so lift is what I need. Here’s a helper function for executing the config.

(defn execute
  [& args]
  (apply lift
         web-group
         :user my-user
         :compute digital-ocean
         args))

Now, open a REPL and use (execute). Because I didn’t use Pallet to provision the nodes, I need to run :bootstrap myself.

user> (use 'net.quanttype.ops.core)
nil
user> (execute :bootstrap)
user> (execute :install)  # custom phase by nginx-crate to install nginx
user> (execute)           # :configure is executed by default
user> (execute :restart)  # custom phase by nginx-crate to restart nginx

Ta-da, everything is ready and the only thing left is to rsync the content to the server.

Conclusion

Now you’ve seen how to automatically install and configure nginx with Pallet. Hopefully sharing this helps people to get started with Pallet. I found that it’s pretty easy to use after a while, but figuring out what to do at first is hard.

Here’s a to-do list for the future:

  • Testing the configuration locally in VirtualBox
  • Automating the pre-Pallet steps
  • Support for DigitalOcean in Pallet

Another thing I’ve been looking at is Docker. It packages applications in isolated containers that can be easily deployed anywhere (as long as you’re running modern-enough Linux). If it catches on, it might a very good idea for future-proofing the setup. I’m not yet quite sure how it fits in this picture, but I’m keeping an eye on it.


Comments or questions? Tweet to me or send me an e-mail.