Plug in to Elixir

I was involved in building the capomastro project, which manages building multiple dependencies in Jenkins, and the subsequent archiving of those dependencies.

Part of it involves handling HTTP notifications from Jenkins when a build transitions through various states.

I’ve been looking at micro-service implementations of the various parts of the service.

I started off with a simple test to get started…pushing a simple JSON notification in the same format as the Jenkins notification plugin sends us.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
defmodule ExMastro.JenkinsNotificationsTest do
  use ExUnit.Case, async: true
  use Plug.Test
 
  @opts ExMastro.JenkinsNotifications.init([])
 
  test "emits appropriate notification event" do
    body = %{name: "mytestjob", url: "job/mytestjob", build: %{number: 11, phase: "STARTED", url: "job/mytestjob/11/"}}
    conn = conn(:post, "/jenkins/notifications/?server=1", Jazz.encode!(body), headers: [{"content-type", "application/json"}])
 
    conn = ExMastro.JenkinsNotifications.call(conn, @opts)
 
    assert conn.status == 201
    assert conn.resp_body == "mytestjob"
  end
end

Line 8 creates a map in the same format as the Jenkins notifications plugin sends.

Line 9 creates a Plug.Conn record, which is what Plug supplies to “plugs” when they’re handling a request, and then manually calls the Plug (in this case, a “module” Plug) to handle the request.

Line 11 calls the Routing plug to handle the connection, allowing it to dispatch based on URL and HTTP method.

Running the tests fails…not unreasonably, as we have no router or dependencies setup.

defmodule ExMastro.Mixfile do
  use Mix.Project
 
  def project do
    [app: :exmastro,
     version: "0.0.1",
     elixir: "~> 0.15.1",
     deps: deps]
  end
 
  # Configuration for the OTP application
  #
  # Type `mix help compile.app` for more information
  def application do
    [applications: [:logger, :cowboy, :plug, :jazz],
     mod: {ExMastro, []}]
  end
 
  def deps do
    [{:cowboy, "~> 1.0.0"},
     {:plug, "~> 0.5.3"},
     {:jazz, github: "meh/jazz"}]
  end
end

Note, this app uses the Plug module, and Cowboy to do the web-serving, and Jazz to encode/decode JSON.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
defmodule ExMastro.JenkinsNotifications do
  import Plug.Conn
  use Plug.Router
 
  plug Plug.Parsers, parsers: [ExMastro.Plugs.Parsers.JSON]
  plug :match
  plug :dispatch
 
  def init(options) do
     options
  end
 
  post "/jenkins/notifications" do
    conn = Plug.Conn.fetch_params(conn)
    # We have the server in conn.params["server"]·
    # And the rest of the build in the conn.params[:json]
    send_resp(conn, 201, conn.params[:json]["name"])
  end
 
  match _ do
    send_resp(conn, 404, "oops")
  end
end

Line 5 is interesting here, Plugs are stackable, which in practice means that they get called in order to handle HTTP requests, and as long as they return a Plug.Conn structure, and don’t call send_response, they will continue.

Line 5 puts a parsing Plug into the top of  stack, parsers are expected to check to see whether or not they can process the content of the request, and if so, they return a tuple of:

{:ok, a map with things to put into the conn.params structure, and the Conn}

This code should look for errors in the body (from the parsing plug), and do something better…

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
defmodule ExMastro.Plugs.Parsers.JSON do
  @moduledoc false
  alias Plug.Conn
 
  def parse(%Conn{} = conn, "application", "json", _headers, opts) do
    case Conn.read_body(conn, opts) do
      {:ok, body, conn} ->
        {:ok, %{json: Jazz.decode!(body)}, conn}
      {:more, _data, conn} ->
        {:error, :too_large, conn}
    end
  end
 
  def parse(conn, _type, _subtype, _headers, _opts) do
    {:next, conn}
  end
end

Line 5 uses pattern matching to pick up on “application/json” content, then attempts to read the body from the Plug.Conn record, handling errors, or…parsing them as JSON, note the ! to parse and raise if it can’t parse the JSON.

This would cause the service to crash and burn, but we can setup a supervision tree to restart it.

Django on Dokku

I’ve been experimenting a bit with Dokku a bit recently, with a view to automating deployment of some small services.

Getting a Django app up and running using Dokku is fairly easy…

I’m using dokku-alt here, as it preconfigures some useful plugins and has some feature improvements over the original.

1) Create virtual machine.

Start a virtual machine wherever you can (I used a .5GiB host in Digital Ocean for this.)

2) Configure DNS entries

Configure a wildcard DNS entry for IP for your virtual machine, I run bind in my internal network at home so, for me this was a simple as creating a zone for it internally.

3) Install Dokku

Follow the dokku-alt instructions for installation.

$ curl -s https://raw.githubusercontent.com/dokku-alt/dokku-alt/master/bootstrap.sh | sudo sh

From now on, it mostly follows the “Getting started with Django on Heroku” article.

4) Prepare local virtualenv

Create a new virtualenv locally (I use virtualenvwrapper), to test your first app, and install the django-toolbelt package from the Cheeseshop.

$ mkvirtualenv djangokku && pip install django-toolbelt

5) Create a new Django app…

$ django-admin.py startproject djangokku

6) Setup the requirements

Change to the djangokku project directory and write a requirements.txt file.

$ cd djangokku
$ pip freeze > requirements.txt

7) Configure Django settings

Update the djangokku/settings.py per the instructions per the Heroku instructions.

Note, for Django 1.6 onwards, you don’t need to set the BASE_DIR as it’s already in the settings.py by default.

Also, I found the STATICFILES_DIRS settings were unnecessary.

8) Configure WSGI to serve static

This would be better done from Nginx.

Configure the WSGI file per the Heroku instructions

I left in the code to configure the environment for the correct settings that’s written out in Django, thus my djangokku/wsgi.py file looks like this:

"""
WSGI config for djangokku project.

It exposes the WSGI callable as a module-level variable named ``application``.

For more information on this file, see

https://docs.djangoproject.com/en/1.6/howto/deployment/wsgi/

"""

import os
os.environ.setdefault("DJANGO_SETTINGS_MODULE", "djangokku.settings")

from django.core.wsgi import get_wsgi_application
from dj_static import Cling

application = Cling(get_wsgi_application())

9) Add a Procfile

The Procfile file “is a mechanism for declaring what commands are run by your application’s dynos on the [Dokku] platform.”

$ echo "web: gunicorn djangokku.wsgi" > Procfile

10) Setup for using git to deploy

Add at least this to your .gitignore file.

*.pyc

Then setup your Django project as a git repository.

$ git init .
$ git add .
$ git commit -am "Initial commit."

11) Configure git for deployments

Add a git remote to deploy directly to the Dokku instance, note that my internal DNS resolves www.dokku via the wildcard to my Digital Ocean instance.

$ git remote add dokku dokku@www.dokku:djangokku

12) Deploy!

$ git push dokku master

This should push your code to your server, and deploy it…

         Running setup.py install for pystache
           pystache: using: version '2.1' of 

           Installing pystache script to /app/.heroku/python/bin
           Installing pystache-test script to /app/.heroku/python/bin
         Running setup.py install for static

           Installing static script to /app/.heroku/python/bin
       Successfully installed Django dj-database-url dj-static django-toolbelt gunicorn psycopg2 pystache static
       Cleaning up...
-----> Preparing static assets
       Running collectstatic...
       69 static files copied to '/app/staticfiles'.

-----> Discovering process types
       Procfile declares types -> web
remote: Sending build context to Docker daemon 2.048 kB
remote: Sending build context to Docker daemon
Step 0 : FROM dokku/djangokku:build
 ---> d158b4a7afb4
Step 1 : ENV GIT_REV 981b43800b11a07eaadd7fbed628bffbc2dc9507
 ---> Running in b4fb0b26d961
 ---> 65ad2d564bb6
Removing intermediate container b4fb0b26d961
Successfully built 65ad2d564bb6
-----> Releasing djangokku ...
-----> Injecting Supervisor ...
-----> Deploying djangokku ...
=====> Application deployed:

http://djangokku.dokku

-----> Cleaning up ...
To dokku@www.dokku:djangokku
 * [new branch]      master -> master

Visiting your newly deployed app should get the familiar “Congratulations on your first Django-powered page.”

13) Databases!

Django apps require Databases…in our case, we’ll connect djangokku to a PostgreSQL database server…

To do this, we need to ssh to the server that’s hosting dokku and create a database using dokku.

$ dokku postgresql:create testing

And link this database to our djangokku application.

dokku postgresql:link djangokku testing

This will redeploy the app, connected to the database:

CREATE ROLE
GRANT
-----> Releasing djangokku ...
-----> Injecting Supervisor ...
-----> Deploying djangokku ...
=====> Application deployed:

http://djangokku.dokku

You can now run the database sync just as you would locally:

$ dokku run djangokku python ./manage.py syncdb

Creating tables ...
Creating table django_admin_log
Creating table auth_permission
Creating table auth_group_permissions
Creating table auth_group
Creating table auth_user_groups
Creating table auth_user_user_permissions
Creating table auth_user
Creating table django_content_type
Creating table django_session

You just installed Django's auth system, which means you don't have any superusers defined.
Would you like to create one now? (yes/no): yes
Username (leave blank to use 'root'): kevin
Email address: kevin@bigkevmcd.com
Password:
Password (again):
Superuser created successfully.
Installing custom SQL ...
Installing indexes ...
Installed 0 object(s) from 0 fixture(s)

That’s the basics of getting a Django app up and running in Dokku.

Branching Strategies

I’ve been using Bazaar for over 6 years, distributed version control was for me, a huge improvement over what we used previously, Subversion, which was itself a huge improvement over CVS.

Twice, we’ve thought about branching, and twice, we’ve ended up with the same basic strategy.

We want to have active development on trunk, a staging deployment, mainly to test code with a production-like setup (and data volumes), and a production release.

The production release has been scheduled as often as a week (and we’ve had more frequently than that), but has sometimes been a month or more apart (after we got out of initial development of features).

What we ended up with was three main branches, named for their deployment environments, with trunk, production and staging branches.

Show-stopper defects found in staging are fixed in staging, and the fix merged back to trunk, same for production.

Trunk is almost always deployable, but, we don’t currently use feature-flags to hide “part-baked” functionality in production, and we rarely use long-lived feature branches to develop features against.

All features/fixes are branches from trunk, and committed right back to trunk once they’ve been reviewed, this approach can sometimes lead to “super-branches”, huge branches which incorporate everything from the infrastructure to the UI, and these are problematic to review, I can definitely recommend keeping branches below 200 lines of diff for ease of reviewing.

Instead of feature-flags, we’ve tended to build from the bottom-up, so…put in the infrastructure to support the UI, and then add the UI at the end.

This hasn’t always worked well, mainly because it’s often that only seeing the UI that a user has their requirements solidified…and that can mean some rework, we’ve gotten round that by having pre-merge-to-trunk customer acceptance demos, but even this can be a bit tricky, as the user doesn’t get to use the software themselves, generally they’re getting a screen-shared demo.

We rarely have show-stopper-fix-in-production bugs, most issues are things that can make their way through the trunk-staging-production process

Scrum, Scrumban and KanBan

We switched a couple of months ago from a failing Scrum-like process to a mostly-Kanban process.

Why did we switch?

We were attempting Scrum, but in reality, our team is too small, and we have a lot of new tasks coming in, these were tasks that couldn’t really be put off, and were pushing out our burndown a lot, our sprints ended with trying to get features landed, and the day before the end invariably had several reviews in the queue.

When we were committing to a set of tasks, our customers weren’t, so every period we ended rushing to get stuff ready for the bi-weekly demo, but many of the “we also spent a fair bit of time doing this” items weren’t being recorded.

Another big failing was that we had our bi-weekly iteration planning meeting late on a friday afternoon (UK time), so after a while, when questions needed to be asked, we didn’t…because it made the end of our week that little bit closer.

Also, there was too much “throw it over the wall” from both the customer and the developers, the demo was failing (as Scrum demos are somewhat likely to) because developers don’t demo “bugs”, we failed to identify the stories of most value to the customer early, they didn’t use the product, until very late in the development cycle, and so we didn’t get into a good iterative pattern.

Personally, the goal (and the gratification) of developing software is people using it, and I believe we left that too late, for various reasons.

So, Scrum just wasn’t working very well.

What were we hoping to gain?

We wanted to have better recording of tasks, when you have a prescribed story list, but lots of interrupt items, we discovered the interrupt items weren’t being recorded very well, this makes commitments to stories within a sprint hard.

We wanted to shorten the release cadence, it’s of course possible to disconnect sprints and releases in a Scrum process, but when things are focussed on the end-of-sprint demo, the expectation is that we’re demoing the stuff that will be available in the release, so we naturally fall into bi-weekly releases.

Tightening the release period, means shorter feedback loops, but this has to be understood as a way of changing things rapidly.

Kanban in practice

Since we moved to Kanban, we’ve mostly brought our releases into a regularly weekly cadence, but as the classic “pigs and chickens” story for Scrum illustrates, without commitment from all parties, this is hard, “common purpose” is way underestimated in companies, different teams have different agendas, and even if they need a feature, that doesn’t mean they’ll have time to actually shepherd that feature through to completion.

We brought in fairly strict Work-in-Progress (WIP) limits for our lanes, and we’ve incrementally added lanes, to reflect our process better, and there’s resistance to adding lanes (which IMO is good), but we’ve increased them from three to eight lanes since we started.

We’ve been using LeankitKanban, which has proved “ok” at best, the UI is not great, but functionally it works.

We don’t have WIP limits for some lanes, specifically our release lanes, but we’ve found that including an “External” lane,  has proved useful to keep an eye on tasks that are dependent on someone from outside our team, we can check-in with the tasks in the lane regularly, and ensure they’re not being forgotten about, we put in the “Blocked” field what it is that we are waiting on, to ensure that it’s clearly defined (or at least as clearly defined as it can be).

We have better recording of our interrupt tasks, I’m now confident that we’re recording pretty much every task that we spend time on, this proves useful in identifying repetitive tasks, as well as the more obvious use of allowing someone to pick up tasks in the event someone’s off on holiday or sick.

We started off overfilling the “Not Started” branch, which made prioritisation harder, it’s demonstrably easier to prioritise a small number of items than a large list, and we we tended to have 15-20 items in “Not Started”, but be putting items at the top of the lane regularly, so the bottom items were never getting anywhere.

We’re now keeping only 6-10 items in the “Not Started” lane, clearly prioritised, and with useful input from stakeholders on the prioritisation of items, this tightens the feedback loop on bugs in particular, as if they’re at the top of the “Not Started” lane, then they will be worked on very soon.

We’ve had problems with incomplete cards, these are problematic because they block the “In Progress” lane for an indeterminate time while we work out what to do with them, now we’re fairly strict about ensuring that before a card can be moved from the Icebox, it’s well enough understood that it can be worked on, this is an ideal opportunity for knowledge sharing to ensure that the card can be implemented by anybody in the team, as well as the opportunity to discuss the card before it’s taken up by a single developer, one of the big problems in remote-working is that you can’t really have  “over-the-desk” conversations, bouncing ideas around, the spontaneity is lost when having to start a Google Hangout, so fixes tend to be made in isolation, properly capturing the conversation around a card as it’s moved into “Not Started” is essential tho’.

Conclusion

The jury’s still out on Kanban, there’s a misconception that Kanban is a process, when it’s really a visualisation of your workflow, if your workflow is unclear, or faulty, Kanban won’t magically fix it, but it provides tools to make it easy to identify bottlenecks in your existing workflow, and it doesn’t define a workflow itself.

The options for lanes are virtually infinite, and you have to create the lanes to reflect your current workflow, identify the problems using  gathered data (and this is one area where Leankit really shines), and then come up with solutions, and repeat the process…

The significance of the friction that WIP generates has to be understood properly, it’s there to force discussions, it should be healthy friction, in a team with good “common purpose”, hitting the WIP limit for a lane is the siren that says “There’s a blockage here, what can I do to unblock it?”.  If you’re unable to unblock it, then a wider conversation is needed, extending the WIP limit without this conversation would be a mistake, but an easy one to make.

One of the other tenets of Kanban is continuous improvement, experimentation is encouraged, with such a loose mechanism, it’s easy to increase the WIP in a lane for a period, and assess the implications for other workflow stages.

We’ve recently tried to limit the number of cards in the “Not Started” lane, this has has meant that we’ve run out of items a couple of times, but we get together and move some items over, but the upside is that we’ve had better prioritisation.

Progress in my projects…

I hibernated over winter, not doing much around the house, but as the weather’s picked up (and with the imminent visit of my mum and dad for the first time), I’ve started to push my projects forward.

I bought a 5m reel of 5050 RGB LEDs, which are destined for under-cupboard lighting in the kitchen, in order to improve this, I’ve been experimenting with Jeenodes, and specifically an LED Node v2a, to drive the strips of LED lighting.

These are basically mini 3v arduinos with built-in radio transceivers, which makes communication between them easy enough, you can send in-memory structures fairly trivially with the RF12 library.

And, there’s a great ledNode sketch that drives the LED node.

I’ve hooked it up to a PIR sensor, and that controls the lighting (the colour of the lighting is controlled separately) via a ZMQ connected service.

I’ve got 5 Jeenodes running (and the kits for a couple more) and two Raspberry Pis providing the horsepower, along with some other bits and pieces.

I’ve also implemented calendar event broadcasting (coming from Google calendars), this was fun to do, and it essentially pulls calendar events for a list of calendars, and emits start/end of those events, services can be written to listen for those events and respond.

Current task is building a ZMQ service that talks to my node-rfxtrx library, which will then be connected to the front-end I’m building, and at that point, I’ll be happy for now :-)

Golang, concurrency and Python

I’ve been working a fair bit with Go recently, my employer has selected it as the language for developing services with, and we’re looking at ways in which Go can solve our problems.

I gave a talk last month looking at the syntax of Go, compared to Python.

Go turns out to be fast, easily compiled, and mostly easily understood, the mantra “Don’t communicate by sharing memory; share memory by communicating.” is essential to understand why Go is great to work with.

Goroutines and channels are great to synchronise processes…but, wait, yes, I’ve been a Python programmer for a long time, and it would be remiss of me to neglect Python’s built-in libraries.

We have the venerable threading library, threading is not great, it does the job (mainly slowing your Python application down), and for quick demos, you need to do a lot go make it safe to work with.

Python 2.6 (released at the start of 2008) brought with it the multiprocessing library, which has Queues and Pipes, which are really not that far removed from Go’s channels, and with a tiny bit of sugar, can prove quite useful, it gets round Python’s GIL by starting separate interpreters which means the interpreter doesn’t have to use the GIL.

from multiprocessing import Queue, Process
 
def main():
    exit_queue = Queue()
    timer_queue = Queue()
    datapoints_queue = Queue()
 
    Process(
        target=send_timed_messages, args=(5, timer_queue, exit_queue)).start()
    Process(
        target=fetch_active_builders_from_offspring,
        args=(timer_queue, datapoints_queue, exit_queue)).start()
    Process(
        target=push_active_builders_to_cosm,
        args=(datapoints_queue, exit_queue)).start()

We create Queues, and hand them off to the subprocesses which do the work…

 
def push_active_builders_to_cosm(datapoints_queue, exit_queue):
    """
    Wait for data from the datapoints_queue, and start a
    process to push it to Cosm
    """
    exit = False
    while not exit:
        message = None
        try:
            message = datapoints_queue.get_nowait()
        except Empty:
            pass
        else:
            _, time_value, value = message.split("::")
            Process(
                target=push_datapoint_to_cosm,
                args=(time_value, value)).start()
        try:
            exit = exit_queue.get_nowait()
        except Empty:
            pass

To exit the process, we push True to the exit_queue in our main process, and it will exit nicely, otherwise it reads from our datapoints queue, and starts another process to push the data to Cosm.

ZeroMQ has a nice method for polling sockets, the Poller object…we can implement a reasonably efficient version using the select module (we could make it more efficient with select.epoll)

 
class QueueSelector(object):
 
    def __init__(self, queues):
        """
        Takes a list of multiprocessing.Queue objects and returns when one of
        them is available.
        """
        self._queues = dict(zip(
            [queue._reader for queue in queues], queues))
 
    def select(self):
        """
        Returns a list of queues that have data waiting...
        """
        (readers, [], []) = select(self._queues.keys(), [], [])
        return [self._queues[reader] for reader in readers]
 
def push_active_builders_to_cosm(datapoints_queue, exit_queue):
    """
    Wait for data from the datapoints_queue, and start a
    process to push it to Cosm
    """
    exit = False
    selector = QueueSelector([datapoints_queue, exit_queue])
    while not exit:
        message = None
        readers = selector.select()
        if datapoints_queue in readers:
            message = datapoints_queue.get()
            _, time_value, value = message.split("::")
            Process(
                target=push_datapoint_to_cosm,
                args=(time_value, value)).start()
        if exit_queue in readers:
            exit = exit_queue.get()

This doesn’t handle write queues, where you’re waiting for the queue to be able to be written to, but that should be fairly easy.

Microservices, ZeroMQ and Node.js

After hibernating for most of the winter (certainly as far as home projects go), the recent spring-like weather (it’s stopped being so cold and miserable), has awakened fresh interest.

I spent some time investigating lots of options, and Stian Eikland’s approach looks good to me.

Basically, it’s a collection of micro-services joined using a ZeroMQ “bus”, I’ve reworked some of his code into CoffeeScript classes, and extended the number of services.

Some services push data to a Broker, and others subscribe to messages from the Broker.

I’ve written my own Cosm service, which looks for sensor messages, and pushes the data to Cosm, using my Cosm API, and today, I completed the first draft of a Google Calendar event publisher, which can publish calendar events to the bus…

To fetch the events from the Google Calendar, it requires read-only access to the private calendar feed, it calculates a 24 hour date period, and fetches those events, and then interprets them and publishes them to the bus.

    getCalendarUrlForToday: (cal) ->
      today = helpers.getFormattedDate()
      tomorrow = helpers.getFormattedDate(1)
      period = "start-min=#{today}&start-max=#{tomorrow}"
      "https://www.google.com/calendar/feeds/#{cal.calendarId}/#{cal.magicCookie}/full?#{period}"
    fetchUrlAndDispatch: (url, callback) ->
        request.get url, (err, response, body) =>
          if not err?
            parser = new xml2js.Parser(parserOptions)
            parser.parseString body, (err, result) =>
              if not err?
                if result.feed.entry?
                  for entry in result.feed.entry
                    for times in entry['gd:when']
                      message =
                        event: 'calendar'
                        start: times.startTime ? undefined
                        end: times.endTime ? undefined
                        title: extractTitleFromEntry entry
                        uid: extractUidFromEntry entry
                      @bus.send message

Using node-cron, it queues processing of the feeds at configurable intervals.

What services that are consuming those events do with the data is up to them…

Next up, a Sunrise/Sunset message producer!

Make your project easy to bootstrap for development

I work across several different Programming Languages, and across at least two different development machines, and across several different projects, both personal, and work-related.

I’m usually happy to dive into someone else’s code (assuming they have a reasonable test-coverage), and fix issues I find, I might even be tempted to pick up a random bug on a project and fix it.

One thing that helps me is if your project is easy to bootstrap for developing.

My rfxcom library is easy enough, just clone the repository from Github run npm install, and then npm test should get the tests passing (you will likely need the equivalent of the “build-essentials” package if you’re not on Ubuntu/Debian, but I guess that’s why it’s “essentials”).

Recently I’ve looked at  a couple of projects, and they’re not so easy, Python has virtualenvs which for most purposes, work similar to npm’s packages, with a suitable “requirements.txt”, you can bootstrap a project fairly easily.

Ruby has this solved too, with Bundler and Gemspecs, I know that I can use rvm to create a new virtualenv, run bundle install and hopefully rake test, and the tests should be running.

If you’re involved in a development project, have a look and see how easy it would be to get it up and running by someone outside your team, if it’s not easy, make it easy.