This post is probably best served with some previous knowledge of Scala, Futures and akka-http, but I've tried to skip the most nitty gritty stuff and jump to the fun part.


Learning new programming languages and frameworks is one of the most important things you can do as an engineer to stay updated and keep the fun. Recently, I have been looking at scala and akka, with special focus on akka-http. I will not go into basics here, there are many great tutorials on akka and scala that you can find on google.

This post will be an not be an introduction to microservices either, but it will look at how the basic building blocks in akka-http and scala that you can use to build a microservice that uses HTTP as the interface to other services. Which, in my opinion is a good choice of protocol, being implementation-agnostic and battle-proven. The general motivation for making microservices is starting to become well known, but for us, it boils down to a couple of things:

  • We can easily change, update and kill services that do a specific task
  • The interfaces between services are well defined and does not require knowledge of inner workings and implementations
  • As stated in the first line, it enables us to continually experiment with new stuff, have fun, learn things and always choose the best tools to achieve a specific task rather than to make boring tradeoffs.

Some background

So, what makes a framework or language good for writing a microservice? I would argue light weight, speed and security features. By light weight I mean a light core lib that can be extended by adding additional libs that do a specific task well.

The language or framework also needs to have support calling other microservices in parallel and merge / compose the results and return it. In our case, akka-http is completely async and Scala Futures and Future composition is very powerful, yet simple to use and reason about. Akka-http is somewhat different than than existing web frameworks such as Django and ruby on rails where the main focus is on the request-response cycle. The goal of akka-http is essentially to connect akka to the web, so it is a really light weight library that can co-exist with an an existing akka application. In fact, my experience with akka-http started when we wanted to expose our batch processing tool (written in akka) to the web in order to trigger jobs on demand.

Getting started

I have always been a fan of just getting my hands dirty right away, rather than reading up forever. I also have good experience with having a clear scope and goal when learning new stuff. So let’s create some code that can handle authenticating and authorizing incoming requests and call appropriate methods.

Domain models, validation and JSON mapping

First, let’s create a model for containing a login request. This could obviously have been done with a regular form post request, but doing it like this let’s me show off the elegant spray JSON serializers, as well as WIX’s accord library for validating object properties (If you need more fine-grained control, let’s say omit some fields or rename them, you can also create a custom object mapper):

Routing and mapping requests

Then, let’s show off the elegant routing directives in akka-http, which I really like. There are actually quite a few built in directives for handling all HTTP methods, as well as conditional GZIP and more. The gist below shows the route for logging in, which will return a token that will be used as the Authorization header in following requests. It also uses the validation provided by WIX and returns an error object that explains what went wrong if so.

This shows some really cool features in akka-http. A route is essentially built by composing directives such as post, complete, pathEndOrSingleSlash. The important part in this case is the entity directive, which enables mapping a request body to a domain object, in our case the LoginRequest. Then, we let WIX do the validation.

Show me the rest of the code

To complete this example and for clarity’s sake, let’s look at the generateLoginToken method that provides a token upon a successful login (I apologize to all scala savants for the possibly messy code):

Creating your own directives

As previously mentioned, there are quite a few built in directives, for instance you can do Basic Auth. However, in our case, we have made our own authentication mechanism, consisting of using tokens (obviously, this example is bit artificial, as basic auth probably is a better choice when building microservices).

When looking at the routing example above, we see that some directives, such as post does not do anything more than wrap the inner directives. The entity directive, on the other hand, returns the LoginRequest object and makes it available to the inner directives.

So, knowing this, let’s create a directive that parses the Authorization header (with the token obtained when logging in) from the request and provides us with the User object that issued the request. Then, the user can be passed on to domain / business logic methods to i.e. determine permissions if we want to do that.

Essentially, we are now creating middleware, as most of you probably are familiar with from Django, Express.js and similar web frameworks - where the goal is either to reject a request before it reaches the domain methods or to augment the request context with i.e. information about the user that issues the request before calling domain methods. Let’s look at the code:

First, we created a directive that just gets the user from looking up the login tokens. The next directive, authenticateWithRoles, does filtering based on a list of roles, which some of you probably are familiar with from Java / Jersey and the @RolesAllowed annotation.

To round off, let’s create a route that uses the authenticateWithRoles directive to only let users with the admin role create new users:

The really cool thing about this kind of middleware is that it’s quite explicit and that you can wrap one route, or you can compose many routes (using the ~ operator) and wrap them. No configuration files, it’s all just code.

Lastly, let’s spin up the server and serve some http:

A few weeks back our team had a discussion about API design. The discussion was mostly about what we could do to design pragmatic and user friendly RESTful APIs for our clients. At some point in the discussion we discussed what idempotency is and what it actually guarantees. That sparked a curiosity in me, and I wanted to explore deeper into that.

Idempotency is defined as: (wikipedia):

Idempotence is the property of certain operations in mathematics and computer science, that can be applied multiple times without changing the result beyond the initial application.

In the context of HTTP APIs, a HTTP method is idempotent if it guarantees that repeating a request multiple of times has the same effect as issuing the request once. This is especially important in the case of network failures. In such cases clients can repeat the same request multiple times without worrying about any unintended effects.

Safe vs Idempotent

Idempotent methods should not be confused with Safe methods. In HTTP, method is safe if they are not expected to cause side effects. Meaning clients can send request to safe methods without worrying about causing any side effects or changes to the resource.

The table below gives an overview of the different properties for the most popular HTTP methods used when implementing RESTful APIs, GET, PUT, POST and DELETE.

Method Safe Idempotent
GET Yes Yes
PUT No Yes

Pragmatic vs Idempotency

Being idempotent is important. However, it does not mean that you have to guarantee that a request one point in time should always return the same result. One example of this is when we apply DELETE on an existing resource multiple times it should return HTTP Status: OK (200) the first time and HTTP Status: Not Found (404) the subsequent times.

A strict implementation of DELETE would expect HTTP Status: OK (200). I believe that approach to be unnecessary cumbersome and pragmatism should trump idempotency in such cases.

Being pragmatic about PUT’s idempotence is common, either by purpose or by carelessness. In the next part we will dive deeper in to the ramifications of relying too much on idempotency of PUT in a concurrent setting.

Concurrency vs Idempotency

A typical semantic used to ensure idempotency for a PUT request is to require the client to send all the values, including the ones that does not change, when updating an object. This approach for PUT that is idempotent but prone to race conditions, both in case of network failure and in highly concurrent environments.

Say Alice tries to update the secret to "A" and because the network is not reliable she loses the connection right before the request was supposed to respond. Now she does not know if the the update got through or not. However, since the PUT is idempotent she can retry the request without worrying of causing other side effects.

This line of thought is correct, if we assume that Alice was the only only one trying to update the secret. Let us assume that another user Bob was simultaneously trying to update the same secret as Alice to "B", and successfully did so in between Alice’s two requests. Then Alice’s request will implicitly overwrite Bobs secret and it will be lost forever.

Even though the example is fairly trivial example it illustrates pretty clearly that idempotency does not ensure safe updates in a concurrent environment. Also, consider how such API semantics in a large distributed environment can cause a a lot of data loss and race conditions if not properly implemented.

Avoiding race conditions

One approach to avoid such race conditions is to implement the API with Optimistic Locking semantics. We can do that by introducing a version number or an hash, to the data model. I prefer using version numbers because they are easy to understand and update.

Assuming that we are using version numbers to implement Optimistic Locking. Each update of a resource must include a strictly monotonically increasing version number of the previous resources. That is if we have a resources

  "secret": "A",
  "version": 1

then the update request for the resource at hand must include version: 2, or it will be rejected,

  "secret": "B",
  "version": 2

Now, let us reconsider the case with Alice and Bob. Let us assume that Alices first request was persisted and that Bob has persisted his secret "B". If Alice retries her request with version: 1 from earlier it will fail. It fails since the version number of her request is less 2. The only way for Alice to update the secret is to explicitly set the version number to 3. Solving the race condition problem illustrated earlier.

Implementing optimistic locking is a trade-off between complexity and reliability, and should only be included if necessary.

Closing Notes

Remember that every API is different and has different requirements. There is no golden rule for API design and you will have to make lot of trade-offs along the way.

For further reading on API designs and principles I highly recommended apigee’s ebooks and 3scales reports.

Update: You can now use MySQL socket factory to connect an app in Google Container Engine to Cloud SQL.

This post is a post-mortem of a two minute job taking us three days. Our two minute job at hand was connecting a Java application running on Google Container Engine to Cloud SQL. Normally a straightforward job, but due to several constraints it turned out to be a bit of a hustle.

If you are already working on the Google Cloud Platform you are likely to be familiar with both Google Container Engine and Cloud SQL. Google Container Engine, henceforth GKE, is a fully fledged cluster management and container orchestration system, powered by Kubernetes. Cloud SQL is fully managed MySQL database service on demand.

According to the Cloud SQL documentation it is recommended to connect from GKE using a separate SQL-Proxy. This proxy will handle authentication to your Cloud SQL instance. Connecting to Cloud SQL from your application will then be as easy as connecting to the host+port of your SQL-Proxy instance, no password or further authentication steps necessary.

Using the SQL-Proxy is not strictly necessary, one could connect a GKE application to a Cloud SQL instance in other more or less manual fashions. We did not chase those paths however as using SQL-Proxy is the solution recommended in the Cloud SQL docs.

As the SQL-Proxy is to run in a separate process there are numerous ways in which we can run it. Deciding upon an approach we gave weight to the following properties to be important: avoiding extra network jumps and transparency — it should be trivial to understand the purpose of the SQL-Proxy and its connection to our Java application.

Try One (the future solution)

We liked to avoid adding a network jump between the SQL-Proxy and our application. As our application is replicated on several nodes in our GKE cluster using a Kubernetes Replication Controller, we wanted to ensure SQL-Proxy was co-located on all the same nodes as our application.

In Pods on Kubernetes you can run side-car containers in addition to your main application container. Two traits of side-car containers are that they will run on the same node as its main container and be available on the localhost interface.

Being loosely-coupled, transparent and obtaining the two traits just mentioned we believe a side-car container is the preferred way of running SQL-Proxy.

Unfortunately, rolling updates when running multiple containers in a pod is currently not supported on Kubernetes. As we rely on rolling updates to deploy at will, we had to abandon this solution for now. This feature is merged in the Kubernetes source and should therefore be available in one of the next releases.

Try Two

As we currently are unable to run SQL-Proxy in the same Pod, a second best option seems to run SQL-Proxy as a separate service. This entails giving up the desired property of ensuring no extra network jumps. As a temporary measure, it seemed a reasonable solution.

The SQL-Proxy however is designed to run on the same node as its application. Snippet from SQL-Proxy source

Then for this solution to be viable we would have to add a separate proxy in a side-car container in the SQL-Proxy Pod. That would leave us with the following flow:

Application -> Proxy -> SQL-Proxy -> Cloud SQL

Adding another step, the extra proxy, means adding complexity to an already at best mediocre solution. There must be something better.

Try Three

Using Docker, keeping your containers simple and single purpose is a common best practice. In our third try we decided to devoid from that principle — running both the SQL-Proxy and our application in the same container.

A feature of Go make this approach more elegant than anticipated: Go-code compiles to binary machine code. The SQL-Proxy is written in Go, therefore we could run its binary in our application’s Docker container without adding anything else than the binary itself.

To ensure having an up-to-date SQL-Proxy we have included a step in our application build script to build the SQL-Proxy from source. Building the SQL-Proxy we start a separate docker container which builds from source, then outputs a tar stream of the binary. We used Dockerception as inspiration for this pattern. Unpacking and mounting the binary file to our final application container is then trivial.

Of the thinkable solutions, this solution provides several desirable properties. Firstly, there will be no extra network jump as SQL-Proxy and application runs in the same container. Furthermore, it is quite transparent in design. If you inspect the application’ Dockerfile you cannot fail to spot the SQL-Proxy being started before the application itself.

Closing Notes

Our journey tells a classic story from using systems not yet mature. Documentation tends to be incomplete and the preferred solutions to your problems are still only on the roadmap. On another note, our experience working with the Google Cloud Platform and GKE has mostly been exhilarating.

For further reading on lessons we have learned running Kubernetes in production check out Andreas Heim’s post.

Everybody should have their own personal light show

The usual recap

In my two previous posts (“Welcome to Unacastle”, “Your own personal theme song”) I’ve been writing about a little side project I’ve got going here at Unacast were I try to leverage beacon technology to create personalized greetings when we enter the office.

As my previous attempts have included smart phones, webhooks and third party apps that has to be installed, I wanted to try something a little more low tech and offline.

Introducing the Pi-LITE + Ti’Be combo

When Birte, our VP of Strategic Partnerships, came back from one of her partner meetings with one of these: tibe It was just what I needed to make this work. The Ti’Be is a small beacon that you can attach to your keys (or anything that you tend to misplace). It connects to an app that among other things can locate the keys for you. Since this is a beacon that I carry with me when I go to work every day, my plan was to create a scanner that triggers an action when the Ti’Be gets in range. This is actually the opposite way of doing things compared to my last experiment.

I remembered that I had a Pi-LITE board for my RaspberryPi laying around, so I decided to make my “own personal light show”.

How it works

I wrote a small Node.js app to run on the RaspberryPi that scans for iBeacons (in this case, just my Ti’Be identified by it’s UUID) and on encounter triggers the LED matrix of the Pi-LITE. The source code can be found on GitHub, feel free to play around with it. I use node-pilite to display stuff on the Pi-LITE and node-bleacon to scan for beacons.

Demo time

sorry about the cheesy music ;)

and a closeup to see the details better

Small update

I’ve gotten a question about how the code backing our “Personal Theme song” looks. As you may remember this is a small side project at the Unacastle where we use the beacon-interaction-webhook-feature of the Writeup app to get our own Personal Theme songs played when we enter the office in the morning.

Below you can see a small Gist of it’s two main parts; the handler.clj handling the webhook call from Writeup and the app.js running on the RaspberryPi that plays the correct mp3 when it receives a websocket message.

The url of the websocket and our ids has been redacted to protect the innocent.