There are two main criteria that one should look for when choosing a database migration framework; it should be simple and it should always roll forward. When we began looking for a migration framework to use in one of our Go applications we could not find one that was simple in the sense that it could read from file as well as from pre-compiled assets, which we tend to use at Unacast, and only rolls forward. As a result of that we ended up implementing our own simple migration framework for Go applications, which I will guide you through in this post.

TLDR; Grab the code on github

Why you should not roll back a database

For some it might sound a little bit odd to say that you should never roll back a database, but let us think about what a rollback is for a moment. While you think about that I will define what I mean rollback and rollforward are.

  • Rollforward: an action in which you take your database from one state to another
  • Rollback: an action in which you take your database from one state to another

The observant reader will notice that the definition for both actions are quite similar, or wait, they are the same. This means that instead of having a rollback you can just write a new rollforward if you are in a situation where a database update has gone wrong.

Another reason is that database roll backs are complicated and there are a lot of things that you need to consider, which you most likely will not, when you are writing a rollback. Let me show an example of something that most people would do when writing a new migration adding a new column:

  1. Write migrate up that adds column X
  2. Write migrate down that removes column X
  3. Runs migration

So far so good. If step 3 fails you can just rollback right away, no stress, but this could also be solved by using a transaction when you run the migration. A more complicated scenario, and probably more realistic one as well, is that migration went well but a couple of hours later the database starts to act weird. What do you do? Should you run the rollback? Probably not. At this point in time you have already gotten some data in the new column X which you most likely want to keep, so the solution for rolling back is not just to remove what you just added. Instead you should write a new migration where you define what you should do to not lose any data. If you would have run the rollback at this point in time, when you have data in column X, you would lose that data. This means that rollbacks are only good during the migration process inside a transaction, but at that point you could as well just use the transaction as rollback mechanism.

Not everyone agrees with always rolling forward, and that is ok, but this is the way we at Unacast think of migrations.

Why could not we just use an existing migration framework

In regard of the roll back issue we could most likely have used an existing framework but not use the roll back feature. The main reason to why we wrote our own framework was that none of the frameworks we could find supported reading migration scripts from assets. In most of our Go applications we compile everything to one file, using go-bindata, meaning that sql files will have a pre-compile step where we generate Go files from the content in the sql scripts. This together with our opinionated view of migrations made us write a new migration framework.

Implementing a migration framework

Before implementing something, it is always good to think about what you want to achieve. The minimum list of features for this project was:

  • Run migrations from assets
  • Run migrations from files
  • Run all migrations, that has not been run before, in one transaction
  • It should use the existing sql.DB package
  • Only roll forward

That does not sound too hard, and it is not as we will see. The last point actually makes things a little bit easier since we are leaving out one feature, rolling back, compared to most other migrations framework. When we know the set of features we need to define the main flow and it turns out it is quite simple:

  1. Verify a migrations table exist, this is needed to keep track of which migrations that has been executed
  2. Get all migrations that has been executed
  3. Get all migrations
  4. Start transaction
  5. Loop over and execute all migrations, ignore those that are already executed
  6. Commit transaction if everything is ok, otherwise roll back transaction (note that this is not roll back of the migration, just a roll back of the transaction)

Now we know everything we need to know to implement the migrations framework. I will not cover all the code, which you can find on github, but there is one thing that I would like to cover. If you look at the signature of the Migrate function it looks like this:

func (migrator *Migrator) Migrate(getFiles GetFiles, getContent GetContent) 

where GetFiles and GetContent has the following signature:

type GetFiles func() []string
type GetContent func(string) string

The rationale behind this approach, instead of giving a folder path where all the files are, is that we can take any function as parameter to the Migrate function that returns a list of strings pointing to where the actual content is and then use the second function to get that content. It also makes it very flexible since the migrations framework is agnostic to where and how the actual content is stored.

When writing your migrations, you will implement a function that has the signature of GetFiles and it will most likely do one of these two things:

  • Return a list of files in a folder or folder tree
  • Return a list of keys that you can use against some map to get content

What your function that implements GetContent should do depends on how you decide you want to use the framework. If reading directly from files the input to GetContent should be file paths, that you get from your GetFiles function, and then your GetContent function just returns the content of those files. If you are using assets your GetContent function should read from the asset framework instead of from disk.

Using the framework

We are still missing a little bit of documentation, but it should be enough to get you started in the readme. Running the migration from inside your application is as simple as:

func runMigrations() {
    db, _ := connectToDb() // should return *sql.DB
    getFiles := func() []string {
        files, _ := assets.AssetDir("migrations/sql")
        return files
    getContent := func(file string) string {
        bytes, _ := assets.Asset(fmt.Sprintf("migrations/sql/%s", file))
        return string(bytes)
    migrator := migrations.New(db)

    migrator.Migrate(getFiles, getContent)

In the example above we are using assets, which have been generated from sql files. The getFiles function returns a list of “file names” in the asset folder migrations/sql. The getContent function will get the output from getFiles as input and will just read the actual asset on each request. With those two defined we can now call Migrate.

Please try it out and let us know what you think. If you have any problems just register a github issue or (even better) send us a PR.

In this blog post, we’d like to present one of our recent hires, Lars Bakke Krogvig. Lars works at Unacast as a Data Engineer and his main responsibility is to make sure the data we receive from our partners flows steadily through our processing pipelines. In his spare time, he’s a road cyclist, and almost as passionate about speed-solving Rubik’s cube as he is passionate about freestyle skiing.


Lars, who are you?

I’m Lars, 27 years old, born and raised in Oslo. I have a Master’s Degree in Applied Mathematics from NTNU in Trondheim. I used to be pretty serious about twin-tip skiing but never got seriously injured. Now I’m more into road cycling, and yes, I suppose I occasionally maybe devote some time to improve at solving Rubik’s cubes. I also like programming, and lots of other things!

What did you do before you started at Unacast

I worked as an Information Management Consultant in the private sector, where I designed, built and maintained reporting systems.

Why did you start working at Unacast?

My main motivation was that I wanted to roll up my sleeves and work more hands-on with design of information flow and implementation of processing pipelines. As a consultant I felt distanced from the problems I solved, I was more of a helping hand than a stakeholder. I wanted to be more involved.

I also want to become better at programming, and so I felt that it was a good idea for me to do more of that in my work (and perhaps slightly less in my spare time).

Another important factor for me was the opportunity to work with emerging technologies, which I believe are more fun and gives me a good skill set for the future.

What kind of problems do you solve, and what are the tools you use?

My main tasks are to rationalize the logic and sequence of what we do with data in Unacast and place that into a structured processing framework.

When working on data problems my job is to figure out where we are and where we want to go, and then to structure the steps in between and make it possible for everyone throughout the company to grasp what actually happens. I try to make complicated processes understandable for everyone, and have everyone around the table wholeheartedly nod and say “this makes sense”.

Regarding tools and methodology, I usually throw together a proof of concept using BigQuery and SQL, and when the concept is validated I put that into our pipelines with Dataflow.

What is the most fun thing about working at Unacast?

I really like that we have short lead times from an idea on a whiteboard to production, and the flexibility the toolset we use provide us with. It is also very rewarding (and sometimes scary) to see my ideas ending up having a big impact on the business, and having the power to influence the company in this way.

Beyond that I really enjoy working in a relaxed environment with colleagues I can have fun with at work. I get to travel to the US and work with people there, which is a new and rewarding experience for me. Everyone seem to tolerate my dry sense of humor, and even respect me enough to not borrow my markers without asking, which is all anyone can ask for.

What will you gain from working at Unacast?

I see this as a rare opportunity to work with leading technologies and processes are becoming standards internationally, but have not yet gained so much momentum here in Norway. I think this is will be a big advantage for me in the long run.

Right now I benefit from being able to work from wherever I want, which gives me a lot of personal freedom few companies are able to offer. I get to learn a lot and from our skilled developers and draw on their experience, which I think is really nice.

All in all, I get to do the things I want to do in a great environment, and have fun along the way!

I really want to use Dataflow, but Java isn't my 🍵
What to do?

A brief intro to Dataflow

Image credit Google

Dataflow is

A fully-managed cloud service and programming model for batch and streaming big data processing.

from Google. It is a unified programming model (recently open sourced as Apache Beam) and a managed service for creating ETL, streaming and batching jobs. It’s also seamlessly integrated with other Google Cloud services like Cloud Storage, Pub/Sub, Datastore, BigTable, BigQuery. The combination of automatic resource management, auto scaling and the integration with the other Google Cloud

So how do we use it?

Here at Unacast we receive large amounts of data, through both files and our APIs, that we need to filter, convert and pass on to storage in e.g. BigQuery. So being able to create both batch (files) and stream (APIs) based data pipelines using one DSL, running on our existing Google Cloud infrastructure was a big win. As Ken wrote in his post on GCP we try to use it every time we need to process a non-trivial amount of data or we just need to run continuously running worker. Ken also mentioned in that post that we found the Dataflow Java SDK less than ideal for defining data pipelines in code. The piping of transformations (pure functions) felt like something better represented in a proper functional language. We had a brief look at the Scala based SCIO by spotify (which is also donated to Apache Beam btw). It looks promising, but we felt that their DSL diverged too much from the “native” Java/Beam one.

Next on our list was Datasplash, a thin Clojure wrapper around the Java SDK with a Clojuresque approach to the pipeline definitions, using concepts such as (threading), map and filter mixed with regular clojure functions, what’s not to like? So we went with Datasplash and have really enjoyed using it in several of our data pipeline projects. Since the Datsplash source is quite extensible and relatively easy to get a grasp of we even have contributed a few enhancements and bugfixes to the project.

And in the blue corner, Datasplash!

It’s time to see of how Datasplash performs in the ring, and to showcase that I’ve chosen to reimplement the StreamingWordExtract example from the Dataflow documentation. A Dataflow-off, so to speak.

The example pipeline reads lines of text from a PubSub topic, splits each line into individual words, capitalizes those words, and writes the output to a BigQuery table

Here’s how it the code looks in it’s entirety, and I’ll talk about some of the highlights specifically about the the pipeline composition bellow.

First we have to create a pipeline instance, and it can in theory be use several times to create parallel pipelines.

Then we apply the different transformation functions with the pipeline as the first argument. Notice that the pipeline has to be run in a separate step, passing the pipeline instance as an argument. This isn’t very functional, but it’s because of the underlying Java SDK.

Inside apply-transforms-to-pipeline we utilize the Threading Macro to start passing the pipeline as the last argument to the read-from-pubsub transformation. The Threading Macro will then pass the result of that transformation as the last argument of the next one, and so on and so forth.

Here we see the actual processing of the data. For each message from PubSub we extract words (and flatten those lists with mapcat), uppercase each word and add them to a simple row json object. Notice the different ways we pass functions to map/mapcat.

Last, but not least we write the results as separate rows to the given BigQuery table.

And that’s it really! No to apply a simple, pure function.

Here’s a quick look at the graphical representation of the pipeline in the Dataflow UI.

This is the Dataflow UI view of the pipeline. 27.770 words have been added to BigQuery


To summarize I’ll say that the experience of building Dataflow pipelines in Clojure using Datasplash has been a pleasant and exciting experience. I would like to emphasize a couple of things I think have turned out to be extra valuable.

  • The code is mostly known Clojure constructs, and the Datasplash specific code try to use the same semantics. Like ds/map and ds/filter.
  • Having a REPL at hand in the editor to test small snippets and function is very underestimated, I’ve found my self using it all the time.
  • Setting up aliases to run different pipelines (locally and in the ☁️ ) with different arguments via Leiningen has also been really handy when testing a pipeline during development.
  • The conciseness and overall feeling of “correctness” when working in an immutable, functional LISP has also been something that I’ve come to love even more now that I’ve tried it in a full fledged project.

First, let me be frank and announce that we’ve been on the Google Cloud Platform (GCP) for little more than a year. And that we’ve not been working full time with GCP for a year, yet. Nonetheless, We feel ready to share some insights and thoughts on building microservices on GCP.

In this blog post will focus on why we at Unacast chose GCP as our cloud provider, why it’s still a good fit for us, and a few lessons learned of using the platform for about a year.

Why Google Cloud Platform

Today the natural choice of cloud is Amazon Web Services (AWS). And with good reason. AWS pioneered many of the great cloud services out there, S3, EC2, Lambda etc. It has as far as we know the longest list of cloud components you can use to build your platform 1. And it’s battle tested at scale by Amazon, Netflix, AirBnB and many others. So why did we choose GCP instead?

Actually, we didn’t. Unacast started out building its platform on a combination of Heroku and AWS. And after some fumbling, sessions of banging our heads against the wall, and some help from a consultant2 we decided to try GCP. And with some effort and a lot of luck it turned out to be the right platform for us. The reason for testing GCP was two fold 1) GCPs big data capabilities and 2) it helps us minimise time used on operations.

There is no secret that Google knows how to handle large amounts of data. And many of the tools provided at GCP is designed for handling storing and processing big data. Tools like Dataflow, Pubsub, BigQuery, Datastore, and BigTable are really powerful tools for data management. GCP also as great environments for running services like App-, Cloud Engine and Dataflow helps us maximise the time used to build business critical features fast, rather than using developer time on keeping the lights on.

The Good Parts


Pubsub is a distributed publish/subscriber queue. It can be used to propagate messages between services, but at Unacast we’ve mostly used it to regulating back pressure. And it works great in scenarios where you want to buffer traffic/load between front-line API endpoints and sub stream services. We use this approach designing write-centric APIs that can handle large unpredictable spikes of requests. NB! Pubsub doesn’t provide any ordering guarantees, and it doesn’t provide any retention unless a subscription is created for a topic.


BigQuery is a great database for building analytics tools. Storing data is cheap and you only pay for querying and storing data. BigQuery is great because of its out of the box capabilities for querying large amounts of data really fast. To put things into perspective, with our thorough usage we’ve seen that BigQuery can query 1GB of data just as fast as 100GB (and probably even more). One thing to remember when using BigQuery is that it’s an append-only database, meaning that you cannot delete single rows, only tables3. In other words, where Cassandra as row-level TTLs BigQuery ships with table-level TTLs. So implementing data retention has to be done differently and may not be straight forward if you’re coming from a standard SQL perspective.

App Engine

App Engine is a scalable runtime environment for Java designed to scale without having to worry about operations. App Engine is great if you need highly scalable APIs. But you can only use Java 7 and libraries whitelisted by Google. Because of this restrictions we’ve got mixed feelings for App Engine. Getting scale without worrying about operations is great but on the other hand the development process is a lot more complex. We would use App Engine where your API doesn’t need much logic or external dependencies, like an API gateway, but for more complex services we would use Container Engine instead.

Container Engine

Container Engine is GCPs answer for hosting linux containers. It’s powered by Kubernetes which is, as of writing, the de facto standard for scheduling and running linux containers in production. On GCP we view Container Engine as the middle ground between Compute and App Engine. Where we believe you get the best tradeoff between operational overhead and flexibility. With Kubernetes you can do interesting things as bundle databases or other services together to increase performance which is impossible in App Engine. However, you have to worry about updating your Kubernetes cluster and keeping the nodes healthy and happy. Adding some operational complexity, work and time spent on not adding features.


Dataflow all the things! Dataflow is GCPs next generation MapReduce. It has streaming and batch capabilities. Dataflow is so good that we try to use it every time we need to process a non-trivial amount of data or we just need to run continuously running workers. As of writing Dataflow only has a SDK for Java. And Java isn’t necessarily the natural language for defining and working with data pipelines. Needless to say we started to look for non-official SDKs that could suit our needs. We found Datasplash, a Dataflow wrapper written in Clojure. Clojure syntax and functional approach works very well when defining data processing pipelines. We’re currently pretty happy with Datasplash/Clojure, at the time of writing we’re running Dataflow pipelines written in Java and Clojure. Time will show if this is the right tool. A caveat with Dataflow on GCP is that it uses Google Compute Engines under the hood. And that means the quota limits for virtual machines can be a show stopper. Make sure to always have large enough limits while you’re evolving your platform.

The not so good


Stackdrivers monitors sucks. Monitoring is hard from the get go. Its hard because it’s hard to know what to monitor and setting up real good monitors. If the monitor is too verbose and sensitive nobody cares when an alert is triggered, and if it’s too sensitive errors in production will go unnoticed. Our opinion setting up custom metrics in Stackdriver is a horrible experience. And that is why we use Datadog for monitoring services and setting up dashboards. To be fair, Stackdriver has some good components too, especially if you’re using AppEngine. Stackdrivers Trace functionality is awesome for tracking what is slow in your application. And the logs module are easy to use and query for interesting info. Our experience is that these two modules works really great out-of-the-box.


Cloud SQL is a great service for running a SQL database with automatic backups, migrate to better hardware, and easy setup and scale read-only replicas. But the SQL engine behind is MySQL. We’ve much respect for what MySQL achieved back in the day but those days are over. But because of the ease of use, infrastructure wise, we’ll probably still be using Cloud SQL in the nearest future. However we think we should always consider using Postgres through or even AWS AuroraDB before settling for Cloud SQL.

Closing notes

We haven’t been able to test all the features of GCP and some of them looks really promising. We’re really excited about the machine learning module. And we hope they’ll support Endpoints for other services than AppEngine soon.

Choosing the right cloud platform isn’t straight forward. It’s hard to know if the services provided by a platform at hand is the right services. We at Unacast have learned from first hand experience that more isn’t necessarily better. And that your first choice and instinct might not always be correct. It was and is still the right choice for us. And after Spotify announced that they we’re moving their infrastructure to GCP, we’re more sure than ever that we chose the right cloud platform.


1. Everything is a platform these days.
2. All consultants aren’t evil.
3. Not entirely true. Deletes are expensive not impossible.


This post is best read with some prior knowledge to Kubernetes. You should be familiar with concepts like pods, services, secrets and deployments. Also, I'm assuming you've been working with kubectl before. Enjoy!

At Unacast we spend a lot of time creating web services usually in the form of JSON APIs. And we’ve spent a lot of time designing and experimenting and researching to design them. We’ve shared what we’ve learned along the way. A lot of these posts has been theoretical but in todays post we’re getting our hands dirty, and we’re going to implement an API that scales when being subject to massive amount of read requests. And we’re doing this using Redis. All the examples will be run using Kubernetes.

We assume that the logic to keep the data in Redis updated has been implemented somewhere else. And we also assume that the rate of adding or updating data (writes) is low. In other words, we expect to have a multiple orders of magnitude more reads than writes. Thus, in our tests we’ll just contain read requests and no writes.

All code snippets included in this post can be found in its full form here.


Before we get started we need to talk about Redis. The Redis team has described Redis quite good on their homepage

Redis is an open source (BSD licensed), in-memory data structure store, used as database, cache and message broker.

In my own words, Redis is really fast and scalable in-memory database with a small footprint on both memory and CPU. What it lacks in features it makes it makes up for in speed and ease of use. Redis isn’t like a relational store where you use SQL to query. But it is shipped with multiple commands for manipulating different types of data structures.

Redis is a really powerful tool and should be a part of every developers toolkit. If Redis isn’t the best fit for you, I’ll still recommend investing time into learning how and when to use a in-memory database.


Redis can be used in multiple ways. Each different approach has different trade-offs and characteristics. We’ll be looking at two different models and test them on scalability. The two models we’re testing are:

  1. Central Redis: one Redis used by multiple instances of the API.

  2. Redis as a sidecar container: Run a read-only slave instance of Redis for every instance of the API.

The performance tests will be run against the same service using the same endpoint for both models. I’ve extracted the endpoint from main.go and included it below.

The snippet does one simple thing. It asks Redis for a string that is stored using the key known-key. And from this simple endpoint we’ll look how Redis behaves under pressure and if it scales. We expect different behavior from the two different architectural approaches. This example might seem constructed, a real world examples that is similar to this approach is verification of api tokens. I agree that this might not be the best way to do token verification but it’s a very simple and elegant design. For a more elegant solution you should consider JSON Web Tokens.

Central Redis

As mentioned above a central Redis architecture is when we use one Redis instance for all API instances. In our case these API instances are replicas of the same API. This is not a restriction but it is a recommended architectural principal to not share databases between different services.

In Unacast we believe in not hosting your own databases. We’ll rather focus on building stuff for our core business and not worry about operations. Normally, we use Google Cloud Platform (GCP) for hosting databases. But hosted Redis isn’t publicly available at GCP so decided to use’s Redis hosting.

Setting up the service using a single Redis is pretty straight forward using has some great guides on how you to get started with their Redis hosting as well. The kube manifest for running a kubernetes deployment and service is added below:

Redis as a sidecar container

Before we describe how to setup a Redis as a sidecar container. We’ve to give short description of what a sidecar is. The sole responsibility of a sidecar container is to support another container. And in this case the job of the Redis sidecar container is to support the API. In Kubernetes we solve this by bundling the API and a Redis Container inside one pod. And for those of us who don’t remember what a Pod is, here is an excerpt for the Kubernetes documentation:

pods are the smallest deployable units of computing that can be created and managed in Kubernetes.

Meaning that if a Redis container is bundled with an API container. They’ll always be deployed together on the same machine. Sharing the same IP and port ranges. So don’t try to bundle two services using the same ports, it’ll simply not work.

The following shows how to bundle the two containers together inside a Kubernetes pod.

By deploying this we’ll have a Redis instances for each pod replica. In this specific case we’ll have three Redis instances. That means we need some mechanism for keeping these instances in sync. Implementing sync functionality is horrible to do on your own [citation needed]. Luckily, Redis can be run in master-slave mode and we’ve a stable Redis instances hosted by By configuring every Redis sidecar instance as a slave of the master run by We can just update the master and not worry about propagating the data the slaves. Our unscientific tests showed us that the Redis master propagates data to the slaves really fast.

NB! A caveat is that you’ve to setup a SSL tunnel to to be able to successfully pair the sidecar instances to’s master instance.

We expect this architecture to scale better than the central Redis approach.


All the tests were run using a Kubernetes cluster:

  • 12 instances of g1-small virtual machines
  • 12 pod replicas

We used vegeta distributed on five n1-standard-4 virtual machines to run the performance tests.

The graphs below are the results from the performance tests. The results focuses on success rate and response times.

Central Redis

Redis as a sidecar container


As expected we see that the sidecar container scales better than the central approach. We observe that the central approach is able to scale to about 15 000 reads/second, while the other can handle over 60 000 reads/second without any problems. Remember that these tests are run on the same hardware and that only a minor change in the APIs architecture resulted in a major performance gain.

Closing Notes

One last thing, remember that utilizing multiple read-only slaves will behave in the same matter as multiple read-only Redis slaves. We prefer using Redis because of its speed, small footprint and ease of use.

We haven’t been running this in production for a long time. So we don’t have any operational experience to share yet. And we intend to share this in the future.

Further work

This post didn’t cover if the Redis as a sidecar container approach scaled linearly as more CPU was added. This is outside of the scope of this post. But our internal testing has shown this to be true. You’re welcome to test this yourself.

At Unacast we’re obsessed with monitoring. One of our mantras is “monitoring over testing”. And notice we haven’t added any monitoring for the Redis instances inside a pod. However if you’re using Datadog, as we do, it’s fairly straight forward to add monitoring by bundling a dd-agent as another sidecar container inside the same pod.

Want more?

If you’re interested in reading more about API design I can recommend the following posts from our archive: