Hot tip: Want to learn react-redux? check out this guide by mr redux himself: https://egghead.io/series/getting-started-with-redux

Summary

This post cointains some thoughts around building a single page app in react-redux and ES6-7 and some patterns that have appeared, as well as how it has affected our workflow. I will not get too technical on how i.e. react-redux bindings and react context works. This post gives some empirical arguments why react-redux with ES6-7 is something you definitely should consider for your next project.

This post might be best served to people that are experienced in front-end development on single page applications and are familiar with frameworks and libraries such as react, ember and angular. But I may be wrong.

Background

Motivation

The main motivation for us in building a single page app in react-redux and es6-7 was actually a bit different for different people in the team, but we can summarize it like this, keeping in mind that we previously had been using angular 1.x for everything frontend-related:

  • Angular 2 is coming, Angular 1 is in maintenance mode and approaching end of life
  • Migration from Angular 1 to 2 is at best cumbersome
  • Angular is a large and complex framework with batteries included, rather than a lightweight library that can be used with other focused libs
  • Going from two-way bindings to unidirectional data flow (Flux) - a need to reason more explicitly about app state and how it relates to the UI
  • Using the new and exiting features of ES6-7
  • Learning new and fun stuff (the most important)

So, it seemed to be a good time to jump on the boat and learn something new. Just to make it clear, I don’t believe there are any silver bullets. Let’s consider this more of a story of where react-redux shines and makes life a as a developer a little easier. I say this a developer who have been doing large angular projects for the last three years - there are definitely many things to love about angular.

Disclaimer

We rely on react-redux-bindings, but there are obviously some features of i.e. the redux store and ES6/7 that is equally valid and useful in angular or in vanilla javascript for that matter. Also, react / redux are completely stand alone libraries that does not require or depend on each other to work. It might be difficult to always separate react from redux in this writing as we are using them very tightly coupled.

The old stack

To summarize, this was the situation: ES5, Angular 1.5, Bootstrap, angular-ui-bootstrap, angular-ui-router, restangular, bower, npm, gulp and less + much more

The build system consisted of a ~300 lines custom gulpfile with pretty advanced features that was hard to grasp for everybody, even myself sometimes. It did a lot of magic stuff in very “creative” ways and it was difficult to get into the code after not having touched it for some time.

The new stack

ES6/7, React-redux / router, redux-dev-tools, material-ui, superagent, webpack, pure npm and no bower.

Webpack is more about configuring the different plugins rather than writing code that works directly on the source files, such as the gulp streaming system. We chose webpack because it seemed to be widely adopted in all examples and tutorials we found, especially on react-redux, both official and non-official sources. Webpack also sports built-in hot reloading that plays really well with what we want to achieve from our development / build-pipeline. To be honest, other options like browserify was not considered or investigated in much depth.

What is Redux

Redux is a very simple and powerful pattern where the entire app state is stored in a single immutable object, and in conjunction with react it can be used to implement the Flux pattern, which basically is about unidirectional data flow in the application, no two-way bindings as in angular.

That’s the redux store / state. The next piece of the puzzle is the reducers. Reducers are simply pure functions that take two inputs - the current state and an action. Based on these two, the new state is computed and returned. The state is immutable, so the reducer will never change the current state.

Let’s look at how this works in pseudo code when a user clicks a button:

  • (User clicks button)
  • → dispatch(action(USER_CLICK))
  • → reduce(currentState, action)
  • → return newState
  • → subscribing react components are notified and calls .render() on themselves.

Reducers will usually work on small and specific parts of the application state, such as the UI state of a subpage and then all reducers are composed to a bigger reducer, or the store - sometimes referred to as the state tree. This is very powerful, because the application can grow arbitrarily large, and you just add more reducers for the different parts and components with a single responsibility that are small, easily testable (pure) functions.

In the example above, the button click could be used to open a modal dialog. Then, the newState object would have a key called isDialogOpen set to true. Then as the view component renders, it will show the modal given that the prop isDialogOpen is true. Declarative and easy to reason about!

React-redux bindings

React-redux bindings and “smart components” enables you to extract just the piece of state you want to render, for instance the logged in user’s name to display in a top bar component, using the connect() method from react-redux bindings. When an action is dispatched - the store will propagate this to subscribers, which are the smart react components that will rerender.

What the connect method essentially does is to map the app state to component props as your are familiar with from react, by slicing off the piece of state you want. It also adds the dispatch method as prop on the component so you can use it to dispatch actions based on i.e. mouse clicks, button clicks, input field focus / blur, routing / url updates, map zooming, panning etc.

Why is react-redux a powerful pattern

React-redux is a powerful pattern for creating and reasoning about UI. It forces you to consider all possible events that can happen in the app and give them a semantic meaning. Lastly, you decide how these events should affect the app state through dispatching actions. All reducers receive actions and can respond with a new state however they see fit. The UI can be seen as just a pure function of the state object, easing overall testability and general reasoning about the UI. For debugging, it’s also great because you can easily reproduce state given a series of actions.

Further, using react-redux with hot reloading is supported, meaning that the source code can be changed in real time (while the state is kept in store). Remember, the UI is a function of the state so we just render what is there at any time. Super simple and powerful when developing on complex navigation and interactions such as modal dialogs, forms, wizards with steps and similar. How many times have you (using live-reload) not done a small change to form, just to reset all form fields when you do a small css change? The great part is that is not really a feature, it is more consequence of the fact that all app state is the store and not scattered around in code that can be hot reloaded.

The redux-devtools are also awesome, giving you an instant overview of the app state and actions dispatched.

This together gives a very developer friendly and fun environment to work in. In Angular, data is scattered around in different controllers, maybe in services, but it the overall app state is arguably more implicit. From my experience, react-redux yields some more code and verbosity, but explicitness trumps implicitness all the time. There are for instance a lot of sugar contained in the react-redux bindings, but we have opted to not use too many of them, as they can be hard to understand and are really very simple convenience functions, such as the connect() function explained above.

Our approach and learnings

Getting started with a npm / js project nowadays is a truly daunting task, and there are enough memes, tweets and rants about this, so I will not add to that. I do believe that it’s best to start simple and learn, so I’m not a huge fan of all these “starter kits”, because they just take your breath away in their complexity and fancy approaches to handle all use cases and corner cases. I think it’s the best way to lose motivation when learning new stuff. Start simple! At a later stage it’s obviously great to borrow ideas from other’s work but it’s not a good idea to solve all types of problems before you have them.

We decided to clone the react-redux counter example and build it out, adding additional tooling as we went along. This was a sweet spot in terms of being easy to understand, yet having some wiring done already. It might a seem a bit contradictory to my claims about starting out simple, and we also had internal discussions on the team whether this was truly the best approach. In hindsight it might have been, but that’s why they call it insight.

What we saw was in many ways that we approached the real-world-example, in terms of structure and features added. For instance, we quickly discovered that using normalizr is a must for storing API data in the store, as reducers don’t deal well with arrays. I think this plays well with my previous argument about not adding a lot of stuff before you know what it is for and why it is useful. It also made us understand better why normalizr is a good pattern for dealing with API data in reducers. Further, using among other the features in ES7 and lodash, you can slice and dice objects in an immutable manner. You can quote me on this; when developing in react-redux, you become very familiar with lodash you will probably use most of the functions at least once.

As the app grew, we saw that Webpack started taking close to 30s on build, and 4-5s on hot-reload, which was getting painful. Using happypack and some modifications to how webpack deals with sourcemaps, we got times down to 6-8s on build and < 1 s on hot-reload.

Closing notes

Our experiences with react-redux so far are very good. It’s far from a silver bullet, and the batteries-not-included approach requires you to reason about your app’s structure and libraries to use for different functionality. We have opted for a module approach, which has its advantages and disadvantages, and the app is continuously being refactored as we learn and discuss along the way. Yet, we are very happy about our decision. If that’s the right choice for you and your team is another question, but I hope the above has given you some more insight - maybe not on the super technical level, but on why the philosophy behind react-redux is a great one.

Look out for code in later posts, we will probably go into more detail as things are getting polished!

In this post we will extend synthetic, a synthetic transaction tester, I wrote about in late February. The previous post can be found here. This post will focus on some pain points we have had with synthetic, and how we solved those problems. Specifically, we will look into two things.

  1. How can we specify synthetic tests more declaratively using YAML
  2. Automate setup of Datadog Monitors

I will go through these in order and provide Go code where it is suitable. Bear in mind that the code has been added for illustrative purposes, and might be out of context. I hope that is OK. And I’m still learning Go so the code provided might not be perfect, I appreciate any feedback.

Declarative Synthetic Test Spec

In the previous post every test had to be written in Go code. At the time of writing that made sense since we only had two running synthetic tests. As we started to add more tests to synthetic a test pattern emerged. Most of our tests performed multiple HTTP request and checked each request returned the expected response. Mostly checking if the status code was correct and that the JSON body contained certain elements. With that in mind we set out to create a declarative way of specifying such tests.

After tinkering a bit we landed on the following data structures and code to run the job specified.

Notice that Schedule does not handle the events outputted by job.Run(). We have omitted this due to readability. leftContains which checks if the map on left-side are fully contained in the map on the right side, also omitted because it was very complex and way too embarrassing to open source as it is right now.

The following example uses the the structure defined above to specify a test job using Go code.

Which is quite much better than to imperatively specify each test. However, you are still writing Go code. And in case you did not remember, our goal is to be able to specify jobs using YAML. Wouldn’t it be great if we could specify something like this instead:

It turns out that is quite easy, if you are willing to use a third-party library. Go does not have a standard library for parsing YAML. But luckily someone has already written a library for it called gopkg.in/yaml.v2. Great!

A big drawback with the YAML approach is that we cannot specify different URLs or other variables for each environment. As described earlier each job is specific to each environment since the URL attribute is fixed per file. One way of fixing this is to specify a job for each environment. This will cause us to update every file when a job changes. That is not scalable. Another approach is to add variables to the YAML specification. And that is exactly what we did. We extended the YAML spec to the following

And then we parse and replace envvars using the following code

In other words, we solved the problem by reading envvars from the YAML file and then string replacing the variables with the correct values.

Putting everything together we are now able to specify a test using YAML and run it without having to write Go code.

Automate Datadog Monitor Setup

Setting up a Datadog Monitor for every environment for every test manually was tedious and caused a lot of errors. Even for three tests it was a crap. It is fair to say that it did not scale. So we decided to automate it! And luckily for us, both DataDog and the DataDog third-party library we are using supports creating and updating Monitors using DataDogs API. The process is quite straight forward and is illustrated below

SetupDatadogMonitor ensures that the monitor is setup either by creating it or updating it. A monitor already exists if the title exists at DataDog. This check is quite weird but the monitor ids at DataDog is integers. And we didn’t want to waste time coming up with a scheme for creating unique integer ids. It’s weird but it works.

Closing thoughts

In this post we looked into how we simplified the process of writing test jobs in synthetic by introducing YAML specifications and automate DataDog monitor setup. Yet, there are a lot of improvements to be done. I have lately been playing around with other hosted services for synthetic transactions. And there is a lot of inspiration to be drawn from these services.

Also, when we are ready we will be extracting synthetic from our code base and open source it.

When designing REST API interfaces, the methods that modifies existing resources seem to bring forth significant design challenges. The two most prominent challenges are idempotency and concurrency control. Ken Grønnbeck earlier wrote about idempotency in REST APIs on the Unacast Labs Blog. As a follow up, this post will discuss different approaches to concurrency control in REST APIs.

TL;DR. Keep the end-to-end flow in mind, minimize change sets and do as little concurrency control as your problem requires.

To ease the flow of the examples that follow we will begin with establishing the interested parties when working with a REST API. The three interested parties we will keep focus on are the api-creator, the integrator and the end-user. The api-creator are the ones building and designing the API, in some cases this might be “the backend team”, for more public facing APIs it might be the core business unit. The integrators are the consumers of the API, they could be another unit in your organization like the frontend team, a business partner or a lone developer using your API in a hobby project. End-users could be anything from your mobile app users to your partner’s users, in fact one end-user could be the user of several of your integrators. Yet, common for end-users is that they are likely oblivous to the fact that the API the api-creator provides exists — and does not want to know a thing about it.

The intended type of concurrency conflict for this post is when a user tries to update a resource which has been modified after the last time this user last saw that resource. The consequence of which is that the user writes over someone else’s change to resource, unknowingly.

As the nature of design challenges are discussions with no definite answers and a myriad of different solutions around — any feedback or correction are most welcome.

Minimize consequences

Reducing the possible consequences of concurrency conflicts is desirable from a API design standpoint. In practice this mean only modifying the fields which the end-user intends to modify. This can be achieved with either more granular endpoints for altering different properties or using PATCH semantics instead of PUT. The endgame here is to in a conflict only let the last request write the field(s) it actually intends to modify.

On another note, if modifying existing resources can be avoided so that you can have immutable resources — you have in all practicality removed concurrency conflicts from your API.

Optimistic locking

If you choose to implement a concurrency control strategy, optimistic locking is likely to be the best bet. Optimistic locking works like the following: try to do an operation, then fail if the resource has changed since last seen. Optimistic locking is opposed to pessimistic locking where a resource is locked from alteration before any changes are done, then released. As pessimistic locking schemes require state to be kept at the server it does not play well in RESTful APIs where it is idiomatic to keep the server stateless.

The general approach for optimistic locking is to along with your payload send a value that identifies which version of the resource you are modifying. If the version to modify from is stale, the request should fail. The version identifier could be a explicit version number or hash of the unmodified resource. Last modified timestamps could also be used, but with caution as clocks might be skewed in a distributed environment.

Identifying the version in a HTTP request one can either put the version identificator as part of the request model in the payload or utilize the if-match request-header from the HTTP/1.1 specification. Note that the HTTP/1.1 specification requires 412 (Precondition failed) to be returned upon conflict.

As a mechanism of concurrency control optimistic locking is a simple yet powerful approach. Implementing it for the api-creator is relatively straightforward. This approach does however impose the risk of end-user’s suffering. A stricter concurrency control strategy requires that the entire end-to-end flow adhere to it. The integrator would be forced to implement graceful resolving of conflicts due to the concurrency control — likely a non-trivial task, definitely so if a GUI is involved.

Last request wins

Skip concurrency control altogether. What does not doing concurrency control in a REST API lead to? It lets the last request win.

There are in fact several upsides to implementing such a simple strategy — or the lack of implementing any. First of all, it is a really clean interface for your integrators, no extraneous HTTP headers or semantics to comprehend. Stronger, is it is a cleaner interface for the end-user as no clutter is added to the interface for handling concurrency conflicts.

Absolutely there are good reasons to use a stricter concurrency control strategy. Most markedly is loss of data. In a last request wins scheme, data from the second last request can be lost. Secondly, having a stricter scheme forces your integrators to think about concurrency control. When omitting a strategy as with last request wins it is more likely that issues with concurrency are not considered. Both of these issues are however approachable by other means than a stricter concurrency control scheme. For instance by implementing versioning of resources, data will never be truly be lost — conflicts can therefore be mended, by manual intervention that is.

Not only is the last request wins approach desirable for the api creator as it entails no work at all beyond clearly communicating it through the API documentation. It is also the simplest possible model for the integrator. As a matter of fact even the end-user’s interfaces would be simpler and more clean.

Opt-in concurrency control

By qualitatively analyzing some well known public APIs like Github, Spotify and etcd we see everything from strict concurrency control to none at all (the last request wins strategy). The most common solution nonetheless seem to be opt-in concurrency control, as in Spotify’s case where you can pass an optional snapshot_id on your update request.

An opt-in approach like Spotify’s clearly reminds your integrators to think about concurrency control and make up their minds whether it makes sense for their end-users.

On the contrary this is not a sound strategy if the same resource can be modified from more than one integrator. An end-user might then overwrite changes made via a opted-in integration from a opted-out integration.

Know your requirements

As shown in this post using concurrency control does impose challenges to your integrators and even their end users. Different applications definitely have different requirements to concurrency control. Some applications are even of such a nature that pessimistic locking best meet the requirements.

The essence here is to know your requirements and how they affect the api-creator, integrators and end-users. Then with this in mind use the first minimize the consequences before applying least concurrency control to fulfill the requirements while keeping all interested parties happy.

What does that deeply nested if statement do exactly?

The last couple of weeks I’ve been doing some java programming for the first time in 5 years, and something that my Scala tuned brain has had a hard time readjust to, is nested if statements with returns in them.

Traditionally Java code has quite a lot of if (someValue != null) checks to avoid the dreaded NullPointerException. In Java 8 java.util.Optional was introduced to mitigate some of this, but if you use a combination of if (someOptional.isPresent()) and someOptional.get() you could end up with just as many nested ifs as you would using traditional _ != null style. In my view this way of doing control flow is quite hard to follow and leads to high Cyclomatic Complexity.

Lets look at an example that I encountered recently (the code has been somewhat altered to protect the innocent):

As we can see the nesting goes deep, and there are several places where the code can “escape” via a return. It’s not obvious what the result will be in any given situation, at least not to me.

I started to think about how I would have solved this particular problem in Scala and I recalled that a former colleague of mine had given me a tip about a functional library for Java called Javaslang. I decided to see if I could solve this problem in a Scala’ish style using that library.

My first attempt looked like this:

It was slightly better than the original as the different operations where separated out into functions that was called from one quite compact return statement at the end. However I thought it had some shortcommings:

  • There where still some nested if statements with returns in them.
  • The map() on line 11 actually discards it’s argument and performs a side effect using the user argument from the encapsualting function instead.

This code could also have been accomplished using the methods available on java.util.Optional, so there wheren’t really any good reason to introduce Javaslang just to do it this way. Javaslang had another trick up it’s sleeve however, one that it has borrowed from languages like Scala and Haskell, namely Pattern matching.

Using that as an approach, I landed on this solution:

So what did we gain from this?

  • If we count the Cases of the Match as one Cyclomatic Complexity point, non of the three code blocks gets above 4.
  • The extraction syntax of Pattern Matching gives us the power to check both that the Option is non empty and it’s value fulfils a predicate in one swoop.
  • The alignment of the the Case lines makes for great readability (in my view).
  • If you get used to the style of looking at the last line/block of a method for it’s return statement, you’ll usually find a compact and concise definition composed of functions doing one separate thing.

Last, but not least, I’ve also included a Scala version of the same solution for comparison.

In this post we look into the concept of Synthetic Transactions, and how to get started with the implementation of a Synthetic Transaction tester.

At Unacast we are building software that scale to process millions of proximity interactions each day. We believe the best approach for building scalable and agile systems is microservices. Meaning, we believe in small, smart and focused applications over large ones. It also enables us to continuously experiment with new stuff, have fun, learn things and always choose the best tools to achieve a specific task rather than to make unnecessary or boring tradeoffs.

I have lately been looking into Go and wanted to gain some experience with building software in it. So I used this opportunity to build a real system using Go.

Monitoring Microservices

However, as you might already know, running and keeping track of multiple services is hard, and monitoring is therefore essential. Since most of our services run using Docker, we use Kubernetes to help us keep our services running if they crash unexpectedly, and we couple Kubernetes together with DataDog to help us monitor all our software environments.

Yet, when bugs, that does not crash a service, find their way into production Kubernetes’ monitoring is of no use. With that in mind it is easy to see that we are in need of some other type of monitoring to see if our services are healthy. We decided on experimenting with a concept called Synthetic Transactions.

Synthetic Transactions

Synthetic Transactions is a monitored transaction that is performed on a system running live in production. Such transactions are used to monitor that the system at hand performs as expected.

In the world of e-commerce a synthetic transaction can be a transaction that continuously tries to place an order and monitors if that order succeeded or not. If it does not succeed, it is an indicator that something is wrong and should get someone’s attention immediately.

At Unacast we use synthetic transactions to monitor our Interactions API. The single purpose of the Interactions API is to accept and ingest interactions for further processing. Since we are a data-oriented company, we are in big trouble if the Interactions API starts failing silently.

Building synthetic transaction tester

Usually we buy such services and there are a lot of great tools out there, such as NewRelic Synthetics and Pingdom. But since the synthetic transactions has to know the inner workings of our infrastructure we decided to try to build it our self.

There are several ways of building synthetic transactions. Ideally, they should be represent a complete transaction. However, I would argue that it is smarter and more pragmatic to build step by step. In this post we will go through the first step of building a synthetic transaction tester. We will share the subsequent steps in future posts.

Step 0: Monitor for expected response

The first natural step is to create a monitor that runs regularly and checks if a known request gets the expected response.

Performing a HTTP request is simple and is easily done with the stdlib alone:

In the code above we specify a SyntheticPayload that is the specific request object we want to send. We specify and send the payload in syntheticHttpRequest and parse the http.Request to specifically check if the http status code returned is 202. If it is not, or the request fails, we suspect that there is something wrong with the API, and returns error codes indicating that some further action should be taken.

In the event where a synthetic transaction fails we use DataDog, integrated with Slack and PagerDuty, to notify us that something fishy is going on. Specifically, we send Events to DataDog using their API from the synthetic transaction tester. We did this using an unofficial DataDog Go library by zorkian and it looked something like this:

This is a simple way of telling DataDog that everything is OK. We use the event tags to separate between error events and events coming from non-critical environments. This is important because no one wants to get a call in the middle of the night because the development or staging environment is not working as optimal.

Finally, we need to be able to schedule these request. For that we used cron for go. Putting all the parts together we got something that looked like the following code snippet.

Disclaimer: The code above is just an a simplification of how it can be implemented, and does not show a complete implementation.

Monitoring the synthetics transaction tester

As you might have guessed the synthetic transaction tester is also a microservice. So how should we proceed to monitor it? It is obvious that it cannot monitor itself. The solution was to monitor the “OK” events we described earlier. If these events suddenly stop arriving at DataDag we know that something is wrong and we can react accordingly.

Further steps

A simple but powerful extension of step 0 is to log metrics such as response time for each request. Such external metrics will be a more accurate measure of response time than just calculating the process time internally in a service. It can also be used to trigger alerts if the system is slow to respond, indicating a more critical issue that requires further investigation.

In the future it will be natural to extend the synthetic transaction service by verifying that data has been processed. In our case, interactions are processed and safely persisted when they reach one of our BigQuery tables after passing through AppEngine, Pub/Sub and Dataflow. It is therefore natural for us to extend the synthetic transactions monitorer to check and verify that the transactions has been persisted as expected.

In addition to verifying that transactions has been persisted as expected. We could also start to measure and deduce the expected processing time of an interaction and use this measurement to monitor if our system as a whole works efficiently. Also, we can use the same numbers to verify if the system delivers as promised according to the SLA.

Finally, an extension could be to support live consumer-driven contract testing. That is, explicitly check and verify that the response payload was correct. By doing so we can go to bed at night without worrying if we have broken the API for any of its consumers.

Enjoyed this post?

We are still learning and eager to share what we are learning along the way. If you enjoyed this post I recommend that you keep in touch because it is a lot more to come. Also, check out some of the posts about microservices written by my awesome colleges: