A walk through our backend technologies

Functional programming lives at the core of our team. We are convinced this paradigm gives us the tools to write correct software in fewer lines of code.

Our current language of choice is Scala, which has a rich ecosystem of libraries, and runs on the ever powerful JVM.

In this post, I'll give you a brief summary of the technologies we are employing, since it's rapidly evolving, making it hard for any job spec to be up-to-date.

Event-driven platform

Our core platform breathes via Apache Pulsar. We have chosen it over other distributed messaging brokers for several reasons. Mainly because it's easy to deal with topics; we can create millions of them on the fly without worrying about performance.

Another reason is that we have an experienced Dev Ops in our team, and also, because we like to use bleeding-edge tech :)

Pulsar doesn't provide any functional client for Scala, so we created our own using Fs2, which builds atop the Java client. Its name is Neutron, and it will be soon open sourced under our Github organization.

Web Sockets

Http4s is our HTTP framework of choice. It integrates quite nicely with Cats Effect and Fs2, and it gives us Web Sockets, which we use to communicate with the frontend, who are also using cutting-edge technologies such as WebRTC and Typescript.

So, our services are mainly Pulsar consumers / producers, and some also expose a few HTTP endpoints.

Strongly-typed domain

If you have seen my recent talks, you might know I preach for newtypes and refinement types. So it shouldn't come as a surprise when I tell you we effectively use the Newtype and Refined libraries, together with some third-party integrations.

For configuration, we also use the excellent Ciris library, which comes with support for refinement types, in addition to other goodies.

Build tools

We use Mill to build our Scala services. It all lives in a mono-repo but each module is built as an individual Docker image.

I like Mill's simplicity because it's all Scala. No magical plugins. Its watch mode has also been a real productivity booster so far. I recommend it to anyone starting out a new application.

For our open-source projects, though, we choose SBT, which has better support for binary compatibility checks (see MiMa) and other good things we don't want to write from scratch.

We also make use of the famous tpolecat compiler flags, brought by mill-tpolecat, as well as a set of compiler plugins we use and recommend: better-monadic-for, context-applied, kind-projector, and wart-remover.

Testing

We have recently adopted Munit, with its Scalacheck support. It was an easy migration from Scalatest and we really like the reports we get out of it so far.

Furthermore, we have adopted Circe Golden to make sure we don't break some of the protocols of communication with the frontend, which is mainly JSON. We had to write some custom code to integrate it with Mill, which can hopefully be open sourced as well.

Summary

As tech evolves, we need to evolve with it. Our backend is in constant motion and we are open to quickly adopt technologies when we see fit.

In the near future, I hope I can introduce more Nix into our system for reproducible builds. In fact, I just introduced it to build this blog!

If all of this sounds interesting to you, please do get in touch!

-- Gabriel.