API First
14 June 2018
In the last few years I've learned a lot about how a business can continue iterating on a product built upon a software stack that is starting to show a bit of age.
One of the lessons I've learned is that products that target businesses should be built API-first, particularly B2B SaaS. What this means is that you start by building the API for the product, the API isn't an after-thought that gets added on when a customer wants to integrate.
What is an API?
API stands for Application Programming Interface. An API essentially provides a User Interface (UI) for machines to be able to interact with your application, it is essentially a Machine Interface for your application.
The functionality that can be accessed via this Machine Interface depends on how you build your application. It can be a sub-set, a super-set, or the same functionality available in your normal UI.
Essentially, it is a set of standards that your company makes available through documentation and examples for how developers at other companies can build programs that can interact with your product.
Most APIs work via normal HTTP requests that your browser sends every time you visit a webpage, just instead of a page, it sends back a set of data that is structured so it is easy for a programmer to use.
What is different between this and how products are normally built?
Most products have an API, but many are built using some sort of "unofficial API". This is because many applications built today actually run on the client, in an environment you don't control. When you're application lives on a server, and the clients (other programs that you're developers build) have to interact with that server, this almost guarantees you are using an API of some sort. Though, it is probably much less formal than what I described above.
This informality causes development teams to learn a number bad habits when working with these APIs. They make breaking changes to these existing "unofficial APIs", APIs get structured so it is very much dependent on how your product functions, and there isn't any centralized documentation for how the API works (besides the code) because it is constantly changing.
These are APIs that you don't necessarily want to expose to your customers for many of the reasons mentioned above. So, you decide you need to build a real API, in addition to your unofficial API. Now you're stuck maintaining two APIs that function differently.
So how is building API First different?
Building API first means that you start designing the systems that make up your product by thinking about how other programs will interact with them. These programs might be your own iOS app or JavaScript app, or a customers' ERP system.
It does not mean that all of these API endpoints are public by default and accessible to your customers outside of your clients. Going API First requires you to have a process for making new API endpoints and resources public, but you may very well not need to worry about making them public for awhile. In fact, if you are still actively iterating on an API, I would encourage you to keep it private until those changes have slowed and the way it works has become stable.
By public, I mean accessible to more than just your company's products. Your company can still maintain control over who gets access to the API and what API endpoints and functionality they get access to.
Going API First means your client software, the software that the end user thinks of as your product, are using the same APIs that your customers are using or might eventually use in production. We call this dogfooding, or eating your own dog food. This means when there is a problem with an API, you have better odds of finding out before your customers and getting a head start on fixing the issue. The engineers that are responsible for maintaining your API have customers who use their product in the same organization, which can dramatically speed up feedback loops, especially on API changes.
It will require more centralized documentation, that is easy to update (maybe even automatically, with something like Swagger). You might even be able to have your CI tool check to see if documentation was updated and not let changes be deployed until it is.
To try and avoid breaking changes, you'll probably have much more generic API endpoints (or resources) that are not tied to the the functionality of a specific feature in your product. Instead, a feature in the product will likely be composed of multiple API requests to different endpoints.
Keeping the output of these APIs well-defined and not dependent on the features of the app allows you to test these APIs in an automated fashion, which can help prevent unintentionally breaking an API. It also makes it easier to migrate functions of your existing API into their own services when that is necessary later on in a project. The service just needs to accept data in the same format and output data in the same format as before.
Building a product API First requires a bit more planning up front. This planning will likely be around:
- What resources will I need available via API endpoints to implement this feature? Which ones already exist?
- How can we make this change non-breaking? Or will we need to version that endpoint?
- Who should get access to this API endpoint? Would we ever want to make it public and expose it to our customers?
- How can I make this API endpoint more generic and useful for more future features? And what's the tradeoff for doing it?
A nice side effect of building API First is that you generally separate your web client from your API in terms of where the code lives is actually more significant than it seems and not always obviously so. Keeping these in the same project (being served from the same server) tends to cause you to take some easy shortcuts that can be hard to revert:
- Using sessions for authentication by default
- Using hash fragments instead of push state for client history
- Serving static content from your API server
Defining a process to propose changes to APIs and have your team review the changes before they are actually implemented will likely be a crucial part of making this work. For a new, private API endpoint making up specific feature, this probably is not a big deal, but any additions or changes to public facing API endpoints should definitely have some sort of review process akin to your code review process (you are doing code reviews aren't you?). Defining the details prior to implementing it can have a number of benefits:
- Allow people to comment on the changes to an existing API or the proposed new API prior to implementation and give visibility.
- Allow the engineers working on your client software to mock responses because they can know what the data coming back from the API will look like.
- Allow QA to start building tests for those API endpoints.
- Additional context for the engineer working on the API.
- If it is a public API, being able to share the documentation and get feedback from customers.
I mentioned a number of different roles above. Engineers working on client software, the person proposing the changes, the engineer that works on the API and will implement the changes, the person getting feedback from customers.
It is possible for this to be 4 different people or all roles being carried out by a single full stack engineer and product manager, it depends on the organization. I worry that adding process like this to adding API endpoints and making changes will discourage full stack developers or teams working in different environments, but I think the end result is worthwhile. Most of the changes to the organization and the architecture of the application, I think, I described above are very positive traits.
It's fairly easy to make this change when starting out. The longer you go, the harder it gets and soon you'll be maintaining two APIs, an unofficial one and an official one.