Supercharge your API development with code generation
At Zoosk Engineering, one of our biggest focuses is to make sure everything we do is as simple and as efficient as possible. This translates into everything from our code and tooling to our processes as well. This means constantly experimenting with new ideas in an effort to improve in these areas. One of those experiments, which we permanently use now, was to use code generation for creating server/client contracts.
The Old Process
Zoosk, like most current web applications, uses a RESTful API to communicate contracts between server and client. Knowing this, our engineering teams are broken down based around this separation into: API, Web, Mobile Web, iOS, and Android. The following graph visually represents a typical feature’s lifecycle from a API/Client tech kickoff to feature completion.
The first step is to have a technical kick-off between the client/server teams. In this meeting these teams agree on the contract for any new endpoints that will need to be created. Once this happens, the server team documents the agreed upon endpoint on our wiki and starts the implementation of the endpoint. Once these contracts are agreed upon, the client teams have what they need to start building their model objects and parsers that correspond to the endpoint spec. When completed, the client team is able to build a subset of the views/controllers while the API finishes work on the endpoint in parallel. But, client work can progress only so far before those teams become blocked by not having functioning API endpoints. In this case, developers typically shift to other projects while waiting for the endpoints to finish development.
Drawbacks
While this process worked well for us for awhile, we began to notice a couple areas that we could improve on.
Context Switching
If the server and client teams begin development of a feature at the same time, (which often happens at Zoosk due to how quickly we move) there is always some time we lose when clients become blocked by the server team. The topic of context switching as a whole is out of the scope of this article, but suffice to say that it’s significant. We do a good job of limiting that time by mocking out endpoints on some of the clients, but that requires work across all of our client teams.
One idea to limit context switching is to ensure all API endpoints are completed a sprint before client teams start working on them. There are a few issues with this process in practice.
- The overall timeline of a feature goes up. We do our work in two week sprints, so this means that all decent sized features now take a minimum of four weeks to go through our pipeline.
- Managing these dependencies between teams adds overhead to the process. It’s not huge, but when you have a number of projects you’re tracking, it adds up.
- Endpoints built in isolation often need to be reworked. Because the endpoints are built in isolation without much collaboration from clients (as clients are busy on other projects), we often end up having to backtrack and make changes to completed endpoints.
We’ve solved our context switching problem, but we’ve introduced inefficiencies in other areas of our process.
Boilerplate Code
Another issue we noticed was that there was a large amount of boilerplate code that would have to be rewritten, albeit slightly differently, for all API endpoints by the server/client teams. Any time you’re spending writing boilerplate code could be time spent NOT writing boilerplate code. From the definition of the term, boilerplate code is redundant and always a candidate for code generation. Each individual team alleviated some of these issues using IDEs or tools built within their teams at varying rates of success.
In PHP
On the server side, we have a lot of code to handle and create responses interchanged between the clients.
| <?php | |
| /** | |
| * @return ApiNode|null | |
| */ | |
| public function createDataNode() | |
| { | |
| /** @var ApiDataNode */ | |
| $dataNode = new ApiDataNode(); | |
| if (!is_null($this->date)) { | |
| $dataNode->setAttribute(API_NODE_ATTR::DATE, $this->date, ApiNode::TYPE_STRING); | |
| } | |
| /** @var ApiNode */ | |
| $nestedGroupNode = new ApiNode(API_NODE_V5::NESTED_GROUP); | |
| $nestedGroupNode->setAttribute(API_NODE_ATTR::NESTED_BOOL, $this->nestedBool, ApiNode::TYPE_BOOL); | |
| /** @var ApiNode */ | |
| $nestedEndpointObjectNode = new ApiNode(API_NODE_V5::NESTED_ENDPOINT_OBJECT); | |
| /** @var ApiListNode */ | |
| $alphanumericList = new ApiListNode(API_NODE_V5::ALPHANUMERIC); | |
| foreach ($this->alphanumericList as $alphanumericListSortKey => $alphanumericListItem) { | |
| /** @var ApiNode */ | |
| $primitiveNode = new ApiNode(API_NODE_V5::ALPHANUMERIC); | |
| $primitiveNode->setAttribute(API_NODE_ATTR::VALUE, $alphanumericListItem, ApiNode::TYPE_STRING); | |
| $alphanumericList->addToList($primitiveNode, $alphanumericListSortKey); | |
| } | |
| $nestedEndpointObjectNode->addChildNode($alphanumericList->getApiNode()); | |
| $nestedEndpointObjectNode->setAttribute(API_NODE_ATTR::DOUBLE_NESTED_GUID, $this->doubleNestedGuid, ApiNode::TYPE_STRING); | |
| $nestedGroupNode->addChildNode($nestedEndpointObjectNode); | |
| $nestedGroupNode->setAttribute(API_NODE_ATTR::NESTED_INT, $this->nestedInt, ApiNode::TYPE_INT); | |
| $nestedGroupNode->setAttribute(API_NODE_ATTR::NESTED_STRING, $this->nestedString, ApiNode::TYPE_STRING); | |
| $nestedGroupNode->addChildNode($this->nestedThing->getApiNode()); | |
| $dataNode->addChildNode($nestedGroupNode); | |
| $dataNode->addChildNode($this->subsetInfo); | |
| $dataNode->addChildNode($this->testObject->getApiNode()); | |
| /** @var ApiListNode */ | |
| $testSubObjectList = new ApiListNode(API_NODE_V5::TEST_SUB_OBJECT); | |
| foreach ($this->testSubObjectList as $testSubObjectListSortKey => $testSubObjectListItem) { | |
| $testSubObjectList->addToList($testSubObjectListItem->getApiNode(), $testSubObjectListSortKey); | |
| } | |
| $dataNode->addChildNode($testSubObjectList->getApiNode()); | |
| return $dataNode; | |
| } | |
| /** | |
| * | |
| */ | |
| private function validateParameters() | |
| { | |
| $intList = $this->validateParam(API_PARAM_V5::INT_LIST, false, API_PARAM_TYPE_V5::STRING); | |
| if (is_null($intList)) { | |
| $this->intList = array(); | |
| } else { | |
| $this->intList = explode(',', $intList); | |
| foreach ($this->intList as $index => $item) { | |
| $this->intList[$index] = intval($item); | |
| } | |
| } | |
| $this->paramDescribedInt = $this->validateParam(API_PARAM_V5::PARAM_DESCRIBED_INT, true, API_PARAM_TYPE_V5::INT); | |
| $this->paramOptionalInt = $this->validateParam(API_PARAM_V5::PARAM_OPTIONAL_INT, false, API_PARAM_TYPE_V5::INT); | |
| $paramPlusAlphanumericList = $this->validateParam(API_PARAM_V5::PARAM_PLUS_ALPHANUMERIC_LIST, true, API_PARAM_TYPE_V5::STRING); | |
| $this->paramPlusAlphanumericList = explode(',', $paramPlusAlphanumericList); | |
| $this->paramString = $this->validateParam(API_PARAM_V5::PARAM_STRING, true, API_PARAM_TYPE_V5::STRING); | |
| ?> |
In javascript
We use the Google Closure Compiler and Tools extensively at Zoosk which led us to a one of a kind integration with Angular and Closure for our web and mobile web SPAs (Single Page Application). That said, as great as these tools have been for supporting a SPA with over 500K lines of javascript, they are definitely not easy on the typing. The jsdoc annotations and coding patterns have been great for code quality and stability but require a lot of extra keystrokes. For example, every new API endpoint requires a few different javascript classes to get the models ready for consumption by our views and controllers.
| /** | |
| * @constructor | |
| * @param {trove.data.TestObjectBuilder} builder | |
| */ | |
| trove.data.TestObject = function(builder) { | |
| this.alphanumericList_ = builder.alphanumericList; | |
| this.primitiveAlphanumeric_ = builder.primitiveAlphanumeric; | |
| this.primitiveString_ = builder.primitiveString; | |
| this.stringSet_ = builder.stringSet; | |
| }; | |
| /** | |
| * @type {!Array.<string>} | |
| */ | |
| trove.data.TestObject.prototype.alphanumericList_; | |
| /** | |
| * @type {!string} | |
| */ | |
| trove.data.TestObject.prototype.primitiveAlphanumeric_; | |
| /** | |
| * @type {!string} | |
| */ | |
| trove.data.TestObject.prototype.primitiveString_; | |
| /** | |
| * @type {!Array.<string>} | |
| */ | |
| trove.data.TestObject.prototype.stringSet_; | |
| /** | |
| * @param {trove.data.TestObject} otherTestObject | |
| * @return {boolean} | |
| */ | |
| trove.data.TestObject.prototype.equals = function(otherTestObject) { | |
| return !!otherTestObject && | |
| goog.array.equals(this.getAlphanumericList(), otherTestObject.getAlphanumericList()) && | |
| goog.array.equals(this.getStringSet(), otherTestObject.getStringSet()) && | |
| this.getPrimitiveAlphanumeric() == otherTestObject.getPrimitiveAlphanumeric() && | |
| this.getPrimitiveString() == otherTestObject.getPrimitiveString(); | |
| }; | |
| /** | |
| * @return {!Array.<string>} | |
| */ | |
| trove.data.TestObject.prototype.getAlphanumericList = function() { | |
| return this.alphanumericList_; | |
| }; | |
| /** | |
| * @return {!string} | |
| */ | |
| trove.data.TestObject.prototype.getPrimitiveAlphanumeric = function() { | |
| return this.primitiveAlphanumeric_; | |
| }; | |
| /** | |
| * @return {!string} | |
| */ | |
| trove.data.TestObject.prototype.getPrimitiveString = function() { | |
| return this.primitiveString_; | |
| }; | |
| /** | |
| * @return {!Array.<string>} | |
| */ | |
| trove.data.TestObject.prototype.getStringSet = function() { | |
| return this.stringSet_; | |
| }; |
In Java
On the Android client, the patterns that we follow there are a lot less verbose, but there is still boilerplate code we have to write for every new endpoint we build.
| package com.zoosk.zoosk.data.objects.json; | |
| import java.util.List; | |
| import com.zoosk.zaframework.lang.JSONArray; | |
| import com.zoosk.zaframework.lang.JSONObject; | |
| class TestObject extends ZObject { | |
| private enum DescriptorKey implements ZObject.DescriptorKey { | |
| ALPHANUMERIC_LIST, | |
| PRIMITIVE_ALPHANUMERIC, | |
| PRIMITIVE_STRING, | |
| STRING_SET; | |
| } | |
| public TestObject(JSONObject jsonObject) { | |
| super(jsonObject); | |
| } | |
| @Override | |
| protected Class<? extends DescriptorKey> putDescriptors(DescriptorMap descriptorMap) { | |
| descriptorMap.put(DescriptorKey.PRIMITIVE_ALPHANUMERIC, String.class, "primitive_alphanumeric"); | |
| descriptorMap.put(DescriptorKey.PRIMITIVE_STRING, String.class, "primitive_string"); | |
| descriptorMap.put(DescriptorKey.STRING_SET, new ZObject.SetDescriptor<String>("string_set", "string") { | |
| @Override | |
| protected void populateList(JSONArray array, Set<String> listToPopulate) { | |
| for (JSONObject jsonObject : array) { | |
| listToPopulate.add(new String(jsonObject.getJSONObject("value")); | |
| } | |
| } | |
| }); | |
| descriptorMap.put(DescriptorKey.ALPHANUMERIC_LIST, new ZObject.ListDescriptor<AlphanumericListItem>("alphanumeric_list", "list_item") { | |
| @Override | |
| protected void populateList(JSONArray array, List<AlphanumericListItem> listToPopulate) { | |
| for (JSONObject jsonObject : array) { | |
| listToPopulate.add(new AlphanumericListItem(jsonObject)); | |
| } | |
| } | |
| }); | |
| return DescriptorKey.class; | |
| } | |
| @SuppressedWarning("unchecked") | |
| public List<AlphanumericListItem> getAlphanumericListItem() { | |
| return (List<AlphanumericListItem>) this.getList(DescriptorKey.ALPHANUMERIC_LIST); | |
| } | |
| public String getPrimitiveAlphanumeric() { | |
| return this.getString(DescriptorKey.PRIMITIVE_ALPHANUMERIC); | |
| } | |
| public String getPrimitiveString() { | |
| return this.getString(DescriptorKey.PRIMITIVE_STRING); | |
| } | |
| @SuppressedWarning("unchecked") | |
| public Set<String> getStringSet() { | |
| return (Set<String>) this.getSet(DescriptorKey.STRING_SET); | |
| } | |
| } | |
In our Wiki
We keep very thorough documentation that must be created for all API endpoints.
Enter Code Generation
Because an API is by definition a well-defined contract between the server and clients, we realized we should be able to easily build code generation tools to alleviate most of the redundant code. We coined this new codebase – Lotus. Lotus is written in Python and provides an Abstract Syntax Language called LSL (Lotus Syntax Language) to define model and endpoint specifications. Using Lotus, we can generate code for all platforms that depends on the API interface.
Here’s an example of what LSL looks like:
| from lotus.declarations import declare_type | |
| declare_type({ | |
| 'api': 'test_object', | |
| 'trove': 'trove.data.TestObject', | |
| 'android': 'TestObject', | |
| 'spec': { | |
| # Primitive types | |
| 'primitive_string': 'string', | |
| 'primitive_alphanumeric': 'alphanumeric', | |
| # Lists | |
| 'alphanumeric_list': ['alphanumeric'], | |
| # Sets | |
| 'string_set': {'string'}, | |
| } | |
| }) | |
| declare_endpoint({ | |
| 'api': 'v5/test/object/get', | |
| 'trove': { | |
| 'parser': 'trove.api.TestObjectParser', | |
| 'command': 'trove.commands.FetchTestObjectCommand' | |
| }, | |
| 'android': { | |
| 'parser': 'TestObjectGet' | |
| }, | |
| 'major_version': 5, | |
| 'minor_version': 0, | |
| 'method': 'POST', | |
| 'auth': True, | |
| 'ssl': False, | |
| 'user_interaction': True, | |
| 'domains': '*', | |
| 'opt_nodes': ['v_test'], | |
| 'params': { | |
| # Required | |
| 'param_string': 'string', | |
| # Described | |
| 'param_described_int': 'int, Something numeric', | |
| # Optional | |
| 'param_optional_int?': 'int', | |
| # Lists | |
| 'int_list?': ['int, Something long'], | |
| # One or more | |
| 'param_plus_alphanumeric_list+': 'alphanumeric', | |
| }) |
All the code referenced earlier in the article will be generated from the minimal LSL mentioned above. This includes:
- PHP code for our API team to parse and build responses
- HTML for the documentation for our internal Wiki
- Google Closure Code for models and parsers for Web and Mobile Web teams
- Java code for Android data models
- XML for .plist files for iOS
- Unit test skeletons
- Live endpoints with mock data
Mock Endpoints on the Server
The other critical improvement Lotus provides us with is the ability to generate mock endpoints as soon as the API contract is agreed on between server/clients. Using LSL, we can generate functional endpoints on the server using mock data based on the data types that LSL supports. This essentially allows us to have working endpoints on day one and permits the clients a much longer window to develop before becoming blocked.
Old Process
In the chart below, you can see how after the advent and maturation of Lotus, the timeline of our feature development has changed significantly. All of the boilerplate code has disappeared, and we can see that the context switching cost has disappeared as well.
New Process
Code Generation Options
Lotus was developed in-house for our API that we’ve built through the years, but there’s one promising code generation tool I’ve seen that’s built around a similar idea. It’s called Swagger. Swagger allows you to build RESTful APIs and generate the corresponding client SDKs very similar to Lotus. This would be impractical if you are already supporting a large API codebase, as we do at Zoosk, but it seems very appealing for greenfield development.
We’re always trying out different ways to be faster and work more efficiently at Zoosk, and in this case, leveraging code generation in Lotus has fundamentally changed the pace at which we’re able to ship features at Zoosk. Happy coding!
