From lmdo to cfctl – The Journey of Developing a DevOps Tool

Photo by Mikael Kristenson on Unsplash

Prologue

I was working in a startup back in year of 2016, of which I was the technical founder. Being the lead and all-rounder for a small tech team, the number of technologies and tools I had to manage were overwhelming. Majority of the technologies in our tool chain were very much bleeding edge at that time. Just to paint the picture, we were using react/react native for SPA and mobile apps; the backends were microservices and completely built on serverless infrastructure using AWS Lambda plus a whole bunch other services such as API gateway, SQS, SNS, RDS, Cognito and CloudFormation. Looking around, no one was setting up their stack similar to ours.

One of the downsides of being the early adopter in the technology adoption life cycle[1] is you won’t find a good eco-system that can service the technology immediately. Most of the open source tools we found were either highly opinionated or focusing on narrow use cases. If we were to adopt those tools, we would end up with a lot of workarounds while still have to get familiar with each of them which only suit our use partially. It becomes even more challenge to onboard a new hire to the team.

Under those circumstances, I spent a week wrote the first version of lmdo in Python because it’s the language choice in the team. It solved the team’s immediate problem by centralising the tool chain, providing a simple point to manage the microservice stacks by packaging services such as CloudFormation, Lambda functions and API gateway together. Coupling with CICD automation and scrum, the teams SDLC (Software Development Lifecycle) was fluid and agile. Any devs in the team regardless he/she is frontend or backend developer can spin up their own testing stack with ease. For new team members, there is only one tool to master.

Since we have full control of the tool chain, many more functionalities were added to lmdo based on the team requirements in a fast fashion over the period. Slightly more than a year later, AWS finally released their first version of SAM (Serverless Application Model) which has pretty much same idea and similar functionalities albeit supporting more languages.

Fast-forward, the technology landscape has changed dramatically. Many are at their mature stage. There are a great deal of tools to choose from. Many of those tools are basically abstractions. By introducing another layer on top, end users would be able to perform and expect the outcome with only knowledges at the meta level. Nonetheless it also has side-effects such as losing the fine detail level of technical skills. Take Terraform for example, it’s easy to use with many wonderful functionalities, supporting multi-clouds (with caveats). However it uses it’s own templating system. I doubt many its users know how to write an equivalent CloudFormations or Google Cloud Deployment Manager without learning it from the beginning. All new AWS services or features are naturally made available to its own CloudFormation but one might have to wait for Terraform version update to get to use it.

Not long ago I was tasked to create and manage a few AWS stacks. Given the size of the stacks and it’s relatively simple job, I naturally wanted to use lmdo. But then I realise it’s not the best tool for job after a few iterations. One is I only need to use its CloudFormation feature, none of the others features are in scope. Secondly lmdo’s template managements doesn’t fit for our current DevOps process. A new tool seemed to be needed. I came up with these requirements:

  • Reverse vendor agnostic: One can easily remove the dependancy of the tool and go back to use awscli or console to create the same stack.
  • No persistent state: Stack state should be queried at real time rather than relying on a on-disk state management.
  • Dependent nested stack auto-uploading.
  • Stack outputs auto-injection to other stacks.
  • Encryption facility that facilitate secrets encryption so they can be stored in the repository.
  • Configuration over convention: To provide maximum flexibility.
  • Manage CloudFormation stacks lifecycle.
  • YAML format for parameter files.

After much research, it’s clear that I will need write this up myself. Over the years, Go has become the go to language for me for writing up a command line tool given that my many years of experience with it and all the goodies it provides such as compatibility for multi-platform, easy distribution as a single binary. Naturally this tool will be written in Go.

Much of the codes and releases you can find in github. I am going to elaborate the template folder structure design here.

In cfctl there are three folders needed to be configured (default stacks.yaml), anything under those three folders are completely up to the user with maximum flexibility as long as the relative paths are given when referencing:

|- templates (1)
|--- ...
|- project
|--- env-values (3)
|--- default
|--- aws-acc-1
|--- aws-acc-2
|--- parameters (2)
|--- ...
|--- stacks.yaml
  1. Template folder:

As a rule of thumb, for the sake of reusability and isolation of concerns:

  • Write generic CloudFormation with parametrised values
  • Break large stacks into smaller or nested stacks

Following those principle, naturally you will have general purpose templates and templates with specific purpose which could be formed by multiple general purpose templates.

2. Template parameter file folder:

This folder contains all the parameter files for the stack templates. The parameter files are written in YAML format however it can be converted back to JSON in one command if needed. cfctl provides a simple templates system using Go template for the parameter values. It allows you to use values from the environment folder and has a few useful helper function such as getting value from stack outputs or system environment variable.

3. Environment specific parameter value file folder:

This folder contains all the values that are used in the parameters files. The values can refer to other values. The values also can be encrypted by user and decrypted by cfctl on the fly using ansible-vault. A typical way organising this folder is by create sub-folders per AWS account. The only one convention is if you have a sub-folder named “default”, all values in it will be loaded every time regardless what target environment is set during deployment.

There are two extra go modules I had also written to facilitate cfctl. One is the graph data structure (you can check it out here) because Go doesn’t have a built-in graph. Basically this graph module allows you to build a graph and store it on a thread-safe store. The Khan sort function allows you to sort the tree and detect circular dependancy. It provide interfaces so that you can easily extending the module, for example, to use your own store, to store your own data on the vertices.

The other module is the vault module. It provides file encryption facility according to ansible-vault’s specifications [2]. With its help you can store secrets along side the template codes safely. cfctl will decrypt those files on the fly during operation without needing to manually decrypt them before hand.

Interestingly, during the course of cfctl’s development, AWS released Beta version of their CloudFormer which also targets many similar problems cfctl trying to solve. However the use case is slightly different here with the introduction of CDK

cfctl is not perfect. There are still many areas it can be improved. Please feel free to provide feedbacks if you are interested. The releases can be downloaded via github.

Reference:

  1. Technology adoption life cycle
  2. Ansible Vault

Disclaimer: The views expressed here are solely those of the author in his private capacity and do not in any way represent the views of any organisation.


From lmdo to cfctl – The Journey of Developing a DevOps Tool was originally published in ITNEXT on Medium, where people are continuing the conversation by highlighting and responding to this story.