So… YAML, our new XML! It’s not used to just store data any more, it’s everywhere we look: configuration of Kubernetes, Ansible, continuous integration systems, build systems, and much more. It’s advertised as user-friendly, easy to read by humans and process by machines. It turns out it’s not all roses… or is it just me?
I have the privilege to entertain and teach you today, on the 13th day of our Java Advent. Sit back and read a story on how I set out to make the world just a bit more type-safe and YAML-free, and how github-workflows-kt and github-actions-typing were born.
But… Why?
It was late 2021 when we were still pushing forward the kotlin-python project where we decided to use GitHub Actions – GitHub’s first-party CI solution. I, personally, liked it a lot, but there was one piece that started to become less and less maintainable for this specific project: its YAML configuration. It was getting really convoluted; the two pipelines had around 20-30 steps each, lots of repetition across them and in each of them, iterating on them involved pushing changes to the repository, sometimes learning about a syntax error or some name mismatch after minutes or even hours. Additionally, I didn’t like the feeling of putting placeholders and logic inside YAML.
Just to give an example, here’s a part of our pipeline prior to converting to Kotlin:
(click here to see the whole file)
I see the following problems, with GitHub’s YAML config and later with YAML in general:
JDK_9="$JAVA_HOME"and./gradlewform a standard preamble for steps whenever Gradle is called, and are repeated multiple times. I needed to copy and paste these, and what if something changes in the preamble, like an extra environment variable needs to be set? It has to be changed in all these places- what other inputs does the
actions/upload-artifactaction provide, and are my input names and types correct? Is it even a correct action name? What other versions are available – maybev2is already deprecated? needs: build_and_test– what an interesting way to set a dependency on another job. It’s not a part of YAML, it’s another thing added by GitHub. On YAML level, it’s just a key-value pair and GitHub adds the semantics- there are two pipelines (separate YAML files) which share common parts, and there’s no way to deduplicate things with pure YAML
- YAML relies on indents to get its structure. Is it always obvious to you how many spaces you need to put? Or maybe tabs? Even if you think you got the indent right (your IDE is smart enough to provide the vertical rulers), now it turns out the key is incorrect on this indent level. Sigh…
- the reference to
${{ matrix.testTask }}keeps repeating, and there’s no mechanism in YAML to ensure thattestTaskis truly one of matrix parameters, and that evenmatrixis a thing. The expression mechanism${{ ... }}is an extension provided by GitHub
Yes, this is my very own rant about YAML. As you see, YAML is indeed simple and readable, but only on the surface: too simplistic to express some constructs in a convenient and safe way. These are all strings, lists, dictionaries… Even if you think you know YAML, you still need to learn how certain mechanisms are implemented in a given tool that uses YAML, like here: expressions inside strings or setting a dependency on another job. It’s like a mirage: promising simplicity, providing headache in the longer run.
I thought there has to be a better way. I decided to try an experimental approach: express it all in a general-purpose programming language (like e.g. Jenkins does with Groovy) and see how it goes – so basically create “the better way” myself, in Kotlin.
Let there be DSL
Before you ask: DSL stands for Domain-Specific Language. In this case, we’re talking about an internal/embedded DSL. Simply put, it’s about using the host programming language’s features to be able to model a specific problem in a human-readable way, and let the DSL implementation handle the rest. Kotlin does provide utilities to efficiently create readable DSLs, so it seemed like a great fit for my little experiment. Some DSLs you may know: SQL is an external DSL, and fluent Hamcrest matchers create an internal DSL within Java/Kotlin.
The Kotlin internal DSL described further is available under https://github.com/krzema12/github-workflows-kt/. Let’s dig deeper into it together!
The foundation
Where do we start designing the DSL? My approach was to take a dead simple workflow and express it in soon-to-be Kotlin DSL. The most frequent task our CIs take care of is building the project, so let’s make the very first step by expressing it in YAML (.github/workflows/build.yaml):
These basic entities are brought to our attention:
- a workflow, with its triggers
- a collection of jobs, with its steps
- steps, with its arguments
Using our DSL, it maps to such Kotlin file (.github/workflows/build.main.kts):
“Whoa, a lot of extra complexity!“, you may think. True, there are several more items, but they look scarier than they really are, so let’s go through them line by line.
This file is in fact an executable Kotlin script. Kotlin compiler allows passing a piece of Kotlin code, and thanks to putting .main.kts in the extension, we don’t even need the main function. To instruct the operating system that Kotlin should be used to interpret it, we need the shebang:
Then this line follows:
which adds a dependency on the DSL library. It’s a regular Maven artifact, hosted on Maven Central so we don’t need any extra repository. Think of it as a replacement for a Gradle/Maven configuration script, only really compact and with limited capabilities.
The bunch of imports speak for themselves. There are usually hidden in the IDE anyway.
Then, something interesting starts to happen. We have a top-level workflow(…) function call. After the call, at the very end of the file, there’s also a call to writeToFile(). Inside, we also see appropriate functions to model jobs and steps. Thanks to the arguments passed to the workflow(…) function and several conventions, executing this script writes a YAML file to .github/workflows/build.yaml. Great! Let’s see the file and confirm everything looks as expected:
Oh my, even more lines… What have we done – from 12 to 33 lines?! Usually this YAML is just a preview of what’s being run on GitHub, and only the Kotlin script is modified. Think of it as a compiler – it does produce some lower-level machine code that is less readable and more verbose, but we usually don’t care that much. Here we have to keep it in our repository because GitHub Actions are not aware of our Kotlin scripts – they only understand the YAMLs. Kotlin becomes our daily interface to author the workflows, and YAML is there only for the GitHub runtime, plus to double check that the DSL does what we expect.
One thing that stands out in the above YAML is the extra check_yaml_consistency job. As already explained, we need to keep both the Kotlin script and the YAML in our repository. Can it happen that someone is not aware of this DSL and edits the YAML by hand? Sure it can. The consistency mechanism is there to guard against the two files getting out of sync – it’s added implicitly by the DSL and runs before your actual workflow (you can opt out of adding the check if your way of using the DSL doesn’t need it). The approach is simple: take the Kotlin script, regenerate the YAML and compare it with the YAML present in the repository. If there’s a perfect match, carry on. Otherwise, fail fast.
Other elements that you may notice are explicit step IDs. In general they are optional in the YAML version, but the DSL adds them preemptively to be able to provide features like access to step outputs.
The rest is still readable, not minified or obfuscated in any way.
The power of Kotlin
All right, we’ve entered the Kotlin world. What does it give us?
The first and foremost feature where all other useful features come from is the compilation phase. With YAML, we just put it in the repository and learn about certain issues at runtime. Here, we have a chance to catch a plethora of issues already in the IDE, and another chance when running the script and generating the YAML.
In the below sections, we’ll go through some of the pain points I enumerated at the beginning, and see how Kotlin and the DSL library addresses them.
Removing repeated parts
They can appear in various forms, like repeated strings inside commands or repeated steps in jobs. For repeated strings, the simplest solution is to extract the value to a variable/constant:
or, if you feel like creating a little piece of abstraction, it can be extracted to a function. This is an extension function because run(…) function is a part of the DSL, and job(…) function is a lambda with receiver of type JobBuilder<*>:
For using Gradle from within GitHub Actions, there’s a dedicated GitHub action providing some extra features like caching. Here I just want to show examples of extracting a common part of a command. You can come up with your own – you have the whole power of Kotlin at hand!
Because the extension function for a single run(…) occurrence works just fine, we can extract complex parts of workflows with multiple steps. These functions can even create jobs given some parameters, so the tool is pretty powerful.
Using actions type-safely
Actions, apart from the name of the CI itself, ale also reusable pieces of logic, for example the most popular https://github.com/actions/checkout. This particular one has 14 inputs, and 0 outputs. A traditional way to learn about them is to go to the action’s repository and either browse the README hoping the inputs are documented and the descriptions are up-to-date, or browse action.y(a)ml file directly.
The DSL proposes another approach. It comes bundled with dozens of actions, each of them providing typed inputs and partially type-safe outputs. It means that you get suggestions in your IDE while writing the workflow in Kotlin, along with documentation provided by the action author. The script won’t compile if you provide incorrect input name or its type:
How is it done? I developed a standardized way of describing action types, see https://github.com/krzema12/github-actions-typing. For most actions where authors didn’t integrate with the typing solution yet, the typings are stored inside the DSL repository (e.g. https://github.com/krzema12/github-workflows-kt/blob/main/actions/actions/checkout/v3/action-types.yml). Some authors successfully onboarded github-actions-typing, like https://github.com/Vampire/setup-wsl where the typings are hosted in its repository: https://github.com/Vampire/setup-wsl/blob/master/action-types.yml. Maintaining the typings hosted as a part of the DSL does add some maintenance overhead, so I really appreciate when action owners decide to maintain them on their own. Automation created in scope of this project takes care of the rest.
Modeling dependencies between jobs
As another example of how a proper programming language can help us model stuff, here’s how we define that job_1 depends on job_2 (job_1 has to run before job_2 starts):
Jobs are represented with its own types (Job) and nothing stands in our way of storing a reference to it in a variable. Then, it’s as simple as passing a reference to a job in another job’s dependencies. It’s logical and type-safe, no need to work on strings and map keys, like in YAML.
Summary
We’ve barely scratched the surface of what’s possible using github-workflows-kt. I hope I managed to draw your attention and show the power of the type-safe approach over plain YAML, especially for more complex workflows. It’s not for everyone – some people will prefer YAML, and Kotlin fans will be more eager to try it out.
If you like what you’ve seen so far, I encourage you to give it a test drive, there’s a documentation available here that will guide you further. I also love feedback – feel free to get in touch via the issues or on the dedicated Slack channel. Let’s work in the open-source spirit to improve the library together!
At this point I’d like to thank all the contributors and early adopters that already provided feedback and introduced great improvements to the library. Without your support, we wouldn’t manage to go this far. I’m looking forward to more adoption of the library and providing even greater feature coverage!
steinhauer.software