Golang Multimodule Monorepos
Intro
I work in a small developer team in a Golang-based private repo. With limited bandwidth, a monorepo seemed optimal to minimizing overhead and to spend more time writing code. Needing a solution for our growing codebase, transitioning to a multimodule monorepo proved effective.
This article will describe intricacies of a multimodule setup, and provide a full walkthrough initializing one. The walkthrough will be a side-by-side module structure, and contain cross-module imports and replace statements. For the full hosted example, see the github link below. Or the git gist at the bottom of this article.
Full Multimodule Walkthrough
Working Towards a Multimodule Monorepo
Less overhead came with working in a single repo: one CI/CD pipeline setup, simpler pull requests, etcetera. Eventually, our team will grow past this monorepo. Too many devs in the pot will produce a lot of merge conflicts. At that time, we’ll simply pull modules out to their own repo.
Directory Structure
This is the directory structure we ended up with. Your circumstances will dictate your own module structure, but here’s a few key takeaways.
$ tree repo/
repo/
├── client
│ ├── go.mod
│ ├── go.sum
│ └── main.go
├── core
│ ├── go.mod
│ └── go.sum
├── integration
│ ├── go.mod
│ └── go.sum
└── toolset
├── go.mod
├── go.sum
└── main.goYou’ll notice we have four modules, not just two for the binaries.
coreholds our generalized code, usable by any moduleclientandtoolsetare our binaries, and hold binary-specific codeintegrationholds our integration tests for all the modules
The core module is like a parent, and is only imported from. Both client and toolset import from core, and integration imports both client and toolset.
A perk to having a separate integration module allows us to test separate module parts, without having to import them into the other. An example would testing our client’s GET request with an auth token to our server, and verifying in the database that a user exists. This also kept our module import graph acyclic, which I’ll touch on soon.
Multimodule Monorepo Aspects
Modularized Code
Code is bundle in its own module. Like the name suggests, this is very modular and therefore rearranged easily. Utilizing go.mod file replace statements, you can easily choose local disk or an origin url.
Take this example of moduleclient importing module core in its go.mod file.
module myvcs.org/repo/clientgo 1.14require (
myvcs.org/repo/core v0.0.0
github.com/stretchr/testify v1.6.1
gopkg.in/yaml.v2 v2.2.8 // indirect
)replace myvcs.org/repo/core v0.0.0 => ../core
Notice how the replacestatement forcore points back one directory. This is in relative path to the client module, and only needs to adjusted slightly should core be moved.
Customizable Import Paths
You can customize golang imports with a thing known as vanity import paths. If you go the semver-free multimodule route, you can use a vanity import path without needing to setup the http meta tag for redirection.
package my-package2import (
// typical import
// github.com/my-organization/repo/module1/my-package1
// slightly shorter vanity import
mysite.org/repo/module1/my-package1
)
...
With a vanity import and replace statement combo, you can use any valid vanity import url to point to any local directory module. Just be sure to edit the importinggo.mod accordingly.
Personally, we found the shorter import path to be convenient. We also put in a vanity import path that we’ll use in the future.
Requires Acyclic Module Importing
You won’t be able to have your modules import each other in a cycle. Normally your code can bounce around a bit within a module, but splitting it into multiple modules walks a finer import line. If you do end up with a module import cycle, it may look like this:
import cycle not allowed
package github.com/repo/modA
imports github.com/repo/modB/pkgB
imports github.com/repo/modA/pkgA
imports github.com/repo/modB/pkgBCircumstantially Semantic Version Maintainable
Bear with me. If you have an established, slowly-changing multimodule repo, even with modules importing each other, using semver tags can be maintainable. Alter one module, commit, tag changes, push and merge. Update other modules accordingly.
However, if you are planning on a fast-paced monorepo, often involving changes across multiple modules that import each other, using semver could be quite a bit of overhead. The small example below demonstrates what a minor instance of that overhead could look like.
module A and module B are semver taggedmodule A imports module B module A needs changes that involves changes in module B
You alter, commit, tag, then push changes to origin module BNow you alter, commit, tag, and push changes to origin module A
This could be even more complicated, with multiple modules importing one another and requiring changes. And if you want to test changes locally, you’ll have to manually add/alter module replace statements, since your go.modfiles point to origin, not your local disk. Gross.
Excel with Semver-Free Modules
Instead of tagging each module, use replace statements to point to your other monorepo modules. Then, make cross-module changes and save with your favorite version control system. It’s that easy.
And, when the day comes to break out of one repo, move your modules out. Remove or change your existing replace statements to point to the new module origin urls. Don’t forget to update the module semver tag, too.
Setting it Up
Here’s a git gist of the README from the walkthrough repo. It will walk you through the entire setup process. For copy-pastable commands and code, check out the README.md in the repo, or this git gist’s raw. Happy coding!
