Skip to content

Bennett Moore

2 posts by Bennett Moore

To the Moon

Over the past year, our team here at MRS had been working on solutions to two seemingly unrelated problems.

First, some of our projects had grown from just one or two repositories in the beginning to sprawling groups of repos that posed a real challenge for keeping project admin and management best practices in sync. For example, when we began implementing pre-commit hooks as a standard part of projects, we had to copy the same code to each repo.

Sometimes we were able to resolve this with other means. For example, we had implemented an automatic issue triaging bot in many projects with some custom JS code that ran in CI. A few months ago, we took its core concepts and created our simple custom issue bot, which we then imported into each project. Now, the ROI for implementing new features for that bot is much higher because they are automatically applied to all projects instead of having to be ported from repo to repo.

However, many times these sorts of DevOps centralizations take a significant amount of development, and the structure of our team prevented us from dedicating lots of resources to such initiatives. Besides, many administrative tasks, such as configuring GitLab branch rules for each repo, cannot be efficiently centralized.

Second, we began to implement AI into more areas of our development workflow, from patching small bugs to designing and implementing large features to automating code review. This exposed an issue: AI agents frequently did not have all of the necessary context in one repo for implementing or reviewing code because other parts of the system were in different repos.

Additionally, as you doubtless have discovered, the world of AI is moving VERY quickly. Standard practices and tooling change practically every month. For example, when AGENTS.md started to become standard as a tool for giving project-level context to agents, we of course wanted to add one to each project we were actively developing. However, in practice, this meant creating and curating AGENTS.md in each of the many repos. The same thing happened with opencode skills and then standard agent skills: we wanted to add our custom-built skills to each project, but this required quite a few MRs.

Our solution to these two problems has been to migrate our polyrepo projects to monorepos.

Frankly, this has revolutionized our per-repo workflows. GitLab/GitHub admin is simpler. Internal innovations are easily applied to multiple projects. The code changes for a feature that affects more than one part of the project (e.g. new password regex rules made in the frontend and backend for redundancy) can be applied simultaneously, in one MR, instead of requiring developers to create two MRs.

Clearly there are benefits to the monorepo approach, but there are also some drawbacks. For example, consider how you might design the CI pipeline for a monorepo. If every project had its own CI suite when it was in its own repo, how would those be merged together? (Stay tuned for a deeper discussion on this in a later post…) If every project has its suite of common development commands, managed in a justfile, how will those work together in the monorepo?

Additionally, now that we have this monorepo, how will we implement such potential advantages as build caching? If App A depends on Lib A, and Lib A has already been compiled, we don’t want to recompile it every time we build App A. Speaking of which, what about dependency management? How will we define the dependency surface for App A? How will we keep track of that dependency graph once it’s been implemented?

Recently, we came upon a tool called moon that solves all of these new problems.

moon is built to help teams efficiently and effectively manage monorepos. There are other tools out there for this, such as nx and turborepo (which we would encourage you to try for yourself), but we chose moon because of its toolchain-agnostic nature and simplicity of setup. Where other tools are optimized for JS/TS-focused repos, moon is fully equipped to help with projects of all types. We are using it for Flutter mobile apps, embedded Qt/C++ apps and libraries, Go tools, and, yes, JS projects.

The impetus for adding moon to our monorepo workflow first came from a need to scale our per-project justfiles. For example, if each project has a deps recipe for installing dependencies, then how do we run those efficiently? While just does support invoking recipes in subdirectories, this doesn’t scale well. How do I quickly install dependencies for a bunch of projects? How do I install all dependencies for all projects without knowing the locations of every single one?

moon solves this problem by scoping tasks (their name for recipes) by project, the list of which is defined at the top of the monorepo.

You go from this:

Terminal window
just app1/deps app2/deps libs/lib1/deps libs/lib3/deps tools/deps

To this:

Terminal window
moon run :deps

Each project can define its own deps task…or you can define a common deps task for certain projects. If a monorepo has 6 Dart packages, all of which use the same set of static analysis tasks, you could define .moon/tasks/dart-lib.yml like this:

# These tasks are inherited by Dart libraries.
inheritedBy:
layer: library
language: dart
tasks:
deps:
command: dart pub get --enforce-lockfile
check:
script: |
dart fix "$projectRoot"
dart format --output json --set-exit-if-changed "$projectRoot"
fix:
script: |
dart fix --apply "$projectRoot"
dart format "$projectRoot"

This completely solves the problem of duplicating tasks between lots of repos; now, those tasks are defined in one place. You can run them locally, in pre-commit hooks, or in CI. Plus, moon automatically generates a project dependency graph and full list of tasks, so that you can see all the available tasks with one moon tasks command.

For more references:

One of the benefits of moon that we weren’t looking for at first but soon found to be extremely useful is its automatic task caching. When you run a task, moon will cache the results and output locally, hashed by the state of the project at the time of the run. Then, if you do not change any relevant files before running the task again, the cached results will be used; if relevant files did change, the task will be fully run again.

The caching behavior of moon is extremely configurable; you can tell it exactly what file changes should trigger a rerun, what files should be treated as build artifacts, what other tasks should be treated as dependencies to force a rerun, and so on.

This caching behavior actually has a very nice application in CI environments. The cache is stored in .moon/cache, which means that you can configure your CI to store that directory in the runner cache and then use it in other jobs. At MRS, we had done this on a lesser level with techniques like NPM and Go caches, but moon provides another level by effectively caching entire jobs and their outputs.

For more information:

Another great feature of moon is its ability to run tasks only when relevant files changed, as determined by the VCS. This concept is similar to how cache hits and misses are determined, but it has an added benefit when running a bunch of tasks at once.

By tracking which tasks have been “affected” by changed files, moon can avoid running a task at all if no input files have changed. You can run something like moon run :check --affected and it will run the check task for only those projects which have been modified.

This is extremely useful in monorepos for things like Git hooks and CI. For example, if every project has a check job defined, I can define a simple pre-commit hook that runs moon run :check to make sure all static checks run before commit. I can do the same thing in CI—but this isn’t efficient in the slightest, nor is it good practice. I want my CI runs to be scoped to the project I changed. If I am working a new feature for app1, then I don’t want or need to run the entire unit test suites (task test, for example) for app2 and lib1. So, I use moon run :check :test --affected in my CI pipeline, and presto! All the static checks and unit tests suites for the project(s) I’m working on get ran without any extra configuration.

In fact, this is such a core use case of moon that they have a dedicated moon ci command with extra options.

For more information:

Monorepos are a great way to address the problems that come with having many repos for one project. They are especially well-suited for today’s increasingly AI-focused software development cycle.

However, monorepos pose unique challenges, and we have found moon to be the tool for resolving them. It does a great job of preserving the things we love about just in smaller repos while providing advanced features for effectively managing and developing in a monorepo.

TO THE MOON!! 🌑 🌘 🌗 🌖 🌕

Should I Really Use a Gitlab Wiki?

Recently, as we’ve been taking on new software projects and continuing to develop existing ones, the role of documentation has come into focus.

In fact, the development of this very site is a direct result of an increased focus by our whole software team to better document the things we learn and already know.

But…WHERE should all this information go? In the past, we always defaulted to Gitlab wikis…but is that the best option? Would an dedicated documentation repository for each project make more sense?

I will note that if your project uses a monorepo, I will think of “dedicated repo” as meaning “dedicated folder inside the repo”; something like apps/docs will work just fine and function the same as a separate repo in a multi-repo project structure.

The Gitlab wiki feature is really a very powerful one. Every group and repository is automatically given a wiki (unless it’s disabled in the project settings), so there’s no additional setup required.

Wikis are designed to be very quick and easy to contribute to. Internally, they are based on a Git repo, but you don’t have to clone the repo or worry about local copies or anything; Gitlab allows you to make and commit all your edits right from the UI and then does the Git operations behind the scenes.

Wikis support easy file uploads for images, PDFs, and whatever else in much the same way that other features do (like uploading files to an issue or MR description). You can attach the file and Gitlab automatically stores it for you with no extra configuration needed.

Gitlab wikis also have some negatives that come with them.

First, these automatic wikis are nice, but they can quickly result in splintered, spread out documentation. It’s very common for one “project” to consist of a group of multiple projects. If each one has its own wiki, including the top-level group, then which one should the project’s documentation go in?

This quickly becomes a problem when you have multiple developers adding things to different wikis. Not only is it likely that some of the more general information will be duplicated, which means unnecessary work, but it also becomes hard to know which wiki is the central source of truth. If two wikis have conflicting information, which version of the information should you choose as being true? You’ll have to ask someone else to confirm, which only serves to take more time for both of you.

Having multiple wikis requires you to remember which one has which pieces of information, which can quickly become a headache when trying to point other team members to something.

The only real way to avoid this problem is to establish from the start which wiki will be the central source of truth and disable the other wikis entirely.

Another disadvantage is a lack of peer review. Gitlab wikis are easy to contribute to…but that can also be a negative, because no outside approval is required to make changes. Anyone on the team can write down whatever they want and the only way for erroneous documentation to be discovered is if another team member stumbles across it and recognizes the error.

Gitlab wikis are also just that—Gitlab wikis. They’re tied to Gitlab. If your project is ever migrated to another code-hosting or issue-tracking site, such as Jira/Bitbucket or Github, then you’ll have to do some work to migrate the wiki with it. It’s fairly easy to clone the wiki’s internal repo and push that to the new site, but you’ll have to go through and update all the Gitlab-specific parts: for example, those lovely file uploads that I mentioned earlier? You’ll have to download each file, change the link that points to it, and figure out a simple way of storing those files in the repo.

There are a lot of advantages to using a dedicated separate repository for documentation. One of the biggest ones is peer review. By creating a separate repository (and setting up the typical workflow, with MRs, protected branches, required approvals, and the like), you force all new documentation to be reviewed by another member of the team.

There are a multitude of reasons why this is a good thing that boil down to the reasons why any peer review on code is a good thing. Reviewers can test any deployment/setup steps that are documented to make sure they work correctly; they can filter information that isn’t really worth documenting; they can suggest spots where more clarity or explanation might be needed; and so on. Code review is core to the functioning of any well-managed project, and documentation must be held to the same rigorous standard.

Another big, closely related advantage is real version control. The Gitlab wiki does use an internal repo, but when editing through the UI, it’s as if every change you make is a push straight to main (which, as any good developer knows, is NOT a good practice). I’ve seen multiple occasions where someone was trying to edit a wiki page at the same time as someone else and their changes conflicted.

With a dedicated repo, this abstraction of easy changes is removed, and that’s a good thing. Team members can check out their own branches to modify content and make sure all their edits are complete and concise. If someone is working on a big new feature, they can make a separate branch of the documentation where they document their new feature as they implement it. While this does require maintaining multiple MRs, we find that to be a small tradeoff for the benefits it brings.

Using a separate repo also simplifies the task of deployment to an external location. For example, if you want to use Starlight (hint: that’s what we used for this site!) to create an actual website for the project documentation, then it’s easy to do so. You can set up the Starlight project in the repository, configure the CI/CD deployments, and so on.

This becomes very important when you want customers to be able to see some or all of the documentation. Gitlab wikis have the same visibility as the project they are a part of; if the project is private (as the majority of ours are), then the wiki is also private. You can’t link to the wiki when talking to customers unless they are made a member of the project, which is its own can of worms…suffice to say it is a can of worms that SHOULD NOT be opened. So, for customers to get access to useful information, you’ll have to have some way of deploying it outside of the project.

Having a separate repo also gives you much more structural freedom. You can define the hierarchy of files however you want, use whatever file formats/types you want, and so forth. For example, in one of my projects, the documentation repo contains both an external site and an internal “wiki”. The external site is for things that we want the customer to be able to access, and the internal wiki is for more development-centric things: running tests, working with hardware, meeting notes, and the like. Having “docs” for external documentation and a “wiki” for internal documentation is a pattern that has served our team well across multiple projects.

One of the only real “cons” to a dedicated repo is that it takes slightly longer to make changes as compared to a Gitlab wiki, but this is only a con if you don’t care about review. We care about making sure documentation is correct and complete and thus much appreciate the slightly slower turnaround time for pushing new information to customers.

It can also be tedious to have to manage MRs in multiple repos; if you implement a new feature in the actual project repo, and then provide documentation for said feature in the docs repo, you now have two MRs to complete one feature.

However, this isn’t a big issue…wikis effectively require the same thing because you are still updating the documentation in a spot that is not the project repo. Plus, if your project uses a monorepo, then the problem is avoided entirely and in fact becomes even more streamlined because the documentation changes and feature changes can be reviewed in a single MR.

We have found using a dedicated separate repository for documentation to be far preferable to using a Gitlab wiki in the projects here at MRS. While this doesn’t mean that it’s the unquestioned best solution for every project, the majority of projects will likely benefit from this structure.

A dedicated repo is much more flexible in terms of deployment and structure, allows for real peer review, and avoids problems with splintered docs in more than one place and over-reliance on Gitlab. If you’re going to be creating a new project anytime soon, we recommend creating your documentation repo right from the start.