Skip to content

Blog

Incremental Development

It’s easy for software engineers to procrastinate. There’s some task that needs to be completed for a project, but it looks overwhelming, and you’ve got lots of other things to work on. So you put it off for a little while longer. When you finally get to it weeks (or months) later, you’ve entirely lost all context on what’s going on, and you lose even more time getting back up to speed.

It’s easy to lose context when juggling lots of projects. You prepare some great design for a project, then some higher priority projects come up, and you don’t come back for months. When you re-read your spec again, you begin to get a sinking feeling: “uggh, that’s going to take a lot of work”.

It’s also easy to get “writer’s block”. You have a great spec, you have all the context, but you have no idea where to start. So you put it off and jump over to another project, starting the whole cycle of procrastination over again.

There’s a common term in software engineering projects - the MVP. Not the most valuable player, but the minimum viable product. Kent Beck defines the MVP as “what I need to make in order to learn something valuable”. What is the bare minimum useful thing we can implement and send to customers for feedback? This is essential for quick iteration towards a high-quality product. You don’t want to implement a huge project and then realize you went the wrong way at the first fork in the road.

As software engineers, I think we can get a lot of value from taking the MVP concept but applying it to our day-to-day development. Remove the “product” and replace it with “change”. What is the minimum valuable change I can make to make some progress towards the project goals? Where can I incrementally move forward, even if it means I’m still far from “finished”? How do I avoid feeling like a failure when there’s still a long way to go?

Instead of getting bogged down with feelings of failure about not reaching the final goal yet, focus on the small wins you can make along the way. As you add up these small wins, you can build momentum, and suddenly you are rushing towards the goals at great speed! Imagine your development progress as a snowball rolling down a snow-covered mountain, growing and growing as it goes. Dave Ramsey calls his method for getting out of debt the “snowball method” for this very reason - it’s not about taking the mathematically most efficient route to the end goal, it’s about building momentum through small wins. Our brains don’t work as mathematical machines. We want to optimize our habits, strategies, and processes to match the psychology behind motivation.

Incremental development is also a great way to fight back against procrastination. It’s much easier to start on a two hour task than it is to start on a two week task. The process of breaking things down can be a great way to get the creative juices flowing again and push past “writer’s block”.

This concept doesn’t just apply to new projects, it also works well for existing projects. Whether it’s a new feature or a large redesign, the process of breaking the change into bite-size pieces will reap huge dividends, both in your motivation and productivity.

In the debate between “wait to act until you have clear direction” versus “act now, redo later if it turns out wrong”, I’m a big proponent of the latter strategy. I think you learn a lot more along the way by prioritizing action. The process of doing something “wrong” will help you gain more clarity on how to do it better.

It’s also important to remember “perfect is the enemy of good”. Often we can get blocked by the idea of “to do it right will take six months”, so we never get started at all. The customer doesn’t need a perfect solution that never gets shipped, they need the good solution you can ship quickly! Often times you might have to accrue a bit of technical debt to get started. Don’t freak out! It’s quite likely that your picture of the “ideal design” will morph several times as you work through some subpar solutions. It’s much more useful to have a hacky prototype that gets you moving than to get stuck on the perfect design that never happens. Embrace the process of prototype, feedback, iteration, repeat.

How do you eat an elephant? One bite at a time. It’s easy to feel overwhelmed as you consider a large project. Make it easier on yourself by breaking it down. Keep recursively breaking it down, until you have a bunch of pieces that won’t take more than two days each to implement. Then get going, rolling your snowball down the mountain!

To the Moon

Over the past year, our team here at MRS had been working on solutions to two seemingly unrelated problems.

First, some of our projects had grown from just one or two repositories in the beginning to sprawling groups of repos that posed a real challenge for keeping project admin and management best practices in sync. For example, when we began implementing pre-commit hooks as a standard part of projects, we had to copy the same code to each repo.

Sometimes we were able to resolve this with other means. For example, we had implemented an automatic issue triaging bot in many projects with some custom JS code that ran in CI. A few months ago, we took its core concepts and created our simple custom issue bot, which we then imported into each project. Now, the ROI for implementing new features for that bot is much higher because they are automatically applied to all projects instead of having to be ported from repo to repo.

However, many times these sorts of DevOps centralizations take a significant amount of development, and the structure of our team prevented us from dedicating lots of resources to such initiatives. Besides, many administrative tasks, such as configuring GitLab branch rules for each repo, cannot be efficiently centralized.

Second, we began to implement AI into more areas of our development workflow, from patching small bugs to designing and implementing large features to automating code review. This exposed an issue: AI agents frequently did not have all of the necessary context in one repo for implementing or reviewing code because other parts of the system were in different repos.

Additionally, as you doubtless have discovered, the world of AI is moving VERY quickly. Standard practices and tooling change practically every month. For example, when AGENTS.md started to become standard as a tool for giving project-level context to agents, we of course wanted to add one to each project we were actively developing. However, in practice, this meant creating and curating AGENTS.md in each of the many repos. The same thing happened with opencode skills and then standard agent skills: we wanted to add our custom-built skills to each project, but this required quite a few MRs.

Our solution to these two problems has been to migrate our polyrepo projects to monorepos.

Frankly, this has revolutionized our per-repo workflows. GitLab/GitHub admin is simpler. Internal innovations are easily applied to multiple projects. The code changes for a feature that affects more than one part of the project (e.g. new password regex rules made in the frontend and backend for redundancy) can be applied simultaneously, in one MR, instead of requiring developers to create two MRs.

Clearly there are benefits to the monorepo approach, but there are also some drawbacks. For example, consider how you might design the CI pipeline for a monorepo. If every project had its own CI suite when it was in its own repo, how would those be merged together? (Stay tuned for a deeper discussion on this in a later post…) If every project has its suite of common development commands, managed in a justfile, how will those work together in the monorepo?

Additionally, now that we have this monorepo, how will we implement such potential advantages as build caching? If App A depends on Lib A, and Lib A has already been compiled, we don’t want to recompile it every time we build App A. Speaking of which, what about dependency management? How will we define the dependency surface for App A? How will we keep track of that dependency graph once it’s been implemented?

Recently, we came upon a tool called moon that solves all of these new problems.

moon is built to help teams efficiently and effectively manage monorepos. There are other tools out there for this, such as nx and turborepo (which we would encourage you to try for yourself), but we chose moon because of its toolchain-agnostic nature and simplicity of setup. Where other tools are optimized for JS/TS-focused repos, moon is fully equipped to help with projects of all types. We are using it for Flutter mobile apps, embedded Qt/C++ apps and libraries, Go tools, and, yes, JS projects.

The impetus for adding moon to our monorepo workflow first came from a need to scale our per-project justfiles. For example, if each project has a deps recipe for installing dependencies, then how do we run those efficiently? While just does support invoking recipes in subdirectories, this doesn’t scale well. How do I quickly install dependencies for a bunch of projects? How do I install all dependencies for all projects without knowing the locations of every single one?

moon solves this problem by scoping tasks (their name for recipes) by project, the list of which is defined at the top of the monorepo.

You go from this:

Terminal window
just app1/deps app2/deps libs/lib1/deps libs/lib3/deps tools/deps

To this:

Terminal window
moon run :deps

Each project can define its own deps task…or you can define a common deps task for certain projects. If a monorepo has 6 Dart packages, all of which use the same set of static analysis tasks, you could define .moon/tasks/dart-lib.yml like this:

# These tasks are inherited by Dart libraries.
inheritedBy:
layer: library
language: dart
tasks:
deps:
command: dart pub get --enforce-lockfile
check:
script: |
dart fix "$projectRoot"
dart format --output json --set-exit-if-changed "$projectRoot"
fix:
script: |
dart fix --apply "$projectRoot"
dart format "$projectRoot"

This completely solves the problem of duplicating tasks between lots of repos; now, those tasks are defined in one place. You can run them locally, in pre-commit hooks, or in CI. Plus, moon automatically generates a project dependency graph and full list of tasks, so that you can see all the available tasks with one moon tasks command.

For more references:

One of the benefits of moon that we weren’t looking for at first but soon found to be extremely useful is its automatic task caching. When you run a task, moon will cache the results and output locally, hashed by the state of the project at the time of the run. Then, if you do not change any relevant files before running the task again, the cached results will be used; if relevant files did change, the task will be fully run again.

The caching behavior of moon is extremely configurable; you can tell it exactly what file changes should trigger a rerun, what files should be treated as build artifacts, what other tasks should be treated as dependencies to force a rerun, and so on.

This caching behavior actually has a very nice application in CI environments. The cache is stored in .moon/cache, which means that you can configure your CI to store that directory in the runner cache and then use it in other jobs. At MRS, we had done this on a lesser level with techniques like NPM and Go caches, but moon provides another level by effectively caching entire jobs and their outputs.

For more information:

Another great feature of moon is its ability to run tasks only when relevant files changed, as determined by the VCS. This concept is similar to how cache hits and misses are determined, but it has an added benefit when running a bunch of tasks at once.

By tracking which tasks have been “affected” by changed files, moon can avoid running a task at all if no input files have changed. You can run something like moon run :check --affected and it will run the check task for only those projects which have been modified.

This is extremely useful in monorepos for things like Git hooks and CI. For example, if every project has a check job defined, I can define a simple pre-commit hook that runs moon run :check to make sure all static checks run before commit. I can do the same thing in CI—but this isn’t efficient in the slightest, nor is it good practice. I want my CI runs to be scoped to the project I changed. If I am working a new feature for app1, then I don’t want or need to run the entire unit test suites (task test, for example) for app2 and lib1. So, I use moon run :check :test --affected in my CI pipeline, and presto! All the static checks and unit tests suites for the project(s) I’m working on get ran without any extra configuration.

In fact, this is such a core use case of moon that they have a dedicated moon ci command with extra options.

For more information:

Monorepos are a great way to address the problems that come with having many repos for one project. They are especially well-suited for today’s increasingly AI-focused software development cycle.

However, monorepos pose unique challenges, and we have found moon to be the tool for resolving them. It does a great job of preserving the things we love about just in smaller repos while providing advanced features for effectively managing and developing in a monorepo.

TO THE MOON!! 🌑 🌘 🌗 🌖 🌕

Docs-Driven Infrastructure

Established infrastructure can often become a black box. You don’t know how it was set up, and there is no documentation. The person who set it up isn’t around to answer your questions or has forgotten the details because it’s been five years. How can you set up new infastructure with good documentation to go along with it?

Deployment scripts or infrastructure as code are a good way to aim for reproducible infrastructure1, but often you are setting up specific infrastructure that only needs to be stood up once. What you really need is a simple guide describing the steps.

Instead of asking your coding agent “How do I set up X?”, give a prompt of the form “Write a README explaining how to set up X”. Then follow the steps in the new documentation. If you run into something incorrect or confusing, revise the documentation and try again. In the end, you should have a guide that accurately describes how to do it, provable by the new infrastructure that you just set up using the guide.

Not only do you end up with a helpful setup guide for the new infrastructure, you also end up with accurate documentation for guiding yourself or other team members when you need to replicate or maintain the infrastructure in the future. It’s a great way to kill two birds with one stone.

  1. Personally, I’m a big fan of Nix flakes.

Guiding Your Agents

AGENTS.md files are a great way to persist long-term “memory” for your coding agents. However, it’s easy for them to become bloated and/or outdated. A recent study has shown this is actually worse than having no AGENTS.md at all.

There is no need to list your tech stack, file structure, or list of just recipes in your AGENTS.md. This is redundant and can be harmful to your agents’ effectiveness when this information becomes outdated. Modern coding agents can quickly figure out the essential details of your codebase for themselves. If you put these details in your AGENTS.md file, you are wasting context space and putting yourself at a high risk of documentation rot - when you update your tech stack or file structure, will you remember to also update AGENTS.md?

In our experience, the best usage of AGENTS.md is for recording guidelines that help your coding agents stay on the right path. Think of it as building some guardrails for your agents. When your agents make mistakes, be sure to update the guidelines to help point them in the right direction for future work.

We’ve found that the following three-tier system works well for organizing agent guidelines:

  • Always Do (no asking)
    • A list of things the agent is allowed to do (and must do) without asking.
  • Ask First (pause for approval)
    • A list of things the agent should pause and ask for permission for.
  • Never Do (hard stop)
    • A list of things the agent should never do.

Sometimes you need to record facts that don’t fit well into the three-tier guidelines system. For these, it works well to add a Long Term Memory section to your AGENTS.md. Be careful to prune this list frequently. Often your agent will want to add items here that provide no value for future situations.

It’s easy for your AGENTS.md to become polluted with information related to specific workflows or tasks (for example, running automated tests or accessing GitLab issues). Agent Skills are a much better fit for this. Whenever possible, move non-essential information from AGENTS.md to your skill files.

In general, be very strict about what is in your AGENTS.md file. It takes up valuable context space and biases every coding agent session. Be sure that your guidelines provide positive benefits with useful guardrails rather than redundant or outdated documentation.

Just Is Just Great

We work on software projects across a variety of programming languages and build systems. Some of them provide an easy way to run development commands (for example, npm run), and others do not.

You need some way to answer the questions “How do I set this up?” and “How do I run this?”. You need some form of documentation of common commands.

It’s easy to write this documentation once and then forget to update it as your project grows.

We needed a system that we can use on any project, no matter the language or build system. Something that would keep the development commands and related documentation in sync.

just is just great because it solves several problems at once while keeping things simple. You only need one simple file and one simple command. just is a command runner that reads all your common project commands (called recipes) from a single file. You can also add explanation comments above each recipe and just will automatically include them in the help output.

For any of our active projects, a new developer can run just to get a list of the available recipes. Our core set of recipes includes:

  • default - to list recipes
  • deps - to install dependencies
  • setup - to run all required commands to set up a local development environment

just automatically runs the first recipe of your file if you run just without any arguments. We take advantage of this by always putting the default recipe at the top, and have it list the available recipes with just --list.

Many of our projects also have a test recipe for running the automated test suite. You’ll also often see at least one of dev, up, run-mobile, or run-docs for running the given project in the local development environment. We use lint and format recipes to trigger the correct linter and formatter for the project’s programming language.

Here is a minimal example that demonstrates all of our core recipes, along with a dev recipe. This is actually the current justfile from our docs-template repository.

# List available recipes
default:
@just --list
# Install dependencies
deps:
npm i
# Set up development environment
setup: deps
pre-commit install
# Run in development environment
dev:
npm run dev

The comments above each recipe are automatically included in the output of just --list:

Terminal window
Available recipes:
default # List available recipes
deps # Install dependencies
dev # Run in development environment
setup # Set up development environment

Why we’re adding a justfile to each of our active projects

Section titled “Why we’re adding a justfile to each of our active projects”

Having a justfile in every active project means that new developers can get started quickly in a project they’ve never worked in before. The “muscle memory” of just to list available recipes and just setup to set up the local environment makes things very convenient.

We can onboard developers to projects quicker, and we can spend more time focused on writing code rather than internal developer documentation.