“Spike-driven development” is a term I’m using for a sort of exploratory development with coding agents, in contrast to the popular (and useful) technique of “spec-driven development”. It’s most useful when I’m working with a new feature, technology, or technique that I’m not very familiar with. In such cases, it’s not very useful to try to develop a comprehensive spec and then hand off to a coding agent. The spec would be useless, because I don’t know what all decisions will come up or even how to best approach them. I’m at the entrance of a dark cave, with no map.
The main focus of “spike-driven development” is exploration. You aren’t going to spend a lot of time writing out long specs. It is more of a “design as you go” approach.
To start out, introduce the goal of your spike to your coding agent of choice. Try to keep the goal narrow and focused, MVP-style. Ask the agent to explore your existing codebase (if you have one) to anchor the new code into the existing patterns and examples.
The coding part is simple. Ask the agent to develop the spike end-to-end, from feature branch to merge request. You don’t want to look at anything until the merge request is ready for your review. Don’t worry about babysitting the agent. “Spike-driven development” is great for firing something off in the background and then switching to work on something that requires more focused attention.
The main focus of spike-driven development should not be the code. It should be the test-and-revise loop that you go through with your agent. Code is cheap. You can clean it up later before you merge to main.
Focus on testing the agent’s output. Does it match your initial expectations? Did the agent develop extras you don’t want? Prune those off early so you don’t get bogged down. Does the output make you think of design decisions you didn’t think of earlier? Good! That is the goal of spike-driven development: quickly iterating to find the important decision points. Discuss the decisions as much as you need with your agent, weighing the different options. Then, send it off to explore the path you chose. If it turns out you don’t like the results of a particular decision, don’t be afraid to backtrack and explore a different route!
Repeat this cycle several times until you are satisfied with the output. Don’t get distracted with including every little detail you ever wanted. Try to stay focused on the core “thing”, and then use follow-up sessions to revise and extend.
Only after you are done critiquing the output should you bother looking at the code.
Hopefully at this point you have a much clearer idea of the shape and size of the metaphorical cave than you did at the beginning. The goal was to find the critical decisions that really define the problem at hand. Now that you have a clearer picture, you have several options for how to proceed with the code that was generated throughout your exploration:
Throw the code away and start fresh. Code is cheap. Don’t feel attached to the code. Now that you have a better idea of the design decisions you want to make, you can sit down and write a clearer spec and proceed from there.
If the code changes are small, you can do a bit of cleanup and ship it. Oftentimes I am surprised by how little code it takes to implement a desired change.
If the code changes are large, split it up into a series of merge requests; there is no reason you should merge your original feature branch with thousands of lines of diffs into main. Work through a plan with your agent to split it up into a series of much smaller merge requests which you can review one by one. Your agent already is an expert with git, so you shouldn’t have to do any of the git wrangling yourself. You’ll probably learn a lot reviewing the diffs of each merge request.
Now that you have completed the initial spike, it should be much easier to keep going. Hopefully you have a much better idea of what you are working with, and it’s not too difficult to revise and extend it over follow-up sessions, maybe using a bit of incremental development.
It’s easy for software engineers to procrastinate. There’s some task that needs to be completed for a project, but it looks overwhelming, and you’ve got lots of other things to work on. So you put it off for a little while longer. When you finally get to it weeks (or months) later, you’ve entirely lost all context on what’s going on, and you lose even more time getting back up to speed.
It’s easy to lose context when juggling lots of projects. You prepare some great design for a project, then some higher priority projects come up, and you don’t come back for months. When you re-read your spec again, you begin to get a sinking feeling: “uggh, that’s going to take a lot of work”.
It’s also easy to get “writer’s block”. You have a great spec, you have all the context, but you have no idea where to start. So you put it off and jump over to another project, starting the whole cycle of procrastination over again.
There’s a common term in software engineering projects - the MVP. Not the most valuable player, but the minimum viable product. Kent Beck defines the MVP as “what I need to make in order to learn something valuable”. What is the bare minimum useful thing we can implement and send to customers for feedback? This is essential for quick iteration towards a high-quality product. You don’t want to implement a huge project and then realize you went the wrong way at the first fork in the road.
As software engineers, I think we can get a lot of value from taking the MVP concept but applying it to our day-to-day development. Remove the “product” and replace it with “change”. What is the minimum valuable change I can make to make some progress towards the project goals? Where can I incrementally move forward, even if it means I’m still far from “finished”? How do I avoid feeling like a failure when there’s still a long way to go?
Instead of getting bogged down with feelings of failure about not reaching the final goal yet, focus on the small wins you can make along the way. As you add up these small wins, you can build momentum, and suddenly you are rushing towards the goals at great speed! Imagine your development progress as a snowball rolling down a snow-covered mountain, growing and growing as it goes. Dave Ramsey calls his method for getting out of debt the “snowball method” for this very reason - it’s not about taking the mathematically most efficient route to the end goal, it’s about building momentum through small wins. Our brains don’t work as mathematical machines. We want to optimize our habits, strategies, and processes to match the psychology behind motivation.
Incremental development is also a great way to fight back against procrastination. It’s much easier to start on a two hour task than it is to start on a two week task. The process of breaking things down can be a great way to get the creative juices flowing again and push past “writer’s block”.
This concept doesn’t just apply to new projects, it also works well for existing projects. Whether it’s a new feature or a large redesign, the process of breaking the change into bite-size pieces will reap huge dividends, both in your motivation and productivity.
In the debate between “wait to act until you have clear direction” versus “act now, redo later if it turns out wrong”, I’m a big proponent of the latter strategy. I think you learn a lot more along the way by prioritizing action. The process of doing something “wrong” will help you gain more clarity on how to do it better.
It’s also important to remember “perfect is the enemy of good”. Often we can get blocked by the idea of “to do it right will take six months”, so we never get started at all. The customer doesn’t need a perfect solution that never gets shipped, they need the good solution you can ship quickly! Often times you might have to accrue a bit of technical debt to get started. Don’t freak out! It’s quite likely that your picture of the “ideal design” will morph several times as you work through some subpar solutions. It’s much more useful to have a hacky prototype that gets you moving than to get stuck on the perfect design that never happens. Embrace the process of prototype, feedback, iteration, repeat.
How do you eat an elephant? One bite at a time. It’s easy to feel overwhelmed as you consider a large project. Make it easier on yourself by breaking it down. Keep recursively breaking it down, until you have a bunch of pieces that won’t take more than two days each to implement. Then get going, rolling your snowball down the mountain!
Over the past year, our team here at MRS had been working on solutions to two seemingly unrelated problems.
First, some of our projects had grown from just one or two repositories in the beginning to sprawling groups of repos that posed a real challenge for keeping project admin and management best practices in sync. For example, when we began implementing pre-commit hooks as a standard part of projects, we had to copy the same code to each repo.
Sometimes we were able to resolve this with other means. For example, we had implemented an automatic issue triaging bot in many projects with some custom JS code that ran in CI. A few months ago, we took its core concepts and created our simple custom issue bot, which we then imported into each project. Now, the ROI for implementing new features for that bot is much higher because they are automatically applied to all projects instead of having to be ported from repo to repo.
However, many times these sorts of DevOps centralizations take a significant amount of development, and the structure of our team prevented us from dedicating lots of resources to such initiatives. Besides, many administrative tasks, such as configuring GitLab branch rules for each repo, cannot be efficiently centralized.
Second, we began to implement AI into more areas of our development workflow, from patching small bugs to designing and implementing large features to automating code review. This exposed an issue: AI agents frequently did not have all of the necessary context in one repo for implementing or reviewing code because other parts of the system were in different repos.
Additionally, as you doubtless have discovered, the world of AI is moving VERY quickly. Standard practices and tooling change practically every month. For example, when AGENTS.md started to become standard as a tool for giving project-level context to agents, we of course wanted to add one to each project we were actively developing. However, in practice, this meant creating and curating AGENTS.md in each of the many repos. The same thing happened with opencode skills and then standard agent skills: we wanted to add our custom-built skills to each project, but this required quite a few MRs.
Our solution to these two problems has been to migrate our polyrepo projects to monorepos.
Frankly, this has revolutionized our per-repo workflows. GitLab/GitHub admin is simpler. Internal innovations are easily applied to multiple projects. The code changes for a feature that affects more than one part of the project (e.g. new password regex rules made in the frontend and backend for redundancy) can be applied simultaneously, in one MR, instead of requiring developers to create two MRs.
Clearly there are benefits to the monorepo approach, but there are also some drawbacks. For example, consider how you might design the CI pipeline for a monorepo. If every project had its own CI suite when it was in its own repo, how would those be merged together? (Stay tuned for a deeper discussion on this in a later post…) If every project has its suite of common development commands, managed in a justfile, how will those work together in the monorepo?
Additionally, now that we have this monorepo, how will we implement such potential advantages as build caching? If App A depends on Lib A, and Lib A has already been compiled, we don’t want to recompile it every time we build App A. Speaking of which, what about dependency management? How will we define the dependency surface for App A? How will we keep track of that dependency graph once it’s been implemented?
Recently, we came upon a tool called moon that solves all of these new problems.
moon is built to help teams efficiently and effectively manage monorepos. There are other tools out there for this, such as nx and turborepo (which we would encourage you to try for yourself), but we chose moon because of its toolchain-agnostic nature and simplicity of setup. Where other tools are optimized for JS/TS-focused repos, moon is fully equipped to help with projects of all types. We are using it for Flutter mobile apps, embedded Qt/C++ apps and libraries, Go tools, and, yes, JS projects.
The impetus for adding moon to our monorepo workflow first came from a need to scale our per-project justfiles. For example, if each project has a deps recipe for installing dependencies, then how do we run those efficiently? While just does support invoking recipes in subdirectories, this doesn’t scale well. How do I quickly install dependencies for a bunch of projects? How do I install all dependencies for all projects without knowing the locations of every single one?
moon solves this problem by scoping tasks (their name for recipes) by project, the list of which is defined at the top of the monorepo.
Each project can define its own deps task…or you can define a common deps task for certain projects. If a monorepo has 6 Dart packages, all of which use the same set of static analysis tasks, you could define .moon/tasks/dart-lib.yml like this:
# These tasks are inherited by Dart libraries.
inheritedBy:
layer: library
language: dart
tasks:
deps:
command: dart pub get --enforce-lockfile
check:
script: |
dart fix "$projectRoot"
dart format --output json --set-exit-if-changed "$projectRoot"
fix:
script: |
dart fix --apply "$projectRoot"
dart format "$projectRoot"
This completely solves the problem of duplicating tasks between lots of repos; now, those tasks are defined in one place. You can run them locally, in pre-commit hooks, or in CI. Plus, moon automatically generates a project dependency graph and full list of tasks, so that you can see all the available tasks with one moon tasks command.
One of the benefits of moon that we weren’t looking for at first but soon found to be extremely useful is its automatic task caching. When you run a task, moon will cache the results and output locally, hashed by the state of the project at the time of the run. Then, if you do not change any relevant files before running the task again, the cached results will be used; if relevant files did change, the task will be fully run again.
The caching behavior of moon is extremely configurable; you can tell it exactly what file changes should trigger a rerun, what files should be treated as build artifacts, what other tasks should be treated as dependencies to force a rerun, and so on.
This caching behavior actually has a very nice application in CI environments. The cache is stored in .moon/cache, which means that you can configure your CI to store that directory in the runner cache and then use it in other jobs. At MRS, we had done this on a lesser level with techniques like NPM and Go caches, but moon provides another level by effectively caching entire jobs and their outputs.
Another great feature of moon is its ability to run tasks only when relevant files changed, as determined by the VCS. This concept is similar to how cache hits and misses are determined, but it has an added benefit when running a bunch of tasks at once.
By tracking which tasks have been “affected” by changed files, moon can avoid running a task at all if no input files have changed. You can run something like moon run :check --affected and it will run the check task for only those projects which have been modified.
This is extremely useful in monorepos for things like Git hooks and CI. For example, if every project has a check job defined, I can define a simple pre-commit hook that runs moon run :check to make sure all static checks run before commit. I can do the same thing in CI—but this isn’t efficient in the slightest, nor is it good practice. I want my CI runs to be scoped to the project I changed. If I am working a new feature for app1, then I don’t want or need to run the entire unit test suites (task test, for example) for app2 and lib1. So, I use moon run :check :test --affected in my CI pipeline, and presto! All the static checks and unit tests suites for the project(s) I’m working on get ran without any extra configuration.
In fact, this is such a core use case of moon that they have a dedicated moon ci command with extra options.
Monorepos are a great way to address the problems that come with having many repos for one project. They are especially well-suited for today’s increasingly AI-focused software development cycle.
However, monorepos pose unique challenges, and we have found moon to be the tool for resolving them. It does a great job of preserving the things we love about just in smaller repos while providing advanced features for effectively managing and developing in a monorepo.
Established infrastructure can often become a black box. You don’t know how it was set up, and there is no documentation. The person who set it up isn’t around to answer your questions or has forgotten the details because it’s been five years. How can you set up new infastructure with good documentation to go along with it?
Deployment scripts or infrastructure as code are a good way to aim for reproducible infrastructure1, but often you are setting up specific infrastructure that only needs to be stood up once. What you really need is a simple guide describing the steps.
Instead of asking your coding agent “How do I set up X?”, give a prompt of the form “Write a README explaining how to set up X”. Then follow the steps in the new documentation. If you run into something incorrect or confusing, revise the documentation and try again. In the end, you should have a guide that accurately describes how to do it, provable by the new infrastructure that you just set up using the guide.
Not only do you end up with a helpful setup guide for the new infrastructure, you also end up with accurate documentation for guiding yourself or other team members when you need to replicate or maintain the infrastructure in the future. It’s a great way to kill two birds with one stone.
AGENTS.md files are a great way to persist long-term “memory” for your coding agents. However, it’s easy for them to become bloated and/or outdated. A recent study has shown this is actually worse than having no AGENTS.md at all.
There is no need to list your tech stack, file structure, or list of just recipes in your AGENTS.md. This is redundant and can be harmful to your agents’ effectiveness when this information becomes outdated. Modern coding agents can quickly figure out the essential details of your codebase for themselves. If you put these details in your AGENTS.md file, you are wasting context space and putting yourself at a high risk of documentation rot - when you update your tech stack or file structure, will you remember to also update AGENTS.md?
In our experience, the best usage of AGENTS.md is for recording guidelines that help your coding agents stay on the right path. Think of it as building some guardrails for your agents. When your agents make mistakes, be sure to update the guidelines to help point them in the right direction for future work.
Sometimes you need to record facts that don’t fit well into the three-tier guidelines system. For these, it works well to add a Long Term Memory section to your AGENTS.md. Be careful to prune this list frequently. Often your agent will want to add items here that provide no value for future situations.
It’s easy for your AGENTS.md to become polluted with information related to specific workflows or tasks (for example, running automated tests or accessing GitLab issues). Agent Skills are a much better fit for this. Whenever possible, move non-essential information from AGENTS.md to your skill files.
In general, be very strict about what is in your AGENTS.md file. It takes up valuable context space and biases every coding agent session. Be sure that your guidelines provide positive benefits with useful guardrails rather than redundant or outdated documentation.
