In a previous article we covered problems with maintaining multiple repositories and 2 ways how to solve it in ideal world. Do you maintain many interdependent repositories one by one and still think that it’s the best approach for your company? Go read it and prove us wrong.
Today we’ll look at a more realistic approach, pitfalls we encountered along the way, and how we fight till the end to conquer them and make 1 repository to rule them all. After all the hard work and tinkering it took reach our goal, I have to say we’re happy with the final results of our monorepo — both from a technical standpoint and from a developer’s experience way. Let’s dive into it.
Not Good Enough Solutions on the Monorepo Market
There are tools like splith used by Symfony, Tomono, Git Subsplit and many more.
And while each of them can handle splitting:
- some won’t keep git history,
- some keep partial git history, e.g empty merge commits,
- some keep git history, but with different hash, and most importantly…
…none can handle creating monorepo from already existing repositories if we don’t count copying all packages to one directory and git init
It took us 2 weeks, 4 carefully scanned READMEs, dozens lines of patch code when the tool didn’t work as expected, and hundreds lines of our own code to find this out. That is open-source expectations vs. reality in practice.
How to Create Monorepo Yourself from Existing Repositories?
This is the most frequent question when it comes to monorepo adoption in companies. And usually the last one because it’s rejected out of hand as requiring too much effort and resources.
Since we already had 15 repositories and a project-base with over 8000 commits in total, the option of dropping git history for all of them was off the table as well.
After struggling to find ways to resolve this problem, one of our programmers Petr Heinz devoted a considerable number of work-hours to coming up with a package: monorepo-tools.
How is Monorepo Tools Better?
Compared to some of the more costly solutions above, it can:
- Merge multiple existing repositories
- Merge them to subdirectories
- Keep their git history
- Keep hash of all commits
- In split repo keep merge commits with per-package files only
- And also do the split while keeping history, hash of commits, etc.
You can read how to use it in this short README. So, the next time consolidating repositories comes up among your team or CTO, there now is tool to create monorepo from existing repositories while still keeping the git history.
Grateful to be Built on The Shoulders of Open-Source Giants
Of course, we’re very grateful — and have to acknowledge — existing open-source projects that already make working with monorepo that much easier. They’ve helped pave the way for some of our own advances and understanding of monorepo architecture, and we’re happy that we could learn from them.
We’d like to also thank Symfony as a project that has helped monorepo approach spread through the PHP ecosystem, simply using it on daily basis. Thank you!
Why You Should(n’t) Keep The Project-Base and Framework in Monorepo
We like to take existing problems, talk about them for long time, examine them from all possible angles, and give them extra than most people would do. This might sound like we’re being pedantic, but in our experience consolidating the project-base and framework has been shown to save money and time. Some people say it’s form of pedantry. Our focus has always been on details, and on the customer’s (in this case developer’s) experience, because our customers — big e-commerce projects — are our priority and what matters the most.
So, back to what we do best! We’re happy to announce that we have perfected a tool which has allowed us to create a single monorepo, and which is able to keep the history and hashes intact. What now?
Well to be honest, we have 2 repositories. 1 monorepo and 1 project-base.
This has been a common way to complete a monorepo:
Or:
Or:
While this method may be the most common, we think the best way to be innovative is to question the status quo.
With 1 monorepo consolidating 15 repositories, we effectively have 15 fewer issues with maintenance. Now there’s only a single location to keep track of.
You might be asking: with 1 monorepo and 1 project-base, however, we have double the number of places to look after (checking PRs, issues etc.). But why? What would be the purpose of using monorepo, if we applied it only on few of repositories
Let’s imagine you change the main Shopsys Controller. How do you update the project-base as well?
- Use dev-master as dependencies? — That could cause a number of errors and fixing them would require switching between 2 projects
- Wait for a stable release? — Now your team has to wait on 2 projects to become stable
And what if somebody would like to send a PR with new feature? How will this influence their motivation to contribute if they have to create yet another PR for another repository in order to incorporate all the feedback from the first PR?
Why Are Frameworks And The Project-Base Separated?
Simply put: it’s easier to keep it that way (this is the status quo approach after all) and…
…we couldn’t think of any other good reasons, so we tried to add the project-base to monorepo.
How To Add The Project-Base To A Monorepo
This is how the standard monorepo structure looks like:
/packages
/CodingStandards
/Migrations
And this is the project-base:
/app
/config
Let the merge begin:
/packages
/CodingStandards
/Migrations
/project-base
/app
/conf ig
As you can imagine, it was not as easy as using a copy-paste operation. We needed to update our Docker setup, Symfony application setup, as well as other functions where a specific location was required with a condition:
if ($isMonorepo) {
return __DIR__ . ‘/../’
} else {
return __DIR__ ;
}
But after all this, even the project-base can be part of your full monorepo experience.
Developer Friendly Monorepo
Back to our initial use case: Let’s imagine you change the main Shopsys Controller. How do you update the project-base as well?
- In the same PR
We gave much care, patience, and attention to this case so contributors would have as smooth and easy an experience as possible. You’ll love it!
Do You Want To Go Monorepo, too?
If you’re totally new to this, there is a nice landing page at gomonorepo.org that can brief you on this topic in under 3 minutes. Want to try it out yourself on your own repositories? Just go to shopsys/monorepo-tools and start with the minimal setup: merging 2 repositories.
Stay tuned for more useful tips from our roadmap.
