Angular Monorepo pains

and how to overcome them

Image borrowed from https://codefresh.io/continuous-integration/using-codefresh-with-mono-repos/ (the best one I found to graphically explain monorepos)

A few months ago, our CTO came back from his vacations and told us he assisted to NG Conf 2019. A great experience that lead him to this conclusion:

"We should develop as Google, using a monorepo"

(curiously this is very similar to NWRL.io's slogan).

At first, as a senior developer, this didn't make too much sense. Our company runs several projects, each one living on their own git repository, each with its own folder structure, dependencies, and deploy toolchains. Of course, after a while, after reading lots of articles, I still didn't found too much sense in changing all our logic to monorepo, but it finally DID make sense.

By the way, what's a monorepo?

The image I borrowed from Codefresh should be clear enough.

A typical old-styled project is usually a monolyth, a huge project that kind-of solves what the organization requires. The sad part is that a monolyth usually lacks of good developing practices, is highly coupled, and is almost impossible to scale.

Over-simplifying reality, when the organization acknowledges its' own flaws it usually tries to migrate to a better approach; this is separating what it can separate from the monolyth and working on several smaller projects that somehow solve smaller requirements. Less coupling , but lots of work integrating each new layer. A visible improvement but for organizations that come from a monolythic universe, integrations will certainly be a PITA.

If every project in the organization , or most projects share certain components between them (both frontend components, or backend libraries), and after looking at the source code you figure out there're too many duplicate modules, components, or logic between all projects, and many projects interact with other projects at the same level, a monorepo makes sense.

A monorepo is a single version controlled repository, that contains many separate projects as if they where a single one. The idea is unifying shared code, avoid duplicate logic, promote team colaboration and other related magic.

Seems good, right? a win-win scenario for the whole organization.
Ehmmmm…, OK, good, probably not that good, but good. As the referred article states:

The frank reality is that, at scale, how well an organization does with code sharing, collaboration, tight coupling, etc. is a direct result of engineering culture and leadership, and has nothing to do with whether a monorepo or a polyrepo is used.

Adopting the monorepo strategy doesn't come without pain.

How to adopt the monorepo strategy?

There are some tools that are supposed to simplify this process. Hackernoon names a few:

• Bit
• Lerna
• Git submodules

As in most cases, the solution that may match your own use case is highly opinionated, so my advice is "You choose your own poison". There's no silver bullet for monorepos, so I will tell you about our case.

How did WE adopt the monorepo strategy?

First I have to say we didn't use any of Hackernoon's suggestions. Our company took NX.dev's article on Monorepos and Automation as a starting point.

Most of our projects are Angular frontend projects with a NodeJS backend, and had this structure:

A frontend folder, with its' own package.json , a backend folder (the frontend's backend api) also with its' own package.json and a toolchain folder (not in the image) with some deploying scripts, usually one script per-environment.

So our first goal was taking this structure and convert it to a monorepo structure, the monorepo structure suggested by NX.dev . This will lead to a single package.json and a single node_module folder.

So the pain begins…

Monorepo pains

Pain 0. Embrace your monorepo tool

This will happen regardless what tool you choose to migrate your projects. You'll have to learn the cli-tool you choose, all the options (or hopefully most of them), all the parameters. As I mentioned before we chose Nrwl.io's nx tool as our poison.

Learning every command variation is not really that hard, you can easily learn them while creating your own monorepo project structure, and in our case, by following the Getting started guide and tutorial for Nx.

A command-line step by step:

# Create your monorepo
npm init nx-workspace monorepo-project
# Go to your new project folder
cd monorepo-project
# Install the schematics for each sub-projects
npm install --save @nrwl/angular
npm install --save @nrwl/express
npm install --save @nrwl/node
# Create the container folders for every sub-project to be included
# more on this later

Pain 1. Including single repo projects in your monorepo

Your organization probably comes from the single repo universe and all your projects, regardless they share or not any code, components or configurations live in a single repo. In our case we use (still do 'cause there's an app that doesn't fit on the monorepo) private Bitbuckets repos, and cloning each of them would be something like:

git clone git@bitbucket.org:your_organization/your_single_repo_project.git

Change the above line for monorepo and it will also work, and it will clone the whole giant project with all you subprojects inside. Take in mind that this WILL happen in the future, and everytime you need to pull the last changes in the code, you WILL pull out the whole monorepo.

The first thing to do is preparing your monorepo project structure so it includes all the single repos meant to be absorbed by the monorepo:

# Create the container folders for every sub-project to be included
ng g @nrwl/angular:application frontend/store
ng g @nrwl/angular:application frontend/store-admin
ng g @nrwl/express:application store --frontendProject=frontend-store
ng g @nrwl/express:application store-admin --frontendProject=frontend-store-admin
ng g @nrwl/angular:lib ui-components
ng g @nrwl/node:lib server-modules
ng g @nrwl/node:lib config

In the example above we are generating the following structure:

• Frontend:
  - store
  - store-admin
• Backend:
  - store
  - store-admin
• Libs:
  - ui-components
  - server-modules
  - config

where store and store-admin are different projects that live under their own single repo, and each one of them has their own particular backend. At first you'll probably going to figure out that there's a lot in common between both projects. Perhaps some UI components, perhaps some configuration; so one of the ideas behind the monorepo is minimizing (or erradicating) repeated code.

Until now, we have only our projects placeholders, but the real code for our projects is not included yet. So we need to include our single repo code inside the monorepo.

Here's what I did:

# Pull latest changes in the develoment branch
git pull
# Create a local feature for including the single repo on the monorepo
git flow feature start singlerepo-store
# Add the remote for the store frontend
git remote add store-front git@bitbucket:user/store-frontend
# Fetch the code from the recently added remote
git fetch store-front
# Create a new branch from this remote using the develop branch
git branch store-front-src store-front/develop
# Now you are on the store-front-src branch, you can edit
# code, create new folders, move files, etc. Be sure to commit any
# changes but DO NOT PUSH THEM. We want to keep them local.
# Do as many changes required to match the folder structure of 
# the monorepo. Remember to delete all non-used files.
# Return to the feature branch
git checkout feature/singlerepo-store
# Merge the code with the adjusted code from the single repo branch
git merge --strategy-option=theirs --allow-unrelated-histories store-front-src
# Make final adjustments to your code, check if everything works. 
# If you have new dependencies from the single repo, include them 
# on the package.json and do an npm install
# Once everything works finish the feature
git flow feature finish singlerepo-store

And you'll have to repeat those steps for every sub-project to be included on the monorepo.
IMPORTANT: Don't just copy & paste the lines from the script, try to understand what you're doing and why you're doing it. Change the repo URL, project name, and branch names to your own. Understand then make it work.

Pain 2. Keeping the code history

In most cases, coming from a single repo universe means there's been a lot of development before the monorepo adoption, therefore a lot of history in every commit. After struggling a while with git scripts I learned that git tracks contents and not files, but I also figured that a file rename or even git mv execution wouldn't fit the need of cleanly keeping the code history.

So I came to an old script (2014) that moves files while keeping their history, using git filter-branch. Does what advertised, but… it's veeeeeery sloooooow. The script takes every parameter given and rewrites the whole history, so if you have many commits, and rename many files it will take several hours to complete.
Also, for many files, my advice is to separate them in blocks and commit/push the new changes, after each of them is processed. I knocked my head against the keyboard many times when a massive process failed because some strange commit.

A valid strategy is first renaming files using git mv, for eg. we had to rename all our backend files from .js to .ts After that, we had to adjust the code so it was type strict, but renaming js to ts (Javascript to Typescript) is a starting point.

For this we used the following command-line:

find project-folder -name '*.js' | awk -F '.js'  '{ print $1 }' | xargs -i echo "git mv '{}.js' '{}.ts' && git add -A '{}.ts'"

This searches for all .js files on the ./project-folder , strips the .js out , and generates a new command line like:

git mv 'project-folder/src/index.js' 'project-folder/src/index.ts' && git add -A 'project-folder/src/index.ts'

Note the enclosing quotes so there's no trouble with file names containing spaces or other troublesome special characters. Executing this command may take a while but not as long as the git-mv-with-history.sh script.

After renaming the files you can apply the script so the folder history is rewritten. The script's comments are pretty clear on the usage:

# Given this example repository structure:
#
#   src/makefile
#   src/test.cpp
#   src/test.h
#   src/help.txt
#   README.txt
#
# The command:
#
#   git-rewrite-history README.txt=README.md  \     <-- rename to markdpown
#                       src/help.txt=docs/    \     <-- move help.txt into docs
#                       src/makefile=src/Makefile   <-- capitalize makefile
#
#  Would restructure and retain history, resulting in the new structure:
#
#    docs/help.txt
#    src/Makefile
#    src/test.cpp
#    src/test.h
#    README.md

I would recommend adjusting the project's structure using the instructions on this pain before making the final steps to include the single repo project on the monorepo. See what work for your own use case.

Pain 3. Migration to Angular 8 (front) and Typescript (back)

Our company had several projects running on Angular 6, others on Angular 7. Every backend project was developed using plain old Node.JS Javascript files. NX.dev tool uses Angular 8 for the frontend and Typescript for the backend.

Migrating to Angular 8 is a fairly easy task, the Angular team provides a schematic to migrate, and some fairly clear instructions, and both take care of almost everything. Just run:

ng update @angular/core --from 7 --to 8 --migrate-only

You will have to double check:

• Routing definitions, no longer using loadChildren: ‘./page/page.module#PageModule' by string but using dynamic imports
• The extra parameter { static: true|false } on each @ContentChild and @ViewChild . Your mileage may vary, but static: false usually works for most cases.
• Any broken dependencies.

Migrating to Typescript can be a full new adventure. In our case we had to pass through an ES5 — ES6 code migration, and then apply some Typescript magic. Just be clear that on the migration stages all your team has to be involved and you'll get reports on "something is not working" on every minute.
Typescript is great, just like the perfect mix between Javascript, strictly typed languages and OOP principles, but beware that what used to work on plain old Javascript may not work after being transpiled from Typescript.

From the Google feedback on Typescript 3.5 thread on Github:

We believe most of these changes were intentional and intended to improve type checking, but we also believe the TypeScript team understands that type checking is always just a tradeoff between safety and ergonomics.

Pain 4. Separating reusable code

One of the main decisions behind a monorepo is reusing code/avoiding repeated code. A huge task when including single repo projects on a monorepo is finding repeated code. But not only repeated code that can be refactored on libraries, also shared UI components, shared configurations, similar methods that can be rewritten or abstracted in some way.

This will require running and checking all code for every subproject, then refactor some code as common or shared modules, and then repeat gain, until your team feels like it's done. There's no ending point for this unless you say so, the idea is refactoring the code in an intelligent way that doesn't break how the team understands every project.

In our case, each backend had some model definitions for the ORM we are using, a love-hate relation between Sequelize and me. Every subproject shares the same database configuration, and also some configurations for our security strategies, and also some external libraries configurations. So most of the work was joining all model definitions as a single model library, common for all subprojects, merging all configuration files and refactoring library imports on each subproject.

We ended up with a project structure like the following:

where all our frontend projects are on the apps/frontend folder and all our backend projects are on the apps/backend .

How do you run a monorepo subproject?

When creating a monorepo using the NX.dev tool you'll be asked to choose between the angular-cli or the nx-cli. As our projects are Angular based frontends with NodeJS + Express.js backends, we are using angular-cli.

Over the hood, NX provides several npm run scripts to run each project and do other things. Under the hood, every script uses the ng command with some extra parameters, and every ng command refers to some section on the angular.json file.

So for example, if my projects are frontend/store , frontend/store-admin , backend/store , backend/store-admin and some shared libraries:

• npm start frontend-store -- -o : Runs the store frontend project. The extra double dash space dash o passes the -o parameter to the underlying ng command so the browser is opened. Any extra parameters you need to pass require a preceding double dash.
• npm start backend-store : Runs the store backend project. Note that as our projects live under apps/frontend and apps/backend folders the project name for the npm start command requires a dash after frontend or backend (depending on what project you want to run).
• npm run build backend-store-admin -- --configuration=local-dev : Builds the store admin backend project using the local-dev environment configuration. When specifying a different environment you can configure the build so it replaces some files depending on the environment. The project is built on the dist/apps/backend/store-admin folder.

Let's say you have configured an extra environment called local-dev, that points to a local database instead of the actual development database. To run the store backend using this environment you'll have to execute the following command:

export NODE_ENV=local-dev && npm run build backend-store-admin -- --configuration=local-dev && node dist/apps/backend/store-admin/main.js

This commands set the NODE_ENV variable to local-dev (there's a slight chance you won't need it, unless you have any configuration that depends on the system environment's variables); builds the store-admin backend using the local-dev settings; and finally executes the compiled main.js file to run the backend.

If remembering all command parameters is hard for you, you can just add a new script in the script section on the package.json file. Trust me, you'll end up learning the parameters anyway.

Pain 5. Adjusting dependencies

Refactoring code is one thing, many imports will have to be changed so they can use the new shared libraries and modules. Some lines of code must be added, some others removed, changes everywhere. And when you feel like everything is ready to run, EVERYTHING WILL FAIL MISERABLY (most of the times).

That nice component library, those extra ngPipes, that NodeJS library, all those wonders that:

• are not prepared for Angular ≥8 compatibility; believe it or not it still happens
• cannot be imported as a ES6 module
• cannot be used in Typescript due the lack of @types

So, you'll have to go to each project repo (hopefully they have one), read reported compatibility issues, update versions (and break your own developed functions), apply workarounds. Worst case is having to remove the conflicting package, find a usable replacement, refactor even more code.

If you're lucky, and the development team didn't use a single user with no reputation developed library, but one with a huge community behind, there's a chance that adjusting dependencies isn't so painful. Anyway it still is a pain.

Pain 6. Deploy toolchains

On a single repo universe, deploy toolchains are "simple":

• Listen to changes on the project's master branch
• Simple build (no extra parameters)
• Blue-green deploy strategy (for Cloud Foundry or similar continuous delivery cloud services).

On the monorepo things aren't that simple anymore.

• There's a single giant project that includes all other subprojects. If there's no particular branching for every subproject, you'll have to disable the automatic deployment when there's a new commit on the master branch. This is explained on Pain 7.
• Compilation requires specific project parameters. OK, this isn't sooo terrible, it's just adding a couple of lines to the build instruction.
• Blue green deployment stills the same.

So far so good? Nope. The problem relies on the monorepo itself. As all projects are included as subprojects, so must be their dependencies. If you have many different projects, expect to have many libraries, so the deploy size will increase with no need, just because the npm install command doesn't know which project requires which libraries. And a huge node_modules folder can be the cause of a deploy failure.

What we did: On the server deploy stage we made custom package.json file as the result of a node script that builds the file using a package black list, so it doesn't include any non required packages. We took special care in excluding large packages as jest, cypress, webpack bundle analyzer, and most angular dependencies. Our frontend projects have less chances of failing the compilation than our backend projects failing on the deploy stage, and by the way, IBM deploys can fail just because the wind blew hard that day…

We also made some slight adjustments to our deploy toolchains, mostly scripting magic to make future deployments easier:

• The build script ow includes a cleaning line, so it deletes everything that is not needed on the final deploy. It also specifies what project has to be built. The rest stays almost the same as the original one, but the building stage is included after cloning the project. Automatic deployment was disabled, now everytime we need a new deploy, we have to run the task by hand (not so terrible either).
• The deploy stage makes some initial checks before running, so it doesn't fail on the first run, where there's no project to backup. The script takes the project, subdomain name, and app name as a parameter, so it's very easy to adjust the settings for every subproject. Only in a few cases there's extra customization required.

Not so painful, but a pain anyway. It required checking every project's deploy toolchain, test it, and set it up. Still no magic here.

Pain 7. Deploying a single monorepo sub-project

Monorepos in a un-elaborated definition are many repos living on a single repo, or many projects enclosed as a single projects, that share some common components, configurations, modules and else. Many of these projects have a different development rhythm, and may suffer from changes faster than other projects in the same monorepo.

As far as we have seen, Nx tool doesn't provide a way of including new apps or libs, that live on other repos (as if they were single repo apps), as git submodules. so every new app lives on the same giant monorepo. Unless your project uses a non-standard branching model (note on this later) where every monorepo has it's own "master branch", every deploy from the master or release branch will drag changes that are probably not meant to be released yet.

With an example, lets suppose your monorepo has the following structure:

Made a sample project for this

• Frontend:
  - store
  - store-admin
• Backend:
  - store
  - store-admin
• Libs:
  - ui-components
  - server-modules
  - config

Let's say your deploy chains listen to any master branch commits and run a deploy pipeline on some cloud provider.

The frontend for the store-admin has almost no changes since the first releases, but the store is always improving, so the code changes a lot, and there're many releases on the go. If your monorepo lives on a single repo, as it were a very large project, everytime the store project requires a deploy, the store-admin will be deployed as well, even if it doesn't have any changes.

Also, if there are changes on any project that have to be reviewed and approved before merging them to the master branch (and then trigger every automatic deploy), those changes WILL be included, unless the team makes cherry picks for every commit that effectively HAS to be included.

Cherry picking is a tedious task, so it's a healthy decision searching for a method to avoid it when deploying some project where not all changes have to be included (yet, this is before review) in sibling projects.

A few ideas on this:

• Feature based commits. Every new feature is worked as a feature branch, which is published and merged upon reviewal. Shouldn't be so hard if your workflow uses features (as git flow).
• Master branch per project. This approach is easier as "it only requires" creating a new master branch for every project that has to be deployed. It doesn't matter if it includes changes that haven't been reviewed yet, because that code is not going to be included in the final compilation. The problem I see is that unless your team has visibility on all reviewed commits , keeping the master branch (the real one, not the new master branch for each project) will require some extra effort.

But wait, isn't creating master branchs or feature branchs the same, you're creating new branchs in the repo? True, it will all depend on your own workflows.

Note about non-standard branching models: Every organization has it's own workflows for development.

• A classic one is using the branching model suggested by git flow: master, develop, release, feature, fix (by the way I like this one).
• Another one would be having several development branches, one for each team member, no feature, fix or release branches, just develop, master and a staging (that would be somewhat like the release branch). Don't like this particular one because if the team grows your projects will end up with too many branches. This becomes unusable in the long term.

Try to find the best of two worlds and add a master branch for each subproject. Yes, complexity is added to the branching model, but you'll be somehow solving your monorepo deploy pain.

Are monorepos evil?

Glad you asked :)
No, they are not evil, but they are a PITA at least on their beginning adoption stages. IMHO it's not a bad idea to have all related projects as a single repo (a monorepo), but the organization has to be prepared for this kind of change, evaluate if this particular requirement can't be solved without a monorepo, or even change the development workflow.

And there's a chance that I find more monorepo pains.