Most of what I build ends up being applications with more than one deployable artifact. Sometimes they're just a front-end client and a simple server. Other times it is that plus a GraphQL schema, multiple Lambdas, and so on.
But among all of that, building the packages was still a problem. Namely, if a local package depended on another local package being built, you need to orchestrate that somehow. And as much as I tried, it always ended up feeling less than ideal — and certainly not easily repeatable.
In my dabbling into other langauges and tooling, I tried using Bazel. In some ways it's great. I used it to great success with a Golang monorepo with multiple build and testing artifacts.
In practice, at the start of 2020 support was growing but still limited. Enough things worked to convince me this is a workable approach though.
At a high level the plugin is pretty straight forward. Yarn's already built the dependency graph. We just need to know where to start from on that graph. That's also relatively easy. If you're in the directory of a package, well we can work out which package it is. If not, we can build everything.
Once we know what we need to build, we have a look at anything it depends on, and if they depend on anything. And so on. Once we know that, we can build a plan for how to build it all with as much parallelisation as you have threads.
Having worked all of that out, there was one last feature I really wanted to include. And to be honest, it's the main thing I've wanted from the start.
I wanted a command that will create a zip file ready for AWS Lambda, Kubernetes or Docker.
Now I hear what you're saying, “what about the Serverless framework?”. While I know it's a valuable tool, and plenty of us use it with much success. It's never fit my requirements. Any abstraction over Cloudformation that obscures the actual Cloudformation templates always ended up getting in my way.
Yarn PnP makes this a bit hard, locally linked packages make this really hard. And vendoring node_modules makes this near impossible. Especially in a monorepo where your dependencies are shared and hoisted up. Meaning you can't just copy the adjacent node_modules folder. We need something smarter. Much smarter.
Once again though, we have access to the dependency graph we've already defined for Yarn. Combining this, with the zipfs tooling in Yarn v2, it was not too much extra work to get this going.
Now, in a package running yarn bundle copies the whole workspace (so likely you repository) into a temporary folder. Then, using Yarn's dependency graph, we chuck out everything we don't need. Delete the local packages that aren't used, and the vendored packages that aren't used.
At this point we have a zip file that looks like your repo, with a bunch of stuff chucked out. Which is great, but there's two remaining issues to tackle.
The first, Yarn PnP. It's great, and it means our zip file is faster to work with and smaller than a node_modules directory. But, we need to run everything via the pnp.js file.
The second, is that as we're recreating the whole workspace in the zip file, and not just your package, you need to know exactly where it is to specify your entrypoint or index file.
The solution was pretty simple. Drop a file called entrypoint.js at the root of the zip file. Have it load pnp.js first, then load your file, referenced in main in your package.json.
And just like that, yarn bundle can create a zip file ready to run in Lambda et al.
This all sounds great, but how do you actually use it?
First, you have to be using Yarn v2/3. If you're not already here's a great getting started guide.
Next install the plugin by running the following command in your Yarn workspace:
yarn plugin import https://yarn.build/v2
or for Yarn v3
yarn plugin import https://yarn.build/latest
This command downloads and installs (or updates) the yarn.build plugin to the latest version. The plugin is downloaded and vendored in you repository. It's not redownloaded on every build.
Currently there's two commands you can run.
yarn build which will run the build script defined in package.json.
And yarn bundle which will create the zip file described above, ready for Lambda et al.
There's still plenty of work to be done on this plugin, but in it's current state it's ready to start being used.
You can find the source here github.com/ojkelly/yarn.build.