Own NPM repository for shared code

Splitting our services into microservices had led us to wonder: what do we do with the shared code? Especially code for basic functions, such as logging or error handling.

Because the code of each microservice lives in its own project folder, and we obviously can’t replicate our shared code in each project folder, we have to place it in a separate shared/common/utils folder outside of any project folder. But how to access an external directory from our projects then? Solutions with npm –link proved to be uneasy to set up, and were causing problems with namespace aliases. Plus, if your common code evolves with even a minor breaking-change, and all your projects are linked to it that way, you have to make the required adaptations in all your projects before you can go on working on them. Not so flexible.

In the end, we decided to go the long, but righteous way: to store such common code in a special project of its own, which we would build as versioned npm package, store on a private repository, and access from any of our projects. It was actually pretty easy to do. Let’s check the few steps to get there:

Host a private NPM repository

Seems like all the thing seen from afar, but it can actually be a breeze, provided you have a server at hand which can run Docker.

We used Verdaccio, which requires no setup at all. Check out https://hub.docker.com/r/verdaccio/verdaccio/ for details. Publishing restrictions on the generated repository are not covered here. Simply make sure you mount a volume to store published packages, as in docker-compose.yml:

volumes:
- /path/to/storage:/verdaccio/storage

Create the shared code library project

We use a NestJs project, compiled with Typescript. Your shared code should reside in /src as usual. Now for the configuration of the library building and publishing:

Notable options within tsconfig.json:

{
"compilerOptions": {
"module": "commonjs",
"declaration": true,
...
"outDir": "./dist",
"rootDir": "./src",
"baseUrl": "./",
"noLib": false
},
"include": ["src/**/*.ts"],
"exclude": ["node_modules"]
}

tconfig.build.json

{
"extends": "./tsconfig.json",
"exclude": ["node_modules", "test", "dist", "**/*spec.ts"]
}

package.json

{
"name": "@ceneau/backoffice",
"version": "1.0.0",
...
"main": "dist/index.js",
"files": [
"dist/**/*",
"*.md"
],
"scripts": {
"build": "tsc -p tsconfig.build.json",
...
},
"publishConfig": {
"access": "restricted"
},
...
}

index.ts should be a barrel file at the root of src/ whose goal is to re-export every objects you want to export in the package. index.ts will be transpiled and become dist/index.js, the root of your exported objects.

With that, the library is ready for building and publishing, but we still have to tell it where to store the generated package, as well as few optional behaviors to follow on each build.

.npmrc (at project’s root) – even if you use yarn

@ceneau:registry=http://url.to.private.npm.repository/

.yarnrc (at project’s root) – obviously if you use yarn to build

version-commit-hooks false
version-git-tag false
version-git-message "Ceneau backoffice common - v%s"

These options handles the generation of tags for each build, and associated git message. Check documentation for more details.

Now, to publish a new version of the shared code:

npm adduser --registry http://url.to.private.npm.repository/

This, if you created your npm repository with no publishing restriction, allows you to create a user within it. You’ll be ask for details and password.

yarn publish

Builds and publishes the package. Hooray !

Use your common library in projects

Now that we’re here, the only remaining trick is to tell your project where to search for your common packages, in addition to the other default npm repositories.

.npmrc (at project’s root) – even if you use yarn

@ceneau:registry=http://url.to.private.npm.repository/

package.json

{
"dependencies": {
"@ceneau/backoffice": "^1",
...
},
...
}

You should be good to go !

Bonus: recover from wrong npm config

If you accidentally changed your global npm registry, your next npm/yarn commands are likely to fail (right now with a 500 error coming from I have no idea). My NPM config was showing some concerning bits:

> npm config list
; userconfig C:\Users\L.npmrc
registry = "http://npm-repo.x.ceneau.com/"

Quick! Hurry and set back the default registry up !

> npm set registry https://registry.npmjs.org/

Running NestJs logic code fast

My 2012 computer, fully loaded with i7 processor and RAM *at the time*, is still quite powerful and very reliable, so I don’t want to change it (ecologically speaking: I don’t want to use more rare resources from the Earth which also come with misery and violence in producing countries; get informed!).
Yet, working with NestJS has first appeared very tedious and extremely long to run, up to 2 minutes.

Avoiding launching the web-server everytime

The first thing I wanted to avoid was to launch the application webserver every time I changed something within the application logic (which is, 99.999% of the time, right?). So I created a script for the case I wanted to develop-and-try :

console.log("[" + (new Date()).toISOString() + "] Script launched.");
import { NestFactory } from '@nestjs/core';
import * as moment from 'moment';
import 'moment/locale/pt-br';
import { AlarmController } from '@ceneau-wa/controller/alarm/alarm.controller';
import { AlarmModule } from '@ceneau-wa/controller/alarm/alarm.module';
import { SymptomOccurrenceDto } from '@ceneau-dto/activity/symptom-occurrence.dto';
console.log("[" + (new Date()).toISOString() + "] Imports done.");

async function run() {
  moment.locale('utc');
  // Prepare controllers and services
  const app = await NestFactory.create(AlarmModule);
  const controller = app.get(AlarmController) as AlarmController;

  // Prepare data to process
  const dto = new SymptomOccurrenceDto();
  dto.entityType = 10;
  dto.entityId = 81;
  dto.positive = true;
  dto.startingDate = 1554214235000;
  dto.endingDate = 1554214237000;
  dto.elementCount = 1;

  // Launch actual logic
  console.log("[" + (new Date()).toISOString() + "] Application starts.");
  await controller.launch(dto);
  console.log("[" + (new Date()).toISOString() + "] Application ends.");
}
run();

This works fine. I create the application context via NestFactory.create, giving it the module I need, and retrieve the component I want to use (here, the AlarmController). Works well!

Minimizing execution time

By using following command to launch above script :

node --nolazy -r ts-node/register -r tsconfig-paths/register src/sendOccurrence.ts 
(don't use this command ! see below)

execution times were terrible, between 90 to 120 seconds.

Surely, it’s the dealing with NestFactory and internal process which causes an horrible overhead, so I tried to not use NestFactory and instead created all services myself with all dependant services, but I was gaining only 2 seconds. I added some profiling time prints, as indicated in above code, and could see most of the time was spent during imports !

Finally, it’s the use of ts-node (install it globally) which saved me, being way faster :

ts-node -T -r tsconfig-paths/register src/sendOccurrence.ts
(-T: "does not check for type errors", saves another 2 seconds in execution)

Execution dropped to 17-20 seconds. Hooray !

Take note that some people totally struggle while using ts-node too, like this guy and his 9-minute start time…

Debugging the process

Unfortunaly, I couldn’t find a way to minimize launching time while debugging the script indicated above. I’m still using a VSCode launch.json with such configuration :

      {
          "type": "node",
          "request": "launch",
          "name": "Slooooow - Debug SendOccurrence",
          "args": ["${workspaceFolder}/src/sendOccurrence.ts"],
          "runtimeArgs": ["--nolazy", "-r", "ts-node/register", "-r", "tsconfig-paths/register"],
          "sourceMaps": true,
          "cwd": "${workspaceRoot}",
          "protocol": "inspector",
          "stopOnEntry": false
      }

And launching time is still 2 minutes…

Note: from VS Code versions after 1.46.0, the internal javascript debugger has an auto-attach mode which seems sweet, and maybe solves this. Turn option “Debug: Toggle Auto Attach” on, and run a npm/yarn command: your breakpoints should be hit. More details here:
https://github.com/microsoft/vscode-js-debug#debug-nodejs-processes-in-the-terminal.

Debugging Typescript with Visual Studio Code and module aliasing

Note: from VS Code versions after 1.46.0, the internal javascript debugger has an auto-attach mode which seems sweet, and maybe make all the text below deprecated. Turn option “Debug: Toggle Auto Attach” on, and run a npm/yarn command: your breakpoints should be hit. More details here:
https://github.com/microsoft/vscode-js-debug#debug-nodejs-processes-in-the-terminal.

Our policy is to go more and more towards Javascript, for client-side applications as well as backoffice jobs, and with that, to embrace technologies such as NodeJS and language add-ons such as TypeScript. Frameworks like Angular (web apps) and NestJS (server-side apps) provide a full range of ready-to-use schematics and release-builds optimizations. Now we want to use an IDE which leverages those tools with powerful IntelliSense and debugging possibilities. IntelliJ IDEA comes as a choice of reference but is not for every budget, so we’ll give a go for Visual Code Studio. This post is about my findings on how to debug a NestJS app (NodeJS app following Angular policies) with VSCode.

Debugging NestJS apps

To avoid ridiculous import paths ‘../../../../adir/amodule’, module aliasing is used in the form of ‘@myalias/amodule’, improving readability and ease in moving code files. The aliases are set in the tsconfig.json file at the root of the project, needing a ‘baseDir’ attribute and a set of ‘paths’ (both to set under the ‘compilerOptions’). VSCode IntelliSense knows this trick perfectly and reacts accordingly, but when it comes to debugging, the default launch.json created by VSCode struggles at matching the paths, and quickly results in :

Debugger listening on ws://127.0.0.1:31189/821980d6-4e30-4360-8ac5-6b47af4faced
For help, see: https://nodejs.org/en/docs/inspectorDebugger attached.
Error: Cannot find module ‘@myalias/app.service’
at Function.Module._resolveFilename (internal/modules/cjs/loader.js:582:15)
at Function.Module._load (internal/modules/cjs/loader.js:508:25)
at Module.require (internal/modules/cjs/loader.js:637:17)

The trick here is to indicate explicitly that the tsconfig-paths module must be used, just like it’s indicated in the default package.json ‘start’ script :
“start”: “ts-node -r tsconfig-paths/register src/main.ts”
In VSCode launch.json file, it’s done by :
“runtimeArgs”: [“–nolazy”, “-r”, “ts-node/register”, “-r”, “tsconfig-paths/register”]

To wrap it up, VSCode launch.json file should look like :

{
// Use IntelliSense to learn about possible attributes.
// Hover to view descriptions of existing attributes.
// For more information, visit: https://go.microsoft.com/fwlink/?linkid=830387
“version”: “0.2.0”,
“configurations”: [
{
“type”: “node”,
“request”: “launch”,
“name”: “Debug Nest Framework”,
“args”: [“${workspaceFolder}/src/main.ts”],
“runtimeArgs”: [“–nolazy”, “-r”, “ts-node/register”, “-r”, “tsconfig-paths/register”],
“sourceMaps”: true,
“cwd”: “${workspaceRoot}”,
“protocol”: “inspector”
}
]
}

Debugging NestJS tests

It’s very very useful to be able to debug the tests written for the application – the same tests which will be automatically run within Continuous Integration. NestJS uses the Jest framework to write tests (close to the Jasmine framework), and here are the launch.json configuration to add to debug all tests or just the test currently shown in the editor :

{
    ...
    "configurations": [
        ...,
        {      
          "type": "node",
          "request": "launch",
          "name": "Debug test - all",
          "program": "${workspaceFolder}/node_modules/.bin/jest",
          "args": ["--runInBand"],
          "console": "integratedTerminal",
          "internalConsoleOptions": "neverOpen",
          "disableOptimisticBPs": true,
          "windows": {
            "program": "${workspaceFolder}/node_modules/jest/bin/jest",
          }
        },
        {
          "type": "node",
          "request": "launch",
          "name": "Debug test - current file",
          "program": "${workspaceFolder}/node_modules/.bin/jest",
          "args": [
            "${fileBasenameNoExtension}",
            "--config",
            "jest.config.js"
          ],
          "console": "integratedTerminal",
          "internalConsoleOptions": "neverOpen",
          "disableOptimisticBPs": true,
          "windows": {
            "program": "${workspaceFolder}/node_modules/jest/bin/jest",
          }
        }

    }
}