Parro Nexus – Local development

Dev: Project setup

Mon, 01 Jan 0001 00:00:00 +0000

Parro Nexus - Repo

The Parro Nexus repo (https://github.com/topicusonderwijs/parro-nexus) is a mono repo containing all services and libraries for Parro Nexus. There are three more repositories for Parro Nexus:

Apps

The apps directory contains all Parro Nexus services. Each app is a NestJS service generated with the service generator.

An app is a NodeJS app which uses the NestJS framework. The root directory of the app therefore contains the usual NodeJS files like the package.json and tsconfig.json. It also contains a Jenkinsfile so the app can be build by Jenkins, a Dockerfile so a docker container can be created and some “dot” files for configuration of npm packages.

Environment variable

Everything which is configurable in an app must be put in environment variables. This makes it easy to config an application when it is running in a docker container on k8s. To use these env vars on your local machine without setting them as actual env vars you can use the .env.* files. Default each app has 2 files .env.template and .env.test. The .env.test is used for the tests and the env.template is used as template to show which variables are available.

For local development you will need to create a file env.local.development, which is ignored by git. In this file you can setup your local values. Since some values are secrets (i.e the database user/pass) it should never end up in the git repo. The easiest way to create a env.local.development file is to copy the env.template and fill it with the appropriate values. When you add new configurable values to a service you MUST add these to the env.template file.

The values from these .env.* files are read in the configuration.ts file located in src/config, using the cross-env npm module. In the package.json you can see the lib used in the start script: "start": "cross-env NODE_ENV=development nest start --watch".

Source code

All source code for the app is located in the src directory. The src directory contains 2 typescript files and 4 subdirectories:

main.module.ts: this file is the entrypoint for NestJS
main.ts: this file is the main typescript file which bootstraps the app
config: for handling all the configuration of the app
listeners: for implementing custom shutdown listeners, to create a gracefull shutdown for the app
modules: this directory contains the NestJS modules (https://docs.nestjs.com/modules) which make up the actual app
schema: this directory contains the generated JSON schema model

The actual app code is located in the src/modules. This directory consists of multiple NestJS modules:

app: this module contains the actual app code
authorization: this module contains the REST oauth authorization code, only used for services which are publicly available
health: this module contains the readiness and liveness endpoints which are used by k8s to determine if the service is up (https://kubernetes.io/docs/tasks/configure-pod-container/configure-liveness-readiness-startup-probes/)
telemetry: this module contains all the setup for the metrics and tracing, which is used to monitor the app

Normally when you create a new app or work on an existing app you only need to change the code in the src/modules/app module. This module contains all the app specific code and can have different setups based on the service needs.

All service use the NestJS framework, so when you are new to this start by reading the docs: https://docs.nestjs.com/

Code generation

To create uniform services we use a code generation tool. This tool makes is possible to generate a skeleton app or lib project. This way we can generate all the boilerplate code in a uniform way, so you can start working on the app code right away.

The generation tool is located in code-gen/schematics. The src dir contains the files which will be added when you create a new service or lib. When you change the code in the code-gen/schematics a Jenkins job will build the code and create a NPM module @parro-connect/schematics. This module is used by the script create_nestjs_service.sh in the apps directory. Since each build creates a new version you must update the version of the package.json when you want to merge changes on the main branch, otherwise the Jenkins build will fail. The easies way is to use the npm command npm run version:for-release. This will bump the version and add an entry to the CHANGELOG.md

The code-gen also contains a git submodule for the JSON schema model code-gen/parro-connect-schema. This submodule makes it possible to generate the model classes in your project. To do so run yarn build from within the code-gen/parro-connect-schema and follow the steps.

Docker

The docker directory contain docker-compose files for running all the services or the local NATS server. To start all the Parro Nexus services at once run the start-compose.sh script. This will build and run the Dockerfile-NestJsService-Local files located in all the app directories. The Dockerfile-NestJsService-Local is a special docker setup which will mount the complete src dir in the container and runs the yarn start:dev command. This makes it possible to change code in the apps and it will be auto reloaded in the container.

Docs

The docs directory contains all the markdown files which makes up this documentation you are now reading.
The docs use the Hugo (https://gohugo.io/) static site generator with the docsy (https://www.docsy.dev/) theme.
To start the docs on your local machine run the start_docs.sh script. This will start a docker container with Hugo and starts a server you can reach on http://localhost:1313/

Libs

The libs directory contain all the Parro Nexus libraries. When you need to create a reusable library (a custom NPM module) you can use the script create_lib.sh. This will use the code generation from the code-gen dir to set up a basic library skeleton project. All libraries are build using Jenkins and are published to the @parro-connect scope (https://docs.npmjs.com/cli/v8/using-npm/scope).

Scripts

The scripts directory contains bash scripts for common tasks.

Dev: NATS

Mon, 01 Jan 0001 00:00:00 +0000

NATS

Parro Nexus relies heavily on NATS for service 2 service communication. When you work on a service locally you will at some point need a working NATS environment for testing or running multiple services.

The Parro Nexus repo contains a NATS setup using docker which can (and should) be used when working on services. This setup uses the NATS docker container and adds some authorization en stream setup.

Prerequisites

You need to have docker installed on your machine in order to work with NATS.

https://docs.docker.com/get-docker/

Start a server

In the scripts directory of the Parro Nexus repo you will find a start_nats.sh script which will start a NATS docker container. The script will use the NATS config file found in docker/nats_conf.conf. This configuration file has a fixed authorization setup with a single Account (see Authentication chapter of NATS page). The credential files which work with this setup are:

./nats-sys-user.creds - System user, for NATS maintenance
./nats-test-user.creds - Actual user, for Parro Nexus services

The start_nats.sh script will also mount all config (./scripts/nats-box-scripts/config), sample messages (./scripts/nats-box-scripts/tools) and a setup (./scripts/nats-box-scripts/setup) script to a nats-box container. The nats-box container contains a NATS CLI which can be used to configurate and monitor the NATS server. When you run the start_nats.sh script you will end up in a shell within this nats-box container. Add this point you can run the setup command to run the ./scripts/nats-box-scripts/setup script. This will create all configuration required for Parro Nexus to run.

To test services you can use one of the sample messages in ./scripts/nats-box-scripts/tools. These are all added to the /usr/local/bin dir so you can just run them as commands:

dlq - Send a message to the connect.v1.dlq.ingester subject
error - Send a message to the connect.v1.error subject
get_org - Send a request to the organisation.v1.get_organisation subject
ingest - Send a message to the message.v1.ingest subject
invalid_msg - Send an invalid message to the message.v1.validate subject
prepared - Send a message to the message.v1.prepared subject
receipt - Send a message to the message.v1.receipt subject
valid_msg - Send a message to the message.v1.validate subject

Stop the server

To stop the server you can kill the running docker containers or run the stop_nats.sh script in the scripts directory.

Dev: Tests

Mon, 01 Jan 0001 00:00:00 +0000

Tests

All services and libraries have tests. There are 2 type of tests implemented:

Unit tests
End 2 End (e2e) tests

These tests are written using the jest framework (since this is used by nestjs).

The unit tests are located near the source in a test directory, and all the files have the extension .spec.ts To run the test run the command yarn test.

The e2e tests are all located in the test directory in the root dir of an app, they have the extension .e2e-spec.ts. To run the e2e tests run yarn test:e2e

The yarn test:e2e test runs the command jest --config ./test/jest-e2e.json. This is important when you want to run the tests from your IDE, since the e2e tests require a config file. When you run the tests from an IDE make sure is also uses this config file otherwise the tests won’t work.

Testcontainers

Most of the e2e tests make use of the testcontainers project. The testcontainers are used to start all dependencies of an application (i.e. postgres or NATS).

This way you can test the actual app without mocking all external dependencies.

Scripts

The package.json files for all apps contain multiple scripts to run tests:

test: Run the unit tests
test:watch: Run the unit test and watch for changes for hot reloading
test:cov: Run the unit tests with code coverage
test:debug: Run the tests, but wait for a debugger before starting
test:e2e: Run the e2e tests
test:e2e-cov: Run the e2e tests with code coverage
test:full-cov: Run all the test (unit and e2e) and combine the code coverages (this is used by the build pipelines)
test:cov-silent: Run the units tests with code coverage, but without output (only errors)
test:e2e-cov-silent: Run the e2e tests with code coverage, but without output (only errors)
test:full-cov-silent: Run all the test (unit and e2e) and combine the code coverages, but without output (only errors)

Dev: Docs

Mon, 01 Jan 0001 00:00:00 +0000

Docs

The docs site for Parro Nexus (the one you are looking at now) is maintained from within the Parro nexus repo. This site renders markdown files using Hugo with the Docsy template.

When you make changes to the markdown, or add new documentation it will auto update the site using a Jenkins job

The Jenkinsfile for this job contains the following steps:

Validate

dockerfile.validate {}

This will validate the Dockerfile locates in the docs dir docs/Dockerfile

Build the site

Since hugo generates all the static html we need to run it against the source of this repository. There is no default jenkins step for hugo so we run the commands inside a docker image from hugo. Since jenkins will mount the current workspace dir inside the container this will have the same effect as running hugo inside the docs directory

docker.image("klakegg/hugo:ext").inside("--entrypoint=\"\"") {
    sh("""
        git submodule update --init --recursive themes/docsy 
        hugo
    """)
}

Build docker image

def img = dockerfile.build {
    stage = "Build image"
    name = "parro-connect/docs"
}

Build the docker container with the generated static html pages

Publish the image

dockerfile.publish {
    stage = "Publish image"
    image = img
    distribute = false
    tags = ["${tag}"]
}

Publish the image to the artifactory

Deploy

stage("Deploy"){
    if(branchName == "main"){
        build   job: "geon/Microservices/deploy_development",
                wait: false,
                propagate: true,
                parameters: [
                    string(name: 'service_name', value: "docs"),
                    string(name: 'new_version', value: tag),
                ]
    }else{
        jenkinsHelper.markStageAsSkipped("Deploy")
    }
}

All commits on the main branch should be published. This is done by a separate job geon/Microservices/deploy_development

Running the docs local

There are 2 ways to run the docs local:

Install hugo
Run it in a container

For both options there is a script in the docs dir:

start_docs.sh -> will start a docker container
start_docs_local_hugo.sh -> will start hugo with the correct startup parameters

Both scripts will start hugo in a watch mode which will show updates as you type. The local site is served at http://localhost:1313