NATS
NATS
NATS (https://docs.nats.io/) is a message broker.
A message broker is an architectural pattern for message validation, transformation, and routing. It mediates communication among applications, minimizing the mutual awareness that applications should have of each other in order to be able to exchange messages, effectively implementing decoupling.
NATS provides a way of sending messages in a pub/sub and request/reply style. Parro Nexus uses NATS for all communication between all services. By using a message broker we get a more decoupled setup, services don’t need to known about each other. The only need to know where to send their messages.
NATS uses subjects for the routing of messages, meaning a service can publish or request a message to an subject and other services can subscribe or reply on these subjects.
For example:
ServiceA needs to get User information based on a username. Instead of making a REST call to the UserService it will sent a request message to NATS with subject user.info. The UserService is subscribed on the subject user.info and will reply on messages received on this channel.
This way there is a decoupling between the services. We can create a complete new UserService and deploy it, and as long as it is responding to the user.info subject ServiveA will keep working.
Jetstream
NATS was build for speed, and with this in mind the core NATS server does not store any message. When a message is published to a subject with no subscribers an error will be thrown.
Since there are lots of use-cases where this is not a viable solution they decided to add support for storing messages. This system is called “Jetstream” (https://docs.nats.io/nats-concepts/jetstream)
For Parro Nexus we deploy a NATS cluster with Jetstream enabled. This makes it possible to create queues of messages and process them when a service is available
Authentication
NATS had several ways of authentication (https://docs.nats.io/nats-concepts/security). For our setup at Topicus we use a JWT token approach. This means the clients get a JWT token which they provide during connect.
We also use NATS accounts. Accounts is the NATS solution for multi-tenancy. When you for example use a cluster for acceptance and production (what Parro Nexus does), you don’t want messages sent by acceptance services to end up in production.
You could use prefixes in your subjects to prevent this, but this gets messy fast. To be able to have a multi-tenant setup you can create Accounts in NATS. An Account is a single tenant, so for Parro Nexus we have an acceptance and an production account.
NATS will then make sure no messages will be shared between these accounts, but you can use the same subject.
To which account you connect is determined by the JWT. Accounts are able to sign JWT tokens, so the NATS server can verify to which account a JWT belongs.
So for the NATS server to check JWT’s it needs to know which accounts are available. This is done by adding the account JWT’s to the server config.
This way the NATS server knows the account JWT’s and it will use these to validate the JWT’s presented by the clients which are connecting.
Since an Account is also only a JWT, the NATS server needs a way to validate these JWT’s as well. Otherwise, you could add a random account JWT to a NATS server and start connecting to it.
To validate account JWT’s NATS uses an concept of an Operator JWT.
The Operator is also a JWT which you add to the server configuration. You create a single Operator per NATS cluster. So with this Operator JWT NATS is able to validate the Account JWT’s and with the valid Account JWT’s it can validate the clients.
Since you want to be able to add/remove accounts at runtime and not only in the config of the NATS server NATS is able to lookup the accounts when they are required. NATS does this by sending a request on a special subject $SYS.REQ.ACCOUNT.*.CLAIMS.LOOKUP.
So when a client connects with a token signed by account production it will check if it knows this account, and otherwise it will request the JWT for this account by sending a message to $SYS.REQ.ACCOUNT.*.CLAIMS.LOOKUP
This makes it possible to create some client which can create accounts and provide the server with the correct JWT when it is needed.
For Parro Nexus (and other Topicus NATS clusters) we created a kubernetes operator for this account lookup.
Operators
To make NATS easier to work with on a kubernetes cluster we developed two custom operators for nats
NATS account operator
The NATS account operator (https://github.com/topicusonderwijs/nats-account-operator) make it possible to create accounts with a Custom Resource Definition (CRD).
The NATS account operator is linked to an NATS cluster and will subscribe to the special subject $SYS.REQ.ACCOUNT.*.CLAIMS.LOOKUP.
When a CRD for an account is created in the k8s cluster it will create an account JWT and store it as a k8s secret. When the NATS cluster requests an account the operator will look up the account in the stored secrets and return the JWT.
apiVersion: nats.topicus.education/v1alpha1
kind: NatsAccount
metadata:
name: "production"
spec:
The NATS account operator can also create JWT’s for the clients which need to connect to the NATS cluster. Parro Nexus does this for all services. The NATS account operator will look up the account in the CRD and create a user JWT and store this as a secret in k8s. The clients (Parro Nexus services) can mount these secrets so the can use them when connecting to the NATS cluster.
apiVersion: nats.topicus.education/v1alpha1
kind: NatsUser
metadata:
name: "some-test-user"
spec:
account: "production"
NATS jetstream operator
The NATS jetstream operator (https://github.com/topicusonderwijs/nats-jetstream-operator) is created to configure the NATS streams. A stream in NATS is the configuration on how messages for certain subjects need to be stored and dispatched.
For example if you want a stream which acts as a queue you need to configure the streams with an retention setting of workqueue. And if you want a maximum of 100 messages stored you need to configure the maxMsgs property.
These configurations can be done from code or through the NATS server. To make it easier to config the streams from k8s we created the NATS jetstream operator.
The NATS jetstream operator takes a CRD and then tells the configured NATS server to create or update a stream. This way you can configure your NATS streams from your infra setup instead of in code.
apiVersion: nats-jetstream.topicus.education/v1alpha1
kind: Stream
metadata:
name: <stream-name>
spec:
creds:
secretName: <required name-of-secret>
fieldRef: <required field-of-seed>
subjects: <required array of subjects i.e ["subject1", "subject2"]>
storage: <required type of storage. Valid values: file, memory>
retention: <required retention type. Valid values: limits, interest, workqueue>
nameOverride: <optional override name when stream name is not same as CRD name>
description: <optional description of the stream>
discard: <optional how to discard messages. Valid values: old, new. Default: old>
duplicateWindow: <optional the window within which to track duplicate messages. Format is in golang Duration i.e 1m, 10s 500ms. Default 0 (no window) >
maxAge: <optional Maximum age of any message in the Stream. Format is in golang Duration i.e 1m, 10s 500ms. Default 0 (unlimited)>
maxBytes: <optional How many bytes the Stream may contain. Default unlimited >
maxConsumers: <optional How many Consumers can be defined for a given Stream. Default -1 (unlimited)>
maxMsgs: <optional How many messages may be in a Stream. Default -1 (unlimited)>
maxMsgSize: <optional The largest message that will be accepted by the Stream. Default -1 (unlimited)>
maxMsgsPerSubject: <optional How many messages may be in a subject. Default -1 (unlimited)>
noAck: <optional Disables acknowledging messages that are received by the Stream. Default false>
replicas: <optional How many replicas to keep for each message in a clustered JetStream. Default 1 max 5 >
Setup for Parro Nexus
NATS is installed and managed by team infra, for the Parro Nexus k8s environments (onderwijs-intern and previder). This means the NATS cluster and the 2 operators are already installed on the k8s clusters.
For Parro Nexus we only manage the NATS configuration. This is done in the infra repo (https://github.com/topicusonderwijs/parro-connect-infra). In the repo we configure a user for each service, and for the services which need streams we add the stream config.
Examples: