This page describes several development aspects of the Cubitt framework.
CQRS micro-services view
This view describes the micro-services involved with the CQRS workings of the Cubitt back-end.
A view on how the CQRS related micro-services in the Cubitt back-end interact.
Orange arrows mean command related flows and purple arrows mean query related flows.
Glossary of elements
| Id | Name | Description |
|---|---|---|
| Client | A client that wants to interact with the Cubitt back-end. | |
| Project Manager | The project manager is a micro-service that manages all projects and handles all requests. It passes the requests to the right micro-service if it exists. | |
| Command handler | The command handler is a micro-service of which only one is spawned per project. it handles all the (transactions of) commands have to applied to the project. After a command is applied successfully, it is stored as an event in the event store and through the event bus, all query micro-services are notified of the change. | |
| Query builder | One query builder is spawned per project. It receives requests for versions not present in the snapshot store and builds it using the events in the event store. Then it both returns the snapshot to the project manager and stores the snapshot in the snapshot store. | |
| Event Bus | An event bus on which all events handled commands are published. Query side micro-services can act upon receiving an event. | |
| Event Store | The event store is a database which stores all events of changes made to project. | |
| Snapshot Store (to be implemented) | The snapshot store is a database which stores JSON representations of different versions of the graph. It acts as a cache. | |
| Continuous graph builder (to be implemented) | One continuous graph builder exists per project, it is a micro-service which listens to event bus. Upon receiving an event, it applies it to the last known version of the project and stores the new snapshot in the snapshot store. This ensures the latest version is always available from the cache. | |
| Continuous push service (to be implemented) | One continuous push service exists per project, it is a micro-service which listens to the event bus. Upon receiving an event, it pushes those events to all clients whom have subscribed themselves to this update service. |
Rationale
As stated before, architectural drawing are basically graphs. This led the authors to take a look at graph databases. A problem however with these graph databases, as well as storing the graph in a NoSQL or relational database, is that reading the graph requires blocking write attempts. By locking the entire graph during read operations, the scalability and performance of the system would be drastically limited.
To solve this problem, Event Sourcing is used. Event Sourcing ensures that all changes to application state are stored as a sequence of events. So every write action, i.e. adding a node, appends an event to the sequence of events. By using an incrementing version number for the events, it is simple to read up to and including the event required, and ignore any write actions that might have occurred in the meantime. Any write actions that might have occurred in the meantime will be appended to the event store. This ensures that read and write operations can be handled concurrently.
More information
More on information on Event Sourcing can be found here.
Replaying every event might become costly if the amount of events grows, so snapshotting is used to mitigate this problem. A snapshot is in essence a pre-constructed state, this can reduce processing time a lot by using more storage space to store these snapshots. For example, if you create and store a snapshot every 50 events and you want to recreate the state after 13861 events. All you do is load the snapshot after 13850 events and replay the 10 events; 13851-13861.
Since different versions of the graph may be requested, i.e. a concurrent user or analysis plugin might request a slightly out of date version, many different graph versions should reside in
memory for every project. This limits the scalability of the system by the amount of RAM and CPU available. Command and Query Responsibility Segregation (CQRS) was introduced as a result of this limitation and of its natural fit with Event Sourcing. CQRS is based on the separation of the interface for reads (Queries) and writes (Commands). This allows the optimization of both interfaces for their use and considerably simplify the design and implementation of all components. The query side is informed of changes to the state (Commands) using Events. Typically, a pub/sub Eventbus is used for the transportation of the events.
More information
More information on CQRS can be found here.
The advantages of the introduction of CQRS are increased scalability and flexibility on the query side. For example, it is possible to create multiple query services that optionally maintain different versions of the graph. Furthermore, it is possible to create a Push Service that pushes events created by users that are working concurrently on the same project. To create this Push service, it only has to listen on the Eventbus and push those events to the client.
Disadvantages of this pattern is the increased operational complexity and the introduction of eventual consistency. The system becomes eventual consistent because changes performed on the write side take time to propagate to the query side. While an event produced by a command is still on the Eventbus, the query side serves an inconsistent version. Since the EventBus should deliver every event at least once, the query side will eventually become consistent.
Module usage view
This view describes what modules are used in which micro-services.
A view on the the module stack in each of the micro-services.
Glossary of elements
| Id | Name | Description |
|---|---|---|
| Event Pusher | The module that handles all external communication for the event pusher. It receives events through the event bus and passes them on to all subscribed clients. | |
| Query builder | The module that handles all external communication for the query builder. It receives a request for a version of the project to be build, then retrieves the required events and passes them all on to the CQRS graph. | |
| Continuous graph builder (to be implemented) | The module that handles all external communication for the continuous graph builder. It listens on the event bus and upon receiving an event, it builds the new version. | |
| Command handler | The module that handles all external communication for the command handler. It receives (transaction of) commands and tries to applies these commands to the CQRS graph. | |
| Common | The common module contains several common elements used in (almost) all other modules. | |
| Commands | The commands module is a library which contains all possible commands and a command factory to parse commands in JSON format. | |
| Events | The events module is a library which contains all possible events and an event factory to parse events in JSON format. | |
| Graph | The graph module is an in memory graph. It offers a basic interface to allow interactions with it. | |
| CQRS graph | The CQRS graph is a wrapper around the graph module. It offers additional features such as transaction management and the ability to handle both events and commands. | |
| Client (to be implemented) | The client is the user program or website which the user will use to interact with the project. | |
| Validator (to be implemented) | A module which allows for semantic checking of changes to the project. | |
| Command sender (to be implemented) | A module which translates user actions in the GUI into commands to be processed by the back-end. | |
| Event receiver (to be implemented) | This module receives event updates from the back-end about changes to the project by other users and applies them to the internal CQRS graph in the client. | |
| GUI (to be implemented) | The graphical user interface in which the user can directly interact with the architecture. These interactions are send to the the back-end by the command sender. |
Rationale
The model shows a high reuse of modules in both frontend and backend. The
reuse decreases the development time as the same feature only has to be developed once. It
also ensures that a certain functionalities behave the same independent of where it is used in
the Cubitt framework. Cubitt is developed in JavaScript (TypeScript) to allow reuse of code in the client, which will most likely be browser based,