NAV

Introduction

Kuzzle is a ready-to-use, on-premises backend that enables you to manage your persistent data and be notified in real-time on whatever happens to it. It also provides you with a flexible and powerful user-management system.

Kuzzle enables you to build modern web applications and complex IoT networks in no time.

  • Persisted data: store your data and perform advanced searches on it.
  • Real-time notifications: subscribe to fine-grained subsets of data.
  • User Management: login, logout and security rules are no more a burden.
  • Extensible: fit Kuzzle to your needs by leveraging the plugin system.

Getting started

In this tutorial you will learn in a few steps how to launch Kuzzle and how to interact with it by persisting data and being notified when data is updated.

Running Kuzzle

Before launching Kuzzle, ensure that your system matches the following pre-requisites:

Thanks to Docker-compose, running Kuzzle is easy. Just grab the standard docker-compose.yml file, copy it into a directory and start Kuzzle:

$ sudo sysctl -w vm.max_map_count=262144
$ wget http://kuzzle.io/docker-compose.yml
$ docker-compose up

Your terminal will show the log messages of Kuzzle’s components starting. After only a few seconds, you should see the following ready message appear:

kuzzle_1         | [✔] Kuzzle server ready

Your Kuzzle server is now ready to be used. For instance, you can hit the main HTTP API endpoint by browsing the page http://localhost:7512?pretty=true or via cURL on the command line:

$ curl "http://localhost:7512/?pretty=true"

Kuzzle will respond you with a list of the existing routes.

Where do we go from here?

Now that Kuzzle is running on your computer, you can dive into playing with it by:

SDK play time

It’s time to play with the Kuzzle SDK. In this section, we will persist a document and subscribe to notifications in Kuzzle using the JS SDK.

Before proceeding, ensure that your system matches the following requisites:

  • A fairly recent version of NodeJS (i.e. v6+) should be installed on your system (instructions here),
  • A Kuzzle server up and running.

Create your first “Hello World” document

Create your playground directory and install the Javascript SDK from the command line using npm:

$ mkdir kuzzle-playground
$ cd kuzzle-playground
$ npm install kuzzle-sdk

Save the following JS code in the new create.js file:

var Kuzzle = require('kuzzle-sdk')

// connect to the Kuzzle server
var kuzzle = new Kuzzle('localhost', {
    defaultIndex: 'playground'
  })

// get a reference to the a collection
var collection = kuzzle.collection('mycollection')

// define the document itself
var document = {
    message: 'Hello, world!'
}

// persist the document into the collection
collection.createDocument(document)

Run your file in NodeJS

$ node create.js

You can find more resources about Kuzzle SDK in the SDK Documentation.

Subscribe to data changes (pub/sub)

Kuzzle provides pub/sub features that allow you to be notified in real-time on the state of your data (check the Room Class Documentation for a deep-dive on notifications).

Let’s get started. Open a new termnial in the playground directory you created before and create the subscribe.js file containing the following code:

var Kuzzle = require('kuzzle-sdk')

// connect to the Kuzzle server
var kuzzle = new Kuzzle('localhost', {
    defaultIndex: 'playground'
  })

// create a reference to the data collection
var collection = kuzzle.collection('mycollection')

// define a filter
var filter = {
    exists: {
        field: 'message'
    }
}

// create a subscription on the collection matching given filters
collection.subscribe(filter, function(error, result) {
    // this function is called each time kuzzle notifies us with a document matching our filters
    console.log('message received from kuzzle:', result)
})

Run your file in NodeJS

$ node subscribe.js

And let it wait for documents entering the scope of the filter.

Now, get back to the previous terminal window and execute once more the create.js script. Take a look at the output of the subscribe.js script. You will see that, each time a document with a message field is persisted in Kuzzle, a notification is shown as standard output.

Where do we go from here?

Now that you are started and operational with Kuzzle, you can fully leverage its power by:

Essentials

Basic concepts

Persistent data

Kuzzle relies on Elasticsearch to store and fetch persistent data. You can perform a variety of CRUD and fine-grained search operations on persistent data. Please refer to the dedicated section for more details.

Real-time notifications

Kuzzle enables you to set up subscriptions to sets of data, in order to be notified in real-time about whatever happens to them. You can create a subscription by selecting a set of data. Selections (also called filters) are expressed in a Domain-specific Language (DSL) that we tailored for this purpose. Please refer to the dedicated section for more details.

SDK

Kuzzle ships with a set of open-source SDK for a variety of languages:

Supported protocols

Kuzzle supports a variety of communication protocols.
For the time being, Kuzzle supports the following protocols:

  • HTTP
  • Websocket
  • Socket.io
  • MQTT (via plugin)

You can directly interact with Kuzzle using the Kuzzle API reference.

Authentication

Kuzzle supports a variety of authentication strategies via PassportJS. Local and OAuth-based authentication is natively supported, but you can also add your own custom strategy.

Please refer to the dedicated section for more details.

Plugins

Kuzzle is extensible in many ways. Plugins enable you to

  • trigger actions on data-related events,
  • intercept the data flow at any point of its lifecycle,
  • add custom methods to the public API,
  • add new communication protocols,
  • add new authentication strategies.

Please refer to the dedicated section for more details.

Installing Kuzzle

Docker

Before launching Kuzzle, ensure that your system matches the following pre-requisites:

Get the standard docker-compose.yml file, copy it into a directory and start Kuzzle:

In this case, you need to increase the maximum virtual memory allowed by typing

$ sudo sysctl -w vm.max_map_count=262144
$ wget http://kuzzle.io/docker-compose.yml
$ docker-compose up

To persist this changes add this line to your /etc/sysctl.conf m.max_map_count=262144

Your terminal will show the log messages of Kuzzle’s components starting. After only a few seconds, you should see the following ready message appear:

kuzzle_1         | [✔] Kuzzle server ready

Your Kuzzle server is now ready to be used. For instance, you can hit the main HTTP API endpoint by browsing the page http://localhost:7512 or via cURL on the command line:

$ curl "http://localhost:7512/?pretty"

Kuzzle will respond you with a list of the existing routes.

Docker-compose will get the missing images for you on the first run, but it will not keep them up to date. To fetch the latest versions of all the images that compose the Kuzzle stack, you can use the following command:

$ docker-compose -f <docker-compose-file.yml> pull

Manually (on Linux)

In this section we will cover the manual installation on Linux systems, since this is the environment all the components of the Kuzzle stack work natively in.

We will run the Kuzzle stack using pm2, from the current user home directory.

Prerequisites

  • A Elasticsearch v5.x instance running on localhost:9200 (preferred version: v5.0, but v2.x is also supported).
  • A Redis v3.x instance running on localhost:6379 (preferred version: v3.2).
  • NodeJS v6.x or upper.
  • gcc and python. On Debian-based systems: sudo apt-get install build-essential python.

Step 1 - Retrieve Kuzzle components source code

  1. Create the Kuzzle root directory:
$ mkdir -p ~/kuzzle
$ cd ~/kuzzle
  1. Create a directory for Kuzzle Proxy and install it:
$ cd ~/kuzzle
$ git clone https://github.com/kuzzleio/kuzzle-proxy.git
$ cd kuzzle-proxy
$ npm install
  1. Create a directory for Kuzzle Core and install it:
$ cd ~/kuzzle
$ git clone https://github.com/kuzzleio/kuzzle.git
$ cd kuzzle
$ npm install
  1. Create a directory for Kuzzle Back Office and install it.

Step 2 - pm2

  1. Install pm2:
$ sudo npm install -g pm2
  1. Create a pm2 configuration file:
$ echo "apps:
   - name: kuzzle-proxy
     cwd: ${KUZZLE_PROXY_INSTALL_DIR}
     script: ${KUZZLE_PROXY_INSTALL_DIR}/index.js
   - name: kuzzle
     cwd: ${KUZZLE_CORE_INSTALL_DIR}
     script: ${KUZZLE_CORE_INSTALL_DIR}/bin/kuzzle
     args: start
     env:
       kuzzle_server__http__port: 7510
       kuzzle_services__proxyBroker__host: localhost
  " > ~/kuzzle/pm2.conf.yml
  1. Run Kuzzle via pm2 and show the logs:
$ pm2 start ~/kuzzle/pm2.conf.yml
$ pm2 logs

After only a few seconds, you will see the following ready message appear:

kuzzle_1         | [✔] Kuzzle server ready

The Kuzzle Back-office can be reached on http://localhost:3000.
Kuzzle HTTP API can be reached on http://localhost:7512/
Socket IO and Websocket channels can be reached over the HTTP server, on port 7512.

Change external services hosts or ports

If you are running some of the service(s) externally, you can configure their host and port using some environment variables and/or a .kuzzlerc file.

Please refer to Kuzzle configuration section for more information.

Running Kuzzle Backoffice

The Kuzzle Backoffice is a handy web application that helps you administrate Kuzzle. You can use it to manage your data, subscribe to realtime notifications and manage security rules.

You can use the publicly hosted Kuzzle Backoffice. If you want to host the Kuzzle Backoffice on your own server, you can download the source code here.

In both cases, you’ll be able to select the Kuzzle server you want to manage once the Backoffice starts up.

Select the Kuzzle server to connect to

The Kuzzle Backoffice automatically looks for a Kuzzle server on localhost:7512. If none is present, you will be prompted to choose a Kuzzle instance to connect to.

You can tell the Backoffice to connect to any Kuzzle server by clicking on the “Choose Environment” dropdown menu, then by selecting “Create new”.

Kuzzle Backoffice is trying to connect to a Kuzzle server

Create a new Backoffice environment by providing the address of the Kuzzle server you want to administrate. You can associate it with a custom name (e.g. “Development” or “My First Kuzzle”) and a color (e.g. red may be a good idea for production environments).

Create an admin account

At this point, Kuzzle is still pristine, which means that no admin account has been set-up: this means that the anonymous user has full rights on the server.

Kuzzle Backoffice prompts the creation of a first admin account

The Backoffice will prompt you with an admin account name and a password. Leave the “Reset anonymous account rights” unchecked, as we will use the anonymous account in the next steps of this tutorial.

Once you created the admin account, use its credentials to log-in.

Configuring Kuzzle

The complete default configuration of Kuzzle is stored in the kuzzlerc file at the root of the installation directory.

Kuzzle uses rc to override its default configuration. The most common ways to do it is:

  • via a .kuzzlerc file (example here);
  • via environment variables prefixed with kuzzle_.

Example 1: configuring Kuzzle via a custom .kuzzlerc file

You can write your custom config in $HOME/.kuzzlerc or any other valid location:

{
  "services": {
    "db": {
      "host": "<ES_HOST>",
      "port": <ES_PORT>
    },
    "proxyBroker": {
      "host": "<PROXY_HOST>"
    }
  }
}

Example 2: configuring Kuzzle via Environment Variables

The name of the environment variables must mimic the structure of the configuration object to override:

  • the variable needs to be prefixed with kuzzle_,
  • the __ correspond to the levels of nesting in the configuration object (e.g. kuzzle_services__proxyBroker__host corresponds to services.proxyBroker.host).
$ kuzzle_services__proxyBroker__host=<PROXY_HOST> node bin/kuzzle start

Environment variables are particularly handy to set your custom configuration through a Docker container. It is very easy to pass environment variables via the environment section of a docker-compose.yml file. Take a look at how we pass environment variables to Kuzzle in our default docker-compose file:

version: '2'

services:
  proxy:
    image: kuzzleio/proxy
    ports:
      - "7512:7512"

  kuzzle:
    image: kuzzleio/kuzzle
    depends_on:
      - proxy
      - redis
      - elasticsearch
    environment:
      - kuzzle_services__db__host=elasticsearch
      - kuzzle_services__internalCache__node__host=redis
      - kuzzle_services__memoryStorage__node__host=redis
      - kuzzle_services__proxyBroker__host=proxy
      - NODE_ENV=production

  redis:
    image: redis:3.2

  elasticsearch:
    image: elasticsearch:5.0

Working with persistent data

Kuzzle relies on Elasticsearch to store and fetch persistent data.

In Kuzzle, data is organized in the following way:

  • Documents are the atomic unit of data. They are defined as JSON structures, in the classical NoSQL fashion and are identified by a unique _id.
  • Documents are grouped into Collections, identified by a unique name.
  • Collections are grouped into Indexes, identified by a unique name.

Document CRUD

Kuzzle ships with a full data CRUD API that enables you to operate in many ways on your documents.

Let’s create a new document, for example, in mycollection, within myindex via the HTTP protocol. This is done by sending a POST request to the API endpoint http://localhost:7512/myindex/mycollection/_create with the body set to

{
  "message": "Hello, world!"
}

Notice that the document is associated to the auto-generated id AVkDBl3YsT6qHI7MxLz0, as we can see in the response:

{
  "status": 200,
  "error": null,
  "requestId": "38d08fa9-449d-47f7-8593-dc136f8b3559",
  "controller": "document",
  "action": "create",
  "collection": "mycollection",
  "index": "myindex",
  "metadata": null,
  "headers": {},
  "result": {
    "_index": "myindex",
    "_type": "mycollection",
    "_id": "AVkDBl3YsT6qHI7MxLz0",
    "_version": 1,
    "result": "created",
    "_shards": {
      "total": 2,
      "successful": 1,
      "failed": 0
    },
    "created": true,
    "_source": {
      "message": "Hello, world!",
      "_kuzzle_info": {
        "author": "-1",
        "createdAt": 1481814465050,
        "updatedAt": null,
        "updater": null,
        "active": true,
        "deletedAt": null
      }
    }
  }
}

Take some time to examine the content of a Kuzzle Response. You may notice that it contains useful information like the name of the controller and action that correspond to the HTTP route we hit with our request, or the complete KuzzleDocument object we just created.

One more thing you may notice is that myindex and mycollection are created on-the-fly along with the document. Let’s verify it by getting the list of collections stored in myindex by sending a GET request to http://localhost:7512/myindex/_list.

{
  "status": 200,
  "error": null,
  "requestId": "51b276c3-3698-4412-b3dc-80d0f84541fb",
  "controller": "collection",
  "action": "list",
  "collection": null,
  "index": "myindex",
  "metadata": null,
  "headers": {},
  "result": {
    "collections": [
      {
        "name": "mycollection",
        "type": "stored"
      }
    ],
    "type": "all"
  }
}

Take a look at the result field in the Response from Kuzzle. It contains an array of collections, each one defined by a name and a type. mycollection is of type stored (which stands for persistent). This is made to distinguish persisted collection from the realtime (or volatile) collections, used to identify real-time documents.

Let’s modify to our brand new document by sending a PUT request to http://localhost:7512/myindex/mycollection/AVkDBl3YsT6qHI7MxLz0/_update with the body set to:

{
  "message": "in a bottle",
  "an_englishman": "in New York"
}

Which gives us the response…

{
  "status": 200,
  "error": null,
  "requestId": "6241ec4d-8529-43ba-9b77-3028b99cd621",
  "controller": "document",
  "action": "update",
  "collection": "mycollection",
  "index": "myindex",
  "metadata": null,
  "headers": {},
  "result": {
    "_index": "myindex",
    "_type": "mycollection",
    "_id": "AVkDBl3YsT6qHI7MxLz0",
    "_version": 2,
    "result": "updated",
    "_shards": {
      "total": 2,
      "successful": 1,
      "failed": 0
    }
  }
}

…telling us that the document has been successfully updated.

Now, we’ll let you figure out what happens when we send a DELETE request to http://localhost:7512/myindex/mycollection/AVkDBl3YsT6qHI7MxLz0 with an empty body (take a look at the API Reference if you don’t want to try).

One thing that Elasticsearch is really good at doing is… Searching! It enables to create extremely precise search queries, thanks to its powerful query DSL. We wrote a comprehensive cookbook to help you understand how it works in detail, but let’s take a look at a couple of simple examples, just to get started.

Say we want to find all the documents within mycollection, via the HTTP protocol. To do it, we send a POST request to http://localhost:7512/myindex/mycollection/_search (we leave the body empty since we have no filters to apply to our query). Depending on the documents you have created in your database, the response will look like:

{
  "status": 200,
  "error": null,
  "requestId": "3b486d49-f1f9-4595-8c10-b63cc5fc1279",
  "controller": "document",
  "action": "search",
  "collection": "mycollection",
  "index": "myindex",
  "metadata": null,
  "headers": {},
  "result": {
    "took": 69,
    "timed_out": false,
    "_shards": {
      "total": 5,
      "successful": 5,
      "failed": 0
    },
    "hits": [
      {
        "_index": "myindex",
        "_type": "mycollection",
        "_id": "AVkDLAdCsT6qHI7MxLz4",
        "_score": 0,
        "_source": {
          "message": "Hey! Ho!"
        },
        "_kuzzle_info": {
          "author": "-1",
          "createdAt": 1481816934209,
          "updatedAt": null,
          "updater": null,
          "active": true,
          "deletedAt": null
        }
      },
      {
        "_index": "myindex",
        "_type": "mycollection",
        "_id": "AVkDK9iNsT6qHI7MxLz3",
        "_score": 0,
        "_source": {
          "message": "Hello, world!"
        },
        "_kuzzle_info": {
          "author": "-1",
          "createdAt": 1481816922252,
          "updatedAt": null,
          "updater": null,
          "active": true,
          "deletedAt": null
        }
      },
      {
        "_index": "myindex",
        "_type": "mycollection",
        "_id": "AVkDLCdRsT6qHI7MxLz5",
        "_score": 0,
        "_source": {
          "message": "Let's go!"
        },
        "_kuzzle_info": {
          "author": "-1",
          "createdAt": 1481816942415,
          "updatedAt": null,
          "updater": null,
          "active": true,
          "deletedAt": null
        }
      }
    ],
    "total": 3,
    "max_score": 0
  }
}

Looks neat. Say that now we only want to query the documents containing the word Hey in the message field. We can achieve this by adding the following query to our body:

{
  "query": {
      "match": {
          "message":"Hey"
      }
    }
}

Which gives, as a result, the following response:

{
  "status": 200,
  "error": null,
  "requestId": "e00cf6d6-8983-498b-8481-96a1fe1b5d46",
  "controller": "document",
  "action": "search",
  "collection": "mycollection",
  "index": "myindex",
  "metadata": null,
  "headers": {},
  "result": {
    "took": 6,
    "timed_out": false,
    "_shards": {
      "total": 5,
      "successful": 5,
      "failed": 0
    },
    "hits": [
      {
        "_index": "myindex",
        "_type": "mycollection",
        "_id": "AVkDLAdCsT6qHI7MxLz4",
        "_score": 0.25811607,
        "_source": {
          "message": "Hey! Ho!"
        },
        "_kuzzle_info": {
          "author": "-1",
          "createdAt": 1481816934209,
          "updatedAt": null,
          "updater": null,
          "active": true,
          "deletedAt": null
        }
      }
    ],
    "total": 1,
    "max_score": 0.25811607
  }
}

Where do we go from here?

Data Validation

One common need when you are dealing with data is validation. Every time you create new documents, update them or publish real-time messages, you may want to check that their content meets a given set of criteria.

The most common example is the “e-mail field validation”. Imagine you provide a registration form to your users to collect their name and e-mail. It’s quite vital to you that the user provides a valid e-mail address.

Instead of leaving you with the burden of coding this logic, Kuzzle natively provides a way to validate your data input against a validation schema. It enables you to define the set document fields to validate and provides you with an extended set of validation rules.

This way, every time Kuzzle receives a data input, it checks it against the validation schema and returns a standard error when the validation fails.

You can specify the validation rules in the kuzzle configuration file in the validation field.

You can take a look at the Kuzzle Data Validation Reference for a straight dive, or keep reading for a smoother introduction.

Basic validation

The place to specify your validation schema is the validation field in your kuzzlerc file. A validation schema has a hierarchical structure, where you specify a set of rules per collection.

validation: {
    "onlineshop": {
        "products": {
            "id": {
                "mandatory": true,
                "type": "number"
            },
            "productDescription": {
                "type": "string",
                "defaultValue": "Sorry, no description available for this product."
            }
        }
    }
}

Let’s take a look at what we just did here.

  • We defined a set of rules for the documents contained in the products collection, within the onlineshop index.
  • We specified the field id as mandatory (which means that it must have a value) and of type Number.
  • We specified the field productDescription as of type String, with a sorry-ish default value.

Take a look at the Validation Fields Reference for a complete insight of all the available specifications.

Complex validation via the DSL

When the validation fields are not enough for your need, you can switch gears and create a complex validation specification via the filtering DSL (the same DSL used to create real-time subscriptions). The idea is pretty simple: you specify a filter that documents must match in order to be valid.

validation: {
    "onlineshop": {
        "products": {
            "validators": [
                // Here goes the filter
                "range": {
                    "price": {
                        "gte": 0
                    }
                }
            ]
        }
    }
}

In the example above, we specified that the value of the attribute price (of documents contained in the products collection) must be greater than 0 (because we do not want to make an online shop that gives products away). We leveraged the range term, that you can look-up in the Real-time filters Reference.

You can take a look at the Kuzzle Data Validation Reference for deeper insight.

Real-time notifications

Besides persisting data and retrieving it via advanced searches, Kuzzle enables you to set up live subscriptions to any set of data.

Live subscriptions are great to keep track of the evolution of a portion of your data you are interested in.

Introduction

Imagine you are developing a collaborative TO-DO list application. All the TO-DO items are persisted in Kuzzle (in a collection called todos) so, once the clients start, they fetch all the items via a simple document search.

But imagine that one of the users (let’s call her Ann), adds a new TO-DO item. In order for other users (let’s call them Tom and Matt) to display the new item, they need to perform a new document search on the todos collection. They will not see the new item until they refresh (or restart) their application.

This cannot be called a “modern” application: it rather looks like an old-school, refresh-ish, one. Like the early ‘90s. Today, such a user-experience wouldn’t be satisfying at all.

A more interesting user-experience would be that clients display the new TO-DO item as soon as it is created. How could we achieve that?

  • By implementing a long-polling mechanism in the clients. Every, say, one second, the clients perform a document search and update their list of TO-DO items. Doesn’t look like a great idea (performances would be rather bad, for example).
  • By introducing a pub/sub mechanism that enables the backend to push the new item to the clients as soon as it is created (looks awesome, doesn’t it?)

The second solution is exactly what we are looking for and Kuzzle ships it natively. We can call it pub/sub, real-time notifications or live subscriptions and it is often used to solve use-cases like this one, where things need to be kept in sync between the client and the server.

Our collaborative TO-DO list clients would subscribe to the todos collection (right after the first document search) in order to be notified in real-time about new TO-DO items. This way, once Ann creates her new item, Tom and Matt see it immediately on their screen.

Concepts

Real-time notifications are triggered by the pub/sub mechanism embedded in Kuzzle and follow the Observer/Observable pattern, in which Kuzzle is the Observable and the client is the Observer.

They are is possible only when the communication happens in a persistent-connection-oriented protocol (like Websockets or MQTT) and therefore not in HTTP.

Clients can subscribe to many types of notifications. Below are some examples:

  1. new documents being created in a given collection (e.g. Ann creates a new TO-DO item);
  2. documents being deleted from a given collection (e.g. Tom deletes a TO-DO item);
  3. changes happening on any document within a collection (e.g. Matt checks an item as “done”);
  4. changes happening on a given set of documents (e.g. clients must play a sound every time an item containing the word “URGENT” is created).

The scope of possibilities is huge. Take a look at the Notifications section in the API Reference for more details.

Examples

But, how does this work in Kuzzle? How do we select the data that we want to subscribe to?

Let’s dive into the implementation of the Collaborative TO-DO list application.

The basic subscription

Once our client has started and initialized with the set of TO-DO items it fetched from the persistence layer, we want it to subscribe to the changes happening on them.

kuzzle
    .collection('todos')
    .subscribe({}, (error, notification) => {
        if (error) {
            throw new Error(error)
        }
        console.log('Something happened and we should do something.', notification)
    })

This code isn’t very useful but it shows the capability to react to a notification coming from the server.

Here, we call the subscribe method on the todos collection with two arguments:

  • The first argument represents the filters, and in this case there’s none, which means that we are subscribing to all the documents in the collection. Filters enable more fine-grained selection on the data we want to subscribe to and are described in the next example.
  • The second argument is the callback, i.e. a function that is called every time a notification is received.

Now, imagine this code is executed on Tom’s client: when Ann creates the new TO-DO item, Tom receives a notification looking like the following:

{
  "status": 200,
  "error": null,
  "index": "<data index>",
  "collection": "<data collection>",
  "controller": "document",
  "action": "create",
  "state": "done",
  "scope": "in",
  "metadata": {},
  "requestId": "<unique request identifier>",
  "result": {
    "_id": "fswdfsrt43e6t2q3rfw",
    "_source": {
      "label": "The new item Ann just created!",
      "checked": "false"
    }
  }
}

The Notification bears some useful information about what just happened:

  • the controller and action attributes show what has happened;
  • the index and collection attributes show where it happened;
  • the result shows the consequence of what just happened (in this case, the newly created document).

We won’t analyze the other attributes for the moment. Take a look at the Notifications section of the API Reference for a comprehensive list of the available notification events.

This subscription is very handy and will notify Tom about the events 1, 2 and 3 of the list above (the controller, action and result will vary depending on the case). But what about the event number 4? How does Tom subscribe to items that only contain the word URGENT in their label field? Looks like a job for the Real-time Filtering DSL coming up in the following section.

Subscription with filters

Kuzzle ships with a powerful Filtering DSL for Live Subscriptions. It is heavily inspired in the Elasticsearch DSL and enables you to perform fine-grained selections on the documents you want to subscribe to.

In our case, we want to select all the documents that contain the URGENT word in the label field. The best pick for this case is the regexp filter.

kuzzle
    .collection('todos')
    .subscribe({
        regexp: {
          label: '.*URGENT.*'
        }
      }, (error, notification) => {
        if (error) {
            throw new Error(error)
        }
        console.log('Something happened and we should do something URGENTLY.', notification)
    })

This way, Tom will be notified about urgent TO-DO items. Take a look at the Filtering DSL Refernce for a comprehensive list of the available filters.

There are a few things that deserve to be noticed here:

  • Tom will be notified either if somebody creates, updates or deletes a document containing the word URGENT;
  • Tom will be notified even if he performs the actions himself (e.g. he is notified right after having created a new TO-DO item).

The last point may seem a little bit inconvenient. What if Tom does not want to receive notifications when the event come from his own actions? Keep reading, the solution is right below.

Subscription with options

The subscribe method can be called with an extra argument, which is an object containing a set of options to be passed to the subscription Room.

We just introduced a new concept here, the Room. A Room is a class representing a single subscription and its constructor is called internally by the subscribe method. The “options” are passed directly to the Room Constructor.

The option we are looking for is subscribeToSelf, which is set to true by default.

let room = kuzzle
    .collection('todos')
    .subscribe(
      { // The Filters object
        regexp: {
          label: '.*URGENT.*'
        }
      },
      { // The Options object
        subscribeToSelf: false
      },
      (error, notification) => { // The callback
        if (error) {
            throw new Error(error)
        }
        console.log('Something happened and we should do something URGENTLY.', notification)
    })

In the code right above, we added the extra “options” object as the second argument to avoid subscribing Tom to his own events.

You may have noticed that, on the very first line, we stored the return value of the subscribe method in a variable. Guess what type is its value? Room.

Security

Kuzzle provides a full set of functionalities to finely define the permissions for your data.

Fresh installation default rights.

When installing Kuzzle for the very first time, no default user is created and the Anonymous user is allowed to perform any action on the data. The only restriction is on the internal data storage used by Kuzzle to store its configuration.

Once a first admin user is created, either via the Kuzzle Back Office or the CLI, the Anonymous permissions are dropped.

You can then use the Back Office to administrate your user rights.

Authentication

The first step to secure your data is to be able to identify your users. Kuzzle ships by default with a local login/password strategy.

If the “local” strategy (i.e. storing the users’ credentials in the local database) doesn’t fit your needs, you can use the Oauth authentication plugin, or develop your own (see Core documentation for more details).

If the authentication request identifies an existing user, Kuzzle generates a JSON Web Token that must be appended to all the subsequent requests.

Permissions

Once you know who is connected, you need a way to grant your users with some privileges to control their access to data.

Users, profiles and roles

Kuzzle associates users to a profile.
You can think to a profile as a user group. All the users that share the same profile will get the same privileges.

Because some sets of permissions can be shared between several profiles, Kuzzle includes an additional level of abstraction below the profile: the roles.

A profile is associated to a set of roles. Each role defines a set of permissions.

Users, profiles and roles

In the simple example above, the editor profile is a superset of the contributor one, which, in turn, extends the default profile.

roles and profiles can be edited in Kuzzle Back Office.

Role definition

A role definition is a hierarchical JSON object in which permissions can be defined at controller and action level.

The role definition is represented as a Javascript object. Each key at the root of the object identifies a controller by its name.

var myRoleDefinition = {
  controllers: {
    < controller|* >: {
      actions: {
        < action|* >: < action permission* >,
        < action|* >: < action permission* >,
        ...
      }
    }
  }
};

Each entry of the controllers, actions tree can be set to either an explicit value or the “*” wildcard.

The action permission value can be set either to:

You can find a comprehensive summary of all the available controllers and actions by sending a GET request to the root endpoint of the Kuzzle API via the HTTP protocol:

$ curl -XGET http://localhost:7512/\?pretty\=true

Take a look at the example below:

var anonymousRole = {
    controllers: {
    auth: {
      actions: {
        login: true,
        checkToken: true,
        getCurrentUser: true
      }
    }
  }
};

The example above is the permission definition set by Kuzzle for the Anonymous user after the first admin user has been created.

In this example, the role grants the user with the permission to perform the login, checkToken and getCurrentUser actions of the auth controller.

Profile definition

A profile definition is a Javascript object that contains an array of roles, identified by their IDs:

var myProfileDefinition = {
  roles: [
    {_id: < role Id > (, restrictedTo: < role restrictions > ) },
    <another role>,
    ...
  ]
};

A role can be applied globally on the profile, or it can be restricted to a list of indexes or index/collections pairs.

For example, if we have a “publisher” role which allows to request any action of the write controller:

var publisherRole = {
    controllers: {
    write: {
      actions: {
        '*': true
      }
    }
  }
};

Then we declare 3 profiles using this role:

var profile1 = {
  roles: [ {_id: 'publisherRole' } ]
};

var profile2 = {
  roles: [
    {
      _id: 'publisherRole',
      restrictedTo: [{index: 'index1'}]
    }
  ]
};

var profile3 = {
  roles: [
    {
      _id: 'publisherRole',
      restrictedTo: [
        {index: 'index1', collections: ['foo', 'bar']},
        {index: 'index2'}
      ]
    }
  ]
};

With this sample profiles:

  • users with profile1 are allowed to use all write controller actions on all indexes and collections.
  • users with profile2 are only allowed to use write controller actions on collections stored in index index1.
  • users with profile3 are only allowed to use write controller actions on:
    • all collections stored in index index2
    • collections foo and bar stored in index index1.
Composition rules

In Kuzzle, permissions follow the Whitelist strategy, which means that an action must be explicitly allowed by at least one role of the user profile (including restrictions).

That means: * If a role allows it, the action is authorized, even if another role denies it. * If no role explicitly allows it, the action if denied, even if no role explicitly denies it as well.

Actions permissions functions

So far, we’ve seen how to set permissions based on the user profile only.

Now, let’s say we have a chat application and want the users to be allowed to edit & delete their own messages only.

This type of rules depends on the context and cannot be expressed as a simple boolean.

Kuzzle lets you define more complex permissions using custom functions, allowing dynamic decision about whether the user is allowed to proceed or not, depending on the context.

In our chat example, the rule can be expressed as:

var role = {
  controllers: {
    write: {
      actions: {
        create: true,
        delete: {
          args: {
            document: {
              index: "$requestObject.index",
              collection: "$requestObject.collection",
              action: {
                get: "$currentId"
              }
            }
          },
          test: "return args.document.content.user.id === $currentUserId"
        }
      }
    }
  }
};

See more details at Core documentation section

Request and Response format

All transactions in Kuzzle are represented by a Request object. The object is created by the client to send a request to Kuzzle and returned by Kuzzle containing the response.

The state of this object evolves along with the lifecycle of the transaction.

The Request object is sealed, which means you cannot add or delete fields once the object is initialized.

Let’s take a look at the structure of the Request object.

Request {
    // members
    status,     // {integer}        The status of the transaction (matches HTTP codes)
    timestamp,  // {integer}
    id,         // {string}         
    context,    // {RequestContext} Contains information about the connection and the current token & user
    input,      // {RequestInput}   The request input params sent by the client
    result,     // {Object}         The raw result sent by the controller (defaults to null)
    error,      // {KuzzleError}    Defaults to null
    response,   // {property}       The final response sent out of Kuzzle (enumerable, get-only property)

    // methods
    setError (error),               // Sets the status to the error.code and fills the error member.
    setResult (obj, [status=200])   // Sets the result and the status code.
}

Let’s take a look at the attributes of this object.

  • The id attribute bears a unique, auto-generated value that identifies the transaction.
  • The timestamp attribute stores the creation date (in seconds after the Epoch time).

Input

The input field contains all the parameters that express the request from the client. It has the following structure:

RequestInput {
    // members
    args,           // {Object}     Parametric arguments. i.e. for REST, taken from the query string
    metadata,       // {Object}
    body,           // {Object}     Content of the resource for REST like routes, main parameters for others
    controller,     // {string}
    action,         // {string}
    resource {
        index,
        collection,
        _id
    }

    // methods
    constructor (data) // data is a JS Object that has the same structure as the Websocket message
}

Context

The context attribute contains information about the state of the connection at the moment the request is sent. It has the following structure:

RequestContext {
    connectionId,   // {scalar}     Unique identifier of the user connection
    protocol,       // {string}
    token,          // {Token}      Auth token
    user            // {User}       Represents the current User associated to the transaction
}

Status codes

The status attribute is a numeric code similar to HTTP status codes. It is used to inform the client about the real status of his request (if an error occurred or not).

List of status codes supported by Kuzzle

1xx Processing
  • 102: Standard status for an unhandled request.
2xx Success
  • 200: Standard status for a successful request.
  • 206: The request (typically a bulk import) is partially successful, but some actions encountered an error. (in this case, error details are returned within error.errors)
4xx Client Error
  • 400: The request is malformed (usually: an argument is missing).
  • 403: The client is not allowed to perform the requested action.
  • 404: The requested resource cannot be found.
5xx Server Error
  • 500: Kuzzle encountered an unexpected error (standard code for internal error).
  • 503: An external service is unavailable

Error objects format

When an error occurred, the error attribute contains KuzzleError object, which inherits from the primitive Javascript Error type.

Life-cycle

Here is how it works.

  • The client initializes the object with the arguments passed to the constructor.
    • The status field is always initialized to 102 (“processing”).
    • The context field is initialized with the connection status and ID.
    • The input field is initialized with the request parameters specified in the options.
    • The fields error, result and response contain null.
  • Kuzzle receives the response. The corresponding controller handles it according to the input field.
    • The raw response of the controller is set to the result field.
    • If an error occurs, Kuzzle updates the error field via the setError method.
    • The status field is update consequently with a HTTP-compliant numeric code.
    • Kuzzle fills the response field with an object compliant with the Kuzzle Response API standard
  • Kuzzle sends the response back to the client.

Command Line Interface

Kuzzle ships with a Command line interface which enables you to:

  • start and stop Kuzzle Core,
  • install and configure plugins,
  • create the first administrator user,
  • reset Kuzzle internal data (use with caution !).
$ ./bin/kuzzle

  Usage: kuzzle [options] [command]


  Commands:

    createFirstAdmin          create the first administrator user
    clearCache                clear internal caches in Redis
    plugins [options] [name]  manage plugins
    reset [options]           delete Kuzzle configuration and users from database
    start [options]           start a Kuzzle instance
    dump                      create a dump of current state of Kuzzle

  Options:

    -h, --help      output usage information
    -V, --version   output the version number
    -d, --debug     make errors more verbose
    -C, --noColors  do not use ANSI coloring

createFirstAdmin

$ ./bin/kuzzle createFirstAdmin

When Kuzzle runs for the first time, no users are defined and the anonymous user is granted with super-admin rights.

The createFirstAdmin command lets you define an administrator user and set your own permissions.

clearCache

$ ./bin/kuzzle clearCache

Kuzzle relies on the Redis service to store some frequently accessed internal data. If you need to restart Kuzzle with a fresh cache, this command can come in hand.

reset

$ ./bin/kuzzle reset --help

    Usage: reset [options]

    delete Kuzzle configuration and users from database

    Options:

      -h, --help             output usage information
      --fixtures <fixtures>  import some fixtures from file
      --mappings <mappings>  load and apply mappings from file
      --noint                non interactive mode

The reset command deletes all currently set configurations and users from the database.

If business data were imported in Kuzzle’s database layer, these are kept intact.

start

$ ./bin/kuzzle start --help

    Usage: start [options]

    start a Kuzzle instance

    Options:

      -h, --help                 output usage information
      -p, --port <port>          Kuzzle port number
          --fixtures <fixtures>  import some fixtures from file
          --mappings <mappings>  load and apply mappings from file

The start command starts a Kuzzle server in the foreground.

dump

$ ./bin/kuzzle dump --help
[ℹ] Creating dump file...
[✔] Done!

[ℹ] Dump has been successfully generated in "dump/20161214-1555-cli" folder
[ℹ] You can send the folder to the kuzzle core team at support@kuzzle.io

The dump command creates a snapshot of the state of Kuzzle. It includes * a core dump of Kuzzle Core, * a dump of the main configuration, * a dump of all the server logs, * a dump of the Node.js engine properties, * a dump of the OS properties, * a dump of the plugins configuration, * a dump of the usage statistics of the Kuzzle Server.

This can be particularly handy to feed a crash report to the support team if you own a Kuzzle License.

The Plugin System

Being able to grab a full-feature backend and run it quickly is very convenient, but Kuzzle is about more than that. As soon as you start developing real-life applications that live out in the wild, you need to extend your backend with your own business-logic. That’s quite legitimate: some logic is sensible and cannot live on the client.

For example, imagine your application needs to leverage a third-party payment system, such as Braintree. You don’t want to put the client in charge of bearing API keys and tokens. Also, you will probably want to keep track of orders and payments, once the transactions successfully end. This should be also a concern of your backend. Also, you are not likely to allow your customers to purchase more items than available, right? At some point, your backend has to validate the transaction by comparing the ordered quantity with the available stocks.

To solve this problem, Kuzzle ships with a powerful Plugin System that enables you to add features to your server in a modular fashion. With Kuzzle Plugins you can

Plugin Types

There are three main Plugin types, each one has specific features.

Core Plugins

They are the most common type of plugins and are meant to extend the Kuzzle Core features. They are plugged on the Kuzzle Core at startup and execute on its same thread. A Core Plugin can extend Kuzzle with the following features:

Listen asynchronously, and perform operations that depend on data-related events. The chunk of data involved with the event is passed to the triggered callback, but the Kuzzle Core continues its execution without waiting for the callback to return, nor receiving the return value of the callback.

Example - “Write a log to a third-party log system every time a document is deleted”. The Logger Plugin, shipped with Kuzzle, uses this feature to log all the data-related events.

Listen synchronously, and perform operations that depend on data-related events. many synchronous listeners can be chained, forming a pipeline. The chunk of data involved with the event is passed to the plugin pipeline and can be modified. The Kuzzle Core waits for the pipeline to return and receives the returned value (which must be the processed chunk of data). The pipeline can even stop a request life-cycle, returning a standard Error to the Kuzzle Core.

Example - “Compare the ordered quantity with the available stocks and return an error if ordered is greater than stocks”.

Add a controller route to expose new actions to the API.

Example - “Expose a checkout API endpoint that handles the Braintree payment process”.

Add an authentication strategy the User authentication system.

Example - “Enable Kuzzle to authenticate users via the OAuth strategy” Kuzzle ships with an Authentication Plugin already bundled in its Community Edition, the Local Strategy Plugin.

Worker Plugins

Workers Plugins are Core Plugins that run on a separate process. The only feature they can add is to asynchronously listen to data-related events. They are useful when performing costly operations as they have no impact on Kuzzle performances.

Example - “Compute a complex data-mining operation and commit the result to a third-party Business-Intelligence platform every time a document is changed”.

Protocol Plugins

Protocol Plugins extend Kuzzle networking capabilities by adding new network protocols.

Example - “Enable Kuzzle to interact with XMPP-oriented services” Kuzzle ships with a Protocol Plugin already bundled in its Community Edition, the Websocket Plugin.

Examples

This example is a very dummy one. We are going to install the Hello World Controller Plugin, which opens a new API endpoint that simply… Greets the client! Not very useful, except for the sake of this example.

To install a Plugin, you just need to make it accessible within the plugins/enabled directory (relative to the path of the Kuzzle installation directory). A common practice is to copy the Plugin code in plugins/available and create a symbolic link in plugins/enabled pointing to it. This way, enabling and disabling a plugin is just a matter of creating or deleting a symbolic link.

Go to the Kuzzle installation directory and type:

$ cd plugins/available
$ git clone https://github.com/kuzzleio/kuzzle-plugin-helloworld.git
$ cd ../enabled
$ ln -s ../available/kuzzle-plugin-helloworld .
$ pm2 restart KuzzleServer

Once Kuzzle has restarted (that’s what the last command is for) you can check the server information at http://localhost:7512/?pretty=true for the new hello endpoint:

{
    ...
    "result": {
        "serverInfo": {
            "kuzzle": {
                ...
                "api": {
                    "routes":
                    ...
                    "kuzzle-plugin-helloworld/hello": {
                      "sayHello": {
                        "name": "sayHello"
                      }
                    }
                }
            }
        }
    }
}

Which means you can call the http://localhost:7512/kuzzle-plugin-helloworld/hello/ route and get the following response:

{
  "requestId": "66265614-e908-4a91-a492-135e40e64aa3",
  "status": 200,
  "error": null,
  "controller": "kuzzle-plugin-helloworld/hello",
  "action": "sayHello",
  "collection": null,
  "index": null,
  "metadata": null,
  "result": "Hello world"
}

To get a deeper insight on how Plugins work in Kuzzle, please refer to the Plugin Reference.

Managing Plugins

As seen in the previous example, managing Plugins is really just a matter of moving files and symbolic links around.

Installing, removing, enabling and disabling Plugins

When starting, Kuzzle (both Core and Proxy) looks for Plugins in the plugins/enabled directory. Valid Plugins are well-formed NPM modules (i.e. they have a package.json at their root) or simple NodeJS requireables (i.e. they have at their root a valid index.js).

A common practice is to copy the Plugin code in plugins/available and create a symbolic link in plugins/enabled pointing to it. This way, enabling and disabling a plugin is just a matter of creating or deleting a symbolic link.

Configuring Plugins

When initializing a Plugin, Kuzzle (both Core and Proxy) calls its init(customConfig, context) method, passing the context and the custom configuration. Custom configuration parameters are specified for each plugin in the plugins object of the Kuzzle configuration file.

{
  "plugins": {
    "kuzzle-plugin-foobar": {
      "option_1": "option_value",
      "option_2": "option_value"
    }
  }
}

Each Plugin is responsible of handling the custom configuration parameters and Kuzzle has no opinion on how to do it. Whether the custom configuration is merged with the defaults or not entirely depends on the implementation of the init function.

Kuzzle in depth

In this section we’ll take a deeper look at the Kuzzle Core internals.

archi_core

Components

The above schema shows the main architecture in Kuzzle, which is composed of the following entities.

  • Kuzzle Proxy: handles the communication beetween the client and Kuzzle (see Connecting to kuzzle), and forwards the input message to the Router.
  • Router: exposes the API routes, normalizes the Request and sends them to the Funnel.
  • Funnel: analyses the Request and forwards it to the appropriate Controller.
  • Controllers: handle the Request (see API reference) and return a response (or an error).
  • Internal Components: Any component internally accessed by controllers.
  • Service Components: Any component used to interact with external services (see below).

Services

In Kuzzle, a Service module is the implementation of the interface to different components of the application (think of a system service).

Kuzzle currently implements the following Services:

A Service can be added to different engines. For example, Redis is used by both the internalCache and the memoryStorage (see default.config.js).

Request Life-Cycle

In this section we are going to focus on how Requests flow between Kuzzle components. We are going to analyze the life-cycle of a Request in order to understand in depth the Kuzzle Core architecture.

Reading content from Kuzzle

By “reading”, we mean any action involving getting content from the persistent layer: getting a single document, count documents, or search contents with advanced filters.

HTTP Request

The schema below shows the Architecture overview showed above and highlights the components involved in reading actions:

read_scenario_http_overview

The following diagram shows how the Request flows between the client application, the different Kuzzle components, and the external services:

read_scenario_http_details

  • The HTTP client asks for a document via a HTTP GET Request. For instance, to retrieve the document ‘739c26bc-7a09-469a-803d-623c4045b0cb’ in the collection users: GET http://kuzzle:7512/mainindex/users/739c26bc-7a09-469a-803d-623c4045b0cb.
  • The proxy forwards the Request through the HTTP Entry point to the Router, which handles it and forwards the formatted Request to the Funnel.

The formatted Request input looks like the following:

{
  "controller": "read",
  "action": "get",
  "resource": {
    "index": "mainindex",
    "collection": "users",
    "_id": "739c26bc-7a09-469a-803d-623c4045b0cb"
  }
}
  • The Funnel Controller validates the data before sending the request to the Document Controller.
  • The Document Controller calls the Read Engine service.
  • The Read Engine service performs an HTTP request to get the data from the data storage.

The content returned by Elasticsearch looks like the following:

{
  "_index": "mainindex",
  "_type": "users",
  "_id": "739c26bc-7a09-469a-803d-623c4045b0cb",
  "_version": 1,
  "found": true,
  "_source": {
      "firstName": "Grace",
      "lastName": "Hopper",
      "age": 85,
      "location": {
          "lat": 32.692742,
          "lon": -97.114127
      },
      "city": "NYC",
      "hobby": "computer"
  }
}
  • Promises functions are resolved to forward the response message back to the HTTP Router.
  • The HTTP Router sends the response to the HTTP client.

Websocket connection

The schema below shows the Architecture overview showed above and highlights the components involved in reading actions:

read_scenario_websocket_overview

The following diagram shows how the Request flows between the client application, the different Kuzzle components, and the external services:

read_scenario_websocket_details

  • The client application opens a Websocket connection to Kuzzle Proxy and emits a “read” event containing the request. For instance, to retrieve the document 739c26bc-7a09-469a-803d-623c4045b0cb in the collection users:
{
  "requestId": "ed4faaff-253a-464f-a6b3-387af9d8483d",
  "action": "get",
  "collection": "users",
  "_id": "739c26bc-7a09-469a-803d-623c4045b0cb"
}

The client then listens to the <requestId> event on the socket, like the following:

  this.socket.once("ed4faaff-253a-464f-a6b3-387af9d8483d", function(response) {
    callback(response);
  });
  • The Kuzzle Websocket plugin handles the Request and forwards the message to the Backend Broker.
  • The Backend Broker sends the message to the Proxy Broker (within the Kuzzle Core) through a Websocket connection.
  • The Proxy Broker forwards the Request through the Proxy Entry Point to the Router, who handles it and forwards the formatted Request to the Funnel.

The formatted Request input looks like the following:

{
  "controller": "read",
  "action": "get",
  "resource": {
    "index": "mainindex",
    "collection": "users",
    "_id": "739c26bc-7a09-469a-803d-623c4045b0cb"
  }
}
  • The Funnel validates the message and forward the request to the Document Controller.
  • The Document Controller forwards the Request input to the Read Engine service.
  • The Read Engine service performs an HTTP request to get the data from Elasticsearch.

The content returned by Elasticsearch looks like the following:

{
  "_index": "mainindex",
  "_type": "users",
  "_id": "739c26bc-7a09-469a-803d-623c4045b0cb",
  "_version": 1,
  "found": true,
  "_source": {
      "firstName": "Grace",
      "lastName": "Hopper",
      "age": 85,
      "location": {
          "lat": 32.692742,
          "lon": -97.114127
      },
      "city": "NYC",
      "hobby": "computer"
  }
}
  • Promises functions are resolved to forward the response message back to the Proxy Broker.
  • The Proxy Broker sends the response to the proxy’s Backend Broker through the Websocket connection.
  • The Proxy triggers the callback, which emits a <requestId> event to the Websocket client.

Subscribing and writing content to Kuzzle

This section explains what happens when clients send new content to Kuzzle.

Kuzzle is able to handle two different types of input:

  • persisted data, via the _create_, _createOrUpdate_, or _delete_ actions.
  • real-time data, via the _publish_ action.

Writing persistent data

This subsection describes the process for persistent data, with an example using the “create” action (see also API Documentation).

persistence_overview

Detailed workflow:

persistence_scenario_details

  • A client sends new content to Kuzzle, either via an HTTP request, through a Websocket connection or using a custom plugin protocol.
  • The router handles the Request and forwards the message to the Funnel.
{
  "status": 102,
  "id": "...", // ... The connection id
  "context": {
    // ... The connection context
  },
  "input": {
    "resource": {
      "index": "mainindex",
      "collection": "users"
    },
    "controller": "document",
    "action": "create",
    "body": {
      "firstName": "Grace",
      "lastName": "Hopper",
      "age": 85,
      "location": {
        "lat": 32.692742,
        "lon": -97.114127
      },
      "city": "NYC",
      "hobby": "computer"
    }
  }
}
  • The Funnel validates the Request and triggers the Plugins Manager with a document:create event. The Plugins Manager calls all pipes and hooks configured by the active plugins (see the Plugin Reference).
  • The Funnel forwards the Request to the Document Controller.
  • The Document Controller sends the request to the Storage Engine service. The Storage Engine sends the request to the database.
  • Once the Storage Engine gets the response back, it replies to the Document Controller.
  • The Document Controller wraps the response in a Kuzzle Response and forwards it back to the user.

Subscribe and Notification scenario

This subsection describes the life-cycle of non persistent data notifications as well as real-time notifications, implementing the Publish/Subscribe pattern.

Remember the Architecture overview and focus on the components involved by pub/sub actions: pubsub_overview

1st step: subscription

The following diagram shows how two different clients, a Websocket and a MQ one, subscribe to data.

pubsub_scenario_details1

  • The client application opens a Websocket or a MQ connection and emits a “subscribe” event with some filters (see the API Documentation). For instance, to be notified about all contents posted to the collection users, containing a field hobby equals to computer:
{
  "requestId": "ed4faaff-253a-464f-a6b3-387af9d8483d",
  "index": "mainindex",
  "collection": "users",
  "action": "on",
  "body": {
    "equals": {
      "hobby": "computer"
    }
  },
  "state": "all"
}

See the Real-time Filters Reference for more details.

The client then listens to the <requestId> event on the socket. Kuzzle will get back to him with a corresponding Room ID and a Room Channel using this event.

Sample Javascript code, using Websocket:

  this.socket.once("ed4faaff-253a-464f-a6b3-387af9d8483d", function(response) {
    callback(response);
  });
  • The Router interprets the input request and transfer the subscription message to the Funnel.
{
  "index": "mainindex",
  "collection": "users",
  "controller": "subscribe",
  "action": "on",
  "filter": {
    "equals": {"hobby": "computer" }
  }
}
  • The Funnel validates the message and transfers it to the Realtime Controller.
  • The Realtime Controller calls the HotelClerk internal component to create the subscription.
  • The HotelClerk calls the DSL component to get a formated filter related to the subscription (see DSL Readme for more details).
  • The HotelClerk creates a channel related to the filters and gives it back to the Realtime Controller.
  • The channel is sent back to the Websocket (or MQ) Router through the internal components.
  • The Websocket (or MQ) Router emits a <requestId> event to the client, containing the subscribed channel ID.

Sample response content:

{
  "status": 200,
  "error": null,
  "index": "mainindex",
  "collection": "users",
  "controller": "subscribe",
  "action": "on",
  "state": "all",
  "requestId": "ed4faaff-253a-464f-a6b3-387af9d8483d",
  "result": {
    "roomId": "78c5b0ba-fead-4535-945c-8d64a7927459",
    "channel": "c5cd8bdc-06a4-4d6e-bae3-61f1a8ac2982"
  }
}
  • The client now listens to this channel events to be notified about new messages corresponding to his subscription filters.
2nd step : notify about real-time actions

The following diagram shows how Kuzzle handles a new message and how subscribed clients are notified:

pubsub_scenario_details2

  • A new content is published to the Notifier component. The _publish_ method can be triggered:
    • either directly by the Document Controller for non persistent data (using the publish action).
    • or by the Plugins Manager when a 'document:create’ event is triggered, to notify users in real-time before the data are sent to the storage Engine.
  • The Notifier calls the DSL component to test registered filters that match the content, and get related rooms.
  • The Notifier uses the Notification Cache engine to store the mapping content/rooms into cache.
  • The Notifier calls the HotelClerk to get the channels related to the rooms.
  • The Notifier broadcasts the message to each related channel to the Websocket and MQ plugins.
  • Finally, the plugins send the message to the clients who subscribed to it.
3rd step : notify about persisted data

pubsub_scenario_details2

  • The Notifier component is notified about a new action by the Document Controller (see write scenario).
  • The Notifier calls the Notification Cache to get the rooms related to the content.
  • The Notifier calls the HotelClerk to get the channels related to the rooms.
  • The Notifier broadcasts the message to each related channel to the Websocket and MQ plugins.
  • Finally, the plugins send the message to the clients who subscribed to it.

Authentication

Kuzzle uses PassportJS to enable authentication through a large amount of providers, for example:

  • local username/password authentication (enabled by default)
  • OAuth2 providers like GitHub or google (using (Oauth plugin)[https://GitHub.com/kuzzleio/kuzzle-plugin-auth-passport-oauth])
  • SAML providers

Remember the Architecture overview and focus on the components involved by reading actions: read_scenario_http_overview

Kuzzle uses the following internal components during the authentication process:

  • The Auth Controller.
  • The Passport Wrapper, which acts as an interface between Kuzzle controllers and the Passport library,
  • The User and Token Repositories, to retrieve users’ data.
  • The Authentication strategy, implemented within a dedicated plugin.

Example - Local Strategy

The “Local” strategy (implemented by the Passport Local Plugin) authenticates a user via a username/password pair (locally stored).

auth_scenario_details_local

  • The user calls the login action of the Auth Controller:
{
  "controller": "auth",
  "action": "login",
  "body": {
    "strategy": "local",
    "username": "<my_username>",
    "password": "<my_password>"
  }
}
  • The Auth Controller calls the authenticate() method of the Passport Wrapper which formats and sends the related request to the Passport local strategy.
  • The Passport local strategy calls the verify() callback method declared by the Local Authentication Plugin to check credentials.
  • The plugin calls the User Repository and check if credentials are good and resolve to an existing user.
  • If a user is found, he is resolved and sent back to the Auth Controller through the internal components.
  • The Auth Controller calls the generateToken() method to get a JWT Token corresponding to the user.
  • The JWT Token is sent back to the client, who will use it in next requests to be authenticated:

Sample response:

{
  "status": 200,
  "error": null,
  "controller": "auth",
  "action": "login",
  "state": "done",
  "requestId": "ed4faaff-253a-464f-a6b3-387af9d8483d",
  "metadata": {},
  "result": {
    "_id": "my_username",
    "jwt": "eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJzdWIiOiJteV91c2VybmFtZSIsIm5hbWUiOiJKb2huIERvZSIsImFkbWluIjp0cnVlfQ.BefoyfAKzwXuGhbYe0iPeG0v9F4HmikvahqwqzQr3pE"
  }
}

Example - OAuth2 Strategy

The “Oauth” strategy, implemented by the Passport Oauth Plugin, authenticates a user via Github, Google+, Facebook, Twitter, or any identity provider using OAUth2 protocol with “Authorization Code” grant type.

For more details about OAuth2 protocol, see here.

auth_scenario_details_oauth2

The authentication flow is a 2-step flow:

1st step: get the OAuth2 Provider’s URL

  • The user calls the login action of the Auth Controller:
{
  "controller": "auth",
  "action": "login",
  "body": {
    "strategy": "GitHub"
  }
}
  • The Auth Controller calls the authenticate() method of the Passport Wrapper which formats and sends the related request to the Passport OAuth2 strategy.
  • The strategy calls a redirection to the OAuth2 Provider.
  • The Passport Wrapper intercepts the redirection request and formats a Kuzzle Response for the client:
{
  "headers":
  {
    "Content-Length": "0",
    "Location": "https://GitHub.com/login/oauth/authorize?response_type=code&redirect_uri=http%3A%2F%2Fkuzzle%2Fapi%2F1.0%2F_login%2Fgithub&client_id=MY_CLIENT_ID"
  },
  "statusCode": 302
}
  • The Auth Controller sends the response to the client, with the redirection URL to the OAUth2 Provider:
{
  "action": "login",
  "controller": "auth",
  "error": null,
  "metadata": {},
  "requestId": "fd4246f9-717c-4503-b50b-3a5bf0f142b5",
  "result": {
    "headers": {
      "Content-Length": "0",
      "Location": "https://GitHub.com/login/oauth/authorize?response_type=code&redirect_uri=http%3A%2F%2Fkuzzle%2Fapi%2F1.0%2F_login%2Fgithub&client_id=MY_CLIENT_ID"
    },
    "statusCode": 302
  },
  "scope": null,
  "state": "done",
  "status": 200
}

2nd step: authenticate the user with the OAuth2 code.

  • The Client sends an HTTP request to the OAuth2 Provider (unless you are using a Kuzzle SDK, this has to be implemented within the client’s application code).
  • The user authenticates to the OAuth2 Provider and allow Kuzzle Application to use his credentials (that is the standard OAuth2 flow, managed at the provider’s side).
  • The OAuth2 Provider sends a HTTP redirect response to the client, containing the OAuth2 authorization code:
HTTP/1.1 302 Found
Location: http://<kuzzle>/_login/GitHub?code=OAUTH2_CODE
  • The client calls again the login action of the Auth Controller, now with the OAuth2 authorization code:
    • either in HTTP, simply following the redirection curl http://<kuzzle>/_login/GitHub?code=OAUTH2_CODE
    • or, with another protocol (for example WebSocket), after having parsed the URL to get the authorization code:
{
  "controller": "auth",
  "action": "login",
  "body": {
    "strategy": "GitHub",
    "code": "OAUTH2_CODE"
  }
}
  • The Auth Controller calls the authenticate() method of the Passport Wrapper which formats and sends the related request to the Passport OAuth2 strategy.
  • The Passport OAuth2 strategy forwards the OAuth2 authorization code to the OAuth2 Provider in order to retrieve the OAuth2 Token.
  • The Passport OAuth2 strategy calls the verify() callback method declared by the OAuth2 Authentication Plugin
  • The plugin calls the User Repository to check for an existing user with the given GitHub ID. (Note: If no related user is found in Kuzzle, the plugin can either deny the authentication or create automatically the user, depending on the settings).
  • The user is resolved and sent back to the Auth Controller through the internal components.
  • The Auth Controller calls the generateToken() method to get a JWT Token related to the user.
  • The JWT Token is sent back to the client, who will use it in next requests to be authenticated.

How to provide your own strategy

Any strategy supported by PassportJS can be implemented in Kuzzle with a dedicated plugin. Please refer to the Plugins Reference).

Advanced Roles Definitions

In the User Guide, we have seen how to assign basic roles to profiles and profiles to users. Here, we are going to learn how to set complex and dynamic permissions.

The privileges for a certain action (restricted to a given set of indexes and collections) must be expressed as a boolean value. So far, we hard-coded this value within the permissions configuration. In some cases, this will not fit your needs. In a collaborative TO-DO list application, for example, a user should not be allowed to update other user’s items. This need is addressed by what we call Permission Closures.

Permission Closures

Instead of hard-coding the permission boolean value, we assign a function (a closure) that computes this value and returns it based on the execution context.

For example, if we need to allow users to update only their own documents, it can be done with this sample role:

let role = {
  controllers: {
    write: {
      actions: {
        update: {
          args: {
            document: {
              index: "$request.input.resource.index",
              collection: "$request.input.resource.collection",
              action: {
                get: "$currentId"
              }
            }
          },
          test: "return args.document.content.user.id === $currentUserId"
        }
      }
    }
  }
};

Where:

The permission function

The permission function is executed in a sandbox with a limited context. Its body is the evaluation of the test parameter given in the role’s definition and must return a boolean value.

The permission function has the following signature:

/**
 * @param {Request} $request              The current action request.
 * @param {string} $currentUserId         The current user Id. Shortcut to request.context.token.userId
 * @param {Object} args                   The result of the evaluated args definition.
 *
 * @return {Boolean}
 */
function ($request, $currentUserId, args) {
  // the function body is built from the "test" parameter.
  // Example, with the sample role above:
  return args.document.content.user.id === $currentUserId;
};
$request

The Request object is the request that is currently being evaluated.

$currentUserId

The $currentUserId variable contains the current user ID. It is an alias for request.context.token.userId.

args

The main purpose of the “closures” behavior is to define a role policy based on the current state of the persistence layer. This means that we need to fetch documents from the storage engine in order to use them within the permission function.

The args object contains these documents, as a result of the evaluation of the fetch definition. Each args item will look like:

{
  content: <the document itself>,
  id: <the document id>
}

In the sample role above (return args.document.content.user.id === $currentUserId), the update action is allowed only if the fetched document contains an attribute user.id which value is the current user ID.

The Fetch Definition

The Fetch Definition allows you to pass some documents fetched from the persistence layer to your permission function.

In our sample role above, we fetch a document variable which contains the document that was requested for update, and we use it in the permission function to test if it is owned by the current user.

args element structure

The args element is the place where we define our Fetch Definitions and has the following structure:

{
  "args": {
    "<some variable>": {
      "index": <index from which to fetch the document(s)>,
      "collection": <collection from which to fetch the document(s)>,
      "action": {
        "<action type (get|mget|search)>": <action type specific parameters>
      }
    },
    "<another variable>": {
      ...
    },
    ...
  }
}

You can define one or more variables inside the args element and, for each of them, the action to use to populate them. Each variable will then be available in your permission function as args.<variable>.

Embedded variables

Some variables are exposed by Kuzzle and can be used within your Fetch Definition:

  • $request: The complete request object being evaluated.
  • $currentId: The current request document ID. It is an alias for $request.input.resource._id.
action types
get

The get action type performs a read/get call. It fetches a document by its ID.

Example:

{
  "args": {
    "currentDocument": {
      "index": "$request.input.resource.index",
      "collection": "$request.input.resource.collection",
      "action": {
        "get": "$currentId"
      }
    },
    "anotherDocument": {
      "index": "myIndex",
      "collection": "myCollection",
      "action": {
        "get": "document_id"
      }
    }
  }
}

In the args field, we declare the following Fetch Definitions:

  • currentDocument, which represents the document that the user wants to update and whose Fetch Definition is composed of:
    • index: the index pointed by the current Request;
    • collection: the collection pointed by the current Request;
    • $currentId: the document ID pointed by the current Request, passed as an argument to the get action.
  • anotherDocument, which represents another document, just as an example, fetched the same way as the previous one but with different parameters.
mget

The mget action type takes a list of document ids for entry and returns the list of matching documents.

{
  "args": {
    "myDocuments": {
      "index": "myIndex",
      "collection": "myCollection",
      "action": {
        "mget": [
          "id_1",
          "id_2",
          ...
        ]
      }
    }
  }
}

In the args field, we declare a multi-valued Fetch Definition. Notice how the mget action takes an array of IDs rather than a single value.

These documents are accessed in the Permission Function as follows:

args.myDocuments = [
  { id: "id_1", content: {name: "Document 1", description: "Cum sociis natoque penatibus et magnis dis parturient montes"},
  }
  { id: "id_2", content: {name: "Document 2", description: "nascetur ridiculus mus. Nulla nunc velit"},
  }
  ...
]

The search action type performs a search on the persistence layer and returns the resulting documents. It behave exactly like a normal document search.

Example:

{
  "args": {
    "myDocuments": {
      "index": "myIndex",
      "collection": "myCollection",
      "action": {
        "search": {
          "filter": {
            "match": {
              "name": "$request.input.body.name"
            }
          }
        }
      }
    }
  }  
}

The search results are available in the Permission Function as an array of documents fetched from myIndex/myCollection, for which the name attribute matches the name attribute of the request:

args.myDocuments = [
  { id: "id_1", content: {name: "foo", description: "Cum sociis natoque penatibus et magnis dis parturient montes"},
  }
  { id: "id_2", content: {name: "foo bar", description: "nascetur ridiculus mus. Nulla nunc velit"},
  }
  ...
]

The content of action.search is directly passed to Elasticsearch.

Please refer to our Elasticsearch Cookbook for additional information on how to build your query.