Upgrading from 2.60 #

This page covers Kuzzle 2.61, starting with its first beta, 2.61.0-beta.1 (2.61.0-beta.2 adds the built-in drivers as packages, 2.61.0-beta.3 pins them to 1.0.0, 2.61.0-beta.4 fixes the index and collection checks in a cluster, 2.61.0-beta.5 documents the OpenSearch 2 and 3 drivers, 2.61.0-beta.6 tells plugins which engine runs, closes the storage connections on shutdown and words storage messages for any engine, 2.61.0-beta.7 waits for plugin requests before closing the storage, 2.61.0-beta.8 moves to the drivers' stable releases, 2.61.0-beta.9 logs what an unhandled rejection contains). 2.61.0 is the content of 2.61.0-beta.9. If you upgrade from an older version, read the previous upgrade notes too.

The API, the configuration and the TypeScript declarations stay compatible with v2.60.1. An Elasticsearch 7 or 8 setup needs no change. Items that may need an action on your side are marked Action.

The changelog is generated from the commits and says what was done. This page says what you may have to check.

Operators #

Elasticsearch 7 and 8 run as built-in storage drivers #

Kuzzle now talks to its database through a storage driver. Elasticsearch 7 and 8 are its built-in drivers, still selected by services.storageEngine.majorVersion: their behavior and their configuration do not change.

Since 2.61.0-beta.2, their code lives in two packages, kuzzle-storage-es-7 and kuzzle-storage-es-8, which kuzzle depends on at pinned versions. npm install brings them with Kuzzle: there is nothing to add to your package.json. npm ls kuzzle-storage-es-8 shows the version your Kuzzle uses.

Since 2.61.0-beta.3, both are pinned to stable versions. If the package of the configured majorVersion is missing (a partial install, a pruned node_modules), Kuzzle does not start and reports services.storage_driver.not_found. Action, only if you see it: run npm install again.

A rolling upgrade from v2.60 with an unchanged configuration works as before: nodes running the built-in drivers are compared by Elasticsearch major version only, and a node older than 2.61 counts as a built-in one. Action, only if your cluster mixes majorVersion values: align them. A majorVersion: "8" node is now refused by "7" nodes, with cluster.join.storage_driver_mismatch (#3139). Such a cluster was splitting its data across two Elasticsearch clusters.

Running another storage engine #

Setting services.storageEngine.driver to the name of a driver package, installed next to kuzzle, runs Kuzzle on that driver's engine. driver takes an npm package name or an npm alias, not a path; a driver installed with npm link or from a directory works too. In that case:

• services.storageEngine.majorVersion is ignored, and the rest of services.storageEngine is handed to the driver. If your configuration sets majorVersion too, Kuzzle logs services.storageEngine.majorVersion is ignored because services.storageEngine.driver is set: remove majorVersion. at startup;
• Kuzzle logs [ℹ] Storage driver: <name>@<version> (<directory>) at startup, instead of the Elasticsearch major version;
• Kuzzle does not start when the driver is missing, when its manifest.json is invalid, when its kuzzleVersion range excludes this Kuzzle version, or when it lacks a method of the storage contract or of a capability it declares (the services.storage_driver.* errors). These messages say what to do, for instance how to install a missing package (#3134, #3137);
• the driver always uses the running Kuzzle, even when another copy of kuzzle sits in node_modules, as in the official Docker image (#3132). Before, such a driver could answer some documented 404 or 400 errors as 500 unexpected_error;
• a request needing a capability the driver does not declare is rejected with services.storage_driver.unsupported_capability (400).

Action, only if you use an external driver in a cluster: every node must run the same driver in the same version. A node running another one fails its startup with cluster.join.storage_driver_mismatch, before it connects to the storage. A node running an external driver also refuses to start next to a node older than 2.61. If the mismatching node may be a stale one (killed without cleaning up, as in a full restart), the starting node logs a warning and waits for its ID card to expire (about 6 seconds by default) before deciding. Upgrading a driver therefore means restarting every node on the new version, not a rolling restart.

Engine version checked by the driver #

Each official driver now refuses to start on an engine it does not support: the built-in Elasticsearch 7 driver on an Elasticsearch 8 cluster, for instance, or an Elasticsearch driver on OpenSearch (which reports itself as Elasticsearch 7.10.2). Action, only if Kuzzle no longer starts: point it at the engine its majorVersion or driver names, or pick the matching driver.

Running Elasticsearch 9 or OpenSearch #

Install the driver next to kuzzle and name it in services.storageEngine.driver:

Unlike the built-in drivers, Kuzzle does not pin these: you choose their version in your package.json. Check your plugins first (see Raw storage clients).

Action, to move from Elasticsearch 8 to Elasticsearch 9:

1. upgrade the Elasticsearch cluster to 9, following Elastic's upgrade guide;
2. npm install kuzzle-storage-es-9 in your application;
3. set services.storageEngine.driver to "kuzzle-storage-es-9" and remove majorVersion;
4. stop every Kuzzle node, then start them all on the new configuration. A rolling restart does not work: a node running kuzzle-storage-es-9 is refused by nodes still running the built-in driver (cluster.join.storage_driver_mismatch).

The OpenSearch drivers do not move your data: copy it from Elasticsearch with a snapshot or a remote reindex.

majorVersion is deprecated #

services.storageEngine.majorVersion is deprecated in the documentation: Kuzzle 3 will select the storage engine with driver alone. It keeps working unchanged in Kuzzle 2, where it still chooses between the built-in Elasticsearch 7 and 8 drivers, and Kuzzle logs no deprecation warning for it.

Without driver, a majorVersion other than "7" or "8", such as "9", stops Kuzzle with services.storage_driver.unsupported_major_version, which points to driver (#3134). Kuzzle 2.60 already refused it, with a plain error.

Index cache of a joining node #

A node joining a cluster could keep, in its cache, an index or a collection that another node had deleted before it joined, and a collection write lock already released: the deletion was announced to the nodes already in the cluster only. When it loads its cache, a node now replaces it with what the storage holds.

Index and collection checks in a cluster #

Each node keeps the list of indexes and collections in memory, and learns about the ones another node creates or deletes through a cluster message that no request waits for. For a few milliseconds after a creation on one node, another node could answer that the index or the collection does not exist (services.storage.unknown_index, unknown_collection), leave it out of index:list or collection:list, or refuse to create an index another node had just deleted (index_already_exists).

A node now checks the storage before giving such an answer, and updates its list. As a result, index:list, collection:list and index:stats read the storage on every call, and a request on an index or a collection that does not exist costs one storage read before it fails. Requests on existing indexes and collections are unchanged.

These storage reads start at most once every services.storageEngine.cacheMissReadInterval milliseconds (default 100) on each node (#3140): a client looping on a missing index no longer costs one read per request. A miss inside the interval waits for the next read and shares it, so the answer is never stale, and any index or collection created or deleted lifts the wait. As a result, a request on an index or a collection that does not exist can fail up to 100 ms later. Action, only if that delay matters to you: set cacheMissReadInterval to 0 to read on every miss, as in 2.61.0-beta.4.

Plugin and application developers #

Raw storage clients #

app.storage.storageClient, app.storage.StorageClient and context.constructors.ESClient are now built by the storage driver. With the built-in drivers, they give the same Elasticsearch client as before, in the same version and with the same API. With an external driver, they give that driver's engine client.

Since 2.61.0-beta.2, that client comes from the driver package's own copy of @elastic/elasticsearch. Action, only if your code checks it with instanceof against a Client class it imports itself: compare its behavior instead, or import the class from the driver's dependencies. To type it, see the drivers' TypeScript page.

Storage error codes #

Since before 2.61, some storage engine errors reached clients as services.storage.unexpected_bad_request or unexpected_error: the error mapping expected messages the current engine clients no longer produce. With 2.61 and every official driver, they come back with their documented services.storage.* code: too_many_operations (thread pool or indexing pressure rejection), cannot_change_mapping, unexpected_properties, invalid_mapping_type, too_many_changes (version conflict) and unknown_query_keyword.

Action, only if your code or your clients match on unexpected_bad_request or unexpected_error for these cases: match the specific code instead.

Storage error messages #

Since 2.61.0-beta.6, storage error messages say "storage engine" instead of "Elasticsearch", also right on OpenSearch (#3143). For instance, services.storage.not_connected now reads "Storage engine is not connected.", services.storage.version_mismatch "Your storage engine version is …", and the Koncorde and Query DSL keyword errors name "the storage engine Query DSL". A storage initialization timeout now names the service storageEngine instead of elasticsearch in core.fatal.service_timeout. Error ids, codes and statuses do not change.

Action, only if your code, your clients or your log alerts match the text of these messages: match the error id instead.

Storage drivers close their connections on shutdown #

Since 2.61.0-beta.6, Kuzzle calls the storage driver's optional close() hook on a graceful shutdown, and every official driver closes its engine connections there. A driver whose close() fails gets a warning in the logs, and the shutdown carries on. Before closing the storage, the shutdown waits for the client requests still running and, since 2.61.0-beta.7, for the requests plugins run through context.accessors.sdk or app.sdk too (#3160). That wait lasts 5 seconds at most, as a plugin may run requests in a loop: past it, Kuzzle logs a warning and closes the storage anyway. An unhandled rejection that happens while Kuzzle shuts down is logged and no longer stops it with exit code 1 in development mode: late plugin work failing on the closed storage does not turn a requested shutdown into a failure. Driver authors: see the storage driver contract.

New exports #

kuzzle now exports the StorageDriver contract and its types, the conformance kit describeStorageDriverConformance, and storageError, which builds Kuzzle's services.storage.* errors. They are meant for driver authors: see the Write storage drivers guide.