Kuzzle Plugin S3 #

S3 has a right system to limit who can upload files to buckets.

Presigned URLs are special disposable Urls generated by S3. It is possible to upload a file directly to one of these URLs so that it can be stored into the bucket.

These URLs must be generated on the server side, this plugin includes among other things the generation of these URLs so that developers can then send their files directly to S3 from a client application.

Installation #

npm install kuzzle-plugin-s3

const s3Plugin = new S3Plugin();
app.plugin.use(s3Plugin);

Plugin configuration #

In your kuzzlerc file, you can change the following configuration variable:

{
  "plugins": {
    "s3": {
      "endpoints": {
        "eu-west-3": {
          "endpoint": "https://s3.eu-west-3.amazonaws.com",
          "forcePathStyle": false,
          "accessKeyIdPath": "foo.s3.eu-west-3.accessKeyId",
          "secretAccessKeyPath": "foo.s3.eu-west-3.secretAccessKey",
          "isMinio": false
        },
        "us-east-1": {
          "endpoint": "https://s3.us-east-1.amazonaws.com",
          "forcePathStyle": false,
          "accessKeyIdPath": "foo.s3.us-east-1.accessKeyId",
          "secretAccessKeyPath": "foo.s3.us-east-1.secretAccessKey",
          "isMinio": false
        },
        "custom-minio": {
          "endpoint": "https://minio.example.com",
          "forcePathStyle": true,
          "accessKeyIdPath": "foo.s3.minio.accessKeyId",
          "secretAccessKeyPath": "foo.s3.minio.secretAccessKey",
          "isMinio": true
        }
      },
      "signedUrlTTL": "20min"
    }
  }
}

Configuration Usage Documentation #

Overview #

The configuration defines S3-compatible endpoints, including AWS S3 and MinIO instances, and specifies credentials paths for secure access. It allows flexible integration with different storage backends.

Configuration Schema #

{
  "plugins": {
    "s3": {
      "endpoints": {
        "region-name": {
          "endpoint": "string",
          "forcePathStyle": "boolean",
          "accessKeyIdPath": "string",
          "secretAccessKeyPath": "string",
          "isMinio": "boolean"
        }
      },
      "signedUrlTTL": "string"
    }
  }
}

Fields Explanation #

endpoints #

• A mapping of region names to their respective configurations.
• region-name: A unique key for each endpoint configuration (e.g., eu-west-3, us-east-1).

Endpoint Fields #

1. endpoint:

   • The URL of the S3-compatible storage service.
   • Example: https://s3.eu-west-3.amazonaws.com for AWS S3 or https://minio.example.com for MinIO.

2. forcePathStyle:

   • Indicates whether to use path-style addressing (e.g., https://endpoint/bucket/object) instead of virtual-hosted style.
   • Default: false for AWS S3; set to true for MinIO.

3. accessKeyIdPath:

   • Vault path where the accessKeyId is securely stored.
   • Example: foo.s3.eu-west-3.accessKeyId.

4. secretAccessKeyPath:

   • Vault path for the secretAccessKey.
   • Example: foo.s3.eu-west-3.secretAccessKey.

5. isMinio:

   • Boolean flag indicating whether the endpoint is a MinIO instance.
   • Example: true for MinIO; false for AWS S3.

signedUrlTTL #

• Specifies the time-to-live for signed URLs.
• Format: A human-readable duration string (e.g., 20min, 1h).

Usage Example #

Configuration File #

{
  "plugins": {
    "s3": {
      "endpoints": {
        "eu-west-3": {
          "endpoint": "https://s3.eu-west-3.amazonaws.com",
          "forcePathStyle": false,
          "accessKeyIdPath": "foo.s3.eu-west-3.accessKeyId",
          "secretAccessKeyPath": "foo.s3.eu-west-3.secretAccessKey",
          "isMinio": false,
          "s3ClientOptions": {
            // any needed option from aws sdk https://docs.aws.amazon.com/AWSJavaScriptSDK/latest/AWS/S3.html#constructor-property
          }
        },
        "minio-instance": {
          "endpoint": "https://minio.example.com",
          "forcePathStyle": true,
          "accessKeyIdPath": "foo.s3.minio.accessKeyId",
          "secretAccessKeyPath": "foo.s3.minio.secretAccessKey",
          "isMinio": true
        }
      },
      "signedUrlTTL": "30min"
    }
  }
}

How the Configuration Is Used #

1. Endpoint Selection:

   • The application determines the endpoint based on the bucketRegion provided in requests.
   • Example:

     const bucketRegion = 'eu-west-3';
     const endpointConfig = config.plugins.s3.endpoints[bucketRegion];

2. Credential Retrieval:

   • The application fetches credentials from the vault using accessKeyIdPath and secretAccessKeyPath. That way, they can be shared for endpoints or not depending on S3 provider.

3. Public Access Management:

   • For AWS S3: Public access blocks are managed via deletePublicAccessBlock.
   • For MinIO: Users are instructed to configure bucket policies or access rules directly.

4. Signed URL Generation:

   • The signedUrlTTL field specifies the expiration time for generated URLs.
   • Example:

     const signedUrl = generateSignedUrl(bucketName, objectKey, endpointConfig, { expiresIn: config.plugins.s3.signedUrlTTL });

Notes #

• Security: Always store sensitive credentials in a secure Do not hardcode them into the configuration.
• Compatibility: Set forcePathStyle to true for MinIO and similar storage backends.
• Customization: Adjust signedUrlTTL based on your use case (e.g., temporary file access or downloads).