Connect a Simple Storage Service (S3) System¶

Simple Storage Service (S3) is used with Amazon Web Services (AWS) for example. In addition to or as an alternative filestore to a pure MongoDB-based setup, the files uploaded by SEAL Operator can be stored in S3. MongoDB is still required for metadata and internal information.

MongoDB, technically able to store files via GridFS, is not bulk storage by design. For a small number of small files accessed only through SEAL Operator, MongoDB is nevertheless sufficient. S3 has no practical size limit and fits better when the data originates from other systems, or when files are large or numerous. Being cloud-hosted, an S3 bucket is also reachable from anywhere. The S3 protocol is not limited to AWS. Many NAS and SAN vendors support it too, which can be an alternative if such a system is already in place at the customer's.

Here, the configuration settings in AWS S3 concerning SEAL Operator are described in brief. For other S3 systems, refer to the corresponding documentation.

Afterwards, the configuration in SEAL Operator is described.

Set Up and Configure an AWS S3 System for SEAL Operator¶

To configure the storage for the file upload the following items are required:

an S3 bucket where the files are stored
an identity and access management (IAM) service user with access to the S3 bucket, its access key and secret

Hint - AWS account

If you do not have an AWS account, go to https://aws.amazon.com/ and create one. This account is the root user of AWS and has administrator rights. Using its access keys is not recommended. Create a specific IAM service user instead as described below.

Create the S3 Bucket¶

Search for the S3 service and create a bucket with the following settings:
- Name: This is the root of the filestore. Therefore, specify a meaningful name, for example, seal-operator-fileupload.
- Region: Select the one closest to you, for example, EU (Frankfurt) eu-central-1.
- Default encryption: Enable Amazon S3 key (SSE-S3).

Create the Access Policy¶

First, create a policy that gives access to only the S3 bucket created before.

Search for the IAM service.
Open the Policies tab and create a new one.

Copy & paste the following policy JSON structure. Replace <bucket_name> with the name specified for the S3 bucket above, for example, seal-operator-fileupload:

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Sid": "VisualEditor0",
      "Effect": "Allow",
      "Action": [
        "s3:PutObject",
        "s3:GetObject",
        "s3:CreateBucket",
        "s3:ListBucket",
        "s3:DeleteObject",
        "s3:DeleteBucket"
      ],
      "Resource": [
        "arn:aws:s3:::<bucket_name>",
        "arn:aws:s3:::<bucket_name>/*"
      ]
    },
    {
      "Sid": "VisualEditor1",
      "Effect": "Allow",
      "Action": "s3:ListAllMyBuckets",
      "Resource": "*"
    }
  ]
}

Save the policy under a recognizable name, for example, SealFilestoreS3Policy.

Hint - reuse

The permissions policy can be reused for other users.

Create an IAM User and Access Key¶

To connect SEAL Operator to an Amazon S3 bucket, you need an IAM user with an access key.

Open the AWS Management Console and navigate to IAM.
Go to Users and create a new user, for example seal-operator-s3.
Assign the required permissions. Use one of the following options:
- Recommended: Attach a policy that grants access to the target S3 bucket. This option follows the principle of least privilege.
- Add the user to a group that already has the required S3 permissions.
After the user has been created, open the user details and go to Security credentials.
In the Access keys section, create a new access key for programmatic access. Save the Access key ID and Secret access key. You need both for the SEAL Operator configuration.

Caution - secret visible only once

Do not leave this dialog before saving the Access key ID and Secret access key. Once you leave it, you cannot access the secret again.

Hint - AWS console UI

The AWS console UI changes from time to time. The important part is that you create an IAM user with an access key that can read/write the S3 bucket used by the Operator.

Configure the S3 Connector¶

In SEAL Operator, activate the connector and specify the keys for the connection:

Open a command prompt.
Export the complete configuration of SEAL Operator from Consul to a YAML file. This ensures that you edit the current configuration.
```
operator config export <filename>.yml --insecure
```
Edit the exported file <filename>.yml.

In the section for the S3 connector, set cstatus to on.

operator:
  connectors:
    s3:
      cstatus: 'on'
      serviceName: operator-fileupload

In the env section, specify the following keys for the seal-operator-fileupload service in the operator-fileupload section:
- FILESTORE_TYPE: mandatory, type how the content of the uploaded files is stored, here s3
- S3_ACCESS_KEY_ID: mandatory, ID of the access key to the S3 system as configured in the S3 system
- S3_SECRET_ACCESS_KEY: mandatory, secret of the access key to the S3 system as configured in the S3 system
- S3_BUCKET: optional, name of the S3 bucket as configured in the S3 system
- S3_REGION: optional, S3 region as configured in the S3 system
- MONGO_FILEUPLOAD_URL: optional, URL of the MongoDB used by the fileupload service to store metadata, job information, and housekeeping data about uploaded files. The fileupload service needs MongoDB even when S3 is used as FILESTORE_TYPE.
- DEFAULT_FILEUPLOAD_PANEL: optional, path to the JSON schema file containing the configuration of the panel
The S3 connection is configured via the operator-fileupload service. There is no dedicated operator-s3 service. In the SEAL Operator configuration:
- the S3 connector points to operator-fileupload.
- the S3-specific environment variables are set under env.service.operator-fileupload.tag.any.
```
env:
  service:
  ...
    operator-fileupload:
      tag:
        any:
          FILESTORE_TYPE: s3
          S3_ACCESS_KEY_ID: '<s3_access_key_id>'
          S3_SECRET_ACCESS_KEY: '<s3_secret_access_key>'
          S3_BUCKET: 'seal-operator-fileupload'
          S3_REGION: 'eu-central-1'
          MONGO_FILEUPLOAD_URL: 'mongodb://<mongodb_server>:27017/operator-s3'
          DEFAULT_FILEUPLOAD_PANEL: '<fileupload_connector_install_dir>/lib/defaultConfig/s3-panel.json'
  ...
```
Literature - keys

For further information about available keys, refer to the Key Reference.
Save the <filename>.yml file.

Re-import the file to Consul.

operator config import <filename>.yml --insecure

Check the imported value, for example for FILESTORE_TYPE:

operator config get FILESTORE_TYPE --service operator-fileupload --insecure

Hint - change panel name in s3-panel.json

Changing the S3 panel name to My Cloud for example is recommended to avoid confusion. You can do that by editing the corresponding S3 configuration file s3-panel.json.

For the file location of the s3-panel.json, refer to Directories and Files under Windows or Directories and Files under Linux.