Documentation

Note: AWS S3 is currently supported only as a Destination. This guide doesn’t cover the DataPrep setup for AWS S3.

Description

‍AWS S3 is an object storage service that offers companies industry-leading scalability, data availability, security, and performance. This means that companies of all sizes can use it to store and protect data for a range of use cases, no matter the amount of data available.

Setup guide

Follow our setup guide to connect AWS S3 to Improvado.

Choose an encryption option

Choose a server-side encryption option:

• SSE-S3 (recommended) - server-side encryption with AWS S3 managed keys. Learn more.
• SSE-KMS - server-side encryption with AWS KMS keys. Learn more.
  Make sure to give the required permissions for the AWS-managed key.
• SSE-KMS with customer-managed key - server-side encryption with customer-managed AWS KMS keys. Learn more.
  Make sure to give the required permissions for an encryption key.
• No storage encryption - not recommended unless the AWS S3 bucket has default encryption.

Important: Make sure to provide your AWS S3 bucket information by following our guide.

‍

Permissions

Enable the following permissions for your AWS S3 bucket:

• ```s3:GetObject```
• ```s3:PutObject```
• ```s3:ListBucket```
• ```s3:DeleteObject```

‍

Complete configuration

On the AWS S3 connection page, fill in the following fields:

1. Enter a name for your Destination connection in the Title.
2. Enter the AWS Access Key ID.
3. Enter the AWS Secret Access Key.
4. Enter the AWS Region.
5. Enter the Bucket Name. {%dropdown-button name="bucket-name"%}

{%dropdown-body name="bucket-name"%}

• Bucket Name can only contain letters, numbers, dashes, and dots and must start and end with a letter or a number.
• Bucket Name length must be between 3 and 63 characters.

{%dropdown-end%}

6. Enter the Folder. {%dropdown-button name="folder"%}

{%dropdown-body name="folder"%}

A forward slash (```/``` ) means that you need to enter the root path.

{%dropdown-end%}

7. Select the File format option from the dropdown.
8. Enter the File name.
9. Select the Separator option from the dropdown.
10. Select the Partition by option from the dropdown. {%dropdown-button name="partition-by"%}

{%dropdown-body name="partition-by"%}

Partition is the way of splitting data for uploading to the file.

{%dropdown-end%}

11. Select the Encryption option from the dropdown. Learn more about all available encryption options here.
12. (SSE-KMS with customer-managed key) Enter the Encryption Key.
13. Select whether you want to Use load by account for this Destination. {%dropdown-button name="use-load-by-account"%}

{%dropdown-body name="use-load-by-account"%}

If enabled, the File name field must include ```{{account}}``` variable.
You must enable this field if you want to use a specific account for data load.

{%dropdown-end%}

How to provide credentials to Improvado

There are three different ways to provide your credentials that you can choose depending on the type of selected Server-Side Encryption:

• Option #1 (Create a user in your AWS account for Improvado) - SSE-S3 and SSE-KMS (any type of key). Recommended option.
• Option #2 (Share Read and Write access with Improvado’s AWS account) - SSE-S3 and SSE-KMS (customer-managed key only)
• Option #3 (Share access with Improvado account using our Canonical ID) - SSE-S3

‍

Option #1 (Create a user in your AWS account for Improvado)

{%docs-informer info%}

Available for SSE-S3 and SSE-KMS (with AWS-managed and customer-managed keys) only.

{%docs-informer-end%}

Create a user in your AWS account for Improvado and enter the following information in Complete configuration:

• Bucket Name
• AWS Access Key ID
• AWS Secret Access Key
• AWS Region

Make sure to enable the following permissions for your AWS S3 bucket.

‍

Option #2 (Share Read and Write access with Improvado’s AWS account)

{%docs-informer info%}

Available for SSE-S3 and SSE-KMS (with customer-managed keys) only.

{%docs-informer-end%}

‍Important: If you’re gonna use this option - notify our Support or your CSM about it and we will create specific users to load data and provide support. We’ll create a Destination connection for you.

Share Read and Write access with Improvado’s AWS Account ID:

1. Create an AWS S3 bucket.
2. Select the Permissions tab on the Bucket Settings page.
3. In the Bucket policy, click the Edit button.

4. Copy & paste the Policy example below.
5. ~Change ```your-bucket-name``` to your real Bucket name.

{
  "Id": "Policy1569503459134",
  "Version": "2012-10-17",
  "Statement": [
      {
          "Sid": "S3Access",
          "Action": [
              "s3:GetObject*",
              "s3:DeleteObject",
              "s3:ListBucket*",
              "s3:PutObject*",
              "s3:ListBucketMultipartUploads",
              "s3:ListMultipartUploadParts",
              "s3:AbortMultipartUpload"
          ],
          "Effect": "Allow",
          "Resource": [
              "arn:aws:s3:::your-bucket-name",
              "arn:aws:s3:::your-bucket-name/*"
          ],
          "Principal": {
              "AWS": [
                  "716309063777"
              ]
          }
      }
  ]
}

5. Save changes.
6. If you use SSE-S3, then Provide your AWS S3 bucket information to Improvado (Option #2).

{%docs-informer info title="Important"%}

If you use SSE-KMS, you have to share access to your KMS key using one of the methods below:

• ‍Method 1 (Add Improvado’s AWS account to KMS Key settings).‍
• Method 2 (Add Improvado’s AWS account ID to your KMS Key policy).

{%docs-informer-end%}

‍Note: AWS allows sharing only customer-managed KMS keys (keys that you created). The AWS-managed KMS key (the key that was created by AWS automatically) cannot be shared.

‍

Method 1 (Add Improvado’s AWS account to KMS Key settings)

The KMS Key ID is required.

1. Create a KMS key.
2. Open the Key settings.
3. Go to Other AWS accounts and click the Add other AWS account button.

4. Paste Improvado’s AWS account ID: ```716309063777```.

5. Save changes.
6. Provide your AWS S3 bucket information to Improvado (Option #2)

‍

Method 2 (Add Improvado’s AWS account ID to your KMS Key policy)

The KMS Key ID is required.

1. Create a KMS key.
2. Open the Key settings.
3. In the Key policy tab, click the Switch to policy view button.

4. Click the Edit button.

5. Copy & paste the Policy example below.
6. ~Change ```example-region-1``` to real key Region.
7. ~Change ```123456789098``` to your real Account ID.
8. ~Change ```111aa2bb-333c-4d44-5555-a111bb2c33dd``` to real Key ID.

{
    "Version": "2012-10-17",
    "Statement":
    [
        {
            "Sid": "Enable IAM User Permissions",
            "Effect": "Allow",
            "Principal": {
                "AWS": "arn:aws:iam::123456789098:root"
            },
            "Action": "kms:*",
            "Resource": "*"
        }
        {
            "Sid": "Allow use of the key",
            "Effect": "Allow",
            "Principal": {
                "AWS": "arn:aws:iam::716309063777:root"
            },
            "Action": [
                "kms:Encrypt",
                "kms:Decrypt",
                "kms:ReEncrypt*",
                "kms:GenerateDataKey*",
                "kms:DescribeKey"
            ],
            "Resource": "arn:aws:kms:example-region-1:123456789098:key/111aa2bb-333c-4d44-5555-a111bb2c33dd"
        }
    ]
}

6. Save changes.
7. Provide your AWS S3 bucket information to Improvado (Option #2)

Provide your AWS S3 bucket information to Improvado (Option #2)

Provide our Support Team or your CSM with the following information:

• Bucket Name
• AWS Region

Our team will create specific users to load data and provide support.

‍

Option #3 (Share access with Improvado account using our Canonical ID)

{%docs-informer info%}

Available for SSE-S3 only.

{%docs-informer-end%}

‍Important: If you’re gonna use this option - notify our Support or your CSM about it and we will create specific users to load data and provide support. We’ll create a Destination connection for you.

Share access with the Improvado account using our Canonical ID:

1. Create an AWS S3 bucket.
2. Select the Permissions tab on the Bucket Settings page.

3. In the Access control list (ACL), click the Edit button.

4. Click the Add grantee button in the Access for other AWS accounts section.
5. Paste ```5f2dfe7db1abc67daeea58fea77fb0c399ed924a2f88cf36d28738b7e1c838ef``` Canonical ID to the Grantee field.
6. Set List and Write permissions for Objects.

7. Click the Save changes button.
8. Provide your AWS S3 bucket information to Improvado (Option #3)

Provide your AWS S3 bucket information to Improvado (Option #3)

Provide our Support Team or your CSM with the following information:

• Bucket Name
• AWS Region

Our team will create specific users to load data and provide support.

‍

Additional information

Folder

Possible parameters:

```/{{workspace_id}}/{{workspace_title}}/{{data_source}}/{{data_table_title}}/{{report_type}}/{{filename}}/{{account}}/{{YYYY}}/{{MM}}/{{DD}}```

• ```{{workspace_id}}``` and ```{{workspace_title}}``` are optional parameters that provide additional information about the workspace used for a destination connection
• ```{{data_source}}``` is a data provider, integration, connector.
• ```{{data_table_title}}``` is an object that contains all extraction orders with the same granularity (dimensional schema).
• ```{{report_type}}``` is a set of such fields as metrics, properties, dimensions, etc.
• ```{{account}}``` - is an optional parameter that allows you to to add specific account for the data load.
• ~ You must enable Use load by account field to add this parameter to the File name.

If you use ```/{{YYYY}}/{{MM}}/{{DD}}``` settings, the data will be added to folders daily. Each new record will not delete the previous one, even for data that contains no date.

‍Data structure of S3 storage by Improvado. You can create a request to the Support Team to add support for different folder structures in a bucket.

File format

Possible formats:

• csv
• csv+gzip
• json
• json+gzip
• parquet
• avro

File name

Possible parameters:

```{{filename}}-{{account}}-{{YYYY}}-{{MM}}-{{DD}}```

• ```{{filename}}``` - is the same as destination table name
• ```{{account}}``` - is an optional parameter that allows you to to add specific account for the data load.
• ~You must enable Use load by account field to add this parameter to the File name

IMPORTANT: you cannot use {{DD}} for partition by month

• ```{{filename}}-{{YYYY}}-{{MM}}-{{DD}}``` – for partition by day
• ```{{filename}}-{{YYYY}}-{{MM}}``` – for partition by month

Also, you can use “_” instead of “-” or do not use any symbols at all, for example:

• ```{{filename}}_{{YYYY}}-{{MM}}-{{DD}}```
• ```{{filename}}{{YYYY}}{{MM}}{{DD}}```

Separator

Possible delimiters that can separate data in your file:

• comma
• semicolon
• tab

Partition by

Possible ways of splitting data:

• Day
• Month