IMPORTANT: This article covers setup of warehouse for load data from Improvado, not customer data warehouse from which data is being extracted. This article doesn't cover setup of customer data warehouse for Data Prep as well.
Possible parameters:
If you use /{YYYY}/{MM}/{DD} settings, the data will be added to folders daily. Each new record will not delete the previous one, even for data that contains no date.
Data structure of S3 storage by Improvado
By request to the support team we are able to support different folder structure in a bucket.
Possible formats:
Possible parameters:
IMPORTANT: you cannot use {{ DD }} for partition by month
Also, you can use “_” instead of “-” or do not use any symbols at all, for example:
Possible ways of splitting data:
SSE-S3 and SSE-KMS (with AWS managed and customer managed keys) encryption types can be used.
Create a user in the your AWS account for Improvado and provides following information:
Created user should have following permissions for the S3 bucket:
SSE-S3 and SSE-KMS (with customer managed keys) encryption types can be used.
Share Read and Write access to the S3 bucket with Improvado’s AWS account (account id=716309063777).
Required information:
{% code-block language="json" %}
{
"Id": "Policy1569503459134",
"Version": "2012-10-17",
"Statement": [
{
"Sid": "S3Access",
"Action": [
"s3:GetObject*",
"s3:DeleteObject",
"s3:ListBucket*",
"s3:PutObject*",
"s3:ListBucketMultipartUploads",
"s3:ListMultipartUploadParts",
"s3:AbortMultipartUpload"
],
"Effect": "Allow",
"Resource": [
"arn:aws:s3:::your-bucket-name",
"arn:aws:s3:::your-bucket-name/*"
],
"Principal": {
"AWS": [
"716309063777"
]
}
}
]
}
{% code-block-end %}
For SSE-KMS you have to share access to your KMS key using one of the methods below.
Note that AWS allows to share only customer managed KMS keys (keys that you created). The AWS managed KMS key (the key that was created by AWS automatically) cannot be shared.
The KMS Key ID is required.
The KMS Key ID is required.
{% code-block language="json" %}
{
"Version": "2012-10-17",
"Statement":
[
{
"Sid": "Enable IAM User Permissions",
"Effect": "Allow",
"Principal": {
"AWS": "arn:aws:iam::123456789098:root"
},
"Action": "kms:*",
"Resource": "*"
}
{
"Sid": "Allow use of the key",
"Effect": "Allow",
"Principal": {
"AWS": "arn:aws:iam::716309063777:root"
},
"Action": [
"kms:Encrypt",
"kms:Decrypt",
"kms:ReEncrypt*",
"kms:GenerateDataKey*",
"kms:DescribeKey"
],
"Resource": "arn:aws:kms:example-region-1:123456789098:key/111aa2bb-333c-4d44-5555-a111bb2c33dd"
}
]
}
{% code-block-end %}
SSE-S3 encryption type can be used.
Share access to Improvado account using our Canonical ID:
Required information:
If you used 2nd or 3rd options - notify our support or CSM about it and we will create specific users to load data and provide support.