Documentation

Note: This article covers the Google Cloud Storage Destination setup process. It doesn’t cover the DataPrep setup for GCS.

You can learn how to extract data from Google Cloud Storage data source here.

Description

‍Google Cloud Storage is a highly available and durable object storage service offered by Google Cloud Platform, designed to store and access large, unstructured data sets with high reliability, scalability, and performance.

Follow our setup guide to connect Google Cloud Storage to Improvado.

Generate a Service Account Key JSON file

In order to use Service Account Key authentication, first, you need to generate a JSON file via Google Cloud Console using official documentation or an interactive step-by-step guide provided by Google.

Alternatively, you can follow the instructions below:

1. In Google Cloud Console, go to IAM & Admin → Service Accounts.

2. Click on the Actions button for your Service account and select Manage keys.

3. In the KEYS tab section, click ADD KEY → Create new key. Choose JSON as a key type and click Create.

4. In the downloaded JSON file, copy your Project ID.

How to connect

You need to share access for your Google Cloud Storage bucket to Improvado Google Service account: improvado-gcs-loader@green-post-223109.iam.gserviceaccount.com with a role at GCS bucket: Storage Object Admin.

Learn more here.

Complete configuration

On the Google Cloud Storage connection page, fill in the following fields:

1. Enter a name for your Destination connection in the Title.
2. Enter the Bucket Name. {%dropdown-button name="bucket-name"%}

{%dropdown-body name="bucket-name"%}

Preferred bucket for GCS uploading.

Bucket Name can only contain letters, numbers, dots, and underscores and must start and end with a letter or number.

Bucket Name length must be between 3 and 222 characters.

{%dropdown-end%}

3. Enter the Filename. {%dropdown-button name="filename"%}

{%dropdown-body name="filename"%}

Possible parameters:

```{{workspace_id}}-{{workspace_title}}-{{data_source}}-{{report_type}}-{{data_table_title}}-{{filename}}-{{dataclass}}-{{YYYY}}-{{MM}}-{{DD}}-{{H}}-{{M}}-{{S}}-{{chunk_id}}```

• ```{{filename}}``` is the same as the destination table name
• ```{{dataclass}}``` - is an optional parameter that describes how the data will be updated in the destination.
  • Possible data class values:
    • ```daily```
    • ```monthly```
    • ```weekly```
    • ```last_day```
    • ```last_day_incremental```
    • ```unknown```
• Note: you cannot use ```{{DD}}``` for partition by month.
  • ```{{filename}}-{{YYYY}}-{{MM}}-{{DD}}``` – for partition by day
  • ```{{filename}}-{{YYYY}}-{{MM}}``` – for partition by month

{%docs-informer warning title="Important"%}

{{YYYY}}/{{MM}}/{{DD}} partitioning is based on a date field in the exported data. If the dataset does not contain an actual date field (for example, no “date” column/field), these partition values cannot be derived and date-based partitioning will not work as expected. In that case, use {{YYYY_today}}/{{MM_today}}/{{DD_today}} to partition by the export run date instead.

{%docs-informer-end%}

{%docs-informer info%}

{{YYYY}}/{{MM}}/{{DD}} refer to the record’s date-based partition values (if applicable), while {{YYYY_today}}/{{MM_today}}/{{DD_today}} always use today’s date.

{%docs-informer-end%}

‍

• Also, you can use “_” instead of “-” or do not use any symbols at all, for example:
  • ```{{filename}}_{{YYYY}}-{{MM}}-{{DD}}```
  • ```{{filename}}{{YYYY}}{{MM}}{{DD}}```

{%dropdown-end%}

4. Select the necessary File Format option from the dropdown. {%dropdown-button name="file-format"%}

{%dropdown-body name="file-format"%}

Possible formats:

• csv
• csv+gzip
• json
• json+gzip
• parquet
• avro

{%dropdown-end%}

5. Select the necessary Separator option from the dropdown. {%dropdown-button name="separator"%}

{%dropdown-body name="separator"%}

Possible delimiters that can separate data in your file:

• comma
• semicolon
• tab

{%dropdown-end%}

6. Select the necessary GCS Region option from the dropdown.
7. Select the necessary Partition by option from the dropdown. {%dropdown-button name="partition-by"%}

{%dropdown-body name="partition-by"%}

Possible ways of splitting data:

• Day (default value)
• Month

{%dropdown-end%}

8. Select the necessary Encryption option from the dropdown. {%dropdown-button name="encryption"%}

{%dropdown-body name="encryption"%}

Possible options:

• Default Cloud Storage encryption;
• Customer-managed encryption keys;
• Customer-supplied encryption keys.

{%dropdown-end%}

9. (Optional) Enter the Root Name. {%dropdown-button name="root-name"%}

{%docs-informer info%}

This field supports letters and numbers only, max 64 characters (no special characters or dynamic values).

{%docs-informer-end%}

10. Select the necessary Use static IP option from the dropdown. {%dropdown-button name="use-static-ip"%}

{%dropdown-body name="use-static-ip"%}

Select Yes for Use static IP option if you allow Improvado to connect your database by the static IPs mentioned on the Destination connection page.

Select No if you have permitted access to your database from any IP. In this case, Improvado will connect your database using dynamic IPs not listed on the Destination connection page.

{%dropdown-end%}

11. Select Workload Identity Federation as the Authentication type (recommended).
12. Upload your Service account key JSON file to the Service account key.
13. Enter the Project ID.
14. (Workload Identity Federation only) Enter the GCP Project Number.
15. (Workload Identity Federation only) Enter the Workload Pool ID. {%dropdown-button name="workload-pool-id"%}

{%dropdown-body name="workload-pool-id"%}

Pool IDs are used as identifiers in IAM.

{%dropdown-end%}

16. (Workload Identity Federation only) Enter the AWS Provider ID. {%dropdown-button name="aws-provider-id"%}

{%dropdown-body name="aws-provider-id"%}

Providers manage and verify identities.

{%dropdown-end%}

17. (Workload Identity Federation only) Enter the Service Account Email. {%dropdown-button name="aws-provider-id"%}

{%dropdown-body name="aws-provider-id"%}

A service account is identified by its email address, which is unique to the account.

{%dropdown-end%}

18. Select the necessary Use load by accounts option from dropdown.
19. Select the necessary Use binary strings option from the dropdown. {%dropdown-button name="use-binary-strings"%}

{%dropdown-body name="use-binary-strings"%}

The "Use binary strings" determines whether string fields are encoded in binary format when loading data.

Recommended setting: "No" (default).

{%dropdown-end%}

Secondary Authentication Option (Workload Identity Federation)

Note: We recommend using the Service Account Key as an authentication method.

With identity federation, you can use Identity and Access Management (IAM) to grant external identities IAM roles, including the ability to impersonate service accounts. This approach eliminates the maintenance and security burden associated with service account keys.

Learn more about Identity Federation here: Workload identity federation | IAM Documentation | Google Cloud.

1. Setup a Workload pool and Provider for your Google Cloud project.
2. Specify the Improvado AWS account ID that you can find on Improvado UI:

3. Paste Improvado AWS Account ID and configure Attribute mapping:
4. ~Set ```attribute.aws_role attribute``` value to ```assertion.arn.extract('assumed-role/{role}/')```.

4. Add a condition to allow only one AWS IAM role which is called: ```workload_identity_fed_file_sender```.
5. ~Set the Condition CEL to ```attribute.aws_role == "workload_identity_fed_file_sender"```.

5. On the Workpool details page, click the Grant access button.

6. Select principals (identities that can access the service account):
7. ~Set ```aws_role attribute``` value to ```workload_identity_fed_file_sender```.

7. Make sure that your service account has atleast Service Account Admin role:

8. Assign Storage Object Admin role to the service account for bucket access:

9. In Complete configuration, enter your Workload Pool ID and AWS Provider ID to the corresponding fields in the Improvado UI.