Documentation

Note: This article covers AWS S3 Data source setup process. You can learn how to load data to AWS S3 Destination here.

How to set up AWS S3 (external buckets) integration

1. On the Data sources page, click on the AWS S3 (external buckets) tile. You can find it using the search field or just open the Flat Data category.
2. On the AWS S3 (external buckets) authorization page, fill in the following fields:
3. ~Enter the Bucket Name
4. ~Enter the Access Key ID
5. ~Enter the Access Secret Key
6. ~Enter the Region
7. Fill in the form and click the Create button.

Step 1. Import page

1. Click to select a file from your computer, or drag and drop your file to this area

{%docs-informer info title="Note"%}

• Supported file formats: CSV, TSV, XLS, XLSX, AVRO
• Maximum size: 10 GB
• A file must begin with column headers starting in column A. If additional information is provided before — it is necessary, use the Header rows to skip field to skip a certain number of rows to import data correctly.

{%docs-informer-end%}

2. After picking the file a processing bar will show the upload progress.
3. ~You can click on the “trash bin” icon to cancel the uploading process of a file
4. After your file has been uploaded and checked for security, let’s move to the settings.
5. Click on the Continue button.

Step 2. Match page

1. Enter the data table name or leave it as is.
2. In the Root path field, enter the full directory path where the system should search for files.
3. In the File mask field, specify a regex pattern to filter filenames or types.
4. Select the fields you need to import. By default, all fields are selected.
5. Check the type. Each field in a data table can be assigned a text, number, or date type.
6. ~The date field is a unique date-key in the database and should be singular. Also, you have the flexibility to choose from a wide range of date formats for this field.
7. ~If you have field with numeric values that you don't want to include in your calculations, change the type from Number to Text.

{%docs-informer info title="Note"%} You can learn more about how to work with dates in Flat Data here. {%docs-informer-end%}

4. The skip rule allows you to exclude empty rows and rows with specific values in a selected field
5. ~Default to 'Do Not Skip': Ensures all data is initially included.
6. ~Multi-Select Options: Combine 'Skip: empty' and 'Skip: value' for targeted exclusions. Selecting 'Do Not Skip' disables other skip options.
7. ~Custom Rule addition: Add one custom rule; the form is disabled afterward to maintain focus
8. ~Custom Rule deletion:  Hover over and click on the red-highlighted basket icon next to the rule to exclude it.‍

‍Usage Note: the component is designed for straightforward value-based selections and does not support complex expressions like '<10'.

6. Click on the Import N fields button to continue.

Step 3. Preview page

1. Check the Data Structure Preview.
2. ~In this step, you see the first row of data that will be extracted after you click the Continue button.

2. Click the Continue button to confirm settings and proceed with data extraction.
3. You'll see your imported file on the Connections page.

Order Details page

Settings tab

Here you can copy your Root path, File Mask and Table meta for your data extraction.