SFTP Data source
Note: This article covers SFTP Data source setup process. You can learn how to load data to SFTP Destination here.
How to set up SFTP integration
- On the Data sources page, click on the SFTP tile. You can find it using the search field or just open the Flat Data category.
- On the SFTP authorization page and enter the Connection name. It’s necessary for differentiating between different SFTP connections within your workspace.
- Optionally, provide your own Public key in ssh-rsa or ssh-ed25519 format to authenticate with your own SSH key. If you leave this field empty, Improvado generates the credentials for you. You can always add or change the key later in the connection's Settings tab.
- Fill in the form and click the Create button.
How to extract data from SFTP
Step 1. Import page
- Click to select a file from your computer, or drag and drop your file to this area
- Supported file formats: csv, tsv, txt, xls, xlsx, avro, parquet, zip, gzip
- Maximum size: 10 GB
- A file must begin with column headers starting in column A. If additional information is provided before — it is necessary, use the Header rows to skip field to skip a certain number of rows to import data correctly.
- After picking the file a processing bar will show the upload progress.
- You can click on the “trash bin” icon to cancel the uploading process of a file
- After your file has been uploaded and checked for security, let’s move to the settings.
- Click on the Continue button.
Step 2. Match page
- Select the fields you need to import. By default, all fields are selected.
- Check the type. Each field in a data table can be assigned a text, number, or date type.
- ~The date field is a unique date-key in the database and should be singular. Also, you have the flexibility to choose from a wide range of date formats for this field.
- ~If you have field with numeric values that you don't want to include in your calculations, change the type from Number to Text.
- The skip rule allows you to exclude empty rows and rows with specific values in a selected field
- Default to 'Do Not Skip': Ensures all data is initially included.
- ~Multi-Select Options: Combine 'Skip: empty' and 'Skip: value' for targeted exclusions. Selecting 'Do Not Skip' disables other skip options.
- ~Custom Rule addition: Add one custom rule; the form is disabled afterward to maintain focus
- ~Custom Rule deletion: Hover over and click on the red-highlighted basket icon next to the rule to exclude it.
Usage Note: the component is designed for straightforward value-based selections and does not support complex expressions like '<10'.
- Choose Write strategy that will be used when extracting data from your files and uploading it into Improvado Storage. Learn more.
- Set the Data table name which will be used for loading data to your destination.
- Click on the Import N fields button to continue.
Step 3. Preview page
- Check the Data Structure Preview.
- ~In this step, you see the first row of data that will be extracted after you click the Continue button.
- Click the Continue button to confirm settings and proceed with data extraction.
- You'll see your imported file on the Connections page.
Connection Details page
You’ll be redirected to the Settings tab of the Connection. Here, you can see your SFTP instance credentials — Host, Port, and Login. The authentication credentials shown depend on how you connected:
- If you did not provide a public key, you’ll see a Password, with an option to regenerate it.
- If you provided a public key when creating the connection, you’ll see Your public key and the Improvado public key instead.
Use the eye icon to reveal a value and the copy icon to copy it.
If you connected with a public key, use the Update credentials section to rotate it — paste your New public key and click Regenerate credentials to overwrite the existing one.
Note: You might see loading in the Connection settings section, that means that your SFTP instance is still being created. It might take some time.
Order Details page
Uploaded files
Here you can see the list of files uploaded to the SFTP Folder path corresponding to the Data table name.
Note: File retention for SFTP integration is two weeks. Files uploaded more than two weeks ago are deleted.
Settings tab
Here you can copy your Folder path and Table meta for your data extraction.
Was this article helpful?
Thanks for the feedback!