SFTP Advanced Settings

Article summary

Did you find this summary helpful?

Thank you for your feedback

We do not recommend changing advanced settings unless you are an experienced Panoply user.

For users who have some experience working with their data in Panoply, there are a number of items that can be customized for this data source.

Files Encoding: The following encodings are available:
- UTF-8
- ISO-8859-1
- Windows-1251
- Windows-1252
- Windows-1254
- Other
  When Other is selected then during the extraction the data source will detect automatically the file’s encoding. This is useful when the user does not know the file encoding or if he is selecting multiple files in multiple different encodings.
Delimeter: For character-delimited files like .csv or .txt, that do not use a comma or a tab for the delimiter, use the dropdown to indicate the correct delimiter to use.
Destination Schema: This is the name of the target schema to save the data. The default schema for data warehouses built on Google BigQuery is panoply. The default schema for data warehouses built on Amazon Redshift is public. This cannot be changed once a source has been collected.
Primary Key - The Primary Key is the field or combination of fields that Panoply will use as the deduplication key when collecting data. Panoply sets the primary key depending on the scenario identified in the following table. To learn more about primary keys in general, see Primary Keys.

`id` column	Enter a primary key	Outcome
yes	no	Panoply will automatically select the `id` column and use it as the primary key.
yes	yes	Not recommended. Panoply will use the `id` column but will overwrite the original source values. If you want Panoply to use your database table's `id` column, do not enter a value into the Primary Key field.
no	no	Panoply creates an `id` column formatted as a GUID, such as `2cd570d1-a11d-4593-9d29-9e2488f0ccc2`.
no	yes	Panoply creates a hashed `id` column using the primary key values entered, while retaining the source columns.

Warning:

Any user-entered primary key will be used across all of the files selected.

Skip XML attributes: When collecting XML files, some of the returning XML fields might have attributes attached to them. Select this option to skip all of the XML attributes and ingest only the XML values. For example, for the data 100, Panoply will ingest the value 100 to the score column
Exclude: The Exclude option allows you to exclude certain data, such as names, addresses, or other personally identifiable information. Enter the column names of the data to exclude.
Parse String: If the data to be collected contains JSON, include the JSON text attributes to be parsed.
Truncate: Truncate deletes all the current data stored in the destination tables, but not the tables themselves. Afterwards Panoply will recollect all the available data for this data source.
Click Save Changes and then click Collect
- The data source appears grayed out while the collection runs.
- You may add additional data sources while this collection runs.
- You can monitor this collection from the Jobs page or the Data Sources page.
- After a successful collection, navigate to the Tables page to review the data results.

Was this article helpful?