- 28 Nov 2022
- 2 Minutes to read
Amazon S3 Advanced Settings
- Updated on 28 Nov 2022
- 2 Minutes to read
We do not recommend changing advanced settings unless you are an experienced Panoply user.
For users who have some experience working with their data in Panoply, there are a number of items that can be customized for this data source.
- Files Encoding: The following encodings are available:
Otheris selected then during the extraction the data source will detect automatically the file’s encoding. This is useful when the user does not know the file encoding or if he is selecting multiple files in multiple different encodings.
- Destination Schema: This is the name of the target schema to save the data. The default schema is Public. This cannot be changed once a source had been collected.
- Primary Key: The primary key is an
idfield that defines the column that contains the table's Primary Key. If this option is left blank and the sheet does not contain an
IDcolumn, Panoply will insert an
id, formatted as a GUID, such as
||User Entered primary key||Outcome|
|yes||no||Panoply will automatically select the
|yes||yes||Not recommended. Panoply will use the
If you want Panoply to use your database table's
|no||no||Panoply creates an
|no||yes||Panoply creates a hashed
WARNING: Any user-entered primary key will be used across all the files selected.
- Incremental Load: An incremental load is only available when the user chooses Collect All Files. This option means that Panoply will only collect files that have been updated since the last successful collection. When selecting specific files, Panoply will collect the entire file. When an incremental key is entered, it will be applied to all files collected.
4. Delimiter: For character-delimited files like .csv or .txt, that do not use a comma or a tab for the delimiter, use the dropdown to indicate the correct delimiter to use.
5. Exclude: The Exclude option allows you to exclude certain data, such as names, addresses, or other personally identifiable information. Enter the column names of the data to exclude.
6. Parse String: If the data to be collected contains JSON, include the JSON text attributes to be parsed.
7. Truncate: Truncate deletes all the current data stored in the destination tables, but not the tables themselves. Afterwards Panoply will recollect all the available data for this data source.
8. Click Save Changes and then Collect.
- The data source appears grayed out while the collection runs.
- You may add additional data sources while this collection runs.
- You can monitor this collection from the Jobs page or the Data Sources page.
- After a successful collection, navigate to the Tables page to review the data results.