Panoply provides a set of advanced options for you to customize the collection from your data sources.
Click Show next to Advanced to expand the advanced options section. The available options vary based on the data source you have selected. In order of frequency, advanced options apply to most, several, or very few data sources.
Schema - The name of the target schema to write the tables to. Default is
public. You cannot change the schema after collection begins. For more, see Database Schemas. Applies to most data sources.
Destination - Panoply selects a default destination. These are the tables where data is stored. For more, see Destinations. Applies to most data sources.
Primary Key - Field(s) to use as the deduplication key when collecting data. Default is
id, which is a Panoply-defined ID. For more, see Primary Keys. Applies to most data sources.
Incremental Key - Used to only pull information that was updated since the last collection and then only update the rows that have been changed. For more, see Incremental Key. Applies to several data sources.
Delimeter - If you choose a character-delimited file with a delimiter other than comma or tab, specify the correct delimiter. Applies to very few data sources.
Exclude - A list of attributes to exclude from the collection process. If there are specific types of data (such as irrelevant or sensitive data) that you want to exclude from your Panoply data warehouse, you can use this setting to manage that. Excluding attributes that are not necessary to the later data analysis can also speed up other processes in the data analysis pipeline.
To exclude a nested attribute, use dot notation.
For more, see How can I exclude sub-tables from a data source? Applies to most data sources.
Parse string - If the data that will be ingested into Panoply contains JSON, include the JSON text attributes to be parsed upon ingestion in the “Parse string” field. As you type the JSON text attributes in the “Parse string” field, another line to type additional text attributes appears automatically. Applies to most data sources.
Truncate table - Check this box to delete the table’s data before collecting the source. Applies to most data sources.
Deep Pagination - A few data sources use inner paging on some attributes in each resource, which can cause slow data ingestion and increase storage usage. If you don’t require the data stored in these nested structures, you can disable deep pagination in order to make ingestion more efficient. Applies to very few data sources.