Advanced Options

Advanced Options

Panoply provides a set of advanced options for you to customize the collection from your data sources.

Click Show next to Advanced to expand the advanced options section. The available options vary based on the data source you have selected. In order of frequency, advanced options apply to most, several, or very few data sources.

advanced options

Schema - The name of the target schema to write the tables to. Default is public. You cannot change the schema after collection begins. For more, see Database Schemas. Applies to most data sources.

Destination - Panoply selects a default destination. These are the tables where data is stored. For more, see Destinations. Applies to most data sources.

Primary Key - Field(s) to use as the deduplication key when collecting data. Panoply sets the primary key depending on the scenario identified in the following table. To learn more about primary keys in general, see Primary Keys.

Source id column Enter a primary key Outcome
yes no Panoply will automatically select the id column and use it as the primary key.
yes yes Not recommended. Panoply will use the id column but will overwrite the original source values.
If you want Panoply to use your source’s id column, do not enter a value into the Primary Key field.
no no Panoply creates an id column formatted as a GUID, such as 2cd570d1-a11d-4593-9d29-9e2488f0ccc2.
no yes Panoply creates a hashed id column using the primary key values entered, while retaining the source columns. WARNING: Any user-entered primary key will be used across all the destination tables.

Incremental Key - Used to only pull information that was updated since the last collection and then only update the rows that have been changed. For more, see Incremental Key. Applies to several data sources.

Delimeter - If you choose a character-delimited file with a delimiter other than comma or tab, specify the correct delimiter. Applies to very few data sources.

Exclude - A list of attributes to exclude from the collection process. If there are specific types of data (such as irrelevant or sensitive data) that you want to exclude from your Panoply data warehouse, you can use this setting to manage that. Excluding attributes that are not necessary to the later data analysis can also speed up other processes in the data analysis pipeline.

To exclude a nested attribute, use dot notation, such as name.firstname.

For more, see How can I exclude sub-tables from a data source? Applies to most data sources.

Parse string - By default, Panoply converts first-level JSON objects into table columns and stores nested JSON as strings. However, you can choose to treat JSON text attributes as JSON objects by entering JSON text attributes to be parsed in the Parse string field.

For example, assume your source data included this JSON object that would be added to the users table: {"phone":"(987) 555-4321", "address":"{"street":"123 Main Street", "city":"Lincoln", "state":"Iowa", "country":"USA"}"}. If you do not enter anything into Parse string, Panoply will create a phone column and an address column, and will store the nested JSON blob as a string in the address column. If you wanted to parse the address attribute, you would type address into the Parse string field. This would create a users_address sub-table with street, city, state, and country columns.

As you enter JSON text attributes in the Parse string field, another line to type additional text attributes appears automatically. Each JSON attribute that you want to parse should be entered into its own row. To select a nested attribute, use dot notation, such as clients.billing.

Truncate table - Check this box to delete the table’s data before collecting the source. Applies to most data sources.

Deep Pagination - A few data sources use inner paging on some attributes in each resource, which can cause slow data ingestion and increase storage usage. If you don’t require the data stored in these nested structures, you can disable deep pagination in order to make ingestion more efficient. Applies to very few data sources.

Getting started is easy! Get all your data in one place in minutes.
Try Free