Advanced Settings

Advanced Settings

Panoply provides a set of advanced settings for you to customize the collection from your data sources. The options available may differ based on the data source in use.

Click on the Advanced tab to open the advanced settings. In order of frequency, advanced options apply to most, several, or very few data sources.

Schema:  The schema field allows you to select the name of the target schema that Panply will use to write the tables. The default schema is public. Once you begin collection, you cannot change the schema. If you want to use a different schema, you will need to add the data source again. For more, see Database Schemas. This setting applies to most data sources.

Destination/Destination Prefix:  The destination defines the table(s) where the incoming data is stored. Panoply defines a default destination, however, users can change that default.  For more, see Destinations. This field applies to most data sources.

Primary Key: The primary key is a field or combination of fields that Panoply uses as the deduplication key when collecting data. Panoply will use an `id` field in the incoming data, if an `id` field exists.  See the table below to understand how Panoply sets the primary key in different scenarios. To learn more about primary keys in general, see Primary Keys.

Source id column Enter a primary key Outcome
yes no Panoply will automatically select the id column and use it as the primary key.
yes yes Not recommended. Panoply will use the id column but will overwrite the original source values.
If you want Panoply to use your source’s id column, do not enter a value into the Primary Key field.
no no Panoply creates an id column formatted as a GUID, such as 2cd570d1-a11d-4593-9d29-9e2488f0ccc2.
no yes Panoply creates a hashed id column using the primary key values entered, while retaining the source columns. WARNING: Any user-entered primary key will be used across all the destination tables.

Incremental Key: The Incremental Key is used to limit the information that is pulled to what was updated since the last collection and then only update the rows that have been changed. For more, see Incremental Key. This applies to several data sources.

Delimeter: If you choose a character-delimited file with a delimiter other than comma or tab, specify the correct delimiter. This applies to very few data sources.

Exclude: Users can add a  list of attributes to exclude from the collection process, if there are specific types of data (such as irrelevant or sensitive data) that you want to exclude from your Panoply data warehouse.  Excluding attributes that are not necessary to the later data analysis can also speed up other processes in the data analysis pipeline.

To exclude a nested attribute, use dot notation, such as name.firstname.

For more, see How can I exclude sub-tables from a data source? Applies to most data sources.

Parse string: By default, Panoply converts first-level JSON objects into table columns and stores nested JSON as strings. However, you can choose to treat JSON text attributes as JSON objects by entering JSON text attributes to be parsed in the Parse string field.

For example, assume your source data included this JSON object that would be added to the users table: {"phone":"(987) 555-4321", "address":"{"street":"123 Main Street", "city":"Lincoln", "state":"Iowa", "country":"USA"}"}. If you do not enter anything into Parse string, Panoply will create a phone column and an address column, and will store the nested JSON blob as a string in the address column. If you wanted to parse the address attribute, you would type address into the Parse string field. This would create a users_address sub-table with street, city, state, and country columns.

As you enter JSON text attributes in the Parse string field, another line to type additional text attributes appears automatically. Each JSON attribute that you want to parse should be entered into its own row. To select a nested attribute, use dot notation, such as clients.billing.

Truncate table: Check this box to delete the table’s data before collecting the source. This applies to most data sources.

Deep Pagination: A few data sources use inner paging on some attributes in each resource, which can cause slow data ingestion and increase storage usage. If you don’t require the data stored in these nested structures, you can disable deep pagination in order to make ingestion more efficient. Applies to very few data sources.

Getting started is easy! Get all your data in one place in minutes.
Try Free