Advanced Settings

Advanced Settings

Panoply documentation is moving! Click here to view the new site!  __________________________________________________________________________________________________________________ Panoply provides a set of advanced settings for you to customize the collection from your data sources. 

Click on Advanced Settings to open the advanced settings. This list is not all of the advanced settings that may be available.  See the individual Data Source documents for detailed descriptions of the advanced settings that are applicable.

Schema: The schema field allows you to select the name of the target schema that Panply will use to write the tables. If your data warehouse is built on Amazon Redshift, the default schema is public. If your data warehouse is built on Google BigQuery, the default schema is panoply. Once you begin collection, you cannot change the schema. On Redshift, if you want to use a different schema, you will need to add the data source again. For more, see Database Schemas. This setting applies to most data sources.

Destination: The destination defines the table(s) where the incoming data is stored. All alphabetic characters are converted to lowercase.

Destination Prefix: The destination prefix is used to define the table(s) where the incoming data is stored. Panoply defines a default destination prefix, however, users can change that default. All alphabetic characters are converted to lowercase.

Primary Key: The primary key is a field or combination of fields that Panoply uses for deduplication when collecting data. If the incoming data has an id field Panoply will recognize it and make it the primary key.  The way Panoply stores the primary key differs based on whether your data warehouse is built in Redshift or BigQuery. To learn more about primary keys, see Primary Keys.

Incremental Key: The Incremental Key is used to limit the information that is pulled to what was updated since the last collection and then only update the rows that have been changed. For more, see Incremental Key. This applies to several data sources.

Exclude: Users can add a list of attributes to exclude from the collection process, if there are specific types of data (such as irrelevant or sensitive data) that you want to exclude from your Panoply data warehouse. Excluding attributes that are not necessary to the later data analysis can also speed up other processes in the data analysis pipeline.

To exclude a nested attribute, use dot notation, such as name.firstname.

Currently, users can only exclude an entire nested array and not specific fields within any object in the arrays. For example, if there is a nested array of objects such as line_items within the orders object, you can exclude the entire line_items array, but not individual fields within line_items.

Parse string:  (this applies only to data warehouses built on Redshift) By default, Panoply converts first-level JSON objects into table columns and stores nested JSON as strings. However, you can choose to treat JSON text attributes as JSON objects by entering JSON text attributes to be parsed in the Parse string field.

For example, assume your source data included this JSON object that would be added to the users table: {"phone":"(987) 555-4321", "address":"{"street":"123 Main Street", "city":"Lincoln", "state":"Iowa", "country":"USA"}"}. If you do not enter anything into Parse string, Panoply will create a phone column and an address column, and will store the nested JSON blob as a string in the address column. If you wanted to parse the address attribute, you would type address into the Parse string field. This would create a users_address sub-table with street, city, state, and country columns.

As you enter JSON text attributes in the Parse string field, another line to type additional text attributes appears automatically. Each JSON attribute that you want to parse should be entered into its own row. To select a nested attribute, use dot notation, such as clients.billing.

Truncate table: Truncate deletes all the current data stored in the destination tables, but not the tables themselves. Afterwards Panoply will recollect all the available data for this data source.

Next Steps

Getting started is easy! Get all your data in one place in minutes.
Try Free