Data Management

Data Management

Using Dynamic Fields

Dynamic fields are used to tell Panoply to use a specific column in your data for one of the properties such as primary key, incremental key or destination prefix. They provide a way to fine tune how Panoply retrieves and stores your data. ... more

Incremental Keys

On data sources that support it, Panoply uses an incremental key to only pull the information that was updated since the last pull and then updates the rows that have been changed. An incremental key can be used to collect only the most up-to-date rows, saving vast amounts of time in the collection process and improving performance. ... more

Nested Data

Nested data may appear in many data sources that produce nested arrays or objects, like document-stores (Mongo), APIs, and even relational databases like Postgres JSON fields. Additionally, some of Panoply’s data enhancement features may produce nested values for flat data. For example, a record that contains an IP field may be enhanced to include a nested object with the geo-location information for that IP address. ... more

Primary Keys

Primary keys are used to ensure uniqueness of the rows in the table. Each of your tables has one or more columns that are used together to define what represents a single unique row in the data warehouse. The process guarantees that no two rows have the same value in their primary key. ... more