Dynamo DB

Dynamo DB

This document describes the Dynamo DB data source. Continue reading to learn more about:

  • Collecting - what should you know about adding the data source.
  • Data Dictionary - what data is available and how it is structured.

Collecting

To configure this data source:

  1. If necessary, whitelist Panoply.
    • To ensure data security, Dynamo DB databases with production data are typically not publicly available. To allow Panoply to access your data, see Whitelisting.
  2. Click Data Sources in the navigation menu.
  3. Click the Add Data Source button.
  4. Search for Dynamo DB and select it.
  5. On the General Tab:
    • API Key: Enter your IAM key
    • API Secret: Enter your IAM secret
    • Region: Select the region
    • Data Available: Select the DynamoDB tables to collect
  6. (Optional) Set the Advanced Settings. We do not recommend changing advanced settings unless you are an experienced Panoply user.
    • Destination Prefix - The default destination is dynamo_<table or view name> , where <table or view name> is a dynamic field. For example, for a table or view name customers, the default destination table is dynamo_customers.
    • For more detailed descriptions of Advanced Settings for the Dynamo DB Data Source, see the Data Dictionary below.

7. Click Save Changes and then click Collect.

  • The data source appears grayed out while the collection runs.
  • You may add additional data sources while this collection runs.
  • You can monitor this collection from the Jobs page or the Data Sources page.
  • After a successful collection, navigate to the Tables page to review the data results.

Data Dictionary

Because Dynamo DB data comes from a database system, Panoply cannot provide a data dictionary. But Panoply does automate the data schema for the collected data. This section includes useful information about the Panoply automations. You can adjust these settings in your data source under Advanced Settings.

  • Destination Prefix: Panoply combines a prefix with the name of the table or view in your database.
    • The default destination is dynamo_<table or view name> , where <table or view name> is a dynamic field. For example, for a table or view name customers, the default destination table is dynamo_customers.
    • To prefix all table names with your own prefix, enter your desired prefix and Panoply will add theΒ <table or view name>Β .
  • Incremental Key - By default, Panoply fetches all of your Dynamo DB data on each run. If you only want to collect some of your data, enter a column name to use as your incremental key. The column must be logically incremental. Panoply will keep track of the maximum value reached during the previous run and will start there on the next run.
    • Incremental Key configurations
      • If no Incremental Key is configured by the user, Panoply collects all the DynamoDB data on each run for the DynamoDB tables or views selected.
      • If the Incremental Key is configured by column name, but not the column value, Panoply collects all data, and then automatically configures the column value at the end of a successful run.
      • If the Incremental Key is configured by column name and the column value (manually or automatically), Panoply will use that value as the place to begin the collection.
        • The value is updated at the end of a successful collection to the last value collected.
        • In future collections, the new value is used as the starting value. So in future collections Panoply looks for data where the IK value is greater than where the collection ended.
    • When an Incremental Key is configured, Panoply will look for that key in each of the selected tables and views. If the table or view does not have the column indicated as the Incremental Key, it must be collected as a separate instance of the data source.
    • A table or view may have some records that have a β€˜null’ value for the incremental key, or they may not capture the incremental key at all. In these situations Panoply omits these records instead of failing the entire data source.

**WARNING:** If you set an incremental key, you can only collect one table per instance of Dynamo DB.

A column in a table uses the same data type for all values in that column. Panoply automatically chooses the data type for each column based on the available values. This is important to note for this data source. If even one value in a column has text, then the entire column is considered data type text.

  • The following metadata columns are added to the destination table(s):

    • __pk: The primary key value as defined in the table
    • __senttime - Formatted as a datetime, such as 2020-04-26T01:26:14.695Z.
    • __updatetime - Formatted as a datetime, such as 2020-04-26T01:26:14.695Z.

Data Type Mapping

Getting started is easy! Get all your data in one place in minutes.
Try Free