Overview

Overview

To collect data from multiple sources into your Panoply data warehouse, you must add data sources.

Panoply currently supports all major databases as data sources, including MySQL, PostgreSQL, SQL, Redshift, MongoDB, Google Datastore, Amazon Aurora, Cassandra, and CouchDB. Always connect to a replication of your production database. Connecting to the production database is against best practices.

Panoply also supports APIs, Message Queues, the Panoply SDKs, and file upload options to add as data sources. For a complete list of data sources, check out our Data Integrations page.

Importing Data

To add a data source:

  1. Click Data Sources in the navigation pane.

  2. Click Add Data Source.

  3. Select your data source from the list.

  4. Enter the relevant credentials. For example, username, password, and URL, IP address, or API details. Or click File Upload to load data without providing credentials. You can also define the following properties:
    • Destination table name, which is the name of the target table where you want to save the data.
    • Primary key for the destination table for de-duplication with a pattern.
    • List of attributes to exclude from the collection process.
    • List of JSON attributes to parse as objects.
  5. Click Collect to begin the data collection process. A progress bar shows you when the data is being analyzed, modeled, and uploaded into Panoply.

Scheduling a Data Source

To schedule a data source:

  1. Click Data Sources in the navigation pane.

  2. In the data source row you want to schedule, click the Schedule icon .

  3. Select the day frequency to run the ingestion, which can be either a specific day of the week or daily.

  4. Select your preferred time option, which can either be an hour interval (every 1, 2, 4, 6, or 12 hours), or a specific hour of the day.

Troubleshooting Data Source Ingestions

On the Data Sources page, each row notes the last runtime for that data source’s ingestion. When a data source ingestion is not successful, the last runtime displays in red text.

To troubleshoot a failed data source ingestion:

  1. Click the red last runtime text to display up to 10 previous runs, including any error messages for failed runs.

  2. For additional information, click Jobs in the navigation pane.

  3. Use the status filter to display only items with an Error status. Items with an Error status are indicated by a red dot.

  4. Click a row to expand and show more detail about the failed ingestion run.