Data Collection FAQs

Data Collection Questions

Q: How can I find out what data was collected by each job?

A: Each table Panoply collects is appended with a __updatetime field that indicates what date and time each row was collected.

Q: How can I find out why some jobs failed?

A: In the Jobs screen, there will be an error message when collecting fails.

Q: Can I use a specific prefix for table names?

A: Yes. You can add your prefix to the destination in the Tables screen under the advanced setting of a data source. The field should be based on a column in the data source that has info about which table this row should go to. This changes per data source. For more information, see the Destination/Destination Prefix paragraph in the Advanced Settings page for your specific data source.

Q: How can I prevent data from being overwritten?

A: Whether we update or append a record is determined by whether or not a record with the same primary key exists in the target table. If such a record exists, we update. If not, we append. The default primary key for a Salesforce record, for example, is the id column. You can override the default primary key by specifying any combination of columns in data source definition.

Q: How can I make sure my data source refreshes?

A: You can schedule collection on an hourly, daily, or on a specific hour of a specific day. Using this feature lets you keep your Panoply DB as up to date as your business requirements dictate. For more information see Scheduling
If you happen to find that you need to update the data coming in from your data sources on-demand, you also have to option to open the data source and click “collect”. This updates the data source on demand without affecting your scheduled data source refresh, so using “collect” will be in addition to any scheduled data source refresh.