I'm working on a project that involves loading batches of 3000+ files to several dozen tables.  There is no user interface, and the tables are simply available for querying.  What are the best practices for testing this type of process?Loading a small set of data and validating each and every member?Loading all the data and validating a subset?Loading all the data and validating counts, averages, and other metrics?Are there other types of testing that can be done, or is some combination best?
How do you test a data load?
testing
I would create two (sets of) test cases. One that runs often, is fast and validates only statistically and another one that runs less often (depending on your project schedule - every week or month) but checks all the data.The fast test would import all the data then check:the number of recordstake a number of records and check their integrity (pick every K records where K could be something like number of records / 100)if you have columns that support aggregate operations which execute fast you might want to check those too in the fast tests.For the test that is executed less often just do a full integrity check either by doing 1-1 comparisons or by computing hashes.