Q:

How do I determine how well a dataset approximates a distribution?

Quite simple, I have some probability distribution p(x), how can I measure whether one empirical density (set of delta masses) is a better approximation than another.  I know that KL-divergence is a well accepted measure between two continuous densities, but it's not clear how to apply that to a set of samples.

A:

For visualization purposes, try a Q-Q plot, which is a plot of the quantiles of your data against the quantiles of the expected distribution.  
If you want a statistical test, the Kolmogorov-Smirnov statistic provides a non-parametric test for whether the data come from $p(x)$, using the maximum difference in the empirical and analytic cdf. 
Of course, you could also evaluate the log-probability of your data under the two  distributions:  $L_1 = \sum_i p_1(X_i)$  vs. $L_2 = \sum_i p_2(X_i)$, and take whichever is larger.  This is equivalent to maximum likelihood density comparison. (However, this may not be valid if $p_1$ and $p_2$ are distributions fit to your data, especially if they have different numbers of fitted parameters; in that case you want to do "model comparison", and there are a variety of tools for this— AIC, BIC, Bayes Factors, Likelihood-ratio test, Cross-validation, etc.)

