GOODNESS OF FIT TESTS

Stefania Donadio, Barbara Mascialino, Paolo Viarengo

INFN Sezione di Genova

The goodness of fit tests are introduced with the aim of verifying that the hypothesis that experimental data come from a random variable whose distribution is well known. This problem is very important both in theoretical and in experimental analysis. The researcher must decide if  theoretical and experimental distribution follow the same functional law. In other words, the problem is concerned with the choice of one of these two alternative hypothesis:

 

H0: FO(x) = FT(x)

H1: FO(x) ¹ FT(x), FO(x) < FT(x), FO(x) > FT(x)

 

Of course, in this kind of tests the acceptance of the null hypothesis H0 means that the researcher will be able to specify the distribution analyzed.

The most important and known test is certainly Pearson's Chi-squared test. It was introduced to study discrete (both quantitative and qualitative) distributions' adaptation. It can be useful also in case of continuous distributions, but the data must be grouped into classes. This test cannot be applied only if the counting of the theoretical frequencies in each class are less than 5. When this is not the case one could try to unify contiguous classes until the minimum theoretical frequency is not reached. 

Among non-parametrical tests the easiest is undoubtedly Kolmogorov test. It is useful to verify the adaptation of a sample coming from a random continuous variable and it is based on the computation of the maximum distance between an empirical repartition function and the theoretical repartition one specified in the null hypothesis H0. If H0 is true, FO(x) ->FT(x) and the difference between the empirical repartition function and the theoretical one will be minimum; otherwise if H0 is false, the difference will be noticeable. So, it is useful to use the following measure:

                 D = sup |FO(x) - FT(x)|              

as a test statistics to check if to accept or to refuse H0. A comparison between Chi-squared test and Kolmogorov test is not possible in an unique way because not always information and conditions are the same. If the observations come from a continuous random variable, these tests are alternative for the same problem. A comparison between the power of these tests is very difficult because Chi-squared test's power depends on the specific clustering of data into groups. 

A problem mathematically similar to Kolmogorov's was studied by Smirnov: it is the classical problem of the two samples. Instead of comparing an empirical distribution with a theoretical one, the researcher can try to find the maximum difference between the two samples' distributions Fn and Gm:

Dmn= sup |Fn(x) - Gm(x)|  

The limit of Kolmogorov-Smirnov test is that it can be applied only on continuous random variables. Conover (1971) and Gibbons and Chakraborti (1992) tried to extend this test to cases of discrete random variables. Other techniques that could allow the application of this test on discrete variables are listed in the observations below.     

Lilliefors test is similar to Kolmogorov test, but is based on the null hypothesis that the random continuous variable is distributed as a normal N(m,s2), when m and s2 are unknown. In practice, being the parameters unknown, the researcher must estimate them from the sample itself (x1,x2,...,xn) and in this way it becomes possible to study the standardized sample (z1,z2,...,zn). The test is performed comparing the empirical repartition function F of (z1,z2,...,zn) with the one of the standardized normal distribution F(z):

D* = sup |FO(z) - F(z)| 

The following two tests can be performed both on continuous and discrete variables:

 Cramer-von Mises test is based on the test statistics:

w2= ò (FO(x) - FT(x))2 dF(x)

This test is satisfying for symmetric and right-skewed distributions.

 Anderson-Darling test: the test is performed on the test statistic:

        A2= ò{[FO(x) –FT(x)]2 / [FT(x)(1 - FT(X))]}dFT(x)

This test seems to be suitable for any data-set (Aksenov and Savageau - 2002) with any skewness (symmetric distributions, left or right skewed). Moreover it seems to be sensible to fat tail of distributions. 

 

Kuiper test is based on a quantity that remains invariant for any shift or re-parameterization, but the test does not work well on distributions' tails:

D* = max (FO(x)-FT(x)) + max (FT(x)-FO(x))

 

Observations

* The problem that Kolmogorov-Smirnov test can be applied only on continuous random variables could be solved in other ways. As a matter of the fact, the application of this test could be enlarged to discrete variables, but only if these ones undergo a certain manipulation. In fact, following Dagum, it is always possible to try to find the best fit of the empirical distribution (this fit could also be  the mixture of more than one fit). Kolmogorov-Smirnov test statistics could be computed between the fitted function (because now it is a continuous one) and the theoretical distribution. At this point little changes must be made on the original test: as the fitted function is an estimation of the empirical distribution, the researcher must in some way consider this fact altering the number of degrees of freedom. Or, in another way, without changing the number of degrees of freedom, it is possible to compare the value of the test statistics found with new critical values from other tables. This technique is able to weight fat tails of distribution, while classical Kolmogorov-Smirnov test is not so sensible in these cases. The whole problem could also be solved in another way: once the researcher has found the best fit of the empirical distribution he could simulate data (transforming them in continuous data) by means of Monte Carlo and then he could apply Kolmogorov-Smirnov test as it is between the continuous simulated data and the theoretical function (that means without any adjustment).     

 

* Another problem concerns the choice of the theoretical function that must be introduced in the null hypothesis of these tests. There are a lot of criteria in order to choose the best theoretical model, but the ones based on Bayes theorem are the best ones. In fact, theoretically speaking a priori one could have a number k of models: 

M1, M2, ..., MK

It is possible to choose the model that maximizes the probability of the posterior distribution. This technique is a generalization of classical criteria based on likelihood ratio. Moreover it has a great internal coherence and allows the researcher to consider the plurality of all possible models. It is based on Bayes factor and it is analytical in the simplest cases, while it needs Winbugs (Gibbs Sampling Technique) or Markov Chain Monte Carlo in the most difficult cases. A much more sophisticated way to achieve the same result is using the BMA technique (Bayesian Model Averaging). In fact it weights the level of uncertainty that exists in the models' selection process. It could be very useful when one has to choose among too many models.

Theoretical models could also be estimated from empirical observations by means of a simple Ferguson process or with a mixture of several Ferguson processes. These techniques are based on a superposition of an a priori weight with an empirical one. In this way at any time the process is able to update the theoretical distribution on data.  

Comparison of multi-dimensional distributions
Comparison among several distributions


Barbara.Mascialino@ge.infn.it

Last update: 03/03/2006