x2.pm
Statistical library package to calculate Pearson's Chi Squared test. This package should be used with statistic.pl and rank.pl
Pearson's Chi-squred test measures the devitation between the observed data and what would be expected if <word1> and <word2> were independent. The higher the score, the less evidence there is in favor of concluding that the words are independent.
Assume that the frequency count data associated with a bigram <word1><word2> is stored in a 2x2 contingency table:
word2 ~word2 word1 n11 n12 | n1p ~word1 n21 n22 | n2p -------------- np1 np2 npp
where n11 is the number of times <word1><word2> occur together, and n12 is the number of times <word1> occurs with some word other than word2, and n1p is the number of times in total that word1 occurs as the first word in a bigram.
The expected values for the internal cells are calculated by taking the product of their associated marginals and dividing by the sample size, for example:
np1 * n1p m11= --------- npp
Then the deviation between observed and expected values for each internal cell is computed to arrive at the Pearson's Chi-Squared test value:
Pearson's Chi-Squared = 2 * [((n11 - m11)/m11)^2 + ((n12 - m12)/m12)^2 + ((n21 - m21)/m21)^2 + ((n22 -m22)/m22)^2]
Ted Pedersen <tpederse@d.umn.edu>
Satanjeev Banerjee <banerjee@cs.cmu.edu>
Bridget Thomson McInnes <bthomson@d.umn.edu>
This measure currently only defined for bigram data stored in 2x2 contingency table.
Mailing List: http://groups.yahoo.com/ngram
Copyright 2000-2004 by Ted Pedersen and Satanjeev Banerjee
This program is free software; you can redistribute it and/or modify it under the terms of the GNU General Public License as published by the Free Software Foundation; either version 2 of the License, or (at your option) any later version.
This program is distributed in the hope that it will be useful, but WITHOUT ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License for more details.
You should have received a copy of the GNU General Public License along with this program; if not, write to the Free Software Foundation, Inc., 59 Temple Place - Suite 330, Boston, MA 02111-1307, USA.