Q:

Mean and variance of words pronounced

I'm working on linguistics and am trying to calculate the average sentence length and how much this average varies.  I worked to leave always one sentence per line
For instance:

La dernière fois qu'on, la dernière fois on l'a pas fait
Ouai

Those are 14 words, a mean of 7 per sentence and a variance of
(7-13)^2/2 + 36/2 = 36, which is really high.

I used the geditcommand and copied an past my work: for instance the begining of the file phrasesAntoine:

Allumlalum...Elal...
Allume  sinon sinon non, mais au moins pour vérifier.
Si si
La dernière fois qu'on, la dernière fois on l'a pas fait
Les amis j'vous présente Bob, Bob le gri-gri.
Salut les amis, Hey
Tiens, on te passe le grigri, celui qui a le grigri a le droit de parler
C'est lui qui a la parole
Il a dû…

I'm looking for a script that would be able to place every line of a text file in an array in order to know their length and find the mean and the variance or any kind of idea that would allow me to find this variance.
Actually "Qu'est-ce que c'est" is 6 words each one separated by a blank or ' or -
I first thought about:
file wc -l >stat

To get this information for each line, but I'm rather new to scripting... I then thought of creating another file named calculator taking the stats as an argument that is the variable $file:
file
int number_of_phrases = $file wc -l;
int mean = /*number of words divided by number of phrases*/
int sum = 0;
int variance =0 ;
for i=0 to number_of_phrases{
    /* here is the calculation of xi-m
    sum = sum + (number of words at line i divided - mean)^2*/
}
variance = sum/number_of_phrase

That is my best guess. Do you have a better idea?

A:

Perl is probably the best language for this kind of work.  Perl's primary author, Larry Wall, is both a unix programmer and a linguist, and the language strongly reflects his interest in linguistics.  There are numerous perl modules for language processing as well as simple text processing.
For example, Lingua::Sentence which is a perl module for splitting paragraphs into sentences.  And numerous other Lingua:: modules.  In fact, Lingua::Sentence and related modules were written for a task very similar to what you are doing now, statistical analysis of texts (in this case, the Europarl corpus, text extracted from the proceedings of the European Parliament)
You could, for example, use Lingua::Sentence to split each paragraph into sentences, count the number of words in each sentence, store that count in an array, and then perform whatever statistical analyses you need on the array.
Perl also has numerous modules for statistical analysis, which you can also find at CPAN (the Comprehensive Perl Archive Network) or you could output the raw data to a file and process it with R or some other stats tool.

