I'm sorting French language words in some text files according to frequency with a focus on insight rather than statistical significance. The challenge is about preserving accented characters and dealing with the article forms in front of vowels(l', d') in the context of shaping word tokens for sorting.The topic of the most frequent words in a file takes many shapes( 1 | 2 | 3 | 4). So I put together this function using GNU utilities:compt1 () {for i in *.txt; do    echo File: $i    sed -e 's/ /\/g' <$i | sed -e 's/^[[:alpha:]][[:punct:]]\(.*\)/\1/' | sed -e 's/\(.*\)/\L\1/' | grep -hEo [[:alnum:]_'-]+ | grep -Fvwf /path_to_stop_words_file | sort | uniq -c | sort -rn done}...which trades spaces for newlines; trims a character followed by punctuation that's at the beginning of the line; then converts everything to lowercase;  uses this compact grep construct which matches word constituent characters to create tokens; then removes the stop words, and finally there is the usual sorting. The stop file contains a segment with individual characters so you have to be careful with how it's used, but the analysis provided on how to create stems for words in different languages is really interesting! Now when I compare the frequency of a significant word with the output of grep -c directly on the files, I think it's close enough within some margin of error. Questions:How could I modify this to merge the frequency of plurals with theirsingular forms i.e. words sharing a common prefix with a varying 1character suffix?I'm trying to assess whether the grep part in particular would work with what's on OSX?1. I cannot provide source data but I can provide this file as an example. The words heure and enfant in the text provide an example. The former appears twice in the text including once as l'heure, and helps validating if the command works or not. The latter appears in both singular and plural forms(enfant/enfants) and would benefit from being merged here.
Frequency of words in non-English language text: how can I merge singular and plural forms etc.?
shell script;text processing;sed;portability;natural language
You really are not going to be able to do this with a simplistic sed script. Im assuming that you will want to reduce to citation forms, collapsing all inflections into a base form.  That means that adjectives like protg, protgs, protge, protges all count as the same thing, the base adjective/participle protg.  Similarly, all inflections of the verb protger  like protge, protgeons, protgeais, protgeasse, protgemes, protgeront, protgeraient, etc.   would all reduce to that base verb.That means you need to know things about the inflectional morphology of the language.  Even worse, you will need to understand something about the actual syntax of language, including for the inflections and to distinguish homographs.I have done very simple approaches to at least the first part of this using Perl.  Its really rather a pain in the butt.  Heres a sample of code I used for generating sort keys for cities and towns on the Iberian peninsula:       # 1st strip leading articles          s/^L'//;    # Catalan          s{ ^            (?:        # Castilian                El              | Los              | La              | Las        # Catalan               | Els              | Les                       | Sa              | Es        # Gallego              | O                     | Os              | A              | As                  )             \s+           }{}x;        # 2nd strip interior particles          s/\b[dl]'//g;   # Catalan          s{             \b            (?:                el  | los | la | las | de  | del | y          # ES              | els | les | i  | sa | es | dels               # CA               | o   | os  | a  | as  | do  | da | dos | das   # GAL            )            \b        }{}gx;That strips the articles and particles so that they dont count for purposes of sortation. But you will have to deal with forms like lautre with a so-called curly-quote, which is really U+2019  RIGHT SINGLE QUOTATION MARK, the preferred form for the apostrophe.  I normalized those into straight ones with a s//'/g first.Oh, and you will have to deal with encodings: MacRoman is not the same as UTF-8 or ISO-8859-1  not by a long shot.Honestly, you probably want to use something like the Snowball stemming algorithm, specifying French as the language.  Certainly Perls Lingua::Stem::Snowball module knows how to do this.  You can search for Perl modules having to do with French linguistics using this query.But stemming will only take you so far.  You wont really do a good job until you apply morphosyntactic analysis  which means you have to generate a parse for the sentences and assign parts of speech to each element there.This requires much more work.  The good news is that there are dedicated tools for this out there, some of which do indeed work on French.  But this really is biting off a great deal, because now youve ventured into the fields of Natural Language Processing and Computational Linguistics.  There is no great home for such questions here, but they might be probably better answered on Linguistics.SE; I dont know.