<commit_before>gawk for beginners
==================

gawk (GNU awk) is a programable unix text utility. It is one of the most complex
of them. Several simple example will be introduced to teach you by examples.

gawk processes the input text file line by line. Each line is devided intos
fields using a set of field separators. The default field separators are *TAB* 
and *space*.  We can refere to a field in the actual line by $1 (first field),
$2 (second field), etc. $0 gives the whole actual input line.

There are some variables maintained by gawk. The most important ones are:

* *NF* number of fields in the actual input line
* *NR* row number in input file
* *FS* field separator(s)

The gawk program can be given in the command line if it is very short or we
can create a separate file for the program. The usual extension for gawk
program file is *.awk*.

gawk reads data from the input file or from the standard input and prints
output to the standard output.

The syntaxis of the gawk programs are very similiar to C programs. gawk can be 
a good start point to learn C.

Add row numbers to a text file
------------------------------

.. code:: gawk

    gawk '{ print NR, $0 }' /etc/passwd

The example above will print back the Linux password file with line numbers to
the standard output. The output of the gawk command can be redirected to 
the a file e.g.

.. code:: gawk

    gawk '{ print NR, $0 }' /etc/passwd > row_num_passwd

Remove columns from input file
------------------------------

.. code:: bash

   gawk -F : '{ print $1, $3, $5 }' /etc/passwd

The example above will print out the Linux user names, user ID and full name.<commit_msg>More gawk genaral info added<commit_after>gawk for beginners
==================

gawk (GNU awk) is a programable unix text utility. It is one of the most complex
of them. Several simple example will be introduced to teach you by examples.

gawk processes the input text file line by line. Each line is devided intos
fields using a set of field separators. The default field separators are *TAB* 
and *space*.  We can refere to a field in the actual line by $1 (first field),
$2 (second field), etc. $0 gives the whole actual input line.

There are some variables maintained by gawk. The most important ones are:

* *NF* number of fields in the actual input line
* *NR* row number in input file
* *FS* field separator(s)

The gawk program can be given in the command line if it is very short or we
can create a separate file for the program. The usual extension for gawk
program file is *.awk*.

gawk reads data from the input file or from the standard input line by line,
processes the input and prints output to the standard output.

The syntaxis of the gawk programs are very similiar to C programs. gawk can be 
a good start point to learn C.

A *gawk* program consists of pattern and action pairs. The pattern must match
the actual input line to trigger the action. All matching actions are 
evaluated, not only the first!

There are some kinds of action:

* Regexp pattern: a regular expression e.g. /^[0-9]+/
* Expression pattern: any expression e.g. NF > 4
* Range: pairs of patterns specif record range e.g. /^ENTITIES/, /^ENDSEC/
* BEGIN: special pattern to initialize
* END: special pattern to clean up
* empty: the empty pattern mathes every input line

Add row numbers to a text file
------------------------------

.. code:: gawk

    gawk '{ print NR, $0 }' /etc/passwd

The example above will print back the Linux password file with line numbers to
the standard output. The output of the gawk command can be redirected to 
the a file e.g.

.. code:: gawk

    gawk '{ print NR, $0 }' /etc/passwd > row_num_passwd

Remove columns from input file
------------------------------

.. code:: bash

   gawk -F : '{ print $1, $3, $5 }' /etc/passwd

The example above will print out the Linux user names, user ID and full name.
{'fraction_non_alphanumeric': 0.09869264291207383, 'fraction_numerical': 0.004870546013842604, 'mean_word_length': 3.142250530785563, 'pattern_counts': {'":': 0, '<': 3, '<?xml version=': 0, '>': 6, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '5090017', 'n_tokens_mistral': 1090, 'n_tokens_neox': 1064, 'n_words': 634}