<commit_before># modernish: a shell modernizer library #

modernish is an ambitious, as-yet experimental, cross-platform POSIX shell
feature detection and language extension library. It aims to extend the
shell language with extensive feature testing and language enhancements,
using the power of aliases and functions to extend the shell language
using the shell language itself.

The name is a pun on Modernizr, the JavaScript feature testing library, -sh,
the common suffix for UNIX shell names, and -ish, still not quite a modern
programming language but perhaps a little closer. jQuery is another source
of general inspiration; like it, modernish adds a considerable feature set
by using the power of the language it's implemented in to extend/transcend
that same language.

That said, the aim of modernish is to build a better shell language, and not
to make the shell language into something it's not. Its feature set is aimed
at solving specific and commonly experiened deficits and annoyances of the
shell language, and not at adding/faking things that are foreign to it, such
as object orientation or functional programming. (However, since modernish
is modular, nothing stops anyone from adding a module attempting to
implement these things.)

The library builds on pure POSIX 2013 Edition (including full C-style shell
arithmetics with assignment, comparison and conditional expressions), so it
should run on any POSIX-compliant shell and operating system. But it does
not shy away from using non-standard extensions where available to enhance
performance or robustness.

Some example programs are in `share/doc/modernish/examples`.

Modernish also comes with a suite of test scripts to detect bugs in
modernish itself, which can be run using `modernish --test` after
installation.


## Getting started ##

Run `install.sh` and follow instructions, choosing your preferred shell
and install location. After successful installation you can run modernish
shell scripts and write your own. Run `uninstall.sh` to remove modernish.


## Two basic forms of a modernish program ##

The **simplest** way to write a modernish program is to source modernish as a
dot script. For example, if you write for bash:

    #! /bin/bash
    . modernish
    use safe
    use sys/base
    ...your program starts here...

The modernish 'use' command load modules with optional functionality. `safe` is
a special module that introduces a new and safer way of shell programming, with
field splitting (word splitting) and pathname expansion (globbing) disabled by
default. The `sys/base` module contains modernish versions of certain basic but
non-standardised utilities (e.g. `readlink`, `mktemp`, `which`), guaranteeing
that modernish programs all have a known version at their disposal. There are
many other modules as well. See below for more information.

The above method makes the program dependent on one particular shell (in this
case, bash). So it is okay to mix and match functionality specific to that
particular shell with modernish functionality.

The **most portable** way to write a modernish program is to use the special
generic hashbang path for modernish programs. For example:

    #! /usr/bin/env modernish
    #! use safe
    #! use sys/base
    ...your program begins here...

A program in this form is executed by whatever shell the user who installed
modernish on the local system chose as the default shell. Since you as the
programmer can't know what shell this is (other than the fact that it passed
some rigorous POSIX compliance testing executed by modernish), a program in
this form *must be strictly POSIX compliant* -- except, of course, that it
should also make full use of the rich functionality offered by modernish.

Note that modules are loaded in a different way: the `use` commands are part of
hashbang comment (starting with `#!` like the initial hashbang path). Only such
lines that *immediately* follow the initial hashbang path are evaluated; even
an empty line in between causes the rest to be ignored.

### Important notes regarding the system locale ###
* modernish, like most shells, fully supports two locales: POSIX (a.k.a.
  C, a.k.a. ASCII) and Unicode's UTF-8. It will work in others, but things
  like converting to upper/lower case, and matching single characters in
  patterns, are not guaranteed.    
  *Caveat:* some shells or operating systems have bugs that prevent (or lack
  features required for) full locale support. If portability is a concern,
  check for `thisshellhas BUG_MULTIBYTE` or `thisshellhas BUG_NOCHCLASS`
  where needed. See Appendix A under [Bugs](#bugs).
* Scripts/programs should *not* change the locale (`LC_*` or `LANG`) after
  initialising modernish. Doing this might break various functions, as
  modernish sets specific versions depending on your OS, shell and locale.
  (Temporarily changing the locale is fine as long as you don't use
  modernish features that depend on it -- for example, setting a specific
  locale just for an external command. However, if you use `harden()`, see
  the [important note](#important-note-on-variable-assignments) in its
  documentation below!)

## Interactive use ##

Modernish is primarily designed to enhance shell programs/scripts, but also
offers features for use in interactive shells. For instance, the new `with`
loop construct from the `loop/with` module can be quite practical to repeat
an action x times, and the `safe` module on interactive shells provides
convenience functions for manipulating, saving and restoring the state of
field splitting and globbing.

To use modernish on your favourite interactive shell, you have to add it to
your `.profile`, `.bashrc` or similar init file.

**Important:** Modernish removes all aliases upon initialising, but it does
depend on other settings, such as the locale. So you have to organise your
`.profile` or similar file in the following order:

* *first*, do everything except aliases and modernish (`PATH`, locale, etc.);
* *then*, `. modernish` and `use` any modules you want;
* *then* define any additional `alias`es you want.

## Internal namespace ##

Function-local variables are not supported by the standard POSIX shell; only
global variables are provided for. Modernish needs a way to store its
internal state without interfering with the program using it. So most of the
modernish functionality uses an internal namespace `_Msh_*` for variables,
functions and aliases. All these names may change at any time without
notice. *Any names starting with `_Msh_` should be considered sacrosanct and
untouchable; modernish programs should never directly use them in any way.*
Of course this is not enforceable, but names starting with `_Msh_` should be
uncommon enough that no unintentional conflict is likely to occur.


## Shell feature testing ##

Modernish includes a battery of shell bug, quirk and feature tests, each of
which is given a special ID. These are easy to query using the `thisshellhas`
function, e.g. `if thisshellhas LOCAL, then` ... That same function also tests
if 'thisshellhas' a particular reserved word or builtin command.

To reduce start up time, the main bin/modernish script only includes the<commit_msg>documentation update re .profile / interactive use<commit_after># modernish: a shell modernizer library #

modernish is an ambitious, as-yet experimental, cross-platform POSIX shell
feature detection and language extension library. It aims to extend the
shell language with extensive feature testing and language enhancements,
using the power of aliases and functions to extend the shell language
using the shell language itself.

The name is a pun on Modernizr, the JavaScript feature testing library, -sh,
the common suffix for UNIX shell names, and -ish, still not quite a modern
programming language but perhaps a little closer. jQuery is another source
of general inspiration; like it, modernish adds a considerable feature set
by using the power of the language it's implemented in to extend/transcend
that same language.

That said, the aim of modernish is to build a better shell language, and not
to make the shell language into something it's not. Its feature set is aimed
at solving specific and commonly experiened deficits and annoyances of the
shell language, and not at adding/faking things that are foreign to it, such
as object orientation or functional programming. (However, since modernish
is modular, nothing stops anyone from adding a module attempting to
implement these things.)

The library builds on pure POSIX 2013 Edition (including full C-style shell
arithmetics with assignment, comparison and conditional expressions), so it
should run on any POSIX-compliant shell and operating system. But it does
not shy away from using non-standard extensions where available to enhance
performance or robustness.

Some example programs are in `share/doc/modernish/examples`.

Modernish also comes with a suite of test scripts to detect bugs in
modernish itself, which can be run using `modernish --test` after
installation.


## Getting started ##

Run `install.sh` and follow instructions, choosing your preferred shell
and install location. After successful installation you can run modernish
shell scripts and write your own. Run `uninstall.sh` to remove modernish.


## Two basic forms of a modernish program ##

The **simplest** way to write a modernish program is to source modernish as a
dot script. For example, if you write for bash:

    #! /bin/bash
    . modernish
    use safe
    use sys/base
    ...your program starts here...

The modernish 'use' command load modules with optional functionality. `safe` is
a special module that introduces a new and safer way of shell programming, with
field splitting (word splitting) and pathname expansion (globbing) disabled by
default. The `sys/base` module contains modernish versions of certain basic but
non-standardised utilities (e.g. `readlink`, `mktemp`, `which`), guaranteeing
that modernish programs all have a known version at their disposal. There are
many other modules as well. See below for more information.

The above method makes the program dependent on one particular shell (in this
case, bash). So it is okay to mix and match functionality specific to that
particular shell with modernish functionality.

The **most portable** way to write a modernish program is to use the special
generic hashbang path for modernish programs. For example:

    #! /usr/bin/env modernish
    #! use safe
    #! use sys/base
    ...your program begins here...

A program in this form is executed by whatever shell the user who installed
modernish on the local system chose as the default shell. Since you as the
programmer can't know what shell this is (other than the fact that it passed
some rigorous POSIX compliance testing executed by modernish), a program in
this form *must be strictly POSIX compliant* -- except, of course, that it
should also make full use of the rich functionality offered by modernish.

Note that modules are loaded in a different way: the `use` commands are part of
hashbang comment (starting with `#!` like the initial hashbang path). Only such
lines that *immediately* follow the initial hashbang path are evaluated; even
an empty line in between causes the rest to be ignored.

### Important notes regarding the system locale ###
* modernish, like most shells, fully supports two locales: POSIX (a.k.a.
  C, a.k.a. ASCII) and Unicode's UTF-8. It will work in others, but things
  like converting to upper/lower case, and matching single characters in
  patterns, are not guaranteed.    
  *Caveat:* some shells or operating systems have bugs that prevent (or lack
  features required for) full locale support. If portability is a concern,
  check for `thisshellhas BUG_MULTIBYTE` or `thisshellhas BUG_NOCHCLASS`
  where needed. See Appendix A under [Bugs](#bugs).
* Scripts/programs should *not* change the locale (`LC_*` or `LANG`) after
  initialising modernish. Doing this might break various functions, as
  modernish sets specific versions depending on your OS, shell and locale.
  (Temporarily changing the locale is fine as long as you don't use
  modernish features that depend on it -- for example, setting a specific
  locale just for an external command. However, if you use `harden()`, see
  the [important note](#important-note-on-variable-assignments) in its
  documentation below!)

## Interactive use ##

Modernish is primarily designed to enhance shell programs/scripts, but also
offers features for use in interactive shells. For instance, the new `with`
loop construct from the `loop/with` module can be quite practical to repeat
an action x times, and the `safe` module on interactive shells provides
convenience functions for manipulating, saving and restoring the state of
field splitting and globbing.

To use modernish on your favourite interactive shell, you have to add it to
your `.profile`, `.bashrc` or similar init file.

**Important:** Upon initialising, modernish adapts itself to
other settings, such as the locale. So you have to organise your
`.profile` or similar file in the following order:

* *first*, define general system settings (`PATH`, locale, etc.);
* *then*, `. modernish` and `use` any modules you want;
* *then* define anything that may depend on modernish.

## Internal namespace ##

Function-local variables are not supported by the standard POSIX shell; only
global variables are provided for. Modernish needs a way to store its
internal state without interfering with the program using it. So most of the
modernish functionality uses an internal namespace `_Msh_*` for variables,
functions and aliases. All these names may change at any time without
notice. *Any names starting with `_Msh_` should be considered sacrosanct and
untouchable; modernish programs should never directly use them in any way.*
Of course this is not enforceable, but names starting with `_Msh_` should be
uncommon enough that no unintentional conflict is likely to occur.


## Shell feature testing ##

Modernish includes a battery of shell bug, quirk and feature tests, each of
which is given a special ID. These are easy to query using the `thisshellhas`
function, e.g. `if thisshellhas LOCAL, then` ... That same function also tests
if 'thisshellhas' a particular reserved word or builtin command.

To reduce start up time, the main bin/modernish script only includes the
{'fraction_non_alphanumeric': 0.052434456928838954, 'fraction_numerical': 0.0006935774726036898, 'mean_word_length': 4.501335368180084, 'pattern_counts': {'":': 0, '<': 3, '<?xml version=': 0, '>': 3, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '2288975', 'n_tokens_mistral': 3611, 'n_tokens_neox': 3461, 'n_words': 2126}