<commit_before>(* this is a multiline comment *)
// this is also a comment, but single line

const:
   [0-9]+

ident:
   [:alpha:] [:alnum:]*

var:
   ident

rf:
   expr

expr:
   add-expr
  
primary-expr: const 
    | var
    | fun-call
    | "(" expr ")"

mult-expr: primary-expr [ "*" mult-expr ]
add-expr: mult-expr [ addop mult-expr ]
addop: "+" | "-"

rel-expr: expr rel-op expr
rel-op: ">" | ">=" | "<" | "<=" | "<>" | "=" 

bool-expr: rel-expr [ bool-op bool-expr ]
bool-op: "and" | "or"

stm-list: stm [ ";" stm-list]

stm: "skip"
   | "if" bool-expr "then" stm-list [";"] "else" stm-list [";"] "end" "if"
   | "while" bool-expr "bound" expr "do" stm-list [";"] "end" "while"
   | var ":=" expr
   | fun-call

fun-call: | ident "(" arg-list ")"
          | ident "::" ident "(" arg-list ")"
     

fundef: "function" ident "(" parm-ety ")" stm-list [";"] "end" "function"

parm-ety: $ | parm
parm: var ["," parm]
arg-ety: $ | arg
arg: expr ["," expr]

def-list: $ | fundef def-list

prog: def-list
<commit_msg>added a pragmatics section about the two types of variables<commit_after>FORMAL GRAMMAR DESCRIPTION:

(* this is a multiline comment *)
// this is also a comment, but single line

const:
   [0-9]+

ident:
   [:alpha:] [:alnum:]*

var:       
   ident

rf:
   expr

expr:
   add-expr
  
primary-expr: const 
    | var
    | fun-call
    | "(" expr ")"

mult-expr: primary-expr [ "*" mult-expr ]
add-expr: mult-expr [ addop mult-expr ]
addop: "+" | "-"

rel-expr: expr rel-op expr
rel-op: ">" | ">=" | "<" | "<=" | "<>" | "=" 

bool-expr: rel-expr [ bool-op bool-expr ]
bool-op: "and" | "or"

stm-list: stm [ ";" stm-list]

stm: "skip"
   | "if" bool-expr "then" stm-list [";"] "else" stm-list [";"] "end" "if"
   | "while" bool-expr "bound" expr "do" stm-list [";"] "end" "while"
   | var ":=" expr
   | fun-call

fun-call: | ident "(" arg-list ")"
          | ident "::" ident "(" arg-list ")"
     

fundef: "function" ident "(" parm-ety ")" stm-list [";"] "end" "function"

parm-ety: $ | parm
parm: var ["," parm]
arg-ety: $ | arg
arg: expr ["," expr]

def-list: $ | fundef def-list

prog: def-list

PRAGMATICS:

Variable's in the grammar can denote both /program variables/ and /parameters/.
Any variable that is assigned to is a program variable.
Only variables appearing in function parameter lists are parameters.

It is an error to use uninitialized variables (since they are neither program
variable nor parameters).

Only parameters may appear in ranking functions ('rf'), since otherwise we
cannot guarantee that each loop iteration takes the same amount of time.

Similarly, only parameters may appear as arguments to non-component functions.


{'fraction_non_alphanumeric': 0.1862378459237098, 'fraction_numerical': 0.0014958863126402393, 'mean_word_length': 3.0286144578313254, 'pattern_counts': {'":': 4, '<': 9, '<?xml version=': 0, '>': 9, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '1948247', 'n_tokens_mistral': 1013, 'n_tokens_neox': 990, 'n_words': 296}