<commit_before>(* <NAME>
 *
 * Copyright (C) 2012, 2014 Facebook
 *
 * This library is free software; you can redistribute it and/or
 * modify it under the terms of the GNU Lesser General Public License
 * version 2.1 as published by the Free Software Foundation, with the
 * special exception on linking described in file license.txt.
 *
 * This library is distributed in the hope that it will be useful, but
 * WITHOUT ANY WARRANTY; without even the implied warranty of
 * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the file
 * license.txt for more details.
 *)
open Common2.Infix

(*****************************************************************************)
(* Prelude *)
(*****************************************************************************)
(* 
 * A (real) Abstract Syntax Tree for C, not a Concrete Syntax Tree
 * as in ast_cpp.ml.
 * 
 * This file contains a simplified C abstract syntax tree. The original
 * C/C++ syntax tree (ast_cpp.ml) is good for code refactoring or
 * code visualization; the type used matches exactly the source. However,
 * for other algorithms, the nature of the AST makes the code a bit
 * redundant. Moreover many analysis are far simpler to write on
 * C than C++. Hence the idea of a SimpleAST which is the
 * original AST where certain constructions have been factorized
 * or even removed.

 * Here is a list of the simplications/factorizations:
 *  - no C++ constructs, just plain C
 *  - no purely syntactical tokens in the AST like parenthesis, brackets, 
 *    braces, commas, semicolons, etc. No ParenExpr. No FinalDef. No
 *    NotParsedCorrectly. The only token information kept is for identifiers
 *    for error reporting. See name below.
 *  - ...
 *  - no nested struct, they are lifted to the toplevel
 *  - no mix of typedef with decl
 *  - sugar is removed, no RecordAccess vs RecordPtAccess, ...
 *  - no init vs expr
 *  - no Case/Default in statement but instead a focused 'case' type
 * 
 * todo: ast_c_simple_build.ml is probably very incomplete, but for now
 * good enough for codegraph purposes on xv6, plan9 and other small C
 * projects.
 * 
 * related work: 
 *  - CIL, but it works after preprocessing; it makes it harder to connect
 *    analysis results to tools like codemap. It also does not handle some of
 *    the kencc extensions.
 *    CIL has two pointer analysis but they were written with bug finding<commit_msg>* lang_c/parsing/ast_c.ml: comments<commit_after>(* <NAME>
 *
 * Copyright (C) 2012, 2014 Facebook
 *
 * This library is free software; you can redistribute it and/or
 * modify it under the terms of the GNU Lesser General Public License
 * version 2.1 as published by the Free Software Foundation, with the
 * special exception on linking described in file license.txt.
 *
 * This library is distributed in the hope that it will be useful, but
 * WITHOUT ANY WARRANTY; without even the implied warranty of
 * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the file
 * license.txt for more details.
 *)
open Common2.Infix

(*****************************************************************************)
(* Prelude *)
(*****************************************************************************)
(* 
 * A (real) Abstract Syntax Tree for C, not a Concrete Syntax Tree
 * as in ast_cpp.ml.
 * 
 * This file contains a simplified C abstract syntax tree. The original
 * C/C++ syntax tree (ast_cpp.ml) is good for code refactoring or
 * code visualization; the types used match exactly the source. However,
 * for other algorithms, the nature of the AST makes the code a bit
 * redundant. Moreover many analysis are far simpler to write on
 * C than C++. Hence the idea of a SimpleAST which is the
 * original AST where certain constructions have been factorized
 * or even removed.

 * Here is a list of the simplications/factorizations:
 *  - no C++ constructs, just plain C
 *  - no purely syntactical tokens in the AST like parenthesis, brackets, 
 *    braces, commas, semicolons, etc. No ParenExpr. No FinalDef. No
 *    NotParsedCorrectly. The only token information kept is for identifiers
 *    for error reporting. See name below.
 *  - ...
 *  - no nested struct, they are lifted to the toplevel
 *  - no anonymous structure (an artificial name is gensym'ed)
 *  - no mix of typedef with decl
 *  - sugar is removed, no RecordAccess vs RecordPtAccess, ...
 *  - no init vs expr
 *  - no Case/Default in statement but instead a focused 'case' type
 * 
 * less: ast_c_simple_build.ml is probably incomplete, but for now
 * is good enough for codegraph purposes on xv6, plan9 and other small C
 * projects.
 * 
 * related work: 
 *  - CIL, but it works after preprocessing; it makes it harder to connect
 *    analysis results to tools like codemap. It also does not handle some of
 *    the kencc extensions.
 *    CIL has two pointer analysis but they were written with bug finding
{'fraction_non_alphanumeric': 0.12454212454212454, 'fraction_numerical': 0.005291005291005291, 'mean_word_length': 3.7033492822966507, 'pattern_counts': {'":': 0, '<': 5, '<?xml version=': 0, '>': 5, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 2, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '3792303', 'n_tokens_mistral': 1309, 'n_tokens_neox': 1230, 'n_words': 661}