<commit_before>// Copyright (C) 2011 Google Inc.
//
// Licensed under the Apache License, Version 2.0 (the "License");
// you may not use this file except in compliance with the License.
// You may obtain a copy of the License at
//
// http://www.apache.org/licenses/LICENSE-2.0
//
// Unless required by applicable law or agreed to in writing, software
// distributed under the License is distributed on an "AS IS" BASIS,
// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
// See the License for the specific language governing permissions and
// limitations under the License.

/**
 * @fileoverview Pseudo-code explaining the source-SES to target-SES
 * translation.
 *
 * <p>See <a href="http://code.google.com/p/google-caja/wiki/SES"
 * >http://code.google.com/p/google-caja/wiki/SES</a> for an
 * explanation of the differences between source-SES and target-SES.
 *
 * <p>Serves as pseudo-code until we can run it. This section explains
 * what more is needed to turn this into running code.
 *
 * We express our translations using JavaScript <a href=
 * "http://wiki.ecmascript.org/doku.php?id=harmony:quasis"
 * >quasiliterals</a> and a hypothetical JavaScript quasi-parser named
 * "js". We additionally assume that the expression syntax {a, b} is
 * shorthand not for <pre>
 * {a: a, b: b}
 * </pre> as currently proposed, but rather for
 * <pre>
 * {
 *   get a() { return a; },
 *   set a(n) { a = n; },
 *   get b() { return b; },
 *   set b(n) { b = n; }
 * }</pre> as proposed at the end of the <a href=
 * "http://wiki.ecmascript.org/doku.php?id=strawman:object_initialiser_shorthand#discussion"
 * >object initializer discussion</a> section.
 *
 * Together, this means that (quasiParser`stuff${{varName}}stuff`) can
 * create an object that can be used as a pattern, placing the values
 * extracted from the specimen into varName, which must have already
 * been declared. As a further elaboration, we assume predicate
 * pattern functions that wrap such a literal
 * (quasiParser`stuff${isThing({varName})}stuff`) and report whether
 * the proposed assignment meets it criteria. If not, that match
 * fails.
 *
 * Separately from the rules below, we also prohibit identifiers
 * ending in ___ (triple underbar) in the input, so that we may use
 * them in the output without fear of capture.
 *
 * The JavaScript quasiParser accepts one additional element besides
 * JavaScript syntax and $ holes. In addition, when an "*", "+", or
 * "?" appears immediately to the right of a $ hole with no
 * intervening space, then it is taken as a quantifier on the binding
 * of the $ hole to consecutive AST elements.
 *
 * Besides the "js" quasi-parser, the following code depends on the
 * following helper functions. Depending on the concrete AST
 * representation chosen, perhaps some of these should instead be uses
 * of AST methods. Refactor as needed.<ul>
 * <li>predicate hole wrappers<ul>
 *     <li>isStatementOrDecl
 *     <li>isExpr
 *     <li>isGlobal       - a global variable name?
 *     <li>isFree         - a free variable name?
 *     <li>isGlobalOrFree
 *     <li>mustNotShadow  - verify name doesn't shadow name in this function
 * </ul>
 * <li>getFirstTokenName  - printed form of first token
 * <li>getChildren        - of an ast node
 * <li>makeAstLike        - like arg, with alternate children
 * </ul>
 *
 * The expanded code assumes bindings for<ul>
 * <li>declareGlobal___
 * <li>defineGlobal___
 * <li>cleanErr___
 * </ul>
 *
 * Additional names reserved for the output that should be prohibited
 * on input<ul>
 * <li>global___
 * <li>e___
 * </ul>Though we should currently reserve all <i>variable</i> names
 * ending in "___" (triple underbar). We should not reserve such names
 * statically as property names, since we cannot reserve them
 * dynamically without a much more expensive translation.
 *
 * <p>The input ast to all of the expand* functions below comes after
 * parsing and scope analysis of a complete Program production, so
 * that questions like isGlobal have an answer. If all the $-hole
 * inputs to a call to the js quasi-parser are asts, then js
 * quasi-parser produces pure ast fragments without scope analysis as
 * its output. If and of the $-hole inputs are one property objects or
 * their predicate wrappers, as explained above, then that call
 * produces an object with a ".test(ast)" method which matches against
 * the ast and, if true, assigns the extracted fragments to these
 * singleton properties.
 *
 * @author <NAME>
 * @provides expandProgramToExpr
 */
<commit_msg>script vs eval scoping in translating source ses.<commit_after>// Copyright (C) 2011 Google Inc.
//
// Licensed under the Apache License, Version 2.0 (the "License");
// you may not use this file except in compliance with the License.
// You may obtain a copy of the License at
//
// http://www.apache.org/licenses/LICENSE-2.0
//
// Unless required by applicable law or agreed to in writing, software
// distributed under the License is distributed on an "AS IS" BASIS,
// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
// See the License for the specific language governing permissions and
// limitations under the License.

/**
 * @fileoverview Pseudo-code explaining the source-SES to target-SES
 * translation.
 *
 * <p>See <a href="http://code.google.com/p/google-caja/wiki/SES"
 * >http://code.google.com/p/google-caja/wiki/SES</a> for an
 * explanation of the differences between source-SES and target-SES.
 *
 * <p>Serves as pseudo-code until we can run it. This section explains
 * what more is needed to turn this into running code.
 *
 * We express our translations using JavaScript <a href=
 * "http://wiki.ecmascript.org/doku.php?id=harmony:quasis"
 * >quasiliterals</a> and a hypothetical JavaScript quasi-parser named
 * "js". We additionally assume that the expression syntax {a, b} is
 * shorthand not for <pre>
 * {a: a, b: b}
 * </pre> as currently proposed, but rather for
 * <pre>
 * {
 *   get a() { return a; },
 *   set a(n) { a = n; },
 *   get b() { return b; },
 *   set b(n) { b = n; }
 * }</pre> as proposed at the end of the <a href=
 * "http://wiki.ecmascript.org/doku.php?id=strawman:object_initialiser_shorthand#discussion"
 * >object initializer discussion</a> section.
 *
 * Together, this means that (quasiParser`stuff${{varName}}stuff`) can
 * create an object that can be used as a pattern, placing the values
 * extracted from the specimen into varName, which must have already
 * been declared. As a further elaboration, we assume predicate
 * pattern functions that wrap such a literal
 * (quasiParser`stuff${isThing({varName})}stuff`) and report whether
 * the proposed assignment meets it criteria. If not, that match
 * fails.
 *
 * Separately from the rules below, we also prohibit identifiers
 * ending in ___ (triple underbar) in the input, so that we may use
 * them in the output without fear of capture.
 *
 * The JavaScript quasiParser accepts one additional element besides
 * JavaScript syntax and $ holes. In addition, when an "*", "+", or
 * "?" appears immediately to the right of a $ hole with no
 * intervening space, then it is taken as a quantifier on the binding
 * of the $ hole to consecutive AST elements.
 *
 * Besides the "js" quasi-parser, the following code depends on the
 * following helper functions. Depending on the concrete AST
 * representation chosen, perhaps some of these should instead be uses
 * of AST methods. Refactor as needed.<ul>
 * <li>predicate hole wrappers<ul>
 *     <li>isStatementOrDecl
 *     <li>isExpr
 *     <li>isGlobal       - a global variable name?
 *     <li>isFree         - a free variable name?
 *     <li>isGlobalOrFree
 *     <li>mustNotShadow  - verify name doesn't shadow name in this function
 * </ul>
 * <li>getFirstTokenName  - printed form of first token
 * <li>getChildren        - of an ast node
 * <li>makeAstLike        - like arg, with alternate children
 * </ul>
 *
 * The expanded code assumes bindings for<ul>
 * <li>declareGlobal___
 * <li>defineGlobal___
 * <li>cleanErr___
 * </ul>
 *
 * <p>If "mustNotShadow" determines that its argument is a variable
 * that shadows another variable in this function, it should throw
 * rather than report a match failure, as this failure is not a
 * license to translate by other means. This condition aborts the
 * overall translation.
 *
 * When translating a Source-SES script program, as might appear
 * between script tags, declareGlobal___ and defineGlobal___ should
 * mutate the virtual global, enabling inter-module linkage. When
 * translating a Source-SES eval program, as might be passed to an
 * "eval" function, declareGlobal___ and defineGlobal___ should bring
 * these names into scope <i>only</i> within this program, i.e., it
 * should not mutate the virtual global.
 *
 * <p>Additional names reserved for the output that should be prohibited
 * on input<ul>
 * <li>global___
 * <li>e___
 * </ul>Though we should currently reserve all <i>variable</i> names
 * ending in "___" (triple underbar). We should not reserve such names
 * statically as property names, since we cannot reserve them
 * dynamically without a much more expensive translation.
 *
 * <p>The input ast to all of the expand* functions below comes after
 * parsing and scope analysis of a complete Program production, so
 * that questions like isGlobal have an answer. If all the $-hole
 * inputs to a call to the js quasi-parser are asts, then js
 * quasi-parser produces pure ast fragments without scope analysis as
 * its output. If and of the $-hole inputs are one property objects or
 * their predicate wrappers, as explained above, then that call
 * produces an object with a ".test(ast)" method which matches against
 * the ast and, if true, assigns the extracted fragments to these
 * singleton properties.
 *
 * @author <NAME>
 * @provides expandProgramToExpr
 */

{'fraction_non_alphanumeric': 0.09363333669659973, 'fraction_numerical': 0.001614367874079306, 'mean_word_length': 3.7020872865275143, 'pattern_counts': {'":': 0, '<': 85, '<?xml version=': 0, '>': 85, 'https://': 0, 'lorem ipsum': 0, 'www.': 2, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '5998158', 'n_tokens_mistral': 2787, 'n_tokens_neox': 2707, 'n_words': 1260}