<commit_before>import Kernel, except: [to_string: 1]

defmodule Macro do
  @moduledoc ~S"""
  Macros are compile-time constructs that are invoked with Elixir's AST
  as input and a superset of Elixir's AST as output.

  Let's see a simple example that shows the difference between functions and macros:

      defmodule Example do
        defmacro macro_inspect(value) do
          IO.inspect(value)
          value
        end

        def fun_inspect(value) do
          IO.inspect(value)
          value
        end
      end

  Now let's give it a try:

      import Example

      macro_inspect(1)
      #=> 1
      #=> 1

      fun_inspect(1)
      #=> 1
      #=> 1

  So far they behave the same, as we are passing an integer as argument.
  But what happens when we pass an expression:

      macro_inspect(1 + 2)
      #=> {:+, [line: 3], [1, 2]}
      #=> 3

      fun_inspect(1 + 2)
      #=> 3
      #=> 3

  The macro receives the representation of the code given as argument,
  while a function receives the result of the code given as argument.
  A macro must return a superset of the code representation. See
  `t:input/0` and `t:output/0` for more information.

  To learn more about Elixir's AST and how to build them programmatically,
  see `quote/2`.

  ## Custom Sigils

  Macros are also commonly used to implement custom sigils. To create a custom
  sigil, define a function with the name `sigil_{identifier}` that takes two
  arguments. The first argument will be the string, the second will be a charlist
  containing any modifiers. If the sigil is lower case (such as `sigil_x`) then
  the string argument will allow interpolation. If the sigil is upper case
  (such as `sigil_X`) then the string will not be interpolated.

  Valid modifiers include only lower and upper case letters. Other characters
  will cause a syntax error.

  The module containing the custom sigil must be imported before the sigil
  syntax can be used.

  ### Examples

      defmodule MySigils do
        defmacro sigil_x(term, [?r]) do
          quote do
            unquote(term) |> String.reverse()
          end
        end
        defmacro sigil_x(term, _modifiers) do
          term
        end
        defmacro sigil_X(term, [?r]) do
          quote do
            unquote(term) |> String.reverse()
          end
        end
        defmacro sigil_X(term, _modifiers) do
          term
        end
      end

      import MySigils

      ~x(with #{"inter" <> "polation"})
      #=>"with interpolation"

      ~x(with #{"inter" <> "polation"})r
      #=>"noitalopretni htiw"

      ~X(without #{"interpolation"})
      #=>"without \#{"interpolation"}"

      ~X(without #{"interpolation"})r
      #=>"}\"noitalopretni\"{# tuohtiw"

  """

  alias Code.Identifier

  @typedoc "Abstract Syntax Tree (AST)"
  @type t :: input

  @typedoc "The inputs of a macro"
  @type input ::
          input_expr
          | {input, input}
          | [input]
          | atom
          | number
          | binary

  @typep input_expr :: {input_expr | atom, metadata, atom | [input]}

  @typedoc "The output of a macro"
  @type output ::
          output_expr
          | {output, output}
          | [output]
          | atom
          | number
          | binary
          | captured_remote_function
          | pid

  @typep output_expr :: {output_expr | atom, metadata, atom | [output]}

  @typedoc """
  A keyword list of AST metadata.

  The metadata in Elixir AST is a keyword list of values. Any key can be used
  and different parts of the compiler may use different keys. For example,
  the AST received by a macro will always include the `:line` annotation,
  while the AST emitted by `quote/2` will only have the `:line` annotation if
  the `:line` option is provided.

  The following metadata keys are public:

    * `:context` - Defines the context in which the AST was generated.
      For example, `quote/2` will include the module calling `quote/2`
      as the context. This is often used to distinguish regular code from code
      generated by a macro or by `quote/2`.
    * `:counter` - The variable counter used for variable hygiene. In terms of
      the compiler, each variable is identified by the combination of either
      `name` and `metadata[:counter]`, or `name` and `context`.
    * `:generated` - Whether the code should be considered as generated by
      the compiler or not. This means the compiler and tools like Dialyzer may not
      emit certain warnings.
    * `:keep` - Used by `quote/2` with the option `location: :keep` to annotate
      the file and the line number of the quoted source.
    * `:line` - The line number of the AST node.

  The following metadata keys are enabled by `Code.string_to_quoted/2`:

    * `:closing` - contains metadata about the closing pair, such as a `}`
      in a tuple or in a map, or such as the closing `)` in a function call
      with parens. The `:closing` does not delimit the end of expression if
      there are `:do` and `:end` metadata  (when `:token_metadata` is true)
    * `:column` - the column number of the AST node (when `:columns` is true)
    * `:delimiter` - contains the opening delimiter for sigils, strings,
      and charlists as a string (such as `"{"`, `"/"`, `"'"`, and the like)
    * `:format` - set to `:keyword` when an atom is defined as a keyword
    * `:do` - contains metadata about the `do` location in a function call with
      `do/end` blocks (when `:token_metadata` is true)
    * `:end` - contains metadata about the `end` location in a function call with
      `do/end` blocks (when `:token_metadata` is true)
    * `:end_of_expression` - denotes when the end of expression effectively
      happens. Available for all expressions except the last one inside a
      `__block__` (when `:token_metadata` is true)
    * `:indentation` - indentation of a sigil heredoc

  The following metadata keys are private:

    * `:alias` - Used for alias hygiene.
    * `:ambiguous_op` - Used for improved error messages in the compiler.
    * `:import` - Used for import hygiene.
    * `:var` - Used for improved error messages on undefined variables.

  Do not rely on them as they may change or be fully removed in future versions
  of the language. They are often used by `quote/2` and the compiler to provide
  features like hygiene, better error messages, and so forth.

  If you introduce custom keys into the AST metadata, please make sure to prefix
  them with the name of your library or application, so that they will not conflict
  with keys that could potentially be introduced by the compiler in the future.
  """
  @type metadata :: keyword

  @typedoc "A captured remote function in the format of &Mod.fun/arity"
  @type captured_remote_function :: fun

  @doc """
  Breaks a pipeline expression into a list.

  The AST for a pipeline (a sequence of applications of `|>`) is similar to the
  AST of a sequence of binary operators or function applications: the top-level
  expression is the right-most `:|>` (which is the last one to be executed), and
  its left-hand and right-hand sides are its arguments:

      quote do: 100 |> div(5) |> div(2)
      #=> {:|>, _, [arg1, arg2]}

  In the example above, the `|>` pipe is the right-most pipe; `arg1` is the AST
  for `100 |> div(5)`, and `arg2` is the AST for `div(2)`.

  It's often useful to have the AST for such a pipeline as a list of function
  applications. This function does exactly that:

      Macro.unpipe(quote do: 100 |> div(5) |> div(2))
      #=> [{100, 0}, {{:div, [], [5]}, 0}, {{:div, [], [2]}, 0}]

  We get a list that follows the pipeline directly: first the `100`, then the
  `div(5)` (more precisely, its AST), then `div(2)`. The `0` as the second
  element of the tuples is the position of the previous element in the pipeline
  inside the current function application: `{{:div, [], [5]}, 0}` means that the
  previous element (`100`) will be inserted as the 0th (first) argument to the
  `div/2` function, so that the AST for that function will become `{:div, [],
  [100, 5]}` (`div(100, 5)`).
  """
  @spec unpipe(t()) :: [t()]
  def unpipe(expr) do
    :lists.reverse(unpipe(expr, []))
  end

  defp unpipe({:|>, _, [left, right]}, acc) do
    unpipe(right, unpipe(left, acc))
  end

  defp unpipe(other, acc) do
    [{other, 0} | acc]
  end

  @doc """
  Pipes `expr` into the `call_args` at the given `position`.
  """
  @spec pipe(t(), t(), integer) :: t()
  def pipe(expr, call_args, position)

  def pipe(expr, {:&, _, _} = call_args, _integer) do
    raise ArgumentError, bad_pipe(expr, call_args)
  end

  def pipe(expr, {tuple_or_map, _, _} = call_args, _integer) when tuple_or_map in [:{}, :%{}] do
    raise ArgumentError, bad_pipe(expr, call_args)
  end

  # Without this, `Macro |> Env == Macro.Env`.
  def pipe(expr, {:__aliases__, _, _} = call_args, _integer) do
    raise ArgumentError, bad_pipe(expr, call_args)
  end

  def pipe(expr, {:<<>>, _, _} = call_args, _integer) do
    raise ArgumentError, bad_pipe(expr, call_args)
  end

  def pipe(expr, {unquote, _, []}, _integer) when unquote in [:unquote, :unquote_splicing] do
    raise ArgumentError,
          "cannot pipe #{to_string(expr)} into the special form #{unquote}/1 " <>
            "since #{unquote}/1 is used to build the Elixir AST itself"
  end

  # {:fn, _, _} is what we get when we pipe into an anonymous function without
  # calling it, for example, `:foo |> (fn x -> x end)`.
  def pipe(expr, {:fn, _, _}, _integer) do
    raise ArgumentError,
          "cannot pipe #{to_string(expr)} into an anonymous function without" <>
            " calling the function; use Kernel.then/2 instead or" <>
            " define the anonymous function as a regular private function"
  end

  def pipe(expr, {call, line, atom}, integer) when is_atom(atom) do
    {call, line, List.insert_at([], integer, expr)}
  end

  def pipe(_expr, {op, _line, [arg]}, _integer) when op == :+ or op == :- do
    raise ArgumentError,
          "piping into a unary operator is not supported, please use the qualified name: " <>
            "Kernel.#{op}(#{to_string(arg)}), instead of #{op}#{to_string(arg)}"
  end

  def pipe(expr, {op, line, args} = op_args, integer) when is_list(args) do
    cond do
      is_atom(op) and Identifier.unary_op(op) != :error ->
        raise ArgumentError,
              "cannot pipe #{to_string(expr)} into #{to_string(op_args)}, " <>
                "the #{to_string(op)} operator can only take one argument"

      is_atom(op) and Identifier.binary_op(op) != :error ->
        raise ArgumentError,
              "cannot pipe #{to_string(expr)} into #{to_string(op_args)}, " <>
                "the #{to_string(op)} operator can only take two arguments"

      true ->
        {op, line, List.insert_at(args, integer, expr)}
    end
  end

  def pipe(expr, call_args, _integer) do
    raise ArgumentError, bad_pipe(expr, call_args)
  end

  defp bad_pipe(expr, call_args) do
    "cannot pipe #{to_string(expr)} into #{to_string(call_args)}, " <>
      "can only pipe into local calls foo(), remote calls Foo.bar() or anonymous function calls foo.()"
  end

  @doc """
  Applies the given function to the node metadata if it contains one.

  This is often useful when used with `Macro.prewalk/2` to remove
  information like lines and hygienic counters from the expression
  for either storage or comparison.

  ## Examples

      iex> quoted = quote line: 10, do: sample()
      {:sample, [line: 10], []}
      iex> Macro.update_meta(quoted, &Keyword.delete(&1, :line))
      {:sample, [], []}

  """
  @spec update_meta(t, (keyword -> keyword)) :: t
  def update_meta(quoted, fun)

  def update_meta({left, meta, right}, fun) when is_list(meta) do
    {left, fun.(meta), right}
  end

  def update_meta(other, _fun) do
    other
  end

  @doc """
  Generates AST nodes for a given number of required argument
  variables using `Macro.var/2`.

  Note the arguments are not unique. If you later on want
  to access the same variables, you can invoke this function
  with the same inputs. Use `generate_unique_arguments/2` to
  generate a unique arguments that can't be overridden.

  ## Examples

      iex> Macro.generate_arguments(2, __MODULE__)
      [{:arg1, [], __MODULE__}, {:arg2, [], __MODULE__}]

  """
  @doc since: "1.5.0"
  @spec generate_arguments(0, context :: atom) :: []
  @spec generate_arguments(pos_integer, context) :: [{atom, [], context}, ...] when context: atom
  def generate_arguments(amount, context), do: generate_arguments(amount, context, &var/2)

  @doc """
  Generates AST nodes for a given number of required argument
  variables using `Macro.unique_var/2`.

  ## Examples

      iex> [var1, var2] = Macro.generate_unique_arguments(2, __MODULE__)
      iex> {:arg1, [counter: c1], __MODULE__} = var1
      iex> {:arg2, [counter: c2], __MODULE__} = var2
      iex> is_integer(c1) and is_integer(c2)
      true

  """
  @doc since: "1.11.3"
  @spec generate_unique_arguments(0, context :: atom) :: []
  @spec generate_unique_arguments(pos_integer, context) :: [
          {atom, [counter: integer], context},
          ...
        ]
        when context: atom
  def generate_unique_arguments(amount, context),
    do: generate_arguments(amount, context, &unique_var/2)

  defp generate_arguments(0, context, _fun) when is_atom(context), do: []

  defp generate_arguments(amount, context, fun)
       when is_integer(amount) and amount > 0 and is_atom(context) do
    for id <- 1..amount, do: fun.(String.to_atom("arg" <> Integer.to_string(id)), context)
  end

  @doc """
  Generates an AST node representing the variable given
  by the atoms `var` and `context`.

  Note this variable is not unique. If you later on want
  to access this same variable, you can invoke `var/2`
  again with the same arguments. Use `unique_var/2` to
  generate a unique variable that can't be overridden.

  ## Examples

  In order to build a variable, a context is expected.
  Most of the times, in order to preserve hygiene, the
  context must be `__MODULE__/0`:

      iex> Macro.var(:foo, __MODULE__)
      {:foo, [], __MODULE__}

  However, if there is a need to access the user variable,
  nil can be given:

      iex> Macro.var(:foo, nil)
      {:foo, [], nil}

  """
  @spec var(var, context) :: {var, [], context} when var: atom, context: atom
  def var(var, context) when is_atom(var) and is_atom(context) do
    {var, [], context}
  end

  @doc """
  Generates an AST node representing a unique variable
  given by the atoms `var` and `context`.

  ## Examples

      iex> {:foo, [counter: c], __MODULE__} = Macro.unique_var(:foo, __MODULE__)
      iex> is_integer(c)
      true

  """
  @doc since: "1.11.3"
  @spec unique_var(var, context) :: {var, [counter: integer], context}
        when var: atom, context: atom
  def unique_var(var, context) when is_atom(var) and is_atom(context) do
    {var, [counter: :elixir_module.next_counter(context)], context}
  end

  @doc """
  Performs a depth-first traversal of quoted expressions
  using an accumulator.
  """
  @spec traverse(t, any, (t, any -> {t, any}), (t, any -> {t, any})) :: {t, any}
  def traverse(ast, acc, pre, post) when is_function(pre, 2) and is_function(post, 2) do
    {ast, acc} = pre.(ast, acc)
    do_traverse(ast, acc, pre, post)
  end

  defp do_traverse({form, meta, args}, acc, pre, post) when is_atom(form) do
    {args, acc} = do_traverse_args(args, acc, pre, post)
    post.({form, meta, args}, acc)
  end

  defp do_traverse({form, meta, args}, acc, pre, post) do
    {form, acc} = pre.(form, acc)
    {form, acc} = do_traverse(form, acc, pre, post)
    {args, acc} = do_traverse_args(args, acc, pre, post)
    post.({form, meta, args}, acc)
  end

  defp do_traverse({left, right}, acc, pre, post) do
    {left, acc} = pre.(left, acc)
    {left, acc} = do_traverse(left, acc, pre, post)
    {right, acc} = pre.(right, acc)
    {right, acc} = do_traverse(right, acc, pre, post)
    post.({left, right}, acc)
  end

  defp do_traverse(list, acc, pre, post) when is_list(list) do
    {list, acc} = do_traverse_args(list, acc, pre, post)
    post.(list, acc)
  end

  defp do_traverse(x, acc, _pre, post) do
    post.(x, acc)
  end

  defp do_traverse_args(args, acc, _pre, _post) when is_atom(args) do
    {args, acc}
  end

  defp do_traverse_args(args, acc, pre, post) when is_list(args) do
    :lists.mapfoldl(
      fn x, acc ->
        {x, acc} = pre.(x, acc)
        do_traverse(x, acc, pre, post)
      end,
      acc,
      args
    )
  end

  @doc """
  Performs a depth-first, pre-order traversal of quoted expressions.
  """
  @spec prewalk(t, (t -> t)) :: t
  def prewalk(ast, fun) when is_function(fun, 1) do
    elem(prewalk(ast, nil, fn x, nil -> {fun.(x), nil} end), 0)
  end
<commit_msg>Add documentation for `Macro.prewalk/2` (#10951)<commit_after>import Kernel, except: [to_string: 1]

defmodule Macro do
  @moduledoc ~S"""
  Macros are compile-time constructs that are invoked with Elixir's AST
  as input and a superset of Elixir's AST as output.

  Let's see a simple example that shows the difference between functions and macros:

      defmodule Example do
        defmacro macro_inspect(value) do
          IO.inspect(value)
          value
        end

        def fun_inspect(value) do
          IO.inspect(value)
          value
        end
      end

  Now let's give it a try:

      import Example

      macro_inspect(1)
      #=> 1
      #=> 1

      fun_inspect(1)
      #=> 1
      #=> 1

  So far they behave the same, as we are passing an integer as argument.
  But what happens when we pass an expression:

      macro_inspect(1 + 2)
      #=> {:+, [line: 3], [1, 2]}
      #=> 3

      fun_inspect(1 + 2)
      #=> 3
      #=> 3

  The macro receives the representation of the code given as argument,
  while a function receives the result of the code given as argument.
  A macro must return a superset of the code representation. See
  `t:input/0` and `t:output/0` for more information.

  To learn more about Elixir's AST and how to build them programmatically,
  see `quote/2`.

  ## Custom Sigils

  Macros are also commonly used to implement custom sigils. To create a custom
  sigil, define a function with the name `sigil_{identifier}` that takes two
  arguments. The first argument will be the string, the second will be a charlist
  containing any modifiers. If the sigil is lower case (such as `sigil_x`) then
  the string argument will allow interpolation. If the sigil is upper case
  (such as `sigil_X`) then the string will not be interpolated.

  Valid modifiers include only lower and upper case letters. Other characters
  will cause a syntax error.

  The module containing the custom sigil must be imported before the sigil
  syntax can be used.

  ### Examples

      defmodule MySigils do
        defmacro sigil_x(term, [?r]) do
          quote do
            unquote(term) |> String.reverse()
          end
        end
        defmacro sigil_x(term, _modifiers) do
          term
        end
        defmacro sigil_X(term, [?r]) do
          quote do
            unquote(term) |> String.reverse()
          end
        end
        defmacro sigil_X(term, _modifiers) do
          term
        end
      end

      import MySigils

      ~x(with #{"inter" <> "polation"})
      #=>"with interpolation"

      ~x(with #{"inter" <> "polation"})r
      #=>"noitalopretni htiw"

      ~X(without #{"interpolation"})
      #=>"without \#{"interpolation"}"

      ~X(without #{"interpolation"})r
      #=>"}\"noitalopretni\"{# tuohtiw"

  """

  alias Code.Identifier

  @typedoc "Abstract Syntax Tree (AST)"
  @type t :: input

  @typedoc "The inputs of a macro"
  @type input ::
          input_expr
          | {input, input}
          | [input]
          | atom
          | number
          | binary

  @typep input_expr :: {input_expr | atom, metadata, atom | [input]}

  @typedoc "The output of a macro"
  @type output ::
          output_expr
          | {output, output}
          | [output]
          | atom
          | number
          | binary
          | captured_remote_function
          | pid

  @typep output_expr :: {output_expr | atom, metadata, atom | [output]}

  @typedoc """
  A keyword list of AST metadata.

  The metadata in Elixir AST is a keyword list of values. Any key can be used
  and different parts of the compiler may use different keys. For example,
  the AST received by a macro will always include the `:line` annotation,
  while the AST emitted by `quote/2` will only have the `:line` annotation if
  the `:line` option is provided.

  The following metadata keys are public:

    * `:context` - Defines the context in which the AST was generated.
      For example, `quote/2` will include the module calling `quote/2`
      as the context. This is often used to distinguish regular code from code
      generated by a macro or by `quote/2`.
    * `:counter` - The variable counter used for variable hygiene. In terms of
      the compiler, each variable is identified by the combination of either
      `name` and `metadata[:counter]`, or `name` and `context`.
    * `:generated` - Whether the code should be considered as generated by
      the compiler or not. This means the compiler and tools like Dialyzer may not
      emit certain warnings.
    * `:keep` - Used by `quote/2` with the option `location: :keep` to annotate
      the file and the line number of the quoted source.
    * `:line` - The line number of the AST node.

  The following metadata keys are enabled by `Code.string_to_quoted/2`:

    * `:closing` - contains metadata about the closing pair, such as a `}`
      in a tuple or in a map, or such as the closing `)` in a function call
      with parens. The `:closing` does not delimit the end of expression if
      there are `:do` and `:end` metadata  (when `:token_metadata` is true)
    * `:column` - the column number of the AST node (when `:columns` is true)
    * `:delimiter` - contains the opening delimiter for sigils, strings,
      and charlists as a string (such as `"{"`, `"/"`, `"'"`, and the like)
    * `:format` - set to `:keyword` when an atom is defined as a keyword
    * `:do` - contains metadata about the `do` location in a function call with
      `do/end` blocks (when `:token_metadata` is true)
    * `:end` - contains metadata about the `end` location in a function call with
      `do/end` blocks (when `:token_metadata` is true)
    * `:end_of_expression` - denotes when the end of expression effectively
      happens. Available for all expressions except the last one inside a
      `__block__` (when `:token_metadata` is true)
    * `:indentation` - indentation of a sigil heredoc

  The following metadata keys are private:

    * `:alias` - Used for alias hygiene.
    * `:ambiguous_op` - Used for improved error messages in the compiler.
    * `:import` - Used for import hygiene.
    * `:var` - Used for improved error messages on undefined variables.

  Do not rely on them as they may change or be fully removed in future versions
  of the language. They are often used by `quote/2` and the compiler to provide
  features like hygiene, better error messages, and so forth.

  If you introduce custom keys into the AST metadata, please make sure to prefix
  them with the name of your library or application, so that they will not conflict
  with keys that could potentially be introduced by the compiler in the future.
  """
  @type metadata :: keyword

  @typedoc "A captured remote function in the format of &Mod.fun/arity"
  @type captured_remote_function :: fun

  @doc """
  Breaks a pipeline expression into a list.

  The AST for a pipeline (a sequence of applications of `|>`) is similar to the
  AST of a sequence of binary operators or function applications: the top-level
  expression is the right-most `:|>` (which is the last one to be executed), and
  its left-hand and right-hand sides are its arguments:

      quote do: 100 |> div(5) |> div(2)
      #=> {:|>, _, [arg1, arg2]}

  In the example above, the `|>` pipe is the right-most pipe; `arg1` is the AST
  for `100 |> div(5)`, and `arg2` is the AST for `div(2)`.

  It's often useful to have the AST for such a pipeline as a list of function
  applications. This function does exactly that:

      Macro.unpipe(quote do: 100 |> div(5) |> div(2))
      #=> [{100, 0}, {{:div, [], [5]}, 0}, {{:div, [], [2]}, 0}]

  We get a list that follows the pipeline directly: first the `100`, then the
  `div(5)` (more precisely, its AST), then `div(2)`. The `0` as the second
  element of the tuples is the position of the previous element in the pipeline
  inside the current function application: `{{:div, [], [5]}, 0}` means that the
  previous element (`100`) will be inserted as the 0th (first) argument to the
  `div/2` function, so that the AST for that function will become `{:div, [],
  [100, 5]}` (`div(100, 5)`).
  """
  @spec unpipe(t()) :: [t()]
  def unpipe(expr) do
    :lists.reverse(unpipe(expr, []))
  end

  defp unpipe({:|>, _, [left, right]}, acc) do
    unpipe(right, unpipe(left, acc))
  end

  defp unpipe(other, acc) do
    [{other, 0} | acc]
  end

  @doc """
  Pipes `expr` into the `call_args` at the given `position`.
  """
  @spec pipe(t(), t(), integer) :: t()
  def pipe(expr, call_args, position)

  def pipe(expr, {:&, _, _} = call_args, _integer) do
    raise ArgumentError, bad_pipe(expr, call_args)
  end

  def pipe(expr, {tuple_or_map, _, _} = call_args, _integer) when tuple_or_map in [:{}, :%{}] do
    raise ArgumentError, bad_pipe(expr, call_args)
  end

  # Without this, `Macro |> Env == Macro.Env`.
  def pipe(expr, {:__aliases__, _, _} = call_args, _integer) do
    raise ArgumentError, bad_pipe(expr, call_args)
  end

  def pipe(expr, {:<<>>, _, _} = call_args, _integer) do
    raise ArgumentError, bad_pipe(expr, call_args)
  end

  def pipe(expr, {unquote, _, []}, _integer) when unquote in [:unquote, :unquote_splicing] do
    raise ArgumentError,
          "cannot pipe #{to_string(expr)} into the special form #{unquote}/1 " <>
            "since #{unquote}/1 is used to build the Elixir AST itself"
  end

  # {:fn, _, _} is what we get when we pipe into an anonymous function without
  # calling it, for example, `:foo |> (fn x -> x end)`.
  def pipe(expr, {:fn, _, _}, _integer) do
    raise ArgumentError,
          "cannot pipe #{to_string(expr)} into an anonymous function without" <>
            " calling the function; use Kernel.then/2 instead or" <>
            " define the anonymous function as a regular private function"
  end

  def pipe(expr, {call, line, atom}, integer) when is_atom(atom) do
    {call, line, List.insert_at([], integer, expr)}
  end

  def pipe(_expr, {op, _line, [arg]}, _integer) when op == :+ or op == :- do
    raise ArgumentError,
          "piping into a unary operator is not supported, please use the qualified name: " <>
            "Kernel.#{op}(#{to_string(arg)}), instead of #{op}#{to_string(arg)}"
  end

  def pipe(expr, {op, line, args} = op_args, integer) when is_list(args) do
    cond do
      is_atom(op) and Identifier.unary_op(op) != :error ->
        raise ArgumentError,
              "cannot pipe #{to_string(expr)} into #{to_string(op_args)}, " <>
                "the #{to_string(op)} operator can only take one argument"

      is_atom(op) and Identifier.binary_op(op) != :error ->
        raise ArgumentError,
              "cannot pipe #{to_string(expr)} into #{to_string(op_args)}, " <>
                "the #{to_string(op)} operator can only take two arguments"

      true ->
        {op, line, List.insert_at(args, integer, expr)}
    end
  end

  def pipe(expr, call_args, _integer) do
    raise ArgumentError, bad_pipe(expr, call_args)
  end

  defp bad_pipe(expr, call_args) do
    "cannot pipe #{to_string(expr)} into #{to_string(call_args)}, " <>
      "can only pipe into local calls foo(), remote calls Foo.bar() or anonymous function calls foo.()"
  end

  @doc """
  Applies the given function to the node metadata if it contains one.

  This is often useful when used with `Macro.prewalk/2` to remove
  information like lines and hygienic counters from the expression
  for either storage or comparison.

  ## Examples

      iex> quoted = quote line: 10, do: sample()
      {:sample, [line: 10], []}
      iex> Macro.update_meta(quoted, &Keyword.delete(&1, :line))
      {:sample, [], []}

  """
  @spec update_meta(t, (keyword -> keyword)) :: t
  def update_meta(quoted, fun)

  def update_meta({left, meta, right}, fun) when is_list(meta) do
    {left, fun.(meta), right}
  end

  def update_meta(other, _fun) do
    other
  end

  @doc """
  Generates AST nodes for a given number of required argument
  variables using `Macro.var/2`.

  Note the arguments are not unique. If you later on want
  to access the same variables, you can invoke this function
  with the same inputs. Use `generate_unique_arguments/2` to
  generate a unique arguments that can't be overridden.

  ## Examples

      iex> Macro.generate_arguments(2, __MODULE__)
      [{:arg1, [], __MODULE__}, {:arg2, [], __MODULE__}]

  """
  @doc since: "1.5.0"
  @spec generate_arguments(0, context :: atom) :: []
  @spec generate_arguments(pos_integer, context) :: [{atom, [], context}, ...] when context: atom
  def generate_arguments(amount, context), do: generate_arguments(amount, context, &var/2)

  @doc """
  Generates AST nodes for a given number of required argument
  variables using `Macro.unique_var/2`.

  ## Examples

      iex> [var1, var2] = Macro.generate_unique_arguments(2, __MODULE__)
      iex> {:arg1, [counter: c1], __MODULE__} = var1
      iex> {:arg2, [counter: c2], __MODULE__} = var2
      iex> is_integer(c1) and is_integer(c2)
      true

  """
  @doc since: "1.11.3"
  @spec generate_unique_arguments(0, context :: atom) :: []
  @spec generate_unique_arguments(pos_integer, context) :: [
          {atom, [counter: integer], context},
          ...
        ]
        when context: atom
  def generate_unique_arguments(amount, context),
    do: generate_arguments(amount, context, &unique_var/2)

  defp generate_arguments(0, context, _fun) when is_atom(context), do: []

  defp generate_arguments(amount, context, fun)
       when is_integer(amount) and amount > 0 and is_atom(context) do
    for id <- 1..amount, do: fun.(String.to_atom("arg" <> Integer.to_string(id)), context)
  end

  @doc """
  Generates an AST node representing the variable given
  by the atoms `var` and `context`.

  Note this variable is not unique. If you later on want
  to access this same variable, you can invoke `var/2`
  again with the same arguments. Use `unique_var/2` to
  generate a unique variable that can't be overridden.

  ## Examples

  In order to build a variable, a context is expected.
  Most of the times, in order to preserve hygiene, the
  context must be `__MODULE__/0`:

      iex> Macro.var(:foo, __MODULE__)
      {:foo, [], __MODULE__}

  However, if there is a need to access the user variable,
  nil can be given:

      iex> Macro.var(:foo, nil)
      {:foo, [], nil}

  """
  @spec var(var, context) :: {var, [], context} when var: atom, context: atom
  def var(var, context) when is_atom(var) and is_atom(context) do
    {var, [], context}
  end

  @doc """
  Generates an AST node representing a unique variable
  given by the atoms `var` and `context`.

  ## Examples

      iex> {:foo, [counter: c], __MODULE__} = Macro.unique_var(:foo, __MODULE__)
      iex> is_integer(c)
      true

  """
  @doc since: "1.11.3"
  @spec unique_var(var, context) :: {var, [counter: integer], context}
        when var: atom, context: atom
  def unique_var(var, context) when is_atom(var) and is_atom(context) do
    {var, [counter: :elixir_module.next_counter(context)], context}
  end

  @doc """
  Performs a depth-first traversal of quoted expressions
  using an accumulator.
  """
  @spec traverse(t, any, (t, any -> {t, any}), (t, any -> {t, any})) :: {t, any}
  def traverse(ast, acc, pre, post) when is_function(pre, 2) and is_function(post, 2) do
    {ast, acc} = pre.(ast, acc)
    do_traverse(ast, acc, pre, post)
  end

  defp do_traverse({form, meta, args}, acc, pre, post) when is_atom(form) do
    {args, acc} = do_traverse_args(args, acc, pre, post)
    post.({form, meta, args}, acc)
  end

  defp do_traverse({form, meta, args}, acc, pre, post) do
    {form, acc} = pre.(form, acc)
    {form, acc} = do_traverse(form, acc, pre, post)
    {args, acc} = do_traverse_args(args, acc, pre, post)
    post.({form, meta, args}, acc)
  end

  defp do_traverse({left, right}, acc, pre, post) do
    {left, acc} = pre.(left, acc)
    {left, acc} = do_traverse(left, acc, pre, post)
    {right, acc} = pre.(right, acc)
    {right, acc} = do_traverse(right, acc, pre, post)
    post.({left, right}, acc)
  end

  defp do_traverse(list, acc, pre, post) when is_list(list) do
    {list, acc} = do_traverse_args(list, acc, pre, post)
    post.(list, acc)
  end

  defp do_traverse(x, acc, _pre, post) do
    post.(x, acc)
  end

  defp do_traverse_args(args, acc, _pre, _post) when is_atom(args) do
    {args, acc}
  end

  defp do_traverse_args(args, acc, pre, post) when is_list(args) do
    :lists.mapfoldl(
      fn x, acc ->
        {x, acc} = pre.(x, acc)
        do_traverse(x, acc, pre, post)
      end,
      acc,
      args
    )
  end

  @doc """
  Performs a depth-first, pre-order traversal of quoted expressions.

  Returns a new ast where each node is the result of invoking `fun` on each
  corresponding node of `ast`.

  ## Examples

      iex> ast = quote do: 5 + 3 * 7
      iex> new_ast = Macro.prewalk(ast, fn
      ...>   {:+, meta, children} -> {:*, meta, children}
      ...>   {:*, meta, children} -> {:+, meta, children}
      ...>   other -> other
      ...> end)
      iex> Code.eval_quoted(ast)
      {26, []}
      iex> Code.eval_quoted(new_ast)
      {50, []}

  """
  @spec prewalk(t, (t -> t)) :: t
  def prewalk(ast, fun) when is_function(fun, 1) do
    elem(prewalk(ast, nil, fn x, nil -> {fun.(x), nil} end), 0)
  end

{'fraction_non_alphanumeric': 0.11668621700879765, 'fraction_numerical': 0.00782991202346041, 'mean_word_length': 2.969848661233993, 'pattern_counts': {'":': 0, '<': 29, '<?xml version=': 0, '>': 138, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '4703566', 'n_tokens_mistral': 11160, 'n_tokens_neox': 10646, 'n_words': 4126}