parameters: (readline, tokeneater=printtoken)
docstring: The tokenize() function accepts two parameters: one representing the
      input stream, and one providing an output mechanism for tokenize().
  
      The first parameter, readline, must be a callable object which provides
      the same interface as the readline() method of built-in file objects.
      Each call to the function should return one line of input as a string.
  
      The second parameter, tokeneater, must also be a callable object. It is
      called once for each token, with five arguments, corresponding to the
      tuples generated by generate_tokens().
docstring_summary: The tokenize() function accepts two parameters: one representing the
      input stream, and one providing an output mechanism for tokenize().
docstring_tokens: The tokenize () function accepts two parameters : one representing the input stream and one providing an output mechanism for tokenize () .
function: def tokenize(readline, tokeneater=printtoken):
      """
      The tokenize() function accepts two parameters: one representing the
      input stream, and one providing an output mechanism for tokenize().
  
      The first parameter, readline, must be a callable object which provides
      the same interface as the readline() method of built-in file objects.
      Each call to the function should return one line of input as a string.
  
      The second parameter, tokeneater, must also be a callable object. It is
      called once for each token, with five arguments, corresponding to the
      tuples generated by generate_tokens().
      """
      try:
          tokenize_loop(readline, tokeneater)
      except StopTokenizing:
          pass
function_tokens: def tokenize ( readline , tokeneater = printtoken ) : try : tokenize_loop ( readline , tokeneater ) except StopTokenizing : pass