parameters: (self, text, identStart=None, tokenSep=None, ignoreWhitespace=True)
docstring: Turn a string into individual tokens.  Three types of tokens are recognized:
      TOKEN_IDENT:   identifiers (those that start with the identStart pattern)
      TOKEN_STRING:  quoted string
      TOKEN_OP:      everything else
  Tokens are separated by white space or the tokenSep pattern.  
  Constructor parameters:
      text:  The string to tokenize
      identStart:  A regular expression describing characters which start an identifier
                   The default expression accepts letters, "_", and "/".
      tokenSep:    A regular expression describing the characters which end a token 
                   (in addition to whitespace).  The default expression accepts
                   anything except alpha-numerics, "_", "/", and ":".
  Usage:
      Invoke getNextToken (or next) to get the next token.  The instance variables
      token, and tokenVal will be populated with the current token type (TOKEN_IDENT,
      TOKEN_STRING, or TOEKN_OP) and value respectively.  nextToken and nextTokenVal
      will also be available for lookahead.   The next method is similar to
      getNextToken but also returns the token value.  A value of None signals end
      of stream.
docstring_summary: Turn a string into individual tokens.  Three types of tokens are recognized:
      TOKEN_IDENT:   identifiers (those that start with the identStart pattern)
      TOKEN_STRING:  quoted string
      TOKEN_OP:      everything else
  Tokens are separated by white space or the tokenSep pattern.  
  Constructor parameters:
      text:  The string to tokenize
      identStart:  A regular expression describing characters which start an identifier
                   The default expression accepts letters, "_", and "/".
      tokenSep:    A regular expression describing the characters which end a token 
                   (in addition to whitespace).  The default expression accepts
                   anything except alpha-numerics, "_", "/", and ":".
  Usage:
      Invoke getNextToken (or next) to get the next token.  The instance variables
      token, and tokenVal will be populated with the current token type (TOKEN_IDENT,
      TOKEN_STRING, or TOEKN_OP) and value respectively.  nextToken and nextTokenVal
      will also be available for lookahead.   The next method is similar to
      getNextToken but also returns the token value.  A value of None signals end
      of stream.
docstring_tokens: Turn a string into individual tokens . Three types of tokens are recognized : TOKEN_IDENT : identifiers ( those that start with the identStart pattern ) TOKEN_STRING : quoted string TOKEN_OP : everything else Tokens are separated by white space or the tokenSep pattern . Constructor parameters : text : The string to tokenize identStart : A regular expression describing characters which start an identifier The default expression accepts letters _ and / . tokenSep : A regular expression describing the characters which end a token ( in addition to whitespace ) . The default expression accepts anything except alpha - numerics _ / and : . Usage : Invoke getNextToken ( or next ) to get the next token . The instance variables token and tokenVal will be populated with the current token type ( TOKEN_IDENT TOKEN_STRING or TOEKN_OP ) and value respectively . nextToken and nextTokenVal will also be available for lookahead . The next method is similar to getNextToken but also returns the token value . A value of None signals end of stream .
function: def __init__(self, text, identStart=None, tokenSep=None, ignoreWhitespace=True):
          """
  Turn a string into individual tokens.  Three types of tokens are recognized:
      TOKEN_IDENT:   identifiers (those that start with the identStart pattern)
      TOKEN_STRING:  quoted string
      TOKEN_OP:      everything else
  Tokens are separated by white space or the tokenSep pattern.  
  Constructor parameters:
      text:  The string to tokenize
      identStart:  A regular expression describing characters which start an identifier
                   The default expression accepts letters, "_", and "/".
      tokenSep:    A regular expression describing the characters which end a token 
                   (in addition to whitespace).  The default expression accepts
                   anything except alpha-numerics, "_", "/", and ":".
  Usage:
      Invoke getNextToken (or next) to get the next token.  The instance variables
      token, and tokenVal will be populated with the current token type (TOKEN_IDENT,
      TOKEN_STRING, or TOEKN_OP) and value respectively.  nextToken and nextTokenVal
      will also be available for lookahead.   The next method is similar to
      getNextToken but also returns the token value.  A value of None signals end
      of stream.                                           
          """
          self.ignoreWhitespace=ignoreWhitespace
          ifDefPy()
          if (isinstance(text, array.array)):
              text = text.tostring()
          endIfDef()
          self.text = asString(text)
          self.textIndex = 0
          self.textLen = len(self.text)
          self.token = None
          self.tokenVal = None
          self.nextToken = None
          self.nextTokenVal = None
          if (identStart == None):
              identStart = "[a-zA-Z_/]"
          if (tokenSep == None):
              tokenSep = "[^a-zA-Z0-9_/:]"
          self.identStart = re.compile(identStart)
          self.tokenSep = re.compile(tokenSep)
          self.getNextToken()
function_tokens: def __init__ ( self , text , identStart = None , tokenSep = None , ignoreWhitespace = True ) : self . ignoreWhitespace = ignoreWhitespace ifDefPy ( ) if ( isinstance ( text , array . array ) ) : text = text . tostring ( ) endIfDef ( ) self . text = asString ( text ) self . textIndex = 0 self . textLen = len ( self . text ) self . token = None self . tokenVal = None self . nextToken = None self . nextTokenVal = None if ( identStart == None ) : identStart = "[a-zA-Z_/]" if ( tokenSep == None ) : tokenSep = "[^a-zA-Z0-9_/:]" self . identStart = re . compile ( identStart ) self . tokenSep = re . compile ( tokenSep ) self . getNextToken ( )