<commit_before>/**
  @file
  @brief Returns all files and subdirectories within a specified parent
  @details When used with getattrs=NO, is not OS specific (uses dopen / dread). 

  If getattrs=YES then the doptname / foptname functions are used to scan all
  properties - any characters that are not valid in a SAS name (v7) are simply 
  stripped, and the table is transposed so theat each property is a column
  and there is one file per row.  An attempt is made to get all properties 
  whether a file or folder, but some files/folders cannot be accessed, and so
  not all properties can / will be populated.

  Credit for the rename approach:
  https://communities.sas.com/t5/SAS-Programming/SAS-Function-to-convert-string-to-Legal-SAS-Name/m-p/27375/highlight/true#M5003 


  usage:

      %mp_dirlist(path=/some/location,outds=myTable)

      %mp_dirlist(outds=cwdfileprops, getattrs=YES)

  @warning In a Unix environment, the existence of a named pipe will cause this 
  macro to hang.  Therefore this tool should be used with caution in a SAS 9 web
  application, as it can use up all available multibridge sessions of the requests
  are resubmitted.
  If anyone finds a way to positively identify a named pipe using SAS (without 
  X CMD) do please raise an issue!


  @param path= for which to return contents
  @param outds= the output dataset to create
  @param getattrs= uses doptname and foptname to return all attributes for each
  file / folder.  


  @returns outds contains the following variables:
   - file_or_folder (file / folder)
   - filepath (path/to/file.name)
   - filename (just the file name)
   - ext (.extension)
   - msg (system message if any issues)
   - OS SPECIFIC variables, if <code>getattrs=</code> is used.

  @version 9.2<commit_msg>fix: documentation<commit_after>/**
  @file
  @brief Returns all files and subdirectories within a specified parent
  @details When used with getattrs=NO, is not OS specific (uses dopen / dread). 

  If getattrs=YES then the doptname / foptname functions are used to scan all
  properties - any characters that are not valid in a SAS name (v7) are simply 
  stripped, and the table is transposed so theat each property is a column
  and there is one file per row.  An attempt is made to get all properties 
  whether a file or folder, but some files/folders cannot be accessed, and so
  not all properties can / will be populated.

  Credit for the rename approach:
  https://communities.sas.com/t5/SAS-Programming/SAS-Function-to-convert-string-to-Legal-SAS-Name/m-p/27375/highlight/true#M5003 


  usage:

      %mp_dirlist(path=/some/location,outds=myTable)

      %mp_dirlist(outds=cwdfileprops, getattrs=YES)

  @warning In a Unix environment, the existence of a named pipe will cause this 
  macro to hang.  Therefore this tool should be used with caution in a SAS 9 web
  application, as it can use up all available multibridge sessions if requests
  are resubmitted.
  If anyone finds a way to positively identify a named pipe using SAS (without 
  X CMD) do please raise an issue!


  @param path= for which to return contents
  @param outds= the output dataset to create
  @param getattrs= YES/NO (default=NO).  Uses doptname and foptname to return 
  all attributes for each file / folder.  


  @returns outds contains the following variables:
   - file_or_folder (file / folder)
   - filepath (path/to/file.name)
   - filename (just the file name)
   - ext (.extension)
   - msg (system message if any issues)
   - OS SPECIFIC variables, if <code>getattrs=</code> is used.

  @version 9.2
{'fraction_non_alphanumeric': 0.06595863610955842, 'fraction_numerical': 0.007825600894354388, 'mean_word_length': 3.479349186483104, 'pattern_counts': {'":': 0, '<': 7, '<?xml version=': 0, '>': 7, 'https://': 2, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '7014983', 'n_tokens_mistral': 1106, 'n_tokens_neox': 1029, 'n_words': 473}