<jupyter_start><jupyter_text>Fixed-Width File Reader Sample Usage**Fixed-width files**: each field in a column is padded to a given number of bytes.**Cython**: Python project to automatically generate Python bindings for C++ scripts.**Python vs C++**: simple but slow vs fast but complicated. Goals* C++ project to read fixed-width files into in-memory Apache Arrow tables.* Includes Cython-made Python bindings for accessibility and integration with Python projects.We use Apache Arrow's CSV reader as a base since it required only fairly minimal changes for this application.* Modify the reader to optionally decode on read-in.* Modify the parser to use field widths instead of a delimiter to separate the input stream into fields.* Modify the converter to optionally convert COBOL-formatted numeric types to standard numeric types. UseFor installation, see this repo's README.First, we'll create an example fixed-width file, including COBOL-formatted numbers, null values and odd spacing.<jupyter_code>%%file sample.fwf
aa           bb     cc  
hello        123}   3.56
hi           9129A  NaN 
     spaces   N/A   7.8 
NA           3{     0<jupyter_output>Writing sample.fwf<jupyter_text>Now we can run the module on the fixed-width file. See the README for a full list of options. Note that the converter uses preset lists/maps for null values and the COBOL values below. These can be modified through:* convert_options.null_values = \[some new list\]* convert_options.\[pos/neg\]\_values = {some new mapping}<jupyter_code>import pyfwfr as pf

convert_options = pf.ConvertOptions(is_cobol=True, strings_can_be_null=True)
parse_options = pf.ParseOptions([13, 7, 4])

table = pf.read_fwf('sample.fwf', parse_options, 
                    convert_options=convert_options)
for column in table.columns:
    print(column)<jupyter_output><Column name='aa' type=DataType(string)>
[
  [
    "hello",
    "hi",
    "spaces",
    null
  ]
]
<Column name='bb' type=DataType(int64)>
[
  [
    -1230,
    91291,
    null,
    30
  ]
]
<Column name='cc' type=DataType(double)>
[
  [
    3.56,
    null,
    7.8,
    0
  ]
]
{'fraction_non_alphanumeric': 0.0866067203028869, 'fraction_numerical': 0.017510648367250355, 'mean_word_length': 3.6873614190687363, 'pattern_counts': {'":': 0, '<': 10, '<?xml version=': 0, '>': 10, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '2', 'dir': 'jupyter-structured-clean-dedup', 'id': '412266', 'license': 'Apache-2.0', 'n_tokens_mistral': 662, 'n_tokens_neox': 616, 'n_words': 240, 'path': 'notebooks/fwfr_sample_usage.ipynb', 'repo_name': 'artemis-analytics/fwfr', 'size': '2113'}