<commit_before>Pandas
======

### Python ###

* `read_table` and `read_csv` come from `_make_parser_function()`, just with different default separation
* `_make_parser_function()` creates a function that calls `_read()`
* `_read()` calls `TextFileReader.read()`
* `TextFileReader` uses an engine for reading, which is `CParserWrapper` by default (`PythonParser` and `FixedWidthFieldParser` are others)
* `TextFileReader.read()` calls the `read()` function of its engine, instantiates a `DataFrame` with `col_dict`, `columns`, `index`
* `CParserWrapper.read()` calls `TextReader.read()` from `pandas.parser` (Cython/C)

### Cython ###
* `TextReader.__cinit__`:
```
self.parser = parser_new()
parser_init(self.parser)
```
* `TextReader.read`:
```
return self._read_rows(None, 1)
```
* `TextReader._read_rows`:
```
tokenize_all_rows(self.parser)
columns = self._convert_column_data()
if len(columns) > 0:
	rows_read = len(list(columns.values())[0])
	parser_consume_rows(self.parser, rows_read)
	parser_trim_buffers(self.parser)
	self.parser_start -= rows_read
return columns
```
* `TextReader._convert_column_data`:
```
num_cols = -1
for i in range(self.parser.lines):
	num_cols = max(num_cols, self.parsers.line_fields[i])
results = {}
for i in range(self.table_width):
	results[i] = self._convert_tokens(i, self.parser_start, self.parser.lines)
```
* `TextReader._convert_tokens`:
```
col_res = None
for dt in ['<i8', '<f8', '|b1', '|O8']:
	col_res = self._convert_with_dtype(dt, i, start, end)
	if col_res is not None:
		break
return col_res
```
* `TextReader._convert_with_dtype`:
```
if dtype[1] == 'i':
	return _try_int64(self.parser, i, start, end)
elif dtype[1] == 'f':
	return _try_double(self.parser, i, start, end)
elif dtype[1] == 'b':
	return _try_bool(self.parser, i, start, end)
elif dtype[1] == 'O':
	return self._string_convert(i, start, end)
```
* `_try_int64`, `_try_double`, `_try_bool`:
```
lines = line_end - line_start
result = np.empty(lines, dtype=np.int64) # np.float64, np.uint8
data = <int64_t *> result.data # <double *>, <uint8_t *>
cdef:
	coliter_t it
	char *word
	int error
coliter_setup(&it, parser, col, line_start)
for i in range(lines):
	word = COLITER_NEXT(it)
	data[i] = str_to_int64(word, INT64_MIN, INT64_MAX, &error)
	# or error = to_double(word, data, parser.sci, parser.decimal)
	# or error = to_boolean(word, data)
	# int:
	if error != 0:
		if error == ERROR_OVERFLOW:
			raise OverflowError(word)
		return None, None
	# double:
	if error != 1:
		if strcasecmp(word, cinf) == 0:
			data[0] = INF
		elif strcasecmp(word, cneginf) == 0:
			data[0] = NEGINF
		else:
			return None, None
		data += 1
	# bool:
	if error != 0:
		return None, None
	data += 1
return result # or result.view(np.bool_)
```
* `_try_double`:
```
lines = line_end - line_start
result = np.empty(lines, dtype=np.float64)
data = <double *> result.data
cdef:
	coliter_t it
	char *word
	int error
coliter_setup(&it, parser, col, line_start)
for i in range(lines):
	word = COLITER_NEXT(it)
	data[i] = str_to_int64(word, INT64_MIN, INT64_MAX, &error)
	if error != 0:
		if error == ERROR_OVERFLOW:
			raise OverflowError(word)
		return None, None
return result
```
### C ###
* `tokenize_all_rows`:
```
return _tokenize_helper(self, -1, 1);
```

* `_tokenize_helper`:
```
parser_op tokenize_bytes; // function of self, nrows
if (self->delim_whitespace)
	tokenize_bytes = tokenize_whitespace;<commit_msg>Erased superfluous section of pandas writeup<commit_after>Pandas
======

### Python ###

* `read_table` and `read_csv` come from `_make_parser_function()`, just with different default separation
* `_make_parser_function()` creates a function that calls `_read()`
* `_read()` calls `TextFileReader.read()`
* `TextFileReader` uses an engine for reading, which is `CParserWrapper` by default (`PythonParser` and `FixedWidthFieldParser` are others)
* `TextFileReader.read()` calls the `read()` function of its engine, instantiates a `DataFrame` with `col_dict`, `columns`, `index`
* `CParserWrapper.read()` calls `TextReader.read()` from `pandas.parser` (Cython/C)

### Cython ###
* `TextReader.__cinit__`:
```
self.parser = parser_new()
parser_init(self.parser)
```
* `TextReader.read`:
```
return self._read_rows(None, 1)
```
* `TextReader._read_rows`:
```
tokenize_all_rows(self.parser)
columns = self._convert_column_data()
if len(columns) > 0:
	rows_read = len(list(columns.values())[0])
	parser_consume_rows(self.parser, rows_read)
	parser_trim_buffers(self.parser)
	self.parser_start -= rows_read
return columns
```
* `TextReader._convert_column_data`:
```
num_cols = -1
for i in range(self.parser.lines):
	num_cols = max(num_cols, self.parsers.line_fields[i])
results = {}
for i in range(self.table_width):
	results[i] = self._convert_tokens(i, self.parser_start, self.parser.lines)
```
* `TextReader._convert_tokens`:
```
col_res = None
for dt in ['<i8', '<f8', '|b1', '|O8']:
	col_res = self._convert_with_dtype(dt, i, start, end)
	if col_res is not None:
		break
return col_res
```
* `TextReader._convert_with_dtype`:
```
if dtype[1] == 'i':
	return _try_int64(self.parser, i, start, end)
elif dtype[1] == 'f':
	return _try_double(self.parser, i, start, end)
elif dtype[1] == 'b':
	return _try_bool(self.parser, i, start, end)
elif dtype[1] == 'O':
	return self._string_convert(i, start, end)
```
* `_try_int64`, `_try_double`, `_try_bool`:
```
lines = line_end - line_start
result = np.empty(lines, dtype=np.int64) # np.float64, np.uint8
data = <int64_t *> result.data # <double *>, <uint8_t *>
cdef:
	coliter_t it
	char *word
	int error
coliter_setup(&it, parser, col, line_start)
for i in range(lines):
	word = COLITER_NEXT(it)
	data[i] = str_to_int64(word, INT64_MIN, INT64_MAX, &error)
	# or error = to_double(word, data, parser.sci, parser.decimal)
	# or error = to_boolean(word, data)
	# int:
	if error != 0:
		if error == ERROR_OVERFLOW:
			raise OverflowError(word)
		return None, None
	# double:
	if error != 1:
		if strcasecmp(word, cinf) == 0:
			data[0] = INF
		elif strcasecmp(word, cneginf) == 0:
			data[0] = NEGINF
		else:
			return None, None
		data += 1
	# bool:
	if error != 0:
		return None, None
	data += 1
return result # or result.view(np.bool_)
```
### C ###
* `tokenize_all_rows`:
```
return _tokenize_helper(self, -1, 1);
```

* `_tokenize_helper`:
```
parser_op tokenize_bytes; // function of self, nrows
if (self->delim_whitespace)
	tokenize_bytes = tokenize_whitespace;
{'fraction_non_alphanumeric': 0.1609231249025417, 'fraction_numerical': 0.014189926711367535, 'mean_word_length': 5.33794466403162, 'pattern_counts': {'":': 0, '<': 14, '<?xml version=': 0, '>': 14, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '4287406', 'n_tokens_mistral': 2564, 'n_tokens_neox': 2404, 'n_words': 591}