<commit_before>.. _theory:

Theory and Implementation Details
=================================
Automated Time Stepping
-----------------------
EXECUTIVE SUMMARY
A means of “automatic time stepping” in RMG was desired in order to eliminate the need for the user to specify times/conversions for mechanism validity testing. An “AUTO” option implementing this “automatic time stepping” was successfully integrated into RMG. Several tests were carried out with the new “AUTO” option. Results were promising, suggesting that the option functioned as intended without being unreasonably computationally expensive. The “AUTO” option offers a more rigorous implementation of RMG’s rate-based mechanism generation algorithm while allowing straightforward generation of reaction mechanisms that encompass chemistry at all time-scales of interest.
MOTIVATION
Originally, the implementation of RMG’s reaction mechanism generation algorithm requires the user to specify times or conversions at which the program will estimate “edge” reaction fluxes and decide whether to include a new species in the “core” reaction model. The choice of which conversions or times to specify is, to a certain extent, arbitrary, and could have a significant impact on the model generated by the program. This arbitrariness can be effectively eliminated by keeping track of the “edge” reaction flux “continuously” within the ODE solver (analogously to how conversion is handled) and terminating ODE solver “integration” when the threshold reaction flux is reached (whereupon a species would be moved from the “edge” to the “core” as in the current RMG implementation). This new method is here referred to as “automatic time stepping”. Implementation of such a method would reduce the number of adjustable parameters that the user must tweak during mechanism generation and provide a more rigorous use of the mechanism generation algorithm. At the same time, such a method would allow straightforward generation of mechanisms that encompass chemistry at all time-scales of interest. 
BACKGROUND 
	RMG is a program for automated reaction mechanism generation initially developed by <NAME> and <NAME> at the Department of Chemical Engineering at M.I.T. Details regarding the program are available elsewhere (see, for example, Refs. 1, 2, 3, and 4). However, the rate-based reaction mechanism generation algorithm used by RMG will be reviewed, as it is a key foundation of the work described here.
	The reaction mechanism is divided into a “core” and an “edge”. The “core” is tracked by integrating the appropriate equations using an ordinary differential equation (ODE) solver. The “edge” includes species that are products of reactions among the “core” species. Periodically during the course of simulating species concentrations using the ODE solver, mechanism validity is tested. Mechanism validity criteria used by RMG are shown in Equation 1, below.
 						(1)
In these criteria, tol is a tolerance specified by the user. If any of these criteria are not valid, the edge species with the highest flux is added to the mechanism “core” and the ODE solver restarts simulation from t = 0. In the original implementation, the times or conversions at which model validity is tested are specified by the user. (See also Figure 1 in the next section for a depiction of the method used.)
	The ODE solvers used by RMG are FORTRAN programs (DASSL or DASPK), while RMG itself is written in Java.
IMPLEMENTATION
In order to achieve the desired behavior for “automatic time stepping”, modifications of the type illustrated in Figure 1 were desired.
 
Figure 1. Modifications to implement “automatic time stepping”.
As the figure shows, a significant required change was the modification of the ODE solver to have access to edge species flux information at each ODE solver time step. Thus, much of the modification effort involved implementing a scheme for passing edge reaction information to the ODE solver. Using the latest CVS version of RMG as a starting point, the implementation ended up requiring changes to the Java classes ReactionModelGenerator and JDASSL (and Global for timing purposes) and the FORTRAN file call_dassl.f (the modified version was renamed to call_dasslAUTO.f90 to distinguish from the original version). Also, a minor change to the Java class ReactionTimeTT was made.
	An example of an ODE solver input file for “automatic time stepping” is shown in Figure 2, below. Differences from “normal” operation are highlighted with boxes.
 
Figure 2. ODE solver input file example for “automatic time stepping”. Boxes highlight differences from the “normal” version.

The first box in the above figure highlights an integer flag signaling to the FORTRAN code that “automatic time stepping” is desired (as opposed to “normal” operation”). The next box highlights the section containing the information needed by the ODE solver to check mechanism validity at each step. In particular, this includes the user-specified tolerance, the number of edge species/reactions, and a line for each edge reaction containing the number of reactants, the number of products, ID numbers corresponding to reactants and products, and finally the rate coefficient, k, for the reaction. It should be noted that in the current implementation, the rate coefficient is considered to be a constant, as additional information regarding temperature/pressure dependence is not passed to solver. Thus, the “automatic time stepping” code (along with other preexisting DASSL code) would need to be modified to handle non-isothermal cases (as well as non-isobaric pressure-dependent cases).
	In order to reproduce existing validity testing for cases with pressure-dependence, special considerations are necessary. In particular, each pressure-dependent network is considered as a “pseudo-species” with an associated “pseudo-reaction” using the kleak associated with the network as the rate coefficient. Monitoring flux to these “pseudo-species” thus allows us to consider whether a pressure-dependent network needs to be enlarged, analogously to how monitoring of species flux allows us to consider whether a species needs to be added to the “core” reaction mechanism.
USAGE NOTES
The modified code still allows for evaluating flux at user-defined time/conversion points, and the use of “automatic time stepping” is invoked by replacing the user-defined time/conversion points with the keyword “AUTO”. Thus, the use of “automatic time stepping” will be referred to as the “AUTO” method in subsequent discussion. At the time of this writing, the “AUTO” method has been implemented and tested in DASSL. Implementation in DASPK is very straightforward due to the fact that the portions of code modified are essentially identical between the two programs; thus, extension to DASPK will simply require cutting and pasting of the modified DASSL code. (Testing the modified DASPK in Windows, however, is currently not possible, as I have not been able to successfully compile DASPK with DAEPACK for use in Windows.) The modified code has been designed to work for both pressure-dependent and non-pressure-dependent cases; also, the code allows automatic stepping with either a conversion or a reaction time as the user-specified termination criterion.
RESULTS
The new option of “AUTO” time-stepping was tested for several cases to verify that the option was working as desired and to investigate the effects on mechanism generation time and on the resulting mechanism. Note: Throughout this section, “1,3-hexadiene” refers to a system involving methane doped with 1,3-hexadiene (in particular, the system defined by the condition files distributed with the March 2007 release of RMG).

Proof of Principle
	The AUTO method was tested on pressure-dependent reaction mechanism generation for the 1,3-hexadiene system described previously, with a goal conversion of 0.9 and use of the ODE solver DASSL. Since the AUTO method was used, no intermediate concentrations/time-steps were specified in the input file. Using the code described previously, mechanism generation completes without error. Figure 3 shows how far the ODE solver “integrated” before the flux threshold was reached by one of the species (or “pseudo-species”).
 
Figure 3. 1,3-hexadiene mechanism generation case study with “AUTO” time steps (tol = 0.05). The points represent the times/conversions at which the ODE solver stopped due to the attainment of threshold flux by one of the edge species. Subsequently, the reaction mechanism would be modified and the ODE solver would restart at t = 0. The attainment of the goal conversion (0.9) is represented by the final (black) points at the right of the plot.

The plot shows that the ODE solver stops at a wide range of times/conversions. In contrast, without using the “AUTO” method, the points in such a plot would occur at discrete time or conversion levels corresponding to the values specified by the user in the input file.

Effects on Generated Mechanism
	One would expect that mechanisms generated using “AUTO” method would be more rigorously valid than those generated using arbitrary user-specified time/conversion steps. In the case of the 1,3-hexadiene system discussed above, mechanisms generated using the “AUTO” method were compared to mechanisms generated using evenly-spaced conversion steps. The results are shown in Table 1, below.
Table 1. Size of mechanisms generated using the “AUTO” method and varying numbers of evenly-spaced conversion steps. As in the previous section, the goal conversion is 0.9 and the tolerance is 0.05.
# of Conversion Steps	Core reactions/species
AUTO	48/15
1	308/31
2	56/17
3	55/16
4	55/16
6	55/16
9	48/15

It should be noted that the mechanism generated with the “AUTO” method and the mechanism generated using nine evenly-spaced conversion steps appear to be essentially identical for this system. Thus, in this case, we approach “AUTO” method behavior as we specify a finer “grid” for conversion steps, as one might intuitively expect.
It is also interesting to note that in this case, the larger mechanisms (corresponding to cases with fewer conversion steps) appeared to contain the smallest (48/15) mechanism as a sub-mechanism (i.e. the larger mechanisms appeared to include the species present in the smallest mechanism, as well as additional species). Thus, in this case, the mechanism generated by the “AUTO” method (and the case with nine conversion steps) would not be expected to be more accurate than the mechanism generated without any conversion steps (assuming the kinetics and thermochemistry of the extra species and reactions is accurate). However, the time required to generate the “AUTO” mechanism is significantly shorter, due to the smaller size of the mechanism. Furthermore, one can imagine that mechanisms of similar size to the largest mechanism reported above could be generated by using tighter tolerances and the “AUTO” method, and one could be more confident in their validity. Thus, even in this case, the use of a low number of conversion steps is not necessarily better than the use of a high number of conversion steps or the “AUTO” method.

Timing Studies
	In order to quantify the computational cost associated with keeping track of the edge species flux during use of the “AUTO” method, several timing studies were undertaken.  Results are shown in Table 2 and Table 3. Table 2 shows times associated with ODE “integration” accumulated over the course of mechanism generation, while Table 3 shows times associated with single instances of ODE “integration”. In the following tables, T3 refers to the time required to perform time-stepping within the ODE solver. T2 is the “FORTRAN” time requirement; that is, it includes time-stepping (T3), as well as the time associated with reading the ODE solver input file, assigning variables, producing output, etc. Finally, T1 is the time associated with the solve() method in the JDASSL class of the Java code; thus, in addition to including the time associated with executing the FORTRAN code (T2), the T1 time also includes time associated with additional tasks such as writing the input file for the FORTRAN ODE solver, assigning ID numbers to core species (and edge species, in the case of the “AUTO” method), and reading and processing the output from the ODE solver. In these tables, times were obtained on a Windows system (my personal laptop) with the FORTRAN code compiled using g95.
Table 2. Times associated with mechanism generation. In cases where uncertainties are reported, three repeat “measurements” were performed; the reported value corresponds to the mean of the three “measurements” and the uncertainty corresponds to the standard deviation. 1,3-hexadiene cases correspond to the same conditions discussed previously (goal conversion = 0.9, tol = 0.05). The mechanisms produced for all three 1,3-hexadiene cases appeared to be essentially identical.
Mechanism	Mechanism generation time (s)	Cumulative T1 (s)	Cumulative T2 (s)	Cumulative T3 (s)
1,3-hexadiene, AUTO	165 ± 4	3.15 ± 0.02	1.91 ± 0.05	1.59 ± 0.05
1,3-hexadiene,
9 conversions	167 ± 4	5.16 ± 0.21	1.79 ± 0.08	0.29 ± 0.03
1,3-hexadiene,
18 conversions	167 ± 4	8.99 ± 0.28	3.16 ± 0.20	0.30 ± 0.05
n-butane, 
AUTO	1115	83.56	73.56	70.39

Table 3. Times associated with a single run of the ODE solver. Methodology: Input files were extracted at various points in the mechanism generation process (using the “AUTO” method). The code was run and timed for the automatic case, and the final time in the output file was recorded. The input files were then manually adjusted to perform non-“AUTO” integration (referred to as “normal” in the table) with the final time from the output-file of the “AUTO” case specified as the end time in the “normal” input file. Reported times are “typical” values usually based on the average of the maximum and minimum time associated with multiple (~5-10) runs. Columns for “Edge Species” and “Edge Reactions” include “pseudo-species” and “pseudo-reactions” associated with pressure-dependent networks, as described previously.
System	Core Species	Edge Species	Edge Reactions	T2normal (s)	T2auto (s)	T2auto/ T2normal
1,3-hexadiene	16	581	1987	0.0625	0.203	3.2
	21	1947	5878	0.0781	0.469	6.0
	31	4416	13189	0.211	1.18	5.6
	34	4829	14823	0.352	2.75	7.8
	40	7982	25666	0.289	1.59	5.5
	44	9143	29480	0.406	3.14	7.8
n-butane	35	1841	5846	0.226	0.617	2.7
mixture (n-butane, n-butanol, acetylene, dimethyl ether)	52	2106	7558	0.586	1.12	1.9

Mechanism generation time results in Table 2 for 1,3-hexadiene show that the time required to generate a mechanism with the “AUTO” method is virtually the same as the time required to generate the same mechanism with conventional use of conversion steps. Table 3 shows, however, that the computational cost associated with tracking edge-species is significant compared to the cost of solving the ODEs, with the “AUTO” cases requiring a longer amount of time by a factor of two to eight (for the same number of ODE time steps). Three phenomena are proposed to explain this “discrepancy”. First, one would expect that in the “AUTO” case, the total time that we must integrate over during mechanism generation will be shorter, in general; that is, the “AUTO” method avoids “wasted” ODE time steps associated with “overshooting” the point where threshold flux has been reached. Second, the time associated with ODE solving is small in comparison to the total time requirement (< 10% of the total time for mechanism generation). Third, and perhaps most significant, there are fewer calls to the ODE solver when the “AUTO” method is used. In the case of the “AUTO” method, each call to the ODE solver is associated with a modification to the reaction mechanism (with the exception of the final integration). However, in the case of the “normal” method, there are extra calls to the ODE solver, as the integration is often stopped despite the mechanism being valid (i.e. no edge species fluxes exceed the threshold). Thus, in order to generate the same mechanism, the “normal” method requires a larger number of calls to the ODE solver, which, in turn is associated with greater overhead such as reading and writing ODE solver input and output files. Values for T1 and T2 in Table 2 for 1,3-hexadiene suggest that this overhead can be significant. (Note that the use of eighteen rather than nine conversions exacerbates the time cost, in accord with expectations based on the aforementioned overhead considerations.) The contribution of this overhead to FORTRAN time requirements is relatively small, but noticeable, while the contribution from the overhead on the Java side appears to be more significant. In fact, in this case, the total ODE solver time-requirement (T1) is noticeably lower in the “AUTO” case (relative to the “normal” cases). Thus, in this case, it appears that the added overhead associated with extra calls to the ODE solver in the “normal” method has outweighed the extra cost associated with tracking the flux at each time step within the ODE solver, using the “AUTO” method. It should be emphasized, however, that this phenomenon of faster “AUTO” operation has only been observed in the one case extensively studied thus far (in which we have been able to obtain the same mechanism using both “AUTO” and “normal” methods), and may not extend to other systems or situations with larger mechanisms. Further study is needed to determine if this behavior is typical.
In Table 2, the last case with n-butane is shown to illustrate performance of the “AUTO” method with a larger mechanism (249/34 core reactions/species rather than 48/15 in the case of 1,3-hexadiene). In Table 3, it is interesting to note that mechanisms with a larger proportion of edge species/reactions relative to core size appear to be associated with larger ratios T2auto/T2normal, as one might expect based on the nature of the added computation associated with the use of the “AUTO” method.

CONCLUSIONS
	The “AUTO” method was implemented and tested in RMG and was determined to function as intended for “automatic time stepping”. Even so, timing studies suggest that tracking the edge flux “continuously” in the ODE solver is associated with a significant time penalty. However, various mitigating factors appear to reduce the impact of this time penalty on overall mechanism generation time when the “AUTO” method is used. Thus, the “AUTO” method appears to be an appealing alternative to user-specification of time/conversion steps, suggesting that it holds promise for routine use in mechanism generation with RMG.

<commit_msg>second draft...still needs further refining<commit_after>.. _theory:

Theory and Implementation Details
=================================
Automated Time Stepping
-----------------------
EXECUTIVE SUMMARY
A means of “automated time stepping” in RMG was desired in order to eliminate the need for the user to specify times/conversions for mechanism validity testing. An “AUTO” option implementing this “automated time stepping” was successfully integrated into RMG. Several tests were carried out with the new “AUTO” option. Results were promising, suggesting that the option functioned as intended without being unreasonably computationally expensive. The “AUTO” option offers a more rigorous implementation of RMG’s rate-based mechanism generation algorithm while allowing straightforward generation of reaction mechanisms that encompass chemistry at all time-scales of interest.
MOTIVATION
Originally, the implementation of RMG’s reaction mechanism generation algorithm requires the user to specify times or conversions at which the program will estimate “edge” reaction fluxes and decide whether to include a new species in the “core” reaction model. The choice of which conversions or times to specify is, to a certain extent, arbitrary, and could have a significant impact on the model generated by the program. This arbitrariness can be effectively eliminated by keeping track of the “edge” reaction flux “continuously” within the ODE solver (analogously to how conversion is handled) and terminating ODE solver “integration” when the threshold reaction flux is reached (whereupon a species would be moved from the “edge” to the “core” as in the current RMG implementation). This new method is here referred to as “automated time stepping”. Implementation of such a method would reduce the number of adjustable parameters that the user must tweak during mechanism generation and provide a more rigorous use of the mechanism generation algorithm. At the same time, such a method would allow straightforward generation of mechanisms that encompass chemistry at all time-scales of interest. 
BACKGROUND 
The reaction mechanism is divided into a “core” and an “edge”. The “core” is tracked by integrating the appropriate equations using an ordinary differential equation (ODE) solver. The “edge” includes species that are products of reactions among the “core” species. Periodically during the course of simulating species concentrations using the ODE solver, mechanism validity is tested. Mechanism validity criteria used by RMG are shown in Equation 1, below. (**add these equations**)
 
In these criteria, tol is a tolerance specified by the user. If any of these criteria are not valid, the edge species with the highest flux is added to the mechanism “core” and the ODE solver restarts simulation from t = 0. In the original implementation, the times or conversions at which model validity is tested are specified by the user.
The ODE solvers used by RMG are FORTRAN programs (DASSL or DASPK), while RMG itself is written in Java.
IMPLEMENTATION
In order to achieve the desired behavior for “automated time stepping”, modifications of the type illustrated in Figure 1 were desired.
 
Figure 1. Modifications to implement “automated time stepping”.
As the figure shows, a significant required change was the modification of the ODE solver to have access to edge species flux information at each ODE solver time step. Thus, much of the modification effort involved implementing a scheme for passing edge reaction information to the ODE solver. The implementation ended up requiring changes to the Java classes ReactionModelGenerator and JDASSL and the FORTRAN file call_dassl.f (the modified version was renamed to call_dasslAUTO.f90 to distinguish from the original version). Also, a minor change to the Java class ReactionTimeTT was made.
An example of an ODE solver input file for “automated time stepping” is shown in Figure 2, below. Differences from “normal” operation are highlighted with boxes.
 
Figure 2. ODE solver input file example for “automated time stepping”. Boxes highlight differences from the “normal” version.

The first box in the above figure highlights an integer flag signaling to the FORTRAN code that “automated time stepping” is desired (as opposed to “normal” operation”). The next box highlights the section containing the information needed by the ODE solver to check mechanism validity at each step. In particular, this includes the user-specified tolerance, the number of edge species/reactions, and a line for each edge reaction containing the number of reactants, the number of products, ID numbers corresponding to reactants and products, and finally the rate coefficient, k, for the reaction. It should be noted that in the current implementation, the rate coefficient is considered to be a constant, as additional information regarding temperature/pressure dependence is not passed to solver. Thus, the “automated time stepping” code (along with other preexisting DASSL code) would need to be modified to handle non-isothermal cases (as well as non-isobaric pressure-dependent cases).
In order to reproduce existing validity testing for cases with pressure-dependence, special considerations are necessary. In particular, each pressure-dependent network is considered as a “pseudo-species” with an associated “pseudo-reaction” using the kleak associated with the network as the rate coefficient. Monitoring flux to these “pseudo-species” thus allows us to consider whether a pressure-dependent network needs to be enlarged, analogously to how monitoring of species flux allows us to consider whether a species needs to be added to the “core” reaction mechanism.

USAGE NOTES
The modified code still allows for evaluating flux at user-defined time/conversion points, and the use of “automated time stepping” is invoked by replacing the user-defined time/conversion points with the keyword “AUTO”. Thus, the use of “automated time stepping” will be referred to as the “AUTO” method in subsequent discussion. The modified code has been designed to work for both pressure-dependent and non-pressure-dependent cases; also, the code allows automatic stepping with either a conversion or a reaction time as the user-specified termination criterion.

RESULTS
The new option of “AUTO” time-stepping was tested for several cases to verify that the option was working as desired and to investigate the effects on mechanism generation time and on the resulting mechanism. Note: Throughout this section, “1,3-hexadiene” refers to a system involving methane doped with 1,3-hexadiene (in particular, the system defined by the condition files distributed with the March 2007 release of RMG). A brief summary of findings follows.
Effects on Generated Mechanism
One would expect that mechanisms generated using “AUTO” method would be more rigorously valid than those generated using arbitrary user-specified time/conversion steps. In the case of the 1,3-hexadiene system discussed above, mechanisms generated using the “AUTO” method were compared to mechanisms generated using evenly-spaced conversion steps. The results are shown in Table 1, below.
Tests suggested that we approach “AUTO” method behavior as we specify a finer “grid” for conversion steps, as one might intuitively expect.
It is also interesting to note that in the test case considered, the "AUTO" method produced a smaller mechanism than cases with a small number of specified conversion steps. Thus, in this case, the mechanism generated by the “AUTO” method (and cases with a sufficiently fine grid of conversion steps) would not be expected to be more accurate than the mechanism generated without any conversion steps (assuming the kinetics and thermochemistry of the extra species and reactions is accurate). However, the time required to generate the “AUTO” mechanism is significantly shorter, due to the smaller size of the mechanism. Furthermore, one can imagine that mechanisms of similar size to the largest mechanism reported above could be generated by using tighter tolerances and the “AUTO” method, and one could be more confident in their validity. Thus, even in this case, the use of a low number of conversion steps is not necessarily better than the use of a high number of conversion steps or the “AUTO” method.
Timing Studies
Tests suggest that the time required to generate a mechanism with the “AUTO” method is virtually the same as the time required to generate the same mechanism with conventional use of conversion steps. Table 3 shows, however, that the computational cost associated with tracking edge-species is significant compared to the cost of solving the ODEs, with the “AUTO” cases requiring a longer amount of time by a factor of two to eight (for the same number of ODE time steps). Three phenomena are proposed to explain this “discrepancy”. First, one would expect that in the “AUTO” case, the total time that we must integrate over during mechanism generation will be shorter, in general; that is, the “AUTO” method avoids “wasted” ODE time steps associated with “overshooting” the point where threshold flux has been reached. Second, the time associated with ODE solving is small in comparison to the total time requirement (< 10% of the total time for mechanism generation). Third, and perhaps most significant, there are fewer calls to the ODE solver when the “AUTO” method is used. In the case of the “AUTO” method, each call to the ODE solver is associated with a modification to the reaction mechanism (with the exception of the final integration). However, in the case of the “normal” method, there are extra calls to the ODE solver, as the integration is often stopped despite the mechanism being valid (i.e. no edge species fluxes exceed the threshold). Thus, in order to generate the same mechanism, the “normal” method requires a larger number of calls to the ODE solver, which, in turn is associated with greater overhead such as reading and writing ODE solver input and output files. Values for T1 and T2 in Table 2 for 1,3-hexadiene suggest that this overhead can be significant. (Note that the use of eighteen rather than nine conversions exacerbates the time cost, in accord with expectations based on the aforementioned overhead considerations.) The contribution of this overhead to FORTRAN time requirements is relatively small, but noticeable, while the contribution from the overhead on the Java side appears to be more significant. In fact, in this case, the total ODE solver time-requirement (T1) is noticeably lower in the “AUTO” case (relative to the “normal” cases). Thus, in this case, it appears that the added overhead associated with extra calls to the ODE solver in the “normal” method has outweighed the extra cost associated with tracking the flux at each time step within the ODE solver, using the “AUTO” method. It should be emphasized, however, that this phenomenon of faster “AUTO” operation has only been observed in the one case extensively studied thus far (in which we have been able to obtain the same mechanism using both “AUTO” and “normal” methods), and may not extend to other systems or situations with larger mechanisms. Further study is needed to determine if this behavior is typical.

CONCLUSIONS
The “AUTO” method was implemented and tested in RMG and was determined to function as intended for “automated time stepping”. Even so, timing studies suggest that tracking the edge flux “continuously” in the ODE solver is associated with a significant time penalty. However, various mitigating factors appear to reduce the impact of this time penalty on overall mechanism generation time when the “AUTO” method is used. Thus, the “AUTO” method appears to be an appealing alternative to user-specification of time/conversion steps, suggesting that it holds promise for routine use in mechanism generation with RMG.


{'fraction_non_alphanumeric': 0.040964013228985886, 'fraction_numerical': 0.013916631192900882, 'mean_word_length': 4.850574712643678, 'pattern_counts': {'":': 0, '<': 7, '<?xml version=': 0, '>': 5, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '4869081', 'n_tokens_mistral': 7215, 'n_tokens_neox': 6720, 'n_words': 4720}