<commit_before>---
layout: article
title: HM-1X Aid
categories: robots
excerpt:
tags: [robots]
color: "#152a55"
image:
  feature: HM-10-2.jpg
  teaser: HM-10-2.jpg
  thumb:
comments: true
---

###Overview

Well, I've had the urge to hack.  It's been strong for awhile now, sadly, with a more than fullt-time job and Bek in graduate school, I've just had no time.  **Until now!**  The new job I have (HMIS Database Manager) has _actual_ vacation time.  I've had almost two weeks off (combined with the Christmas holiday).  The first few days were obviously spent working...**But!** After turning my phone and email (the work email) I was able to...catch up on family time.  Sigh.  Then, clean house.  Then, get bored.  But with a few days left I actually got some hacking in.

I downloaded Visual Studio Express on my work computer a few months ago.  (Shh.  Don't tell the boss.) But I've not had time to write a single line of code.  This holiday was a good time to learn C\#!

Two weeks later I had this monster,

<div class="flex-video">
<iframe width="420" height="315" src="https://www.youtube.com/embed/XtrS7ENc1EU" frameborder="0" allowfullscreen></iframe>
</div>

It is meant to be a GUI for the HM-10, HM-11, and HM-15 modules.  The highlights,

[HM-1X Aid Download](https://github.com/Ladvien/HM-1X-Build/raw/master/setup.exe)

The source can be found,

[HM-1X Aid Source](https://github.com/Ladvien/HM-1X_Aid_v01)

1. It uses threads to prevent any "Sleeping."  Keeps the UI healthy.
2. IO.SerialPorts _should_ allow a lot of USB-to-UART chips to work.
3. Basic Terminal App features (view as ASCII, HEC, Decimal, etc.) like RealTerm and the others. BUT! With one feature I've longed for in a terminal app for so damn long.  The ability to save your settings on exit.  No more selecting 9600 baud for the billionth time.
4. I've put a lot of command validity checks in the system.  For example, if you were to type "AT+CON0123**S**6789012" Would not be a valid command normally, but terminal will convert the "S" to "F."
5. I have also imbued it with a bit my understanding of the HM-1X datasheets.  This is probably the greatest selling point of the program, sadly.
6. **C# is my new favorite.  Don't judge me :P.**

I thought I would take some time to go into the code involved in the little terminal, not for posterity, merely for my own reference.

**"Attention .NET Users! This is the law. Disperse immediately, or we will use lethal force to clear the area. You have been warned. You now have 20 seconds to comply..."**


![](http://www.geek.com/wp-content/uploads/2015/09/stallone-judge-dredd-590x332.jpg)


### Object Oriented Programming

I have post as much lately.  I think it is a combination of losing LMR and not having time since Bek started school.  But I have definitely been writing code.  The focus has been on learning OOP design.  This this C\# program was probably my first _real_ object-oriented program.  But I have also been writing in [Swift](https://developer.apple.com/ios/?cid=wwa-us-kwg-features) for iOS, which is an OOP language as well.

Arlight, so what's the difference between OOP and the other thing? And why learn OOP, I thought robot used microcontollers whichi are much too small to handle OOP?

Well, as I've learned, every robot builder is _already_ an object oriented programmer.

#### Difference between OOP and Procedural programming

I will not attempt to explain more poorly what I've learned, rather, I'm going to list what has helped me get a better understanding.

1. [Object-Oriented programming](https://www.youtube.com/watch?v=lbXsrHGhBAU) (video)
2. [Procedural Programming](https://www.youtube.com/watch?v=b4n9KYD05jg) (video)
3. [<NAME>' Design Patterns Series](https://www.youtube.com/watch?v=vNHpsC5ng_E&list=PLF206E906175C7E07) (examples in Java, but it helped me the most.  Got to see OOP in action.)

I believe every roboticist is both an object oriented programmer and a procedural programmer.  They create objects with specific functions (PCBs, STLs, etc.), retaining the plans to make copies of such objects in the future.  They instantiate those objects (print the STL, send the PCB file to OSHPark, etc).  Each of these objects created usually has a specific purpose.  The design of the object, for instance a motor-controller, is often only accessible by a specific input such as UART connection.  And its outputs are controlled by how the inputs are digested by the motor-controller itself. The robot-builder may decide to add an accelerometer to the motor-controller by copying the design files and adding the needed accelerometer circuit (inheritance).  It seems like all the concepts are there,

1. [Abstraction](https://en.wikipedia.org/wiki/Abstraction_%28computer_science%29)
2. [Encapsulation](https://en.wikipedia.org/wiki/Encapsulation_%28computer_programming%29)
3. [Polymorhphism](https://en.wikipedia.org/wiki/Polymorphic_code)
4. [Inheritance](https://en.wikipedia.org/wiki/Inheritance_%28object-oriented_programming%29)

However, each roboticist must line the insides of objects with procedures that move data through in a predictable manner.  This means, at some granularity point, objects are filled with small runs of procedural programming (I realize Procedural Programming and programming procedurally are not the same, but hey, it's my article and I'm trying to make a point).

Moving away from the hippy-dippy stuff; for my personal development as a robot builder I will be taking time to learn both, since both seem roots of computational-thinking.

In application, I want to know procedural programming since microcontrollers often have small memory sets and the code needs to move from input to output with little overhead (Hmm, procedural programming is a linear style and a line is the shortest distance between two points, there's gotta be something there).  But I also want to know how to connect my robot to large systems, such as a Raspberry Pi, PC, or the whooooole Internet.  And to it do it effectively I need to be able to pass data between procedural and object programs.

### Avoiding Sleep

<a href="http://gulpjs.com">
  <img height="220" width="320" style="float: left; margin: 10px;" src="http://www.frugal-cafe.com/public_html/frugal-blog/frugal-cafe-blogzone/wp-content/uploads/2010/11/coffee-smiley-face-ballistik-coffee-boy-flickr.jpg">
</a>

One bit I noticed writing in OOP is it usually takes full advantage of [threading](https://en.wikipedia.org/wiki/Thread_%28computing%29).  Most microcontrollers have one core (yes, yes, except the Parrallax), so threading is not something done often on a microcontroller.  But my program is walking between the microcontroller world and the big-boy-PC world.  As I stated above, at some point the HM-1X module, more specifically the CC254X the Texas Instrument chip at the heart of the HM-1X, would need to pass its data to the PC.  

Here in lies a dilemma best explained by a picture,

<a href="http://gulpjs.com">
  <img height="220" width="320" style="float: left; margin: 10px;" src="/images/threadingCars.png">
</a>

For the HM-1X Aid the data is passed through the Serial connection.  Of course, as I stated above, I am using the Systems.IO.Ports.SerialPorts framework to handle the incoming serial data.  A nifty little aspect of IO.Ports is it actually has the [DataReceivedEvent](https://msdn.microsoft.com/en-us/library/system.io.ports.serialport.datareceived%28v=vs.110%29.aspx) on a separate thread.  This event is triggered by an RX interrupt on whatever USB-to-UART chip the IO.Ports is working with.  This allows data to be handled as it is coming in.  Excellent.  (Coincdentally, iOS handles Bluetooth in the CoreBluetooth with an interrupt, but it will only be on a separate thread if setup as such.)

Now, I mentioned at some point methods for handling data probably go from procedural to object-oriented when moving upstream from a microcontroller to a PC.  A USB-to-UART bridge is a perfect example.  Receiving UART data for a microntroller looks like this,

{% highlight c %}
int incomingByte = 0;   // for incoming serial data

void setup() {
        Serial.begin(9600);     // opens serial port, sets data rate to 9600 bps
}

void loop() {

        // send data only when you receive data:
        if (Serial.available() > 0) {
                // read the incoming byte:
                incomingByte = Serial.read();

                // say what you got:
                Serial.print("I received: ");
                Serial.println(incomingByte, DEC);
        }
}
{% endhighlight %}

Basically, the microcontroller is looping over a check to see if there is any data available in the RX buffer.  Whenever the `loop()` finds there are available data, then it will run through some procedures on handling those data.  This method works perfectly fine, if all your microcontroller is doing is serial data centric.  But what happens if your microcontroller is has a never-ending stream of data?  Whatever code is after the `if(Serial.available() > 0)` will never execute, since the available serial data will never be less than 0.  This is referred to as using a "blocking" serial data method.  I'm thinking "blocking" means it is preventing the code from doing anything else


Hmm, wouldn't be better to handle serial data "immediately" when it comes in?  That's where interrupts come in.  

The `Serial.onReceive()` is an interrupt vector which fires every time the serial data comes in.  It then stores it then copies the data received from the interrupt buffer into the `string_buffer`.

{% highlight c %}
void MyFunction(){
   int i;
   int length = Serial.available();
   int string_buffer[32];   

   //copy data out of the receive buffer
   for(i = 0; i < length; i++){
      string_buffer[i] = Serial.read();
   }

   //run a string compare function, or otherwise parse the received data
   if(MySpecialStingCompare(string_buffer,"Hello Arduino")){
      Serial.println("Hello World");
   }
}

void setup(){
   Serial.begin(9600);
   Serial.onReceive(MyFunction);
}

void loop(){
   //do nothing
}

{% endhighlight %}

This is a non-blocking method of handling serial data.  It takes advantage of a hardware level peripheral on the Atmel chips known as a [interrupt vector](https://en.wikipedia.org/wiki/Interrupt_vector_table).  This particular interrupt is fired any time the RX pin receives a series of LOWs which resemble incoming data.  

This method has the advantage of freeing the main processor to do other things in the main loop and handle data _only_ when new data is received.  This saves us from having to make a comparison every clock cycle.  Also, and more importantly, it allows the microcontroller to immediately update data important to the purpose of the main process; this is critical in processes which are time sensitive, such as remote control on a quadcopter.

For example, if we wrote a radio controller for our quadcopter using a Bluetooth device which talks to an Arduino Pro Mini.  If we used the blocking method the Arduino receives any data from the radio controller, like, oh I don't know, "Don't run into that tree!" the main process on the Arduino would not have this information until it gets back to the `Serial.available() > 0`.  Not cool, if milliseconds matter.

However, if you were to write the same radio controller using non-blocking, interrupt based serial communication, then whenever you send the signal to the Bluetooth device, and that device sends it serially to the Arduino, the Arduino will basically bookmark its place in the main process and handle any data regarding its immediate crash.

Good stuff.

Alright, so that's how a microcontroller handles things procedurally--but what does this have to do with my C# program? I dunno, I'm DATKing this article.

Let's take a look at C\#, though.

{% highlight c# %}
string InputData = "";

// Read Data.
private void DataReceivedHandler(object sender, SerialDataReceivedEventArgs e)
{
    // Gets incoming data from open port and passes it to a handler.
    InputData = ComPort.ReadExisting();
    captureBuffer = InputData;
    captureBuffer = "";

}

{% endhighlight %}

This is probably the simplest version of a `DataReceivedHandler()` using C\#.  In short, it fires whenever data is received; so it's non-blocking.  The method as I have it written simply grabs the data from the interrupt buffer using `ComPor.ReadExisting()` and shoves it into the `inputData`.  Pretty simple, right?  And it is, which is one reason I think so many [poo-poo](http://www.sparxeng.com/blog/software/must-use-net-system-io-ports-serialport) using IO.Ports.SerialPort framework.  But how is different than the Arduino HAL `Serial.onReceive()`?  The main difference is the the C\# is actually on a separate thread from the main thread.

Now, I'm going to dare to oversimplify what little understand regarding threads.  For this article, I would like to think of a thread as a separate process.  Of course, we can't call the second main process, "main process".  Computers don't like ambiguity.  Instead, let's get creative and call the second process something wild, fun, and bizarre! Like..."second process".

Since we have cleared up definitions let's talk about how this is different than the Arduino HAL.  In C\# the second process is pretty much like the main process.  There is a lot of complexity to it, but the compiler takes care of most of it.  With one grand honking exception: The user interface (UI) runs on the main process and cannot be updated by any other process.  Instead, a protocol saving data in a space both processes can access it and a flag to look there are used.  This is the good old snail-mail mailbox, where the flag is raised when you want the post-person to look for letter.

1. Can't update UI when main process is busy.  Arduino and not updating an OLED until finished with Serial data.
2. Advantages of interrupts; received data immediately added to main process
3. Advantage of threads; off-load clock cycle intensive operations to a different process to maintain responsiveness of UI and other main process fun.


One last note, I started porting this to Mac and Linux using MonoDevelopment, but it would require writing my own interrupt handler.  Or faking it with a timeout method.  Nether option is worth it to me.  Too many other projects and too little time.
<commit_msg>Writing<commit_after>---
layout: article
title: HM-1X Aid
categories: robots
excerpt:
tags: [robots]
color: "#152a55"
image:
  feature: HM-10-2.jpg
  teaser: HM-10-2.jpg
  thumb:
comments: true
---

###Overview

Well, I've had the urge to hack.  It's been strong for awhile now, sadly, with a more than fullt-time job and Bek in graduate school, I've just had no time.  **Until now!**  The new job I have (HMIS Database Manager) has _actual_ vacation time.  I've had almost two weeks off (combined with the Christmas holiday).  The first few days were obviously spent working...**But!** After turning my phone and email (the work email) I was able to...catch up on family time.  Sigh.  Then, clean house.  Then, get bored.  But with a few days left I actually got some hacking in.

I downloaded Visual Studio Express on my work computer a few months ago.  (Shh.  Don't tell the boss.) But I've not had time to write a single line of code.  This holiday was a good time to learn C\#!

Two weeks later I had this monster,

<div class="flex-video">
<iframe width="420" height="315" src="https://www.youtube.com/embed/XtrS7ENc1EU" frameborder="0" allowfullscreen></iframe>
</div>

It is meant to be a GUI for the HM-10, HM-11, and HM-15 modules.  The highlights,

[HM-1X Aid Download](https://github.com/Ladvien/HM-1X-Build/raw/master/setup.exe)

The source can be found,

[HM-1X Aid Source](https://github.com/Ladvien/HM-1X_Aid_v01)

1. It uses threads to prevent any "Sleeping."  Keeps the UI healthy.
2. IO.SerialPorts _should_ allow a lot of USB-to-UART chips to work.
3. Basic Terminal App features (view as ASCII, HEC, Decimal, etc.) like RealTerm and the others. BUT! With one feature I've longed for in a terminal app for so damn long.  The ability to save your settings on exit.  No more selecting 9600 baud for the billionth time.
4. I've put a lot of command validity checks in the system.  For example, if you were to type "AT+CON0123**S**6789012" Would not be a valid command normally, but terminal will convert the "S" to "F."
5. I have also imbued it with a bit my understanding of the HM-1X datasheets.  This is probably the greatest selling point of the program, sadly.
6. **C# is my new favorite.  Don't judge me :P.**

I thought I would take some time to go into the code involved in the little terminal, not for posterity, merely for my own reference.

**"Attention .NET Users! This is the law. Disperse immediately, or we will use lethal force to clear the area. You have been warned. You now have 20 seconds to comply..."**


![](http://www.geek.com/wp-content/uploads/2015/09/stallone-judge-dredd-590x332.jpg)


### Object Oriented Programming

I have post as much lately.  I think it is a combination of losing LMR and not having time since Bek started school.  But I have definitely been writing code.  The focus has been on learning OOP design.  This this C\# program was probably my first _real_ object-oriented program.  But I have also been writing in [Swift](https://developer.apple.com/ios/?cid=wwa-us-kwg-features) for iOS, which is an OOP language as well.

Arlight, so what's the difference between OOP and the other thing? And why learn OOP, I thought robot used microcontollers whichi are much too small to handle OOP?

Well, as I've learned, every robot builder is _already_ an object oriented programmer.

#### Difference between OOP and Procedural programming

I will not attempt to explain more poorly what I've learned, rather, I'm going to list what has helped me get a better understanding.

1. [Object-Oriented programming](https://www.youtube.com/watch?v=lbXsrHGhBAU) (video)
2. [Procedural Programming](https://www.youtube.com/watch?v=b4n9KYD05jg) (video)
3. [<NAME>' Design Patterns Series](https://www.youtube.com/watch?v=vNHpsC5ng_E&list=PLF206E906175C7E07) (examples in Java, but it helped me the most.  Got to see OOP in action.)

I believe every roboticist is both an object oriented programmer and a procedural programmer.  They create objects with specific functions (PCBs, STLs, etc.), retaining the plans to make copies of such objects in the future.  They instantiate those objects (print the STL, send the PCB file to OSHPark, etc).  Each of these objects created usually has a specific purpose.  The design of the object, for instance a motor-controller, is often only accessible by a specific input such as UART connection.  And its outputs are controlled by how the inputs are digested by the motor-controller itself. The robot-builder may decide to add an accelerometer to the motor-controller by copying the design files and adding the needed accelerometer circuit (inheritance).  It seems like all the concepts are there,

1. [Abstraction](https://en.wikipedia.org/wiki/Abstraction_%28computer_science%29)
2. [Encapsulation](https://en.wikipedia.org/wiki/Encapsulation_%28computer_programming%29)
3. [Polymorhphism](https://en.wikipedia.org/wiki/Polymorphic_code)
4. [Inheritance](https://en.wikipedia.org/wiki/Inheritance_%28object-oriented_programming%29)

However, each roboticist must line the insides of objects with procedures that move data through in a predictable manner.  This means, at some granularity point, objects are filled with small runs of procedural programming (I realize Procedural Programming and programming procedurally are not the same, but hey, it's my article and I'm trying to make a point).

Moving away from the hippy-dippy stuff; for my personal development as a robot builder I will be taking time to learn both, since both seem roots of computational-thinking.

In application, I want to know procedural programming since microcontrollers often have small memory sets and the code needs to move from input to output with little overhead (Hmm, procedural programming is a linear style and a line is the shortest distance between two points, there's gotta be something there).  But I also want to know how to connect my robot to large systems, such as a Raspberry Pi, PC, or the whooooole Internet.  And to it do it effectively I need to be able to pass data between procedural and object programs.

### Avoiding Sleep

<a href="http://gulpjs.com">
  <img height="220" width="320" style="float: left; margin: 10px;" src="http://www.frugal-cafe.com/public_html/frugal-blog/frugal-cafe-blogzone/wp-content/uploads/2010/11/coffee-smiley-face-ballistik-coffee-boy-flickr.jpg">
</a>

One bit I noticed writing in OOP is it usually takes full advantage of [threading](https://en.wikipedia.org/wiki/Thread_%28computing%29).  Most microcontrollers have one core (yes, yes, except the Parrallax), so threading is not something done often on a microcontroller.  But my program is walking between the microcontroller world and the big-boy-PC world.  As I stated above, at some point the HM-1X module, more specifically the CC254X the Texas Instrument chip at the heart of the HM-1X, would need to pass its data to the PC.  

Here in lies a dilemma best explained by a picture,

<img height="277" width="640" style="margin: 10px;" src="/images/threadingCars.png">

For the HM-1X Aid the data is passed through the Serial connection.  Of course, as I stated above, I am using the Systems.IO.Ports.SerialPorts framework to handle the incoming serial data.  A nifty little aspect of IO.Ports is it actually has the [DataReceivedEvent](https://msdn.microsoft.com/en-us/library/system.io.ports.serialport.datareceived%28v=vs.110%29.aspx) on a separate thread.  This event is triggered by an RX interrupt on whatever USB-to-UART chip the IO.Ports is working with.  This allows data to be handled as it is coming in.  Excellent.  (Coincdentally, iOS handles Bluetooth in the CoreBluetooth with an interrupt, but it will only be on a separate thread if setup as such.)

Now, I mentioned at some point methods for handling data probably go from procedural to object-oriented when moving upstream from a microcontroller to a PC.  A USB-to-UART bridge is a perfect example.  Receiving UART data for a microntroller looks like this,

{% highlight c %}
int incomingByte = 0;   // for incoming serial data

void setup() {
        Serial.begin(9600);     // opens serial port, sets data rate to 9600 bps
}

void loop() {

        // send data only when you receive data:
        if (Serial.available() > 0) {
                // read the incoming byte:
                incomingByte = Serial.read();

                // say what you got:
                Serial.print("I received: ");
                Serial.println(incomingByte, DEC);
        }
}
{% endhighlight %}

Basically, the microcontroller is looping over a check to see if there is any data available in the RX buffer.  Whenever the `loop()` finds there are available data, then it will run through some procedures on handling those data.  This method works perfectly fine, if all your microcontroller is doing is serial data centric.  But what happens if your microcontroller is has a never-ending stream of data?  Whatever code is after the `if(Serial.available() > 0)` will never execute, since the available serial data will never be less than 0.  This is referred to as using a "blocking" serial data method.  I'm thinking "blocking" means it is preventing the code from doing anything else


Hmm, wouldn't be better to handle serial data "immediately" when it comes in?  That's where interrupts come in.  

The `Serial.onReceive()` is an interrupt vector which fires every time the serial data comes in.  It then stores it then copies the data received from the interrupt buffer into the `string_buffer`.

{% highlight c %}
void MyFunction(){
   int i;
   int length = Serial.available();
   int string_buffer[32];   

   //copy data out of the receive buffer
   for(i = 0; i < length; i++){
      string_buffer[i] = Serial.read();
   }

   //run a string compare function, or otherwise parse the received data
   if(MySpecialStingCompare(string_buffer,"Hello Arduino")){
      Serial.println("Hello World");
   }
}

void setup(){
   Serial.begin(9600);
   Serial.onReceive(MyFunction);
}

void loop(){
   //do nothing
}

{% endhighlight %}

This is a non-blocking method of handling serial data.  It takes advantage of a hardware level peripheral on the Atmel chips known as a [interrupt vector](https://en.wikipedia.org/wiki/Interrupt_vector_table).  This particular interrupt is fired any time the RX pin receives a series of LOWs which resemble incoming data.  

This method has the advantage of freeing the main processor to do other things in the main loop and handle data _only_ when new data is received.  This saves us from having to make a comparison every clock cycle.  Also, and more importantly, it allows the microcontroller to immediately update data important to the purpose of the main process; this is critical in processes which are time sensitive, such as remote control on a quadcopter.

For example, if we wrote a radio controller for our quadcopter using a Bluetooth device which talks to an Arduino Pro Mini.  If we used the blocking method the Arduino receives any data from the radio controller, like, oh I don't know, "Don't run into that tree!" the main process on the Arduino would not have this information until it gets back to the `Serial.available() > 0`.  Not cool, if milliseconds matter.

However, if you were to write the same radio controller using non-blocking, interrupt based serial communication, then whenever you send the signal to the Bluetooth device, and that device sends it serially to the Arduino, the Arduino will basically bookmark its place in the main process and handle any data regarding its immediate crash.

Good stuff.

Alright, so that's how a microcontroller handles things procedurally--but what does this have to do with my C# program? I dunno, I'm DATKing this article.

Let's take a look at C\#, though.

{% highlight c# %}
string InputData = "";

// Read Data.
private void DataReceivedHandler(object sender, SerialDataReceivedEventArgs e)
{
    // Gets incoming data from open port and passes it to a handler.
    InputData = ComPort.ReadExisting();
    captureBuffer = InputData;
    captureBuffer = "";

}

{% endhighlight %}

This is probably the simplest version of a `DataReceivedHandler()` using C\#.  In short, it fires whenever data is received; so it's non-blocking.  The method as I have it written simply grabs the data from the interrupt buffer using `ComPor.ReadExisting()` and shoves it into the `inputData`.  Pretty simple, right?  And it is, which is one reason I think so many [poo-poo](http://www.sparxeng.com/blog/software/must-use-net-system-io-ports-serialport) using IO.Ports.SerialPort framework.  But how is different than the Arduino HAL `Serial.onReceive()`?  The main difference is the the C\# is actually on a separate thread from the main thread.

Now, I'm going to dare to oversimplify what little understand regarding threads.  For this article, I would like to think of a thread as a separate process.  Of course, we can't call the second main process, "main process".  Computers don't like ambiguity.  Instead, let's get creative and call the second process something wild, fun, and bizarre! Like..."second process".

Since we have cleared up definitions let's talk about how this is different than the Arduino HAL.  In C\# the second process is pretty much like the main process.  There is a lot of complexity to it, but the compiler takes care of most of it.  With one grand honking exception: The user interface (UI) runs on the main process and cannot be updated by any other process.  Instead, a protocol saving data in a space both processes can access it and a flag to look there are used.  This is the good old snail-mail mailbox, where the flag is raised when you want the post-person to look for letter.

1. Can't update UI when main process is busy.  Arduino and not updating an OLED until finished with Serial data.
2. Advantages of interrupts; received data immediately added to main process
3. Advantage of threads; off-load clock cycle intensive operations to a different process to maintain responsiveness of UI and other main process fun.


One last note, I started porting this to Mac and Linux using MonoDevelopment, but it would require writing my own interrupt handler.  Or faking it with a timeout method.  Nether option is worth it to me.  Too many other projects and too little time.

{'fraction_non_alphanumeric': 0.07073484874856407, 'fraction_numerical': 0.011626692658474606, 'mean_word_length': 4.029411764705882, 'pattern_counts': {'":': 0, '<': 25, '<?xml version=': 0, '>': 29, 'https://': 28, 'lorem ipsum': 0, 'www.': 14, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 6, 'profanity_word_list.json': 2, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '6120646', 'n_tokens_mistral': 8200, 'n_tokens_neox': 7582, 'n_words': 4113}