<commit_before>---
layout: post
title:  "Revisiting SimCity 2000 - Episode II"
author: keith
date:   2015-12-30 05:00:35
categories: Offbeat
tags: [Machine Learning, Neural Networks, Data Science, Video Games, SimCity, Simulation, Hex Editing]
---
<br>
[Go back to Episode I](/offbeat/2015/12/29/exploring_simcity_pt1.html)
<br>

## Looking Under the Hood - Continued
In this post I will delve into the details of extracting useful information from our .SC2 file.<commit_msg>New Post<commit_after>---
layout: post
title:  "Revisiting SimCity 2000 - Episode II"
author: keith
date:   2015-12-30 05:00:35
categories: Offbeat
tags: [Machine Learning, Neural Networks, Data Science, Video Games, SimCity, Simulation, Hex Editing]
---
<br>
[Go back to Episode I](/offbeat/2015/12/29/exploring_simcity_pt1.html)
<br>

## Looking Under the Hood - Continued
In this post I will delve into the details of extracting useful information from our .SC2 file.

Directly referencing David Moew's guide...
{% highlight text %}
Each segment has an 8-byte header:

Bytes 1-4: Type of segment
Byres 5-8: Number of bytes in this segment, except for this 8-byte header

The remaining bytes in each segment are data.

The data in most SimCity segments is compressed using a form of run-length
encoding.  When this is done, the data in the segment consists of a series
of chunks of two kinds.  The first kind of chunk has first byte from 1 to
127;  in this case the first byte is a count telling how many data bytes
follow.  The second kind of chunk has first byte from 129 to 255.  In this
case, if you subtract 127 from the first byte, you get a count telling how
many times the following single data byte is repeated.  Chunks with first
byte 0 or 128 never seem to occur.
{% endhighlight %}
{'fraction_non_alphanumeric': 0.07344632768361582, 'fraction_numerical': 0.04463276836158192, 'mean_word_length': 4.133333333333334, 'pattern_counts': {'":': 0, '<': 7, '<?xml version=': 0, '>': 7, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 4, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '2754476', 'n_tokens_mistral': 585, 'n_tokens_neox': 508, 'n_words': 264}