<commit_before>TODO
------------------------------------------
- [ ] add tile type (regular,stone,rocks,water,sand)
- [ ] fill registry with prices
- [ ] save current time
- [ ] load - load current time and adjust all queues (tick with timeUnits)
- [ ] serialize state of the registries (basically only task registry)
- [ ] remove building after collected when in manual mode
- [ ] include error handler which is gathering error message and codes
- [ ] buy locked area
- [ ] handle area states for every island -> which ones are still locked
- [ ] path finding- but where would be the starting point?
- [ ] do workers jsut walk around idle?

30.07.2015
- [x] write out rewards when a task has finished
- [x] task system (?)
- [x] Include logging
- [x] add world level system based on experience
- [x] tasks are based on world level

27.07.2015
- [x] start -> if CM_MANUAL check if it is not done (pending)
- [x] check that every ID (sign / first col) in CSV file is unique
- [x] extract registries into individual files
- [x] registry - show line number and proper error message

24.07.2015
- [x] load - check if there are all required regular josb running
- [x] registry - verify foreign keys (mainly sign) - report error
- [x] price registry - extract prices into individual files like HT.txt
- [x] convert all txt files to the new named csv format
- [x] every island is split into areas which are visible or not and user needs to buy areas
- [x] add hidden attribute to tiles -> TS_LOCKED bit
- [x] read area definition from file containig tiles
- [x] world -> load islands.txt and process this file
- [x] build island map (1 Home base and 1 Hut and x tress and y fruits spread randomly)
- [x] island - define supported buildings / resources
- [x] island define size (make Tiles size dynamic)<commit_msg>adding more TODOs<commit_after>TODO
------------------------------------------
- [ ] add tile type (regular,stone,rocks,water,sand)
- [ ] fill registry with prices
- [ ] load - load current time and adjust all queues (tick with timeUnits)
- [ ] remove building after collected when in manual mode
- [ ] include error handler which is gathering error message and codes
- [ ] buy locked area
- [ ] handle area states for every island -> which ones are still locked
- [ ] path finding- but where would be the starting point?
- [ ] do workers jsut walk around idle?
- [ ] BUG - after loading - regular tasks disappear when finished

04.08.2015
- [x] save current time
- [x] serialize state of the registries (basically only task registry)

30.07.2015
- [x] write out rewards when a task has finished
- [x] task system (?)
- [x] Include logging
- [x] add world level system based on experience
- [x] tasks are based on world level

27.07.2015
- [x] start -> if CM_MANUAL check if it is not done (pending)
- [x] check that every ID (sign / first col) in CSV file is unique
- [x] extract registries into individual files
- [x] registry - show line number and proper error message

24.07.2015
- [x] load - check if there are all required regular josb running
- [x] registry - verify foreign keys (mainly sign) - report error
- [x] price registry - extract prices into individual files like HT.txt
- [x] convert all txt files to the new named csv format
- [x] every island is split into areas which are visible or not and user needs to buy areas
- [x] add hidden attribute to tiles -> TS_LOCKED bit
- [x] read area definition from file containig tiles
- [x] world -> load islands.txt and process this file
- [x] build island map (1 Home base and 1 Hut and x tress and y fruits spread randomly)
- [x] island - define supported buildings / resources
- [x] island define size (make Tiles size dynamic)
{'fraction_non_alphanumeric': 0.10416666666666667, 'fraction_numerical': 0.016233766233766232, 'mean_word_length': 3.186862967157418, 'pattern_counts': {'":': 0, '<': 3, '<?xml version=': 0, '>': 11, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '1800729', 'n_tokens_mistral': 1051, 'n_tokens_neox': 998, 'n_words': 544}