<commit_before><!-- START doctoc generated TOC please keep comment here to allow auto update -->
<!-- DON'T EDIT THIS SECTION, INSTEAD RE-RUN doctoc TO UPDATE -->
#   Table of Content

- [Name](#name)
- [Status](#status)
- [Description](#description)
  - [Module List](#module-list)
- [Install](#install)
- [Usage](#usage)
- [Author](#author)
- [Copyright and License](#copyright-and-license)

<!-- END doctoc generated TOC please keep comment here to allow auto update -->

#   Name

pykit:
A collection of python libs those are used in project s2: Storage-Service at
baishancloud.com

#   Status

This library is in beta phase.

It has been used heavily in our object storage service, as a foundamental
library of our devops platform.

#   Description

There are several native modules like `utfjson`,
along with several external modules like `jobq`, which are imported with
`script/git-subrepo`(`git-subrepo` is also maintained by itself).

##  Module List

There is a `README.md` for each module.

| name               | description                                                                           |
| :--                | :--                                                                                   |
| script/git-subrepo | A shell script maintaining sub module                                                 |
| jobq               | Process serial of input elements with several functions concurrently and sequentially |
| mysqlconnpool      | Mysql connection pool with MySQLdb in python                                          |
| utfjson            | force `json.dump` and `json.load` in `utf-8` encoding                                 |

#   Install

This package does not support installation.

Just clone it and copy it into your project source folder.

```
cd your_project_folder
git clone https://github.com/baishancloud/pykit.git
```

#   Usage

```
from pykit import jobq

def add1(args):
    return args + 1

def printarg(args):
    print args

jobq.run([0, 1, 2], [add1, printarg])
# > 1
# > 2
# > 3
```

#   Author

<NAME> (张炎泼) <<EMAIL>>

#   Copyright and License

The MIT License (MIT)

Copyright (c) 2015 <NAME> (张炎泼) <<EMAIL>><commit_msg>add description about how to update sub-repo<commit_after><!-- START doctoc generated TOC please keep comment here to allow auto update -->
<!-- DON'T EDIT THIS SECTION, INSTEAD RE-RUN doctoc TO UPDATE -->
#   Table of Content

- [Name](#name)
- [Status](#status)
- [Description](#description)
  - [Module List](#module-list)
- [Install](#install)
- [Usage](#usage)
  - [Update sub repo](#update-sub-repo)
- [Author](#author)
- [Copyright and License](#copyright-and-license)

<!-- END doctoc generated TOC please keep comment here to allow auto update -->

#   Name

pykit:
A collection of python libs those are used in project s2: Storage-Service at
baishancloud.com

#   Status

This library is in beta phase.

It has been used heavily in our object storage service, as a foundamental
library of our devops platform.

#   Description

There are several native modules like `utfjson`,
along with several external modules like `jobq`, which are imported with
`script/git-subrepo`(`git-subrepo` is also maintained by itself).

##  Module List

There is a `README.md` for each module.

| name               | description                                                                           |
| :--                | :--                                                                                   |
| script/git-subrepo | A shell script maintaining sub module                                                 |
| jobq               | Process serial of input elements with several functions concurrently and sequentially |
| mysqlconnpool      | Mysql connection pool with MySQLdb in python                                          |
| utfjson            | force `json.dump` and `json.load` in `utf-8` encoding                                 |

#   Install

This package does not support installation.

Just clone it and copy it into your project source folder.

```
cd your_project_folder
git clone https://github.com/baishancloud/pykit.git
```

#   Usage

```
from pykit import jobq

def add1(args):
    return args + 1

def printarg(args):
    print args

jobq.run([0, 1, 2], [add1, printarg])
# > 1
# > 2
# > 3
```

##  Update sub repo

>   You do not need to read this chapter if you are not a maintainer.

First update sub repo config file `.gitsubrepo`
and run `git-subrepo`.

`git-subrepo` will fetch new changes from all sub repos and merge them into
current branch `mater`:

```
./script/git-subrepo/git-subrepo
```

`git-subrepo` is a tool in shell script.
It merges sub git repo into the parent git working dir with `git-submerge`.

#   Author

<NAME> (张炎泼) <<EMAIL>>

#   Copyright and License

The MIT License (MIT)

Copyright (c) 2015 <NAME> (张炎泼) <<EMAIL>>
{'fraction_non_alphanumeric': 0.0979020979020979, 'fraction_numerical': 0.006170300287947347, 'mean_word_length': 2.159844054580897, 'pattern_counts': {'":': 0, '<': 21, '<?xml version=': 0, '>': 28, 'https://': 2, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 20, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '2509271', 'n_tokens_mistral': 1387, 'n_tokens_neox': 1299, 'n_words': 516}