<commit_before># ETCD Load Testing

This is a etcd load test module written in go-language. It basically creates 
artificial load on a running etcd instance. The test runs for a random set of
keys and values, which can be specified in the configuration file. You can 
configure many other parameters in the config file. See the sample config file 
*etcd_load.cfg* for more information. 


##Features
 - Memory information
  - gives the memory information about the running etcd instance -- before, 
  	after and difference, when requests are made
  -	To get this info, use the "-mem" flag while running the module
  - Also, if the instance is running on a remote machine, then you need to use
  	the "-remote" flag as well.
 - Key value distribution
  - This feature basically allows you to specify the distribution of the 
  	key-values, that is how many keys lie in a particular value range, specified
  	by "value-range" parameter under section : "section-args", in etcd_load.cfg
  - See the "pct" parameter under the section :"section-args", in etcd_load.cfg
 - Value range
  - This allows you to specify value ranges, which will be used for the -- Key
  	value distribution feature.
  - Note : length (Value Range) = length (Key Value distribution) + 1
 - There are other configuration options as well, like -- log-file, remote-flag,
 	remote-host-user, etcd. . Some of these can be specified using commandline 
 	flags as well, in which case the flags will override the cfg-file values. To
 	know more about them see the default config file -- "etcd_load.cfg" and for
 	help regarding flags, use
 		- go run etcd_load.go -h

##Setup
 - Make sure that the following packages are in your GOPATH. Set your 
   GOPATH if its not already set.
   In go, to get a "package" you can simply do : go get package. 
   In this use case do it under your GOPATH.
 - Packages :
  - "github.com/coreos/go-etcd/etcd"
  - "code.google.com/p/gcfg"
  - "code.google.com/p/go.crypto/ssh"
 - Example : 
  - go get "code.google.com/p/gcfg"
 - Set up a default config file, like the one available in the repo.
  - Below is a sample config file, for more details take a look at etcd_load.cfg
   
## Sample Config File
```
		[section-args]
		etcdhost="127.0.0.1"
		etcdport="4001"
		operation=create
		keycount="100"
		operation-count="200"
		log-file=log
		threads=5 
		pct="5,74,10,10,1"
		value-range="0,256,512,1024,8192,204800"
		remote-flag=False
		ssh-port=22
		remote-host-user=root
```

##Build
 - Now, to run etcd_load test, use the following steps
```
 $ go build etcd_load.go report.go
```

##Running the Test

 - $ ./etcd_load -c etcd_load.cfg

 - Examples :

  	[remote etcd instance]
   	```
 	$ ./etcd_load -c etcd_load.cfg -mem -remote -o create  
 	$ ./etcd_load -c etcd_load.cfg -h 10.10.10.1 -p 4001 -o create 
   	```

    [local etcd instance]
    ```
 	$ ./etcd_load -c etcd_load.cfg -h 127.0.0.1 -o create 
	```

	Note that the "-c" flag is compulsory, that is you need to have a default 
	config file that must be input using the -c flag
	To know more about the flags :: do -- go run etcd_load.go -h

<commit_msg>Readme Update<commit_after># ETCD Load Testing

This is a etcd load test module written in go-language. It basically creates 
artificial load on a running etcd instance. The test runs for a random set of
keys and values, which can be specified in the configuration file. You can 
configure many other parameters in the config file. See the sample config file 
*etcd_load.cfg* for more information. 


##Features
 - Memory information
  - gives the memory information about the running etcd instance -- before, 
  	after and difference, when requests are made
  -	To get this info, use the "-mem" flag while running the module
  - Also, if the instance is running on a remote machine, then you need to use
  	the "-remote" flag as well.
 - Key value distribution
  - This feature basically allows you to specify the distribution of the 
  	key-values, that is how many keys lie in a particular value range, specified
  	by "value-range" parameter under section : "section-args", in etcd_load.cfg
  - See the "pct" parameter under the section :"section-args", in etcd_load.cfg
 - Value range
  - This allows you to specify value ranges, which will be used for the -- Key
  	value distribution feature.
  - Note : length (Value Range) = length (Key Value distribution) + 1
 - There are other configuration options as well, like -- log-file, remote-flag,
 	remote-host-user, etcd. . Some of these can be specified using commandline 
 	flags as well, in which case the flags will override the cfg-file values. To
 	know more about them see the default config file -- "etcd_load.cfg" and for
 	help regarding flags, use
 		- go run etcd_load.go -h

##Setup
 - Make sure that the following packages are in your GOPATH. Set your 
   GOPATH if its not already set.
   In go, to get a "package" you can simply do : go get package. 
   In this use case do it under your GOPATH.
 - Packages :
  - "github.com/coreos/go-etcd/etcd"
  - "code.google.com/p/gcfg"
  - "code.google.com/p/go.crypto/ssh"
 - Example : 
  - go get "code.google.com/p/gcfg"
 - Set up a default config file, like the one available in the repo.
  - Below is a sample config file, for more details take a look at etcd_load.cfg
   
## Sample Config File
```
		[section-args]
		etcdhost="127.0.0.1"
		etcdport="4001"
		operation=create
		keycount="100"
		operation-count="200"
		log-file=log
		threads=5 
		pct="5,74,10,10,1"
		value-range="0,256,512,1024,8192,204800"
		remote-flag=False
		ssh-port=22
		remote-host-user=root
```

##Build
 - Now, to run etcd_load test, use the following steps
```
 $ go build etcd_load.go report.go
```

##Running the Test

 - $ ./etcd_load -c config_file -flag
  - flag : -help , -h , -p , -o , -k , -oc , -log , -mem , -remote
  - For more information, do :
   - $ go run etcd_load.go -help

 - Examples :

  	[remote etcd instance]
   	```
 	$ ./etcd_load -c etcd_load.cfg -mem -remote -o create  
 	$ ./etcd_load -c etcd_load.cfg -h 10.10.10.1 -p 4001 -o create 
   	```

    [local etcd instance]
    ```
 	$ ./etcd_load -c etcd_load.cfg -h 127.0.0.1 -o create 
	```

	Note that the "-c" flag is compulsory, that is you need to have a default 
	config file that must be input using the -c flag
	To know more about the flags :: do -- go run etcd_load.go -h


{'fraction_non_alphanumeric': 0.09260712603986815, 'fraction_numerical': 0.0207188824360383, 'mean_word_length': 3.022727272727273, 'pattern_counts': {'":': 0, '<': 3, '<?xml version=': 0, '>': 3, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 6, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '841742', 'n_tokens_mistral': 2178, 'n_tokens_neox': 2002, 'n_words': 858}