<commit_before># Objective 
The Node-DC is an open source collaboration project to develop workloads representing realistic use cases of Node.js in the data center. It started in 2016 with collaboration from the open source community, Node.js experts and companies playing important roles in Node.js’s evolution and production deployments in the data centers. 

Workloads APIs, functionalities and metrics will be used to evaluate performance of the complete software stack for Node.js, Operating Systems, containers, virtualization, network stacks as well as various data center configuration choices. These are relevant for hardware and software developers, researchers and Node.js community. One priority for this project is to update the workloads in order to keep them up with the fast moving eco-system of Node.js. 

# Contributing and Community   

## Contribution to individual projects: 
If you want to contribute code to a project, first you need to fork the project. Next step is to send a pull request (PR) for review to the repository. For small changes, you can implement at first step and for large changes, it is recommended to initially send the architecture and data flow, before investing significant time into the detailed implementation. The PR will be reviewed by the project team members. You need one "Looks Good To Me (LGTM)" and no objections from the project members. Once you have gained the required approval, the project maintainers will merge the PR.

## Contribution to use cases: 
We are constantly evaluating the relevant use cases. The list below includes the current status and is evolving based on the continuous feedback we are receiving from the community and major datacenter deployments for Node.js: 

  - Node-DC-EIS (Node.js - Data Center - Employee Information Services): Current status at v0.9 with v1.0 by Q1 2017
    
  - Node-DC-SSR (Node.js - Data Center - Server Side Rendering): Under consideration as next candidate

  - We are looking for community suggestions for other relevant use cases

Feedback can be sent either by creating issues or sending e-mail listed at https://github.com/Node-DC

# Node-DC-EIS (Node.js - Data Center - Employee Information Services) 
 
This is first in a series of upcoming workloads for modelling use cases of Node.js in Data Center (Node-DC). This workload is modelling various functionalities of Employee Information Services implemented in Node.js in Data Center.

# Open for contributions to take from v0.9 to v1.0  
 
Following changes are being worked on from v0.9 to v1.0
  - Feedback from community
  - Change from Mongoose to MongoDB driver
  - Enable DB caching in microservices 
  - Enable clustering of microservices 
  - Increase unique lastnames from 777 to at least 5000 or more 
  
Following changes are being considered as optional:
  - Options for alternate DB choices
  - Unit tests as part of PR acceptance criterion 
  - Changing client from Python to Node.js 
  - Workload phases ramp-up, measurement, and ramp-down to be able to set for a given time instead of #requests 
  - Node internal stats like events in the loop etc.
  - Reduce the size of output file by processing response time samples every n sec interval instead of post-processing at the end of complete run
  - Containerize the client, server and DB for easy testing and setup of the workload 

# Node-DC-EIS Architecture  
Node-DC-EIS follows a 3-tier model and consists of three components client, Node.js server and DB.

## Client 
Client controls various phases like ramp-up, measurement and ramp-down as well as issues requests, tracks response time and errors. At the end of run, it validates the runs and post-processes the transactions log producing final metrics and graphs.

## Node.js server 
Node.js server accepts all requests from client and responds back after retrieving employee information from the DB. Node.js cluster code is architected to scale by default. 

In Node.js server configuration file "Node-DC-EIS-cluster/config/configuration.js", parameter 'cpu_count' can be used to scale to various number of CPUs. 
  - Default value is '-1' which will use all available CPUs. 
  - Monolithic mode can be set by 'cpu_count' to 1. 

## DB 
Employee Information is stored in DB (current implementation using MongoDB). For each run, DB is populated based on configuration parameters set in the "Node-DC-EIS/blob/master/Node-DC-EIS-client/config.json".     

# Node-DC-EIS Metrics  
Node-DC-EIS produces two primary metrics:
  - Throughput ( n Node-DC-EIS requests / second ): It is calculated by "requests in measurement phase / measurement time" 
  - p99 Response Time: It is 99 percentile of response time. 
	
Report also contains detailed response time data like minimum, maximum, average and p99 (99 percentile) of response time. 
Post-processing also produces response time graph as well as other histograms like memory utilization over the run length of the workload.  

# Node-DC-EIS default and configurable options for research and testing  
Node-DC-EIS sets all parameters by default to model typical Node.js server deployment. Many important parameters have been defined in the configuration files to make it easy to be able to evaluate, test and validate wide variety of deployments. 

Most parameters can be set in the client configuration file and these are passed to Node.js server and DB. But some parameters must be set at the start of Node.js server and DB. Such parameters must be defined in configuration files for respective components. 

## Parameters in client configuration file ( Node-DC-EIS/blob/master/Node-DC-EIS-client/config.json ) :
  - "MT_interval" : "100" // Total time for measuring time requests
	- "request" : "10000",  // Total number of requests to be issued
	- "concurrency" : "200", // That many requests can be issued in parallel
  - "run_mode" : "1",  // The type of run. 1 for time-based, 2 for request-based
  - "interval" : "15", // Data collection happens every interval specified in seconds
	- "rampup_rampdown": "100", // After this many requests, execution will enter measurement phase
	- "tempfile":"temp_log", // Default name of the log file
	- "total_urls":"100", // Number of urls which will be used to issue requests 
	- "server_ipaddress":"localhost", // Server IP address
	- "server_port":"9000",
	- "root_endpoint": "/",
	- "nameurl_count":"25", // 25 % of get_ratio type of requests are nameurl_count
	- "zipurl_count":"25", // 25 % of get_ratio type of requests are zipurl_count
	- "idurl_ratio":"50" // 50 % of get_ratio type of requests are idurl_count
	
	- "db_params":  // Parameters below are related to DB
	- "dbrecord_count": "10000", // DB is created with this many number of employee records
	- "name_ratio":"25", // DB is created such that each last name search will find ~this many employee records  
	- "zip_ratio":"25" // DB is created such that each last zipcode search will find ~this many employee records 
	
	- "url_params": // Parameters set the ratio for type of requests. total below must be 100 and post and delete must be equal to maintain approximately same number of empoyee records during a run
	- "get_ratio": "90", // Out of total requests this many requests are of getID type which has sub-breakup into name, zip and id urls
	- "post_ratio":"5", // Out of total requests this many requests are of post type
	- "delete_ratio":"5" // Out of total requests this many requests are of delete type
	
	- "memory_params": // Parameters below helps in profiling memory usages by Node.js server
	- "memstat_interval": "1", 
	- "memlogfile": "memlog_file"

## Parameters in Node.js server configuration file "Node-DC-EIS-cluster/config/configuration.js"

	- 'cpu_count' : 0, // default 0 means monolithic mode, -1 means total number of CPUs available

  	- 'enable_caching' : false, // Caching of MongoDB in Node.js
  	- 'cache_max_size' : 100000,
  	- 'cache_expiration' : 1200000


# Node-DC-EIS Workload Modes  

This workload has two modes 

  - Cluster mode (includes a monolithic mode when setting CPU count by cpu_count = 0 in Node.js server configuration file)

  - Microservices mode


# Content  

Node-DC-EIS workload code, which contains following directories,
Server Codebase:
  - Node-DC-EIS-cluster, and
  - Node-DC-EIS-microservices

Client driver codebase
  - Node-DC-EIS-client

Multiple instance scripts
  - 

#### NOTE : 
If proxy needs to be set up, make sure the it has been properly set.
  - (npm config set proxy http://proxy.example.com:8080)
  - (npm config set https-proxy http://proxy.example.com:8080)

# Client Setup  

## Linux Client: Required Modules and Installations<commit_msg>Update README.md with latest information<commit_after># Objective 
The Node-DC is an open source collaboration project to develop workloads representing realistic use cases of Node.js in the data center. It started in 2016 with collaboration from the open source community, Node.js experts and companies playing important roles in Node.js’s evolution and production deployments in the data centers. 

Workloads APIs, functionalities and metrics will be used to evaluate performance of the complete software stack for Node.js, Operating Systems, containers, virtualization, network stacks as well as various data center configuration choices. These are relevant for hardware and software developers, researchers and Node.js community. One priority for this project is to update the workloads in order to keep them up with the fast moving eco-system of Node.js. 

# Contributing and Community   

## Contribution to individual projects: 
If you want to contribute code to a project, first you need to fork the project. Next step is to send a pull request (PR) for review to the repository. For small changes, you can implement at first step and for large changes, it is recommended to initially send the architecture and data flow, before investing significant time into the detailed implementation. The PR will be reviewed by the project team members. You need one "Looks Good To Me (LGTM)" and no objections from the project members. Once you have gained the required approval, the project maintainers will merge the PR.

## Contribution to use cases: 
We are constantly evaluating the relevant use cases. The list below includes the current status and is evolving based on the continuous feedback we are receiving from the community and major datacenter deployments for Node.js: 

  - Node-DC-EIS (Node.js - Data Center - Employee Information Services): Current release is at v1.0.0.
    
  - Node-DC-SSR (Node.js - Data Center - Server Side Rendering): Under consideration as next candidate

  - We are looking for community suggestions for other relevant use cases

Feedback can be sent either by creating issues or sending e-mail listed at https://github.com/Node-DC

# Node-DC-EIS (Node.js - Data Center - Employee Information Services) 
 
This is first in a series of upcoming workloads for modelling use cases of Node.js in Data Center (Node-DC). This workload is modelling various functionalities of Employee Information Services implemented in Node.js in Data Center.

# Open for contributions  
  - Feedback from community
  - Change from Mongoose to MongoDB driver
  - Enable DB caching in microservices 
  - Enable clustering of microservices 
  - Increase unique lastnames from 777 to at least 5000 or more 
  
Following changes are being considered as optional:
  - Options for alternate DB choices
  - Unit tests as part of PR acceptance criterion 
  - Changing client from Python to Node.js 
  - Workload phases ramp-up, measurement, and ramp-down to be able to set for a given time instead of #requests 
  - Node internal stats like events in the loop etc.
  - Reduce the size of output file by processing response time samples every n sec interval instead of post-processing at the end of complete run
  - Containerize the client, server and DB for easy testing and setup of the workload 

# Node-DC-EIS Architecture  
Node-DC-EIS follows a 3-tier model and consists of three components client, Node.js server and DB.

## Client 
Client controls various phases like ramp-up, measurement and ramp-down as well as issuing requests, tracks response time and errors. At the end of run, it validates the runs and post-processes the transactions log producing final metrics and graphs.

## Node.js server 
Node.js server accepts all requests from client and responds back after retrieving employee information from the DB. Node.js cluster code is architected to scale by default. 

In Node.js server configuration file "Node-DC-EIS-cluster/config/configuration.js", parameter 'cpu_count' can be used to scale to various number of CPUs. 
  - Default value is '0' which will use single CPU core or called monolithic mode.
  - With the value of '-1', it will use all available CPUs. 
  - It will use number of processes as specified by any other positive value 

## DB 
Employee Information is stored in DB (current implementation using MongoDB). For each run, DB is populated based on configuration parameters set in the "Node-DC-EIS/blob/master/Node-DC-EIS-client/config.json".     

# Node-DC-EIS Metrics  
Node-DC-EIS produces two primary metrics:
  - Throughput ( n Node-DC-EIS requests / second ): It is calculated by "requests in measurement phase / measurement time" 
  - p99 Response Time: It is 99 percentile of response time. 
	
Report also contains detailed response time data like minimum, maximum, average, p95 and p99 (99 percentile) of response time. 
Post-processing also produces response time graph as well as other histograms like memory utilization over the run length of the workload.  

# Node-DC-EIS default and configurable options for research and testing  
Node-DC-EIS sets all parameters by default to model typical Node.js server deployment. Many important parameters have been defined in the configuration files to make it easy to be able to evaluate, test and validate wide variety of deployments. 

Most parameters can be set in the client configuration file and these are passed to Node.js server and DB. But some parameters must be set at the start of Node.js server and DB. Such parameters must be defined in configuration files for respective components. 

## Parameters in client configuration file ( Node-DC-EIS/blob/master/Node-DC-EIS-client/config.json ) :
  - "MT_interval" : "100" // Total time for measuring time requests
	- "request" : "10000",  // Total number of requests to be issued
	- "concurrency" : "200", // That many requests can be issued in parallel
  - "run_mode" : "1",  // The type of run. 1 for time-based, 2 for request-based
  - "interval" : "10", // Data collection happens every interval specified in seconds
	- "rampup_rampdown": "100", // After this many requests, execution will enter measurement phase
	- "tempfile":"temp_log", // Default name of the log file
	- "total_urls":"100", // Number of urls which will be used to issue requests 
	- "server_ipaddress":"localhost", // Server IP address
	- "server_port":"9000",
	- "root_endpoint": "/",
	- "nameurl_count":"25", // 25 % of get_ratio type of requests are nameurl_count
	- "zipurl_count":"25", // 25 % of get_ratio type of requests are zipurl_count
	- "idurl_ratio":"50" // 50 % of get_ratio type of requests are idurl_count
	
	- "db_params":  // Parameters below are related to DB
	- "dbrecord_count": "10000", // DB is created with this many number of employee records
	- "name_ratio":"5", // DB is created such that each last name search will find ~this many employee records  
	- "zip_ratio":"5" // DB is created such that each last zipcode search will find ~this many employee records 
	
	- "url_params": // Parameters set the ratio for type of requests. total below must be 100 and post and delete must be equal to maintain approximately same number of empoyee records during a run
	- "get_ratio": "90", // Out of total requests this many requests are of getID type which has sub-breakup into name, zip and id urls
	- "post_ratio":"5", // Out of total requests this many requests are of post type
	- "delete_ratio":"5" // Out of total requests this many requests are of delete type
	
	- "memory_params": // Parameters below helps in profiling memory usages by Node.js server
	- "memstat_interval": "1", 
	- "memlogfile": "memlog_file"

## Parameters in Node.js server configuration file "Node-DC-EIS-cluster/config/configuration.js"

	- 'cpu_count' : 0, // default 0 means monolithic mode, -1 means total number of CPUs available
  	- 'enable_caching' : false, // Caching of MongoDB in Node.js
  	- 'cache_max_size' : 100000,
  	- 'cache_expiration' : 1200000


# Node-DC-EIS Workload Modes  

This workload has two modes 

  - Cluster mode (includes a monolithic mode when setting CPU count by cpu_count = 0 in Node.js server configuration file)

  - Microservices mode


# Content  

Node-DC-EIS workload code, which contains following directories,
Server Codebase:
  - Node-DC-EIS-cluster, and
  - Node-DC-EIS-microservices

Client driver codebase
  - Node-DC-EIS-client

Multiple instance scripts
  - multiple_instance_config.json // Contains configuration
  - run_multiple_instance.sh      // Main driver script
 
#### NOTE : 
If proxy needs to be set up, make sure the it has been properly set.
  - (npm config set proxy http://proxy.example.com:8080)
  - (npm config set https-proxy http://proxy.example.com:8080)

# Client Setup  

## Linux Client: Required Modules and Installations
{'fraction_non_alphanumeric': 0.06420289027246244, 'fraction_numerical': 0.012794882047181128, 'mean_word_length': 3.8606329816768463, 'pattern_counts': {'":': 40, '<': 3, '<?xml version=': 0, '>': 3, 'https://': 2, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '2510529', 'n_tokens_mistral': 4724, 'n_tokens_neox': 4387, 'n_words': 2374}