<commit_before>node-Tor
===

Javascript implementation of a Tor (or Tor like) anonymizer network (The Onion Router https://www.torproject.org/)

Example of implementations:

* Peersm (http://www.peersm.com) : Anonymous peer to peer download inside browsers and distributed database inside browsers (torrent like but better and safe)

* iAnonym :Anonymity into your browser everywhere from any device, see https://www.github.com/Ayms/iAnonym and http://www.ianonym.com

## Presentation:

This is an unofficial and extended implementation of the Tor (or Tor like) protocol (Onion Proxy and Onion Router) which anonymizes communications via the Tor (or Tor like) network. This allows to simply connect to the Tor (or Tor like) network and use it, as well as creating and adding nodes into the network, creating complementary and/or parallel networks, implementing completely, partially or not the Tor protocol or a derived one, using completely, partially or not the Tor network, it can be used to create separated Tor like networks.

####There are numerous possibilities of uses for node-Tor

####The most challenging goals are now to put the OP and the OR inside the browsers.

####It's done for the OP, see latest screenshots here (Peersm/OR DB): http://www.peersm.com

####It's already quite fast while no optimization efforts have been made.

Now we can envision the OR inside the browsers too.

This repository is a subset of the complete version, the complete version is stable but not public for now, see below "Why is the complete version not public?" and "node-Tor Status".

See live test possibility in section "Tests".

## Why is the complete version not public?

####The complete version does include the OP and OR parts, websocket protocol extension, socks proxy interface + all communication interfaces for both (tcp, http, tls, websockets), the OP inside the browser (+ everything that is required : crypto, websocket, array buffer, etc - see [iAnonym](https://www.github.com/Ayms/iAnonym) ) and proxy auto config mechanisms.

Speechless to say that it's a huge and complicate work, but javascript's magic makes that the complete code is only about 6000 "normal" lines, the complete code with third party modules minified is about 400 KB.

It does offer numerous powerfull and easy possibilities to access anonymizer networks or create your own and do whatever you like inside them. It's really fast and much faster than whatever exists today.

Then we are a bit concerned about what people could do with it, that's why it's not public for now.

##js OP : Anonymous peer to peer download inside browsers and distributed database inside browsers (torrent like but better and safe)

####This section does explain the OR-DB mechanisms to download files from the internet or peer to peer from browsers anonymously. You don't need to install anything since the entry point to the anonymizer network is the javascript OP inside your browser.

####The usual trivial question is how can you be sure that the js OP code loaded by the browser from our site is trustable? It is if you use https with our site combined with the excellent [Interception Detector] (http://www.ianonym.com/intercept.html)

####The OR-DB does act like an Exit Node except that it does only relay data from one browser to another.

####The OR-DB does not know to whom it is talking to and does not know what data it is relaying, it only knows to which anonymized streams it has to relay data, and only knows the hash identifying the source of data, not it's real name.

####So, unlike Torrents, nobody knows what you are doing, therefore you can not be tracked and/or caught.

####It will be of course compatible with popular CDNs such as MEGA.

The configuration can be summarized as follow:

	[Browser 1] <-- Websocket1-1 (ephemeral for direct download)		--> 	OR1-1 	<--> OR1-2 	<--> OR1-3 <--> Web site
				<-- Websocket1-2 (permanent for peer to peer download)	-->		OR1-1b	<--> OR1-2b	<--> OR-DB
	[Browser 2] <-- Websocket2-1 (ephemeral for direct download)		--> 	OR2-1 	<--> OR2-2 	<--> OR2-3 <--> Web site
				<-- Websocket2-2 (permanent for peer to peer download)	-->		OR2-1b	<--> OR2-2b	<--> OR-DB

Each active browser has one permanent circuit/stream established with each OR-DB in the network, and some circuits established with exit nodes.

In a future version the browser could have only one permanent circuit/stream established with only one OR-DB and other circuits with other OR-DBs could be establisehd on demand dynamically.
			
RELAY_DB_INFO streams are used by browsers to tell the OR-DB what files they have.

A file is represented by a link (https://www.download.com/file), which can be an internet link or generated by the system if this is a file that you have stored yourself inside your browser.

This link is processed to get a unique reference called a "Hash_name" below.

So there are public hash_names (derived from the downloading link) and private ones for the files that you have uploaded yourself inside your browser, others don't know them unless you give them the hash_names.

Files are stored inside your browser using IndexedDB. Files are stored keeping reference to the file length so if an unexpected event does occur or someone closes his browser during the download, the transfer can be resumed from another source.

This does allow several users to stream files simultanously too.

The index for storing the files is the hash of the file, so if you download several time the same file with different hash_name (from different locations for example), only one will be stored with a hash_name corresponding to the latest one downloaded.
			
RELAY_DB_INFO streams payload is a number of n:
* Hash_name Length	[1 byte]
* Hash_name			[Hash_name Length bytes]
* Hash_file Length	[1 byte]
* Hash_file			[Hash_file Length bytes]
* File_size Length	[1 byte]
* File_size			[Size Length bytes]
* Operation			[1 byte] //0:removed, 1 added

while [CELL_LEN - n*(Hash Lenght+Hash Length bytes+Size Length+Size Lenght bytes) - 14 bytes]>0

When receiving a RELAY_DB_INFO stream, the OR-DB does associate the files information to the socket/CIC/Stream where it has been received (Files[Hash]=Array([Socket,CIC,Stream,nb]) where nb is the number of current files being served.

RELAY_DB_QUERY (forward and backward) streams payload is :
* Hash_name Length			[1 byte]
* Hash_name					[Hash Length bytes]
* Bytes_available_length 	[1 byte]
* Bytes_available			[Bytes_available_length]
* Transaction_id			[16 bytes]

The initiating OP has chosen a streamId1 to send the RELAY_DB_QUERY message that will be used to stream data if the operation is successfull.

When receiving a RELAY_DB_QUERY stream the OR-DB looks if it can associate it to a socket,CIC,Stream choosing the one having the lowest nb, generates a Transaction_id (associated to the incoming socket/CIC/stream) and then send it forward.

When receiving a RELAY_DB_QUERY stream, the browser does respond with a:

(1) RELAY_DB_CONNECTED stream if it can serve the file:
* Transaction_id		[16 bytes]
* File_size Length		[1 byte] 
* File_size 			[File_size Length bytes]
* File_type Length		[2 bytes] 
* File_type 			[File_type Length bytes]

The browser has chosen a streamId2 to reply, the OR_DB does associate streamId2 to StreamId1. 

RELAY_DB_CONNECTED is sent backward to streamId1 (without Transaction_id) - The OR_DB increments nb. The browser receiving RELAY_DB_CONNECTED does not try to download the file connecting to an exit node and waits for data.

The browser starts sending RELAY_DB_DATA streams on streamId2:
* data

When receiving RELAY_DB_DATA streams, the OR_DB sends backward the RELAY_DB_DATA stream to StreamId1.

When download is finished, nb is decremented.

If incoming stream is destroyed, the OR_DB sends a RELAY_DB_END stream forward with Reason INCOMING_STREAM_DESTROYED and decrements nb.

If upcoming stream is destroyed, the OR_DB sends a RELAY_DB_END stream backward with Reason RESET if it can find another browser to serve the file or Reason UPCOMING_STREAM_DESTROYED

(2) RELAY_DB_END stream if it can not serve the file:
* Transaction_id	[16 bytes]
* Reason		[1 byte]

with Reason set to UNAVAILABLE.

When receiving a RELAY_DB_END stream, the OR_DB looks if another browser can serve the file (limited to NB_DB attempts), then restarts the process, if not sends it backward (with payload set at Reason only).

When receiving a RELAY_DB_END stream, the receiving browser tries downloading the file connecting to an exit node if Reason is not RESET (and reset the file being dowloaded if it was the case) or reset the file being downloaded if Reason is RESET and waits for new data.

Flow control: to be documented.

Studies are ongoing to implement the OR-DB inside browsers too.

## Tests : 

See an example of communication in [logs OP and OR] (https://github.com/Ayms/node-Tor/blob/master/test/example.txt)

You can try it live :
* set the socks proxy V5 interface of your browser to IP 192.168.3.11 port 8101 (on Firefox : Options/Advanced/Network/Parameters/Manual configuration of proxy), clear the cache/history, close your browser and reopen it
* enter url http://www.lepoint.fr or http://www.monip.org

For security reasons this test address is restricted to the domains www.lepoint.fr, www.monip.org and related IP addresses, then while loading the pages the network will not fetch resources outside of these domains (for example facebook widget on www.lepoint.fr will display "outside of authorized domains") but this is enough to give an idea and you can navigate inside the domains.

www.lepoint.fr is a huge public site, so it's a good test site.

www.monip.org allows you to see your anonymized IP address (if you refresh the page you will see that your anonymized IP address can change since different circuits can be used inside the network).

Depending on the "health" of the circuits and associated routers when you try it, the delay for page loading might vary but it is usually fast.

The test configuration is :

	[Browser] <--socks--> [node-Tor_OP:8100] <-----> [node-Tor_OR] <-----> [Tor Network] <-----> [Site] 

## Install :

Install node.js on supported platforms : Unix, Windows, MacOS

For the record, you need the following to run node-Tor :

	node.js version >= v0.7.5 (getDiffieHelman function needed)
	openssl version >= 1.0.1 (aes counter mode encryption required - evp_aes_128_ctr)
	python >= 2.6 (node.js's requirement)
	
####Current release of node.js is v0.10.5 with openSSL 1.0.1e (at last...)
	
Then as usual :

	npm install node-Tor<commit_msg>flow control, hash_name peersm<commit_after>node-Tor
===

Javascript implementation of a Tor (or Tor like) anonymizer network (The Onion Router https://www.torproject.org/)

Example of implementations:

* Peersm (http://www.peersm.com) : Anonymous peer to peer download inside browsers and distributed database inside browsers (torrent like but better and safe)

* iAnonym :Anonymity into your browser everywhere from any device, see https://www.github.com/Ayms/iAnonym and http://www.ianonym.com

## Presentation:

This is an unofficial and extended implementation of the Tor (or Tor like) protocol (Onion Proxy and Onion Router) which anonymizes communications via the Tor (or Tor like) network. This allows to simply connect to the Tor (or Tor like) network and use it, as well as creating and adding nodes into the network, creating complementary and/or parallel networks, implementing completely, partially or not the Tor protocol or a derived one, using completely, partially or not the Tor network, it can be used to create separated Tor like networks.

####There are numerous possibilities of uses for node-Tor

####The most challenging goals are now to put the OP and the OR inside the browsers.

####It's done for the OP, see latest screenshots here (Peersm/OR DB): http://www.peersm.com

####It's already quite fast while no optimization efforts have been made.

Now we can envision the OR inside the browsers too.

This repository is a subset of the complete version, the complete version is stable but not public for now, see below "Why is the complete version not public?" and "node-Tor Status".

See live test possibility in section "Tests".

## Why is the complete version not public?

####The complete version does include the OP and OR parts, websocket protocol extension, socks proxy interface + all communication interfaces for both (tcp, http, tls, websockets), the OP inside the browser (+ everything that is required : crypto, websocket, array buffer, etc - see [iAnonym](https://www.github.com/Ayms/iAnonym) ) and proxy auto config mechanisms.

Speechless to say that it's a huge and complicate work, but javascript's magic makes that the complete code is only about 6000 "normal" lines, the complete code with third party modules minified is about 400 KB.

It does offer numerous powerfull and easy possibilities to access anonymizer networks or create your own and do whatever you like inside them. It's really fast and much faster than whatever exists today.

Then we are a bit concerned about what people could do with it, that's why it's not public for now.

##js OP : Anonymous peer to peer download inside browsers and distributed database inside browsers (torrent like but better and safe)

####This section does explain the OR-DB mechanisms to download files from the internet or peer to peer from browsers anonymously. You don't need to install anything since the entry point to the anonymizer network is the javascript OP inside your browser.

####The usual trivial question is how can you be sure that the js OP code loaded by the browser from our site is trustable? It is if you use https with our site combined with the excellent [Interception Detector] (http://www.ianonym.com/intercept.html)

####The OR-DB does act like an Exit Node except that it does only relay data from one browser to another.

####The OR-DB does not know to whom it is talking to and does not know what data it is relaying, it only knows to which anonymized streams it has to relay data, and only knows the hash identifying the source of data, not it's real name.

####So, unlike Torrents, nobody knows what you are doing, therefore you can not be tracked and/or caught.

####It will be of course compatible with popular CDNs such as MEGA.

The configuration can be summarized as follow:

	[Browser 1] <-- Websocket1-1 (ephemeral for direct download)		--> 	OR1-1 	<--> OR1-2 	<--> OR1-3 <--> Web site
				<-- Websocket1-2 (permanent for peer to peer download)	-->		OR1-1b	<--> OR1-2b	<--> OR-DB
	[Browser 2] <-- Websocket2-1 (ephemeral for direct download)		--> 	OR2-1 	<--> OR2-2 	<--> OR2-3 <--> Web site
				<-- Websocket2-2 (permanent for peer to peer download)	-->		OR2-1b	<--> OR2-2b	<--> OR-DB

Each active browser has one permanent circuit/stream established with each OR-DB in the network, and some circuits established with exit nodes.

In a future version the browser could have only one permanent circuit/stream established with only one OR-DB and other circuits with other OR-DBs could be establisehd on demand dynamically.
			
RELAY_DB_INFO streams are used by browsers to tell the OR-DB what files they have.

A file is represented by a link (https://www.download.com/file), which can be an internet link or generated by the system if this is a file that you have stored yourself inside your browser.

This link is processed to get a unique reference called a "Hash_name" below.

So there are public hash_names (derived from the downloading link) and private ones for the files that you have uploaded yourself inside your browser, others don't know them unless you give them the hash_names.

Files are stored inside your browser using IndexedDB. Files are stored keeping reference to the file length so if an unexpected event does occur or someone closes his browser during the download, the transfer can be resumed from another source.

This does allow several users to stream files simultanously too.

The index for storing the files is the hash of the file, so if you download several time the same file with different hash_name (from different locations for example), only one will be stored with a hash_name corresponding to the latest one downloaded.
			
RELAY_DB_INFO streams payload is a number of n:
* Hash_name Length	[1 byte]
* Hash_name			[Hash_name Length bytes]
* Hash_file Length	[1 byte]
* Hash_file			[Hash_file Length bytes]
* File_size Length	[1 byte]
* File_size			[Size Length bytes]
* Operation			[1 byte] //0:removed, 1 added

while [CELL_LEN - n*(Hash Lenght+Hash Length bytes+Size Length+Size Lenght bytes) - 14 bytes]>0

When receiving a RELAY_DB_INFO stream, the OR-DB does associate the files information to the socket/CIC/Stream where it has been received (Files[Hash]=Array([Socket,CIC,Stream,nb]) where nb is the number of current files being served.

RELAY_DB_QUERY (forward and backward) streams payload is :
* Hash_name Length			[1 byte]
* Hash_name					[Hash Length bytes]
* Bytes_available_length 	[1 byte]
* Bytes_available			[Bytes_available_length]
* Transaction_id			[16 bytes]

The initiating OP has chosen a streamId1 to send the RELAY_DB_QUERY message that will be used to stream data if the operation is successfull.

When receiving a RELAY_DB_QUERY stream the OR-DB looks if it can associate it to a socket,CIC,Stream choosing the one having the lowest nb, generates a Transaction_id (associated to the incoming socket/CIC/stream) and then send it forward.

Note that the Bytes_available field is for the receiver to advertise how many bytes he already has for this file (in case a download was interrupted), the sender will get this information and resume the download from this number of bytes.

When receiving a RELAY_DB_QUERY stream, the browser does respond with a:

(1) RELAY_DB_CONNECTED stream if it can serve the file:
* Transaction_id		[16 bytes]
* File_size Length		[1 byte] 
* File_size 			[File_size Length bytes]
* File_type Length		[2 bytes] 
* File_type 			[File_type Length bytes]

The browser has chosen a streamId2 to reply, the OR_DB does associate streamId2 to StreamId1. 

RELAY_DB_CONNECTED is sent backward to streamId1 (without Transaction_id) - The OR_DB increments nb. The browser receiving RELAY_DB_CONNECTED does not try to download the file connecting to an exit node and waits for data.

The browser starts sending RELAY_DB_DATA streams on streamId2:
* data

When receiving RELAY_DB_DATA streams, the OR_DB sends backward the RELAY_DB_DATA stream to StreamId1.

When download is finished, nb is decremented.

If incoming stream is destroyed, the OR_DB sends a RELAY_DB_END stream forward with Reason INCOMING_STREAM_DESTROYED and decrements nb.

If upcoming stream is destroyed, the OR_DB sends a RELAY_DB_END stream backward with Reason RESET if it can find another browser to serve the file or Reason UPCOMING_STREAM_DESTROYED

(2) RELAY_DB_END stream if it can not serve the file:
* Transaction_id	[16 bytes]
* Reason		[1 byte]

with Reason set to UNAVAILABLE.

When receiving a RELAY_DB_END stream, the OR_DB looks if another browser can serve the file (limited to NB_DB attempts), then restarts the process, if not sends it backward (with payload set at Reason only).

When receiving a RELAY_DB_END stream, the receiving browser tries downloading the file connecting to an exit node if Reason is not RESET (and reset the file being dowloaded if it was the case) or reset the file being downloaded if Reason is RESET and waits for new data.

The receiver can stop a download sending a RELAY_DB_END to the sender relayed by the ORDB, in that case the partial data downloaded so far will be stored in indexedDB at the receiver side, he will be able to resume the download later.

Flow control: 

The sender does monitor its available bandwidth and does provide 50% of its bandwidth to the receivers, so 50%/(receivers nb) for each receiver, both sender and receiver are updating a window associated to the stream they are using to exchange data, the sender does decrement the window size by 1 for each packet received and waits for a RELAY_DB_SENDME message from the receiver if the window size drops to 0 before reinitializing the window size and sending data again. The receiver does the same and send a RELAY_DB_SENDME each time the window size comes under 80% of the initial window size.

This is assuming that half of the sender upload bandwidth is the minimal bandwidth available in all the path between the sender and the receiver, it might not be the case since some nodes in the path might have a lower bandwidth, in that case the sender's or receiver's circuit with the ORDB is destroyed to get another one performing better.

Hash_names management:

Hash_names are used to identify a download. If the file was originally downloaded from the web, hash_name is the hash of the link (public), if the file was uploaded inside the browser, hash_name is a hash provided by the system (private). So the same file can have different hash_name which will split the traffic better than having a unique reference like its hash for all the same files. The hash information of a file does allow to detect if a file was not modified for malicious reasons.

A name is associated to the file, for http://aaa.com/myfile.ext (public) the name is myfile.ext, for private references the name is a random string or the name of the file that was uploaded inside the browser. For private references you must provide to the receivers the hash_name and the file extension since they will not know the extension after the download.

You can encrypt files too that you have downloaded or uploaded, in that case the name of the encrypted file will become name.ext.enc, the system will keep reference to the initial type of the file and store it in indexedDB as a binary file. The sender must provide to receivers the hash_name, the encryption key and the extension of the file. While downloading an encrypted file the receiver will receive the initial type of the file so he can decrypt it and store it with the right format (but nothing else since the ORDB must not know any other information about the files that it is relaying), then he can open/save it using the extension provided by the sender.

The type of a file is used for consistency reasons when storing a file in indexedDB, but senders can change the real type of the file if they don't want the ORDB to be aware of it.

So, basically, people must exchange information to download private and/or encrypted files, by usual means (blogs, emails, etc)

Non anonym Peer to Peer (without the traffic being relayed by the ORDB): for those that do not care about anonymity, the system might implement WebRTC mechanisms so the trunking does not go through the ORDB after the ORDB has set the connexion between the two peers but goes directly between the browser of the sender and the receiver.

Others:

Studies are ongoing to implement the OR-DB inside browsers too.

## Tests : 

See an example of communication in [logs OP and OR] (https://github.com/Ayms/node-Tor/blob/master/test/example.txt)

You can try it live :
* set the socks proxy V5 interface of your browser to IP 192.168.3.11 port 8101 (on Firefox : Options/Advanced/Network/Parameters/Manual configuration of proxy), clear the cache/history, close your browser and reopen it
* enter url http://www.lepoint.fr or http://www.monip.org

For security reasons this test address is restricted to the domains www.lepoint.fr, www.monip.org and related IP addresses, then while loading the pages the network will not fetch resources outside of these domains (for example facebook widget on www.lepoint.fr will display "outside of authorized domains") but this is enough to give an idea and you can navigate inside the domains.

www.lepoint.fr is a huge public site, so it's a good test site.

www.monip.org allows you to see your anonymized IP address (if you refresh the page you will see that your anonymized IP address can change since different circuits can be used inside the network).

Depending on the "health" of the circuits and associated routers when you try it, the delay for page loading might vary but it is usually fast.

The test configuration is :

	[Browser] <--socks--> [node-Tor_OP:8100] <-----> [node-Tor_OR] <-----> [Tor Network] <-----> [Site] 

## Install :

Install node.js on supported platforms : Unix, Windows, MacOS

For the record, you need the following to run node-Tor :

	node.js version >= v0.7.5 (getDiffieHelman function needed)
	openssl version >= 1.0.1 (aes counter mode encryption required - evp_aes_128_ctr)
	python >= 2.6 (node.js's requirement)
	
####Current release of node.js is v0.10.x with openSSL 1.0.1e (at last...)
	
Then as usual :

	npm install node-Tor
{'fraction_non_alphanumeric': 0.055631757301920284, 'fraction_numerical': 0.008512183314507019, 'mean_word_length': 4.297926907458859, 'pattern_counts': {'":': 0, '<': 39, '<?xml version=': 0, '>': 47, 'https://': 10, 'lorem ipsum': 0, 'www.': 30, 'xml': 0}, 'pii_count': 2, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 43, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '3413623', 'n_tokens_mistral': 6833, 'n_tokens_neox': 6446, 'n_words': 3728}