<commit_before>\newcommand{\numhosts}{192}
\newcommand{\storageperhost}{256GB}

\newcommand{\childrenperparent}{192}

\newcommand{\fortynine}{$\frac{n}{2} - 1$}
\newcommand{\fiftyone}{$\frac{n}{2} + 1$}

\newcommand{\filechecksize}{256 bytes}

\newcommand{\maxcorruption}{85\%}
\newcommand{\inversemaxcorruption}{15\%}
\newcommand{\maxredundancy}{6$\frac{2}{3}$}

\newcommand{\maxslicesize}{1mb}

% the real set of commands - above to be replaced
\newcommand{\quorumsize}{128}
\newcommand{\defaultM}{28}
\newcommand{\maxsegmentsize}{1mb}

\documentclass[twocolumn]{article}

\begin{document}
\frenchspacing

\title{Sia : Decentralized, Compensated, Self-Repairing Computer Storage}

\author{
{\rm <NAME>}\\
Rensselaer Polytechnic Institute
}

\maketitle

\subsection*{Abstract}
Sia is a decentralized platform for computer storage acting as a marketplace for storage.
On the supply side, host computers contribute storage in return for Sia's internal cryptocurrency - Siacoin.
On the demand side, consumers use Siacoin to rent storage space.
The price of storage is set by a market making algorithm that is sensitive to the volume of supply and demand.
Files on Sia are distributed to a large number of randomly selected hosts.
Erasure Coding is used to give the files redundancy, enabling files to be spread over hundreds of machines with highly reliable network redundancies as low as 1.2.
Spreading the files over hundreds of hosts means that files can be downloaded in parallel from hundreds of places and achieve high throughput.
As explored in the paper, Sia should be a cheaper and superior solution to cloud storage when compared against the existing centralized alternatives.

\section{Introduction}

Sia is a decentralized platform for storing data on the cloud.
The storage on Sia is block level, like an unformatted hard drive.
This makes Sia file system agnostic - you store information on Sia by setting and requesting arrays of bits.
It is very easy to put a file system such as NTFS or ext4 on top of Sia.
It is expected that the average user will not interact directly with Sia, but rather with an application such as a music player that streams directly from Sia.

Because Sia operates at the block level, it does not make as much sense to talk about files being stored on Sia.
Instead we talk about sectors of storage, a sector being a logical set of bits stored on Sia.
Files can be stored within sectors, across multiple sectors, or any other way that a file can be translated into a series of bits.
Sia is random access, which means you can download any part of a sector without needed to download the whole sector.

Each sector is given to \quorumsize{} machines that monitor the health of the sector.
If any of these machines goes offline, it is replaced by the network, keeping the sector available and in good health.
The sector is broken into \quorumsize{} segments, of which M are redundant, where M is chosen by the uploader.
Any M machines can go offline at the same time and the sector will still be recoverable - this is because Reed-Solomon coding is used.
If a machine goes offline or fails, it will be replaced by a new machine within a few hours, bringing the sector back up to its original redundancy.
Having a sector split between \quorumsize{} machines and being able to tolerate M faults every day means that even in scenarios of high paranoia M can be relatively small.

Reed-Solomon codes are maximum-distance separable.
This means that redundancy is maximally efficient.
For example, take a 100mb sector that gets broken up into 128 pieces with M=28.
You have 100 non-redundant pieces and 28 redundant pieces.
Of these, ANY 28 pieces can be lost and the full sector can still be recovered.
Together, all 128 pieces consume 128mb of physical storage.
The default value for M on Sia is \defaultM{}.

Each sector is stored on \quorumsize{} machines, and the sector can be downloaded in parallel from all of them.
This allows Sia to obtain very high throughput for downloads, even if each machine is only offering a little bandwidth.
Additionally, for situations requiring reduced latency, only the closest \quorumsize{} - M machines need to be contacted.
Because a random set of machines are hosting each sector, the closest \quorumsize{} - M machines are likely to be very close when M is large, regardless of the location of the downloader.

Each sector stored on Sia is accompanied by approximately 40kb of overhead that allows the network to monitor and error-check the sector.
Multiple files can be stored on a single sector; 40kb is not a minimum file size it is a minimum sector size.
Each sector is broken into \quorumsize{} segments after redundancy is added.
The maximum size of a segment is limited to \maxsegmentsize.
The maximum size of a sector therefore depends on the value of M, a larger M means a smaller maximum sector size.
Sectors are just a way for the network to orgainze, even though the maximum sector size is small, files and information can be stored across many sectors.
Optimizing for smaller sectors means that each file is stored on that many more machines, which means the benefits of parallel downloads are increased.

Another property of Sia is that it becomes very difficult to censor data.
Forcefully taking sectors off of the network requires corrupting or destroying more than M of the \quorumsize{} segments that compose the sector.
Because sectors on Sia are randomly distributed among a global set of hosts, it is very difficult for a single entity (even as powerful as the NSA) to attack a sufficient number of machines to corrupt a segment, especially when M is set to be large.

This network design results in some great features:
\begin{itemize}
	\item High Speed
	\item Low Latency
	\item High Robustness and Security
	\item Low Overhead
\end{itemize}

\section{Managing Sectors}

When uploading, a sector can either be encrypted or public.
Encrypted sectors are great for housing personal or private files.
Public sectors are great for hosting embeddable web content such as images and videos.
Any sector can be downloaded by anybody, and a sector may end up on any malicious machine.
The best way to protect a file or sector is to use client-side encryption.

Sectors can also be mutable or immutable.
The contents of a mutable sector can be altered at any time, but only by authorized parties.
Authorization is done using public key cryptography.
It is expected that most sectors will be mutable.
Immutable sectors cannot be altered after they have been uploaded.
This is achieved by authorizing nobody to edit a file.
Immutable sectors are appropriate for public and/or shared data - if you use a public sector, you want assurance that the data will be the same every time you access it.

Each sector is associated with a balance of siacoins, which is used to pay for the storage that the sector consumes.
Each block, a volume of siacoins is subtracted from the sector balance according to the current price of storage.
When the balance runs out, the sector is deleted, and space on the network is cleared for new data.
Anybody can add to the balance of a sector at any time, whether the sector is mutable or immutable.
Immutable sectors cannot be deleted from the network unless the siacoin balance fully depletes.

\section{Use Cases}

Sia can be used as an efficient replacement to many of today's web services.
One obvious example it BitTorrent.
With BitTorrent, every seeder needs essentially the entire file that's being seeded, resulting in a huge global redundancy of the file.
With Sia, a file only needs a small amount of redundancy to provide the same amount of throughput.
Another issue is that torrents die once the last seeder disappears.
The half life of a torrent can be as low as a few months.
With Sia, a file will stay around as long as the sectors are paid for.
Because anyone can pay for a sector, anyone can secure the longevity of a file on Sia without needing to go through the hassle of seeding it continuously.
Sia is not free, but neither is BitTorrent.
The cost of BitTorrent is seeding files, which consumes hard drive space and bandwidth.
Anybody with both of these resources can participant in Sia and gain Siacoins, enabling them to also download from Sia.
Sia is a comprehensive replacement for BitTorrent.

Sia makes sense as the storage layer for any streaming application.
This includes content embedded into web pages.
By embedding images and videos into Sia instead of hosting them on a centralized server, you can achieve lower storage costs, lower bandwidth costs, higher throughput, and equivalent (or even reduced) latencies.
Sia automatically acts as a CDN, meaning users across the world receive similar service to local users.
Sia could also be used for non-streaming applications, such as for save data in a video game, or even for loading new areas in a video game.

Sia is also elastic, meaning you only need to rent exactly as much storage as you are currently using.
Instead of needing to plan for the future and buying a 1TB drive, if you are only consuming 650GB then you can rent exactly 650GB of Sia storage.
You can expand the volume of storage you are renting at any time, within minutes.

On sufficiently fast connections, Sia could potentially be used as a replacement for hard drives.
The throughput on Sia can potentially exceed a gigabit per second, which is nearly as much throughput as an SSD.
The bottleneck would be latency, which would still be around a hundred milliseconds per request (except for sectors with very high redundancy).
By moving enough of your operating system into RAM, you can avoid this problem and replace your local hard drive.

\section{Economics}

\section{Overview}

Sia is built out of many small blockchains of \numhosts{} hosts each.
These blockchains participate in a deterministic consensus algorithm instead of using a block mining algorithm such as proof-of-work.
Blockchains are assembled randomly from hosts on the network - hosts cannot control which blockchain they are participating in.
Each host must be contributing exactly \storageperhost{} to the network in order to be placed in a blockchain.
A single machine operate as a group of hosts each contributing exactly \storageperhost{}, as long as the storage is unique.
Blockchains will be honest as long as at least a simple majority of the hosts within the blockchain are honest.

The proof-of-storage algorithm makes use of the fact that data is redundant - each host is storing different Reed-Solomon coded pieces of the files uploaded to the network.
By gathering file pieces from each host, the network can use the pieces to check for honesty and corruption simultaneously.

Each blockchain has a set of files which it tracks.
As files are added and deleted, the network performs load-balancing between blockchains to keep the file distribution even.

Hosts wanting to join the network are put into a queue, where they wait until there is sufficient storage demand to justify creating a new blockchain.
If there are many hosts joining the queue but few new files being added to the network, the price of storage will drop.
If there are many files being uploaded and few hosts joining the queue, the price of storage will increase.
The pricing algorithm tries to always keep hosts in a queue instead of putting files in a queue.
<commit_msg>add basic economics, leave out advanced economics<commit_after>\newcommand{\numhosts}{192}
\newcommand{\storageperhost}{256GB}

\newcommand{\childrenperparent}{192}

\newcommand{\fortynine}{$\frac{n}{2} - 1$}
\newcommand{\fiftyone}{$\frac{n}{2} + 1$}

\newcommand{\filechecksize}{256 bytes}

\newcommand{\maxcorruption}{85\%}
\newcommand{\inversemaxcorruption}{15\%}
\newcommand{\maxredundancy}{6$\frac{2}{3}$}

\newcommand{\maxslicesize}{1mb}

% the real set of commands - above to be replaced
\newcommand{\quorumsize}{128}
\newcommand{\defaultM}{28}
\newcommand{\hostincomefee}{3.9\%}
\newcommand{\maxsegmentsize}{1mb}

\documentclass[twocolumn]{article}

\begin{document}
\frenchspacing

\title{Sia : Decentralized, Compensated, Self-Repairing Computer Storage}

\author{
{\rm <NAME>}\\
Rensselaer Polytechnic Institute
}

\maketitle

\subsection*{Abstract}
Sia is a decentralized platform for computer storage acting as a marketplace for storage.
On the supply side, host computers contribute storage in return for Sia's internal cryptocurrency - Siacoin.
On the demand side, consumers use Siacoin to rent storage space.
The price of storage is set by a market making algorithm that is sensitive to the volume of supply and demand.
Files on Sia are distributed to a large number of randomly selected hosts.
Erasure Coding is used to give the files redundancy, enabling files to be spread over hundreds of machines with highly reliable network redundancies as low as 1.2.
Spreading the files over hundreds of hosts means that files can be downloaded in parallel from hundreds of places and achieve high throughput.
As explored in the paper, Sia should be a cheaper and superior solution to cloud storage when compared against the existing centralized alternatives.

\section{Introduction}

Sia is a decentralized platform for storing data on the cloud.
The storage on Sia is block level, like an unformatted hard drive.
This makes Sia file system agnostic - you store information on Sia by setting and requesting arrays of bits.
It is very easy to put a file system such as NTFS or ext4 on top of Sia.
It is expected that the average user will not interact directly with Sia, but rather with an application such as a music player that streams directly from Sia.

Because Sia operates at the block level, it does not make as much sense to talk about files being stored on Sia.
Instead we talk about sectors of storage, a sector being a logical set of bits stored on Sia.
Files can be stored within sectors, across multiple sectors, or any other way that a file can be translated into a series of bits.
Sia is random access, which means you can download any part of a sector without needed to download the whole sector.

Each sector is given to \quorumsize{} machines that monitor the health of the sector.
If any of these machines goes offline, it is replaced by the network, keeping the sector available and in good health.
The sector is broken into \quorumsize{} segments, of which M are redundant, where M is chosen by the uploader.
Any M machines can go offline at the same time and the sector will still be recoverable - this is because Reed-Solomon coding is used.
If a machine goes offline or fails, it will be replaced by a new machine within a few hours, bringing the sector back up to its original redundancy.
Having a sector split between \quorumsize{} machines and being able to tolerate M faults every day means that even in scenarios of high paranoia M can be relatively small.

Reed-Solomon codes are maximum-distance separable.
This means that redundancy is maximally efficient.
For example, take a 100mb sector that gets broken up into 128 pieces with M=28.
You have 100 non-redundant pieces and 28 redundant pieces.
Of these, ANY 28 pieces can be lost and the full sector can still be recovered.
Together, all 128 pieces consume 128mb of physical storage.
The default value for M on Sia is \defaultM{}.

Each sector is stored on \quorumsize{} machines, and the sector can be downloaded in parallel from all of them.
This allows Sia to obtain very high throughput for downloads, even if each machine is only offering a little bandwidth.
Additionally, for situations requiring reduced latency, only the closest \quorumsize{} - M machines need to be contacted.
Because a random set of machines are hosting each sector, the closest \quorumsize{} - M machines are likely to be very close when M is large, regardless of the location of the downloader.

Each sector stored on Sia is accompanied by approximately 40kb of overhead that allows the network to monitor and error-check the sector.
Multiple files can be stored on a single sector; 40kb is not a minimum file size it is a minimum sector size.
Each sector is broken into \quorumsize{} segments after redundancy is added.
The maximum size of a segment is limited to \maxsegmentsize.
The maximum size of a sector therefore depends on the value of M, a larger M means a smaller maximum sector size.
Sectors are just a way for the network to orgainze, even though the maximum sector size is small, files and information can be stored across many sectors.
Optimizing for smaller sectors means that each file is stored on that many more machines, which means the benefits of parallel downloads are increased.

Another property of Sia is that it becomes very difficult to censor data.
Forcefully taking sectors off of the network requires corrupting or destroying more than M of the \quorumsize{} segments that compose the sector.
Because sectors on Sia are randomly distributed among a global set of hosts, it is very difficult for a single entity (even as powerful as the NSA) to attack a sufficient number of machines to corrupt a segment, especially when M is set to be large.

This network design results in some great features:
\begin{itemize}
	\item High Speed
	\item Low Latency
	\item High Robustness and Security
	\item Low Overhead
\end{itemize}

\section{Managing Sectors}

When uploading, a sector can either be encrypted or public.
Encrypted sectors are great for housing personal or private files.
Public sectors are great for hosting embeddable web content such as images and videos.
Any sector can be downloaded by anybody, and a sector may end up on any malicious machine.
The best way to protect a file or sector is to use client-side encryption.

Sectors can also be mutable or immutable.
The contents of a mutable sector can be altered at any time, but only by authorized parties.
Authorization is done using public key cryptography.
It is expected that most sectors will be mutable.
Immutable sectors cannot be altered after they have been uploaded.
This is achieved by authorizing nobody to edit a file.
Immutable sectors are appropriate for public and/or shared data - if you use a public sector, you want assurance that the data will be the same every time you access it.

Each sector is associated with a balance of siacoins, which is used to pay for the storage that the sector consumes.
Each block, a volume of siacoins is subtracted from the sector balance according to the current price of storage.
When the balance runs out, the sector is deleted, and space on the network is cleared for new data.
Anybody can add to the balance of a sector at any time, whether the sector is mutable or immutable.
Immutable sectors cannot be deleted from the network unless the siacoin balance fully depletes.

\section{Use Cases}

Sia can be used as an efficient replacement to many of today's web services.
One obvious example it BitTorrent.
With BitTorrent, every seeder needs essentially the entire file that's being seeded, resulting in a huge global redundancy of the file.
With Sia, a file only needs a small amount of redundancy to provide the same amount of throughput.
Another issue is that torrents die once the last seeder disappears.
The half life of a torrent can be as low as a few months.
With Sia, a file will stay around as long as the sectors are paid for.
Because anyone can pay for a sector, anyone can secure the longevity of a file on Sia without needing to go through the hassle of seeding it continuously.
Sia is not free, but neither is BitTorrent.
The cost of BitTorrent is seeding files, which consumes hard drive space and bandwidth.
Anybody with both of these resources can participant in Sia and gain Siacoins, enabling them to also download from Sia.
Sia is a comprehensive replacement for BitTorrent.

Sia makes sense as the storage layer for any streaming application.
This includes content embedded into web pages.
By embedding images and videos into Sia instead of hosting them on a centralized server, you can achieve lower storage costs, lower bandwidth costs, higher throughput, and equivalent (or even reduced) latencies.
Sia automatically acts as a CDN, meaning users across the world receive similar service to local users.
Sia could also be used for non-streaming applications, such as for save data in a video game, or even for loading new areas in a video game.

Sia is also elastic, meaning you only need to rent exactly as much storage as you are currently using.
Instead of needing to plan for the future and buying a 1TB drive, if you are only consuming 650GB then you can rent exactly 650GB of Sia storage.
You can expand the volume of storage you are renting at any time, within minutes.

On sufficiently fast connections, Sia could potentially be used as a replacement for hard drives.
The throughput on Sia can potentially exceed a gigabit per second, which is nearly as much throughput as an SSD.
The bottleneck would be latency, which would still be around a hundred milliseconds per request (except for sectors with very high redundancy).
By moving enough of your operating system into RAM, you can avoid this problem and replace your local hard drive.

\section{Economics}

Sia compensates hosts using a cryptocurrency.
This cryptocurrency, Siacoin, will be easily exchangeable for bitcoins and subsequently USD through a trustless exchange.
When the currency launches, the mining rate will be set to 10,000 siacoins per day.
The number of coins mined will decrease daily until the 4 year mark, at which point the network will mine coins such that the annual inflation rate is kept permanently at 5\%.
This means that there is no cap to the number of siacoins that will be produced.
No coins will be premined.

An inflationary model has been chosen for Siacoin to emphasize the current hosts on the network as opposed to the early adopters.
Initially, the number of coins generated will be very high, and the early adopters will be rewarded.
In the long run however we want the people actively contributing resources to the network to be the people that are most heavily rewarded.
We also believe that an inflationary model will result in a currency that is much less volatile.

Hosts on Sia will have two incomes: the first is the income from mining new coins, and the second is the income from rental payments.
This creates a unique dynamic where it's entirely possible that the raw cost of storage will be greater than the network price of storage due to the additional income hosts receive from mining.
Both incomes for hosts have a \hostincomefee{} fee which is distributed among the owners of Siastock.

Siastock is a 100\% premined currency that will be initially owned completely by the developers.
During an IPO or a pre-IPO, some percentage of Siastock will be traded for seed money.
Someone who owns 1\% of all Siatock will receive (1\% * \hostincomefee{}) = 0.039\% of all host income on the Sia network.
This model of developer and backer revenue has been chosen instead of premining because it enables us to make Sia inflationary while still having a guaranteed income.
It also makes it clearer exactly how much value is going towards the developers and stock owners.

The only way to access the storage on Sia is to use siacoins.
The storage on Sia has a certain value, and the only way to access that value is to use Siacoin.
If the price of Siacoin drops significantly, access to Sia resources becomes very cheap, which should in turn increase the demand for Siacoin and bring the price back up.
The internal value of Sia, combined with the fact that this value can only be accessed using Siacoin gives Siacoin an internal value that is roughly equivalent to the value of all the resources on Sia.
Siastock provides a guaranteed income that is equivalent a percentage of all host income in siacoins, Siastock also has an internal value.

% price ups and downs
% pricing Siacoin
% pricing Siastock

\section{Overview}

Sia is built out of many small blockchains of \numhosts{} hosts each.
These blockchains participate in a deterministic consensus algorithm instead of using a block mining algorithm such as proof-of-work.
Blockchains are assembled randomly from hosts on the network - hosts cannot control which blockchain they are participating in.
Each host must be contributing exactly \storageperhost{} to the network in order to be placed in a blockchain.
A single machine operate as a group of hosts each contributing exactly \storageperhost{}, as long as the storage is unique.
Blockchains will be honest as long as at least a simple majority of the hosts within the blockchain are honest.

The proof-of-storage algorithm makes use of the fact that data is redundant - each host is storing different Reed-Solomon coded pieces of the files uploaded to the network.
By gathering file pieces from each host, the network can use the pieces to check for honesty and corruption simultaneously.

Each blockchain has a set of files which it tracks.
As files are added and deleted, the network performs load-balancing between blockchains to keep the file distribution even.

Hosts wanting to join the network are put into a queue, where they wait until there is sufficient storage demand to justify creating a new blockchain.
If there are many hosts joining the queue but few new files being added to the network, the price of storage will drop.
If there are many files being uploaded and few hosts joining the queue, the price of storage will increase.
The pricing algorithm tries to always keep hosts in a queue instead of putting files in a queue.

{'fraction_non_alphanumeric': 0.0346201938105658, 'fraction_numerical': 0.00578305720537668, 'mean_word_length': 4.4920600858369095, 'pattern_counts': {'":': 0, '<': 5, '<?xml version=': 0, '>': 5, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 18, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '1955120', 'n_tokens_mistral': 6291, 'n_tokens_neox': 5931, 'n_words': 4130}