<commit_before><p align="center"><img src="https://www.lucidchart.com/publicSegments/view/7eba83d0-9717-482f-8957-c2343f598a3d/image.jpeg" height="400"/></p>

# <p align="center">1 - Abstract</p>
Automated composition of computer programs has been a standing challenge since the early days of artificial intelligence, and has no clear solutions with modern-day research in deep learning. Indeed, modeling latent context representations in language proves to be a difficult task singularly, and combined with applying structured procedural knowledge in a generative fashion quickly becomes intractable for complex domain specific languages. There is a clear need for research in devising efficiently learned combinations of these independent problems. We investigate the nature of such an architecture, and determine the current assumptions and limitations made by previous attempts at neural program synthesis. From our observations, we propose an architecture using the recent Generative Adversarial Networks and Hyper Networks to generate syntactically correct and behaviorally useful python source code from input and output examples, pseudocode, or even erroneous existing code.

# <p align="center">2 - Introduction</p>
## Motivation for our Research.
There a number of reasons outside of  the scientific domain that motivate research in the problem of neural program synthesis. The most obvious motivator is the ability reduce the amount of time required to develop large programs that make heavy use of APIs for file conversion, handling http requests, and even data analytics. An amount of time saved by developers in a company has a proportional amount of income to be saved. Multiplied across dozens of software developers in a single firm, savings figures quickly measure in the hundreds of thousands over the course of a single business year. A second obvious motivator is a reduction in the knowledge barrier for scientific computing, which can be argued to increase innovation trends and education standards, which induce higher productivity in research and industry. From a more theoretical perspective, the difference between neural program synthesis, and neural question answering is a matter of dataset choice. Our research will bring deeper insight to general natural language understanding.

## Problem Setting.
In our investigation, we focus on building a generative machine learning model for automatic synthesis of python source code.  The operands to this model are some high dimensional vector description of the desired program behavior. We make no assumptions regarding the encoding structure of this vector, other than assuming a correlation with latent program behavior. From this representation, our model shall produce executable python source code that matches our initial program description. Having identified the learning space for our investigation, we begin to decompose this space into compartmentalized subspaces that have more tractable representations. We then consider what sort of knowledge is necessary to develop computer programs: an understanding of program syntax, and a prediction of program behavior. In our investigation, we shall explicitly model these properties; however, in other more commonly used methods, one or both of these properties is assumed to be latent. Our hypothesis is that explicit models for contextual properties within data distributions allows for more robust generative models for those distributions.

# <p align="center">3 - Related Work</p>
## Neural Program Synthesis.
This approach to learning programs will produce a sequence of tokens representing executable source code using high level base functions, or APIs. These Methods have been commonly used previously to interact with high level APIs of program such as microsoft excel to learn--for example--how to filter and transform a list of grid cells from user examples. In this implementation, the recurrent neural networks outputs a sequence of function tokens that determine the order in which base functions are applied to grid cell content. This approach is the most effective for problems that have a limited number of base functions or operations. For generalizing the composition of python source code, base functions would have to be encoded as single characters, and this approach would become a naive end-to-end code generator, which has not been successful in past research.

## Neural Program Induction.
This approach to learning programs differs from program synthesis in that there is no explicit representation of base functions, and there are no human observable function tokens that can be executed by an interpreter. Instead, this method assumes a neural network will learn the latent procedure of an algorithm in an end-to-end manner. These methods include a large number of memory networks invented in the past few years. In practice, these algorithms are able to learn simple procedural tasks such as binary addition, and sorting, but are not able to learn more complex functions including recursion. These methods are also limited in that learning an update to the internal procedure of the network is nearly impossible, and requires a large amount of examples, and potentially rebuilding the model. For these models to generalize to the composition of high level programming languages the amount of data required for training is intractable, and has not been successfully attempted.

# <p align="center">4 - Human Perspective</p>
In order to fully realize the problems and assumptions made by prior approaches in this field, we consider the human approach to writing python source code. Engineering students at UC Berkeley are  given a text explanation of the intended behavior of a function, along with multiple input and output examples of that function. These students are able to consistently compose python code to solve arbitrarily complex tasks given this limited information. In such coding homework, these students are not explicitly given a table of syntax rules, so we can infer that sytnes rules have been learned implicitly by these students. In addition to learning syntax, in order to compose programs that accomplish a certain procedural scheme, these students must be able to predict the behavior of code internally while writing--otherwise solving homework would require trial-and-error, and would be intractable to solve.

# <p align="center">5 - Problems and Assumptions</p>
The primary problem existing with programming by example methods for neural program induction is that the required amount of different examples for complex programs can become intractable quickly, and even more so when performing training on that data. Programming by example methods for neural program synthesis require significantly less data to generate functioning programs, however, the complexity of programs is limited by the number of accessible base functions. More recently, synthesis methods that allow for base functions to be learned explicitly solve this problem, but assume a latent representation of final program behavior, and of program syntax. We hypothesize that learning complex programs becomes an easier task when representations for both program behavior and program syntax are made explicit.

*<p align="center">Research Idea by <NAME>, with collaboration from <NAME> and <NAME></p>*<commit_msg>lmao just corrected my name<commit_after><p align="center"><img src="https://www.lucidchart.com/publicSegments/view/7eba83d0-9717-482f-8957-c2343f598a3d/image.jpeg" height="400"/></p>

# <p align="center">1 - Abstract</p>
Automated composition of computer programs has been a standing challenge since the early days of artificial intelligence, and has no clear solutions with modern-day research in deep learning. Indeed, modeling latent context representations in language proves to be a difficult task singularly, and combined with applying structured procedural knowledge in a generative fashion quickly becomes intractable for complex domain specific languages. There is a clear need for research in devising efficiently learned combinations of these independent problems. We investigate the nature of such an architecture, and determine the current assumptions and limitations made by previous attempts at neural program synthesis. From our observations, we propose an architecture using the recent Generative Adversarial Networks and Hyper Networks to generate syntactically correct and behaviorally useful python source code from input and output examples, pseudocode, or even erroneous existing code.

# <p align="center">2 - Introduction</p>
## Motivation for our Research.
There a number of reasons outside of  the scientific domain that motivate research in the problem of neural program synthesis. The most obvious motivator is the ability reduce the amount of time required to develop large programs that make heavy use of APIs for file conversion, handling http requests, and even data analytics. An amount of time saved by developers in a company has a proportional amount of income to be saved. Multiplied across dozens of software developers in a single firm, savings figures quickly measure in the hundreds of thousands over the course of a single business year. A second obvious motivator is a reduction in the knowledge barrier for scientific computing, which can be argued to increase innovation trends and education standards, which induce higher productivity in research and industry. From a more theoretical perspective, the difference between neural program synthesis, and neural question answering is a matter of dataset choice. Our research will bring deeper insight to general natural language understanding.

## Problem Setting.
In our investigation, we focus on building a generative machine learning model for automatic synthesis of python source code.  The operands to this model are some high dimensional vector description of the desired program behavior. We make no assumptions regarding the encoding structure of this vector, other than assuming a correlation with latent program behavior. From this representation, our model shall produce executable python source code that matches our initial program description. Having identified the learning space for our investigation, we begin to decompose this space into compartmentalized subspaces that have more tractable representations. We then consider what sort of knowledge is necessary to develop computer programs: an understanding of program syntax, and a prediction of program behavior. In our investigation, we shall explicitly model these properties; however, in other more commonly used methods, one or both of these properties is assumed to be latent. Our hypothesis is that explicit models for contextual properties within data distributions allows for more robust generative models for those distributions.

# <p align="center">3 - Related Work</p>
## Neural Program Synthesis.
This approach to learning programs will produce a sequence of tokens representing executable source code using high level base functions, or APIs. These Methods have been commonly used previously to interact with high level APIs of program such as microsoft excel to learn--for example--how to filter and transform a list of grid cells from user examples. In this implementation, the recurrent neural networks outputs a sequence of function tokens that determine the order in which base functions are applied to grid cell content. This approach is the most effective for problems that have a limited number of base functions or operations. For generalizing the composition of python source code, base functions would have to be encoded as single characters, and this approach would become a naive end-to-end code generator, which has not been successful in past research.

## Neural Program Induction.
This approach to learning programs differs from program synthesis in that there is no explicit representation of base functions, and there are no human observable function tokens that can be executed by an interpreter. Instead, this method assumes a neural network will learn the latent procedure of an algorithm in an end-to-end manner. These methods include a large number of memory networks invented in the past few years. In practice, these algorithms are able to learn simple procedural tasks such as binary addition, and sorting, but are not able to learn more complex functions including recursion. These methods are also limited in that learning an update to the internal procedure of the network is nearly impossible, and requires a large amount of examples, and potentially rebuilding the model. For these models to generalize to the composition of high level programming languages the amount of data required for training is intractable, and has not been successfully attempted.

# <p align="center">4 - Human Perspective</p>
In order to fully realize the problems and assumptions made by prior approaches in this field, we consider the human approach to writing python source code. Engineering students at UC Berkeley are  given a text explanation of the intended behavior of a function, along with multiple input and output examples of that function. These students are able to consistently compose python code to solve arbitrarily complex tasks given this limited information. In such coding homework, these students are not explicitly given a table of syntax rules, so we can infer that sytnes rules have been learned implicitly by these students. In addition to learning syntax, in order to compose programs that accomplish a certain procedural scheme, these students must be able to predict the behavior of code internally while writing--otherwise solving homework would require trial-and-error, and would be intractable to solve.

# <p align="center">5 - Problems and Assumptions</p>
The primary problem existing with programming by example methods for neural program induction is that the required amount of different examples for complex programs can become intractable quickly, and even more so when performing training on that data. Programming by example methods for neural program synthesis require significantly less data to generate functioning programs, however, the complexity of programs is limited by the number of accessible base functions. More recently, synthesis methods that allow for base functions to be learned explicitly solve this problem, but assume a latent representation of final program behavior, and of program syntax. We hypothesize that learning complex programs becomes an easier task when representations for both program behavior and program syntax are made explicit.

*<p align="center">Research Idea by <NAME>, with collaboration from <NAME> and <NAME></p>*
{'fraction_non_alphanumeric': 0.029272259113701043, 'fraction_numerical': 0.004201111261688576, 'mean_word_length': 5.277754147171416, 'pattern_counts': {'":': 0, '<': 39, '<?xml version=': 0, '>': 39, 'https://': 2, 'lorem ipsum': 0, 'www.': 2, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '3328901', 'n_tokens_mistral': 2926, 'n_tokens_neox': 2762, 'n_words': 2135}