BACKGROUND

Various types of machine learning models can be trained to generate source code under a variety of circumstances. Some machine learning models may be trained to translate source code from one programming language to another. Other machine learning models may be trained to flag and/or fix potential issues with source code that might be missed by a compiler, such as security issues, logical issues, etc.

A significant challenge with these types of machine learning models is obtaining sufficient training data. For example, it may be difficult to obtain enough labeled pairs of source code snippets (e.g., that implement the same function in two different programming languages) to sufficiently train a machine learning model to perform its task (e.g., translating between the two different programming languages) accurately. Even when sufficient training data is available for supervised training, problems such as overfitting can occur. Unlike these machine learning models, humans are capable of detecting patterns in a relatively small number of examples. However, some programming tasks, such as migrating a large code base from one form to another, are cumbersome and/or time-consuming for humans to perform at scale.

SUMMARY

Implementations are described herein for using machine learning to perform a variety of tasks related to migration of source code based on a relatively small number (“few shots”) of demonstrations. For example, during inference, one or more demonstration tuples (e.g., pairs, triplets, etc.) that include a source code snippet before and after a planned source code migration may be used to condition a machine learning model, such as an autoregressive language model, to perform a task established by the demonstration tuple(s). Once conditioned based on these demonstration tuple(s), the machine learning model may be used to process yet-to-be-migrated source code in a manner consistent with the demonstration(s). Thus, for instance, if the demonstration(s) include examples of source code snippets being converted from C to C++, then the yet-to-be-migrated source code may likewise be converted from C to C++.

In some implementations, a user such as a programmer may provide these demonstrations explicitly, e.g., by typing source code text into a field. In other implementations, these demonstrations may be selected explicitly by the programmer, e.g., from a list. Alternatively, a programmer may provide a free-form natural language input that conveys the programmer's desire or intent (e.g., “I want to migrate source code from Python2 to Python3”), and demonstrations that sufficiently match the intent (e.g., are sufficiently semantically similar) may be selected from a library of existing demonstrations.

In various implementations, the machine learning model may be an autoregressive language model (e.g., trained to perform natural language processing, or “NLP”) such as a bidirectional encoder representations from transformers (BERT)-based model (also referred to herein as a “transformer” model). By conditioning such an autoregressive language model with demonstration(s), the autoregressive language model is effectively “primed” to perform a task established by the demonstration(s), e.g., by being more likely to select output candidates that are aligned with the demonstrated task.

For example, during conditioning the demonstrations may be processed using the autoregressive language model to generate intermediate embedding(s). These intermediate embeddings may then be provided as inputs (e.g., acting as a form of memory) for subsequent iterations of the autoregressive language model in which yet-to-be-migrated source code is processed. As a result of these intermediate embeddings being processed by the model in conjunction with the yet-to-be-migrated source code, candidate outputs of the model that are more closely aligned with the demonstrated task may be assigned greater probabilities than others. In particular, these higher-probability outputs may more closely resemble, semantically, source code transformations shown by the demonstrations.

In some implementations, the autoregressive language model may have been trained previously on one or more corpuses related specifically to computer programming, as opposed to general-purpose corpuses such as encyclopedias, newspapers, magazines, etc. These computer programming-related corpuses can include source code (e.g., multiple code bases in a variety of different programming languages) and natural language documentation about computer programming. Training the autoregressive language model specifically using computer-programming-related corpuses enables the model, upon conditioning with demonstrations pertinent to a source code migration, to more accurately generate post-migration source code.

Natural language computer programming documentation may include any natural language that describes source code itself, instructs readers how to create source code, and/or provides guidelines for how source code should be written to comport with, for instance, standards, desired styles, proprietary mandates, operational requirements, security requirements, and so forth. Natural language computer programming documentation may include, but is not limited to, educational literature such as textbooks, programming language guides or tutorials, inline comments in source code files, comments that are provided when a programmer commits revised source code to a version control system (VCS), and so forth.

In addition to improving its accuracy in generating post-migration source code, training the autoregressive language model based on natural language computer programming documentation enables the model to be used for other purposes. For example, the autoregressive language model, upon suitable conditioning, may be used to generate natural language snippets that describe source code (e.g., its purpose, function, intent, inner workings, etc.), and/or natural language snippets that describe an intent behind a source code migration (e.g., migrating from C to C++, upgrading source code for multi-thread performance, etc.). To this end, in some implementations, a natural language snippet that describes the source code (e.g., its function, purpose, or intent) and/or an intent behind the migration may be provided as part of one or more demonstrations. In some implementations, these generated natural language snippets may be added to the post-migration source code, e.g., as inline comments and/or as suggestions for comments to be submitted to a VCS.

In some implementations, the autoregressive language model may be used in multiple stages to generate intermediate output(s). For example, source code in a first programming language may be converted to an intermediate form, such as natural language describing how the source code operates, or a canonical programming language such as pseudocode. Then, the intermediate form may be converted to a second programming language. In some such implementations, translating from an intermediate form to the second programing language may result in less syntactical errors than translating directly from the first programming language to the second programming language, e.g., because conversion from the first programming language to the intermediate form reduces syntactical noise while leaving intact the semantic meaning/intent behind the source code.

In some implementations, a method implemented using one or more processors may include: conditioning an autoregressive language model based on one or more demonstration tuples, wherein one or more of the demonstration tuples includes a first version of a first source code snippet that exists prior to a planned migration and a second version of the first source code snippet that is desired after the planned migration, and wherein the autoregressive language model is trained on one or more corpuses of source code and one or more corpuses of natural language documentation on the subject of computer programming; processing a pre-migration version of a source code file based on the conditioned autoregressive language model; and based on the processing, generating a post-migration version of the source code file.

In various implementations, one or more of the demonstration tuples includes a third source code snippet, an example input for the third source code snippet, and a target output of the third source code snippet given the example input. In various implementations, in the second version of the first source code snippet, at least a first token may be transformed into a target nomenclature; and in the post-migration version of the source code file, at least a second token that is different than the first token may be transformed into the target nomenclature. In various implementations, the target nomenclature captures a desired coding style used by an entity or a desired coding style espoused by computer programming educational literature that is included in one or more of the corpuses of natural language documentation about computer programming.

In various implementations, the method may include receiving one or more of the demonstration tuples via textual input provided by a user. In various implementations, the method may include selecting one or more of the demonstration tuples from a library of existing demonstration tuples based on user input. In various implementations, the user input may include a free-form natural language input spoken or typed by a user, and the selecting is based on semantic similarity between the free-form natural language input and the selected one or more of the demonstration tuples.

In various implementations, one or more of the demonstration tuples may include a natural language snippet that describes the first source code snippet, and wherein the method includes, based on the processing, generating another natural language snippet that describes the source code file. In various implementations, the autoregressive language model may be trained exclusively on the one or more corpuses of source code and the one or more corpuses of natural language documentation on the subject of computer programming.

In some implementations, the method may further include: performing a semantic comparison of the pre-migration source code file and the post-migration source code file; and based on the semantic comparison: selecting another post-migration version of the source code file from a distribution generated by the autoregressive language model; or performing supervised training on the autoregressive language model based on the pre-migration and post-migration versions of the source code file.

In another aspect, a method may be implemented using one or more processors and may include: conditioning an autoregressive language model based on one or more demonstration tuples, wherein one or more of the demonstration tuples includes a first version of a first source code snippet that exists prior to a planned migration, an example input for the first source code snippet, and a target output of the first source code snippet given the example input, and wherein the autoregressive language model is trained exclusively on one or more corpuses of source code and one or more corpuses of natural language documentation on the subject of computer programming; processing a pre-migration version of a source code file based on the conditioned autoregressive language model; and based on the processing, generating a post-migration version of the source code file.

In addition, some implementations include one or more processors of one or more computing devices, where the one or more processors are operable to execute instructions stored in associated memory, and where the instructions are configured to cause performance of any of the aforementioned methods. Some implementations also include one or more non-transitory computer readable storage media storing computer instructions executable by one or more processors to perform any of the aforementioned methods.

It should be appreciated that all combinations of the foregoing concepts and additional concepts described in greater detail herein are contemplated as being part of the subject matter disclosed herein. For example, all combinations of claimed subject matter appearing at the end of this disclosure are contemplated as being part of the subject matter disclosed herein.

