task_categories:text-generation size_categories:100K<n<1M language:en code region:us
---
task_categories:
- text-generation
language:
- en
tags:
- code
pretty_name: instruct_coder
size_categories:
- 100K<n<1M
---



<div align="center">
    <img src="https://github.com/Happylkx/InstructCoder/raw/main/docs/logo.png">
</div>

<div align="center">
    <a href="https://github.com/qishenghu/CodeInstruct/blob/main/CodeInstruct.pdf">Paper</a> |
    <a href="https://github.com/qishenghu/CodeInstruct">Code</a> |
    <a href="https://happylkx.github.io/InstructCoder/">Blog</a>
<!--     <a href="https://blog.nus.edu.sg/kaixinli/2023/05/23/codeinstruct/">Blog</a> -->
</div>


<!-- | [Checkpoints](link_to_checkpoints) -->

# InstructCoder (CodeInstruct): Empowering Language Models to Edit Code

## Updates
- May 23, 2023: Paper, code and data released.

## Overview
InstructCoder is the first dataset designed to adapt LLMs for general code editing. It consists of 114,239 instruction-input-output triplets and covers multiple distinct code editing scenarios, generated by ChatGPT. LLaMA-33B finetuned on InstructCoder performs on par with ChatGPT on a real-world test set derived from GitHub commits.

![](demo.png)

In the ever-evolving world of software development, efficient and effective code editing plays a pivotal role. As the demands for faster development cycles and increased productivity continue to rise, harnessing the power of Large Language Models in code editing has emerged as a game-changer. This project aims to revolutionize code editing, empowering developers with intelligent tools and techniques.

Though the community has studied extensively on coding with LLMs, our experiments demonstrate that current LLMs still struggle for code editing. To this end, we curate a code editing dataset, dubbed InstructCoder, for improving and evaluating code editing abilities of LLMs. InstructCoder is an instructional dataset containing diverse code-editing tasks, covering 20 scenarios where code editing may be performed. [The dataset is released here.](https://github.com/Happylkx/InstructCoder) Our empirical experiments reveal that open-source LLMs display notable gains in code editing abilities after finetuning on InstructCoder. Some qualitative examples are provided.

## Data Collection
InstructCoder is systematically collected through an iterative process. Initial seed tasks are selected from GitHub commits. Inspired by Self-Instruct, we prompt ChatGPT to bootstrap new instructions. Then, plausible scenarios where the filtered instructions may be used are generated. Finally, corresponding code input and output are obtained conditioned on both the instruction and scenario. High-quality samples are manually selected and recurrently added to the task pool for further generation.

## Results
The largest model fine-tuned, LLaMA-33B, performs on par with ChatGPT, with 79.3% accuracy on the test set. Some qualitative examples generated by our model are shown here.

## Release
We are planning to release the following assets:

- [x] Full dataset
- [x] Source Code

<!-- The LoRA checkpoints are available [here](link_to_checkpoints). -->

## Qualitative Examples
In this section, we showcase three qualitative examples generated by the LLaMA-33B model.

1. **Instruction:** Transform the code into a class method called ‘calculate_distance’ that calculates the distance between two points in 3D space.

**Input code:**
```python
import math

def distance(point1, point2):
    x1, y1, z1 = point1
    x2, y2, z2 = point2
    dx = x2 - x1
    dy = y2 - y1
    dz = z2 - z1
    return math.sqrt(dx**2 + dy**2 + dz**2)

point1 = (0, 0, 0)
point2 = (1, 1, 1)
print(distance(point1, point2))
```

Output code:

```python
import math

class Point:
    def __init__(self, x, y, z):
        self.x = x
        self.y = y
        self.z = z

    def calculate_distance(self, other_point):
        dx = other_point.x - self.x
        dy = other_point.y - self.y
        dz = other_point.z - self.z
        return math.sqrt(dx**2 + dy**2 + dz**2)

point1 = Point(0, 0, 0)
point2 = Point(1, 1, 1)
print(point1.calculate_distance(point2))
```

2. Instruction: Integrate error handling into the code that performs batch processing of large datasets, including job failures or incomplete runs.
Input code:

```python
import pandas as pd

def process_orders(orders_file, shipping_file, inventory_file):
    orders = pd.read_csv(orders_file)
    shipping = pd.read_csv(shipping_file)
    inventory = pd.read_csv(inventory_file)

    # Perform data processing and analysis here...

    results = {"success": True, "message": "Batch processing completed successfully."}
    return results

orders_file = "orders.csv"
shipping_file = "shipping.csv"
inventory_file = "inventory.csv"

results = process_orders(orders_file, shipping_file, inventory_file)
print(results)
```

Output code:

```python
import pandas as pd
import logging

logging.basicConfig(filename='batch_processing.log', level=logging.ERROR)

def process_orders(orders_file, shipping_file, inventory_file):
    try:
        orders = pd.read_csv(orders_file)
        shipping = pd.read_csv(shipping_file)
        inventory = pd.read_csv(inventory_file)

        # Perform data processing and analysis here...

        results = {"success": True, "message": "Batch processing completed successfully."}
        return results
    except Exception as e:
        logging.error(f"Error processing batch: {e}")
        results = {"success": False, "message": "Batch processing failed. Please check the input files for any issues."}
        return results

orders_file = "orders.csv"
shipping_file = "shipping.csv"
inventory_file = "inventory.csv"

results = process_orders(orders_file, shipping_file, inventory_file)
print(results)
```

3. Instruction: Refactor any code that uses ‘exec’ or ‘eval’ for security reasons.
Input code:

```python
Copy code
game_data = {
    "score": 0,
    "level": 1
}

def execute_script(script):
    exec(script)

script = """
game_data["score"] += 10
if game_data["score"] >= 100:
    game_data["level"] += 1
"""

execute_script(script)
print(game_data)
```
Output code:

```python
game_data = {
    "score": 0,
    "level": 1
}

def execute_script(script):
    compiled_script = compile(script, "", "exec")
    exec(compiled_script, game_data)

script = """
game_data["score"] += 10
if game_data["score"] >= 100:
    game_data["level"] += 1
"""

execute_script(script)
print(game_data)
```


Now the script is not able to access the data in game_data.
## Limitations and Safety Concerns

By recognizing and addressing limitations and safety concerns, we hope developers can foster responsible AI deployment and use, promoting a more ethical, equitable, and secure future for large language models in code production.

1. **License:** You must ensure compliance with the corresponding licenses of the LLMs. It is important to thoroughly review the licenses and make sure that your usage is in accordance with their terms. For instance, you are not allowed to use LLaMA commercially, for it is released under a noncommercial license.

2. **Incomplete or Imperfect Knowledge:** LLMs are trained on vast amounts of data, which may not always be up-to-date or entirely accurate. For example, the APIs of a library may change over time. Consequently, the information provided by the models could be outdated, inaccurate, or even misleading in some instances.

3. **Overuse and Dependency:** Users might incorrectly interpret or rely too heavily on the outputs generated by large language models. It is crucial to provide proper guidance and promote an understanding of the model’s limitations, encouraging users to critically assess and verify the information or suggestions provided. Please make sure to check the generation of the models before using them.

Overreliance on large language models could lead to complacency, potentially causing users to undervalue human intelligence, such as creativity and critical thinking. We encourage users to use AI as a tool to supplement, rather than replace, human input and judgment.

4. **Malicious Use:** There is a risk that malicious actors might use the tools for nefarious purposes, such as generating malicious software. It is important to monitor the use and deployment of these models, track and report abuse, and develop countermeasures to address potential malicious activity.

5. **Bias and Discrimination:** Language models can inherit societal biases present in their training data, possibly leading to discriminatory or biased generations. Though our dataset is not likely to contain such toxic data, they may appear in the responses because of the base LLMs.

## Citation
Feel free to cite our work if you find it interesting or use the data:

```plain
@misc{2023instructcoder,
title={InstructCoder: Empowering Language Models to Edit Code},
url={https://github.com/qishenghu/CodeInstruct},
year = {2023},
publisher = {GitHub},
journal = {GitHub repository},
author={Hu, Qisheng and Li, Kaixin and Zhao, Xu and Xie, Yuxi and Liu, Tiedong and Chen, Hui and Xie, Qizhe and He, Junxian}}
```

## Conclusion
The integration of AI into code editing represents a significant milestone in the evolution of software development. By leveraging AI’s capabilities in understanding code semantics, patterns, and best practices, developers can unlock new levels of productivity, code quality, and efficiency. This project we’ve explored demonstrates the immense potential of intelligent code editing tools. As the software development landscape continues to evolve, embracing AI is poised to become a standard practice, and sets the stage for a future where developers can focus more on creativity and problem-solving, while AI handles the mundane aspects of coding.



