File size: 3,022 Bytes
d26280a |
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 |
# Roadmap
<img width="800" alt="image" src="https://github.com/gpt-engineer-org/gpt-engineer/assets/4467025/1af9a457-1ed8-4ab3-90e6-4fa52bbb8c09">
There are three main milestones we believe will greatly increase gpt-engineer's reliability and capability:
- [x] Continuous evaluation of our progress 🎉
- [ ] Test code and fix errors with LLMs
- [ ] Make code generation become small, verifiable steps
## Our current focus:
- [x] **Continuous evaluation of progress 🎉**
- [x] Create a step that asks “did it run/work/perfect” in the end of each run [#240](https://github.com/gpt-engineer-org/gpt-engineer/issues/240) 🎉
- [x] Collect a dataset for gpt engineer to learn from, by storing code generation runs 🎉
- [ ] Run the benchmark multiple times, and document the results for the different "step configs" [#239](https://github.com/gpt-engineer-org/gpt-engineer/issues/239)
- [ ] Improve the default config based on results
- [ ] **Self healing code**
- [ ] Run the generated tests
- [ ] Feed the results of failing tests back into LLM and ask it to fix the code
- [ ] **Let human give feedback**
- [ ] Ask human for what is not working as expected in a loop, and feed it into LLM to fix the code, until the human is happy
- [ ] **Improve existing projects**
- [ ] Decide on the "flow" for the CLI commands and where the project files are created
- [ ] Add an "improve code" command
- [ ] Benchmark capabilities against other tools
## Experimental research
This is not our current focus, but if you are interested in experimenting: Please
create a thread in Discord #general and share your intentions and your findings as you
go along. High impact examples:
- [ ] **Make code generation become small, verifiable steps**
- [ ] Ask GPT4 to decide how to sequence the entire generation, and do one
prompt for each subcomponent
- [ ] For each small part, generate tests for that subpart, and do the loop of running the tests for each part, feeding
results into GPT4, and let it edit the code until they pass
- [ ] **Ad hoc experiments**
- [ ] Try Microsoft guidance, and benchmark if this helps improve performance
- [ ] Dynamic planning: Let gpt-engineer plan which "steps" to carry out itself, depending on the
task, by giving it few shot example of what are usually "the right-sized steps" to carry
out for such projects
## Codebase improvements
By improving the codebase and developer ergonomics, we accelerate progress. Some examples:
- [ ] Set up automatic PR review for all PRs with e.g. Codium pr-agent
- [ ] LLM tests in CI: Run super small tests with GPT3.5 in CI, that check that simple code generation still works
# How you can help out
You can:
- Post a "design" as a google doc in our Discord and ask for feedback to address one of the items in the roadmap
- Submit PRs to address one of the items in the roadmap
- Do a review of someone else's PR and propose next steps (further review, merge, close)
Volunteer work in any of these will get acknowledged.
|