|
name: "CodeDebugCritic_Flow" |
|
description: |-2 |
|
Given a problem description and a candidate solution, test the code and provide useful feedback concerning the correctness of the solution and the potential mistakes. |
|
|
|
|
|
input_interface: |
|
- "problem_description" |
|
- "input_description" |
|
- "output_description" |
|
- "io_examples_and_explanation" |
|
- "public_tests_individual_io" |
|
- "code" |
|
|
|
|
|
output_interface: |
|
- "testing_results_summary" |
|
- "all_tests_passed" |
|
- "code_feedback" |
|
|
|
|
|
subflows_config: |
|
CodeTestingCritic: |
|
_target_: .CF_CodeTesting.instantiate_from_default_config |
|
CodeCriticWrongAttempt: |
|
_target_: .CF_CodeCriticWrongAttempt.instantiate_from_default_config |
|
|
|
topology: |
|
|
|
- goal: "Test the code on the public tests and provide a results summary." |
|
|
|
|
|
input_interface: |
|
_target_: aiflows.interfaces.KeyInterface |
|
additional_transformations: |
|
- _target_: aiflows.data_transformations.KeyMatchInput |
|
|
|
|
|
flow: CodeTestingCritic |
|
|
|
|
|
output_interface: |
|
_target_: aiflows.interfaces.KeyInterface |
|
additional_transformations: |
|
- _target_: .src.data_transformations.CorrectnessFlag |
|
input_key: "public_tests_results" |
|
output_key: "all_tests_passed" |
|
- _target_: .src.data_transformations.TestingResultsSummaryGeneration |
|
output_key: "testing_results_summary" |
|
|
|
single_test_error_message: True |
|
|
|
no_error_template: |2- |
|
${.issue_title} |
|
All of the executed tests passed. |
|
|
|
compilation_error_template: |2- |
|
${.issue_title} |
|
The execution resulted in a compilation error. |
|
## Compilation error message: |
|
{{error_message}} |
|
timeout_error_template: |2- |
|
${.issue_title} |
|
The execution timed out, the solution is not efficient enough. |
|
runtime_error_template: |2- |
|
${.issue_title} |
|
The execution resulted in a runtime error on the following test. |
|
## [Failed test] Input |
|
``` |
|
{{test_input}} |
|
``` |
|
## [Failed test] Runtime error message |
|
{{error_message}} |
|
single_test_error_template: |2- |
|
${.issue_title} |
|
The Python code does not solve the problem in the problem description due to logical errors. It fails the following test: |
|
## [Failed test] Input |
|
``` |
|
{{test_input}} |
|
``` |
|
## [Failed test] Expected output |
|
``` |
|
{{expected_output}} |
|
``` |
|
## [Failed test] Generated output |
|
``` |
|
{{generated_output}} |
|
``` |
|
all_tests_header: |2- |
|
${.issue_title} |
|
The Python code does not solve the problem in the problem description due to logical errors. It fails on the following tests. |
|
test_error_template: |2- |
|
## [Failed test {{idx}}] |
|
### [Failed test {{idx}}] Input |
|
``` |
|
{{test_input}} |
|
``` |
|
### [Failed test {{idx}}] Expected output |
|
``` |
|
{{expected_output}} |
|
``` |
|
### [Failed test {{idx}}] Generated output |
|
``` |
|
{{generated_output}} |
|
``` |
|
tests_separator: "\n\n" |
|
|
|
issue_title: "# Issue with the last proposed solution" |
|
|
|
|
|
- goal: "Generate feedback grounded in the test results summary." |
|
|
|
|
|
input_interface: |
|
_target_: aiflows.interfaces.KeyInterface |
|
additional_transformations: |
|
- _target_: aiflows.data_transformations.KeyMatchInput |
|
|
|
|
|
flow: CodeCriticWrongAttempt |
|
|
|
|
|
output_interface: |
|
_target_: aiflows.interfaces.KeyInterface |
|
additional_transformations: |
|
- _target_: aiflows.data_transformations.KeyRename |
|
old_key2new_key: |
|
api_output: "code_feedback" |
|
|
|
reset: true |
|
|
|
|
