I've seen too many cases when a student has completed a programming exercise (without cheating) but can't say, for example, what type of values a variable contains during program runtime (in case of Python programs).It looks like the ability to solve exercises should not be the only (or even main?) criteria for assessment.Do you know any research papers (or books or posts) that tackle this issue and/or recommend other assessment techniques / exercise types?
Students can solve programming exercises but not explain their solutions. What to do?
curriculum design;resource request;grading;assessments
If I understand your problem correctly, it's that students can create programs that behave correctly without understanding why they behave correctly. I assume that they do this by some combination of brute force trying things and SO search.Using program correctness for grading has the desirable qualities of being objective and automation friendly.So how could you keep these qualities while testing for actual understanding?  While I agree that good code structure, variable naming, and possibly comments can offer proof of understanding, they are all highly subjective.What I would do instead is construct exam exercises that are relatively small requirements changes to homework assignments that students have completed earlier, ideally at least 2 weeks earlier.  Then give each student access to the code (s)he submitted for the homework as a starting point for completing the exam exercises.  Tune the original homework size and the exam requirements changes so that a student who understands the code and has well-written code can complete the exam exercise quickly, but one that started from scratch would not likely be able to complete the work.This accomplishes 4 things:It solves your original problem of testing whether students understand the code they have written.It maintains the objectivity and automation friendliness of grading on program behavior.It rewards production of readable, maintainable code.It prepares students for the world of coding outside of a class environment, where the majority of time spent is not on isolated exercises, but extending an existing code base.Disclaimer: I work in industry and have no research to support this.