Microtask programming is a form of crowdsourcing for programming in which implementation work is decomposed into short, self-contained microtasks. Each microtask offers a specific goal (e.g., write a unit test) as well as all of the required context and environment support necessary to accomplish this goal. Key to microtasking is the choice of workflow, which delineates the microtasks developers may complete and how contributions from each are aggregated to generate the final software product. Existing approaches either rely on a single developer to manually generate all microtasks, limiting their potential scalability, or impose coordination requirements which limit their effectiveness. Inspired by behavior-driven development, we describe a novel workflow for decomposing programming into microtasks in which each microtask involves identifying, testing, implementing, and debugging an individual behavior within a single function. We apply this approach to the implementation of microservices, demonstrating the first approach for implementing a microservice through microtasks. To evaluate our approach, we conducted a user study in which a small crowd worked to implement a simple microservice and test suite. We found that the crowd was able to use a behavior-driven microtask workflow to successfully complete 350 microtasks and implement 13 functions, quickly onboard and submit their first microtask in less than 24 minutes, contribute new behaviors in less than 5 minutes, and together implement a functioning microservice with only four defects. We discuss these findings and their implications for incorporating microtask work into open source projects.
Other work has explored preconfigured environments which enable teachers to manage a crowd of programming students. Codeopticon @cite_34 enables instructors to continuously monitor multiple students and help them code. OverCode @cite_28 and Foobaz @cite_19 help mentors to cluster student code submissions, enabling teachers to give feedback on clusters of submissions rather than individual submissions. Codepilot @cite_25 reduces the complexity of programming environments for novice programmers by integrating a preconfigured environment for real-time collaborative programming, testing, bug reporting, and version control into a single, simplified system.
@cite_28 @cite_19 @cite_34 @cite_25
2076771354 2179578783 2241837413 2611714150
In MOOCs, a single programming exercise may produce thousands of solutions from learners. Understanding solution variation is important for providing appropriate feedback to students at scale. The wide variation among these solutions can be a source of pedagogically valuable examples and can be used to refine the autograder for the exercise by exposing corner cases. We present OverCode, a system for visualizing and exploring thousands of programming solutions. OverCode uses both static and dynamic analysis to cluster similar solutions, and lets teachers further filter and cluster solutions based on different criteria. We evaluated OverCode against a nonclustering baseline in a within-subjects study with 24 teaching assistants and found that the OverCode interface allows teachers to more quickly develop a high-level view of students' understanding and misconceptions, and to provide feedback that is relevant to more students' solutions. Current traditional feedback methods, such as hand-grading student code for substance and style, are labor intensive and do not scale. We created a user interface that addresses feedback at scale for a particular and important aspect of code quality: variable names. We built this user interface on top of an existing back-end that distinguishes variables by their behavior in the program. Therefore our interface not only allows teachers to comment on poor variable names, they can comment on names that mislead the reader about the variable's role in the program. We ran two user studies in which 10 teachers and 6 students created and received feedback, respectively. The interface helped teachers give personalized variable name feedback on thousands of student solutions from an edX introductory programming MOOC. In the second study, students composed solutions to the same programming assignments and immediately received personalized quizzes composed by teachers in the previous user study. One-on-one tutoring from a human expert is an effective way for novices to overcome learning barriers in complex domains such as computer programming. But there are usually far fewer experts than learners. To enable a single expert to help more learners at once, we built Codeopticon, an interface that enables a programming tutor to monitor and chat with dozens of learners in real time. Each learner codes in a workspace that consists of an editor, compiler, and visual debugger. The tutor sees a real-time view of each learner's actions on a dashboard, with each learner's workspace summarized in a tile. At a glance, the tutor can see how learners are editing and debugging their code, and what errors they are encountering. The dashboard automatically reshuffles tiles so that the most active learners are always in the tutor's main field of view. When the tutor sees that a particular learner needs help, they can open an embedded chat window to start a one-on-one conversation. A user study showed that 8 first-time Codeopticon users successfully tutored anonymous learners from 54 countries in a naturalistic online setting. On average, in a 30-minute session, each tutor monitored 226 learners, started 12 conversations, exchanged 47 chats, and helped 2.4 learners. Novice programmers often have trouble installing, configuring, and managing disparate tools (e.g., version control systems, testing infrastructure, bug trackers) that are required to become productive in a modern collaborative software development environment. To lower the barriers to entry into software development, we created a prototype IDE for novices called CodePilot, which is, to our knowledge, the first attempt to integrate coding, testing, bug reporting, and version control management into a real-time collaborative system. CodePilot enables multiple users to connect to a web-based programming session and work together on several major phases of software development. An eight-subject exploratory user study found that first-time users of CodePilot spontaneously used it to assume roles such as developer tester and developer assistant when creating a web application together in pairs. Users felt that CodePilot could aid in scaffolding for novices, situational awareness, and lowering barriers to impromptu collaboration.