<commit_before>.. _Using External Graders:

###########################
Using External Graders
###########################


.. _External Grader Overview:
*******************
Overview
*******************

An external grader is a service that receives student responses to a problem, processes those responses, and returns feedback and a problem grade to the edX platform. You build and deploy an external grader separately from the edX platform.

See the following sections for more information:

* :ref:`External Grader Example`
* :ref:`External Graders and XQueue`
* :ref:`The XQueue Interface`
* :ref:`Building an External Grader`
* :ref:`Create a Code Response Problem`

.. _External Grader Example:

***************************
External Grader Example
***************************

An external grader is particularly useful for software programming courses where students are asked to submit complex code.  The grader can run tests that you define on that code and return results to a student.

For example, you define a problem that requires students to submit Python code, and create a set of tests that an external grader can run to verify the submissions. When a student enters Python code for the problem and clicks **Check**, the code is sent to the grader for testing.  If the code passes all tests, the grader returns the score and a string indicating that the solution is correct.

.. image:: Images/external-grader-correct.png
 :alt: Image of a students view of a programming problem that uses an external grader, with a correct result 


The external grader can return a string with results, which the student can see by clicking **See full output**. This can be particularly useful when the solution is not correct and you want to return information about the failed tests. For example:

.. image:: Images/external-grader-incorrect.png
 :alt: Image of a students view of a programming problem that uses an external grader, with a correct result 

.. _External Graders and XQueue:

**************************************
External Graders and XQueue
**************************************

The edX Platform communicates with your external grader through XQueue.  XQueue provides students' input to the grader; it then receives results from the grader and returns them to students.  

The queue for your course must be set up in one of two modes:

*  **Pull**

*  **Push**

In most situations, edX recommends that you use Pull mode. Pull mode will prevent your grader from being overloaded by submissions it is not ready to process.

You determine which mode to use when you are building your course and your grader. You must communicate this decision to your edX Program Manager. The student experience is not affected by this decision.

==================
Pull Mode
==================

In Pull mode, student submissions are collected in XQueue, where they remain until the grader actively retrieves, or pulls, the next submission from the queue for grading.

The external grader polls the XQueue through a RESTful interface at a regular interval. When the external grader receives a submission, it runs the tests on it, then pushes the response back to XQueue through the RESTful interface. XQueue then delivers the response to the edX Learning Management System.

==================
Push Mode
==================

In Push mode, XQueue actively pushes student submissions to the external grader, which passively waits for the next submission to grade. When the external grader receives a submission, it runs the tests on it, then synchronously delivers the graded response back to the XQueue. XQueue then delivers the response to the edX Learning Management System.


============================
External Grader Workflow
============================

The following steps show the complete process:

#. The student either enters code or attaches a file for a problem, then clicks Check.
#. XQueue either pushes the code to the external grader, or waits until the external grader pulls the code.
#. The external grader runs the tests that you created on the code.
#. The external grader returns the grade for the submission, as well as any results in a string, to XQueue. 
#. The XQueue delivers the results to the edX Learning Management System.
#. The student sees the problem results and the grade.


==================
The XQueue Name
==================

Your course will use a specific XQueue name. You use this name when creating problems in edX Studio. You get this name from your edX Program Manager. As edX hosts many XQueues for different courses, it is critical that you use the exact XQueue name in your problems, as described in the section :ref:`Create a Code Response Problem`. 


.. _The XQueue Interface:

**************************************
The XQueue Interface
**************************************

The student submission sent from XQueue to the grader, and the response sent from the grader to XQueue, are JSON objects, as described below.

======================================================
Inputs to the External Grader
======================================================

The grader receives student submissions as a JSON object with two keys:

* **student_response**: A string containing the student's code submission.  The string comes from either input the student enters in the edX Learning Management System or a file the student attaches.

* **grader_payload**: An optional string that you can specify when creating the problem. For more information, see the section :ref:`Create a Code Response Problem`.

For example::

 {
   "xqueue_body":
   "{
     "student_response": "def double(x):\n return 2*x\n", 
     "grader_payload": "problem_2"
    }"
 }

======================================================
External Grader Responses
======================================================
<commit_msg>Add info to external graders chapter<commit_after>.. _Using External Graders:

###########################
Using External Graders
###########################


.. _External Grader Overview:
*******************
Overview
*******************

An external grader is a service that receives student responses to a problem, processes those responses, and returns feedback and a problem grade to the edX platform. You build and deploy an external grader separately from the edX platform.

See the following sections for more information:

* :ref:`External Grader Example`
* :ref:`External Graders and XQueue`
* :ref:`The XQueue Interface`
* :ref:`Building an External Grader`
* :ref:`Create a Code Response Problem`

.. _External Grader Example:

***************************
External Grader Example
***************************

An external grader is particularly useful for software programming courses where students are asked to submit complex code.  The grader can run tests that you define on that code and return results to a student.

For example, you define a problem that requires students to submit Python code, and create a set of tests that an external grader can run to verify the submissions. When a student enters Python code for the problem and clicks **Check**, the code is sent to the grader for testing.  If the code passes all tests, the grader returns the score and a string indicating that the solution is correct.

.. image:: Images/external-grader-correct.png
 :alt: Image of a students view of a programming problem that uses an external grader, with a correct result 


The external grader can return a string with results, which the student can see by clicking **See full output**. This can be particularly useful when the solution is not correct and you want to return information about the failed tests. For example:

.. image:: Images/external-grader-incorrect.png
 :alt: Image of a students view of a programming problem that uses an external grader, with a correct result 

.. _External Graders and XQueue:

**************************************
External Graders and XQueue
**************************************

The edX Platform communicates with your external grader through XQueue.  XQueue provides students' input to the grader; it then receives results from the grader and returns them to students.  

The queue for your course must be set up in one of two modes:

*  **Pull**

*  **Push**

In most situations, edX recommends that you use Pull mode. Pull mode will prevent your grader from being overloaded by submissions it is not ready to process.

You determine which mode to use when you are building your course and your grader. You must communicate this decision to your edX Program Manager. The student experience is not affected by this decision.

==================
Pull Mode
==================

In Pull mode, student submissions are collected in XQueue, where they remain until the grader actively retrieves, or pulls, the next submission from the queue for grading.

The external grader polls the XQueue through a RESTful interface at a regular interval. When the external grader receives a submission, it runs the tests on it, then pushes the response back to XQueue through the RESTful interface. XQueue then delivers the response to the edX Learning Management System.

For example code of an external grader that uses Pull mode, see the `Stanford-Online repository xqueue_pull_ref <https://github.com/Stanford-Online/xqueue_pull_ref>`_.

==================
Push Mode
==================

In Push mode, XQueue actively pushes student submissions to the external grader, which passively waits for the next submission to grade. When the external grader receives a submission, it runs the tests on it, then synchronously delivers the graded response back to the XQueue. XQueue then delivers the response to the edX Learning Management System.

For example code of an external grader that uses Push mode, see the `edX repository xserver <https://github.com/edx/xserver>`_.


============================
External Grader Workflow
============================

The following steps show the complete process:

#. The student either enters code or attaches a file for a problem, then clicks Check.
#. XQueue either pushes the code to the external grader, or waits until the external grader pulls the code.
#. The external grader runs the tests that you created on the code.
#. The external grader returns the grade for the submission, as well as any results in a string, to XQueue. 
#. The XQueue delivers the results to the edX Learning Management System.
#. The student sees the problem results and the grade.


==================
The XQueue Name
==================

Your course will use a specific XQueue name. You use this name when creating problems in edX Studio. You get this name from your edX Program Manager. As edX hosts many XQueues for different courses, it is critical that you use the exact XQueue name in your problems, as described in the section :ref:`Create a Code Response Problem`. 


.. _The XQueue Interface:

**************************************
The XQueue Interface
**************************************

The student submission sent from XQueue to the grader, and the response sent from the grader to XQueue, are JSON objects, as described below.

.. note:: XQueue does not send the the student ID to the external grader. Your grader cannot access student IDs or associate student IDs with submissions.

For the code for the XQueue interface, see the file `urls.py in the edX XQueue repository <https://github.com/edx/xqueue/blob/master/queue/urls.py>`_.

======================================================
Inputs to the External Grader
======================================================

The grader receives student submissions as a JSON object with two keys:

* **student_response**: A string containing the student's code submission.  The string comes from either input the student enters in the edX Learning Management System or a file the student attaches.

* **grader_payload**: An optional string that you can specify when creating the problem. For more information, see the section :ref:`Create a Code Response Problem`.

For example::

 {
   "xqueue_body":
   "{
     "student_response": "def double(x):\n return 2*x\n", 
     "grader_payload": "problem_2"
    }"
 }

======================================================
External Grader Responses
======================================================

{'fraction_non_alphanumeric': 0.14909650757637144, 'fraction_numerical': 0.00032412284255732923, 'mean_word_length': 4.919424460431655, 'pattern_counts': {'":': 6, '<': 6, '<?xml version=': 0, '>': 6, 'https://': 3, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '131632', 'n_tokens_mistral': 2875, 'n_tokens_neox': 2709, 'n_words': 1656}