user295: 
A kōan (公案?) (/ˈkoʊ.ɑːn/; Chinese: 公案; pinyin: gōng'àn; Korean: 공안 (kong'an); Vietnamese: công án) is a story, dialogue, question, or statement, (1) which is used in Zen practice to provoke the "great doubt" (2) and test a student's progress (3) in Zen practice. (4)

Kōan Wikipedia

What are these stories, dialogues, questions, or statements? Where do they come from? How are they formulated?
What is "great doubt"? What is the goal of provoking "great doubt"?
What is being tested? What are the possible responses? How does the response quantify progress?
What is the overall goal of kōans? 

Perhaps considering a few kōans it is possible to illustrate the process from choosing or formulating the story, question, statement, or dialog, inspiring "great doubt", and how this measures progress and ultimately relates to the goal kōans / "great doubt" is supposed to achieve.
Looking for an explanation based on any tradition using kōans including but not limited to zen or mahayana. Objective is to better understand the following content.

user10712: The original connotation of the term in Chinese pertains to public records. Koans are often accounts of pupils interacting with masters, or their expression which was part of such interaction, and is contained in such classics as "The Blue Cliff Record." By their character and what has been preserved within them, they are thought to catalyze a shift in consciousness toward waking up in those who contemplate them during meditation and labours. Formal ceremonies, whereby pupils assigned such koans discuss them with their teacher and within the brief exchange disclose glimmerings of insight, have become part of specialized Buddhist tradition, particularly amongst monks in Rinzai Zen.
The problem of embarking on any great adventure of humility is that it takes an effort of ego to initiate. The great mystery of waking up is that at some point an effort must be made, and then, in its aftermath, a collapse. One of the essential qualities of Zen Buddhism is its emphasis on arationality, transconceptual process, or transcendent insight. These koans, acting as rocks for a nutcracker, become a catalyst around which awakening might emerge, and in part due to the arising of this collapse, a return to unknowing (great doubt).
In the context of the interview with the master, nothing is being tested, though the ceremony may give the impression of this having occurred. There is no real limit to possible responses, though social constraints and protocols will have a bearing on the outcome. Since there is no clear, predetermined response requested or specified, it becomes the role of the teacher to act as an authoritative 'spotter', inasmuch as all are monks present but some have more experience at recognizing the qualities of realization.
There is a trope that koans are riddles, puzzles, with particular solutions, and some of them are even presented in such a fashion, recursively inquiring as to the rationale of some simple, ambiguous, or confusing remark. Yet koans are only incidentally instructive or quizzing by character, and the response cannot, because of the very human and artistic form of that which they are part, quantify anything. Instead, progress is always qualified and conditioned.

user-1: I'm going to try my best and not give too much away, but unless I use a specific example, I'm not sure how else to proceed.  Let's use:

Mumonkan - Case 4: The Western Barbarian With No Beard
The Case:
Wakuan said, "Why has the Western Barbarian no beard?"
Mumon's Comment:
Study should be real study, enlightenment should be real
enlightenment. You should once meet this barbarian directly to be
really intimate with him. But saying you are really intimate with him
already divides you into two.
Mumon's Verse:
Don't discuss your dream
Before a fool.
Barbarian with no beard
Obscures the clarity.

First, some preliminaries.  So what is a koan?  Koans are usually a secondhand account of an exchange between a master and a student or between two enlightened masters.  Read as literature, these exchanges were either teaching moments or a test of another's quality of experience and/or current state of mind.  Practically speaking, koans are teaching tools.  Every koan points to something.  Sometimes it's an aspect of the enlightened mind (like in Gutei Raises a Finger or The Buddha Holds Up a Flower), other times it can be some psychological hang up like in The Western Barbarian Has No Beard or in The Golden Carp Falls Out of the Net. There are also what are known as break through koans that are designed to mess your head up completely to really advance your practice.  These would include things like Mu or Nansen Cuts the Cat in Two.  (I think Hakuin breaks koans into five classes, but this is the general gist.)  Like a legal precedent (koan literally means "legal case"), koans are records of past insights that can help a student form a basis of enlightened understanding (or supramundane knowledge as the Theravada would have it).
Each koan is broken into two basic parts.  First, you have the case (the quoted text above).  To go with it, there is what is known as the "turning phrase" (hua tou in Chinese) that express the essence of the koan.  The hua tou is given to you by your teacher when you begin to work with a particular case.  In meditation, you simple say that phrase on your out breath.  In the case above, on every exhalation you'd just say "no beard".
Koans are an insight practice.  In order to work with them effectively, you first have to develop some mastery over working with mushin or emptiness.  In Theravada, this would roughly equate to an extremely deep state of access concentration.  We Zen folk believe, much like you Elder folks, that unless the mind is quelled, there is no way for insight to occur.  It would be an exercise in futility to ruminate intellectually over a koan although this remains a problem that many students run into.  To avoid that and guide the student toward an experience of emptiness (e.g. deep access concentration), we use the mu koan to tame the mind (you can read more about that here).
Once mu or emptiness is established, the student can then start working with other koans.  Working with a koan can be hell or heaven.  If you've practiced jhana meditation at all, you know how oppressive deep concentration can sometimes feel.  In the case of our example, saying "no beard", "no beard" to yourself, sincerely, on every out breath, with no intellectualizing over what "no beard" means, gives rise to a very deep experience of emptiness.  The Mu koan equates this with "swallow[ing] a red-hot iron ball, which you cannot spit out even if you try."  This is the Great Doubt.  It's important to distinguish this kind of Doubt from the small doubt you'd have of, say, the Falcons winning the Superbowl. The Great Doubt is visceral and is the product of a full suspension of intellectual examination and the concentration of the mind in a state of emptiness.  And it can suck.  (That being said, sometimes piti can arise and become so menacingly strong that it can sweep you right into the 1st jhana.  But still, you say "no beard", "no beard".  Sometimes you'll be able to relax into the 2nd jhana, other times you go right back to the suck.  But always - "no beard", "no beard".)
So that's the meditation side of things.  Now we come out of meditation with our minds made calm and apply it to our Barbarian.  First, some context.  The Western Barbarian is, of course, Bodhidharma.  He has a beard.  Everybody knows that.  To say he doesn't is like denying the 800 lbs. gorilla in the room who is throwing feces at you while ripping up your first edition copy of The Fellowship of the Ring.  This simian is an asshole and you really need to acknowledge him.  As a student works with "no beard" things are going to come up (we are yogacara peeps so we'd say that they arise from the alaya-vijnana or store house consciousness).  For this koan, what arises will probably be something about your psychological makeup or habitual patterning that you'd rather sweep under the rug.  This koan says no, no sir.  Tell me what's there.  You leverage emptiness to pry something out of your psyche.
I would actually say it's unwise to try to use koans to quantify progress.  Some people can answer hundreds of koans and not reach enlightenment.  Other can get there with just a handful.  It's kind of like a game of psychological Jenga.  There are pieces that can be taken quite easily at no risk to our mundane selves.  You'll answer those very rapidly.  Other pieces - danger!  These can take longer to dislodge, but when you pull them out the entire structure can come toppling over.  Your teacher is basically like the foreman of a wrecking crew.  It falls on his or her experience to select those koans that will cause your little Jenga tower to come down in the least amount of moves.  Koans can also be used to prod or gauge a student's mind.  If you are having trouble with a particular koan, it can help point to those areas where your teacher might want to attack.  In post-enlightenment practice, they also help demonstrate where you are still blocked.
Koans really aren't all that mystical.  They are actually very practical in their purpose and execution.  I hope that gave you at least some inkling into how they "work".

295: <blockquote>
  <p>A kōan (公案?) (/ˈkoʊ.ɑːn/; Chinese: 公案; pinyin: gōng'àn; Korean: 공안 (kong'an); Vietnamese: công án) is a <strong><em>story, dialogue, question, or statement,</em></strong> (1) which is used in Zen practice to <strong><em>provoke the "great doubt"</em></strong> (2) and <strong><em>test a student's progress</em></strong> (3) in Zen practice. (4)</p>
</blockquote>

<p><a href="https://en.wikipedia.org/wiki/K%C5%8Dan" rel="nofollow noreferrer">Kōan Wikipedia</a></p>

<ol>
<li>What are these stories, dialogues, questions, or statements? Where do they come from? How are they formulated?</li>
<li>What is "great doubt"? What is the goal of provoking "great doubt"?</li>
<li>What is being tested? What are the possible responses? How does the response quantify progress?</li>
<li>What is the overall goal of kōans? </li>
</ol>

<p>Perhaps considering a few kōans it is possible to illustrate the process from choosing or formulating the story, question, statement, or dialog, inspiring "great doubt", and how this measures progress and ultimately relates to the goal kōans / "great doubt" is supposed to achieve.</p>

<p>Looking for an explanation based on any tradition using kōans including but not limited to <a href="/questions/tagged/zen" class="post-tag" title="show questions tagged &#39;zen&#39;" rel="tag">zen</a> or <a href="/questions/tagged/mahayana" class="post-tag" title="show questions tagged &#39;mahayana&#39;" rel="tag">mahayana</a>. Objective is to better understand the <a href="https://buddhism.stackexchange.com/search?q=koan">following content</a>.</p>
 10712: <p>The original connotation of the term in Chinese pertains to public records. Koans are often accounts of pupils interacting with masters, or their expression which was part of such interaction, and is contained in such classics as "The Blue Cliff Record." By their character and what has been preserved within them, they are thought to catalyze a shift in consciousness toward waking up in those who contemplate them during meditation and labours. Formal ceremonies, whereby pupils assigned such koans discuss them with their teacher and within the brief exchange disclose glimmerings of insight, have become part of specialized Buddhist tradition, particularly amongst monks in Rinzai Zen.</p>

<p>The problem of embarking on any great adventure of humility is that it takes an effort of ego to initiate. The great mystery of waking up is that at some point an effort must be made, and then, in its aftermath, a collapse. One of the essential qualities of Zen Buddhism is its emphasis on arationality, transconceptual process, or transcendent insight. These koans, acting as rocks for a nutcracker, become a catalyst around which awakening might emerge, and in part due to the arising of this collapse, a return to unknowing (great doubt).</p>

<p>In the context of the interview with the master, nothing is being tested, though the ceremony may give the impression of this having occurred. There is no real limit to possible responses, though social constraints and protocols will have a bearing on the outcome. Since there is no clear, predetermined response requested or specified, it becomes the role of the teacher to act as an authoritative 'spotter', inasmuch as all are monks present but some have more experience at recognizing the qualities of realization.</p>

<p>There is a trope that koans are riddles, puzzles, with particular solutions, and some of them are even presented in such a fashion, recursively inquiring as to the rationale of some simple, ambiguous, or confusing remark. Yet koans are only incidentally instructive or quizzing by character, and the response cannot, because of the very human and artistic form of that which they are part, quantify anything. Instead, progress is always qualified and conditioned.</p>
 -1: <p>I'm going to try my best and not give too much away, but unless I use a specific example, I'm not sure how else to proceed.  Let's use:</p>
<blockquote>
<p>Mumonkan - Case 4: The Western Barbarian With No Beard</p>
<p>The Case:</p>
<p>Wakuan said, &quot;Why has the Western Barbarian no beard?&quot;</p>
<p>Mumon's Comment:</p>
<p>Study should be real study, enlightenment should be real
enlightenment. You should once meet this barbarian directly to be
really intimate with him. But saying you are really intimate with him
already divides you into two.</p>
<p>Mumon's Verse:</p>
<p>Don't discuss your dream<br />
Before a fool.<br />
Barbarian with no beard
<br />Obscures the clarity.</p>
</blockquote>
<p>First, some preliminaries.  So what is a koan?  Koans are usually a secondhand account of an exchange between a master and a student or between two enlightened masters.  Read as literature, these exchanges were either teaching moments or a test of another's quality of experience and/or current state of mind.  Practically speaking, koans are teaching tools.  Every koan points to something.  Sometimes it's an aspect of the enlightened mind (like in Gutei Raises a Finger or The Buddha Holds Up a Flower), other times it can be some psychological hang up like in The Western Barbarian Has No Beard or in The Golden Carp Falls Out of the Net. There are also what are known as break through koans that are designed <strike>to mess your head up completely</strike> to really advance your practice.  These would include things like Mu or Nansen Cuts the Cat in Two.  (I think Hakuin breaks koans into five classes, but this is the general gist.)  Like a legal precedent (koan literally means &quot;legal case&quot;), koans are records of past insights that can help a student form a basis of enlightened understanding (or supramundane knowledge as the Theravada would have it).</p>
<p>Each koan is broken into two basic parts.  First, you have the case (the quoted text above).  To go with it, there is what is known as the &quot;turning phrase&quot; (<em>hua tou</em> in Chinese) that express the essence of the koan.  The <em>hua tou</em> is given to you by your teacher when you begin to work with a particular case.  In meditation, you simple say that phrase on your out breath.  In the case above, on every exhalation you'd just say &quot;no beard&quot;.</p>
<p>Koans are an insight practice.  In order to work with them effectively, you first have to develop some mastery over working with mushin or emptiness.  In Theravada, this would roughly equate to an extremely deep state of access concentration.  We Zen folk believe, much like you Elder folks, that unless the mind is quelled, there is no way for insight to occur.  It would be an exercise in futility to ruminate intellectually over a koan although this remains a problem that many students run into.  To avoid that and guide the student toward an experience of emptiness (e.g. deep access concentration), we use the mu koan to tame the mind (you can read more about that <a href="https://buddhism.stackexchange.com/a/19092/698">here</a>).</p>
<p>Once mu or emptiness is established, the student can then start working with other koans.  Working with a koan can be hell or heaven.  If you've practiced jhana meditation at all, you know how oppressive deep concentration can sometimes feel.  In the case of our example, saying &quot;no beard&quot;, &quot;no beard&quot; to yourself, sincerely, on every out breath, with no intellectualizing over what &quot;no beard&quot; means, gives rise to a very deep experience of emptiness.  The Mu koan equates this with &quot;swallow[ing] a red-hot iron ball, which you cannot spit out even if you try.&quot;  This is the Great Doubt.  It's important to distinguish this kind of Doubt from the small doubt you'd have of, say, the Falcons winning the Superbowl. The Great Doubt is visceral and is the product of a full suspension of intellectual examination and the concentration of the mind in a state of emptiness.  And it can suck.  (That being said, sometimes piti can arise and become so menacingly strong that it can sweep you right into the 1st jhana.  But still, you say &quot;no beard&quot;, &quot;no beard&quot;.  Sometimes you'll be able to relax into the 2nd jhana, other times you go right back to the suck.  But always - &quot;no beard&quot;, &quot;no beard&quot;.)</p>
<p>So that's the meditation side of things.  Now we come out of meditation with our minds made calm and apply it to our Barbarian.  First, some context.  The Western Barbarian is, of course, Bodhidharma.  He has a beard.  Everybody knows that.  To say he doesn't is like denying the 800 lbs. gorilla in the room who is throwing feces at you while ripping up your first edition copy of The Fellowship of the Ring.  This simian is an asshole and you really need to acknowledge him.  As a student works with &quot;no beard&quot; things are going to come up (we are yogacara peeps so we'd say that they arise from the alaya-vijnana or store house consciousness).  For this koan, what arises will probably be something about your psychological makeup or habitual patterning that you'd rather sweep under the rug.  This koan says no, no sir.  Tell me what's there.  You leverage emptiness to pry something out of your psyche.</p>
<p>I would actually say it's unwise to try to use koans to quantify progress.  Some people can answer hundreds of koans and not reach enlightenment.  Other can get there with just a handful.  It's kind of like a game of psychological Jenga.  There are pieces that can be taken quite easily at no risk to our mundane selves.  You'll answer those very rapidly.  Other pieces - danger!  These can take longer to dislodge, but when you pull them out the entire structure can come toppling over.  Your teacher is basically like the foreman of a wrecking crew.  It falls on his or her experience to select those koans that will cause your little Jenga tower to come down in the least amount of moves.  Koans can also be used to prod or gauge a student's mind.  If you are having trouble with a particular koan, it can help point to those areas where your teacher might want to attack.  In post-enlightenment practice, they also help demonstrate where you are still blocked.</p>
<p>Koans really aren't all that mystical.  They are actually very practical in their purpose and execution.  I hope that gave you at least some inkling into how they &quot;work&quot;.</p>
