Middle-earth canon

The term Middle-earth canon, also called Tolkien's canon, is used for the published writings of J. R. R. Tolkien regarding Middle-earth as a whole. The term is also used in Tolkien fandom to promote, discuss and debate the idea of a consistent fictional canon within a given subset of Tolkien's writings.
The terms have been used by reviewers, publishers, scholars, authors and critics such as John Garth, Tom Shippey, Jane Chance and others to describe the published writings of J. R. R. Tolkien on Middle-earth as a whole. Other writers look to the entire body of work of the author as a "Tolkien canon", rather than a subset defined by the fictional "Middle-earth" setting.
Tolkien's works.
The works on Middle-earth published by Tolkien during his lifetime include "The Hobbit", "The Lord of the Rings", "The Adventures of Tom Bombadil", and "The Road Goes Ever On". After Tolkien's death his son Christopher published "The Silmarillion" with many textual changes to knit several mostly unfinished manuscripts together as a coherent narrative. Further posthumous publications (with text more closely following Tolkien's original) include "Unfinished Tales", "The Letters of J. R. R. Tolkien", "Bilbo's Last Song", "The Children of Húrin", "Beren and Luthien" and "The Fall of Gondolin".
Christopher Tolkien also published the 12-volume "History of Middle-earth", containing many texts, drafts, and notes by Tolkien (both early and late), together with Christopher's own extensive notes placing these in context.
Further works authorized by the Tolkien Estate include "The History of The Hobbit" in two volumes by John D. Rateliff and "The Annotated Hobbit" by Douglas A. Anderson, both with notes and early drafts by Tolkien. Linguistic material by Tolkien concerning Middle-earth has also been published with the permission of the Estate in two periodical publications. The Qenya and Gnomish Lexicons, in full, appear in "Parma Eldalamberon" Numbers 11–16; other mostly self-contained fragments, notes, and poems appear in various issues of "Vinyar Tengwar". All of this material together constitutes a collection which, much like real-world histories and mythologies, contains numerous points of obscurity, omission, or apparent contradiction.
"The Hobbit".
Although Tolkien said that "The Hobbit" was conceived separately from his mythological stories, early drafts show that it was set in that world, referring explicitly to characters and places which appeared in his "Book of Lost Tales" which would later become "The Silmarillion". The Necromancer was originally Thû, the precursor of Sauron; Thorin's grandfather was imprisoned in the same dungeons that held Beren and Lúthien; the Elven king was Thingol and his land Menegroth.
When he revised "The Hobbit" to bring the story of the finding of the Ring in line with "The Lord of the Rings", Tolkien retained the original version as the tale Bilbo told to justify his acquisition of the Ring.
"The Lord of the Rings".
"The Lord of the Rings" picks up the story of "The Hobbit" some sixty years later, with some of the same locations and characters. Tolkien now explicitly linked the story to the "Silmarillion" tales, but placed it some six thousand years later in time. This reframing made some details in "The Hobbit", such as the goblins' ready recognition of the ancient swords Orcrist and Glamdring, difficult to reconcile into a single history.
Other details from "The Hobbit" don't quite mesh with "The Lord of the Rings". Frodo and his companions, for example, cover much the same territory in the Trollshaws as Bilbo and the Dwarves, but take much longer to reach Rivendell, and the geography is described differently. Several adjustments to "The Hobbit" only increased the discrepancies; and in the 1960s, Tolkien began rewriting "The Hobbit" more in the style and tone of "The Lord of the Rings", adjusting the journey and landmarks to fit the later story, but ultimately abandoned the effort.
Writings after "The Lord of the Rings".
According to Christopher Tolkien, despite J. R. R. Tolkien's desire to bring the older "Silmarillion" stories to a publishable state, much time was spent instead trying to bring consistency to the works already published. The unpublished manuscripts were left in various states of completion. These older stories had existed and changed over many decades, partly in response to "The Lord of the Rings"; as he reworked the material, he made substantial changes, up to and including the abandonment of major themes and entire tales, and wholesale rewrites and revisions of otherwise seemingly complete narratives.
Towards the end of his life, the focus of Tolkien's writing shifted from story telling inspired by his philological pursuits to more philosophical concerns, and Tolkien never finished a unified, systematic, and internally consistent narrative.
The "Silmarillion" compilation.
"The Silmarillion" was compiled by Christopher Tolkien (long involved in J. R. R. Tolkien's creative process) and published in 1977, four years after Tolkien’s death. It presents an abridged cycle of Tolkien's drafts of his Elvish legends, drawing material from the earliest "Book of Lost Tales" to drafts written long after "The Lord of the Rings". Most of the original texts have subsequently appeared in the "History of Middle-earth". Christopher's goal was a version resembling what he thought at the time his father might have produced.
Christopher observed that absolute consistency among the Middle-earth tales could only be achieved by losing much that was good in them:
"a complete consistency (either within the compass of "The Silmarillion" itself or between "The Silmarillion" and other published writings of my father's) is not to be looked for, and could only be achieved, if at all, at heavy and needless cost."
He went on to say:
 "My father came to conceive "The Silmarillion" as a compilation... and it is to some extent a compendium in fact and not only in theory." 
Throughout his commentaries in "Unfinished Tales" and the twelve volumes of "The History of Middle-earth", Christopher Tolkien points out differences between various versions of the original texts and the final editorial selections and occasional alterations in "The Silmarillion". In the Introduction of "Unfinished Tales" he observes that such selection was necessary to publishing a unified narrative; but in some cases he later came to feel that he went too far, for example in the ruin of Doriath:
"I think now that this was a mistaken view, and that the undoubted difficulties could have been and should have been, surmounted without so far overstepping the bounds of editorial function." 
Editing for consistency can be seen by comparing the chapter "Of the Voyage of Eärendil" in "The Silmarillion" with its corresponding section in the "History of Middle-earth" Volume V ("The Lost Road and Other Writings"). The "Quenta Silmarillion" of the 1930s was Tolkien's final text for this section, and Christopher carried it forward into "The Silmarillion" nearly word for word with editorial modifications—for consistency with other works—primarily limited to nomenclature: Fionwë edited to Eönwë, Lindar to Vanyar, etc. For example:
From "The Silmarillion": "In the front of that fire came Glaurung the golden, father of dragons, in his full might; and in his train were Balrogs, and behind them came the black armies of the Orcs in multitudes such as the Noldor had never before seen or imagined." — p. 151.
From "The Lost Road": "In the front of that fire came Glómund the golden, the father of dragons, and in his train were Balrogs, and behind them came the black armies of the Orcs in multitudes such as the Gnomes had never before seen or imagined." — p. 280.
In the continuing development of the published history of Middle-earth, Christopher Tolkien quotes in "The Children of Húrin" his father's own words on his fictional universe:
"once upon a time... I had in mind to make a body of more or less connected legend... I would draw some of the great tales in fullness, and leave many only placed in the scheme, and sketched."
Christopher Tolkien offers this justification for exercising his editorial authority to produce "The Children of Húrin" as a separate book:
"...it has seemed to me that there was a good case for presenting my father's long version of the legend of the Children of Húrin as an independent work, between its own covers, with a minimum of editorial presence, and above all in continuous narrative without gaps or interruptions, if this could be done without distortion or invention, despite the unfinished state in which he left parts of it."
Ethan Gilsdorf reviewing "The Children of Húrin" wrote of the editorial function: "Of almost equal interest is Christopher Tolkien's task editing his father's abandoned projects. In his appendix, he explains his editorial process this way: "While I have had to introduce bridging passages here and there in the piecing together of different drafts, there is no element of extraneous 'invention' of any kind, however slight." He was criticized for having monkeyed with his father's text when putting "The Silmarillion" together. This pre-emptive strike must be meant to allay the fears of Tolkien's most persnickety readers."
Fictional canon for Middle-earth.
As a result of the manner of its creation, the secondary world of Middle-earth is complicated. Its creator developed various elements of his fiction over the course of decades, making substantial changes including the abandonment of major themes, facts and entire tales, and undertook wholesale rewrites and revisions of otherwise "complete" narratives. The author's opinions on the relationships of his texts to each other often changed. In his letters, Tolkien comments upon the intertextual relationships of his works:
"I am doubtful myself about the undertaking [of finishing "The Silmarillion"]. Part of the attraction of the L.R. ["The Lord of the Rings"] is, I think, due to the glimpses of a large history in the background: an attraction like that of viewing far off an unvisited island, or seeing the towers of a distant city gleaming in a sunlit mist. To go there is to destroy the magic, unless new unattainable vistas are again revealed."
The quest by some readers for a consistent fictional canon within some subset of Tolkien's writings was noted by Verlyn Flieger. Since the degree of narrative consistency that might be expected from a series of novels is not always found in Tolkien's work, Flieger attributed the need on the part of some readers to find consistency within the stories to the sense of reality that Tolkien strove to instil in his work, although the search for a definitive fictional canon has been seen as ultimately irrelevant to appreciation of his tales.
The desire for a Middle-earth canon arises from the need of some readers to form an internal consistency between the stories, a need related to their "willing suspension of disbelief". Tolkien, in his essay "On Fairy Stories", claimed that no individual fantasy story can be successful without maintaining an "inner consistency of reality". An author, he says:
"... makes a Secondary World which your mind can enter. Inside it, what he relates is 'true': it accords with the laws of that world. ... The moment disbelief arises, the spell is broken; the magic, or rather art, has failed."
W.H. Auden, former student of Tolkien, supports this notion in his review of one of Tolkien's books:"Of any imaginary world the reader demands that it seem real, and the standard of realism demanded today is much stricter than in the time, say, of Malory."
External links.
"These links are to Tolkien websites where Middle-earth canon issues are discussed by readers, fans, etc."