<story><title>Wikimedia Foundation spending</title><url>https://en.wikipedia.org/wiki/Wikipedia:Wikipedia_Signpost/2017-02-27/Op-ed</url></story><parent_chain><item><author>rspeer</author><text>That gives you Wikitext encapsulated in XML. How do you get at the &lt;i&gt;content&lt;/i&gt; of the Wikitext?&lt;p&gt;I work on a Wikitext parser [1]. So do many other people, in different ways. Wikitext syntax is horrible and it mixes content and presentation indiscriminately (for example, it contains most of HTML as a sub-syntax).&lt;p&gt;The problem is basically unsolvable, as the result of parsing a Wiki page is defined only by a complete implementation of MediaWiki (with all its recursively-evaluated template pages, PHP code, and Lua code), but if you run that whole stack what you get in the end is HTML -- just the presentation, not the content you presumably wanted.&lt;p&gt;So people solve various pieces of the problem instead, creating approximate parsers that oversimplify various situations to meet their needs.&lt;p&gt;One of these solutions is DBPedia [2], but if you use DBPedia you have to watch out for the parts that are  false or misleading due to parse errors.&lt;p&gt;[1] &lt;a href=&quot;https:&amp;#x2F;&amp;#x2F;github.com&amp;#x2F;LuminosoInsight&amp;#x2F;wikiparsec&quot; rel=&quot;nofollow&quot;&gt;https:&amp;#x2F;&amp;#x2F;github.com&amp;#x2F;LuminosoInsight&amp;#x2F;wikiparsec&lt;/a&gt;&lt;p&gt;[2] &lt;a href=&quot;http:&amp;#x2F;&amp;#x2F;wiki.dbpedia.org&amp;#x2F;&quot; rel=&quot;nofollow&quot;&gt;http:&amp;#x2F;&amp;#x2F;wiki.dbpedia.org&amp;#x2F;&lt;/a&gt;</text></item><item><author>jacquesm</author><text>&amp;gt; and it&amp;#x27;s still hard to get any sort of machine readable data out of it.&lt;p&gt;Huh? &lt;a href=&quot;https:&amp;#x2F;&amp;#x2F;dumps.wikimedia.org&amp;#x2F;&quot; rel=&quot;nofollow&quot;&gt;https:&amp;#x2F;&amp;#x2F;dumps.wikimedia.org&amp;#x2F;&lt;/a&gt;&lt;p&gt;Doesn&amp;#x27;t that qualify?</text></item><item><author>avar</author><text>I was very actively involved in MediaWiki development &amp;amp; Wikimedia ops (less so though) in 2004-2006 back when IIRC there were just 1-4 paid Wikimedia employees.&lt;p&gt;It was a very different time, and the whole thing was run much more like a typical open source project.&lt;p&gt;I think the whole project has gone in completely the wrong direction since then. Wikipedia itself is still awesome, but what&amp;#x27;s not awesome is that the typical reader &amp;#x2F; contributor experience is pretty much the same as it was in 2005.&lt;p&gt;Moreover, because of the growing number of employees &amp;amp; need for revenue the foundation&amp;#x27;s main goal is still to host a centrally maintained site that must get your pageviews &amp;amp; donations directly.&lt;p&gt;The goal of Wikipedia should be to spread the content as far &amp;amp; wide as possible, the way OpenStreetMap operates is a much better model. Success should be measured as a function of how likely any given person is to see factually accurate content sourced from Wikipedia, and it shouldn&amp;#x27;t matter if they&amp;#x27;re viewing it on some third party site.&lt;p&gt;Instead it&amp;#x27;s still run as one massive monolithic website, and it&amp;#x27;s still hard to get any sort of machine readable data out of it. This IMO should have been the main focus of Wikimedia&amp;#x27;s development efforts.</text></item></parent_chain><comment><author>10165</author><text>&amp;quot;That gives you Wikitext encapsulated in XML.&amp;quot;&lt;p&gt;avar: &amp;quot;The goal of Wikipedia should be to spread the content as far &amp;amp; wide as possible, the way OpenStreetMap operates is a better model.&amp;quot;&lt;p&gt;I am confused.&lt;p&gt;Doesn&amp;#x27;t OSM data come encapsulated in XML or some binary format?&lt;p&gt;As for dispersion of content, I could have sworn I have seen Wikipedia content on non-Wikipedia websites.  Is there some restriction that prohibits this?&lt;p&gt;I have seen Wikipedia data offered in DNS TXT records as well.</text></comment>
<story><title>Wikimedia Foundation spending</title><url>https://en.wikipedia.org/wiki/Wikipedia:Wikipedia_Signpost/2017-02-27/Op-ed</url></story><parent_chain><item><author>rspeer</author><text>That gives you Wikitext encapsulated in XML. How do you get at the &lt;i&gt;content&lt;/i&gt; of the Wikitext?&lt;p&gt;I work on a Wikitext parser [1]. So do many other people, in different ways. Wikitext syntax is horrible and it mixes content and presentation indiscriminately (for example, it contains most of HTML as a sub-syntax).&lt;p&gt;The problem is basically unsolvable, as the result of parsing a Wiki page is defined only by a complete implementation of MediaWiki (with all its recursively-evaluated template pages, PHP code, and Lua code), but if you run that whole stack what you get in the end is HTML -- just the presentation, not the content you presumably wanted.&lt;p&gt;So people solve various pieces of the problem instead, creating approximate parsers that oversimplify various situations to meet their needs.&lt;p&gt;One of these solutions is DBPedia [2], but if you use DBPedia you have to watch out for the parts that are  false or misleading due to parse errors.&lt;p&gt;[1] &lt;a href=&quot;https:&amp;#x2F;&amp;#x2F;github.com&amp;#x2F;LuminosoInsight&amp;#x2F;wikiparsec&quot; rel=&quot;nofollow&quot;&gt;https:&amp;#x2F;&amp;#x2F;github.com&amp;#x2F;LuminosoInsight&amp;#x2F;wikiparsec&lt;/a&gt;&lt;p&gt;[2] &lt;a href=&quot;http:&amp;#x2F;&amp;#x2F;wiki.dbpedia.org&amp;#x2F;&quot; rel=&quot;nofollow&quot;&gt;http:&amp;#x2F;&amp;#x2F;wiki.dbpedia.org&amp;#x2F;&lt;/a&gt;</text></item><item><author>jacquesm</author><text>&amp;gt; and it&amp;#x27;s still hard to get any sort of machine readable data out of it.&lt;p&gt;Huh? &lt;a href=&quot;https:&amp;#x2F;&amp;#x2F;dumps.wikimedia.org&amp;#x2F;&quot; rel=&quot;nofollow&quot;&gt;https:&amp;#x2F;&amp;#x2F;dumps.wikimedia.org&amp;#x2F;&lt;/a&gt;&lt;p&gt;Doesn&amp;#x27;t that qualify?</text></item><item><author>avar</author><text>I was very actively involved in MediaWiki development &amp;amp; Wikimedia ops (less so though) in 2004-2006 back when IIRC there were just 1-4 paid Wikimedia employees.&lt;p&gt;It was a very different time, and the whole thing was run much more like a typical open source project.&lt;p&gt;I think the whole project has gone in completely the wrong direction since then. Wikipedia itself is still awesome, but what&amp;#x27;s not awesome is that the typical reader &amp;#x2F; contributor experience is pretty much the same as it was in 2005.&lt;p&gt;Moreover, because of the growing number of employees &amp;amp; need for revenue the foundation&amp;#x27;s main goal is still to host a centrally maintained site that must get your pageviews &amp;amp; donations directly.&lt;p&gt;The goal of Wikipedia should be to spread the content as far &amp;amp; wide as possible, the way OpenStreetMap operates is a much better model. Success should be measured as a function of how likely any given person is to see factually accurate content sourced from Wikipedia, and it shouldn&amp;#x27;t matter if they&amp;#x27;re viewing it on some third party site.&lt;p&gt;Instead it&amp;#x27;s still run as one massive monolithic website, and it&amp;#x27;s still hard to get any sort of machine readable data out of it. This IMO should have been the main focus of Wikimedia&amp;#x27;s development efforts.</text></item></parent_chain><comment><author>shshhdhs</author><text>Thanks for working on that! Didn&amp;#x27;t know it was so bad. The following is a possibly stupid idea, but I&amp;#x27;d like to hear your thoughts:&lt;p&gt;What if you just render the content into HTML and then &amp;quot;screen scraped&amp;quot; the text, and then convert into a more useful format (MarkDown, JSON, etc). Is that plausible?</text></comment>