<commit_before><!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Transitional//EN" "http://www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd">
<html xmlns="http://www.w3.org/1999/xhtml">
 <head>
  <link href="static/screen.css" type="text/css" rel="stylesheet" />
  <link href="static/prettify.css" type="text/css" rel="stylesheet" />
  <script type="text/javascript" src="static/prettify.js"></script>
  <meta http-equiv="Content-Type" content="text/html; charset=utf-8" />
  <title>gpodder.net Feed-Service</title>
 </head>

 <body>
  <h1>gpodder.net Feed-Service</h1>

  <p>The gpodder.net Feed-Service is a webservice for parsing podcast feeds and
  returning the simplified contents as JSON.</p>

  <h2>Usage</h2>

  <h3>Examples</h3>
  <pre class="prettyprint">
   SERVER=http://mygpo-feedservice.appspot.com/parse
   curl --header "Accept: application/json" "$SERVER?<strong>url</strong>=http://feeds.feedburner.com/linuxoutlaws&amp;<strong>inline_logo</strong>=1&amp;<strong>scale_logo</strong>=30" #<a href="/parse?url=http://feeds.feedburner.com/linuxoutlaws&inline_logo=1&scale_logo=30" title="see HTML response">^</a>
   curl --header "Accept: application/json" "$SERVER?<strong>url</strong>=http://leo.am/podcasts/floss&amp;<strong>url</strong>=http://feeds.twit.tv/floss_video_large"</a> #<a href="/parse?url=http://leo.am/podcasts/floss&url=http://feeds.twit.tv/floss_video_large" title="see HTML response">^</a>
   curl --header "Accept: application/json" "$SERVER?<strong>url</strong>=http://www.dancarlin.com/cswdc.xml&amp;<strong>strip_html</strong>=1"</a> #<a href="/parse?url=http://www.dancarlin.com/cswdc.xml&strip_html=1" title="see HTML response">^</a>
   curl --header "Accept: application/json" -d "url=http://feeds.feedburner.com/linuxoutlaws" $SERVER #<a href="/parse?url=http://feeds.feedburner.com/linuxoutlaws" title="see HTML response">^</a></pre>

  <h3>Requests</h3>
  <p>Parameters to /parse (either GET or POST as application/x-www-form-urlencoded)
   <ul>
    <li><strong>url</strong>: The URL of the feed that should be parsed (required). This parameter can be repeated multiple times. The values can be URL-encoded.</li>
    <li><strong>inline_logo</strong>: If set to 1, the (unscaled) logos are included in the response as <a href="http://en.wikipedia.org/wiki/Data_URI_scheme">data URIs</a> (default 0).</li>
    <li><strong>scale_logo</strong>: If inline_logo is set to 1, scales the included logo down to the given size. The resulting image is fitted into a square with the given side-length. If the given size is greater than the original size, the image won't be scaled at all.</li>
    <li><strong>logo_format</strong>: If inline_logo is set to 1, the inlined image is converted to the specified format (either <em>png</em> or <em>jpeg</em>). If this option is not used, the original format is preserved.</li>
    <li><strong>strip_html</strong>: If set to 1, all HTML tags are removed from text fields. If your client is not able to display HTML, use this option to reduce bandwidth usage.</li>
    <li><strong>use_cache</strong>: Feeds are cached by the service according to the feed's caching headers. If use_cache is set to 1 (default) feeds are retrieved from the cache if possible. If set to 0, feeds are always fetched from their URL. Do not use 0 as a default value in your application.</li>
   </ul>
  </p>
  <p>Headers to /parse
   <ul>
    <li><a href="if-mod-since"></a><strong>If-Modified-Since</strong>: Time when all requested feeds have been accessed the last time. The response will only contain podcasts that have been modified in the meantime.</li>
    <li><strong>User-Agent</strong>: Clients should send a descriptive User-Agent string. In case of abuse of the service, misbehaving and/or generic user-agents might be blocked.</li>
    <li><a name="accept"></a><strong>Accept</strong>: Clients should send <em>Accept: application/json</em> to indicate that they are prepared to receive JSON data. If you send a different Accept header, you will receive a HTML formatted response.</li>
    <li><a name="accept-encoding"></a><strong>Accept-Encoding</strong>: Google App Engine gzips responses based on <em>Accept-Encoding</em> and <em>User-Agent</em> (<a href="http://code.google.com/appengine/docs/python/runtime.html#Responses">see documentation</a>). Include "gzip" in both headers to ensure gzip compression.</li>
   </ul>
  </p>

  <h3>Responses</h3>
  <ul>
   <li>Each response contains a list of feeds, at least one for each <em>url</em>-Parameter</li>
   <li>HTTP-Redirects are followed automatically (this is reflected in the <em>urls</em> field)</li>
   <li><a href="http://cyber.law.harvard.edu/rss/rssRedirect.html">RSS-Redirects</a> are followed by additionally including the new feed in the response</li>
   <li>Each feed contains
    <ul>
     <li><strong>title</strong>: the title of the feed</li>
     <li><strong>link</strong>: the feeds website</li>
     <li><strong>description</strong>: a description of the feed, potentially including HTML characters</li>
     <li><strong>author</strong>: the feed's author</li>
     <li><strong>language</strong>: the feed's language</li>
     <li><a name="urls" /><strong>urls</strong>: the redirect-chain of the URL passed in the <em>url</em> parameter. This can be used to match the requested URLs to the entries in the response.</li>
     <li><strong>new_location</strong>: the referred to location, if the feed uses <a href="http://cyber.law.harvard.edu/rss/rssRedirect.html">RSS-Redirects</a>. The new location will also be included in the response</li>
     <li><a name="logo" /><strong>logo</strong>: the URL of the feed's logo</li>
     <li><strong>logo_data</strong>: the feed's logo as a <a href="http://en.wikipedia.org/wiki/Data_URI_scheme">data URI</a>, if <em>inline_logo</em> has been used. To save bandwidth, the logo is not included if it changed since the date sent in <em><a href="#if-mod-since">If-Modified-Since</a></em></li>
     <li><strong>content_types</strong>: the content types of the feed, either audio, video or image</li>
     <li><strong>hub</strong>: the endpoint URL of the <a href="http://code.google.com/p/pubsubhubbub/">hub</a> through which the feed is published</li>
     <li><strong>errors</strong>: a dictionary of occured errors, where the key contains an <a href="#error-codes">error code</a> and the value a string representation.</li>
     <li><strong>http_last_modified</strong>: the Unix timestamp of the last modification of the feed (according to the HTTP header).</li>
     <li><strong>http_etag</strong>: the HTTP E-Tag of the feed</li>
     <li><strong>episodes</strong>: the list of episodes</li>
    </ul>
   </li>
   <li>Each episode contains
    <ul>
     <li><strong>guid</strong>: an unique endentifier for the episode (provided by the feed in the GUID property)</li>
     <li><strong>title</strong>: the title of the episode</li>
     <li><strong>short_title</strong>: the non-repetitive part of the episode title. If an episode number is found, it is also removed and provided separately.</li>
     <li><strong>number</strong>: the episode number which is parsed from the title</li>
     <li><strong>description</strong>: the description of the episode, potentially including HTML characters</li>
     <li><strong>link</strong>: the website link for the episode</li>
     <li><strong>released</strong>: the Unix timestamp of the episode's release</li>
     <li><strong>author</strong>: the episode's author</li>
     <li><strong>duration</strong>: the episode's duration in seconds</li>
     <li><strong>language</strong>: the episode's language</li>
     <li><strong>files</strong>: a list of all files linked by the episode. Each files is represented by an object containing url, filesize (in Bytes) and mimetype.</li>
    </ul>
   </li>
   <li><a name="error-codes" />Current Error Codes
    <ul>
     <li><strong>fetch-feed</strong>: The feed could not be retrieved. The URL is given in the <a href="#urls">urls</a> list</li>
     <li><strong>fetch-logo</strong>: The feed's logo could not be retrieved. Its URL is given in the <a href="#logo">logo</a> field</li>
     <li><strong>hub-subscription</strong>: An error occured while subscribing to the feed's hub for instant updates. This is generally not critical for the client</li>
    </ul>
   </li>
  </ul>
  <p>Headers
   <ul>
    <li><strong>Last-Modified</strong>: The timestamp of the latest change to any of the requested feeds. This value can be used in the If-Modified-Since parameter to subsequent requests. This header is not sent for the HTML formatted response.</li>
    <li><strong>Content-Type</strong>: <em>application/json</em> if your request contains <em>Accept: application/json</em>, otherwise the response will contain the HTML representation with <em>text/html</em>.</li>
    <li><strong>Content-Encoding</strong>: <em>gzip</em> if the response is compressed. See <em><a href="#accept-encoding">Accept-Encoding</a></em> for details.</li>
    <li><strong>Vary</strong>: Contains the request headers for which the response can vary. Currently this is <em>Accept, User-Agent, Accept-Encoding</em>.</li>
   </ul>
  </p>

  <h3>Using this Instance</h3>

  <p>If you intend to use <em>this instance</em> of the webservice in your
  application, please contact <a
  href="mailto:<EMAIL>"><EMAIL></a> beforehand.

  <h2>Source Code</h2>
  <p>The sourcecode is available from <a
  href="http://repo.or.cz/w/mygpo-feedservice.git">http://repo.or.cz/w/mygpo-feedservice.git</a>
  under the terms of the AGPLv3. Please report bugs on the <a href="https://bugs.gpodder.org/enter_bug.cgi?product=mygpo-feedservice">bugtracker</a> and mention the exact URL of your failed request.</p>


 </body>
</html>
<commit_msg>update repository location (now on GitHub)<commit_after><!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Transitional//EN" "http://www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd">
<html xmlns="http://www.w3.org/1999/xhtml">
 <head>
  <link href="static/screen.css" type="text/css" rel="stylesheet" />
  <link href="static/prettify.css" type="text/css" rel="stylesheet" />
  <script type="text/javascript" src="static/prettify.js"></script>
  <meta http-equiv="Content-Type" content="text/html; charset=utf-8" />
  <title>gpodder.net Feed-Service</title>
 </head>

 <body>
  <h1>gpodder.net Feed-Service</h1>

  <p>The gpodder.net Feed-Service is a webservice for parsing podcast feeds and
  returning the simplified contents as JSON.</p>

  <h2>Usage</h2>

  <h3>Examples</h3>
  <pre class="prettyprint">
   SERVER=http://mygpo-feedservice.appspot.com/parse
   curl --header "Accept: application/json" "$SERVER?<strong>url</strong>=http://feeds.feedburner.com/linuxoutlaws&amp;<strong>inline_logo</strong>=1&amp;<strong>scale_logo</strong>=30" #<a href="/parse?url=http://feeds.feedburner.com/linuxoutlaws&inline_logo=1&scale_logo=30" title="see HTML response">^</a>
   curl --header "Accept: application/json" "$SERVER?<strong>url</strong>=http://leo.am/podcasts/floss&amp;<strong>url</strong>=http://feeds.twit.tv/floss_video_large"</a> #<a href="/parse?url=http://leo.am/podcasts/floss&url=http://feeds.twit.tv/floss_video_large" title="see HTML response">^</a>
   curl --header "Accept: application/json" "$SERVER?<strong>url</strong>=http://www.dancarlin.com/cswdc.xml&amp;<strong>strip_html</strong>=1"</a> #<a href="/parse?url=http://www.dancarlin.com/cswdc.xml&strip_html=1" title="see HTML response">^</a>
   curl --header "Accept: application/json" -d "url=http://feeds.feedburner.com/linuxoutlaws" $SERVER #<a href="/parse?url=http://feeds.feedburner.com/linuxoutlaws" title="see HTML response">^</a></pre>

  <h3>Requests</h3>
  <p>Parameters to /parse (either GET or POST as application/x-www-form-urlencoded)
   <ul>
    <li><strong>url</strong>: The URL of the feed that should be parsed (required). This parameter can be repeated multiple times. The values can be URL-encoded.</li>
    <li><strong>inline_logo</strong>: If set to 1, the (unscaled) logos are included in the response as <a href="http://en.wikipedia.org/wiki/Data_URI_scheme">data URIs</a> (default 0).</li>
    <li><strong>scale_logo</strong>: If inline_logo is set to 1, scales the included logo down to the given size. The resulting image is fitted into a square with the given side-length. If the given size is greater than the original size, the image won't be scaled at all.</li>
    <li><strong>logo_format</strong>: If inline_logo is set to 1, the inlined image is converted to the specified format (either <em>png</em> or <em>jpeg</em>). If this option is not used, the original format is preserved.</li>
    <li><strong>strip_html</strong>: If set to 1, all HTML tags are removed from text fields. If your client is not able to display HTML, use this option to reduce bandwidth usage.</li>
    <li><strong>use_cache</strong>: Feeds are cached by the service according to the feed's caching headers. If use_cache is set to 1 (default) feeds are retrieved from the cache if possible. If set to 0, feeds are always fetched from their URL. Do not use 0 as a default value in your application.</li>
   </ul>
  </p>
  <p>Headers to /parse
   <ul>
    <li><a href="if-mod-since"></a><strong>If-Modified-Since</strong>: Time when all requested feeds have been accessed the last time. The response will only contain podcasts that have been modified in the meantime.</li>
    <li><strong>User-Agent</strong>: Clients should send a descriptive User-Agent string. In case of abuse of the service, misbehaving and/or generic user-agents might be blocked.</li>
    <li><a name="accept"></a><strong>Accept</strong>: Clients should send <em>Accept: application/json</em> to indicate that they are prepared to receive JSON data. If you send a different Accept header, you will receive a HTML formatted response.</li>
    <li><a name="accept-encoding"></a><strong>Accept-Encoding</strong>: Google App Engine gzips responses based on <em>Accept-Encoding</em> and <em>User-Agent</em> (<a href="http://code.google.com/appengine/docs/python/runtime.html#Responses">see documentation</a>). Include "gzip" in both headers to ensure gzip compression.</li>
   </ul>
  </p>

  <h3>Responses</h3>
  <ul>
   <li>Each response contains a list of feeds, at least one for each <em>url</em>-Parameter</li>
   <li>HTTP-Redirects are followed automatically (this is reflected in the <em>urls</em> field)</li>
   <li><a href="http://cyber.law.harvard.edu/rss/rssRedirect.html">RSS-Redirects</a> are followed by additionally including the new feed in the response</li>
   <li>Each feed contains
    <ul>
     <li><strong>title</strong>: the title of the feed</li>
     <li><strong>link</strong>: the feeds website</li>
     <li><strong>description</strong>: a description of the feed, potentially including HTML characters</li>
     <li><strong>author</strong>: the feed's author</li>
     <li><strong>language</strong>: the feed's language</li>
     <li><a name="urls" /><strong>urls</strong>: the redirect-chain of the URL passed in the <em>url</em> parameter. This can be used to match the requested URLs to the entries in the response.</li>
     <li><strong>new_location</strong>: the referred to location, if the feed uses <a href="http://cyber.law.harvard.edu/rss/rssRedirect.html">RSS-Redirects</a>. The new location will also be included in the response</li>
     <li><a name="logo" /><strong>logo</strong>: the URL of the feed's logo</li>
     <li><strong>logo_data</strong>: the feed's logo as a <a href="http://en.wikipedia.org/wiki/Data_URI_scheme">data URI</a>, if <em>inline_logo</em> has been used. To save bandwidth, the logo is not included if it changed since the date sent in <em><a href="#if-mod-since">If-Modified-Since</a></em></li>
     <li><strong>content_types</strong>: the content types of the feed, either audio, video or image</li>
     <li><strong>hub</strong>: the endpoint URL of the <a href="http://code.google.com/p/pubsubhubbub/">hub</a> through which the feed is published</li>
     <li><strong>errors</strong>: a dictionary of occured errors, where the key contains an <a href="#error-codes">error code</a> and the value a string representation.</li>
     <li><strong>http_last_modified</strong>: the Unix timestamp of the last modification of the feed (according to the HTTP header).</li>
     <li><strong>http_etag</strong>: the HTTP E-Tag of the feed</li>
     <li><strong>episodes</strong>: the list of episodes</li>
    </ul>
   </li>
   <li>Each episode contains
    <ul>
     <li><strong>guid</strong>: an unique endentifier for the episode (provided by the feed in the GUID property)</li>
     <li><strong>title</strong>: the title of the episode</li>
     <li><strong>short_title</strong>: the non-repetitive part of the episode title. If an episode number is found, it is also removed and provided separately.</li>
     <li><strong>number</strong>: the episode number which is parsed from the title</li>
     <li><strong>description</strong>: the description of the episode, potentially including HTML characters</li>
     <li><strong>link</strong>: the website link for the episode</li>
     <li><strong>released</strong>: the Unix timestamp of the episode's release</li>
     <li><strong>author</strong>: the episode's author</li>
     <li><strong>duration</strong>: the episode's duration in seconds</li>
     <li><strong>language</strong>: the episode's language</li>
     <li><strong>files</strong>: a list of all files linked by the episode. Each files is represented by an object containing url, filesize (in Bytes) and mimetype.</li>
    </ul>
   </li>
   <li><a name="error-codes" />Current Error Codes
    <ul>
     <li><strong>fetch-feed</strong>: The feed could not be retrieved. The URL is given in the <a href="#urls">urls</a> list</li>
     <li><strong>fetch-logo</strong>: The feed's logo could not be retrieved. Its URL is given in the <a href="#logo">logo</a> field</li>
     <li><strong>hub-subscription</strong>: An error occured while subscribing to the feed's hub for instant updates. This is generally not critical for the client</li>
    </ul>
   </li>
  </ul>
  <p>Headers
   <ul>
    <li><strong>Last-Modified</strong>: The timestamp of the latest change to any of the requested feeds. This value can be used in the If-Modified-Since parameter to subsequent requests. This header is not sent for the HTML formatted response.</li>
    <li><strong>Content-Type</strong>: <em>application/json</em> if your request contains <em>Accept: application/json</em>, otherwise the response will contain the HTML representation with <em>text/html</em>.</li>
    <li><strong>Content-Encoding</strong>: <em>gzip</em> if the response is compressed. See <em><a href="#accept-encoding">Accept-Encoding</a></em> for details.</li>
    <li><strong>Vary</strong>: Contains the request headers for which the response can vary. Currently this is <em>Accept, User-Agent, Accept-Encoding</em>.</li>
   </ul>
  </p>

  <h3>Using this Instance</h3>

  <p>If you intend to use <em>this instance</em> of the webservice in your
  application, please contact <a
  href="mailto:<EMAIL>"><EMAIL></a> beforehand.

  <h2>Source Code</h2>
  <p>The sourcecode is available from <a
  href="https://github.com/gpodder/mygpo-feedservice">https://github.com/gpodder/mygpo-feedservice</a>
  under the terms of the AGPLv3. Please report bugs on the <a href="https://bugs.gpodder.org/enter_bug.cgi?product=mygpo-feedservice">bugtracker</a> and mention the exact URL of your failed request.</p>


 </body>
</html>

{'fraction_non_alphanumeric': 0.14914838115697407, 'fraction_numerical': 0.004398751981995806, 'mean_word_length': 4.4614525139664805, 'pattern_counts': {'":': 0, '<': 675, '<?xml version=': 0, '>': 675, 'https://': 4, 'lorem ipsum': 0, 'www.': 10, 'xml': 6}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '2605143', 'n_tokens_mistral': 6187, 'n_tokens_neox': 5970, 'n_words': 1957}