<commit_before>""" The core class bits of TileStache.

Two important classes can be found here.

Layer represents a set of tiles in TileStache. It keeps references to
providers, projections, a Configuration instance, and other details required
for to the storage and rendering of a tile set. Layers are represented in the
configuration file as a dictionary:

    {
      "cache": ...,
      "layers": 
      {
        "example-name":
        {
          "provider": { ... },
          "metatile": { ... },
          "preview": { ... },
          "projection": ...,
          "stale lock timeout": ...,
          "cache lifespan": ...,
          "write cache": ...,
          "bounds": { ... },
          "allowed origin": ...,
          "maximum cache age": ...,
          "redirects": ...,
          "tile height": ...,
          "jpeg options": ...,
          "png options": ...
        }
      }
    }

- "provider" refers to a Provider, explained in detail in TileStache.Providers.
- "metatile" optionally makes it possible for multiple individual tiles to be
  rendered at one time, for greater speed and efficiency. This is commonly used
  for the Mapnik provider. See below for more information on metatiles.
- "preview" optionally overrides the starting point for the built-in per-layer
  slippy map preview, useful for image-based layers where appropriate.
  See below for more information on the preview.
- "projection" names a geographic projection, explained in TileStache.Geography.
  If omitted, defaults to spherical mercator.
- "stale lock timeout" is an optional number of seconds to wait before forcing
  a lock that might be stuck. This is defined on a per-layer basis, rather than
  for an entire cache at one time, because you may have different expectations
  for the rendering speeds of different layer configurations. Defaults to 15.
- "cache lifespan" is an optional number of seconds that cached tiles should
  be stored. This is defined on a per-layer basis. Defaults to forever if None,
  0 or omitted.
- "write cache" is an optional boolean value to allow skipping cache write
  altogether. This is defined on a per-layer basis. Defaults to true if omitted.
- "bounds" is an optional dictionary of six tile boundaries to limit the
  rendered area: low (lowest zoom level), high (highest zoom level), north,
  west, south, and east (all in degrees).
- "allowed origin" is an optional string that shows up in the response HTTP
  header Access-Control-Allow-Origin, useful for when you need to provide
  javascript direct access to response data such as GeoJSON or pixel values.
  The header is part of a W3C working draft (http://www.w3.org/TR/cors/).
- "maximum cache age" is an optional number of seconds used to control behavior
  of downstream caches. Causes TileStache responses to include Cache-Control
  and Expires HTTP response headers. Useful when TileStache is itself hosted
  behind an HTTP cache such as Squid, Cloudfront, or Akamai.
- "redirects" is an optional dictionary of per-extension HTTP redirects,
  treated as lowercase. Useful in cases where your tile provider can support
  many formats but you want to enforce limits to save on cache usage.
  If a request is made for a tile with an extension in the dictionary keys,
  a response can be generated that redirects the client to the same tile
  with another extension.
- "tile height" gives the height of the image tile in pixels. You almost always
  want to leave this at the default value of 256, but you can use a value of 512
  to create double-size, double-resolution tiles for high-density phone screens.
- "jpeg options" is an optional dictionary of JPEG creation options, passed
  through to PIL: http://effbot.org/imagingbook/format-jpeg.htm.
- "png options" is an optional dictionary of PNG creation options, passed
  through to PIL: http://effbot.org/imagingbook/format-png.htm.
- "pixel effect" is an optional dictionary that defines an effect to be applied
   for all tiles of this layer. Pixel effect can be any of these: blackwhite,
  greyscale, desaturate, pixelate, halftone, or blur.

The public-facing URL of a single tile for this layer might look like this:

    http://example.com/tilestache.cgi/example-name/0/0/0.png

Sample JPEG creation options:

    {
      "quality": 90,
      "progressive": true,
      "optimize": true
    }

Sample PNG creation options:

    {
      "optimize": true,
      "palette": "filename.act"
    }

Sample pixel effect:

    {
        "name": "desaturate",
        "factor": 0.85
    }

Sample bounds:

    {
        "low": 9, "high": 15,
        "south": 37.749, "west": -122.358,
        "north": 37.860, "east": -122.113
    }

Metatile represents a larger area to be rendered at one time. Metatiles are
represented in the configuration file as a dictionary:

    {
      "rows": 4,
      "columns": 4,
      "buffer": 64
    }

- "rows" and "columns" are the height and width of the metatile measured in
  tiles. This example metatile is four rows tall and four columns wide, so it
  will render sixteen tiles simultaneously.
- "buffer" is a buffer area around the metatile, measured in pixels. This is
  useful for providers with labels or icons, where it's necessary to draw a
  bit extra around the edges to ensure that text is not cut off. This example
  metatile has a buffer of 64 pixels, so the resulting metatile will be 1152
  pixels square: 4 rows x 256 pixels + 2 x 64 pixel buffer.

The preview can be accessed through a URL like /<layer name>/preview.html:

    {
      "lat": 33.9901,
      "lon": -116.1637,
      "zoom": 16,
      "ext": "jpg"
    }

- "lat" and "lon" are the starting latitude and longitude in degrees.
- "zoom" is the starting zoom level.
- "ext" is the filename extension, e.g. "png".
"""

import logging
from wsgiref.headers import Headers
from StringIO import StringIO
from urlparse import urljoin
from time import time

from Pixels import load_palette, apply_palette, apply_palette256

try:
    from PIL import Image
except ImportError:
    import Image

from ModestMaps.Core import Coordinate

_recent_tiles = dict(hash={}, list=[])

def _addRecentTile(layer, coord, format, body, age=300):
    """ Add the body of a tile to _recent_tiles with a timeout.
    """
    key = (layer, coord, format)
    due = time() + age
    
    _recent_tiles['hash'][key] = body, due
    _recent_tiles['list'].append((key, due))
    
    logging.debug('TileStache.Core._addRecentTile() added tile to recent tiles: %s', key)
    
    # now look at the oldest keys and remove them if needed
    cutoff = 0
    for i, (key, due_by) in enumerate(_recent_tiles['list']):
        # new enough?
        if time() < due_by:
            cutoff = i
            break
        
        logging.debug('TileStache.Core._addRecentTile() removed tile from recent tiles: %s', key)
        
        try:
            del _recent_tiles['hash'][key]
        except KeyError:
            pass
    _recent_tiles['list'] = _recent_tiles['list'][cutoff:]

def _getRecentTile(layer, coord, format):
    """ Return the body of a recent tile, or None if it's not there.
    """
    key = (layer, coord, format)
    body, use_by = _recent_tiles['hash'].get(key, (None, 0))
    
    # non-existent?
    if body is None:
        return None
    
    # new enough?
    if time() < use_by:<commit_msg>Changed _addRecentTile to use in-place update<commit_after>""" The core class bits of TileStache.

Two important classes can be found here.

Layer represents a set of tiles in TileStache. It keeps references to
providers, projections, a Configuration instance, and other details required
for to the storage and rendering of a tile set. Layers are represented in the
configuration file as a dictionary:

    {
      "cache": ...,
      "layers": 
      {
        "example-name":
        {
          "provider": { ... },
          "metatile": { ... },
          "preview": { ... },
          "projection": ...,
          "stale lock timeout": ...,
          "cache lifespan": ...,
          "write cache": ...,
          "bounds": { ... },
          "allowed origin": ...,
          "maximum cache age": ...,
          "redirects": ...,
          "tile height": ...,
          "jpeg options": ...,
          "png options": ...
        }
      }
    }

- "provider" refers to a Provider, explained in detail in TileStache.Providers.
- "metatile" optionally makes it possible for multiple individual tiles to be
  rendered at one time, for greater speed and efficiency. This is commonly used
  for the Mapnik provider. See below for more information on metatiles.
- "preview" optionally overrides the starting point for the built-in per-layer
  slippy map preview, useful for image-based layers where appropriate.
  See below for more information on the preview.
- "projection" names a geographic projection, explained in TileStache.Geography.
  If omitted, defaults to spherical mercator.
- "stale lock timeout" is an optional number of seconds to wait before forcing
  a lock that might be stuck. This is defined on a per-layer basis, rather than
  for an entire cache at one time, because you may have different expectations
  for the rendering speeds of different layer configurations. Defaults to 15.
- "cache lifespan" is an optional number of seconds that cached tiles should
  be stored. This is defined on a per-layer basis. Defaults to forever if None,
  0 or omitted.
- "write cache" is an optional boolean value to allow skipping cache write
  altogether. This is defined on a per-layer basis. Defaults to true if omitted.
- "bounds" is an optional dictionary of six tile boundaries to limit the
  rendered area: low (lowest zoom level), high (highest zoom level), north,
  west, south, and east (all in degrees).
- "allowed origin" is an optional string that shows up in the response HTTP
  header Access-Control-Allow-Origin, useful for when you need to provide
  javascript direct access to response data such as GeoJSON or pixel values.
  The header is part of a W3C working draft (http://www.w3.org/TR/cors/).
- "maximum cache age" is an optional number of seconds used to control behavior
  of downstream caches. Causes TileStache responses to include Cache-Control
  and Expires HTTP response headers. Useful when TileStache is itself hosted
  behind an HTTP cache such as Squid, Cloudfront, or Akamai.
- "redirects" is an optional dictionary of per-extension HTTP redirects,
  treated as lowercase. Useful in cases where your tile provider can support
  many formats but you want to enforce limits to save on cache usage.
  If a request is made for a tile with an extension in the dictionary keys,
  a response can be generated that redirects the client to the same tile
  with another extension.
- "tile height" gives the height of the image tile in pixels. You almost always
  want to leave this at the default value of 256, but you can use a value of 512
  to create double-size, double-resolution tiles for high-density phone screens.
- "jpeg options" is an optional dictionary of JPEG creation options, passed
  through to PIL: http://effbot.org/imagingbook/format-jpeg.htm.
- "png options" is an optional dictionary of PNG creation options, passed
  through to PIL: http://effbot.org/imagingbook/format-png.htm.
- "pixel effect" is an optional dictionary that defines an effect to be applied
   for all tiles of this layer. Pixel effect can be any of these: blackwhite,
  greyscale, desaturate, pixelate, halftone, or blur.

The public-facing URL of a single tile for this layer might look like this:

    http://example.com/tilestache.cgi/example-name/0/0/0.png

Sample JPEG creation options:

    {
      "quality": 90,
      "progressive": true,
      "optimize": true
    }

Sample PNG creation options:

    {
      "optimize": true,
      "palette": "filename.act"
    }

Sample pixel effect:

    {
        "name": "desaturate",
        "factor": 0.85
    }

Sample bounds:

    {
        "low": 9, "high": 15,
        "south": 37.749, "west": -122.358,
        "north": 37.860, "east": -122.113
    }

Metatile represents a larger area to be rendered at one time. Metatiles are
represented in the configuration file as a dictionary:

    {
      "rows": 4,
      "columns": 4,
      "buffer": 64
    }

- "rows" and "columns" are the height and width of the metatile measured in
  tiles. This example metatile is four rows tall and four columns wide, so it
  will render sixteen tiles simultaneously.
- "buffer" is a buffer area around the metatile, measured in pixels. This is
  useful for providers with labels or icons, where it's necessary to draw a
  bit extra around the edges to ensure that text is not cut off. This example
  metatile has a buffer of 64 pixels, so the resulting metatile will be 1152
  pixels square: 4 rows x 256 pixels + 2 x 64 pixel buffer.

The preview can be accessed through a URL like /<layer name>/preview.html:

    {
      "lat": 33.9901,
      "lon": -116.1637,
      "zoom": 16,
      "ext": "jpg"
    }

- "lat" and "lon" are the starting latitude and longitude in degrees.
- "zoom" is the starting zoom level.
- "ext" is the filename extension, e.g. "png".
"""

import logging
from wsgiref.headers import Headers
from StringIO import StringIO
from urlparse import urljoin
from time import time

from Pixels import load_palette, apply_palette, apply_palette256

try:
    from PIL import Image
except ImportError:
    import Image

from ModestMaps.Core import Coordinate

_recent_tiles = dict(hash={}, list=[])

def _addRecentTile(layer, coord, format, body, age=300):
    """ Add the body of a tile to _recent_tiles with a timeout.
    """
    key = (layer, coord, format)
    due = time() + age
    
    _recent_tiles['hash'][key] = body, due
    _recent_tiles['list'].append((key, due))
    
    logging.debug('TileStache.Core._addRecentTile() added tile to recent tiles: %s', key)
    
    # now look at the oldest keys and remove them if needed
    cutoff = 0
    for i, (key, due_by) in enumerate(_recent_tiles['list']):
        # new enough?
        if time() < due_by:
            cutoff = i
            break
        
        logging.debug('TileStache.Core._addRecentTile() removed tile from recent tiles: %s', key)
        
        try:
            del _recent_tiles['hash'][key]
        except KeyError:
            pass
    del _recent_tiles['list'][:cutoff]

def _getRecentTile(layer, coord, format):
    """ Return the body of a recent tile, or None if it's not there.
    """
    key = (layer, coord, format)
    body, use_by = _recent_tiles['hash'].get(key, (None, 0))
    
    # non-existent?
    if body is None:
        return None
    
    # new enough?
    if time() < use_by:
{'fraction_non_alphanumeric': 0.08534927969556945, 'fraction_numerical': 0.011416145691764067, 'mean_word_length': 2.8749341758820433, 'pattern_counts': {'":': 74, '<': 9, '<?xml version=': 0, '>': 5, 'https://': 0, 'lorem ipsum': 0, 'www.': 2, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '6914813', 'n_tokens_mistral': 4309, 'n_tokens_neox': 4040, 'n_words': 1886}