<commit_before>OFS is a bucket/object storage library.

It provides a common API for doing storing bitstreams (+ a bit of related
metadata) in 'bucket/object' stores such as:

  * Filesystem (+ pairtree)
  * S3
  * Archive.org
  * Riak (in progress)
  * **add a backend here**

Why use the library:

  * Abstraction: write common code but use different storage backends
  * More than a filesystem, less than a database - support for metadata as well
    bitstreams
  * Extra features:

      * Sharding - automatic sharding of files to support distributed file-system like structure (planned)


Example Usage
=============

(local version - depends on 'pairtree', and 'simplejson')::

    >>> from ofs.local import OFS

    >>> o = OFS()
    (Equivalent to 'o = OFS(storage_dir = "data", uri_base="urn:uuid:", hashing_type="md5")')

    # Claim a bucket - this will add the bucket to the list of existing ones
    >>> uuid_id = o.claim_a_bucket()
    >>> uuid_id
    '4aaa43cdf5ba44e2ad25acdbd1cf2f70'

    # Choose a bucket name - if it exists, a new UUID one will be formed instead and returned
    >>> bucket_id = o.claim_a_bucket("foo")
    >>> bucket_id
    'foo'
    >>> bucket_id = o.claim_a_bucket("foo")
    >>> bucket_id
    '1bf93208521545879e79c13614cd12f0'

    # Store a file:
    >>> o.put_stream(bucket_id, "foo.txt", open("foo....))
    {'_label': 'foo.txt', '_content_length': 10, '_checksum': 'md5:10feda25f8da2e2ebfbe646eea351224', '_last_modified': '2010-08-02T11:37:21', '_creation_date': '2010-08-02T11:37:21'}

    # or:
    >>> o.put_stream(bucket_id, "foo.txt", "asidaisdiasjdiajsidjasidji")
    {'_label': 'foo.txt', '_content_length': 10, '_checksum': 'md5:10feda25f8da2e2ebfbe646eea351224', '_last_modified': '2010-08-02T11:37:21', '_creation_date': '2010-08-02T11:37:21'}

    # adding a file with some parameters:
    >>> o.put_stream(bucket_id, "foooo", "asidaisdiasjdiajsidjasidji", params={"original_uri":"http://...."})
    {'_label': 'foooo', 'original_uri': 'http://....', '_last_modified': '2010-08-02T11:39:11', '_checksum': 'md5:3d690d7e0f4479c5a7038b8a4572d0fe', '_creation_date': '2010-08-02T11:39:11', '_content_length': 26}
<commit_msg>Altered the README to reflect new get_url<commit_after>OFS is a bucket/object storage library.

It provides a common API for doing storing bitstreams (+ a bit of related
metadata) in 'bucket/object' stores such as:

  * Filesystem (+ pairtree)
  * S3
  * Archive.org
  * Riak (in progress)
  * **add a backend here**

Why use the library:

  * Abstraction: write common code but use different storage backends
  * More than a filesystem, less than a database - support for metadata as well
    bitstreams
  * Extra features:

      * Sharding - automatic sharding of files to support distributed file-system like structure (planned)


Example Usage
=============

(local version - depends on 'pairtree', and 'simplejson')::

    >>> from ofs.local import OFS

    >>> o = OFS()
    (Equivalent to 'o = OFS(storage_dir = "data", uri_base="urn:uuid:", hashing_type="md5")')

    # Claim a bucket - this will add the bucket to the list of existing ones
    >>> uuid_id = o.claim_a_bucket()
    >>> uuid_id
    '4aaa43cdf5ba44e2ad25acdbd1cf2f70'

    # Choose a bucket name - if it exists, a new UUID one will be formed instead and returned
    >>> bucket_id = o.claim_a_bucket("foo")
    >>> bucket_id
    'foo'
    >>> bucket_id = o.claim_a_bucket("foo")
    >>> bucket_id
    '1bf93208521545879e79c13614cd12f0'

    # Store a file:
    >>> o.put_stream(bucket_id, "foo.txt", open("foo....))
    {'_label': 'foo.txt', '_content_length': 10, '_checksum': 'md5:10feda25f8da2e2ebfbe646eea351224', '_last_modified': '2010-08-02T11:37:21', '_creation_date': '2010-08-02T11:37:21'}

    # or:
    >>> o.put_stream(bucket_id, "foo.txt", "asidaisdiasjdiajsidjasidji")
    {'_label': 'foo.txt', '_content_length': 10, '_checksum': 'md5:10feda25f8da2e2ebfbe646eea351224', '_last_modified': '2010-08-02T11:37:21', '_creation_date': '2010-08-02T11:37:21'}

    # adding a file with some parameters:
    >>> o.put_stream(bucket_id, "foooo", "asidaisdiasjdiajsidjasidji", params={"original_uri":"http://...."})
    {'_label': 'foooo', 'original_uri': 'http://....', '_last_modified': '2010-08-02T11:39:11', '_checksum': 'md5:3d690d7e0f4479c5a7038b8a4572d0fe', '_creation_date': '2010-08-02T11:39:11', '_content_length': 26}

    # Get the underlying URL pointing to a resource
    >>> o.get_url(bucket_id, "foo")
      [typical local pairtree response:]
   "file:///opt/ofs_store/pairtree_root/1b/f9/32/......./obj/foo"
      [typical remote response]
   "http://..."
   "ftp://..."

{'fraction_non_alphanumeric': 0.15407789972024963, 'fraction_numerical': 0.08091241661286852, 'mean_word_length': 4.06318082788671, 'pattern_counts': {'":': 2, '<': 3, '<?xml version=': 0, '>': 72, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '280668', 'n_tokens_mistral': 1896, 'n_tokens_neox': 1657, 'n_words': 414}