<commit_before># Copyright (c) 2013 OpenStack Foundation
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
#    http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or
# implied.
# See the License for the specific language governing permissions and
# limitations under the License.

"""
Middleware that will provide Static Large Object (SLO) support.

This feature is very similar to Dynamic Large Object (DLO) support in that
it allows the user to upload many objects concurrently and afterwards
download them as a single object. It is different in that it does not rely
on eventually consistent container listings to do so. Instead, a user
defined manifest of the object segments is used.

----------------------
Uploading the Manifest
----------------------

After the user has uploaded the objects to be concatenated, a manifest is
uploaded. The request must be a PUT with the query parameter::

    ?multipart-manifest=put

The body of this request will be an ordered list of segment descriptions in
JSON format. The data to be supplied for each segment is:

=========== ========================================================
Key         Description
=========== ========================================================
path        the path to the segment object (not including account)
            /container/object_name
etag        the ETag given back when the segment object was PUT,
            or null
size_bytes  the size of the complete segment object in
            bytes, or null
range       (optional) the (inclusive) range within the object to
            use as a segment. If omitted, the entire object is used.
=========== ========================================================

The format of the list will be:

  .. code::

    [{"path": "/cont/object",
      "etag": "etagoftheobjectsegment",
      "size_bytes": 10485760,
      "range": "1048576-2097151"}, ...]

The number of object segments is limited to a configurable amount, default
1000. Each segment must be at least 1 byte. On upload, the middleware will
head every segment passed in to verify:

 1. the segment exists (i.e. the HEAD was successful);
 2. the segment meets minimum size requirements;
 3. if the user provided a non-null etag, the etag matches;
 4. if the user provided a non-null size_bytes, the size_bytes matches; and
 5. if the user provided a range, it is a singular, syntactically correct range
    that is satisfiable given the size of the object.

Note that the etag and size_bytes keys are still required; this acts as a guard
against user errors such as typos. If any of the objects fail to verify (not
found, size/etag mismatch, below minimum size, invalid range) then the user
will receive a 4xx error response. If everything does match, the user will
receive a 2xx response and the SLO object is ready for downloading.

Behind the scenes, on success, a json manifest generated from the user input is
sent to object servers with an extra "X-Static-Large-Object: True" header
and a modified Content-Type. The items in this manifest will include the etag
and size_bytes for each segment, regardless of whether the client specified
them for verification. The parameter: swift_bytes=$total_size will be
appended to the existing Content-Type, where total_size is the sum of all
the included segments' size_bytes. This extra parameter will be hidden from
the user.

Manifest files can reference objects in separate containers, which will improve
concurrent upload speed. Objects can be referenced by multiple manifests. The
segments of a SLO manifest can even be other SLO manifests. Treat them as any
other object i.e., use the Etag and Content-Length given on the PUT of the
sub-SLO in the manifest to the parent SLO.

-------------------
Range Specification
-------------------

Users now have the ability to specify ranges for SLO segments.
Users can now include an optional 'range' field in segment descriptions
to specify which bytes from the underlying object should be used for the
segment data. Only one range may be specified per segment.

  .. note::

     The 'etag' and 'size_bytes' fields still describe the backing object as a
     whole.

If a user uploads this manifest:

  .. code::

     [{"path": "/con/obj_seg_1", "etag": null, "size_bytes": 2097152,
       "range": "0-1048576"},
      {"path": "/con/obj_seg_2", "etag": null, "size_bytes": 2097152,
       "range": "512-1550000"},
      {"path": "/con/obj_seg_1", "etag": null, "size_bytes": 2097152,
       "range": "-2048"}]

The segment will consist of the first 1048576 bytes of /con/obj_seg_1,
followed by bytes 513 through 1550000 (inclusive) of /con/obj_seg_2, and
finally bytes 2095104 through 2097152 (i.e., the last 2048 bytes) of
/con/obj_seg_1.

  .. note::


     The minimum sized range is 1 byte. This is the same as the minimum
     segment size.


-------------------------
Retrieving a Large Object
-------------------------

A GET request to the manifest object will return the concatenation of the
objects from the manifest much like DLO. If any of the segments from the
manifest are not found or their Etag/Content Length have changed since upload,
the connection will drop. In this case a 409 Conflict will be logged in the
proxy logs and the user will receive incomplete results. Note that this will be
enforced regardless of whether the user perfomed per-segment validation during
upload.

The headers from this GET or HEAD request will return the metadata attached
to the manifest object itself with some exceptions::

    Content-Length: the total size of the SLO (the sum of the sizes of
                    the segments in the manifest)
    X-Static-Large-Object: True
    Etag: the etag of the SLO (generated the same way as DLO)

A GET request with the query parameter::

    ?multipart-manifest=get

will return a transformed version of the original manifest, containing
additional fields and different key names.

A GET request with the query parameters::

    ?multipart-manifest=get&format=raw

will return the contents of the original manifest as it was sent by the client.
The main purpose for both calls is solely debugging.

When the manifest object is uploaded you are more or less guaranteed that
every segment in the manifest exists and matched the specifications.
However, there is nothing that prevents the user from breaking the
SLO download by deleting/replacing a segment referenced in the manifest. It is
left to the user to use caution in handling the segments.

-----------------------<commit_msg>Corrects spelling error in swift/common/middleware/slo.py<commit_after># Copyright (c) 2013 OpenStack Foundation
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
#    http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or
# implied.
# See the License for the specific language governing permissions and
# limitations under the License.

"""
Middleware that will provide Static Large Object (SLO) support.

This feature is very similar to Dynamic Large Object (DLO) support in that
it allows the user to upload many objects concurrently and afterwards
download them as a single object. It is different in that it does not rely
on eventually consistent container listings to do so. Instead, a user
defined manifest of the object segments is used.

----------------------
Uploading the Manifest
----------------------

After the user has uploaded the objects to be concatenated, a manifest is
uploaded. The request must be a PUT with the query parameter::

    ?multipart-manifest=put

The body of this request will be an ordered list of segment descriptions in
JSON format. The data to be supplied for each segment is:

=========== ========================================================
Key         Description
=========== ========================================================
path        the path to the segment object (not including account)
            /container/object_name
etag        the ETag given back when the segment object was PUT,
            or null
size_bytes  the size of the complete segment object in
            bytes, or null
range       (optional) the (inclusive) range within the object to
            use as a segment. If omitted, the entire object is used.
=========== ========================================================

The format of the list will be:

  .. code::

    [{"path": "/cont/object",
      "etag": "etagoftheobjectsegment",
      "size_bytes": 10485760,
      "range": "1048576-2097151"}, ...]

The number of object segments is limited to a configurable amount, default
1000. Each segment must be at least 1 byte. On upload, the middleware will
head every segment passed in to verify:

 1. the segment exists (i.e. the HEAD was successful);
 2. the segment meets minimum size requirements;
 3. if the user provided a non-null etag, the etag matches;
 4. if the user provided a non-null size_bytes, the size_bytes matches; and
 5. if the user provided a range, it is a singular, syntactically correct range
    that is satisfiable given the size of the object.

Note that the etag and size_bytes keys are still required; this acts as a guard
against user errors such as typos. If any of the objects fail to verify (not
found, size/etag mismatch, below minimum size, invalid range) then the user
will receive a 4xx error response. If everything does match, the user will
receive a 2xx response and the SLO object is ready for downloading.

Behind the scenes, on success, a json manifest generated from the user input is
sent to object servers with an extra "X-Static-Large-Object: True" header
and a modified Content-Type. The items in this manifest will include the etag
and size_bytes for each segment, regardless of whether the client specified
them for verification. The parameter: swift_bytes=$total_size will be
appended to the existing Content-Type, where total_size is the sum of all
the included segments' size_bytes. This extra parameter will be hidden from
the user.

Manifest files can reference objects in separate containers, which will improve
concurrent upload speed. Objects can be referenced by multiple manifests. The
segments of a SLO manifest can even be other SLO manifests. Treat them as any
other object i.e., use the Etag and Content-Length given on the PUT of the
sub-SLO in the manifest to the parent SLO.

-------------------
Range Specification
-------------------

Users now have the ability to specify ranges for SLO segments.
Users can now include an optional 'range' field in segment descriptions
to specify which bytes from the underlying object should be used for the
segment data. Only one range may be specified per segment.

  .. note::

     The 'etag' and 'size_bytes' fields still describe the backing object as a
     whole.

If a user uploads this manifest:

  .. code::

     [{"path": "/con/obj_seg_1", "etag": null, "size_bytes": 2097152,
       "range": "0-1048576"},
      {"path": "/con/obj_seg_2", "etag": null, "size_bytes": 2097152,
       "range": "512-1550000"},
      {"path": "/con/obj_seg_1", "etag": null, "size_bytes": 2097152,
       "range": "-2048"}]

The segment will consist of the first 1048576 bytes of /con/obj_seg_1,
followed by bytes 513 through 1550000 (inclusive) of /con/obj_seg_2, and
finally bytes 2095104 through 2097152 (i.e., the last 2048 bytes) of
/con/obj_seg_1.

  .. note::


     The minimum sized range is 1 byte. This is the same as the minimum
     segment size.


-------------------------
Retrieving a Large Object
-------------------------

A GET request to the manifest object will return the concatenation of the
objects from the manifest much like DLO. If any of the segments from the
manifest are not found or their Etag/Content Length have changed since upload,
the connection will drop. In this case a 409 Conflict will be logged in the
proxy logs and the user will receive incomplete results. Note that this will be
enforced regardless of whether the user performed per-segment validation during
upload.

The headers from this GET or HEAD request will return the metadata attached
to the manifest object itself with some exceptions::

    Content-Length: the total size of the SLO (the sum of the sizes of
                    the segments in the manifest)
    X-Static-Large-Object: True
    Etag: the etag of the SLO (generated the same way as DLO)

A GET request with the query parameter::

    ?multipart-manifest=get

will return a transformed version of the original manifest, containing
additional fields and different key names.

A GET request with the query parameters::

    ?multipart-manifest=get&format=raw

will return the contents of the original manifest as it was sent by the client.
The main purpose for both calls is solely debugging.

When the manifest object is uploaded you are more or less guaranteed that
every segment in the manifest exists and matched the specifications.
However, there is nothing that prevents the user from breaking the
SLO download by deleting/replacing a segment referenced in the manifest. It is
left to the user to use caution in handling the segments.

-----------------------
{'fraction_non_alphanumeric': 0.10136168353600815, 'fraction_numerical': 0.01893249836161072, 'mean_word_length': 3.4779915226605804, 'pattern_counts': {'":': 32, '<': 3, '<?xml version=': 0, '>': 3, 'https://': 0, 'lorem ipsum': 0, 'www.': 2, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 6, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '4914575', 'n_tokens_mistral': 3623, 'n_tokens_neox': 3321, 'n_words': 1944}