<commit_before>===============================================
SwiftHLM (Swift Hight-Latency Media) middleware
===============================================

# (C) Copyright 2016 IBM Corp.
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
#    http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or
# implied.
# See the License for the specific language governing permissions and
# limitations under the License.

### Version: 0.1

### Authors:
<NAME> (<EMAIL>)
<NAME> (<EMAIL>)

### Content:
1. Description
2. Requirements
3. Install
4. Configure
5. Activate
6. HLM Backend
7. External Interface and Usage Examples 
8. References

1. Description
===============================================

SwiftHLM middleware is useful for running OpenStack Swift on top of high
latency media (HLM) storage, such as tape or optical disk archive based
backends, allowing to store cheaply and access efficiently large amounts of
infrequently used object data.

SwiftHLM can be added to OpenStack Swift (without modifying Swift itself) to
extend Swift's interface and thus allow to explicitly control and query the
state (on disk or on HLM) of Swift objects data, including efficient prefetch
of bulk of objects from HLM to disk when those objects need to be accessed.
This function previously missing in Swift can be seen similar to Amazon Glacier
[1], either through the Glacier API or the Amazon S3 Lifecycle Management API
[2].

BDT Tape Library Connector (open source) [3] and IBM Spectrum Archive [4] are
examples of HLM backends that provide important and complex functions to manage
HLM resources (tape mounts/unmounts to drives, serialization of requests for
tape media and tape drives resources) and can use SwiftHLM functions for a
proper integration with Swift.

Access to data stored on HLM could be done transparently, without using
SwiftHLM, but that does not work well in practice for many important use cases
and for various reasons as discussed in [5]. In [5] it is also explained how
SwiftHLM function can be orthogonal and complementary to Swift (ring to ring)
tiering [6].

SwiftHLM version 0.1 provides the following basic HLM functions on the external
Swift interface:
- MIGRATE (container or an object from disk to HLM)
- RECALL (i.e. prefetch a container or an object from HLM to disk) 
- STATUS (get status for an object or a container objects).

Detailed (still exemplary and not standardized) syntax and usage examples are 
provided below in section "External Interface and Usage Examples".


2. Requirements
=============================================== 

- OpenStack Swift Juno (tested) or Kilo (tested) or a later release (not
  tested)
- HLM backend that supports SwiftHLM functions, see HLM Backend section below
  for details
- Python 2.7+


3. Install
===============================================

    # python setup.py install


4. Configure
===============================================

  a) If Swift is installed from source
    Modify Swift's configuration file /etc/swift/proxy-server.conf to include hlm middleware.
    In section [pipeline:main] add hlm keyword into pipeline, e.g.:

      pipeline = catch_errors gatekeeper healthcheck proxy-logging cache bulk tempurl formpost slo dlo ratelimit tempauth hlm staticweb container-quotas account-quotas proxy-logging proxy-server

    Add a new section:
      # High latency media (hlm) middleware
      [filter:hlm]
      use = egg:swifthlm#swifthlm

  b) If Swift installed as part of Spectrum Scale 4.1.1 and later:

    # mmces service stop OBJ --all

    Retrieve your current Swift middleware pipeline setting:
    # mmobj config list --ccrfile proxy-server.conf --section pipeline:main --property pipeline

    Example output for the previous command:
    pipeline = healthcheck cache formpost tempurl swift3 s3token authtoken keystoneauth container-quotas account-quotas staticweb bulk slo dlo proxy-server

    Insert hlm before proxy-server and write back the configuration - for the above example:
    # mmobj config change --ccrfile proxy-server.conf --section pipeline:main --property pipeline --value "healthcheck cache formpost tempurl swift3 s3token authtoken keystoneauth container-quotas account-quotas staticweb bulk slo dlo hlm proxy-server"

    Register the swifthlm middleware:
    # mmobj config change --ccrfile proxy-server.conf --section filter:hlm --property use --value egg:swifthlm#swifthlm

5. Activate
===============================================

To activate the middleware, restart Swift services:


  a) Swift installed from source:
    # swift-init main reload

  b) Spectrum Scale 4.1.1 or later:
    # mmces service start OBJ --all


6. HLM Backend
===============================================

An HLM backend that supports SwiftHLM functions (MIGRATE, RECALL, STATUS) is
exposed to Swift in the same way as if SwiftHLM is not used (via a file system
interface and a Swift ring definition), plus it needs to additionally support
processing and responding requests from SwiftHLM middleware for performing
SwiftHLM functions.

This additional SwiftHLM middleware to HLM backend interface is aimed to be
configurable and a particular HLM backend needs to support at least one of the
two options:
a) The backend accepts and responds requests from SwiftHLM via a CLI interface
b) SwiftHLM communicates to the backend via a pair of object extended
attributes, the requested state (RS) set by SwiftHLM and read by the backend,
and the current state (CS) set by the backend and read by the SwiftHLM. E.g. if
a Swift application requests migrating a container to HLM, SwiftHLM middleware<commit_msg>Updated README.txt to explicit that a backend needs to be installed and configured before using SwiftHLM<commit_after>===============================================
SwiftHLM (Swift Hight-Latency Media) middleware
===============================================

# (C) Copyright 2016 IBM Corp.
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
#    http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or
# implied.
# See the License for the specific language governing permissions and
# limitations under the License.

### Version: 0.1

### Authors:
<NAME> (<EMAIL>)
<NAME> (<EMAIL>)

### Content:
1. Description
2. Requirements
3. Install
4. Configure
5. Activate
6. HLM Backend
7. External Interface and Usage Examples 
8. References

1. Description
===============================================

SwiftHLM middleware is useful for running OpenStack Swift on top of high
latency media (HLM) storage, such as tape or optical disk archive based
backends, allowing to store cheaply and access efficiently large amounts of
infrequently used object data.

SwiftHLM can be added to OpenStack Swift (without modifying Swift itself) to
extend Swift's interface and thus allow to explicitly control and query the
state (on disk or on HLM) of Swift objects data, including efficient prefetch
of bulk of objects from HLM to disk when those objects need to be accessed.
This function previously missing in Swift can be seen similar to Amazon Glacier
[1], either through the Glacier API or the Amazon S3 Lifecycle Management API
[2].

BDT Tape Library Connector (open source) [3] and IBM Spectrum Archive [4] are
examples of HLM backends that provide important and complex functions to manage
HLM resources (tape mounts/unmounts to drives, serialization of requests for
tape media and tape drives resources) and can use SwiftHLM functions for a
proper integration with Swift.

Access to data stored on HLM could be done transparently, without using
SwiftHLM, but that does not work well in practice for many important use cases
and for various reasons as discussed in [5]. In [5] it is also explained how
SwiftHLM function can be orthogonal and complementary to Swift (ring to ring)
tiering [6].

SwiftHLM version 0.1 provides the following basic HLM functions on the external
Swift interface:
- MIGRATE (container or an object from disk to HLM)
- RECALL (i.e. prefetch a container or an object from HLM to disk) 
- STATUS (get status for an object or a container objects).

Detailed (still exemplary and not standardized) syntax and usage examples are 
provided below in section "External Interface and Usage Examples".


2. Requirements
=============================================== 

- OpenStack Swift Juno (tested) or Kilo (tested) or a later release (not
  tested)
- HLM backend that supports SwiftHLM functions, see HLM Backend section below
  for details
- Python 2.7+


3. Install
===============================================

    # python setup.py install


4. Configure
===============================================

  a) If Swift is installed from source
    Modify Swift's configuration file /etc/swift/proxy-server.conf to include hlm middleware.
    In section [pipeline:main] add hlm keyword into pipeline, e.g.:

      pipeline = catch_errors gatekeeper healthcheck proxy-logging cache bulk tempurl formpost slo dlo ratelimit tempauth hlm staticweb container-quotas account-quotas proxy-logging proxy-server

    Add a new section:
      # High latency media (hlm) middleware
      [filter:hlm]
      use = egg:swifthlm#swifthlm

  b) If Swift installed as part of Spectrum Scale 4.1.1 and later:

    # mmces service stop OBJ --all

    Retrieve your current Swift middleware pipeline setting:
    # mmobj config list --ccrfile proxy-server.conf --section pipeline:main --property pipeline

    Example output for the previous command:
    pipeline = healthcheck cache formpost tempurl swift3 s3token authtoken keystoneauth container-quotas account-quotas staticweb bulk slo dlo proxy-server

    Insert hlm before proxy-server and write back the configuration - for the above example:
    # mmobj config change --ccrfile proxy-server.conf --section pipeline:main --property pipeline --value "healthcheck cache formpost tempurl swift3 s3token authtoken keystoneauth container-quotas account-quotas staticweb bulk slo dlo hlm proxy-server"

    Register the swifthlm middleware:
    # mmobj config change --ccrfile proxy-server.conf --section filter:hlm --property use --value egg:swifthlm#swifthlm

5. Activate
===============================================

To activate the middleware, restart Swift services:


  a) Swift installed from source:
    # swift-init main reload

  b) Spectrum Scale 4.1.1 or later:
    # mmces service start OBJ --all

Note: Before SwiftHLM can be used (Section 7), an HLM Backend needs to be
installed and configured (Section 6).

6. HLM Backend
===============================================

An HLM backend that supports SwiftHLM functions (MIGRATE, RECALL, STATUS) is
exposed to Swift in the same way as if SwiftHLM is not used (via a file system
interface and a Swift ring definition), plus it needs to additionally support
processing and responding requests from SwiftHLM middleware for performing
SwiftHLM functions.

This additional SwiftHLM middleware to HLM backend interface is aimed to be
configurable and a particular HLM backend needs to support at least one of the
two options:
a) The backend accepts and responds requests from SwiftHLM via a CLI interface
b) SwiftHLM communicates to the backend via a pair of object extended
attributes, the requested state (RS) set by SwiftHLM and read by the backend,
and the current state (CS) set by the backend and read by the SwiftHLM. E.g. if
a Swift application requests migrating a container to HLM, SwiftHLM middleware
{'fraction_non_alphanumeric': 0.10965021086579013, 'fraction_numerical': 0.007773091871330522, 'mean_word_length': 4.509794988610478, 'pattern_counts': {'":': 0, '<': 11, '<?xml version=': 0, '>': 11, 'https://': 0, 'lorem ipsum': 0, 'www.': 2, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '5844155', 'n_tokens_mistral': 3238, 'n_tokens_neox': 2987, 'n_words': 1607}