<commit_before># dkrcp
Copy files between host's file system, containers, and images.
#####ToC
[Copy Semantics](#copy-semantics)  
&nbsp;&nbsp;&nbsp;&nbsp;[Images as SOURCE/TARGET](#images-as-sourcetarget)  
&nbsp;&nbsp;&nbsp;&nbsp;[Interweaved Copying](#interweaved-copying)  
&nbsp;&nbsp;&nbsp;&nbsp;[Permissions](#permissions)  
&nbsp;&nbsp;&nbsp;&nbsp;[follow-link](#follow-link)  
&nbsp;&nbsp;&nbsp;&nbsp;[Examples](#examples)  
[Installing](#install)  
[Testing](#testing)  
[Motivation](#motivation)

```
Usage:  [OPTIONS] SOURCE [SOURCE]... TARGET 

  SOURCE - Can be either: 
             host file path     : {<relativepath>|<absolutePath>}
             image file path    : [<nameSpace>/...]{<repositoryName>:[<tagName>]|<UUID>}::{<relativepath>|<absolutePath>}
             container file path: {<containerName>|<UUID>}:{<relativepath>|<absolutePath>}
             stream             : -
  TARGET - See SOURCE.

  Copy SOURCE to TARGET.  SOURCE or TARGET must refer to either a container/image.
  <relativepath> within the context of a container/image is relative to
  container's/image's '/' (root).

OPTIONS:
    --ucpchk-reg=false        Don't pull images from registry. Limits image name
                                resolution to Docker local repository for  
                                both SOURCE and TARGET names.
    --author="",-a            Specify maintainer when TARGET is an image.
    --change[],-c             Apply specified Dockerfile instruction(s) when
                                TARGET is an image. see 'docker commit'
    --message="",-m           Apply commit message when TARGET is an image.
    --follow-link=false,-L    Follow symbolic link when link specified as a 
                                SOURCE argument. Different from 'cp -aL'!
    --help=false,-h           Don't display this help message.
    --version=false           Don't display version info.
```

Supplements [```docker cp```](https://docs.docker.com/engine/reference/commandline/cp/) by:
  * Facilitating image creation or adaptation by simply copying files.  When copying to an existing image, its state is unaffected, as copy preserves its immutability by creating a new layer.
  * Enabling the specification of multiple copy sources, including other images, to improve operational alignment with Linux [```cp -a```](https://en.wikipedia.org/wiki/Cp_%28Unix%29) and minimize layer creation when TARGET refers to an image.
  * Supporting the direct expression of copy semantics where SOURCE and TARGET arguments concurrently refer to containers.
 
#### Copy Semantics
```dkrcp``` relies on ```docker cp```, therefore, ```docker cp```'s [documentation](https://docs.docker.com/engine/reference/commandline/cp/) describes much of the expected behavior of ```dkrcp```, especially when specifying a single SOURCE argument.  Due to this reliance ```docker cp``` explanations concerning:
  * tarball streams ' - ',
  * container's root directory treated as the current one when copying to/from relative paths,
  * working directory of ```docker cp``` anchoring relative host file system paths,
  * desire to mimic ```cp -a``` recursive navigation and preserve file permissions,
  * ownership UID/GID permission settings when copying to/from containers, 
  * use of ':' as means of delimiting a container UUID/name from its associated path,
  * inability to copy certain files,

are all applicable to ```dkrcp```.

However, the following tabular form offers an equivalent description of copy behavior but visually different than the documention associated to ```docker cp```.

|         | SOURCE File  | SOURCE Directory | [SOURCE Directory Content](https://github.com/WhisperingChaos/dkrcp/blob/master/README.md#source-directory-content-an-existing-directory-path-appended-with-) | SOURCE Stream |
| :--:    | :----------: | :---------:| :----------: | :----------: |
| **TARGET exists as file.** | Copy SOURCE content to TARGET. | Error |Error | Error |
| **TARGET name does not exist but its parent directory does.** | TARGET File name assumed. Copy SOURCE content to TARGET.| TARGET directory name assumed. Create TARGET directory. Copy SOURCE content to TARGET. | Identical behavior to adjacent left hand cell. | Error |
| **TARGET name does not exist, nor does its parent directory.** | Error | Error | Error | Error|
| **TARGET exists as directory.** | Copy SOURCE to TARGET. | Copy SOURCE to TARGET. | Copy SOURCE content to TARGET. | Copy File/Directory to TARGET. |
| **[TARGET assumed directory](https://github.com/WhisperingChaos/dkrcp/blob/master/README.md#target-assumed-directory-the-rightmost-last-name-of-a-specified-path-suffixed-by---it-is-assumed-to-reference-an-existing-directory) doesn't exist but its parent directory does.** | Error | Copy SOURCE content to TARGET. | Copy SOURCE content to TARGET. | Error |

######TARGET assumed directory: The rightmost, last, name of a specified [path](https://en.wikipedia.org/wiki/Path_%28computing%29) suffixed by '/' - indicative of a directory.

######SOURCE Directory Content: An existing directory path appended with '/.'.

The multi-SOURCE copy semantics simply converge to the row labeled: '**TARGET exists as directory.**' above.  In this situation any SOURCE type, whether it a file, directory, or stream is successfully copied, as long as the TARGET refers to a preexisting directory, otherwise, the operation fails.

##### Images as SOURCE/TARGET
A double colon '```::```' delimiter classifies the file path as referring to an image, differentiating it from the single one denoting a container reference.  Therefore, an argument referencing an image involving a tag would appear similar to: '```repository_name:tag::etc/hostname```'.

When processing image arguments, ```dkrcp``` prefers binding to images known locally to Docker Engine that match the provided name and will ignore remote ones unless directed to include them.  To search remote registries, specify the option: ```--ucpchk-reg=true```.  Enabling this feature will cause ```dkrcp``` to initiate ```docker pull``` iff the specified image name is not locally known.  Note, when enabled, ```--ucpchk-reg``` applies to both SOURCE and TARGET image references. Therefore, in situations where the TARGET image name doesn't match a locally existing image but refers to an existing remote image, this remote image will be pulled and become the one referenced by TARGET.

Since copying to an existing TARGET image first applies this operation to a derived container (an image replica), its effects are "reversible".  Failures involving existing images simply delete the derived container leaving the repository unchanged.  However, when adding a new image to the local repository, the repository's state is first updated to reflect a [```scratch```](https://docs.docker.com/engine/userguide/eng-image/baseimages/#creating-a-simple-base-image-using-scratch) version of the image.  This ```scratch``` image is then updated in the same way as any existing TARGET image.  In this situation, a failure removes both the container and ```scratch``` image reverting the local repository's state.

######Copy *from* an *existing image*:
  * Convert the referenced image to a container via [```docker create```](https://docs.docker.com/engine/reference/commandline/create).
  * Copy from this container using ```docker cp```.
  * Destroy this container using [```docker rm```](https://docs.docker.com/engine/reference/commandline/rm/).

######Copy *to* an *existing image*:
  * Convert the referenced image to a container via ```docker create```.
  * Copy to this container using ```docker cp```.
    * When both the SOURCE and TARGET involve containers, the initial copy strategy streams ```docker cp``` output of the SOURCE container to a receiving ```docker cp``` that accepts this output as its input to the TARGETed container.  As long as the TARGET references an existing directory, the copy succeeds completing the operation.  However, if this copy strategy should fail, a second strategy executes a series of ```docker cp``` operations.  The first copy in this series, replicates the SOURCE artifacts to a temporary directory in the host environment executing ```dkrcp```.  A second ```docker cp``` then relays this SOURCE replica to the TARGET container.
  * If copy succeeds, ```dkrcp``` converts this container's state to an image using [```docker commit```](https://docs.docker.com/engine/reference/commandline/commit/).  
    * Specifying an image name as a TARGET argument propagates this name to ```docker commit``` superseding the originally named image.
    * When processing multiple SOURCE arguments, ```dkrcp``` delays the commit until after iterating over all of them.
  * If copy fails, ```dkrcp``` bypasses the commit.
  * Destroy this container using ```docker rm```.

######Copy *to create* an *image*:
  * Execute a ```docker build``` using [```FROM scratch```](https://docs.docker.com/engine/userguide/eng-image/baseimages/#creating-a-simple-base-image-using-scratch).
  * Continue with [Copy *to* an *existing image*](https://github.com/WhisperingChaos/dkrcp/blob/master/README.md#copy-to-an-existing-image).

#####Interweaved Copying
The behavior of ```dkrcp``` in situations where the same container assumes both SOURCE and TARGET roles is undefined and may change.  Preliminary testing indicates that non-overlapping directory references are copied as expected.  In fact, a fully overlapping root to root copy operation succeeds but no time has been invested to determine its correctness.  Therefore, exercise caution when copying between the same SOURCE and TARGET containers.  Copy operations involving the same SOURCE and TARGET image fortunately avert this uncertainty.

When operating on the same SOURCE and TARGET image, ```dkrcp``` converts both to independent container instances.  The use of independent containers prevents possible entanglement of the copy streams.  Therefore ```dkrcp```'s behavior (not its implementation) should be identical to: copy from source container to host then copy from host to target container.

#####Permissions
Since ```dkrcp``` wraps ```docker cp``` it applies file system permissions according to ```docker cp``` semantics.  ```docker cp``` currently replaces Linux ```UID:GID``` file system settings with the ```UID:GID``` of the account executing ```docker cp``` when copying from a container.  It then reverses this behavior when copying to a TARGET container, by replacing both the SOURCE ```UID:GID``` with the Linux root ID ('1').  Caution should be exercised as these permission semantics will eliminate custom ```UID:GID``` settings applied to SOURCE or TARGET file system objects.  The same permission semantics apply to images.  

#####follow-link
(```--follow-link,-L```)'s usual behavior replaces a symbolic link with with a physical copy of it's dereferenced object.  When coupled with ```cp -aL``` link replacement occurs for every element of the recursively produced list of subdirectories/files for a SOURCE argument that's a directory.  Currently, ```dkrcp```'s limits ```--follow-link``` behavior to only those symbolic links specified as SOURCE arguments.  Therefore, a SOURCE argument referencing a symbolic link that's associated to file is replaced by a copy of the file with similar behavior applied to a SOURCE argument symbolic link associated to a directory.  However, in situations involving a SOURCE argument referencing a symbolically linked directory or an actual one, ```dkrcp``` eschews typical ```--forward-link``` behavior by preserving, instead of replacing, symbolic links of the SOURCE directory's recursively enumerated file and subdirectory symbolic links.  This behavior demonstrated below mirrors the current [design](https://github.com/docker/docker/issues/21146) of ```docker cp --follow-link```. 
```
# host 'xlink' is a symbolic link to a directory.  this directory also contains
# a symbolic link named 'xfilelink'. 

root@d2c40c688111:/# ls -al xlink
lrwxrwxrwx 1 root root 2 Mar 14 19:41 xlink -> /x
root@d2c40c688111:/# cd xlink
root@d2c40c688111:/xlink# ls -al
total 8
drwxr-xr-x  2 root root 4096 Mar 14 19:42 .
drwxr-xr-x 42 root root 4096 Mar 14 19:41 ..
-rw-r--r--  1 root root    0 Mar 14 19:35 xfile
lrwxrwxrwx  1 root root    8 Mar 14 19:42 xfilelink -> /x/xfile

# specify 'xlink' symbolic link as SOURCE argument to 'dkrcp' targeting a container.
root@d2c40c688111:/# dkrcp --follow-link -- xlink d1b47bc8:/

# result of '--follow-link' copy.  notice 'xlink' conversion to physical directory
# but 'xfilelink' remains a link.  'xfilelink' would have been converted to a physical
# file if 'dkrcp --follow-link' fully implemented 'cp -aL'.
/ # ls -al xlink
total 8
drwxr-xr-x    2 root     root          4096 Mar 14 19:42 .<commit_msg>root UID:GID incorrect replace with (0:0)<commit_after># dkrcp
Copy files between host's file system, containers, and images.
#####ToC
[Copy Semantics](#copy-semantics)  
&nbsp;&nbsp;&nbsp;&nbsp;[Images as SOURCE/TARGET](#images-as-sourcetarget)  
&nbsp;&nbsp;&nbsp;&nbsp;[Interweaved Copying](#interweaved-copying)  
&nbsp;&nbsp;&nbsp;&nbsp;[Permissions](#permissions)  
&nbsp;&nbsp;&nbsp;&nbsp;[follow-link](#follow-link)  
&nbsp;&nbsp;&nbsp;&nbsp;[Examples](#examples)  
[Installing](#install)  
[Testing](#testing)  
[Motivation](#motivation)

```
Usage:  [OPTIONS] SOURCE [SOURCE]... TARGET 

  SOURCE - Can be either: 
             host file path     : {<relativepath>|<absolutePath>}
             image file path    : [<nameSpace>/...]{<repositoryName>:[<tagName>]|<UUID>}::{<relativepath>|<absolutePath>}
             container file path: {<containerName>|<UUID>}:{<relativepath>|<absolutePath>}
             stream             : -
  TARGET - See SOURCE.

  Copy SOURCE to TARGET.  SOURCE or TARGET must refer to either a container/image.
  <relativepath> within the context of a container/image is relative to
  container's/image's '/' (root).

OPTIONS:
    --ucpchk-reg=false        Don't pull images from registry. Limits image name
                                resolution to Docker local repository for  
                                both SOURCE and TARGET names.
    --author="",-a            Specify maintainer when TARGET is an image.
    --change[],-c             Apply specified Dockerfile instruction(s) when
                                TARGET is an image. see 'docker commit'
    --message="",-m           Apply commit message when TARGET is an image.
    --follow-link=false,-L    Follow symbolic link when link specified as a 
                                SOURCE argument. Different from 'cp -aL'!
    --help=false,-h           Don't display this help message.
    --version=false           Don't display version info.
```

Supplements [```docker cp```](https://docs.docker.com/engine/reference/commandline/cp/) by:
  * Facilitating image creation or adaptation by simply copying files.  When copying to an existing image, its state is unaffected, as copy preserves its immutability by creating a new layer.
  * Enabling the specification of multiple copy sources, including other images, to improve operational alignment with Linux [```cp -a```](https://en.wikipedia.org/wiki/Cp_%28Unix%29) and minimize layer creation when TARGET refers to an image.
  * Supporting the direct expression of copy semantics where SOURCE and TARGET arguments concurrently refer to containers.
 
#### Copy Semantics
```dkrcp``` relies on ```docker cp```, therefore, ```docker cp```'s [documentation](https://docs.docker.com/engine/reference/commandline/cp/) describes much of the expected behavior of ```dkrcp```, especially when specifying a single SOURCE argument.  Due to this reliance ```docker cp``` explanations concerning:
  * tarball streams ' - ',
  * container's root directory treated as the current one when copying to/from relative paths,
  * working directory of ```docker cp``` anchoring relative host file system paths,
  * desire to mimic ```cp -a``` recursive navigation and preserve file permissions,
  * ownership UID/GID permission settings when copying to/from containers, 
  * use of ':' as means of delimiting a container UUID/name from its associated path,
  * inability to copy certain files,

are all applicable to ```dkrcp```.

However, the following tabular form offers an equivalent description of copy behavior but visually different than the documention associated to ```docker cp```.

|         | SOURCE File  | SOURCE Directory | [SOURCE Directory Content](https://github.com/WhisperingChaos/dkrcp/blob/master/README.md#source-directory-content-an-existing-directory-path-appended-with-) | SOURCE Stream |
| :--:    | :----------: | :---------:| :----------: | :----------: |
| **TARGET exists as file.** | Copy SOURCE content to TARGET. | Error |Error | Error |
| **TARGET name does not exist but its parent directory does.** | TARGET File name assumed. Copy SOURCE content to TARGET.| TARGET directory name assumed. Create TARGET directory. Copy SOURCE content to TARGET. | Identical behavior to adjacent left hand cell. | Error |
| **TARGET name does not exist, nor does its parent directory.** | Error | Error | Error | Error|
| **TARGET exists as directory.** | Copy SOURCE to TARGET. | Copy SOURCE to TARGET. | Copy SOURCE content to TARGET. | Copy File/Directory to TARGET. |
| **[TARGET assumed directory](https://github.com/WhisperingChaos/dkrcp/blob/master/README.md#target-assumed-directory-the-rightmost-last-name-of-a-specified-path-suffixed-by---it-is-assumed-to-reference-an-existing-directory) doesn't exist but its parent directory does.** | Error | Copy SOURCE content to TARGET. | Copy SOURCE content to TARGET. | Error |

######TARGET assumed directory: The rightmost, last, name of a specified [path](https://en.wikipedia.org/wiki/Path_%28computing%29) suffixed by '/' - indicative of a directory.

######SOURCE Directory Content: An existing directory path appended with '/.'.

The multi-SOURCE copy semantics simply converge to the row labeled: '**TARGET exists as directory.**' above.  In this situation any SOURCE type, whether it a file, directory, or stream is successfully copied, as long as the TARGET refers to a preexisting directory, otherwise, the operation fails.

##### Images as SOURCE/TARGET
A double colon '```::```' delimiter classifies the file path as referring to an image, differentiating it from the single one denoting a container reference.  Therefore, an argument referencing an image involving a tag would appear similar to: '```repository_name:tag::etc/hostname```'.

When processing image arguments, ```dkrcp``` prefers binding to images known locally to Docker Engine that match the provided name and will ignore remote ones unless directed to include them.  To search remote registries, specify the option: ```--ucpchk-reg=true```.  Enabling this feature will cause ```dkrcp``` to initiate ```docker pull``` iff the specified image name is not locally known.  Note, when enabled, ```--ucpchk-reg``` applies to both SOURCE and TARGET image references. Therefore, in situations where the TARGET image name doesn't match a locally existing image but refers to an existing remote image, this remote image will be pulled and become the one referenced by TARGET.

Since copying to an existing TARGET image first applies this operation to a derived container (an image replica), its effects are "reversible".  Failures involving existing images simply delete the derived container leaving the repository unchanged.  However, when adding a new image to the local repository, the repository's state is first updated to reflect a [```scratch```](https://docs.docker.com/engine/userguide/eng-image/baseimages/#creating-a-simple-base-image-using-scratch) version of the image.  This ```scratch``` image is then updated in the same way as any existing TARGET image.  In this situation, a failure removes both the container and ```scratch``` image reverting the local repository's state.

######Copy *from* an *existing image*:
  * Convert the referenced image to a container via [```docker create```](https://docs.docker.com/engine/reference/commandline/create).
  * Copy from this container using ```docker cp```.
  * Destroy this container using [```docker rm```](https://docs.docker.com/engine/reference/commandline/rm/).

######Copy *to* an *existing image*:
  * Convert the referenced image to a container via ```docker create```.
  * Copy to this container using ```docker cp```.
    * When both the SOURCE and TARGET involve containers, the initial copy strategy streams ```docker cp``` output of the SOURCE container to a receiving ```docker cp``` that accepts this output as its input to the TARGETed container.  As long as the TARGET references an existing directory, the copy succeeds completing the operation.  However, if this copy strategy should fail, a second strategy executes a series of ```docker cp``` operations.  The first copy in this series, replicates the SOURCE artifacts to a temporary directory in the host environment executing ```dkrcp```.  A second ```docker cp``` then relays this SOURCE replica to the TARGET container.
  * If copy succeeds, ```dkrcp``` converts this container's state to an image using [```docker commit```](https://docs.docker.com/engine/reference/commandline/commit/).  
    * Specifying an image name as a TARGET argument propagates this name to ```docker commit``` superseding the originally named image.
    * When processing multiple SOURCE arguments, ```dkrcp``` delays the commit until after iterating over all of them.
  * If copy fails, ```dkrcp``` bypasses the commit.
  * Destroy this container using ```docker rm```.

######Copy *to create* an *image*:
  * Execute a ```docker build``` using [```FROM scratch```](https://docs.docker.com/engine/userguide/eng-image/baseimages/#creating-a-simple-base-image-using-scratch).
  * Continue with [Copy *to* an *existing image*](https://github.com/WhisperingChaos/dkrcp/blob/master/README.md#copy-to-an-existing-image).

#####Interweaved Copying
The behavior of ```dkrcp``` in situations where the same container assumes both SOURCE and TARGET roles is undefined and may change.  Preliminary testing indicates that non-overlapping directory references are copied as expected.  In fact, a fully overlapping root to root copy operation succeeds but no time has been invested to determine its correctness.  Therefore, exercise caution when copying between the same SOURCE and TARGET containers.  Copy operations involving the same SOURCE and TARGET image fortunately avert this uncertainty.

When operating on the same SOURCE and TARGET image, ```dkrcp``` converts both to independent container instances.  The use of independent containers prevents possible entanglement of the copy streams.  Therefore ```dkrcp```'s behavior (not its implementation) should be identical to: copy from source container to host then copy from host to target container.

#####Permissions
Since ```dkrcp``` wraps ```docker cp``` it applies file system permissions according to ```docker cp``` semantics.  ```docker cp``` currently replaces Linux ```UID:GID``` file system settings with the ```UID:GID``` of the account executing ```docker cp``` when copying from a container.  It then reverses this behavior when copying to a TARGET container, by replacing both the SOURCE ```UID:GID``` with the Linux root ID ('0:0').  Caution should be exercised as these permission semantics will eliminate custom ```UID:GID``` settings applied to SOURCE or TARGET file system objects.  The same permission semantics apply to images.  

#####follow-link
(```--follow-link,-L```)'s usual behavior replaces a symbolic link with with a physical copy of it's dereferenced object.  When coupled with ```cp -aL``` link replacement occurs for every element of the recursively produced list of subdirectories/files for a SOURCE argument that's a directory.  Currently, ```dkrcp```'s limits ```--follow-link``` behavior to only those symbolic links specified as SOURCE arguments.  Therefore, a SOURCE argument referencing a symbolic link that's associated to file is replaced by a copy of the file with similar behavior applied to a SOURCE argument symbolic link associated to a directory.  However, in situations involving a SOURCE argument referencing a symbolically linked directory or an actual one, ```dkrcp``` eschews typical ```--forward-link``` behavior by preserving, instead of replacing, symbolic links of the SOURCE directory's recursively enumerated file and subdirectory symbolic links.  This behavior demonstrated below mirrors the current [design](https://github.com/docker/docker/issues/21146) of ```docker cp --follow-link```. 
```
# host 'xlink' is a symbolic link to a directory.  this directory also contains
# a symbolic link named 'xfilelink'. 

root@d2c40c688111:/# ls -al xlink
lrwxrwxrwx 1 root root 2 Mar 14 19:41 xlink -> /x
root@d2c40c688111:/# cd xlink
root@d2c40c688111:/xlink# ls -al
total 8
drwxr-xr-x  2 root root 4096 Mar 14 19:42 .
drwxr-xr-x 42 root root 4096 Mar 14 19:41 ..
-rw-r--r--  1 root root    0 Mar 14 19:35 xfile
lrwxrwxrwx  1 root root    8 Mar 14 19:42 xfilelink -> /x/xfile

# specify 'xlink' symbolic link as SOURCE argument to 'dkrcp' targeting a container.
root@d2c40c688111:/# dkrcp --follow-link -- xlink d1b47bc8:/

# result of '--follow-link' copy.  notice 'xlink' conversion to physical directory
# but 'xfilelink' remains a link.  'xfilelink' would have been converted to a physical
# file if 'dkrcp --follow-link' fully implemented 'cp -aL'.
/ # ls -al xlink
total 8
drwxr-xr-x    2 root     root          4096 Mar 14 19:42 .
{'fraction_non_alphanumeric': 0.10729329173166927, 'fraction_numerical': 0.009009360374414977, 'mean_word_length': 4.124100719424461, 'pattern_counts': {'":': 0, '<': 29, '<?xml version=': 0, '>': 33, 'https://': 26, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '4834592', 'n_tokens_mistral': 7192, 'n_tokens_neox': 6604, 'n_words': 3013}