<jupyter_start><jupyter_text>How to use the Data Lab *Store Client* ServiceThis notebook documents how to use the Data Lab virtual storage system via the store client service. This can be done either from a Python script (e.g. within this notebook) or from the command line using the datalab command. The storage manager service interfaceThe store client service simplifies access to the Data Lab virtual storage system. This section describes the store client service interface in case we want to write our own code against that rather than using one of the provided tools. The store client service accepts an HTTP GET call to the appropriate endpoint for the particular operation:| Endpoint | Description | Req'd Parameters ||----------|-------------|------------|| /get | Retrieve a file | name || /put | Upload a file | name || /load | Load a file to vospace | name, endpoint || /cp | Copy a file/directory | from, to || /ln | Link a file/directory | from, to || /lock | Lock a node from write updates | name || /ls | Get a file/directory listing | name || /access | Determine file accessability | name || /stat | File status info | name,verbose || /mkdir | Create a directory | name || /mv | Move/rename a file/directory | from, to || /rm | Delete a file | name || /rmdir | Delete a directory | name || /tag | Annotate a file/directory | name, tag |For example, a call to http://datalab.noirlab.edu/storage/get?name=vos://mag.csv will retrieve the file '_mag.csv_' from the root directory of the user's virtual storage.  Likewise, a python call using the _storeClient_ interface such as "_storeClient.get('vos://mag.csv')_" would get the same file. Virtual storage identifiersFiles in the virtual storage are usually identified via the prefix "_vos://_". This shorthand identifier is resolved to a user's home directory of the storage space in the service.  As a convenience, the prefix may optionally be omitted when the parameter refers to a node in the virtual storage. Navigation above a user's home directory is not supported, however, subdirectories within the space may be created and used as needed. AuthenticationThe storage manager service requires a DataLab security token. This needs to be passed as the value of the header keyword "X-DL-AuthToken" in any HTTP GET call to the service. If the token is not supplied anonymous access is assumed but provides access only to public storage spaces.  From Python codeThe store client service can be called from Python code using the datalab module. This provides methods to access the various functions in the storeClient subpackage.  InitializationThis is the setup that is required to use the store client. The first thing to do is import the relevant Python modules and also retrieve our DataLab security token.<jupyter_code># Standard notebook imports
from getpass import getpass
from dl import authClient, storeClient<jupyter_output><empty_output><jupyter_text>Comment out and run the cell below if you need to login to Data Lab:<jupyter_code>## Get the authentication token for the user
#token = authClient.login(input("Enter user name: (+ENTER) "),getpass("Enter password: (+ENTER) "))
#if not authClient.isValidToken(token):
#    raise Exception('Token is not valid. Please check your usename/password and execute this cell again.')<jupyter_output><empty_output><jupyter_text>Listing a file/directoryWe can see all the files that are in a specific directory or get a full listing for a specific file.  In this case, we'll list the default virtual storage directory to use as a basis for changes we'll make below.<jupyter_code>listing = storeClient.ls (name = 'vos://')
print (listing)<jupyter_output>cutout.fits,public,results,tmp<jupyter_text>The *public* directory shown here is visible to all Data Lab users and provides a means of sharing data without having to setup special access.  Similarly, the *tmp* directory is read-protected and provides a convenient temporary directory to be used in a workflow.  File Existence and InfoAside from simply listing files, it's possible to test whether a named file already exists or to determine more information about it.<jupyter_code># A simple file existence test:
if storeClient.access ('vos://public'):
    print ('User "public" directory exists')
if storeClient.access ('vos://public', mode='w'):
    print ('User "public" directory is group/world writable')
else:
    print ('User "public" directory is not group/world writable')
    
if storeClient.access ('vos://tmp'):
    print ('User "tmp" directory exists')        
if storeClient.access ('vos://tmp', mode='w'):
    print ('User "tmp" directory is group/world writable')
else:
    print ('User "tmp" directory is not group/world writable')<jupyter_output>User "public" directory exists
User "public" directory is not group/world writable
User "tmp" directory exists
User "tmp" directory is not group/world writable<jupyter_text>Uploading a fileNow we want to upload a new data file from our local disk to the virtual storage:<jupyter_code>storeClient.put (to = 'vos://newmags.csv', fr = './newmags.csv')
print(storeClient.ls (name='vos://'))<jupyter_output>(1 / 1) ./newmags.csv -> vos://newmags.csv
cutout.fits,newmags.csv,public,results,tmp<jupyter_text>Downloading a fileLet's say we want to download a file from our virtual storage space, in this case a query result that we saved to it in the "How to use the Data Lab query manager service" notebook:<jupyter_code>storeClient.get (fr = 'vos://newmags.csv', to = './mymags.csv')<jupyter_output>(1/1) [====================] [   142B] newmags.csv<jupyter_text>It is also possible to get the contents of a remote file directly into your notebook by specifying the location as an empty string:<jupyter_code>data = storeClient.get (fr = 'vos://newmags.csv', to = '')
print (data)<jupyter_output>id,g,r,i
001,22.3,12.4,21.5
002,22.3,12.4,21.5
003,22.3,12.4,21.5
004,22.3,12.4,21.5
005,22.3,12.4,21.5
006,22.3,12.4,21.5
007,22.3,12.4,21.5<jupyter_text>Loading a file from a remote URLIt is possible to load a file directly to virtual storage from a remote URL )e.g. an "accessURL" for an image cutout, a remote data file, etc) using the "storeClient.load()" method:<jupyter_code>url = "http://datalab.noirlab.edu/svc/cutout?col=&siaRef=c4d_161005_022804_ooi_g_v1.fits.fz&extn=31&POS=335.0,0.0&SIZE=0.1"
storeClient.load('vos://cutout.fits',url)<jupyter_output><empty_output><jupyter_text>Creating a directoryWe can create a directory on the remote storage to be used for saving data later:<jupyter_code>storeClient.mkdir ('vos://results')<jupyter_output><empty_output><jupyter_text>Copying a file/directoryWe want to put a copy of the file in a remote work directory:<jupyter_code>storeClient.mkdir ('vos://temp')
print ("Before: " + storeClient.ls (name='vos://temp/'))
storeClient.cp (fr = 'vos://newmags.csv', to = 'vos://temp/newmags.csv',verbose=True)
print ("After: " + storeClient.ls (name='vos://temp/'))
print(storeClient.ls('vos://',format='long'))<jupyter_output>-rw-rw-r-x  demo01  2963520  22 Nov 2021 14:22  cutout.fits
-rw-rw-r-x  demo01     142  30 Nov 2021 14:58  newmags.csv
drwxrwxr-x  demo01       0  14 Jul 2020 10:01  public/
drwxrwxr-x  demo01       0  22 Nov 2021 14:22  results/
drwxrwxr-x  demo01       0  30 Nov 2021 14:58  temp/
drwxrwx---  demo01       0  14 Jul 2020 10:01  tmp/<jupyter_text>Notice that in the *ls()* call we append the directory name with a trailing '/' to list the contents of the directory rather than the directory itself.  Linking to a file/directory**WARNING**: Linking is currently **not** working in the Data Lab storage manager. This notebook will be updated when the problem has been resolved.Sometimes we want to create a link to a file or directory.  In this case, the link named by the *'fr'* parameter is created and points to the file/container named by the *'target'* parameter.<jupyter_code>storeClient.ln ('vos://mags.csv', 'vos://temp/newmags.csv')
print ("Root dir: " + storeClient.ls (name='vos://'))
print ("Temp dir: " + storeClient.ls (name='vos://temp/'))<jupyter_output>Root dir: cutout.fits,newmags.csv,public,results,temp,tmp
Temp dir: newmags.csv<jupyter_text>Moving/renaming a file/directoryWe can move a file or directory:<jupyter_code>storeClient.mv(fr = 'vos://temp/newmags.csv', to = 'vos://results')
print ("Results dir: " + storeClient.ls (name='vos://results/'))<jupyter_output>Results dir: newmags.csv<jupyter_text>Deleting a fileWe can delete a file:<jupyter_code>print ("Before: " + storeClient.ls (name='vos://'))
storeClient.rm (name = 'vos://mags.csv')
print ("After: " + storeClient.ls (name='vos://'))<jupyter_output>Before: cutout.fits,newmags.csv,public,results,temp,tmp
After: cutout.fits,newmags.csv,public,results,temp,tmp<jupyter_text>Deleting a directoryWe can also delete a directory, doing so also deletes the contents of that directory:<jupyter_code>storeClient.rmdir(name = 'vos://temp')<jupyter_output><empty_output><jupyter_text>Tagging a file/directory**Warning**: Tagging is currently **not** working in the Data Lab storage manager. This notebook will be updated when the problem has been resolved.We can tag any file or directory with arbitrary metadata:<jupyter_code>storeClient.tag('vos://results', 'The results from my analysis')<jupyter_output><empty_output><jupyter_text>Cleanup the demo directory of remaining files<jupyter_code>storeClient.rm (name = 'vos://newmags.csv')
storeClient.rm (name = 'vos://results')
storeClient.ls (name = 'vos://')<jupyter_output><empty_output><jupyter_text>Using the datalab commandThe datalab command provides an alternate command line way to work with the query manager through the query subcommands, which is especially useful if you want to interact with the query manager from your local computer. Please have the `datalab` command line utility installed first (for install instructions see https://github.com/astro-datalab/datalab ). The cells below are commented out. Copy and paste any of them (without the comment sign) and run locally.  Log in once<jupyter_code>#!datalab login<jupyter_output><empty_output><jupyter_text>and enter the credentials as prompted.  Downloading a fileLet's say we want to download a file from our virtual storage space:<jupyter_code>#!datalab get fr="vos://mags.csv" to="./mags.csv"<jupyter_output><empty_output><jupyter_text>Uploading a fileNow we want to upload a new data file from our local disk:<jupyter_code>#!datalab put fr="./newmags.csv" to="vos://newmags.csv"<jupyter_output><empty_output><jupyter_text>Copying a file/directoryWe want to put a copy of the file in a remote work directory:<jupyter_code>#!datalab cp fr="vos://newmags.csv" to="vos://temp/newmags.csv"<jupyter_output><empty_output><jupyter_text>Linking to a file/directorySometimes we want to create a link to a file or directory:<jupyter_code>#!datalab ln fr="vos://temp/mags.csv" to="vos://mags.csv"<jupyter_output><empty_output><jupyter_text>Listing a file/directoryWe can see all the files that are in a specific directory or get a full listing for a specific file:<jupyter_code>#!datalab ls name="vos://temp"<jupyter_output><empty_output><jupyter_text>Creating a directoryWe can create a directory:<jupyter_code>#!datalab mkdir name="vos://results"<jupyter_output><empty_output><jupyter_text>Moving/renaming a file/directoryWe can move a file or directory:<jupyter_code>#!datalab mv fr="vos://temp/newmags.csv" to="vos://results"<jupyter_output><empty_output><jupyter_text>Deleting a fileWe can delete a file:<jupyter_code>#!datalab rm name="vos://temp/mags.csv"<jupyter_output><empty_output><jupyter_text>Deleting a directoryWe can also delete a directory:<jupyter_code>#!datalab rmdir name="vos://temp"<jupyter_output><empty_output><jupyter_text>Tagging a file/directoryWe can tag any file or directory with arbitrary metadata:<jupyter_code>#!datalab tag name="vos://results" tag="The results from my analysis"<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.10887603721398038, 'fraction_numerical': 0.01684686949962283, 'mean_word_length': 5.27669647553919, 'pattern_counts': {'":': 0, '<': 100, '<?xml version=': 0, '>': 101, 'https://': 1, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 4, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '27', 'dir': 'jupyter-structured-clean-dedup', 'id': '1593', 'license': 'BSD-3-Clause', 'n_tokens_mistral': 3787, 'n_tokens_neox': 3627, 'n_words': 1336, 'path': '04_HowTos/StoreClient/How_to_use_the_Data_Lab_StoreClient.ipynb', 'repo_name': 'noaodatalab/notebooks_default', 'size': '11931'}