<commit_before>Qubole Data Service Python SDK
==============================

A Python module that provides the tools you need to authenticate with, and use the Qubole Data Service API.


Installation
------------

$ python setup.py install # may need to do this as root


This should place a command line utility 'qds.py' somewhere in your path

$ which qds.py
/usr/bin/qds.py 


CLI
---

qds.py allows running Hive, Hadoop and Pig commands against QDS. Users can run commands synchronously - or submit a command and check it's status.

$ qds.py -h # will print detailed usage

Examples:

# run a hive query and print the results
$ qds.py --token '<PASSWORD>' hivecmd run --query "show tables"

# pass in api token from bash var 
$ export QDS_API_TOKEN=xxyyzz

# run the example hadoop command
$ qds.py hadoopcmd run jar 's3://paid-qubole/HadoopAPIExamples/jars/hadoop-0.20.1-dev-streaming.jar' -files 's3n://paid-qubole/HadoopAPIExamples/WordCountPython/mapper.py,s3n://paid-qubole/HadoopAPIExamples/WordCountPython/reducer.py' -mapper mapper.py -reducer reducer.py -numReduceTasks 1 -input 's3n://paid-qubole/default-datasets/gutenberg' -output 's3n://example.bucket.com/wcout'

# check the status of command # 12345678
$ qds.py hivecmd check 12345678
{"status": "done", ... }


SDK API
-------

An example Python application needs to do the following:

1.  Set the api_token:

        from qubole import Qubole

        Qubole.configure(api_token='<KEY>')

2.  Use the Command classes defined in commands.py to execute commands. To run Hive Command:

        from commands import *

        hc=HiveCommand.create(query='show tables')
        print "Id: %s, Status: %s" % (str(hc.id), hc.status)<commit_msg>README GFM friend friend<commit_after>Qubole Data Service Python SDK
==============================

A Python module that provides the tools you need to authenticate with, and use the Qubole Data Service API.


Installation
------------

$ python setup.py install # may need to do this as root


This should place a command line utility 'qds.py' somewhere in your path

$ which qds.py
/usr/bin/qds.py 


CLI
---

qds.py allows running Hive, Hadoop and Pig commands against QDS. Users can run commands synchronously - or submit a command and check it's status.

$ qds.py -h # will print detailed usage

Examples:

// run a hive query and print the results
$ qds.py --token '<PASSWORD>' hivecmd run --query "show tables"

// pass in api token from bash var 
$ export QDS_API_TOKEN=<PASSWORD>

// run the example hadoop command
$ qds.py hadoopcmd run jar 's3://paid-qubole/HadoopAPIExamples/jars/hadoop-0.20.1-dev-streaming.jar' -files 's3n://paid-qubole/HadoopAPIExamples/WordCountPython/mapper.py,s3n://paid-qubole/HadoopAPIExamples/WordCountPython/reducer.py' -mapper mapper.py -reducer reducer.py -numReduceTasks 1 -input 's3n://paid-qubole/default-datasets/gutenberg' -output 's3n://example.bucket.com/wcout'

// check the status of command # 12345678
$ qds.py hivecmd check 12345678
{"status": "done", ... }


SDK API
-------

An example Python application needs to do the following:

1.  Set the api_token:

        from qubole import Qubole

        Qubole.configure(api_token='<KEY>')

2.  Use the Command classes defined in commands.py to execute commands. To run Hive Command:

        from commands import *

        hc=HiveCommand.create(query='show tables')
        print "Id: %s, Status: %s" % (str(hc.id), hc.status)
{'fraction_non_alphanumeric': 0.12445478336725792, 'fraction_numerical': 0.01628380343123001, 'mean_word_length': 3.4908616187989554, 'pattern_counts': {'":': 2, '<': 8, '<?xml version=': 0, '>': 8, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '6214578', 'n_tokens_mistral': 1180, 'n_tokens_neox': 1096, 'n_words': 376}