<commit_before># gooddata-agent

A simple command line tool for uploading data to GoodData's WebDAV storage

## Usage

*Download:*

from http://gooddata.s3.amazonaws.com/agent/gdc-agent-0.9.jar

*Configure:*

Create and edit the properties file (e.g. "my.properties") based on the commented sample configuration file below. 

*Run:*

    java -jar target/gdc-agent-0.9.jar my.properties

Note: the following configuration parameters can be overriden using
command line parameters:
  * `jdbc.driver`: `--jdbc-driver`
  * `jdbc.driverPath`: `--jdbc-driverPath`
  * `jdbc.password`: `--jdbc-password`
  * `jdbc.url`: `--jdbc-url`
  * `jdbc.username`: `--jdbc-username`
  * `gdc.password`: <PASSWORD>`
  * `gdc.username`: `--gdc-username`

For example:

    java -jar target/gdc-agent-0.8.jar --jdbc-username=admin --jdbc-password=<PASSWORD> my.properties

*Triggering server side ETL:*

The agent can optionally run a server side ETL process. It will
automatically send information about the uploaded file in the ETL
parameters: `gdc_agent_zip`/ `gdc_agent_manifest` and
`gdc_agent_zip_url_nocreds`/`gdc_agent_manifest_url_nocreds`.

Moreover, if the `gdc.etl.send_credentials` property is set to
`true`, the agent tool will also sent the following parameters:
`gdc_username`, `gdc_password` and `gdc_agent_zip_url` or
`gdc_agent_manifest_url`.

Note the gdc_agent_zip_url and gdc_agent_manifest_url parameters
will include the full URL of the uploaded file including the
credentials, for example
`https://user%40company.com:P4$$w0rd@secure-di.gooddata.com/project-uploads/fqp21nrdqm666u5nwgl0upsgee4a4xv2/data-20141223161932.zip`.

In addition, you can provide custom ETL parameters using
property keys prefixed `gdc.etl.param.` or `gdc.etl.hidden_param.`

*Sample Configuration:*

    ##########################################
    ### Source data on a local file system ###
    ##########################################

    # include all CSV files from that directory in the zip file
    filesystem.input_dir=/tmp/exports/to_gooddata
    filesystem.wildcard=*.csv

    #################################
    ### Source data in a database ###
    #################################
    jdbc.driver_path=/usr/share/java/mysql-connector-java-5.1.5.jar
    jdbc.driver=com.mysql.jdbc.Driver
    jdbc.username=extracts
    jdbc.password=*******
    jdbc.url=jdbc:mysql://localhost/test<commit_msg>ETL parameters related minor update<commit_after># gooddata-agent

A simple command line tool for uploading data to GoodData's WebDAV storage

## Usage

*Download:*

from http://gooddata.s3.amazonaws.com/agent/gdc-agent-0.9.jar

*Configure:*

Create and edit the properties file (e.g. "my.properties") based on the commented sample configuration file below. 

*Run:*

    java -jar target/gdc-agent-0.9.jar my.properties

Note: the following configuration parameters can be overriden using
command line parameters:
  * `jdbc.driver`: `--jdbc-driver`
  * `jdbc.driverPath`: `--jdbc-driverPath`
  * `jdbc.password`: <PASSWORD>`
  * `jdbc.url`: `--jdbc-url`
  * `jdbc.username`: `--jdbc-username`
  * `gdc.password`: `--gdc-password`
  * `gdc.username`: `--gdc-username`

For example:

    java -jar target/gdc-agent-0.8.jar --jdbc-username=admin --jdbc-password=<PASSWORD> my.properties

*Triggering server side ETL:*

The agent can optionally run a server side ETL process. It will
automatically send information about the uploaded file in the ETL
parameters: `gdc_agent_zip`/ `gdc_agent_manifest` and
`gdc_agent_zip_url_nocreds`/`gdc_agent_manifest_url_nocreds`.

Moreover, if the `gdc.etl.send_credentials` property is set to
`true`, the agent tool will also sent the following parameters:
`gdc_username`, `gdc_password` and `gdc_agent_zip_url` or
`gdc_agent_manifest_url`.

Note you can use the gdc_agent_zip_url or gdc_agent_manifest_url variable
in the FileDownloader CloudConnect component as it will include the full URL of the uploaded file including the
credentials, for example
`https://user%40company.com:P4$$w0rd@secure-di.gooddata.com/project-uploads/fqp21nrdqm666u5nwgl0upsgee4a4xv2/data-20141223161932.zip`.

In addition, you can provide custom ETL parameters using
property keys prefixed `gdc.etl.param.` or `gdc.etl.hidden_param.`

_Caution:_ never put any sensitive information directly into your
CloudConnect graph or into the workspace.prm CloudConnect configuration file.

*Sample Configuration:*

    ##########################################
    ### Source data on a local file system ###
    ##########################################

    # include all CSV files from that directory in the zip file
    filesystem.input_dir=/tmp/exports/to_gooddata
    filesystem.wildcard=*.csv

    #################################
    ### Source data in a database ###
    #################################
    jdbc.driver_path=/usr/share/java/mysql-connector-java-5.1.5.jar
    jdbc.driver=com.mysql.jdbc.Driver
    jdbc.username=extracts
    jdbc.password=*******
    jdbc.url=jdbc:mysql://localhost/test
{'fraction_non_alphanumeric': 0.16427145708582835, 'fraction_numerical': 0.015169660678642715, 'mean_word_length': 4.643018018018018, 'pattern_counts': {'":': 0, '<': 7, '<?xml version=': 0, '>': 7, 'https://': 2, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 2, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 3, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '1705363', 'n_tokens_mistral': 1688, 'n_tokens_neox': 1551, 'n_words': 425}