<commit_before>---
layout: default
title: Sqoop
category: Services
type: SERVICES
---
# Using Sqoop

Sqoop is a tool to migrate data from structured datastores into the Haddop file system (HDFS). It works much like any other tool installed with Hadoop, and requires a command with arguments.

## Using Sqoop on Oracle 12c
An example command, run on our system looks like this:

    sqoop import --verbose --connect "jdbc:oracle:thin:hadoop/hadoop@//numbertwo:1521/orcl" --username hadoop -P --table TYLER.TWEETS

### Command Explained
The _--verbose_ option was for debugging purposes, to find the error messages as we tried to make it work with our database instance.

The _--connect_ command is for connecting to the database.

The part in the string is our database's connect string, what sqoop will use to connect to the database and migrate the data.

_--username_ and _-P_ are for giving sqoop access to the database as a user with the right permissions on the specific table that needs to be imported. _-P_ will make sqoop ask for the password in the command window as a prompt, this is more secure than entering the password into the command itself.

###Things to watch out for

The connect string is case sensitive, and will cause many java exceptions to be thrown if there are any mistakes.

The table that is being imported will have to be set up with a primary key, or the import will fail. This may be something that doesn't happen often,<commit_msg>one last edition to sqoop file<commit_after>---
layout: default
title: Sqoop
category: Services
type: SERVICES
---
# Using Sqoop

Sqoop is a tool to migrate data from structured datastores into the Haddop file system (HDFS). It works much like any other tool installed with Hadoop, and requires a command with arguments.

## Using Sqoop on Oracle 12c
An example command, run on our system looks like this:

    sqoop import --verbose --connect "jdbc:oracle:thin:hadoop/hadoop@//numbertwo:1521/orcl" --username hadoop -P --table TYLER.TWEETS

### Command Explained
The _--verbose_ option was for debugging purposes, to find the error messages as we tried to make it work with our database instance.

The _--connect_ command is for connecting to the database.

The part in the string is our database's connect string, what sqoop will use to connect to the database and migrate the data.

_--username_ and _-P_ are for giving sqoop access to the database as a user with the right permissions on the specific table that needs to be imported. _-P_ will make sqoop ask for the password in the command window as a prompt, this is more secure than entering the password into the command itself.

TYLER.TWEETS is the table which contains the data that we want to import. This table specifically holds data from a twitter data dump.

###Things to watch out for

The connect string is case sensitive, and will cause many java exceptions to be thrown if there are any mistakes.

The table that is being imported will have to be set up with a primary key, or the import will fail. This may be something that doesn't happen often,
{'fraction_non_alphanumeric': 0.04839233517375771, 'fraction_numerical': 0.003897369275738876, 'mean_word_length': 4.0741350906095555, 'pattern_counts': {'":': 0, '<': 3, '<?xml version=': 0, '>': 3, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '7128619', 'n_tokens_mistral': 814, 'n_tokens_neox': 771, 'n_words': 493}