<commit_before>
////
  Licensed to the Apache Software Foundation (ASF) under one
  or more contributor license agreements.  See the NOTICE file
  distributed with this work for additional information
  regarding copyright ownership.  The ASF licenses this file
  to you under the Apache License, Version 2.0 (the
  "License"); you may not use this file except in compliance
  with the License.  You may obtain a copy of the License at
 
      http://www.apache.org/licenses/LICENSE-2.0
 
  Unless required by applicable law or agreed to in writing, software
  distributed under the License is distributed on an "AS IS" BASIS,
  WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
  See the License for the specific language governing permissions and
  limitations under the License.
////

Compatibility Notes
-------------------

Sqoop uses JDBC to connect to databases and adheres to
published standards as much as possible. For databases which do not
support standards-compliant SQL, Sqoop uses alternate codepaths to
provide functionality. In general, Sqoop is believed to be compatible
with a large number of databases, but it is tested with only a few.

Nonetheless, several database-specific decisions were made in the
implementation of Sqoop, and some databases offer additional settings
which are extensions to the standard.

This section describes the databases tested with Sqoop, any
exceptions in Sqoop's handling of each database relative to the
norm, and any database-specific settings available in Sqoop.

Supported Databases
~~~~~~~~~~~~~~~~~~~

While JDBC is a compatibility layer that allows a program to access
many different databases through a common API, slight differences in
the SQL language spoken by each database may mean that Sqoop can't use
every database out of the box, or that some databases may be used in
an inefficient manner.

When you provide a connect string to Sqoop, it inspects the protocol scheme to
determine appropriate vendor-specific logic to use. If Sqoop knows about
a given database, it will work automatically. If not, you may need to
specify the driver class to load via +\--driver+. This will use a generic
code path which will use standard SQL to access the database. Sqoop provides
some databases with faster, non-JDBC-based access mechanisms. These can be
enabled by specfying the +\--direct+ parameter.

Sqoop includes vendor-specific support for the following databases:

[grid="all"]
`-----------`--------`--------------------`---------------------
Database    version  +\--direct+ support? connect string matches
----------------------------------------------------------------
HSQLDB      1.8.0+   No                   +jdbc:hsqldb:*//+
MySQL       5.0+     Yes                  +jdbc:mysql://+
Oracle      10.2.0+  No                   +jdbc:oracle:*//+
PostgreSQL  8.3+     Yes (import only)    +jdbc:postgresql://+
----------------------------------------------------------------

Sqoop may work with older versions of the databases listed, but we have
only tested it with the versions specified above.

Even if Sqoop supports a database internally, you may still need to
install the database vendor's JDBC driver in your +$SQOOP_HOME/lib+
path on your client. Sqoop can load classes from any jars in
+$SQOOP_HOME/lib+ on the client and will use them as part of any
MapReduce jobs it runs; unlike older versions, you no longer need to
install JDBC jars in the Hadoop library path on your servers.

MySQL
~~~~~

JDBC Driver: http://www.mysql.com/downloads/connector/j/[MySQL
Connector/J]

MySQL v5.0 and above offers very thorough coverage by Sqoop. Sqoop
has been tested with +mysql-connector-java-5.1.13-bin.jar+.

zeroDateTimeBehavior
^^^^^^^^^^^^^^^^^^^^

MySQL allows values of +\'0000-00-00\'+ for +DATE+ columns, which is a
non-standard extension to SQL. When communicated via JDBC, these
values are handled in one of three different ways:

- Convert to +NULL+.
- Throw an exception in the client.
- Round to the nearest legal date (+\'0001-01-01\'+).

You specify the behavior by using the +zeroDateTimeBehavior+
property of the connect string. If a +zeroDateTimeBehavior+ property
is not specified, Sqoop uses the +convertToNull+ behavior.

You can override this behavior. For example:

----
$ sqoop import --table foo \
    --connect jdbc:mysql://db.example.com/someDb?zeroDateTimeBehavior=round
----

+UNSIGNED+ columns
^^^^^^^^^^^^^^^^^^

Columns with type +UNSIGNED+ in MySQL can hold values between 0 and
2^32 (+4294967295+), but the database will report the data type to Sqoop
as +INTEGER+, which will can hold values between +-2147483648+ and
+\+2147483647+. Sqoop cannot currently import +UNSIGNED+ values above
+2147483647+.

+BLOB+ and +CLOB+ columns
^^^^^^^^^^^^^^^^^^^^^^^^^

Sqoop's direct mode does not support imports of +BLOB+, +CLOB+, or
+LONGVARBINARY+ columns. Use JDBC-based imports for these
columns; do not supply the +\--direct+ argument to the import tool.

Importing views in direct mode
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

Sqoop is currently not supporting import from view in direct mode. Use
JDBC based (non direct) mode in case that you need to import view (simply
omit +--direct+ parameter).

Direct-mode Transactions
^^^^^^^^^^^^^^^^^^^^^^^^

For performance, each writer will commit the current transaction
approximately every 32 MB of exported data. You can control this
by specifying the following argument _before_ any tool-specific arguments: +-D
sqoop.mysql.export.checkpoint.bytes=size+, where _size_ is a value in
bytes. Set _size_ to 0 to disable intermediate checkpoints,
but individual files being exported will continue to be committed
independently of one another.

IMPORTANT: Note that any arguments to Sqoop that are of the form +-D
parameter=value+ are Hadoop _generic arguments_ and must appear before
any tool-specific arguments (for example, +\--connect+, +\--table+, etc).

PostgreSQL
~~~~~~~~~~

Sqoop supports JDBC-based connector for PostgreSQL: http://jdbc.postgresql.org/

The connector has been tested using JDBC driver version "9.1-903 JDBC 4" with
PostgreSQL server 9.1.

Importing views in direct mode
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

Sqoop is currently not supporting import from view in direct mode. Use
JDBC based (non direct) mode in case that you need to import view (simply
omit +--direct+ parameter).

Oracle
~~~~~~

JDBC Driver:
http://www.oracle.com/technology/software/tech/java/sqlj_jdbc/htdocs/jdbc_112010.html[Oracle
JDBC Thin Driver] - Sqoop is compatible with +ojdbc6.jar+.

Sqoop has been tested with Oracle 10.2.0 Express Edition. Oracle is
notable in its different approach to SQL from the ANSI standard, and
its non-standard JDBC driver. Therefore, several features work
differently.

Dates and Times
^^^^^^^^^^^^^^^<commit_msg>SQOOP-683: Documenting sqoop.mysql.export.sleep.ms - easy throttling feature for direct MySQL exports<commit_after>
////
  Licensed to the Apache Software Foundation (ASF) under one
  or more contributor license agreements.  See the NOTICE file
  distributed with this work for additional information
  regarding copyright ownership.  The ASF licenses this file
  to you under the Apache License, Version 2.0 (the
  "License"); you may not use this file except in compliance
  with the License.  You may obtain a copy of the License at
 
      http://www.apache.org/licenses/LICENSE-2.0
 
  Unless required by applicable law or agreed to in writing, software
  distributed under the License is distributed on an "AS IS" BASIS,
  WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
  See the License for the specific language governing permissions and
  limitations under the License.
////

Compatibility Notes
-------------------

Sqoop uses JDBC to connect to databases and adheres to
published standards as much as possible. For databases which do not
support standards-compliant SQL, Sqoop uses alternate codepaths to
provide functionality. In general, Sqoop is believed to be compatible
with a large number of databases, but it is tested with only a few.

Nonetheless, several database-specific decisions were made in the
implementation of Sqoop, and some databases offer additional settings
which are extensions to the standard.

This section describes the databases tested with Sqoop, any
exceptions in Sqoop's handling of each database relative to the
norm, and any database-specific settings available in Sqoop.

Supported Databases
~~~~~~~~~~~~~~~~~~~

While JDBC is a compatibility layer that allows a program to access
many different databases through a common API, slight differences in
the SQL language spoken by each database may mean that Sqoop can't use
every database out of the box, or that some databases may be used in
an inefficient manner.

When you provide a connect string to Sqoop, it inspects the protocol scheme to
determine appropriate vendor-specific logic to use. If Sqoop knows about
a given database, it will work automatically. If not, you may need to
specify the driver class to load via +\--driver+. This will use a generic
code path which will use standard SQL to access the database. Sqoop provides
some databases with faster, non-JDBC-based access mechanisms. These can be
enabled by specfying the +\--direct+ parameter.

Sqoop includes vendor-specific support for the following databases:

[grid="all"]
`-----------`--------`--------------------`---------------------
Database    version  +\--direct+ support? connect string matches
----------------------------------------------------------------
HSQLDB      1.8.0+   No                   +jdbc:hsqldb:*//+
MySQL       5.0+     Yes                  +jdbc:mysql://+
Oracle      10.2.0+  No                   +jdbc:oracle:*//+
PostgreSQL  8.3+     Yes (import only)    +jdbc:postgresql://+
----------------------------------------------------------------

Sqoop may work with older versions of the databases listed, but we have
only tested it with the versions specified above.

Even if Sqoop supports a database internally, you may still need to
install the database vendor's JDBC driver in your +$SQOOP_HOME/lib+
path on your client. Sqoop can load classes from any jars in
+$SQOOP_HOME/lib+ on the client and will use them as part of any
MapReduce jobs it runs; unlike older versions, you no longer need to
install JDBC jars in the Hadoop library path on your servers.

MySQL
~~~~~

JDBC Driver: http://www.mysql.com/downloads/connector/j/[MySQL
Connector/J]

MySQL v5.0 and above offers very thorough coverage by Sqoop. Sqoop
has been tested with +mysql-connector-java-5.1.13-bin.jar+.

zeroDateTimeBehavior
^^^^^^^^^^^^^^^^^^^^

MySQL allows values of +\'0000-00-00\'+ for +DATE+ columns, which is a
non-standard extension to SQL. When communicated via JDBC, these
values are handled in one of three different ways:

- Convert to +NULL+.
- Throw an exception in the client.
- Round to the nearest legal date (+\'0001-01-01\'+).

You specify the behavior by using the +zeroDateTimeBehavior+
property of the connect string. If a +zeroDateTimeBehavior+ property
is not specified, Sqoop uses the +convertToNull+ behavior.

You can override this behavior. For example:

----
$ sqoop import --table foo \
    --connect jdbc:mysql://db.example.com/someDb?zeroDateTimeBehavior=round
----

+UNSIGNED+ columns
^^^^^^^^^^^^^^^^^^

Columns with type +UNSIGNED+ in MySQL can hold values between 0 and
2^32 (+4294967295+), but the database will report the data type to Sqoop
as +INTEGER+, which will can hold values between +-2147483648+ and
+\+2147483647+. Sqoop cannot currently import +UNSIGNED+ values above
+2147483647+.

+BLOB+ and +CLOB+ columns
^^^^^^^^^^^^^^^^^^^^^^^^^

Sqoop's direct mode does not support imports of +BLOB+, +CLOB+, or
+LONGVARBINARY+ columns. Use JDBC-based imports for these
columns; do not supply the +\--direct+ argument to the import tool.

Importing views in direct mode
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

Sqoop is currently not supporting import from view in direct mode. Use
JDBC based (non direct) mode in case that you need to import view (simply
omit +--direct+ parameter).

Direct-mode Transactions
^^^^^^^^^^^^^^^^^^^^^^^^

For performance, each writer will commit the current transaction
approximately every 32 MB of exported data. You can control this
by specifying the following argument _before_ any tool-specific arguments: +-D
sqoop.mysql.export.checkpoint.bytes=size+, where _size_ is a value in
bytes. Set _size_ to 0 to disable intermediate checkpoints,
but individual files being exported will continue to be committed
independently of one another.

Sometimes you need to export large data with Sqoop to a live MySQL cluster that
is under a high load serving random queries from the users of your application.
While data consistency issues during the export can be easily solved with a
staging table, there is still a problem with the performance impact caused by
the heavy export.

First off, the resources of MySQL dedicated to the import process can affect
the performance of the live product, both on the master and on the slaves.
Second, even if the servers can handle the import with no significant
performance impact (mysqlimport should be relatively "cheap"), importing big
tables can cause serious replication lag in the cluster risking data
inconsistency.

With +-D sqoop.mysql.export.sleep.ms=time+, where _time_ is a value in
milliseconds, you can let the server relax between checkpoints and the replicas
catch up by pausing the export process after transferring the number of bytes
specified in +sqoop.mysql.export.checkpoint.bytes+. Experiment with different
settings of these two parameters to archieve an export pace that doesn't
endanger the stability of your MySQL cluster.

IMPORTANT: Note that any arguments to Sqoop that are of the form +-D
parameter=value+ are Hadoop _generic arguments_ and must appear before
any tool-specific arguments (for example, +\--connect+, +\--table+, etc).
Don't forget that these parameters are only supported with the +\--direct+
flag set.

PostgreSQL
~~~~~~~~~~

Sqoop supports JDBC-based connector for PostgreSQL: http://jdbc.postgresql.org/

The connector has been tested using JDBC driver version "9.1-903 JDBC 4" with
PostgreSQL server 9.1.

Importing views in direct mode
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

Sqoop is currently not supporting import from view in direct mode. Use
JDBC based (non direct) mode in case that you need to import view (simply
omit +--direct+ parameter).

Oracle
~~~~~~

JDBC Driver:
http://www.oracle.com/technology/software/tech/java/sqlj_jdbc/htdocs/jdbc_112010.html[Oracle
JDBC Thin Driver] - Sqoop is compatible with +ojdbc6.jar+.

Sqoop has been tested with Oracle 10.2.0 Express Edition. Oracle is
notable in its different approach to SQL from the ANSI standard, and
its non-standard JDBC driver. Therefore, several features work
differently.

Dates and Times
^^^^^^^^^^^^^^^
{'fraction_non_alphanumeric': 0.11096947333109694, 'fraction_numerical': 0.014022140221402213, 'mean_word_length': 3.677125823658613, 'pattern_counts': {'":': 0, '<': 3, '<?xml version=': 0, '>': 3, 'https://': 0, 'lorem ipsum': 0, 'www.': 6, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 2, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '3928858', 'n_tokens_mistral': 4116, 'n_tokens_neox': 3836, 'n_words': 1977}