<commit_before>// Licensed to the Apache Software Foundation (ASF) under one
// or more contributor license agreements.  See the NOTICE file
// distributed with this work for additional information
// regarding copyright ownership.  The ASF licenses this file
// to you under the Apache License, Version 2.0 (the
// "License"); you may not use this file except in compliance
// with the License.  You may obtain a copy of the License at
//
//   http://www.apache.org/licenses/LICENSE-2.0
//
// Unless required by applicable law or agreed to in writing,
// software distributed under the License is distributed on an
// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
// KIND, either express or implied.  See the License for the
// specific language governing permissions and limitations
// under the License.

[[developing]]
= Developing Applications With Apache Kudu

:author: Kudu Team
:imagesdir: ./images
:icons: font
:toc: left
:toclevels: 3
:doctype: book
:backend: html5
:sectlinks:
:experimental:

Kudu provides C++, Java and Python client APIs, as well as reference examples to illustrate
their use.

WARNING: Use of server-side or private interfaces is not supported, and interfaces
which are not part of public APIs have no stability guarantees.

== Viewing the API Documentation
include::installation.adoc[tags=view_api]

== Working Examples

Several example applications are provided in the
link:https://github.com/apache/kudu/tree/master/examples[examples directory]
of the Apache Kudu git repository. Each example includes a `README` that shows
how to compile and run it. The following list includes some of the
examples that are available today. Check the repository itself in case this list goes
out of date.

`cpp/example.cc`::
  A simple C++ application which connects to a Kudu instance, creates a table, writes data to it, then drops the table.
`java/java-example`::
  A simple Java application which connects to a Kudu instance, creates a table, writes data to it, then drops the table.
`java/collectl`::
  A small Java application which listens on a TCP socket for time series data corresponding to the Collectl wire protocol.
  The commonly-available collectl tool can be used to send example data to the server.
`java/insert-loadgen`::
  A Java application that generates random insert load.
`python/dstat-kudu`::
  An example program that shows how to use the Kudu Python API to load data into a new / existing Kudu table
  generated by an external program, `dstat` in this case.
`python/graphite-kudu`::
  An example plugin for using graphite-web with Kudu as a backend.

These examples should serve as helpful starting points for your own Kudu applications and integrations.

== Maven Artifacts
The following Maven `<dependency>` element is valid for the Apache Kudu public release
(since 1.0.0):

[source,xml]
----
<dependency>
  <groupId>org.apache.kudu</groupId>
  <artifactId>kudu-client</artifactId>
  <version>1.10.0</version>
</dependency>
----

Convenience binary artifacts for the Java client and various Java integrations (e.g. Spark, Flume)
are also available via the link:http://repository.apache.org[ASF Maven repository] and
link:https://mvnrepository.com/artifact/org.apache.kudu[Maven Central repository].

== Example Impala Commands With Kudu

See link:kudu_impala_integration.html[Using Impala With Kudu] for guidance on installing
and using Impala with Kudu, including several `impala-shell` examples.

== Kudu Integration with Spark

Kudu integrates with Spark through the Data Source API as of version 1.0.0.
Include the kudu-spark dependency using the --packages option:

Use the kudu-spark_2.10 artifact if using Spark with Scala 2.10. Note that Spark 1 is no
longer supported in Kudu starting from version 1.6.0. So in order to use Spark 1 integrated
with Kudu, version 1.5.0 is the latest to go to.
[source]
----
spark-shell --packages org.apache.kudu:kudu-spark_2.10:1.5.0
----

Use kudu-spark2_2.11 artifact if using Spark 2 with Scala 2.11.

NOTE: kudu-spark versions 1.8.0 and below have slightly different syntax.
See the documentation of your version for a valid example. Versioned documentation can be found
on the link:http://kudu.apache.org/releases/[releases page].

[source]
----
spark-shell --packages org.apache.kudu:kudu-spark2_2.11:1.10.0
----

Below is a minimal Spark SQL "select" example for a Kudu table created with
Impala in the "default" database. We first import the kudu spark package,
then create a DataFrame, and then create a view from the DataFrame. After those
steps, the table is accessible from Spark SQL.

[source,scala]
----
import org.apache.kudu.spark.kudu._

// Create a DataFrame that points to the Kudu table we want to query.
val df = spark.read.options(Map("kudu.master" -> "master1.foo.com,master2.foo.com,master3.foo.com",
                                "kudu.table" -> "default.my_table")).kudu
// Create a view from the DataFrame to make it accessible from Spark SQL.
df.createOrReplaceTempView("my_table")
// Now we can run Spark SQL queries against our view of the Kudu table.
spark.sql("select * from my_table").show()
----

Below is a more sophisticated example that includes both reads and writes:

[source,scala]
----
import org.apache.kudu.client._
import collection.JavaConverters._

// Read a table from Kudu
val df = spark.read
  .options(Map("kudu.master" -> "kudu.master:7051", "kudu.table" -> "kudu_table"))
  .format("kudu").load

// Query using the Spark API...
df.select("id").filter("id >= 5").show()

// ...or register a temporary table and use SQL
df.createOrReplaceTempView("kudu_table")
val filteredDF = spark.sql("select id from kudu_table where id >= 5").show()

// Use KuduContext to create, delete, or write to Kudu tables
val kuduContext = new KuduContext("kudu.master:7051", spark.sparkContext)

// Create a new Kudu table from a DataFrame schema
// NB: No rows from the DataFrame are inserted into the table
kuduContext.createTable(
    "test_table", df.schema, Seq("key"),
    new CreateTableOptions()
        .setNumReplicas(1)
        .addHashPartitions(List("key").asJava, 3))

// Insert data
kuduContext.insertRows(df, "test_table")

// Delete data
kuduContext.deleteRows(filteredDF, "test_table")

// Upsert data
kuduContext.upsertRows(df, "test_table")

// Update data
val alteredDF = df.select("id", $"count" + 1)
kuduContext.updateRows(filteredRows, "test_table")

// Data can also be inserted into the Kudu table using the data source, though the methods on
// KuduContext are preferred
// NB: The default is to upsert rows; to perform standard inserts instead, set operation = insert
// in the options map
// NB: Only mode Append is supported
df.write
  .options(Map("kudu.master"-> "kudu.master:7051", "kudu.table"-> "test_table"))
  .mode("append")
  .format("kudu").save

// Check for the existence of a Kudu table
kuduContext.tableExists("another_table")

// Delete a Kudu table
kuduContext.deleteTable("unwanted_table")
----

=== Upsert option in Kudu Spark
The upsert operation in kudu-spark supports an extra write option of `ignoreNull`. If set to true,
it will avoid setting existing column values in Kudu table to Null if the corresponding DataFrame
column values are Null. If unspecified, `ignoreNull` is false by default.
[source,scala]
----
val dataFrame = spark.read
  .options(Map("kudu.master" -> "kudu.master:7051", "kudu.table" -> simpleTableName))
  .format("kudu").load
dataFrame.createOrReplaceTempView(simpleTableName)
dataFrame.show()
// Below is the original data in the table 'simpleTableName'
+---+---+
|key|val|
+---+---+
|  0|foo|
+---+---+

// Upsert a row with existing key 0 and val Null with ignoreNull set to true
val nullDF = spark.createDataFrame(Seq((0, null.asInstanceOf[String]))).toDF("key", "val")
val wo = new KuduWriteOptions
wo.ignoreNull = true
kuduContext.upsertRows(nullDF, simpleTableName, wo)
dataFrame.show()
// The val field stays unchanged
+---+---+
|key|val|<commit_msg>[docs] fix spark integration examples<commit_after>// Licensed to the Apache Software Foundation (ASF) under one
// or more contributor license agreements.  See the NOTICE file
// distributed with this work for additional information
// regarding copyright ownership.  The ASF licenses this file
// to you under the Apache License, Version 2.0 (the
// "License"); you may not use this file except in compliance
// with the License.  You may obtain a copy of the License at
//
//   http://www.apache.org/licenses/LICENSE-2.0
//
// Unless required by applicable law or agreed to in writing,
// software distributed under the License is distributed on an
// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
// KIND, either express or implied.  See the License for the
// specific language governing permissions and limitations
// under the License.

[[developing]]
= Developing Applications With Apache Kudu

:author: <NAME>
:imagesdir: ./images
:icons: font
:toc: left
:toclevels: 3
:doctype: book
:backend: html5
:sectlinks:
:experimental:

Kudu provides C++, Java and Python client APIs, as well as reference examples to illustrate
their use.

WARNING: Use of server-side or private interfaces is not supported, and interfaces
which are not part of public APIs have no stability guarantees.

== Viewing the API Documentation
include::installation.adoc[tags=view_api]

== Working Examples

Several example applications are provided in the
link:https://github.com/apache/kudu/tree/master/examples[examples directory]
of the Apache Kudu git repository. Each example includes a `README` that shows
how to compile and run it. The following list includes some of the
examples that are available today. Check the repository itself in case this list goes
out of date.

`cpp/example.cc`::
  A simple C++ application which connects to a Kudu instance, creates a table, writes data to it, then drops the table.
`java/java-example`::
  A simple Java application which connects to a Kudu instance, creates a table, writes data to it, then drops the table.
`java/collectl`::
  A small Java application which listens on a TCP socket for time series data corresponding to the Collectl wire protocol.
  The commonly-available collectl tool can be used to send example data to the server.
`java/insert-loadgen`::
  A Java application that generates random insert load.
`python/dstat-kudu`::
  An example program that shows how to use the Kudu Python API to load data into a new / existing Kudu table
  generated by an external program, `dstat` in this case.
`python/graphite-kudu`::
  An example plugin for using graphite-web with Kudu as a backend.

These examples should serve as helpful starting points for your own Kudu applications and integrations.

== Maven Artifacts
The following Maven `<dependency>` element is valid for the Apache Kudu public release
(since 1.0.0):

[source,xml]
----
<dependency>
  <groupId>org.apache.kudu</groupId>
  <artifactId>kudu-client</artifactId>
  <version>1.10.0</version>
</dependency>
----

Convenience binary artifacts for the Java client and various Java integrations (e.g. Spark, Flume)
are also available via the link:http://repository.apache.org[ASF Maven repository] and
link:https://mvnrepository.com/artifact/org.apache.kudu[Maven Central repository].

== Example Impala Commands With Kudu

See link:kudu_impala_integration.html[Using Impala With Kudu] for guidance on installing
and using Impala with Kudu, including several `impala-shell` examples.

== Kudu Integration with Spark

Kudu integrates with Spark through the Data Source API as of version 1.0.0.
Include the kudu-spark dependency using the --packages option:

Use the kudu-spark_2.10 artifact if using Spark with Scala 2.10. Note that Spark 1 is no
longer supported in Kudu starting from version 1.6.0. So in order to use Spark 1 integrated
with Kudu, version 1.5.0 is the latest to go to.
[source]
----
spark-shell --packages org.apache.kudu:kudu-spark_2.10:1.5.0
----

Use kudu-spark2_2.11 artifact if using Spark 2 with Scala 2.11.

NOTE: kudu-spark versions 1.8.0 and below have slightly different syntax.
See the documentation of your version for a valid example. Versioned documentation can be found
on the link:http://kudu.apache.org/releases/[releases page].

[source]
----
spark-shell --packages org.apache.kudu:kudu-spark2_2.11:1.10.0
----

Below is a minimal Spark SQL "select" example. We first import the kudu spark package,
then create a DataFrame, and then create a view from the DataFrame. After those
steps, the table is accessible from Spark SQL.

NOTE: There is also a Spark link:https://github.com/apache/kudu/tree/master/examples/quickstart/spark[quickstart]
 guide and another link:https://github.com/apache/kudu/tree/master/examples/scala/spark-example[example]
 available.

NOTE: You can use the Kudu CLI tool to create table and generate data by
 `kudu perf loadgen kudu.master:7051 -keep_auto_table` for the following two examples.

[source,scala]
----
import org.apache.kudu.spark.kudu._

// Create a DataFrame that points to the Kudu table we want to query.
val df = spark.read.options(Map("kudu.master" -> "kudu.master:7051",
                                "kudu.table" -> "default.my_table")).format("kudu").load
// Create a view from the DataFrame to make it accessible from Spark SQL.
df.createOrReplaceTempView("my_table")
// Now we can run Spark SQL queries against our view of the Kudu table.
spark.sql("select * from my_table").show()
----

Below is a more sophisticated example that includes both reads and writes:

[source,scala]
----
import org.apache.kudu.client._
import org.apache.kudu.spark.kudu.KuduContext
import collection.JavaConverters._

// Read a table from Kudu
val df = spark.read
  .options(Map("kudu.master" -> "kudu.master:7051", "kudu.table" -> "kudu_table"))
  .format("kudu").load

// Query using the Spark API...
df.select("key").filter("key >= 5").show()

// ...or register a temporary table and use SQL
df.createOrReplaceTempView("kudu_table")
val filteredDF = spark.sql("select key from kudu_table where key >= 5").show()

// Use KuduContext to create, delete, or write to Kudu tables
val kuduContext = new KuduContext("kudu.master:7051", spark.sparkContext)

// Create a new Kudu table from a DataFrame schema
// NB: No rows from the DataFrame are inserted into the table
kuduContext.createTable(
    "test_table", df.schema, Seq("key"),
    new CreateTableOptions()
        .setNumReplicas(1)
        .addHashPartitions(List("key").asJava, 3))

// Check for the existence of a Kudu table
kuduContext.tableExists("test_table")

// Insert data
kuduContext.insertRows(df, "test_table")

// Delete data
kuduContext.deleteRows(df, "test_table")

// Upsert data
kuduContext.upsertRows(df, "test_table")

// Update data
val updateDF = df.select($"key", ($"int_val" + 1).as("int_val"))
kuduContext.updateRows(updateDF, "test_table")

// Data can also be inserted into the Kudu table using the data source, though the methods on
// KuduContext are preferred
// NB: The default is to upsert rows; to perform standard inserts instead, set operation = insert
// in the options map
// NB: Only mode Append is supported
df.write
  .options(Map("kudu.master"-> "kudu.master:7051", "kudu.table"-> "test_table"))
  .mode("append")
  .format("kudu").save

// Delete a Kudu table
kuduContext.deleteTable("test_table")
----

=== Upsert option in Kudu Spark
The upsert operation in kudu-spark supports an extra write option of `ignoreNull`. If set to true,
it will avoid setting existing column values in Kudu table to Null if the corresponding DataFrame
column values are Null. If unspecified, `ignoreNull` is false by default.
[source,scala]
----
val dataFrame = spark.read
  .options(Map("kudu.master" -> "kudu.master:7051", "kudu.table" -> simpleTableName))
  .format("kudu").load
dataFrame.createOrReplaceTempView(simpleTableName)
dataFrame.show()
// Below is the original data in the table 'simpleTableName'
+---+---+
|key|val|
+---+---+
|  0|foo|
+---+---+

// Upsert a row with existing key 0 and val Null with ignoreNull set to true
val nullDF = spark.createDataFrame(Seq((0, null.asInstanceOf[String]))).toDF("key", "val")
val wo = new KuduWriteOptions
wo.ignoreNull = true
kuduContext.upsertRows(nullDF, simpleTableName, wo)
dataFrame.show()
// The val field stays unchanged
+---+---+
|key|val|
{'fraction_non_alphanumeric': 0.10246981675553105, 'fraction_numerical': 0.010357296071581786, 'mean_word_length': 4.1346758967904345, 'pattern_counts': {'":': 0, '<': 22, '<?xml version=': 0, '>': 42, 'https://': 6, 'lorem ipsum': 0, 'www.': 2, 'xml': 2}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 3, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '5731778', 'n_tokens_mistral': 5119, 'n_tokens_neox': 4870, 'n_words': 1950}