<commit_before>---
title: "SDK"
date: 2019-03-13T18:28:09-07:00
draft: false
weight: 3
---

# Connectors

They provide SQL integration with any database. Here is a list of the [existing connectors](https://github.com/cloudera/hue/tree/master/desktop/libs/notebook/src/notebook/connectors).

Connectors are pluggable and can new engines can be supported. Feel free to contact the community.

## SQL Autocomplete

Close to 100% of [Hive and Impala grammar](https://github.com/cloudera/hue/blob/master/desktop/core/src/desktop/static/desktop/js/autocomplete/jison) is supported which makes the autocomplete extremly powerful. Other languages defaults to a generic SQL grammar.

See [How to write your own SQL parser](/developer/parsers/). Integrating [Apache Calcite](https://calcite.apache.org/docs/reference.html), [ZetaSql](https://github.com/google/zetasql)... would make SQL users even happier with a lot more Databases!


## SQL Connectors

[SqlAlchemy](https://www.sqlalchemy.org) is the prefered way if the HiveServer2 API is not supported by the database. The implementation is in [`sql_alchemy.py`](https://github.com/cloudera/hue/blob/master/desktop/libs/notebook/src/notebook/connectors/sql_alchemy.py) and is depends on the repective SqlAlchemy dialects.

With the JDBC proxy, query editor with any JDBC compatible database. View the [JDBC connector](https://github.com/cloudera/hue/blob/master/desktop/libs/notebook/src/notebook/connectors/jdbc.py).

**Note** In the long term, SqlAlchemy is prefered as more "Python native".

### Solr SQL

[Solr connector](https://github.com/cloudera/hue/blob/master/desktop/libs/notebook/src/notebook/connectors/solr.py).

### Oozie Jobs

MapReduce, Pig, Java, Shell, Sqoop, DistCp [Oozie connector](https://github.com/cloudera/hue/blob/master/desktop/libs/notebook/src/notebook/connectors/oozie_batch.py).

### Spark / Livy

Based on the [Livy REST API](https://livy.incubator.apache.org/docs/latest/rest-api.html)

* [Notebook connector](https://github.com/cloudera/hue/blob/master/desktop/libs/notebook/src/notebook/connectors/spark_shell.py)
  * PySpark
  * Scala
  * Spark SQL
* [Batch connector](https://github.com/cloudera/hue/blob/master/desktop/libs/notebook/src/notebook/connectors/spark_batch.py)

## Jobs

The Job Browser is generic and can list any type of jobs, queries and provide bulk operations like kill, pause, delete... and access to logs and recommendations.

Here is its [API](https://github.com/cloudera/hue/tree/master/apps/jobbrowser/src/jobbrowser/apis).

## Files
Here is an example on how the File Browser can list HDFS, S3 files and [ADLS](https://issues.cloudera.org/browse/HUE-7248).<commit_msg>HUE-8888 [docs] Add link to fsmanager in File SDK section<commit_after>---
title: "SDK"
date: 2019-03-13T18:28:09-07:00
draft: false
weight: 3
---

# Connectors

They provide SQL integration with any database. Here is a list of the [existing connectors](https://github.com/cloudera/hue/tree/master/desktop/libs/notebook/src/notebook/connectors).

Connectors are pluggable and can new engines can be supported. Feel free to contact the community.

## SQL Autocomplete

Close to 100% of [Hive and Impala grammar](https://github.com/cloudera/hue/blob/master/desktop/core/src/desktop/static/desktop/js/autocomplete/jison) is supported which makes the autocomplete extremly powerful. Other languages defaults to a generic SQL grammar.

See [How to write your own SQL parser](/developer/parsers/). Integrating [Apache Calcite](https://calcite.apache.org/docs/reference.html), [ZetaSql](https://github.com/google/zetasql)... would make SQL users even happier with a lot more Databases!


## SQL Connectors

[SqlAlchemy](https://www.sqlalchemy.org) is the prefered way if the HiveServer2 API is not supported by the database. The implementation is in [`sql_alchemy.py`](https://github.com/cloudera/hue/blob/master/desktop/libs/notebook/src/notebook/connectors/sql_alchemy.py) and is depends on the repective SqlAlchemy dialects.

With the JDBC proxy, query editor with any JDBC compatible database. View the [JDBC connector](https://github.com/cloudera/hue/blob/master/desktop/libs/notebook/src/notebook/connectors/jdbc.py).

**Note** In the long term, SqlAlchemy is prefered as more "Python native".

### Solr SQL

[Solr connector](https://github.com/cloudera/hue/blob/master/desktop/libs/notebook/src/notebook/connectors/solr.py).

### Oozie Jobs

MapReduce, Pig, Java, Shell, Sqoop, DistCp [Oozie connector](https://github.com/cloudera/hue/blob/master/desktop/libs/notebook/src/notebook/connectors/oozie_batch.py).

### Spark / Livy

Based on the [Livy REST API](https://livy.incubator.apache.org/docs/latest/rest-api.html)

* [Notebook connector](https://github.com/cloudera/hue/blob/master/desktop/libs/notebook/src/notebook/connectors/spark_shell.py)
  * PySpark
  * Scala
  * Spark SQL
* [Batch connector](https://github.com/cloudera/hue/blob/master/desktop/libs/notebook/src/notebook/connectors/spark_batch.py)

## Jobs

The Job Browser is generic and can list any type of jobs, queries and provide bulk operations like kill, pause, delete... and access to logs and recommendations.

Here is its [API](https://github.com/cloudera/hue/tree/master/apps/jobbrowser/src/jobbrowser/apis).

## Files

Various storage systems like Hadoop HDFS, AWS S3 and Azure [ADLS](https://issues.cloudera.org/browse/HUE-7248) can be interacted with. The [`fsmanager.py`](https://github.com/cloudera/hue/blob/master/desktop/core/src/desktop/lib/fsmanager.py) is the main router to each API.
{'fraction_non_alphanumeric': 0.12450378924575965, 'fraction_numerical': 0.010826416456153013, 'mean_word_length': 4.690965092402464, 'pattern_counts': {'":': 0, '<': 3, '<?xml version=': 0, '>': 3, 'https://': 29, 'lorem ipsum': 0, 'www.': 2, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '2003823', 'n_tokens_mistral': 1947, 'n_tokens_neox': 1792, 'n_words': 468}