Execute a SQL command and return a string representing the results.
__init__(spark_session: Optional[SparkSession] = None, catalog: Optional[str] = None, schema: Optional[str] = None, ignore_tables: Optional[List[str]] = None, include_tables: Optional[List[str]] = None, sample_rows_in_table_info: int = 3)[source]¶
Initialize a SparkSQL object.
Parameters
spark_session – A SparkSession object.
If not provided, one will be created.
catalog – The catalog to use.
If not provided, the default catalog will be used.
schema – The schema to use.
If not provided, the default schema will be used.
ignore_tables – A list of tables to ignore.
If not provided, all tables will be used.
include_tables – A list of tables to include.
If not provided, all tables will be used.
sample_rows_in_table_info – The number of rows to include in the table info.
Defaults to 3.
classmethod from_uri(database_uri: str, engine_args: Optional[dict] = None, **kwargs: Any) → SparkSQL[source]¶
Creating a remote Spark Session via Spark connect.
For example: SparkSQL.from_uri(“sc://localhost:15002”)
get_table_info(table_names: Optional[List[str]] = None) → str[source]¶
get_table_info_no_throw(table_names: Optional[List[str]] = None) → str[source]¶
Get information about specified tables.
Follows best practices as specified in: Rajkumar et al, 2022
(https://arxiv.org/abs/2204.00498)
If sample_rows_in_table_info, the specified number of sample rows will be
appended to each table description. This can increase performance as
demonstrated in the paper.