Catalog

Access to Spark’s catalog for inspecting and managing databases, tables, functions, views, and cached tables.

Example

const dbs = await spark.catalog.listDatabases();
const tables = await spark.catalog.listTables("analytics");
await spark.catalog.cacheTable("analytics.events");

See

Spark source: Catalog.scala

Methods

cacheTable()

cacheTable(tableName, storageLevel?): Promise<void>;

Defined in: catalog.ts:131

Cache the specified table in-memory with an optional storage level.

Parameters

Parameter	Type
`tableName`	`string`
`storageLevel?`	`StorageLevel`

Returns

Promise<void>

clearCache()

clearCache(): Promise<void>;

Defined in: catalog.ts:141

Remove all cached tables from the in-memory cache.

Returns

Promise<void>

createExternalTable()

createExternalTable(tableName, options?): DataFrame;

Defined in: catalog.ts:198

Create an external table based on the dataset in a data source.

Returns a DataFrame associated with the external table.

Parameters

Parameter	Type
`tableName`	`string`
`options?`	{ `options?`: `Record`<`string`, `string`>; `path?`: `string`; `schema?`: `StructType`; `source?`: `string`; }
`options.options?`	`Record`<`string`, `string`>
`options.path?`	`string`
`options.schema?`	`StructType`
`options.source?`	`string`

Returns

DataFrame

createTable()

createTable(tableName, options?): DataFrame;

Defined in: catalog.ts:168

Create a table based on the dataset in a data source.

When path is specified, an external table is created from the data at the given path. Otherwise a managed table is created.

Returns a DataFrame associated with the new table.

Parameters

Parameter	Type
`tableName`	`string`
`options?`	{ `description?`: `string`; `options?`: `Record`<`string`, `string`>; `path?`: `string`; `schema?`: `StructType`; `source?`: `string`; }
`options.description?`	`string`
`options.options?`	`Record`<`string`, `string`>
`options.path?`	`string`
`options.schema?`	`StructType`
`options.source?`	`string`

Returns

DataFrame

currentCatalog()

currentCatalog(): Promise<string>;

Defined in: catalog.ts:120

Get the current default catalog name.

Returns

Promise<string>

currentDatabase()

currentDatabase(): Promise<string>;

Defined in: catalog.ts:109

Get the current database name.

Returns

Promise<string>

databaseExists()

databaseExists(dbName): Promise<boolean>;

Defined in: catalog.ts:79

Check if a database exists.

Parameters

Parameter	Type
`dbName`	`string`

Returns

Promise<boolean>

dropGlobalTempView()

dropGlobalTempView(viewName): Promise<boolean>;

Defined in: catalog.ts:103

Drops the global temporary view. Returns true if the view existed.

Parameters

Parameter	Type
`viewName`	`string`

Returns

Promise<boolean>

dropTempView()

dropTempView(viewName): Promise<boolean>;

Defined in: catalog.ts:97

Drops the local temporary view. Returns true if the view existed.

Parameters

Parameter	Type
`viewName`	`string`

Returns

Promise<boolean>

functionExists()

functionExists(functionName, dbName?): Promise<boolean>;

Defined in: catalog.ts:85

Check if a function exists.

Parameters

Parameter	Type
`functionName`	`string`
`dbName?`	`string`

Returns

Promise<boolean>

getDatabase()

getDatabase(dbName): DataFrame;

Defined in: catalog.ts:58

Get the database with the specified name. Returns a single-row DataFrame.

Parameters

Parameter	Type
`dbName`	`string`

Returns

DataFrame

getFunction()

getFunction(functionName, dbName?): DataFrame;

Defined in: catalog.ts:68

Get the function with the specified name. Returns a single-row DataFrame.

Parameters

Parameter	Type
`functionName`	`string`
`dbName?`	`string`

Returns

DataFrame

getTable()

getTable(tableName, dbName?): DataFrame;

Defined in: catalog.ts:63

Get the table or view with the specified name. Returns a single-row DataFrame.

Parameters

Parameter	Type
`tableName`	`string`
`dbName?`	`string`

Returns

DataFrame

isCached()

isCached(tableName): Promise<boolean>;

Defined in: catalog.ts:91

Returns true if the table is currently cached in-memory.

Parameters

Parameter	Type
`tableName`	`string`

Returns

Promise<boolean>

listCatalogs()

listCatalogs(pattern?): DataFrame;

Defined in: catalog.ts:53

List all catalogs. Returns a DataFrame with catalog metadata.

Parameters

Parameter	Type
`pattern?`	`string`

Returns

DataFrame

listColumns()

listColumns(tableName, dbName?): DataFrame;

Defined in: catalog.ts:43

List all columns of a table. Returns a DataFrame with column metadata.

Parameters

Parameter	Type
`tableName`	`string`
`dbName?`	`string`

Returns

DataFrame

listDatabases()

listDatabases(pattern?): DataFrame;

Defined in: catalog.ts:33

List all databases. Returns a DataFrame with database metadata.

Parameters

Parameter	Type
`pattern?`	`string`

Returns

DataFrame

listFunctions()

listFunctions(dbName?, pattern?): DataFrame;

Defined in: catalog.ts:48

List all functions in a database. Returns a DataFrame with function metadata.

Parameters

Parameter	Type
`dbName?`	`string`
`pattern?`	`string`

Returns

DataFrame

listTables()

listTables(dbName?, pattern?): DataFrame;

Defined in: catalog.ts:38

List all tables in a database. Returns a DataFrame with table metadata.

Parameters

Parameter	Type
`dbName?`	`string`
`pattern?`	`string`

Returns

DataFrame

recoverPartitions()

recoverPartitions(tableName): Promise<void>;

Defined in: catalog.ts:156

Recover all partitions of the given table and update the catalog.

Parameters

Parameter	Type
`tableName`	`string`

Returns

Promise<void>

refreshByPath()

refreshByPath(path): Promise<void>;

Defined in: catalog.ts:151

Invalidate and refresh cached data for any DataFrame containing the given path.

Parameters

Parameter	Type
`path`	`string`

Returns

Promise<void>

refreshTable()

refreshTable(tableName): Promise<void>;

Defined in: catalog.ts:146

Invalidate and refresh all cached data and metadata for the given table.

Parameters

Parameter	Type
`tableName`	`string`

Returns

Promise<void>

setCurrentCatalog()

setCurrentCatalog(catalogName): Promise<void>;

Defined in: catalog.ts:126

Set the current default catalog.

Parameters

Parameter	Type
`catalogName`	`string`

Returns

Promise<void>

setCurrentDatabase()

setCurrentDatabase(dbName): Promise<void>;

Defined in: catalog.ts:115

Set the current database.

Parameters

Parameter	Type
`dbName`	`string`

Returns

Promise<void>

tableExists()

tableExists(tableName, dbName?): Promise<boolean>;

Defined in: catalog.ts:73

Check if a table exists.

Parameters

Parameter	Type
`tableName`	`string`
`dbName?`	`string`

Returns

Promise<boolean>

uncacheTable()

uncacheTable(tableName): Promise<void>;

Defined in: catalog.ts:136

Remove the specified table from the in-memory cache.

Parameters

Parameter	Type
`tableName`	`string`

Returns

Promise<void>