SparkSession

Defined in: spark-session.ts:170

The client handle for a Spark Connect session.

Holds the transport, session identifier, and runtime-adapter hooks (for example the Arrow decoder). All DataFrame operations are scheduled against a SparkSession; most applications create one session at startup and reuse it.

Construct a session with the runtime-specific builder, for example SparkSessionBuilder from @spark-connect-js/node.

Example

import { SparkSessionBuilder } from "@spark-connect-js/node";

const spark = await SparkSessionBuilder
  .remote("sc://localhost:15002")
  .build();

const df = await spark.sql("SELECT 1 AS n");
console.log(await df.collect());

See

Spark source: SparkSession.scala

Properties

catalog

readonly catalog: Catalog;

Defined in: spark-session.ts:204

Access the session catalog for inspecting databases, tables, and columns.

---

conf

readonly conf: RuntimeConfig;

Defined in: spark-session.ts:210

Read and write Spark configuration entries on the connected server.

---

sessionId

readonly sessionId: string;

Defined in: spark-session.ts:171

---

udf

readonly udf: UDFRegistration;

Defined in: spark-session.ts:207

Register Java UDFs and UDAFs as SQL functions.

Accessors

read

Get Signature

get read(): DataFrameReader;

Defined in: spark-session.ts:213

Returns a DataFrameReader for building Read plans.

Returns

DataFrameReader

---

readStream

Get Signature

get readStream(): DataStreamReader;

Defined in: spark-session.ts:222

Returns a DataStreamReader for building streaming Read plans. The resulting DataFrame carries isStreaming: true and can only be consumed via df.writeStream.

Returns

DataStreamReader

---

streams

Get Signature

get streams(): StreamingQueryManager;

Defined in: spark-session.ts:227

Manage the streaming queries running on this session.

Returns

StreamingQueryManager

Methods

addTag()

addTag(tag): void;

Defined in: spark-session.ts:423

Tag every subsequent operation on this session with tag. Tags are carried on ExecutePlanRequest.tags and let you cancel a group of operations with interruptTag.

Parameters

Parameter	Type
tag	string

Returns

void

Throws

InvalidInputError if the tag contains , or is empty.

---

clearTags()

clearTags(): void;

Defined in: spark-session.ts:439

Drop all active tags.

Returns

void

---

createDataFrame()

Call Signature

createDataFrame(rows, schema?): DataFrame;

Defined in: spark-session.ts:298

Create a DataFrame from a plain array of row objects. The runtime adapter encodes the rows to Arrow IPC internally, so callers do not need to build the Arrow bytes themselves.

Type inference walks the first non-null value per column:

• string becomes Utf8 (never Dictionary<Int32, Utf8>, which Spark LocalRelation misreads as bare integer indices)
• number becomes Int32 if every value is an integer, otherwise Float64
• boolean becomes Bool
• bigint becomes Int64
• Date becomes Timestamp[ms]
• null and undefined in a column with a typed sibling are preserved as null

For richer types (Decimal, Struct, Array, Map, Binary), build the Arrow IPC bytes yourself and pass a Uint8Array to the other overload.

Parameters

Parameter	Type
rows	Row[]
schema?	string

Returns

DataFrame

Call Signature

createDataFrame(data, schema?): DataFrame;

Defined in: spark-session.ts:307

Create a DataFrame from pre-built Arrow IPC streaming bytes.

Parameters

Parameter	Type	Description
data	Uint8Array	Arrow IPC streaming format bytes. File-format input (magic prefix ARROW1\0\0) is rejected.
schema?	string	Optional DDL-formatted schema string (e.g. "id INT, name STRING").

Returns

DataFrame

Throws

InvalidInputError on empty input or file-format bytes.

---

getTags()

getTags(): string[];

Defined in: spark-session.ts:434

Return a snapshot of the currently active tags.

Returns

string[]

---

interruptAll()

interruptAll(): Promise<string[]>;

Defined in: spark-session.ts:447

Interrupt every running operation in this session.

Returns

Promise<string[]>

---

interruptOperation()

interruptOperation(operationId): Promise<string[]>;

Defined in: spark-session.ts:458

Interrupt a single running operation by its operation ID.

Parameters

Parameter	Type
operationId	string

Returns

Promise<string[]>

---

interruptTag()

interruptTag(tag): Promise<string[]>;

Defined in: spark-session.ts:452

Interrupt every running operation tagged with tag.

Parameters

Parameter	Type
tag	string

Returns

Promise<string[]>

---

range()

range(
   startOrEnd,
   end?,
   step?,
   numPartitions?): DataFrame;

Defined in: spark-session.ts:269

Create a DataFrame with a single id column containing a sequence of integers from start (inclusive) to end (exclusive), incrementing by step.

Mirrors PySpark’s spark.range(start, end, step, numPartitions).

Parameters

Parameter	Type	Default value
startOrEnd	number	undefined
end?	number	undefined
step?	number	1
numPartitions?	number	undefined

Returns

DataFrame

Example

spark.range(10)            // 0, 1, 2, ..., 9
  spark.range(1, 10)         // 1, 2, 3, ..., 9
  spark.range(0, 10, 2)      // 0, 2, 4, 6, 8

---

removeTag()

removeTag(tag): void;

Defined in: spark-session.ts:429

Remove a previously added tag. No-op if the tag wasn’t set.

Parameters

Parameter	Type
tag	string

Returns

void

---

sql()

sql(query): DataFrame;

Defined in: spark-session.ts:243

Execute a SQL query.

Parameters

Parameter	Type
query	string

Returns

DataFrame

---

stop()

stop(): Promise<void>;

Defined in: spark-session.ts:491

Stop the session: releases server-side state and closes the transport.

Returns

Promise<void>

---

table()

table(tableName): DataFrame;

Defined in: spark-session.ts:254

Read a catalog table or temp view as a DataFrame. Shorthand for spark.read.table(name).

Parameters

Parameter	Type
tableName	string

Returns

DataFrame

---

version()

version(): Promise<string>;

Defined in: spark-session.ts:483

Return the Apache Spark version reported by the connected server.

One AnalyzePlan RPC. Result is not cached; call once and store if you need it repeatedly.

Mirrors pyspark.sql.SparkSession.version.

Returns

Promise<string>

---

builder()

static builder(): SparkSessionBuilder;

Defined in: spark-session.ts:197

Returns

SparkSessionBuilder