Configuration

The connection URL

The client accepts the full sc:// connection-string grammar:

sc://host:port[/[;key=value][;key=value]...]

Reserved keys with first-class handling:

Key	Meaning
use_ssl	true swaps the channel to grpc.credentials.createSsl(). The presence of token implies use_ssl=true.
token	Bearer token. Attached as authorization: Bearer <token> on every RPC. Token-over-insecure (use_ssl=false with token) is rejected.
user_id	Sent as userContext.userId on every RPC.
user_agent	Prefix to the canonical user-agent string <your prefix> spark-connect-js/<ver> (node <ver>; <platform>).
session_id	Reuse an existing server-side session by UUID. Validated at parse time.
grpc_max_message_size	Override the gRPC max-message-size in bytes for both inbound and outbound.

Any unreserved ;key=value pair is attached as gRPC metadata on every outgoing RPC.

Examples

connect("sc://localhost:15002");
connect("sc://spark.internal:443/;use_ssl=true");
connect("sc://spark.internal:443/;token=abc;use_ssl=true");
connect("sc://spark.internal:443/;use_ssl=true;tenant=acme;x-deploy=blue");
connect("sc://localhost:15002/;session_id=550e8400-e29b-41d4-a716-446655440000");

To skip the URL string, use the builder directly:

import { SparkSession } from "@spark-connect-js/node";

const spark = SparkSession.builder()
  .remote("sc://spark.internal:15002")
  .getOrCreate();

Builder methods

SparkSession.builder()
  .remote("sc://localhost:15002")   // required, connection URL
  .sessionId("550e8400-e29b-41d4-a716-446655440000") // optional, reuse a server-side session
  .transport(customTransport)       // optional, plug in a custom transport
  .arrowDecoder(customDecoder)      // optional, plug in a custom Arrow decoder
  .arrowEncoder(customEncoder)      // optional, needed for createDataFrame with plain rows
  .getOrCreate();                   // return the session

getOrCreate() constructs the session without opening the channel. The channel opens lazily on the first RPC, gated by a handshake deadline: if the channel isn’t ready within handshakeTimeoutMs (default 10 seconds, 0 disables), the RPC rejects with errorClass: "CONNECTION_TIMEOUT" instead of hanging on an unreachable or misconfigured endpoint.

For low-level transport tuning (custom ChannelCredentials, retry policy override, max-message-size override, handshakeTimeoutMs) construct a GrpcTransport directly and inject it via .transport(...). The retry policy also carries maxConsecutiveNoProgressReattaches (default 3), the ceiling on stream reattaches that deliver no data. The GrpcTransportOptions shape is documented in the API reference.

Data-source options

Reader and writer options pass through to Spark unchanged. The full list is in the Spark SQL data sources documentation.

CSV

Option	Default	Description
header	false	Treat the first line as a header.
sep	,	Field separator.
quote	"	Quote character.
escape	\\	Escape character inside quotes.
inferSchema	false	Infer types; requires a second pass.
nullValue	""	String that represents null.
dateFormat		Date parse pattern.
timestampFormat		Timestamp parse pattern.
mode	PERMISSIVE	PERMISSIVE, DROPMALFORMED, FAILFAST.
multiLine	false	Allow newlines inside quoted fields.

JSON

Option	Default	Description
multiLine	false	One JSON object may span multiple lines per file.
allowComments	false	Permit // comments.
allowSingleQuotes	true	Permit '...' strings.
mode	PERMISSIVE	PERMISSIVE, DROPMALFORMED, FAILFAST.
primitivesAsString	false	Keep every primitive value as a string.

Parquet and ORC

Option	Default	Description
compression	snappy	none, snappy, gzip, lz4, zstd.
mergeSchema	false	Merge schemas across files. Expensive for wide tables.

All formats

Option	Description
pathGlobFilter	Restrict file discovery with a glob pattern.
recursiveFileLookup	Include files in nested directories.
modifiedBefore / modifiedAfter	Filter by file modification time.

Write modes

Mode	Description
append	Add to existing data.
overwrite	Replace existing data.
ignore	No-op if the target exists.
error / errorifexists	Fail if the target exists (the default).

Runtime configuration

session.conf mirrors PySpark’s spark.conf, routed through Spark Connect’s Config RPC. Every method is async because each call is a server roundtrip.

await spark.conf.set("spark.sql.shuffle.partitions", "50");
const v = await spark.conf.get("spark.sql.shuffle.partitions"); // "50"
const all = await spark.conf.getAll();                          // Record<string, string>
const allShuffle = await spark.conf.getAll("spark.sql.shuffle.");
const ok = await spark.conf.isModifiable("spark.sql.shuffle.partitions"); // true
await spark.conf.unset("spark.sql.shuffle.partitions");

get returns string | undefined; the server returns undefined for unset keys.

Server-side defaults still live in spark-defaults.conf or --conf flags on start-connect-server.sh. session.conf is for per-session overrides made from client code.