DataFrame

Defined in: data-frame.ts:100

A distributed collection of rows with a named schema, obtained from a SparkSession (for example via spark.read.parquet(path) or spark.sql(...)).

DataFrame is lazy. Transformation methods (select, filter, join, withColumn, etc.) return a new DataFrame that wraps an extended logical plan; no work is performed on the server until an action (collect, count, show, write.save, etc.) is called.

Example

Read, transform, collect

const df = await spark.read.parquet("s3://bucket/events");

const recent = df
  .filter(col("ts").gte(lit("2026-01-01")))
  .groupBy("country")
  .count();

const rows = await recent.collect();

See

Spark source: Dataset.scala

Type Parameters

Type Parameter	Default type
R extends Row	Row

Accessors

stat

Get Signature

get stat(): DataFrameStat;

Defined in: data-frame.ts:767

Access statistical functions (corr, cov, crosstab, etc.).

Returns

DataFrameStat

---

write

Get Signature

get write(): DataFrameWriter;

Defined in: data-frame.ts:774

Returns a DataFrameWriter for persisting the contents of this DataFrame.

Returns

DataFrameWriter

---

writeStream

Get Signature

get writeStream(): DataStreamWriter;

Defined in: data-frame.ts:794

Returns a DataStreamWriter for launching a streaming query against the data in this (streaming) DataFrame.

Only valid on streaming DataFrames (those whose source plan is built via spark.readStream). The server will reject the write if the input plan isn’t streaming.

Returns

DataStreamWriter

Methods

agg()

agg(...exprs): DataFrame;

Defined in: data-frame.ts:197

Aggregate over the whole DataFrame. Shorthand for groupBy().agg(...), matching PySpark and Scala.

Parameters

Parameter	Type
…exprs	Column[]

Returns

DataFrame

Example

df.agg(count("*").alias("rows"), avg("salary").alias("mean"))

---

alias()

alias(name): DataFrame<R>;

Defined in: data-frame.ts:525

Assign an alias to this DataFrame, useful for self-joins.

Parameters

Parameter	Type
name	string

Returns

DataFrame<R>

---

as()

as<NewR>(): DataFrame<NewR>;

Defined in: data-frame.ts:132

Narrow this DataFrame’s row type for typed collect(), first(), etc. The cast is compile-time only. Matches Scala’s Dataset.as[T] precedent.

Shape-preserving transformations (filter, where, limit, sort, orderBy, etc.) keep the narrowed type. Shape-changing transformations (select, withColumn, drop, groupBy().agg(), etc.) reset to Row. Re-narrow with another .as<NewR>() after them if needed.

Type Parameters

Type Parameter
NewR extends Row

Returns

DataFrame<NewR>

Example

const rows = await df.as<{ count: bigint; mean: number }>().collect();
  // rows is typed Array<{ count: bigint; mean: number }>

---

cache()

cache(): Promise<DataFrame<R>>;

Defined in: data-frame.ts:804

Persist this DataFrame with the default storage level (MEMORY_AND_DISK). Returns this DataFrame for method chaining.

Returns

Promise<DataFrame<R>>

---

coalesce()

coalesce(numPartitions): DataFrame<R>;

Defined in: data-frame.ts:645

Return a new DataFrame that is reduced to the given number of partitions. Unlike repartition(), coalesce avoids a full shuffle and tries to combine existing partitions.

Parameters

Parameter	Type	Description
numPartitions	number	Target number of partitions

Returns

DataFrame<R>

---

col()

col(name): Column;

Defined in: data-frame.ts:146

Return a Column bound to this DataFrame. Use to disambiguate columns in self-joins and same-schema joins where the unqualified column name would be ambiguous on the server side.

Parameters

Parameter	Type
name	string

Returns

Column

Example

const a = df.alias("a");
  const b = df.alias("b");
  a.join(b, a.col("id").eq(b.col("id")));

---

collect()

collect(): Promise<R[]>;

Defined in: data-frame.ts:924

Execute the plan and collect all result rows into a JS array.

For large datasets, prefer toLocalIterator() or forEach() to avoid loading everything into memory.

Returns

Promise<R[]>

---

columns()

columns(): Promise<string[]>;

Defined in: data-frame.ts:1055

Return the column names as a string array. Uses the AnalyzePlan.Schema RPC to resolve the schema without executing.

Returns

Promise<string[]>

---

count()

count(): Promise<bigint>;

Defined in: data-frame.ts:942

Return the number of rows.

Uses an aggregate count plan. The full dataset is not collected. Returns a bigint because Spark’s count(*) is LongType. Wrap in Number(...) when you know the row count fits in a JS safe integer.

Returns

Promise<bigint>

---

createGlobalTempView()

createGlobalTempView(viewName): Promise<void>;

Defined in: data-frame.ts:887

Register as a global temporary view. Throws if the view already exists.

Parameters

Parameter	Type
viewName	string

Returns

Promise<void>

---

createOrReplaceGlobalTempView()

createOrReplaceGlobalTempView(viewName): Promise<void>;

Defined in: data-frame.ts:876

Register as a global temporary view, replacing if it already exists.

Parameters

Parameter	Type
viewName	string

Returns

Promise<void>

---

createOrReplaceTempView()

createOrReplaceTempView(viewName): Promise<void>;

Defined in: data-frame.ts:854

Register this DataFrame as a temporary view with the given name. The view is session-scoped and will be dropped when the session ends.

Parameters

Parameter	Type
viewName	string

Returns

Promise<void>

---

createTempView()

createTempView(viewName): Promise<void>;

Defined in: data-frame.ts:865

Register as a temporary view. Throws if the view already exists.

Parameters

Parameter	Type
viewName	string

Returns

Promise<void>

---

crossJoin()

crossJoin(other): DataFrame;

Defined in: data-frame.ts:290

Alias for join with joinType=“cross”.

Parameters

Parameter	Type
other	DataFrame

Returns

DataFrame

---

cube()

cube(...columns): GroupedData;

Defined in: data-frame.ts:202

Multi-dimensional cube aggregation (all grouping-column combinations).

Parameters

Parameter	Type
…columns	(string | Column)[]

Returns

GroupedData

---

describe()

describe(...cols): DataFrame;

Defined in: data-frame.ts:684

Compute summary statistics (count, mean, stddev, min, max) for columns.

Parameters

Parameter	Type
…cols	string[]

Returns

DataFrame

---

distinct()

distinct(): DataFrame<R>;

Defined in: data-frame.ts:404

Alias for dropDuplicates() with no arguments.

Returns

DataFrame<R>

---

drop()

drop(...columnNames): DataFrame;

Defined in: data-frame.ts:295

Drop one or more columns by name.

Parameters

Parameter	Type
…columnNames	string[]

Returns

DataFrame

---

dropDuplicates()

dropDuplicates(...columnNames): DataFrame<R>;

Defined in: data-frame.ts:394

Remove duplicate rows, optionally considering only a subset of columns.

Parameters

Parameter	Type
…columnNames	string[]

Returns

DataFrame<R>

---

dropna()

dropna(how?, cols?): DataFrame<R>;

Defined in: data-frame.ts:500

Drop rows with null values.

Parameters

Parameter	Type	Default value
how	"any" | "all"	"any"
cols	string[]	[]

Returns

DataFrame<R>

---

dtypes()

dtypes(): Promise<[string, string][]>;

Defined in: data-frame.ts:1065

Return column names and their data types as [name, type] pairs. Uses the AnalyzePlan.Schema RPC.

Returns

Promise<[string, string][]>

---

except()

except(other): DataFrame;

Defined in: data-frame.ts:445

Return rows in this but not in other (distinct).

Parameters

Parameter	Type
other	DataFrame

Returns

DataFrame

---

exceptAll()

exceptAll(other): DataFrame;

Defined in: data-frame.ts:450

Return rows in this but not in other (duplicates kept).

Parameters

Parameter	Type
other	DataFrame

Returns

DataFrame

---

explain()

explain(mode?): Promise<string>;

Defined in: data-frame.ts:1098

Return the query execution plan as a string.

Parameters

Parameter	Type	Default value	Description
mode	"extended" | "simple" | "codegen" | "cost" | "formatted"	"simple"	Explain mode: “simple”, “extended”, “codegen”, “cost”, “formatted”

Returns

Promise<string>

---

fillna()

fillna(value, cols?): DataFrame<R>;

Defined in: data-frame.ts:490

Replace null values. If cols is empty, applies to all columns.

Parameters

Parameter	Type	Default value
value	string | number | boolean	undefined
cols	string[]	[]

Returns

DataFrame<R>

---

filter()

filter(condition): DataFrame<R>;

Defined in: data-frame.ts:160

Filter rows by a boolean Column expression or a SQL predicate string. A string is parsed server-side as SQL via expr(...).

Parameters

Parameter	Type
condition	string | Column

Returns

DataFrame<R>

Example

df.filter(col("status").eq(lit("active")));
  df.filter("status = 'active' AND region IN ('EU', 'US')");

---

first()

first(): Promise<R | null>;

Defined in: data-frame.ts:1017

Return the first row as a Row object, or null if the DataFrame is empty.

Returns

Promise<R | null>

---

forEach()

forEach(fn): Promise<void>;

Defined in: data-frame.ts:995

Process each row with a callback as it streams from the server.

Parameters

Parameter	Type
fn	(row) => void

Returns

Promise<void>

Example

await df.forEach((row) => console.log(row.name, row.salary));

---

getStorageLevel()

getStorageLevel(): Promise<StorageLevel>;

Defined in: data-frame.ts:841

Get the storage level used for caching this DataFrame. Returns the StorageLevel if cached, or NONE if not cached.

Returns

Promise<StorageLevel>

---

groupBy()

groupBy(...columns): GroupedData;

Defined in: data-frame.ts:185

Group by one or more columns, returning a GroupedData handle for aggregation.

Parameters

Parameter	Type
…columns	(string | Column)[]

Returns

GroupedData

---

head()

head(n?): Promise<R[]>;

Defined in: data-frame.ts:1025

Return the first n rows as an array (alias for limit + collect).

Parameters

Parameter	Type	Default value
n	number	1

Returns

Promise<R[]>

---

hint()

hint(name, ...parameters): DataFrame<R>;

Defined in: data-frame.ts:541

Attach an optimizer hint to this DataFrame.

Parameters

Parameter	Type
name	string
…parameters	(string | number | boolean)[]

Returns

DataFrame<R>

Examples

df.hint("broadcast")

df.join(right.hint("broadcast"), ...)

---

intersect()

intersect(other): DataFrame;

Defined in: data-frame.ts:435

Return rows present in both DataFrames (distinct).

Parameters

Parameter	Type
other	DataFrame

Returns

DataFrame

---

intersectAll()

intersectAll(other): DataFrame;

Defined in: data-frame.ts:440

Return rows present in both DataFrames (duplicates kept).

Parameters

Parameter	Type
other	DataFrame

Returns

DataFrame

---

isEmpty()

isEmpty(): Promise<boolean>;

Defined in: data-frame.ts:1075

Returns true if the DataFrame has no rows. Uses head(1) to check and stops after the first row.

Returns

Promise<boolean>

---

join()

join(
   other,
   condition?,
   joinType?): DataFrame;

Defined in: data-frame.ts:262

Join with another DataFrame.

Parameters

Parameter	Type	Default value	Description
other	DataFrame	undefined	The right side DataFrame
condition?	Column	undefined	Join condition (a boolean Column expression)
joinType?	| "inner" | "full_outer" | "left_outer" | "right_outer" | "left_semi" | "left_anti" | "cross"	"inner"	Type of join (default: “inner”)

Returns

DataFrame

---

limit()

limit(n): DataFrame<R>;

Defined in: data-frame.ts:214

Limit the number of rows.

Parameters

Parameter	Type
n	number

Returns

DataFrame<R>

---

melt()

melt(
   ids,
   values,
   variableColumnName,
   valueColumnName): DataFrame;

Defined in: data-frame.ts:757

Alias for unpivot().

Parameters

Parameter	Type
ids	(string | Column)[]
values	| (string | Column)[] | undefined
variableColumnName	string
valueColumnName	string

Returns

DataFrame

---

offset()

offset(n): DataFrame<R>;

Defined in: data-frame.ts:409

Skip the first N rows.

Parameters

Parameter	Type
n	number

Returns

DataFrame<R>

---

orderBy()

orderBy(...columns): DataFrame<R>;

Defined in: data-frame.ts:251

Alias for sort().

Parameters

Parameter	Type
…columns	(string | Column)[]

Returns

DataFrame<R>

---

persist()

persist(storageLevel?): Promise<DataFrame<R>>;

Defined in: data-frame.ts:814

Persist this DataFrame with the given storage level. Returns this DataFrame for method chaining.

Parameters

Parameter	Type	Default value	Description
storageLevel	StorageLevel	MEMORY_AND_DISK	How to store the cached data

Returns

Promise<DataFrame<R>>

---

printSchema()

printSchema(): Promise<void>;

Defined in: data-frame.ts:1113

Print the schema to the console in a tree format. Convenience method that calls schema() and formats the output.

Returns

Promise<void>

---

randomSplit()

randomSplit(weights, seed?): DataFrame<Row>[];

Defined in: data-frame.ts:716

Randomly split this DataFrame into multiple DataFrames by weight.

Parameters

Parameter	Type
weights	number[]
seed?	number

Returns

DataFrame<Row>[]

---

repartition()

repartition(numPartitions, ...columns): DataFrame<R>;

Defined in: data-frame.ts:620

Return a new DataFrame partitioned by the given number of partitions. This results in a full shuffle of the data.

Parameters

Parameter	Type	Description
numPartitions	number	Target number of partitions
…columns	(string | Column)[]	Optional partitioning columns

Returns

DataFrame<R>

---

repartitionByRange()

repartitionByRange(numPartitions, ...columns): DataFrame<R>;

Defined in: data-frame.ts:660

Return a new DataFrame partitioned by the given columns using range partitioning.

Parameters

Parameter	Type	Description
numPartitions	number	Target number of partitions
…columns	(string | Column)[]	Partitioning columns

Returns

DataFrame<R>

---

replace()

replace(to, subset?): DataFrame;

Defined in: data-frame.ts:702

Replace values matching old with new, optionally restricted to a column subset.

Parameters

Parameter	Type	Default value
to	Record<string, string | number | boolean | null>	undefined
subset	string[]	[]

Returns

DataFrame

---

rollup()

rollup(...columns): GroupedData;

Defined in: data-frame.ts:208

Multi-dimensional rollup aggregation (hierarchical subtotals).

Parameters

Parameter	Type
…columns	(string | Column)[]

Returns

GroupedData

---

sameSemantics()

sameSemantics(other): Promise<boolean>;

Defined in: data-frame.ts:898

Returns true if both DataFrames have the same logical plan.

Parameters

Parameter	Type
other	DataFrame

Returns

Promise<boolean>

---

sample()

sample(
   fraction,
   withReplacement?,
seed?): DataFrame<R>;

Defined in: data-frame.ts:476

Return a random sample of rows.

Parameters

Parameter	Type	Default value
fraction	number	undefined
withReplacement	boolean	false
seed?	number	undefined

Returns

DataFrame<R>

---

schema()

schema(): Promise<Record<string, unknown>>;

Defined in: data-frame.ts:1085

Return the schema of the DataFrame as a plain object. Uses the AnalyzePlan.Schema RPC to resolve column names and types without executing the query.

Returns

Promise<Record<string, unknown>>

---

select()

select(...columns): DataFrame;

Defined in: data-frame.ts:175

Project (select) a subset of columns.

Parameters

Parameter	Type
…columns	(string | Column)[]

Returns

DataFrame

---

selectExpr()

selectExpr(...exprs): DataFrame;

Defined in: data-frame.ts:559

Select columns using SQL expression strings. Each string is parsed by the server as an expression.

Parameters

Parameter	Type
…exprs	string[]

Returns

DataFrame

Example

df.selectExpr("age * 2 as doubled_age", "name")

---

semanticHash()

semanticHash(): Promise<number>;

Defined in: data-frame.ts:908

Returns a hash code of the logical plan.

Returns

Promise<number>

---

show()

show(numRows?, truncate?): Promise<void>;

Defined in: data-frame.ts:1125

Pretty-print the first numRows rows to the console as an ASCII table.

Mirrors PySpark’s df.show() behaviour. If truncate is true, strings longer than 20 characters are truncated with ....

Parameters

Parameter	Type	Default value
numRows	number	20
truncate	boolean	true

Returns

Promise<void>

---

sort()

sort(...columns): DataFrame<R>;

Defined in: data-frame.ts:226

Sort by one or more columns (ascending by default). Use col(“x”).desc() for descending order.

Parameters

Parameter	Type
…columns	(string | Column)[]

Returns

DataFrame<R>

---

sortWithinPartitions()

sortWithinPartitions(...columns): DataFrame<R>;

Defined in: data-frame.ts:587

Sort within each partition (non-global sort).

Parameters

Parameter	Type
…columns	(string | Column)[]

Returns

DataFrame<R>

---

summary()

summary(...statistics): DataFrame;

Defined in: data-frame.ts:693

Compute specified statistics for numeric and string columns.

Parameters

Parameter	Type
…statistics	string[]

Returns

DataFrame

---

tail()

tail(n): Promise<Row[]>;

Defined in: data-frame.ts:1042

Return the last n rows as an array.

Maps to Spark Connect’s Relation.Tail.

Parameters

Parameter	Type
n	number

Returns

Promise<Row[]>

---

take()

take(n): Promise<R[]>;

Defined in: data-frame.ts:1033

Return the first n rows as an array. Alias for head(). Matches PySpark’s take() semantics.

Parameters

Parameter	Type
n	number

Returns

Promise<R[]>

---

toDF()

toDF(...columnNames): DataFrame;

Defined in: data-frame.ts:512

Return a new DataFrame with renamed columns (positional).

Parameters

Parameter	Type
…columnNames	string[]

Returns

DataFrame

---

toLocalIterator()

toLocalIterator(): AsyncIterableIterator<R>;

Defined in: data-frame.ts:978

Async iterator that yields rows one at a time. Only one batch is in memory at a time.

Returns

AsyncIterableIterator<R>

Example

for await (const row of df.toLocalIterator()) {
    console.log(row);
  }

---

transform()

transform<T>(fn): T;

Defined in: data-frame.ts:580

Apply a user-defined function to this DataFrame and return the result. This is purely client-side; it just calls fn(this).

Enables fluent pipeline composition:

Type Parameters

Type Parameter
T extends DataFrame<Row>

Parameters

Parameter	Type
fn	(df) => T

Returns

T

Example

df.transform(withDoubledAge).transform(withSalaryBand)

---

union()

union(other): DataFrame;

Defined in: data-frame.ts:420

Return a new DataFrame with rows from both this and other (duplicates kept).

Parameters

Parameter	Type
other	DataFrame

Returns

DataFrame

---

unionAll()

unionAll(other): DataFrame;

Defined in: data-frame.ts:425

Alias for union().

Parameters

Parameter	Type
other	DataFrame

Returns

DataFrame

---

unionByName()

unionByName(other, allowMissingColumns?): DataFrame;

Defined in: data-frame.ts:430

Union by column name (rather than position), keeping duplicates.

Parameters

Parameter	Type	Default value
other	DataFrame	undefined
allowMissingColumns	boolean	false

Returns

DataFrame

---

unpersist()

unpersist(blocking?): Promise<DataFrame<R>>;

Defined in: data-frame.ts:828

Remove this DataFrame from the cache.

Parameters

Parameter	Type	Default value	Description
blocking	boolean	false	Whether to block until the operation completes

Returns

Promise<DataFrame<R>>

---

unpivot()

unpivot(
   ids,
   values,
   variableColumnName,
   valueColumnName): DataFrame;

Defined in: data-frame.ts:738

Unpivot from wide format to long format.

Parameters

Parameter	Type
ids	(string | Column)[]
values	| (string | Column)[] | undefined
variableColumnName	string
valueColumnName	string

Returns

DataFrame

---

where()

where(condition): DataFrame<R>;

Defined in: data-frame.ts:170

Alias for filter().

Parameters

Parameter	Type
condition	string | Column

Returns

DataFrame<R>

---

withColumn()

withColumn(name, expression): DataFrame;

Defined in: data-frame.ts:308

Add or replace a column.

Parameters

Parameter	Type
name	string
expression	Column

Returns

DataFrame

Example

df.withColumn("doubled", col("value").multiply(lit(2)))

---

withColumnRenamed()

withColumnRenamed(existing, newName): DataFrame;

Defined in: data-frame.ts:332

Rename a single column.

Parameters

Parameter	Type
existing	string
newName	string

Returns

DataFrame

---

withColumns()

withColumns(colMap): DataFrame;

Defined in: data-frame.ts:319

Add or replace multiple columns at once.

Parameters

Parameter	Type
colMap	Record<string, Column>

Returns

DataFrame

---

withColumnsRenamed()

withColumnsRenamed(colsMap): DataFrame;

Defined in: data-frame.ts:345

Rename multiple columns at once.

Parameters

Parameter	Type	Description
colsMap	Record<string, string>	mapping of { existingName: newName }

Returns

DataFrame

---

withWatermark()

withWatermark(eventTimeColumn, delayThreshold): DataFrame<R>;

Defined in: data-frame.ts:373

Attach an event-time watermark to a streaming DataFrame.

Bounds how late an event can arrive before Spark considers it dropped for stateful streaming operators (windowed aggregations, stream-stream joins, dropDuplicates). No effect on a batch DataFrame.

Parameters

Parameter	Type	Description
eventTimeColumn	string	name of the event-time column
delayThreshold	string	Spark interval string, e.g. "10 minutes", "1 hour"

Returns

DataFrame<R>

Example

spark.readStream.format("rate").load()
    .withWatermark("timestamp", "10 minutes")
    .groupBy(window(col("timestamp"), "5 minutes"))
    .agg(count("*").alias("events"))

---

writeTo()

writeTo(tableName): DataFrameWriterV2;

Defined in: data-frame.ts:782

Returns a DataFrameWriterV2 for writing to the given table using the DataSource V2 API (catalog-aware, supports create/replace/append/overwrite).

Parameters

Parameter	Type
tableName	string

Returns

DataFrameWriterV2