DataFrameReader

Defined in: spark-session.ts:653

Fluent reader for loading data into a DataFrame. Returned by spark.read; configure the format, options, and schema, then terminate with a format shortcut (csv, json, parquet, orc, text) or .load().

Example

spark.read.parquet("s3://bucket/events/");

spark.read
  .schema("id INT, name STRING")
  .option("header", "true")
  .csv("/data/people.csv");

See

Spark source: DataFrameReader.scala

Constructors

Constructor

new DataFrameReader(session): DataFrameReader;

Defined in: spark-session.ts:659

Parameters

Parameter	Type
session	SparkSession

Returns

DataFrameReader

Methods

csv()

csv(path): DataFrame;

Defined in: spark-session.ts:740

Shortcut for .format(“csv”).load(path).

Parameters

Parameter	Type
path	string

Returns

DataFrame

---

format()

format(fmt): this;

Defined in: spark-session.ts:693

Parameters

Parameter	Type
fmt	string

Returns

this

---

json()

json(path): DataFrame;

Defined in: spark-session.ts:735

Shortcut for .format(“json”).load(path).

Parameters

Parameter	Type
path	string

Returns

DataFrame

---

load()

load(path): DataFrame;

Defined in: spark-session.ts:715

Trigger a Read plan node. The resulting DataFrame is lazy; no data is fetched until .collect() or an action is called.

This maps to Spark Connect’s Relation.Read with ReadType.DataSource: { format: “parquet”, paths: […], options: {…} }

Parameters

Parameter	Type
path	string

Returns

DataFrame

---

option()

option(key, value): this;

Defined in: spark-session.ts:698

Parameters

Parameter	Type
key	string
value	string

Returns

this

---

options()

options(opts): this;

Defined in: spark-session.ts:703

Parameters

Parameter	Type
opts	Record<string, string>

Returns

this

---

orc()

orc(path): DataFrame;

Defined in: spark-session.ts:750

Shortcut for .format(“orc”).load(path).

Parameters

Parameter	Type
path	string

Returns

DataFrame

---

parquet()

parquet(path): DataFrame;

Defined in: spark-session.ts:745

Shortcut for .format(“parquet”).load(path).

Parameters

Parameter	Type
path	string

Returns

DataFrame

---

schema()

schema(schema): this;

Defined in: spark-session.ts:668

Set the schema for the data source. Accepts a DDL-formatted string (e.g. “name STRING, age INT”) or a StructType with a toDDL() method.

Parameters

Parameter	Type
schema	| string | { toDDL: string; }

Returns

this

---

table()

table(tableName): DataFrame;

Defined in: spark-session.ts:726

Read a named table (catalog table or temp view).

Parameters

Parameter	Type
tableName	string

Returns

DataFrame

---

text()

text(path): DataFrame;

Defined in: spark-session.ts:755

Shortcut for .format(“text”).load(path).

Parameters

Parameter	Type
path	string

Returns

DataFrame