DataFrameWriter

Defined in: data-frame-writer.ts:30

Writes the contents of a DataFrame to an external storage system or a catalog table.

Obtained via DataFrame.write. Mirrors Spark’s path-oriented DataFrameWriter (V1). For catalog-aware, atomic table writes, use DataFrameWriterV2 via df.writeTo(table).

Example

await df.write.format("parquet").mode("overwrite").save("/path/to/output");
await df.write.mode("append").saveAsTable("analytics.events");

See

Spark source: DataFrameWriter.scala

Constructors

Constructor

new DataFrameWriter(df): DataFrameWriter;

Defined in: data-frame-writer.ts:39

Parameters

Parameter	Type
df	DataFrame

Returns

DataFrameWriter

Methods

bucketBy()

bucketBy(
   numBuckets,
   col, ...
   cols): this;

Defined in: data-frame-writer.ts:89

Bucket the output by the given columns with a fixed number of buckets. Only applicable when saving to a table (saveAsTable).

Parameters

Parameter	Type
numBuckets	number
col	string
…cols	string[]

Returns

this

---

csv()

csv(path): Promise<void>;

Defined in: data-frame-writer.ts:168

Shortcut for .format(“csv”).save(path).

Parameters

Parameter	Type
path	string

Returns

Promise<void>

---

format()

format(fmt): this;

Defined in: data-frame-writer.ts:44

Set the output format (e.g. “parquet”, “json”, “csv”, “orc”, “delta”).

Parameters

Parameter	Type
fmt	string

Returns

this

---

insertInto()

insertInto(tableName): Promise<void>;

Defined in: data-frame-writer.ts:146

Insert the DataFrame’s contents into the given table. Unlike saveAsTable, insertInto does not create the table; it must already exist.

Parameters

Parameter	Type
tableName	string

Returns

Promise<void>

---

json()

json(path): Promise<void>;

Defined in: data-frame-writer.ts:163

Shortcut for .format(“json”).save(path).

Parameters

Parameter	Type
path	string

Returns

Promise<void>

---

mode()

mode(m): this;

Defined in: data-frame-writer.ts:56

Set the save mode:

• “append” - Append to existing data
• “overwrite” - Overwrite existing data
• “error” (default) - Error if data already exists
• “ignore” - Silently ignore if data already exists

Parameters

Parameter	Type
m	SaveMode

Returns

this

---

option()

option(key, value): this;

Defined in: data-frame-writer.ts:62

Set a single write option.

Parameters

Parameter	Type
key	string
value	string

Returns

this

---

options()

options(opts): this;

Defined in: data-frame-writer.ts:68

Set multiple write options.

Parameters

Parameter	Type
opts	Record<string, string>

Returns

this

---

orc()

orc(path): Promise<void>;

Defined in: data-frame-writer.ts:178

Shortcut for .format(“orc”).save(path).

Parameters

Parameter	Type
path	string

Returns

Promise<void>

---

parquet()

parquet(path): Promise<void>;

Defined in: data-frame-writer.ts:173

Shortcut for .format(“parquet”).save(path).

Parameters

Parameter	Type
path	string

Returns

Promise<void>

---

partitionBy()

partitionBy(...columns): this;

Defined in: data-frame-writer.ts:74

Partition the output by the given column names.

Parameters

Parameter	Type
…columns	string[]

Returns

this

---

save()

save(path): Promise<void>;

Defined in: data-frame-writer.ts:118

Save the DataFrame to the given path.

Sends a WriteOperation command through the Spark Connect RPC.

Parameters

Parameter	Type
path	string

Returns

Promise<void>

---

saveAsTable()

saveAsTable(tableName): Promise<void>;

Defined in: data-frame-writer.ts:131

Save the DataFrame as a named table.

Parameters

Parameter	Type	Description
tableName	string	The fully qualified or unqualified table name

Returns

Promise<void>

---

sortBy()

sortBy(...columns): this;

Defined in: data-frame-writer.ts:80

Sort the output within each partition by the given column names.

Parameters

Parameter	Type
…columns	string[]

Returns

this

---

text()

text(path): Promise<void>;

Defined in: data-frame-writer.ts:183

Shortcut for .format(“text”).save(path).

Parameters

Parameter	Type
path	string

Returns

Promise<void>