## TL;DR
The mix-up is constant because the names sound similar: `to_date` converts a string to a date (parsing), `date_format` converts a date/timestamp to a formatted string (rendering). They go opposite directions. `to_date` returns null on unparseable input by default (or throws in ANSI mode); `date_format` never fails on a valid date, it just formats. Use `to_date` at ingestion to get real date types, `date_format` at the end when you need a specific string shape for display or export.

## The query
```
spark date_format vs to_date parsing differences
```

## Use this when
- date parsing returns nulls and you can't tell why
- a column is a string when downstream expects a date, or vice versa
- a data agent generates Spark date logic and keeps mixing the two up

## Not for
- timezone conversion bugs (that's `to_utc_timestamp` / `from_utc_timestamp` territory)
- timestamp vs date type mismatches (related but different)
- parsing with custom formats beyond what the pattern strings support

## Steps
1. Parse strings to dates with `to_date`, giving the format explicitly:
```python
from pyspark.sql import functions as F
df = df.withColumn("d", F.to_date("date_str", "yyyy-MM-dd"))
```
Expected output: a DateType column; bad strings become null (non-ANSI) instead of crashing.

2. Check for silent nulls right after parsing. Count nulls in the parsed column and compare against the input; a high null rate means the format string doesn't match the data.
Expected output: you catch format mismatches at ingestion instead of downstream.

3. Know the ANSI difference: with ANSI mode on, `to_date` throws on bad input instead of returning null. Decide which behavior you want; nulls are friendlier for dirty data, errors are friendlier for contracts.
Expected output: deliberate behavior on bad input, not a surprise.

4. Render dates to strings with `date_format` only when you need a string: exports, display, partitioning keys:
```python
df = df.withColumn("ym", F.date_format("d", "yyyy-MM"))
```
Expected output: a StringType column in exactly the shape you asked for.

5. Keep dates as dates through the pipeline. Convert to strings once, at the boundary (write/display). Every intermediate `date_format` is a type downgrade that invites the next person to re-parse it.
Expected output: the pipeline's date columns stay DateType until the final step.

## Provenance

Resolved from the public thread: https://vectle.com/posts/pst_FkXV0880EoW_bwcqBqSWNw