Skip to content

Add effect/dataframe - #10641

Open
fubhy wants to merge 90 commits into
feat/arrow-supportfrom
feat/dataframe-support
Open

fubhy wants to merge 90 commits into
feat/arrow-supportfrom
feat/dataframe-support

Conversation

@fubhy

@fubhy fubhy commented Nov 14, 2026 •

Copy link
Copy Markdown
Member

Stack

main
├─ #10643 effect/encoding/FlatBuffers
│   └─ #10649 effect/arrow
│       ├─ this PR: effect/dataframe
│       └─ #10655 effect/encoding/Parquet (carries #10652 by merge)
│           └─ #10666 ArrowParquet, Parquet files as Arrow record batches
├─ #10652 effect/encoding/Thrift
└─ #10657 effect/encoding/Compression

Each layer depends only on the ones below it. #10655 joined the stack so that #10666 can read Parquet files as Arrow record batches. It carries #10652 (effect/encoding/Thrift) by merge until that lands on main. #10657 supplies optional codecs through the codecs option of the Parquet reader. Arrow Flight and Protobuf / gRPC / ConnectRPC are out of scope.

Summary

Experimental effect/dataframe: typed columnar frames on top of effect/arrow, plus lazy Query plans that run in memory, batch by batch over streamed tables, or as SQL through effect/sql. Query operations cover filters, projections, sorts, groups and aggregates, joins, windows and reshaping. Semantics follow Polars 2.1, and column names and types are checked at compile time. The SQL engine targets PostgreSQL (and PGlite) only; other dialects are future work.

One type model

The in-memory engine decides result types for every engine.

  • Table columns are checked structurally against their Schema (nested lists and structs included; BigInt needs a 64-bit integer, BigDecimal any decimal). Literal, enum and refined fields are also checked with Schema.is on their values.
  • fromRows, empty, validate and Query.table return Normalize<T> rows: optional keys become required and undefined becomes null, recursively.
  • The SQL compiler runs the plan through the memory engine on zero-row inputs, exposes the resulting Arrow schema on Compiled.schema, casts every output column to text and decodes against that schema. Results have the same types as the memory engine whatever the driver returns, empty results included. The PGlite suite compares types strictly now.

One row rule

E.isRowWise (no window functions) is the single expression-level rule.

  • Predicate pushdown never moves a window predicate, and such a filter is a barrier.
  • Streaming runs a filter or aggregate per batch only when its expressions are row-wise. Otherwise it collects first.
  • SQL always computes a filter predicate in a subquery, so window predicates see the unfiltered rows.
  • Windows inside grouped aggregate arguments run per group, as in Polars.

Rounding

round was checked per operand type against PostgreSQL 18 (PGlite) and Polars 2.1. PostgreSQL's ROUND(numeric) rounds half away from zero and ROUND(float8) half to even. ROUND(float8, int) doesn't exist. The engines round half to even everywhere:

  • Floats round x * 10^d in float64 (ROUND(float8)), so 2.675 gives 2.68 and 1.005 gives 1 at two places, as in Polars.
  • Decimals round half to even at d places into Decimal(min(p+1, 38), s), as in Polars. The memory engine used to ignore round on decimals. The SQL engine uses a FLOOR-based CASE rather than ROUND(numeric).
  • Integers are unchanged. Negative decimals fail with a DataFrameError.

Other review fixes

Memory and streaming: head no longer pulls an extra batch and head(0) pulls nothing; zero-row batches aggregate like an empty table; each named table is read once per run, Query.stream included; explain doesn't read streamed tables; negative half-even decimal casts; UInt32 floorDivide/mod by zero give null; decimal sums and running sums fail on 64-bit overflow instead of wrapping; list children of sliced Arrow input are trimmed; slice computes its end from the unclamped start (Polars: slice(-5, 1) on 3 rows is empty).

SQL: boolean sum and cumSum; temporal, duration and decimal VALUES literals keep sub-millisecond precision; an empty partition list emits OVER (); countDistinct on empty input is 0; hidden helper columns use one prefix per compile that no plan column starts with; sqrt of negatives is NaN like the memory engine; errors carry the driver's message.

Rejected: the finding that decimal add should respect the declared precision. Polars types add, multiply and sum as Decimal(38, s), so the real limit is 64-bit storage, and that limit is enforced.

Second review round

Fixed, each with a regression test that fails before the fix:

  • Filters and slices move only below operators whose expressions are all row-wise, so sorting by cumSum(x) then filtering keeps Polars' order. The same operator-level rule decides what streams per batch. Gold case window/filter-after-sort-by-window.
  • Every computed decimal is stored through one 64-bit check: literals, abs/negate of the minimum and rolling sums fail instead of wrapping. Out-of-range casts give null (f64 1e308 to decimal(18,18), UInt64 max, negative decimals to UInt64). Decimal rolling means are no longer in unscaled units.
  • Partial sums for streaming are exact bigints checked once at the end, so a sum's success no longer depends on batch boundaries.
  • DateTimeUtc fields accept only UTC timestamps within the DateTime range; UTC/Etc/UTC/+00:00 zones read as DateTime.Utc. DateTimeZoned is documented as unsupported.
  • Streamed results without rows keep their column types, nested declared columns included.
  • The SQL engine checks every table it reads against its declared columns: non-nullable columns have no nulls, and literal, enum and refined fields hold only values the schema accepts (in the same statement as the query since the third round).
  • SQL temporal casts go through their units in both directions, staying exact beyond 2^53.
  • SQL computes window functions inside aggregate arguments in a subquery partitioned by the group keys, then aggregates.
  • SQL boolean min/max (plain, windowed, cumulative) use BOOL_AND/BOOL_OR; unique([]) keeps the first row.
  • DuckDB is gated out: Sql.Dialect is gone, Sql.compile(plan) takes no options, and Sql.engine/Sql.layer are values. The DuckDB lowering had no driver or tests here, so it isn't shipped.

Third review round

The invariant behind these fixes: a table column's declared physical type is fixed once, from the Schema plus options.types, and every constructor, validator, stream batch and SQL read is converted to it exactly, through the same code.

  • Plan.TableColumn carries decimal precision/scale and temporal units. A BigDecimal field needs options.types in fromRows, validate and Query.table (breaking for Query.table and validate without it), so empty streams now give typed results: sum over an empty decimal stream is a BigDecimal, not 0.
  • A declared decimal needs a precision that is an integer from 1 to 38 and a 32-bit scale. Query.table, DataFrame.validate and DataFrame.fromRows fail for anything else with a DataFrameError that names the column, and so does reading the table of a plan that carries one.
  • A decimal override in options.types must be 128-bit, like a decimal field of an Arrow schema. Overrides of 32, 64 or 256 bits used to be declared as 128-bit decimals; they now fail with a DataFrameError that names the column.
  • Decimal scales work over the whole 32-bit range, negative ones included. A rescale by 20 digits or more is decided from the 64 bits of storage without building the power of ten: zero stays zero, a value scaled down rounds to zero, a value scaled up leaves storage (null from an explicit cast, a DataFrameError from a declaration, a common type or arithmetic). Casts to and from a negative scale used to die with a RangeError, as did the int32 extremes.
  • Expr.castDecimal fails for a precision outside 1 to 38 or a scale that isn't a 32-bit integer, a product fails when the sum of its scales leaves 32 bits, and a cast to text fails when the scale is more than a string can hold. The SQL compiler takes scales from -1000 to 1000, the range of PostgreSQL's numeric, and fails for others.
  • An integer and a decimal with a negative scale meet in a decimal of scale 0, so comparisons, when, coalesce, join keys and Query.concat no longer round the integer to the decimal's scale. Only an explicit cast rounds.
  • A BigDecimal literal has a precision of at least its scale and at most 38, so a literal with a scale above 38 is a valid 128-bit decimal. A literal of a plan with a precision outside 1 to 38, a scale outside 32 bits or more digits than its precision fails with a DataFrameError when it is evaluated.
  • Casts between floats and decimals, mean and rollingMean keep finite values at scales above 308, where 10^scale is infinite as a float.
  • Expr.isIn compares a decimal by value: 7.00 is one of 7, 7n and a BigDecimal equal to it at any scale, and no longer one of 700, the unscaled units it is stored as. BigDecimal, DateTime and Duration candidates are compared with eq, so they match, run on SQL and survive the JSON form of a plan. A candidate with a fraction no longer kills a query over 64-bit integers. Typed candidates are joined in halves, so a list of thousands stays a shallow plan. On SQL, candidates with a fraction go in a list of their own against the column as a float, so one of them doesn't make PostgreSQL round a decimal or a 64-bit integer and its other candidates. A UInt64 above the signed range is one of its value. Candidates of the value's kind are supported and follow each engine's comparison rules, as eq does: a decimal next to a number that is not an integer, and a zoned timestamp next to a DateTime outside a UTC session, can differ on a database. Candidates of another kind than the value (a string for an integer) are documented as not portable.
  • Streamed batches are converted to the declared types before any operator sees them, and lossy conversions fail. Partial aggregates and group keys of undeclared tables are combined in their common type, so batches at scales 1 and 2 sum to 2.43 (it was 13.5), and 1.2/1.20 group together.
  • fromRows and empty share validate's type check, so a DateTimeUtc field with a UTC override reads as DateTime.Utc and other zones fail.
  • Schema value checks look through unions, suspend, structs and arrays; recursive schemas fail with a DataFrameError instead of overflowing the stack.
  • SQL relation contract: the checks and the result are one statement. Each table is read once through a NOT MATERIALIZED CTE; PostgreSQL never inlines one that calls a volatile function, so a nextval view is evaluated once for both, and stable tables are inlined at no measurable cost (MATERIALIZED cost 1.3-1.9x). Column types are checked first against the catalog with the memory engine's rules.
  • SQL checks are generated from one table covering every column type: Null columns, DateTime range, UTC-only, exact unit and decimal fit. timestamptz reads as UTC whatever the session zone; intervals with months are rejected.
  • SQL reads declared table columns as exactly the types the memory engine conforms them to, decimal and every temporal unit included.
  • Mixed-unit temporal operands meet in the coarser unit before arithmetic, comparisons, joins and concat, on both engines. Durations rescale toward zero, as in Polars 2.1 (memory used to round down).

Fourth review round

  • Implicit conversions to a common type go through one checked conversion, K.coerce, and fail when the common type can't hold a value. This covers concat, join keys, unpivot, streamed key and partial merges, and operand promotion. Expr.cast still gives null. An integer literal the column type can't hold widens the type, as in Polars (Int8 + 300 is Int16).
  • Partial sums stay exact, and the result type and overflow are applied once at the end, through the same function as the in-memory path.
  • Temporal casts that leave 64 bits give null instead of wrapping, as in Polars.
  • A struct schema's own checks run on whole rows in memory. SQL rejects such tables when the query compiles.
  • SQL reads each table once per statement through a MATERIALIZED CTE that both the checks and the result read, so the returned rows are exactly the checked rows for every relation (TABLESAMPLE views included). The cost is 1.2-2x on PGlite, and filters can't use indexes. Two Query.table nodes with the same name share one read.
  • SQL checks every declared column against its whole physical type, including integer ranges, Float32 round trips, the 64-bit decimal bound and finite dates. The checks run before projection, count or filters can drop the column.
  • SQL reads every column as its declared type, so enums, char(n), varchar, smallint and real behave as in memory.

Fifth review round

  • Every batch of a streamed step has the column types of that step's whole result, so partial aggregates and group keys never need converting. A streamed concat reads the first batch of each input, finds the common types the same way the in-memory concat does, and converts every batch to them. A declared table converts each batch to its declared types. An undeclared table's batches must keep the columns and types of its first batch, as in an Arrow stream; otherwise the query fails. This replaces the per-batch common-type tracking from the fourth round, and streamed results now match concat for the same input order, failures included.
  • Filters stay above a concat, and filters on join keys go to the left side only, since the join keeps the left key type.
  • Integer literals are fitted to the other operands wherever they appear, in coalesce/fillNull, when and binary operations.
  • The SQL compiler names everything it invents (CTE sources, helper and check columns) with a prefix no table or column name in the plan starts with, so a generated CTE can't shadow a real table. The column type check qualifies the columns it reads, so a missing column can't resolve to a catalog column.

Sixth review round

  • An expression's type depends only on the input schema and the plan, never on data, batch boundaries or rewrites. Integer literals are fitted from plan constants: a when, coalesce or fillNull of integer literals spans the range of all its literals, whichever rows select them. Other expressions built from literals count as Int32, and a cast always fixes the type.
  • Every streamed node emits at least one batch, possibly without rows, unless its columns are unknown. A declared table without batches, and head(0), take their columns from the declarations without reading the source. A result whose columns depend on an undeclared table of which no batch was read fails, asking for the table's columns, instead of inventing a frame without columns.
  • DataFrame.fromArrowTable of a table without batches keeps its fields' nullability.
  • Names the SQL compiler invents use a short prefix (__, or __1_, __2_, … when a user name starts with it), so PostgreSQL never truncates them. Two columns of one relation that share their first 63 bytes are rejected when the query compiles.

Seventh review round

  • A table named more than once in a plan is read lazily: one shared read per run, started only when some node first pulls its batches. Zero-row plans over repeated tables, such as head(concat([t, t]), 0), read nothing, and plans that need rows still read the table once and give every use the same rows.
  • The SQL compiler compares table and column names the way PostgreSQL keeps them (the first 63 bytes, whole characters) wherever declarations share a source. Declarations whose names match in their first 63 bytes read one relation and one column once per statement, and each keeps its own checks.

Eighth review round

  • Rows rebuilt for a check (a struct with an optional field) keep __proto__ as an own property. Before, the assignment set the prototype, so a refinement saw a row without the field through fromRows, validate and Query.table. DF-C032 has accepted and rejected rows for a top-level and a nested struct.
  • Merges the Arrow head with the structural-validation wording of ArrowIpc.

Status

Ready for re-review. Known limits: the common type of more than two batch types depends on their order, as in Polars, since the supertype isn't associative; memory fails on decimal mixes beyond 64 bits that PostgreSQL's numeric accepts; decimals are stored in 64 bits (failures, never wrapping); timestamps and durations on PostgreSQL keep microsecond precision; casting a float to a decimal goes through float64; Time table columns and nested table columns aren't supported on SQL; a decimal mean is computed in float64; boolean rollingMin/rollingMax aren't typed for booleans.

Tests

  • test/dataframe/DataFrame.test.ts and Query.test.ts. The Polars gold corpus (regenerated with generate_gold.py and Polars 2.1.0) runs three ways: inputs inline, as catalog frames, and as streams of two-row batches.
  • typetest/DataFrame.tst.ts and packages/sql/pglite/test/DataFrameSql.test.ts.

Validation at eddf419102: pnpm check, pnpm lint, pnpm jsdocs --check, pnpm doctest --run packages/effect/src/dataframe packages/effect/src/arrow (27 in 12 files), pnpm vitest --run test/dataframe test/arrow test/encoding (1,233 in 23 files), pnpm test-types DataFrame (50 tests, 134 assertions), PGlite suite (84).

Closes EFF-3804

@changeset-bot

changeset-bot Bot commented Nov 14, 2026 •

Copy link
Copy Markdown

🦋 Changeset detected

Latest commit: eddf419

The changes in this PR will be included in the next version bump.

This PR includes changesets to release 35 packages
Name Type
effect Patch
@effect/opentelemetry Patch
@effect/vitest Patch
@effect/ai-amazon-bedrock Patch
@effect/ai-anthropic Patch
@effect/ai-cloudflare Patch
@effect/ai-openai-compat Patch
@effect/ai-openai Patch
@effect/ai-openrouter Patch
@effect/ai-typesafe Patch
@effect/atom-react Patch
@effect/atom-solid Patch
@effect/atom-vue Patch
@effect/platform-browser Patch
@effect/platform-bun Patch
@effect/platform-cloudflare Patch
@effect/platform-deno Patch
@effect/platform-node-shared Patch
@effect/platform-node Patch
@effect/sql-clickhouse Patch
@effect/sql-d1 Patch
@effect/sql-libsql Patch
@effect/sql-mssql Patch
@effect/sql-mysql2 Patch
@effect/sql-pg Patch
@effect/sql-pglite Patch
@effect/sql-sqlite-bun Patch
@effect/sql-sqlite-do Patch
@effect/sql-sqlite-node Patch
@effect/sql-sqlite-react-native Patch
@effect/sql-sqlite-wasm Patch
@effect/docgen Patch
@effect/doctest Patch
@effect/graphql-generator Patch
@effect/openapi-generator Patch

Not sure what this means? Click here to learn what changesets are.

Click here if you're a maintainer who wants to add another changeset to this PR

@effect-janitor effect-janitor Bot added the 4.0 label Nov 14, 2026
@fubhy
fubhy marked this pull request as draft November 14, 2026 17:01
@fubhy

fubhy commented Nov 14, 2026

Copy link
Copy Markdown
Member Author

@effect-bot just doodling (heavy slop) here for some side quests but might be worth evaluating nonetheless

@effect-bot

Copy link
Copy Markdown
Contributor

Added your request to the existing evaluation task (EFF-3804), with the exploratory framing noted. The review will assess the Arrow/dataframe design and suitability for Effect, and report findings back here without pushing changes to the PR.

@effect-bot effect-bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Verdict: REQUEST_CHANGES on d4eeba3.

I would keep exploring the idea, but I would not merge this version into Effect. Arrow-backed ingestion and typed queries fit Effect's streaming and SQL use cases. The public frame/expression/plan split is understandable, and an independent Polars oracle is a useful foundation. The blockers are concrete: valid queries silently change results under optimization or rebatching, Arrow round-trips can change values, and several paths return values that contradict the advertised TypeScript row type.

The design needs one source of truth for physical types, runtime value types, nullability, and whether an expression depends on other rows. Today that information is independently approximated in schema conformance, the optimizer, the streaming classifier, SQL inference, and driver decoding. Reuse the existing expression classification consistently and carry output type metadata to the engine boundary rather than adding more casts or operation-specific exceptions.

Given the exploratory framing, I recommend first accepting a smaller, independently interoperable Arrow boundary and a sound typed in-memory frame/query contract. Defer the SQL/streaming expansion until the same typed operations produce the same result across full frames, arbitrary batches, and driver representations, except for explicitly documented backend differences. A handwritten FlatBuffers/Arrow implementation can be justified, but its layout/normalization invariants and reference interoperability coverage need to be explicit before core ownership is reasonable.

Validation on this head: nix develop -c pnpm install and nix develop -c pnpm check passed; the five dataframe runtime files plus the PGlite integration file passed (612 tests); pnpm test-types DataFrame.tst.ts passed on both configured targets (40 executed tests); targeted Arrow/dataframe doctests passed (21 examples). JSDoc checking passed. Root pnpm lint failed dprint formatting on arrow/internal/ipc.ts and the three dataframe gold JSON files; oxlint completed before that failure. Focused temporary public-API probes reproduced the findings. No tracked source/test changes or pushes were made. Passing oracle tests currently miss physical-layout boundaries and valid stateful-expression compositions; SQL comparisons also normalize away some runtime type differences. Keep regression coverage at these public boundaries rather than multiplying internal-kernel tests.

The inline comments detail the specific defects. Experimental status is appropriate for API evolution, but it does not make silent data corruption or unsound row types acceptable.

@fubhy

fubhy commented Nov 15, 2026

Copy link
Copy Markdown
Member Author

@effect-bot Apply your suggestions. Take over this pull request but split it up into Apache Arrow & DataFrame separately as stacked PRs. Do not add Arrow Flight yet since that is blocked by adding broader Protobuf / gRPC / ConnectRPC support later (and also separately).

@effect-bot

Copy link
Copy Markdown
Contributor

Added this takeover request to EFF-3804: apply the review suggestions and split the work into stacked PRs, with Apache Arrow first and DataFrame on top. Arrow Flight is explicitly out of scope until the separate Protobuf / gRPC / ConnectRPC work is in place.

effect-bot pushed a commit that referenced this pull request Nov 15, 2026
Typed columnar data frames and lazy queries on top of effect/arrow, with in-memory, streaming and SQL engines. Split out of #10641.
@effect-bot
effect-bot force-pushed the feat/dataframe-support branch from d4eeba3 to 096cbe4 Compare November 15, 2026 11:28
@effect-bot effect-bot mentioned this pull request Nov 15, 2026
@effect-bot effect-bot changed the title Add typed columnar data frames with Arrow support Add effect/dataframe Nov 15, 2026
@effect-bot
effect-bot changed the base branch from main to feat/arrow-support November 15, 2026 11:29
@effect-janitor effect-janitor Bot removed the 4.0 label Nov 15, 2026
effect-bot pushed a commit that referenced this pull request Nov 15, 2026
Typed columnar data frames and lazy queries on top of effect/arrow, with in-memory, streaming and SQL engines. Split out of #10641.
@effect-bot
effect-bot force-pushed the feat/dataframe-support branch from 096cbe4 to 62266a8 Compare November 15, 2026 12:09
@effect-bot
effect-bot marked this pull request as ready for review November 15, 2026 12:10
effect-bot pushed a commit that referenced this pull request Nov 15, 2026
Apache Arrow schemas, columns, record batches and tables with IPC stream and file codecs. Split out of #10641.
effect-bot pushed a commit that referenced this pull request Nov 15, 2026
Typed columnar data frames and lazy queries on top of effect/arrow, with in-memory, streaming and SQL engines. Split out of #10641.
@effect-bot
effect-bot force-pushed the feat/dataframe-support branch from 62266a8 to 0a6efc3 Compare November 15, 2026 12:46
effect-bot pushed a commit that referenced this pull request Nov 15, 2026
Typed columnar data frames and lazy queries on top of effect/arrow, with in-memory, streaming and SQL engines. Split out of #10641.
@effect-bot
effect-bot force-pushed the feat/dataframe-support branch from 0a6efc3 to d1c6e67 Compare November 15, 2026 13:37
effect-bot pushed a commit that referenced this pull request Nov 15, 2026
Apache Arrow schemas, columns, record batches and tables with IPC stream and file codecs. Split out of #10641.
effect-bot pushed a commit that referenced this pull request Nov 15, 2026
Typed columnar data frames and lazy queries on top of effect/arrow, with in-memory, streaming and SQL engines. Split out of #10641.
@effect-bot
effect-bot force-pushed the feat/dataframe-support branch from 509d3f0 to d7121cc Compare November 15, 2026 19:30
@effect-bot
effect-bot force-pushed the feat/dataframe-support branch from 691ab52 to 8968ad5 Compare November 16, 2026 09:26
@effect-bot

Copy link
Copy Markdown
Contributor

Two stack changes that touch this PR only indirectly:

  • feat/arrow-support got one commit, 07a5358e28, which stops ArrowIpc.decode from decoding a message's metadata again on every chunk. It only changes arrow/internal/ipc.ts, which this branch doesn't touch, so I left feat/dataframe-support at 8968ad5116 and did not rebase it. It will pick the commit up on the next rebase.
  • Add effect/encoding/Parquet #10655 (Parquet) now also sits on feat/arrow-support, next to this PR, with Add ArrowParquet to effect/arrow #10666 (Parquet files as Arrow record batches) on top. DataFrame does not depend on either. The stack section above has the graph.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants