Repository navigation
Add effect/dataframe - #10641
Add effect/dataframe#10641fubhy wants to merge 90 commits into
Conversation
🦋 Changeset detectedLatest commit: eddf419 The changes in this PR will be included in the next version bump. This PR includes changesets to release 35 packages
Not sure what this means? Click here to learn what changesets are. Click here if you're a maintainer who wants to add another changeset to this PR |
|
@effect-bot just doodling (heavy slop) here for some side quests but might be worth evaluating nonetheless |
|
Added your request to the existing evaluation task (EFF-3804), with the exploratory framing noted. The review will assess the Arrow/dataframe design and suitability for Effect, and report findings back here without pushing changes to the PR. |
effect-bot
left a comment
There was a problem hiding this comment.
Verdict: REQUEST_CHANGES on d4eeba3.
I would keep exploring the idea, but I would not merge this version into Effect. Arrow-backed ingestion and typed queries fit Effect's streaming and SQL use cases. The public frame/expression/plan split is understandable, and an independent Polars oracle is a useful foundation. The blockers are concrete: valid queries silently change results under optimization or rebatching, Arrow round-trips can change values, and several paths return values that contradict the advertised TypeScript row type.
The design needs one source of truth for physical types, runtime value types, nullability, and whether an expression depends on other rows. Today that information is independently approximated in schema conformance, the optimizer, the streaming classifier, SQL inference, and driver decoding. Reuse the existing expression classification consistently and carry output type metadata to the engine boundary rather than adding more casts or operation-specific exceptions.
Given the exploratory framing, I recommend first accepting a smaller, independently interoperable Arrow boundary and a sound typed in-memory frame/query contract. Defer the SQL/streaming expansion until the same typed operations produce the same result across full frames, arbitrary batches, and driver representations, except for explicitly documented backend differences. A handwritten FlatBuffers/Arrow implementation can be justified, but its layout/normalization invariants and reference interoperability coverage need to be explicit before core ownership is reasonable.
Validation on this head: nix develop -c pnpm install and nix develop -c pnpm check passed; the five dataframe runtime files plus the PGlite integration file passed (612 tests); pnpm test-types DataFrame.tst.ts passed on both configured targets (40 executed tests); targeted Arrow/dataframe doctests passed (21 examples). JSDoc checking passed. Root pnpm lint failed dprint formatting on arrow/internal/ipc.ts and the three dataframe gold JSON files; oxlint completed before that failure. Focused temporary public-API probes reproduced the findings. No tracked source/test changes or pushes were made. Passing oracle tests currently miss physical-layout boundaries and valid stateful-expression compositions; SQL comparisons also normalize away some runtime type differences. Keep regression coverage at these public boundaries rather than multiplying internal-kernel tests.
The inline comments detail the specific defects. Experimental status is appropriate for API evolution, but it does not make silent data corruption or unsound row types acceptable.
|
@effect-bot Apply your suggestions. Take over this pull request but split it up into Apache Arrow & DataFrame separately as stacked PRs. Do not add Arrow Flight yet since that is blocked by adding broader Protobuf / gRPC / ConnectRPC support later (and also separately). |
|
Added this takeover request to EFF-3804: apply the review suggestions and split the work into stacked PRs, with Apache Arrow first and DataFrame on top. Arrow Flight is explicitly out of scope until the separate Protobuf / gRPC / ConnectRPC work is in place. |
Typed columnar data frames and lazy queries on top of effect/arrow, with in-memory, streaming and SQL engines. Split out of #10641.
d4eeba3 to
096cbe4
Compare
Typed columnar data frames and lazy queries on top of effect/arrow, with in-memory, streaming and SQL engines. Split out of #10641.
096cbe4 to
62266a8
Compare
Apache Arrow schemas, columns, record batches and tables with IPC stream and file codecs. Split out of #10641.
3760249 to
6e6a98a
Compare
Typed columnar data frames and lazy queries on top of effect/arrow, with in-memory, streaming and SQL engines. Split out of #10641.
62266a8 to
0a6efc3
Compare
Typed columnar data frames and lazy queries on top of effect/arrow, with in-memory, streaming and SQL engines. Split out of #10641.
0a6efc3 to
d1c6e67
Compare
Apache Arrow schemas, columns, record batches and tables with IPC stream and file codecs. Split out of #10641.
d175dc8 to
18d2b91
Compare
Typed columnar data frames and lazy queries on top of effect/arrow, with in-memory, streaming and SQL engines. Split out of #10641.
509d3f0 to
d7121cc
Compare
…s its columns can't hold exactly
…chemas a frame can't do that for
…nions are checked as Schema.is checks them
…icter builders and checks
691ab52 to
8968ad5
Compare
|
Two stack changes that touch this PR only indirectly:
|
A row is rebuilt for a check when a field is optional, and assigning __proto__ set the prototype instead of the field, so the check saw an object without it.
…the data frame internals
…an't have before values are converted
…ect decimal widths, cast targets and SQL scales that can't be held
…erals a valid checked type and convert floats at scales past 308
… units and compare typed candidates with eq
… list of their own on SQL and compare UInt64 candidates unsigned
Stack
Each layer depends only on the ones below it. #10655 joined the stack so that #10666 can read Parquet files as Arrow record batches. It carries #10652 (
effect/encoding/Thrift) by merge until that lands onmain. #10657 supplies optional codecs through thecodecsoption of the Parquet reader. Arrow Flight and Protobuf / gRPC / ConnectRPC are out of scope.Summary
Experimental
effect/dataframe: typed columnar frames on top ofeffect/arrow, plus lazyQueryplans that run in memory, batch by batch over streamed tables, or as SQL througheffect/sql. Query operations cover filters, projections, sorts, groups and aggregates, joins, windows and reshaping. Semantics follow Polars 2.1, and column names and types are checked at compile time. The SQL engine targets PostgreSQL (and PGlite) only; other dialects are future work.One type model
The in-memory engine decides result types for every engine.
BigIntneeds a 64-bit integer,BigDecimalany decimal). Literal, enum and refined fields are also checked withSchema.ison their values.fromRows,empty,validateandQuery.tablereturnNormalize<T>rows: optional keys become required andundefinedbecomesnull, recursively.Compiled.schema, casts every output column to text and decodes against that schema. Results have the same types as the memory engine whatever the driver returns, empty results included. The PGlite suite compares types strictly now.One row rule
E.isRowWise(no window functions) is the single expression-level rule.Rounding
roundwas checked per operand type against PostgreSQL 18 (PGlite) and Polars 2.1. PostgreSQL'sROUND(numeric)rounds half away from zero andROUND(float8)half to even.ROUND(float8, int)doesn't exist. The engines round half to even everywhere:x * 10^din float64 (ROUND(float8)), so2.675gives2.68and1.005gives1at two places, as in Polars.dplaces intoDecimal(min(p+1, 38), s), as in Polars. The memory engine used to ignoreroundon decimals. The SQL engine uses a FLOOR-based CASE rather thanROUND(numeric).decimalsfail with aDataFrameError.Other review fixes
Memory and streaming:
headno longer pulls an extra batch andhead(0)pulls nothing; zero-row batches aggregate like an empty table; each named table is read once per run,Query.streamincluded;explaindoesn't read streamed tables; negative half-even decimal casts; UInt32floorDivide/modby zero give null; decimal sums and running sums fail on 64-bit overflow instead of wrapping; list children of sliced Arrow input are trimmed;slicecomputes its end from the unclamped start (Polars:slice(-5, 1)on 3 rows is empty).SQL: boolean
sumandcumSum; temporal, duration and decimalVALUESliterals keep sub-millisecond precision; an empty partition list emitsOVER ();countDistincton empty input is 0; hidden helper columns use one prefix per compile that no plan column starts with;sqrtof negatives is NaN like the memory engine; errors carry the driver's message.Rejected: the finding that decimal add should respect the declared precision. Polars types add, multiply and sum as
Decimal(38, s), so the real limit is 64-bit storage, and that limit is enforced.Second review round
Fixed, each with a regression test that fails before the fix:
cumSum(x)then filtering keeps Polars' order. The same operator-level rule decides what streams per batch. Gold casewindow/filter-after-sort-by-window.abs/negateof the minimum and rolling sums fail instead of wrapping. Out-of-range casts give null (f641e308todecimal(18,18), UInt64 max, negative decimals to UInt64). Decimal rolling means are no longer in unscaled units.DateTimeUtcfields accept only UTC timestamps within theDateTimerange;UTC/Etc/UTC/+00:00zones read asDateTime.Utc.DateTimeZonedis documented as unsupported.min/max(plain, windowed, cumulative) useBOOL_AND/BOOL_OR;unique([])keeps the first row.Sql.Dialectis gone,Sql.compile(plan)takes no options, andSql.engine/Sql.layerare values. The DuckDB lowering had no driver or tests here, so it isn't shipped.Third review round
The invariant behind these fixes: a table column's declared physical type is fixed once, from the Schema plus
options.types, and every constructor, validator, stream batch and SQL read is converted to it exactly, through the same code.Plan.TableColumncarries decimal precision/scale and temporal units. ABigDecimalfield needsoptions.typesinfromRows,validateandQuery.table(breaking forQuery.tableandvalidatewithout it), so empty streams now give typed results:sumover an empty decimal stream is aBigDecimal, not0.Query.table,DataFrame.validateandDataFrame.fromRowsfail for anything else with aDataFrameErrorthat names the column, and so does reading the table of a plan that carries one.options.typesmust be 128-bit, like a decimal field of an Arrow schema. Overrides of 32, 64 or 256 bits used to be declared as 128-bit decimals; they now fail with aDataFrameErrorthat names the column.DataFrameErrorfrom a declaration, a common type or arithmetic). Casts to and from a negative scale used to die with aRangeError, as did the int32 extremes.Expr.castDecimalfails for a precision outside 1 to 38 or a scale that isn't a 32-bit integer, a product fails when the sum of its scales leaves 32 bits, and a cast to text fails when the scale is more than a string can hold. The SQL compiler takes scales from -1000 to 1000, the range of PostgreSQL'snumeric, and fails for others.when,coalesce, join keys andQuery.concatno longer round the integer to the decimal's scale. Only an explicit cast rounds.BigDecimalliteral has a precision of at least its scale and at most 38, so a literal with a scale above 38 is a valid 128-bit decimal. A literal of a plan with a precision outside 1 to 38, a scale outside 32 bits or more digits than its precision fails with aDataFrameErrorwhen it is evaluated.meanandrollingMeankeep finite values at scales above 308, where10^scaleis infinite as a float.Expr.isIncompares a decimal by value: 7.00 is one of7,7nand aBigDecimalequal to it at any scale, and no longer one of700, the unscaled units it is stored as.BigDecimal,DateTimeandDurationcandidates are compared witheq, so they match, run on SQL and survive the JSON form of a plan. A candidate with a fraction no longer kills a query over 64-bit integers. Typed candidates are joined in halves, so a list of thousands stays a shallow plan. On SQL, candidates with a fraction go in a list of their own against the column as a float, so one of them doesn't make PostgreSQL round a decimal or a 64-bit integer and its other candidates. A UInt64 above the signed range is one of its value. Candidates of the value's kind are supported and follow each engine's comparison rules, aseqdoes: a decimal next to a number that is not an integer, and a zoned timestamp next to aDateTimeoutside a UTC session, can differ on a database. Candidates of another kind than the value (a string for an integer) are documented as not portable.2.43(it was13.5), and1.2/1.20group together.fromRowsandemptysharevalidate's type check, so aDateTimeUtcfield with a UTC override reads asDateTime.Utcand other zones fail.suspend, structs and arrays; recursive schemas fail with aDataFrameErrorinstead of overflowing the stack.NOT MATERIALIZEDCTE; PostgreSQL never inlines one that calls a volatile function, so anextvalview is evaluated once for both, and stable tables are inlined at no measurable cost (MATERIALIZEDcost 1.3-1.9x). Column types are checked first against the catalog with the memory engine's rules.Nullcolumns,DateTimerange, UTC-only, exact unit and decimal fit.timestamptzreads as UTC whatever the session zone; intervals with months are rejected.Fourth review round
K.coerce, and fail when the common type can't hold a value. This covers concat, join keys, unpivot, streamed key and partial merges, and operand promotion.Expr.caststill gives null. An integer literal the column type can't hold widens the type, as in Polars (Int8 + 300is Int16).MATERIALIZEDCTE that both the checks and the result read, so the returned rows are exactly the checked rows for every relation (TABLESAMPLEviews included). The cost is 1.2-2x on PGlite, and filters can't use indexes. TwoQuery.tablenodes with the same name share one read.Float32round trips, the 64-bit decimal bound and finite dates. The checks run before projection,countor filters can drop the column.char(n),varchar,smallintandrealbehave as in memory.Fifth review round
coalesce/fillNull,whenand binary operations.Sixth review round
when,coalesceorfillNullof integer literals spans the range of all its literals, whichever rows select them. Other expressions built from literals count as Int32, and a cast always fixes the type.head(0), take their columns from the declarations without reading the source. A result whose columns depend on an undeclared table of which no batch was read fails, asking for the table's columns, instead of inventing a frame without columns.DataFrame.fromArrowTableof a table without batches keeps its fields' nullability.__, or__1_,__2_, … when a user name starts with it), so PostgreSQL never truncates them. Two columns of one relation that share their first 63 bytes are rejected when the query compiles.Seventh review round
head(concat([t, t]), 0), read nothing, and plans that need rows still read the table once and give every use the same rows.Eighth review round
__proto__as an own property. Before, the assignment set the prototype, so a refinement saw a row without the field throughfromRows,validateandQuery.table.DF-C032has accepted and rejected rows for a top-level and a nested struct.ArrowIpc.Status
Ready for re-review. Known limits: the common type of more than two batch types depends on their order, as in Polars, since the supertype isn't associative; memory fails on decimal mixes beyond 64 bits that PostgreSQL's numeric accepts; decimals are stored in 64 bits (failures, never wrapping); timestamps and durations on PostgreSQL keep microsecond precision; casting a float to a decimal goes through float64; Time table columns and nested table columns aren't supported on SQL; a decimal mean is computed in float64; boolean
rollingMin/rollingMaxaren't typed for booleans.Tests
test/dataframe/DataFrame.test.tsandQuery.test.ts. The Polars gold corpus (regenerated withgenerate_gold.pyand Polars 2.1.0) runs three ways: inputs inline, as catalog frames, and as streams of two-row batches.typetest/DataFrame.tst.tsandpackages/sql/pglite/test/DataFrameSql.test.ts.Validation at
eddf419102:pnpm check,pnpm lint,pnpm jsdocs --check,pnpm doctest --run packages/effect/src/dataframe packages/effect/src/arrow(27 in 12 files),pnpm vitest --run test/dataframe test/arrow test/encoding(1,233 in 23 files),pnpm test-types DataFrame(50 tests, 134 assertions), PGlite suite (84).Closes EFF-3804