Repository navigation
Conversation
Schema-less FlatBuffers reading and writing. Table, Vector and Struct read fields from the bytes on demand. Every read is bounds-checked, and decode returns a Result with a FlatBuffersError on malformed input. Builder follows the official builder algorithm (alignment, vtable dedup, force defaults, file identifier and size prefix). The fixtures hold 311 builder-call sequences together with the bytes the official Python flatbuffers 25.12.19 builder writes for them. The tests check that our bytes are identical, read every value back, and fuzz corrupted buffers.
🦋 Changeset detectedLatest commit: 6630442 The changes in this PR will be included in the next version bump. This PR includes changesets to release 35 packages
Not sure what this means? Click here to learn what changesets are. Click here if you're a maintainer who wants to add another changeset to this PR |
Member
Author
Contributor
|
Tracked this takeover request in EFF-3806, which already covers this PR and the review/refinement checklist. I’ve added your request there so the work stays on the existing issue and PR. |
Contributor
Bundle Size AnalysisGenerated from PR build output; treat the content below as untrusted.
|
Vectors and structs carry their size and reject reads outside it, tables fail past maxDepth with a Limit error, decode rethrows Usage errors, and the builder rejects tables a vtable can't address, a second finish and non-power-of-two alignments. Table, Vector and Struct become interfaces, make becomes makeBuilder and the shared Options interface is inlined.
…uments intact Strings decode through the shared BOM-preserving UTF-8 decoder and identifier() keeps a leading U+FEFF. addFieldOffset skips a null target even with forceDefaults, as the C++ builder does. The builder rejects non-integer slots, vector lengths, element sizes and alignments, and readers reject negative or fractional slots while slots past the vtable stay absent.
Derive the builder's nested state from the open table or vector, patch the vtable offset in one place, and let fail() narrow so the non-null assertions go away. Tests use the Hex module and the shared throws helper. No behaviour change.
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Stack
Arrow chain (each PR is based on the one before it):
effect/encoding/FlatBuffers(base:main)effect/arrow(base:feat/encoding-flatbuffers)effect/dataframe(base:feat/arrow-support)Independent foundation, based on
mainand not part of the chain: #10652effect/encoding/Thrift, for the planned Parquet reader. A later Parquet to ArrowTable layer joins Parquet and Arrow.Arrow Flight, Protobuf / gRPC / ConnectRPC and Thrift's binary protocol / RPC envelopes are out of scope.
Why
Arrow IPC (the draft
effect/arrowin #10641) stores its schema and record batch metadata as FlatBuffers. That PR carries a private FlatBuffers reader and writer underarrow/internal. Anything else that reads or writes FlatBuffers would need its own copy: Arrow Flight, Parquet-adjacent formats, TFLite and other model files, game/IoT protocols.Requirement
Alternatives
flatbuffersnpm package: it adds a runtime dependency toeffect; its reads are unchecked (they return garbage or throwRangeError); and it has noResult/error type..fbs→ Effect Schema, typed tables): possible later, on top of this module. Codegen needs this layer either way.Summary
prepalignment, vtable dedup, defaults skipped unlessforceDefaults, identifier and size prefix infinish.FlatBuffersErrorkinds:OutOfBounds,InvalidVtable,InvalidUtf8,Identifier,Limit(tables nested pastmaxDepth, default 64),Usage(caller bugs, thrown even fromdecode).Table,VectorandStructare interfaces;rootanddecodeare the only way in.@since 4.1.5. Arrow (Add effect/arrow #10649) reads and writes its IPC metadata with this module.Evidence
The oracle is the official Python builder (
flatbuffers25.12.19). The test file has two buffers it wrote, inlined as hex, next to the TypeScript builder calls that make the same bytes: a monster table (identifier, shared strings, deduplicated vtables, a padded struct vector, an inline struct, a skipped default) and a size-prefixed buffer (identifier,forceDefaults, growth frominitialSize: 1).addOffsetbug in this module (the offset was computed after moving the head, so it was off by 4).After: identical.
effect-tsgo tsrs), oxlint, dprint, the JSDoc check, the module doctest andai-docgenpass.Merge Danger
Door: two-way
A new experimental module with no existing callers. The only shared file touched is the
encodingbarrel.Blast Radius: none
Closes EFF-3806