Skip to content

[ADD] base_sparse_field_jsonb - #3480

Open
bosd wants to merge 8 commits into
OCA:19.0from
bosd:19.0-add-base_sparse_field_jsonb
Open

bosd wants to merge 8 commits into
OCA:19.0from
bosd:19.0-add-base_sparse_field_jsonb

Conversation

@bosd

@bosd bosd commented Jan 2, 2026

Copy link
Copy Markdown

References
· PostgreSQL JSONB documentation: https://www.postgresql.org/docs/current/datatype-json.html
· GIN index documentation: https://www.postgresql.org/docs/current/gin.html
· OCA attribute_set: https://github.com/OCA/odoo-pim
· Related issue: OCA/odoo-pim#153

@rvalyi

rvalyi commented Jan 3, 2026

Copy link
Copy Markdown
Member

@bosd I'm not sure I understand why you depend on attribute_set here. You want to build a supercharged attribute_set module with JSONB, right? That sounds legit but should this be done inside the server-tools base_sparse_fields_jsonb module? Wouldn't it be better to leave base_sparse_fields_jsonb without attribute_set dependency and create another attribute_set_jsonb module in OCA/odoo-pim or make attribute_set depend on base_sparse_fields_jsonb eventually?

Disclaimer I didn't read your code carefully yet.

@bosd

bosd commented Jan 3, 2026

Copy link
Copy Markdown
Author

Wouldn't it be better to leave base_sparse_fields_jsonb without attribute_set dependency and create another attribute_set_jsonb module in OCA/odoo-pim or make attribute_set depend on base_sparse_fields_jsonb eventually?

@rvalyi I was indeed contemplating about this. Placing it in a separeate module in the pim repo would keep this base implementation more basic, clean and flexible. But there are some more modules / improvements to get to a full working stack. Wanted to prevent, that we end up with a lot of glue modules and puzzle to get it up and running.

I will later separate the latest commits and place it in their own module.

This module as it sits here, could potentially be merged into base_sparse, but for now i think it is better to keep it separate.

@rvalyi

rvalyi commented Jan 24, 2026

Copy link
Copy Markdown
Member

@bosd this starts to look pretty solid. kudos for the work! I would say what is missing for now are some tests inside this module. You can probably adapt some tests from the original module https://github.com/odoo/odoo/blob/19.0/addons/base_sparse_field/tests/test_sparse_fields.py
Tell us when it's ready for review. cc @sebastienbeau

@bosd
bosd force-pushed the 19.0-add-base_sparse_field_jsonb branch 2 times, most recently from dd4086a to d9495a2 Compare January 24, 2026 14:58
@bosd
bosd marked this pull request as ready for review January 24, 2026 15:02
@bosd

bosd commented Jan 24, 2026

Copy link
Copy Markdown
Author

@rvalyi , @sebastienbeau , cc: @alexis-via It is ready for review!

@bosd

bosd commented Aug 4, 2026

Copy link
Copy Markdown
Author

@sebastienbeau Can you please review?

@OCA-git-bot OCA-git-bot added series:19.0 mod:base_sparse_field_jsonb Module base_sparse_field_jsonb labels Aug 6, 2026
@bosd
bosd force-pushed the 19.0-add-base_sparse_field_jsonb branch from 49b4032 to 021d66b Compare August 6, 2026 06:45
@bosd
bosd force-pushed the 19.0-add-base_sparse_field_jsonb branch from 021d66b to f41ab09 Compare August 21, 2026 18:47
@bosd

bosd commented Sep 30, 2026

Copy link
Copy Markdown
Author

@Cedric-Pigeon @acsonefho @sbidoul Can you please review? 🙏

@bosd

bosd commented Oct 2, 2026

Copy link
Copy Markdown
Author

@bamidev Can you please review?

@bamidev bamidev left a comment •

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ok, I've left some review comments, but it would be nice to use jsonb for sparse fields rather than json manually (de)serialized into text columns indeed!

Comment thread base_sparse_field_jsonb/hooks.py Resolved
Comment thread base_sparse_field_jsonb/README.md Outdated
**What this module provides:**

- **JSONB Storage**: Serialized fields use PostgreSQL JSONB column type
- **GIN Indexes**: Automatic creation of GIN indexes for fast key/value lookups

@bamidev bamidev Oct 7, 2026 •

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm not sure if adding a gin index on each jsonb field (for these sparse fields) is the way to go.
I'd leave the field definition itself responsible for it, and only create the gin index if index=True.

But maybe it can be argued whether it is a good idea to have the index attribute on SerializedJsonb fields default to True.

Copy link
Copy Markdown
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Agreed, makes more sense to let the field decide. I'll only create the GIN index when index=True. I'd keep the default False to stay in line with how Odoo handles index elsewhere, people who filter on it can opt in.

_logger.info("base_sparse_field_jsonb: No GIN indexes on TEXT columns found.")


def post_init_hook(env):

@bamidev bamidev Oct 7, 2026 •

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ok, altering the TEXT columns to JSONB during installation, also leaves the database in an unclean/broken state after you uninstall the module. Maybe an uninstall_hook that reverts them back to TEXT might be nice to have.

Copy link
Copy Markdown
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fair point. I'll add an uninstall_hook that casts the columns back to TEXT so base_sparse_field keeps working after removal.

Comment thread base_sparse_field_jsonb/hooks.py Resolved
@bosd

bosd commented Oct 7, 2026

Copy link
Copy Markdown
Author

Thanks a lot for the thorough review @bamidev, very useful. I'll push the fixes shortly and ping you again.

bosd added 2 commits October 7, 2026 21:29
- Patch the original Serialized class in place, so modules importing it
  directly from base_sparse_field (queue_job, server_environment) get a
  JSONB column too
- Select columns from the registry serialized fields instead of names
- Only drop indexes on the column being converted
- Only create a GIN index when the field has index=True
- Add an uninstall_hook converting the columns back to TEXT
- Remove the hand written README.md
Recent Odoo 19 check_indexes replaces an index using another method than
the expected one, turning the GIN index of indexed serialized fields into a
btree one. Hide the index attribute of serialized fields while Odoo checks
the indexes, and manage their GIN index afterwards.

@rvalyi rvalyi left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks @bosd.
I used GLM 5.3-flash in Hermes to review and I fully endorse the review here:

I re-reviewed the head commit (e007107) against the Odoo 19.0
framework source; the blocker I raised on f09dc07 is fixed, and the new tests
hit the real code path instead of the helper.

The mechanism is right. check_indexes builds expected from every stored
field that has a column_type, so a serialized field stays in the loop; by
setting field.index to False for the duration of the call, core falls into
the elif not index and tablename == existing.get(indexname, ...) branch and
only logs "Keep unexpected index" (odoo/orm/registry.py:910-911) instead of
dropping the GIN index and rebuilding it as btree. The index keeps Odoo's own
name, there is no duplicate btree, and update_gin_index recreates it after
core's pass. test_check_indexes_keeps_gin / test_check_indexes_creates_gin
exercise exactly that path (registry.check_indexes), which is the assertion
that was missing.

Remaining points, none of them blocking for me:

  1. update_gin_index should be as forgiving as core. create_index at the end
    of the wrapper is called without the cr.savepoint() +
    psycopg2.OperationalError protection that core applies to its own index
    creation (odoo/orm/registry.py:973-981), so any failure (lock timeout,
    permission, a column that was not converted yet) aborts the whole module
    install/upgrade where core would just warn and continue.

  2. The installation cost is still undocumented. ALTER TABLE ... ALTER COLUMN TYPE jsonb (hooks.py:71) takes an ACCESS EXCLUSIVE lock and rewrites the
    table, and every converted table is locked until the end of the install
    transaction; the GIN creation at models/fields.py:128 takes its own lock as
    well. On an attribute_set / queue_job-sized table this is a
    maintenance-window operation, and a single TEXT row that is not valid JSON
    aborts the install. A short paragraph in CONFIGURE (or USAGE, under
    "Migration from TEXT") would save someone a bad surprise.

  3. The GIN index sits outside what the ORM believes about the field: as long as
    the module is loaded, field.index says True while the index is GIN and
    not known to check_indexes. That is the point of the design, but it means
    this patch is coupled to the 19.0 implementation of check_indexes.
    A comment stating that (and that it must be re-checked when the series is
    ported) would help the next maintainer. Also, model_names is iterated
    twice (the map() and then the call to the original), so
    model_names = list(model_names) at the top is cheap insurance if a caller
    ever passes an iterator.

  4. The uninstall caveat is worth one more sentence: until the process is
    restarted, Serialized.column_type is still ("jsonb", "jsonb"), so a
    module update in that window converts the columns back to JSONB right after
    uninstall_hook put them back to TEXT. "Restart the server afterwards" is
    there; saying why makes it actionable. And the round-trip is not byte
    identical: the column default dropped by convert_column_to_jsonb is not
    restored, and jsonb::text reformats the JSON. Harmless for
    base_sparse_field, which only does json.loads, but worth a line.

  5. Still there from the previous round, all minor: SerializedJsonb = Serialized (models/fields.py:191) has no consumer outside this unreleased
    PR — its only importers are the tests, which can use Serialized or
    fields.Serialized (test_direct_import_is_patched already asserts the
    identity); "installable": True / "auto_install": False are redundant
    manifest keys; the DESCRIPTION table still claims "~30% larger" storage for
    TEXT and "O(log n)" for key existence, neither of which is measured; and
    USAGE's installation list should mention the Registry.check_indexes patch
    next to the Serialized one.

  6. Very minor: index="btree_not_null" (or "trigram") on a serialized field
    is accepted by core's assert, gets hidden by the wrapper, and comes out as a
    plain, non-partial GIN index. Either document that only True is honoured
    or handle it explicitly.

Nothing else: the in-place patching of Serialized is still the right call
(single class shared with odoo.fields.Serialized, so the queue_job /
server_environment case is covered), the column-specific index drop is properly
scoped through pg_depend, and the uninstall hook now leaves base_sparse_field
in a working state.

- create the GIN index under a savepoint, log instead of aborting the
  install/update when it fails, like Odoo does for its own indexes
- document the table rewrite and locks taken when converting the columns,
  and that invalid JSON aborts the installation
- document the coupling with the 19.0 Registry.check_indexes
- warn when a serialized field uses an index value other than True
- accept an iterator for model_names in check_indexes
- explain the uninstall caveats
- drop the SerializedJsonb alias and redundant manifest keys
- remove unmeasured performance claims from the description
@bosd

bosd commented Nov 16, 2026

Copy link
Copy Markdown
Author

Thanks @rvalyi for the thorough re-review! All six points are addressed in 2b4c8e3:

  1. update_gin_index now creates the index under cr.savepoint() and logs on psycopg2.OperationalError, like core does. A test covers the failure path.
  2. USAGE ("Migration from TEXT") now documents the table rewrite under ACCESS EXCLUSIVE, that locks are held until the end of the install, the GIN lock, and that a single invalid JSON row aborts the install. It includes a query to check the data beforehand.
  3. A docstring in check_indexes now states that it relies on the 19.0 "Keep unexpected index" branch and must be re-checked when porting. model_names = list(model_names) is added, and a test passes an iterator.
  4. The uninstall section now explains why the restart is needed (the class is still patched, so an update in that window converts the columns back to JSONB). It also notes that the round trip is not byte identical (the dropped default is not restored, and jsonb::text reformats the JSON).
  5. Removed the SerializedJsonb alias (tests now use fields.Serialized) and the redundant manifest keys. Replaced the unmeasured performance claims in DESCRIPTION. USAGE now lists the Registry.check_indexes patch.
  6. Index values other than True now log a warning, and a plain GIN index is created. This is documented, and a test covers it.

@bamidev could you take another look? Your earlier points are addressed in f09dc07 and e007107.

bosd added 2 commits November 19, 2026 16:35
PostgreSQL refuses to change the type of a column read by a view or a
materialized view ("cannot alter type of a column used by a view or
rule"), which made the conversion of a serialized column fail and the
module install or update crash on databases having such a view (e.g. a
reporting materialized view reading the attributes).

Drop those views (and the views built on them) before the conversion and
recreate them from their definition, with their indexes, afterwards. A
view whose definition does not fit the new type is left dropped with a
warning for its module to recreate, as Odoo does when it converts a column.
…inition

The view definition is literal SQL: a LIKE pattern or a modulo made
odoo.tools.SQL fail with 'not enough arguments for format string'.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

mod:base_sparse_field_jsonb Module base_sparse_field_jsonb series:19.0

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants