Repository navigation
Conversation
5fba59d to
820929f
Compare
|
@bosd I'm not sure I understand why you depend on attribute_set here. You want to build a supercharged attribute_set module with JSONB, right? That sounds legit but should this be done inside the server-tools base_sparse_fields_jsonb module? Wouldn't it be better to leave base_sparse_fields_jsonb without attribute_set dependency and create another attribute_set_jsonb module in OCA/odoo-pim or make attribute_set depend on base_sparse_fields_jsonb eventually? Disclaimer I didn't read your code carefully yet. |
@rvalyi I was indeed contemplating about this. Placing it in a separeate module in the pim repo would keep this base implementation more basic, clean and flexible. But there are some more modules / improvements to get to a full working stack. Wanted to prevent, that we end up with a lot of glue modules and puzzle to get it up and running. I will later separate the latest commits and place it in their own module. This module as it sits here, could potentially be merged into base_sparse, but for now i think it is better to keep it separate. |
7d44c4a to
820929f
Compare
|
@bosd this starts to look pretty solid. kudos for the work! I would say what is missing for now are some tests inside this module. You can probably adapt some tests from the original module https://github.com/odoo/odoo/blob/19.0/addons/base_sparse_field/tests/test_sparse_fields.py |
dd4086a to
d9495a2
Compare
|
@rvalyi , @sebastienbeau , cc: @alexis-via It is ready for review! |
|
@sebastienbeau Can you please review? |
49b4032 to
021d66b
Compare
021d66b to
f41ab09
Compare
|
@Cedric-Pigeon @acsonefho @sbidoul Can you please review? 🙏 |
|
@bamidev Can you please review? |
| **What this module provides:** | ||
|
|
||
| - **JSONB Storage**: Serialized fields use PostgreSQL JSONB column type | ||
| - **GIN Indexes**: Automatic creation of GIN indexes for fast key/value lookups |
There was a problem hiding this comment.
I'm not sure if adding a gin index on each jsonb field (for these sparse fields) is the way to go.
I'd leave the field definition itself responsible for it, and only create the gin index if index=True.
But maybe it can be argued whether it is a good idea to have the index attribute on SerializedJsonb fields default to True.
There was a problem hiding this comment.
Agreed, makes more sense to let the field decide. I'll only create the GIN index when index=True. I'd keep the default False to stay in line with how Odoo handles index elsewhere, people who filter on it can opt in.
| _logger.info("base_sparse_field_jsonb: No GIN indexes on TEXT columns found.") | ||
|
|
||
|
|
||
| def post_init_hook(env): |
There was a problem hiding this comment.
Ok, altering the TEXT columns to JSONB during installation, also leaves the database in an unclean/broken state after you uninstall the module. Maybe an uninstall_hook that reverts them back to TEXT might be nice to have.
There was a problem hiding this comment.
Fair point. I'll add an uninstall_hook that casts the columns back to TEXT so base_sparse_field keeps working after removal.
|
Thanks a lot for the thorough review @bamidev, very useful. I'll push the fixes shortly and ping you again. |
- Patch the original Serialized class in place, so modules importing it directly from base_sparse_field (queue_job, server_environment) get a JSONB column too - Select columns from the registry serialized fields instead of names - Only drop indexes on the column being converted - Only create a GIN index when the field has index=True - Add an uninstall_hook converting the columns back to TEXT - Remove the hand written README.md
Recent Odoo 19 check_indexes replaces an index using another method than the expected one, turning the GIN index of indexed serialized fields into a btree one. Hide the index attribute of serialized fields while Odoo checks the indexes, and manage their GIN index afterwards.
rvalyi
left a comment
There was a problem hiding this comment.
Thanks @bosd.
I used GLM 5.3-flash in Hermes to review and I fully endorse the review here:
I re-reviewed the head commit (e007107) against the Odoo 19.0
framework source; the blocker I raised on f09dc07 is fixed, and the new tests
hit the real code path instead of the helper.
The mechanism is right. check_indexes builds expected from every stored
field that has a column_type, so a serialized field stays in the loop; by
setting field.index to False for the duration of the call, core falls into
the elif not index and tablename == existing.get(indexname, ...) branch and
only logs "Keep unexpected index" (odoo/orm/registry.py:910-911) instead of
dropping the GIN index and rebuilding it as btree. The index keeps Odoo's own
name, there is no duplicate btree, and update_gin_index recreates it after
core's pass. test_check_indexes_keeps_gin / test_check_indexes_creates_gin
exercise exactly that path (registry.check_indexes), which is the assertion
that was missing.
Remaining points, none of them blocking for me:
-
update_gin_indexshould be as forgiving as core.create_indexat the end
of the wrapper is called without thecr.savepoint()+
psycopg2.OperationalErrorprotection that core applies to its own index
creation (odoo/orm/registry.py:973-981), so any failure (lock timeout,
permission, a column that was not converted yet) aborts the whole module
install/upgrade where core would just warn and continue. -
The installation cost is still undocumented.
ALTER TABLE ... ALTER COLUMN TYPE jsonb(hooks.py:71) takes an ACCESS EXCLUSIVE lock and rewrites the
table, and every converted table is locked until the end of the install
transaction; the GIN creation at models/fields.py:128 takes its own lock as
well. On anattribute_set/queue_job-sized table this is a
maintenance-window operation, and a single TEXT row that is not valid JSON
aborts the install. A short paragraph in CONFIGURE (or USAGE, under
"Migration from TEXT") would save someone a bad surprise. -
The GIN index sits outside what the ORM believes about the field: as long as
the module is loaded,field.indexsays True while the index is GIN and
not known tocheck_indexes. That is the point of the design, but it means
this patch is coupled to the 19.0 implementation ofcheck_indexes.
A comment stating that (and that it must be re-checked when the series is
ported) would help the next maintainer. Also,model_namesis iterated
twice (themap()and then the call to the original), so
model_names = list(model_names)at the top is cheap insurance if a caller
ever passes an iterator. -
The uninstall caveat is worth one more sentence: until the process is
restarted,Serialized.column_typeis still("jsonb", "jsonb"), so a
module update in that window converts the columns back to JSONB right after
uninstall_hookput them back to TEXT. "Restart the server afterwards" is
there; saying why makes it actionable. And the round-trip is not byte
identical: the column default dropped byconvert_column_to_jsonbis not
restored, andjsonb::textreformats the JSON. Harmless for
base_sparse_field, which only doesjson.loads, but worth a line. -
Still there from the previous round, all minor:
SerializedJsonb = Serialized(models/fields.py:191) has no consumer outside this unreleased
PR — its only importers are the tests, which can useSerializedor
fields.Serialized(test_direct_import_is_patchedalready asserts the
identity);"installable": True/"auto_install": Falseare redundant
manifest keys; the DESCRIPTION table still claims "~30% larger" storage for
TEXT and "O(log n)" for key existence, neither of which is measured; and
USAGE's installation list should mention theRegistry.check_indexespatch
next to theSerializedone. -
Very minor:
index="btree_not_null"(or"trigram") on a serialized field
is accepted by core's assert, gets hidden by the wrapper, and comes out as a
plain, non-partial GIN index. Either document that onlyTrueis honoured
or handle it explicitly.
Nothing else: the in-place patching of Serialized is still the right call
(single class shared with odoo.fields.Serialized, so the queue_job /
server_environment case is covered), the column-specific index drop is properly
scoped through pg_depend, and the uninstall hook now leaves base_sparse_field
in a working state.
- create the GIN index under a savepoint, log instead of aborting the install/update when it fails, like Odoo does for its own indexes - document the table rewrite and locks taken when converting the columns, and that invalid JSON aborts the installation - document the coupling with the 19.0 Registry.check_indexes - warn when a serialized field uses an index value other than True - accept an iterator for model_names in check_indexes - explain the uninstall caveats - drop the SerializedJsonb alias and redundant manifest keys - remove unmeasured performance claims from the description
|
Thanks @rvalyi for the thorough re-review! All six points are addressed in 2b4c8e3:
@bamidev could you take another look? Your earlier points are addressed in f09dc07 and e007107. |
PostgreSQL refuses to change the type of a column read by a view or a
materialized view ("cannot alter type of a column used by a view or
rule"), which made the conversion of a serialized column fail and the
module install or update crash on databases having such a view (e.g. a
reporting materialized view reading the attributes).
Drop those views (and the views built on them) before the conversion and
recreate them from their definition, with their indexes, afterwards. A
view whose definition does not fit the new type is left dropped with a
warning for its module to recreate, as Odoo does when it converts a column.
…inition The view definition is literal SQL: a LIKE pattern or a modulo made odoo.tools.SQL fail with 'not enough arguments for format string'.
References
· PostgreSQL JSONB documentation: https://www.postgresql.org/docs/current/datatype-json.html
· GIN index documentation: https://www.postgresql.org/docs/current/gin.html
· OCA attribute_set: https://github.com/OCA/odoo-pim
· Related issue: OCA/odoo-pim#153