Skip to content

[fix][ml] Confirm retried cursor metadata writes in ZooKeeper - #27652

Open
poorbarcode wants to merge 4 commits into
apache:masterfrom
poorbarcode:codex/confirm-cursor-metadata-writes
Open

poorbarcode wants to merge 4 commits into
apache:masterfrom
poorbarcode:codex/confirm-cursor-metadata-writes

Conversation

@poorbarcode

@poorbarcode poorbarcode commented Nov 17, 2026 •

Copy link
Copy Markdown
Contributor

Background

Following a ZooKeeper connectivity interruption, a broker repeatedly failed to open a durable subscription with BadVersion. The cursor znode already existed. Unloading the topic let the broker reload the persisted cursor, and the subscription recovered.

Motivation

ZooKeeper can commit a cursor write before the client loses the response. Replaying a create then returns NODEEXISTS, which Pulsar maps to BadVersion. Replaying an update with the original expected version also returns BadVersion. The broker needs to recognize its own committed write while continuing to report conflicts when another writer wins.

Modifications

  • Add an optional publication ID to ManagedCursorInfo. Assign a UUID on the first attempt and preserve it across retries of the same write. A new cursor property update clears the ID inherited from the previous record.
  • Let writers opt in through the existing put API with Option.VerifyPublication. Its documentation requires a unique publication ID in the serialized value and preservation of the ID, serialized bytes, and expected version across retries of the same write. Other metadata record types can use the same verification path.
  • Verify opted-in conditional writes to fixed, persistent ZooKeeper paths after a version conflict or an ambiguous write response. This includes individual creates and updates issued after another operation in a batch fails. Confirm success only if the stored version is the expected version plus one and the bytes, including the publication ID, match. A mismatch preserves the original write error.
  • Use an asynchronous single-node getData so verification also works for valid records larger than the server’s multiRead response limit. Return the observed Stat, including its session ownership information.
  • Keep the original write pending during transient verification read failures. Retry the read with exponential backoff and jitter so another open of the same cursor can join the pending create. Permanent read failures and store closure complete the pending operation exceptionally.
  • Document how to chain dependent operations when a retry or verification read delays write completion.
  • Add regression coverage for a committed write whose response is lost, caller retries, competing publications, cursor serialization with compression, and verification after transport or session changes.

Local verification passed the repository-wide sanityCheck and all 47 tests in ZKMetadataStoreTest, MetaStoreImplTest, ManagedCursorInfoMetadataTest, and ManagedCursorPropertiesTest.

Does this pull request potentially affect one of the following parts:

  • Dependencies (add or upgrade a dependency)
  • The public API
  • The schema
  • The default values of configurations
  • The threading model
  • The binary protocol
  • The REST endpoints
  • The admin CLI options
  • The metrics
  • Anything that affects deployment

The metadata-store API gains the optional VerifyPublication write option. The persisted managed cursor protobuf gains an optional string field. Brokers can still read older cursor records that omit it.

Comment thread managed-ledger/build.gradle.kts Outdated
file.writeText(source.replace(declaration, "import org.apache.pulsar.metadata.api.HasPublicationId;\n\n" +
"${declaration}HasPublicationId, "))
}
}

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This task doesn't follow Gradle best practices. It doesn't declare inputs and outputs etc. and would break up-to-date checks.

Is it really justified to add this type of solution?

HasPublicationId.getPublication() isn't called in production code in this PR. Is something missing?

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Regarding the interface question, I've added setPublication(String) and documented the contract on getPublication():

Confirming a write after a bad-version response through this getter would require deserializing the stored record. Since the publication ID is included in the serialized bytes, the metadata store can compare those bytes directly, together with the expected next version, to avoid that CPU overhead. Verification therefore does not need to call this getter.

This getter and setPublication(String) define a readable and writable publication property for implementing records. Implementations must serialize this property as part of the record. Each logical write must use a unique publication ID and preserve it across retries of that write.

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

the comment isn't addressed regarding the gradle task. If the ManagedCursorInfo were to be modified, the postprocessing should be done in an additional doLast step directly in the lightproto task. However, based on the current state of this PR, I think that the HasPublicationId interface should be completely removed. I've made the point in the comment on the interface file.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Removed the custom doLast action and the GenerateLightProtoTask import. managed-ledger/build.gradle.kts is restored to its pre-PR contents, and ManagedCursorInfo uses the standard LightProto output. All 47 focused tests and the repository-wide sanityCheck passed.

Comment on lines +21 to +25
/** A metadata record that persists a unique ID for each logical write and reuses it on retries. */
public interface HasPublicationId {
/** Returns the identifier stored in the record, or null or empty for a record written without one. */
String getPublication();
}

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It looks like this interface is completely unnecessary since there isn't any production code that uses it

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I've added setPublication(String) and documented the contract on getPublication():

Confirming a write after a bad-version response through this getter would require deserializing the stored record. Since the publication ID is included in the serialized bytes, the metadata store can compare those bytes directly, together with the expected next version, to avoid that CPU overhead. Verification therefore does not need to call this getter.

This getter and setPublication(String) define a readable and writable publication property for implementing records. Implementations must serialize this property as part of the record. Each logical write must use a unique publication ID and preserve it across retries of that write.

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This review comment isn't resolved. There's no production code that uses this interface and therefore it's completely unnecessary. Instead of adding this interface, you can document the behavior of VerifyPublication option and the related semantics.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Removed HasPublicationId. The VerifyPublication Javadoc now documents the unique ID requirement, preserving the ID, serialized bytes and expected version across retries, and why comparing bytes avoids deserialization. Other record types can use the same verification path through the existing put option.

@poorbarcode
poorbarcode requested a review from lhotari November 18, 2026 02:51

@lhotari lhotari left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Please address the comment about removing the HasPublicationId.

@poorbarcode
poorbarcode requested a review from lhotari November 18, 2026 13:09
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants