Repository navigation
Conversation
Two appends to one session ran concurrently: they raced over the session row and over the in-memory event list, and on SQLite one of them was aborted with a lock error. adk-python keeps one lock per session and holds it across the stale check, the commit and the update of the last update time. Hold a per-session lock across AppendEvent, keyed by app, user and session, and drop the lock once nobody holds or waits for it, like adk-python's reference-counted locks. Append to the in-memory session only after the commit, so that a failed append does not leave its event behind and the in-memory order matches the order the events were committed in. The in-memory session still receives the untrimmed event, so temp: keys stay readable for the rest of the invocation and do not reach storage. Lock the session row FOR UPDATE for the rest of the transaction, like adk-python, so an append from another process cannot write the row in between the read and the commit. Fixes google#1671
caohy1988
left a comment
There was a problem hiding this comment.
Same-session appends are now serialized, failed writes no longer change the in-memory event list, and temporary state stays readable after a successful commit. Two problems with the new locking should be fixed before merge.
AppendEvent now locks the session row before taking the app and user state locks, while Create still takes app and user state first and inserts the session last. The new comment notes that a Create for an existing session ID can block on the session-row lock; on MySQL that opposite order can deadlock, and the database may abort the valid append as the victim. I reproduced this against MySQL 8: a concurrent duplicate Create caused AppendEvent to fail with a deadlock error and left the event uncommitted. The same run succeeded when only service.go was restored to the base. Please give Create and AppendEvent a consistent lock order (for example app state, user state, then session), keep the stale-session check under the session-row lock, and add this concurrent duplicate-create case to the tests.
The new per-session mutex also waits in sync.Mutex.Lock without consulting the request context, and the holder keeps it for the whole transaction. A second AppendEvent whose context is already canceled, or whose deadline expires while it is queued, cannot return until the first request finishes. On the base implementation the canceled call reached the database layer and returned context.Canceled immediately. Please make lock acquisition context-aware (and drop the waiter's ref if it cancels), so a canceled append can leave the queue before the earlier transaction commits.
…ctx while queued Two problems from review of the per-session append lock: Create writes app state, then user state, then inserts the session row. AppendEvent took the session row first, so a concurrent duplicate Create held the state rows while waiting for the session row and the append held the session row while waiting for the state rows. MySQL aborts one side of that cycle, and the append was the victim, leaving its event uncommitted. Take the locks in Create's order: app, user, session. The stale check stays under the session-row lock. The per-session lock waited in sync.Mutex.Lock, which ignores the request context, so a canceled append stayed queued until the holder's transaction committed. Waiting now selects on ctx.Done() and returns ctx.Err(), and a canceled waiter drops its reference so an unused entry is still reclaimed. Tests: a recorded-statement test pins the row-lock order, a concurrent duplicate Create test checks the append still commits, and a canceled append test checks it returns context.Canceled while the holder is still inside its transaction. Signed-off-by: Hemang Varshney <218273197+hemv-857@users.noreply.github.com>
|
@caohy1988 — both problems are fixed and pushed in 1. Lock order. Tests added:
2. Context-aware acquisition. Waiting is a select on The local suite is SQLite-only, so the order test pins the sequence rather than reproducing the MySQL deadlock directly. |
|
Synced with |
golangci-lint v2.15.0 enables the modernize waitgroupgo analyzer, which flags the Add(1)/go func/defer Done pattern. Queue the goroutines with WaitGroup.Go instead; the synchronization behaviour is unchanged.
|
Lint follow-up: golangci-lint v2.15.0 enables the |
Summary
Fixes #1671
Two appends to one session ran concurrently in the database session
service. They raced over the session row and over the in-memory event
list, and on SQLite one of them was aborted with a lock error. This
brings the Go backend in line with adk-python, which keeps one lock per
session and holds it across the stale check, the commit and the update of
the last update time.
What changed
AppendEvent, keyed by app, user andsession. The lock is dropped once nobody holds or waits for it, so a
long-lived service does not accumulate one lock per session it has seen.
This matches adk-python's reference-counted per-session locks
(
database_session_service.py,_with_session_lock).added to the local session before it was written, so an append the
database rejected still showed up in
Events()and in the session state.The in-memory session still receives the untrimmed event, so
temp:keys stay readable for the rest of the invocation and do not reach
storage, the way the in-memory and the Vertex AI session services do.
FOR UPDATEfor the rest of thetransaction, like adk-python, so an append from another process cannot
write the row in between the read and the commit. The app and user state
rows were already locked this way; the session row was not.
SQLite has no row-level locking and ignores the clause, so the
FOR UPDATEis a no-op there; SQLite serializes writers anyway.What behaves differently on the current release
AppendEventcalls for one session no longer interleave.On SQLite the loser previously failed with
database table is locked: database is deadlocked (6); it now fails with the stale-session error.Appends that used to collide now all succeed when they share one handle.
in-memory session: neither in
Events()nor in the session state.in, rather than the order the calls were made in.
Getreturns them intimestamp order, which is a different order, as before.
the stale-session error, and that append's state key is still lost — the
same as adk-python. Only the error changes: a stale-session error
instead of a database lock error. On SQLite that was already loud; on
Postgres it was silent.
AppendEventandLastUpdateTime()overupdatedAtis unchanged; fix(session/database): read and write updatedAt under the session lock #1244 still tracks it.mainwrote the fieldtwice, once inside the transaction and once after it, and both writes
were unguarded; this change leaves one unguarded write, so the race
surface shrinks but the race remains. The per-session lock is what
keeps a concurrent append from seeing the window between the commit and
the write.
Delete,GetandListtake no lock, which is what adk-python doesas well (
delete_session,get_sessionandlist_sessionstake none).They were considered and left alone here.
For #1244
This change removes the in-transaction
updatedAtwrite that #1244converts to
setUpdatedAt. Whichever of the two lands second has to keepexactly one write to
updatedAt— asetUpdatedAtcall before thecommit, or the remaining post-commit write converted. If both are
removed,
updatedAtnever advances and every append after the first on ahandle fails as stale.
Testing
TestDatabaseService_AppendEvent_LocksTheSessionRow— the session rowis read
for update.TestDatabaseService_AppendEvent_FailedAppendLeavesNoEventInMemory— arejected append leaves no event and no state key behind.
TestDatabaseService_AppendEvent_KeepsTempStateReadableInMemory— atemp:key stays readable in the live session and does not reachstorage.
TestDatabaseService_AppendEvent_SerializesAppendsToOneSession— fourappends to one session never have two of them inside a transaction at
the same time, all four are stored, and every state key survives.
TestKeyedMutex_DropsTheEntryWhenItIsUnusedandTestKeyedMutex_SerializesOneKeyAtATime— the lock helper itself.TDD: with
session/database/service.goreverted tomain, three of thefour behaviour tests fail and the suite is red.
TestDatabaseService_AppendEvent_KeepsTempStateReadableInMemorypasses onmainon purpose: it guards a behaviourmainalready had and that anearlier version of this PR broke, and the mutant that turns it red is
trimming the event before the in-memory append.
Each guard was also reverted on its own: dropping the per-session lock,
moving the in-memory append back before the write, trimming the event
before the in-memory append, and dropping the
FOR UPDATEeach one turnsits own test red. The lock entry is dropped in a separate test, which
fails when the entry is never dropped.
go test -race -mod=readonly -count=1 -shuffle=on work— greengolangci-lint run(v2.3.1, the version CI pins) — 0 issuesgofumptandgoimports— clean