Skip to content

[BugFix][Transform] Skip pipelining when a copy reads a loop-carried buffer - #3728

Open
losewayy wants to merge 2 commits into
tile-ai:mainfrom
losewayy:fix/3301-f077-loop-carried-copy-index
Open

losewayy wants to merge 2 commits into
tile-ai:mainfrom
losewayy:fix/3301-f077-loop-carried-copy-index

Conversation

@losewayy

Copy link
Copy Markdown

Problem

Part of #3301 (F077). A pipelined loop that stores a copy's source index into shared memory for the next iteration silently returns wrong data:

index_shared[0] = 0
for k in T.Pipelined(4, num_stages=1):
    offset = index_shared[0]
    T.copy(A[offset : offset + 64], a_shared)
    for i in T.Parallel(64):
        B[k * 64 + i] = a_shared[i]
    index_shared[0] = (k + 1) * 64

On 0176ba14, every iteration copies A[0:64], so 192 of 256 elements are wrong with num_stages=1 or 2. With 4-element tiles, which are too small for TMA and take the software pipeline, 12 of 16 are wrong. The serial loop is correct.

Approach

Both pipeline schedulers issue the next iteration's copy before this iteration's store.

  • Warp specialization: the producer evaluates offset before its back-pressure wait, while the store stays in the consumer after its release, and no barrier orders the two. After the split decides which prefix statements move to the producer, the loop now takes the existing sequential fallback if a producer-side read can observe a write that is not an earlier producer-side statement. Accesses are compared by backing data, and tile ops report their operands through GetAccessRegions(). A store that the split already moves ahead of the read, as in test_producer_bind_kernel, and writes to a copy's own destination keep warp specialization.
  • Software pipeline: PipelinePlanning falls back to the sequential schedule when a copy, or a binding that feeds its operands, reads a buffer that a later statement writes. For explicit order/stage annotations, the same check rejects a schedule that runs the store in a different stage from its reader, or before it, and accepts one that keeps the store after the copy in the same stage.

#2822 made the planner follow store -> binding -> copy chains within one iteration and left the warp-specialized scheduler out of scope; neither path checked the loop back-edge.

Falling back is the smallest correct change: with num_stages >= 2 the producer can run more than one iteration ahead, so moving a barrier cannot order the store before the next copy. The check works at buffer granularity, so a loop whose consumer writes a disjoint region of the copied global tensor also falls back.

Tested on RTX 5070 Ti (sm_120), CUDA 13.3:

  • The new test fails on 0176ba14 with wrong values, or without a diagnostic for the unsafe explicit schedule, and passes with this change. It covers the TMA path at 1 and 2 stages, a T.fill update, a non-TMA copy, and both explicit schedules.
  • test_producer_bind_kernel still emits tl::tma_load, and a loop that updates the copied tile in place keeps TMA and warp specialization.
  • The pipeline planning, software pipeline injection and transitive copy dependency tests pass, and the existing issue tests that use T.Pipelined give the same results as 0176ba14. In test_tilelang_transform_producer_consumer_ws.py, the two wide shared-memory cases fail on both revisions because this GPU cannot allocate 128/144 KB of dynamic shared memory.

…buffer

A pipelined loop can stage a copy's source index through a shared buffer that the loop body updates for the next iteration (`offset = index_shared[0]; T.copy(A[offset : offset + 64], a_shared); ...; index_shared[0] = (k + 1) * 64`). Both pipeline schedulers issue the next iteration's copy before that store, so the copy reads the stale index and the kernel silently returns wrong data (tile-ai#3301, F077).

Warp specialization: the producer evaluates the index before its back-pressure wait while the store stays in the consumer, and no barrier orders the two. Once the split is planned, fall back to the sequential loop if a producer-side read can observe a write other than an earlier producer-side statement. Accesses are compared by backing data and tile-op access regions, so the prefix stores the split already moves ahead of the read and writes to a copy's own destination keep warp specialization.

Software pipeline: PipelinePlanning falls back to the sequential schedule when a copy, or a binding that feeds it, reads a buffer that a later statement writes. An explicit order/stage schedule that runs such a store in a different stage from its reader, or before it, is rejected with a diagnostic.
@github-actions

Copy link
Copy Markdown

👋 Hi! Thank you for contributing to the TileLang project.

Please remember to run pre-commit run --all-files in the root directory of the project to ensure your changes are properly linted and formatted. This will help ensure your contribution passes the format check.

We appreciate you taking this step! Our team will review your contribution, and we look forward to your awesome work! 🚀

@coderabbitai

coderabbitai Bot commented Oct 31, 2026 •

Copy link
Copy Markdown
Contributor

Explain Change →

No actionable comments were generated in the recent review.

Walkthrough

The changes add buffer access ordering checks to warp specialization and pipeline planning. Pipeline planning detects loop-carried reads through copy operands and handles hazards in automatic and explicit schedules. CUDA regression tests cover indexed copies and stage arrangements.

Changes: Buffer read ordering

Layer / File(s) Summary
Warp-specialization ordering guard
src/cuda/transform/producer_consumer_ws.cc
A buffer access collector records reads and writes. BuildWSBlock returns the original block when a write violates producer-side read ordering.
Pipeline hazard checks and regression coverage
src/transform/pipeline_planning.cc, testing/python/issue/test_tilelang_issue_3301_f077.py
Pipeline planning detects loop-carried copy-read hazards. Automatic planning falls back to a sequential loop, while an explicitly conflicting schedule raises an error. Regression tests check indexed copies and stage arrangements.

Estimated code review effort: 3 (Moderate) | ~25 minutes

Change: Bug fix · Severity of issue fixed: Medium

Suggested labels: bug, UX

Merge Risk: Minimal · up to b91cf

This change makes pipelined loops that copy from a loop-carried buffer fall back to sequential execution, or reject unsafe explicit schedules, instead of producing wrong values. The supplied evidence shows no concrete remaining merge-blocking risk, and the regression test covers the reported failure cases.

Pre-merge checks: Passed 4

Check name Status Explanation
Title check Passed The title names the main change: a pipelining fix that skips pipelining when a copy reads a buffer updated across loop iterations. The changeset matches this: warp-specialized and software pipeline pa…
Description check Passed The description is closely related to the changeset. It explains the bug (a pipelined copy reads a loop-carried shared-memory index), the cause (both schedulers issue the next copy before the store), …
Linked Issues check Passed Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check Passed Check skipped because no linked issues were found for this pull request.
Full details: Title check

Explanation

The title names the main change: a pipelining fix that skips pipelining when a copy reads a buffer updated across loop iterations. The changeset matches this: warp-specialized and software pipeline paths fall back to sequential execution on loop-carried reads, and a regression test is added. The title is a single sentence and specific, though its length (75 characters) is at the upper end of concise.

Full details: Description check

Explanation

The description is closely related to the changeset. It explains the bug (a pipelined copy reads a loop-carried shared-memory index), the cause (both schedulers issue the next copy before the store), the fix in the warp-specialization and software-pipeline paths, the explicit-schedule diagnostics, and the regression test and validation performed.

Finishing Touches

Generate unit tests (beta)

  • Create a new PR
  • Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

Share

Recent review info

Run configuration

Commits

Reviewing files that changed from the base of the PR and between 0340b44 and b91cf3a.

Files selected for processing: 2

  • src/cuda/transform/producer_consumer_ws.cc
  • testing/python/issue/test_tilelang_issue_3301_f077.py
Files skipped from review as they are similar to previous changes: 2

  • src/cuda/transform/producer_consumer_ws.cc
  • testing/python/issue/test_tilelang_issue_3301_f077.py

Included review availability: This review used your included allowance. Your plan provides up to 8 included reviews per hour; 7 remain after this review.

@chatgpt-codex-connector

chatgpt-codex-connector Bot commented Oct 31, 2026 •

Copy link
Copy Markdown

Codex Review Summary

This comment shows the latest Codex review activity on this pull request.

Review Status Commit Review trigger
📝 Code Review ⚠️ Failed 2026-10-31T18:53:40.161137Z
We couldn’t complete this review. Comment @codex review to try again.
0340b44 PR opened
ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review" or "@codex security review".

Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

bug Something isn't working UX Improvements to user experience, usability, or design flow.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant