Repository navigation
feat(61156998): date_trunc and T-SQL function translation for Superset (release 3.2.0) - #55
Merged
Merged
Conversation
date_trunc:
- build the DATEADD/DATEDIFF expression in one place, shared by the dialect
hook and the raw SQL rewrite, instead of two copies of the same formula
- drop the milliseconds/microseconds grains: DATEDIFF(millisecond, ...)
overflows a 32-bit int 24 days past any fixed epoch, so they always failed
- never rewrite inside string literals: a value like 'date_trunc(''day'', x)'
used to be silently corrupted
- never rewrite KQL: the translation now runs only on the T-SQL branch
- translate nested date_trunc calls instead of only the outermost one
- unknown grain stays untouched in raw SQL and raises CompileError when it
comes through func.date_trunc
ORDER BY:
- explain why the TOP fallback exists and mark the silent max_top_n cap
- document how to override max_top_n from create_engine / Superset
tests:
- cover literals, KQL passthrough, nesting, removed grains, CompileError
- check max_top_n through create_engine, the way callers actually set it
- integration tests for ORDER BY without LIMIT, ORDER BY inside a wrapped
query, and the resulting date for every supported grain
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The integration suite needed a real cluster and credentials, so it never ran in CI and both the ORDER BY and date_trunc fixes rested on unverified guesses. It now runs against the local Kusto emulator, which is the real query engine, and is wired into the QA workflow. Verified against the emulator, which also corrected one assumption: - a top-level ORDER BY needs no TOP, Kusto sorts it as it is; the previous fix added one anyway and silently capped such queries at max_top_n rows - a nested ORDER BY without TOP is rejected outright, not sorted incorrectly: "OTR0001: The ORDER BY clause is not valid in views, inline functions, derived tables, sub-queries, and common table expressions, unless TOP ... is specified" So the TOP fallback now applies to nested SELECTs only. - every date_trunc grain returns the expected timestamp, week is Monday-based and week_sun Sunday-based, and int epoch 0 is accepted as a DATEDIFF base To connect to the emulator at all: - take the scheme from the URL driver and stop discarding url.port, so kustosql+http://localhost:8080 works - skip authentication for plain HTTP, which the emulator does not support - add the kustosql.http dialect, which also lets a local Superset point at the emulator Test setup: - conftest targets the emulator by default, falls back to KUSTO_URL for a real cluster, ignores the .env.sample placeholders, and skips the suite when nothing is reachable instead of failing it - drop the duplicated get_kcsb from the KQL tests - make emulator / make emulator-stop, plus integration steps in the QA workflow Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Measured against a copy of the production Superset metadata (81 035 SQLLab queries and 1 784 virtual datasets on the Kusto SQL databases, 16 266 of the queries failed). The failures name what to fix, and every replacement below was executed against the Kusto emulator before being added. Generalise the date_trunc rewrite into a table of call translations: - datetrunc(...) — the T-SQL 2022 spelling, and a bare unquoted grain (31 failures) - date_part(unit, x) -> DATEPART(unit, x) (18) - startofday/week/month/year(x) — KQL habits with no T-SQL counterpart (27) - ifnull(a, b) -> COALESCE(a, b) (32) - DATE(x), to_date(x) -> truncation to the day (135): Kusto has one datetime type, so CAST/CONVERT to DATE keeps the time of day and cannot serve here Left alone on purpose, because no exact equivalent exists: FORMAT, DATE_FORMAT, TRY_CAST, JSON_VALUE, OPENJSON. Kusto reports those better than a guess would. Send a leading WITH as T-SQL, not KQL: - 207 production CTE queries failed, none succeeded; 87 of them never reached Kusto at all, because a query not starting with "select" was sent as KQL - Kusto does execute CTEs — verified on the emulator, including a CTE whose body needs the date_trunc rewrite - language detection now skips leading whitespace, semicolons and comments, and matches whole words: `withdrawals | count` is a KQL table, not a CTE Checked on the whole corpus: 335 of the 54 279 queries are rewritten, and not one string literal is altered or paren balance changed. 481 of the failed queries are now handled differently (320 rewritten, 175 no longer misrouted to KQL). Two defects the corpus and its tests caught in this change itself: - the language check was a regex with a repeated alternation, which backtracked catastrophically on queries with a long comment header; it is a linear scan now - the first scanner draft ran a regex per character, ~10x slower than needed Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
An apostrophe inside a comment ("-- don't ask") was read as the start of a
string literal, so the scanner swallowed the rest of the query and never
translated the real call after it. Comments are now skipped exactly like
literals, in both the scanner and the argument splitter, which also stops the
rewriter from editing text inside comments at all.
Found by replaying a copy of the production corpus: of 54 827 queries that
already succeeded, only 3 are changed by the rewriter, and all three are the
same query whose DATE_TRUNC('week', ...) Kusto never accepted anyway.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
…ules
Superset's filter search compiles ILIKE to lower(col) LIKE lower('%text%'),
and Kusto rejects it: a LIKE pattern must be a string literal (OTR0001), while
lower('%text%') is a call. Plain LIKE is not a fix either — on a column it
matches case-sensitively (both verified against the emulator).
So the cursor's rewriter now folds lower('LITERAL') into the lowered literal:
the pattern becomes a literal again and the match stays case-insensitive.
lower(column) and lower(<expression>) pass through untouched.
The rewrite moves after parameter substitution, so a pattern bound via
%(name)s folds the same way as one inlined by literal_binds.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
An unbounded `until curl` loop would hang the job until the GitHub timeout if the emulator ever failed to start. Five minutes is plenty: locally it answers in a few seconds. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
ruff is unpinned (>=0.9.3) and newer releases flag the long __init__ signatures that PLR0913 already excuses. The same three findings fail the lint job on main. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Egorrko
changed the title
skrepkaq
approved these changes
Sep 18, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Makes the T-SQL dialect usable from Superset charts and SQL Lab against Kusto. Running in Dodo production Superset for about a month (the Superset fork pins this branch).
Tickets: 61156998 (date_trunc), 68302686 (ILIKE).
date_trunc(61156998)func.date_trunc(grain, col)compiles toDATEADD(part, DATEDIFF(part, epoch, x), epoch)on thekustosqldialect. Unsupported grains raiseCompileError.date_trunc('grain', x)typed in SQL Lab is rewritten to the same form inCursor.execute(), so it also covers the path where Superset bypasses the dialect.weekis Monday-based like PostgreSQL. Sub-day grains use a 2000-01-01 epoch to stay inside int32 (secondtherefore works for years 1932–2068).Other functions Kusto's T-SQL emulation lacks
Rewritten in the same pass, each verified against the Kusto emulator:
datetrunc,date_part→DATEPART,ifnull→COALESCE,startofday/week/month/year,to_date,date(truncate to day). Anything without an exact equivalent (FORMAT,TRY_CAST,JSON_VALUE, …) is left alone so Kusto reports it.The rewriter is a linear scanner: string literals and SQL comments are opaque, nested calls are handled recursively, unbalanced input is passed through untouched. KQL queries are never rewritten.
ILIKE (68302686)
Kusto requires a
LIKEpattern to be a string literal, but SQLAlchemy'silikecompiles tolower(x) LIKE lower('%p%').lower('literal')is folded to the lowercased literal.Query language detection
is_tsql_query()now treats a leadingWITHas T-SQL (CTEs used to be sent as KQL and fail), skips leading comments, semicolons and whitespace, and requires a whole-wordselect/with.Nested
ORDER BYwithoutLIMITKusto rejects
ORDER BYinside a subquery unless it carriesTOP. Superset'sWRAP_SQLlimit method produces exactly that shape, so a nestedORDER BYwithout its own limit getsTOP {max_top_n}(default 500 000, configurable viacreate_engine(url, max_top_n=...)/ Superset engine parameters). Top-levelORDER BYis unaffected.Test infrastructure
kustosql+http://dialect for the Kusto emulator (plain HTTP, no auth, custom port).make emulator/make emulator-stop; integration tests run against the emulator by default and are skipped when it is unreachable.KUSTO_URLstill targets a real cluster.Verification
kustainer-linux: 51 passed, 1 skipped (auth error test, not applicable to the emulator).main.Known limits
date_trunc('second', …)overflows outside 1932–2068.lower(N'…')is not folded.[bracketed identifier]is read as a string opener.Release
Bumps the version to 3.2.0 (minor: new features, no breaking changes for existing
kustosql+https/kustokql+httpsusers). After merge: tag3.2.0, GitHub release,make releaseto PyPI.🤖 Generated with Claude Code