Skip to content

ASIM custom unifying parsers (follow-up to #14880): the docs "line to add" table drops disabled/pack by construction; Im_DnsCustom eventtype=lookup vs Query-gating leaves; watchlist example uses the built-in-only key #15172

Closed

Description

@spasaintk

Follow-up to #14880, as invited on closing. Everything below was re-verified today (2026-09-22) against the live Learn page normalization-manage-parsers (updated_at: 2026-09-18T22:36, git_commit_id: 4d8e8ee47d16…) and this repository at 2e3336d052. Six of the seven points in #14880 are closed by PR #15162 plus the republished docs — thank you. What remains is a mechanism, not a typo list, so it is filed once rather than per row.

1. The "line to add" table drops disabled and pack by construction

The page says the table is built from ParserParams in Parsers/ASim<Schema>/Parsers/im<Schema>.yaml. That list is a strict subset of the custom unifying stub's signature, so the two flags that only the stub declares fall out of the generated row and will regress on every regeneration. Measured today:

count
stubs under ASIM/deploy/EmptyCustomUnifyingParsers/ declaring disabled 12 of 13 (all but Im_AgentEventCustom)
stubs declaring pack 13 of 13
table rows forwarding disabled=disabled 4 of 13 (DhcpEvent, FileEvent, RegistryEvent, WebSession)
rows omitting it although the stub declares it 8
rows omitting pack=pack although the stub declares it 1 (ProcessEvent)

The new prose ("also forward disabled=disabled … even when it isn't included in the schema's row below") documents the gap rather than closing it. Two fixes, either sufficient:

  • generate the rows from the stub's functionParameters (the thing the reader is editing) instead of from ParserParams; or
  • add a lockstep check to the PR gate (.script/tests/asimParsersTest/) that im<Schema>.yaml's ParserParams ∪ {disabled,pack} equals the stub signature, so the two sources cannot drift silently.

2. Im_DnsCustom's eventtype='lookup' default and the Dns leaves disagree — a leaf can return zero rows silently

The stub now defaults eventtype:string='lookup' (aligned with imDns.yaml). The schema page says EventType "is set to Query", and the corpus is split three ways on what the parameter means (Parsers/ASimDns/Parsers/, measured at 2e3336d052):

evidence count
leaf parameter defaults eventtype:string='Query' 14
leaf parameter defaults eventtype:string='lookup' 4
body literals EventType = 'Query' 10
body literals EventType = 'lookup' 0

and the predicates that consume it:

  • vimDnsGigamonVSeries.yaml:83 gates eventtype == 'Query' → under the stub's 'lookup' default this leaf contributes zero rows, with no error;
  • vimDnsMicrosoftSysmonWindowsEvent.yaml:89 gates eventtype=='lookup' → a caller passing the documented 'Query' gets zero rows from it;
  • vimDnsMicrosoftSysmon.yaml:89 accepts both (in~ ('Query','lookup'));
  • vimDnsMicrosoftNXlog.yaml:259 maps lookup → Query before comparing.

This is the same class as point 4 of #14880 (a default that silently empties a union arm). Suggested resolution: pick one canonical spelling (the NXlog normalisation iff(eventtype == "lookup", "Query", eventtype) is the one that is backward-compatible for every caller) and have the gate check every Dns leaf's eventtype predicate against it.

3. The watchlist example shows the built-in-only key

The "Use a modified version of a built-in parser" example is CallerContext = Exclude_Im_Dns, SourceSpecificParser = Exclude_Im_Dns_AzureFirewall. Those underscore keys are what the read-only built-ins consult. The workspace-deployed unifiers the same page tells the reader to deploy key on the bare alias — Parsers/ASimDns/Parsers/ASimDns.yaml:43-56 reads ExcludeASimDns, ExcludeASimDnsBuiltIn, ExcludeASimDnsCiscoUmbrella, … (and, as measured, ExcludeASimASimDnsAzureFirewall, a key with a doubled prefix nobody would guess). A reader who follows the example mutes the built-in but not the deployed copy, or vice versa. One sentence stating which family of keys applies to which deployment — or a single spelling in both — would close it.

4. The public docs mirror never synced

MicrosoftDocs/defender-docs@public (and repo_sync_working_branch) still serve the pre-#14880 normalization-manage-parsers.md and a normalization-common-fields.md with ms.date: 11/17/2021 and no AdditionalEntities — six days after the live pages changed. Anyone reading or vendoring from the public repo still sees the broken table. Probably a docs-team routing item; noting it here because #14880's fix is invisible there.

Corrections to #14880 (our side)

  • Point 6 said "every stub declares disabled:bool=false". Im_AgentEventCustom declares none — then and now.
  • We reported AssetEntity had no documented row but never compared its stub to imAssetEntity.yaml. Your entitytype_in → assettype_in rename fixed a real defect we missed.

Not verified

Whether the built-in _Im_<Schema> binds the custom stub positionally or by name. PR #15162's body says the parameters were appended "to preserve existing positional argument slots"; the built-in bodies are not in this repository, so it cannot be checked from here. If binding is positional, the table's named-argument advice is safe but the stub's parameter ORDER becomes load-bearing and would deserve a gate of its own.

Activity

  1. yummyblabla commented on Oct 6, 2026

    @yummyblabla
    Collaborator

    Acknowledge on the issue. We will address the issues that you have as soon as possible.

  2. yummyblabla commented on Oct 8, 2026

    @yummyblabla
    Collaborator

    Regarding issue 1,
    I have removed the disabled parameters from the unifying custom parser. It did not make sense that the unifying parser can be disabled. However, the 'line-to-add' in the documentation should be updated to what is supported now. the parameter 'disabled' is kept in case the user wants to disable/enable their custom parsers.
    The pack parameter in ProcessEvent has been omitted as it is currently not supported in the ParserParams. This should be the only anomaly in the arguments.

  3. yummyblabla commented on Oct 8, 2026

    @yummyblabla
    Collaborator

    Regarding issue 2, the default value for eventtype is now 'Query'. From my findings, 'lookup' was a legacy value and the early parsers were written to handle either lookup or query, but not the recent parsers. There should not be any issues with changing the default to Query.

  4. yummyblabla commented on Oct 8, 2026

    @yummyblabla
    Collaborator

    Regarding issue 3, the language is made clear. A PR will be made to fix typos made in the keys for workspace deployed parsers.

  5. yummyblabla commented on Oct 8, 2026

    @yummyblabla
    Collaborator

    Regarding issue 4, it looks like the team that owns public documentation have stopped maintaining defender-docs

  6. yummyblabla commented on Oct 8, 2026

    @yummyblabla
    Collaborator

    spasaintk Let me know if any of the changes that were implemented seems reasonable for you.

  7. yummyblabla commented on Oct 15, 2026

    @yummyblabla
    Collaborator

    spasaintk I will be closing the issue. If you have any further questions about the changes, feel free to reopen or open another issue.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Labels

ASIMParserParser specialty review needed

Type

No type

Projects

No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions