Repository navigation
fix(cli-internal): unblock e2e for release (gen2 migration, drift, e2e infra) - #14998
Merged
Merged
Conversation
Gen2 migration e2e apps failed the Gen2 sandbox deploy and the rollback
push for two reasons, both fixed here.
The generated Gen2 auth construct re-emitted login-derived standard
attributes without an AttributeDataType, and Cognito rejected the
UserPool update against the imported Gen1 pool ("Invalid
AttributeDataType input" / attribute modification errors). The generated
auth override now sets the UserPool Schema via addPropertyOverride:
verbatim from the deployed Gen1 auth template when available, otherwise
built from the live pool's attributes with their data types, constraints
and Required:false.
@aws-cdk/toolkit-lib 1.32.0 (pulled transitively by the floating
backend-deployer range) forces Express mode when a hotswap falls back to
a full deployment, which breaks the refactor's replacement update
("Replacement type updates not supported on stack with
disable-rollback"). The generated Gen2 package.json now pins
@aws-cdk/toolkit-lib to 1.40.0 via overrides.
Migration app snapshots are updated. Adopted from the unmerged branches
combined/gen2-migration-e2e and run-cb-e2e/zimzha/gen2-cognito-schema,
where all nine gen2-migration e2e apps passed.
---
Prompt: I am going to release amplify CLI. The process to release
amplify-cli is documented in:
https://quip-amazon.com/C223Aqn7XrI6/Deprecated-Runbook-Amplify-CLI-Release-Process.
However, we couldn't get AmplifyCLI-E2E-Testing codebuild task to pass,
which is blocking the release. I want you to first understand how to run
the AmplifyCLI-E2E-Testing Codebuild job. Meanwhile you can spin up more
agents to analyze the historial codebuild job, and understand what are
the failures. Some of the job maybe created to test a fix to certain
failure, using a fix branch. After the analysis, I'd like you to adopt
all the necessary not-merged historical fix that would fix the
constantly failing or flaky tests. Then evaluate if any additional fix
is needed. Ultimately, I'd like all the tests to be able to pass after
some retry. Meanwhile, I'd like a report on all the failing tests, the
root cause of the failure, whether it is flaky, and what can we do to
fix it. In summary, I'd like you to iterate towards a passing e2e run on
codebuild.
Co-authored-by: Galib Sarayev <sarayev@amazon.com>
After gen2-migration refactor --rollback, DynamoDB tables keep the Gen2 stack tags that CloudFormation propagated onto their StreamSpecification while they lived in the Gen2 stack. The Gen1 template does not declare stream tags, so drift detection reported each leftover tag and the subsequent gen2-migration lock failed (seen on the discussions and mood-board e2e apps). Drift detection now filters StreamSpecification tag entries that are absent from the template and whose key is a known Amplify Gen2 tag (amplify:deployment-type, created-by, gen2-migration/post-refactor). Any other stream tag drift is still reported. Adopted from the unmerged branch run-cb-e2e/zimzha/gen2-cognito-schema. --- Prompt: I am going to release amplify CLI. The process to release amplify-cli is documented in: https://quip-amazon.com/C223Aqn7XrI6/Deprecated-Runbook-Amplify-CLI-Release-Process. However, we couldn't get AmplifyCLI-E2E-Testing codebuild task to pass, which is blocking the release. I want you to first understand how to run the AmplifyCLI-E2E-Testing Codebuild job. Meanwhile you can spin up more agents to analyze the historial codebuild job, and understand what are the failures. Some of the job maybe created to test a fix to certain failure, using a fix branch. After the analysis, I'd like you to adopt all the necessary not-merged historical fix that would fix the constantly failing or flaky tests. Then evaluate if any additional fix is needed. Ultimately, I'd like all the tests to be able to pass after some retry. Meanwhile, I'd like a report on all the failing tests, the root cause of the failure, whether it is flaky, and what can we do to fix it. In summary, I'd like you to iterate towards a passing e2e run on codebuild.
The e2e HTML reporter serialized the jest results with a plain JSON.stringify, which throws on circular references (for example error objects attached to failed tests) and loses the run report. Circular references are now replaced with "[Circular]". Adopted from the unmerged branch run-cb-e2e/zimzha/gen2-cognito-schema. --- Prompt: I am going to release amplify CLI. The process to release amplify-cli is documented in: https://quip-amazon.com/C223Aqn7XrI6/Deprecated-Runbook-Amplify-CLI-Release-Process. However, we couldn't get AmplifyCLI-E2E-Testing codebuild task to pass, which is blocking the release. I want you to first understand how to run the AmplifyCLI-E2E-Testing Codebuild job. Meanwhile you can spin up more agents to analyze the historial codebuild job, and understand what are the failures. Some of the job maybe created to test a fix to certain failure, using a fix branch. After the analysis, I'd like you to adopt all the necessary not-merged historical fix that would fix the constantly failing or flaky tests. Then evaluate if any additional fix is needed. Ultimately, I'd like all the tests to be able to pass after some retry. Meanwhile, I'd like a report on all the failing tests, the root cause of the failure, whether it is flaky, and what can we do to fix it. In summary, I'd like you to iterate towards a passing e2e run on codebuild.
Windows e2e shards that enable the Pinpoint SMS channel failed with "Failed to enable the SMS channel" in most recent batches whenever their position-based region was ap-southeast-1 (w_function_permissions_env_5 13/21, w_notifications_analytics_compatibility_sms_2 15/22) or eu-central-1 (w_notifications_sms_update 8/22), while the same tests on Linux in us-west-2 / us-east-2 never failed (0/26). The test files that enable the SMS channel are now pinned to us-west-2 through FORCE_REGION_MAP and the e2e workflow specs are regenerated. Only CLI_REGION is added to the 12 affected shards; shard composition and identifiers are unchanged. --- Prompt: I am going to release amplify CLI. The process to release amplify-cli is documented in: https://quip-amazon.com/C223Aqn7XrI6/Deprecated-Runbook-Amplify-CLI-Release-Process. However, we couldn't get AmplifyCLI-E2E-Testing codebuild task to pass, which is blocking the release. I want you to first understand how to run the AmplifyCLI-E2E-Testing Codebuild job. Meanwhile you can spin up more agents to analyze the historial codebuild job, and understand what are the failures. Some of the job maybe created to test a fix to certain failure, using a fix branch. After the analysis, I'd like you to adopt all the necessary not-merged historical fix that would fix the constantly failing or flaky tests. Then evaluate if any additional fix is needed. Ultimately, I'd like all the tests to be able to pass after some retry. Meanwhile, I'd like a report on all the failing tests, the root cause of the failure, whether it is flaky, and what can we do to fix it. In summary, I'd like you to iterate towards a passing e2e run on codebuild.
Each e2e shard gets a second in-build attempt through retry, but the parent test account session assumed at the start of the build lasts one hour. When the first attempt ran longer, the retry called STS/Organizations with the expired session, jq failed on the empty account list and the build exited with status 3 before running any test. Since 2026-08-10 this cut the retry short in 193 failed builds (gen2-migration shards, l_api_9a_api_6c, searchable and others). retry now re-assumes the parent test account role from the CodeBuild role before every attempt after the first, when _loadTestAccountCredentials is available (all current callers define it). Verified with a local simulation of retry that the second attempt picks up freshly loaded credentials. --- Prompt: I am going to release amplify CLI. The process to release amplify-cli is documented in: https://quip-amazon.com/C223Aqn7XrI6/Deprecated-Runbook-Amplify-CLI-Release-Process. However, we couldn't get AmplifyCLI-E2E-Testing codebuild task to pass, which is blocking the release. I want you to first understand how to run the AmplifyCLI-E2E-Testing Codebuild job. Meanwhile you can spin up more agents to analyze the historial codebuild job, and understand what are the failures. Some of the job maybe created to test a fix to certain failure, using a fix branch. After the analysis, I'd like you to adopt all the necessary not-merged historical fix that would fix the constantly failing or flaky tests. Then evaluate if any additional fix is needed. Ultimately, I'd like all the tests to be able to pass after some retry. Meanwhile, I'd like a report on all the failing tests, the root cause of the failure, whether it is flaky, and what can we do to fix it. In summary, I'd like you to iterate towards a passing e2e run on codebuild.
integration_test failed in two of three runs on 2026-10-21 because apt-get update exited 100 when the git-core launchpad PPA configured in the build image returned 503 for its package index. The packages the job installs come from the Ubuntu archive, so that index is not needed. The index refresh is now retried once and the job continues with the indexes that were fetched; if a required index is really missing, the following apt-get install still fails. --- Prompt: I am going to release amplify CLI. The process to release amplify-cli is documented in: https://quip-amazon.com/C223Aqn7XrI6/Deprecated-Runbook-Amplify-CLI-Release-Process. However, we couldn't get AmplifyCLI-E2E-Testing codebuild task to pass, which is blocking the release. I want you to first understand how to run the AmplifyCLI-E2E-Testing Codebuild job. Meanwhile you can spin up more agents to analyze the historial codebuild job, and understand what are the failures. Some of the job maybe created to test a fix to certain failure, using a fix branch. After the analysis, I'd like you to adopt all the necessary not-merged historical fix that would fix the constantly failing or flaky tests. Then evaluate if any additional fix is needed. Ultimately, I'd like all the tests to be able to pass after some retry. Meanwhile, I'd like a report on all the failing tests, the root cause of the failure, whether it is flaky, and what can we do to fix it. In summary, I'd like you to iterate towards a passing e2e run on codebuild.
The two Schema override tests asserted on fragments of the generated auth resource with toContain / toMatch. They now snapshot the complete generated file, per the coding guidelines, so regressions in the rest of the output are caught too. The snapshots encode the same behavior the previous assertions checked. --- Prompt: I am going to release amplify CLI. The process to release amplify-cli is documented in: https://quip-amazon.com/C223Aqn7XrI6/Deprecated-Runbook-Amplify-CLI-Release-Process. However, we couldn't get AmplifyCLI-E2E-Testing codebuild task to pass, which is blocking the release. I want you to first understand how to run the AmplifyCLI-E2E-Testing Codebuild job. Meanwhile you can spin up more agents to analyze the historial codebuild job, and understand what are the failures. Some of the job maybe created to test a fix to certain failure, using a fix branch. After the analysis, I'd like you to adopt all the necessary not-merged historical fix that would fix the constantly failing or flaky tests. Then evaluate if any additional fix is needed. Ultimately, I'd like all the tests to be able to pass after some retry. Meanwhile, I'd like a report on all the failing tests, the root cause of the failure, whether it is flaky, and what can we do to fix it. In summary, I'd like you to iterate towards a passing e2e run on codebuild.
drift.md now lists every false-positive filter in detect-stack-drift.ts, including the AppSync ApiKey /Expires filter that was missing and the new Gen2 DynamoDB stream tag filter. gen2-migration/generate.md documents the always-emitted auth UserPool Schema override and the @aws-cdk/toolkit-lib pin in the generated root package.json. --- Prompt: I am going to release amplify CLI. The process to release amplify-cli is documented in: https://quip-amazon.com/C223Aqn7XrI6/Deprecated-Runbook-Amplify-CLI-Release-Process. However, we couldn't get AmplifyCLI-E2E-Testing codebuild task to pass, which is blocking the release. I want you to first understand how to run the AmplifyCLI-E2E-Testing Codebuild job. Meanwhile you can spin up more agents to analyze the historial codebuild job, and understand what are the failures. Some of the job maybe created to test a fix to certain failure, using a fix branch. After the analysis, I'd like you to adopt all the necessary not-merged historical fix that would fix the constantly failing or flaky tests. Then evaluate if any additional fix is needed. Ultimately, I'd like all the tests to be able to pass after some retry. Meanwhile, I'd like a report on all the failing tests, the root cause of the failure, whether it is flaky, and what can we do to fix it. In summary, I'd like you to iterate towards a passing e2e run on codebuild.
Simone319
marked this pull request as ready for review
October 23, 2026 09:16
xyos
approved these changes
Oct 23, 2026
Contributor
Author
|
AmplifyCLI-PR-Testing succeeded in:
|
Simone319
enabled auto-merge (squash)
October 23, 2026 10:11
sharonyajain
approved these changes
Oct 23, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description of changes
AmplifyCLI-E2E-Testinghas not gone green since July, which blocks the release. This PR collects every change needed to get both split batches (Linux and Windows) green: two Gen2 migration codegen fixes and one drift-detection fix that were sitting on unmerged branches, plus three small e2e-infrastructure fixes for failures that were environmental. With this branch, both batches pass, with only known-flaky shards needing oneRETRY_FAILED_BUILDSround.The v1→v2 transformer migration tests that failed from July to September (
<apiId>GraphQLSchema already exists in stack) were caused by a CloudFormation defect and have been fixed service-side; no change here (and #14963 is no longer needed).Gen2 migration: generated auth UserPool Schema
All nine
gen2_migratione2e apps failed the Gen2 sandbox deploy or the rollback push because the generated Gen2 auth construct re-emits login-derived attributes without anAttributeDataType, and Cognito rejects any Schema change on the imported Gen1 pool. The auth renderer now always emitscfnUserPool.addPropertyOverride('Schema', …): verbatim from the deployed Gen1 auth template when it is available, otherwise built from the live pool's attributes (declarable standard and custom attributes, each with its data type andRequired: false).TS.jsValuenow renders arrays of objects. Migration app snapshots are updated.This contains the change from draft #14993 and extends it with the verbatim Gen1 Schema.
Gen2 migration: CDK toolkit pin in the generated app
@aws-cdk/toolkit-lib1.32.0, which the floating backend-deployer range pulls in, forces Express mode when a sandbox hotswap falls back to a full deployment, breaking the refactor's replacement update. The generated rootpackage.jsonnow pins@aws-cdk/toolkit-libto 1.40.0 throughoverrides. This replaces the broader dependency pins in draft #14990, which hit npmERESOLVE.Drift: leftover Gen2 DynamoDB stream tags
After
gen2-migration refactor --rollback, tables keep the Gen2 stack tags that CloudFormation propagated onto theirStreamSpecification, and the followinggen2-migration lockfailed on that drift. A new false-positive filter drops/StreamSpecification/Tags/Nentries that are absent from the template and whose key is a known Amplify Gen2 tag; any other stream tag change is still reported.E2E infrastructure
us-west-2viaFORCE_REGION_MAP. On Windows they failed in most batches whenever their position-based region wasap-southeast-1oreu-central-1("Failed to enable the SMS channel"), while the same tests never failed on Linux inus-west-2/us-east-2. OnlyCLI_REGIONis added to the 12 affected shards in the generated specs.retrynow re-assumes the parent test account before each re-attempt. The parent session lasts one hour, so a first attempt longer than that made the retry fail before running any test (193 failed builds since 2026-08-10).integration_testsetup retriesapt-get updateand continues when only the third-party git-core PPA index fails (it returned 503 repeatedly).Docs
drift.mdlists the current false-positive filters (including the ApiKey/Expiresfilter from #14991, which was missing).gen2-migration/generate.mddocuments the Schema override and the toolkit-lib pin.Notes for reviewers
Required: trueon the refactor update, while the verbatim path emitsRequired: truefrom the template and passes e2e. Worth deciding whether the fallback should stay.lists workout programs including a newly created onecan miss a just-created item (unpaginated, eventually consistent list on an owner-auth model); it failed once in three runs. A follow-up could page throughnextTokenand retry briefly.l_api_9a_api_6candl_searchable_migration(AppSync returns "API not found" when creating the OpenSearch data source; service-side), container tests (ECS deployment awaiter), andupb(split-mode S3 cache race).Issue #, if available
Supersedes drafts #14993 and #14990 (the
/Expirespart of #14990 already landed as #14991).Description of how you validated changes
RETRY_FAILED_BUILDSround for flaky shards (integration_testPPA 503,l_api_9a_api_6c). Theintegration_testapt change was validated in a scoped batch where the firstapt-get updatehit the 503 and the retry passed. Final split e2e on this PR's head (7fbae99): green with no batch retries — Linux 160/160, Windows 134/134. One shard (l_gen2_migration_fitness_tracker) passed on its in-build second attempt; its first attempt ran ~1h53m, so this also exercised the retry credential refresh.packages/amplify-cliunit tests for the auth generator, package.json generator and drift detection: 3 suites / 50 tests pass locally (inline snapshots regenerated); the full unit suite (testjob) passed in both e2e runs above.Checklist
yarn testpassesBy submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.