Skip to content

Commit 64a5c8b

Browse files
azure-sdk-automation[bot]connieyCopilot
authored
Sync eng/common directory with azure-sdk-tools for PR 17155 (#3827)
New-TestResources: Resolve deployment state instead of blindly retrying on failure New-AzResourceGroupDeployment retries reused the same -Name as the original deployment. If the original was still running server-side (e.g. a long-running template), the retry collided with it (Code=DeploymentActive) and every retry failed the same way. Resolve-DeploymentAfterFailure now asks ARM for the existing deployment's actual state before deciding what to do: - still running -> poll it to completion (bounded, 60 min timeout) instead of starting a redundant concurrent deployment, which could otherwise cause resource-level conflicts and leave an orphaned duplicate deployment record behind - already succeeded/failed/canceled -> return that result as-is so normal failure handling reports the real error instead of masking it with a retry - not found at all -> safe to submit a new deployment under a unique name, truncating the base name so the result stays within ARM's 64-character deployment-name limit Also resets $deployment to $null at the start of each template-file iteration so a stale value from a previous template can't leak through if the try block throws without assigning a new one. Co-authored-by: Connie Yau <conniey@microsoft.com> Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: ce6c37d3-fb5b-4100-8c45-fc8ee31b4170
1 parent b7bd3f7 commit 64a5c8b

2 files changed

Lines changed: 143 additions & 9 deletions

File tree

‎eng/common/TestResources/New-TestResources.ps1‎

Lines changed: 11 additions & 9 deletions
Original file line numberDiff line numberDiff line change
@@ -601,6 +601,10 @@ try {
601601

602602
# Deploy the templates
603603
foreach ($templateFile in $templateFiles) {
604+
# Reset so a stale value from a previous iteration can't leak through if the try block
605+
# below throws without reassigning it.
606+
$deployment = $null
607+
604608
# Deployment fails if we pass in more parameters than are defined.
605609
Write-Verbose "Removing unnecessary parameters from template '$($templateFile.jsonFilePath)'"
606610
$templateJson = Get-Content -LiteralPath $templateFile.jsonFilePath | ConvertFrom-Json
@@ -652,15 +656,13 @@ try {
652656
}
653657

654658
if (!$deployment -or $deployment.ProvisioningState -ne 'Succeeded') {
655-
Write-Warning "Initial deployment attempt failed, retrying..."
656-
$deployment = Retry -Attempts 4 -Action {
657-
New-AzResourceGroupDeployment `
658-
-Name $BaseName `
659-
-ResourceGroupName $resourceGroup.ResourceGroupName `
660-
-TemplateFile $templateFile.jsonFilePath `
661-
-TemplateParameterObject $templateFileParameters `
662-
-Force:$Force
663-
}
659+
Write-Warning "Initial deployment attempt failed or did not report success; checking whether it's still active server-side..."
660+
$deployment = Resolve-DeploymentAfterFailure `
661+
-DeploymentName $BaseName `
662+
-ResourceGroupName $resourceGroup.ResourceGroupName `
663+
-TemplateFile $templateFile.jsonFilePath `
664+
-TemplateFileParameters $templateFileParameters `
665+
-Force:$Force
664666
}
665667

666668
if ($deployment.ProvisioningState -ne 'Succeeded') {

‎eng/common/TestResources/TestResources-Helpers.ps1‎

Lines changed: 132 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -12,6 +12,138 @@ function LogVsoCommand([string]$message) {
1212
Write-Host $message
1313
}
1414

15+
<#
16+
.SYNOPSIS
17+
Resolves the outcome of a deployment after New-AzResourceGroupDeployment threw or
18+
returned a non-'Succeeded' state, without starting a redundant concurrent deployment.
19+
20+
.DESCRIPTION
21+
A non-'Succeeded' result (or a thrown error) from New-AzResourceGroupDeployment doesn't
22+
necessarily mean the deployment failed - it can also mean the deployment is still
23+
genuinely running server-side (e.g. long-running templates like database server
24+
provisioning that can take well past this cmdlet's own client-side wait). Starting a
25+
second, concurrent deployment in that case - whether reusing the same name or a new one -
26+
races the still-active original: same name fails fast with Code=DeploymentActive, but even
27+
a different name can still cause resource-level conflicts if both deployments try to
28+
manage the same underlying resource, and either way leaves an orphaned duplicate
29+
deployment record behind.
30+
31+
This function asks ARM for the deployment's actual current state before deciding what to
32+
do:
33+
- Not found at all: nothing is in flight server-side, so it's safe to submit a brand
34+
new deployment under a unique name.
35+
- Still running: poll it to completion (bounded by PollTimeoutMinutes) instead of
36+
submitting another deployment.
37+
- Already succeeded: treat it as a success.
38+
- Already failed/canceled: return it as-is so the caller's normal failure handling
39+
(HandleDeploymentFailure) reports the real error, instead of masking it with a retry.
40+
41+
.PARAMETER DeploymentName
42+
The name of the deployment that was originally attempted (and may or may not still be
43+
active server-side).
44+
45+
.PARAMETER ResourceGroupName
46+
The resource group the deployment was made (or attempted) against.
47+
48+
.PARAMETER TemplateFile
49+
Path to the ARM template file to deploy, used only if a brand new deployment needs to be
50+
submitted (i.e. no existing deployment record is found).
51+
52+
.PARAMETER TemplateFileParameters
53+
Template parameters to pass to New-AzResourceGroupDeployment, used only if a brand new
54+
deployment needs to be submitted.
55+
56+
.PARAMETER Force
57+
Passed through to New-AzResourceGroupDeployment if a brand new deployment needs to be
58+
submitted.
59+
60+
.PARAMETER PollTimeoutMinutes
61+
Maximum time to poll a still-active existing deployment before giving up and throwing.
62+
Defaults to 60 minutes.
63+
64+
.OUTPUTS
65+
A deployment object of the same type/shape returned by New-AzResourceGroupDeployment and
66+
Get-AzResourceGroupDeployment (has a ProvisioningState property callers can check the same
67+
way they already do for the initial deployment attempt), in one of these states:
68+
- 'Succeeded', 'Failed', or 'Canceled' - the pre-existing deployment reached that
69+
terminal state (either immediately, or after polling); or
70+
- whatever New-AzResourceGroupDeployment returns for a freshly submitted retry
71+
deployment (the "not found at all" case above).
72+
73+
Throws if the pre-existing deployment never reaches a terminal state within
74+
PollTimeoutMinutes, or if the new retry deployment submission itself throws.
75+
#>
76+
function Resolve-DeploymentAfterFailure(
77+
[string] $DeploymentName,
78+
[string] $ResourceGroupName,
79+
[string] $TemplateFile,
80+
[hashtable] $TemplateFileParameters,
81+
[switch] $Force,
82+
[int] $PollTimeoutMinutes = 60
83+
) {
84+
$terminalStates = @('Succeeded', 'Failed', 'Canceled')
85+
$existing = Get-AzResourceGroupDeployment -ResourceGroupName $ResourceGroupName -Name $DeploymentName -ErrorAction SilentlyContinue
86+
87+
if ($existing) {
88+
if ($existing.ProvisioningState -in $terminalStates) {
89+
Write-Warning "Deployment '$DeploymentName' already reached terminal state '$($existing.ProvisioningState)' server-side."
90+
return $existing
91+
}
92+
93+
Write-Warning "Deployment '$DeploymentName' is still active server-side (state '$($existing.ProvisioningState)'). Polling instead of starting a redundant concurrent deployment."
94+
$start = Get-Date
95+
$deadline = $start.AddMinutes($PollTimeoutMinutes)
96+
$lastLoggedState = $null
97+
$nextHeartbeat = $start
98+
99+
while ((Get-Date) -lt $deadline) {
100+
$existing = Get-AzResourceGroupDeployment -ResourceGroupName $ResourceGroupName -Name $DeploymentName -ErrorAction SilentlyContinue
101+
if (!$existing) {
102+
# The deployment record disappeared between polls; fall through to submitting
103+
# a new deployment below.
104+
break
105+
}
106+
107+
$now = Get-Date
108+
if ($existing.ProvisioningState -ne $lastLoggedState -or $now -ge $nextHeartbeat) {
109+
$elapsedMinutes = [int] ($now - $start).TotalMinutes
110+
Write-Host " Deployment '$DeploymentName' provisioningState = $($existing.ProvisioningState) (elapsedMinutes = $elapsedMinutes)"
111+
$lastLoggedState = $existing.ProvisioningState
112+
$nextHeartbeat = $now.AddMinutes(1)
113+
}
114+
115+
if ($existing.ProvisioningState -in $terminalStates) {
116+
return $existing
117+
}
118+
119+
Start-Sleep -Seconds 15
120+
}
121+
122+
if ($existing -and $existing.ProvisioningState -notin $terminalStates) {
123+
throw "Timed out after $PollTimeoutMinutes minute(s) waiting for deployment '$DeploymentName' to finish. Latest provisioningState: '$($existing.ProvisioningState)'."
124+
}
125+
}
126+
127+
# No existing deployment record was found: nothing is actually in flight server-side (e.g.
128+
# the client call failed before ARM even registered the deployment), so it's safe to submit
129+
# a brand new deployment. ARM deployment names are limited to 64 characters; truncate only
130+
# the base portion (not the unique "-retry-<suffix>" we append) so the result always stays
131+
# within the limit, even when $DeploymentName is already close to 64 chars.
132+
Write-Warning "No deployment named '$DeploymentName' was found; submitting a new deployment under a unique name."
133+
$retrySuffix = "-retry-$([Guid]::NewGuid().ToString('N').Substring(0, 8))"
134+
$maxBaseLength = 64 - $retrySuffix.Length
135+
$truncatedBaseName = if ($DeploymentName.Length -gt $maxBaseLength) { $DeploymentName.Substring(0, $maxBaseLength) } else { $DeploymentName }
136+
$retryDeploymentName = "$truncatedBaseName$retrySuffix"
137+
Write-Host "Submitting new deployment as '$retryDeploymentName'"
138+
139+
return New-AzResourceGroupDeployment `
140+
-Name $retryDeploymentName `
141+
-ResourceGroupName $ResourceGroupName `
142+
-TemplateFile $TemplateFile `
143+
-TemplateParameterObject $TemplateFileParameters `
144+
-Force:$Force
145+
}
146+
15147
function Retry([scriptblock] $Action, [int] $Attempts = 5) {
16148
$attempt = 0
17149
$sleep = 5

0 commit comments

Comments
 (0)