Invoke-AzAksRunCommand with subdirectories in -CommandContextAttachment doesn't work due to the use of backslashes as path separators

[Taha-cmd]

Description

A directory, which contains kubernetes manifests on the top level, can be applied to an AKS cluster using Invoke-AzAksRunCommand -CommandContextAttachment <path> -Command "kubectl apply -f ."

However, if the directory contains sub-directories, then Invoke-AzAksRunCommand fails with the misleading error message:
Failed to run command due to cluster perf issue, container command- in aks-command namespace did not start within 30s on your cluster, retry may helps. If issue persist, you may need to tune your cluster with better performance (larger node/paid tier).

Invoke-AzAksRunCommand creates a pod with an init-container, which does the following:

• command:
  • bash
  • -c
  • echo extracting user context && unzip -o /cmd-context/context.zip -d /command-state/command-files
    && echo copying default SA && cp -r /var/run/secrets/kubernetes.io/serviceaccount
    /command-state/k8s-identity && echo wait for AAD token && inotifywait -e modify
    -t 60 /command-state/k8s-identity/token && sleep 5s

By looking into the logs of the init-container created by Invoke-AzAksRunCommand , we noticed that it fails with exit code 1 after executing the unzip step. According to the man page of unzip , it returns 1 when there are errors OR warnings. By inspecting the logs, we see indeed such a warning, which reads: warning: /cmd-context/context.zip appears to use backslashes as path separators. We believe that this root cause of the issue.

Issue script & Debug output

Invoke-AzAksRunCommand -ResourceGroupName "" -Name "" `
-CommandContextAttachment "directory with top level k8s manifests" `
-Command "kubectl apply -f ." `
-Force # this works fine

Invoke-AzAksRunCommand -ResourceGroupName "" -Name "" `
-CommandContextAttachment "directory with sub-directories containing k8s manifests" `
-Command "kubectl apply -f . --recursive" `
-Force # this fails

Environment data

Name                           Value
----                           -----
PSVersion                      7.2.8
PSEdition                      Core
GitCommitId                    7.2.8
OS                             Microsoft Windows 10.0.19044
Platform                       Win32NT
PSCompatibleVersions           {1.0, 2.0, 3.0, 4.0…}
PSRemotingProtocolVersion      2.3
SerializationVersion           1.1.0.1
WSManStackVersion              3.0

Module versions

ModuleType Version    PreRelease Name
---------- -------    ---------- ---- 
Script     9.0.1                 Az

ModuleType Version    PreRelease Name
---------- -------    ---------- ----
Script     5.1.0                 Az.Aks

Error output

RequestId      : <>
Message        : Failed to run command due to cluster perf issue, container command-93cd5d2981594abab3e9a06ae14a7c2f in aks-command namespace did not start within 30s on your  
                 cluster, retry may helps. If issue persist, you may need to tune your cluster with better performance (larger node/paid tier).
ServerMessage  : KubernetesOperationError: Failed to run command due to cluster perf issue, container command-93cd5d2981594abab3e9a06ae14a7c2f in aks-command namespace did     
                 not start within 30s on your cluster, retry may helps. If issue persist, you may need to tune your cluster with better performance (larger node/paid tier).    
                 (System.Collections.Generic.List`1[Microsoft.Rest.Azure.CloudError])
ServerResponse : {BadRequest}
RequestMessage : {POST <clusterId>}
InvocationInfo : {Invoke-AzAksRunCommand}
Line           : Invoke-AzAksRunCommand -ResourceGroupName "" `

Position       : At C:\Development\Invoke-AzAksRunCommand bug repro\Deploy-Manifests.ps1:1 char:1
                 + Invoke-AzAksRunCommand -ResourceGroupName "" `
                 + ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
StackTrace     :    at Microsoft.Azure.Management.ContainerService.ManagedClustersOperations.BeginRunCommandWithHttpMessagesAsync(String resourceGroupName, String
                 resourceName, RunCommandRequest requestPayload, Dictionary`2 customHeaders, CancellationToken cancellationToken)
                    at Microsoft.Azure.Management.ContainerService.ManagedClustersOperations.RunCommandWithHttpMessagesAsync(String resourceGroupName, String resourceName,     
                 RunCommandRequest requestPayload, Dictionary`2 customHeaders, CancellationToken cancellationToken)
                    at Microsoft.Azure.Management.ContainerService.ManagedClustersOperationsExtensions.RunCommandAsync(IManagedClustersOperations operations, String
                 resourceGroupName, String resourceName, RunCommandRequest requestPayload, CancellationToken cancellationToken)
                    at Microsoft.Azure.Management.ContainerService.ManagedClustersOperationsExtensions.RunCommand(IManagedClustersOperations operations, String
                 resourceGroupName, String resourceName, RunCommandRequest requestPayload)
                    at Microsoft.Azure.Commands.Aks.RunAzureRmAksCommand.<ExecuteCmdlet>b__42_0()
                    at Microsoft.WindowsAzure.Commands.Utilities.Common.AzurePSCmdlet.ConfirmAction(Boolean force, String continueMessage, String processMessage, String        
                 target, Action action, Func`1 useShouldContinue)
                    at Microsoft.Azure.Commands.ResourceManager.Common.AzureRMCmdlet.ConfirmAction(Boolean force, String continueMessage, String processMessage, String
                 target, Action action)
                    at Microsoft.Azure.Commands.Aks.RunAzureRmAksCommand.ExecuteCmdlet()
                    at Microsoft.WindowsAzure.Commands.Utilities.Common.CmdletExtensions.<>c__3`1.<ExecuteSynchronouslyOrAsJob>b__3_0(T c)
                    at Microsoft.WindowsAzure.Commands.Utilities.Common.CmdletExtensions.ExecuteSynchronouslyOrAsJob[T](T cmdlet, Action`1 executor)
                    at Microsoft.WindowsAzure.Commands.Utilities.Common.CmdletExtensions.ExecuteSynchronouslyOrAsJob[T](T cmdlet)
                    at Microsoft.WindowsAzure.Commands.Utilities.Common.AzurePSCmdlet.ProcessRecord()

[dingmeng-xue]

@YanaXu , please look into this issue.