Updating Settings.json and Pull pipeline fails.

[BoostSebastian]

Updating Setting.json and Pull pipeline fails.

Using latest files from repo and tried to specify to use 1.9 of AzOps.

When going from setting.json:

            "Core.SkipPim": true,
            "Core.SkipPolicy": false,
            "Core.SkipResource": true,
            "Core.SkipChildResource": true,
            "Core.SkipResourceGroup": false,

to

            "Core.SkipPim": true,
            "Core.SkipPolicy": false,
            "Core.SkipResource": false,
            "Core.SkipChildResource": true,
            "Core.SkipResourceGroup": false,

or

            "Core.SkipPim": true,
            "Core.SkipPolicy": false,
            "Core.SkipResource": false,
            "Core.SkipChildResource": false,
            "Core.SkipResourceGroup": false,

I end up with the following error message and can't work around it:

Where-Object: The term 'Where-Object' is not recognized as a name of a cmdlet, function,
script file, or executable program. Check the spelling of the name, or if a
path was included, verify that the path is correct and try again.

It is always stopping at Initialize step but at different amount of time, all from 4 minutes to 10 minutes
Steps to reproduce

1. Can't reproduce, happends everytime I try to redo the steps.

[daltondhcp]

Hi Sebastian,

Just tried in my own environment with the same settings and are unable to reproduce the issue. Have you made any other changes to the settings file? Can you share some more context on at what resource/scope it is trying to discover when it fails?

/Johan

[BenjaminEngeset]

Same here. In general getting a lot of different exceptions on the Initialize task. Different errors and different timespans. Happends on most of the AzOps installations.

settings.json:

{
    "Version": 1,
    "DynamicTree": {
        "AzOps": {
            "Core.AutoGeneratedTemplateFolderPath": ".",
            "Core.AutoInitialize": true,
            "Core.DefaultDeploymentRegion": "westeurope",
            "Core.EnrollmentAccountPrincipalName": "",
            "Core.ExcludedSubOffer": [
                "AzurePass_2014-09-01",
                "FreeTrial_2014-09-01",
                "AAD_2015-09-01"
            ],
            "Core.ExcludedSubState": [
                "Disabled",
                "Deleted",
                "Warned",
                "Expired"
            ],
            "Core.ExportRawTemplate": false,
            "Core.IgnoreContextCheck": false,
            "Core.IncludeResourcesInResourceGroup": ["*"],
            "Core.IncludeResourceType": ["*"],
            "Core.InvalidateCache": true,
            "Core.OfferType": "MS-AZR-0017P",
            "Core.PartialMgDiscoveryRoot": [],
            "Core.SkipPim": true,
            "Core.SkipPolicy": false,
            "Core.SkipResource": false,
            "Core.SkipChildResource": false,
            "Core.SkipResourceGroup": false,
            "Core.SkipResourceType": [
                "microsoft.alertsmanagement/smartDetectorAlertRules",
                "Microsoft.PowerPlatform/accounts",
                "Microsoft.PowerPlatform/enterprisePolicies",
                "microsoft.visualstudio/account",
                "Microsoft.VSOnline/plans",
                "Microsoft.Insights/scheduledQueryRules"
            ],
            "Core.SkipRole": false,
            "Core.SubscriptionsToIncludeResourceGroups": ["*"],
            "Core.TemplateParameterFileSuffix": ".json",
            "Core.ThrottleLimit": 10,
            "Core.WhatifExcludedChangeTypes":[
                "NoChange",
                "Ignore"
            ]
        },
        "|AzureDevOps": {
            "!Condition": "EnvAzDevPipeline -or -not OSWindows",
            "AzOps.Core.State": "%PIPELINE_WORKSPACE%/s/root"
        },
        "|GitHub": {
            "!Condition": "EnvGithubAction",
            "AzOps.Core.State": "%GITHUB_WORKSPACE%/root"
        },
        "PSFramework": {
            "Message.Info.Maximum": 3
        }
    }
}

[daltondhcp]

Thanks for sharing @bengeset96. The last two (related to child resources), looks like a downstream issue with the Export ARM API (internal server error)
For the others, it would be great to get some more context on what specific resources/resource groups that are failing (i.e. the lines before the terminating errors appears)

To simplify debugging, if possible, please send over some example pipeline logs over email to me on jodahlbo[at]microsoft.com.

[BenjaminEngeset]

@daltondhcp

I've sent you some logs from the various runs on your email.

Can also enable debugging on the agent if needed for further investigation.

[jsandquist]

@BoostSebastian @bengeset96 I recognize a few of these, see #656, but after we reset the logging level to default 3 (instead of 9) and also changed the throttle limit to 1 instead of default 10 (although unsure if related at all) we no longer see this on a daily basis, but only once every week or so.

[BenjaminEngeset]

@jsandquist

I've tried to throttle it down as well, but did not have any impact on the AzOps installations. I'll continue debugging and trying different scenarios. The larger the environment is, the more often I get it.

[daltondhcp]

Update: In 1.9.1, we introduced error handling to handle the "Internal Server Error" issues thrown from the ExportResources API. Please let us know if you observe improvements in the situation.

[BoostSebastian]

@daltondhcp I tried today to reimplement everything from scratch but exactly the same error.

Test-Path: Value cannot be null. (Parameter 'The provided Path argument was null or an empty collection.')

##[error]PowerShell exited with code '1'.
Finishing: Initialize

This happends right after our resource group with Exchange 2019 servers pool
This is maybe 4/10 and the rest at what I can only assume is random

[daltondhcp]

@daltondhcp I tried today to reimplement everything from scratch but exactly the same error.

Test-Path: Value cannot be null. (Parameter 'The provided Path argument was null or an empty collection.')

##[error]PowerShell exited with code '1'.
Finishing: Initialize

This happends right after our resource group with Exchange 2019 servers pool This is maybe 4/10 and the rest at what I can only assume is random

This looks like a completely different error, right?

[BoostSebastian]

@daltondhcp I don't know, getting different errors each time I run the Pipeline and at different lines.
Here is four runs in a row.
Using 1.9.1.

[BenjaminEngeset]

We're having the same issues as I can tell @BoostSebastian. I have spent quite much time troubleshooting, but still not any closer.

Johan has been kind and I have sent some logs, but they did not tell much.

Starting from scratch with blank repository state and different agent has not helped.

[BoostSebastian]

@BenjaminEngeset, I tried a blank repo but no success. I could back it up to 1.7 but same error again when changing to include resources and child-resources.

This seems to be the most commin error message:

Write-PSFMessage: The term 'Write-PSFMessage' is not recognized as a name of a cmdlet, function

[BoostSebastian]

I have tried to run many times now.
One success and rest of the runs is failed on Initialize step.
At 1.9.1 it now always show.

Write-PSFMessage: The term 'Write-PSFMessage' is not recognized as a name of a cmdlet, function

[BoostSebastian]

Any Updates :) Still the same Issues.

[daltondhcp]

Hey @BoostSebastian - sorry for the slow updates. We are aiming to ship AzOps 2.0 very soon that will introduce lots of performance improvements and also from what we've seen address the above. Keep your eyes peeled.

[BoostSebastian]

@daltondhcp Souund awesome :) I will try as soon it is released.