Pre Build (Merge and Test Data Files) - [ERROR] Environment configuration for '[ENV]' contains Mandatory settings, which is not allowed!

[pmatthews05]

Working through Walkthrough guide. Using Microsoft Hosted Agent.

After uploading the certificates to my Keyvault, and creating an environment using the following script https://github.com/ykuijs/M365DSC_CICD/blob/main/Supportscripts/PrepareKeyvault.ps1

I've updated the [env]#Generic.psd1 file, but not really changed anything else within the Data files.

When I'm running the following pipeline https://github.com/ykuijs/M365DSC_CICD/blob/main/Pipelines/build-template.yaml pipeline, I've reached the step in the pipeline "Pre Build (Merge and Test Data Files)" and I'm hitting the following errors.

Starting discovery in 1 files.
Discovery found 1 tests in 8ms.
Running tests.

Running tests from 'C:\Users\VssAdministrator\AppData\Local\Temp\tmpau0tcs.tests.ps1'
Describing --- Check M365-DSC-CompositeResources configuration ---
 Context Mandatory
##[error]    [-] Mandatory 29ms (26ms|3ms)
##[error]     RuntimeException: Cannot index into a null array.
##[error]     at <ScriptBlock>, C:\Users\VssAdministrator\AppData\Local\Temp\tmpau0tcs.tests.ps1:5
Tests completed in 138ms
Tests Passed: 0, Failed: 1, Skipped: 0, Inconclusive: 0, NotRun: 0
2025-01-17 16:32:40 [VERBOSE] Test-M365DSCPowershellDataFile: ^ Elapsed: 0.2060283 Seconds              TotalTime: 11.7617955 seconds
2025-01-17 16:32:40 [FAILURE] Test-M365DSCPowershellDataFile: Pester[5.7.1]  Tests:1  Passed:0  Failed:1 
2025-01-17 16:32:40 [FAILURE] PreBuild.ps1:   [ERROR] Environment configuration for '******' contains Mandatory settings, which is not allowed!
2025-01-17 16:32:40 [INFO] PreBuild.ps1:  
2025-01-17 16:32:40 [INFO] PreBuild.ps1:  
2025-01-17 16:32:40 [INFO] PreBuild.ps1: ---------------------------------------------------------
2025-01-17 16:32:40 [INFO] PreBuild.ps1:  Starting Basic/Tenant merge, Tokenizing and QA testing
2025-01-17 16:32:40 [INFO] PreBuild.ps1: ---------------------------------------------------------
2025-01-17 16:32:40 [INFO] PreBuild.ps1:  
2025-01-17 16:32:40 [INFO] PreBuild.ps1: ---------------------------------------------------------------------
2025-01-17 16:32:40 [INFO] PreBuild.ps1: Processing environment [1/1]: ******
2025-01-17 16:32:40 [INFO] PreBuild.ps1: ---------------------------------------------------------------------
2025-01-17 16:32:40 [INFO] PreBuild.ps1:  
2025-01-17 16:32:40 [INFO] PreBuild.ps1: Merging basic config with environment-specific config
2025-01-17 16:32:40 [INFO] PreBuild.ps1: Exporting Original ConfigData to file
2025-01-17 16:32:40 [INFO] PreBuild.ps1:   Sorting and exporting data on keys: Priority
2025-01-17 16:32:42 [INFO] PreBuild.ps1:  
2025-01-17 16:32:42 [INFO] PreBuild.ps1: Replacing tokens in ConfigData
2025-01-17 16:32:42 [INFO] PreBuild.ps1: - Token Replaced:  Forest_Code           <-  "******"
2025-01-17 16:32:42 [INFO] PreBuild.ps1: - Token Replaced:  Tenant_Name           <-  "******"
2025-01-17 16:32:42 [INFO] PreBuild.ps1: - Token Replaced:  Tenant_ShortName      <-  "***"
2025-01-17 16:32:42 [INFO] PreBuild.ps1: - Token Replaced:  TenantGuid            <-  "21ebac7a-****-****-****-65a5a92e9a7c"
2025-01-17 16:32:42 [INFO] PreBuild.ps1: - Token Replaced:  TenantId              <-  "****.onmicrosoft.com"
2025-01-17 16:32:42 [INFO] PreBuild.ps1: Exporting Tokenized ConfigData to file
2025-01-17 16:32:43 [INFO] PreBuild.ps1:  
2025-01-17 16:32:43 [INFO] PreBuild.ps1: Testing for presence of all Required parameters in merged configuration data
2025-01-17 16:32:43 [INFO] Test-M365DSCPowershellDataFile: Test(s) selected: Required
2025-01-17 16:32:43 [INFO] Test-M365DSCPowershellDataFile: Load Example data from module M365DSC.CompositeResources
2025-01-17 16:32:43 [DEBUG] Test-M365DSCPowershellDataFile: ^ Elapsed: 0.0898294 Seconds              TotalTime: 0.0900976 seconds
2025-01-17 16:32:43 [INFO] Test-M365DSCPowershellDataFile: Create Hashtables for reference data 
2025-01-17 16:32:55 [DEBUG] Test-M365DSCPowershellDataFile: ^ Elapsed: 11.9182929 Seconds             TotalTime: 12.0085133 seconds
2025-01-17 16:32:55 [INFO] Test-M365DSCPowershellDataFile: Create pester rules
2025-01-17 16:32:56 [DEBUG] Test-M365DSCPowershellDataFile: ^ Elapsed: 1.1250578 Seconds              TotalTime: 13.1336808 seconds
2025-01-17 16:32:56 [INFO] Test-M365DSCPowershellDataFile: Execute pesterscript
Pester v5.7.1

I require guidance what I need to do to get rid of this error.

[martincaddick]

You've got a way to go yet Paul. Easiest thing to do at the moment is make all of the mandatory files blank. You'll have plenty of time to work on that bit later.

[pmatthews05]

Hi @martincaddick,
Thank you for your reply.

However, I'm confused. I would have thought that by taking the repo exactly as it is, I would be able to deploy a basic setup, by following the whitepaper. So far I've hit a couple of roadblocks that has required me to change the files/pipelines to move onto the next step.

Can you explain to me what in the mandatory files is causing the error Cannot index into a Null array?
I also don't understand what is causing the error [ERROR] Environment configuration for '******' contains Mandatory settings, which is not allowed! I thought it merged Mandatory setting and environment settings together?

I can just make the Mandatory files blank, but if I don't understand why this is happening, I'm not going to avoid this issue again in the future.

[martincaddick]

From 5.1.1 in the pdf.
Mandatory
All data files that contain settings that are mandatory for all
environments. Settings in this file must be present in the Basic file and
cannot be specified in any of the Environment specific files.
These two requirement are validated using unit tests.

So, if Mandatory says
Setting = Value
then it must be set like that in Basic AND not be set to something different in the Environment files.

There are bugs in the repo(s) which I and a few others are working thru, however, persevere because it is worth it and you will learn a heap along the way. It's worthwhile reading thru all of the issues including the pending pull request. I spent days trying to unxderstand why I was getting stuck on the deployment.yaml / yml file issue in postbuild.ps1

[pmatthews05]

@martincaddick thank you for pointing step 5.1.1 out. I must have skimmed over that. I will continue with this knowledge.

I personally think Mandatory is being used incorrectly in it's logic, as if all files have to be in Basic and cannot be in Environment, why not just say all Basic are mandatory? ( I understand this probably wasn't your decision).

I feel that the logic should be as follows:

• Basic as the baseline
  • All environments will have this setting, the same settings. (It is Mandatory and will be deployed for all environments, and no need to put these settings in the Environments).
• Mandatory
  • These are the "variables" that you must set in all environments, as they are different for each environment, but required, a bit like a variable file in terraform, the values that are set in Mandatory are default values.
• Environments
  • Any other values you wish to set can be set in Environment, but they are not mandatory between environments. These are unique to the environment.

I work in multi tenant environment which is growing, hence why I believe the logic should be the above.

[martincaddick]

@pmatthews05
Look up the Serenity Prayer ;-)
For me personally, 'Basic' means 'Enterprise' but I can live with it.

On a more practical note,

Try this in your Mandatory Exchange file

@{
    NonNodeData = @{
        Exchange = @{
            MailTips = @{
                MailTipsAllTipsEnabled = $True
                MailTipsExternalRecipientsTipsEnabled = $True
                MailTipsGroupMetricsEnabled = $True
                MailTipsMailboxSourcedTipsEnabled = $True
            }
        }
    }
}

This in your Basic Exchange file

@{
    NonNodeData = @{
        Exchange = @{
            MailTips = @{
                MailTipsAllTipsEnabled = $True
                MailTipsExternalRecipientsTipsEnabled = $True
                MailTipsGroupMetricsEnabled = $True
                MailTipsLargeAudienceThreshold = 25
                MailTipsMailboxSourcedTipsEnabled = $True
            }
        }
    }
}

and finally this in your Environment Exchange file

@{
    NonNodeData = @{
        Exchange = @{
            MailTips = @{
                MailTipsLargeAudienceThreshold = 50
            }
        }
    }
}

Your security people has said you absolutely must have those settings, at your enterprise level (basic) all of your tenants will comply with the security settings and will have some preferred defaults and then the specific environment will be able to customise the settings that it can without going against the security requirements.

[ykuijs]

Hi @pmatthews05, @martincaddick, sorry for not replying sooner. Have been a little busy. Great to see you are supporting each other solving these issues!

To provide you with some more insights:

Data files setup

• The Basic layer are all settings that should apply to all tenants. Everything you configure here will be used for all tenants.
• The Environment layer are the settings that are specific for an environment. Either because these are environment specific, like a UL for example, or because you want to override a setting from the Basic layer (SettingX = 1 in Basic, but for this environment it should be SettingX = 2)

These two layers are merged into a single data set which is then used to compile the MOF file for the environment.

The Mandatory layer are all settings that should be set exactly as specified in these files. In the past the solution just merged this on top of the merge of the Basic/Environment overriding all settings that were set in the other two layers. But then I had a discussion with someone that had an interesting view:

• When using this merge approach, it is possible that you set "SettingX = 1" in the Basic layer, "SettingX = 2" in the Environment layer and "SettingX = 3" in the Mandatory layer. So there are three different values of SettingX and no matter what you set on a low level, the Mandatory will always win.
• Of course this works, but can also cause confusion: I change SettingX to the value of 4 in the Environment layer, but when deploying that to my environment, the setting isn't changed to 4.

Because of this, we decided to change the merge approach to a unit test approach. Now the Mandatory layer is validated against:

• The Basic layer, requiring all settings in the Mandatory layer to be configured exactly as specified in the Mandatory layer.
• The Environment layer, preventing all settings in the Mandatory layer from being overridden with different values.

Using this method, the values in the data files are always consistent with the current state of the environment. Which was not the case with the merge method.

I have also added this explanation to paragraph 5.1.1 of the whitepaper, which will be release soon.

Files in repository

I have checked the files in the repository and found that the Mandatory file for Teams contained information. I have now removed that. So if you start from scratch, it should not start complaining about mandatory settings. These updates will be push shortly.