Special characters encoding issue

[bartvermeersch]

Can someone point me whether this issue is caused by the CICD process or by Microsoft365DSC itself?

When applying this DSC, the é character becomes é

@{
    Config_AzureAD = @{
    }
    NonNodeData = @{
        AzureAD = @{
            Users = @(
            @{
                UniqueID            = "Fred.Smith2"
                UserPrincipalName  = "Fred.Smith2@{{TenantId}}"
                FirstName          = "Frédéric"
                LastName           = "Smith"
                DisplayName        = "Frédéric J. Smith"
                Ensure             = "Present"
            }

            )
        }
    }
}

Frédéric is rendered as Frédéric

:32.7580117Z VERBOSE: [fv-az899-25]: LCM:  [ Start  Resource ]  
:32.7590683Z [[AADUser][email protected]::[AzureAD]AzureAD_Configuration]
:32.7595264Z VERBOSE: [fv-az899-25]: LCM:  [ Start  Test     ]  
:32.7601767Z [[AADUser][email protected]::[AzureAD]AzureAD_Configuration]
:32.8209316Z VERBOSE: [fv-az899-25]:                            
:32.8220614Z [[AADUser][email protected]::[AzureAD]AzureAD_Configuration] Testing configuration of 
:32.8225314Z Azure AD User [email protected]
:32.8235164Z VERBOSE: [fv-az899-25]:                            
:32.8239543Z [[AADUser][email protected]::[AzureAD]AzureAD_Configuration] Getting configuration of 
:32.8246096Z Office 365 User [email protected]
:33.7070529Z VERBOSE: [fv-az899-25]:                            
:33.7081710Z [[AADUser][email protected]::[AzureAD]AzureAD_Configuration] Getting Office 365 User 
:33.7085204Z [email protected]
:33.7500688Z VERBOSE: [fv-az899-25]:                            
:33.7512909Z [[AADUser][email protected]::[AzureAD]AzureAD_Configuration] The specified User doesn't 
:33.7517496Z already exist.
:33.7528349Z VERBOSE: [fv-az899-25]:                            
:33.7533540Z [[AADUser][email protected]::[AzureAD]AzureAD_Configuration] Current Values: 
:33.7611257Z DisplayName=$null
:33.7633370Z Ensure=Absent
:33.7657875Z FirstName=$null
:33.7665297Z LastName=$null
:33.7760972Z UserPrincipalName=$null
:33.7763235Z VERBOSE: [fv-az899-25]:                            
:33.7779000Z [[AADUser][email protected]::[AzureAD]AzureAD_Configuration] Target Values: 
:33.7865617Z DisplayName=Frédéric J. Smith
:33.7889723Z Ensure=Present
:33.7903915Z FirstName=Frédéric
:33.7917125Z LastName=Smith
:33.7976809Z [email protected]
:33.7992970Z Verbose=True
:34.3136176Z VERBOSE: [fv-az899-25]:                            
:34.3149301Z [[AADUser][email protected]::[AzureAD]AzureAD_Configuration] Test-TargetResource returned 
:34.3154633Z False
:34.3165195Z VERBOSE: [fv-az899-25]: LCM:  [ End    Test     ]  
:34.3176093Z [[AADUser][email protected]::[AzureAD]AzureAD_Configuration]  in 1.4850 seconds.
:34.3186687Z VERBOSE: [fv-az899-25]: LCM:  [ Start  Set      ]  
:34.3197438Z [[AADUser][email protected]::[AzureAD]AzureAD_Configuration]
:34.3213652Z VERBOSE: [fv-az899-25]:                            
:34.3219230Z [[AADUser][email protected]::[AzureAD]AzureAD_Configuration] Setting configuration of 
:34.3225629Z Office 365 User [email protected]
:34.3912936Z VERBOSE: [fv-az899-25]:                            
:34.3924482Z [[AADUser][email protected]::[AzureAD]AzureAD_Configuration] Getting configuration of 
:34.3928188Z Office 365 User [email protected]
:34.5647666Z VERBOSE: [fv-az899-25]:                            
:34.5659024Z [[AADUser][email protected]::[AzureAD]AzureAD_Configuration] Getting Office 365 User 
:34.5662899Z [email protected]
:34.6125073Z VERBOSE: [fv-az899-25]:                            
:34.6136531Z [[AADUser][email protected]::[AzureAD]AzureAD_Configuration] The specified User doesn't 
:34.6141422Z already exist.
:35.0528304Z VERBOSE: [fv-az899-25]:                            
:35.0544581Z [[AADUser][email protected]::[AzureAD]AzureAD_Configuration] Creating Office 365 User 
:35.0548699Z [email protected]
:35.0591231Z VERBOSE: [fv-az899-25]:                            
:35.0596808Z [[AADUser][email protected]::[AzureAD]AzureAD_Configuration] Creating new user with 
:35.0612018Z values: AccountEnabled=True
:35.0659483Z DisplayName=Frédéric J. Smith
:35.0672178Z GivenName=Frédéric
:35.0759288Z UserPrincipalName=Fred.Smith2@xxxxx
:35.5288503Z VERBOSE: [fv-az899-25]: LCM:  [ End    Set      ] 

I checked the psd1 file and it is UTF8withBOM

Preamble          :
BodyName          : utf-8
EncodingName      : Unicode (UTF-8)
HeaderName        : utf-8
WebName           : utf-8
WindowsCodePage   : 1200
IsBrowserDisplay  : True
IsBrowserSave     : True
IsMailNewsDisplay : True
IsMailNewsSave    : True
IsSingleByte      : False
EncoderFallback   : System.Text.EncoderReplacementFallback
DecoderFallback   : System.Text.DecoderReplacementFallback
IsReadOnly        : True
CodePage          : 65001

[ykuijs]

Hi @bartvermeersch, is this change also reflected in the object in Microsoft 365 or just in the logging?

Background: This is caused by different encoding of files. Different applications use different encodings, resulting in characters being interpreted incorrectly. CodeX means something different in a different encoding.

[bartvermeersch]

Also in the object I'm afraid.

[bartvermeersch]

I got a bit further, all files are UTF8BOM, but the generated MOF is UTF-16 LE

[bartvermeersch]

I'm stuck, I added some logging:

Write-Log -Object (Get-Content -Path (Resolve-Path -Path $finalPsdPath).Path -Raw)
At the end of the prebuilt script, it returns the é characters correctly (D:\a\1\s\Output\envxxx\envxxx.psd1)

At the beginning of the built script, the same file returns é characters?

What am I missing?

[bartvermeersch]

I was able to fix the encoding of the MOF by removing

pwsh: true

from the Pre Build task.

https://github.com/ykuijs/M365DSC_CICD/blob/9035794433e9e0eb43228f309055599ffe4ca3e7/Pipelines/build-template.yaml#L66C11-L66C21

Unfortunately, the final outcome remans wrong. In deploy.ps1 the MOF is UT8 and contains é characters (perfect), but the Start-DscConfiguration returns é instead...

[ykuijs]

Hi @bartvermeersch, this is an issue that I have seen several times before. File encoding is a recurring problem unfortunately.

However I just had a call with @FabienTschanz. He has done some investigation on this issue and found a possible solution, which he shared here:

• New-M365DSCReportFromConfiguration: en dash in EXOGroupSettings DisplayName causes DSCParser exception microsoft/Microsoft365DSC#5202 (comment)
• New-M365DSCReportFromConfiguration: en dash in EXOGroupSettings DisplayName causes DSCParser exception microsoft/Microsoft365DSC#5202 (comment)

Could you please check if the solution shared there works for you? We are thinking about adding some logic to the code that checks for this solution and give a warning if that is not configured correctly. But first want to be sure it really solves the issue. So it would be much appreciated if you could test it on your end.