Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Update-M365DSCDependencies: ... does not support PowerShell Core #5433

Open
fasteiner opened this issue Nov 20, 2024 · 39 comments
Open

Update-M365DSCDependencies: ... does not support PowerShell Core #5433

fasteiner opened this issue Nov 20, 2024 · 39 comments

Comments

@fasteiner
Copy link
Contributor

Description of the issue

When running the update Dependencies command in PowerShell Core this happens:
14:22:46 WARNING: The dependency {Az.Accounts} does not support PowerShell Core. Please run Update-M365DSCDependencies in Windows PowerShell.
14:22:46 WARNING: The dependency {Az.ResourceGraph} does not support PowerShell Core. Please run Update-M365DSCDependencies in Windows PowerShell.
14:22:46 WARNING: The dependency {Az.Resources} does not support PowerShell Core. Please run Update-M365DSCDependencies in Windows PowerShell.
14:22:46 WARNING: The dependency {Az.SecurityInsights} does not support PowerShell Core. Please run Update-M365DSCDependencies in Windows PowerShell.
14:22:46 WARNING: The dependency {ExchangeOnlineManagement} does not support PowerShell Core. Please run Update-M365DSCDependencies in Windows PowerShell.
14:22:48 WARNING: The dependency {Microsoft.PowerApps.Administration.PowerShell} does not support PowerShell Core. Please run Update-M365DSCDependencies in Windows PowerShell.
14:22:48 WARNING: The dependency {MicrosoftTeams} does not support PowerShell Core. Please run Update-M365DSCDependencies in Windows PowerShell.

All of this modules are PowerShell Core compatible, therefore you must have a mistake in your dependencies.

Also, this requires me to install the module as administrator, which is not favorable and also I wonder why the Scope switch is being ignored again?

I'll try downgrading now, but this needs to be fixed!

Microsoft 365 DSC Version

1.24.1113.1

Which workloads are affected

other

The DSC configuration

n/a

Verbose logs showing the problem

n/a

Environment Information + PowerShell Version

OsName               : Microsoft Windows Server 2019 Standard
OsOperatingSystemSKU : StandardServerEdition
OsArchitecture       : 64-bit
WindowsVersion       : 1809
WindowsBuildLabEx    : 17763.1.amd64fre.rs5_release.180914-1434
OsLanguage           : en-US
OsMuiLanguages       : {en-US}

PS C:\Users\svsteiner>
PS C:\Users\svsteiner> $PSVersionTable

Name                           Value
----                           -----
PSVersion                      7.4.6
PSEdition                      Core
GitCommitId                    7.4.6
OS                             Microsoft Windows 10.0.17763
Platform                       Win32NT
PSCompatibleVersions           {1.0, 2.0, 3.0, 4.0…}
PSRemotingProtocolVersion      2.3
SerializationVersion           1.1.0.1
WSManStackVersion              3.0
@fasteiner
Copy link
Contributor Author

last working version: 1.24.626.1

@FabienTschanz
Copy link
Collaborator

@fasteiner This is not a problem but expected behavior. Microsoft365DSC only supports the use case where the PowerShell modules are installed under C:\Program Files\WindowsPowerShell\Modules... path. This is due to internal restrictions in using the Cmdlets, but you can still use PowerShell 7 without problems. The only thing you have to take into consideration is to update the modules from inside Windows PowerShell (running as an administrator).

We do understand that this might not be beneficial for all, but it's the only supported way to use Microsoft365DSC.

@fasteiner
Copy link
Contributor Author

@FabienTschanz Up to version 1.24.626.1, this module was fully supported in PowerShell Core, which aligned well with our automation environment. However, with its integration into our PowerShell 7-only workflows, we are now faced with the challenge of transitioning away from this module due to the decision to discontinue support for PowerShell Core.

This decision is perplexing, given that the module had already achieved full compatibility with PowerShell 7. Moving backward in compatibility undermines advancements and forces us to sunset this module in favor of developing our own implementation—a decision that is both time-consuming and resource-intensive.

We encourage reconsideration of this approach to ensure continuity and compatibility for users who rely on modern PowerShell environments.

@FabienTschanz
Copy link
Collaborator

FabienTschanz commented Nov 23, 2024

@fasteiner As a fellow engineer, I fully understand your situation. I personally am also not happy with that, so I'll take it up on myself and check what we from the community can do to make it fully compatible with PowerShell Core. But this will take time since I'm only working on this module in my own free time as a contributor and not a fully-fledged member.

One thing I would like to mention here: I was the one who updated the module to "only" allow installation of the required modules within Windows PowerShell. This is because of some dependencies and statements that Microsoft365DSC requires the modules to be installed in the Windows PowerShell module directory. Why that is, and what this requirement is still a hard dependency, is something I'm not aware of. I will take a look and start my preparations.

One question at last: Running Update-M365DSCDependencies in Windows PowerShell installs all the modules there, and then they are also available inside of PowerShell Core because PowerShell Core considers modules from the Windows PowerShell. For now, isn't it possible for you to simply run the update command in Windows PowerShell and afterwards switch to PowerShell Core?

@fasteiner
Copy link
Contributor Author

@FabienTschanz
that's nice to hear!

Ok, but the module worked before that requirement was introduced (at least in PowerShell 7), would it make sense to revert back to the state where this was not required to be installed in Windows PowerShell?

regarding your question:

I tried that unfortunately, running in PowerShell Core still was not possible:
Image

Therefore, I switched to an older version.

@FabienTschanz
Copy link
Collaborator

FabienTschanz commented Nov 24, 2024

@fasteiner Backwards compatibility is essential for modules this big. It's not easy to introduce something new without breaking things in a completely different location. The best example is PnP.PowerShell, the current module version requires PowerShell 5.1 and does not work with PowerShell 7, however if we were to update to a newer version, we would kill support for PowerShell 5.1 and require e.g. 7.2. That has to be done with a breaking change release and is not feasible unless we introduce new major functions... Currently, that's not in sight.

What I think could work: If we are running in PowerShell Core and no modules are present, we can install them in the current environment (except of PnP.PowerShell which requires it to be installed in PowerShell 5.1). I'm not sure how that would be handled by PowerShell Core if the modules are both present in 5.1 and Core, that's something that we need to take a look at.

Edit: What's the output of Get-Module XYZ -ListAvailable when run in PowerShell Core, XYZ to be replaced with the modules that need updating? Were they all installed within PowerShell 5.1?

@FabienTschanz
Copy link
Collaborator

@fasteiner With the latest release, you'll only have to install the PnP.PowerShell module in version 1.12.0 in the Windows PowerShell context. The other modules can be installed in PowerShell Core (using Update-M365DSCDependencies or Update-M365DSCModule).

@Borgquite
Copy link
Contributor

@FabienTschanz I presume you’re aware the LCM (which provides the ‘configuration engine’ for Microsoft365DSC and is triggered by the Start/Test/Set-DscConfiguration cmdlets) is a Windows PowerShell 5.1 feature as yet unsupported in PowerShell Core and therefore until this is replaced I understand any part of the project that depends on the LCM will have a 5.1 dependency?

PowerShell/PowerShell#12724
https://learn.microsoft.com/en-us/powershell/dsc/overview?view=dsc-2.0
PowerShell/DSC#529

@Borgquite
Copy link
Contributor

Borgquite commented Dec 5, 2024

@FabienTschanz P.S. If prerequisite modules are only installed in C:\Program Files\PowerShell\7\Modules (not C:\Program Files\WindowsPowerShell\Modules) I suspect any functionality relying on the LCM will no longer work.

@FabienTschanz
Copy link
Collaborator

@Borgquite To be perfectly honest, at the time when I investigated this, I was not aware of the fact that the LCM requires PowerShell 5.1. I will check on my other machine if that actually is the case.

@Borgquite
Copy link
Contributor

Borgquite commented Dec 5, 2024

@FabienTschanz Sure. As I understand it:

@Borgquite
Copy link
Contributor

Borgquite commented Dec 5, 2024

@ykuijs
Copy link
Member

ykuijs commented Dec 6, 2024

The issue @fasteiner is describing has been introduced because of a change to support PSv7 earlier this year. An extra parameter was added to the dependencies file to specify if a dependency had to be installed in PSv7 or not. As it turns out, this has issues and needs to get fixed 😄

As @Borgquite mentions, DSC still has some requirements on PSv5.1: Whenever you are using the LCM (all *-DscConfiguration cmdlets), you will use PSv5.1 since the LCM is built on top of that version. However all functions that do not use the LCM will (or better yet should) work in both PSv5.1 and PSv7.

@Borgquite
Copy link
Contributor

Borgquite commented Dec 6, 2024

In terms of making use of PowerShell Core and PnP.PowerShell 2.x, has any thought been given to using syntax like this in the relevant modules?

$codetorun = {
    Write-Output $PSVersionTable
}

if ($PSVersionTable.PSVersion.Major -le 5)
{
    pwsh $codetorun
}
else
{
    & $codetorun
}

Would need us to be quite specific about what gets installed where i.e.

  • Install PowerShell Core with PnP.PowerShell module (system-wide, for LCM)
  • Install remaining modules in Windows PowerShell 5.1 (system-wide)

But might allow the use of the latest PnP.PowerShell module without giving up the LCM?

@FabienTschanz
Copy link
Collaborator

I thought about a similar thing, why don't we switch the entire stack of Microsoft365DSC to PowerShell Core once and for all and introduce a "PowerShell" runspace management so we can, similar to the LCM, run the code fast without always spinning up a whole new PS session where the module imports are necessary? So the entry point of Microsoft365DSC and the functions is a kind of "proxy" access to the underlying PowerShell Core implementation.

I'd be glad to try some things out there.

@Borgquite
Copy link
Contributor

Borgquite commented Dec 10, 2024

@FabienTschanz Sorry, not quite sure about the details of what you're saying here? Are you saying that you want to run all the internals of Microsoft 365 DSC in PowerShell Core, with a Windows PowerShell 5.1 compatibility layer in the Get-, Set- and Test-TargetResource functions to allow them to be used by the LCM?

Or are you wanting to get rid of the LCM dependency? Be aware that there's no real on-premises LCM alternative in PowerShell Core for configuring resources or performing drift/change management at present, and one of the really useful things about this project is the ability to use it alongside other DSC resources to manage a hybrid environment using Windows PowerShell 5.1 and the LCM, so I'd vote to retain 5.1 compatibility unless a new LCM is released.

@FabienTschanz
Copy link
Collaborator

@Borgquite I‘m refering to a „compatibility“ layer as you named it. I definitely do not intend to get rid of the LCM since it‘s the „behind the scenes“ actor that keeps DSC going for as long as we like, so we need it for an unknown time period until DSCv3 is released and does its thing. At that point, we‘ll be able to adopt the new technology. But for now, I‘d probably look for something similar in technology in what the LCM does (with its own runspaces where the resources are run in) and try to adopt a layer that enables us to use PowerShell Core inside of the LCM engine.

@Borgquite
Copy link
Contributor

@FabienGilbert Ah, understood. I guess I can see advantages and disadvantages to moving everything to a PowerShell Core layer:

Pros

  • More consistency, predictable behaviour between modules
  • Ability to new PowerShell Core features
  • No need to worry about any other modules going PS Core only
  • Simpler installation requirements (install PS Core, install all modules in PS Core directories)

Cons

  • Additional prerequisites (at present, there's no need to install PS Core if you don't want to, and in restricted environments, that can be a problem)
  • Performance - the compatibility layer will probably have a performance impact for now
  • Additional code complexity, to provide the compatibility layer

I could see benefits to what you're suggesting. Perhaps start with trying to migrate the PnP.SharePoint resources over in a clean way, and see how that goes?

@FabienTschanz
Copy link
Collaborator

If @ykuijs agrees in migrating PnP.PowerShell over to PowerShell Core we can give it a try. What do you think, or @NikCharlebois? We should probably consider moving some core stuff over and start preparing a possible shift for the future. Laying the groundwork for the next generation of Microsoft365DSC 😄

@ykuijs
Copy link
Member

ykuijs commented Dec 10, 2024

I would love to migrate everything to PSv7, however since the LCM still runs on PSv5.1 this is not possible yet.

It would be a possibility to implement a check on the PowerShell version and then use a different version of PnP.PowerShell. But that would mean that the resources become much more complex since in v2 cmdlets might have changed names and/or parameters. In case of using PS Core (e.g. in an export) we would have to use the PnP v2 cmdlets and in case of using PSv5.1, we would have to use the v1 cmdlets. Also the prerequisites would both have to be installed.

So right now, I do not think that is a feasible and usable option. Towards the future with DSCv3 however, that would be an option. Since in that version the LCM no longer exists and you can use PSv7 to apply and test the state of the resources.

@NikCharlebois Agree with this?

@Borgquite
Copy link
Contributor

Borgquite commented Dec 15, 2024

If we did it right, I think @FabienTschanz is suggesting resources can run in PowerShell Core under the hood, called via PS 5.1 in the LCM, so the code would be simpler. I would prefer this myself (retaining PS 5.1 compatibility with the LCM). There’s nothing being added in DSCv3 that we don’t already have in v2 as far as I am aware - if we want to test and set multiple resources, report, and monitor and correct drift, we basically have to reimplement the LCM, as Microsoft won’t be doing it for us.

Personally I think Microsoft have made a big mistake in making DSCv3 so incompatible and toothless compared to what went before - from what I hear, the expectation is that people will use other ‘higher order tools’ (read, Ansible, Puppet, Chef) to do what the LCM currently does. So we either have it build our own Ansible/Puppet/Chef ‘lite’ in PS7 - or we could try to retain PS 5.1 LCM compatibility, which I’d prefer.

PowerShell/DSC#529

From a personal note in my position (a nonprofit working in countries where the cloud is unreliable and impractical) using the LCM is really important - we need on-premises support for LOB apps things like Windows Server but still need some compatibility with stuff like Microsoft365DSC to cross-manage a limited hybrid cloud footprint for services like email. I’d love to see Microsoft365DSC move ahead in a way that allows us to continue to easily leverage it in our hybrid environment, both on-premises and in-cloud, without having to splash out on expensive cloud LCMs like Azure Automation, or third party ‘higher order’ tools, which don’t add any value or cost benefit, or aren’t practical for anyone who has to remain in hybrid for the foreseeable future. Or is the plan that we reimplement the missing LCM features in PowerShell Core once DSCv3 is release?

@FabienTschanz
Copy link
Collaborator

@Borgquite You exactly hit the nail on the head with your statement and what I thought of creating. Since the LCM will be falling apart for PS7, a replacement technology is needed sooner or later - I'm aware that PS5.1 won't be dying anytime soon, so the LCM will continue to work, but it's not understandable why there is no replacement technology available after years of PowerShell being developed independently from the OS.

Any other tools like Ansible, Puppet, Azure Config Manager etc. are all good and so, but if there is no support pretty much out of the box, I'm almost certain that customer will migrate away from this solution if setup and integration with a new technology proves to be of the same effort as redoing your configuration with, let's say, Terraform or any other solution that includes M365 configuration.

In my opinion, if Microsoft is not doing anything to replace the LCM, it should be part of the Microsoft365DSC tool to independently start working on a solution to address this issue. It will later come bite us in a place where we don't want to get bitten. There are so many new features in PS7 we could use when the migration is complete. From massive performance improvements to a streamlined platform, without dependency issues and so on. I believe the work we have to put in to create such a solution to be a worthy investment when it enables us to continue forward with big steps, especially now that the technology stack is available, many issues fixed and from what I see, a confident group of Microsoft employees and development teams working on a project I sometimes thought won't be able to sustain on the market. I have high hopes for this project and love the collaboration between the members, we should really use that to our advantage and create something awesome.

@Borgquite
Copy link
Contributor

Borgquite commented Dec 16, 2024

@FabienTschanz Indeed - based on the above, we shouldn't expect a full replacement for LCM until DSCv3.1 (and then, it only something that 'might be discussed'). I was slightly incorrect before - looks like you will be able to create a full configuration in DSCv3, but I doubt there will be periodic reporting, monitoring for drift detection, or automatic remediation as under the LCM for 1.1, all of which currently are available and supported features inside Microsoft365DSC. Given DSCv3's configuration file format switches to YAML as well, I'm not sure how the existing Export/ReverseDSC stuff will work based on what looks like a lack of support for exporting in classic PowerShell resources at the moment - you're looking at a ton of probably breaking changes to get Microsoft365DSC fully ported to DSCv3. (PowerShell/DSC#73 and PowerShell/DSC#171)

At the moment Microsoft are recommending that existing DSC 1.1 existing configurations should not try to migrate to v3. Given that shouldn't we be looking at getting Microsoft365DSC playing nicely with DSC 1.1 and modules which only support PowerShell Core 7 now, instead of resting all our hopes on a potential future release of DSCv3.1 which may include an LCM, but which may or may not actually get here?

@Borgquite
Copy link
Contributor

Borgquite commented Feb 13, 2025

Just to add something else here - in case we are looking forward to DSCv3 as the potential solution. As I understand it within the DSC community there has been some discussion over support for PowerShell 'script-based' / 'classic' / 'MOF-based resources' vs newer 'class-based' resources. Just to be clear, Microsoft365DSC appears to be primarily script-based at this time.

Thankfully from my perspective, the outcome of that discussion was 'while DSCv3 represents a major change to the DSC platform, DSCv3 is able to invoke PowerShell DSC Resources, including script-based and class-based DSC Resources, as they exist today', so Microsoft365DSC should still work as-is with DSCv3.

However, looking at the latest DSCv3 release notes, support for 'classic' (script-based) resources appears to now be limited to only the Microsoft.Windows/WindowsPowerShell adapter, and not the Microsoft.DSC/PowerShell adapter. If I understand correctly, this means that under DSCv3, classic (script-based) resources like Microsoft365DSC will still only run on Windows PowerShell 5.1, not PowerShell Core 7+.

To me this leaves two viable options that I can think of:

  • Rewrite the entire Microsoft365DSC module as class-based resource to allow continued use of PS7 modules after DSC v3 release.
  • Or implement the suggestion from me and @FabienTschanz to create a compatibility layer allowing cmdlets from PS7 modules to be called 'under the hood', when the underlying resource is executing on WPS5.1 (as it will on DSC 1.1 and 3.0). This means that the resource could continue to be used in DSC 1.1, 2.0 and 3.0 in the forseeable future, including the LCM, without having to put in a major refactoring effort.

@FabienTschanz
Copy link
Collaborator

Rewriting everything to class-based resources would be a real pain, but it probably needs to happen sooner or later. I don't know if we can mix class-based and script-based resources with DSCv3 (but probably yes from what there is written), but personally, I would refrain from this unless we can build a bigger prototype and compare entire workloads from DSCv1 with DSCv3.

My suggestion is still the compatibility layer for PS5.1 up to 7 which will be an alternative to the LCM. Though that would take a lot of time to build as well, it's probably a longer term solution until we can purely switch to DSCv3 with PowerShell 7.

@Borgquite
Copy link
Contributor

Borgquite commented Feb 13, 2025

You can certainly mix class-based and script-based resources now (PSResourceRepository within ComputerManagementDsc is an example of that) but I'd be concerned about the work needed in terms of testing and debugging (described fairly well by @ryanspletzer in the issue above) - perhaps things have improved since then, but most DSC resources I see are still well and truly 'stuck' in script-based resources, with very, very few contributors willing to work on much-requested feature improvements or much-needed bug fixes even to them, let alone up for refactoring entire modules, unit tests, integration tests etc.

This resource is a bit different as it has Microsoft employees in it but to me switching from script-based to class-based is a classic time-consuming sink hole of programmers' time - a lot of work for no real benefit at the end for actual end-user SysOps / DevOp roles who just need the module to work for their jobs (from my point of view as both a programmer and an end-user).

It's what the guy who created VBA and co-founded Trello and Stack Overflow (a former Microsoft employee, and a pretty successful guy) would call classic 'fire and motion' and I'd say time spent rewriting to class-based would be much better spent implementing some sort of compat layer and bug fixing / implementing new resources.

@salbeck-sit
Copy link
Contributor

@Borgquite @FabienTschanz Interesting thread.
I'd concur with the idea that a replacement for the LCM needs to be 'invented'. This could be done via a runspace-based approach with the critical issue being that global- and module-/script-level variables - and thereby connections - can survive from one resource to the next (runspaces can do just that, see https://learn-powershell.net/2013/04/19/sharing-variables-and-live-objects-between-powershell-runspaces/ - unsure if this is still supported i Pwsh). Otherwise it would be possible to utilize https://github.com/microsoft/CloudConfigurationManager
The base script/function would need to be able to consume a DSC Configuration and call Test/Get/Set for each included resource, passing global variables to each. It also requires the ability to convert CIMInstance-based parts of the configuration to parameters in such calls. In short, mimicking what the LCM does today (perhaps except for the variation in apply-methods)

@FabienTschanz
Copy link
Collaborator

@Borgquite I suggest we use the Cloud Configuration Manager as mentioned by @salbeck-sit. It already contains a bunch of functions that wrap nicely around the .ps1 file and allow for Test and Set to be called, e.g. Start-CCMConfiguration and Test-CCMConfiguration.

The caveat is that it doesn't support MOF-based files, only .ps1 files. But there is no dependency on the LCM, so we could potentially introduce a massive performance boost if we enable the usage of PowerShell 7. Still, some thoughts need to be put in the dependency management so that we can support both Windows PowerShell as well as PowerShell 7.

For me, I'd first start thinking about how the Cloud Configuration Manager integrates into the workflow of us users, add missing functionality to it (e.g. validating a configuration, maybe support for MOF-files (well probably not)...), and what other things come to mind.

What do you think? Do we want to go with this approach? If yes, who would like to work on it? As soon as we know which direction we will take, let's open a different issue and not "misuse" this one 👀

FYI @NikCharlebois @ykuijs

@Borgquite
Copy link
Contributor

Borgquite commented Feb 27, 2025

@Borgquite Had a look at the Cloud Configuration Manager. If I'm honest, there are a few issues I'm unclear about.

It seems to take a very Microsoft365DSC-centric approach to configuration. I presume it works beautifully with the .ps1 files generated by Microsoft365DSC itself which are 'just' a series of configurations - I don't think it supports any of the dynamic configuration functionality included in the DSC 1.1 engine. That might mean it is fine for the built-in export/compare requirements of Microsoft365DSC, but couldn't be used as a full replacement for DSC 1.1 for more complex configuration requirements.

In other words it appears very much like an on-premises implementation of DSC 2.0 / Azure Machine Configuration - supporting 'bare' PS1 files, but without parameters, flow control, configuration data, the Node keyword, composite configurations etc. However it appears to be limited at present even compared with DSC 2.0 and AMC in two ways. Firstly, it can only call MSFT_ resources (which again means it's no good for calling other DSC resources alongside Microsoft365DSC)

https://github.com/microsoft/CloudConfigurationManager/blob/8d92e50f2e18a404ad2e8071cc97dfe234d2b9e9/Modules/CloudConfigurationManager/CloudConfigurationManager.psm1#L317

And secondly more interestingly it uses the Get-/Test-/Set-TargetResource functions directly, instead of using DSC 1.1 and 2.0's Invoke-TargetResource, so at present it would only support 'classic' script-based resources, not class-based ones. This has implications when it comes to DSC 3.0, which will only support PS5.1 as a platform for classic script-based resources (see discussion above). So if we want Microsoft365DSC to use PS7 after DSC 3.0 is released, I think still the only options are to implement the PS5.1 -> PS7 compatibility layer, or to switch to class-based resources, regardless of whether we use CloudConfigurationManager or not.

https://github.com/microsoft/CloudConfigurationManager/blob/8d92e50f2e18a404ad2e8071cc97dfe234d2b9e9/Modules/CloudConfigurationManager/CloudConfigurationManager.psm1#L334

As far as I can see it doesn't presently support periodic reporting, monitoring for drift detection (with event logging), or automatic drift remediation either.

Overall I think it has limited usefulness - perhaps it could be used to replace Microsoft365DSC's Export and Compare functionality. But it wouldn't take away the need to keep Microsoft365DSC compatible with DSC 1.1 or 2.0, so that more complex configurations (requiring flow control etc, or mixing Microsoft365DSC with resources from other modules like ActiveDirectoryDsc) can still take place.

@salbeck-sit
Copy link
Contributor

@Borgquite I agree, CCM is a very bare-bones implementation of applying a configuration. This means that building a configuration - ie the script-file containing the completed configuration - is a different thing from applying it where in DSC 1.1 you can mix the two.
I haven't really played with CCM but it's probably the closest thing we have that could replace DSC 1.1.
It will need extra work to provide the features that would be needed to be able to report on drift etc. Microsoft365DSC tackles this by expecting every resource to log 'drift' to an eventlog instead of relying on 'ApplyAndAutocorrect' that the LCM can do in DSC 1.1.
Indeed, the whole Export-aspect of Microsoft365DSC would need to be adressed to be able to provide actual configurations that could be compared to one another (another form of 'drift') and/or cloned to other tenants. Things that I wouldn't want to live without

@Borgquite
Copy link
Contributor

Borgquite commented Feb 27, 2025

Thanks. I'd still say, while admittedly messy, the advantage of just doing PS 5.1 -> PS 7 compatibility layer is the number of problems that it fixes with minimal effort:

  • Want to move to PS7 but no LCM - just use the PS 5.1 LCM
  • Want to use PS7 modules but retain PS 5.1 compatibility - done
  • Want to use PS7 modules but retain DSCv3 compatibility - no need to rewrite as class-based resources

To me it's clear - it solves all the problems with minimal work. None of the other options (move to PS7 and replace LCM with CCM, or move to PS7, replace LCM with CCM including update to use Invoke-TargetResource, switch to class-based resources) solve all of the issues at once.

@FabienTschanz
Copy link
Collaborator

FabienTschanz commented Feb 27, 2025

Thank you so much for the valuable input. Gotta wrap my head around a bit, just to stay clear.

  • If we want to run Test-/Set in PS5.1, then there is a compatibility layer in place which is built on top of PS7. This guarantees that we can use PS7 everywhere, including for all of the dependencies. This is used particularly by the LCM to retain compatibility.
  • For PS7, there is no LCM. Meaning, the only way would be a custom DSC engine which doesn't exist. So no support for calling M365DSC from PS7 directly because everything would need to be rewritten for DSCv3?
  • This would mean, that cross-platform won't be supported for the foreseeable future until we can move to PS7 and M365DSC is compatible with DSCv3 and a DSC engine capable to replace the LCM?

I hope I understood that correctly. Feel free to correct me where I'm mistaken, especially about the PS7 part.

@Borgquite
Copy link
Contributor

Borgquite commented Feb 28, 2025

Yes. On the second bullet point, you could use Cloud Configuration Manager temporarily as a limited LCM on PS7, but it won't fix the problem when DSCv3 comes out. When DSCv3 is released, I think we either need the compatibility layer, or migrate everything to class-based resources, if we want to call PS7 modules inside Microsoft365DSC.

@FabienTschanz
Copy link
Collaborator

Got it. I'll see to create a separate issue with the requirements for this compatibility layer and link this one here. Already looking at how this can be implemented.

@FabienTschanz
Copy link
Collaborator

@Borgquite Guess the "simple" approach with just creating a runspace that runs PowerShell 7 isn't possible. From PowerShell 5.1, I can only start other runspaces that are PS5.1 as well. So I think invoking a command in the Microsoft365DSC module, which in term initializes a PS7 process that accepts the arguments passed to the PS5.1 module from the LCM is probably the way to go. Or do you have another idea?

@Borgquite
Copy link
Contributor

Borgquite commented Feb 28, 2025

@FabienTschanz Oh that's frustrating, how about one of these?

  1. PowerShell Remoting. This is recommended as an option by Microsoft
  • Install PSDesiredStateConfiguration 2.0.7 module in $env:PSModulePath for Windows PowerShell 7 to get Invoke-DscResource working in PSCore 7.
  • Install Microsoft365DSC PowerShell module into $env:PSModulePath for Windows PowerShell 5.1 (so that it's available from both 5.1 and 7)
  • Run Enable-PSRemoting from an elevated PowerShell 7 session
  • Run $PSCoreSession = New-PSSession -ConfigurationName PowerShell.7 from an elevated Windows PowerShell session
  • Then do something like if $PSVersionTable.PSEdition -eq 'Desktop' { Invoke-Command -Session $PSCoreSession -ScriptBlock { Invoke-DscResource -Name <ResourceName> -Method <Get/Set/Test> -ModuleName Microsoft365DSC -Property $PSBoundParameters } } else { do PS7 specific resource code }

It may need $using:PSBoundParameters and any other parameters going through the scriptblock, so they are expanded before being sent to PS7...

You could try to define $PSCoreSession inside a module scope (Microsoft365DSC or maybe MSCloudLoginAssistant) so that maybe it won't need recreating on each resource, for performance?

If you find running Invoke-DscResource doesn't work you could also try to call the Get-/Set-/Test-TargetResource functions directly like CloudConfigurationManager does but perhaps this might make the 'script-based' (rather than class-based) nature of Microsoft365DSC even more entrenched than it already is?

Advantages is that if you can retain a single PSCoreSession, I'd expect it to perform quite well. Disadvantage is that it would require modules installed in the right place and the requirement to run Enable-PSRemoting as an initial step.

  1. Run pwsh.exe direct (discussed in this reddit post)
  • Set up PSDesiredStateConfiguration 2.0.7 and Microsoft365DSC as above
  • Do something like if $PSVersionTable.PSEdition -eq 'Desktop' { pwsh.exe { Invoke-DscResource -Name <ResourceName> -Method <Get/Set/Test> -ModuleName Microsoft365DSC -Property $PSBoundParameters } } else { do PS7 specific resource code }

Same as before re using Invoke-DscResource vs Get-/Set-Test-TargetResource

Advantages are that it's much easier to set up. Disadvantages are that it may not perform as well - you're loading a new PSCore executable inside each resource, including loading all the modules, on each Get/Set/Test run, so performance may suck.

  1. Use new PowerShellProcessInstance - this reddit post has it in reverse (5.1 from 7) but perhaps it can be reversed? Looks like you may end up with a remoting session anyway but perhaps easier to set up? Beyond my abilities though.

@FabienTschanz
Copy link
Collaborator

@Borgquite I took the time and experimented today with Enter-PSSession and New-PSSession and I finally got it working after a couple of hours of kicking WinRM in it's booty... Your suggestion was almost spot-on, very thankful for it. Still took me a long time with WinRM, but here's how I got it working:

  1. Run Enable-PSRemoting -Force - Kinda obvious, but necessary
  2. Open an elevated PS5.1 session
  3. Execute the following commands (to execute a sample resource):
$PSCoreSession = New-PSSession -ComputerName localhost -ConfigurationName PowerShell.7 -EnableNetworkAccess
Invoke-Command -Session $PSCoreSession -ScriptBlock {
    Invoke-DscResource -Name EXOAddressList -Method Get -ModuleName Microsoft365DSC -Property @{Name="Test"}
}

And yeah... that's basically it. It does its job without issues.

Next steps:

  • Testing how $PSBoundParameters handles itself when passed to the session
  • How authentication works with the parameters and if it stays connected
  • How to display Verbose, Warning etc. streams
  • Where to put that logic
  • And how we can update all resources easily

Thanks again, keeping you posted.

@FabienTschanz
Copy link
Collaborator

Alriiiight, great news. I was just able to create a sample module, execute it with PS5.1, have a PS7 session spawn from inside Microsoft365DSC, relay the parameters to that session and invoke the Get-TargetResource function. Here's the flow of it:

Image

Caching behavior of the session works really fine, execution is smooth, except of the initial loading time. Same as with the LCM, the first resource takes a couple of seconds. But other than that, it performs beyond my expectations.

The sample module looks like the following:

Import-Module D:\M365DSC\Microsoft365DSC\Modules\Microsoft365DSC\Microsoft365DSC.psd1 -Force # Only necessary because I didn't copy the updated function definition to the default $PSModulePath

function Get-TargetResource { # These are the normal functions. No changes here.
    [CmdletBinding()]
    param (
        [Parameter(Mandatory)]
        [string]$Name
    )

    Write-Host "Get-TargetResource called with Name: $Name"

    $result = @{
        Name = $Name
        Ensure = 'Present'
        Path = 'C:\Temp'
    }

    Write-Output $result
}

function Test-TargetResource {} # Omitted

function Set-TargetResource {} # Omitted

function Export-TargetResource {} # Omitted

if ($PSVersionTable.PSVersion.Major -lt 6) {
    # Rename the function definition so we can call it later
    Rename-Item -Path function:\Get-TargetResource -NewName Get-TargetResource2

    # Redefine the function. Currently not yet dynamically with only the body replacement
    function Get-TargetResource { 
        [CmdletBinding()]
        param (
            [Parameter(Mandatory)]
            [string]$Name,

            [Parameter()]
            [System.String]
            $ApplicationId,

            [Parameter()]
            [System.String]
            $TenantId,

            [Parameter()]
            [System.String]
            $CertificateThumbprint
        )

        Write-Host "Overridden Get-TargetResource called with Name: $Name"
        if ($PSBoundParameters['Verbose'].IsPresent) {
            $PSBoundParameters.Remove('Verbose') | Out-Null
            $PSBoundParameters.Add('Verbose', $true)
        }

        Invoke-Pwsh7Resource -Name 'EXOAddressList' -FunctionName 'Get' -Parameters $PSBoundParameters
    }
}

Export-ModuleMember -Function *-TargetResource

The code for the Invoke-Pwsh7Resource and Initialize-PowerShellCoreSession in M365DSCUtil.psm1:

function Invoke-Pwsh7Resource {
    [CmdletBinding()]
    param (
        [Parameter(Mandatory = $true)]
        [string]$Name,

        [Parameter(Mandatory = $true)]
        [ValidateSet('Get', 'Set', 'Test', 'Export')]
        [string]$FunctionName,

        [Parameter(Mandatory = $true)]
        [hashtable]$Parameters
    )

    if (-not $script:PSCoreSessionInitialized)
    {
        Initialize-PowerShellCoreSession
    }

    $result = Invoke-Command -Session $PSCoreSession -ScriptBlock {
        Invoke-DscResource -Name $using:Name -Method $using:FunctionName -ModuleName Microsoft365DSC -Property $using:Parameters
    }

    return $result
}

function Initialize-PowerShellCoreSession {
    $script:PSCoreSession = New-PSSession -ComputerName localhost -ConfigurationName PowerShell.7 -EnableNetworkAccess
    $script:PSCoreSessionInitialized = $true
}

I'll take a look if I can define and replace the body dynamically so that we don't have to rewrite and add all of the functions again. This would be quite a pain. Otherwise, I think that looks pretty neat tbh 😄

@FabienTschanz
Copy link
Collaborator

FabienTschanz commented Mar 1, 2025

Edited post.

Integration with the LCM is also working - Just created a test resource based on a simplified AADGroup version. Test-DscConfiguration correctly calls into the module, the call is redirected into PS7, Test-TargetResource is then executed and returns its result. This all works without issues.

But there is one drawback: Invoke-DscResource is very slow because it always fetches all DSC resources using Get-DscResource. This gives an overhead of ~0.6 seconds for each resource. If the current path of the module is passed down (for script-based imports), then the performance can be optimized to be almost equally as fast as a native execution. We're talking about native being ~0.13s and with the compatibility level ~0.21s, down from ~1s per resource execution when using Invoke-DscResource. At least on my machine, that differs greatly on other computers of course. Would love to see how this performs on a greater scale.

@Borgquite I reverse engineered the PSDesiredStateConfiguration 2.0.7 module, and they can call both script- and class based resources. The code is actually so simple is laughable:

script-based:

    $path = $resource.Path
    $type = $resource.ResourceType

    Write-Debug "Importing $path ..."
    Import-module -Scope Local -Name $path -Force -ErrorAction stop

    $functionName = "$Method-TargetResource"

    Write-Debug "calling $name\$functionName ..."
    $global:DSCMachineStatus = $null
    $output = & $type\$functionName @Property
    return Get-InvokeDscResourceResult -Output $output -Method $Method

class-based:

    $path = $resource.Path
    $type = $resource.ResourceType

    Write-Debug "Importing $path ..."
    $iss = [System.Management.Automation.Runspaces.InitialSessionState]::CreateDefault2()
    $powershell = [PowerShell]::Create($iss)
    $script = @"
using module "$path"

return [$type]::new()
"@

    $null= $powershell.AddScript($script)
    $dscType=$powershell.Invoke() | Select-object -First 1
    foreach($key in $Property.Keys)
    {
        $value = $Property.$key
        $dscType.$key = $value
    }

    $global:DSCMachineStatus = $null
    $output = $dscType.$Method()
    return Get-InvokeDscResourceResult -Output $output -Method $Method

We can easily mimic this behavior and adapt if the resource is either script or class-based. So we won't be dependent on this module either. Execution time is blazingly fast and everything's properly integrated. I think we might have hit the jackpot here for a solution that benefits all of us.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

5 participants