[vcpkg/fortran] Switch to ifort as the Fortran compiler #19413

Neumann-A · 2021-08-06T12:00:09Z

Depends on Add Intel oneAPI env variables. vcpkg-tool#148 being merged
waiting for next vcpkg-tool release
~~- [ ] Add Microsoft.VisualStudio.ComponentGroup.UWP.VC.BuildTools to Windows Visual Studio #19554 for uwp builds~~
Use helper port.
~~Update Lapack to 3.10~~ extracted
No longer limited to dynamic linkage for Fortran
No x86 host support.

(will update versions if this can actually correctly run in CI)

also closes #19983

github-actions

This is a new experimental fast check for PR issues. Please let us know if this bot is helpful!

After committing all other changes, the version database must be updated

git add -u && git commit
git checkout 5742e878fc71b4fcb9c019b50f1f887f22db58df -- versions
./vcpkg x-add-version --all

Diff

diff --git a/versions/baseline.json b/versions/baseline.json
index 7995aa8..127a8da 100644
--- a/versions/baseline.json
+++ b/versions/baseline.json
@@ -3001,8 +3001,8 @@
       "port-version": 1
     },
     "lapack-reference": {
-      "baseline": "3.8.0",
-      "port-version": 5
+      "baseline": "3.10.0",
+      "port-version": 0
     },
     "lastools": {
       "baseline": "2020-05-09",
@@ -6612,6 +6612,10 @@
       "baseline": "2021-05-22",
       "port-version": 1
     },
+    "vcpkg-fortran": {
+      "baseline": "1",
+      "port-version": 0
+    },
     "vcpkg-gfortran": {
       "baseline": "3",
       "port-version": 0
diff --git a/versions/l-/lapack-reference.json b/versions/l-/lapack-reference.json
index 96db87c..cf67f47 100644
--- a/versions/l-/lapack-reference.json
+++ b/versions/l-/lapack-reference.json
@@ -1,5 +1,10 @@
 {
   "versions": [
+    {
+      "git-tree": "17d524cc89464b4cdddf4ab9ca3d25b72b6d6ba9",
+      "version-string": "3.10.0",
+      "port-version": 0
+    },
     {
       "git-tree": "1887fc1fcb0c96df1ea24fffc9b045330426e3b6",
       "version-string": "3.8.0",

Neumann-A · 2021-08-06T18:54:01Z

ports/vcpkg-fortran/z_vcpkg_load_environment_from_batch.cmake

+### BATCH_FILE_PATH
+Arguments to pass to the batch file
+#]===]
+function(z_vcpkg_load_environment_from_batch)


@strega-nil @ras0219-msft : Should I put this into scripts/cmake ? The only reason I did not do it here is because I didn't want to force a world rebuild until the required changes to vcpkg-tool are merged.
I also need this for the community intel triplet. (also to load the required env variables)

github-actions

This is a new experimental fast check for PR issues. Please let us know if this bot is helpful!

All manifest files must be formatted

./vcpkg format-manifest ports/*/vcpkg.json

Diff

diff --git a/ports/lapack-reference/vcpkg.json b/ports/lapack-reference/vcpkg.json
index cf79815..c99f3b9 100644
--- a/ports/lapack-reference/vcpkg.json
+++ b/ports/lapack-reference/vcpkg.json
@@ -3,9 +3,7 @@
   "version-string": "3.10.0",
   "description": "LAPACK — Linear Algebra PACKage http://www.netlib.org/lapack/",
   "dependencies": [
-    {
-      "name": "vcpkg-fortran"
-    }
+    "vcpkg-fortran"
   ],
   "default-features": [
     "blas-select"
diff --git a/ports/vcpkg-ifort/vcpkg.json b/ports/vcpkg-ifort/vcpkg.json
index f7c4366..f4525fb 100644
--- a/ports/vcpkg-ifort/vcpkg.json
+++ b/ports/vcpkg-ifort/vcpkg.json
@@ -3,8 +3,7 @@
   "version-string": "1",
   "description": "Metaport to install ifort dll dependencies from Intel oneAPI",
   "supports": "windows & !arm",
-  "dependencies" : 
-  [
+  "dependencies": [
     "vcpkg-fortran"
   ]
 }

After committing all other changes, the version database must be updated

git add -u && git commit
git checkout c801072693edcd0704d187131d9078fcd089dbf2 -- versions
./vcpkg x-add-version --all

Diff

diff --git a/versions/baseline.json b/versions/baseline.json
index e135641..43969b2 100644
--- a/versions/baseline.json
+++ b/versions/baseline.json
@@ -3009,8 +3009,8 @@
       "port-version": 1
     },
     "lapack-reference": {
-      "baseline": "3.8.0",
-      "port-version": 5
+      "baseline": "3.10.0",
+      "port-version": 0
     },
     "lastools": {
       "baseline": "2020-05-09",
@@ -6620,10 +6620,18 @@
       "baseline": "2021-05-22",
       "port-version": 1
     },
+    "vcpkg-fortran": {
+      "baseline": "1",
+      "port-version": 0
+    },
     "vcpkg-gfortran": {
       "baseline": "3",
       "port-version": 0
     },
+    "vcpkg-ifort": {
+      "baseline": "1",
+      "port-version": 0
+    },
     "vcpkg-pkgconfig-get-modules": {
       "baseline": "2021-04-02",
       "port-version": 1
diff --git a/versions/l-/lapack-reference.json b/versions/l-/lapack-reference.json
index 96db87c..e090549 100644
--- a/versions/l-/lapack-reference.json
+++ b/versions/l-/lapack-reference.json
@@ -1,5 +1,10 @@
 {
   "versions": [
+    {
+      "git-tree": "d013c13a916469a4a6de4fbbbd164dc3fc607eab",
+      "version-string": "3.10.0",
+      "port-version": 0
+    },
     {
       "git-tree": "1887fc1fcb0c96df1ea24fffc9b045330426e3b6",
       "version-string": "3.8.0",

Neumann-A · 2021-08-13T11:22:20Z

@BillyONeal: VMs not completely setup?

    The license for Visual Studio expires in 1 day.
    Project 'ALL_BUILD' could not be loaded because it's missing install components. To fix this launch Visual Studio installer with the following selections:
    C++ (v142) Universal Windows Platform tools

BillyONeal · 2021-08-13T17:16:06Z

@BillyONeal: VMs not completely setup?

Nothing has changed in the VM image in the last month (#19508 isn't landed yet). Note that we install "Universal Windows Platform Tools" but not "Universal Windows Platform Tools (v142)" right now

vcpkg/scripts/azure-pipelines/windows/deploy-visual-studio.ps1

Line 11 in eb9cb29

'Microsoft.VisualStudio.Workload.Universal',

This implies installing the "Universal Windows Platform Development" workload, but I don't see lines for the following optional checkboxes (they are checked on my development workstation but not CI :) ):

Neumann-A · 2021-08-13T18:29:42Z

According to https://docs.microsoft.com/en-us/visualstudio/install/workload-component-id-vs-build-tools?view=vs-2019 the missing optional component is Microsoft.VisualStudio.ComponentGroup.UWP.VC.BuildTools

… into use_oneapi_fortran

…ted. (If UWP would ninja this would probably not be a VM update issue)

github-actions

This is a new experimental fast check for PR issues. Please let us know if this bot is helpful!

PRs must add only one version and must not modify any published versions

When making any changes to a library, the version or port-version in vcpkg.json or CONTROL must be modified.

Error: Local changes detected for vcpkg-cmake but no changes to version or port version.
-- Version: 2022-04-12
-- Old SHA: ef4a9463f802c1cd6f8b31e560529aeaee0faff4
-- New SHA: 60b5c6f101857827ca5912cf728672f41830d9dc
-- Did you remember to update the version or port version?
-- Pass `--overwrite-version` to bypass this check.
***No files were updated.***
Error: Local changes detected for vcpkg-fortran but no changes to version or port version.
-- Version: 2022-02-22
-- Old SHA: ec6affed6b4aca0be5d5c59cc5c614f43afbc177
-- New SHA: 06093715131e8d56f325052595c8e1fc8710e936
-- Did you remember to update the version or port version?
-- Pass `--overwrite-version` to bypass this check.
***No files were updated.***

After committing all other changes, the version database must be updated

git add -u && git commit
git checkout c37cc7836a0e1cfd55747be8ec472eafa8055276 -- versions
./vcpkg x-add-version --all

Diff

diff --git a/versions/baseline.json b/versions/baseline.json
index fc44c8d7..99f7189c 100644
--- a/versions/baseline.json
+++ b/versions/baseline.json
@@ -3322,7 +3322,7 @@
     },
     "lapack-reference": {
       "baseline": "3.10.0",
-      "port-version": 0
+      "port-version": 1
     },
     "lastools": {
       "baseline": "2020-05-09",
@@ -7304,6 +7304,10 @@
       "baseline": "2022-02-06",
       "port-version": 0
     },
+    "vcpkg-fortran": {
+      "baseline": "2022-02-22",
+      "port-version": 0
+    },
     "vcpkg-get-python-packages": {
       "baseline": "2022-04-11",
       "port-version": 0
@@ -7316,6 +7320,10 @@
       "baseline": "2021-11-16",
       "port-version": 1
     },
+    "vcpkg-ifort": {
+      "baseline": "2022-02-22",
+      "port-version": 0
+    },
     "vcpkg-pkgconfig-get-modules": {
       "baseline": "2022-02-10",
       "port-version": 0

JackBoosY · 2022-04-20T11:17:45Z

Will rerun x64-osx tests after others finished.

github-actions

This is a new experimental fast check for PR issues. Please let us know if this bot is helpful!

PRs must add only one version and must not modify any published versions

When making any changes to a library, the version or port-version in vcpkg.json or CONTROL must be modified.

Error: Local changes detected for lapack-reference but no changes to version or port version.
-- Version: 3.10.0#1
-- Old SHA: 0ce53c448a021788bc0249e2d75b5a6b96e799bb
-- New SHA: d4eae50df1a1b64c8a1ebfb5e2bcca6a77922c1b
-- Did you remember to update the version or port version?
-- Pass `--overwrite-version` to bypass this check.
***No files were updated.***

JackBoosY · 2022-04-27T14:15:50Z

/azp run

azure-pipelines · 2022-04-27T14:16:00Z

Azure Pipelines successfully started running 1 pipeline(s).

# Conflicts: # versions/v-/vcpkg-cmake.json

BillyONeal · 2022-04-28T18:21:44Z

Related: #15053 ?

Neumann-A · 2022-04-28T19:15:12Z

Related: #15053 ?

So so. 15053 switches the whole windows toolchain to intel. 15053 contains stuff from this PR though.

BillyONeal

Our default behavior has to work without the user installing extra Intel stuff for users who do not care. This probably prevents using ifort by default since it appears to require a separate login-walled installation of Intel's OneAPI toolchains.
Capturing a port for the contract of "we need Fortran support", as you have done here with vcpkg-fortran, seems reasonable and consistent with other things that we do. (I'm personally concerned about potential command line differences between them but I'm told that lapack and blas are substantially all our Fortran customers and that I probably shouldn't worry about that)
- We do think it's reasonable to give users a mechanism to choose a Fortran compiler to use, as you are trying to demonstrate in WIP: [x64-windows-intel] new Triplet: #15053 . A triplet setting, as demonstrated there, would work, but is inefficient because it forces a world rebuild to change this even though a small number of ports may be affected. (Of course, this is true for all triplet settings) Overlays are a solution that affects only things that actually need Fortran, but providing an overlay forces the user to implement the complete interface somehow rather than just saying "I want ifort" or "I want gfortran". Overlays have a nice property in that it lets you do development on the "not default" option without triggering world rebuilds for other customers.
- We do not think "try to use ifort if we can find it" is OK, it needs to be an explicit user decision, for example, in the triplet.
- This contract/metaport probably shouldn't have a "supports" clause, unless we know that it's categorically impossible to provide Fortran there. So !uwp may be fine but windows is not. (Linux customers may want to use gfortran from their system's package manager, for example.)
[You mentioned this but I'm indicating that we're considering it] This PR breaks x86 hosts (you need to be an amd64 host in order to run this). Some years ago we had a substantial number of x86-only customers. We accidentially broke this ourselves though in [7zip] Upgrade vcpkgTools 7zip to 19.00 msi #19365 and didn't switch back to CMake to extract things for 4 ish months and it seems nobody noticed which suggests that this wouldn't be a big deal.
Given that UWP is already broken for gfortran et al. here, don't worry about UWP interaction in whatever the solution is.

I'm sorry if these observations sound a bit "wishy washy" -- the situation is that we aren't 100% sure what we want the future to be here so we've resorted to listing constraints we believe exist on the solution space to help you decide (a) how you should interpret our response as to whether we want this, and (b) whether you want to continue investigating this path.

Suggested path forward:

Add vcpkg-fortran as the representation of the contract, as you have done here. Its behavior should match today's behavior (gfortran).
Document the contract (all the cmake functions, variables, environment variables, etc.) expected to be provided by vcpkg-fortran.
Create a new port intended to be an overlay for vcpkg-fortran which selects ifort. That probably needs to live in a different directory like the hypothetical "backports" we have been talking about, but for purposes of making forward progress on this PR I don't think you need a concrete answer to that question yet.

ports/vcpkg-cmake/vcpkg_cmake_build.cmake

ports/vcpkg-fortran/portfile.cmake

ports/vcpkg-fortran/copyright

ports/lapack-reference/FindLAPACK.cmake

ports/lapack-reference/portfile.cmake

ports/vcpkg-fortran/z_vcpkg_load_environment_from_batch.cmake

Neumann-A · 2022-04-28T22:21:27Z

Our default behavior has to work without the user installing extra Intel stuff for users who do not care. This probably prevents using ifort by default since it appears to require a separate login-walled installation of Intel's OneAPI toolchains.

With that you are basically saying to take the flang-classic route used in #23987. The main reason I am hesitant about that route is the build times required for the LLVM fork of flang-classic and the fact that #23987 loses x86 target which can still be covered by gfortran. Would be good if that could be reduced somehow

lapack and blas are substantially all our Fortran customers

that is because the Fotran support situation has not been solved and vcpkg is focused around windows. On other platforms Fortran support is easily available. Only windows is a problem.

A triplet setting, as demonstrated there, would work, but is inefficient because it forces a world rebuild to change

It is not a triplet setting but a toolchain setting which sets CMAKE_Fortran_(COMPILER|FLAGS) explicitly. This is only required on windows due to vcpkg cleaning the environment/path. On other platforms CMake will find the fortran compiler without extra help and without needing extra flags.
Getting rid of the world rebuild due to the triplet hash is an open issue #19984.
The only thing which would be nice to have is z_vcpkg_load_environment_from_batch to not have to manually setup the passthrough. Should this be made into a different PR?

rather than just saying "I want ifort" or "I want gfortran". Overlays have a nice property in that it lets you do development on the "not default" option without triggering world rebuilds for other customers.

Personally I would like to use version-string for that in the main repo ;). basically have a PR with version-string : "gfortran"; version-string : "ifort", version-string : "flang-classic", The additional overlays for classic users could also be provided. The question here is what should the end default be in the main repo?
Again this is a windows only issues. All other platforms will control the Fortran compiler via env FC.

This contract/metaport probably shouldn't have a "supports" clause, unless we know that it's categorically impossible to provide Fortran there. So !uwp may be fine but windows is not. (Linux customers may want to use gfortran from their system's package manager, for example.)

All other platforms have native fortran support and don't require the contract/extra port. I don't intend to force the compiler there.

...This PR breaks x86 hosts .... We accidentially broke this ourselves ... nobody noticed which suggests that this wouldn't be a big deal..... Given that UWP is already broken for gfortran et al. here, don't worry about UWP interaction in whatever the solution is.

Ok won't care about x86 host and UWP. This also means I should go with #23987 which additionally adds arm64 support but loses x86 target support. I could make a port vcpkg-fortran depending on nothing (for !windows) and depending on vcpkg-gfortran for x86 or vcpkg-flang-classic for the rest.

the situation is that we aren't 100% sure what we want the future to be here

I personally think that should be clear -> flang-new will be the future path which probably will still require 2 years to mature.

BillyONeal · 2022-05-05T18:18:10Z

With that you are basically saying to take the flang-classic route used in #23987. The main reason I am hesitant about that route is the build times required for the LLVM fork of flang-classic and the fact that #23987 loses x86 target which can still be covered by gfortran. Would be good if that could be reduced somehow

Yeah, we wouldn't want to require people build a whole compiler to get BLAS. But clang is available in binary forms today, it seems like it would be reasonable for flang to be similarly available. (It may even involve some amount of hosting by GitHub or the vcpkg team in support of this or...)

Personally I would like to use version-string for that in the main repo ;).

Then we would need to have version versions! It'd be turtles all the way down! I would point to the "alternatives" / "options" RFC as a solution here whenever that finally gets to the top of our backlog.

This contract/metaport probably shouldn't have a "supports" clause, unless we know that it's categorically impossible to provide Fortran there. So !uwp may be fine but windows is not. (Linux customers may want to use gfortran from their system's package manager, for example.)
All other platforms have native fortran support and don't require the contract/extra port. I don't intend to force the compiler there.

The port should still exist to document the dependency even if it becomes empty on the platforms where Fortran compilers are assumed to be "ambiently available".

Neumann-A · 2022-05-05T19:42:15Z

But clang is available in binary forms today, it seems like it would be reasonable for flang to be similarly available.

It is not. Also building the flang1 and flang2 doesn't require that much time. Only building clang with the classig-flang toolchain takes forever (1h on my machine). Due to MT(d) MD(d) stuff there needs to be extra stuff taken into account in the building of clang with flang on windows. I haven't figured out how clang embeds /DEFAULTLIB: into object files.
It probably gets better if flang-new is finally ready to produce binaries maybe it will then be included in the LLVM windows package but currently only ifort/ifx is available in binary form on windows (unless you plan to pull flang from conda.)

Then we would need to have version versions!

that already exists: It is called port-version:

finally gets to the top of our backlog.

Probably when we are all grey and wiser ;)

The port should still exist to document the dependency even if it becomes empty on the platforms where Fortran compilers are assumed to be "ambiently available".

probably can do that and just assume all other platforms use gfortran even though there is ifort and ifx also available.

Neumann-A added 2 commits August 6, 2021 13:55

add fortran helper port

e913fac

update lapack port and switch to new fortran helper

7c044f0

github-actions bot requested changes Aug 6, 2021

View reviewed changes

Neumann-A added 2 commits August 6, 2021 14:34

[skip ci] cleanup

b013d52

[skip ci] add vcpkg-ifort to copy dlls + more cleanup

265dc58

Neumann-A commented Aug 6, 2021

View reviewed changes

JonLiu1993 assigned PhoebeHui Aug 9, 2021

PhoebeHui added the category:port-feature The issue is with a library, which is requesting new capabilities that didn’t exist label Aug 9, 2021

Merge remote-tracking branch 'upstream/master' into use_oneapi_fortran

748c1a6

github-actions bot requested changes Aug 13, 2021

View reviewed changes

Neumann-A added 4 commits August 13, 2021 08:33

[skip actions] apply format diff

aa1bae1

[skip actions] consider 19 20 and 21 and a bit of cleanup

7f18045

[skip actions] fix vcpkg-ifort

1598ae8

[skip actions] make it work in static builds

d9578d8

BillyONeal mentioned this pull request Aug 13, 2021

Add Microsoft.VisualStudio.ComponentGroup.UWP.VC.BuildTools to Windows Visual Studio #19554

Closed

Neumann-A mentioned this pull request Aug 15, 2021

[Lapack] Disable UWP support #19587

Closed

Neumann-A added 3 commits August 16, 2021 15:12

Merge remote-tracking branch 'upstream/master' into use_oneapi_fortran

95e522e

[skip actions] fix uwp builds.

4645f90

[skip actions] apply format diff

5dae300

Neumann-A mentioned this pull request Aug 17, 2021

[lapack-reference|fortran] Add new port #12805

Merged

Neumann-A added 6 commits August 18, 2021 10:30

Merge remote-tracking branch 'upstream/master' into use_oneapi_fortran

958dcf4

Merge branch 'use_oneapi_fortran' of https://github.com/Neumann-A/vcpkg…

165a35d

… into use_oneapi_fortran

[skip actions] update vcpkg-fortran to the changes made in rollup

5bbd0ff

[skip actions] switch to version-date and revert changes to the portfile

ebe763f

[skip actions] version stuff

0c88af0

add lapack-reference:x64-uwp=skip on the ci baseline until VM is upda…

920d824

…ted. (If UWP would ninja this would probably not be a VM update issue)

github-actions bot approved these changes Aug 18, 2021

View reviewed changes

Merge remote-tracking branch 'upstream/master' into use_oneapi_fortran

d6c3f7d

github-actions bot requested changes Apr 20, 2022

View reviewed changes

version stuff

a83fda7

Neumann-A marked this pull request as draft April 23, 2022 20:07

github-actions bot requested changes Apr 23, 2022

View reviewed changes

Alexander Neumann added 3 commits April 27, 2022 14:50

Merge remote-tracking branch 'upstream/master' into use_oneapi_fortran

b0d0816

detect intel fortran compiler to apply patch in lapack.

c51de5e

version stuff

5f56132

github-actions bot approved these changes Apr 27, 2022

View reviewed changes

JackBoosY marked this pull request as ready for review April 27, 2022 14:15

JackBoosY marked this pull request as draft April 27, 2022 14:15

Alexander Neumann added 3 commits April 28, 2022 13:27

Merge remote-tracking branch 'upstream/master' into use_oneapi_fortran

32dd4db

# Conflicts: # versions/v-/vcpkg-cmake.json

version date bump

e111bf2

version stuff

bf206e2

github-actions bot approved these changes Apr 28, 2022

View reviewed changes

BillyONeal requested changes Apr 28, 2022

View reviewed changes

Neumann-A closed this May 8, 2022

Neumann-A mentioned this pull request May 9, 2022

[vcpkg/Fortran|world rebuild] Windows: Use ifort (x86|x64) or flang(arm64) #24622

Draft

7 tasks

Neumann-A mentioned this pull request Feb 18, 2023

Update vcpkg-tool to 2023-02-16. #29664

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[vcpkg/fortran] Switch to ifort as the Fortran compiler #19413

[vcpkg/fortran] Switch to ifort as the Fortran compiler #19413

Neumann-A commented Aug 6, 2021 •

edited

Loading

github-actions bot left a comment

Neumann-A Aug 6, 2021

github-actions bot left a comment

Neumann-A commented Aug 13, 2021

BillyONeal commented Aug 13, 2021

Neumann-A commented Aug 13, 2021

github-actions bot left a comment

JackBoosY commented Apr 20, 2022

github-actions bot left a comment

JackBoosY commented Apr 27, 2022

azure-pipelines bot commented Apr 27, 2022

BillyONeal commented Apr 28, 2022

Neumann-A commented Apr 28, 2022

BillyONeal left a comment

Neumann-A commented Apr 28, 2022

BillyONeal commented May 5, 2022

Neumann-A commented May 5, 2022

[vcpkg/fortran] Switch to ifort as the Fortran compiler #19413

[vcpkg/fortran] Switch to ifort as the Fortran compiler #19413

Conversation

Neumann-A commented Aug 6, 2021 • edited Loading

github-actions bot left a comment

Choose a reason for hiding this comment

Neumann-A Aug 6, 2021

Choose a reason for hiding this comment

github-actions bot left a comment

Choose a reason for hiding this comment

Neumann-A commented Aug 13, 2021

BillyONeal commented Aug 13, 2021

Neumann-A commented Aug 13, 2021

github-actions bot left a comment

Choose a reason for hiding this comment

JackBoosY commented Apr 20, 2022

github-actions bot left a comment

Choose a reason for hiding this comment

JackBoosY commented Apr 27, 2022

azure-pipelines bot commented Apr 27, 2022

BillyONeal commented Apr 28, 2022

Neumann-A commented Apr 28, 2022

BillyONeal left a comment

Choose a reason for hiding this comment

Neumann-A commented Apr 28, 2022

BillyONeal commented May 5, 2022

Neumann-A commented May 5, 2022

Neumann-A commented Aug 6, 2021 •

edited

Loading