Xamarin.Android task cannot find NDK with latest Azure image(s)

[thomasphillipsgb]

Android application type

Classic Xamarin.Android (MonoAndroid12.0, etc.)

Affected platform version

Azure pipeline

Description

Making use of the Xamarin.Android task we have been able to build and deploy Xamarin.Android applications. New images were released which seems to have affected this task; the NDK path is no longer found when executing the "_ResolveMonoAndroidSdks" step resulting in a build failure.

Reviewing the latest releases for these images, I see there was work around NDK paths, symlinks, and environment variable changes.
actions/runner-images#5879
actions/runner-images#5930

The image we're using is macOS-latest, here are the old and new logs showing the "_ResolveMonoAndroidSdks" step resolving the NDK path:
Old working image (20220724.1)

New broken image (20220801.1)

Do we know if the Xamarin.Android task needs to be updated to reflect this?

Steps to Reproduce

1. Make use of the Azure Pipeline Xamarin.Android task for a Xamarin.Android application using the macOS-latest image.
2. See the task eventually fails with error XA5104: Could not locate the Android NDK. Please make sure the Android NDK is installed in the Android SDK Manager, or if using a custom NDK path, please ensure the $(AndroidNdkDirectory) MSBuild property is set to the custom path.

Did you find any workaround?

No response

Relevant log output

No response

[jonathanpeppers]

Did this start failing like last night?

I have a build yesterday that did:

Android NDK: /Users/runner/Library/Android/sdk/ndk/24.0.8215888/

https://dev.azure.com/jonathanpeppers/boots/_build/results?buildId=31&view=logs&j=a82d9e72-1de4-5bd6-c188-90296135479e&t=65080fd3-055e-56ae-0221-9a17a9cb5386&l=365

It is macOS-latest:

https://github.com/jonathanpeppers/boots/blob/7bf7e0875d8e4a19c3c51559521d7d0f1aa15f89/azure-pipelines.yml#L35

[jonathanpeppers]

Started a new one, let's see what it does: https://dev.azure.com/jonathanpeppers/boots/_build/results?buildId=33&view=results

[jonathanpeppers]

I think it also worked: https://dev.azure.com/jonathanpeppers/boots/_build/results?buildId=33&view=logs&j=a82d9e72-1de4-5bd6-c188-90296135479e&t=65080fd3-055e-56ae-0221-9a17a9cb5386&l=365

[thomasphillipsgb]

Thanks for the response! Our builds started failing this morning, ones running yesterday were still using the old image, due to the staggered rollout.

I do see your build is using MSBuild directly, whereas I'm using this badger for building my Xamarin Android project
https://docs.microsoft.com/en-gb/azure/devops/pipelines/tasks/build/xamarin-android?view=azure-devops

[jonathanpeppers]

It seems like the bots should still have r23 and r24, can you add some ls commands to a build to find out if they are there?

[jonathanpeppers]

Mine is also using XamarinAndroidVersion: 13.0.0-0, which it downloaded and installed.

[GreatBarrier86]

Instead of the suggested mitigation of installing r21, is there a way to change the project to just use the latest version? I'm obviously having the same problem too.

@jonathanpeppers, there's something here i dont understand. I dont think we're specifying the required NDK version in our manifest or csproj file, so what's causing it to use r21 specifically instead of just $LatestVersion.

_ResolveMonoAndroidSdks:
MonoAndroid Tools: /Library/Frameworks/Xamarin.Android.framework/Libraries/xbuild/Xamarin/Android/
Android Platform API level: 31
TargetFrameworkVersion: v12.0
Android NDK:
Android SDK: /Users/runner/Library/Android/sdk/
Android SDK Build Tools: /Users/runner/Library/Android/sdk/build-tools/30.0.2/
Java SDK: /Library/Java/JavaVirtualMachines/Temurin-Hotspot-11.jdk/Contents/Home/
Application Java class: android.app.Application

"/Users/runner/work/8/s/Mobile/CPSMobile/CPSMobile.Android/CPSMobile.Android.csproj" (PackageForAndroid target) (1) ->
(_BuildApkEmbed target) ->
/Library/Frameworks/Mono.framework/External/xbuild/Xamarin/Android/Xamarin.Android.Common.targets(1954,3): error XA5104: Could not locate the Android NDK. Please make sure the Android NDK is installed in the Android SDK Manager, or if using a custom NDK path, please ensure the $(AndroidNdkDirectory) MSBuild property is set to the custom path. [/Users/runner/work/8/s/Mobile/CPSMobile/CPSMobile.Android/CPSMobile.Android.csproj]

[jonathanpeppers]

I found something that prevents the selection of r25: dotnet/android-tools#190

Versions with Android NDK r24 or lower should work, though. As long as one of those is installed.

[GreatBarrier86]

According to https://github.com/actions/virtual-environments/blob/main/images/macos/macos-11-Readme.md, the following NDKs are installed.
21.4.7075529
23.2.8568313
24.0.8215888

Wouldn't one of these be auto-selected? Or is there a configuration problem on my manfiest/csproj side? It just seems like a manual step shouldn't be necessary

Edit: and on top of that, that list doesn't include r25. I'm not seeing how this would work, or I just have a fundamental misunderstanding of the implementation. :)

[DevenCC]

Hey there, following the mitigation proposed in this thread, I've added this task prior to my Android build task and at least the NDK is now present.

    - task: CmdLine@2
      displayName: "XA5104 Workaround"
      inputs:
        script: '${ANDROID_HOME}/tools/bin/sdkmanager --install "ndk;21.4.7075529"'

This might not be an elegant permanent solution, since as @GreatBarrier86 stated, I also don't understand why 21.4.7075529 is not found automatically. However, this could help unblock those failing pipelines in the meantime at least.

[GreatBarrier86]

@DevenCC, did you have any success referencing a later version of the NDK that, according to that link, is also installed on the image? I tried doing the r24 and had no success.

[DevenCC]

@GreatBarrier86 I tried the same task with 24.0.8215888 and it worked fine as well.
I am assuming here that using any of the ndk versions stating in the MacOS-latest virtual environments would work.
I do not know if any other specific version would work however.

For my specific needs however, I went for version 21.4.7075529 as that is what my project was using prior to this issue.

[jtorvald]

I'm experiencing the same issue. This week we had one re-run on devops and that one magically succeeded while the previous one failed. All the builds after kept failing. The temp-fix from @DevenCC helps to get a successful build.

[jonathanpeppers]

Is someone able to share a .binlog of a failing build? And if you have a working example to compare against, that would also be helpful. Thanks!

You can build with -bl:nameyourlog.binlog, and then archive the file during your build.

[jtorvald]

@jonathanpeppers I tried to make one but I can't find where the binlog is placed. I copied the whole source and output directory but can't find it.

[mikequ-taggysoft]

Our App Center Xamarin.Forms Android builds just broke with the same error, too. We don't use any MSBuild task script or specify any specific NDK path/version in our Xamarin project. Things always built fine with just the source code in our DevOps repo.

I wonder why it's not using whatever is the latest on the build agent after this update. Is there a setting in the Xamarin.Forms project that we have to configure?

[jtorvald]

@mikequ-taggysoft look at the temporary fix from @DevenCC with task: CmdLine@2. Copy that block into your pipeline.yml before the task task: XamarinAndroid@1
That will make sure the NDK gets installed and your build will succeed.

[mikequ-taggysoft]

@jtorvald We're not using Azure Pipelines for the project. We build it on-commit or manually via App Center which is connected to the repo.

[jtorvald]

@mikequ-taggysoft sorry, I skipped over that. I guess you can do the same in AppCenter with a post-clone hook script and to add this line to it:

$ANDROID_HOME/tools/bin/sdkmanager --install "ndk;21.4.7075529"

Documentation about post-clone hook can be found here

[mikequ-taggysoft]

@jtorvald Thanks! We do use post clone scripts and adding that command to it did resolve the issue.

Is there any way to find out exactly which part of our Xamarin project depends on NDK v21 (or the Xamarin.Android framework itself does?) Is it possible to use the latest version somehow?

[GreatBarrier86]

Something else is strange here. I dont have any NDK version installed on my local build machine and i have no trouble building my app at all. If it works fine for me without the NDK, why is it necessary on the azure build machine? Is it something MacOS specific?

Also, i agree with @mikequ-taggysoft. If we can eliminate the specific version requirement, that would be ideal.

[dellis1972]

Are you building Debug or Release builds locally? only certain features of a Release build will need the NDK.
Also the sdk I have includes an ndk in it. So it might be that your local install is picking that up.

[thomasphillipsgb]

This is what we define;

  - task: XamarinAndroid@1
    displayName: 'Build ${{ parameters.ProjectPathMain }}'
    inputs:
      projectFile: ${{ parameters.ProjectPathMain }}
      outputDirectory: '$(build.binariesdirectory)/${{ parameters.BuildConfiguration }}'
      configuration: ${{ parameters.BuildConfiguration }}
      msbuildVersionOption: latest
      msbuildArguments: '/p:AndroidNdkDirectory="$(ANDROID_NDK_HOME)"'
      jdkOption: 'JDKVersion'
      createAppPackage: true

This is what was used which resulted in that error; error XA5105: Toolchain utility 'armv7a-linux-androideabi16-clang' for target Arm was not found. Tried in path: "/Users/runner/Library/Android/sdk/ndk/25.0.8775105/toolchains/llvm/prebuilt/darwin-x86"

[jonathanpeppers]

@Maxgamerboy1 can you use this to get a .binlog of your build:

msbuildArguments: -bl:$(Build.ArtifactStagingDirectory)msbuild.binlog

Then save this file as an artifact, and share it?

You might consider using the plain MSBuild yaml task instead of XamarinAndroid. I'm not sure who created it, or if it is maintained. It seems like the MSBuild task would do everything and you just need to use the -t:SignAndroidPackage target?

[thomasphillipsgb]

you can email at the address on my GitHub profile, thanks.

@jonathanpeppers i've emailed you a binlog from an errored build
i'll look to use MSBuild directly tomorrow

[GreatBarrier86]

@jonathanpeppers, i shared the XamarinAndroidVersion above in this post here: #7233 (comment). Is that satisfactory to get what you need for this issue? Not the NDK one, but the issue of 2022 on macos-latest not supporting Android 13.

It also occurs to me that my success with the NDK was on macos-latest. @Maxgamerboy1's issue was on windows 2022.

[jonathanpeppers]

@Maxgamerboy1 @GreatBarrier86 you both have Xamarin.Android 12.x, and you need to use 13.x. What Azure DevOps pool are you using?

They supposedly rolled out an update that contains Xamarin.Android 13.x with: actions/runner-images#6045

[GreatBarrier86]

Do you want a new log with -v:diag in it because i did have this problem again today.

@jonathanpeppers , this is the pool i'm using

[jonathanpeppers]

If you see XamarinAndroidVersion says 12.x with macos-latest, you might consider filing an issue here: https://github.com/actions/runner-images/issues

Or we could reopen: actions/runner-images#6038

[GreatBarrier86]

Whichever you think is best.

[GreatBarrier86]

You might consider using the plain MSBuild yaml task instead of XamarinAndroid. I'm not sure who created it, or if it is maintained. It seems like the MSBuild task would do everything and you just need to use the -t:SignAndroidPackage target?

@jonathanpeppers, do you recommend using the regular MSBuild task instead of Xamarin.Android? I created a new pipeline to develop Xamarin.Android and it put in both the Xamarin.Android step AND the MSBuild one. If we should be using the other one, that's fine, but it would be good to know.

According to here, that task was last updated 2 months ago
https://github.com/microsoft/azure-pipelines-tasks/tree/master/Tasks/XamarinAndroidV1

[thomasphillipsgb]

@Maxgamerboy1 @GreatBarrier86 you both have Xamarin.Android 12.x, and you need to use 13.x. What Azure DevOps pool are you using?

They supposedly rolled out an update that contains Xamarin.Android 13.x with: actions/runner-images#6045

@jonathanpeppers I'm using the Azure Pipelines pool, running the macOS-latest image, specifically that errored build used image version 20220829.1.

I changed to use the MSBuild task for my android project and it still fails to find the NDK, and it continues to resolve Xamarin.Android 12.0.0.3... Should we reopen that issue actions/runner-images#6038

[GreatBarrier86]

@Maxgamerboy1, according to here (actions/runner-images#6038 (comment)), macos-latest doesn't actually have the latest software. Can you get it to work with macos-12?

[thomasphillipsgb]

@GreatBarrier86 thanks for pointing that out, it works using macos-12!
I'll be switching over to MSBuild task instead of the XamarinAndroid one too.

I'd be happy to mark this as resolved

[GreatBarrier86]

@Maxgamerboy1, any reason you're using the MSBuild task over the XamarinAndroid one? just curious myself. I'd like to use whichever one is recommended.

[thomasphillipsgb]

@GreatBarrier86 no real reason, i'm seeing Pepper's builds making use of it, and with the revelation that we don't know who is maintaining that task, I think it's a smart move to hop over the MSBuild task

[GreatBarrier86]

@Maxgamerboy1, i'm not sure i agree that is unsupported, but i have no inside knowledge. According to GitHub, the Xamarin.Android task was just updated 2 months ago. I'm not sure if it's really "unsupported". Read what i wrote up here: #7233 (comment)

[jonathanpeppers]

@GreatBarrier86 no real reason, i'm seeing Pepper's builds making use of it, and with the revelation that we don't know who is maintaining that task, I think it's a smart move to hop over the MSBuild task

I don't think I use the XamarinAndroid .yaml task, where did you see that? If someone is working on lately, it isn't the Xamarin.Android team, so I'm not sure about its status.

Personally, I don't see what value it has over the regular MSBuild task. If there is some value, we should just put that logic in Xamarin.Android/.NET 6 instead?

[GreatBarrier86]

Unless i misunderstand, he's saying he saw that you do NOT use Xamarin.Android, but instead use MSBuild.

[thomasphillipsgb]

Unless i misunderstand, he's saying he saw that you do NOT use Xamarin.Android, but instead use MSBuild.

Yes, that's correct. Sorry for the confusion; XamarinAndroid task- I've stopped using, MSBuild task - I've now switched over to use :D

[salgiza]

Could someone post an example of how the MSBuild task should be configured to get the same result as this basic XamarinAndroid task?

We are having the same problem and I don't really mind using one or the other as long as it works, but all the docs for Azure DevOps use the XamarinAndroid task, and I'm totally unfamiliar with the MSBuild task 😢

[thomasphillipsgb]

@salgiza sure! Although, see my response here #7233 (comment) the issue is relating more to the build image itself rather than the task used to build.

to switch over, in your case, it should look something like this;

  - task: MSBuild@1
    inputs:
      solution: '**/*Droid*.csproj'
      configuration: '$(buildConfiguration)'
      msbuildArguments: '-p:OutDir="$(outputDirectory)" -t:SignAndroidPackage'

[salgiza]

Thanks a lot @Maxgamerboy1 , with all the messages in the thread, I had missed the one that said that changing the vmImage from macos-latest to macos-12 should fix the problem. Seeing the comments from @jonathanpeppers, though, I decided to do both (change the vmImage, and migrate from the XamarinAndroid task to MSBuild), and our pipeline is working again!

[jonathanpeppers]

Ok, so I think we've solved this one -- the macos-12 image has Xamarin.Android 13.x, which supports up to Android NDK 24.

Let me know if I missed anything outstanding here, thanks!

[GreatBarrier86]

@Maxgamerboy1, one thing I noticed, and I'm not sure what I should be using here, but the Xamarin.Android step's msbuild line shows /t:PackageForAndroid instead of /t:SignAndroidPackage. My next step after this is the Android signing task, so I'm not sure which to use. For now, i'm going with `/t:PackageForAndroid' to mimic what the original task had.

[thomasphillipsgb]

Ah, sorry, didn't know you were doing signing in a separate step. Yes, you'll need to use the PackageForAndroid target in that case.

[GreatBarrier86]

I certainly don't have to sign in a different step. Does the msbuild task also zipalign? If so, i don't see why i can't use the msbuild task for both.

[thomasphillipsgb]

If using the /t:SignAndroidPackage it should zipalign, then keysign for you, here's a snippet from one of my old builds showing the _Sign step;

2022-09-05T14:26:12.3762630Z _Sign:
2022-09-05T14:26:12.3763980Z   /Users/runner/Library/Android/sdk/build-tools/32.0.0/zipalign -p 4 "obj/release/android/bin/com.[---].apk" "/Users/runner/work/1/b/release//com.[---]-Signed.apk" 
2022-09-05T14:26:13.1999960Z   /Library/Java/JavaVirtualMachines/Temurin-Hotspot-11.jdk/Contents/Home/bin/java -jar /Library/Frameworks/Xamarin.Android.framework/Libraries/xbuild/Xamarin/Android/apksigner.jar sign --ks "/Users/runner/.local/share/Xamarin/Mono for Android/debug.keystore" --ks-pass pass:android --ks-key-alias androiddebugkey --key-pass pass:android --min-sdk-version 26 --max-sdk-version 30  /Users/runner/work/1/b/release/com.[---]-Signed.apk 

you'll also note prior to this step, MSBuild creates a debug keystore under the _CreateAndroidDebugSigningKey step.

essentially, if you wanna zipalign and sign using your own keys, use /t:PackageForAndroid, else let the /t:SignAndroidPackage do it for you

[GreatBarrier86]

@Maxgamerboy1, I'm leaving this here for anyone else that happens to run across this same question. You can pass in MSBuild arguments to sign the APK file, regardless of whether its your key or not. Maybe it will help someone in the future because i did need to research how to do this.

/p:OutputPath=$(build.binariesdirectory)/$(BuildConfiguration)/$(TestOutputFolder) /t:SignAndroidPackage /p:AndroidKeyStore=true /p:AndroidSigningKeyAlias=$(KeystoreAlias) /p:AndroidSigningKeyPass=$(CertificatePassword) /p:AndroidSigningKeyStore=$(Build.SourcesDirectory)/MyKeystore.file /p:AndroidSigningStorePass=$(CertificatePassword)

Taken from here: https://www.jenx.si/2020/04/05/xamarin-forms-developers-how-to-sign-android-apk/