Ubuntu 18.04 agents have unexpected default locale #3120

dougbu · 2021-04-06T17:43:31Z

Description
We (ASP.NET Core team) are attempting to move from Ubuntu 16.04 to 18.04 Microsoft-hosted agents. Our builds are failing because libicu60 is not installed on the newer agents. See also https://github.com/actions/virtual-environments/blob/main/images/linux/Ubuntu1604-README.md (where libicu55 is installed) versus https://github.com/actions/virtual-environments/blob/main/images/linux/Ubuntu1804-README.md (where the string icu isn't matched at all).

Area for Triage:

Packages

Question, Bug, or Feature?:

Bug

But if there's a task to install libicu60 (the version aligned with Ubuntu 18.04) in our pipelines, I'm all ears.

Virtual environments affected

Image version

Image version where you are experiencing the issue.
20210330.1

Expected behavior
A description of what you expected to happen.
Expect our tests to continue working correctly. Note https://docs.microsoft.com/en-us/dotnet/core/install/linux-ubuntu#dependencies lists libicu60 as a prerequisite for .NET Core on Ubuntu.

Actual behavior
A description of what is actually happening.
libicu60 is missing from the image and some of our tests are failing due to string ordering issues. See for example failures in https://dev.azure.com/dnceng/public/_build/results?buildId=1073540

Repro steps
A description with steps to reproduce the issue. If your have a public example or repo to share,
please provide the link.

Rerun one of the following PR validation builds

The text was updated successfully, but these errors were encountered:

al-cheb · 2021-04-06T18:54:02Z

Hello, @dougbu
The libicu60 is already pre-installed on Ubuntu Server 18.04.

jobs:
  build:
    runs-on: ubuntu-18.04
    steps:

      - name: libicu60
        run: dpkg -l | grep libicu60

dougbu · 2021-04-06T19:10:45Z

@al-cheb was that changed in the last day or so❔ And, are you saying https://github.com/actions/virtual-environments/blob/main/images/linux/Ubuntu1804-README.md is not up-to-date❔

Separately, are the agents used for Azure Pipelines and GitHub Actions different despite the fact https://docs.microsoft.com/en-us/azure/devops/pipelines/agents/hosted?view=azure-devops&tabs=yaml#software links to the above README.md file❔

The missing package would completely explain the test failures we're seeing for the listed PRs.

dougbu · 2021-04-06T19:38:32Z

Shoot. The agents aren't appropriate for our builds but the problem isn't as simple as adding libicu60. For example, I tried a manual installation of the package in aspnet/BuildTools#998 and saw similar output to you @al-cheb: libicu60 is already the newest version (60.2-3ubuntu3.1). at https://dev.azure.com/dnceng/public/_build/results?buildId=1074813&view=logs&j=5818c23b-5f5e-5596-064a-6c64d403371a&t=28aca32c-5ccf-5757-b167-7b688276e977&l=17. This is now basically a documentation issue because https://github.com/actions/virtual-environments/blob/main/images/linux/Ubuntu1804-README.md doesn't list the included libicu60 package.

Not sure where to look next to enable us to use the newer Ubuntu agents…

al-cheb · 2021-04-07T07:37:27Z

@al-cheb was that changed in the last day or so❔ And, are you saying https://github.com/actions/virtual-environments/blob/main/images/linux/Ubuntu1804-README.md is not up-to-date❔

We don't include every version of default pre-installed packages in the documentation file.

Separately, are the agents used for Azure Pipelines and GitHub Actions different despite the fact https://docs.microsoft.com/en-us/azure/devops/pipelines/agents/hosted?view=azure-devops&tabs=yaml#software links to the above README.md file❔

It's the same image Azure Pipelines and GitHub Action .

PS. I can reproduce the issue on my test fresh vm with Azure Ubuntu Server 18.04:

  Starting test execution, please wait...
  [xUnit.net 00:00:00.9741394]     Compare_DetectsAllFieldRemovals(typeToCheck: "public struct ComparisonScenarios.StructToRemoveFi"...) [FAIL]
  [xUnit.net 00:00:00.9811200]     Compare_DetectsAllFieldRemovals(typeToCheck: "public class ComparisonScenarios.ClassToRemoveFiel"...) [FAIL]
  Failed   Compare_DetectsAllFieldRemovals(typeToCheck: "public struct ComparisonScenarios.StructToRemoveFi"...)
  Error Message:
   Assert.Equal() Failure
  Expected: List<BreakingChange> [BreakingChange { Kind = Removal, MemberId = "public const System.Int32 ConstToChangeValue = 1", TypeId = "public struct ComparisonScenarios.StructToRemoveFi"... }, BreakingChange { Kind = Removal, MemberId = "public const System.Int32 ConstToMakeField = 2", TypeId = "public struct ComparisonScenarios.StructToRemoveFi"... }, BreakingChange { Kind = Removal, MemberId = "public readonly System.Int32 FieldToMakeWritable", TypeId = "public struct ComparisonScenarios.StructToRemoveFi"... }, BreakingChange { Kind = Removal, MemberId = "public static readonly System.Int32 StaticFieldToM"..., TypeId = "public struct ComparisonScenarios.StructToRemoveFi"... }, BreakingChange { Kind = Removal, MemberId = "public static readonly System.Int32 StaticFieldToM"..., TypeId = "public struct ComparisonScenarios.StructToRemoveFi"... }, ...]
  Actual:   List<BreakingChange> [BreakingChange { Kind = Removal, MemberId = "public System.Int32 FieldToMakeReadonly", TypeId = "public struct ComparisonScenarios.StructToRemoveFi"... }, BreakingChange { Kind = Removal, MemberId = "public System.Int32 FieldToRemove", TypeId = "public struct ComparisonScenarios.StructToRemoveFi"... }, BreakingChange { Kind = Removal, MemberId = "public const System.Int32 ConstToChangeValue = 1", TypeId = "public struct ComparisonScenarios.StructToRemoveFi"... }, BreakingChange { Kind = Removal, MemberId = "public const System.Int32 ConstToMakeField = 2", TypeId = "public struct ComparisonScenarios.StructToRemoveFi"... }, BreakingChange { Kind = Removal, MemberId = "public readonly System.Int32 FieldToMakeWritable", TypeId = "public struct ComparisonScenarios.StructToRemoveFi"... }, ...]

  Failed   Compare_DetectsAllFieldRemovals(typeToCheck: "public class ComparisonScenarios.ClassToRemoveFiel"...)
  Error Message:
   Assert.Equal() Failure
  Expected: List<BreakingChange> [BreakingChange { Kind = Removal, MemberId = "public const System.Int32 ConstToChangeValue = 1", TypeId = "public class ComparisonScenarios.ClassToRemoveFiel"... }, BreakingChange { Kind = Removal, MemberId = "public const System.Int32 ConstToMakeField = 2", TypeId = "public class ComparisonScenarios.ClassToRemoveFiel"... }, BreakingChange { Kind = Removal, MemberId = "public readonly System.Int32 FieldToMakeWritable", TypeId = "public class ComparisonScenarios.ClassToRemoveFiel"... }, BreakingChange { Kind = Removal, MemberId = "public static readonly System.Int32 StaticFieldToM"..., TypeId = "public class ComparisonScenarios.ClassToRemoveFiel"... }, BreakingChange { Kind = Removal, MemberId = "public static readonly System.Int32 StaticFieldToM"..., TypeId = "public class ComparisonScenarios.ClassToRemoveFiel"... }, ...]
  Actual:   List<BreakingChange> [BreakingChange { Kind = Removal, MemberId = "public System.Int32 FieldToMakeReadonly", TypeId = "public class ComparisonScenarios.ClassToRemoveFiel"... }, BreakingChange { Kind = Removal, MemberId = "public System.Int32 FieldToRemove", TypeId = "public class ComparisonScenarios.ClassToRemoveFiel"... }, BreakingChange { Kind = Removal, MemberId = "public const System.Int32 ConstToChangeValue = 1", TypeId = "public class ComparisonScenarios.ClassToRemoveFiel"... }, BreakingChange { Kind = Removal, MemberId = "public const System.Int32 ConstToMakeField = 2", TypeId = "public class ComparisonScenarios.ClassToRemoveFiel"... }, BreakingChange { Kind = Removal, MemberId = "public readonly System.Int32 FieldToMakeWritable", TypeId = "public class ComparisonScenarios.ClassToRemoveFiel"... }, ...]

al-cheb · 2021-04-07T10:40:23Z

Looks like the tests are fixed - aspnet/BuildTools#1000

dougbu · 2021-04-07T19:17:40Z

@al-cheb that PR validated using Ubuntu 16.04. The root cause of the test failures is an unexpected locale (C.UTF-8) on the 18.04 agents. Will let you know when I've tested the solution. Still catching up on the morning's email…

dougbu · 2021-04-07T20:35:30Z

Did a quick test with aspnet/BuildTools#999. On Ubuntu 16.04 agents, the default locale is en_US.UTF-8 while on 18.04 agents, it's C.UTF-8. This isn't a great idea for .NET Core 2.1 and the difference between the two platforms is unexpected. Does this repo set the locale for the agents❔ Or is this a change Canonical made between the two versions❔

I only see LC_ALL mentioned in https://github.com/actions/virtual-environments/blob/b7f276c003aea42575b52247bdb2183e355fca2f/images/macos/provision/configuration/environment/bashrc but there are other ways to control the locale…

Ubuntu 16.04

Script contents:
locale
========================== Starting Command Output ===========================
/bin/bash --noprofile --norc /home/vsts/work/_temp/46cde479-ce11-437b-a209-2ffec3c7eea6.sh
LANG=en_US.UTF-8
LANGUAGE=
LC_CTYPE="en_US.UTF-8"
LC_NUMERIC="en_US.UTF-8"
LC_TIME="en_US.UTF-8"
LC_COLLATE="en_US.UTF-8"
LC_MONETARY="en_US.UTF-8"
LC_MESSAGES="en_US.UTF-8"
LC_PAPER="en_US.UTF-8"
LC_NAME="en_US.UTF-8"
LC_ADDRESS="en_US.UTF-8"
LC_TELEPHONE="en_US.UTF-8"
LC_MEASUREMENT="en_US.UTF-8"
LC_IDENTIFICATION="en_US.UTF-8"
LC_ALL=

Ubuntu 18.04

Script contents:
locale
========================== Starting Command Output ===========================
/bin/bash --noprofile --norc /home/vsts/work/_temp/b531dbb7-bdb2-4246-95f9-6c8990e6bda0.sh
LANG=C.UTF-8
LANGUAGE=
LC_CTYPE="C.UTF-8"
LC_NUMERIC="C.UTF-8"
LC_TIME="C.UTF-8"
LC_COLLATE="C.UTF-8"
LC_MONETARY="C.UTF-8"
LC_MESSAGES="C.UTF-8"
LC_PAPER="C.UTF-8"
LC_NAME="C.UTF-8"
LC_ADDRESS="C.UTF-8"
LC_TELEPHONE="C.UTF-8"
LC_MEASUREMENT="C.UTF-8"
LC_IDENTIFICATION="C.UTF-8"
LC_ALL=

Workaround

Fortunately, the workaround is pretty simple and no locale-gen is needed. I just set LC_ALL, LANG and LANGUAGE to en_US.UTF-8.

But, if it's an unintended change between the two images, I suggest that be corrected.

dougbu · 2021-04-08T00:31:35Z

Renamed to reflect current understanding of the problem.

/btw

@al-cheb was that changed in the last day or so❔ And, are you saying https://github.com/actions/virtual-environments/blob/main/images/linux/Ubuntu1804-README.md is not up-to-date❔

We don't include every version of default pre-installed packages in the documentation file.

Ah, makes sense 😺

al-cheb · 2021-04-08T06:40:13Z

We don't have any plans to change the default locale on Azure Ubuntu images provided by Canonical publisher - https://stackoverflow.com/questions/55673886/what-is-the-difference-between-c-utf-8-and-en-us-utf-8-locales (e.g. - https://lintut.com/how-to-set-up-system-locale-on-ubuntu-18-04/)

OS	locale	locale -a	/etc/default/locale
Ubuntu Server 16.04	LANG=en_US.UTF-8 LANGUAGE= LC_CTYPE="en_US.UTF-8" LC_NUMERIC="en_US.UTF-8" LC_TIME="en_US.UTF-8" LC_COLLATE="en_US.UTF-8" LC_MONETARY="en_US.UTF-8" LC_MESSAGES="en_US.UTF-8" LC_PAPER="en_US.UTF-8" LC_NAME="en_US.UTF-8" LC_ADDRESS="en_US.UTF-8" LC_TELEPHONE="en_US.UTF-8"	C C.UTF-8 en_US.utf8 POSIX	LANG=en_US.UTF-8
Ubuntu Server 18.04	LANG=C.UTF-8 LANGUAGE= LC_CTYPE="C.UTF-8" LC_NUMERIC="C.UTF-8" LC_TIME="C.UTF-8" LC_COLLATE="C.UTF-8" LC_MONETARY="C.UTF-8" LC_MESSAGES="C.UTF-8" LC_PAPER="C.UTF-8" LC_NAME="C.UTF-8" LC_ADDRESS="C.UTF-8" LC_TELEPHONE="C.UTF-8" LC_MEASUREMENT="C.UTF-8" LC_IDENTIFICATION="C.UTF-8" LC_ALL=	C C.UTF-8 POSIX en_US.utf8	LANG=C.UTF-8
Ubuntu Server 20.04	LANG=C.UTF-8 LANGUAGE= LC_CTYPE="C.UTF-8" LC_NUMERIC="C.UTF-8" LC_TIME="C.UTF-8" LC_COLLATE="C.UTF-8" LC_MONETARY="C.UTF-8" LC_MESSAGES="C.UTF-8" LC_PAPER="C.UTF-8" LC_NAME="C.UTF-8" LC_ADDRESS="C.UTF-8" LC_TELEPHONE="C.UTF-8" LC_MEASUREMENT="C.UTF-8" LC_IDENTIFICATION="C.UTF-8" LC_ALL=	C C.UTF-8 POSIX en_US.utf8	LANG=C.UTF-8
Ubuntu Server 20.10	LANG=C.UTF-8 LANGUAGE= LC_CTYPE="C.UTF-8" LC_NUMERIC="C.UTF-8" LC_TIME="C.UTF-8" LC_COLLATE="C.UTF-8" LC_MONETARY="C.UTF-8" LC_MESSAGES="C.UTF-8" LC_PAPER="C.UTF-8" LC_NAME="C.UTF-8" LC_ADDRESS="C.UTF-8" LC_TELEPHONE="C.UTF-8" LC_MEASUREMENT="C.UTF-8" LC_IDENTIFICATION="C.UTF-8" LC_ALL=	C C.UTF-8 POSIX en_US.utf8	LANG=C.UTF-8

dougbu · 2021-04-08T19:51:29Z

🆗 I'll stick w/ our workaround.

/cc @JunTaoLuo @wtgodbe, please do the same in the branches and repos you're updating.

dougbu · 2021-04-08T19:52:05Z

See the bottom of #3120 (comment) for the workaround I meant

dougbu added the needs triage label Apr 6, 2021

This was referenced Apr 6, 2021

Use Ubuntu 18.04 build agents dotnet/efcore#24592

Merged

Prep for branding & baselines aspnet/BuildTools#997

Merged

[release/2.1] Update branding to 2.1.28 dotnet/aspnetcore#31547

Merged

al-cheb added OS: Ubuntu question Further information is requested and removed needs triage labels Apr 6, 2021

dougbu mentioned this issue Apr 7, 2021

[release/2.1] Update branding to 2.1.28 II dotnet/aspnetcore#31569

Closed

al-cheb self-assigned this Apr 7, 2021

dougbu changed the title ~~Ubuntu 18.04 agents lack libicu60 package~~ Ubuntu 18.04 agents have unexpected default locale Apr 8, 2021

dougbu closed this as completed Apr 8, 2021

shivammathur mentioned this issue May 19, 2021

Odd iconv behavior converting from utf8 to ascii shivammathur/setup-php#459

Closed

5 tasks

sbouchet mentioned this issue May 25, 2021

fix tests path jbosstools/jbosstools-hibernate#1346

Merged

hololeap mentioned this issue Jul 20, 2021

Failing citeproc tests jgm/pandoc#7449

Closed

rajsite mentioned this issue Aug 1, 2023

Add formatting API to date column ni/nimble#1381

Merged

1 task

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Ubuntu 18.04 agents have unexpected default locale #3120

Ubuntu 18.04 agents have unexpected default locale #3120

dougbu commented Apr 6, 2021

al-cheb commented Apr 6, 2021 •

edited

Loading

dougbu commented Apr 6, 2021

dougbu commented Apr 6, 2021

al-cheb commented Apr 7, 2021 •

edited

Loading

al-cheb commented Apr 7, 2021

dougbu commented Apr 7, 2021

dougbu commented Apr 7, 2021

dougbu commented Apr 8, 2021

al-cheb commented Apr 8, 2021 •

edited

Loading

dougbu commented Apr 8, 2021

dougbu commented Apr 8, 2021

Ubuntu 18.04 agents have unexpected default locale #3120

Ubuntu 18.04 agents have unexpected default locale #3120

Comments

dougbu commented Apr 6, 2021

al-cheb commented Apr 6, 2021 • edited Loading

dougbu commented Apr 6, 2021

dougbu commented Apr 6, 2021

al-cheb commented Apr 7, 2021 • edited Loading

al-cheb commented Apr 7, 2021

dougbu commented Apr 7, 2021

dougbu commented Apr 7, 2021

Ubuntu 16.04

Ubuntu 18.04

Workaround

dougbu commented Apr 8, 2021

al-cheb commented Apr 8, 2021 • edited Loading

dougbu commented Apr 8, 2021

dougbu commented Apr 8, 2021

al-cheb commented Apr 6, 2021 •

edited

Loading

al-cheb commented Apr 7, 2021 •

edited

Loading

al-cheb commented Apr 8, 2021 •

edited

Loading