Use system calls to get terminal size on Linux / Mac #4497

alexarchambault · 2025-02-06T20:01:45Z

This makes Mill use my native-terminal library (formerly windows-ansi) to get the terminal size. That way, Mill doesn't run tput commands every ~100 ms to query the terminal size.

lihaoyi · 2025-02-07T02:53:54Z

runner/client/src/mill/runner/client/MillProcessLauncher.java

+    if (mill.main.client.Util.hasConsole())
+      try {
+        NativeTerminal.getSize();
+        canUse = true;
+      } catch (Throwable ex) {
+        canUse = false;
+      }
+    else canUse = false;
+
+    canUseNativeTerminal = canUse;
+  }
+
  static void writeTerminalDims(boolean tputExists, Path serverDir) throws Exception {
    String str;

    try {
-      if (java.lang.System.console() == null) str = "0 0";
+      if (!mill.main.client.Util.hasConsole()) str = "0 0";


It seems we're checking mill.main.client.Util.hasConsole() twice here; once in the static blockl and once in writeTerminalDims. We can probably skip one of those checks right?

hasConsole caches its result now, so calling it twice shouldn't incur any cost

lihaoyi · 2025-02-07T03:07:14Z

Trying this out on my macbook seems to add about ~200ms latency (~150ms -> ~350ms), running ./mill dist.installLocal then time ~/Github/mill/mill-assembly.jar version repeatedly in an empty folder. @alexarchambault can you see if you get similar numbers? I wonder where that 200ms is going, but if we can't figure out we may have to stick with the shelling-out-to-tput approach even if it's a bit ugly

alexarchambault · 2025-02-07T11:22:29Z

I'm getting the same numbers :/ Printing the duration of some of the calls to get the terminal size, I get things like

First NativeTerminal.getSize duration: 182
Second NativeTerminal.getSize duration: 0
First tput thing duration: 19
Second tput thing duration: 13

So once the native things are setup, getting the terminal size is sub-ms.

Upon initialization, jansi reads a binary file (like libjansi.jnilib) in its resources, writes it in a temporary directory, and loads it with System.load. Manually circumventing the resources / temp dir stuff speeds things up a lot, and gives back the same total time for time …/mill/mill-assembly.jar version.

lihaoyi · 2025-02-07T14:14:02Z

If the resource/tempdir stuff is circumventable, does that mean that it isn't necessary for the jni functionality?

alexarchambault · 2025-02-07T17:41:42Z

If the resource/tempdir stuff is circumventable, does that mean that it isn't necessary for the jni functionality?

Not necessarily, we can load the native library from elsewhere on disk if we want to. I just pushed code that loads it via the coursier cache (this needs coursier/coursier#3272, so this is going to need new coursier and coursier-interface releases). If the native library is already in cache, there's no need to write a temporary file.

From native images, we can also statically link the native library, so that nothing needs to be loaded at runtime. But this needs the JNI library to push static libraries somewhere (like here or here), and jansi doesn't. It worked fine for other JNI libraries in Scala CLI, although I haven't seen that being done elsewhere.

alexarchambault · 2025-02-07T18:06:24Z

From Java 22 onwards, it should be possible to use FFM, JLine is able to for sure. This doesn't need to load custom native libraries upfront (although I have very little experience with it for now).

lihaoyi · 2025-02-08T02:01:47Z

Unpacking the native library and caching it via coursier sounds reasonable. We just need to make sure Coursier is not on the hot path so we don't need to classload all the coursier/scala code when the library is already unpacked.

alexarchambault · 2025-02-17T22:21:44Z

(Seems this is surfacing issues with newer versions of coursier)

Conflicts: build.mill scalalib/src/mill/scalalib/JsonFormatters.scala

lihaoyi · 2025-03-01T02:49:32Z

I'm having trouble trying to ./mill dist.installLocal this PR on my M1 macbook

runner.client.buildInfoMembers java.lang.RuntimeException: Cannot get jansi version from compile class path
    scala.sys.package$.error(package.scala:27)
    build_.runner.package_$client$.$anonfun$buildInfoMembers$7(package.mill:25)
    scala.Option.getOrElse(Option.scala:201)
    build_.runner.package_$client$.$anonfun$buildInfoMembers$3(package.mill:25)

alexarchambault · 2025-03-11T16:33:32Z

I'm having trouble trying to ./mill dist.installLocal this PR on my M1 macbook

I have a Mac ARM too, and I'm getting no issues. Could you have jansi in your ~/.ivy2/local? That's the only scenario I can think of, where that exception would be thrown.

What does ./mill show runner.client.compileClasspath say? It should contain a JAR for jansi (and the code throws if it doesn't look like jansi-<version>.jar)

lihaoyi · 2025-03-12T09:32:46Z

Hah, indeed I have a ~/.ivy2/local/org.fusesource.jansi/jansi/2.4.1/jars/jansi.jar jar on the classpath. Let me try removing it and see if that helps

lihaoyi · 2025-03-12T09:37:36Z

Seems to work now!

lihaoyi · 2025-03-12T09:42:59Z

runner/client/src/mill/runner/client/MillProcessLauncher.java

@@ -248,15 +250,81 @@ static int getTerminalDim(String s, boolean inheritError) throws Exception {

  private static AtomicReference<String> memoizedTerminalDims = new AtomicReference();

+  private static final boolean canUseNativeTerminal;
+
+  static {


This block is pretty messy, and I'm wondering if there's some way we can make it obvious that the fast path is fast and does not classload any heavy libraries. That is the case now, but the fact that it's not obvious from the code makes it likely to regress accidentally if people touch this code.

Perhaps we could do something like what we do for the JVM versions in

mill/runner/client/src/mill/runner/client/MillProcessLauncher.java

Lines 134 to 152 in 392f32d

if (jvmId != null) {

// Fast path to avoid calling `CoursierClient` and paying the classloading cost

// when the `javaHome` JVM has already been initialized for the configured `jvmId`

// and is ready to use directly

Path millJavaHomeFile = Paths.get(".").resolve(out).resolve(millJavaHome);

if (Files.exists(millJavaHomeFile)) {

String[] savedJavaHomeInfo = Files.readString(millJavaHomeFile).split(" ");

if (savedJavaHomeInfo[0].equals(jvmId)) {

javaHome = savedJavaHomeInfo[1];

}

}

if (javaHome == null) {

javaHome = CoursierClient.resolveJavaHome(jvmId).getAbsolutePath();

Files.createDirectories(millJavaHomeFile.getParent());

Files.write(millJavaHomeFile, (jvmId + " " + javaHome).getBytes());

}

}

, where we do a fast-path check whether a cache/marker file exists on disk, and put the complex code inside the conditional so it only runs in the slow path?

I'm going to try to make that clearer from the code, but it already goes first through a fast path, then through a slower one. The fast path goes from the beginning of the static { block up to File jansiLib = …. It calls a method on coursier.paths.CachePath, but it's a fast thing, that shouldn't load much more coursier classes (the method is here, and it loads that class, which itself loads directories-jvm classes - it's all pure Java classes). It does all that to compute the coursier "archive cache" location (where it unpacks archives), and the location of the jansi dynamic library inside it.

Yes I'm aware it currently goes through all lightweight JVM classes, and the manual startup-time benchmarks verify it. But we should refactor the code to make it obvious at a glance at a single if-conditional rather than having to read through 70 lines of imperative code and dig through third-party libraries, otherwise it is prone to regress when someone overlooks it and makes changes later

Just put the loading logic aside in a separate class, with two methods, tryLoadFast and loadSlow (did you mean that kind of thing?)

lihaoyi · 2025-03-12T09:57:52Z

Added some rough measurements and this cuts down the time for getDimensions from ~5ms to ~0ms, which is nice

alexarchambault · 2025-03-13T11:09:52Z

Seems this puts jansi in ~/.ivy2/local…

Conflicts: integration/feature/init/src/MillInitMavenTests.scala

alexarchambault · 2025-03-14T11:57:29Z

It seems these consistently fail on CI, but not locally, I'm not sure why

integration.feature[output-directory].packaged.server.test 2 tests failed: 
  mill.integration.OutputDirectoryTests mill.integration.OutputDirectoryTests.Output directory elsewhere in workspace
  mill.integration.OutputDirectoryTests mill.integration.OutputDirectoryTests.Output directory outside workspace

lihaoyi · 2025-03-18T00:14:58Z

It seems these consistently fail on CI, but not locally, I'm not sure why

integration.feature[output-directory].packaged.server.test 2 tests failed: 
  mill.integration.OutputDirectoryTests mill.integration.OutputDirectoryTests.Output directory elsewhere in workspace
  mill.integration.OutputDirectoryTests mill.integration.OutputDirectoryTests.Output directory outside workspace

Not sure, but you can use the commented-out debug section of run-tests.yml to try and minimize what is running in CI and figure add logging to debug it

alexarchambault and others added 2 commits February 6, 2025 20:57

Use system calls to get terminal size on Linux / Mac

3548d0a

[autofix.ci] apply automated fixes

3932da7

alexarchambault marked this pull request as ready for review February 6, 2025 21:15

lihaoyi reviewed Feb 7, 2025

View reviewed changes

Load jansi native lib ourselves to speed things up

068a0f7

alexarchambault marked this pull request as draft February 17, 2025 10:27

alexarchambault added 4 commits February 17, 2025 11:27

Merge branch '0.12.x' into native-terminal

baea11d

Fix coursier interface version

e033ef5

Fix

253cb6b

Add missing coursier codecs

39f2c86

alexarchambault and others added 3 commits February 28, 2025 15:28

Merge branch '0.12.x' into native-terminal

f2fe2d1

Conflicts: build.mill scalalib/src/mill/scalalib/JsonFormatters.scala

Tweaking

383cac1

[autofix.ci] apply automated fixes

c2a21c1

alexarchambault marked this pull request as ready for review February 28, 2025 19:43

Merge branch '0.12.x' into pr/native-terminal

ab5829f

lihaoyi reviewed Mar 12, 2025

View reviewed changes

alexarchambault added 2 commits March 12, 2025 16:50

Merge branch '0.12.x' into pr/native-terminal

9a85417

Add comments

c4e8887

alexarchambault added 4 commits March 13, 2025 12:32

Merge branch '0.12.x' into pr/native-terminal

302387c

Don't publish jansi in ~/.ivy2/local in integration test

1d9cb82

Merge branch '0.12.x' into pr/native-terminal

9ee14f7

Merge branch '0.12.x' into pr/native-terminal

54b6843

Conflicts: integration/feature/init/src/MillInitMavenTests.scala

Merge branch '0.12.x' into pr/native-terminal

16e17be

alexarchambault marked this pull request as draft March 25, 2025 23:48

alexarchambault added 7 commits March 26, 2025 00:48

debug

c3625e6

debug

ee96932

Adjust logback outfile file upon startup

c525f4a

debug

ba089f6

fixup Adjust logback outfile file upon startup

813a8b6

Merge branch '0.12.x' into pr/native-terminal

b80a92d

Revert debug stuff

439a9c1

alexarchambault marked this pull request as ready for review March 26, 2025 16:45

alexarchambault marked this pull request as draft March 26, 2025 16:45

alexarchambault and others added 2 commits March 26, 2025 18:00

Make fast path / slow path more explicit

ad391e1

[autofix.ci] apply automated fixes

6802c56

alexarchambault marked this pull request as ready for review March 26, 2025 18:37

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Use system calls to get terminal size on Linux / Mac #4497

Use system calls to get terminal size on Linux / Mac #4497

alexarchambault commented Feb 6, 2025

lihaoyi Feb 7, 2025

alexarchambault Mar 26, 2025

lihaoyi commented Feb 7, 2025

alexarchambault commented Feb 7, 2025

lihaoyi commented Feb 7, 2025

alexarchambault commented Feb 7, 2025

alexarchambault commented Feb 7, 2025

lihaoyi commented Feb 8, 2025

alexarchambault commented Feb 17, 2025

lihaoyi commented Mar 1, 2025

alexarchambault commented Mar 11, 2025

lihaoyi commented Mar 12, 2025

lihaoyi commented Mar 12, 2025

lihaoyi Mar 12, 2025

alexarchambault Mar 12, 2025 •

edited

Loading

lihaoyi Mar 18, 2025 •

edited

Loading

alexarchambault Mar 26, 2025

lihaoyi commented Mar 12, 2025

alexarchambault commented Mar 13, 2025

alexarchambault commented Mar 14, 2025

lihaoyi commented Mar 18, 2025

	if (jvmId != null) {

	// Fast path to avoid calling `CoursierClient` and paying the classloading cost
	// when the `javaHome` JVM has already been initialized for the configured `jvmId`
	// and is ready to use directly
	Path millJavaHomeFile = Paths.get(".").resolve(out).resolve(millJavaHome);
	if (Files.exists(millJavaHomeFile)) {
	String[] savedJavaHomeInfo = Files.readString(millJavaHomeFile).split(" ");
	if (savedJavaHomeInfo[0].equals(jvmId)) {
	javaHome = savedJavaHomeInfo[1];
	}
	}

	if (javaHome == null) {
	javaHome = CoursierClient.resolveJavaHome(jvmId).getAbsolutePath();
	Files.createDirectories(millJavaHomeFile.getParent());
	Files.write(millJavaHomeFile, (jvmId + " " + javaHome).getBytes());
	}
	}

Use system calls to get terminal size on Linux / Mac #4497

Are you sure you want to change the base?

Use system calls to get terminal size on Linux / Mac #4497

Conversation

alexarchambault commented Feb 6, 2025

lihaoyi Feb 7, 2025

Choose a reason for hiding this comment

alexarchambault Mar 26, 2025

Choose a reason for hiding this comment

lihaoyi commented Feb 7, 2025

alexarchambault commented Feb 7, 2025

lihaoyi commented Feb 7, 2025

alexarchambault commented Feb 7, 2025

alexarchambault commented Feb 7, 2025

lihaoyi commented Feb 8, 2025

alexarchambault commented Feb 17, 2025

lihaoyi commented Mar 1, 2025

alexarchambault commented Mar 11, 2025

lihaoyi commented Mar 12, 2025

lihaoyi commented Mar 12, 2025

lihaoyi Mar 12, 2025

Choose a reason for hiding this comment

alexarchambault Mar 12, 2025 • edited Loading

Choose a reason for hiding this comment

lihaoyi Mar 18, 2025 • edited Loading

Choose a reason for hiding this comment

alexarchambault Mar 26, 2025

Choose a reason for hiding this comment

lihaoyi commented Mar 12, 2025

alexarchambault commented Mar 13, 2025

alexarchambault commented Mar 14, 2025

lihaoyi commented Mar 18, 2025

alexarchambault Mar 12, 2025 •

edited

Loading

lihaoyi Mar 18, 2025 •

edited

Loading