java.nio.BufferUnderflowException while trying to run android tests with proguarded build on compileSdkVersion 31

[dmitry-ryadnenko-izettle]

The same as #70.

Reproduced with AGP 7.0.4, 7.1.1. When compileSdkVersion is set to 31. Everything works great when compileSdkVersion is 30.

R8 version is 3.1.51 (AGP 7.1.1 one).

This blocks us from upgrading a lot of dependencies, including upgrading Kotlin to the latest version.

We run Flank using Java 11.

java.nio.BufferUnderflowException
	at java.base/java.nio.Buffer.nextGetIndex(Buffer.java:651)
	at java.base/java.nio.HeapByteBuffer.getInt(HeapByteBuffer.java:402)
	at com.linkedin.dex.spec.AnnotationOffItem.<init>(AnnotationOffItem.kt:13)
	at com.linkedin.dex.spec.AnnotationSetItem$Companion.create(AnnotationSetItem.kt:18)
	at com.linkedin.dex.parser.AnnotationUtilsKt.checkIfAnnotationIsInherited(AnnotationUtils.kt:89)
	at com.linkedin.dex.parser.AnnotationUtilsKt.getTestAnnotation(AnnotationUtils.kt:77)
	at com.linkedin.dex.parser.AnnotationUtilsKt.getClassAnnotationValues(AnnotationUtils.kt:46)
	at ftl.run.platform.android.CreateAndroidTestContextKt.isParametrizedClass(CreateAndroidTestContext.kt:210)
	at ftl.run.platform.android.CreateAndroidTestContextKt.getParametrizedClasses(CreateAndroidTestContext.kt:200)
	at ftl.run.platform.android.CreateAndroidTestContextKt.getFlankTestMethods(CreateAndroidTestContext.kt:153)
	at ftl.run.platform.android.CreateAndroidTestContextKt.calculateShardsNormally(CreateAndroidTestContext.kt:86)
	at ftl.run.platform.android.CreateAndroidTestContextKt.calculateShards(CreateAndroidTestContext.kt:112)
	at ftl.run.platform.android.CreateAndroidTestContextKt.access$calculateShards(CreateAndroidTestContext.kt:1)
	at ftl.run.platform.android.CreateAndroidTestContextKt$setupShards$2$1$1.invokeSuspend(CreateAndroidTestContext.kt:49)
	at kotlin.coroutines.jvm.internal.BaseContinuationImpl.resumeWith(ContinuationImpl.kt:33)
	at kotlinx.coroutines.DispatchedTask.run(DispatchedTask.kt:106)
	at kotlinx.coroutines.EventLoopImplBase.processNextEvent(EventLoop.common.kt:274)
	at kotlinx.coroutines.BlockingCoroutine.joinBlocking(Builders.kt:85)
	at kotlinx.coroutines.BuildersKt__BuildersKt.runBlocking(Builders.kt:59)
	at kotlinx.coroutines.BuildersKt.runBlocking(Unknown Source)
	at kotlinx.coroutines.BuildersKt__BuildersKt.runBlocking$default(Builders.kt:38)
	at kotlinx.coroutines.BuildersKt.runBlocking$default(Unknown Source)
	at ftl.presentation.RunStateKt.runBlockingWithObservingRunState(RunState.kt:17)
	at ftl.domain.RunTestAndroidKt.invoke(RunTestAndroid.kt:63)
	at ftl.presentation.cli.firebase.test.android.AndroidRunCommand.run(AndroidRunCommand.kt:58)
	at picocli.CommandLine.executeUserObject(CommandLine.java:1939)
	at picocli.CommandLine.access$1300(CommandLine.java:145)
	at picocli.CommandLine$RunLast.executeUserObjectOfLastSubcommandWithSameParent(CommandLine.java:2358)
	at picocli.CommandLine$RunLast.handle(CommandLine.java:2352)
	at picocli.CommandLine$RunLast.handle(CommandLine.java:2314)
	at picocli.CommandLine$AbstractParseResultHandler.execute(CommandLine.java:2179)
	at picocli.CommandLine$RunLast.execute(CommandLine.java:2316)
	at picocli.CommandLine.execute(CommandLine.java:2078)
	at ftl.Main$main$1.invoke(Main.kt:12)
	at ftl.Main$main$1.invoke(Main.kt:10)
	at ftl.run.exception.ExceptionHandlerKt.withGlobalExceptionHandling(ExceptionHandler.kt:28)
	at ftl.run.exception.ExceptionHandlerKt.withGlobalExceptionHandling(ExceptionHandler.kt:17)
	at ftl.Main.main(Main.kt:10)

[dmitry-ryadnenko-izettle]

Also, funny thing, this issue blocks us from using M1 Macbooks. Because there is a bug in Room 2.3 that breaks kapt on M1 Macbooks. It is fixed in Room 2.4. To upgrade to Room 2.4 we need to update our compileSdkVersion to 31. And after we update our compileSdkVersion to 31 Flank breaks because of this here issue.

[kkoser]

Sorry for the delay, I'll look into this this week. Interesting that room 2.4 breaks it, could be some code DTP doesn't understand or just a bad transitive dependency. Hopefully that will be enough to repro and dig in

[dmitry-ryadnenko-izettle]

@kkoser That's because Room's 2.3 SQLite doesn't support M1 chips. See https://stackoverflow.com/questions/68884589/caused-by-java-lang-exception-no-native-library-is-found-for-os-name-mac-and-o. But this Room issue is not related to this here issue.

[dmitry-ryadnenko-izettle]

The issue we have is that we can't upgrade to compileSdkVersion 31.

[dmitry-ryadnenko-izettle]

@kkoser I'm trying to build and run this, but I'm getting this error when I run assembled parser.jar

Exception in thread "main" java.lang.NoSuchMethodError: java.nio.ByteBuffer.position(I)Ljava/nio/ByteBuffer;
	at com.linkedin.dex.spec.DexFile.<init>(DexFile.kt:34)
	at com.linkedin.dex.parser.DexParser$Companion$readDexFiles$1$2.invoke(DexParser.kt:92)
	at com.linkedin.dex.parser.DexParser$Companion$readDexFiles$1$2.invoke(DexParser.kt:48)
	at kotlin.sequences.TransformingSequence$iterator$1.next(Sequences.kt:210)
	at kotlin.sequences.SequencesKt___SequencesKt.toCollection(_Sequences.kt:752)
	at kotlin.sequences.SequencesKt___SequencesKt.toMutableList(_Sequences.kt:782)
	at kotlin.sequences.SequencesKt___SequencesKt.toList(_Sequences.kt:773)
	at com.linkedin.dex.parser.DexParser$Companion.readDexFiles(DexParser.kt:93)
	at com.linkedin.dex.parser.DexParser$Companion.findTestMethods(DexParser.kt:76)
	at com.linkedin.dex.parser.DexParser$Companion.findTestNames(DexParser.kt:64)
	at com.linkedin.dex.parser.DexParserCommand.run(DexParser.kt:38)
	at com.github.ajalt.clikt.parsers.Parser.parse(Parser.kt:180)
	at com.github.ajalt.clikt.parsers.Parser.parse(Parser.kt:17)
	at com.github.ajalt.clikt.core.CliktCommand.parse(CliktCommand.kt:396)
	at com.github.ajalt.clikt.core.CliktCommand.parse$default(CliktCommand.kt:393)
	at com.github.ajalt.clikt.core.CliktCommand.main(CliktCommand.kt:411)
	at com.github.ajalt.clikt.core.CliktCommand.main(CliktCommand.kt:436)
	at com.linkedin.dex.parser.DexParser$Companion.main(DexParser.kt:55)
	at com.linkedin.dex.parser.DexParser.main(DexParser.kt)

It looks like this is connected somehow to JVM versions I use to build or run parser.jar. Do you have any quick hint how to get around this?

[dmitry-ryadnenko-izettle]

Alright, managed to get around this by doing this

        (this.byteBuffer as Buffer).position(0)

[dmitry-ryadnenko-izettle]

Can't reproduce this locally without using Flank. Which JDK do you use for building this project?

[dmitry-ryadnenko-izettle]

Did some investigation today. Removed all the tests from the app. Still got the error. Kept removing code from the androidTest source set. Managed to find out that the cause of the error is withing our Dagger setup. Removing our Dagger setup stops the error from occurring. It will be hard to find the cause of error by bisecting our Dagger setup. Because in Dagger setups everything is interconnected.

Next thing I plan to do is to build Flank locally and to try to run it and reproduce the issue with debugger turned on.

[cckroets]

Keep us posted @dmitry-ryadnenko-izettle. I did sanity check that the dex-test-parser tests pass with a vanilla build of 7.1.2, compileSdkVersion 31, and R8 enabled. I'm interested to hear what in the Dagger setup is causing this.

[dmitry-ryadnenko-izettle]

@kkoser @cckroets

Alright. So I managed to figure out that the exception (trying to read beyond buffer's limits, BufferUnderflowException) is thrown while trying to run DexFile.getClassAnnotationValues() on this interface

@ClassTokenAnnotation(
    favoriteClasses = {String.class, int.class, void.class, int[].class, Object[][][].class})
public interface SimplerInterface {
  int myField = 1;
}

Where ClassTokenAnnotation is this.

public @interface ClassTokenAnnotation {
  Class<?>[] favoriteClasses();
}

I'm not sure though that these class definitions are 100% the same as in my code. I found class names while debugging the code. And then I searched for these classes and found them on GitHub here. And after a short investigation I still have no idea how these classes ended up in our dex files. Through some dependencies of course, but it's hard to say through which ones. Probably some Dagger stuff.

Can you help me with the investigation from this point? It's hard for me to continue the investigation myself because it's hard to navigate through all this byte buffer logic, and I just not familiar enough with the domain to understand what's going on.

[dmitry-ryadnenko-izettle]

I think I have a fix. Please check the PR above.

[cckroets]

Thanks for the detailed investigation, will take a look at the PR.

It's interesting that those classes from annotation-tools are on the test's runtime classpath, rather than just on the annotation-processor or some other tooling classpath.

Assuming you are using Gradle, if you don't expect that they should be there, you can run ./gradlew androidDependencies to try and debug why they are being used in the first place

[cckroets]

Thanks for fixing!

Fixed by #73