Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

jpf-nhandler's tests are failing on amd based machines #11

Closed
Harsh4902 opened this issue Jul 8, 2024 · 8 comments · Fixed by #12
Closed

jpf-nhandler's tests are failing on amd based machines #11

Harsh4902 opened this issue Jul 8, 2024 · 8 comments · Fixed by #12

Comments

@Harsh4902
Copy link
Contributor

  • test results on arm based machines
converter.JPF2JVMTest > convertClassTest PASSED
converter.JPF2JVMTest > convertIntegerTest PASSED
converter.JPF2JVMTest > convertStringTest PASSED
converter.JPF2JVMTest > convertHashMapTest PASSED
converter.JPF2JVMTest > convertArrayTest PASSED
converter.JVM2JPFTest > convertClassTest PASSED
converter.JVM2JPFTest > convertIntegerTest PASSED
converter.JVM2JPFTest > convertStringTest PASSED
converter.JVM2JPFTest > convertHashMapTest PASSED
converter.JVM2JPFTest > convertArrayTest PASSED
converter.specific.java_textTest > convertDecimalFormatTest PASSED
converter.specific.java_textTest > convertSimpleDateFormatTest PASSED
converter.specific.java_ioTest > convertFileTest PASSED
converter.specific.java_langTest > convertClassTest PASSED
converter.specific.java_langTest > convertStringTest PASSED
WARNING: An illegal reflective access operation has occurred
WARNING: Illegal reflective access by converter.specific.java_util_regexTest (file:/root/jpf/jpf-nhandler/build/classes/java/test/) to field java.util.regex.Matcher.text
WARNING: Please consider reporting this to the maintainers of converter.specific.java_util_regexTest
WARNING: Use --illegal-access=warn to enable warnings of further illegal reflective access operations
WARNING: All illegal access operations will be denied in a future release
converter.specific.java_util_regexTest > convertMatcherTest PASSED
converter.specific.java_util_regexTest > convertPatternTest PASSED
converter.specific.JPF2JVMjava_utilTest > convertRandomTest PASSED
converter.specific.JPF2JVMjava_lang_reflectTest > convertConstructorTest PASSED
converter.specific.JPF2JVMjava_lang_reflectTest > convertFieldTest PASSED
converter.specific.JPF2JVMjava_lang_reflectTest > convertMethodTest PASSED
on_the_fly.StringTest > testToLowerCase PASSE
on_the_fly.StringTest > testGetBytes PASSED
on_the_fly.StringTest > testSplit PASSED
on_the_fly.StringTest > testTrim PASSED
on_the_fly.StringTest > testLastIndexOf PASSED
on_the_fly.StringTest > testToUpperCase PASSED
on_the_fly.StringTest > testEqualsIgnoreCase PASSED
on_the_fly.StringTest > testToCharArray PASSED
on_the_fly.StringTest > testMatches PASSED
on_the_fly.StringTest > testReplace PASSED
on_the_fly.StringTest > testConcat PASSED
on_the_fly.StringTest > testEquals PASSED
on_the_fly.StringTest > testIntern PASSED
on_the_fly.StringTest > testIndexOf PASSED
Test Execution: SUCCESS
Summary: 35 tests, 35 passed, 0 failed, 0 skipped
  • test results on amd based machines
on_the_fly.StringTest > testToLowerCase FAILED
    java.lang.AssertionError at StringTest.java:115
on_the_fly.StringTest > testGetBytes FAILED
    java.lang.AssertionError at StringTest.java:95
on_the_fly.StringTest > testSplit FAILED
    java.lang.AssertionError at StringTest.java:123
on_the_fly.StringTest > testTrim FAILED
    java.lang.AssertionError at StringTest.java:135
on_the_fly.StringTest > testLastIndexOf FAILED
    java.lang.AssertionError at StringTest.java:76
on_the_fly.StringTest > testToUpperCase FAILED
    java.lang.AssertionError at StringTest.java:107
on_the_fly.StringTest > testEqualsIgnoreCase FAILED
    java.lang.AssertionError at StringTest.java:44
on_the_fly.StringTest > testToCharArray FAILED
    java.lang.AssertionError at StringTest.java:52
on_the_fly.StringTest > testMatches FAILED
    java.lang.AssertionError at StringTest.java:85
on_the_fly.StringTest > testReplace FAILED
    java.lang.AssertionError at StringTest.java:149
on_the_fly.StringTest > testConcat FAILED
    java.lang.AssertionError at StringTest.java:142
on_the_fly.StringTest > testEquals FAILED
    java.lang.AssertionError at StringTest.java:34
on_the_fly.StringTest > testIntern FAILED
    java.lang.AssertionError at StringTest.java:157
on_the_fly.StringTest > testIndexOf FAILED
    java.lang.AssertionError at StringTest.java:63
converter.specific.java_util_regexTest > convertMatcherTest FAILED
    java.lang.AssertionError at java_util_regexTest.java:65
converter.specific.java_util_regexTest > convertPatternTest FAILED
    java.lang.AssertionError at java_util_regexTest.java:50
converter.specific.java_textTest > convertDecimalFormatTest FAILED
    java.lang.AssertionError at java_textTest.java:50
converter.specific.java_textTest > convertSimpleDateFormatTest FAILED
    java.lang.AssertionError at java_textTest.java:76
converter.specific.java_ioTest > convertFileTest FAILED
    java.lang.AssertionError at java_ioTest.java:49
converter.specific.java_langTest > convertClassTest FAILED
    java.lang.AssertionError at java_langTest.java:43
converter.specific.java_langTest > convertStringTest FAILED
    java.lang.AssertionError at java_langTest.java:55
converter.specific.JPF2JVMjava_utilTest > convertRandomTest FAILED
    java.lang.AssertionError at JPF2JVMjava_utilTest.java:47
converter.specific.JPF2JVMjava_lang_reflectTest > convertConstructorTest FAILED
    java.lang.AssertionError at JPF2JVMjava_lang_reflectTest.java:70
converter.specific.JPF2JVMjava_lang_reflectTest > convertFieldTest FAILED
    java.lang.AssertionError at JPF2JVMjava_lang_reflectTest.java:93
converter.specific.JPF2JVMjava_lang_reflectTest > convertMethodTest FAILED
    java.lang.AssertionError at JPF2JVMjava_lang_reflectTest.java:47
converter.JVM2JPFTest > convertClassTest FAILED
    java.lang.AssertionError at JVM2JPFTest.java:127
converter.JVM2JPFTest > convertIntegerTest FAILED
    java.lang.AssertionError at JVM2JPFTest.java:70
converter.JVM2JPFTest > convertStringTest FAILED
    java.lang.AssertionError at JVM2JPFTest.java:58
converter.JVM2JPFTest > convertHashMapTest FAILED
    java.lang.AssertionError at JVM2JPFTest.java:97
converter.JVM2JPFTest > convertArrayTest FAILED
    java.lang.AssertionError at JVM2JPFTest.java:82
converter.JPF2JVMTest > convertClassTest FAILED
    java.lang.AssertionError at JPF2JVMTest.java:107
converter.JPF2JVMTest > convertIntegerTest FAILED
    java.lang.AssertionError at JPF2JVMTest.java:66
converter.JPF2JVMTest > convertStringTest FAILED
    java.lang.AssertionError at JPF2JVMTest.java:56
converter.JPF2JVMTest > convertHashMapTest FAILED
    java.lang.AssertionError at JPF2JVMTest.java:86
converter.JPF2JVMTest > convertArrayTest FAILED
    java.lang.AssertionError at JPF2JVMTest.java:76
Test Execution: FAILURE
Summary: 35 tests, 0 passed, 35 failed, 0 skipped

35 tests completed, 35 failed
@cyrille-artho
Copy link
Member

This is strange, because I have an Intel Mac where the tests pass. Maybe it's the exact JDK version that is slightly different? I have the following:

$ java -version
openjdk version "11.0.23" 2024-04-16
OpenJDK Runtime Environment (build 11.0.23+9)
OpenJDK 64-Bit Server VM (build 11.0.23+9, mixed mode)

@Harsh4902
Copy link
Contributor Author

I also have the same configuration, but still it's failing.

openjdk version "11.0.23" 2024-04-16
OpenJDK Runtime Environment (build 11.0.23+9-post-Ubuntu-1ubuntu122.04.1)
OpenJDK 64-Bit Server VM (build 11.0.23+9-post-Ubuntu-1ubuntu122.04.1, mixed mode, sharing)

@cyrille-artho
Copy link
Member

Maybe it is an Ubuntu issue then? I have access to one Ubuntu machine that I can use (without docker), and I also get all tests failing there.
Ubuntu and Mac OS are normally quite similar when it comes to running Java code, so I'm surprised that Ubuntu causes all tests to fail here. The CI workflow actually also uses Ubuntu, and the last commit had all tests pass.

@cyrille-artho
Copy link
Member

On the upside, if you find the reason for one test failure, you probably found them all. I recommend looking into a simple test for this. It's probably just the way the main JPF project (jpf-core) is configured. I am not an expert in Gradle, but I guess the problem lies here (in build.gradle):

    // jpf-core
    implementation files(jpf.Jar.getFullPath())

@Harsh4902
Copy link
Contributor Author

Harsh4902 commented Jul 8, 2024 •
edited
Loading

@cyrille-artho Yeah I am working on it. I also found one interesting thing, test cases of package converter and converter.specific are working fine. You can test it by using following commands:
./gradlew clean test --tests converter.*
./gradlew clean test --tests converter.specific.*

@cyrille-artho
Copy link
Member

You mean they work in isolation? Perhaps something in the other tests changes the global state of the JVM (by setting a static variable). For some reason, I currently can't test JPF at all on the machine I have access to. Gradle does not run.

@Harsh4902
Copy link
Contributor Author

You mean they work in isolation? Perhaps something in the other tests changes the global state of the JVM (by setting a static variable). For some reason, I currently can't test JPF at all on the machine I have access to. Gradle does not run.

Yeah, test cases of package converter and converter.specific are running fine when we run it separately by using above-mentioned commands.

@cyrille-artho
Copy link
Member

I was able to compile nhandler on Ubuntu 20 (on an older computer), but in that case, all tests fail, regardless of whether some are run in isolation or whether all 35 unit tests are run.
However, the order in which Gradle runs tests is not necessarily deterministic, so there may be a test within converter that can interfere with other tests when run early, but perhaps not when most other tests have already concluded.
Please keep looking for possible reasons for the test failure. Perhaps jpf-core is not found in my setup, which would explain tests always failing.
A test can also "pollute" the environment of other tests by affecting the system-wide configuration through the Java library, so setting static fields (or calling methods that set them) is not even the only way to cause such potential problems.

@Harsh4902 Harsh4902 mentioned this issue Jul 10, 2024
Merged
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

jpf-nhandler's tests are failing on amd based machines Harsh4902/jpf-nhandler
2 participants
@cyrille-artho @Harsh4902