PSA: Empty Test Suite

I’m writing this post, and tagging it with “Empty Test Suite” all over the place, because Googling “Empty Test Suite” won’t tell you what I discovered.  In fact, the StackOverflow method of debugging this led me on a couple of hours of bad adventure.

The setup:

  • Android Things developer preview 0.6
  • Raspberry Pi connected hardware
  • IntelliJ
  • JUnit tests using the AndroidJunitTestRunner

Almost all my tests (200+ at this point) are run on-device as they require hardware.  After hours of fiddling with Gradle, I get my tests, 45 minutes worth, running on the device.  Even hooked it up to the CI pipeline.

Then I went on a couple of hours worth of refactoring.  Run CI again, and bang! Empty Test Suite.

Testing started at 2:07 PM ...
 02/13 14:07:17: Launching Tests in ''
 $ adb push C:\Users\rodley\Documents\bubbles\Bubbles\build\outputs\apk\blue\debug\Bubbles-blue-debug.apk /data/local/tmp/
 $ adb shell pm install -g -t -r "/data/local/tmp/"

$ adb push C:\Users\rodley\Documents\bubbles\Bubbles\build\outputs\apk\androidTest\blue\debug\Bubbles-blue-debug-androidTest.apk /data/local/tmp/
 $ adb shell pm install -g -t -r "/data/local/tmp/"

No apk changes detected since last installation, skipping installation of C:\Users\rodley\.gradle\caches\modules-2\files-2.1\\orchestrator\1.0.1\12d61be26b643c6413d207248660bce8f6d8b236\orchestrator-1.0.1.apk
 $ adb shell am force-stop
 No apk changes detected since last installation, skipping installation of C:\Users\rodley\.gradle\caches\modules-2\files-2.1\\test-services\1.0.1\c24c3f3ccf05dfdd86dbcd6c7a648d6c4a429178\test-services-1.0.1.apk
 $ adb shell am force-stop
 Running tests

$ adb shell CLASSPATH=$(pm path app_process / am instrument -r -w -e targetInstrumentation -e package -e debug false
 Client not ready yet..
 Started running tests
 Tests ran to completion.

Empty test suite.

That error message seems pretty clear.  The test runner couldn’t find any tests to run.  To me, this said that I’d broken my build configuration – either grabbing a bad dependency, picking the wrong test runner or otherwise hosing the build.gradle somehow.  Setting up connected testing was just painful enough in those particular ways that this seemed almost inevitable.

Well, the error message wasn’t clear, or at least complete.  What happened was that my Android Application was blowing up on a simple null pointer in the onCreate and crashing the process, which the test runner, bless its heart, interprets as an empty suite.  If I had, first time out, looked at the logcat:

02-13 19:03:04.404 10994-10994/? E/InstrumentationResultPrinter: Failed to mark test No Tests as finished after process crash
02-13 19:03:04.407 10994-10994/? E/MonitoringInstr: Exception encountered by: Dumping thread state to outputs and pining for the fjords.
 java.lang.NullPointerException: Attempt to invoke virtual method 'void' on a null object reference

I would have seen that I’d written an easily fixed bit of crappe code, not a painful gradle misconfiguration.  If the tests had been running locally, not connected, the crash would have reported to the console just like the erroneous “Empty Test Suite” message, and again, the issue would have been obvious.

Empty Test Suite?  Check the logcat.