PSA: Empty Test Suite

I’m writing this post, and tagging it with “Empty Test Suite” all over the place, because Googling “Empty Test Suite” won’t tell you what I discovered.  In fact, the StackOverflow method of debugging this led me on a couple of hours of bad adventure.

The setup:

  • Android Things developer preview 0.6
  • Raspberry Pi connected hardware
  • IntelliJ
  • JUnit tests using the AndroidJunitTestRunner

Almost all my tests (200+ at this point) are run on-device as they require hardware.  After hours of fiddling with Gradle, I get my tests, 45 minutes worth, running on the device.  Even hooked it up to the CI pipeline.

Then I went on a couple of hours worth of refactoring.  Run CI again, and bang! Empty Test Suite.

Testing started at 2:07 PM ...
 02/13 14:07:17: Launching Tests in 'com.farlo.android.bubbles.control.rule'
 $ adb push C:\Users\rodley\Documents\bubbles\Bubbles\build\outputs\apk\blue\debug\Bubbles-blue-debug.apk /data/local/tmp/com.farlo.android.bubbles
 $ adb shell pm install -g -t -r "/data/local/tmp/com.farlo.android.bubbles"
 Success

$ adb push C:\Users\rodley\Documents\bubbles\Bubbles\build\outputs\apk\androidTest\blue\debug\Bubbles-blue-debug-androidTest.apk /data/local/tmp/com.farlo.android.bubbles.test
 $ adb shell pm install -g -t -r "/data/local/tmp/com.farlo.android.bubbles.test"
 Success

No apk changes detected since last installation, skipping installation of C:\Users\rodley\.gradle\caches\modules-2\files-2.1\com.android.support.test\orchestrator\1.0.1\12d61be26b643c6413d207248660bce8f6d8b236\orchestrator-1.0.1.apk
 $ adb shell am force-stop android.support.test.orchestrator
 No apk changes detected since last installation, skipping installation of C:\Users\rodley\.gradle\caches\modules-2\files-2.1\com.android.support.test.services\test-services\1.0.1\c24c3f3ccf05dfdd86dbcd6c7a648d6c4a429178\test-services-1.0.1.apk
 $ adb shell am force-stop android.support.test.services
 Running tests

$ adb shell CLASSPATH=$(pm path android.support.test.services) app_process / android.support.test.services.shellexecutor.ShellMain am instrument -r -w -e targetInstrumentation com.farlo.android.bubbles.test/android.support.test.runner.AndroidJUnitRunner -e package com.farlo.android.bubbles.control.rule -e debug false android.support.test.orchestrator/android.support.test.orchestrator.AndroidTestOrchestrator
 Client not ready yet..
 Started running tests
 Tests ran to completion.

Empty test suite.

That error message seems pretty clear.  The test runner couldn’t find any tests to run.  To me, this said that I’d broken my build configuration – either grabbing a bad dependency, picking the wrong test runner or otherwise hosing the build.gradle somehow.  Setting up connected testing was just painful enough in those particular ways that this seemed almost inevitable.

Well, the error message wasn’t clear, or at least complete.  What happened was that my Android Application was blowing up on a simple null pointer in the onCreate and crashing the process, which the test runner, bless its heart, interprets as an empty suite.  If I had, first time out, looked at the logcat:

02-13 19:03:04.404 10994-10994/? E/InstrumentationResultPrinter: Failed to mark test No Tests as finished after process crash
02-13 19:03:04.407 10994-10994/? E/MonitoringInstr: Exception encountered by: com.farlo.android.bubbles.BubblesApp@d5f45a0. Dumping thread state to outputs and pining for the fjords.
 java.lang.NullPointerException: Attempt to invoke virtual method 'void com.farlo.android.bubbles.Cabinet.setupMonitoredDevices(android.content.Context)' on a null object reference

I would have seen that I’d written an easily fixed bit of crappe code, not a painful gradle misconfiguration.  If the tests had been running locally, not connected, the crash would have reported to the console just like the erroneous “Empty Test Suite” message, and again, the issue would have been obvious.

Empty Test Suite?  Check the logcat.

About JR
Software guy, startup guy, non-fiction glutton, south shore inhabitant

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Google+ photo

You are commenting using your Google+ account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s

%d bloggers like this: