Android User Interface Testing on Amazon’s Device Farm Part 2

In part 1 of this post I looked at how to create Robotium instrumentation tests for an Android application so that it could be tested on Amazon’s Device Farm. At the end of that post, we had two apk files: the application and the tests. The process of testing on the AWS device farm is simple and effective.


You will need an AWS account, you can sign up and get a free usage tier here. Once you account is configured, head on over to the Device  Farm and testing can commence.


Create a Project

Tests are organized into projects. Projects can contain tests to be run on iOS, Android and Fire OS, they just provide a mechanism to allow you to organize your work.

Define a Test Run

At the heart of a Device Farm test is a Test Run. Here, you upload the code under test, the test code and select which devices to test against.

Step 1: Select the application to test

It’s wizard driven, so as soon as you click the button to define a Test Run, you are prompted to choose an apk file from your local drive.

Define the Android application to test

Once uploaded, Device Farm performs a brief analysis of the manifest of the Android APK and reports it’s findings.

Manifest Analysis Complete
Manifest analysis complete

Step 2: Configure the test type

Device Farm presents a list of the test types it supports. At the time of writing, several of these were still in the preview phase of development. For all of the tests except Built-In: Fuzz you have to supply test code. Built-In: Fuzz simulates random user interactions. In our case, we selected the Instrumentation test type and uploaded the Robotium code we generated previously.

Select test type and upload the code
Select test type and upload the code

Step 3: Select Devices

You now need to specify which devices to test against. All users are offered the opportunity to use the built-in Top Devices pool which consists of 5 devices. To use other devices (and there are many), you will need to define your own device pools. You need Amazon to approve your use of these devices: simply a matter of filling in a form and waiting a few days for a response.

Device Selection

Step 4: Specify Device State

The last edit screen in the wizard allows the specification of the state of the device. There are two main areas to configure: the physical state of the device (network connection, location, enabled hardware) and the addition of pre-installed apps and data.


Define device state

These settings will be applied as part of the initialization of the devices on which our app is to be tested.

Step 5: Reality Check

As with other AWS wizards, we get to review our chosen settings before the tests are run.


Step 6: Run and Wait for the Results

It takes several minutes for the tests to run, dependent on the complexity of the tests and the availability of the devices in the device pool. Eventually, you will get either a green bar indicating your tests have all passed.

Wait for the results
Wait for the results

Or, some tests will fail

Tests Failed!
Tests Failed!


Exploring the Results

Simply clicking on the results lines shown in the screen-shots above takes you to the details of the test:

Results for each device




Detailed results for each device are then available

Details of each test run

The tabs at the top allow you to access recorded screenshots:

Screenshots from the test

Performance records

Performance records

And, perhaps most important of all the output files from the tests along with the captured LogCat output as the test proceeded.

Log files from the test

Learning Tree whose generosity allows me to write these posts have a couple of great courses that might be of interest to you: course 4711 provides a great overview of the new features in Amazon’s Web Service suite including Device Farm. For more on Android Development, take a look at course 2771 Android Application Development & Programming

Type to search

Do you mean "" ?

Sorry, no results were found for your query.

Please check your spelling and try your search again.