Automated Android* Application Testing Testing is an important part of the application development process. For Android, it is particularly important because the devices are very different from each other in the following ways:*.Screen size and resolution*.Android version*.Form factor*.Instruction set of the processor*.The presence of the front camera, NFC, external keyboard, etc.Your Android application should be tested on many devices.Application testing processesinclude many types of testing.Lets take a look at manual functional testing. The tester needs to carefully check all the functionality and reset thedevice to an initial state. The tester repeats these steps for each application and each device. Done manually, this process is very time-consuming.Automated functional testing can be performed regularly without additional costs. For example, test a build every night on all devices, analyze the results in the morning, and fix the bugs.In this article I will review several tools for automated functional testing. Ill review only tools included in AndroidSDK or distributed under Open Source license.The concept of automatedtestingOur goal is to automate actions that are performed manually for maximum precision. Lets review these actions. Well use several applications and several Android devices.For each application and eachdevice we need to perform the following steps:1.Install the application on the device2.Launch the application3.Test the application using the selected method4.Remove the application5.Reset the device to the initial stageAt each step you need to collect and analyze data such as logs and screenshots. Below, we’ll discuss the tools that automate these steps.Control Android devicesFirst, you need to select the computer that will run the automated test and install theAndroid SDK on it. Ill use a desktop computer with Linux*as an example. You need to disable lock screen and increase time before sleep mode to the maximum level on each device.For some testing methods, you need to disable screen orientation change.There are two utilities in the Android SDK to control Android devices: ADB and monkeyrunner*. Ill describe in detail how to automate actions that are used in manual testing.Control the Android device using ADBADB (Android Debug Bridge) is a command line tool to control Android devices. The ADB home page is:developer.android/tools/help/adb.htmlThe ADB tool is located in the directory/platform-tools/.You should put this directory in thePATHenvironment variable.Checking the ADB installationInstall and configure the Android SDK, then connect Android devices to your computer and run the command:adb devicesThis command will list all plugged-in Android devices. Ifthe list of devices is not empty, then ADB is working properly.Working with multiple devicesYou should use “-s” parameter to specify which device ADB should use.adb -s [serial_number] [command]For example:adb -s [serial_number] logcatThe serial number of the device can be obtained from the output of «adb devices» command. Parameter-sallows you to work simultaneously with multiple connected devices.Basic ADB commandsOpen a console on the device:adb shellRun a command on the device:adb shell [command]Many standard Linux utilities are included in Android: ls, cat, dmesg, ...Install the application from the apk file:adb install example.apkUninstall the application:adb uninstall [package]Get a package name from the apk file:aapt dump badging example.apk | grep PackageDownload a file from the device to the computer:adb pull [path-on-device] [file]Upload a file from the computer to the device:adb push [file] [path-on-device]Note:Only read access is allowed to most directories on the Android device. Write access is allowed to/sdcard(but youcan’t run programs from this directory) and/data/local/tmp.Start an application:adb shell am start -n [package]/[activity]Run the specified activity.You can extract the activity name from the apk file:aapt dump badging example.apk | greplaunchable-activityReading logsLogcat is a command that reads logs from Android devices.Logcat homepage:developer.android/tools/help/logcat.htmlRead logs from the device (blocks until you press Ctrl-C):adb logcatClear log buffer on the device:adb logcat -cDump log buffer on the device (shows current buffer contents, non-blocking):adb logcat -dExample:1adb logcat -c # clear thebuffer log2# Action3adb logcat -d > file.log # save the current contents of the log buffer to file.logCapture screenshot using screencapThescreencaputility saves the current contents of the screen to a graphic file:1adb shell screencap /sdcard/screen.png2adb pull /sdcard/screen.png screen.png3adb shell rm/sdcard /screen.pngThescreencaputility is available on phones with Android 4.x and above. On previous Android versions you can capture screenshots using monkeyrunner.BASH script to test the application using ADBScript:app_test.shControl the Android device with MonkeyRunnerThe monkeyrunner tool provides APIs for scripts that control Android devices. You can write a Python* script with monkeyrunner that installs an Android application, launches it, emulates user actions, takes screenshots, and saves them to your computer. Monkeyrunner uses Jython* to run scripts.The monkeyrunner home page and API reference:developer.android/tools/help/monkeyrunner_concepts.htmlReading logs with monkeyrunnerFile log.py:01# coding: utf-802fromcom.android.monkeyrunner importMonkeyRunner, MonkeyDevice03 04deflog(fn, device):05 msg =device.shell(logcat -d)06 f_log =open(fn, at)07 ifmsg isNone:08 msg =None09 f_log.write(msg.encode(utf-8))10 f_log.close() 11 device.shell(logcat -c)12 13if__name__ ==__main__:14 device =MonkeyRunner.waitForConnection()15 device.shell(logcat -c) # Clear logs buffer16 # ...17 log(example.log, device) # Write logsStarting:monkeyrunner log.pyThe script will write logs to a file example.log in the currentdirectory.Capture screenshot using MonkeyRunnerFile screen.py:1# coding: utf-82fromcom.android.monkeyrunner importMonkeyRunner, MonkeyDevice3 4if__name__ ==__main__:5 device =MonkeyRunner.waitForConnection()6 image =device.takeSnapshot()7 image.writeToFile(screenshot.png,png)Starting:monkeyrunner screen.pyThe script takes a screenshot and saves it to a filescreenshot.png in the current directory.Example of device control using monkeyrunnerScript:monkeyrunner_test.pyStarting:monkeyrunner monkeyrunner_test.pyAutomated Testing MethodsTesting with Monkey*Imagine that the device you are testing is given to a very active and creative monkey—the Monkey tool is designedto simulate this situation.The Monkey tool, a part of the Android SDK, sends a stream of random user events.Command line parameters specify the number of user actions, the ratio of each event type, and aname of a package (so Monkey will not go beyond the limits of the tested application and will not start sending SMS to all the contacts in the Address Book).Examples and a list of parameters are on the Monkey home page:developer.android/tools/help/monkey.htmlThe main advantage of the Monkey tool is zero maintenance costs. Furthermore, stress testing can detect non-trivial bugs.Disadvantages of testing withthe Monkey tool:*.Monkey can’t simulate complex workloads such as authentication. In such cases, application functionalityremains untested.*.Games with complex control that require quick reactions and complex gestures will be completed at the beginning, or do not begin.*.It’s very difficult to reproduce errors found by Monkey.*.Monkey doesn’t check application status during the test.Automated testing with Monkey is a good starting point for any application. It is possible that this method will show adequate results for a particular application.If the testing quality is low, you should use other testing methods.Testing with MonkeyRunnerYou can not only develop Android device control scriptsusing MonkeyRunner, you can also write scripts to test the application on a specific device.Advantages:*.Flexible.Disadvantages:*.The complexity of writing scripts, even in simple cases.Developing monkeyrunner scripts takes a lot of time, so this method is usually not justified.However, in special cases, this method might work.Testing with getevent and sendeventGetevent and sendevent utilities allow the user to record a sequence of events and replay the sequence.Rootpermissions are not required to run these tools.Advantages:*.Event sequences can berecorded without additional costs in manual testing, if it is carried out.*.Programming skills are not required to record an event sequence.Disadvantages:*.Sequence should be recorded separately for each application and each device.If you change the interface of an application, all recorded actions shouldbe redone.*.This method doesn’t check application statusduring the test. If application response is delayed (for example, web page loading), the testing results will be incorrect.*.Rapid and complex sequences will be played longer than recorded. So it is not always suitable for testing dynamic games where response time is critical.Record the sequence of events:01# Record event sequence02# Do actions on the device, press Ctrl-C to finish03adb shell getevent -t > events.txt04# Convert event sequence to script05./decode_events.py events.txt > events.sh06# Load the script to the device07adb push events.sh /data/local/tmp/08# Set the permissions09adb shell chmod755 /data/local/tmp/events.sh10# Run script11adb shell sh /data/local/tmp/events.shScript:decode_events.pyReplay the recorded sequence of events on the device.Testing with Robotium*Robotium is not a part of Android SDK, and it’s distributed under the Open Source license. The Robotiumhome page is:code.google/p/robotium/.Robotium scripts define actions on the application UI level rather than the input device level.For example, a script needs totap on an «OK» button. The monkeyrunner script will be implemented as “Tap at the screen point (x0, y0)”. A Robotium script will be implemented as “Press the button with the text “OK” .”When actions are described at the interface level, the testing script can be made independent of interface layout, screen resolution, and orientation.Additionally, Robotium allowsyou to check the application response to the action.For example, after clicking the «OK» button, the list with the “Item 1” item should appear.You can check list element names with Robotium. If you check the application state after each step, it is easy to find at whichstep the error occurred.Disadvantages:*.You need to develop a test script in Java* for each application. This requires programming skills and time.*.When the application interface changes, the event sequence must beredone.*.It’s harder to write Robotium scripts compared to recording events using getevent / sendevent.In general, Robotium allows you to develop the highest quality test cases with adequate cost.Testing methods comparisonTesting methodAdvantagesDisadvantagesMonkey -a streamof random user actionsNo maintenance costs.Independent of device.Stress testing can detect non-trivialerrors.Quality oftesting varies from application to application.Bug reports are hard to reproduce.Monkey doesn’t check thestate of the application during testing.monkeyrunner – device control scriptFlexibilityThe complexity of scripting, even for simple applications.getevent/sendevent - record/replay user actionsProgramming skills are not required to record event sequence.Recordedsequence of actions will only fit one device at a fixed screen orientation.When theapplication interface changes, the eventsequence must be redone.This method doesn’t check state of application duringtesting.Robotium - test script API to verify the statusActions are described at the applicationUI level.Script may be independent of the screen resolution and screenorientation.Script can check applicationstate after an action.The complexity of writing scripts inJava.If you change the application interface,you will need to modify the script.Result analysisNow we need to analyze the logs and screenshots collected during the automated testing process for errors.Log analysisYou can do a search for strings:*.I/DEBUG*.FATAL EXCEPTION*.WIN DEATHThis list can be extended witherror messages that were found during manual testing.Screenshot analysisYou can prepare a series of screenshots at key moments of testing and compare them to the screen content during automated testing. This will determine if the automated testing process is running correctly.Its useful to compare the initial screenshot with the screenshot after the application launches. This helps to detect cases when the application silently fails.Monkeyrunner allows you to compare two screenshots with the specified tolerance inpercents:1image1 =device.takeSnapshot()2# ...3image2 =device.takeSnapshot()4ifimage2.sameAs(image1,0.1):5 printimage1 and image2 are the same (10%)Unfortunately, there is no MonkeyImage API to load an image from a file. You can write a custom function to compare images using, for example, thePython* ImagingLibrary.Reset the device to the initial stateDevices should be returned tothe initial state after testing.This can be achieved in several ways:*.Press the «Back» buttonmultiple times.*.Reboot the device.*.Restart the zygote process.The first option is usually the most appropriate.Press the Back button multiple timesPress the Back button using monkeyrunner:1fori inxrange(0, 10):2 device.press(KEYCODE_BACK, MonkeyDevice.DOWN_AND_UP)3 time.sleep(0.5)In practice, this is a good option because it emulates the behavior of a real user.ConclusionIn this article, we covered some methods of automated testing for Android applications. We reviewed advantages and disadvantages of automated testing methods.We also discussed the Monkey and monkeyrunner tools included in the Android SDK and the Robotium tool.Automated testing does not replace other types of testing.A properly constructed testing process that combines different testing methods including automated testing is an essential part of a high qualityapplication development process.
Posted on: Mon, 20 Oct 2014 12:45:59 +0000
Recently Viewed Topics
© 2015