Abbot
(with support from Costello)
Applying what I learned with JUnit to writing GUI tests, and building a better
'bot.
Contents
The Pain (Problems and Goals)
On the subject of why this testing is so important, I won't repeat here what
has been ably described in
Test
Infected. If you haven't read that document, please do so before
proceeding here.
Often there's a mismatch between interface specifications and the actual
implementations. When you have a test case that explicitly tests for a given
behavior, you've codified the interface specification. And if you've
documented the test, it's a lot easier to verify whether a specification has
been met than either inspecting the code or running the app by hand.
Side effects. Often a developer will want to make some optimizations to a
piece of code on which a lot of user-interface behavior depends. Sure, the
new table sorter will be ten times faster and more memory efficient, but are
you sure the changes won't affect the report generator? Run the tests and
find out. A developer is much more likely to run a test that has encapsulated
how to set up and test a report than he is to launching the application by
hand and trying to track down who knows how to do that particular test.
GUI Testing
Getting developers to test is not easy, especially if the testing itself
requires additional learning. Developers will not want to learn the details
of specific application procedures when it has no bearing on their immediate
work, nor should they have to. If I'm working on a pie chart graph, I don't
really want to know the details of connecting to the database and making
queries simply to get an environment set up for testing.
Describing expected behavior before writing the code can clarify the
developer's goals and avoid overbuilding useless feature sets.
Scripts vs compiled code
How can I test a GUI prior to writing code? One alternative (and useful in
certain cases) is developing a mockup in a RAD tool. Unfortunately, the
usefulness is relatively short-lived; it's not really possible (at this point
in time) to automatically generate a very interesting interface. If your
entire interface consists of buttons and forms, you may not really need a gui
tester anyway. Mockups don't convert well to tests for the actual developed
code, and RAD tool output usually requires some hand modification
afterwards.
I could write code, a la JFCUnit, but this typically is more easily done after
the GUI code has been written. It also requires a programmer to write the
test.
What if the script could plainly describe the GUI components of interest, and
simply describe the actions to take on those components? Providing you know
the basic building blocks, you can edit the scripts by hand or if you don't
know the building blocks, in a script editor. No compilation necessary.
I wanted scripts to be hand-editable, with no separate compilation step. I
wanted to be able to drop scripts into directory hierarchies as needed, and
have the test framework pick them up automatically, similar to how JUnit
auto-generates test cases.
How to Test
One issue in defining a GUI test is that you need to map from a semantic
action (select the second item in the records table) onto a programmatic
action (myTable.setSelectedIndex(1)
). This comprises two separate
problems. First, the target of the action, "records table", must be somehow
translated into an actual code object. Second, the semantic event "select the
second item" must be translated into a programmatic action on that code
object.
Tracking components
There are many methods of identifying a component in a GUI hierarchy. We want
the tests to be flexible enough to not break simply because another button or
panel was added to the GUI layout.
- Component Names Java provides for naming components, but since no
one ever names them, this method of identification is mostly useless. Worse,
some auto-generated components (frames for otherwise frameless windows,
windows for popup menus, and most dialog instantiations) have auto-generated
names, which aren't particularly helpful.
- Position in hierarchy This method guarantess a unique match, but
also guarantees your script will break when that hierarchy changes, even for
otherwise trivial modifications (like inserting a scrollpane). Each component
would need to store its parent reference and index within that parent. Note
that this implies each parent reference might need to store the same
information for itself.
- Parent window title This is useful as a first-pass discriminator in
multi-window applications.
- Component class This helps discriminate from the available
components list, but is not likely sufficient on its own to identify a
component.
- Custom tags Here's where the real component resolution is done.
Many component classes will have some aspect or property that can be used to
uniquely identify them, typically a label (the text of a button or menu, a
labelFor component, a window's title), but potentially any identifying
element may be used. Abbot uses custom component testers to get the unique
tag for any given component. These testers are dynamically loaded, so we
don't have to know a priori how to get a tag for a given component class.
Recording events
Ideally, there would be a programmatic function on every component that
performed a given user action, similar to the doClick
method on
AbstractButton
. Given that this is not the case, we want to
provide for both implementing such functions and loading them dynamically, and
constructing semantic events from low-level events when corresponding
functions are not available. Having dedicated semantic functions makes
writing scripts easier, but supporting the low-level events means they are not
absolutely required.
The most basic events to support are those that correspond to low-level OS
events for user input, namely
- Pointer motion
- Button press/release
- Key press/release
In addition, the most common semantic events that affect user input are
windows opening and closing, so we throw in support for those (Alternatively,
the time delta between events could be preserved for replay, but doing so has
not yet proved useful in very many cases)
- Wait for window open/closed
The easiest way to write a script is to record a series of actions for later
replay, adding checkpoints or assertions along the way to verify correctness.
Eventually, you might want to add heuristics that strip out meaningless
events, but keeping track of a few basic events is sufficient for good
functionality.
Over and above event stream support, it's useful to have custom, class-based
component functions that provide higher-level semantic events. For example, a
custom table might export a select(int row, int col)
action to
select a cell within the table, or sortByColumn(int col)
to
invoke a click on the column header which would cause a sort in the table.
The custom actions use existing building blocks (low-level events or existing
semantic actions) to construct the higher-level semantic event.