Posted by kusno mudiarto at July 23rd, 2006

If you ever read my background, you’ll know that I’m currently working on developing web browser for mobile phone in my company. Well, developing Web Browser is not an easy task; A regular web browser can easily have thousands of requirements. Making sure that whenever we implement something, it doesn’t break other part is becoming more and more difficult. I realized that the unit testing that we’ve done so far is not adequate. It is not automatic enough, and user have to recompile the code to enable or disable individual unit test. It is also very hard to test complex interaction between module, let’s say between css, java script, DOM, etc.

Another problem is that our browser is targeted for mobile device, and supposed to be very portable across different compiler. It also has to have small footprint, so I can’t put all unit test scenario that I want to implement in the phone. That’s important too, since I need to make sure that porting the web browser to different platform/compiler doesn’t break the functionality.

So generally the requirements that I need for our new unit testing are:

  • Running test with one click
  • Easy to add new test, easy to verify unit test
  • No need to recompile the code - need to separate the code and the testing data itself
  • C++ (our software is written in C++)

I was looking for unit testing method that has those requirements, but couldn’t find any that satisfy my needs. Most of them fail at requirements #3 - separate the code and the testing data. Some of the closest one are DDTUnit and JTestCase. But unfortunately they are written in java & for java. I think it is time that we start developing our own unit test framework.

The Design.
First is the part for every module / function to be tested. I was thinking that for every subsystem needs to be tested, we can write some simple test runner / recorder. Its responsibility are:

  • receiving input with (binary)XML like format, and then construct a function call based on the XML data, and,
  • recording the output / result of the function, and then send it to the unit test subsystem; Unit test subsystem will transmit it to the unit test server outside from the device; but that’s a different story.

This test runner / recorder is very simple, we should be able to make a simple test runner / recorder with about 10 - 15 lines of code.

The second part is the unit test subsystem itself, which will receive XML data from the test server, execute the test runner, wait for output and then post the output back to the server. So once again its responsibility are:

  • Receiving the XML test instruction from test server
  • Calling the correct test runner (find it based on name) to execute the data, and then wait for the output to arrive
  • Post the data to the server, and then wait for the next test from the server.
  • All traffic to and from server is sent as HTTP data

So far so good; the test runner & unit test subsystem now becoming part of the code on the device(of course we should be able to turn it off with a flick of a compiler switch); but they are pretty small, because they don’t contain any test code. Now the last part is the test server itself. This one should be pretty straightforward also, its responsibility are:

  • Sending xml data to client
  • Receiving xml data from client
  • Save the data to disk
  • Compare the data with whatever reference data we have on disk, and if it matches, then the test pass, if not, the test fail. I think we can use XMLUnit to do some diff, it looks like it will be very helpful.

With this way of working, now we can separate code & testing data, and it will be very easy to develop the unit test server. We can write it in any kind of language, and we are not limited to any hardware / software limitation of mobile device. The other good thing is that now we can test the device from anywhere on the world with internet connection, no need to stand by right next to the test server and do the test.

Unit Test Server

The unit test server jobs are pretty simple; but we don’t have much time to develop it, since it is not our core software. I have to do a bit of skunk work to do it ;-) .. not sure what’s my boss reaction if I put it on the schedule :) . The choice for language for unit test server got me thinking for about 1 sec.. and then I decided that Ruby will be a very good choice. And boy how glad I was to make that decision. I wrote a simple unit test server using Webrick, and it is a joy to write. I got all the support & tools that I need, the code is simple .. maybe only about 200 - 300 lines (hey, I’m still a beginner, I think some ruby expert can do it in about 50 lines :-) ) and it is pretty functional although not complete yet. It has a web interface to handle & report the test result, so I can run only specific test, or collection of tests, and doesn’t have to do all the test everytime.

Currently it is only taking XML data; but I think in the future, it is as easy to put ruby code as the test data, and also ruby code as the validator, so I can do more flexible test later.
Future planned enhancements:

  • Using ruby script as test data generator & test validator
  • Using html as input, and the layout data as output and compare it with previous result; or maybe even compare it with other 3rd party browser
  • Automatic generation of test report; saving test reports; repeat failed test
  • Multiple user for the test server; currently only one client per test server.
  • etc.

Even though I just started with ruby, I already falling in love with it. Hope my wife doesn’t read it ;-) . I think this unit test framework can be useful to test other kind of software / plateform; Since the data itself is platform independent (just a regular XML data); and it also separates data & the code, so it is easy to add additional test code whenever we found any problem in the future. And last but not least, the data can be saved in the version control along with the code, so we can always test the code with the right data that comes along with it.

Unfortunately the code for this unit test framework is not available; it is part of the code that belong to our workplace; but I hope I can convince my boss to make it open source; just to give something back to the community ;-)