It's good to see that LibreOffice project has more and more tests improving the functional quality of the software. In this post, I also write about a new test suit added to the project. At Collabora, we are working on a project about a new chart functionality and as a preparation we decided to add a test suit which covers the chart layouting code better than the tests LO already had. With this we can map the actual state of the chart code to test cases and so make sure it's functionality remains intact.
Different kind of automatic tests in LO
There are different forms of automatic tests in the project. The question was which one is the most effective for testing a bigger part of the code (chart layout) not only a small functionality. One form of testing is when we use CppUnit assertions to compare some properties of the actual test case with the expected values. These tests used to test a very specific test case. For example it can check whether one test document has 2 pages when it is imported into LibreOffice. This kind of tests can lead to code duplication when we test the same thing (e.g. page number) on different test documents. That's why this kind of CppUnit test is not effective when we need to test one bigger part of the software sistematically, which may need to test the same thing (e.g. page number) on different test documents, which documents might be important use cases.
Other tests use an XML dump functionality to dump the test document's layout to an XML file and use this file as a reference with which the test document can be compared later. Adding new test cases to this kind of test suits is easier, needs lesser code change compared to first form of tests. However this kind of tests checks the whole layout of the document in general. So when one of these tests fails, you need to check the XML file, understand it's structure and find out why the pointed difference is there. It's clear that this kind of test failures might be more difficult to understand compared to the first form of tests where the test failures point to a very specific document property not to a reference XML file.
New chart test suit
An important difference between these two forms are the generality level of what it tests. First form tests some specific properties of one specific test case, while second form test the whole layout of the test document. To avoid the issues of these two forms I chose to write a test suit designed to test the software functionality on a middle level of generality.
I added a test suit which contains similar tests as the first form, comparing some specific document properties to extected values using CppUnit assertions, but the expected values are not hard coded, but they are written into and read from a simple structured reference file. It's implemented on a way which makes easy to add a new test document for an existing test case and generate the reference file without extending the test code. On the other hand we get a helpful error message when one test failes, since the test case is more specific than an XML dump test.
Testing chart functionality
After I had the form I spent some time on adding the most common use cases to the test suit. I added test cases for different components of charts (axis, chartwall, legend, grid, data series, etc.) and also for common chart types (columnchart, barchart, piechart, etc.). All test cases contains more subcases, testing functionally distinctable use cases. The sistematic testing of the chart functionality also pointed out some issues of the software:
Now with the new chart test suit bigger part of the chart functionality is covered with tests, but there are still use cases which are not tested. For example some exotic chart types have no tests yet, like bublechart, netchart or 3D charts. Tested document formats are also limited to LibreOffice native formats (ODS, ODP), but these tests are easy to extend to Microsoft Office file formats too, for compatibility testing. You can find new test suit at chart2/qa/extras/chart2dump/ in case you need to add new test cases.