XSLBench is a performance benchmark of XSLT processors. The benchmark measures transformation times for a small set of XML files and rates the engines based on their deviation from the average transform time for all engines. The tests are designed to stress XSLT use in web page construction, document rendering and document statistics computation.
The table below shows an average of the test scores of the processors. Each test score is calculated as a percentage of the time taken to run a specified transform against the average of all the processors. This figure allows you to compare the speed differential between the processors. An engine having the value 100% is a completely average performer. An engine with the value 50% is twice as quick than the average and so on.
Processor |
Average % Score |
22.23 |
|
50.73 |
|
53.07 |
|
66.39 |
|
125.16 |
|
129.31 |
|
130.40 |
|
132.58 |
|
190.12 |
WARNING(S):
An attempt was made to benchmark the following processors but failed for various reasons. I will look at these again the next time around.
The test environment and procedure can be found here here.
The table and graph show the performance of each processor on each of the tests. The times are measured in milli-seconds and give the average of all the runs for that processor/test. A description of the tests can be found here. The tests were run on a PIII 550Mhz with 196Mb running Windows 2000.
Processor |
Test1 (ms) |
Test2 (ms) |
Test3 (ms) |
Napa 0.3 |
1.6 |
55 |
41 |
MS XSLT (Sept '00) |
4.5 |
118 |
79 |
Sun XSLTC (Beta 4) |
5.8 |
111 |
66 |
XT |
6.2 |
129 |
116 |
Saxon 5.5 |
10.6 |
247 |
242 |
Sabletron 0.44 |
16.5 |
266 |
107 |
Xalan J 1.2D02 |
9.2 |
247 |
305 |
Oracle J V2 |
10.7 |
388 |
169 |
Xalan C 0.4 |
3.6 |
540 |
540 |
As many people kindly pointed out, playing with your Java environment can make a big difference to a XSL Processors performance. For these tests I have standized on the use of JDK 1.3. Using other JVM can improve the performance of some processors/test combination.
The memory setting that always had the quickest overall score for all tests on all processors was -Xms64M (initial heap=64Mb) -Xmx128M (maximum heap=128Mb). The critical factor was setting the initial pool size at all. Once set larger values did improve the performance but only by a small amount. The maximum size was only important in that mx>=ms.
There is some variation in the results of processors based on which XML Parser the processor is using. In these tests I used the following combinations.
I also tested Saxon against JAXP1.0 (ProjectX) but this turned out to be slightly slower than Ælfred. If anybody knows of better combinations than these please let me know and I will try and benchmark that pair.
The tests used on the benchmark have been designed to mimic common XSLT usage. They have been kept deliberately simple to allow the largest range of XSLT processors to be compared independently of their conformance or otherwise to the standard.
There are currently three tests in the benchmark.
This test is typical of many web pages constructed with XSLT. The style sheet contains almost a complete HTML page. During the transformation some table-structured data is extracted from the XML and inserted into the output page. The test is dominated by the output speed of the processor when reading from the style sheet.
The test is typical of transforming large input texts but keeping the same document order. The style sheet uses templates to match on elements in the input document and ‘wrap’ the text with HTML tags. The XML file used is a Shakespeare play. The test is dominated by the cost of template resolution and output speed.
This test is typical of data analysis style sheets. The same XML file is used as in test 2 but this time the style sheet calculates the speeches and average lines per a speech for each scene in the play. The test is dominated by the speed of XPath expression evaluation.
The benchmark is executed using a test driver for each processor. The driver executes a given transformation 200 times with the average transformation time of the last 100 runs reported as the result. If the engine supports any form a speed up (i.e. compiled style sheets) that could be reasonably be used in a production environment the driver can exploit it. The driver is required to supply output either via stdout or a file. There are no restrictions on the form of the output, i.e. character encoding, whitespace handling, etc. In fact the validity of the results is not checked during the runs. For each test the driver is executed three times and the average of the runs reported.
You can download a copy of the benchmark sample files and drivers from here.
This benchmark was out of necessity fairly simple. In the future I would like to add some more tests and cover a bigger range of processors. I think the tests should remain fairly targeted at highlighting specific performance issues. If you have any thoughts on tests or even better some sample files I would love to hear from you. Equally driver files or improvments to the existing drivers are most welcome.
Kevin Jones
kjouk@yahoo.co.uk