Saturday, June 27, 2015

Benchmarking C compilers for Commodore Amiga

There are a lot of great programming tools for classic Amiga computers, some of them absolutely unique for this platform, like AMOS or Amiga E. AMOS is a very powerful Basic dialect with a wide set of graphics and sound commands that allow you to create entertainment software, especially games, in no time. Amiga E is an advanced, multi-paradigm programming language, featuring flexible types, powerful module system and very short compilation times.

There is also AREXX, which is Amiga port of REXX programming language popular on IBM mainframes. Every decent Amiga software offers so called "AREXX port", which is a set of commands that can be called from an AREXX script. This is a very powerful tool, which allows you to write a script that, for example, calls a graphics software to make thumbnails for a set of images, than an archiver to put them into a compressed file, and than another program to upload that file to an ftp server. Modern software very rarely allows you to control it with external scripts, and even if it does, you must use some clunky tools like AutoIt, which simulate keystrokes and mouse clicks. Sometimes a program can be called from command line, but than you must learn how to parse its output if you want to know more than just whether the operation failed or not. In AmigaOS, thanks to AREXX, automation is a heck of a lot easier, because there is a single communication standard with clearly defined interfaces for every software written for this platform.

Of course there are also traditional programming languages avaiable, like ubiquitous ANSI C. Doing some research on Amiga programming tools I managed to make a couple of them work. Three of them run natively on classic Amiga: Aztec C 5.0a, Dice C 3.16, and SAS C 6.58. Another two are cross compilers, which run under Linux on PC, but produce native Amiga executables: vbcc 0.9b and m68k-amigaos-gcc, which is a port of gcc 2.95 generating code for Motorola MC68xxx CPU family. There is also a vbcc build for Windows available on Github.
I decided to make a simple benchmark to compare the compilation times, executable sizes and their execution times. Of course you cannot make definite judgement just by one benchmark, but I needed at least a basic reference to start with. At one time a popular benchmark was a sieve of Eratosthenes, so I decided to implement one. I also used AmigaOS timer device to measure execution time of the generated executables. If you are interested in the source code you can get it from here.

Except from cross compiling, I made all tests on Amiga 600 with 2MB Chip RAM, 1MB FastRAM and Workbench 2.0. As a rule of thumb, you should always make tests on real hardware - emulators are not 100% cycle perfect, and they always run on top of a multitasking operating system, which introduces additional interference. I also ran each test 10 times, rejected the highest and the lowest result, and took an average from the remaining ones. I didn't measure compilation time for cross compilers, as they run on PC, and not on Amiga.
All in all, the results are as follows:

CompilerCompilation timeExecutable sizeExecution time
Aztec C 5.0a37 seconds12032 bytes3114 miliseconds
gcc 2.95-12072 bytes809 miliseconds
Dice C 3.1631 seconds7092 bytes1332 miliseconds
SAS C 6.5855 seconds14612 bytes1597 miliseconds
vbcc 0.9b-12764 bytes1306 miliseconds

As you can see, for this particular test, Dice C is a clear winner among the native compilers: not only was it the quickest, but also produced the smallest and the fastest executable code. Execution time of the code compiled by Dice C was almost on par with the code produced by vbcc, which ran on incomparably more powerful machine and could perform a lot more optimisations. Not surprisingly, the fastest executable was produced by gcc.

No comments: