Rugg/Feldman benchmarks

The Rugg/Feldman benchmarks are a series of seven short BASIC programming language programs that are used to test the performance of BASIC implementations on various microcomputers. They were published by Tom Rugg and Phil Feldman in the June 1977 issue of the US computer magazine, Kilobaud .

The article reported that Integer BASIC, an interpreter program written by Steve Wozniak for the Apple II computer, was much faster than the other programs tested. This sparked widespread comments about the tests, including a lengthy letter from Bill Gates. A follow-up article in the October 1977 issue addressed these concerns in depth and added many new machines and BASICs to the set of results.

John Coll added an eighth test using transcendental functions in an article in the February 1978 issue of the British magazine Personal Computer World (PCW). This expanded set became known as the PCW Benchmarks, and was particularly popular as a test for UK-designed machines like the Grundy NewBrain and BBC Micro.

The benchmark was widely used through the late 1970s, and appears as a standard in many computer magazines and journals. In the 1980s it was not as widely used in the US as the Creative Computing Benchmark or Byte Sieve, but remained in common use in the UK.

History

First tests

The benchmark suite was introduced to test claims that were being made by vendors that their BASIC was much faster than others running on the same machines. The authors were unable to find a set of standardized benchmarks and decided to write their own. ^[1] The tests deliberately ignored string and floating point performance, as many BASICs of the era, especially those descended from Tiny BASIC, lacked these features. Strings were further hindered by major differences in syntax between different versions. ^{[2] [lower-alpha 1]}

The two ran the benchmarks on every machine they could find, typically at friends' houses. All of the major 8-bit CPUs were tested, including the Intel 8080, Zilog Z80, Motorola 6800 and MOS 6502. For comparison, they also ran it on a CDC Cyber 174 supercomputer, which is so fast that they had to add code to time the run using the system's clock as they could not operate a stopwatch quickly enough. They used this machine for two reasons; one was simply to see how fast such a machine ran, and the other was to ensure no one micro would end up at the top of the list and then claim they had been proven to be the fastest. ^[3]

The article makes a special note of the BASIC that shipped with IMSAI computers, as it basically didn't work. The machine was very new at the time and the BASIC was described by the company as "preliminary". During this period, different IMSAI machines were delivered with different versions of BASIC. Each version had different features, but none of them were able to run even the majority of the programs. ^[4]

The Apple II, using the original Integer BASIC, finished well in front of all the other machines. As the tests did not make use of any floating point features, this result was not surprising given the much simpler internal representation of numbers. ^{[lower-alpha 2]} The next fastest, albeit significantly slower, was Zapple BASIC on a Zilog Z80 add-in card in an Altair 8800. The rest of the list contained a number of very closely spaced entries, dominated by what would later be known as Microsoft BASIC. Across the entire suite of 8080 and Z80 machines and versions of BASICs, the spread was only 20%. In contrast, the 6800-based machines were isolated entirely at the bottom of the list, 30 to 40% slower than the 8080 and Z80 entries. ^[4]

As the article went to press, the editors of Kilobaud visited several local computer stores to test the programs on newer machines. This process added the production version of IMSAI BASIC, North Star BASIC and the 11 kB BASIC on the Poly-88. IMSAI's results were at the bottom of the pack, while North Star and Poly were in the middle. ^[5]

Follow-up

Rugg and Feldman revisited the suite in the October 1977 issue. The new article opened by noting they had been deluged with mail about the original article. Among them was a letter from Bill Gates, who they introduce as the author of "Altair BASIC (8080 and 6800 versions), OSI BASIC and PET BASIC". The last entry refers to a member of the "1977 trinity" of machines, the Commodore PET, which was made available in prototype form for inclusion in the article. ^[6]

Gates complained that the original test series "let an integer BASIC be compared against... more powerful BASIC... using floating point." He suggested that the test include a DEFINT A-Z at the start, which would make newer versions of Altair BASIC use integer math as well. Instead, Rugg and Feldman took another approach and eliminated all integer-only BASICs from the new test results. ^[6] This did not eliminate the Apple II, which by this time had introduced the MS-derived Applesoft BASIC on cassette. ^[7]

Gates also noted that the results for the 6800 machine were not indicative of this processor. The machine they used, the Altair 680, runs the CPU at half its rated speed. He suggested a more typical 6800-based machine would be slightly faster than the 8080. However, the 6800 once again put in a poor showing even on newer machines running at higher speeds. ^[6]

Gates also suggested that the Cyber 173's time was likely due to it being a compiler rather than an interpreter. The authors point out this was not the case for the TRW BASIC they used in the original test, and then use this as a segue to compare the differences between compilers and interpreters. ^[6]

Gates concluded his letter by noting that a number of the results were identical in spite of being on different machines. He suggests this is because the BASICs in question contained "signatures" from Altair BASIC and are thus "illegitimate software". He did not specify which ones he claimed were stolen, and the authors responded by saying that if "Bill can stop people from selling them through legal means, we'll stop listing them." ^[8]

Rugg and Feldman conclude the article by mentioning some of the other concerns that were raised after the original article. One common issue was the lack of more advanced mathematical functions, which they acknowledge, but suggest this is something best left to the reader. The other was the lack of string manipulation, but they note that the syntax of string handling differed considerably between platforms and thus could not be made in a single version. ^{[8] [lower-alpha 3]}

In this series of tests, the list was topped by the OSI Challenger, a 6502-based machine that had been "souped up" to 2 MHz, double that of typical 1 MHz 6502 machines of the era like the Apple II and PET. When running at its normal 1 MHz speed, the Challenger was just beaten by Zapple BASIC on Z80 machines running at 4 MHz. PET BASIC was next, only slightly behind the Challenger. ^[9] They conclude that the 6502 is the highest performing of the CPUs, agreeing with comments Gates had made in his letter. The 6800 once again ends up in last place. ^[10]

PCW version

As part of a longer article discussing new entries into the computer market, including the TRS-80, John Coll used the Rugg/Feldman tests to benchmark a variety of machines available to him in the UK in October 1977. He added an eighth test to exercise the math routines, and provided the resulting run times both on their own as well as the additional time compared to Test 7, in keeping with the earlier concept of each test modifying the last. ^[11] The results were published in the first issue of Personal Computer World in February 1978, ^[12] with a short follow-up in their November 1978 issue. ^[13]

Use

As one of the earliest sets of BASIC benchmarks, the F/R tests were seen primarily in the late 1970s and early 1980s. It was a standard among reviews in Kilobaud, used to compare the many new varieties of BASIC that continued to appear for early microcomputers. ^[14] Compute! used it for their 1979 review of the Challenger 1P, ^[15] and 68 Journal used it to demonstrate the extremely high performance of BASIC09. ^[16] InfoWorld used it for their 1981 review of a new BASIC for the TRS-80, ^[17] and the TRS-80 Color Computer as a whole. ^[18]

After that point the Byte Sieve began to become popular and the number of articles referring to the F-R benchmark become less common, but it could be found even such rarefied sources as the HP Journal. ^[19]

The PCW versions remained very popular in the UK, and can be found in many reviews of UK-centric machines like the Grundy NewBrain, various Sinclair Research machines, ^[20] and the BBC Micro. ^[21]

Programs

The programs were designed to allow a user to type in the first test, run it, and then modify it in-place to run the subsequent tests. This meant the user did not have to type in seven different programs, but simply modify a single one. ^[3] The first seven listings are the originals from the 1977 article, ^[1] the eighth is the PCW addition. ^[22]

Benchmark 1

300 PRINT"S" 400 FOR K=1 TO 1000 500 NEXT K 700 PRINT"E" 800 END

FOR-NEXT loops are one of the most fundamental constructs in the BASIC language, and if the performance of these loops is slow it is highly likely that any program running in that BASIC will be slow as well. A famous example of this is Atari BASIC, which had several problems that greatly slowed the performance of FOR-NEXT loops compared to contemporary examples, and BASIC programs on the Atari were generally very slow as a result. ^[23]

Benchmark 2

300 PRINT"S" 400 K=0 500 K=K+1 600 IF K<1000 THEN 500 700 PRINT"E" 800 END

This test ultimately performs the same operations as test 1, but in this case, it uses an explicit test and jump rather than using the built-in FOR-NEXT construct. Generally, this program runs much slower than 1 because most BASICs parse the parameters in the FOR, including the line number, only once when it is first encountered. Using the IF, as in test 1, causes it to parse the values every time through the loop. Moreover, most BASICs do not simply store the parsed line number of the top of the loop, but the memory address, whereas a THEN requires the interpreter to scan through the program for the corresponding line number, in this case 500. Although this test does not show it due to its small size, this search takes increasingly long times as the program length grows. Some versions of BASIC optimized GOTO using explicit labels, ^[24] or pushing the GOTO targets on a stack to make them perform like NEXT. ^[25] Turbo-Basic XL did both, and ran much faster than any similar BASIC as a result. ^[24]

Benchmark 3

300 PRINT"S" 400 K=0 500 K=K+1 510 LET A=K/K*K+K-K 600 IF K<1000 THEN 500 700 PRINT"E" 800 END

Test 3 is an expansion of test 2, this time adding some basic mathematics and variable access. By comparing the time to run tests 2 and 3, one can get an idea of the performance of the language's math library. ^[1]

Benchmark 4

300 PRINT"S" 400 K=0 500 K=K+1 510 LET A=K/2*3+4-5 600 IF K<1000 THEN 500 700 PRINT"E" 800 END

The same as test 3 except that the variable K is replaced by numeric constants. This requires the interpreter to convert the values from their textual representation into their internal storage format, which takes time (unless this is performed once and for all, before execution, as in some more advanced BASICs). Some idea of the performance of this conversion functionality can be determined by comparing this time to benchmark 3. ^[1]

Benchmark 5

300 PRINT"S" 400 K=0 500 K=K+1 510 LET A=K/2*3+4-5 520 GOSUB 820 600 IF K<1000 THEN 500 700 PRINT"E" 800 END 820 RETURN

Test 5 introduces a subroutine call. Long programs in early versions of BASIC would make extensive use of subroutines and thus the efficiency of the calling mechanism was important. Depending on the way the system worked, the return might cache the location of the calling line in a fashion similar to NEXT, and thus run very quickly. Others might store the line number of the calling line, and thus require the code to scan the program listings to RETURN. ^[1]

Benchmark 6

300 PRINT"S" 400 K=0 430 DIM M(5) 500 K=K+1 510 LET A=K/2*3+4-5 520 GOSUB 820 530 FOR L=1 TO 5 540 NEXT L 600 IF K<1000 THEN 500 700 PRINT"E" 800 END 820 RETURN

Test 6 defines a small array at the start and adds another FOR-NEXT loop inside the main loop. This has little effect on the code, but is used to set a baseline for test 7. ^[1]

Benchmark 7

300 PRINT"S" 400 K=0 430 DIM M(5) 500 K=K+1 510 LET A=K/2*3+4-5 520 GOSUB 820 530 FOR L=1 TO 5 535 M(L)=A 540 NEXT L 600 IF K<1000 THEN 500 700 PRINT"E" 800 END 820 RETURN

This assigns a value to each of the array elements every time through the loop. Comparing the time needed to run 7 to 6 indicates the efficiency of array access. ^[1]

Benchmark 8

300 PRINT"S" 400 K=0 500 K=K+1 530 A=K^2 540 B=LOG(K) 550 C=SIN(K) 600 IF K<100 THEN 500 700 PRINT"E" 800 END

Test 8 was added by PCW, ^[11] performing several transcendental functions in order to test their performance. This code does not include the code from test 7, which breaks the original pattern of each test adding to the last. Instead, the associated article listed both the time to run 8 and 7 and 8 combined. ^[12]

Sample results

This list is not meant to be exhaustive, but instead a quick overview of the results seen on some popular systems of the early microcomputer and home computer era. The first table includes a selection of machines from the original tests in June 1977, with exceptions as noted. ^[1] Test time is in seconds.

System	CPU	BASIC	Test 1	Test 2	Test 3	Test 4	Test 5	Test 6	Test 7	Test 8	Notes
Apple II	6502 @ 1 MHz	Integer BASIC	1.3	3.1	7.2	7.2	8.8	18.5	28.0		PCW's numbers are about 0.1s longer
Apple II	6502 @ 1 MHz	Applesoft BASIC	1.3	8.5	16.0	17.8	19.1	28.6	44.8	55.5	from PCW, for comparison with above
Altair 8800	8080 @ 2 MHz	Altair BASIC 4.0	1.7	10.2	21.0	22.5	24.3	36.7	52.4
Altair 8800	8080 @ 2 MHz	Altair BASIC 4.0	1.9	7.5	20.6	20.9	22.1	38.8	57.0	67.8	from PCW, including FP
Altair 8800/TDL ZPU	Z80 @ 4 MHz	Zapple	1.7	9.5	20.6	21.7	23.7	36.7	52.4		Z80 add-on board, Zapple BASIC shipped with it
Compucolor 8001	8080 @ 2 MHz	Compucolor 8k BASIC	2.1	13.1	27.0	29.0	31.3	47.5	67.8
SWTPC 6800	6800 @ 898 kHz	SWT 8k BASIC	1.4	9.0	16.8	18.1	20.0	31.0	45.2	53.7	from PCW
SWTPC 6800	6800 @ 898 kHz	Altair 680 BASIC	14.9	24.7	96.1	105.3	109.8	174.1	204.5
Altair 680	6800 @ 500 kHz	Altair 680 BASIC 1.1	2.6	16.4	30.9	33.7	36.6	56.0	81.9	96.9	from PCW

System	CPU	BASIC	Test 1	Test 2	Test 3	Test 4	Test 5	Test 6	Test 7	Test 8	Notes
IBM 5100			4.5	21.1	57.4	54.5	59.0	88.2	174.9	201.6
BBC Micro	6502 @ 2 MHz	BBC BASIC	0.8	3.1	8.1	8.7	9.0	13.9	21.1	49.9	fastest of the English/American 8-bit BASICs
Acorn Electron	6502A @ 2 MHz	BBC BASIC	1.11	4.01	11.12	11.76	12.40	18.72	28.71	72.53	from PCW [26]
Atari 800XL	6502 @ 1.79 MHz	Atari BASIC	2.2	7.3	19.7	24.1	26.3	40.3	60.1	na	no explanation for lack of test 8
Commodore 64	6510 @ 1 MHz	MS BASIC	1.2	9.3	17.6	19.5	21.0	29.5	47.5	119.3
ZX80	Z80 @ 3.25 MHz	Sinclair BASIC	1.5	4.7	9.2	9.0	12.7	25.9	39.2	NA	integer only
ZX Spectrum	Z80 @ 3.58 MHz	Sinclair BASIC	4.4	8.2	20.0	19.2	23.1	53.4	77.6	239.1
Acorn Atom	6502 @ 1 MHz	Atom BASIC	0.5	5.1	9.5	10.8	13.9	19.1	31.1	92	from DatorHobby 1/84 [27]
Luxor ABC 80	Z80 @ 3 MHz	ABC BASIC	1.1	2.3	11.1	12.1	12.6	17.7	23.9	136	from DatorHobby 1/84
ABC 80 (using integer variables instead of floating point)	Z80 @ 3 MHz	ABC BASIC	0.3	1.1	3.5	3.5	3.6	5.8	9.3	65	from Mikrodatorn 1982
ABC 800 (using single precision floating point)	Z80 @ 3 MHz	ABC BASIC II	0.9	1.8	6.0	5.9	6.3	11.6	19.6	29	from Mikrodatorn 1982
ABC 800 (using double precision floating point)	Z80 @ 3 MHz	ABC BASIC II	1.2	2.2	10.0	10.6	11.0	17.8	26.4	144	from Mikrodatorn 1982
VIC 20	6502 @ 1 MHz	MS BASIC	1.4	8.3	15.5	17.1	18.3	27.2	42.7	99	from DatorHobby 1/84
Dragon 32/64	6809E @ 0.89 MHz	MS Extended BASIC	1.6	10.2	19.7	21.6	23.3	34.3	50.0	129	from DatorHobby 1/84
Spectravideo SV-328	Z80 @ 3.6 MHz	MS Extended BASIC	1.6	5.4	17.9	19.6	20.6	30.7	42.2	236	from DatorHobby 1/84
TI-99/4A	9900 @ 3 MHz	TI BASIC	2.9	8.8	22.8	24.5	26.1	61.6	84.4	382	from DatorHobby 1/84
Oric Telestrat	6502 @ 1 MHz	Oric BASIC	0.5	3.6	11.4	13.2	13.4	18.7	26.3	109	Cartridge BASIC, measured by community member
Oric Atmos	6502 @ 1 MHz	Oric BASIC 1.1	1.6	15.2	25.4	27.4	33.0	45.6	68.5	136	measured by community member
Oric 1	6502 @ 1 MHz	Oric BASIC 1.0	1.8	17.1	29.0	31.4	38.0	51.8	77.8	230	measured by community member
Tangerine Microtan 65	6502 @ 0.75 MHz	MS Extended BASIC	1.9	12.8	24.7	27.8	29.6	43.2	68.9	243	with TANEX expansion? Measured by community member.
VTech CreatiVision	6502 @ 2 MHz	CreatiVision BASIC (1983)	20	60	79	119	180	298	398	120	executes in VRAM [28]
VTech Laser 2001	6502 @ 2 MHz	VTech Laser 2001 BASIC	1.7	6.7	11.6	12.9	15.3	26.0	39.7	64.5	same hardware as CreatiVision but executes in CPU RAM

Notes

1. At the time there were two major dialects of string handling used in BASIC. The first had been introduced by Hewlett-Packard as part of their HP 2000 line and used array slicing for manipulation. The second had been introduced by Digital Equipment (DEC) and used string functions instead. Microsoft BASIC, based on the DEC dialect, became the de facto standard and by the late 1970s the earlier syntax had largely disappeared with the exception of Atari BASIC.
2. Integer BASIC used 16-bit binary numbers, while most BASICs used 32- or 40-bit floating point formats, which requires longer to process. See Microsoft Binary Format for details.
3. The issue with string syntax stems from the difference between BASICs that were inspired by DEC's vs. HP's dialects. Microsoft BASIC was based on DEC's BASIC-PLUS variety and uses the MID/LEFT/RIGHT functions to manipulate strings, while HP used an "array-of-chars" system more similar to FORTRAN 77 and C that is found in Atari BASIC, North Star BASIC, and others. The two are different enough that porting directly from one to the other is not trivial, and such portability was a goal of the benchmark suite.

References

Citations

1. RuggFeldmanJun1977, p. 66.
2. RuggFeldmanJun1977, p. 67.
3. RuggFeldmanJun1977, p. 68.
4. RuggFeldmanJun1977, p. 69.
5. RuggFeldmanJun1977, p. 70.
6. RuggFeldmanOct1977, p. 22.
7. Herzfeld, Andy (12 February 2014). "MacBasic – The Sad Story of MacBasic". Folklore.org.
8. RuggFeldmanOct1977, p. 24.
9. RuggFeldmanOct1977, p. 23.
10. RuggFeldmanOct1977, p. 25.
11. PCWFeb1978, p. 57.
12. PCWFeb1978, p. 58.
13. PCWNov1978.
14. Didday, Rich (January 1978). "A tale of Four BASICs; which one for your 6800?" (PDF). Kilobaud. pp. 50–59.
15. Russell, W. Keith (Fall 1979). "The Challenger 1P". Compute!. pp. 100–104.
16. Harmon, Tom (September 1980). "OS/9-BASIC09". 68 Micro Journal. pp. 26–27.
17. Puckett, Dale (31 August 1981). "Extended Color BASIC for TRS-80 from Microsoft". InfoWorld. pp. 26–28.
18. Puckett, Dale (16 August 1982). "Radio Shack Color Computer". InfoWorld. pp. 57–61.
19. Bidwell, John; Palermo, David (May 1982). "Advanced Multilingual Computer Systems for Measurement Automation and Computer-Aided Engineering Applications" (PDF). Hewlett-Packard Journal. 33 (5): 3–7.
20. Personal Computer World, 1983 and 1984 Benchtest Summaries
21. Ochocki, Peter. "Personal Computer World Benchmark Tests".
22. PCWNov1978, p. 58.
23. Wilkinson, Bill (1982). Inside Atari DOS. Optimized Systems Software. p. 191. ISBN   0-942386-02-7 . Retrieved 2009-04-04.
24. "Turbo-Basic XL" (PDF). Happy-Computer. December 1985. pp. 34–47.
25. BASIC XL (PDF). Optimized Systems Software. 1983. p. 35.
26. Benchtest:Acorn Electron. October 1983.
27. Dator Hobby Nr1 1984.
28. CreativEmu Forum thread.

Bibliography

• Rugg, Tom; Feldman, Phil (June 1977). "BASIC Timing Comparisons... information for speed freaks". Kilobaud. pp. 66–70.
• Rugg, Tom; Feldman, Phil (October 1977). "BASIC timing comparisons... revised and updated". Kilobaud. pp. 20–25.
• Coll, John (February 1978). "Direct Addressing: Where to get your personal computer". Personal Computer World. pp. 55–58.
• Coll, John (November 1978). "Benchmarks Again". Personal Computer World. p. 52.

Further reading

• Computer Handbook: Businessman's Guide to Choosing and Using Computer Systems, lists the PCW versions as part of their benchmark sets.