[callahan]

Edit -- Some of this was wrong. See updated benchmarks in my reply below.

TL;DR: After testing, I've found that SCSI2SD v6's max performance is slower than a high-end 10,000RPM SCSI drive, but it still often outperforms such drives in normal usage on an SGI Indy.

Based on discussions here and elsewhere, I decided to do some benchmarks using my own hardware. My test system:
SGI Indy, 180Mhz R5K, 256MB RAM
Two identically imaged drives with Irix 6.5.22 and a smattering of Nekoware and demos (each drive was tested when it was the only drive on the bus, primarily because I can't get my SCSI2SD to work with another drive.)
-- SCSI2SD v6 w/ Transcend high performance SD card. More on my SCSI2SD settings below.
-- Fujitsu MAJ3182MP UW-160 10,000RPM SCSI disk (68 pin interface w/ a 68-50 pin adapter).

First, I benchmarked both drives using diskperf 1.2. I found that the 10 second test interval produced wildly varying and often nonsensical results (near zero throughput or throughput nearly twice the theoretical maximum speed of the Indy's Fast SCSI bus), so in the end I used 60 second test intervals. Note this is 60 seconds per test (i.e. 60 seconds for 4MB forward write, 60 seconds for 4MB forward read, etc.), so a complete test suite takes ~75 minutes. I ran each test twice on each drive. The two test runs gave very similar results (minus a couple of outliers on the SCSI2SD data), with 1-sigma values generally less than .2MB/S.

Below I've attached graphs with the following diskperf test results:

• Sequential read/write test results (average of two runs, using the forward test results only; backward tests showed the same trends), and
  
• Random read/write test results (average of two runs)
  

These results showed that at best, the SCSI2SD performed roughly on par with the Fujitsu control, especially with read/writes less than 64kb. With larger files, it still sometimes performed on par with the Fujitsu but, at worst (large file write performance), had less than half the measured throughput.

Next, I performed a suite of three tests representing real-world use cases. These results found, in contrast to the above, that performance with the SCSI2SD was better than the Fujitsu in every day applications. I repeated the first two test series twice, again values from the tests matched nicely. I only performed the tar tests once per drive.

Test 1: System boot-up (time from the first text displayed on the mini-root to the visual login window)

• SCSI2SD: 44 seconds
  
• Fujitsu: 51 seconds
  

Test 2: Firefox loading (from open command to complete rendering of www.google.com)

• SCSI2SD: 37 seconds
  
• Fujitsu: 52 seconds
  

Note: Tests 1 and 2 are obviously read-heavy (avoiding the SCSI2SD's greatest flaw according to diskperf), but I still did not expect it to do so well based on the neck and neck read performance from the diskperf tests.

Test 3A/B: tar and untar a large set of files of varying, but mostly small, size (I used /usr/nekoware on the tested system, test tar was ~489MB. I did not/not use the verbose option)
3A: Archive

• SCSI2SD: 6 minutes, 10 seconds
  
• Fujitsu: 3 minutes, 54 seconds
  

3B: Extract

• SCSI2SD: 11 minutes, 24 seconds
  
• Fujitsu: 4 minutes, 43 seconds
  

Note: These results seem to confirm the diskperf results, showing a large performance difference when writing large files.

After seeing this data, I will continue to use my SCSI2SD as my "everyday" Indy drive. It performs better than a spinning disk in the tasks I perform the most often, uses much less power, and is quieter. However, if your workload includes many operations with large sets of files, a conventional disk may be considerably faster (but then, why on earth are you using a system/device with only a Fast SCSI bus?).

More on my SCSI2SD config:
Firmware 6.2.1, excerpts from my config. xml (My full SCSI2SD config xml file is available on my github)

Code:

<parity>true</parity>
<enableScsi2>true</enableScsi2>
<selLatch>true</selLatch>
<startupDelay>6</startupDelay>
<scsiSpeed>0</scsiSpeed>

Note that I did not/not try a newer firmware version primarily because it was a giant pain to get the SCSI2SD working with my Indy originally, so I do not want to risk messing anything up.

[KayBee]

Hey that’s a great report. Very interesting reading. I will follow your lead with my O2 and share it here when I do.

KB

[jpstewart]

Wow, that looks like a thorough test.  I'm very impressed and have a lot of respect for your work.

[callahan]

UPDATE:

It turns out the last post's methodology wasn't great. So, updates!

New TL;DR: With top-tier SD cards the SCSI2SD v6 can perform better than conventional 10k and 15k RPM disks on a Fast SCSI (10MB/S) bus, except for sequential write. The quality of your SD card is very important to overall performance. Also, firmware 6.1.3 performs better (on tested SGI systems) than 6.3.0.


This update is driven by two developments. First, after getting some nonsense data from my Octane over the last week, I realized that it is critical to use the -D flag with SGI's diskperf utility. Using this flag removed variability with 10 second runs on the Indy and Indigo2 and kept my Octane from giving nonsensical results (e.g. random read speeds of >250 MB/S). Second, I bought a new top-tier SD card to compare SCSI2SD performance. The results surprised me!

Methodology notes:

I used two high-performance SD cards with identical drive images, SCSI2SD settings, and firmware.

1. Transcend S64GSDC500S 64GB (amazon link), currently ~$25. This is a good card, U3/V30 rated, listed transfer rates of up to 95 MB/s Read; 60 MB/s write. Tests with this card are sometimes shorthanded as $testsystem_T
   
2. Sandisk Extreme Pro SDSDXXY-128G-GN4IN 128GB (amazon link), current ~$38. This is a top-tier card, also U3/V30 rated but claims "Shot speeds up to 90MB/s, transfer speeds up to 170MB/s". Tests with this card are sometimes shorthanded as $testsystem_San
   

Note: Both of these are very good SD cards. If you use a more normal/bargain SD card, I would expect worse--possibly very bad--performance using the SCSI2SD.

All diskperf tests used the following command line (and relied on previously created 1GB test files): diskperf -W -D -n [description] -r4k -t 10 [$testpath]. All tests were performed with a single user logged in via ssh. The systems tested were:

1. SGI Indy, 180Mhz R5000 processor, 256MB RAM, IRIX 6.5.22. On this device, each test was performed with only the test drive connected (i.e. the test drive was the system and test disk).
   
2. SGI Indigo2, 175MHz R10000 processor, 640MB RAM, IRIX 6.5.22. On this device, each test was performed on the secondary SCSI controller and the test device was the only device on the bus.
   
3. SGI Octane, 2x360MHz R12000 processors, 1.5GB RAM, IRIX 6.5.30. On this device 50/68 pin drives were connected externally to the secondary SCSI controller. 80 pin drives were connected internally. The Fujitsu MAX drive was the booted/system drive for all tests (including the test of the MAX drive itself).
   

The drives tested were:

1. SCSI2SD v6, Firmware 6.1.3, Transced 64GB SD card
   
2. SCSI2SD v6, Firmware 6.1.3 and Firmware 6.3.0, Sandisk Extreme Pro 128GB SD card
   
3. Fujitsu MAJ3182MP 18.2GB 10k RPM UW-160 disk
   
4. Fujitsu MAX-series 36GB 15k RPM U320 disk
   
5. HP 3008B26C 300GB 15k RPM U320 disk 
   

I've attached graphs to this post comparing sequential and random read/write tests for all drives. My key takeaways:

1. The SCSI2SD/Sandisk duo was fast at random reads/writes (should be no surprise). It outperformed even the 15k RPM U320 drives in the Octane at random read performance up until 32kb blocks (where the conventional drives were able to surpass the SCSI2SD's Fast SCSI transfer limit). On the Indy/Indigo2, it outperformed all drives at all block sizes.
   
2. The only clear loss for the SCSI2SD against other Fast SCSI devices was in sequential write performance, where it topped out around 4 MB/S while the 10k and 15k RPM disks were able to sustain transfers at the Fast SCSI bus limit. 
   
3. The older 6.1.3 firmware outperformed the newest 6.3.0 release handily. I tested both with and without "turbo mode" enabled. I found turbo slightly increased most speeds, but also added variability that caused a drops at some block sizes. I've reverted my device to 6.1.3, no-turbo. My SCSI2SD config is still available on my github.
   
4. Using a SCSI2SD on a Indy/Indigo2 appears to be largely a no-brainier -- minus cost. At ~$140 for SCSI2SD v6+SD card, my current SCSI2SD setup is over twice as expensive as all of the other disks I tested combined (each was less than $20 on ebay).
   
5. On anything with a Ultra/Wide SCSI bus, the SCSI2SD probably won't make sense except for a very specific use cases (e.g. CD-ROM emulation).
   

               

Not a key takeway, but I was surprised that the SCSI2SD performed measurably better on the Octane, despite being limited to Fast SCSI speeds, which both the Indy and Indigo2 support. Any ideas why?

I also repeated the real-world benchmarks with the new Sandisk card and added Octane results for comparison. My key takeaway remains that in real-world tests, the performance boost with a SCSI2SD was nearly enough to make a top-spec Indy feel like a high-spec Indigo2. Emphasis on the "feel" and "nearly". Obviously it doesn't make up for the difference in raw CPU or graphics muscle. And in some workloads, the sequential write performance will still make a SCSI2SD slower than an Indy with a good conventional disk. I've attached graphs with times for various tasks. A few further methodology notes:

1. The Indy and Indgo2's IRIX installs are configured very similarly, but they are not identical. The Octane has a newer IRIX version and more software installed.
   
2. As before, boot time was measured from the first text on miniroot window to the visual login screen.
   
3. As before, Firefox (current nekoware version) was measured from the command to a fully rendered Google.com.
   

   

That's it for now!

---

Reached the attachment limit, here are the promised firmware/SD card comparisons.

       

[Raion]

Interesting data. I think I'll stick with spinners for the Octane, but I want to get a SCSI2SD, I just wish that the code/plans were still FOSS so there was competition to drive prices down.

[KayBee]

Major Eaton : We have top men working on it right now.
Jones : Who?
Maj. Eaton : Calla... Han.

Fun to read, and for selfish reasons (Octane inclusion) this is excellent.

Cheers to you,

KB

[KayBee]

Callahan, I have an external CDROM for the Octane, so as you stated in another post in theory I could use that to connect the SCSI2SD as you did I believe. My question is, the external CDROM enclosure I have is heavy-duty with big internal power supply and fan. Does the external 68 pin SCSI supply power to the SCSI2SD or must I have the enclosure plugged in?

I think it's a reasonable thought because on the O2, the 50 pin ribbon cable from the internal CDROM supplied power to the SCSI2SD.

Thanks again.

KB

[nintendoeats]

I would like to mention, as described in another thread, that using a quality SD card is key. It took my ~6 hours to install IRIX 6.5 + NFS on a cheap class 10 card with a 250MHz I2, and the system was painfully slow to use.