[jan-jaap]

How easy is it to setup bridging? Is it a very simple option that you pass through the /proc interface or using something like ifconfig or route?

Setting up Linux as a router is easy:

Code:

sysctl -w net.ipv4.ip_forward=1

But your favorite distribution probably has a file where you can add this, e.g. /etc/sysctl.conf in Debian

Let's assume your LAN uses IP range 192.168.1.x and the FDDI systems use 192.168.2.x. The FDDI router box has one IP in the LAN range (let's say 192.168.1.10) and one in the FDDI range (192.168.2.1). There's a LAN-to-internet router of course (192.168.1.1). This router offers a default route (to internet) and a static route for the 192.168.2.x range via 192.168.1.10 (the FDDI router). That way all systems on my LAN can transparently reach the FDDI range. I could configure the FDDI route in every system in my LAN and save the traffic to/from my LAN router but decided against it (flexibility and easy of use vs. traffic).

The tricky part is SGI systems with both an ethernet and an FDDI interface. You end up with two routes to these systems: one via 10Mbit ethernet and one via 100Mbit FDDI. In the past I had my SGI systems in my LAN, so the fastest way from a system on LAN (let's say 192.168.1.30) to Onyx (let's say 192.168.1.20 & 192.168.2.20) would be via the 192.168.2.x network. No amount of DNS trickery, 'routed' magic and hop weights could convince anything to leave the 192.168.1.x network and go via 192.168.2.x instead.

I ended up disabling the 10Mb/s ethernet interface on these SGI systems (wired but disabled in IRIX), and these days my SGIs sit on their own IP range. The IP ranges by themselves are not the solution: the fastest way from e.g. my Origin 200 (Gbit ethernet) to my Crimson (10Mb/s ethernet, 100MB/s FDDI) is via FDDI, even though the ethernet interfaces are all in one IP range and FDDI in another. If someone has a pointer to the recommend way to solve this I'd like to hear it. There's got to be a more intelligent solution than e.g. an IP range for every possible wire speed.

Have you setup a ring topology?
My understanding was that if you have a single host in your ring topology that goes down, packets could be dropped. Is there a built in bypass to the PCI card to prevent that? Maybe I should just read books on the subject... I have them somewhere from my network study courses.

I keep a copy of the old Practical Guide to FDDI on my server, which has most of the info you need to build an FDDI network.

You are right about the dual ring topology. The downside of a ring is that all systems are expected to be up and running -- loose more than one node and your ring is gone. Systems on the ring are connected with 'DAS' cards (dual attach, A & B ports). But FDDI also knows the concept of point-to-point connections ('SAS', single attach, M ports). An FDDI concentrator normally sits on the ring, so it's uplink is DAS, and it offers a number of M ports to connect stations. Effectively, a concentrator allows you to build a local tree topology which is attached to the ring. The stations attached to the M ports don't need to be powered up all the time, the concentrator takes care of that. It behaves a lot like an ethernet switch.

So, my FDDI topology looks like this:

Code:

LAN <-> FDDI router
         /    \
         |ring|
         \    /
      Concentrator
      |     |    |
     SGI   SGI  SGI

Technically I have a ring, but it's the most minimal ring possible with only two stations on it. This setup doesn't need bypass switches.

FDDI networking is well supported in IRIX, all the way down to the installation miniroot (but not the PROM). If I boot a miniroot on an old 4D series over 10Mb/s ethernet, it will detect FDDI interface and run the actual network installation using that.

NB: It just occurred to me that I might also have set up the FDDI router as a bridge, with FDDI and SGI ethernet in the same IP range. Setting up Linux as a bridge isn't too complicated, *but*
(1) The SGI ethernet interfaces are directly attached to the LAN router, the FDDI interfaces via the FDDI router. You'd have to set up a non-standard netmask on the LAN router to send part of the range via the FDDI router (bridge).
(2) FDDI uses a larger MTU than ethernet, so packets going FDDI -> ethernet need to be fragmented. Not sure how that works with a transparent bridge.

Ultimately, I don't care so much about whether multiple IP ranges are in use, I care that traffic automatically chooses the fastest route between two systems. I also have some vague plans to move the LAN to 802.1X Port-Based Network Authentication for improved security, at which time things need to be redesigned anyway.

[ghost180sx]

Wow, that is the most fantastic response ever. That's a lot of detail and I don't think I'm going to be grabbing any FDDI equipment anytime soon, but if I do, I know where to get my setup info!

[jan-jaap]

Finished the 19" rack wiring last weekend.

The light equipment (switches) is in the top of the rack, the heavy equipment (disk arrays etc) in the bottom.



Visible in the picture, top to bottom:

* Cisco WS-C1400 FDDI concentrator. It has two interface cards installed: one with 8 MIC ports, two of which are the A/B uplink to my 'ring', and one card with 10 SC ports.

* IBM branded Brocade 5000 SAN switch, 32 * 4Gb, fully licensed. The reason I went with this is is because (a) FC is backwards compatible with two older generations, so the 4Gb switch is the last that will talk to 1 Gb equipment (e.g. my Onyx1), while 4Gb is the fastest you can find in any SGI, and (b) the noise level is 'acceptable' rather than the 2Gb switches which sound like a jet engine about to take off.

* Two 24 port Netgear managed ethernet switches. When I bought these, the 24 port models were the 'biggest' with passive coolig only. No noise in normal operation.

* Cyclades ACS 48 serial port server.

Most of this equipment has ports at the front. This allows me to quickly hook up a system for an experiment if necessary. The exception is the FDDI concentrator. That's why I made a spare cable to the front. You may notice the SC-SC fiber coupler labeled 'FDDI slot1/port8' under the right power cable of the SAN switch. By lucky coincidence it "just fits" there without major surgery to the cable management panel.

There's an APC AP7953 rack PDU mounted in the rack, between this and the console server I have full remote access and management capabilities to everything.

[Irinikus]

Jan-Jaap, you have an awesome setup! (I really appreciate your attention to detail!!!!)

You should share more pics of your complete setup, as you definitely have the best setup here!

[jan-jaap]

Another update. I finished the floor tiles under the patch strips, it needed a much bigger hole to allow all wires to pass into the gutter under the floor. This is the final setup:



There are "two umbilical cords"going from the patch strips to the 19" rack: one with ~ 70 network, serial cables and one with ~ 40 fiber optic cables, FC and FDDI. It allows the 19" rack to be pulled around 2.5m from the wall for maintenance.

With the rack pushed back, it looks like this:



Finally, with the rack in place, and the 4D/380 VGX by it's side. The 19" rack is a 42U HP 10000G2 series, the PowerSeries isn't quite that tall, but still around 1.60m. I need to clean up the stuff by the side and on top of the 4D/380. Can't run it like this

[Irinikus]

That looks incredibly good!!!

[jwhat]

Hi Jan-Jaap,

stunningly tidy, but also complex wiring setup.

How do you keep a record of the connectivity (physical connections, VLAN and sub-nets) ?

I run quite a few VLAN and associated IP sub-nets and often have trouble remembering the details of what is where and so try to keep track via a spreadsheet and some Visio network diagrams.

Your setup puts my wiring to shame.

Cheers from Oz,

John.

[jan-jaap]

stunningly tidy, but also complex wiring setup.

How do you keep a record of the connectivity (physical connections, VLAN and sub-nets) ?

I run quite a few VLAN and associated IP sub-nets and often have trouble remembering the details of what is where and so try to keep track via a spreadsheet and some Visio network diagrams.

In my previous computer room, a lot of the wiring was ad-hoc. Also, several systems were in the garage, there almost everything was plugged in as needed. Still, the amount of wiring was substantial.

When I had my current computer room constructed, I wanted things to 'just work'. If you keep patching cables into whatever you're working with today, you will always be wondering what's plugged and what's not. You will break things. You will trip over cables. Some systems will never see any use because they are not plugged in but something else is. You will be wondering why you've got a 32 port FC switch between 20 systems of which 4 or 5 are wired.

So, I decided to wire 'everything'. Every system is always ready to run. I can even turn them on and off remotely. That means a lot of wire (there's ~ 700m CAT cable and 300m of fibre in this room). The moment make that decision you basically give up flexibility. A bundle of a couple of dozen network cables is tidy but not flexible. Unbundle them and you will end up with a humongous knot. If you want to be able to re-purpose cables (flexibility) you also need to keep extra length everywhere. At this scale, that literally means a hundred little rolls of cable left and right.

Also, I don't need that much flexibility. I have enough SGI systems. I pretty much have a specimen of every series made. The last time I bought an SGI was years ago. I'm not someone who re-arranges the furniture 'just because'. OK, so between flexibility and tidiness I chose the latter. Once you give up flexibility, you can start thinking about how to hide that kilometer of cable in the room.

But I'm not completely crazy. I put in a decent amount of currently unused extra cable. Another concession to maintenance is the patch strips behind the 19" rack. Initially I planned to simply run custom made patch cables from the switches in the 19" rack to the various systems, but the idea that the rack would be 'forever' tied to the room bothered me, it would have made installation even more complicated, and it was basically impossible to find FDDI cables in the right lengths and with the right connectors on either ends. Hence the patch strips behind the rack were born.

There are four patch strips (A ... D) , and they define the numbering of all cables. A1 .. A20 or so are FC. B1 .. B14 are FDDI. The C range is RJ45 fanning out direction my desk, and the D range the same, but direction the opposite wall where my desksides and most of my workstations live. I considered using different colors for serial and ethernet, but decided against it: the cable fanning out into the room is all grey anyway (cut to size, from a roll). It might look pretty on the side of the switches, but not so at the patch strips. Also, what colors to use for currently unassigned cable? In short: everything is black.

All cables from systems to patch strips are labeled on either end. The cables from patch strips to 19" rack are only labeled on the rack side. If I ever undo those patch strips I'll be in hell and will have to dig up the connectivity tester. Of course I keep a spreadsheet mapping cables to systems.

VLANs: I have them, but most are not related to my SGIs. It does mean all my switches are managed though, and the correct VLAN parameters set for every port. Can't simply plug something into any port and expect it to work. More flexibility lost. Of course inter-switch links are multi-VLAN so I can make something 'appear' where I need it.

So there you have it. These are the choices I made. It took years to complete, and other people would probably have done things differently. Also, I've said it before: this is not a business network. In a business network you can't tie down everything for good. I've got the density of a server room with the ambience of an office.

[Intuition]

Wow, been such a while since I checked this thread.

This is an epic project with a monumental result.

Thanks for sharing all the pics and info.     

[jan-jaap]

With the wiring mostly in place I decided to hang some of the posters I collected / had printed over the years:



I especially like the "IRIX releases" poster. It's from 2000 so it ends around IRIX 6.5.9 but I think it's excellent.



Other side of the room. Normally there's a 24" screen with the keyboard/mouse on the right, but I repurposed it as a second screen for my "work from home" setup.



Above the Wacom table there's a memorabilia from InfiniteReality project. I believe it's a wafer of GE chips. Here's a better view:



The wafer is broken. Ah, well...