Cluster Filesystems
#1
Cluster Filesystems
Does anyone here have any experience with cluster filesystems? I've got a a fibre channel SAN setup here (IBM DS3512) and I thought it would be neat to share it between a couple of my main Linux boxes with a cluster filesystem. 

A little research suggested that OCFS2 would be the simplest for me. The ocfs2-tools package on Debian contains everything needed to configure the cluster heartbeat and all that stuff. Setting up the two hosts was indeed straightforward. I managed to get them accessing one filesystem on the SAN array without any problems. But the performance of OCFS2 was very disappointing. An rsync backup job to the array seemed vastly slower than before the change to OCFS2, but that was hard to quantify due to the nature of the data being backed up. The weekly job to tar up the array's contents and write it out to tape (LTO5) was also quite slow, running at 58 MB/sec.

I've reverted back to XFS on the array and sharing it via NFS for now. (Which is how it was accessed before the conversion to OCFS2.) The rsync to the array takes about half as much time. Backing up from the array now runs over 100 MB/sec and completes in 3 hours and 21 minutes vs 6 hours and 7 minutes under OCFS2. The difference in performance is huge, for both reading and writing.

So that's got me wondering if there was something wrong with my OCFS2 configuration or if that's just the nature of the beast. And, is it OCFS2 that is slow or all cluster filesystems? 

Before I go through the process of configuring GFS2, does anyone here have experience with it? Or with SGI's CXFS? CXFS is commercially licensed so not really a viable option for my little hobby project, but I wonder how it compares with traditional XFS.

I'd love to hear your thoughts and comments if you have experience with clustered filesystems. Are there other open source options I should be looking at beyond OCFS2 and GFS2?

SGI:  Indigo, Indigo2, Octane, Origin 300
Sun:  SPARCstation 20 (x4), Ultra 2, Blade 2500, T5240
HP:  9000/380, 425e, C8000
Digital: DECstation 5000/125, PWS 600au
jpstewart
Developer

Trade Count: (1)
Posts: 444
Threads: 6
Joined: May 2018
Location: SW Ontario, CA
Find Reply
08-21-2023, 12:39 AM
#2
RE: Cluster Filesystems
Very, very interesting question, I'm cooking pop corn.

You could try to test Proxmox cluster file systems (https://pve.proxmox.com/wiki/Storage, https://pve.proxmox.com/wiki/Proxmox_Clu...m_(pmxcfs)) and compare performances.

I tried Proxmox CFS in the past but no success, I always have installed a NFS server as "SAN controller", converting a SAN to a NAS if a DFS was necessary.

Indy Indy R4400 150 MHz, 256 MB RAM, 24-bit graphics, 6.5.22
  Octane2 Octane2 R12000 400 MHz, 1024 MB RAM, ODYSSEY V6, 6.5.30

IBM 44p model 170 (7044-170), POWER3-II 64 bits 400 MHz, 1 GB RAM
AlphaServer DS10 466 MHz, 128 MB RAM
HP C3600 PA-RISC 8600 552MHz, 2 GB RAM
HP ProDesk 400 G2 MINI Intel Core i5 6th Gen 2,50GHz 16GB, Debian GNU/Linux (NAS)
HP t610 Debian GNU/Linux, 2 GB RAM (NAS)
Linux-RISC
RISC and Unix lover

Trade Count: (0)
Posts: 51
Threads: 4
Joined: Apr 2022
Location: Spain
Website Find Reply
08-21-2023, 11:06 AM


Forum Jump:


Users browsing this thread: 1 Guest(s)