Disc Images vs Tarballs -- the Master Thread
#1
Disc Images vs Tarballs -- the Master Thread
So we've had some debates on the merits of each in the past and such. So I wanted to compile a proper master thread for two reasons:

1. Give up to date info on what IRIXNet is doing about disc preservation and how we plan to tackle disc image stuff. 

2. Give reasons for our current/past opinions, and also explain how we're gonna change that up. 

3. Give people (myself being also guilty) an outlet to hash this out. 

Ok. With that out of the way, here's what we're doing about disc images in the near future:

One of our US servers with big storage space is going offline soon. I'm not ready to replace it yet, so it's likely some of our services will require the use of mirrors (ftp.irixnet.org being the big one) for the foreseeable future once it shuts off. Once we've got something to replace it, I will bring that back online and start thinking of how to handle adding disc images. 

What's current/non-negotiable:

They will either use .img or .efs file extensions. Not .iso or proprietary .mdf/mds or .nrg or whatever. This is because these aren't ISO-9660 or Joliet/Rockridge type images, nor are they UDF or customized ISO FS (as in the case of most video games on CD/DVD) I'm aware some of the more stupid programs on OSes like macOS or Windows won't properly recognize them with that, but that's not a problem I can resolve since we're not the programmer of those programs. You'll need to find workarounds or use a better program. 

They will be stored separate from network install files. 

They will not be given storage priority (Meaning that space-constrained boxes will not host them)

As for /why/ we've not allocated them in the past, and why we've been resistant to it, here's why:

For much of the earlier history, I couldn't afford massive servers with terabytes of disc space. In my experience, deduplication doesn't work on efs images properly. I don't know why, but it's probably to do with how block-level deduplication works and needs to be explained by someone smarter than me. In any case, when I experimented with it at length, it simply straight up didn't work, and the dedup percentage for ZFS plummeted. That meant I simply had less storage density out of them.

Additionally, you cannot mount/use EFS images on IRIX. No loopback FS. No burning capability out of the box. No efs2tar type program. It means you can't use them on the OS that it's designed for, so I elected for tarballs. 

Tarballs are easier to use for network installs, and efs2tar only works on WSL, macOS and GNU/Linux, not on any other OS including vanilla Windows or the BSDs or whatever. That makes it harder than you think for interoperability reasons. 

We've had jrra.zone and other sources as well, covering what we couldn't, so there was outlets.

Some people are OK with anarchy/disorganized/not properly named sources or without interop guidelines. I try to keep things a bit more orderly and high quality. It doesn't please everyone, and I get that, but it's the majority that counts there.

Anyways, let's hash it out. If you're frustrated with my approach, tell me.

I'm the system admin of this site. Private security technician, licensed locksmith, hack of a c developer and vintage computer enthusiast. 

https://contrib.irixnet.org/raion/ -- contributions and pieces that I'm working on currently. 

https://codeberg.org/SolusRaion -- Code repos I control

Technical problems should be sent my way.
Raion
Chief IRIX Officer

Trade Count: (9)
Posts: 4,239
Threads: 533
Joined: Nov 2017
Location: Eastern Virginia
Website Find Reply
09-12-2021, 04:12 PM
#2
RE: Disc Images vs Tarballs -- the Master Thread
Thanks for posting that.  Quite frankly, I'm the sort of person for whom it doesn't matter if we agree or disagree.  What matters to me is that you've got well thought out reasons for doing things your way rather than just making rash decisions for no reason.  I respect your (or anyone's) reasoning when I can see it all laid out so clearly.

That said, I do totally agree with your reasoning about the lack of support for loopback mounting on IRIX.  That's kind of a show-stopper for disk images.

Like you, I like things well structured with clear guidelines and naming conventions.  So thank you!

SGI:  Indigo, Indigo2, Octane, Origin 300
Sun:  SPARCstation 20 (x4), Ultra 2, Blade 2500, T5240
HP:  9000/380, 425e, C8000
Digital: DECstation 5000/125, PWS 600au
jpstewart
Developer

Trade Count: (1)
Posts: 444
Threads: 6
Joined: May 2018
Location: SW Ontario, CA
Find Reply
09-14-2021, 11:20 PM
#3
RE: Disc Images vs Tarballs -- the Master Thread
The only rebuttal I have for Disc Images is that I'm attempting to create install source drives (upgraded install disc media, using a removable SCSI reader that use CF chips, not optical). Once I get a working example, I'll start a new thread about it!

I can use dvhtool to extract the MR & other items from an install disc (Part 7 & 8 ) and install them on new media. Without the first disc, I don't have the layout to examine, nor the files to add. Sure, the boot process gets them somehow via netboot.

Since I'm emulating the correct partition layout I just need the right files. It's much harder get what I need from another source.

So for anyone using physical media (or preparing physical media) the lack of images is a big deal.

I really hate the entire netbooting thing, sure it's cool if you've got a working setup, but the setup is just too much trouble for installing Irix (especially on older hardware with 10Mb Ethernet). When I can just pop in a disc (granted it's like 7 discs now). If I can get this larger install media idea to work, I'd have all the files in a install source HDD/Flash chip then just hit go.

So I'm big of install media (of some sort), the tarballs are useless to me other than looking at the package listing to tell what's on what disc.

I'd rather just have the disc images and instruct people on burning them for each major platform or using Linux loopback mounting to read the EFS root and extract the files they want.

But that's me.
weblacky
I play an SGI Doctor, on daytime TV.

Trade Count: (10)
Posts: 1,716
Threads: 88
Joined: Jan 2019
Location: Seattle, WA
Find Reply
09-15-2021, 01:50 AM
#4
RE: Disc Images vs Tarballs -- the Master Thread
My rebuttal is only that 10mbit ethernet is faster and more consistent than most CD drives that can be read over SCSI speeds. It's not that hard to get IRIX setup for that, it just requires dedicated hardware. I understand not everyone wants to commit to that.

I'm the system admin of this site. Private security technician, licensed locksmith, hack of a c developer and vintage computer enthusiast. 

https://contrib.irixnet.org/raion/ -- contributions and pieces that I'm working on currently. 

https://codeberg.org/SolusRaion -- Code repos I control

Technical problems should be sent my way.
Raion
Chief IRIX Officer

Trade Count: (9)
Posts: 4,239
Threads: 533
Joined: Nov 2017
Location: Eastern Virginia
Website Find Reply
09-15-2021, 02:58 AM


Forum Jump:


Users browsing this thread: 1 Guest(s)