Reverse-engineering the tardist file format

Reverse-engineering the tardist file format - Printable Version

+- IRIX Network Forums (//forums.irixnet.org)
+-- Forum: SGI/MIPS (//forums.irixnet.org/forum-3.html)
+--- Forum: Development/Porting (//forums.irixnet.org/forum-9.html)
+--- Thread: Reverse-engineering the tardist file format (/thread-780.html)

Reverse-engineering the tardist file format - Unxmaal - 12-18-2018

I have two main projects: a fork of irixboot, at https://github.com/unxmaal/irixboot/ , and irix_ansible, at https://github.com/unxmaal/irix_ansible/ .

The combined goal of these two projects is to allow a user to automatically deploy and configure a full IRIX installation to a bare-metal SGI system with minimal (or no) user intervention.

Currently irixboot has a few gaps where user input is required:

User has to boot the SGI into PROM, format drive, and run inst
User has to provide inst with media locations fixed with auto-generated selections file yesterday
User has to interact with inst to complete install, rqsall, and reboot
User has to login as EZSetup, or at least root, to
configure networking
install wget, openssh, python from bundles copied to /tmp by inst
enable sshd

At this point, irix_ansible can take over and complete the remaining configuration tasks, including tasks for security, nekoware, and remote x11.

My current thoughts on how to solve the networking/ssh/python gap is to have irixboot's ansible generate the required networking and init scripts, then package them into a small tardist that irixboot would include as one of the installation media.

This would allow IRIX to start for the first time with a properly-configured network and with sshd running as a service.

I've searched quite a bit for new or old Linux-compatible software that can create a tardist file. At this point I don't believe there is any. I did find 'epm' that claims to create tardists, but on further investigation requires 'gendist', which only exists on IRIX.

I've done some crude examinations of extracted tardists, and they seem to consist of at least 3 files:

<package name>: binary, with some files listed
<package name>.idb: list of file paths and their offset location in the .sw file
<package name>.sw: large file that appears to be a concatenation of all of the actual files in the package

Any resources or guidance would be much appreciated.

If I've gone down a dark path here, and there's a better way to transfer some files to an installation target (and perhaps execute them!) during install time, please let me know!

RE: Reverse-engineering the tardist file format - dexter1 - 12-18-2018

This reminds me of a package called roboinst which was used to deploy IRIX machines with the combination of a file server with different configuration files for installing tardists and console access to each IRIX machine. Ah, those were the days of installing hundreds of IRIX machines :-) ...

Anyway, to reiterate: On IRIX, tardists are the most likely choice for a package system, since IRIX installations are based around it. Gendist/swpkg is used to build tardists and inst/swmgr installs them.

Tardists are uncompressed tar archives of an inst package and, as you already have found out, contain at least:

- a binary spec file without extension which is named after the base product name, like "ftn" if the base product is called "ftn".
This file has all the info on a base product, like its version number, creation date, amount of images and their subsystems, all dependencies, incompatibilities, whether it is a patch for another product, etc...

- an IDB file with extension .idb, e.g. "ftn.idb", which contains (in ASCII) the location (offset) of each file in an image file and its cksum, its destination location in an install, extra flags relevant for inst, and scripts which will be run when a file is installed or removed.

- one or more image files with filename base-product-name.image-name, e.g. "ftn.sw" or "ftn.man", which contains for that image all the files in compressed format (done by the Unix compress program which creates .Z files)

So far so good. The problem is of course that the tools to make these tardists are IRIX only. But there may be a way to create them in linux after all:

For the image files and idb files there is a tool called mydb.c mentioned before in other threads which does the extraction for you.
See http://persephone.cps.unizar.es/~spd/src/other/mydb.c
So with some programming you can do the reverse: create image files from an IDB file by compressing and concatenating all files, noting their offset and cksum.

The spec file is binary and needs a bit of reverse engineering, which i have invested some time to do this. I have made a crude c program available here on github and with a simple

Code:
cc -o specfile specfile.c

./specfile myproduct

you can try it out on an existing spec file.

I did this reversing with hexedit on several known packages generated with gendist from the ASCII spec and idb files. I'm not done yet, but this will give you an idea of how feasible (or difficult) it is to create tardists on non-IRIX systems.

RE: Reverse-engineering the tardist file format - callahan - 12-19-2018

FWIW, Roboinst is discussed starting on page 147 of techpub 007-1364-140

RE: Reverse-engineering the tardist file format - Unxmaal - 12-19-2018

(12-18-2018, 10:55 PM)dexter1 Wrote: This reminds me of a package called roboinst which was used to deploy IRIX machines with the combination of a file server with different configuration files for installing tardists and console access to each IRIX machine. Ah, those were the days of installing hundreds of IRIX machines :-) ...

I'd looked into roboinst, but decided against it as it requires another running IRIX machine.

Discord user Northsky reminded me that IRIX runs both telnet and ftp at boot. It's likely I won't need to create tardists, since I can use telnet for command/control and ftp for file transfer.

However... it'd still be neat to be able to generate tardists at-will from within irixboot.