[Raion]

On another note I DID manage to get schily tar installed. I'll be testing this soon, but I'll be making a tarball of both it and schily make probably soon.

Edit: Well that didn't work. Inst still balks:

sudo inst -f neko_make_42.tardist

Default distribution to install from: neko_make_42.tardist

For help on inst commands, type "help overview".


Inst 4.1 Main Menu

1. from [source ...]            Specify location of software to be installed
2. open [source ...]            Specify additional software locations
3. close [source ...]          Close a software distribution location
4. list [keywords] [names]      Display information about software subsystems
5. go                          Perform software installation and removal now
6. install [keywords] [names]  Select subsystems to be installed
7. remove [keywords] [names]    Select subsystems to be removed
8. keep [keywords] [names]      Do not install or remove these subsystems
9. step [keywords] [names]      Interactive mode for install/remove/keep
10. conflicts [choice ...]      List or resolve installation conflicts
11. help [topic]                Get help in general or on a specific word
12. view ...                    Go to the View Commands Menu
13. admin ...                    Go to the Administrative Commands Menu
14. quit                        Terminate software installation

Inst> list
Unpacking tardist file into temporary distribution directory /var/tmp/tardista00DlL
ERROR: tar terminated abnormally

Inst> quit

All the same, I did manage to get star and it's far faster/better than bsdtar. I'll keep on planning to package libarchive for other ports, but I'll probably be using "star" as my personal tar.

So the command inst uses is : /sbin/tar xBfRRRo -

The problem flag commands appear to be B and RRR which do the following:

Code:

When specified as RRR, all
       filenames (including    the targets of symbolic    links) are made
       directory relative; that is,    they are extracted as ./filename.
       This    mode is    intended for the use of    programs running as root, such
       as tardist, that want to be sure that they do not overwrite system
       files when extracting archives, either by accident, or through a
       trojan horse    attack.     Since the t,x and other options referring to
       specific files use pattern matching,    you must omit the leading '/'
       when    specifying directory or    filenames with those options when
       using the R option.    Thus, to extract /a/b/c    with a relative
       pathname, you would do:
        tar xvR    a/b/c
       Earlier releases prepended a    leading    '.', requiring the use of:
        tar xvR    ./a/b/c
       The way to check for    sure is    to use:
        tar tvR    | head -2
       and then use    whichever format is used in the    output list.

       If specified    as RRR,    all directory information is stripped from
       pathnames in    the archive, including the pathnames of    symbolic
       links.  All files will be written to    the current directory; no
       directories will be created;    symbolic links will only refer to
       files in the    current    directory.  Note that if two or    more different
       files in the    archive    have the same filename but have    different
       pathnames, the last file extracted will overwrite any earlier file
       with    the same filename.
...

B       Force input and output blocking to 20 blocks    per record, and    cause
       the input processing    to ignore input    record boundaries,
       concatenating all input into    a contiguous stream. This option
       allows tar to work through pipes, networks, and other
       communications channels where the record blocking may not be
       maintained.

This indicates that yeah, it's trying to do it at a blocksize of 20.

star __does__ by default use a blocksize of 20, so B is not a problem. It also strips leading slashes by default. 

This could cause issues if we tried to wrapper inst, and I'm not willing to risk a clobbering install myself to do that. But the RRR functionality could potentially be implemented. 

Either way, I could modify the star program to ignore -B with a printf message "IRIX compatibility flag detected, ignoring" but that won't fix the -RRR flag. It does make you appreciate the thoughtfulness with which IRIX devs did at least implement the functionality.

[jpstewart]

You'll be happy to know that my wrapper (shell script) for IRIX tar should be ready for a beta test by this time tomorrow.  It's piping input and output to gzip/bzip2/xz compression when asked via the z/j/J flags as used in more modern tar implementations.  All other options are passed to IRIX tar so everything inst does should "just work".

It's still missing a few features but I'd like to get some feedback on it before going much further.

There's also a C version that does exactly the same thing.  It's a little ugly, so I'm not yet ready to share that code.  But the script is sufficient for testing out the basic idea and it's a lot easier to edit and debug.

Oh, thanks for posting exact tar command that inst uses.  I was going to ask about, but you're one step ahead of me!

Stay tuned....  (And please forgive me for only having about an hour a day to work on the script and C versions.)

[Raion]

LOL no worries. I'm still offering star for now and looking into having someone add the -B and -RRR flags and trying that, but I'm unsure if that'll ever come to fruition

[jpstewart]

Alright.  Here's the wrapper script that I've created to add z, j, and J options to IRIX tar for bzip, gzip2, and xz compression as implemented by GNU tar and others.  But first some notes:

I normally use bash but this script is written for the standard IRIX /bin/sh using a minimal set of sh features.  It is known to work on IRIX 6.5.22 but should also work on older versions, too.  If anybody can try it out on 5.3 or even 4.0.x let me know how that goes.  (I'd test it myself but my Indigo needs a new battery first.)

The preceeding means that there are some ugly constructs.  I'm not fond of the weird while loop I've used but sh doesn't have the for (( ; ; )) syntax that would look cleaner.  Even worse is the use of cut to pick things apart character by character but the ${name:x:y} expansion isn't available in the shells at hand.  I've used expr for arithmetic where I could have used let or ((..)) on 6.5.x but it doesn't really seem to matter much given the other limitations and that it would have then precluded use on earlier IRIX.  (And while that wasn't one of Raion's requirements I figured I might as well go for it since we were 90% of the way there anyway.)  So to the other shell scripters reading this, please don't criticize the code for being antiquated.  That's a necessary evil unless we can guarantee a newer shell is installed.  And if anyone wants to suggests improvements, please keep them compatible with /bin/sh on IRIX 6.5.22.

I also couldn't use any of the standard getopt methods because of tar's weird syntax. 

Specifically, the first argument to tar (the "key" in the man page) doesn't necessarily start with a -.  It can, but in practice rarely does.  The other oddity is that the b and f options aren't immediately followed by their mandatory arguments.  They cause tar to take their arguments next from the list after the key (in the order the letters appear), but there can be other intervening letters in the key.  The first example from the tar man page is: 'tar cOdbf 20 /dev/rmt/tps0d7ns filelist' where "20" is the argument to b even though it immediately follows f, and then /dev/rmt/... is the argument to f despite the intervening number.  The getopt routines can't parse that, so it has to be done manually.

So far I've only handled creating, listing, and extracting archives from the c, t, and x modes;  r, u, X, and C are yet to come.  I also haven't yet handled the case where tar is called in a pipe with no filename.  Please remember this is just an early prototype!

Also, there may be some issues with weird filenames containing " or $ characters due to the way my script handles quoting and the shell expanding $ within double quotes.  But if you use such weird filenames, I'm inclined to say "not my problem".  LOL.

My recommendation is to rename /sbin/tar to /sbin/tar.real then save this script as /sbin/tar (don't forget to 'chmod +x' it!) since the goal was to replace the system tar with one that can handle compression.  Alternatively, you can also save this script anywhere in your PATH as tar.sh and call it directly as such for testing purposes.

You can configure the script where indicated to point to the real tar program, gzip, bzip2, and xz.  If they're not in your PATH, specify full paths to them.  Otherwise the script will honour your PATH.  I recommend always using a full path for REAL_TAR so that you don't accidentally call this script recursively.

Test this out and let me know what you think of it.  (And by "you" I mean anyone, not just Raion.  The more the better!)  It is not well tested yet.  I'm better at writing code than testing it. 

Please be aware that I may not have time to respond for the next few days.  (Due to other work Sunday to Tuesday or Wednesday.)  But rest assured I will get back to you towards the end of next week and I will read your comments and suggestions between now and then.

Any feedback is greatly appreciated.

Code:

#!/bin/sh

# This script is a wrapper for the IRIX tar program to extend its abilities to
# handle compressed archives in a manner similar to more modern tar programs.

###################
#
# USER CONFIGURATION
# the following defaults should be sufficient or can be edited if needed
#
###################

REAL_TAR="/sbin/tar.real"
GZIP="gzip"
BZIP2="bzip2"
XZ="xz"

###################
#
# END USER CONFIGURATION
# there should be no need to edit anything below here
#
###################

# this is a minor optimization:
# check for one of the z j J compression options
# if none is found, we'll skip further processing and call real tar
case "$1" in
*z*|*j*|*J*)
# compression in use;  must run full script
;;
*)
# everything else means no cmpression, bail out early
${REAL_TAR} "$@"
exit $?
;;
esac

# initialize these to empty values
# if they're still empty after parsing arguments, that's an error
COMPRESSOR=
MODE=
OUT_KEY=
OUT_ARGS=
FILENAME=

# tar expects first argument to be a "key" of single letter options
# we must parse this one character at a time
# we can ignore a leading - if it is present
IN_KEY="${1#-}"
shift
I=1 # index into the "key"
while [ "$I" -le "${#IN_KEY}" ]; do
CHAR=`echo ${IN_KEY} | cut -c $I-$I`

case "$CHAR" in
# check for mode characters
c|x|t)
MODE=${CHAR}
OUT_KEY="${OUT_KEY}${CHAR}"
;;
r|u|X|C)
MODE=${CHAR}
echo "ERROR: Mode ${MODE} is not yet supported by this script" >&2
exit 1
;;
# check for compression modes which DO NOT get passed through
z)
COMPRESSOR="${GZIP}"
;;
j)
COMPRESSOR="${BZIP2}"
;;
J)
COMPRESSOR="${XZ}"
;;
# check for filename argument
f)
OUT_KEY="${OUT_KEY}f"
OUT_ARGS="- "
FILENAME="$1"
shift
;;
# check for blocksize argument
b)
OUT_KEY="${OUT_KEY}b"
OUT_ARGS="$1 "
shift
;;
# passthrough other key letters
*)
OUT_KEY="${OUT_KEY}${CHAR}"
;;
esac

I=`expr $I + 1`
done

# exit if we didn't find a filename  TODO: use stdin/stdout instead of erroring
if [ -z "${FILENAME}" -o "${FILENAME}" = "-" ]; then
echo "ERROR: No tar file was specified." >&2
exit 2
fi

case "$MODE" in
c)
# create compressed archive
${REAL_TAR} ${OUT_KEY} ${OUT_ARGS} "$@" | ${COMPRESSOR} -c >"${FILENAME}"
exit $?
;;
x|t)
# decompress an archive
${COMPRESSOR} -dc "${FILENAME}" | ${REAL_TAR} ${OUT_KEY} ${OUT_ARGS} "$@"
exit $?
;;
# TODO:  add support for r u X C modes
*)
echo "ERROR: Not sure whether to compress or decompress" >&2
exit 3
;;
esac

[jpstewart]

I install GNU tar on all my IRIX systems, too.  I just type 'gtar' on the command line to use it and leave IRIX tar alone where it can still be used by things like inst.  But that requires knowing when to use gtar instead of regular tar.  Or people could install GNU tar as simply 'tar' and put it first in their PATH, but that requires understanding how PATH works, that it can vary between root and other user accounts, and other intricacies that new users might not know.

My understanding was that Raion wants to replace (as opposed to supplement) IRIX tar with something that handles compression while retaining compatibility with the way inst and other system tools call it.  That simplifies things for new users since they don't have to know the differences between multiple versions of tar. 

To maintain inst compatibility, we either need to take the easier (IMHO) route of wrapping IRIX tar with something that supports compression or the harder route of re-working a more modern tar to handle options with the same meaning and syntax as IRIX tar does.

So that's why I'm doing what I'm doing, and hopefully I'm on the same page as Raion!

And on a related note, I'm still working on the wrapper.  I discovered this morning that I completely forgot the block of code in the C language version to just pass through to IRIX tar when there's no compression in use.  Big oops there.  Bad tunnel vision when it comes to supporting compression, I guess!

Any feedback on the preliminary script version?  How is performance in particular?  It's so short, I'm not sure if the C language version is worth the hassle.

[Raion]

I haven't been able to extensively test it, but it works quite fast on my Octane2's and Onyx2. I will be testing it on my R5k Indy and R4600 Indy soon.

As far as tar goes, star is faster than gtar and more compatible with xz archives, so that's worth looking into since I built it and tardist'd it if you're currently using gtar. I certain think with libarchive and star there's little reason to package gnu tar in the new "nekoware".

[jpstewart]

Here's the next draft of the script.  I believe it now supports all of tar's mode letters (c C r t u x X) and being called in a pipe with no filename (or a filename of -).  Be forewarned, however, that using it in a pipe in conjunction with the C r or u modes might not work correctly at the moment.  Also, in those three modes, the filename must use the .tar.gz, .tar.bz2, or .tar.xz extension (depending on compression method) in order to process correctly.  The alternative forms such as .tgz will confuse the script (although not the compression programs themselves).

Internally to the script I've renamed some of the variables.  I discovered the gzip binary uses the GZIP variable for passing options so I've renamed the variables in the "user configuration" section to TAR_REAL, TAR_GZIP, etc. so that they all follow the same pattern.  And they can all be overridden in your environment without having to edit the script.  E.g., to use a custom compression program without renaming/replacing the IRIX native tar, you could do:

Code:

TAR_REAL=/sbin/tar TAR_GZIP=/path/to/custom/compressor ./tar.sh czf ...

The program names can be full paths but they don't need to be.  It still respects your PATH.

As with the earlier version, this has not been extensively tested.  Bug reports and feedback are encouraged.

There's a small bug in the C language implementation.  I've probably just screwed up a pointer manipulation somewhere but it's setting me back again.  I still only have about an hour a day to share between the two versions.  So that one is still a work in progress....

Code:

#!/bin/sh

# This script is a wrapper for the IRIX tar program to extend its abilities to
# handle compressed archives in a manner similar to more modern tar programs.

###################
#
# USER CONFIGURATION
# the following defaults should be sufficient or can be edited if needed
# they can also be overriden by environment variables of the same name in the calling shell
#
###################

TAR_REAL="${TAR_REAL-/sbin/tar.real}"
TAR_GZIP="${TAR_GZIP-gzip}"
TAR_BZIP2="${TAR_BZIP2-bzip2}"
TAR_XZ="${TAR_XZ-xz}"

###################
#
# END USER CONFIGURATION
# there should be no need to edit anything below here
#
###################

# this is a minor optimization:
# check for one of the z j J compression options
# if none is found, we'll skip further processing and call real tar
case "$1" in
*z*|*j*|*J*)
# compression in use;  must run full script
;;
*)
# everything else means no cmpression, bail out early
${TAR_REAL} "$@"
exit $?
;;
esac

# initialize these to empty values
# if they're still empty after parsing arguments, that's an error
COMPRESSOR=
EXTENSION=
MODE=
OUT_KEY=
OUT_ARGS=
FILENAME=

# tar expects first argument to be a "key" of single letter options
# we must parse this one character at a time
# we can ignore a leading - if it is present
IN_KEY="${1#-}"
shift
I=1 # index into the "key"
while [ "$I" -le "${#IN_KEY}" ]; do
CHAR=`echo ${IN_KEY} | cut -c $I-$I`

case "$CHAR" in
# check for mode characters
c|C|r|u|t|x|X)
MODE=${CHAR}
OUT_KEY="${OUT_KEY}${CHAR}"
;;
# check for compression modes which DO NOT get passed through
z)
COMPRESSOR="${TAR_GZIP}"
EXTENSION=".gz"
;;
j)
COMPRESSOR="${TAR_BZIP2}"
EXTENSION=".bz2"
;;
J)
COMPRESSOR="${TAR_XZ}"
EXTENSION=".xz"
;;
# check for filename argument
f)
FILENAME="$1"
# we'll have to add f and appropriate filename when building the real command
shift
;;
# check for blocksize argument
b)
OUT_KEY="${OUT_KEY}b"
OUT_ARGS=" $1 "
shift
;;
# check for comparison tree path
X)
OUT_KEY="${OUT_KEY}X"
OUT_ARGS=" '$1' "
shift
;;
# passthrough other key letters
*)
OUT_KEY="${OUT_KEY}${CHAR}"
;;
esac

I=`expr $I + 1`
done

if [ -z "${COMPRESSOR}" ]; then
echo "ERROR:  Not sure which compression program to use." >&2
exit 1
fi

case "$MODE" in
c)
# create compressed archive
if [ \! -z "${FILENAME}" -a "${FILENAME}" \!= "-" ]; then
# compress to file
${TAR_REAL} ${OUT_KEY}f ${OUT_ARGS} - "$@" | ${COMPRESSOR} -c >"${FILENAME}"
else
# do not redirect to file
${TAR_REAL} ${OUT_KEY}f ${OUT_ARGS} - "$@" | ${COMPRESSOR} -c
fi
exit $?
;;
t|x|X)
# decompress an archive
if [ \! -z "${FILENAME}" ]; then
# decompress from file
${COMPRESSOR} -dc "${FILENAME}" | ${TAR_REAL} ${OUT_KEY}f ${OUT_ARGS} - "$@"
else
# default to reading from stdin
${COMPRESSOR} -dc | ${TAR_REAL} ${OUT_KEY}f ${OUT_ARGS} - "$@"
fi
exit $?
;;
C|r|u)
# these modes update an existing tar file so we'll decompress, run tar, and compress again
if [ \! -z "${FILENAME}" -a "${FILENAME}" \!= "-" ]; then
# operate on existing file
${COMPRESSOR} -d "${FILENAME}" && \
${TAR_REAL} ${OUT_KEY}f ${OUT_ARGS} ${FILENAME%$EXTENSION} "$@" && \
${COMPRESSOR} "${FILENAME%$EXTENSION}"
else
# just pipe everything through -- not sure this makes sense though
${COMPRESSOR} -dc | ${TAR_REAL} ${OUT_KEY} ${OUT_ARGS} "$@" | ${COMPRESSOR} -c
fi
exit $?
;;
*)
echo "ERROR: Unsupported mode "${MODE}".  Not sure whether to compress or decompress." >&2
exit 2
;;
esac

[jpstewart]

At last, here's the long overdue C version of the tar wrapper to implement compression.  It's at the end of this post.  (Save it as "tar_wrapper.c" without the ".txt" extension.  The forum doesn't accept ".c" files.)  There's no copyright or license notice at the moment.  As far as I'm concerned it is now property of Raion and the IRIX Network.  We can add license terms later if need be.

This is for anyone who is interested to try out.  The more the better!

Compiling:

Running 'cc -o tar_wrapper tar_wrapper.c' is sufficient.  Add -O2 or -O3 if you want the compiler to optimize it, but it's so small it doesn't make much difference. 

By default it assumes that the real IRIX native tar has been renamed to /sbin/tar.real, and that gzip, bzip2, and xz are all in your path.  These can be overridden by hand-editing the source or by defining the macros DEFAULT_TAR, DEFAULT_GZIP, DEFAULT_BZIP2, and/or DEFAULT_XZ on the compiler command line.  Note that you must provide a C-style string in double quotes, which will need to be backslash-escaped in the shell.  E.g., 'cc -DDEFAULT_GZIP=\"/path/to/gzip\" -DDEFAULT_TAR=\"tar\" ...'.  You can either specify an absolute path or rely on the command being found in your $PATH.

Two additional flags are available for debugging.  Add -DDEBUG when compiling and it will print a trace (on stderr so as not to corrupt stdout which may be in use by tar itself) as it parses options and then will print the full command line that it executes.  Add -DNO_RUN and it will only show the trace without actually running any commands.

Installing:

(These instructions are basically the same as for the earlier script, but I'm putting them here for anyone who misses the earlier post.)

To replace the system tar, which is found in /sbin/tar and a symlink to it at /usr/bin/tar, rename /sbin/tar to /sbin/tar.real and copy the freshly compiled tar_wrapper to /sbin/tar.

Alternatively, you can put the compiled program anywhere you like and call it directly on the command line.  In this case you must set DEFAULT_TAR to the correct path.  Be especially careful not to accidentally call this script recursively.  Using the absolute path to system tar is highly recommended in all cases, and mandatory if you name this wrapper simply tar and put it in your $PATH.

Known Bugs:

These are things I know aren't fully functional yet.  But I want to get this out for beta testing and feedback while I fix them.  Or other people can offer patches if they like!

• Very little checking is done to ensure we don't overflow command line and filename buffers.  I know that needs to be fixed ASAP.
  
• All arguments passed through to real tar get enclosed in single quotes to ensure filenames with spaces and other characters don't get mangled by the shell.  But filenames containing single quotes / apostrophes will get mangled by the simple quoting in use.  Until that's fixed you can't pass such filenames on the command line; you have to find another way to tell tar which file(s) to include.
  
• Modes r and u append to and update an existing tar file.  If it is compressed, we have to de-compress it, operate on the resulting tar file, and then re-compress the tar file.  File names like "foo.tar.gz" become "foo.tar" when de-compressed.  Currently only .gz, .bz2, and .xz extensions are recognized and removed from the resulting call to the real tar program.  So if you named your file "foo.tbz2" instead (which bunzip2 will uncompress to "foo.tar"), it won't work with this wrapper (yet).  The wrapper needs to properly handle the full set of filename extensions supported by the various compressors.
  

Possible Future Enhancements:

These are things that might be nice to add, after the known bugs (and any more that come to light!) are dealt with.  These are in no particular order.  I'd like feedback on other people's priorities here.  What's important to you, and what's not?

• Use environment variables to allow overriding paths to tar, gzip, bzip2, and xz at run-time.  (Currently they can only be changed when compiling.)
  
• Support additional compression options such as the --lzma option from other tar implementations.
  
• Support auto-determining compression method from the filename suffix like GNU & libarchive.
  
• Support the --use-compress-program option like the others.
  

[Raion]

Hi jp,

Just tested the script version on my box and it seems to work fine. I think with a manpage that'd work well - this was on my Onyx2 so I'll be testing speed differences on slower boxen to see how they work out.

Nice work, and I think i can definitely incorporate this into the first of many "patches" for IRIX.

[jpstewart]

I think with a manpage that'd work well

I'll see what I can do for a man page over the next few days.  It's been years since I've had to write one of those!

Nice work, and I think i can definitely incorporate this into the first of many "patches" for IRIX.

Thanks!  I'm glad I could help out.