[IRIXdejazel]

Hi everyone,

I hope this is the right place to put this question. I tried checking the FAQ first, but it is currently down.

At work, we are trying to set up a Mac to work as a system for backing up directories from an SGI O2 running IRIX 6.5 (or 6.5.x, I guess).

From what I read on the web, the easiest/best way is to use wget, which would allow us to move entire directories over the local network to the Mac.
We managed to find an already compiled version for the Mac, and it worked when we connected to the O2 and dragged a directory (with all of its many directories and sub-directories) over to the Mac.

A problem arose when we realized that wget appears to work one way. We also need to be able to send those directories back to the O2. I assume wget on the O2 would allow us to do that.

A 2nd problem arose after we managed to get the wget source (from the GNU project) to configure, but when running the make, I get an error claiming that "The MIPSpro C Compiler (license FEATURE string = cc) requires a license password." And then a half a page of how to contact SGI to get a license. Obviously not gonna happen.

We have not tried to compile this under the root user, as I assume that won't work either. Though I could be wrong.

Prior to this, we backed up our work on tapes for years (decades?) but the tape drives are becoming unreliable. Thus, the possible solution of using a local network to just push/grab the files to/from the Mac. They only other thing we have regularly done is use ftp to move files one at a time to the machines that need them. But that is usually only 3 files. To back up all of the work requires countless files and directories.

So I guess what I am asking is, is there any way to get an already compiled version of wget that will run on my machine? My first instinct is no, judging from what that configure file was doing. But I freely admit near-total ignorance in this area. I have never tried to build Unix software on this machine before. We have a proprietary software package that I use to create visual display database scenes of airports for flight simulators. I have never had to use the machine for anything other than that, so my "Unix guru" experience is lacking.

I know I could tar -cf the directory to an archive file, then ftp it over that way, but I was hoping to be able to use wget, as it seems very powerful and flexible and would likely serve our needs very well. If I could get it to work on both ends.

So, if wget cannot be installed unless built on the host system, then I guess I need some form of compiler already ready to go, so I can use that to compile the wget source. But where do I get something like that? Or instructions or tricks on how to fix the compiler that I have, so that it will run without complaining about the lack of a license.

==Dave

[dexter1]

Welcome Glad you can join us and ask for help.

So for transferring files to and from a Mac there are probably half a dozen options, some easier, some faster:

- Use a common NFS server. This requires the nfs subsystem on the O2 which is retrievable from http://ftp.irisware.net/pub/irix-os/irix...nfs.tar.gz
- Use scp/sftp. Both the O2 and Mac can run an openssh server which you can use to copy files with sftp or scp.
- Use a statically linked wget on the O2. I have made a Nekoware package which has a static build of the wget binary here: https://tudl1910.home.xs4all.nl/nekoware....4.tardist . It will complain about having unmet dependencies, but installing the neko_wget.sw.static should be fine. Mind you that you should supply --no-check-certificates as an option to wget to disable certificate checking.

The compiler moaning about not having a license is merely cosmetical. AFAIK it should still work, although there are a few dependencies which have to be met in order to build wget. This is all solvable because lots of libraries have been ported already, but only if you like hassling with development on the O2.

[mamorim01]

They only other thing we have regularly done is use ftp to move files one at a time to the machines that need them. But that is usually only 3 files. To back up all of the work requires countless files and directories.

...

I know I could tar -cf the directory to an archive file, then ftp it over that way, but I was hoping to be able to use wget, as it seems very powerful and flexible and would likely serve our needs very well. If I could get it to work on both ends.

...

I am unfortunately not able to provide advice regarding installing/compiling wget under Irix, but have you tried mget and mput from within ftp? This would allow you to send multiple batches of files using wildcards (ie, mget *.tgz), the main drawback being it is not recursive (in other words, it will not copy subdirectories). Apologies if that is an obvious suggestion but I noticed you mentioned ftp'ing files one at a time, so this might be a workaround. Note that if you try to mget or mput files you may be prompted to confirm each file before the transfer begins, this setting can be overridden through the prompt command.

Having an openssh server installed on the O2 side, as suggested by dexter1, would allow you to recursively transfer files from the Mac using scp. This is a fairly powerful tool which you can use from within the command line, or even under a GUI such as Transmit, depending on your preferences. I am using a similar arrangement to upload files to a set of raspberry pi's (each one running openssh server) and it works wonders.

[Unxmaal]

Get this:
http://techpubs.spinlocksolutions.com/ir...ekodeps.pl

Just copypasta it to vi on your SGI box, the old fashioned way. Save it in /usr/nekoware/.

Run it to generate the configs:
perl nekodeps.pl --config

Since nekochan died, you'll need to use a mirror. Modify the config file at around line 60 to read:
default_mirrors => [
'mirror.larbob.org/nekoware',

Run the script again with the bootstrap options

perl nekodeps.pl --bootstrap --ion

That'll install a new copy of wget.

If you need any other packages (like neko_ssh, which you will), modify the script again at line 60:
default_mirrors => [
'http://mirror.larbob.org/nekoware',

(For some reason the new version of wget doesn't like the old url style.)

Then you can run
perl nekodeps.pl -l
for a full packages list, or
perl nekodeps.pl <pkgname>
to install a package.

[IRIXdejazel]

Thank you, everyone, for your helpful replies. It is always eye-opening to realize that even when you might think you know something, you actually know only a small percentage of what actual expert people know! 

I am not sure which helpful advice I will try first, but I will look into all of them. I will let you know what I found to work the best, or what works at all maybe.

I was aware of the mput and mget under ftp. But as you pointed out, that is not helpful if I want to move hundreds of files under dozens of nested directories.

This whole process is kind of neat, as it has been ages since I have needed to dig this far into something to make it work. Well, at least, digging into something where someone else knows far more than I do. Many of the problems with the database building software I have had to figure out on my own over the years, as no one here, or pretty much anywhere, uses that age-old software anymore.

Thanks again! If there are other suggestions, keep them coming if you wish. I may never know what may turn up to be useful now or in the future.

==Dave

[Raion]

Wiki is fixed, sorry about that. You can read the FAQ now.

[IRIXdejazel]

I figured I would give everyone an update on the wget situation.
I am still having problems. The result is this long post.

I decided to try to compile my own version from source, but could not get it to complete. The compiler would complain about not finding something; I would find whatever it was looking for and put it in a directory that I thought it needed to be in, and it would continue. Until it found another thing it needed. Rinse, repeat. Eventually, it hit some kind of error that I could not figure out (I forgot what it was and have since given up on compiling it.)

I then tried the wget-static version 1.19.4 suggested by dexter1. I managed to use the Software Manager to install it (where it complained as predicted; but I just told it to ignore the thing it was complaining about, and wget-static installed.)

However, when I try to use it to pull stuff from the Mac, it does this (I edited the user, password, and host address):

/usr/nekoware/bin/wget-static -r -nH -l 20 -d --cut-dirs=0 ftp://user:password[at]10.100.100.99//tncm_lb01
Setting --cut-dirs (cutdirs) to 0
Setting --cut-dirs (cutdirs) to 0
DEBUG output created by Wget 1.19.4 on irix6.5.

Reading HSTS entries from /usr/visual/.wget-hsts
URI encoding = 'ISO8859-1'
logging suppressed, strings may contain password
Converted file name 'tncm_lb01' (UTF-8) -> 'tncm_lb01' (ISO8859-1)
Using '.listing' as listing tmp file.
--2018-12-06 20:19:06--  ftp://user:*password*[at]10.100.100.99//tncm_lb01
           => '.listing'
Segmentation fault (core dumped)

Before I go any further, I want to point out that I also found an already compiled binary of wget version 1.8.2. I don't recall where I found it, but it actually works. The only thing is, that version does not have the --preserve-permissions option. So when I pull directories from the backup disk on the Mac, the permissions get changed, and the proprietary software that I need won't rebuild. I found out what was wrong, and fixed the permissions, and it does build after that, but that is a step that I would rather not have to take. The already compiled version on the Mac is 1.11.4, and it works also, and has the --preserve-permission option. I would like to have that option on the O2 on the other end, if I can.

I also found a compiled version 1.10.2
It does this when I try to use it:

usr/local/bin/wget --version
 2457:usr/local/bin/wget: rld: Fatal Error: Cannot Successfully map soname 'libssl.so.0.9.7' under any of the filenames /usr/lib32/libssl.so.0.9.7:/usr/local/lib/VsuCNAirport/libssl.so.0.9.7:/usr/local/lib/libssl.so.0.9.7:/usr/lib32/internal/libssl.so.0.9.7:/lib32/libssl.so.0.9.7:/opt/lib32/libssl.so.0.9.7: 

Another already compiled version 1.14 does this:

wget_1.14 -r -nH -l 20  --cut-dirs=0 ftp://user:password[at]10.100.100.99//tncm_lb01
--2018-12-06 19:55:21--  ftp://user:*password*[at]10.100.100.99//tncm_lb01
           => '.listing'
 2433:wget_1.14: rld: Fatal Error: attempted access to unresolvable symbol in wget_1.14: getaddrinfo


I then tried to do what Unxmaal suggested. Since my O2 cannot connect to the internet, I used the Mac to do that. I had to modify the procedure slightly to make it work, but I did manage to get all the distribution packages or dependencies downloaded to the Mac, which I then used the wget 1.8.2 on the O2 to pull them to the O2, where I used the Software Manager to massage the installation of everything, which worked. I was surprised by that.

But when I try to use it, it does this:

/usr/nekoware/bin/wget -r -nH -l 20 -d --cut-dirs=0 ftp://user:password[at]10.100.100.99//tncm_lb01
Setting --cut-dirs (cutdirs) to 0
DEBUG output created by Wget 1.11.3 on irix6.5.

Using `.listing' as listing tmp file.
--2018-12-06 00:46:39--  ftp://user:*password*[at]10.100.100.99//tncm_lb01
           => `.listing'
Trace/BPT trap (core dumped)

I then figured that maybe the 1.19.4 version could be re-installed, since all those other packages were now on the disk. But they apparently are not what it needs, as it still won't finish the install unless I tell it to ignore the conflicts.

The ftp version of the wget command is the one that works both ways, and does exactly what I need it to do, except for messing up the permissions. The http does not function in either direction, for reasons that I do not know. But since it works ftp-wise, I guess it doesn't matter.

So, the TL;DR version is: is there an already compiled version hiding somewhere that will actually work for me? Or is there a way to make 1.19.4 or 1.11.3 work correctly? (I am giving up on the other 2 varieties, since there seems to be nothing there except the binary. So probably no way to make any changes there.) Or, a way to actually make it compile from scratch from the source?

Alternatively, I guess I will just use the 1.8.2 version on the O2 and be glad that at least that works, even though I will have to manually change permissions on several files after pulling them back from the backup.

This whole process has been interesting, frustrating, and somewhat fun. It would be more fun if it would do what I want.

Thanks for all your ideas and help!

==Dave
(I had to use [at] instead of the normal "at" symbol, as it would not let me post otherwise, saying I was including contact info.)

[dexter1]

Looks like you've found a bug in the wget-static binary of 1.19.4 which i've build. I can debug that version with a bit of time.

On compile: The neko_wget has a release notes file which states environment variables and build procedure. It's under optional software and is printed below.

Code:

SOURCE/VERSION
--------------
wget-1.19.4 / 6


IRIX/MIPSPRO VERSION
--------------------
6.5.22 / 7.4.4m


ENVIRONMENT VARIABLES
---------------------
setenv CC c99
setenv MAKE /usr/nekoware/bin/gmake
setenv PERL /usr/nekoware/bin/perl
setenv CFLAGS '-O2 -mips4 -n32'
setenv CPPFLAGS -I/usr/nekoware/include
setenv LDFLAGS '-L/usr/nekoware/lib'
setenv LIBS '-lunistring -lintl -liconv'
setenv OPENSSL_CFLAGS '-I/usr/nekoware/include'
setenv OPENSSL_LIBS '-L/usr/nekoware/lib -lssl -lcrypto'
setenv PATH /usr/nekoware/bin:$PATH

For builds on mips3 systems, use:

setenv CFLAGS '-O2 -mips3 -n32'


CONFIGURE
---------
./configure --prefix=/usr/nekoware --disable-rpath --with-ssl=openssl


BUILD
-----
gmake
gmake check
gmake install


KNOWN DEPENDENCIES
------------------
neko_gettext.sw.lib 10 maxint
neko_libiconv.sw.lib 5 maxint
neko_libidn2.sw.lib 1 maxint
neko_libunistring.sw.lib 1 maxint
neko_openssl.sw.lib 33 maxint
neko_pcre.sw.lib 4 maxint
neko_zlib.sw.lib 9 maxint
gmake to build the package
perl to make the documentation and the unittests

ERRORS/MISCELLANEOUS
--------------------
A recent perl is required to create the documentation which is why the path has to be changed to /usr/nekoware/bin to point to the p
erl from Nekoware

There is a more recent version 1.19.5, but its test suite depends on the glibc function 'fmemopen' which is not ported to IRIX.

If you're running the testsuite without libidn2 the test will pass. However i wanted to include unicode strings for foreign domainna
mes and now the test fails at 5 points
============================================================================
Testsuite summary for wget 1.19.4
============================================================================
# TOTAL: 93
# PASS:  77
# SKIP:  11
# XFAIL: 0
# FAIL:  5
# XPASS: 0
# ERROR: 0
============================================================================
I think the failed test are not errors from wget, but from perl, sine it runs the HTTP::Daemon for the necessary tests.

For bootstrapping nekoware downloads a static build is provided with the subsystem neko_wget.sw.static : It installs wget-static whi
ch only depends on a dynamically linked libc.

You can build the static version by setting:
setenv LDFLAGS '-B static -L/usr/nekoware/lib'
and do the configure with:
./configure --prefix=/usr/nekoware --with-ssl=openssl
and replace the libunistring.so and libiconv.so in the Makefile's in . and ./src with the static versions libunistring.a and libicon
v.a

You can see that there are quite a few dependencies, but openssl and zlib are more or less essential, the rest are convenience libraries.

The wget 1.8.4 binary can work around permissions if you package the files into a tar archive first before getting them via wget. Then after having received it untar the package and the file permissions should be preserved.

Still, wget is a bit of a kludge getting files back and forth, especially when there are many versions and none of them seems to operate fully. As an alternative an NFS server or basic ftp/sftp server sould not be too hard to set up.

EDIT: i suspect i fell into the pthread trap again: i think i forgot to link with -lpthread. The getaddrinfo error points to this possibility.

[IRIXdejazel]

Looks like you've found a bug in the wget-static binary of 1.19.4 which i've build. I can debug that version with a bit of time.

On compile: The neko_wget has a release notes file which states environment variables and build procedure. It's under optional software and is printed below.
...
You can see that there are quite a few dependencies, but openssl and zlib are more or less essential, the rest are convenience libraries.

The wget 1.8.4 binary can work around permissions if you package the files into a tar archive first before getting them via wget. Then after having received it untar the package and the file permissions should be preserved.

Thanks. I realized that we could have just used tar and ftp to make and recover our backups, but I was trying to use something that would easily allow the recovery of the entire backup, but also just one or two files as needed, in one step. That's why wget looked so attractive.

Still, wget is a bit of a kludge getting files back and forth, especially when there are many versions and none of them seems to operate fully. As an alternative an NFS server or basic ftp/sftp server sould not be too hard to set up.

EDIT: i suspect i fell into the pthread trap again: i think i forgot to link with -lpthread. The getaddrinfo error points to this possibility.

I am starting to think that I am in way over my head with this wget attempt. I feel like I have learned a lot, but also realize that there is so much I do not understand. Plus, I don't want to keep bugging folks for hand-holding and spoon-feeding, which is what it is starting to look like. I mean, I don't even understand the dependencies thing, or where to find them, or where to put them so that it works. That's why the already-built and running version sounded so nice.

We have previously used ftp to send one or only a few files back and forth. But that is unwieldy for backing up a whole directory tree with dozens of sub-directories and thousands of files. Unless of course we use tar and ftp, which for reasons above, I was trying to find a way around.

If the nice folks here want to continue trying to help me, that would be nice. But I can also understand the idea that because I am apparently so green with regards to this, there may not be any way to actually help me.

I am assuming it must be possible, since there is the 1.8.4 version that works great, except for the permissions thing, and the version that works on the 4 core Mac running Snow Leopard seems to have no issues. It even works on my 2012 Mini running Mojave (I just tried last night.) I just don't know if it is anything that I can fix on my own.

Quite a learning experience though!

Thanks,
Dave

[dexter1]

I've tested the wget segfault bug on my freshly booted O2, as far as i am able to with an anonymous ftp retrieve, but sofar i'm unable to reproduce it:

Code:

odin:/local/frank> wget-static -r -nH -l 20 -d --cut-dirs=0 ftp://ftp:anonymous@ftp.gwdg.de/pub/sub/frwi/infedi/
Setting --cut-dirs (cutdirs) to 0
Setting --cut-dirs (cutdirs) to 0
DEBUG output created by Wget 1.19.4 on irix6.5.

Reading HSTS entries from /home/frank/.wget-hsts
URI encoding = 'ISO8859-1'
logging suppressed, strings may contain password
Converted file name 'pub/sub/frwi/infedi/index.html' (UTF-8) -> 'pub/sub/frwi/infedi/index.html' (ISO8859-1)
Using 'pub/sub/frwi/infedi/.listing' as listing tmp file.
--2018-12-14 16:44:23--  ftp://ftp:*password*@ftp.gwdg.de/pub/sub/frwi/infedi/
          => 'pub/sub/frwi/infedi/.listing'
Resolving ftp.gwdg.de (ftp.gwdg.de)... 2001:638:60f:110::1:2, 134.76.12.6
Caching ftp.gwdg.de => 2001:638:60f:110::1:2 134.76.12.6
Connecting to ftp.gwdg.de (ftp.gwdg.de)|2001:638:60f:110::1:2|:21... failed: Address family not supported by protocol family.
Connecting to ftp.gwdg.de (ftp.gwdg.de)|134.76.12.6|:21... connected.
Created socket 3.
Releasing 0x10419eb8 (new refcount 1).
220-Welcome to ftp.gwdg.de
220-
220
Logging in as ftp ...
--> USER ftp

331 Please specify the password.

--> PASS anonymous

230 Login successful.
Logged in!
==> SYST ...
--> SYST

215 UNIX Type: L8
done.    
UNIX TYPE L8: I know it and I will use "LIST -a" as standard list command
==> PWD ...
--> PWD

257 "/"
done.
==> TYPE I ...
--> TYPE I

200 Switching to Binary mode.
done.  changing working directory
Prepended initial PWD to relative path:
  pwd: '/'
  old: 'pub/sub/frwi/infedi'
 new: '/pub/sub/frwi/infedi'
==> CWD (1) /pub/sub/frwi/infedi ...
--> CWD /pub/sub/frwi/infedi

250 Directory successfully changed.
done.
conaddr is: 134.76.12.6
==> PASV ...
--> PASV

227 Entering Passive Mode (134,76,12,6,120,11).
trying to connect to 134.76.12.6 port 30731
Created socket 4.
done.    ==> LIST ...
--> LIST -a

150 Here comes the directory listing.
(skipping "LIST")done.

pub/sub/frwi/infedi/.listing         [ <=>                                                       ]     251  --.-KB/s    in 0s      

Closed fd 4
226 Directory send OK.
2018-12-14 16:44:23 (3.15 MB/s) - 'pub/sub/frwi/infedi/.listing' saved [251]

DIRECTORY; perms 755; size: 48; month: Oct; day: 10; year: 1994 (no tm);
Ignoring `.' and `..';
Skipping.
DIRECTORY; perms 755; size: 115; month: Oct; day: 10; year: 1994 (no tm);
Ignoring `.' and `..';
Skipping.
PLAINFILE; perms 755; size: 6144; month: Oct; day: 10; year: 1994 (no tm); infedi
PLAINFILE; perms 755; size: 6656; month: Oct; day: 10; year: 1994 (no tm); infedi.txt
Removed 'pub/sub/frwi/infedi/.listing'.
Converted file name 'pub/sub/frwi/infedi/infedi' (UTF-8) -> 'pub/sub/frwi/infedi/infedi' (ISO8859-1)
Converted file name 'pub/sub/frwi/infedi/infedi' (UTF-8) -> 'pub/sub/frwi/infedi/infedi' (ISO8859-1)
--2018-12-14 16:44:23--  ftp://ftp:*password*@ftp.gwdg.de/pub/sub/frwi/infedi/infedi
          => 'pub/sub/frwi/infedi/infedi'
==> CWD not required.
conaddr is: 134.76.12.6
==> PASV ...
--> PASV

227 Entering Passive Mode (134,76,12,6,120,247).
trying to connect to 134.76.12.6 port 30967
Created socket 4.
done.    ==> RETR infedi ...
--> RETR infedi

150 Opening BINARY mode data connection for infedi (6144 bytes).
done.
Length: 6144 (6.0K)

pub/sub/frwi/infedi/infedi       100%[==========================================================>]   6.00K  --.-KB/s    in 0.003s  

Closed fd 4
226 Transfer complete.
2018-12-14 16:44:24 (1.88 MB/s) - 'pub/sub/frwi/infedi/infedi' saved [6144]

Converted file name 'pub/sub/frwi/infedi/infedi.txt' (UTF-8) -> 'pub/sub/frwi/infedi/infedi.txt' (ISO8859-1)
Converted file name 'pub/sub/frwi/infedi/infedi.txt' (UTF-8) -> 'pub/sub/frwi/infedi/infedi.txt' (ISO8859-1)
--2018-12-14 16:44:24--  ftp://ftp:*password*@ftp.gwdg.de/pub/sub/frwi/infedi/infedi.txt
          => 'pub/sub/frwi/infedi/infedi.txt'
==> CWD not required.
conaddr is: 134.76.12.6
==> PASV ...
--> PASV

227 Entering Passive Mode (134,76,12,6,120,84).
trying to connect to 134.76.12.6 port 30804
Created socket 4.
done.    ==> RETR infedi.txt ...
--> RETR infedi.txt

150 Opening BINARY mode data connection for infedi.txt (6656 bytes).
done.
Length: 6656 (6.5K)

pub/sub/frwi/infedi/infedi.txt   100%[==========================================================>]   6.50K  --.-KB/s    in 0s      

Closed fd 4
226 Transfer complete.
2018-12-14 16:44:24 (31.0 MB/s) - 'pub/sub/frwi/infedi/infedi.txt' saved [6656]

Closed fd 3
FINISHED --2018-12-14 16:44:24--
Total wall clock time: 0.7s
Downloaded: 2 files, 12K in 0.003s (3.59 MB/s)
odin:/local/frank>

The segfault occurs at the resolve stage. Do you have any issues with name resolution on your network from the O2?
If not, i can try to squeeze in some time to make a new wget-static with extra libraries linked in, to see if that removes the segfault.