[gijoe77]

so I've been installing everything on the ftp, and I must of done something that is really screwing up my o2.  Basically its not very responsive, it seems to have long periods of time between processing certain commands, for example if I try to uncompress and untar a file, it takes way too long - for example a 200mb .tgz file is taking anywhere from 30-50 minutes to uncompress/untar.

here is a video showing the issue:
https://youtu.be/05Z4iH2Y3bY



the thing that is interesting is that not all commands seem to be having an issue.  here is a diskperf:

Code:

-bash-4.2# diskperf -W -D -n "o2 500GB SSD" /tmp/testfile
#---------------------------------------------------------
# Disk Performance Test Results Generated By Diskperf V1.2
#
# Test name     : o2 500GB SSD
# Test date     : Fri Nov  9 11:42:19 2018
# Test machine  : IRIX o2 6.5 10070056 IP32
# Test type     : XFS data subvolume
# Test path     : /tmp/testfile
# Request sizes : min=4096 max=33553920
# Parameters    : direct=1 time=10 scale=1.000 delay=0.000
# XFS file size : 524288000 bytes
#---------------------------------------------------------
# req_size  fwd_wt  fwd_rd  bwd_wt  bwd_rd  rnd_wt  rnd_rd
#  (bytes)  (MB/s)  (MB/s)  (MB/s)  (MB/s)  (MB/s)  (MB/s)
#---------------------------------------------------------
       4096    9.72    8.72    9.78    7.49    9.74    7.50
       8192   14.82   13.21   14.87   11.53   14.81   11.48
      16384   19.94   17.52   20.02   15.34   19.85   15.41
      32768   24.15   20.98   24.22   19.20   24.18   19.22
      65536   27.02   24.22   27.08   22.75   27.05   22.82
     131072   28.60   25.26   28.66   23.70   28.65   23.78
     262144   29.48   25.18   29.56   23.66   29.53   23.72
     524288   29.94   25.88   30.00   25.47   29.98   25.48
    1048576   30.18   26.19   30.23   25.09   30.22   25.09
    2097152   30.55   26.53   30.63   25.70   30.61   25.66
    4194304   31.15   27.22   31.24   26.81   31.22   26.79
    8388608   31.58   27.63   31.79   27.39   31.81   27.49
   16777216   31.62   27.91   32.15   27.70   32.14   27.83
-bash-4.2# 

here is a mkfile command:

Code:

-bash-4.3# time mkfile 10g /tmp/tempfile

real    6m6.163s
user    0m0.160s
sys     1m0.216s

Currently the o2 has a 500GB SSD, I cloned the disk to a 15K 300GB brand new SCSI disk and it's having the same issue, so I proved out my HDD.  I've been chkconfig'ing off most things, I've been disabling stuff in /etc/rc2.d, I just can't seem to figure out what the issue is.  I figured I'd post and perhaps someone might have some ideas where to look next.

here is my current chkconfig:

Code:

-bash-4.3# chkconfig -s
        Flag                 State               
        ====                 =====               

        autofs               on
        desktop              on
        ipaliases            on
        jaleolic             on
        network              on
        nsd                  on
        routed               on
        rtmond               on
        savecore             on
        sdpd                 on
        snetd                on
        sysevent             on
        tfxd                 on
        timed                on
        verbose              on
        visuallogin          on
        vswap                on
        windowsystem         on
        xdm                  on
        xlv                  on
        ad_elmd              off
        appletalk            off
        array                off
        audit                off
        autoconfig_ipaddress off
        automount            off
        aw_flexlm            off
        cesag                off
        cpumeter             off
        dl_lucid             off
        dl_stone             off
        dl_sws               off
        dl_vic               off
        dl_wire              off
        dl_wiretap           off
        esp                  off
        fcagent              off
        flexlm               off
        fontserver           off
        fsr_xfs              off
        gated                off
        ldap                 off
        lockd                off
        lp                   off
        maple_license        off
        mediad               off
        mrouted              off
        named                off
        nds                  off
        neko_ldap            off
        netwr_client         off
        nfs                  off
        noiconlogin          off
        nostickytmp          off
        nss_fasttrack        off
        ntp                  off
        pmcd                 off
        pmie                 off
        privileges           off
        proclaim_relayagent  off
        proclaim_server      off
        proxymngr            off
        quickpage            off
        rarpd                off
        rsvpd                off
        rwhod                off
        sar                  off
        sendmail             off
        sendmail_cf          off
        sesdaemon            off
        sesinetd             off
        sgi_apache           off
        Shout_Server         off
        soundscheme          off
        Thomson_Licenses     off
        timeslave            off
        ts                   off
        webface              off
        webface_apache       off
        yp                   off
        ypmaster             off
        ypserv               off

my /etc/rc* files:

Code:

-bash-4.3# ls -l /etc/rc*
-rwxr-xr-x 1 root sys 1037 Mar 13  2018 /etc/rc0
-rwxr-xr-x 1 root sys 1441 Mar 13  2018 /etc/rc2
-rwxr-xr-x 1 root sys  439 Mar 13  2018 /etc/rc3

/etc/rc0.d:
total 4
-rwxr-xr-x 1 root sys 1443 Aug 17 22:45 K00ad_elmd
lrwxr-xr-x 1 root sys   14 Aug 30 08:43 K00qx -> /etc/init.d/qx
lrwxr-xr-x 1 root sys   17 Jun  8 17:28 K01atalk -> /etc/init.d/atalk
lrwxr-xr-x 1 root sys   21 Jun  8 17:28 K01qpage -> ../init.d/qpageserver
lrwxr-xr-x 1 root sys   19 Jun  8 17:28 K01teleffect -> ../init.d/teleffect
lrwxr-xr-x 1 root sys   14 Jun  8 17:28 K02midi -> ../init.d/midi
lrwxr-xr-x 1 root sys   13 Jun  8 17:28 K02pcp -> ../init.d/pcp
lrwxr-xr-x 1 root sys   14 Jun  8 17:28 K02pmie -> ../init.d/pmie
lrwxr-xr-x 1 root sys   18 Jun  8 17:28 K02sdpd -> ../init.d/run-sdpd
lrwxr-xr-x 1 root sys   20 Aug 29 21:09 K02wdl_lucid -> /etc/init.d/dl_lucid
lrwxr-xr-x 1 root sys   13 Jun  8 17:28 K02xdm -> ../init.d/xdm
lrwxr-xr-x 1 root sys   18 Jun  8 17:28 K03announce -> ../init.d/announce
lrwxr-xr-x 1 root sys   18 Jun  8 17:28 K05availmon -> ../init.d/availmon
lrwxr-xr-x 1 root sys   22 Aug 29 21:09 K06s+w_stop -> /etc/init.d/stone+wire
lrwxr-xr-x 1 root sys   16 Jun  8 17:28 K09mediad -> ../init.d/mediad
lrwxr-xr-x 1 root sys   19 Jun  8 17:28 K09proxymngr -> ../init.d/proxymngr
lrwxr-xr-x 1 root sys   14 Jun  8 17:28 K15cron -> ../init.d/cron
lrwxr-xr-x 1 root sys   13 Jun  8 17:28 K17nds -> ../init.d/nds
lrwxr-xr-x 1 root sys   14 Jun  8 17:28 K20mail -> ../init.d/mail
lrwxr-xr-x 1 root sys   14 Jun  8 17:28 K21ldap -> ../init.d/ldap
lrwxr-xr-x 1 root sys   13 Jun  8 17:28 K24esp -> ../init.d/esp
-rwxr-xr-x 1 root sys  653 Sep 28 17:23 K24neko_ldap
lrwxr-xr-x 1 root sys   19 Jun  8 17:28 K24neko_ldap.O -> ../init.d/neko_ldap
lrwxr-xr-x 1 root sys   12 Jun  8 17:28 K25lp -> ../init.d/lp
lrwxr-xr-x 1 root sys   18 Jun  8 17:28 K25sysevent -> ../init.d/Sysevent
lrwxr-xr-x 1 root sys   15 Jun  8 17:28 K27audit -> ../init.d/audit
lrwxr-xr-x 1 root sys   25 Jun  8 17:28 K27nss_fasttrack -> /etc/init.d/nss_fasttrack
lrwxr-xr-x 1 root sys   22 Jun  8 17:28 K27sgi_apache -> /etc/init.d/sgi_apache
lrwxr-xr-x 1 root sys   16 Jun  8 17:28 K30flexlm -> ../init.d/flexlm
lrwxr-xr-x 1 root sys   17 Jun  8 17:28 K32fcagent -> ../init.d/fcagent
lrwxr-xr-x 1 root sys   19 Jun  8 17:28 K32sesdaemon -> ../init.d/sesdaemon
lrwxr-xr-x 1 root sys   26 Jun  8 17:28 K34webface_apache -> /etc/init.d/webface_apache
lrwxr-xr-x 1 root sys   17 Jun  8 17:28 K35webface -> ../init.d/webface
lrwxr-xr-x 1 root sys   20 Jun  8 17:28 K38aliases-ip -> ../init.d/aliases-ip
lrwxr-xr-x 1 root sys   17 Jun  8 17:28 K40network -> ../init.d/network
-rwxr-xr-x 1 root sys  657 Jun 10 14:00 K50eclipse
lrwxr-xr-x 1 root sys   15 Jun  8 17:28 K75array -> ../init.d/array
lrwxr-xr-x 1 root sys   20 Jun  8 17:28 K75pppstartup -> ../init.d/pppstartup
lrwxr-xr-x 1 root sys   18 Jun  8 17:28 K80postinst -> ../init.d/postinst
lrwxr-xr-x 1 root sys   12 Jun  8 17:28 K80ts -> ../init.d/ts
lrwxr-xr-x 1 root sys   14 Jun  8 17:28 K81dlif -> ../init.d/dlif
lrwxr-xr-x 1 root sys   17 Jun  8 17:28 K82sysetup -> ../init.d/sysetup
lrwxr-xr-x 1 root sys   14 Jun  8 17:28 K83swap -> ../init.d/swap
lrwxr-xr-x 1 root sys   21 Jun  8 17:28 K84filesystems -> ../init.d/filesystems
lrwxr-xr-x 1 root sys   24 Jun  8 17:26 K90nlskeyserver -> /etc/init.d/nlskeyserver
lrwxr-xr-x 1 root sys   20 Jun  8 17:26 K98jaleolic -> /etc/init.d/jaleolic
lrwxr-xr-x 1 root sys   13 Jun  8 17:28 K98usr -> ../init.d/usr
lrwxr-xr-x 1 root sys   24 Jun  8 17:26 K99Shout_Server -> /etc/init.d/Shout_Server
lrwxr-xr-x 1 root sys   28 Jun  8 17:26 K99Thomson_Licenses -> /etc/init.d/Thomson_Licenses
lrwxr-xr-x 1 root sys   20 Jun  8 17:28 K99disk_patch -> ../init.d/disk_patch

/etc/rc2.d:
total 5
lrwxr-xr-x 1 root  sys    18 Jun  8 17:26 S00announce -> ../init.d/announce
lrwxr-xr-x 1 root  sys    20 Jun  8 17:26 S00disk_patch -> ../init.d/disk_patch
lrwxr-xr-x 1 root  sys    13 Jun  8 17:26 S04usr -> ../init.d/usr
lrwxr-xr-x 1 root  sys    21 Jun  8 17:26 S12filesystems -> ../init.d/filesystems
lrwxr-xr-x 1 root  sys    18 Jun  8 17:26 S16postinst -> ../init.d/postinst
lrwxr-xr-x 1 root  sys    17 Jun  8 17:26 S20sysetup -> ../init.d/sysetup
lrwxr-xr-x 1 root  sys    14 Jun  8 17:26 S21perf -> ../init.d/perf
lrwxr-xr-x 1 root  sys    20 Jun  8 17:26 S23autoconfig -> ../init.d/autoconfig
lrwxr-xr-x 1 root  sys    17 Jun  8 17:26 S30network -> ../init.d/network
lrwxr-xr-x 1 root  sys    22 Jun  8 17:26 S31proclaim -> ../init.d/run-proclaim
lrwxr-xr-x 1 root  sys    20 Jun  8 17:26 S32aliases-ip -> ../init.d/aliases-ip
lrwxr-xr-x 1 root  sys    14 Jun  8 17:26 S34snmp -> ../init.d/snmp
lrwxr-xr-x 1 root  sys    26 Jun  8 17:26 S34webface_apache -> /etc/init.d/webface_apache
lrwxr-xr-x 1 root  sys    15 Jun  8 17:26 S35audit -> ../init.d/audit
lrwxr-xr-x 1 root  sys    17 Jun  8 17:26 S35webface -> ../init.d/webface
lrwxr-xr-x 1 root  sys    14 Jun  8 17:26 S41ldap -> ../init.d/ldap
lrwxr-xr-x 1 root  sys    16 Jun  8 17:26 S45flexlm -> ../init.d/flexlm
lrwxr-xr-x 1 root  sys    18 Jun  8 17:26 S48savecore -> ../init.d/savecore
lrwxr-xr-x 1 root  sys    14 Jun  8 17:26 S49swap -> ../init.d/swap
lrwxr-xr-x 1 root  sys    14 Jun  8 17:26 S50mail -> ../init.d/mail
lrwxr-xr-x 1 root  sys    21 Jun  8 17:26 S51aw_flexlm -> /etc/init.d/aw_flexlm
lrwxr-xr-x 1 root  sys    20 Jun  8 17:26 S58rmtmpfiles -> ../init.d/rmtmpfiles
lrwxr-xr-x 1 root  sys    12 Jun  8 17:26 S60lp -> ../init.d/lp
lrwxr-xr-x 1 root  sys    15 Jun  8 17:26 S62array -> ../init.d/array
lrwxr-xr-x 1 root  sys    25 Jun  8 17:26 S63nss_fasttrack -> /etc/init.d/nss_fasttrack
lrwxr-xr-x 1 root  sys    22 Jun  8 17:26 S63sgi_apache -> /etc/init.d/sgi_apache
lrwxr-xr-x 1 root  sys    14 Jun  8 17:26 S75cron -> ../init.d/cron
lrwxr-xr-x 1 root  sys    20 Jun  8 17:26 S75pppstartup -> ../init.d/pppstartup
lrwxr-xr-x 1 root  sys    13 Jun  8 17:26 S76esp -> ../init.d/esp
lrwxr-xr-x 1 root  sys    18 Jun  8 17:26 S77sysevent -> ../init.d/Sysevent
lrwxr-xr-x 1 root  sys    13 Jun  8 17:26 S83nds -> ../init.d/nds
lrwxr-xr-x 1 root  sys    12 Jun  8 17:26 S85ts -> ../init.d/ts
lrwxr-xr-x 1 root  sys    19 Jun  8 17:26 S88configmsg -> ../init.d/configmsg
lrwxr-xr-x 1 root  sys    16 Jun  8 17:26 S90chkdev -> ../init.d/chkdev
lrwxr-xr-x 1 root  sys    24 Jun  8 17:26 S90nlskeyserver -> /etc/init.d/nlskeyserver
lrwxr-xr-x 1 root  sys    18 Jun  8 17:26 S90roboinst -> ../init.d/roboinst
lrwxr-xr-x 1 root  sys    16 Jun  8 17:26 S91ts -> ../init.d/ts_old
lrwxr-xr-x 1 root  sys    17 Jun  8 17:26 S92fcagent -> ../init.d/fcagent
lrwxr-xr-x 1 root  sys    19 Jun  8 17:26 S92sesdaemon -> ../init.d/sesdaemon
lrwxr-xr-x 1 root  sys    18 Jun  8 17:26 S95availmon -> ../init.d/availmon
lrwxr-xr-x 1 root  sys    20 Jun  8 17:26 S96fontserver -> ../init.d/fontserver
lrwxr-xr-x 1 root  sys    16 Jun  8 17:26 S97mediad -> ../init.d/mediad
lrwxr-xr-x 1 root  sys    19 Jun  8 17:26 S97proxymngr -> ../init.d/proxymngr
lrwxr-xr-x 1 root  sys    20 Jun  8 17:25 S98jaleolic -> /etc/init.d/jaleolic
lrwxr-xr-x 1 root  sys    14 Jun  8 17:26 S98midi -> ../init.d/midi
lrwxr-xr-x 1 root  sys    13 Jun  8 17:26 S98pcp -> ../init.d/pcp
lrwxr-xr-x 1 root  sys    14 Jun  8 17:26 S98pmie -> ../init.d/pmie
lrwxr-xr-x 1 root  sys    16 Jun  8 17:26 S98rtmond -> ../init.d/rtmond
lrwxr-xr-x 1 root  sys    18 Jun  8 17:26 S98sdpd -> ../init.d/run-sdpd
lrwxr-xr-x 1 root  sys    13 Jun  8 17:26 S98xdm -> ../init.d/xdm
lrwxr-xr-x 1 root  sys    24 Jun  8 17:26 S99Shout_Server -> /etc/init.d/Shout_Server
lrwxr-xr-x 1 root  sys    28 Jun  8 17:26 S99Thomson_Licenses -> /etc/init.d/Thomson_Licenses
lrwxr-xr-x 1 root  sys    17 Jun  8 17:26 S99atalk -> /etc/init.d/atalk
lrwxr-xr-x 1 root  user   15 Jun  8 17:26 S99avidd -> ../init.d/avidd
lrwxr-xr-x 1 root  user   14 Jun  8 17:26 S99errs -> ../init.d/errs
lrwxr-xr-x 1 root  sys    22 Jun  8 17:26 S99netwr_client -> ../init.d/netwr_client
lrwxr-xr-x 1 root  sys    21 Jun  8 17:26 S99qpage -> ../init.d/qpageserver
lrwxr-xr-x 1 root  sys    19 Jun  8 17:26 S99teleffect -> ../init.d/teleffect
lrwxr-xr-x 1 root  sys    14 Aug 30 08:43 _S47qx -> /etc/init.d/qx
-rwxr-xr-x 1 root  sys   657 Jun 10 14:00 _S50eclipse
-rwxr-xr-x 1 root  sys   653 Sep 28 17:23 _S60neko_ldap
lrwxr-xr-x 1 root  sys    20 Aug 31 21:42 _S89sesinetd -> /etc/init.d/sesinetd
-rw-r--r-- 1 root  sys   130 Aug 31 00:58 _S98wp
lrwxr-xr-x 1 root  sys    20 Aug 29 21:09 _S98zdl_lucid -> /etc/init.d/dl_lucid
-rwxr-xr-x 1 root  sys  1443 Aug 17 22:45 _S99ad_elmd
lrwxr-xr-x 1 root  sys    23 Aug 30 01:24 _S99bg_mod_link -> /etc/init.d/bg_mod_link
lrwxr-xr-x 1 root  sys    22 Aug 29 21:09 _S99stone+wire -> /etc/init.d/stone+wire
lrwxr-xr-x 1 60000 user   23 Sep  2 02:01 _maple_license -> ../init.d/maple_license

/etc/rc3.d:
total 0
-bash-4.3# 

any ideas?

[mamorim01]

No sensible ideas, sorry. It seems to be idling out (0%) when the decompression freezes, my gut feeling is that it could be memory or controller related, although an issue with the disk itself would have been my first thought as well. 

An entirely wild guess since you seem to have disk_patch in your rc.d folder.

http://nixdoc.net/man-pages/irix/man1/diskpatch.1.html

Ignore this, disk_patch seems to be scheduled by cron and should not be active otherwise.

Quite a few software license servers in your rc.d folder, as well. These give me the willies.

[gijoe77]

I cloned my disk before I started installing everything on the FTP (sometimes I'm not an idiot I guess...), I put the older-cloned disk into the o2 and it was running normal, here is a video of how the o2 should be running:

https://youtu.be/zUj8CbuPacw

From what I can tell I have ruled out a bad hdd (since I cloned my SSD to a brand new 15k SCSI disk, the problem was also seen on that SCSI disk). I don't know if I can rule out some sort of OS corruption, but I think I would be seeing some serious issues if it were an OS issue.

What I really want to do is go item by item/line by line of the processes or services that get started as the OS loads and try to find the offending process. I was under the impression that would be the /etc/rc2.d/S* scripts, is there some other place that gets loaded at startup that a program might have installed some startup script?

[mamorim01]

What I really want to do is go item by item/line by line of the processes or services that get started as the OS loads and try to find the offending process.  I was under the impression that would be the /etc/rc2.d/S* scripts, is there some other place that gets loaded at startup that a program might have installed some startup script?

If you suspect a software process then that is a good place to start.

Scripts are stored in the /etc/init.d folder with symbolic links to start (sXXfile) and stop/kill (kXXfile) placed at the relevant runlevel folders (typically rc0.d at shutdown and rc3.d or rc2.d at startup for multiuser with or without networking). As you know, you can later on run top or ps to confim that the relevant startup daemon has not been started and to get a picture of which process are still active as you go through the list. If you take batches of startup script links out the rc3.d and rc.2 folders you will eventually narrow it down to the offending one. Comparing the contents of these folders with the contents of your original drive to determine which new scripts have been installed since can further help narrow down the problem.

I guess you could also try to start in single user mode (run level 1), I do not believe that startup daemons are run under that mode. The hard drive would be working just fine in that mode if your theory is correct.

[gijoe77]

so I basically disabled all the startup scripts in /etc/rc2.d (except the ones that at least need to be started to boot, like S30network!! ) and still it runs very slow.

I ran the system in single user mode, in system administrator mode (init 1), and it was still working slow.

I mounted the disk externally on a different system and ran "xfs_repair" on it, still nothing.

I made sure it was actually the gtar in the nekoware dir and ldd'd it to make sure it wasn't using anything I've been compiling and playing with in my /usr/local path, everything looks correct.

I'm at a loss, I think I'm just gonna have to restore from backup, reinstall everything again like I was doing all summer but do more testing after I install each package. It's a bit of a bummer, but oh well I'll take this inconvenience over unrecoverable data loss anytime

[mamorim01]

That is quite interesting. I too am at a loss, starting the system in administrator mode should work just fine if a rogue non-system process was originally at fault. Since the problem has replicated on the cloned disk, could it possibly be due to a damaged system file or swap file maybe?

I do agree that a complete restore could be a feasible workaround.

[Jacques]

I noticed on the 'slow' version you're running bash and on the 'normal' one xwsh, a coincidence?

[dexter1]

Those diskperf number are absolutely fine.

An IDE diagnostic run may shake something loose. These should run within 50 minutes.

If nothing comes out of that i would bite the bullet and do a normal irix 6.5 install on a clean disk. The only way to be sure. Do keep your old disk or back it up, it looks like there is a lot of unusual software on it.

[gijoe77]

  I'll take this inconvenience over unrecoverable data loss anytime

I had to say that....  ^^^^^   but anyway....

I backed up my SSD to a brand new 15K 300GB SCSI disk, tested it, it worked fine.

I started cloning an old backup disk to the SSD, but then I thought of something I wanted to try, so I canceled that (it had already destroyed all the original SSD data).  I wanted to restore my SSD drive back to the original state, so  I put the SSD-cloned 15k SCSI disk into the o2 as the master and I was going to clone it back to the SSD.  

wouldnt you believe it, something went wrong with the SSD-backup - xfs errors everywhere, system wouldn't even boot, it would just start dumping memory to /var/adm/crash and everything got all messed up.  I ran xfs_repair on it, and it did a ton of stuff.  I'm not sure if its bootable or not yet, and there was a *LOT* of stuff in the lost+found directory.  Oh well, at least its not all gone, I have most of the things I need to have backed up in other spots and my older cloned drives will be of help.  I'm probably going to have to spend a week or two trying to rebuild the system from all the salvageable bits.  bummer.  I was going to do n+1 backups of the SSD, but didnt think it was needed.  Well I guess that was a silly mistake.

I did make some interesting observations.  the SSD blows away the results of diskperf, but in real applications it seems to not be really blowing away spinning disks on an o2.  so I used the with the same compressed archive for my testing on the SSD and a 10K and 15k (time gtar zxvf <filename>) :       http://www.mediafire.com/file/1izsdjfsq3...0.tgz/file 

I also used a totally different o2 (200mhz R5k), and the tests were sort of surprising.  for this particular test the SSD wasn't necessarily faster than the disks .  I'm currently copying whats left of the SSD filesystem and am hoping I could salvage all my custom scripts and all the notes I had on porting software, should take 20 hours lol   Once I get everything back in order I'll have to see where I was keeping notes on the completion times for SSD vs 15k vs 10k and I'll post a follow up and my thoughts maybe tomorrow

[Trippynet]

To be honest, I'm not too surprised overall. The limiting factor on the O2 is the SCSI speed, which is 40MB/s. Most later HDDs can easily saturate this, so it's only the seek time that you improve on. Hence if you do something with lots and lots of very small files, I would expect a performance improvement (albeit not a massive one), but for a lot of normal tasks, I wouldn't expect much improvement overall. It's mainly the lack of noise where the SSD will win against most HDDs.