The start of a LONG Fuel repair thread...
#31
RE: The start of a LONG Fuel repair thread...
Well, until I get a working graphics card I'm at a stand-still. And of course I now cannot find any real options to buy one. So far it's been 3 days without word from the dealer (no response to emails) and sources suddenly don't have any posted. I've reached out to B&B today in Canada (I'm close to Canada anyway) and we'll see what I get back.

I really didn't expect this much resistance on getting a Fuel...and I'm still not there yet. I guess we go the slow way...since the fast way apparently dried up.
weblacky
I play an SGI Doctor, on daytime TV.

Trade Count: (10)
Posts: 1,716
Threads: 88
Joined: Jan 2019
Location: Seattle, WA
Find Reply
11-05-2021, 09:47 PM
#32
RE: The start of a LONG Fuel repair thread...
Guys, I'd also like to say that offering sales advice in threads like this is off-topic and not super helpful. If you wanna help, that's great. Take it to his wanted thread or DMs please!

I'm the system admin of this site. Private security technician, licensed locksmith, hack of a c developer and vintage computer enthusiast. 

https://contrib.irixnet.org/raion/ -- contributions and pieces that I'm working on currently. 

https://codeberg.org/SolusRaion -- Code repos I control

Technical problems should be sent my way.
Raion
Chief IRIX Officer

Trade Count: (9)
Posts: 4,240
Threads: 533
Joined: Nov 2017
Location: Eastern Virginia
Website Find Reply
11-06-2021, 02:17 AM
#33
RE: The start of a LONG Fuel repair thread...
Maybe I missed it -- but you've verified everything works fine without the V10 installed, right? You can boot and install irix etc without graphics in a fuel with no problems (I did it recently), run some diags and stress tests.

Also don't be afraid to run without the side panel on so you can get the L1 cable out; it's fine. Just the blue shroud needs to be on.
vvuk
O2

Trade Count: (0)
Posts: 43
Threads: 4
Joined: Aug 2021
Location: California
Find Reply
11-07-2021, 07:49 AM
#34
RE: The start of a LONG Fuel repair thread...
(11-07-2021, 07:49 AM)vvuk Wrote:  Maybe I missed it -- but you've verified everything works fine without the V10 installed, right?  You can boot and install irix etc without graphics in a fuel with no problems (I did it recently), run some diags and stress tests.

Also don't be afraid to run without the side panel on so you can get the L1 cable out; it's fine.  Just the blue shroud needs to be on.

No, I cannot boot my fuel without a graphics card. Without a graphics card the voltage/temp/fan sensor i2c network is broken (incomplete) and you lose contact with bedrock i2c completely. 

It’s possible you can can do that with env off, but with this unit I cannot turn env off and boot (I tried, I think) and it refused without communication with the i2c network.  It may be related to my L1 revision. 

I’ve never heard of someone being able to boot a headless fuel, you’d have to document how you did that because from what I know you must have a graphics card to get to PROM on a fuel, unless someone else can shine some light in that as most nekonomicon pages describe a dead graphics card as preventing booting. 

However, it’s possible they either added or removed such an ability based on L1 firmware.  Mine is shown in my logs, I cannot upgrade it until the system is happy. 

It also alerted me that my card had an active short, without it who knows the damage that could have resulted.  

No, I need my voltage monitoring for going further since it’s been damaged. 

With the shorted-open graphics, I get to a blinking white LED, pre-power up stage…but cannot power up due to unresponsive card (my guess) because I fail pre-power checks once I try.  

Without the card my env system breaks, “env off” does not bypass those checks for me (I tried that).  

The V10 I bought should be shipping Monday or Tuesday timeframe.  That should allow completion of the i2c sensor network and respond to the power-on signal.  

Best I can do, for now.
weblacky
I play an SGI Doctor, on daytime TV.

Trade Count: (10)
Posts: 1,716
Threads: 88
Joined: Jan 2019
Location: Seattle, WA
Find Reply
11-07-2021, 02:45 PM
#35
RE: The start of a LONG Fuel repair thread...
My fuels I owned wouldn't either.

Keep in mind my extraordinary bad luck with fuels and you will begin to understand why I developed a dislike of them lawl.

I'm the system admin of this site. Private security technician, licensed locksmith, hack of a c developer and vintage computer enthusiast. 

https://contrib.irixnet.org/raion/ -- contributions and pieces that I'm working on currently. 

https://codeberg.org/SolusRaion -- Code repos I control

Technical problems should be sent my way.
Raion
Chief IRIX Officer

Trade Count: (9)
Posts: 4,240
Threads: 533
Joined: Nov 2017
Location: Eastern Virginia
Website Find Reply
11-07-2021, 03:30 PM
#36
RE: The start of a LONG Fuel repair thread...
(11-07-2021, 02:45 PM)weblacky Wrote:  No, I cannot boot my fuel without a graphics card. Without a graphics card the voltage/temp/fan sensor i2c network is broken (incomplete) and you lose contact with bedrock i2c completely. 

It’s possible you can can do that with env off, but with this unit I cannot turn env off and boot (I tried, I think) and it refused without communication with the i2c network.  It may be related to my L1 revision. 

I’ve never heard of someone being able to boot a headless fuel, you’d have to document how you did that because from what I know you must have a graphics card to get to PROM on a fuel, unless someone else can shine some light in that as most nekonomicon pages describe a dead graphics card as preventing booting. 

I just did this ~4 weeks ago. I bought a fuel that came without graphics (owner wanted to keep the V12). While waiting for a graphics card to come in, I wanted to set up the ATX power supply, SATA controller/boot drive, and work through some netboot issues.

There was nothing special to do. You do not need a graphics card to get to PROM -- you can access serial PROM output while connected to the L1 port by pressing Control-D, or connect to the first serial port on the back; the system will default to console=d mode when there is no graphics. (IRIX does install without graphics support in this case, e.g. GFX=NONE; you have to do a bit of work to tell inst that you do in fact have a graphics card later on and reinstall a few packages to get working graphics.) A _dead_ graphics card I can see preventing booting, if it's half-dead (i.e. system sees it but it's not working; it'll likely keep trying to shove you into graphics console mode). A completely missing one should be fine.

I did not have to disable env monitoring for any of this. In fact I definitely did not wanted to do so, because I was constantly monitoring the power levels due to the new power supply for the first bit. I had full operational env monitoring the entire time. I was able to fully install IRIX, and I used the machine for a few days doing some headless things over ssh until the video card arrived.

I had zero errors from env or any L1 warnings at any point. I don't have full logs unfortunately (or maybe I do, I'll do some digging).

You actually have a newer L1 version than I do -- yours:

Code:
Firmware Image B: Rev. 1.28.3, Built 03/20/2004 00:01:57

mine (from version command output):

Code:
L1 1.20.6 (Image B), Built 03/12/2003 11:01:29    [1MB image]

(You can also see if there's a different version in the other flash slot, via "flash status v"; if there is, you can boot it by using the flash command, see "help flash". Note I don't know how to reset back to A/B physically in case something fails, so ymmv there.)

Even with a different L1 version, the physical I2C topology would not change. So if you are having problems communicating with bedrock env I2C that resolves itself when a graphics card is inserted, my guess is that there's a break in the i2c lines on the mainboard and the graphics card plugged in ends up providing another route to the bedrock i2c lines causing it to "appear". Since you have a hint that I2C is not working, I would check connectivity between the SDA/SCL pins on all the DS1780s on the mainboard. Every SDA should be connected to every other SDA, same with SCL. (And if you manage to trace them, they should all connect to whatever chip is the i2c leader.)

I would really try to resolve all these issues without graphics first. You run the risk of there being damage on the mainboard that is frying the graphics card in some way, and that any replacement card you get will end up damaged in a similar way.
vvuk
O2

Trade Count: (0)
Posts: 43
Threads: 4
Joined: Aug 2021
Location: California
Find Reply
11-07-2021, 05:00 PM
#37
RE: The start of a LONG Fuel repair thread...
A lot of this stuff is over my head here guys but my question is if there's a potential software cause to why this doesn't work because I had three fuels and mostly working condition over my time working with them and all of them failed spectacularly in different ways. So I had a lot of time to test with and without graphics. Never was I able to get a system to boot headless.

If the answer is no then I suspect that there might be another point of failure on the board that might be common and potentially caused by the defective power supplies. Just a personal theory though?

I'm the system admin of this site. Private security technician, licensed locksmith, hack of a c developer and vintage computer enthusiast. 

https://contrib.irixnet.org/raion/ -- contributions and pieces that I'm working on currently. 

https://codeberg.org/SolusRaion -- Code repos I control

Technical problems should be sent my way.
Raion
Chief IRIX Officer

Trade Count: (9)
Posts: 4,240
Threads: 533
Joined: Nov 2017
Location: Eastern Virginia
Website Find Reply
11-07-2021, 06:46 PM
#38
RE: The start of a LONG Fuel repair thread...
Okay, hoping you were right, I tried booting without V10 and without Kbd/ms...got the same result I documented. The Fuel physically stopped me because it's missing an I2C element (ERROR: I2C:not present) which by all accounts appears to be the V10's DS1780.  You see below those elements are missing once the V10 is removed.

In my previous posts they come back and show up fine when the card is inserted, as I replaced the chip...and the chip is directly feed from the XIO connector (not a VRM on the card).  So my assertion is that the graphics card is the last component on the sensor bus and completes the entire thing.

The way I interpret what I’m seeing in my logs (all posted in this thread) is that the graphics card’s DS1780 is charged with monitoring all the XIO voltages (those are all blank when card is removed in env) and when I say “lose communication with bedrock” I specifically see row 5 on the last env temp table say:

5 BEDROCK          Wait Pwr  Not currently available

this changes to fully filled out when my shorted (now open short) non-starting V10 with replaced DS1780 IC is installed. “Not currently Available” is replaced with temps just like the lines above it. 

This doesn’t produce an alert or claim the env monitoring isn’t running, env claims running and configured.  But it’s missing info, which I interpret as an error. Because that info is present when the card is inserted.


I'm well aware that i2c is a shared bus (not linear, p2p)...but since the V10 is "missed" and yet the ENV monitoring claims it's up and running without a config error I assume my i2c bus lines are fine (how else could I talk to them all but the missing one when the V10 is removed).  I assume you're operating under a bug/feature that SGI closed to prevent people from doing exactly what you're suggesting.  My gamble is that without Bedrock temps the system is in a position to overheat and fail in an undetectable way, so SGI likely closed that loophole and I have an actual error that stops me (yet doesn't stop env).  Because, while I believe what you're saying, I haven't seen anyone else claim being able to boot headless on a Fuel and I can find at least 3+ postings of people saying what I'm seeing...the fuel will not let you start without the monitoring from the graphics card.

I urge you to please post your pre-power up log with env and power printouts, and see what it shows.  If you show exactly what I show (missing XIO voltages and missing bedrock temps)...then likely your old firmware is giving you a loophole feature.  My board has the same firmware for both flash slots...so very likely it was produced with this firmware in place.

While I won't pressure you, I'd bet fair money if you actually upgraded your firmware...you'd be sitting right here with me...unable to start without a graphics card.

Log shown below:

Code:
SGI SN1 L1 Controller

Firmware Image B: Rev. 1.28.3, Built 03/20/2004 00:01:57





001a01-L1>INFO: 001a01 will power up system in  5 seconds...

INFO: 001a01 powering up the system.

eERROR: 001a01 auto power up error.



ERROR: command not found.

001a01-L1>env

Environmental monitoring is enabled and running.



Description    State       Warning Limits     Fault Limits       Current

-------------- ----------  -----------------  -----------------  -------

           12V   Wait Pwr  10%  10.80/ 13.20  20%   9.60/ 14.40    0.19

        12V IO   Wait Pwr  10%  10.80/ 13.20  20%   9.60/ 14.40    0.19

            5V   Wait Pwr  10%   4.50/  5.50  20%   4.00/  6.00    0.26

          3.3V   Wait Pwr  10%   2.97/  3.63  20%   2.64/  3.96    0.89

          2.5V   Wait Pwr  10%   2.25/  2.75  20%   2.00/  3.00    0.00

          1.5V   Wait Pwr  10%   1.35/  1.65  20%   1.20/  1.80    0.00

        5V aux   Wait Pwr  10%   4.50/  5.50  20%   4.00/  6.00    5.04

      3.3V aux   Wait Pwr  10%   2.97/  3.63  20%   2.64/  3.96    3.30

PIMM0 12V bias   Wait Pwr  10%  10.80/ 13.20  20%   9.60/ 14.40    0.19

     Fuel SRAM   Wait Pwr  10%   2.25/  2.75  20%   2.00/  3.00    0.29

      Fuel CPU   Wait Pwr  10%   1.13/  1.38  20%   1.00/  1.50    0.03

    PIMM0 1.5V   Wait Pwr  10%   1.35/  1.65  20%   1.20/  1.80    0.00

PIMM0 3.3V aux   Wait Pwr  10%   2.97/  3.63  20%   2.64/  3.96    3.29

  PIMM0 5V aux   Wait Pwr  10%   4.50/  5.50  20%   4.00/  6.00    5.02

  XIO 12V bias <not present>

        XIO 5V <not present>

      XIO 2.5V <not present>

  XIO 3.3V aux <not present>



Description    State       Warning RPM  Current RPM

-------------- ----------  -----------  -----------

FAN 0  EXHAUST   Wait Pwr          920            0

FAN 1       HD   Wait Pwr         1560            0

FAN 2      PCI   Wait Pwr         1120            0

FAN 3    XIO 1   Wait Pwr         1600            0

FAN 4    XIO 2   Wait Pwr         1600            0

FAN 5       PS   Wait Pwr         1349            0



                              Advisory   Critical   Fault      Current

Description       State       Temp       Temp       Temp       Temp       

----------------- ----------  ---------  ---------  ---------  --------- 

0 NODE 0           Wait Pwr    [Autofan Control]    75C/167F   11C/ 51F

1 NODE 1           Wait Pwr    [Autofan Control]    75C/167F   11C/ 51F

2 NODE 2           Wait Pwr    [Autofan Control]    75C/167F   11C/ 51F

3 PIMM             Wait Pwr    [Autofan Control]    75C/167F   11C/ 51F

4 ODYSSEY        <not present>

5 BEDROCK          Wait Pwr  Not currently available


001a01-L1>reset
Code:
ERROR: power appears off.
Code:
001a01-L1>power up
Code:
ERROR: I2C:not present

001a01-L1>reboot_l1





SGI SN1 L1 Controller

Firmware Image B: Rev. 1.28.3, Built 03/20/2004 00:01:57





001a01-L1>* poINFO: 001a01 will power up system in  5 seconds...

INFO: 001a01 powering up the system.

wERROR: 001a01 auto power up error.

er

001a01:

Supply          State Voltage    Margin  Value

--------------  ----- ---------  ------- -----

           12V    off    0.188V      N/A

        12V IO     NC    0.188V      N/A

            5V     NC    0.260V      N/A

          3.3V     NC    0.894V   normal     0

          2.5V    off    0.000V   normal     0

          1.5V     NC    0.000V   normal     0

        5V aux     NC    5.044V      N/A

      3.3V aux     NC    3.302V      N/A

PIMM0 12V bias     NC    0.188V      N/A

     Fuel SRAM     NC    0.286V   normal     0

      Fuel CPU    off    0.028V   normal   119

    PIMM0 1.5V     NC    0.000V   normal     0

PIMM0 3.3V aux     NC    3.285V      N/A

  PIMM0 5V aux     NC    5.018V      N/A

  XIO 12V bias     <not present>

        XIO 5V     <not present>

      XIO 2.5V     <not present>

  XIO 3.3V aux     <not present>


001a01-L1>reb   reboot_l1





SGI SN1 L1 Controller

Firmware Image B: Rev. 1.28.3, Built 03/20/2004 00:01:57





001a01-L1>INFO: 001a01 will power up system in  5 seconds...

INFO: 001a01 powering up the system.

ERROR: 001a01 auto power up error.



001a01-L1>env check

Environmental monitoring is enabled and running.

001a01-L1>flash status

Flash image B currently booted



Image      Status        Revision    Built

-----   -------------   ----------   -----

  A     valid           1.28.3       03/20/2004 00:01:57

  B     default         1.28.3       03/20/2004 00:01:57

001a01-L1>



My Mainboard is: 030-1707-005 Rev. A


Fingers Crossed on my replacement V10 card!

(11-07-2021, 06:46 PM)Raion Wrote:  A lot of this stuff is over my head here guys but my question is if there's a potential software cause to why this doesn't work because I had three fuels and mostly working condition over my time working with them and all of them failed spectacularly in different ways. So I had a lot of time to test with and without graphics. Never was I able to get a system to boot headless.

If the answer is no then I suspect that there might be another point of failure on the board that might be common and potentially caused by the defective power supplies. Just a personal theory though?

There's a lot in this statement, but I've asserted (and still agree) for a while that bad power from PSUs caused many of these failures (not all of course, I do believe your heat argument).  Since I found the DS1780 on my V10 is directly connected to the mainboard XIO 5V line...yeah that's pretty much the 5V AUX line...right into PSU.  There is current limiting but not conditioning/regulation so it makes total sense that as the PSU turns (and for some reason they TURNED FAST compared to other SGIs) fragile low-voltage devices suffer.

I'm super interested to see inside my PSU (when I get there) to see if it blew something (like a protection diode that prevents versus voltage spikes or something) or if the caps are just WAY out of it.

Since ENV doesn't seem to produce a config error when missing the V10, I assume the later checks where put in place to prevent going forward (without reprogramming ENV) in later firmwares.  I know they did things like close the "Carnage" keyword and change how "Security" works (once "ON", behavior to turn it off was changed, right?).  So I know they made changes all the time, given customer scenarios.
(This post was last modified: 11-07-2021, 08:22 PM by weblacky.)
weblacky
I play an SGI Doctor, on daytime TV.

Trade Count: (10)
Posts: 1,716
Threads: 88
Joined: Jan 2019
Location: Seattle, WA
Find Reply
11-07-2021, 08:11 PM
#39
RE: The start of a LONG Fuel repair thread...
With my Fuel _with_ a V10 installed, when I look at env before booting, I see:
Code:
env
Environmental monitoring is enabled and running.

... (omitted power) ...

Description    State       Warning RPM  Current RPM
-------------- ----------  -----------  -----------
FAN 0  EXHAUST   Wait Pwr          920            0
FAN 1       HD   Wait Pwr         1560            0
FAN 2      PCI   Wait Pwr         1120            0
FAN 3    XIO 1   Wait Pwr         1600            0
FAN 4    XIO 2   Wait Pwr         1600            0
FAN 5       PS   Wait Pwr         1600            0

                           Advisory   Critical   Fault      Current
Description    State       Temp       Temp       Temp       Temp
-------------- ----------  ---------  ---------  ---------  ---------
NODE 0           Wait Pwr   60C/140F   65C/149F   70C/158F   21C/ 69F
NODE 1           Wait Pwr   60C/140F   65C/149F   70C/158F   20C/ 68F
NODE 2           Wait Pwr   60C/140F   65C/149F   70C/158F   21C/ 69F
PIMM             Wait Pwr   60C/140F   65C/149F   70C/158F   21C/ 69F
ODYSSEY          Wait Pwr   60C/140F   65C/149F   70C/158F   21C/ 69F
BEDROCK          Wait Pwr  Not currently available

Note the "Not currently available" on BEDROCK.  So that's normal.  My guess is the monitoring chip for BEDROCK just doesn't have power applied until the system is fully running.

I can pull my V10 temporarily if you'd like, but I'm not sure what it would tell you; the system would boot fine, like it did before.  I guarantee this system ran fine without a graphics board, and I can't image why SGI would want to limit the possibility to do that -- it would be a valuable option in case of graphics card damage when data needs to be recovered or similar.  There's no reason to explicitly forbid it.  I'm hesitant to upgrade my firmware because, well, things are working fine.

However I don't see what in your logs actually prevents things from booting; I see "ERROR: 001a01 auto power up error." but nothing beyond that. I see from an earlier log something like "system failed pre-power check"; is that the current issue? Are you seeing anything from the PROM after you start booting? Issue a "pwr up" and then when the L1 prompt comes back, hit Control-D.
vvuk
O2

Trade Count: (0)
Posts: 43
Threads: 4
Joined: Aug 2021
Location: California
Find Reply
11-07-2021, 08:35 PM
#40
RE: The start of a LONG Fuel repair thread...
Yeah vvuk, I'll tell you from experience, if you've got a good stable ATX PSU in there and no other problems, don't mess with things, please.

I've had too many fuels die by being a curious cat.

I'm the system admin of this site. Private security technician, licensed locksmith, hack of a c developer and vintage computer enthusiast. 

https://contrib.irixnet.org/raion/ -- contributions and pieces that I'm working on currently. 

https://codeberg.org/SolusRaion -- Code repos I control

Technical problems should be sent my way.
Raion
Chief IRIX Officer

Trade Count: (9)
Posts: 4,240
Threads: 533
Joined: Nov 2017
Location: Eastern Virginia
Website Find Reply
11-07-2021, 08:59 PM


Forum Jump:


Users browsing this thread: 1 Guest(s)