IRIX Network Forums
06/02 - 03 Downtime - Printable Version

+- IRIX Network Forums (//forums.irixnet.org)
+-- Forum: Site-Related (//forums.irixnet.org/forum-39.html)
+--- Forum: Site Discussion (//forums.irixnet.org/forum-29.html)
+--- Thread: 06/02 - 03 Downtime (/thread-1264.html)

Pages: 1 2

06/02 - 03 Downtime - Raion - 06-03-2019

Hey this is just a short statement regarding the down time we had today and yesterday evening.

I made the horrible mistake of having a bad web host for the forum and wiki. 

I will be moving this off shortly after which I will release full details of all of the issues that I underwent with them.  For safety reasons until I have every infrastructure move to another provider I am going to refrain from giving out a lot of details.

Please keep watching the thread at the moment.

As far as I can tell there doesn't appear to be any data loss and I'm hoping to take tomorrow and get us moved off because I'm not going to risk this again. 

Thank you guys for your patience and understanding

RE: 06/02 - 03 Downtime - Raion - 06-04-2019

Now that backups are taken and downloaded against host retaliation, allow me to talk at length about the horror that is: TechVPS (https://techvps.com)

I signed up with them after I did a one-month trial a few months ago. The performance was more than good, pricing phenomenal, and they answered basic sales and support questions within 4 hours.

But then this happens. 

It was about 9PM EST on June 2nd when I noticed the forum was down (I was in the middle of a lengthy post I was trying to recover). This resulted in a panic as I couldn't SSH or ping the VPS node, which led me to believe a node crash. I opened a ticket within 15 mins as I was also locked out of SolusVM (a VPS control panel solution.) It is copied below.

I may as well have pissed into a black hole. No response, not to this fucking minute, has anyone gotten back to me. 

I decided to go to sleep, as I normally do on Sunday, around 1AM EST.

I got up at 9. Still no forums, still no correspondence. 

I put in a management request, and even though i work in the hosting industry and hate this, I fired off another response.

This provider uses WHMCS, like most smaller-scale providers do. The ticket doesn't get responded to, it will stay in "Open" status. The status for it to have been answered, is usually "Answered" or "Awaiting Response" (This is usually configurable within WHMCS or your ticket software.)

So this means nobody has answered the ticket, which tells you nothing about this.

Well, I headed off to Annapolis to deliver MassiveRobot his PI 4D/35. and on the way up, it's back up. I finished the job and got back home at 11PM, where I have begun frantically backing things up (if you noticed a blip slightly before this post came up, that was me backing stuff up.)

Then the moment of truth as I logged into the box:

root@kagura ~ % w
11:14PM  up 78 days, 22:12, 1 user, load averages: 0.59, 0.44, 0.36

This means that the server was up the entire time and it was a network issue. 

This raises several questions for me, as a client:

1. Would it have been too much to ask to fucking email anyone about this? Update your status page? Etc. At my hosting company I work at (which I don't use for this because I don't mix work and personal projects) if an issue lasts beyond 20 mins, we open a status post, which posts to twitter, facebook and our own RSS powered status page. Pretty nifty. 

2. Why did my ticket not get a reply???

3. What are these morons doing?

Looking back, several flags:

I only ever spoke to two people at the company:

Dementi Borodin - an obviously foreign frontline worker.
John - has no surname in the ticket system, who handled some issues for me with setup.

I was also frantic because my backups I had done, mostly relying on ZFS snaps, came back as nonfunctional. I was of course frantic, worrying that we'd lose months of data. 

That's not gonna happen again. Praetor, and several others behind the scenes here, are being recruited to keep this backed up at multiple sources. 

Downtime is inevitable for a host, but TechVPS has handled this about as well as a dog handles chocolate. The worst possible way. 

Fuck TechVPS. Now that I have backups, I could give two shits if they try to turn us off. I'm going to spend the next day or two moving our asses off this joker of a VPS company. Meanwhile, they will be dragged through the mud repeatedly.

I'd like to thank everyone who has contacted me with messages of encouragement and more through this difficult time. You live and you learn, and this place means too much for me to lose it all.


And for TechVPS, if you read this - and you wanna make it right, refund me at least half of my hosting plan. I'll give you 48 hours with your management ticket if three fucking days go by from the time that was filed and I get nothing? You're getting charged back on EVERY cent I've paid. and I'll make sure your reputation is ruined. Fucking cocksuckers.

RE: 06/02 - 03 Downtime - Raion - 06-04-2019

Moved us over successfully to a new provider. I've charged back everything with TechVPS. Knowing paypal, I'll get every cent back.

RE: 06/02 - 03 Downtime - Raion - 06-04-2019

TechVPS resells servers from Colocrossing and according to LowEndTalk this generally is a source of issues.

RE: 06/02 - 03 Downtime - Trippynet - 06-04-2019

Good work! Guess we're all a bit jittery after having the rug pulled under us with Nekochan (which had its fair share of downtime previously it has to be said). Many thanks for all your efforts and for sorting things out. Big thumbs up from me!

RE: 06/02 - 03 Downtime - Raion - 06-04-2019

Gods above you had no idea how stressful it was. Especially since I was concerned about the backup situation.

Over the next couple of weeks I'm going to be aggressively restructuring how I do backups and we're going to have at least three other people with copies of all the site information stored in safe spots.

RE: 06/02 - 03 Downtime - BackPlaner - 06-05-2019

(06-04-2019, 10:55 PM)Raion Wrote:  Gods above you had no idea how stressful it was. Especially since I was concerned about the backup situation.

Over the next couple of weeks I'm going to be aggressively restructuring how I do backups and we're going to have at least three other people with copies of all the site information stored in safe spots.

Thank god you noticed!
Nice job Smile

RE: 06/02 - 03 Downtime - dexter1 - 06-08-2019

Great job Raion! Thanks for taking care of this.

Any word from TechVPS about this already? Or did they decide to stick their heads in the sand?

RE: 06/02 - 03 Downtime - vishnu - 06-08-2019

Just speaking personally for myself, I have a static IP address from Comcast, name resolution from Pair Networks and my webserver is in my house, which works out very nicely in terms of control issues... Smile

RE: 06/02 - 03 Downtime - Raion - 06-08-2019

I live in a rural area, have unreliable internet and power. This is not an acceptable way to host a website that gets 5,000 views a week.

We cannot use home-hosting. Period.