[Raion]

It's been brought to my attention I tend to blog more than write good topics, but I will do better I guess 

September will mark three years I walked away from my daily Linux admin job. There wasn't one single factor, but several. The first was I felt I was lured into the job under false promises. I wanted to go into compliance and resources, a section of the company where we deal with abuse, fraud, DMCAs, hacks, problem customers, etc. Instead they wanted me in admin. 

What e thought my job was gonna be:

Monitoring, handling non automation issues, and working on specialist issues.

What I got:

Babysitting frontliners that were mostly outsourced, having morons escalate problems they could have solved, being sidelined from crises, and getting yelled at because I'm not a generalist and no, I don't like dealing with broken python/node shit. 

The issue is that the industry is ending traditional sysadmin and monitoring jobs. And that's what I prefer. DevOps brings deadlines, agile, needless automation and such and it's a pathetic attempt to merge and squish specialists. It's ick IMHO. I know people who are optimistic, but it's not my thing. I was miserable there, and had to multitask to the point I was making sometimes serious mistakes.

So yeah, I think it's ugly. I'm not a fan. Bad way to run things and a corporate fad that's bad for users and staff alike. 

What do I want this discussion to be? I dunno. I just wish I wasn't alone in this frustration. I now work for an FFL, and it's fine for now, but not forever. I seriously am considering going into IBM i or MVS/JCL.

[Shiunbird]

IBM i, woa.

Look, I recommend a lot of the articles from softpanorama.org. I think you will love it. I'm sad they are closing shop...
The ones about Agile and DevOps are excellent.

And yes, my occupation was mostly something like sysadmin (but for video conferencing infrastructure) and, with Zoom and other SaaS crap, all I do is avoid security fuck ups. Seriously, the vendors pass all ISO certifications, but when it's time to integrate with our systems, they want full root. EVERY. SINGLE. TIME. WTF!

I'm about to leave my decent corporate job for a startup of 8 employees, a paycut and no benefits. Why?

1. Incompetent manager bullied me into anxiety and two years of hell, threatening to accuse me of racism. Same man tried to break into a colleague's room during a conference (to sleep with her) and punched a guy during a Christmas part (we are not from the same office - he is in the US and I am in Europe) and broke havoc in our AWS environment. Someone protects him.

I got a raise and a promotion and "thanks for the hard work". Resolving the security gap cost me 6 months of unpaid overtime, and two year later, still drowning in anxiety.

Ten years ago, we were two engineers and maximum a manager in an implementation call. Now we have 5 managers from each side on top of the two engineers, and things that should take 10 minutes take two hours, things that should take a day take a month, and things that should take a month take 2 years.

And yes - dealing with idiots who have not a effing clue.
Vendor last week, when I was still in sick leave, trying to enable an integration for days. I returned, googled the error message: "feature not supported in trial accounts".
More than once a week, I have to say to others: "Look, I have a degree in Music, I should NOT be the one telling you that."

2. The new place would give me freedom from standup calls, idiocy, and I'd work with the co-founder, who is an excellent coder. I'd learn a lot and part of my job would be making sure that the application runs everywhere from Raspberry Pis to AWS and Azure. His main focus is full portability, so I'd have the chance to get my hands dirty.

For the last 4-5 years, I've been suffering a lot at work. The work that I used to happily do alone now take 6 people who don't have a clue and I MUST work with them. I don't mind having no time because I am busy, but being idle, looping over, making mistakes and not having time to breath drives the anxiety through the roof.

Be happy you get to deal with broken python/node. They forced us to go to Snaplogic to build integrations and the thing is a pain to use and a pain to troubleshoot. I managed to successfully rebel and got my own VM with Powershell (for tasks requiring AD) and bash (yes, you don't like it, but gosh, snaplogic...). My colleagues were not so lucky.

Then you got a bunch of people doing metawork for the sake of justifying their paycheques and, seriously, send me one more employee satisfaction survey or internal newsletter, I dare you!

[Raion]

Python and node weren't fun.

For one, our retarded marketing team called us top tier node hosting. In truth, it was experimental trash. Users had to setup node to run using nohup or pm2 as their user on shared cpanel hosting. It regularly would crash, ant it wouldn't reboot. I told them to write crons to check/restart it. It had to run on a passive ftp port and be redirected by Apache. No logs, no help, and lots of morons being attracted by this crappy fad.

Python was worse. passenger_wsgi was used and it didn't log. No integration. And I hate python,it should die forever.

On top of that we were using PHP wrong for the first 2 years I was there. DSO/modphp, prefork, suexec/ruid2. You may be asking what the hell am I on?

Well, prefork is horrible for memory and concurrency. It is DDOS vulnerable. It's crap. It's also a security issue. SuPHP and lsapi are better, as is fcgi. We moved to lsapi.

Oh, on top of that we had a mixed cfengine/salt setup. It was awful. But I've never found a config management system I liked.

[Shiunbird]

Users had to setup node to run using nohup or pm2 as their user on shared cpanel hosting. 

woa...

I share your pain.
The biggest problem of the industry is the lack of people who actually know what they are doing.
We train specialists of depth, but the work calls for strong generalists, due to all the crazy integrations that happen all the time.

[vishnu]

I loathe cpanel with the red hot fire of a thousand burning suns. I have never, and will never, use a hosting provider that doesn't offer a shell account.

[Raion]

Cpanel is fine, it's better than it's competition, e.g. plesk.

We offered shells, but it's not a root shell, it's a managed provider for the noobs.

I won't drag my former employer, I just think they were and continue to be greedy.

[vishnu]

The shells offered by the hosting providers I've used were always chrooted to $HOME, but were otherwise unencumbered. When you're trying restore a multi-gigabyte mysql database it's a heck of a lot easier to have a shell prompt than cpanel... 😜

[Raion]

Phpmyadmin has an upload limit. But if you have a database over 200M, it's VERY likely you need a dedicated host. On a shared host a 1G database pegged the IO for several minutes, and I had to suspend the dude. Didn't matter that LVE/Cloud Linux (which uses CageFS for isolation) had locked down his IO, between that and everything else we told him VPS or screw off. You can argue we overbooked, but it's also on the other hand not conducive. IRIXNet's forum DB is under 100M. No compression or anything here, just optimizations and no extra triggers or stored procedures needed.

[weblacky]

I'm sort of taken aback by how much of this is relatable.  I guess it's already been templated on how to abuse people in this industry but my story isn't too different from what Raion himself posted.  It was my first real job and I was there for nearly 7 years, most people didn't last year let alone two.  I directly report it to the CEO in a midsize company and it was its own brand of hell.  The only reason I stayed as long as I did was because of the employees. We had a very good hiring process and the majority of our employees were very hard-working, decent, educated people that you could learn a lot from and they were eager to learn. 

Oddly enough the owners and their high standards did allow us to hire good people and it was those people that I stayed for because I was helping them in my capacity.  After all, we were all overworked and underpaid for years. And this was during the deepest portion of the last recession.  I left all that behind in 2013 and really haven't looked back. 

But at the time I had trouble seeing just how abusive it was and just how much was taken from me.  I passed up on monumental opportunities because I was so fixated on my employers issues and problems and solutions as I was being beaten into submission.

There are some things I don't begrudge my old employer for and I actually feel may be the correct way to do things in order to get work out of people but at the same time the reward system wasn't compensatory with the expectations that were set.

If it wasn't for the other employees being sort of a gang of buds I would've left a long time ago but I stuck it out. It wasn't until I had missed so much that I realized how much I needed to just leave.

The company has since downsized and had other issues since my departure which is both indirectly a result of my leaving and a direct result of the poor management that I had to deal with.

All at all in today's world, especially with what's been going on recently, a lot of people won't speak up or won't risk their jobs because the salary is just good enough to live comfortably and keep your mouth closed.  I witnessed quite a few frauds but I never witnessed anything where somebody was harassed or otherwise injured so it was more of just a good old boys club of lies. The client would lie to us and we would lie to the client.  It was almost expected.

But the tidbits being brought up in the previous posts do ring with bells of truth in my own experience.  I'm out of all that now and even though I don't make anywhere near what I did, I'm at least happy.

Creating something of value really isn't easy. Especially in this industry where it really needs to result in a finalized product. Unless it's some sort of formula or technique that can be used in a larger system the system itself is what is valuable.  In a company you normally have multiple individuals with different disciplines working toward that goal. When it's just you even if you try to wear the many hats you sometimes come up short.

I applaud those people who had the guts and jumped out of their situations to try to find something else. Often times it is only in the purview of those of us who aren't already committed to families or relationships and therefore have to be a higher earners.  When you can still take those kinds of chances.

[Shiunbird]

All at all in today's world, especially with what's been going on recently, a lot of people won't speak up or won't risk their jobs because the salary is just good enough to live comfortably and keep your mouth closed.  I witnessed quite a few frauds but I never witnessed anything where somebody was harassed or otherwise injured so it was more of just a good old boys club of lies. The client would lie to us and we would lie to the client.  It was almost expected.

My place of employment, everyone.

I am on the edge of taking the leap from 40.000+ people big mega corp to 8 people startup, benefits and pay cut. Final decision coming in a few days, pending a document that narrates abuses I've suffered years ago. Once I have it, I am most probably gone.