[Raion]

As part of my attempt, in good faith to be more open as to our infrastructure, and to not be defensive when people try to offer advice, I'm going to be using this thread as a "lecture" of sorts on the network structure of IRIX Network, answer some common questions and explain why things are the way they are. 

Let me introduce you to the three servers of IRIX Network:

Kagura - an iocage node housing "renka" which manages the forums. 

Murakumo - an iocage node housing the ftp/rsync/http mirror usftp/shiki (currently having issues but will be back soon!)

Naraku - a massive six-disk sandy bridge server that handles all disk-intensive parts of the site as well as static html.


All of these currently utilize the following tech:

FreeBSD 12.1-RELEASE
NGINX
MariaDB
PostgreSQL
PHP 7.3

You'll notice there's not redundancy, but there is fault tolerance. Parts of the site can continue to function if one or more servers go down. This is by design as we can't currently afford direct fault tolerance.

This is also supplemented primarily by Yomi and Zelan, two of my home backup servers running FreeBSD and NetBSD respectively. This will be joined by a Solaris server sooner or later. 

Allow me to answer now some FAQs:

1. Why do you insist on using FreeBSD?

For 90% of IRIX Network's run, other than help from praetor, and previously unxmaal, I have been 100% responsible for keeping things up to date. The other staff mostly are bureacratic and administrative checks, and I carry out whatever we collectively decide on (I have as much say as the others, I'm not petty enough to threaten to get my way). 

I am most comfortable using FreeBSD. I hate Linux pretty heavily, and I'm probably a bit infamous for my dislike of it and GNU. 

2. Why do you use iocage?

Security and isolation. It allows the actual "server" that houses actual user data and files to be isolated, not have any isolation, etc. 

3. Why not use AWS or Linode?

Both don't exactly have great FreeBSD support, and the costs are not any better. Also, barring some minor downtime, we've had a far better track record than nekochan.net, which was hosted on a speakeasy dsl (In other words, server in Peter's house)

4. Why not Apache?

Because it's not what I'm comfortable with. Apache is fine if you have a low traffic site.

5. But your hosting costs are over $100/mo, you could save money on Linux!

Most shared Linux hosting can't handle the amount of traffic or disk usage we have. I worked for three major webhosting companies: Bluehost, A2 and InMotion. All of them have disk space restrictions and are far more afraid of DMCA than the companies I host with. They also use CloudLinux, which strangles any site that uses too much I/O, RAM, CPU etc. In short, we wouldn't save any money there.

I hear you talking OpenVZ. OpenVZ has a shitty track record of reliability, and all servers running under it have privacy issues, and over subscribe all resources, often leading to situations out of control. Murakumo and Kagura are under KVM, which would cost the same to host Linux. The reason the costs are so high is due to disk space, primarily. 

6. Why does irixnet.org emails have //forums.irixnet.org and //wiki.irixnet.org

Unlike other vintage forums at large who cater to a mixed audience (not even SGI specifically), I think it is wholly onerous and unnecessary to force our users to use SSL. To hack around MyBB and Dokuwiki's limitations, I use // to indicate https and http agnosticism. If you access the forums over https, it'll have a green lock. 

IF I set it to http, it breaks https

IF I set it https, we can't use http.

Got it? Good. Yes, I could use load balancers and stuff, but it's very rare I actually get serious whiners, and I'm fine with a simple solution. 

7. Why did you pick MyBB?

License (LGPL), performance, security, and compatibility. It works great on 7.3 and has regular updates. Xenforo is proprietary, and both Dodoid and I take exception to that. PHPBB is trash, and didn't like our little // hack. SMF? Your funeral. 

8. Why are/were you so defensive when people try to help?

Short answer: People aren't always seeing the full picture, only the symptoms. It's easy to make assumptions as to the actual solution, but assumptions often miss a lot of things. 

Long answer: I'm inexperienced in running a forum of this size. A lot of stress and pressure is on you, and a lot like an anxious driver, sometimes the noise can cause you to blow up. Ever seen Dumb and Dumber? Remember when Lloyd screamed in Mental's ear?


That was me, and sometimes threads got deleted, and people got lectured in DMs or what-not. I can't change that. I do try to be more understanding now, but I mean a lot of this happened months or even over a year ago. I won't beg, but cut me a bit of slack if you really are still upset about that. 

9. Email issues, what was the story there?

I sometimes get this, not as much as I used to. Since about 2012-ish, Google and several other email providers who provide free email addresses have been cutting down on spam as much as possible. They have a variety of ways to accomplish this:

1. Greylisting, which is used by AOL and Yahoo
2. Silent filtering, which Google never says it does, but yet it does in fact do regularly.
3. Spamboxing, all of them do this
4. Bouncing the message with a 55X SMTP error. 

The transient embargo of G-Mail that happened here was because of no. 2. Look, I get it. Google is a popular email provider. But they're absolutely pure evil. They infringe on software freedoms, dominate the net like MS did in the 90s and early 2000s, and they censor, interfere in elections and also are working with all of the big media and social media companies to quash small operations like ours via censorship. This isn't a conspiracy. 

I embargoed it because our email provider, Zoho, wasn't able to solve it. Unlike the 90s and early 2000s, you cannot simply setup php to send mail(); and expect it to work reliably anymore. It simply doesn't. mail(); uses /bin/sendmail or postfix or exim in an unauthenticated fashion. There's no proof you are who you say you are. It's a lot like driving a car with a fake registration. That's why Google doesn't accept it. 

Google had blocked our domain and silently been trashing notifications. This is not because of spamming, or the .cc we were using or whatever. It's because Google doesn't like small forums like ours. 

I finally bit the bullet and pay $15USD to sendgrid a month to handle our email, and they can actually get through to Google if there's a problem. It's a pain, and it's not cheap, but it works. 

C'est la vie with that. 


If there's any further questions or comments, do let me know. I do keep this site running with **extreme** care. And thankfully users like larbob donate resources, alongside others who have helped on occasion.

EDIT: Clarification on who was the subject of "competitor" since the language used was incredibly vague.

[Raion]

Next, I'm going to talk about three things:

Risk Management
Backups
Exploit Mitigation

As all of these are important with a site and infrastructure this complex.

Risk Management

A significant number of attacks occur in three specific actions on a website:

Uploading files, specifically malicious ones
User-input fields, which include posts, logins etc
Query strings in a URL

In order to mitigate this, compromises must be had. And for exploits, some file formats are safer than others. Among other reasons, this is why zips and pdfs aren't acceptable uploads throughout much of the infrastructure. ZIP has known vulnerabilities, and this is in part because of its combination archiving and compression system plus its age and ubiquity in the face of superior open formats, like 7z, tar+xz etc. PDFs historically, especially on Windows, could execute privileged code inside of them.

I'm not worried about IRIX systems, but modern user systems and the server. Those are the two things that should be very carefully protected. PDFs, SCR and ZIP files are some of the most widespread virus vectors that are relevant for us. And because of that, it's important to understand why they are banned. Yes, there's antivirus scanning solutions but they are:


A) Hard to integrate into existing apps
B) Don't catch everything. Antivirus relies on known definitions, and all it takes is a few bytes in a file to be slightly different to elude them. 

For user-input fields, a lot of this can be mitigated by: 

1. Blocking certain PHP functions that are nonessential, such as mail(), exec(), system() etc. These can be exploited in untrusted strings. 
2. Having your apps know how to sanitize user inputs. 
3. Blocking exec, symlink and setuid on the filesystem housing the webserver. 

I do all of these to help isolate the filesystem, so even if something gets through, it'll get stuck even before it has a chance to execute. Though, if we used perl, CGI or some other languages, this would not be totally possible as those rely on UNIX exec bits. 

Query strings in a URL are mostly evaded using NGINX's ^rewrite regex rules to hide the query strings from the user. 


Backups

Because everything uses FreeBSD/zfs, I utilize ZFS snapshots to save the state of the server and transfer it. This is combined with iocage images, and regular database downloads remotely to ensure that we have a good healthy set of fallback states. In AWS and other services, most of the time FreeBSD uses UFS, and snapshot capabilities are far less advanced and wouldn't work for our use case. There's proprietary ways of managing that, and I'm sure Linux has its own facilities, but I ain't interested lol. 

Exploit Mitigation

As I've previously described, keeping as much of the system read-only as possible helps, as well as turning off exec, setuid, symlinks as well as utilizing features of PHP like open_basedir, zfs dataset isolation, and iocage, helps mitigate the issue. Most of the servers you can't even access the sites inside without getting root access on the host node (Very hard) and then knowing how to console into the iocage. 

Overall, I try to keep things organized and well sorted.