Hey people, sorry for not updating for so long. I didn't die, it just turned out that C++ support ended up being far more complicated than I initially intended it to be (mostly because of my own stupidity), and also uni started back up so I haven't had as much time each to work on this as I would have liked. This post isn't going to be quite as comprehensive about things I did as my previous posts have been. A lot has happened in the past few weeks,
including this project getting featured on the BSD Now podcast 
, me getting into contact with Eschaton (the first person to attempt a port of LLVM for IRIX), and best of all, a new contributor joining me! Vladimir Vukićević is probably the first person I've talked to more than once who has a Wikipedia article written about them, and he's been absolutely invaluable. He's taken the initiative on reexamining and fixing assumptions and hacks I've made both now and in the past, as well as extending LLVM's support for IRIX through debugging bugs caused by clang/LLD interactions with rld. He's the first to actually get clang and LLD to run on IRIX, and I'm genuinely so grateful for all of his help. A small community is beginning to form around this project, and I'm happy knowing that even if I were to quit for some reason (I'm not planning on it), the project would continue progressing without my involvement.
Starting from the beginning, after the last update, I took a break for a few days from LLVM and worked on other stuff, so when I came back to LLVM, I immediately dived headfirst into C++ support. I picked up where I left off before the shared libraries bug with getting libc++ and libc++abi up and running. I figured out what was causing that bizarre cmake failure from last time, it turned out that cmake was missing an IRIX platform file, as well as IRIX definitions inside the libc++ CMakeLists.txt. I added the platform file from the SGUG cmake port, and I added some IRIX flags to the CMakeLists.txt.
I was onto the build process, which at this point mainly consisted of me fighting with both the libc++ headers, and GCC's include-fixed headers. The main sources of frustration were anything involving va_list's and wchar_t. clang and GCC both define wchar_t themselves, however the IRIX headers have a typedef-d wchar_t for MIPSPro, causing a type redefinition error on both clang and GCC. The IRIX headers define va_list as a char*, which GCC doesn't expect, and so the GCC include-fixed headers provide alternate functions that use its own builtin __gnuc_va_list type. clang doesn't know about this however and so it tries to use the normal IRIX va_list, inevitably causing an error because it can't find functions using the IRIX va_list. There wasn't really any easy way to patch this within the libc++ support files and so I ended up simply hacking away at the include-fixed headers so that the IRIX va_list functions would be available when building with clang.
There were some other bugs in libc++ I had to fix through defining preprocessor macros and turning off certain functionality, unfortunately my memory's pretty fuzzy on them since it's been nearly 3 weeks since I was working on them. I also extended libxg to provide strerror_r. Around this time I built libc++abi and got a whole lot of errors related to relocations on read only segments
I disabled the errors by passing -Wl,-z,notext and they went away, however unfortunately this would still end up coming into play later on.
I got libc++ and libc++ abi to both build, and I linked a C++ hello world, however rld surprised me with this:
iswblank was a relatively easy fix, it turns out that IRIX exports iswblank in their C library as _iswblank, so I simply defined an iswblank macro for libc++ that pointed at _iswblank. Those other 2 errors were more involved. I didn't know what they were so I demangled the function names, and realized they were the aligned operator new, and aligned operator delete. I had to create a posix_memalign in libxg for libc++ so that I could use libc++'s aligned memory allocation function, and thus enable the missing operator overloads that were causing this error.
Finally I had resolved all the missing functions, and I went to run my hello world program, only to hit this
I ran the program in GDB (without any debugging symbols) and realized it was segfaulting when it was calling a constructor. I peppered some printf statements around and confirmed that it was crashing right here
Now here's the rather dumb part. I ended up messing around for a little over a week in my off hours trying to get the LLD output to more closely match Binutils LD in terms of the ELF segment layout, I assumed this because, for one thing, I was still quite leery of my previous hack I had done to get LLD to output an executable text segment at the DT_MIPS_BASE_ADDRESS, it resulted in a very suspicious looking "ghost" executable segment, that while it technically worked, was definitely one of those things I cannot explain. See for yourself:
On top of this ghost segment issue, I was also leery of the relocation errors I was getting prior. Binutils LD only has two PT_LOAD segments, one for executable code and read only data, and one read/write segment for everything else. The eh_frame section is inside that read/write segment, and so I assumed that was potentially causing a bug in rld, causing it to improperly relocate runtime data.
I'm not gonna bore you guys with the details, because, well, it was really boring. I managed to get the LLD output to more or less mirror the Binutils LD output, however it still didn't work. It was still crashing in the exact same place. I paused on the project for a few days since I was starting to feel quite drained.
I came back though, refreshed and ready to get to work, and so I decided this time around, screw libc++, let's see what happens if we try and use GNU libstdc++. I created 4 test binaries using a combo of g++, clang++, Binutils LD, and LLD. Out of the 4, only a combination of g++ and Binutils LD actually produced a working binary that didn't just immediately segfault. (I later on got a test binary from MIPSPro however it ended up being irrelevant). I decided to step through all of the nonworking binaries in GDB, and that's when I finally noticed something that I'd completely missed the first time around, the programs were not only all segfaulting at the same place, but they were segfaulting when they were trying to run operations using member variables of ostream objects.
This didn't make any sense though, the objects themselves were all allocated in memory, why were they all full of invalid member variables? I then noticed that, wait a second, the objects aren't on the stack like a normal variable, they're all off in memory occupied by the shared library, this means they're all global variables.
How is a global variable initialized? Good question, in C, the compiler is responsible for assigning values to global variables, at compile time, it allocates space in the .data section for the variable, and writes in the constant value to it, and the linker will simply add this data section into the final binary. C++ is somewhat different though, the compiler will perform the same operation for any constant initializers, such as say, int x = 5; For non constant initializers though, such as int* x = new int[5]; the variable will be initialized at program startup, before the main function is even called. The specific way the runtime initializes variables can differ between compilers though, for most modern platforms, GCC's runtime will look through the entries in the .init_array section and call each global constructor placed inside to initialize any global variables. Other operating systems such as IRIX differ though, there, GCC's runtime will instead look backwards through the .ctors section in order to call any global constructors. Regardless of the method however, the global variables should always be initialized by the global constructors before the main function is even entered so that they're immediately available for use with valid contents.
I didn't make the connection before, but std::cout is one of those global objects, and just like other global objects, it needs to be initialized by the runtime before it can be used, explaining why it was causing a segfault everytime I tried using it without it being initialized. Just to double check that my theory about the global constructors not being executed was correct though, I decided to make a test program that simply initialized a global object of a custom class, then called member functions on it to read its contents. As I suspected, none of the values that were returned had been initialized
This should have returned 5 and Hello! for X and Y respectively.
Around this time was when Vladimir Vukićević joined me, and he made rapid progress on cleaning up the various fixes and hacks I'd applied, making it easier for newcomers to get started on setting up an IRIX clang/LLD cross compiler, natively hosting clang/LLD on IRIX, and debugging/fixing his own issues he ran into while cross compiling software with clang/LLD.
It was pretty easy to get a working binary out of clang and Binutils LD, I passed -fno-use-init-array to clang so that it would revert to the older .ctors method of generating a global constructors list, then passed the object file through to Binutils LD, resulting in a working hello world. clang with LLD proved to be more tricky though. Even with the command line flag, it still wasn't calling any of the global constructors at all, even though they were now being stored in the .ctors section instead of the .init_array. I dumped a list of the symbols present in the clang/Binutils LD binary, and a list of symbols present in the clang/LLD binary, and I realized that clang/LLD was missing
a lot of the symbols necessary to actually call all the global constructors.
This was pretty baffling to me, at first I assumed that maybe these symbols just weren't being read in from the crt objects GCC provides, however I was able to confirm that LLD was seeing them by placing a printf at the function where LLD loads in object file symbols, and it directly printed out the list of symbols that were missing from the final binary. I then assumed that maybe LLD was somehow deciding they were unnecessary and was discarding them? This didn't really make a whole lot of sense though so I opened up the crtn.o file that contained these symbols and realized something right away
Do you remember this screenshot?
It was from one of my earlier posts, it turns out that the local symbols, which were all related to global object initialization, were mixed in with the global symbols. I figured that the way LLD was preserving symbols for the final binary was only including symbols from the position before the global and weak symbols appeared. I searched through the LLD code and I found my suspicions confirmed
Continued in part 2 since I've hit the file limit for this post
NEED -> 
. As always, the code is up on 
, me getting into contact with Eschaton (the first person to attempt a port of LLVM for IRIX), and best of all, a new contributor joining me! Vladimir Vukićević is probably the first person I've talked to more than once who has a Wikipedia article written about them, and he's been absolutely invaluable. He's taken the initiative on reexamining and fixing assumptions and hacks I've made both now and in the past, as well as extending LLVM's support for IRIX through debugging bugs caused by clang/LLD interactions with rld. He's the first to actually get clang and LLD to run on IRIX, and I'm genuinely so grateful for all of his help. A small community is beginning to form around this project, and I'm happy knowing that even if I were to quit for some reason (I'm not planning on it), the project would continue progressing without my involvement.
