LLVM For IRIX - Printable Version
+- IRIX Network Forums (//forums.irixnet.org)
+-- Forum: SGI/MIPS (//forums.irixnet.org/forum-3.html)
+--- Forum: Development/Porting (//forums.irixnet.org/forum-9.html)
+--- Thread: LLVM For IRIX (/thread-3043.html)
+- IRIX Network Forums (//forums.irixnet.org)
+-- Forum: SGI/MIPS (//forums.irixnet.org/forum-3.html)
+--- Forum: Development/Porting (//forums.irixnet.org/forum-9.html)
+--- Thread: LLVM For IRIX (/thread-3043.html)
Pages:
1
2
LLVM For IRIX - aurxenon - 08-06-2021
Hey all, I started working on getting clang/LLVM to output ELF binaries for IRIX a few weeks ago. I noticed a while back that LLVM already has 99% of what you need to get working binaries for IRIX. Its MIPS support is pretty good and it already supports a wide range of MIPS processors, and it also has support for the 3 major MIPS ABIs that IRIX uses, o32, n32, and n64. As a matter of fact, you don't even need to necessarily modify LLVM in any way to be able to get freestanding object files that can theoretically be linked on IRIX. To my understanding, user Eschaton attempted to use LLVM in this way. I wanted to take it a step further however, and get full blown IRIX support integrated into clang and LLVM, as well as to be able to natively host the full LLVM suite on IRIX itself.
I started off with just attempting to get LLVM itself to build & run on IRIX, the main changes here were just small #ifdef __sgi patches to some files to fix some build errors, however I pretty quickly ran into a brick wall, linking all the LLVM static libraries to produce the various executables quickly caused the n32 GCC/binutils to OOM. After banging my head against the wall for a few days trying to get a full n64 toolchain. I gave up and switched gears to instead try and teach clang about IRIX. This was actually relatively easy, LLVM has a very clean & easy to read codebase, and it already has plenty of OS drivers built in. For IRIX, I decided to mostly copy Hurd's driver file, and just make changes here and there to specify IRIX's dynamic linker and system directories. There were some other miscellaneous changes I had to make in other files, mostly relating to target triplets, although I did also have to specify an IRIX OS Target, which again, wasn't hard at all.
Finally I built LLVM with only clang enabled, and I tested it... my hello world didn't build... clang was specifying that SGUG's LD link using elf32btsmipn32, which it didn't support. I made a quick patch to clang to instead pass elf32bmipn32 if the target OS is IRIX. This time the build failed again, but for a different reason; for some reason binutils couldn't find the C runtime or libgcc even though I had them in the sysroot I had passed to binutils when I built it. I then had an epiphany, I could just have clang output object files, then link them on a real SGI, similar to what the OpenVMS developers did when they were porting LLVM. I uploaded helloworld.o to an Octane, then ran gcc on it to link it, and whaddya know, it printed "Hello world from LLVM on an SGI!"
Building helloworld.c on my desktop
Linking and running helloworld.o on IRIX
That was pretty cool, but I wanted to see if my clang/LLVM patches in combo with SGUG binutils on the Octane could handle something a bit more complex, so I got to work on building figlet. I immediately ran into some compilation errors. These were related to types, so I assumed that I needed to use the same include-fixed that GCC uses on IRIX. I added a system include to the IRIX driver for a /usr/include-fixed in addition to the usual /usr/include. However even with these fixes, figlet still didn't build.
I was beginning to think that maybe I'd just gotten lucky after all with the hello world program. But then I realized that the IRIX standard C library enables/disables various types and prototypes depending on what flags the compiler has enabled by default. I didn't feel like putting in the work to add them to clang yet, so I instead printed out all the predefines GCC has on IRIX, then stuck all of them into an irix-defines.h file, and added that to the top of all figlet source files so all of the predefines would apply before any IRIX headers were included. This actually worked, although it resulted in incredibly spammy output during the build since a lot of preprocessor macros were getting redefined by clang, or by the various system headers. I got my figlet object files and uploaded them all to the Octane. Figlet linked and actually ran!
There's plenty of work to be done to get a useful LLVM toolchain for IRIX, for one, the various predefines GCC has for IRIX need to be added to clang so you don't have to stick an irix-defines.h into every source file you want to compile. I also want to have clang/LLVM ported directly to IRIX so you don't need to compile software using two different computers. Finally, I want to eventually be able to have a full LLVM toolchain that uses LLD, as well as LLDB. A few longer term goals would be to have full C ABI compatibility with MIPSPro so there aren't any issues with struct alignment, to get LLD to emit debugging symbols that can be read by DBX, and finally, to clean up the patches enough to upstream IRIX support.
Link to the LLVM fork:
https://github.com/aurxenon/llvm-project-irix
One other thing I forgot to note, the target triplet is a bit broken at the moment, it thinks it's building for mips64-sgi-irix6.5, even though it then builds for 32 bit MIPS, with n32 ABI. Not entirely sure what's causing that, but I'll probably figure it out at some point.
RE: LLVM For IRIX - robespierre - 08-06-2021
(08-06-2021, 07:11 PM)aurxenon Wrote: One other thing I forgot to note, the target triplet is a bit broken at the moment, it thinks it's building for mips64-sgi-irix6.5, even though it then builds for 32 bit MIPS, with n32 ABI. Not entirely sure what's causing that, but I'll probably figure it out at some point.N32 is a 64-bit ABI.
RE: LLVM For IRIX - aurxenon - 08-06-2021
Oops, thank you. I always assumed that n32 was just simply a faster 32 bit backport of n64. I didn't realize it was more akin to x32.
RE: LLVM For IRIX - Raion - 08-06-2021
Me and Aur also chatted regarding this prior to release and I was helping with the first few attempts in logistics ways, and yeah, I was confused because MIPS64 and MIPS32 are used confusingly in all compiler docs. Keeping in mind those are both referring usually to 32-bit and 64-bit MIPS ISAs incompatible with MIPS III/IV I was confused as well, but we realized in this context it's just a bits identifier.
I'm proud of Aur for getting this all together, and all the credit goes to him.
RE: LLVM For IRIX - aurxenon - 08-11-2021
Hey guys, done quite a bit of work since the previous comment. Immediately after I got clang to be able to emit object files for IRIX, I went to work on fixing the missing predefines the IRIX standard library relies on. Raion helped out by dumping some of the MIPSPro predefines, these were mostly to enable/disable features for different MIPS microarchitectures, so I added these to clang's MIPS target, while the SGI defines I added to the IRIX driver. There were 2 predefines that clang already implemented, but that I had to change for IRIX, __mips, and _MIPS_ISA, these were set by default to 64 and _MIPS_ISA_MIPS64 respectively, while MIPSPro sets these to mips3/mips4 and _MIPS_ISA_MIPS3/_MIPS_ISA_MIPS4 depending on which microarchitecture you're building for. Even after I finished implementing all these predefines however, this wasn't enough to build a hello world however, there were still plenty of errors from clang being unable to find various types and function prototypes from the IRIX standard library. I then dumped GCC and clang's predefines and I ended up writing a tiny Python script to compare clang and GCC's predefine output to figure out which predefines I needed to add to clang.
This revealed about a hundred predefines that GCC had, but clang didn't have. This was too much to just add to clang manually, especially since I suspected the vast majority of them weren't even used by anything on IRIX, so I ended up extending the script to check and see which GCC predefines got actually used inside the IRIX standard library. My suspicions were confirmed when I only uncovered about 8 or so of these predefines actually getting used inside the IRIX standard library. _LANGUAGE_C was a big predefine that I was missing, without it, sgidefs.h, which defines some of the types the standard library uses elsewhere, literally appeared to the clang preprocessor as a blank file. I added these few missing-and-important predefines to clang, and then finally, I managed to get clang to compile a hello world object file without needing me to manually stick a big list of predefines at the front.
From here, I initially began working on adding IRIX support to LLVM's libc++, however, I changed my mind and instead decided to kill two birds with one stone, get a proper IRIX linker for my host machine, and also implement IRIX support for LLD. I quickly fixed the bug I was having before where the binutils LD wasn't detecting the object files or shared libraries it needed, turns out I wasn't actually passing the paths to the IRIX /usr/lib32 inside my sysroot. I also made sure to download my GCC installation from the Octane as well, since I don't have a working compiler-rt yet, and GCC's libraries provide a good enough equivalent for now. I then discovered that LLD expected to use MIPS target emulation elf32btsmipn32 rather than elf32bmipn32, so I commented out my previous "fix" I added to clang.
After my 2 previous fixes, I was left with this:
I was extremely confused since I've used those exact same object files and libraries to link object files emitted from clang on the Octane under binutils LD before, and I've never had it error out like this. Everything looked fine under readelf, so I initially assumed that maybe I was accidentally linking the GCC libraries, which are built for the MIPS3 microarchitecture, with MIPS4 libraries. However even after fixing the paths so they specifically pointed to MIPS3 files in all cases, I still got the same error. From here I ended up spending quite a while searching the internet, looking at ELF documentation until I finally stumbled across a page from Oracle about ELF symbol tables.
Something I had read prior finally clicked:
LLD was complaining that even after it had read what it thought was the last STB_LOCAL symbol according to .symtab's sh_info variable defined in the section header, it was seeing that there were still more local symbols mixed in with the global and weak symbols. In other words, the crtbegin.o provided by GCC doesn't follow the ELF specification. Unlike figuring out what the issue meant, fixing it was rather simple, all I did was patch out the check for mixed local and global/weak symbols inside LLD, causing LLD to emit warnings rather than error out. LLD now only emitted one more error related to library parsing. I got to work on figuring out why LLD was now complaining that libgcc_s.so had an invalid sh_entsize. I checked and it turned out that the .dynamic section of libgcc_s.so was what LLD had a problem with. I did some more reading on ELF and I discovered that sh_entsize essentially defines the size of each entry inside the dynamic array.
At first, in order to get around the sh_entsize, I wrote some code for LLD that literally just set .dynamic's sh_entsize to sizeof(Elf_Dyn). I then had to rebuild the entirety of LLVM (since the change was inside a header file), only to discover I had wasted my time, LLD didn't like me directly modifying the Elf_Dyn section header in memory, and it crashed. I didn't really feel like spending the time to figure out why LLD was segfaulting after modifying the section headers, plus I knew that the libgcc_s.so file specifically was the problem anyways, so I simply took the easy way out and directly overwrote libgcc_s.so's .dynamic's sh_entsize with a value of 8 in a hex editor. The reason I picked 8 is because that's the native size of each entry under the n32 ABI. After I rebuilt LLVM, I tested out my new libgcc_s.so, and hey, it worked fine. I was getting some new errors, but these were just missing symbol definitions rather than invalid ELF headers.
One of the missing symbol definitions was _rld_new_interface, which is supposed to be filled in at runtime by IRIX's runtime linker, rld, and the other one was a missing function from libm. These were easily fixed by passing both -lm and --allow-shlib-undefined. However, 4 more errors cropped up from /usr/lib32/mips3/crt1.o even after this, 3 of them were unknown ELF relocation types, and 1 other error was complaining about an invalid type of ELF relocation being applied.
This is what I saw after I looked up the unknown relocation value in the SGI ELF64 documentation. According to the addendum at the bottom of the table, this is "intended for address reset records in an Event Location section, (see Section 2.10)". This would make sense since LLD was complaining about .MIPS.event.init having an unknown relocation type. After reading the Event Location section (the aforementioned section 2.10), from what I can understand, R_MIPS_SCN_DISP is used to monitor what the program is currently doing, which is where the events name of the section comes from. This info seems to be mainly used for debugging in conjunction with DWARF, but supposedly it's also used for processor bug workarounds. Anyways, it didn't really seem like LLD should have to touch this, so I implemented R_MIPS_SCN_DISP into LLD, but had it simply do nothing when encountering this relocation type.
Finally, only one more error to go (for real this time). LLD didn't like that .MIPS.event.text held a non ABS relocation, in this case, R_MIPS_CALL_HI16, against the __istart symbol. R_MIPS_CALL_HI16 performs lazy resolution of symbols, essentially, it waits until the first time a symbol is called in order to actually determine the location of this symbol. It appears that LLD wanted the absolute location to be found out right away rather than wait until a function call is made. I simply patched the error out and instead had LLD emit a warning.
Finally after all of this, clang could compile hello world and have LLD link it to get an a.out.
(I guess libm and libc also don't follow the ELF specification)
The resulting binary actually ran on the Octane!
Of course I couldn't stop there, I compiled figlet and linked it using clang and LLD, and it built just fine. I then went to run it on the Octane, and I got a segmentation violation. I inspected the headers with readelf and they were quite obviously corrupted
I got worried for a few minutes that my fixes to LLD hadn't worked after all, but then I happened to notice something. figlet was larger by a couple dozen bytes on my host machine, curious as to why this was, I inspected figlet under readelf on my host, and it was perfectly normal
It turned out that FTP was clobbering the figlet binary while it was being uploaded to the Octane, I have no idea why this didn't happen to any of the other object files or the hello world when they were being uploaded to the octane, but to get around this, I simply switched to SFTP rather than using the builtin FTP server. I reuploaded figlet, and just like the hello world did, it ran perfectly
Certainly quite an amazing sight.
I've committed the changes I made to LLD as well as the changes to the predefines to GitHub. My next goal after this is to implement IRIX support into libc++, then get compiler-rt working. After that, hopefully I can get a native clang/LLD build that can run on IRIX itself.
RE: LLVM For IRIX - robespierre - 08-11-2021
That's very cool. Thanks for the detailed writeup, the journey is the most helpful part of this.
(08-11-2021, 08:12 PM)aurxenon Wrote: It turned out that FTP was clobbering the figlet binary while it was being uploaded to the Octane
Do you reflexively type BINARY or IMAGE every time you start FTP? Many FTPs default to ASCII and may implement character conversions in that mode.
RE: LLVM For IRIX - aurxenon - 08-11-2021
(08-11-2021, 09:34 PM)robespierre Wrote: That's very cool. Thanks for the detailed writeup, the journey is the most helpful part of this.I normally use FileZilla to upload and download files from the Octane, so I just went and checked my default transfer settings, it appears that FileZilla defaults to treating files without an extension as ASCII files when uploading under FTP. I guess this explains why the object files and the a.out got transferred correctly, but not the figlet executable.
(08-11-2021, 08:12 PM)aurxenon Wrote: It turned out that FTP was clobbering the figlet binary while it was being uploaded to the Octane
Do you reflexively type BINARY or IMAGE every time you start FTP? Many FTPs default to ASCII and may implement character conversions in that mode.
RE: LLVM For IRIX - Raion - 08-11-2021
Aurxenon is getting a very nice server for his troubles here and some money from me. If anyone wants to tip him, I don't mind handling it. I have cash app, crypto and Patreon. Aur is a real bro here and I recommend everyone support him as he's doing things that nobody else is bothering with.
Aurxenon, you can also drop PayPal.me or other means of direct payment if you want. That would be fine by me.
RE: LLVM For IRIX - vishnu - 08-12-2021
Raion, so I'm inferring you *don't* have Paypal? Patreon always takes a chunk, which I'm fine with, I have no problem with a payment company that wants to stay in business, but Paypal doesn't take anything if you're sending to what they define as "friends and family." Disclaimer: This is AFAIK, I haven't read either of their terms and services.
RE: LLVM For IRIX - Raion - 08-12-2021
I don't have Paypal, I quit it. They want KYC that I find unacceptable for a company that's neither a bank nor a credit union but in a weird limbo between a credit institution, processor and "bank". So I chose to leave it behind. CashApp is free, Crypto is free, and Patreon's fraction cut is small.
As for updates on this, the big issue right now is libc++, which expects a unicode/modern locale system, neither of which we currently have, and adding such a thing is not possible. Because of that, our goal is to try and get it built without any of those necessary types, but it's proving difficult. Worst case, we can fall back to libstdc++v3 but... we'll see how far we get.