[robespierre]

Note that there were some functions that did not follow the expected ABI calling convention (manual or automated optimizations?) and would therefore require some more work.

The O32 ABI is more strict. Every argument to a function must have space on the stack in addition to being passed in registers. The stack location for saving every register is strictly defined.
The N32 and 64 ABIs are much more relaxed. Stack space is not required for arguments in registers. If registers must be saved, the save location is up to the function.
These ABIs also allow more flexibility in how functions interact with the link editor. Any instruction can be marked with a relocation tag to be filled in later when objects are linked together.
When IPA (interprocedural analysis) is used, the compiler can reason about possible call graphs between functions. A function "B" that is only called by one other function "A" (for example), may be able to share stack space or temporary registers with "A", or be inlined directly into it. This is in addition to the ABI rules for non-leafs (functions which call others), leafs which use stack space, and leafs which do not use stack space.

To touch code at assembly level, it must be coded at assembly level. This holds true for SGI MIPSpro C/C++ where no asm() construct is available to my knowledge.

MIPSpro does not support an 'asm' extension. The recommended way to write assembly is to first make an outline of the function in C with whatever argument and return types are needed, compile with cc -S, and then edit the emitted "out.s" file, to be assembled with as.

To implement this double call before and after the system function, the wrapper function in libinstw.so would just need to call first the prologue function, then the system libinst.so function, and finally the epilogue function. In the source code, prologue functions are called pre functions, and epilogue functions are called post functions. Both are stored in the file prepost.c (conveniently named).

One of the common uses for wrapping functions is to change what arguments they receive. In your framework, how is this accomplished? Does the dispatcher put the arguments on its stack where a prologue callee can modify them? What about epilogues that wish to modify the return values?