GEM only works under IRIX 6.2+. The rld error is probably something about not having glBindTextureEXT (or something). OpenGL 1.0 has some extensions to speed up texture mapping (which are an integral part of OpenGL 1.1). However, these don't exist on IRIX 5.3. If you recompile GEM (see the next question), things should work fine.

I don't have access to an IRIX machine, so don't expect any builds from me. Upgrading to IRIX 6.2+ is worth it.