Software mode is unreliable, that's pretty much all there is to it. Make sure you don't have any high resolution texture or model paks installed as software mode hates those in particular. You should try using R1GL and then adjusting the settings, you can make it look pretty much identical to software mode with the following:
set gl_coloredlightmaps 0
set gl_texturemode GL_NEAREST
vid_restart