There is not a specific maximum file size. However, if SketchUp is slow when orbiting the model, or when selecting and changing elements, that is usually a sign that the model may be too large for us to render.
nXtRender and IRender nXt use the same extraction logic, and the same rendering engine, so IRender nXt will not help with this problem.
Try removing some geometry if possible to make the file size manageable.
And, of course, make sure you have a minimum of 2GB of RAM in your machine