Is there a way to see how much memory I'm using in the JavaVM?

Yes. The standard java.lang.Runtime package can be accessed within the Netscape JavaVM.

Retrieve Free and Totals with freeMemory() and totalMemory(). gc() will force the garbage collector to run. Calling gc() two or three times is usually required to reclaim everything. runFinalization() actually frees the memory (and some other runtime associated structures).

HINT: You can configure the memory sizes from JavaScript:
var rt=java.lang.Runtime.getRuntime()
Alert("JavaVM Memory: Free: "+ rt.freeMemory() +" of Total: "+ rt.totalMemory())

NOTE: You must NOT call gc() and runFinalization() from a LiveConnect thread (called from JavaScript). The garbage collector requires a standard JavaVM thread context.