Mongo uses memory mapped files to access data, which results in large numbers being displayed in tools like top for the mongod process. This is normal when using memory-mapped files. Basically, the amount of mapped datafile is shown in the virtual size parameter, and resident bytes shows how much data is being cached in RAM. The larger your data files, the higher the vmsize of the mongod process. (This is also why Mongo is best ran on 64 bit operating systems.)
If other processes on the box need more ram, the operating system’s virtual memory manager will relinquish some memory from the cache — and the resident bytes on mongod process will drop.
You can get a feel for the “inherent” memory footprint of Mongo by starting it fresh, with no connections, with an empty /data/db directory and looking at the resident bytes. (Running with —nojni option will result in even lower core memory usage — and the move to spidermonkey, forthcoming, will make that case closer to the norm.)