While starting Nexus I was getting following error, this was on a machine that was running at 1GB of memory.

Error occurred during initialization of VM
Could not reserve enough space for object heap

So the solution was to edit /usr/local/nexus/bin/jws/wrapper.conf
and add wrapper.java.additional.4=-Xmx128m parameter this setup java to use only 128mb of memory for the new VM.

Solution 2

After further analysis of the config file there are following options that can be set as well, just need to uncomment them

# Size Java memory, in MB (-Xms)
#wrapper.java.initmemory=128
# Size Java memory, in MB (-Xmx)
#wrapper.java.maxmemory=256