Two JVM options are often used to tune JVM heap size: -Xmx for maximum  heap size, and -Xms for initial heap size. Here are some common mistakes  I have seen when using them:
1. Only setting -Xms JVM option and its value is greater than the  default maximum heap size, which is 64m. The default minimum heap size  seems to be 0. For example,   
      java -Xms128m BigApp
      Error occurred during initialization of VM
      Incompatible initial and maximum heap sizes specified  
      The correct command should be java -Xms128m -Xmx128m BigApp.  It’s a good idea to set the minimum and maximum heap size to the same  value. In any case, don’t let the minimum heap size exceed the maximum  heap size.
Heap size is larger than your computer’s physical memory. For example,   
      java -Xmx2g BigApp
      Error occurred during initialization of VM
      Could not reserve enough space for object heap
      Could not create the Java virtual machine.  
      The fix is to make it lower than the physical memory: java -Xmx1g BigApp  
    2.Incorrectly use mb as the unit, where m or M should be used instead.  
      java -Xms256mb -Xmx256mb BigApp
      Invalid initial heap size: -Xms256mb
      Could not create the Java virtual machine.  
    3 The heap size is larger than JVM thinks you would ever need. For example,  
      java -Xmx256g BigApp
      Invalid maximum heap size: -Xmx256g
      The specified size exceeds the maximum representable size.
      Could not create the Java virtual machine.  
      The fix is to lower it to a reasonable value: java -Xmx256m BigApp  
    4.The value is not expressed in whole number. For example,  
      java -Xmx0.9g BigApp
      Invalid maximum heap size: -Xmx0.9g
      Could not create the Java virtual machine.  
      The correct command should be java -Xmx928m BigApp  
NOTE:  
How to set java heap size in Tomcat?
Stop Tomcat server, set  environment variable CATALINA_OPTS, and then restart Tomcat. Look at the  file tomcat-install/bin/catalina.sh or catalina.bat for how this  variable is used. For example,  
set CATALINA_OPTS=”-Xms512m -Xmx512m”  (Windows)
export CATALINA_OPTS=”-Xms512m -Xmx512m”  (ksh/bash)
setenv CATALINA_OPTS “-Xms512m -Xmx512m”  (tcsh/csh)  
In catalina.bat or catallina.sh, you may have noticed CATALINA_OPTS,  JAVA_OPTS, or both can be used to specify Tomcat JVM options. What is  the difference between CATALINA_OPTS and JAVA_OPTS? The name  CATALINA_OPTS is specific for Tomcat servlet container, whereas  JAVA_OPTS may be used by other java applications (e.g., JBoss). Since  environment variables are shared by all applications, we don’t want  Tomcat to inadvertently pick up the JVM options intended for other apps.  I prefer to use CATALINA_OPTS.  
How to set java heap size in JBoss?
Stop JBoss server, edit  $JBOSS_HOME/bin/run.conf, and then restart JBoss server. You can change  the line with JAVA_OPTS to something like:  
JAVA_OPTS=”-server -Xms128m -Xmx128m”  
How to set java heap size in Eclipse?
You have 2 options:
1. Edit eclipse-home/eclipse.ini to be something like the following and restart Eclipse.  
-vmargs
-Xms64m
-Xmx256m  
2. Or, you can just run eclipse command with additional options at  the very end. Anything after -vmargs will be treated as JVM options and  passed directly to the JVM. JVM options specified in the command line  this way will always override those in eclipse.ini. For example,  
eclipse -vmargs -Xms64m -Xmx256m  
How to set java heap size in NetBeans?
Exit NetBeans, edit the file netbeans-install/etc/netbeans.conf. For example,  
netbeans_default_options=”-J-Xms512m -J-Xmx512m -J-XX:PermSize=32m -J-XX:MaxPermSize=128m -J-Xverify:none  
How to set java heap size in Apache Ant?
Set environment variable  ANT_OPTS. Look at the file $ANT_HOME/bin/ant or %ANT_HOME%\bin\ant.bat,  for how this variable is used by Ant runtime.  
set ANT_OPTS=”-Xms512m -Xmx512m”  (Windows)
export ANT_OPTS=”-Xms512m -Xmx512m”  (ksh/bash)
setenv ANT_OPTS “-Xms512m -Xmx512m”  (tcsh/csh)  
How to set java heap size in jEdit?
jEdit is a java application,  and basically you need to set minimum/maximum heap size JVM options when  you run java command. jEdit by default runs with a default maximum heap  size 64m. When you work on large files, you are likely to get these  errors:  
java.lang.OutOfMemoryError: Java heap space
at java.lang.String.concat(String.java:2001)
at org.gjt.sp.jedit.buffer.UndoManager.contentInserted(UndoManager.java:160)
at org.gjt.sp.jedit.Buffer.insert(Buffer.java:1139)
at org.gjt.sp.jedit.textarea.JEditTextArea.setSelectedText(JEditTextArea.java:2052)
at org.gjt.sp.jedit.textarea.JEditTextArea.setSelectedText(JEditTextArea.java:2028)
at org.gjt.sp.jedit.Registers.paste(Registers.java:263)  
How to fix it? If you click a desktop icon, or Start menu item to  start jEdit: right-click the icon or menu item, view its property, and  you can see its target is something like:  
C:\jdk6\bin\javaw.exe -jar “C:\jedit\jedit.jar”  
You can change that line to:  
C:\jdk6\bin\javaw.exe -Xmx128m -Xms128m -jar “C:\jedit\jedit.jar”  
If you run a script to start jEdit: just add these JVM options to the java line inside the script file:  
java -Xmx128m -Xms128m -jar jedit.jar  
If you start jEdit by running java command: just add these JVM options to your java command:  
java -Xmx128m -Xms128m -jar jedit.jar  
Note that when you run java with -jar option, anything after -jar  jar-file will be treated as application arguments. So you should always  put JVM options before -jar. Otherwise, you will get error:  
C:\jedit>java -jar jedit.jar -Xmx128m
Unknown option: -Xmx128m
Usage: jedit [] []  
How to set java heap size in JavaEE SDK/J2EE SDK/Glassfish/Sun Java System Application Server?
Stop the application server, edit
$GLASSFISH_HOME/domains/domain1/config/domain.xml, search for XML element name java-config and jvm-options. For example,  
-Xmx512m
-XX:NewRatio=2
-XX:MaxPermSize=128m
…  
You can also change these settings in the web-based admin console, typically at 
http://localhost:4848/, or 
https://localhost:4848/.  Go to Application Server near the top of the left panel, and then on  the right panel, click JVM Settings -> JVM Options, and you will see a  list of existing JVM options. You can add new ones and modify existing  ones there.  
Yet another option is to use its Command Line Interface (CLI) tool command, such as:  
./asadmin help create-jvm-options
./asadmin help delete-jvm-options  
They may be a bit hard to use manually, but are well suited for automated scripts.