How to set java heap size in JavaEE SDK/J2EE SDK/Glassfish/Sun Java System Application Server

Stop the application server, edit
$GLASSFISH_HOME/domains/domain1/config/domain.xml, search for XML element name java-config and jvm-options. For example,

-Xmx512m
-XX:NewRatio=2
-XX:MaxPermSize=128m
...

You can also change these settings in the web-based admin console, typically at http://localhost:4848/, or https://localhost:4848/. Go to Application Server near the top of the left panel, and then on the right panel, click JVM Settings -> JVM Options, and you will see a list of existing JVM options. You can add new ones and modify existing ones there.

Yet another option is to use its Command Line Interface (CLI) tool command, such as:

./asadmin help create-jvm-options
./asadmin help delete-jvm-options

They may be a bit hard to use manually, but are well suited for automated scripts.