PhyML can be really demanding for RAM with larghe trees, this information is in PyML stdout. It should be checked and tested by RAPPAS to verify if the process can run on the current architecture.
Alternatively, RAPPAS can advice to use PAML if RAM requirements of PhyML are too high.