Both packages can be installed at the same time in RHEL.
Just use;
redhat-switch-mail (system-config-mail in RHEL4) to choose which package you want to use.
Make sure the new packages configuration files are correct (sendmail.mc or main.cf)
After that, make sure that the correct package is set to turn on in the correct runlevels,
chkconfig <sendmail or postfix> on
service <old service> stop
service <new service> restart

Your new email service is running.