First off, check to see what Oracle RPMs are installed, and try removing them.
rpm -qa | grep oracle
rpm -e oracle-xe-univ
If you get an error here, move on to the next steps to manually remove XE. Remove the XE directories.
rm -Rf /usr/lib/oracle/xe
rm -Rf /etc/oratab
rm -Rf /etc/init.d/oracle-xe
rm -Rf /etc/sysconfig/oracle-xe
Your system no longer has any XE files and you can start again.