If you are able to determine what means the attacker used to get into your system, you should try to close that hole. For instance, perhaps you see several FTP entries just before the user logged in. Disable the FTP service and check and see if there is an updated version, or if any of the lists know of a fix.
Check all your log files, and make a visit to your security lists and pages and see if there are any new common exploits you can fix. You can find Caldera security fixes at http://www.caldera.com/tech-ref/security/. Red Hat has not yet seperated their security fixes from bug fixes, but their distribution errata is available at http://www.redhat.com/errata
Debian now has a security mailing list and web page. See: http://www.debian.com/security/ for more information.
It is very likely that if one vendor has released a security update, that most other Linux vendors will as well.
There is now a linux security auditing project. They are methodically going through all the user space utilities and looking for possible security exploits and overflows. From their announcement:
"We are attempting a systematic audit of Linux sources with a view to being as secure as OpenBSD. We have already uncovered (and fixed) some problems, but more help is welcome. The list is unmoderated and also a useful resource for general security discussions. The list address is: security-audit@ferret.lmh.ox.ac.uk To subscribe, send a mail to: security-audit-subscribe@ferret.lmh.ox.ac.uk"
If you don't lock the attacker out, they will likely be back. Not just back on your machine, but back somewhere on your network. If they were running a packet sniffer, odds are good they have access to other local machines.