-----BEGIN PGP SIGNED MESSAGE----- An HTML version of this tech tip is available from: http://www.cert.org/pub/tech_tips/root_compromise.html ------------------------------------------------------------------------ CERT® Coordination Center Steps for Recovering from a UNIX Root Compromise ------------------------------------------------------------------------ This document sets out suggested steps for responding to a UNIX root compromise. Your response should be carried out in several stages: Introduction A. Before you get started 1. If you have a security policy, consult your security policy 2. If you do not have a security policy i. Consult with management ii. Consult with your legal council iii. Contact law enforcement agencies iv. Notify others within your organization 3. Document all of the steps you take in recovering B. Regain control 1. Disconnect compromised system(s) from the network 2. Copy an image of the compromised system(s) C. Analyze the intrusion 1. Look for modifications made to system software and configuration files 2. Look for modifications to data 3. Look for tools and data left behind by the intruder 4. Review log files 5. Look for signs of a network sniffer 6. Check other systems on your network 7. Check for systems involved or affected at remote sites D. Contact CERT/CC and other sites involved 1. Incident Reporting 2. Contact the CERT Coordination Center 3. Obtain contact information for other sites involved E. Recover from the intrusion 1. Install a clean version of your operating system 2. Disable unnecessary services 3. Install all vendor security patches 4. Consult CERT advisories, summaries, and vendor-initiated bulletins 5. Caution use of data from backups 6. Change passwords F. Improve the security of your system and network 1. Review security using the UNIX configuration guidelines document 2. Review the security tools document 3. Install security tools 4. Enable maximal logging 5. Configure firewalls to defend networks G. Reconnect to the Internet H. Update your security policy 1. Document lessons learned from being root compromised 2. Calculate the cost of this incident 3. Incorporate necessary changes (if any) in your security policy Document revision history ------------------------------------------------------------------------ Introduction This document sets out suggested steps for responding to a UNIX root compromise. Note that all action taken during your recovery from a root compromise should be in accordance with your organization's policies and procedures. A. Before you get started 1. If you have a security policy, consult your security policy 2. If you do not have a security policy i. Consult with management Depending on how your organization is structured, it may be important to notify management in order to facilitate internal coordination of your recovery effort. ii. Consult with your legal council Before you get started in your recovery, your organization needs to decide if pursuing a legal investigation is an option. Note that the CERT Coordination Center is involved in providing technical assistance and facilitating communications in response to computer security incidents involving hosts on the Internet. We do not have legal expertise and cannot offer legal advice or opinions. For legal advice, we recommend that you consult with your legal counsel. Your legal counsel can provide you with legal options (both civil and criminal) and courses of action based on you or your organization's needs. It is up to you how you wish to pursue this incident. You may wish to secure your systems or to contact law enforcement to investigate the case. If you are interested in determining the identity of or pursuing action against the intruder, we suggest that you consult your management and legal counsel to see if any local, state, or federal laws have been violated. Based on that, you could then choose to contact a law enforcement agency and see if they wish pursue an investigation. We encourage you to discuss the root compromise activity with your management and legal counsel to answer the following questions: + What is your legal status in terms of your ability to trap intruders or trace connections (i.e., do you have a login banner stating that connections can be tracked or traced? See CERT Advisory CA-92:19, "Keystroke Logging Banner"). + What are your legal responsibilities if your site is aware of the activity and does not take steps to prevent it? + Have any local, state, or federal laws have been violated? + Should an investigation should be pursued? + Should you report the activity to local, state, or national) law enforcement? iii. Contact law enforcement agencies In general, if you are interested in pursuing any type of investigation or legal prosecution, we'd encourage you to first discuss the activity with your organization's management and legal counsel and to notify any appropriate law enforcement agencies (in accordance with any policies or guidelines at your site). Keep in mind that unless one of the parties involved contacts law enforcement, any efforts to trap or trace the intruder may be to no avail. We suggest you contact law enforcement before attempting to set a trap or tracing an intruder. U.S. sites interested in an investigation can contact their local Federal Bureau of Investigation (FBI) field office. To find contact information for your local FBI field office, please consult your local telephone directory or see the FBI's field offices web page available at: http://www.fbi.gov/fo/fo.htm For more information, please see the web page of the FBI National Computer Crime Squad (NCCS): http://www.fbi.gov/programs/nccs/compcrim.htm You may wish to contact the U.S. Secret Service for incidents involving the following: + theft or abuse of credit card information (e.g., credit card fraud, the exchange of credit card information) + threats to the President of the United States (e.g., threatening email messages) + impersonation of the President of the United States (e.g., the creation of forged email appearing to come from the President) To contact the Secret Service: Secret Service main phone number: +1 202 435-7700 Financial Crimes Division - Electronic Crimes Section Phone: +1 202 435-5850 Fax: +1 202 435 7607 Non-U.S. sites may want to discuss the activity with their local law enforcement agency to determine the appropriate steps that should be taken with regard to pursuing an investigation. iv. Notify others within your organization In addition to notifying management and legal council at your site, you may also need to notify others within your organization who may be directly affected by your recovery process (e.g., other administrators or users). 3. Document all of the steps you take in recovering The importance of documenting every step you take in recovery can not be overstated. Recovering from a root compromise can be a hectic and time consuming process in which hasty decisions are often made. Documenting the steps you take in recovery will help prevent hasty decisions and give you a record of all the steps you took to recover, which will be useful for your future reference. Documenting the steps you take in recovery also may be useful if there is a legal investigation. B. Regain control 1. Disconnect compromised system(s) from the network To regain control, you will need to disconnect all compromised machines from your network. You may wish to operate in single user mode to ensure that you have complete control of the machine; however, by rebooting or changing to single user mode, you may lose some useful information because all processes executing at the time of discovery will be killed. Therefore, you may wish to work through steps in section C.5. Look for signs of a network sniffer to determine if the compromised system is currently running a network sniffer. Operating in single user mode will prevent users, intruders, and intruder processes from accessing or changing state on the compromised machine while you are going through the recovery process. If you do not disconnect the compromised machine from the network, you run the risk that the intruder may be connected to your machine and may be undoing your steps as you try to recover the machine. 2. Copy an image of the compromised system(s) Before analyzing the intrusion we encourage you to create a backup of your system. This will provide a "snapshot" of the file system at the time that the root compromise was first discovered. You may need to refer back to this backup in the future. If you have an available disk which is the same size and model as the disk in the compromised system, you can use the dd command to make an exact copy of the compromised system. For example, on a Linux system with two SCSI disks, the following command would make an exact replica of the compromised system (/dev/sda) to the disk of the same size and model (/dev/sdb). # dd if=/dev/sda of=/dev/sdb Please read the dd man page for more information. There are many other ways to create a backup of your system. Creating a low level backup is important in case you ever need to restore the state of the compromised machine when it was first discovered. Also, files may be needed for a legal investigation. Label, sign, and date the backup and keep the backup in a secure location to maintain integrity of the data. C. Analyze the intrusion With your system disconnected from the network, you can now thoroughly review log files and configuration files for signs of intrusion, intruder modifications, and configuration weaknesses. 1. Look for modifications made to system software and configuration files Verify all system binaries and configuration files. When looking for modifications of system software and configuration files, keep in mind that any tool you are using on the compromised system to verify the integrity of binaries and configuration files could itself be modified. Also keep in mind that the kernel (operating system) itself could be modified. Thus, we encourage you to boot from a trusted kernel and obtain a known clean copy of any tool you intend to use in analyzing the intrusion. We urge you to thoroughly check all of your system binaries against distribution media. We have seen an extensive range of Trojan horse binaries installed by intruders. Some of the binaries which are commonly replaced by Trojan horses are: telnet, in.telnetd, login, su, ftp, ls, ps, netstat, ifconfig, find, du, df, libc, sync, inetd, and syslogd. Other items to check are any binaries referenced in /etc/inetd.conf, critical network and system programs, and shared object libraries. Because some Trojan horse programs could have the same timestamps as the original binaries and give the correct sum values, we recommend you use cmp to make a direct comparison of the binaries and the original distribution media. Alternatively, you can check the MD5 results on suspect binaries against a list of MD5 checksums from known good binaries. Ask your vendor if they make MD5 checksums available for their distribution binaries. Additionally, verify your configuration files against copies that you know to be unchanged. Some of the specific things you may want to inspect in your configuration files are: + Check your /etc/passwd file for entries that do not belong. + Check to see if /etc/inetd.conf has been modified. + If you allow the r commands (rlogin, rsh, rexec), ensure there is nothing that does not belong in /etc/hosts.equiv or in any .rhosts files. + Check for new SUID and SGID files. The following example will print out all SUID and SGID files within your filesystem. # find / \( -perm -004000 -o -perm -002000 \) -type f -print 2. Look for modifications to data Data on root compromised systems is often modified by intruders. We encourage you to verify the integrity of web pages, ftp archives, files in users' home directories, and any other data files on your system. 3. Look for tools and data left behind by the intruder Intruders will commonly install custom-made tools for continued monitoring or access to a root compromised system. The common classes of files left behind by intruders are as follows: + Network Sniffers A network sniffer is a utility which will monitor and log network activity to a file. Intruders commonly use network sniffers to capture username and password data that is passed in cleartext over the network. (see section C.5 below) + Trojan Horse Programs Trojan horse programs are programs which appear to function properly, but either add or remove features. Intruders use Trojan horse programs to hide their activity, capture username and password data, and create backdoors for future access to a root compromised system. (see section C.1 above) + Vulnerability Exploits A majority of root compromises are a result of machines running vulnerable versions of software. Intruders often use tools to exploit known vulnerabilities and gain root access. These tools are often left behind on the system in "hidden" directories. + Other Intruder Tools The types of intruder tools listed above is not intended to be a conclusive list. There may be other tools left behind by an intruder. Some of the other types of tools you may find are tools to: + probe systems for vulnerabilities + launch widespread probes of many other sites + launch denial of service attacks + use your computing and networking resources + Intruder Tool Output You may find log files from any number of intruder tools. These log files may contain information about other sites involved, vulnerabilities of your compromised machine(s), and vulnerabilities at other sites. We encourage you to search thoroughly for such tools and output files. Be sure to use a known clean copy of any tool which you use to search for intruder tools. The following list includes the most common places intruder tools are found on compromised systems. + Look for unexpected ASCII files in the /dev directory. Some of the Trojan binaries rely on configuration files which are often found in /dev. + If an intruder has created a new account and home directory, look very carefully through that directory. There may be hidden files or directories. + Look for files or directories with strange names such as "..." (three dots) or ".. " (two dots and some whitespace). Intruders often try and hide files within such directories. 4. Review log files Reviewing your log files will help you get a better idea of how your machine was compromised, what happened during the compromise, and what remote hosts accessed your machine. Keep in mind when reviewing any log files from a root compromised machine that any of the logs could have been modified by the intruder. You may need to look in your /etc/syslog.conf file to find where syslog is logging messages. Below is a list of some of the more common UNIX log file names, their function, and what to look for in those files. Depending on how your system is configured, you may or may not have the following log files. + messages The messages log will contain a wide variety of information. The thing to look for in this file is anomalies. Anything out of the ordinary should be inspected. Also, look for events that occurred around the known time of the intrusion. + xferlog If the compromised system has a functioning ftp server, xferlog will contain log files for all of the ftp transfers. This may help you discover what intruder tools have been uploaded to your system, as well as what information has been downloaded from your system. + utmp This file contains binary information for every user currently logged in. This file is only useful to determine who is currently logged in. One way to access this data is the who command. + wtmp Every time a user successfully logs in, logs out, or your machine reboots, the wtmp file is modified. This is a binary file; thus, you need to use a tool to obtain useful information from this file. One such tool is last. The output from last will contain a table which associates user names with login times and the host name where the connection originated. Checking this file for suspicious connections (e.g., from unauthorized hosts) may be useful in determining other hosts that may have been involved and finding what accounts on your system may have been compromised. + secure Some versions of UNIX (RedHat Linux for example) log tcp wrapper messages to the secure log file. Every time a connection is established with one of the services running out of inetd that uses tcp wrappers, a log message is appended to this log file. When looking through this log file, look for anomalies such as services that were accessed which are not commonly used, or connections from unfamiliar hosts. The common item to look for when reviewing log files is anything that appears out of the ordinary. 5. Look for signs of a network sniffer When a root compromise occurs, there is the potential for intruders to install a network monitoring program, commonly called a sniffer (or packet sniffer), to capture user account and password information. The first step to take in determining if a sniffer is installed on your system is to see if any process currently has any of your network interfaces in promiscuous mode. If any interface is in promiscuous mode, then a sniffer could be installed on your system. Note that detecting promiscuous interfaces will not be possible if you have rebooted your machine or are operating in single user mode since your discovery of this intrusion. There are a couple of tools designed for this purpose. + cpm available for download from: ftp://coast.cs.purdue.edu/pub/tools/unix/cpm/cpm.1.2.tar.gz + ifstatus available for download from: ftp://coast.cs.purdue.edu/pub/tools/unix/ifstatus/ifstatus2.1.tar.gz Keep in mind that some legitimate network monitors and protocol analyzers will set a network interface in promiscuous mode. Detecting an interface in promiscuous mode does not necessarily mean that an intruder's sniffer is running on a system. Another issue to consider is that sniffer log files tend to grow quickly in size. You may want to use utilities such as df to determine if part of the filesystem is larger than expected. Remember that df is often replaced by a Trojan horse program when sniffers are installed; therefore, be sure to obtain a known clean copy of that utility if you do use it. If you find that a packet sniffer has been installed on your systems, we strongly urge you to examine the output file from the sniffer to determine what other machines are at risk. Machines at risk are those that appear in the destination field of a captured packet. Many common sniffers will log each connection as follows: -- TCP/IP LOG -- TM: Tue Nov 15 15:12:29 -- PATH: not_at_risk.domain.com(1567) => at_risk.domain.com(telnet) For sniffer logs of this particular format, you can obtain a list of affected machines by executing the following command: % grep PATH: $sniffer_log_file | awk '{print $4}' | \ awk -F\( '{print $1}'| sort -u You may need to adjust the command for your particular case. You should be aware that there may be other machines at risk in addition to the ones that appear in the sniffer log. This may be because the intruder has obtained previous sniffer logs from your systems, or through other attack methods. For more information, we encourage you to review CERT Advisory CA-94:01, available from: http://www.cert.org/advisories/CA-94.01.ongoing.network.monitoring.attacks.html The advisory includes a description of sniffer activity and suggested approaches for addressing this problem. Please send us a list of all hosts you know to be affected. This will help us determine the scope of the problem. 6. Check other systems on your network We encourage you to check all of your systems, not just those that you know to be compromised. Include in your check any systems associated with the compromised system through shared network-based services (such as NIS and NFS) or through any method of trust (such as systems in hosts.equiv or .rhosts files, or a Kerberos server). In examining other systems on your network, we encourage you to use our Intruder Detection Checklist: ftp://ftp.cert.org/pub/tech_tips/intruder_detection_checklist 7. Check for systems involved or affected at remote sites While examining log files, intruder output files, and any files modified or created during and since the time of the intrusion, look for information that leads you to suspect that another site may be linked with the compromise. We often find that other sites linked to a compromise (whether upstream or downstream of the compromise) have often themselves been victims of a compromise. It is therefore important that any other potential victim sites are identified and notified as soon as possible. D. Contact CERT/CC and other sites involved 1. Incident Reporting Intruders will frequently use compromised accounts or hosts to launch attacks against other sites. If you find evidence of compromise or intruder activity at any other sites, we encourage you to contact those sites. Tell them what you have found, explain that this may be a sign of compromise or intruder activity at their site, and suggest that they may wish to take steps to determine if/how the compromise occurred and prevent a recurrence. When contacting other sites, please give them as much detail as possible including date/timestamps, timezone, and what to do if they have follow-up information. We would appreciate a "cc" to cert@cert.org on any correspondence. If you like, you can let the site know that you are working with us on on this incident (please include the assigned CERT tracking number in the subject line of your messages). Also let them know that we can offer assistance on how to recover from the compromise. 2. Contact the CERT Coordination Center We would appreciate it you would complete and return an Incident Reporting Form as this will help us better assist you, and allow us to relate ongoing intruder activities. This also provides us a better overview of trends in attack profiles and provides input for other CERT documents such as Advisories and Summaries. We prefer that Incident Reporting Forms are sent to us via email. The Incident Reporting Forms are available from: ftp://ftp.cert.org/pub/incident_reporting_form Our contact information is as follows: Email: cert@cert.org (monitored during business hours) Telephone: +1-412-268-7090 24-hour hotline Fax: +1-412-268-6989 CERT Coordination Center personnel answer business days (Monday-Friday) 08:30-17:00 EST/EDT (GMT-5)/(GMT-4), on call for emergencies during other hours. CERT Coordination Center Software Engineering Institute Carnegie Mellon University Pittsburgh, PA USA 15213-3890 3. Obtain contact information for other sites involved If you need contact information for a .COM, .EDU, .NET, or .ORG top-level domain, we encourage you to use the InterNIC's whois database. For information about using 'whois', see the InterNIC's Whois tutorial at: http://rs.internic.net/tools/whois.html You can query the InterNIC's whois database via the Web at the following URL: http://rs.internic.net/cgi-bin/whois For information about finding contact information for other top level domains, please see the following URL: http://rs.internic.net/help/other-reg.html To find contact information for other incident response teams, you may also want to check the contact list of the Forum of Incident Response and Security Teams (FIRST), available in: http://www.first.org/team-info/ We do not recommend sending email to "root" or "postmaster" of a machine that is suspected of being involved in intruder activity. If that machine is the source of an intruder attack, it is possible that that machine itself may be compromised and the intruder may have root access and/or be reading or intercepting email sent to that host. If you are still unsure of a site or contact details, please get in touch with us. E. Recover from the intrusion 1. Install a clean version of your operating system Keep in mind that if a machine is root compromised, anything on that system could have been modified, including the kernel, binaries, datafiles, running processes, and memory. In general, the only way to trust that a machine is free from backdoors and intruder modifications is to reinstall the operating system from the distribution media and install all of the security patches before connecting back to the network. Merely determining and fixing the vulnerability that was used to initially compromise this machine may not be enough. We encourage you to restore your system using known clean binaries. In order to put the machine into a known state, you should re-install the operating system using the original distribution media. 2. Disable unnecessary services Configure your system to offer only the services that the system is intended to offer, and no others. Check to ensure that there are no weaknesses in the configuration files for those services, and that those services are available only to the intended set of other systems. In general, the most conservative policy is to start by disabling everything and only enabling services as they are needed. 3. Install all vendor security patches We strongly encourage you to ensure that the full set of security patches for each of your systems is applied. This is a major step in defending your systems from attack, and its importance cannot be overstated. We encourage you to check with your vendor regularly for any updates or new patches that relate to your systems. 4. Consult CERT advisories, summaries, and vendor-initiated bulletins We encourage you to consult past CERT advisories, summaries, and vendor-initiated bulletins, and to follow the instructions that are relevant to your particular configuration. Be sure that you have installed all applicable patches or workarounds described in the CERT publications. Remember to check the advisories periodically to ensure that you have the most current information. Past CERT advisories are available from: http://www.cert.org/advisories/index.html ftp://ftp.cert.org/pub/cert_advisories/ Past CERT summaries are available from: ftp://ftp.cert.org/pub/cert_summaries/ Vendor-initiated bulletins are available from: ftp://ftp.cert.org/pub/cert_bulletins/ 5. Caution use of data from backups When restoring data from a backup, ensure that the backup itself is from an uncompromised machine. Keep in mind that you could re-introduce a vulnerability that would allow an intruder to gain unauthorized access. Also, if you are only restoring users' home directories and data files, keep in mind that any of those files could contain Trojan horse programs. You may want to pay close attention to .rhosts files in users' home directories. 6. Change passwords After all security holes or configuration problems have been patched or corrected, we suggest that you change the passwords of ALL accounts on the affected system(s). Ensure that passwords for all accounts are not easy to guess. You may want to consider using vendor-supplied or third-party tools to enforce your password policies. F. Improve the security of your system and network 1. Review security using the UNIX Configuration Guidelines document To help you assess the security of your system(s), please refer to our UNIX Configuration Guidelines document. This document may be useful to you in checking your system for common UNIX configuration problems that are often exploited by intruders. ftp://ftp.cert.org/pub/tech_tips/UNIX_configuration_guidelines 2. Review the security tools document Consider using some of the software security tools that are available, such as Tripwire®, COPS, and the TCP wrapper package. A description of some tools that can be used to help secure a system and deter break-ins is available from: ftp://ftp.cert.org/pub/tech_tips/security_tools 3. Install security tools Install all security tools before you connect your machine back to the network. Also, this is a good time to take an MD5 checksum snapshot of the newly restored system using a tool such as Tripwire®. 4. Enable maximal logging Make sure that logging/auditing/accounting programs are enabled (for example, process accounting) and that they are set to an appropriate level (for example, sendmail logging should be level 9 or higher). Backup your logs, and/or consider writing your logs to a different machine, an append-only file system, or a secure logging host. 5. Configure firewalls to defend networks Consider filtering certain TCP/IP services at your firewall or router. For some suggestions, please refer to "Packet Filtering for Firewall Systems," available from: ftp://ftp.cert.org/pub/tech_tips/packet_filtering G. Reconnect to the Internet If you disconnected from the Internet, the best time to reconnect is after you have completed all the steps listed above. H. Update your security policy The CERT Coordination Center recommends that every site develop their own computer security policy. Each organization may have a specialized culture and security requirements that are specific to their own organization. Please refer to RFC 2196 "Site Security Handbook" for information about developing computer security policies and procedures for sites that have systems on the Internet. This document is available from: ftp://ftp.isi.edu/in-notes/rfc2196.txt 1. Document lessons learned from being root compromised Document and review your lessons learned from going through the process of recovering from a root compromise. This will help you decide the appropriate revisions necessary for your security policy. 2. Calculate the cost of this incident For many organizations, changes simply are not made in security policy until they understand the cost of security, or lack thereof. Calculating the cost of an incident will give you a measurement as to the importance of security for your organization. You may find calculating the cost of this incident useful for explaining to management that security is important to your organization. 3. Incorporate necessary changes (if any) in your security policy The last step to take in this process is to make the changes to your security policy. Be sure to inform members of your organization as to the changes that have been made and how that may affect them. ------------------------------------------------------------------------ CERT is registered in the U.S. Patent and Trademark Office. Tripwire is a registered trademark of the Purdue Research Foundation, and it is also licensed to VCC. Copyright 1998 Carnegie Mellon University. Conditions for use, disclaimers, and sponsorship information can be found in http://www.cert.org/legal_stuff.html and ftp://ftp.cert.org/pub/legal_stuff. If you do not have FTP or web access, send mail to cert@cert.org with "copyright" in the subject line. ------------------------------------------------------------------------ Revision History June 15, 1998 Fixed some broken URLs April 8, 1998 Added trademark information for Tripwire April 6, 1998 Fixed a typo in the find command in Section C.1. April 3, 1998 Fixed URL in Copyright statement April 1, 1998 Version 2.14 Released ------------------------------------------------------------------------ -----BEGIN PGP SIGNATURE----- Version: 2.6.2 iQCVAwUBNYWB0nVP+x0t4w7BAQEZDQQAy6EjB65EcL0joblIvD7zwo4jDZReo84B BUwXnuAiNapM5aSf+f8TVAl7+87T9+uyMZLtQYHyjGP7qQN/YllcKhEq0O6MKFSZ PnNQH3W6jUumaYKJYOuoCefOYJy9/Zbhbv+jnIAV5uSxdh5hAfn0nMwfwkuHQTyx 2LZfhKMjJWk= =eaTd -----END PGP SIGNATURE-----