How to Perform A Autopsy (Oh God I've Been Owned)
A Text for Admins in the Wonderful World Of Unix 
Digital Ebola 
--------------------------------------------------------------------------------

No matter how good you are, eventually it's going to happen. You are going to 
get owned. Now, while this might not happen as often to a senior level admin 
that has been playing with UNIX for a long time, it will happen at one point. 
This is mathematics. Someone is going to find a bug somewhere, or maybe you just
overlooked something. Never the less, it happened, stop crying like a baby, and
plan your next action. It doesnt cost 80 thousand dollars to bring boxes back 
from the molestations of a 14 year old, but it does take some effort. You have
essential 3 options which I will explain:

1. Reinstall. Time consuming, but very effective. Quicker then the other methods

2. DE-Own the box. A little more time consuming then a reinstall, but you saved 
   your configurations. 

3. Set up booby traps and wait. 


OPTION #1 

Reinstalling is such a pain. You have custom scripts to replace, custom 
configurations for any of the services you may be running, and possibly even 
custom software that has been written in-house. I can not stress to you the 
importance of a back-up in any case. If you are not backing things up, you are
not doing your job. Period. Don't go blaming your loss on the hacker that got 
you, because the same thing could have happened in the event of a hardware loss
or a Layer 1 disaster. If you were smart, restore from backups, fix the original
security hole, and your on your way. If you were not smart, well, you are going
to spend some time reconfiguring, but this is still faster then the next methods
I am about to detail. 

OPTION #2

DE-Owning the box is time consuming. No doubt about it. You will gain alot of 
information by doing this. What it takes is patience, and a lot of reading. 
First of all, you need to assess the visable damage. Was there any webpage 
defacements? Was there a nasty issue.net? Are users complaining of data loss?
Are the logs still intact? Also, did the hacker leave behind a history file?
Some hackers are way sloppy, either they are too much of a novice to know, 
or they just plain don't care. At any rate, this is information you have to 
collect. 

After your damage assessment is complete, you can then begin to fix your box.
Chances are great, your hacker has left several backdoors in the system. 
The golden rule of thumb is: Anything that runs as root can be a backdoor. 
Some examples are, /bin/login , /root/.bash_profile , or any of your startup 
scripts such as /etc/rc.x or /etc/init.d so forth and so on. If you know UNIX,
you will know what I am talking about. Another more simple backdoor is a 
suid shell somewhere on the box. You check for all SUID programs by doing this:

owned$ find / -perm -4000 -print > suid

You are looking for anything out of the ordinary such as this:

-rwsr-sr-x    1 root     root       426980 May  9 01:00 .bash

Now, why would there be a SUID program in someones home directory that is owned
by root? When did root put that there? Chances are, root did not. The kid that 
was playing around as root did. And in most cases, executing that program as 
a standard user will produce a root shell. Granted, there has been some 
improvements on some versions of shells that has better UID control, but there
are alot of systems that will let you chmod +s sh and then run it as a user. 
It runs as root. You become root. 

Another even simpler backdoor can be found by simply checking your passwd file.
You would be surprised the number of admins that never watch their passwd file.
In this case you would be looking for:

digi:x:1000:1000:Digital Ebola,,,:/home/digi:/bin/bash
digi2:x:0:0:Digital Ebola,,,:/home/digi:/bin/bash

The hacker would then proceed to telnet into your machine as a normal user, 
and then su to their rootshell. I must say, I have personally seen a backdoor
like this last for up to 78 days. 

Another form of backdooring is the rootkit. Hackers today are mostly unoriginal
people, either be it from lack of skill or from impatience. This is a good thing
for you the admin. Why? Their lack of creativity will allow you to find their 
back doors easily due to the public archiving of these rootkits. You should 
download every rootkit you can find, and do file compares between them and the
binaries of your flavor/distro. Common trojaned services are telnetd, identd and
even sshd. This takes time. Read the rootkit instructions, try the default 
methods of accessing. Most people never even bother to customized a pre-built 
rootkit. 

Now, if you have had a real professional come in your machine, then I can 
honestly say that you may never find all the backdoors. I do not mean to 
kill off your hopes of a recovery, but there are some out there that have 
true finesse. I'll give you a example. Hacker comes in box via public exploit,
just as any kiddie. He see this system as one to keep. He does not modify a 
webpage, he does not packet from the box. He wants to keep his access, to be
the ghost in the machine. He trojans the kernel.

Dear admin, I must ask how well do you know your kernel configuration? Do you
REALLY know the modules that you are loading? You can do ANYTHING from the   
kernel. End of story. I want to hide my processes, I want to become invisible. 
I will make the kernel do what I want. A easy way to do this from a hackers 
standpoint, is to install LIDS (if you are a Linux admin, if you are actual UNIX
admin, similar tactics can apply). Yes, the Linux Intrusion Detection System. 
This thing can hide processes, make files undeleteable, and even monitor 
modifications. And most admins have never even noticed that their kernel has 
been recompiled, and their machine rebooted. You would think that Admins would
pay more attention to their uptime, but there are ways of handling that too. 
Someone that can write custom kernel modules can stay in your machine for as 
long as they wish. Or at least until you recompile your kernel. Which means, 
after your intrusion, a kernel compile is a must. The heart of your operating
system has complete control, if the hacker controls your kernel, and knows how
to manipulate it, you are going to be fighting him for a long time. If you even
relize he is there. 

OPTION #3

Setting up booby traps can be fun. It can also give you a good deal of 
information. I am not talking about buying a pre-built "honey-pot". I am 
talking about rigging the system to where you can watch your intruder. 
See what he is doing. See where he is coming from. I will talk more about 
what you can do for perfoming a autopsy in a moment. For now, lets concentrate
on what you need to know to be able to get to that point. 

First thing, is isolating what the intruder has done, assess the damage. 
Don't fix the problems, just write it all down. You must make it appear that 
you don't have a clue that he is there. Now, the trick way to watch him, is to 
work around what he has done. If he has not messed with your login service, you
will be able to hide a ttysnoops server. Once he is logged in, you can watch 
his terminal session real time. Chances are, he has rigged the logs to wipe 
themselves after each logout. Start another log daemon, make it look like a 
service that is meant to be on the system. The hacker will most likely not 
notice. Log everything he does. Install a sniffer, and hide the process and log
that too. This bringing me to the next section, doing the actual tracking.

TRACKING 

You are now watching your server close for anything out of the ordinary. You
see hacker log in. You are watching his every movements. What do you do? 
First thing, is to see where he is coming from. Is there a DNS reverse? or is 
it a IP? Does the IP reverse? In most of these situations, most admins do not
know what to do beyond a simple nslookup, and if the IP does not reverse, 
the admin thinks that its hopeless. Not so. You can perform a ARIN whois.
ARIN is the American Registery for Internet Numbers. They are the people in 
charge of assigning IP's and they work closely with other agencys around the 
world that perform that same function. If your attackers IP will not reverse, 
go to ARIN (http://www.arin.net) and lookup the IP. It will come back to the 
provider of that IP, and even sometimes a individule. At this point, you may 
call these people and ask them who is in charge of that IP. If you are 
suspecting a break-in, most providers are happy to help you out any way they 
can, short of giving out personal information. You can then go from there, to
contacting the admin of that system, to see if its a actual user on his 
system, or maybe he has suffered a break-in as well. 

For IPs that do reverse, you can get the contact information of the domain 
that is being utilized thru a whois with network solutions. Generally, the 
contact of the domain will have some clue to who is using that machine, and
you will be able to compare notes if they have had a break-in as well. 

Another thing you can do, if you were blessed with a web defacement, is to 
check the web defacement mirrors. Chances are, if they hit you, they hit others
as well. You can then call the other people that had break-ins and compare
notes. Hackers that do web defacements are often very blatant about who they 
are. A search on metacrawler or another popular search engine will often yield 
interesting information. Maybe even point you to a home page with contact info!
Maybe the attacker IRC's. Most likely he is IRCing from his home machine. Be
advised, IRC is a very anonymous medium. Just because someone says they are 
someone, does not mean they are that person. It is noteworthy to check it out, 
but please, realize that IRC is not true to life. 

Things you can do to make your autopsy go better...

I have already stressed the importance of backups. Any admin worth his title
knows to back his data up. Another thing you can do is use a loghost in addition
to logging locally to your machine. A loghost is wonderful. It basically allows
your machines to be penetrated, and yet you still have a full account of the 
connections. Make your loghost as close to unbreakable as you can. Do not run 
any other kinds of services on this box. Ideally, this machine will be local, 
so you will not have to even run telnetd. You can rig a cronjob to backup your
networks logs to tape everynight, or even better, back up via CD burner. You 
will thank yourself later. 

You should make it policy to log every event on the system no matter how 
small. It will develope a since of timing. If timings are wrong on your 
daily events, either you have a malfunction or you have a intruder. There are 
kernel modules availiable on the internet that allow you to log every command,
regardless of shell, thru your kernel. Installing this and logging to a remote 
host is very effective for keeping your system monitored properly. Install 
tripwire, or a like binary, and keep its database updated. Watch for little 
changes. Everything you watch and log now, will make your life easier in the 
event of a intrusion. 

CONCLUSION

I see news reports of a hacker that is caught and fined 250,000 dollars. This 
is the supposive cost of restoring the system. This is outrageous. The 
higher the dollar amount, the longer the hacker goes away to prison. The 
sentence can be longer for hacking a machine, then it is for murder. Have we 
as a society decided that a Sun Enterprise 3500 is more important then a human
life? Yes, hacking is wrong. So is murder. Taking a human life should be more 
of a charge then the taking of a server. It does not cost thousands of dollars
to replace data. It does take time. It does take work. I can have no sympathy 
for a person that is too ignorant to back their data up. After all, are you 
going to blame a hacker for a hardware malfunction? Before you go condemning 
all hackers for your intrusion, please realize, that I could not have written 
this text, without intruding on machines at one point in my life. There will 
always be someone out there with more skill, or that has a piece of knowledge 
that you don't. Accept it, it is the reason we got into computers in the first 
place. If you are recovering from a intrusion, all you can do is learn from it,
and become a wiser person. You might get the hacker in the end, you might send 
him to jail, but it is one of many. There will always be people out there that
can get in. Some suggestions so you do not become a statistic: watch your distro
/flavor's homepage for security updates. Watch bugtraq and securityfocus and 
any other security site possible. There are people that develope new 
vunerabilities every day. Watch out for these, and adjust your policies 
accordingly. Read about intrusion detection systems and use them. 
And if you do have a have a intrusion, don't just blame the hacker
and don't just blame yourself. It's a learning experience, not a very fun one, 
but you will live thru it, and perhaps in the end teach someone else how to 
get thru it. 

EOF