Could a hostname change by itself?

Recently, starting last weekend, 2007-10-21, I observed distributed brute-force attacks on port 22 (ssh) for about 2-3 days. Luckily, the rate of connections was rather low (I guess it was distributed, too) so that my servers performance was not affected. Strangely enough, by the way, I haven't found any report about this supposedly big attack.

Sometime on Monday, I got annoyed and decided to retrieve the sources' IP addresses and block them. One of my servers provided the IP addresses readily in /var/log/auth.log, thus a command like

grep "Illegal user mysql from ::ffff:[^ ]*$" /var/log/auth.log \
| sed -e 's/^.*::ffff:\(.*\)/\1/' | sort | uniq

was sufficient. However, another server was configured to log hostnames instead of IP addresses if available. In this case, the above command gives us a mixture of IPs and hostnames.

While I was trying to find an elegant way to convert the hostnames to IP adresses I received a Cron message from this very server.

Date: Mon, 22 Oct 2007 11:21:01 +0200 (CEST)
From: Cron Daemon <root@myserver>
Subject: Cron <root@journal>  routine_checks -cron

Mon Oct 22 11:21:01 CEST 2007 domain was not = myserver but imec.msu.ru
Mon Oct 22 11:21:01 CEST 2007 ipaddress was not = $myip but 212.192.229.215

You can tell that I didn't need any coffee to stay awake at this time, even though it was a late damn sysadmin's Monday morning ;-) Moreover, a few weeks ago, we were facing a password leakage of one user of myserver. Back then, I took every measure to guarantee that the server was not compromised, but of course, I was still highly alerted, and all logging facilities were running on full power.

I remember that my first thought was: Damn it. So now I knew someone was still on my server. And it was actively being misused. And, worst of all, I would at least spend a night to set it up anew.

Well, after some rather nervous 30 minutes I started wondering how the IP address could change while I could still reach my server at the same old IP address ;-) Also, a few other things seemed strange. However, the machine was running fine. Thus, I was slowly coming down.

After some time of searching for rootkits and signs of an attack I gave in (a bit) and focused again on the retrieval of the IP addresses. On a terminal where I was logged in as (non-root) user I did something like the following. (Please don't try it! It's plain stupid and wrong!)

grep "PAM: Authentication failure for mysql from" /var/log/auth.log \
| sed -e 's/^.* from \([^ ]*\)$/\1/' \
| while read myhostname ; do
  case $myhostname in 
    1*|2*|3*|4*|5*|6*|7*|8*|9*|0*) echo $myhostname;;
    * ) hostname $myhostname;;
  esac
done

As I said, this is wrong in several ways, or let's say there is 1 major error in it. Everything else is just way oversimplified.

Sometimes a wrong line helps

But, dude, was I lucky that I tried this wrong line again because this time I got an error message which explained quite a lot :-)

hostname: you must be root to change the host name

Got it? I leave the combinatorial work to my readers. Here is a (admittedly long) 1-liner of how to "correctly" retrieve a list of hostnames and IP addresses e.g. from a typical Debian /var/log/auth.log and convert all hostnames to IP addresses where needed:

egrep "PAM: Authentication failure for (mysql|root) from" /var/log/auth.log \
| sed -e 's/^.* from \([^ ]*\)$/\1/' \
| while read myhostname ; do echo $myhostname \
| awk 
'/^(25[0-5]|2[0-4][0-9]|[01]?[0-9][0-9]?)\.(25[0-5]|2[0-4][0-9]|[01]?[0-9][0-9]?)\.(25[0-5]|2[0-4][0-9]|[01]?[0-9][0-9]?)\.(25[0-5]|2[0-4][0-9]|[01]?[0-9][0-9]?)$/ { print; exitcode=1 } END { exit exitcode }' \
&& { MYDNS=($(host -t a $myhostname)) ; echo ${MYDNS[${#MYDNS[@]}-1]} ; } ; done \
>list_of_ip_addresses

The long regular expression used in awk here, checks whether the value of $myhostname is an IP address. If it is it gets printed and the exit code is 1 which causes to stop the line at &&, else the exit code is 0 and everything after && is executed, too. The bash code handling $MYDNS is a way/trick to get the last field of a string (here the output of host) by means of bash arrays. Other tricks like this can be found in GreyCat's BashFAQ.

Finally, if you use the above code to get the IP addresses you might want to add something like grep -v SERVFAIL to get rid of errors of the DNS queries, or else manually check the list_of_ip_addresses.