Intrusion detection with git – the basics

How can you make sure all your system binaries and configuration files have not been compromised by an intruder? You can of course run rkhunter[1] or AIDE[2] on a regular base or via cron, but this is only going to show you, that a MD5 checksum has been changed.

Using git as an intrusion detection system is actually more useful than you might think. You can, for example also see what configuration has been changed or overwritten by an automated repository update or you can roll back to a previous working version of a specific apache vhost. If multiple system engineers work on the same machine, you will be able to evaluate what they have done and even make sure it was actually them.

Throughout this post series I will show you the power of git, sudo and gpg to create several layers of defence as well as a completely automated and comprehensible system documentation.

Cautions

Before you are going to git your entire filesystem, you need to consider a few important aspects.

First, the repository must explicitly be restricted to the administrative user root, otherwise everyone is able to search the git index for sensitive information.

Secondly, you only want to monitor important stuff and don’t want to be bothered by vim or shell history files or even fast-growing logfiles. Data directories and other locations such as the storage of your database should also be excluded or you might see git changes in almost realtime.

Note about compatibility

As I have secured many CentOS machines, I will use it as a reference. Please keep in mind that some differences might apply on other versions of Linux and especially on BSD or Solaris.

Git repository

First things first: Create a new git repository at the very root of your file system.

cd /
git init
touch /.gitignore

The crucial point is to immediately restrict access to this repository and the corresponding gitignore.

chown root:root /.git
chown root:root /.gitignore
chmod 700 /.git
chmod 600 /.gitignore

Before you are going to add everything to git, you should populate your .~/gitignore. Here is a very basic setup which is still manageable for beginners:

#######################################################################
# User defined wants
#######################################################################

# Comment this line out if you want to see sudo activity
/var/db/sudo/*


#######################################################################
# Standard Excludes
#######################################################################

######## 1.0) base ########
#/bin
/dev/
/home/
/lib
/lib64
/lost+found/
/media/
/mnt/
/opt/
/proc/
/run/
#/sbin
/shared/
/srv/
/sys/
/tmp/

######## 2.0) /boot ########
/boot/vmlinuz*
/boot/*.img
/boot/*.hmac
/boot/*.gz

######## 3.0) /etc ########
/etc/selinux/targeted/modules/active/modules/*.pp

######## 4.0) /root ########
/root/.bash_history
/root/.viminfo

######## 5.0) /usr ########
#/usr/bin/
/usr/games/
/usr/include/
/usr/lib/
/usr/lib64/
/usr/libexec/
#/usr/sbin/
/usr/share/
/usr/src/
/usr/tmp

######## 5.1) /usr/local ########
#/usr/local/bin/
/usr/local/games/
/usr/local/include/
/usr/local/lib/
/usr/local/lib64/
/usr/local/libexec/
#/usr/local/sbin/
/usr/local/share/
/usr/local/src/

######## 6.0) /var ########
/var/cache/
/var/log/
/var/mail/
/var/spool/
/var/tmp/
/var/www/

######## 6.1) /var/lib ########
/var/lib/git*
/var/lib/pear/
/var/lib/logrotate.status
/var/lib/mysql/
/var/lib/ntp/drift
/var/lib/plymouth/boot-duration
/var/lib/php/session
/var/lib/postfix/master.lock
/var/lib/rpm/
/var/lib/systemd/random-seed
/var/lib/tripwire/*
/var/lib/yum/history/
/var/lib/yum/yumdb/
/var/lib/NetworkManager/

Note:

  • Remove the comment from */bin and */sbin directories, if you only want to monitor configuration changes and not add system binaries to the index.
  • If you want to keep track of shared libraries as well, you need to fine-tune various lib sections

Further adjust the gitconfig to your needs an then add the files and do the first commit.

git add .
git commit -am 'Initial commit'

Validate the repository

As this is your first commit, you should carefully inspect all indexed files to see if something is missing or wrong:

git ls-tree --name-only -r HEAD

Also do a reverse check and inspect all files being ignored:

git ls-files -o -i --exclude-standard

If everything looks OK, you are done for now, otherwise edit your gitconfig and repeat the above steps.

Detect anomalies

Whenever you ssh into your machine, try to routinize the following actions to identify a possible intrusion detection:

  1. Become root
  2. Go to the filesystem root
  3. Check git status and git log

If there is anything that looks suspicious you can always diff it against previous commits.

Daily work: Alter your system

Whatever you do, if it is updating or just changing a configuration, try to be strict to yourself: Do one task after the other and commit each single task (such as adding a new vhost or updating php) into the index separately.
If you train yourself to do this, you will end up with a very well documented system over time and also have a good overview about unwanted changes.

Troubleshooting

In case you add new files or folders to your gitignore and they are not being recognized after you have committed, you will probably need to rebuild the index. Simply remove all cached files and re-add them:

git rm -r -f --cached .
git add .
git commit -am 'Meaningful commit message'

Further information

Have a look at my github[3] repository, which will always provide the latest gitignore file. Feel free to use and/or contribute.

[1] http://rkhunter.sourceforge.net/
[2] http://aide.sourceforge.net/
[3] https://github.com/cytopia/git-ids

Is this really enough protection?

Of course not :-). You are still missing the power of git combined with gpg and multiple git identities via sudo. This will also eliminate the possibility that an intruder can just commit changes to your index.

So stay tuned for the second part, which will handle all the advanced and fun stuff.

4 comments on “Intrusion detection with git – the basics”

  1. Nick Reply

    I am usually making a repo in /etc folder so I can keep track of changes in configuration files. Root folder is even better I guess but the repo might get too big I am afraid. Cool idea though.

  2. Robert Reply

    nice idea. kudos.
    typo: “.~/gitignore”
    any idea when the next part will be forthcoming?

    Cheers!

Leave a Reply to Janelle Cancel reply

Your email address will not be published.