Fighting Spammers with Splunk 4.1

I haven't blogged about Splunk for a while, but with Release 4.1 there are some new cool features, that helps us tremendously in fighting spammers.

Most of you may not know, that a substantial part of running an ISP mail platform consists of fighting abuse.

Almost all abuse cases are caused by Spammers. Their main target is to control as many mailbox accounts as possible, where they can send spam from. Spammers can send Spam through bot-nets, acquire passwords through phishing attacks, or just do a smpt-auth brute-force attack.

Anyways, the Internet is 24x7, Spammers never sleep and we do neither. There is always someone on-call.

I want to show you, how we currently detect abuse on our platform, and what we do against it.

Detecting Abuse

There are many ways how to detect abuse. I don't want to go into too many details, and also, I don't want to reveal everything we do, as Spammers might read this :-)

One popular way is to count the number of "User unknowns" status messages from the remote mail server. Spammers own address-lists. Often the quality of these lists is bad and these lists contain email addresses that don't exist anymore.

What we do is running a saved search periodically and if we count a certain amount of such user unknown error codes, we trigger an alarm by e-mail and SMS.

SMS Alarming

If the saved search triggers an alarm, it calls a perl script. The script gets different arguments, like the job-id for this search. Using this information, a REST call is done to the Splunk Search-Head to give out the results for this search. The results are formated in XML, which then is parsed using XPATH. After that, an email message is send to an SMS-Gatewas resulting in following alarm on a mobile phone.

The SMS shows basic information, so that the poor guy on-call can decide if it is a false positive or not. If it's obviously spam, things get interesting.

Finding the Abuser
As I mentioned earlier, Splunk does not only trigger an SMS, but also sends an email for this alert.

Clicking on the result link opens the Splunk Web Interface.

For some searches you can just click on any field to drill down to the events, but because we use "fillnull", we can't do that. Instead we will just modify the search slightly. This will give us all events.

From here we can select the options for the field "transaction_id". You can see an option "Show This Transaction Only". Pressing this will open a single transaction:

Blocking the Abuser

Now comes the interesting part. You can see two options, "Block smtp auth", and "Block webmail". Clicking on these, will issue an HTTP Post to another system, blocking all access to this account.

Job almost done.

Informing the Account Owner

As all access is blocked now, we will report the incident to our abuse team. The abuse team's tracking system will receive all the necessary information, as well as the original log-entries as an evidence. After that, the account owner will be informed about this abuse case.


Splunk 4.1 makes life for us much easier. We don't have to copy-paste information into various applications. Everything can be done by simply clicking on it.


Dan Goldburt said…
This comment has been removed by the author.
Anonymous said…
How are you able to create a workflow for notification? This is amazing!
The Backboner said…
Hi Mika,

Nice blog-entry. We've to speak once later about this...

Chris said…
Hi Mika,
the new workflow feature does make Life easier. I wish I had more time to play with it myself. Nice blog entry.

Popular posts from this blog

Opensolaris, Huawei E220, Swisscom and Sunrise

Adjusting ZFS resilvering speed

Heating up the Data Pipeline (Part 1)