Fighting Spammers with Splunk 4.1

I haven't blogged about Splunk for a while, but with Release 4.1 there are some new cool features, that helps us tremendously in fighting spammers.

Most of you may not know, that a substantial part of running an ISP mail platform consists of fighting abuse.

Almost all abuse cases are caused by Spammers. Their main target is to control as many mailbox accounts as possible, where they can send spam from. Spammers can send Spam through bot-nets, acquire passwords through phishing attacks, or just do a smpt-auth brute-force attack.

Anyways, the Internet is 24x7, Spammers never sleep and we do neither. There is always someone on-call.

I want to show you, how we currently detect abuse on our platform, and what we do against it.

Detecting Abuse

There are many ways how to detect abuse. I don't want to go into too many details, and also, I don't want to reveal everything we do, as Spammers might read this :-)

One popular way is to count the number of "User unknowns" status messages from the remote mail server. Spammers own address-lists. Often the quality of these lists is bad and these lists contain email addresses that don't exist anymore.

What we do is running a saved search periodically and if we count a certain amount of such user unknown error codes, we trigger an alarm by e-mail and SMS.

SMS Alarming

If the saved search triggers an alarm, it calls a perl script. The script gets different arguments, like the job-id for this search. Using this information, a REST call is done to the Splunk Search-Head to give out the results for this search. The results are formated in XML, which then is parsed using XPATH. After that, an email message is send to an SMS-Gatewas resulting in following alarm on a mobile phone.



The SMS shows basic information, so that the poor guy on-call can decide if it is a false positive or not. If it's obviously spam, things get interesting.

Finding the Abuser
As I mentioned earlier, Splunk does not only trigger an SMS, but also sends an email for this alert.



Clicking on the result link opens the Splunk Web Interface.



For some searches you can just click on any field to drill down to the events, but because we use "fillnull", we can't do that. Instead we will just modify the search slightly. This will give us all events.



From here we can select the options for the field "transaction_id". You can see an option "Show This Transaction Only". Pressing this will open a single transaction:



Blocking the Abuser

Now comes the interesting part. You can see two options, "Block smtp auth", and "Block webmail". Clicking on these, will issue an HTTP Post to another system, blocking all access to this account.

Job almost done.

Informing the Account Owner

As all access is blocked now, we will report the incident to our abuse team. The abuse team's tracking system will receive all the necessary information, as well as the original log-entries as an evidence. After that, the account owner will be informed about this abuse case.



Summary

Splunk 4.1 makes life for us much easier. We don't have to copy-paste information into various applications. Everything can be done by simply clicking on it.

Comments

Dan Goldburt said…
This comment has been removed by the author.
April 17, 2010 at 9:41 PM
Anonymous said…
How are you able to create a workflow for notification? This is amazing!
April 19, 2010 at 8:38 PM
Unknown said…
Hi Mika,

Nice blog-entry. We've to speak once later about this...

Pascal
April 20, 2010 at 7:04 AM
Chris said…
Hi Mika,
the new workflow feature does make Life easier. I wish I had more time to play with it myself. Nice blog entry.
April 21, 2010 at 9:42 AM
Post a Comment

Popular posts from this blog

Heating up the Data Pipeline (Part 1)

Image
Pre-Processing Data I often hear the question from our customers, how data can be transformed prior to indexing in Splunk. Damien from  Baboonbones  has done a tremendous job in creating add-ons   providing custom inputs for Splunk. Most of his custom inputs provide the means to pre-process data by allowing custom event handlers to be written. Sometimes you still want to pre-process data that gets collected from Splunk's standard input types, like file monitors, Windows EventLogs, scripted inputs etc. Also, not everyone is capable of writing custom event handlers. A requirement these customers have, is that they have rolled out a large number of Splunk Universal Forwarders and they do not want to install another agent. To summarize, the solution capable of pre-processing data, should be easy to use, be easily integrated and be build on top of their existing architecture. How to plumb Splunk Pipelines Splunk has its own fittings to connect a Univer...
Read more

Heating up the Data Pipeline (Part 3)

Image
Welcome back to the "Heating up the Data Pipeline" blog series. In part 1 we talked about how to route data from Splunk to a 3rd party system. In part 2 walked through a simple data flow that passes data collected from Splunk Forwarders through Apache NiFi back to Splunk over the HTTP Event Collector. In this part, we will look at a more complex use case, where we route events to an index, based on the sending host's classification. The classification will be looked up from a CSV file. We will make use of Apache NiFi's new Record-Oriented  data handling capabilities, which will look initially a bit more complicated, but once you grasp it, it will make further Use Cases easier and faster to build. High-Level Dataflow We will again start with our ListentTCP input, but this time we will send the data to another Processor Group. The Processor Group will again emit events suitable for Splunk HEC. Note that we have increased the Max Batch Size from 1 ...
Read more

Heating up the Data Pipeline (Part 2)

Image
In Part 1 we went through how to route events from Splunk to a 3rd party system without losing metadata. Now I'll show you how events can be transformed using Apache NiFi and be sent back to Splunk into the  HTTP Event Collector . Note: The following is not a step-by-step documentation. To learn how to use Apache NiFi you should read the G etting Started Guide . Simple Pass-Through Flow As a first exercise we will create a simple flow, that only passes data through NiFi, without applying any complex transformations. The following picture shows a high-level NiFi flow, that receives events in our custom uncooked event format with a TCP listener, then sends the data further into a transformation "black box" (aka Processor Group), which emits events in a format, that can be ingested into a Splunk HTTP Event Collector input. Apache Nifi currently provides a rapidly growing number of processors (currently 266), which can be used for data ingestion, transfo...
Read more