I’ve recently been researching Splunk and have been impressed with its power, flexibility, and ease of use. This blog is not intended to be a step-by-step tutorial, but rather is aimed to show some initial findings, overview one way to integrate Splunk with DNN, and paint the picture of some potential use cases.

So What is Splunk?
If you don’t already know what Splunk is, Splunk is a software company based in San Francisco that produces software for searching, monitoring, and analyzing machine generated big data via a web style interface. Splunk’s software helps organizations with operational intelligence, log management, application management, enterprise security and compliance.

Installing Splunk was simple and after clicking around a little while it was evident that Splunk is an intuitive software. From a UI standpoint, it makes logical sense and the flow is easy to understand. And it didn’t take long to see and understand how powerful it is.

As you may imagine, I began to wonder if and how I could integrate Splunk with DNN.

DNN + Splunk: One Way to Connect the Two
One of Splunk’s powerful features is that it can literally suck in all types, styles, and formats of data. This data can be machine data, log files, or even data from a REST API. There are several mechanisms for getting data into Splunk, but for this scenario, DNN’s web API implementation makes this an easy fit. On the DNN side, a developer can easily create a custom module using web services to expose any DNN data on an endpoint, which Splunk can then access. If you’d like to go the custom module route, check out my other blog series on module development. However, I did not write a custom module to test the integration.

For my initial investigation into Splunk I chose to use DNN Sharp’s API Endpoint module as it allows easy configuration of end points. Splunk is architected to consume any type of data and then it makes that data extremely easy to search, create visualizations and/or alerts with. These searches, visualizations, and alerts can be very basic or very complex in nature.

Another thing to note is that Splunk is architected to do this at scale and can easily parse enormous amounts of data. For example, every time you drink from a Coca-Cola “Freestyle” machine at a fast food restaurant, the data from your drink selection is logged and Splunk helps analyze the data, denote trends, and sends alerts. So yes, those Coke machines (all across the world) are connected IOT devices and Coke is a Splunk customer. See how Coke is using Splunk in the Splunk Conf 2014 Keynote replay session. Imagine how much data that is on a global scaled --> Splunk is helping Coke make sense of it.

Side note: Check out the blog I wrote on using Particle & Splunk to monitor temperature

So, my first goal was simple: see if I could get data from DNN into Splunk.

Sticking along the thought process of “data logs” I figured why not expose the DNN event log on an endpoint and see what I could make happen. Obviously, the event log may not be the best use case as site administrators can clear logs or processes to automatically clear logs sometimes exist. However, for this initial test it is a good candidate. To get the event log data on an end point I used the DNN Sharp API Endpoint module to make a SQL query on the event log view and return it as JSON.

With the event log now sitting out there as JSON on a DNN end point now all I needed to do was get it into Splunk…

Getting REST Data Into Splunk
The Splunk side of this configuration only took a few minutes to configure and keep in mind I’m no Splunk guru (read, it’s easy!). Splunk is similar to DNN in that it’s extensible. Splunk extensions can be found on the Apps and Ad-Ons sections of the Splunk website. I tell you this because ultimately, I followed a blog by Damien Dallimore on getting REST data into Splunk which used a modular input extension and that was all it took. I simply completed the required fields in the Splunk REST Modular Input as shown below.

I chose to poll the data every 60 seconds. With this information inputted I clicked save and returned to the Data Inputs screen of Splunk and chose my newly created data source.

BOOM! I was seeing DNN event log info in Splunk!

Searching, Visualizations, & Alerts in Splunk
With data in Splunk now I needed to proceed to using Splunk to make sense of the data. Splunk’s searching functionality makes it very easy to search for, well... anything you'd like. I’m not yet knowledgeable enough to fully explain all the capabilities, but what I can easily see is that you can select your data source, click on keywords, add them to the source's search criteria and set your desired timeframe for the search. It’s feels as if you have a Google search bar and all your searches are performed on your data source and intellisense & syntax highlighting for your search are provided too!

Once you have a search returning data you can then create visualizations or alerts. And yes, there are tons of visualizations provided by Splunk. These visualizations can be saved as reports or live as “panels” that reside on dashboards. Dashboards can have as many panels as you want and you can have multiple dashboards if you like. Also, you can easily embed these panels into DNN or any other location by clicking the “convert to HTML” link that each panel has. Being able to display this info anywhere you like is a neat feature. Are your mental light bulbs turning on yet?

So, I created a few visualizations based on event log data that was available. I created a number-based-visualization to show a large number that represented a count of 404 errors, a line graph showing the number of failed logins, and a chart showing the 404’s over time. So, in just minutes Splunk was already helping me understand that I have some issues going on with one of my sites. I believe one reason for the 404's is that I've renamed some pages that I think bots are targeting trying to register. Anyways, I've got work to do... don't judge!

Opening Up Possibilities
Now you may be looking at this and thinking to yourself, yeah this is neat, but I could create a custom module to make something similar to this happen. And you would be correct, but keep in mind the potential use cases, flexibility, and scalability of Splunk in comparison to a custom module. You could easily have all your customers as data sources and create dashboards to help you (and your customers) quickly understand what’s going on with your customer's applications. You could also do data mashups of data from a DNN website/web app, some IOT device out in space, and any other data source you can think of to provide valuable insight. And again, Splunk has no problem doing this with massive amounts of data.

With just a little research into Splunk it didn’t take long to get my mind spinning with all the possibilities within DNN and beyond. Think about your current DNN use cases, requirements of your customers, and the exploding IOT market and you’ll soon see the light.

Here are some ideas I had right off the bat:

• Dashboards that visually communicate mission critical data to your client’s management tier
  • Think a specific page of dashboards only visible to your client’s leadership showing application performance dashboards.
  • Think about posting monitors in your client’s offices (or your office) showing performance or usability metrics 
  • Dashboard Examples:
    • Failed Logins – to denote potential bot attacks 
    • New Users – to denote growth or potential bot attacks 
    • 404’s – to denote when pages aren’t found, SEO issues, or other concerns 
    • Etc., etc., etc. 
• Alerts – Any type of alert you can imagine based on the data you expose 
• Proactive Client Consulting 
  • If you represent a web firm you can shift your business from being reactive to proactive by monitoring your client’s data and then reaching out to them when trends are noticed
    • Consider your SEO person suggesting redirects be put in place when 404’s for a specific page continuously occur
    • Consider shutting down account creation when a rapid account creation occurs over a short period of time in the scenario of bot registration attacks 
    • Have you developed custom modules for your clients? You could create visualizations and alerts from the data of those modules to help provide business intelligence & insights to your clients 
• IOT 
  • As I mentioned earlier I also wrote a blog on using Particle & Splunk to monitor temperature. Check it out for more potential use cases. 
  • Many clients are seeking or will be seeking IOT solutions in the future. If you are working with connected devices you could easily setup web services in DNN to log data from these microcontrollers, sensors, motors, or actuators. If you’re logging data into DNN (or any other location) you can easily feed that data to Splunk to parse and analyze the data. Presenting data from IOT devices coupled with data from your DNN instance could potentially provide a holistic picture for your client’s business and deliver critical insights to them.
    • DNN firms like nvisionative are already building IOT based solutions that integrate with DNN. Splunk opens even more possibilities for organizations like this.

Splunk FTW!
As you can see the power and flexibility Splunk provides is really nice. I believe Splunk could be a game-changer especially for those with large amounts of data to parse, anybody in the IOT space, and much more. I hope this blog has provided you with an introductory glimpse into some of the capabilities of Splunk and even got you thinking of potential ways to integrate Splunk into your applications or customer's environments. I am still learning about it and hope you will too. I know that I'm just scratching the surface here in my initial findings.

Find out more about Splunk at http://www.Splunk.com

In the past few years I’ve been getting into IOT. You may have seen tweets or blog about the Tech-Turkey project I’ve been working on or flame throwing pumpkins at Halloween. I’ve learned and used Arduinos, Raspberry Pi’s, and Particle Photons and Electrons. It has been fun to learn more and get into the connected world… the internet of things!

I’ve been keeping ServoCity in business and even recently worked to get a custom PCB created. Every step of the way I’ve been learning different things and realizing just how much more there is to learn. Recently I’ve started learning more about Splunk.

What is Splunk
If you don’t already know what Splunk is, Splunk is a software company based in San Francisco that produces software for searching, monitoring, and analyzing machine generated big data via a web style interface. Splunk’s software helps organizations with operational intelligence, log management, application management, enterprise security and compliance. 

Side note: In my first exploration into Splunk I wrote a blog about using Splunk with DNN that may interest you.

Particle & Splunk
DNN is a web application, but what if I wanted to get data from an IOT device? That’s when we call on Particle. If you’re not familiar with Particle, it makes it really easy to bring real world objects online. Particle is one of my favorite IOT platforms. It makes awesome microcontrollers, provides a nice IDE, has awesome documentation, and a great community. Connecting to Particle’s cloud is straight forward and even southerners can do it! See my presentation at our user group on DNN & Particle.

If you’re not familiar with Splunk, it makes it really easy to pull in data (machine data) and make sense of it. I’m talking about parsing vast amounts of data, creating visualizations and/or alerts and making it simple to understand. Even southerners can use it too!

Both Particle and Splunk are industry leaders and have some really big names behind their companies and as clients of their companies.

So why not bring Particle & Splunk together?

Reading Temperature with Particle
To use the awesomeness that both solutions bring us we’ll first need to read the temperature and post it to a webservice. Here again, Particle makes this easy. I used a basic temperature reader in a bread board layout for this experiment.

Then, in Particle’s IDE I used the basic tutorial level code to read an analog value and post it to a Particle cloud variable. Cloud variables are accessible via web services. That is, I can make a GET request and parse the JSON object to get the data. Epic.

And all you have to do is “flash” the firmware to the device and then hit the API endpoint. In my case I accessed the service and saw the following JSON

Now we were cooking with oil! The next step was to get this data into Splunk.

Getting Particle’s RESTful Data Into Splunk
Getting RESTful data into Splunk is really straightforward thanks to Splunk’s extensibility. Splunk has an extensions gallery that can be found on the Apps and Ad-Ons sections of the Splunk website. I tell you this because ultimately, I followed a blog by Damien Dallimore on getting REST data into Splunk which used a modular input extension and that was all it took. I simply completed the required fields in the Splunk REST Modular Input as shown below.

After clicking save, the data from my Particle temperature reader was showing up in Splunk!

Creating Dashboards from the Particle Data
Once data shows up in Splunk you can literally perform any search query you want on the data and create/configure dashboards, panels, reports, alerts and more. Splunk is very powerful in this regard and scale to infinity. However, for this scenario I just wanted log the temperature over time from one device, as well as the temperature’s highest, lowest, and average. Splunk, again, made this very simple.

After clicking on the “result” field I created some visualizations and voila! Out popped some neat dashboards showing all my data in a way that’s easy to understand.

If you are like me, you kind of want to see things in action. So for those of you like me who are visual learners, here’s a quick video of the solution in action.

An IOT Combination That’s Hard to Beat!
As you can see, both solutions are awesome and the opportunities are endless. Consider the possibilities here… Particle is easy to deploy and post data to the net and Splunk can easily connect, suck in data, and bring instant insights. The more data you give to Splunk the more knowledge you’re going to have. Splunk can handle this at scale too… I mean massive scale. Why not connect thousands of devices and pump all the data into Splunk and tune it to your liking! I believe that’s what they refer to as operational intelligence 😊 Now my mind is spinning with possibilities. Is yours?