Write Unraid Docker Logs to Splunk HTTP Event Collector (HEC)

Prerequisites

  • Splunk Service with IP (Cloud or Docker container works)
    • Mine is a docker container inside Unraid server at http://192.168.68.46:8000
  • Docker container that you want to information logged to

What is Splunk?

Splunk is used for monitoring and searching through big data. It indexes and correlates information in a container that makes it searchable, and makes it possible to generate alerts, reports and visualizations.

https://cloudian.com/guides/splunk-big-data/splunk-data-analytics-splunk-enterprise-or-splunk-hunk/

Essentially it is a place to collect and gather all your logs from different places which allows you to consolidate them and search them for specific fields.

Docker Container

In this Docker Container I have an .NET Application which utilizes Hangfire to run scheduled tasks at a specific interval. The contents of the docker container and what it does really doesn’t matter at all, as long as it logs and follows some sort of format that you can parse. I will be following the container below for C# which utilizes Nlog to format all of my output into a consistent format.

NLog Logging Initialization
using NLog;
using NLog.Config;
using NLog.Targets;

namespace Aincrad.Common.Utilities
{
    public class AincradLogging
    {
        public static void Initialize()
        {
            var config = new LoggingConfiguration();
            var consoleTarget = new ConsoleTarget
            {
                Name = "console",
                Layout = "${date}|${level:uppercase=true}|${callsite:includeNamespace=false:className=true:fileName=false:includeSourcePath=false:methodName=false}|${message}",
            };
            config.AddTarget(consoleTarget);
            config.AddRuleForAllLevels(consoleTarget);
            LogManager.Configuration = config;
            LogManager.GetCurrentClassLogger().Info("Initialized Aincrad Logging!");
        }
    }
}

In the above code, the main line you care about is the Layout one. What it does is format the logs so that they look like the below logs.

2021/11/01 19:31:31.768|INFO|AincradLogging|Initialized Aincrad Logging!
2021/11/01 19:31:30.047|DEBUG|Program|The current jobs are currently scheduled to run.

This follows the format of <TIME>|<LOG_LEVEL>|<CALLING_CLASS>|<MESSAGE>

  • <TIME>- Time that the log is generated by the application, NOT the time it was received by splunk (though the same)
  • <LOG_LEVEL> – The severity of the log, DEBUG,INFO,ERROR,etc. You can sort in splunk on this field easily
  • <CALLING_CLASS> – In C#, I value knowing what the exact class is writing to the logger when debugging
  • <MESSAGE> – Whatever other information that you want to pass to the log.

Splunk Instance (Cloud, Local, or LAN)

For this it will work with any type of Splunk so you can host it anywhere. For me I have it in my docker container on my UnRAID server so that all my other containers are able to write to it.

  1. Navigate to your Splunk Web UI and log in at http://192.168.68.46:8000 (Port 8000 is default web port)

2. Navigate in the top menu bar to Settings > DATA > Data Inputs

3. Navigate to HTTP Event Collector, Recieve data over HTTP or HTTPs

4. Navigate to New Token in the top right corner of the page

5. Add a name and description for your collector, Click Next

6. Click through Next for Input Settings, Review, and Done. Default settings are fine

7. Save the Token Value in this final generation page, you will need that

8. Navigate to your docker container page and add the following arguments to your docker container

--log-driver=splunk --log-opt splunk-token=7fc97e9f-e1o6-4tr2-8d8a-0b51d11f83bd --log-opt splunk-url=http://192.168.68.46:8088
  • –log-driver=splunk – This tells the docker container to use a different logging driver
  • –log-opt splunk-token={TOKEN} – This is the Splunk token you got from HEC creation page
  • –log-opt splunk-url – The endpoint for your splunk instance, port defaults to 8088 (different from web ui)

9. Restart your container and log into Splunk and search at http://192.168.68.46:8000/en-US/app/search/search

source="http:siglerdev-blog"  | rex "\{\"line\":\"(?<time>.+)\|(?<log_level>.+)\|(?<Controller>.+)\|(?<Message>.+\}|.+?)\"" | fields - _time | table time log_level Controller Message | sort -time

Once you’ve reached this far you can throw it into a dashboard and do your own queries for it!

Thank you for reading this and let me know if you have any other questions.

Leave a Reply

Your email address will not be published.