01 October 2021

Working with Splunk

The other articles in this series:

• Splunk and Fluentd
• Rejecting Fluend

Configuration File Layout

In the documentation $SPLUNKHOME refers to the location Splunk is installed. Configuration is in $SPLUNKHOME/etc. Every folder has a default and optionally local subfolder, configuration you create or modify should only go in local in case an upgrade overwrites default, local takes precedence. Most of the configuration you create will be in etc/system/local. When you use the gui to create configuration the app that writes the configuration file will create a file in its local folder. I’ve found that these configs can safely get moved to system/local, and that the gui will even preserve the file location if I modify it later. Moving configuration requires a stop before making the change. As you become familiar with the configuration you will rely on find and grep to help locate where the configuration you seek currently resides.

Splunk has a lot of configuration files scattered about. If that weren’t bad enough, apps install their code into etc/apps, so when you try to put etc under version control you end up with a huge repository. I use a script to add just conf files. #!/bin/bash cd /opt/splunk/etc find . -name '*.conf' | xargs git add

Getting Splunk

Downloading Splunk requires registration of a Splunk account. You’ll then have to download each splunk and splunk forwarder package you need. This requires several clicks for each. If you want to download with the command line, you’ll need to cancel the download and cut and paste the wget script (which at least they put right on the page once you start the download).

Every time you need to update Splunk you’ll need to repeat this process.

You can use these links to shortcut in once you’ve registered with splunk:

https://www.splunk.com/en_us/download/splunk-enterprise.html https://www.splunk.com/en_us/download/universal-forwarder.html

Post Install Tasks

web.conf

Every time you access Splunk it will alert you about a new version, and new versions are released very frequently.

Add the following to $SPLUNKHOME/etc/system/local/web.conf

[settings]
# stop nuisance new release notices
updateCheckerBaseURL = 0
# running behind a proxy
tools.proxy.on = True

The second setting will be needed when you put Splunk behind a proxy. The free version has no access control, you need Apache or Nginx in front of it. The default port is 8000.

Local User, Accept License, and Switch the License

The splunk CLI requires an admin user to be created, and frequently requires logon by that user (the credential is cached for a limited time), Splunk have made it clear they will ever allow disabling this logon even for root. Every time you install or update Splunk, it is required to accept their license. Finally on your servers but not forwarders will always be installed with an enterprise trial license, using this license will result in your eventually being locked out of you data, switching to the free license is an important setup step to protect yourself.

Here are some task steps for ansible to help do this:

user-seed.conf

[user_info]
USERNAME = admin
PASSWORD = ***secret***

- name: seed admin user
  ansible.builtin.template:
    src: templates/user-seed.conf.j2
    dest: "/etc/system/local/user-seed.conf"
    owner: ""
    group: ""
- name: accept the license
  ansible.builtin.shell:
    cmd: ""
    tags: [skip_ansible_lint]
  loop:
     - "/bin/splunk stop"
     - "/bin/splunk start --accept-license --answer-yes"

# On servers only

- name: switch splunk license to free
  ansible.builtin.lineinfile:
    path: "/etc/system/local/server.conf"
    state: present
    regexp: '[license]'
    insertafter: EOF
    line: |
      [license]
      active_group = Free
      #
  when: splunk_server

Setting up Indexes

By default Splunk creates a main index, in the simplest configuration can send everything there. Setting up indexes for different events can improve performance and allows finer grained control over event retention.

The default index settings are to keep everything forever. Splunk subdivides indexes into buckets, when it is time to remove old events, Splunk can only delete entire buckets not individual events, for smaller installations the defaults will never roll buckets. If for an index you only want to keep 90 days, you need Splunk to use a bucket for no more than 90 days. Making your buckets too small can hurt performance.

The values for aging buckets (which are not available in the gui) are: frozenTimePeriodInSecs and maxHotSpanSecs. There was some issue that could come up if those numbers were round, so add a small random integer to your values. freezing buckets is archiving or deleting them, the HotSpanSecs is how long a bucket can remain in use.