Splunk LabOps: automated security telemetry for live Linux infrastructure.

Verified engineering results

The implementation combines working infrastructure, recorded ingestion results and automated collector tests.

Secure collection from an internet-facing Linux server

The Debian host initiates key-authenticated SSH connections to the Hetzner VPS to retrieve ssh.service journal records and /var/log/fail2ban.log. Both collectors send normalized events to Splunk's HTTPS Event Collector on loopback. Splunk Web is also bound to loopback; HEC credentials remain on the collector host, not the VPS.

The collector uses separate index-scoped HEC credentials, a configured CA and certificate fingerprint pinning. A dedicated certificate with a localhost IP subject alternative name is a future hardening improvement.

Data lifecycle and replay behavior

Original journal timestamps are retained in Splunk event time, and each event includes a journal cursor. A first run reads up to 24 hours of SSH history; later runs resume after the last saved cursor. The cursor is written atomically only after HEC returns a success response. This favors replay over silently dropping events if a submission is interrupted.

Combined configured index-size targets, including Splunk system indexes, are 12 GiB. They are retention settings, not a hard disk quota. Runtime data is persisted on the workstation's LUKS-encrypted ext4 filesystem.

Two live ingestion paths

Hetzner Debian VPS
  ├─ ssh.service journal → cursor-based Python collector
  └─ Fail2Ban log       → offset/rotation-aware Python collector
               ↓
     Authenticated HTTPS HEC
               ↓
       Splunk index: labops_vps
               ↓
      SPL security investigation

index=labops_vps (source="journal:ssh" OR source="fail2ban:log")
| table _time host source event_type src_ip message
| sort - _time

Engineering beyond installation

Container identity and filesystem permissions

Initial Splunk provisioning failed while reading /opt/splunk/etc/splunk.version. Diagnosis showed that the persisted top-level configuration directory was traversable by the Splunk UID but not by the initialization identity. I changed permissions only on that directory rather than recursively opening the tree or deleting persistent configuration. The documented subsequent Ansible run finished with failed=0.

Collector failure handling

The SSH and Fail2Ban collectors have 28 unit tests covering authentication parsing, IPv4/IPv6 extraction, original timestamps, incremental checkpoints, incomplete lines, rotation, truncation and failed HEC submissions. SSH collection saves its cursor after an accepted event. Fail2Ban collection maintains a separate state file with log identity and byte offset, processes complete lines and handles the immediate uncompressed rotated predecessor.

Structured SSH authentication analysis

Successful authentication events include parsed account name, authentication method, source IP and, where OpenSSH provides them, key algorithm and SHA-256 fingerprint. Splunk queries can group successful logins by account and key without relying exclusively on raw-message text.

index=labops_vps source="journal:ssh" event_type="ssh_authentication_success"
| where isnotnull(key_fingerprint)
| stats count as successful_logins dc(key_fingerprint) as distinct_keys by username auth_method key_type

Fail2Ban event ingestion

The second collector retrieves live Fail2Ban log entries over existing SSH access and classifies detected failures, bans, unbans, log rollover and service errors. A real log-rollover event was indexed and verified in Splunk; ban/unban scenarios are covered by synthetic tests rather than presented as observed attacks.

index=labops_vps source="fail2ban:log"
| table _time host event_type level component message

From collection to detection and response

Next milestones add Apache web-server telemetry, then SPL detections, investigation workflows and security response automation across the existing SSH and Fail2Ban data sources.

Related engineering projects

LabOps complements my AWS Infrastructure Automation & Kubernetes lab, Porter reliability platform, and T-Pot honeynet.