Splunk Integration

Overview

This guide covers integrating Anchore Enterprise data streaming with Splunk using the HTTP Event Collector (HEC). Once configured, vulnerability reports and policy evaluations will flow into Splunk for search, alerting, and dashboard visualization.

Prerequisites

Architecture

┌─────────────────────┐       ┌─────────────────────┐       ┌─────────────────────┐
│  Anchore Enterprise │       │     Fluent Bit      │       │       Splunk        │
│                     │       │                     │       │                     │
│  Reports Worker     │──────►│  Tail + JSON Parse  │──────►│  HTTP Event         │
│  Event Files        │ NDJSON│  Splunk Output      │ HTTPS │  Collector (HEC)    │
│                     │       │                     │       │                     │
└─────────────────────┘       └─────────────────────┘       └─────────────────────┘
                                                                     │
                                                                     ▼
                                                            ┌─────────────────────┐
                                                            │  Splunk Index       │
                                                            │  - Search           │
                                                            │  - Dashboards       │
                                                            │  - Alerts           │
                                                            └─────────────────────┘

Splunk Configuration

Step 1: Enable HTTP Event Collector

Enable HEC globally in Splunk:

Via Splunk Web UI:

  1. Navigate to Settings > Data Inputs > HTTP Event Collector
  2. Click Global Settings
  3. Set All Tokens to Enabled
  4. Configure Default Source Type to anchore:events
  5. Click Save

Via REST API:

curl -k -u admin:<password> -X POST \
  https://<splunk-host>:8089/servicesNS/nobody/splunk_httpinput/data/inputs/http/http/enable

Step 2: Create HEC Token

Create a dedicated HEC token for Anchore events:

Via Splunk Web UI:

  1. Navigate to Settings > Data Inputs > HTTP Event Collector
  2. Click New Token
  3. Configure:
    • Name: anchore_events
    • Source type: anchore:events
    • Index: main (or create a dedicated index)
  4. Click Submit
  5. Copy the generated token value

Via REST API:

curl -k -u admin:<password> -X POST \
  https://<splunk-host>:8089/servicesNS/nobody/splunk_httpinput/data/inputs/http \
  -d "name=anchore_events" \
  -d "sourcetype=anchore:events" \
  -d "index=main"

The response will include the token value.

For better data management, create a dedicated index for Anchore events:

Via Splunk Web UI:

  1. Navigate to Settings > Indexes
  2. Click New Index
  3. Configure:
    • Index Name: anchore_events
    • Max Size: Based on your retention needs
  4. Click Save
  5. Update your HEC token to use this index

Via REST API:

curl -k -u admin:<password> -X POST \
  https://<splunk-host>:8089/services/data/indexes \
  -d "name=anchore_events" \
  -d "datatype=event"

Fluent Bit Configuration

Configure Fluent Bit to forward events to Splunk HEC:

[OUTPUT]
    Name            splunk
    Match           anchore.events
    Host            ${SPLUNK_HEC_HOST}
    Port            ${SPLUNK_HEC_PORT}
    TLS             On
    TLS.Verify      On
    Splunk_Token    ${SPLUNK_HEC_TOKEN}
    Splunk_Send_Raw Off
    Event_Host      anchore-enterprise
    Event_Sourcetype anchore:events
    Event_Index     anchore_events
    Retry_Limit     5

Configuration Options

ParameterDescriptionExample
HostSplunk HEC hostnamesplunk.example.com
PortSplunk HEC port8088
TLSEnable TLS encryptionOn
TLS.VerifyVerify TLS certificatesOn
Splunk_TokenHEC authentication tokenyour-token-here
Splunk_Send_RawSend raw JSON eventsOff
Event_HostHost field value in Splunkanchore-enterprise
Event_SourcetypeSourcetype for eventsanchore:events
Event_IndexTarget Splunk indexanchore_events
Retry_LimitNumber of retry attempts5

Environment Variables

Set these environment variables for Fluent Bit:

VariableDescriptionExample
SPLUNK_HEC_HOSTSplunk HEC hostnamesplunk.example.com
SPLUNK_HEC_PORTSplunk HEC port8088
SPLUNK_HEC_TOKENHEC authentication tokenyour-hec-token

TLS Configuration

For production deployments, always enable TLS verification:

[OUTPUT]
    Name            splunk
    ...
    TLS             On
    TLS.Verify      On
    TLS.CA_File     /path/to/ca-bundle.crt

If using self-signed certificates (not recommended for production):

[OUTPUT]
    Name            splunk
    ...
    TLS             On
    TLS.Verify      Off

Verification

Step 1: Test HEC Connectivity

Test the HEC endpoint directly:

curl -k -X POST "https://<splunk-host>:8088/services/collector/event" \
  -H "Authorization: Splunk <your-token>" \
  -d '{"event": "test event from anchore"}'

Expected response:

{"text":"Success","code":0}

Step 2: Check Fluent Bit Logs

Verify Fluent Bit is connecting to Splunk:

# Kubernetes
kubectl logs <fluent-bit-pod> | grep -i splunk

# Docker
docker logs <fluent-bit-container> 2>&1 | grep -i splunk

Look for:

  • [output:splunk:splunk.0] worker #0 started
  • No connection errors

Step 3: Search for Events in Splunk

Run a search in Splunk to verify events are arriving:

index=anchore_events sourcetype="anchore:events"

Or search for specific event types:

index=anchore_events event="image.vulnerability_report"

index=anchore_events event="tag.policy_evaluation"

Event Schema

Vulnerability Report Event

{
  "event": "image.vulnerability_report",
  "timestamp": "2024-01-15T10:30:45.123Z",
  "account_name": "admin",
  "resource_id": "sha256:abc123...",
  "payload": {
    "image_digest": "sha256:abc123...",
    "total_added": 15,
    "total_removed": 3,
    "added": [
      {
        "vulnerability_id": "CVE-2024-1234",
        "severity": "Critical",
        "package_name": "openssl",
        "package_version": "1.1.1k",
        "fixed_in": "1.1.1l",
        "url": "https://nvd.nist.gov/vuln/detail/CVE-2024-1234"
      }
    ],
    "removed": []
  }
}

Policy Evaluation Event

{
  "event": "tag.policy_evaluation",
  "timestamp": "2024-01-15T10:31:00.456Z",
  "account_name": "admin",
  "resource_id": "docker.io/library/alpine:latest",
  "payload": {
    "result": "fail",
    "policy_id": "default",
    "image_digest": "sha256:abc123...",
    "findings": [
      {
        "gate": "vulnerabilities",
        "trigger": "package",
        "action": "stop",
        "message": "Critical vulnerability found: CVE-2024-1234"
      }
    ]
  }
}

Splunk Searches

Basic Searches

All Anchore Events:

index=anchore_events sourcetype="anchore:events"

Vulnerability Reports Only:

index=anchore_events event="image.vulnerability_report"

Policy Evaluations Only:

index=anchore_events event="tag.policy_evaluation"

Failed Policy Evaluations:

index=anchore_events event="tag.policy_evaluation" payload.result="fail"

Vulnerability Analysis

Critical Vulnerabilities:

index=anchore_events event="image.vulnerability_report"
| spath path=payload.added{} output=vulns
| mvexpand vulns
| spath input=vulns
| where severity="Critical"
| table _time, account_name, resource_id, vulnerability_id, package_name, fixed_in

Top 10 Most Common CVEs:

index=anchore_events event="image.vulnerability_report"
| spath path=payload.added{} output=vulns
| mvexpand vulns
| spath input=vulns
| stats count by vulnerability_id
| sort -count
| head 10

Vulnerabilities by Severity:

index=anchore_events event="image.vulnerability_report"
| spath path=payload.added{} output=vulns
| mvexpand vulns
| spath input=vulns
| stats count by severity
| sort -count

Images with Most Vulnerabilities:

index=anchore_events event="image.vulnerability_report"
| stats sum(payload.total_added) as total_vulns by resource_id
| sort -total_vulns
| head 10

Policy Analysis

Policy Violations by Gate:

index=anchore_events event="tag.policy_evaluation" payload.result="fail"
| spath path=payload.findings{} output=findings
| mvexpand findings
| spath input=findings
| stats count by gate
| sort -count

Recent Policy Failures:

index=anchore_events event="tag.policy_evaluation" payload.result="fail"
| table _time, account_name, resource_id, payload.policy_id
| sort -_time
| head 20

Dashboards

Creating a Vulnerability Dashboard

Create a new dashboard in Splunk with the following panels:

Panel 1: Vulnerability Count Over Time

index=anchore_events event="image.vulnerability_report"
| timechart sum(payload.total_added) as "New Vulnerabilities"

Panel 2: Severity Distribution

index=anchore_events event="image.vulnerability_report"
| spath path=payload.added{} output=vulns
| mvexpand vulns
| spath input=vulns
| stats count by severity

Panel 3: Top Vulnerable Images

index=anchore_events event="image.vulnerability_report"
| stats sum(payload.total_added) as vulns by resource_id
| sort -vulns
| head 10

Creating a Policy Compliance Dashboard

Panel 1: Pass/Fail Ratio

index=anchore_events event="tag.policy_evaluation"
| stats count by payload.result

Panel 2: Policy Compliance Over Time

index=anchore_events event="tag.policy_evaluation"
| timechart count by payload.result

Panel 3: Recent Failures

index=anchore_events event="tag.policy_evaluation" payload.result="fail"
| table _time, account_name, resource_id, payload.policy_id
| sort -_time

Alerting

Critical Vulnerability Alert

Create an alert for new critical vulnerabilities:

Search:

index=anchore_events event="image.vulnerability_report"
| spath path=payload.added{} output=vulns
| mvexpand vulns
| spath input=vulns
| where severity="Critical"
| stats count as critical_count by resource_id
| where critical_count > 0

Alert Settings:

  • Trigger: Number of results > 0
  • Throttle: 1 hour per resource_id
  • Action: Email, Slack, or PagerDuty

Policy Failure Alert

Create an alert for policy failures:

Search:

index=anchore_events event="tag.policy_evaluation" payload.result="fail"
| stats count by resource_id, payload.policy_id

Alert Settings:

  • Trigger: Number of results > 0
  • Throttle: Based on your requirements
  • Action: Your preferred notification method

Troubleshooting

No Events in Splunk

  1. Verify HEC is enabled:

    curl -k "https://<splunk-host>:8089/services/data/inputs/http?output_mode=json" \
  -u admin:<password>

  2. Test HEC endpoint:

    curl -k -X POST "https://<splunk-host>:8088/services/collector/event" \
  -H "Authorization: Splunk <token>" \
  -d '{"event": "test"}'

  3. Check Fluent Bit logs for errors:

    docker logs <fluent-bit-container> 2>&1 | tail -50

  4. Verify network connectivity:

    # From Fluent Bit container
curl -k https://<splunk-host>:8088/services/collector/health

Authentication Errors

If you see 401 Unauthorized errors:

  1. Verify the HEC token is correct
  2. Check the token is enabled in Splunk
  3. Ensure the token has permission to write to the target index

TLS Errors

If you see certificate errors:

  1. Verify the CA certificate is correct
  2. Check certificate chain is complete
  3. For testing only: Set TLS.Verify Off (not recommended for production)

Missing Fields

If fields are not appearing in Splunk:

  1. Verify the sourcetype is set correctly
  2. Check field extractions in Splunk
  3. Use spath command to extract JSON fields in searches

Performance Tuning

High Volume Environments

For high-volume deployments:

  1. Increase Fluent Bit workers:

    [SERVICE]
    Workers     4

  2. Enable compression:

    [OUTPUT]
    Name            splunk
    ...
    compress        gzip

  3. Batch events:

    [OUTPUT]
    Name            splunk
    ...
    Batch_Size      2048

Splunk Indexer Optimization

  1. Create a dedicated index for Anchore events
  2. Configure appropriate retention policies
  3. Consider using indexed extractions for frequently searched fields

Next Steps

Last modified January 15, 2026