Prometheus

Anchore Enterprise exposes prometheus metrics in the API of each service if the config.yaml that is used by that service has the metrics.enabled key set to true.

Each service exports its own metrics and is typically scraped by a Prometheus installation to gather the metrics. Anchore Enterprise does not aggregate or distribute metrics between services. You should configure your Prometheus deployment or integration to check each Anchore Enterprise service’s API using the same port it exports for the /metrics route.

Monitoring in Kubernetes and/or Helm Chart

Prometheus is very commonly used for monitoring Kubernetes clusters. Prometheus is supported by core Kubernetes services. There are many guides on using Prometheus to monitor a cluster and services deployed within, and also many other monitoring systems can consume Prometheus metrics.

The Anchore Enterprise Helm Chart includes a quick way to enable the Prometheus metrics on each service container:

• Set: helm install myanchore anchore/enterprise --set anchoreConfig.metrics.enabled=true

• Or, set it directly in your customized values.yaml

  ## @param anchoreConfig.metrics.enabled Enable Prometheus metrics for all Anchore services
  ## @param anchoreConfig.metrics.auth_disabled Disable auth on Prometheus metrics for all Anchore services
  ##
  metrics:
    enabled: true
    auth_disabled: false

To deploy the Prometheus container and expose it externally, you can configure an Ingress controller, such as NGINX, within the Helm chart values:

  # Prometheus Configuration
  prometheus:
    ingress:
      enabled: true
      ingressClassName: "nginx" # Specify your Ingress controller
      hosts:
        - anchoredemo.com # The hostname to access Prometheus
      paths: # Note: paths is typically an array
        - /prometheus
      pathType: Prefix # Or ImplementationSpecific, Exact. 'Prefix' is common.
    prometheusSpec:
      #retention: 10d
      externalUrl: "https://anchoredemo.com/prometheus" # Ensure this matches your ingress path
      routePrefix: "/prometheus"
      podMonitorSelectorNilUsesHelmValues: false
      podMonitorSelector:
        matchLabels:
          release: "kube-prom-stack"

Because Anchore Enterprise scales horizontally, there can often be multiple replicas of each service running at any given time. This scaling behavior makes it essential to use a monitoring approach that can dynamically discover and track all active pods, so, to monitor the Anchore Enterprise Pods we will be deploying a PodMonitor. Since Anchore Enterprise creates multiple instances of certain services (for example, analyzers), relying on static service discovery is not always the most practical method and therefore the PodMonitor is our recommended method for pod discovery/tracking.

The PodMonitor resource allows Prometheus to:

• Dynamically detect new or removed Anchore Enterprise pods as the deployment scales up or down.

• Automatically update its scrape configuration without manual intervention.

• Collect metrics directly from the /metrics endpoints of each pod, even when no Service is defined.

If you run the kubectl get svc command in your Anchore Enterprise namespace, you will notice that the analyzer does not have a service so it is not reachable through a loadbalancer. PodMonitor will figure this out for Prometheus as each pod uses a different port. The PodMonitor will find the Anchore Enterprise pods and edit the Prometheus config to allow it to automatically discover and scrape the metrics.

PodMonitor Example

You must set up the PodMonitor and configure the necessary secrets used to collect the metrics from /metric endpoints on the pod. The following is an example that can be used as a template for your own configuration:

kubectl apply -f - <<EOF
  apiVersion: monitoring.coreos.com/v1
  kind: PodMonitor
  metadata:
    name: anchore-enterprise-metrics
    namespace: monitoring
    labels:
      team: my-monitoring-team
      release: kube-prom-stack
  spec:
    namespaceSelector:
      matchNames:
        - anchore #Set this to match the namespace Anchore is deployed in
    selector:
      matchLabels:
        app.kubernetes.io/part-of: anchore # ADJUST THIS label selector
    podMetricsEndpoints:
    - path: /metrics # The endpoint path for metrics
      # By omitting the 'port' field, Prometheus will attempt to scrape 
      # the '/metrics' path on ALL declared ports of the selected pods.
      scheme: http # Assuming metrics are exposed over HTTP
      # Interval and scrapeTimeout values can be uncommented below and customized if needed
      # interval: 30s
      # scrapeTimeout: 10s
      basicAuth:
        username:
          name: anchore-metrics-credentials # Name of the Secret
          key: username # Key inside the Secret
        password:
          name: anchore-metrics-credentials # Name of the Secret
          key: password # Key inside the Secret
EOF

kubectl apply -f - <<EOF
# This secret stores the credentials that Prometheus will use
# to authenticate when scraping metrics from the Anchore service pods.
apiVersion: v1
kind: Secret
metadata:
  name: anchore-metrics-credentials
  namespace: monitoring # Must be the same namespace as the PodMonitor above
type: Opaque
data:
  # The username to use for Basic Auth (usually 'admin' for Anchore metrics endpoint)
  username: YWRtaW4= # Base64 of 'admin'
  
  # IMPORTANT: Replace <YOUR_BASE64_OUTPUT> with the Base64-encoded version of
  # the ANCHORE_ADMIN_PASSWORD you have configured in your deployment.
  # Keep the quotes, otherwise the value is not parsed as a string.
  password: "<YOUR_BASE64_OUTPUT>"
EOF

The Secret must be created in the same namespace as the PodMonitor, not the namespace Prometheus runs in. The Prometheus Operator resolves basicAuth credentials from the namespace of the PodMonitor that references them, so a Secret placed elsewhere is never found and the scrape fails to authenticate.

It is imperative that users carefully review the commented lines within the code snippet and modify the configuration to align with their specific environment. Specifically the selectors for the namespace (matchNames) and labels (matchLabels) to match your deployment, as they define the scope of the pods the PodMonitor tracks. Any new pods subsequently created with matching labels in the specified namespaces will be automatically discovered and monitored by the PodMonitor based on this configuration. For comprehensive details on the resource definition, refer to the official PodMonitor documentation here;

Grafana Dashboards

Anchore publishes a set of Grafana dashboards that visualize the metrics described on this page. They are built for a Kubernetes deployment scraped by a PodMonitor, as configured above.

Prerequisites

• Anchore Enterprise metrics are enabled (anchoreConfig.metrics.enabled=true).
• A PodMonitor is scraping the Anchore pods, as described in PodMonitor Example. The dashboards rely on the namespace and container labels that Prometheus adds automatically when scraping pods.
• Grafana has a Prometheus data source pointing at the Prometheus that performs the scraping.

The Anchore Enterprise dashboard also includes CPU and memory panels that read container_cpu_usage_seconds_total and container_memory_rss. These are exported by cAdvisor in the kubelet rather than by Anchore, and most Prometheus deployments in Kubernetes scrape them by default. If yours does not scrape the kubelet, those two panels stay empty and the rest of the dashboard is unaffected.

Import a Dashboard Using the Grafana UI

This works with any Grafana installation and is the simplest option.

1. In Grafana, select Dashboards, then New, then Import.
2. Upload the JSON file, or paste its contents into the import box.
3. Select your Prometheus data source when prompted, then select Import.

Import a Dashboard Using a ConfigMap

Many Grafana installations run a dashboard sidecar that imports any ConfigMap carrying a particular label. If yours does, you can deploy the dashboards as ConfigMaps so they survive a Grafana restart and can be managed with the rest of your manifests.

The sidecar decides which ConfigMaps to import using two settings: the namespace it watches and the label it selects on. Both vary by installation, so read them from the sidecar container before you create the ConfigMap:

kubectl get pod -n <grafana-namespace> -l app.kubernetes.io/name=grafana \
  -o jsonpath='{.items[0].spec.containers[?(@.name=="grafana-sc-dashboard")].env}'

Look for NAMESPACE, LABEL, and LABEL_VALUE in the output. A NAMESPACE of ALL means the sidecar imports matching ConfigMaps from any namespace. Then create and label the ConfigMap accordingly:

kubectl create configmap anchore-enterprise-k8s \
  --namespace <watched-namespace> \
  --from-file=anchore-enterprise-k8s.json

kubectl label configmap anchore-enterprise-k8s \
  --namespace <watched-namespace> \
  <label>=<label-value>

Dashboard Variables

Each dashboard exposes variables you can set from the dropdown menus at the top of the page:

The Total DB Connections panel on the Database dashboard reads pg_stat_database_numbackends, which is exported by postgres_exporter rather than by Anchore. If you do not run postgres_exporter, that single panel stays empty. Every other panel on the three dashboards uses metrics that Anchore exports itself.

The specific strategy for monitoring services with prometheus is outside the scope of this document. But, because Anchore exposes metrics on the /metrics route of all service ports, it should be compatible with most monitoring approaches (daemon sets, side-cars, etc).

For info on deploying Prometheus in a docker-compose deployment see Deploy using Docker Compose.

Metrics of Note

Anchore services export a range of metrics. The following list shows some Anchore services that can help you determine the health and load of an Anchore deployment.

• anchore_global_images_analysis_status
  • This metric provides a global count of images broken down by analysis status.
  • The analysis_status label indicates the state: not_analyzed, analyzing, analyzed, or analysis_failed.
  • As the not_analyzed count grows you can expect longer analysis times. Adding more analyzers to a system can help drain the queue faster and keep wait times to a minimum.
  • Example: anchore_global_images_analysis_status{analysis_status=“not_analyzed”} 185.0 anchore_global_images_analysis_status{analysis_status=“analyzing”} 1.0 anchore_global_images_analysis_status{analysis_status=“analyzed”} 452.0 anchore_global_images_analysis_status{analysis_status=“analysis_failed”} 0.0
  • This metric is exported from the catalog service and is based on the database state.
• anchore_monitor_runtime_seconds
  • This is a Summary metric that records the duration of each async watcher process as it executes on a duty cycle. The function label identifies the watcher and the status label indicates success or fail.
  • As the system grows, these durations will become longer to account for more tags to check for updates, repos to scan for new tags, and user notifications to process.
• anchore_tmpspace_available_bytes
  • This metric tracks the available space in the “tmp_dir” location for each container. This is most important for the instances that are analyzers where this can indicate how much disk is being used for analysis and how much overhead there is for analyzing large images.
  • This is expected to be consumed in cycles, with usage growing during analysis and then flushing upon completion. A consistent growth pattern here may indicate left over artifacts from analysis failures or a large layer_cache setting that is not yet full. The layer cache (see Layer Caching) is located in this space and thus will affect the metric.
• process_resident_memory_bytes
  • This is the memory actually consumed by the instance, where each instance is a service process of Anchore. Anchore is fairly memory intensive for large images and in deployments with lots of analyzed images due to lots of json parsing and marshalling, so monitoring this metric will help inform capacity requirements for different components based on your specific workloads. Lots of variables affect memory usage, so while we give recommendations in the Capacity Planning document, there is no substitute for profiling and monitoring your usage carefully.