Troubleshooting

This guide will walk through some general troubleshooting tips with your Anchore Enterprise instance. When troubleshooting Anchore Enterprise, the recommend approach is to first verify all Anchore services are up, use the event subsystem to narrow down particular issues, and then navigate to the logs for specific services to find out more information.

Throughout this guide, AnchoreCTL commands will be executed to assist with troubleshooting. For more information on AnchoreCTL, please reference the CLI section.

Verifying Services
Events
Logs
Removing a Repository and Images

Verifying Healthy Service Status

There is a minimum set of anchore service components that must be running for a deployment to be functional:

Service analyzer
Service policy_engine
Service catalog
Service apiext
Service simplequeue

You can verify which services have registered themselves successfully, along with their status, by running: anchorectl system status

# anchorectl system status
 ✔ Status system
┌─────────────────┬────────────────────┬─────────────────────────────┬──────┬────────────────┬────────────┬──────────────┐
│ SERVICE         │ HOST ID            │ URL                         │ UP   │ STATUS MESSAGE │ DB VERSION │ CODE VERSION │
├─────────────────┼────────────────────┼─────────────────────────────┼──────┼────────────────┼────────────┼──────────────┤
│ reports         │ anchore-quickstart │ http://reports:8228         │ true │ available      │ 0.0.10     │ 4.0.3        │
│ apiext          │ anchore-quickstart │ http://api:8228             │ true │ available      │ 0.0.17     │ 4.0.3        │
│ rbac_manager    │ anchore-quickstart │ http://rbac-manager:8228    │ true │ available      │ 0.0.10     │ 4.0.3        │
│ rbac_authorizer │ anchore-quickstart │ http://rbac-authorizer:8228 │ true │ available      │ 0.0.10     │ 4.0.3        │
│ reports_worker  │ anchore-quickstart │ http://reports_worker:8228  │ true │ available      │ 0.0.10     │ 4.0.3        │
│ simplequeue     │ anchore-quickstart │ http://queue:8228           │ true │ available      │ 0.0.17     │ 4.0.3        │
│ analyzer        │ anchore-quickstart │ http://analyzer:8228        │ true │ available      │ 0.0.17     │ 4.0.3        │
│ policy_engine   │ anchore-quickstart │ http://policy-engine:8228   │ true │ available      │ 0.0.17     │ 4.0.3        │
│ notifications   │ anchore-quickstart │ http://notifications:8228   │ true │ available      │ 0.0.10     │ 4.0.3        │
│ catalog         │ anchore-quickstart │ http://catalog:8228         │ true │ available      │ 0.0.17     │ 4.0.3        │
└─────────────────┴────────────────────┴─────────────────────────────┴──────┴────────────────┴────────────┴──────────────┘

Note: If specific services are down, you can investigate the logs for the services. For more information, see Logs.

The -vvv and –json options

Passing the -vvv option to AnchoreCTL can often help narrow down particular issues by displaying the client configuration and client functions as they are running:

# Example system status with -vvv

# anchorectl -vvv system status
[0000]  INFO anchorectl version: 1.0.0-rc3
[0000] DEBUG application config:
  url: http://localhost:8228
  username: admin
  password: '******'
...
[0000] DEBUG command config:
  format: text
[0000] DEBUG checking if new version of anchorectl is available
[0000] TRACE worker stopped component=eventloop
[0000] TRACE bus stopped component=eventloop
┌─────────────────┬────────────────────┬─────────────────────────────┬──────┬────────────────┬────────────┬──────────────┐
│ SERVICE         │ HOST ID            │ URL                         │ UP   │ STATUS MESSAGE │ DB VERSION │ CODE VERSION │
├─────────────────┼────────────────────┼─────────────────────────────┼──────┼────────────────┼────────────┼──────────────┤
│ reports_worker  │ anchore-quickstart │ http://reports_worker:8228  │ true │ available      │ 0.0.10     │ 4.0.3        │
│ simplequeue     │ anchore-quickstart │ http://queue:8228           │ true │ available      │ 0.0.17     │ 4.0.3        │
│ analyzer        │ anchore-quickstart │ http://analyzer:8228        │ true │ available      │ 0.0.17     │ 4.0.3        │
...
...

Passing the --json option to AnchoreCTL commands will output the API response data in JSON, which often contains much more information than what the CLI outputs by default for both regular successful operations, and for operations that are resulting in an error:

# anchorectl -o json system status
 ✔ Status system
{
  "serviceStates": [
    {
      "baseUrl": "http://reports_worker:8228",
      "hostid": "anchore-quickstart",
      "serviceDetail": {
...
...

Events

If you’ve successfully verified that all of the Anchore Enterprise services are up, but are still running into issues operating Anchore, a good place check is the event log.

The event log subsystem provides users with a mechanism to inspect asynchronous events occurring across various Anchore Enterprise services. Anchore events include periodically-triggered activities such as vulnerability data feed sync in the policy_engine service, image analysis failures originating from the analyzer service, and other informational or system fault events. The catalog service may also generate events for any repositories or image tags that are being watched when Anchore Enterprise encounters connectivity, authentication, authorization, or other errors in the process of checking for updates.

The event log is aimed at troubleshooting most common failure scenarios, especially those that happen during asynchronous operations, and to pinpoint the reasons for failures that can be used subsequently to help with corrective actions. Events can be cleared from Anchore Enterprise in bulk or individually.

Viewing Events

Running the following command will give a list of recent Anchore events: anchorectl event list

# Viewing list of recent Anchore events

# anchorectl event list
 ✔ List events
┌──────────────────────────────────┬──────────────────────────────────────────────┬───────┬───────────────────────────────────────────────────────┬─────────────────┬────────────────┬────────────────────┬─────────────────────────────┐
│ UUID                             │ EVENT TYPE                                   │ LEVEL │ RESOURCE ID                                           │ RESOURCE TYPE   │ SOURCE SERVICE │ SOURCE HOST        │ TIMESTAMP                   │
├──────────────────────────────────┼──────────────────────────────────────────────┼───────┼───────────────────────────────────────────────────────┼─────────────────┼────────────────┼────────────────────┼─────────────────────────────┤
│ 329ff24aa77549458e2656f1a6f4c98f │ system.image_analysis.registry_lookup_failed │ error │ dockerr.io/alpine:3.4                                 │ image_reference │ catalog        │ anchore-quickstart │ 2022-08-24T22:08:29.026352Z │
│ 4010f105cf264be6839c7e8ca1a0c46e │ system.image_analysis.registry_lookup_failed │ error │ dockerr.io/alpine:latest                              │ image_reference │ catalog        │ anchore-quickstart │ 2022-08-24T22:08:28.991101Z │
│ 6924eb83313746ff8b842a88654e3ac1 │ system.image_analysis.registry_lookup_failed │ error │ dockerr.io/alpine:3.12                                │ image_reference │ catalog        │ anchore-quickstart │ 2022-08-24T22:08:28.956321Z │
│ efdcf727647c458f85cb6464926e474d │ system.image_analysis.registry_lookup_failed │ error │ dockerr.io/nginx:latest                               │ image_reference │ catalog        │ anchore-quickstart │ 2022-08-24T22:08:28.920222Z │
...
│ 1eb04509b2bc44208cdc7678eaf76fef │ user.image.analysis.completed                │ info  │ docker.io/ubuntu:latest                               │ image_tag       │ analyzer       │ anchore-quickstart │ 2022-08-24T22:06:13.736004Z │
│ 6f735f8db7e84ce19b221d3b024318af │ user.image.analysis.processing               │ info  │ docker.io/ubuntu:latest                               │ image_tag       │ analyzer       │ anchore-quickstart │ 2022-08-24T22:06:13.128912Z │
│ 480eb191f87440b48c9f8cfa6529badf │ user.image_tag.added                         │ info  │ docker.io/ubuntu:latest                               │ image_tag       │ catalog        │ anchore-quickstart │ 2022-08-24T22:06:08.307039Z │
...
└──────────────────────────────────┴──────────────────────────────────────────────┴───────┴───────────────────────────────────────────────────────┴─────────────────┴────────────────┴────────────────────┴─────────────────────────────┘

Details about a specific event

If you would like more information about a specific event, you can run the following command: anchorectl event get <event-id>

# Details about a specific Anchore event

# anchorectl event get 1eb04509b2bc44208cdc7678eaf76fef
 ✔ Fetched event
UUID: 1eb04509b2bc44208cdc7678eaf76fef
Event:
  Event Type: user.image.analysis.completed
  Level: info
  Message: Image analysis available
  Resource:
    Resource ID: docker.io/ubuntu:latest
    Resource Type: image_tag
    User Id: admin
  Source:
    Source Service: analyzer
    Base Url: http://analyzer:8228
    Source Host: anchore-quickstart
    Request Id:
  Timestamp: 2022-08-24T22:06:13.736004Z
  Category:
  Details:
Created At: 2022-08-24T22:06:13.832881Z

Note: Depending on the output from the detailed events, looking into the logs for a particular servicename (example: policy_engine) is the next troubleshooting step.

Logs

Anchore services produce detailed logs that container information about user interactions, internal processes, warnings and errors. The verbosity of the logs is controlled using the log_level setting in config.yaml (for manual installations) or the corresponding ANCHORE_LOG_LEVEL environment variable (for docker-compose or Helm installations) for each service.

The log levels are DEBUG, INFO, WARN, ERROR, and FATAL, where the default is INFO. Most of the time, the default level is sufficient as the logs will container WARN, ERROR and FATAL messages as well. But for deep troubleshooting, it is always recommended to increase the log level to DEBUG in order to ensure the availability of the maximum amount of information.

Anchore logs can be accessed by inspecting the docker logs for any anchore service container using the regular docker logging mechanisms, which typically default to displaying to the stdout/stderr of the containers themselves - for example:

# docker ps
...
33c809f1803a        anchore/anchore-engine:latest       "/docker-entrypoint.…"   22 hours ago        Up 22 hours (healthy)   8228/tcp                         aevolume_engine-catalog_1
...

# docker logs aevolume_engine-analyzer_1
[service:worker] 2019-12-06 00:54:20+0000 [-] [MainThread] [anchore_engine.twisted/makeService()] [INFO] Initializing configuration
[service:worker] 2019-12-06 00:54:20+0000 [-] [MainThread] [anchore_engine.twisted/makeService()] [INFO] Initializing logging
[service:worker] 2019-12-06 00:54:20+0000 [-] [MainThread] [anchore_engine.service/initialize()] [DEBUG] Invoking instance-specific handler registration
[service:worker] 2019-12-06 00:54:20+0000 [-] [MainThread] [anchore_engine.service/_register_instance_handlers()] [INFO] Registering api handlers
[service:worker] 2019-12-06 00:54:20+0000 [-] [MainThread] [anchore_engine.service/_process_stage_handlers()] [INFO] Processing init handlers for bootsrap stage: pre_config
[service:worker] 2019-12-06 00:54:20+0000 [-] [MainThread] [anchore_engine.service/_process_stage_handlers()] [DEBUG] Executing 0 stage pre_config handlers
[service:worker] 2019-12-06 00:54:20+0000 [-] [MainThread] [anchore_engine.service/_configure()] [INFO] Loading and initializing global configuration
[service:worker] 2019-12-06 00:54:20+0000 [-] [MainThread] [anchore_engine.service/_configure()] [INFO] Configuration complete
...
...

The logs themselves are also persisted as logfiles inside the Anchore service containers. Executing a shell into any Anchore service container and navigating to /var/log/anchore, you will find the service log files. For example, using the same analyzer container service as described previously.

# docker exec -t -i  aevolume_engine-analyzer_1 /bin/bash
[anchore@687818c10b93 anchore-engine]$ cat /var/log/anchore/anchore-worker.log
2019-12-06 00:54:20+0000 [-] [MainThread] [anchore_engine.twisted/makeService()] [INFO] Initializing configuration
2019-12-06 00:54:20+0000 [-] [MainThread] [anchore_engine.twisted/makeService()] [INFO] Initializing logging
2019-12-06 00:54:20+0000 [-] [MainThread] [anchore_engine.service/initialize()] [DEBUG] Invoking instance-specific handler registration
2019-12-06 00:54:20+0000 [-] [MainThread] [anchore_engine.service/_register_instance_handlers()] [INFO] Registering api handlers
2019-12-06 00:54:20+0000 [-] [MainThread] [anchore_engine.service/_process_stage_handlers()] [INFO] Processing init handlers for bootsrap stage: pre_config
2019-12-06 00:54:20+0000 [-] [MainThread] [anchore_engine.service/_process_stage_handlers()] [DEBUG] Executing 0 stage pre_config handlers
2019-12-06 00:54:20+0000 [-] [MainThread] [anchore_engine.service/_configure()] [INFO] Loading and initializing global configuration
2019-12-06 00:54:20+0000 [-] [MainThread] [anchore_engine.service/_configure()] [INFO] Configuration complete
...
...

Removing a Repository and Images

There may be a time when you wish to stop a repository analysis while the analysis is running, such as accidentally watching an image with a large number of tags. For more information, see Removing a Repository and All Images.

Warning: Be careful when deleting images. In this flow, Anchore deletes the image, not just the repository/tag combo. Because of this, deletes may impact more than the expected repository since an image may have tags in multiple repositories or even registries.

Last modified October 25, 2023