Introduction
The event log subsystem provides the users with a mechanism to inspect asynchronous events occurring across various Anchore Enterprise services. Anchore events include periodically triggered activities such as vulnerability data feed syncs in the policy-engine service, image analysis failures originating from the analyzer service, and other informational or system fault events. The catalog service may also generate events for any repositories or image tags that are being watched, when the engine encounters connectivity, authentication, authorization or other errors in the process of checking for updates. The event log is aimed at troubleshooting most common failure scenarios (especially those that happen during asynchronous engine operations) and to pinpoint the reasons for failures, that can be used subsequently to help with corrective actions. Events can be cleared from anchore-engine in bulk or individually.
The Anchore events (drawn from the event log) can be accessed through the Anchore Enterprise API and AnchoreCTL, or can be emitted as webhooks if your Anchore Enterprise is configured to send webhook notifications. For API usage refer to the document on using the Anchore Enterprise API.
Accessing Events
The anchorectl command can be used to list events and filter through the results, get the details for a specific event and delete events matching certain criteria.
# anchorectl event --help
Event related operations
Usage:
event [command]
Available Commands:
delete Delete an event by its ID or set of filters
get Lookup an event by its event ID
list Returns a paginated list of events in the descending order of their occurrence
Flags:
-h, --help help for event
Use " event [command] --help" for more information about a command.
For help regarding global flags, run --help on the root command
For a list of the most recent events:
anchorectl event list
✔ List events
┌──────────────────────────────────┬──────────────────────────────────────────────┬───────┬─────────────────────────────────────────────────────────────────────────┬─────────────────┬────────────────┬────────────────────┬─────────────────────────────┐
│ UUID │ EVENT TYPE │ LEVEL │ RESOURCE ID │ RESOURCE TYPE │ SOURCE SERVICE │ SOURCE HOST │ TIMESTAMP │
├──────────────────────────────────┼──────────────────────────────────────────────┼───────┼─────────────────────────────────────────────────────────────────────────┼─────────────────┼────────────────┼────────────────────┼─────────────────────────────┤
│ 8c179a3b27a543fe9285cf4feb65561d │ system.image_analysis.registry_lookup_failed │ error │ docker.io/alpine:3.4 │ image_reference │ catalog │ anchore-quickstart │ 2022-08-24T23:08:30.54001Z │
│ 48c18a84575d45efbf5b41e0f3a87177 │ system.image_analysis.registry_lookup_failed │ error │ docker.io/alpine:latest │ image_reference │ catalog │ anchore-quickstart │ 2022-08-24T23:08:30.510193Z │
│ f6084efd159c43a1a0518b6df5e58505 │ system.image_analysis.registry_lookup_failed │ error │ docker.io/alpine:3.12 │ image_reference │ catalog │ anchore-quickstart │ 2022-08-24T23:08:30.480625Z │
│ 4464b8f83df046388152067122c03610 │ system.image_analysis.registry_lookup_failed │ error │ docker.io/alpine:3.8 │ image_reference │ catalog │ anchore-quickstart │ 2022-08-24T23:08:30.450983Z │
...
│ 60f14821ff1d407199bc0bde62f537df │ system.image_analysis.restored_from_archive │ info │ sha256:89020cd33be2767f3f894484b8dd77bc2e5a1ccc864350b92c53262213257dfc │ image_digest │ catalog │ anchore-quickstart │ 2022-08-24T22:53:12.662535Z │
│ cd749a99dca8493889391ae549d1bbc7 │ system.analysis_archive.image_archived │ info │ sha256:89020cd33be2767f3f894484b8dd77bc2e5a1ccc864350b92c53262213257dfc │ image_digest │ catalog │ anchore-quickstart │ 2022-08-24T22:48:45.719941Z │
...
└──────────────────────────────────┴──────────────────────────────────────────────┴───────┴─────────────────────────────────────────────────────────────────────────┴─────────────────┴────────────────┴────────────────────┴─────────────────────────────┘
Note: Events are ordered by the timestamp of their occurrence, the most recent events are at the top of the list and the least recent events at the bottom.
There are a number of ways to filter the event list output (see anchorectl event list --help for filter options):
For troubleshooting events related to a specific event type:
# anchorectl event list --event-type system.analysis_archive.image_archive_failed
✔ List events
┌──────────────────────────────────┬──────────────────────────────────────────────┬───────┬──────────────┬───────────────┬────────────────┬────────────────────┬────────────────────────────┐
│ UUID │ EVENT TYPE │ LEVEL │ RESOURCE ID │ RESOURCE TYPE │ SOURCE SERVICE │ SOURCE HOST │ TIMESTAMP │
├──────────────────────────────────┼──────────────────────────────────────────────┼───────┼──────────────┼───────────────┼────────────────┼────────────────────┼────────────────────────────┤
│ 35114639be6c43a6b79d1e0fef71338a │ system.analysis_archive.image_archive_failed │ error │ nginx:latest │ image_digest │ catalog │ anchore-quickstart │ 2022-08-24T22:48:23.18113Z │
└──────────────────────────────────┴──────────────────────────────────────────────┴───────┴──────────────┴───────────────┴────────────────┴────────────────────┴────────────────────────────┘
To filter events by level such as ERROR or INFO:
anchorectl event list --level info
✔ List events
┌──────────────────────────────────┬─────────────────────────────────────────────┬───────┬─────────────────────────────────────────────────────────────────────────┬───────────────┬────────────────┬────────────────────┬─────────────────────────────┐
│ UUID │ EVENT TYPE │ LEVEL │ RESOURCE ID │ RESOURCE TYPE │ SOURCE SERVICE │ SOURCE HOST │ TIMESTAMP │
├──────────────────────────────────┼─────────────────────────────────────────────┼───────┼─────────────────────────────────────────────────────────────────────────┼───────────────┼────────────────┼────────────────────┼─────────────────────────────┤
│ 60f14821ff1d407199bc0bde62f537df │ system.image_analysis.restored_from_archive │ info │ sha256:89020cd33be2767f3f894484b8dd77bc2e5a1ccc864350b92c53262213257dfc │ image_digest │ catalog │ anchore-quickstart │ 2022-08-24T22:53:12.662535Z │
│ cd749a99dca8493889391ae549d1bbc7 │ system.analysis_archive.image_archived │ info │ sha256:89020cd33be2767f3f894484b8dd77bc2e5a1ccc864350b92c53262213257dfc │ image_digest │ catalog │ anchore-quickstart │ 2022-08-24T22:48:45.719941Z │
...
Note: Event listing response is paginated, anchorectl displays the first 100 events matching the filters. For all the results use the –all flag.
All available options for listing events:
# anchorectl event list --help
Returns a paginated list of events in the descending order of their occurrence. Optional query parameters may be used for filtering results
Usage:
event list [flags]
Flags:
--all return all events (env: ANCHORECTL_EVENT_ALL)
--before string return events that occurred before the ISO8601 formatted UTC timestamp
(env: ANCHORECTL_EVENT_BEFORE)
--event-type string filter events by a prefix match on the event type (e.g. "user.image.")
(env: ANCHORECTL_EVENT_TYPE)
-h, --help help for list
--host string filter events by the originating host ID (env: ANCHORECTL_EVENT_SOURCE_HOST_ID)
--level string filter events by the level - INFO or ERROR (env: ANCHORECTL_EVENT_LEVEL)
-o, --output string the format to show the results (allowable: [text json json-raw id]; env: ANCHORECTL_FORMAT) (default "text")
--page int32 return the nth page of results starting from 1. Defaults to first page if left empty
(env: ANCHORECTL_PAGE)
--resource-type string filter events by the type of resource - tag, imageDigest, repository etc
(env: ANCHORECTL_EVENT_RESOURCE_TYPE)
--service string filter events by the originating service (env: ANCHORECTL_EVENT_SOURCE_SERVICE_NAME)
--since string return events that occurred after the ISO8601 formatted UTC timestamp
(env: ANCHORECTL_EVENT_SINCE)
For help regarding global flags, run --help on the root command
Event listing displays a brief summary of the event, to get more detailed information about the event such as the host where the event has occurred or the underlying the error:
# anchorectl event get c31eb023c67a4c9e95278473a026970c
✔ Fetched event
UUID: c31eb023c67a4c9e95278473a026970c
Event:
Event Type: system.image_analysis.registry_lookup_failed
Level: error
Message: Referenced image not found in registry
Resource:
Resource ID: docker.io/aerospike:latest
Resource Type: image_reference
User Id: admin
Source:
Source Service: catalog
Base Url: http://catalog:8228
Source Host: anchore-quickstart
Request Id:
Timestamp: 2022-08-24T22:08:28.811441Z
Category:
Details: cannot fetch image digest/manifest from registry
Created At: 2022-08-24T22:08:28.812749Z
Clearing Events
Events can be cleared/deleted from the system in bulk or individually. Bulk deletion allows for specifying filters to clear the events within a certain time window. To delete all events from the system:
# anchorectl event delete --all
Use the arrow keys to navigate: ↓ ↑ → ←
? Are you sure you want to delete all events:
▸ Yes
No
⠙ Deleting event
c31eb023c67a4c9e95278473a026970c
329ff24aa77549458e2656f1a6f4c98f
649ba60033284b87b6e3e7ab8de51e48
4010f105cf264be6839c7e8ca1a0c46e
...
Delete events before a specified timestamp (can also use --since instead of --before to delete events that were generated after a specified timestamp):
# anchorectl event delete --before 2022-08-24T22:08:28.629543Z
✔ Deleted event
ce26f1fa1baf4adf803d35c86d7040b7
081394b6e62f4708a10e521a960c54d7
d21b587dea5844cc9c330ba2b3d02d2e
7784457e6bf84427a175658f134f3d6a
...
Delete a specific event:
# anchorectl event delete fa110d517d2e43faa8d8e2dfbb0596af
✔ Deleted event
fa110d517d2e43faa8d8e2dfbb0596af
Sending Events as Webhook Notifications
In addition to access via API and AnchoreCTL, the Anchore Enterprise may be configured to send notifications for events as they are generated in the system via its webhook subsystem. Webhook notifications for event log records is turned off by default. To turn enable the ’event_update’ webhook, uncomment the ’event_log’ section under ‘services->catalog’ in config.yaml, as in the following example:
services:
...
catalog:
...
event_log:
notification:
enabled: True
# (optional) notify events that match these levels. If this section is commented, notifications for all events are sent
level:
- error
Note: In order for events to be sent via webhook notifications, you’ll need to ensure that the webhook subsystem is configured in config.yaml (if it isn’t already) - refer to the document on subscriptions and notifications for information on how to enable webhooks in Anchore Enterprise. Event notifications will be sent to ’event_update’ webhook endpoint if it is defined, and the ‘general’ webhook endpoint otherwise.