If a custom CA certificate is required to access an external resource then the Trust Store in Anchore needs to be propagated to the following locations:
- The Operating System provided trust store.
- The Python Certifi trust store.
- NodeJS runtime for the UI.
When might you need to add a CA Cert to Anchore Enterprise?
- Using an SSL terminating network proxy in your Anchore deployment environment.
- Anchore needs to be able to reach external https endpoints from vulnerability feeds to container registries.
- Using a Container Registry with self-signed certificate or custom CA.
- You can update the trust store OR use the –insecure option when configuring the registry in Anchore.
The operating system trust store is read by the skopeo utility (the tool used to interact with container registries) and python requests library that is used to access container registries to read manifests and pull image layers.
Adding your certificate(s)
Approach 1
The first approach is centred around creating a new Anchore Enterprise image and inserting the CA certs into the right places. You might need to perform this for both the Anchore Enterprise COre and Anchore Enterprise UI image.
The following Dockerfile illustrates an example of how this general process can be automated to produce your own container with a new custom CA cert installed.
1. Create Dockerfile
Example Dockerfile updating the certifi trust store for the Python Anchore Enterprise Image
FROM docker.io/anchore/enterprise:v5.X.X
USER root:root
COPY ./custom-ca.pem /home/anchore/venv/lib/python3.11/site-packages/certifi/
# This is to verify the CA's are in the correct format
RUN openssl crl2pkcs7 -nocrl -certfile /home/anchore/venv/lib/python3.11/site-packages/certifi/custom-ca.pem | openssl pkcs7 -print_certs -noout
COPY ./custom-ca.pem /etc/pki/ca-trust/source/anchors/
RUN update-ca-trust && trust list
RUN /usr/bin/cat /home/anchore/venv/lib/python3.11/site-packages/certifi/custom-ca.pem >> /home/anchore/venv/lib/python3.11/site-packages/certifi/cacert.pem
USER anchore:anchore
We suggest adding an indicator to the resulting image name that designates it as being custom built. Ex: enterprise:v5.X.X-custom
2. Build Custom Image using Dockerfile
sudo docker build -t anchore/enterprise:v5.X.Xcustom .
You will need to perform this on each build, store the new enterprise image in a private registry and update your Helm or Compose deployment to use the new image reference.
Approach 2 (Recommended)
This approach is about injecting the secrets/ca certs into the containers at runtime and therefore doesn’t require a new image to be built.
Docker Compose
Enterprise
The entrypoint of the enterprise container will enumerate certificates mounted at /home/anchore/certs, combine them with its built-in CAs and populate them all via the environment variables:
- REQUESTS_CA_BUNDLE
- SSL_CERT_DIR
The above should configure the running services to use the custom CA(s). Please note that doing an exec into a container may not include the /docker-entrypoint.sh.
For all services using the enterprise container:
volumes:
- ./ldap-combined-ca-cert-bundle.pem:/home/anchore/certs/ldap-combined-ca-cert-bundle.pem:ro
UI
Simply supply your own custom certificate(s) as environment variable(s) and volume mount(s). The example below is supplying a custom CA for use with LDAP in the Anchore Enterprise UI image.
ui:
image: docker.io/anchore/enterprise-ui:v5.9.0
volumes:
- ./license.yaml:/license.yaml:ro
- ./config-ui.yaml:/config/config-ui.yaml:z
- ./ldap-combined-ca-cert-bundle.pem:/home/anchore/certs/ldap-combined-ca-cert-bundle.pem:ro
environment:
- NODE_EXTRA_CA_CERTS=/home/anchore/certs/ldap-combined-ca-cert-bundle.pem
Helm
For Helm deployments, first create the Kubernetes secret that will store your cert(s). The example below is supplying multiple certs in a custom-ca-cert secret and anchore K8s namespace.
kubectl create secret generic custom-ca-cert --from-file=ldap-ca-cert.pem=./ldap-ca-cert.pem --from-file=db-ssl-ca-cert.pem=./db-ssl-ca-cert.pem -n anchore
Ensure you have the CA cert secret in the same namespace as your deployment. (eg. -n anchore)
Now update your Helm values file to reference your secret, and CA cert file
certStoreSecretName: "custom-ca-cert"
ui:
ldapsRootCaCertName: "ldap-ca-cert.pem"
anchoreConfig:
database:
ssl: true
sslRootCertFileName: "db-ssl-ca-cert.pem"
Please note there are other certs you can supply and configure anchoreConfig.internalServicesSSL & anchoreConfig.keys.privateKeyFileName
Additional Background
Operating System
To add a certificate to the operating system trust store the CA certificate should be placed in the /etc location that is appropriate for the container image being used.
- For Anchore 4.5.X and newer, the base container is Red Hat Universal Base Image 9.X, which stores certs in
/etc/pki/ca-trust/source/anchors/and requires user to run update-ca-trust command as root to update the system certs.
Anchore Enterprise UI - Node.js
The Anchore Enterprise UI is powered by Node.js and as such, when the UI makes calls to external services such as LDAP it might require a certificate. Please note that Node.js can also pull certificates from the Operating System store.
- Anchore Enterpise loads the certificate into the NODE_EXTRA_CA_CERTS environment variable
Anchore Enterprise - Python
Certifi is a curated list of trusted certificate authorities that is used by the Python requests HTTP client library. The Python requests library is used by Anchore for all HTTP interactions, including when communicating with Anchore Feed service, when webhooks are sent to a TLS enabled endpoint and inbetween Anchore services if TLS has been configured. To update the Certifi trust store the CA certificate should be appended onto the cacert.pem file provided by the Certifi library.
- For Enterprise 5.1.x and newer, Python was upgraded to python 3.11, certifi’s cacert.pem is installed in
/home/anchore/venv/lib/python3.11/site-packages/certifi/cacert.pem
Debugging
How to know if you need a custom cert?
Have a proxy or custom CA in place? Can’t ignore self-signed certs? Then yes.
Ask your IT / Infrastructure Team, otherwise you can test the connections from your Anchore deployment/server to the service in question.
curl https://myregistry.example.com (if you see ssl verify errors then you might require a custom ca)
If you are able to ignore self-signed certs, you can do this for Container Registries in Anchore
Fetch, test and use the Custom Cert
If you have identified that you need to add a custom CA cert into Anchore. You can run the following to fetch and test the certificate before redeploying Anchore.
# fetch the cert
openssl s_client -showcerts -servername myregistry.example.com -connect myregistry.example.com:443 > cacert.pem
# test the cert
curl -v --cacert=cacert.pem myregistry.example.com
You can take this certificate and add this to your Anchore deployment as described above.