Overview
You've built a perfect ClickHouse deployment for your LLM observability stack. Your team pushes the image to AWS ECR, ready for production. Then it happens: the security scanner flags critical CVEs in the base image, and your deployment is blocked before it even reaches production. Sound familiar? This guide walks you through the exact process to unblock that deployment using Docker Hardened Images (DHI). You'll learn how to replace the default developer-friendly ClickHouse base with a hardened one, pass security scans, and finally go live.
What You Need
- Access to the ClickHouse source Dockerfile (or the ability to modify the image build process)
- Docker CLI installed and configured
- Docker Hardened Images repository access or equivalent hardened base image
- Container registry where you will push the rebuilt image (e.g., AWS ECR, Docker Hub)
- Security scanning tool (e.g., Trivy, Snyk, or built-in scanner in your registry)
- Kubernetes cluster or deployment target
- Basic knowledge of ClickHouse architecture (especially layered storage and network ports)
Step-by-Step Guide
Step 1: Identify the Blocked Deployment and Understand the CVE Findings
When your pipeline flags CVEs, don't ignore them. Start by reviewing the scanner report. In the original scenario, the vulnerabilities were in the base image (e.g., OS-level packages), not in ClickHouse itself. Verify this by checking the CVE list against ClickHouse's own codebase. If the CVEs are in transitive dependencies that ClickHouse never touches (like unused libraries), you have a case for a workaround. But the security team will likely still reject it. Note the exact base image (e.g., debian:bullseye-slim) and the packages flagged.
Step 2: Evaluate the ClickHouse Base Image
ClickHouse's official Docker image is built for developer convenience, not enterprise hardening. It includes many utility packages that are not required for runtime. For example, it might include curl, bash, or perl that the scanner finds vulnerabilities in. Understand the layered architecture: ClickHouse runs as a single binary, so you can often strip away everything except the library dependencies. Check the official Dockerfile on GitHub to see how ClickHouse is compiled and what the base image provides.
Step 3: Select a Docker Hardened Image as Replacement
Docker Hardened Images (DHI) provides minimal, security-optimized base images. Choose a DHI variant that matches the ClickHouse binary's runtime requirements. Typically, you'll want a distroless or scratch-based image that contains only glibc, libstdc++, and OpenSSL. For example, use gcr.io/distroless/cc-debian12:latest or a focused hardened base from your organization's private registry. The goal is to reduce the attack surface to the bare minimum. If your team already has approved base images (e.g., from a hardened registry), use those.
Step 4: Rebuild the ClickHouse Image Using the Hardened Base
Create a new Dockerfile that starts from the hardened base image instead of the original Debian slim. Copy only the ClickHouse binary, config files, and essential libraries. Use multi-stage builds to compile ClickHouse from source in a builder stage, then copy only the required artifacts to the final stage. Example structure:
# Build stage
FROM clickhouse/clickhouse-server:latest AS builder
# Final stage
FROM gcr.io/distroless/cc-debian12
COPY --from=builder /usr/bin/clickhouse /usr/bin/clickhouse
COPY --from=builder /etc/clickhouse-server /etc/clickhouse-server
# ... other minimal files
EXPOSE 8123 9000
ENTRYPOINT ["/usr/bin/clickhouse", "server"]
Remove any unnecessary binaries, scripts, or package managers. If ClickHouse needs specific shared libraries (e.g., libssl), add them explicitly from the builder stage. Test locally to ensure the server starts and queries work.
Step 5: Re-scan and Validate
Build the new image and tag it (e.g., yourregistry/clickhouse:hardened-1.0). Run a full security scan using the same tool that initially blocked you. The number of CVEs should be near zero (typically 0 or 1 low/negligible). Document the remaining findings. For each, confirm they are not exploitable or are in unused code paths. This evidence will help in Step 6.
Step 6: Obtain Security Approval
Present the rebuilt image and scan report to your security team. Highlight that the hardened base eliminated all critical and high CVEs. Explain that the remaining low-impact findings are in libraries that ClickHouse does not load at runtime. Attach the evidence from Step 5. Since you've followed a reputable hardened image standard, your security team is far more likely to approve. In the original issue, the team used DHI to get the all-clear.
Step 7: Deploy to Production
Once approved, update your Kubernetes manifests or deployment configuration to reference the new hardened image. Roll out the update in a canary or staging environment first to verify performance and stability. Monitor ClickHouse logs and query performance. Because the hardened image removes many packages, the binary runs leaner, which can actually improve startup time and memory usage. After confirmation, promote to production. Your blocked deployment is now production-ready.
Tips for Success
- Automate scanning in your CI/CD pipeline to catch issues early before pushing to a registry.
- Pin your base image versions to avoid unexpected CVEs from upstream changes. Use a tool like
renovateto manage updates deliberately. - Consider multi-stage builds even for production images to keep the final layer as small as possible.
- Regularly rebuild your hardened images to pick up security patches in the base image itself.
- Document the rationale for each package you include – this helps security audits later.
- Use runtime security controls (e.g., Pod Security Standards, seccomp profiles) alongside hardened images.
By following these steps, you transform a blocked, insecure ClickHouse container into a production-grade, security-team–approved deployment. Docker Hardened Images are the key to bridging developer velocity and enterprise security.