Introduction

To enable the use of HPA and VPA in Kubernetes, we will need to install Metrics Server in the cluster. By default, the certificates used by kubelets will not be recognized by the metrics server.

We will need to either get metrics server to ignore insecure TLS (which is not secure) or by modifying the kubelet configuration to rotate its certificates using the cluster CA.

Sample Errors

If you install the Metrics Server directly with the following command:

kubectl apply -f https://github.com/kubernetes-sigs/metrics-server/releases/latest/download/high-availability-1.21+.yaml

You may encounter the following errors:

E0208 03:38:49.733438       1 scraper.go:149] "Failed to scrape node" err="Get \"https://10.0.0.71:10250/metrics/resource\": tls: failed to verify certificate: x509: cannot validate certificate for 10.0.7.71 because it doesn't contain any IP SANs" node="sg-hub-svr-k8s-master-01"
E0208 03:38:49.742354       1 scraper.go:149] "Failed to scrape node" err="Get \"https://10.0.0.81:10250/metrics/resource\": tls: failed to verify certificate: x509: cannot validate certificate for 10.0.7.81 because it doesn't contain any IP SANs" node="sg-hub-svr-k8s-worker-01"
I0208 03:38:59.441222       1 server.go:192] "Failed probe" probe="metric-storage-ready" err="no metrics to serve"

Resolution

Since our cluster is already bootstrapped, we need to do the following steps:

  1. Install the Kubelet Serving Certificate Approver in your cluster first:
kubectl apply -f https://raw.githubusercontent.com/alex1989hu/kubelet-serving-cert-approver/main/deploy/standalone-install.yaml

This is a custom approving controller which approvers kubelet CSR that kubelet use to serve TLS endpoints.

  1. Install Metrics Server in your cluster:
kubectl apply -f https://github.com/kubernetes-sigs/metrics-server/releases/latest/download/components.yaml
  1. Edit MachineConfig for all your nodes in your cluster:

Sample Config:

machine:
  kubelet:
    extraArgs:
      rotate-server-certificates: true
Remember to do the same for all your nodes in your cluster!
  1. Test the metrics server
➜  kubectl top nodes
NAME                       CPU(cores)   CPU(%)   MEMORY(bytes)   MEMORY(%)
sg-hub-svr-k8s-master-01   91m          4%       779Mi           58%
sg-hub-svr-k8s-worker-01   33m          1%       515Mi           15%