Securing Prometheus with Basic Authentication

When you install Prometheus, it is accessible to anyone who knows the endpoint. Recently, Prometheus introduced a built-in mechanism to add basic authentication to each HTTP request, eliminating the need to install a separate proxy like nginx for basic auth configuration. With this mechanism, Prometheus does not store passwords; instead, it computes the hash and compares it with the existing one for the user.

Let's begin by installing the bcrypt Python module to create a password hash:

sudo apt-get -y install python3-bcrypt

Next, create a simple script that prompts for input and returns the hash for the password:

vim generate_password.py

Here's an example of the "generate_password.py" script:

import getpass
import bcrypt

password = getpass.getpass("Enter a password: ")
hashed_password = bcrypt.hashpw(password.encode("utf-8"), bcrypt.gensalt())
print(hashed_password.decode())

Run the script and enter "admin" for the password:

python3 generate_password.py

Copy the generated hash and create an additional Prometheus configuration file:

sudo vim /etc/prometheus/web.yml

Here's an example of the "web.yml" file:

---
basic_auth_users:
    admin: $2b$12$QoGsg.4dlUa4ONm.V/Evu.ndHgM3I2mVyA2zAJduKkb25CEvxxUSu

Now, provide this configuration to Prometheus by updating the systemd service definition:

sudo vim /etc/systemd/system/prometheus.service

Within the "prometheus.service" file, add the "--web.config.file" option:

...
ExecStart=/usr/local/bin/prometheus \
  ...
  --web.config.file=/etc/prometheus/web.yml

After making changes to the systemd service, reload it:

sudo systemctl daemon-reload

Restart Prometheus:

sudo systemctl restart prometheus

Check the status to ensure there are no errors:

sudo systemctl status prometheus

Now, you can test basic authentication by visiting Prometheus and reloading the page.

Additionally, update the Grafana data source configuration to include a username and password:

sudo vim /etc/grafana/provisioning/datasources/datasources.yaml

Here's an example of the "datasources.yaml" file:

---
apiVersion: 1

datasources:
  - name: Prometheus
    type: prometheus
    url: http://localhost:9090
    isDefault: true
    basicAuth: true
    basicAuthUser: admin
    secureJsonData:
      basicAuthPassword: devops123

Restart Grafana:

sudo systemctl restart grafana-server

Upon checking the targets section, you'll notice that the Prometheus target is down. This is because Prometheus now requires a username and password for scraping itself. To update the Prometheus target, edit the Prometheus configuration file:

sudo vim /etc/prometheus/prometheus.yml

Add basic authentication to the "prometheus" job:

...
  - job_name: "prometheus"
    basic_auth:
      username: admin
      password: devops123
    static_configs:
      - targets: ["localhost:9090"]

Check the Prometheus configuration and reload it:

promtool check config /etc/prometheus/prometheus.yml
curl -X POST -u admin:devops123 http://localhost:9090/-/reload

Verify that the Prometheus target is up and running in the UI.