{"id":4723,"date":"2025-12-13T11:36:14","date_gmt":"2025-12-13T11:36:14","guid":{"rendered":"https:\/\/www.it-react.com\/?p=4723"},"modified":"2025-12-13T16:16:14","modified_gmt":"2025-12-13T16:16:14","slug":"building-a-scalable-monitoring-stack-with-prometheus-grafana-part-1","status":"publish","type":"post","link":"https:\/\/www.it-react.com\/index.php\/2025\/12\/13\/building-a-scalable-monitoring-stack-with-prometheus-grafana-part-1\/","title":{"rendered":"Building a Scalable Monitoring Stack with Prometheus & Grafana"},"content":{"rendered":"\n
I used to have a very simple monitoring strategy for my home lab: if the fans were spinning and the LEDs were blinking, everything was fine. If the room was quiet, I had a problem.<\/p>\n\n\n\n
While that approach has a certain rustic charm, it is terrible for uptime and even worse for troubleshooting. You can’t fix a performance bottleneck if you don’t have the data to prove it exists. “It feels slow” is not a metric.<\/p>\n\n\n\n
Today, we are going to fix that. We are going to build a professional-grade monitoring stack from scratch. We will use Prometheus<\/strong> (to store data), Grafana<\/strong> (to visualize it), and Exporters<\/strong> (to collect it).<\/p>\n\n\n\n
And because I like to keep my host OS clean, we are going to run the whole stack in Docker.<\/p>\n\n\n\n
The Architecture: Hub and Spoke<\/h3>\n\n\n\n
Before we touch the terminal, let\u2019s understand what we are building. We are designing this to be scalable from day one.<\/p>\n\n\n\n
\n
The Hub:<\/strong> This is our central VM running Prometheus and Grafana.<\/li>\n\n\n\n
The Spokes:<\/strong> These are the agents (Node Exporters) running on every server we want to monitor.<\/li>\n<\/ol>\n\n\n\n
For this tutorial, we will set up the Hub and monitor the Hub itself, plus an external Windows server to prove that this setup is hybrid-ready.<\/p>\n\n\n\n
Section 1: The Environment<\/h3>\n\n\n\n
I am starting with a clean installation of Ubuntu Server<\/strong>. First, let’s get our tools installed. While there is a newer “Docker Compose V2”, on standard Ubuntu repositories, the classic Docker Compose V1<\/strong> is incredibly stable and robust.<\/p>\n\n\n\n
Let’s install Docker and the Compose tool:<\/p>\n\n\n\n
To verify we are ready, let’s check the version:<\/p>\n\n\n\n
docker-compose version\n# Output: docker-compose version 1.29.2, build unknown<\/code><\/pre>\n\n\n\n
Section 2: Structure is Everything<\/h3>\n\n\n\n
If there is one thing I\u2019ve learned after years of managing servers, it\u2019s that “temporary” setups tend to become permanent. Dumping config files in your home folder is a nightmare waiting to happen when you need to debug this six months from now.<\/p>\n\n\n\n
Let\u2019s create a proper, dedicated workspace.<\/p>\n\n\n\n
Section 3: The “Brain” Configuration (Prometheus)<\/h3>\n\n\n\n
Prometheus works on a “pull” model. It wakes up every few seconds and reaches out to specific targets to grab data. We need to tell it where to look.<\/p>\n\n\n\n
Paste the following. We are defining two jobs: one to monitor the Linux Hub itself, and one for a future Windows machine.<\/p>\n\n\n\n
global:\n scrape_interval: 15s\n\nscrape_configs:\n # Job 1: Monitor Prometheus itself\n - job_name: 'prometheus'\n static_configs:\n - targets: ['localhost:9090']\n\n # Job 2: Monitor our Linux Host (The Hub)\n - job_name: 'node_exporter'\n static_configs:\n - targets: ['node_exporter:9100']\n\n # Job 3: Monitor an external Windows Server\n - job_name: 'windows-exporter'\n static_configs:\n - targets: ['192.168.1.15:9182'] # <--- Replace with your Windows IP<\/code><\/pre>\n\n\n\n
Note:<\/strong> For the Linux job, we use the hostname node_exporter<\/code>. Since we are running inside Docker, Prometheus resolves the container name automatically.<\/p>\n\n\n\n
Section 4: The Stack Definition<\/h3>\n\n\n\n
Now, let\u2019s define our infrastructure as code. Create the docker-compose.yml<\/code> file in your root folder (~\/ops\/monitoring<\/code>).<\/p>\n\n\n\n
nano docker-compose.yml<\/code><\/pre>\n\n\n\n
Here is the full stack definition:<\/p>\n\n\n\n
version: '3.8'\n\nservices:\n prometheus:\n image: prom\/prometheus:latest\n container_name: prometheus\n volumes:\n - .\/prometheus\/prometheus.yml:\/etc\/prometheus\/prometheus.yml\n ports:\n - \"9090:9090\"\n command:\n - '--config.file=\/etc\/prometheus\/prometheus.yml'\n networks:\n - monitoring_net\n\n node_exporter:\n image: prom\/node-exporter:latest\n container_name: node_exporter\n # We mount the HOST paths so the container can see the VM's metrics\n volumes:\n - \/proc:\/host\/proc:ro\n - \/sys:\/host\/sys:ro\n - \/:\/rootfs:ro\n command:\n - '--path.procfs=\/host\/proc'\n - '--path.sysfs=\/host\/sys'\n - '--collector.filesystem.mount-points-exclude=^\/(sys|proc|dev|host|etc)($$|\/)'\n networks:\n - monitoring_net\n\n grafana:\n image: grafana\/grafana:latest\n container_name: grafana\n ports:\n - \"3000:3000\"\n volumes:\n - .\/grafana_data:\/var\/lib\/grafana\n environment:\n - GF_SECURITY_ADMIN_PASSWORD=admin\n networks:\n - monitoring_net\n\nnetworks:\n monitoring_net:\n driver: bridge<\/code><\/pre>\n\n\n\n
Section 5: The “Oops” Moment (Troubleshooting)<\/h3>\n\n\n\n
Time to fire it up.<\/p>\n\n\n\n
docker-compose up -d<\/code><\/pre>\n\n\n\n
When I first ran this, I checked the status (docker-compose ps<\/code>) and saw that Grafana had crashed with an Exit 1<\/code> status.<\/p>\n\n\n\n
The error was clear: GF_PATHS_DATA='\/var\/lib\/grafana' is not writable<\/code>. The Grafana container runs as user ID 472, but the folder I created was owned by my user. It didn’t have permission to write.<\/p>\n\n\n\n
The Fix:<\/strong> We need to change the ownership of that folder to match the ID that Grafana expects.<\/p>\n\n\n\n
![\"\"](\"https:\/\/www.it-react.com\/wp-content\/uploads\/2025\/12\/image-1-1024x380.png\")