Observer

Observer is a service that monitors HTTP endpoints for health and availability. It periodically checks configured HTTP endpoints and records their status, response time, and content validation results.

Project Overview

Observer is designed to provide reliable monitoring of HTTP endpoints with the following features:

• Configurable Health Checks: Monitor multiple HTTP endpoints with customizable check intervals and timeouts
• Response Validation: Validate responses using status codes and regex patterns
• Metrics Exposure: Expose Prometheus metrics for monitoring and alerting
• Persistent Storage: Store observation results in a PostgreSQL database
• Buffered Processing: Efficiently process and store observations using a configurable buffer
• Graceful Shutdown: Handle shutdowns gracefully to ensure no data loss
• Automatic Reload: Reloads target in an interval automatically and pick up new changes

Project Architecture

The project is structured into several packages:

Main Components

• cmd: Contains the main entry points for the application

  • main.go: Sets up the application and command-line interface
  • observe.go: Implements the observe command to start monitoring
  • migrate.go: Implements the migrate command for database migrations

• internal: Contains internal application code

  • config: Configuration loading and validation
  • observer: Core monitoring functionality
    • registry.go: Manages targets and observers
    • observer.go: Implements the health check loop
    • target.go: Defines target configuration and validation
    • buffer.go: Buffers observations before storage

• pkg: Reusable packages

  • backoff: Implements retry with backoff functionality
  • buffer: Generic buffer implementation
  • domain: Core domain models
  • logger: Logging functionality
  • metrics: Prometheus metrics
  • storage: Storage interfaces and implementations

Component Interaction

1. The application loads configuration from a YAML file
2. The registry loads targets from a targets YAML file
3. For each target, an observer is created to periodically check the endpoint
4. Observations are collected in a buffer
5. When the buffer reaches a threshold (items count or time), observations are stored in the database
6. Metrics are exposed via an HTTP endpoint for Prometheus scraping

How to Run

Prerequisites

• Go 1.24 or later
• PostgreSQL database
• Docker and Docker Compose (optional)

Configuration

1. Copy the sample configuration files:

   cp config.sample.yml config.yml
   cp targets.sample.yml targets.yml
   

2. Edit the configuration files to match your environment:

   • config.yml: Main application configuration
   • targets.yml: HTTP endpoints to monitor

Configuration Parameters

The config.yml file contains the following configuration parameters:

Environment

• environment: Specifies the current running environment (e.g., development, production)

HTTP Server

• http.addr: Address and port the HTTP server will listen on (default: ":8080")
• http.readTimeout: Maximum duration for reading the entire request, including the body (default: 1m)
• http.readHeaderTimeout: Amount of time allowed to read request headers (default: 10s)
• http.writeTimeout: Maximum duration before timing out writes of the response (default: 2m)
• http.idleTimeout: Maximum amount of time to wait for the next request when keep-alives are enabled (default: 2m)
• http.requestTimeout: Maximum time allowed for processing a single request (default: 10s)
• http.maxHeaderBytes: Maximum number of bytes the server will read parsing the request header (default: 0, no limit)
• http.metricsPath: URL path where metrics are exposed (default: "/metrics")

Database

• database.username: Database authentication username
• database.password: Database authentication password
• database.host: Database server hostname or IP address
• database.port: Database server port
• database.sslMode: SSL mode for database connection (e.g., disable, require)
• database.name: Database name
• database.maxOpenConnections: Maximum number of open connections to the database (default: 10)
• database.maxIdleConnections: Maximum number of idle connections in the connection pool (default: 8)
• database.connMaxLifetime: Maximum amount of time a connection may be reused (default: 3m)
• database.connMaxIdleTime: Maximum amount of time a connection may be idle (default: 3m)

Graceful Shutdown

• gracefulShutdownTimeout: Maximum duration to wait for ongoing requests to complete during shutdown (default: 30s)

Observer Service

• observer.targetsFilePath: Path to the file containing target configurations (default: targets.yml)
• observer.targetsReloadInterval: How often the targets file should be checked for changes (default: 10s)
• observer.bufferItemsThreshold: Number of items that triggers a buffer flush (default: 100)
• observer.bufferTimeThreshold: Maximum time before a buffer flush is triggered (default: 10s)
• observer.bufferMaxLength: Maximum number of items the buffer can hold (default: 10000)

Example configuration:

environment: production

http:
  addr: ":9090"
  readTimeout: 30s
  metricsPath: /prometheus/metrics

database:
  username: dbuser
  password: securepassword
  host: db.example.com
  port: 5432
  name: observer_db
  maxOpenConnections: 20

gracefulShutdownTimeout: 1m

observer:
  targetsFilePath: /etc/observer/targets.yml
  bufferItemsThreshold: 200
  bufferTimeThreshold: 5s

Running with Docker Compose

1. Start the PostgreSQL database:

   make start-docker-compose
   

2. Run the application:

   make migrate-up  # Run database migrations
   go run cmd/* -c config.yml observe  # Start monitoring
   

Running with Docker

1. Build the Docker image:

   docker build -t observer .
   

2. Run the container:

   docker run -v /path/to/config:/app/configs -p 8080:8080 observer
   

Make Commands

The project includes several make commands to help with development:

Dependency Management

• make dependency: Install required development tools
• make install-mockgen: Install mockgen for generating mocks
• make install-golangci-lint: Install golangci-lint for linting
• make install-goimports: Install goimports for import formatting

CI Commands

• make fix-imports: Fix import formatting
• make lint: Run linting
• make test: Run tests
• make generate: Generate code (mocks, etc.)
• make install-hooks: Install git hooks
• make uninstall-hooks: Uninstall git hooks

Development Environment

• make start-docker-compose: Start PostgreSQL with Docker Compose
• make stop-docker-compose: Stop Docker Compose services
• make remove-docker-compose: Remove Docker Compose containers
• make clean: Clean up the environment

Database Migrations

• make create-migration name=<NAME>: Create a new database migration
• make migrate-up: Run database migrations

Target Configuration

Targets are configured in the targets.yml file with the following properties:

• url: (required) The HTTP endpoint to check
• interval: (optional) Time between checks (default: 30s)
• timeout: (optional) Maximum time for a request (default: 5s)
• regexPattern: (optional) Regular expression to validate the response body
• validStatusCodes: (optional) HTTP status codes considered successful (default: 200-205)

Example:

- url: https://example.com
  interval: 15s
  timeout: 3s
  regexPattern: ".*status.*:.*ok.*"
  validStatusCodes: ["200", "2xx"]