Modern infrastructure requires continuous monitoring and rapid incident response. However, manually configuring and managing alerts is not only labor-intensive but also susceptible to human error.
Automating alert provisioning allows you to enforce consistency, secure sensitive credentials, and integrate monitoring into your deployment pipelines.
This guide dives deep into how you can use the SigNoz Terraform Provider to define and manage alert configurations as code, making your observability setup resilient and adaptable.
Here’s what we’ll cover:
Why Automate Alert Provisioning?
It’s a good idea to automate your alert provisioning for various reasons.
First of all, configuring things manually often leads to discrepancies between environments (development, staging, production). Automating alerts ensures that all environments adhere to the same monitoring standards, reducing the likelihood of configuration drift and improving consistency and uniformity.
Also, when alerts are defined as code, every change is tracked in your version control system. This audit trail makes it easier to trace and review changes, collaborate with team members, and roll back configurations if issues arise.
Something else to consider is that as your infrastructure grows, manually managing alerts becomes unsustainable. Automation allows you to quickly and efficiently update your alerting rules across multiple services without the need for repetitive manual intervention.
Automation also helps improve security. Storing sensitive information like API tokens as environment variables or in secret management systems helps maintain security. Automating the process also minimizes human exposure to critical credentials.
And finally, defining alerts as code enables you to integrate monitoring configurations into your CI/CD pipelines. This leads to continuous testing, validation, and deployment of alert rules alongside application updates.
So as you can see, there are many compelling reasons to go the automation route. Now let’s see how you can do this in practice.
What Are SigNoz and Terraform?
SigNoz is an open-source observability platform designed to collect, analyze, and visualize metrics, logs, and traces from your applications. Its most helpful features include:
-
It has comprehensive monitoring abilities: Provides detailed insights into system performance, error rates, and user behaviors.
-
It comes equipped with real-time analytics: Enables proactive issue detection and performance optimization.
-
It’s community-driven: As an open-source solution, it benefits from community contributions, transparency, and customization.
-
It’s cost-effective: Offers powerful observability capabilities without the hefty licensing fees of proprietary solutions.
Terraform is an Infrastructure as Code (IaC) tool developed by HashiCorp. It allows you to define and provision infrastructure using declarative configuration files. Terraform’s core advantages include:
-
Its declarative syntax: You specify the desired state of your infrastructure, and Terraform handles the implementation.
-
Its version Control: Configuration files can be managed in Git repositories, enabling traceability and rollback of changes.
-
Powerful automation: Facilitates automated provisioning and updates, reducing manual effort and errors.
-
Multi-cloud support: Manages resources across different cloud providers with a consistent workflow.
So you might be wondering: why should you use Terraform with SigNoz?
First of all, Terraform ensures that your infrastructure is managed consistently across different environments, reducing the risk of configuration drift. It also simplifies managing multiple alerts and resources, making it easier to scale your observability setup.
Beyond this, automating the provisioning process reduces manual setup efforts and minimizes the potential for human error.
And finally, Terraform configurations can be version-controlled, allowing teams to track changes over time and collaborate more effectively.
Overview of the Setup
This setup utilizes the SigNoz Terraform Provider to manage alerts and notification channels within SigNoz Cloud. The configuration includes:
-
Provider configuration: Establishes the connection to SigNoz using the API endpoint and a securely provided API token.
-
Notification channels: Defines where alerts are sent (for example, via email) to ensure the right teams are notified.
-
Alert rules: Specifies the conditions under which alerts are triggered, including thresholds and evaluation windows.
-
External variables: Enhances flexibility by allowing critical values (like CPU thresholds and email addresses) to be managed externally.
Prerequisites
Before diving into the setup, make sure you have the following:
-
SigNoz Cloud account: If you don’t have one, sign up for SigNoz Cloud to host your observability data and configure alerts.
-
Terraform installed: Install Terraform on your machine. Terraform is the tool you’ll use to manage your infrastructure as code.
-
SigNoz API token:
-
Log in to your SigNoz Cloud dashboard.
-
Navigate to Settings > API Tokens.
-
Click Generate API Token.
-
Copy the token, as you’ll need it to authenticate Terraform with SigNoz.
-
-
Basic knowledge of Terraform: Familiarity with Terraform’s syntax and concepts, including writing configuration files and running Terraform commands, is essential.
-
Text editor: Use any code editor like Visual Studio Code or Sublime Text to write your Terraform configuration files.
Steps to Set Up the Project
1. Understand the signoz_alert Resource
The signoz_alert resource allows you to create and manage alert rules in SigNoz via Terraform. It supports various alert types, conditions, and configurations. Understanding this resource is crucial as it forms the basis of your alert configuration.
2. Set Up Your Terraform Configuration
Create a new directory for your Terraform configuration:
mkdir signoz-terraform
cd signoz-terraform
Create a main.tf file with the following content:
terraform {
required_providers {
signoz = {
source = "SigNoz/signoz"
version = "0.1.3" # Use the latest version from the Terraform Registry
}
}
}
provider "signoz" {
endpoint = "https://api.us.signoz.cloud" # Replace with your SigNoz Cloud API endpoint
api_token = var.signoz_api_token
}
variable "signoz_api_token" {}
The provider block configures the SigNoz provider, where endpoint specifies the API endpoint and api_token is passed through a variable for security.
3. Define a Notification Channel (Optional)
If you plan to send alerts to specific channels, define them using signoz_notification_channel. For example, create a channels.tf file:
resource "signoz_notification_channel" "email_channel" {
name = "Email Channel"
type = "email"
receivers {
email_config {
to = ["alerts@example.com"]
}
}
}
Defining a notification channel ensures that alerts are sent to the correct recipients, enhancing the utility of your alerting system.
4. Create an Alert Using the signoz_alert Resource
Create an alerts.tf file to define your alert:
resource "signoz_alert" "cpu_high_usage" {
alert = "High CPU Usage Alert"
alert_type = "METRIC_BASED_ALERT"
severity = "critical"
description = "Alert when CPU usage exceeds 80% over 5 minutes"
rule_type = "threshold_rule"
broadcast_to_all = false
disabled = false
eval_window = "5m0s"
frequency = "1m0s"
version = "v4"
condition = jsonencode({
compositeQuery = {
builderQueries = {
A = {
aggregateOperator = "avg"
dataSource = "metrics"
metricName = "cpu_usage_user"
reduceTo = "avg"
filters = {
items = []
op = "AND"
}
groupBy = []
}
}
queryType = "builder"
panelType = "graph"
unit = "%"
}
op = ">"
target = 80
matchType = "EQUALS"
selectedQueryName = "A"
targetUnit = "%"
})
preferred_channels = [signoz_notification_channel.email_channel.name]
labels = {
severity = "critical"
team = "DevOps"
}
}
This configuration creates a high CPU usage alert with specific conditions and notifications. The condition parameter is crucial as it defines the alert triggering logic.
5. Provide the API Token
Set the signoz_api_token as an environment variable:
export TF_VAR_signoz_api_token="YOUR_SIGNOZ_API_TOKEN"
This ensures that your API token is securely used by Terraform without hardcoding it in your configuration files.
6. Initialize Terraform
Run:
terraform init
This command initializes your Terraform working directory, downloading necessary plugins, and preparing the environment.
7. Review the Execution Plan
Generate the execution plan:
terraform plan
This step previews the changes Terraform will make, allowing you to verify the configuration before applying it.
8. Apply the Configuration
Apply the changes:
terraform apply
Type yes when prompted. This command applies the configuration, creating or updating resources as specified.
9. Verify the Alert in SigNoz Cloud
To do this, follow these steps:
-
Log in to your SigNoz Cloud dashboard.
-
Navigate to Alerts.
-
Confirm that the “High CPU Usage Alert” is listed.
-
Click on the alert to view its details and ensure it matches your configuration.
10. Modify the Alert (Optional)
To change the CPU usage threshold to 75%, follow these steps:
11. Destroy the Resources (Optional)
To remove the alert and notification channel:
terraform destroy
Type yes to confirm. This command will delete the resources created by Terraform.
Best Practices and Security Considerations
In modern infrastructure automation, robust best practices and security measures are paramount.
Use version pinning
To ensure your alert provisioning remains reliable and maintainable, start with strict version control. Avoid using the latest tag and instead specify an exact version number. This ensures your infrastructure configuration remains consistent and predictable.
By pinning your provider version (for example, use version = “0.1.3” instead of version = “>= 0.1.3”.), you eliminate unexpected behavior that can arise from upstream changes. This practice is critical for long-term stability, especially when your infrastructure scales across multiple environments.
Externalize Credentials
Security is non-negotiable. Instead of embedding sensitive details like API tokens in your codebase, leverage environment variables or dedicated secret management tools such as HashiCorp Vault or AWS Secrets Manager.
For instance, storing your SigNoz API token as an environment variable (TF_VAR_signoz_api_token) not only mitigates the risk of credential exposure but also simplifies the process of credential rotation. Also, enforce access control policies around your configuration repositories and CI/CD pipelines to further secure these secrets.
Use Version Control
A mature setup also demands rigorous infrastructure version control. Hosting your Terraform configuration in a Git repository with branch protection and code review policies allows you to track changes meticulously, roll back problematic updates, and maintain an audit trail. This traceability is essential when troubleshooting issues or validating compliance during audits.
You should also document your configuration decisions extensively—explain why a particular CPU threshold was chosen or why specific labels (like severity and team) are used. Such documentation becomes invaluable for onboarding new team members or when revisiting configurations months later.
Integrating with CI/CD Pipelines
Integrating Terraform with your CI/CD pipeline is a cornerstone of a modern, automated deployment strategy. A well-architected pipeline not only validates your infrastructure changes but also ensures that your alerting rules remain in sync with your evolving application environment.
Continuous Integration (CI) involves automatically merging code changes into a shared repository and running automated tests on each commit. In practice, embedding Terraform plan into your pull request workflow provides early feedback, catching misconfigurations before they reach production. For instance, a GitHub Actions workflow can automatically check your changes:
name: Terraform CI/CD
on:
push:
branches: [main]
pull_request:
branches: [main]
jobs:
terraform:
runs-on: ubuntu-latest
steps:
- name: Checkout Repository
uses: actions/checkout@v3
- name: Setup Terraform
uses: hashicorp/setup-terraform@v2
- name: Terraform Init
run: terraform init
- name: Terraform Plan
run: terraform plan -no-color
env:
TF_VAR_signoz_api_token: ${{ secrets.SIGNOZ_API_TOKEN }}
This workflow uses GitHub secrets to securely manage your API tokens while validating the configuration changes. Continuous Delivery (CD) takes this further by automating deployments. Once your plan is approved, an automated Terraform apply step (often scheduled during off-peak hours or coordinated with application deployments) ensures smooth, coordinated rollouts.
Advanced pipelines can also include automated rollback mechanisms. For example, if a deployment triggers an anomaly, scripts can automatically revert to a previous version using your version control history—minimizing downtime and reinforcing the feedback loop between application performance and infrastructure configuration.
Advanced Customizations and Troubleshooting
As your observability requirements evolve, you may need to implement advanced customizations. One powerful approach is using multi-metric composite alerts. Instead of triggering an alert on a single threshold, you can design rules that combine multiple conditions—for example, firing only when both CPU usage and memory consumption exceed critical levels. This nuanced alerting minimizes false positives and ensures alerts are issued only during genuine performance issues.
Terraform’s modular design is especially useful here. By creating reusable modules that encapsulate your alert configurations, you can parameterize key variables—such as thresholds, evaluation windows, and notification channels—across a microservices architecture. This modularity enforces consistency while simplifying management and scaling.
Troubleshooting advanced configurations starts with reviewing your terraform plan output to ensure every change aligns with expectations. If an alert isn’t triggering as expected, inspect the JSON structure generated by the jsonencode function. Even minor syntax errors can cause significant issues.
When integrating with incident management tools like PagerDuty or Opsgenie, run comprehensive end-to-end tests in a staging environment. For example, deploy a test alert to a dedicated channel to verify that the complete alerting pipeline—from condition detection to incident escalation—is functioning correctly.
In one real-world scenario, a misconfigured composite query in an alert’s JSON payload led to intermittent failures. By enabling detailed provider logs and iteratively validating the JSON output, the issue was rapidly isolated and resolved. Such experiences underscore the importance of rigorous logging, validation, and testing in production-grade setups.
Conclusion
Automating alert provisioning is a transformative approach to managing observability in modern infrastructures.
By defining alerts and notification channels as code, you make your systems more consistent, scalable, secure, and easily integratabtle with CI/CD. You can set up uniform alert rules across all environments, quickly update and deploy monitoring configs, easily handle secure credentials, and automate CI/CD workflows that stay in sync with application changes. They also become easier to integrate with CI/CD workflows.
I hope you’ve enjoyed this tutorial and have learned something new. I’m always open to suggestions and discussions on LinkedIn. Hit me up with direct messages.
If you’ve enjoyed my writing and want to keep me motivated, consider leaving starts on GitHub and endorsing me for relevant skills on LinkedIn.
Till the next one, happy coding!