CloudOps Runbooks as Code: Automating Operations on AWS

Convert manual operational procedures into executable Terraform and AWS Lambda runbooks for 90% faster incident response.

The Problem with Traditional Runbooks

Every operations team has experienced it: a critical incident occurs at 3 AM, and the on-call engineer scrambles to find the relevant runbook in a wiki that hasn't been updated in months. The documented steps don't match the current infrastructure. Commands need to be adapted. Precious minutes tick by while the engineer tries to figure out what's changed since the runbook was written.

Traditional runbooks suffer from documentation drift—the gap between what's documented and what's actually deployed widens over time. They're also manual by nature, requiring engineers to copy-paste commands, interpret outputs, and make judgment calls under pressure. CloudOps Runbooks as Code solves these problems by transforming operational procedures into executable, version-controlled automation.

What Are Runbooks as Code?

Runbooks as Code transforms operational procedures from static documentation into executable automation. Instead of prose describing what to do, you have tested, version-controlled code that actually does it. This code lives alongside your infrastructure definitions, is reviewed through the same pull request process, and can be executed manually or automatically in response to alerts.

The key insight is that operational procedures are just another form of infrastructure—they should be treated with the same rigor as your Terraform modules or Kubernetes manifests. When you update your infrastructure, you update the corresponding runbooks in the same commit. When you test your infrastructure, you test your runbooks. There's no drift because the runbooks are the source of truth.

AWS Systems Manager Automation Documents

AWS Systems Manager (SSM) Automation documents are a powerful way to implement runbooks as code on AWS. These YAML or JSON documents define a series of steps that can interact with AWS services, run commands on EC2 instances, execute Lambda functions, and integrate with external systems. They can be triggered manually, on a schedule, or automatically in response to CloudWatch alarms or EventBridge events.

SSM Automation documents support parameters, conditional logic, branching, and approval steps. They provide detailed execution logs and can be rolled back if steps fail. Most importantly, they can be version-controlled in your Git repository and deployed through your CI/CD pipeline alongside your infrastructure code.

description: Automated EC2 instance recovery runbook
schemaVersion: '0.3'
assumeRole: '{{ AutomationAssumeRole }}'
parameters:
  InstanceId:
    type: String
    description: ID of the unhealthy instance
  AutomationAssumeRole:
    type: String
    default: arn:aws:iam::ACCOUNT:role/SSMAutomationRole

mainSteps:
  - name: checkInstanceState
    action: aws:executeAwsApi
    inputs:
      Service: ec2
      Api: DescribeInstanceStatus
      InstanceIds:
        - '{{ InstanceId }}'
    outputs:
      - Name: InstanceState
        Selector: $.InstanceStatuses[0].InstanceState.Name
        
  - name: stopInstance
    action: aws:changeInstanceState
    inputs:
      InstanceIds:
        - '{{ InstanceId }}'
      DesiredState: stopped
      
  - name: startInstance
    action: aws:changeInstanceState
    inputs:
      InstanceIds:
        - '{{ InstanceId }}'
      DesiredState: running
      
  - name: verifyRecovery
    action: aws:waitForAwsResourceProperty
    inputs:
      Service: ec2
      Api: DescribeInstanceStatus
      InstanceIds:
        - '{{ InstanceId }}'
      PropertySelector: $.InstanceStatuses[0].InstanceStatus.Status
      DesiredValues:
        - ok

Lambda-Based Runbooks for Complex Logic

While SSM Automation handles many scenarios well, complex runbooks often require custom logic that's better expressed in a programming language. AWS Lambda functions can implement sophisticated remediation logic, integrate with external APIs, query databases, and make nuanced decisions based on multiple data sources.

A typical pattern combines SSM Automation for orchestration with Lambda functions for complex steps. The automation document invokes Lambda functions at key decision points, passing context and receiving instructions on how to proceed. This gives you the best of both worlds: the auditability and execution framework of SSM with the flexibility of custom code.

import boto3
import json

def lambda_handler(event, context):
    """
    Automated database failover runbook.
    Triggered by CloudWatch alarm for high replication lag.
    """
    rds = boto3.client('rds')
    sns = boto3.client('sns')
    
    primary_instance = event['primary_instance']
    replica_instance = event['replica_instance']
    
    # Step 1: Verify replication lag exceeds threshold
    lag = get_replication_lag(rds, replica_instance)
    if lag < 300:  # Less than 5 minutes
        return {'action': 'none', 'reason': 'Lag within acceptable range'}
    
    # Step 2: Check replica health before promotion
    replica_status = rds.describe_db_instances(
        DBInstanceIdentifier=replica_instance
    )['DBInstances'][0]
    
    if replica_status['DBInstanceStatus'] != 'available':
        notify_team(sns, f"Replica {replica_instance} not healthy for promotion")
        return {'action': 'manual_intervention', 'reason': 'Replica unhealthy'}
    
    # Step 3: Promote replica to standalone
    rds.promote_read_replica(
        DBInstanceIdentifier=replica_instance,
        BackupRetentionPeriod=7
    )
    
    # Step 4: Update application configuration
    update_connection_string(replica_instance)
    
    # Step 5: Notify team
    notify_team(sns, f"Database failover completed. New primary: {replica_instance}")
    
    return {
        'action': 'failover_completed',
        'new_primary': replica_instance,
        'old_primary': primary_instance
    }

Terraform Integration for Infrastructure Runbooks

Some operational procedures involve infrastructure changes that are best managed through Terraform. Scaling up capacity, adding new nodes to a cluster, or modifying security groups can be automated by triggering Terraform runs from your runbooks. This ensures infrastructure changes follow the same review and approval process as normal operations.

Tools like Terraform Cloud, Atlantis, or custom CI/CD pipelines can expose APIs for triggering Terraform operations. Your runbooks can call these APIs to apply pre-approved changes, passing variables that customize the operation. For example, a capacity scaling runbook might trigger a Terraform run that adjusts Auto Scaling group sizes based on current demand.

Connecting Runbooks to Observability

The real power of runbooks as code emerges when they're connected to your observability stack. CloudWatch alarms, Datadog monitors, or PagerDuty alerts can automatically trigger runbook execution when specific conditions are detected. This creates a closed loop where issues are not just detected but automatically remediated.

Start conservatively with runbooks that handle well-understood, low-risk scenarios like restarting a failed service or clearing a full disk. As you gain confidence, expand to more complex scenarios. Always include guardrails: maximum execution frequency, human approval for destructive actions, and automatic rollback if remediation fails.

• CloudWatch → EventBridge → SSM Automation

  Native AWS integration for automatic runbook triggering

• PagerDuty Incident Workflows

  Trigger runbooks from incident creation with context passed automatically

• Datadog Workflow Automation

  Build visual runbook workflows that respond to monitor alerts

• Custom webhooks

  Integrate any monitoring tool that supports webhook notifications

Testing and Validating Runbooks

Runbooks as code should be tested like any other code. Unit tests can validate individual steps in isolation. Integration tests can execute runbooks against test environments to verify end-to-end behavior. Chaos engineering exercises can trigger runbooks by creating the conditions they're designed to handle.

Include runbook testing in your CI/CD pipeline. When someone modifies a runbook, automated tests should verify it still works correctly. This prevents the documentation drift problem that plagues traditional runbooks—if the runbook doesn't work, the tests fail, and the change doesn't get deployed.