Building Internal Developer Platforms with Backstage

The Cognitive Load Crisis

Modern cloud development requires expertise in an overwhelming number of tools:

• Version Control: Git, GitHub/GitLab workflows, branch protection rules
• CI/CD: GitHub Actions, Jenkins, CircleCI configuration
• Containers: Docker, Kubernetes YAML, Helm charts
• Cloud Infrastructure: AWS IAM, VPCs, security groups, Terraform
• Databases: RDS provisioning, connection management, migrations
• Observability: Prometheus, Grafana, log aggregation, tracing
• Security: Secrets management, vulnerability scanning, compliance policies

Instead of expecting every developer to become a Kubernetes expert, platform teams build "Golden Paths"—pre-configured, opinionated workflows that handle 80% of use cases. Developers self-serve infrastructure without needing to understand the complexity underneath.

What is Backstage?

Backstage is an open-source Internal Developer Platform framework created by Spotify and donated to the CNCF (Cloud Native Computing Foundation). It provides:

Think of Backstage as "the developer homepage." Instead of bookmarking 15 different tools, developers access everything through one portal.

Architecture: How Backstage Works

1. Service Catalog (The Inventory)

Every service, API, database, and tool in your infrastructure is described by a catalog-info.yaml file stored in its Git repository:

# catalog-info.yaml (stored in service repo)
apiVersion: backstage.io/v1alpha1
kind: Component
metadata:
  name: user-service
  description: User authentication and profile management
  tags:
    - api
    - authentication
    - production
  annotations:
    github.com/project-slug: company/user-service
    circleci.com/project-slug: github/company/user-service
    pagerduty.com/service-id: PUSER123
spec:
  type: service
  lifecycle: production
  owner: team-auth
  system: core-platform
  providesApis:
    - user-api
  consumesApis:
    - payment-api
    - email-api
  dependsOn:
    - resource:postgres-users-db
    - resource:redis-sessions

Backstage automatically discovers these files, builds a dependency graph, and creates a searchable catalog. Developers can instantly see: "What APIs does this service depend on?" or "Who owns this database?"

2. Software Templates (The Scaffolding)

Templates automate the "create new service" workflow. Instead of copying old projects and manually changing names, developers use a form:

# templates/python-api/template.yaml
apiVersion: scaffolder.backstage.io/v1beta3
kind: Template
metadata:
  name: python-api-template
  title: Python REST API
  description: Create a new Python FastAPI service
spec:
  owner: platform-team
  type: service
  
  parameters:
    - title: Service Information
      required:
        - name
        - owner
      properties:
        name:
          title: Service Name
          type: string
          description: Unique service name (e.g., payment-service)
        owner:
          title: Owner Team
          type: string
          ui:field: OwnerPicker
  
  steps:
    # 1. Clone template repository
    - id: fetch-template
      name: Fetch Skeleton
      action: fetch:template
      input:
        url: ./skeleton
        values:
          name: ${{ parameters.name }}
          owner: ${{ parameters.owner }}
    
    # 2. Create GitHub repository
    - id: create-repo
      name: Create GitHub Repository
      action: github:repo:create
      input:
        repoUrl: github.com?repo=${{ parameters.name }}
        description: ${{ parameters.description }}
        defaultBranch: main
    
    # 3. Create Kubernetes namespace
    - id: create-k8s-namespace
      name: Provision Kubernetes Namespace
      action: kubernetes:create-namespace
      input:
        name: ${{ parameters.name }}
        labels:
          owner: ${{ parameters.owner }}
    
    # 4. Create CI/CD pipeline
    - id: create-pipeline
      name: Setup CI/CD
      action: github:actions:create
      input:
        repoUrl: ${{ steps['create-repo'].output.repoUrl }}
        workflowFile: .github/workflows/deploy.yml
    
    # 5. Register in service catalog
    - id: register
      name: Register in Catalog
      action: catalog:register
      input:
        catalogInfoPath: '/catalog-info.yaml'
  
  output:
    links:
      - title: Repository
        url: ${{ steps['create-repo'].output.repoUrl }}
      - title: CI/CD Pipeline
        url: ${{ steps['create-pipeline'].output.pipelineUrl }}

Result: Developer fills in a 5-field form, clicks "Create," and in 3 minutes has: Git repo, CI/CD pipeline, Kubernetes namespace, monitoring dashboards, and documentation—all configured with organizational best practices.

Implementation Guide: Deploy Backstage

Step 1: Install Backstage

# Create new Backstage app
npx @backstage/create-app@latest

# Follow prompts:
# - App name: company-backstage
# - Database: PostgreSQL (recommended for production)

cd company-backstage

# Start development server
yarn dev

# Backstage now running at http://localhost:3000

Step 2: Configure Authentication

Integrate with your SSO provider (GitHub, Google, Okta, Azure AD):

# app-config.yaml
auth:
  environment: production
  providers:
    github:
      production:
        clientId: ${GITHUB_CLIENT_ID}
        clientSecret: ${GITHUB_CLIENT_SECRET}
    google:
      production:
        clientId: ${GOOGLE_CLIENT_ID}
        clientSecret: ${GOOGLE_CLIENT_SECRET}
        signIn:
          resolvers:
            - resolver: emailMatchingUserEntityProfileEmail

Step 3: Populate Service Catalog

Add existing services to the catalog by creating catalog-info.yaml files:

1. Create catalog-info.yaml in each service repository
2. Configure Backstage to auto-discover these files
3. Or manually register each service via UI

# app-config.yaml - Auto-discovery
catalog:
  providers:
    github:
      myOrg:
        organization: 'my-company'
        catalogPath: '/catalog-info.yaml'  # Look for this file
        filters:
          branch: 'main'
          repository: '.*'  # All repos

Step 4: Install Essential Plugins

Extend Backstage with plugins for your tech stack:

# Install plugins
yarn add --cwd packages/app \
  @backstage/plugin-kubernetes \
  @backstage/plugin-cost-insights \
  @backstage/plugin-circleci \
  @backstage/plugin-pagerduty

# Configure Kubernetes plugin (shows pod status, logs)
# app-config.yaml
kubernetes:
  serviceLocatorMethod:
    type: 'multiTenant'
  clusterLocatorMethods:
    - type: 'config'
      clusters:
        - url: https://kubernetes.default.svc
          name: production
          authProvider: 'serviceAccount'

Building Your First Template

Let's create a template that provisions a complete microservice:

1. Create Template Skeleton

# templates/microservice/skeleton/
├── src/
│   ├── main.py          # Templated Python code
│   └── config.py
├── Dockerfile
├── k8s/
│   ├── deployment.yaml  # Kubernetes manifests
│   └── service.yaml
├── .github/
│   └── workflows/
│       └── deploy.yml   # CI/CD pipeline
├── catalog-info.yaml    # Service metadata
└── README.md

# Example: src/main.py with template variables
from fastapi import FastAPI

app = FastAPI(title="${{ values.name }}")

@app.get("/")
def read_root():
    return {"service": "${{ values.name }}", "owner": "${{ values.owner }}"}

2. Define Template Actions

Use Backstage actions to automate infrastructure provisioning:

• fetch:template - Copy skeleton files with variable substitution
• publish:github - Create GitHub repository and push code
• kubernetes:create-namespace - Provision K8s namespace
• catalog:register - Add service to Backstage catalog
• http:backstage:request - Call custom APIs (e.g., create database)

Advanced: Golden Paths for Common Patterns

Create templates for your organization's most common use cases:

Each template encodes best practices: security scanning, resource limits, auto-scaling, observability. Developers get production-ready infrastructure without needing to learn every tool.

Measuring Success: Key Metrics

Track these metrics to measure IDP adoption and impact:

• Time to First Deployment: How long from "day 1" to "first code in production"?
  Target: <1 day (vs 2-3 weeks baseline)
• Infrastructure Ticket Volume: Requests to ops team for provisioning
  Target: 80% reduction
• Template Adoption Rate: % of new services using templates
  Target: >90%
• Developer Satisfaction: Survey score for "ease of deployment"
  Target: 8+/10
• Service Discovery: "Can you find who owns X service?"
  Target: <30 seconds

Common Pitfalls & Solutions

Pitfall 1: "Build Everything at Once"

Symptom: Trying to catalog every service and create templates for every use case before launch.

Solution: Start small. Pick 1-2 critical workflows (e.g., "create REST API"), implement templates, measure adoption. Expand based on feedback. Aim for 80/20 rule: 20% of templates handle 80% of use cases.

Pitfall 2: "No Team Ownership"

Symptom: Backstage deployed but nobody maintains it. Templates break, docs go stale.

Solution: Assign dedicated platform team (2-3 engineers). Treat IDP as a product with roadmap, SLAs, and customer support (internal Slack channel for questions).

Pitfall 3: "Templates Too Rigid"

Symptom: Developers bypass templates because they can't customize for edge cases.

Solution: Provide "escape hatches." Templates handle common cases, but allow manual configuration for 20% of unique needs. Document both paths clearly.

HostingX Managed IDP Service

Building and maintaining a production-grade Internal Developer Platform requires:

• Dedicated platform engineering team (2-3 engineers)
• 3-6 months initial development (catalog, templates, plugins)
• Ongoing maintenance: plugin updates, template improvements, user support
• Integration with existing tools (CI/CD, K8s, cloud providers)

HostingX's Managed IDP Service provides Backstage implementation and hosting:

• ✅ Pre-configured Backstage instance with enterprise SSO
• ✅ Custom templates for your tech stack (REST APIs, workers, pipelines)
• ✅ Pre-integrated plugins (GitHub, K8s, CI/CD, monitoring)
• ✅ Service catalog auto-discovery from your repositories
• ✅ TechDocs generation and hosting
• ✅ 24/7 platform support and maintenance
• ✅ Hosted infrastructure (no ops burden on your team)

Conclusion: From Gatekeepers to Enablers

Traditional ops teams are seen as gatekeepers—bottlenecks that slow down feature delivery. Developers submit tickets and wait days for simple infrastructure changes.

Internal Developer Platforms flip this dynamic. Platform teams become enablers, building self-service tools that accelerate delivery. Instead of answering repetitive questions ("How do I deploy?"), they encode solutions in templates and automation.

The result: developers ship faster, ops teams focus on high-value work (improving platforms, not provisioning individual resources), and organizations achieve the true promise of DevOps—autonomous teams moving at high velocity with built-in guardrails.

Backstage is the open-source foundation for this transformation. The question isn't whether to build an IDP, but how quickly you can get started.