Configuration Management and Workflow Orchestration with Git and Temporal

By Marvin Tutt, Chief Executive Officer, Caia Tech

Introduction

Git and Temporal, two open source projects developed over decades with massive investment from the world’s leading technology companies, can be combined to create powerful infrastructure patterns. This article explores how these tools work together for configuration management, workflow orchestration, and infrastructure automation.

Understanding Git Beyond Version Control

Git, created in 2005 by Linus Torvalds for Linux kernel development, has capabilities that extend far beyond tracking code changes. After 19 years of continuous development by thousands of contributors, Git provides a robust foundation for infrastructure automation.

Git’s Infrastructure Capabilities

Cryptographic Integrity: Every commit in Git uses SHA-1 hashing to create an immutable audit trail. Any modification to historical data is immediately detectable through hash verification.

Distributed Architecture: Git operates without requiring a central server. Each repository clone contains the complete history, enabling resilient, offline-capable operations.

Branch Isolation: Git branches provide perfect isolation for parallel development. This same mechanism works for isolating configurations, environments, or customer data.

Efficient Storage: Git’s packfile format and delta compression make it efficient for storing configuration files, documentation, and structured data—not just source code.

Understanding Temporal for Workflow Orchestration

Temporal emerged from Uber’s need to orchestrate millions of workflows reliably. The team that originally built Uber’s Cadence workflow engine founded Temporal to bring enterprise-grade workflow orchestration to the broader market.

Temporal’s Core Capabilities

Durable Execution: Workflows in Temporal survive server crashes, network failures, and data center outages. The system automatically resumes execution exactly where it left off.

Deterministic Replay: Temporal can replay any workflow execution exactly as it originally ran, providing debugging capabilities that make complex distributed system failures understandable.

Built-in Reliability: Retry logic, timeout handling, and error management are built into Temporal’s execution model. Developers write business logic while Temporal handles the distributed systems complexity.

Event Sourcing: Every action in a workflow is recorded as an event, creating a complete, queryable history of all executions.

The Git + Temporal Architecture Pattern

When combined, Git and Temporal create a pattern where Git serves as the source of truth for all configuration and state, while Temporal orchestrates the workflows that interact with this data.

How They Work Together

1. Configuration stored in Git: All system configuration, infrastructure definitions, and application settings live in Git repositories
2. Temporal orchestrates changes: Workflows read from Git, process changes, and write results back
3. Git hooks trigger workflows: Commits to specific branches or paths trigger Temporal workflows
4. Complete audit trail: Both systems maintain detailed history of all changes and executions

Example: Configuration Management System

Consider a configuration management system built with Git and Temporal:

# Stored in Git: configs/production/database.yaml
database:
  host: db.production.internal
  port: 5432
  max_connections: 100
  ssl_enabled: true

# Temporal workflow
@workflow.defn
class ConfigurationWorkflow:
    @workflow.run
    async def apply_configuration(self, config_path: str):
        # Fetch configuration from Git
        config = await workflow.execute_activity(
            fetch_from_git,
            args=[config_path],
            start_to_close_timeout=timedelta(seconds=10)
        )
        
        # Validate configuration
        validation_result = await workflow.execute_activity(
            validate_config,
            args=[config],
            start_to_close_timeout=timedelta(seconds=5)
        )
        
        if not validation_result.is_valid:
            raise Exception(f"Invalid configuration: {validation_result.errors}")
        
        # Apply configuration to systems
        apply_result = await workflow.execute_activity(
            apply_to_systems,
            args=[config],
            start_to_close_timeout=timedelta(minutes=5),
            retry_policy=RetryPolicy(maximum_attempts=3)
        )
        
        # Commit results back to Git
        await workflow.execute_activity(
            commit_to_git,
            args=[f"Applied configuration from {config_path}", apply_result],
            start_to_close_timeout=timedelta(seconds=10)
        )
        
        return apply_result

This pattern provides version control, validation, reliable application, and complete auditability—all using two open source tools.

Practical Implementation Patterns

Multi-Environment Management

Using Git branches for environment isolation:

• main branch contains production configurations
• staging branch contains staging configurations
• development branch contains development configurations
• Temporal workflows handle promotions between branches
• Git’s merge mechanisms prevent configuration drift

Long-Running Process Orchestration

Temporal excels at workflows that run for extended periods:

@workflow.defn
class MonitoringWorkflow:
    @workflow.run
    async def monitor_system(self, repo_path: str):
        while True:
            # Check Git for configuration updates
            current_config = await workflow.execute_activity(
                fetch_from_git,
                args=[repo_path],
                start_to_close_timeout=timedelta(seconds=10)
            )
            
            # Apply any changes
            if current_config != self.last_config:
                await workflow.execute_activity(
                    apply_changes,
                    args=[current_config],
                    start_to_close_timeout=timedelta(minutes=5)
                )
                self.last_config = current_config
            
            # Wait before next check
            await workflow.sleep(timedelta(minutes=5))

Infrastructure as Code

Store infrastructure definitions in Git, use Temporal to orchestrate provisioning:

# infrastructure/servers.yaml in Git
servers:
  web:
    count: 3
    type: t3.medium
    region: us-east-1
  database:
    count: 2
    type: r5.large
    region: us-east-1

Temporal workflows read these definitions and orchestrate the actual provisioning, handling failures, retries, and rollbacks automatically.

Implementation Considerations

Advantages

No Vendor Lock-in: Both Git and Temporal are open source with vibrant communities. Organizations maintain complete control over their infrastructure.

Proven Scale: Git handles the Linux kernel development with thousands of contributors. Temporal processes millions of workflows at companies like Uber and Netflix.

Comprehensive Tooling: Thousands of tools integrate with Git. Temporal provides SDKs for Go, Java, Python, TypeScript, and other languages.

Built-in Resilience: Git’s distributed nature and Temporal’s durable execution provide natural disaster recovery capabilities.

Challenges

Learning Curve: Using Git as a configuration database and Temporal for orchestration requires different thinking than traditional approaches.

Operational Requirements: Self-hosting requires operational expertise in running distributed systems.

Not Universal: High-frequency trading systems, real-time streaming analytics, and certain data-intensive applications may require different approaches.

Getting Started

Basic Setup

1. Install Temporal locally:

curl -sSf https://temporal.download/cli.sh | sh
temporal server start-dev

2. Create a Git repository for configurations:

git init config-repo
cd config-repo
mkdir -p configs/production configs/staging

3. Write your first workflow:

from temporalio import workflow
import subprocess

@workflow.defn
class GitSyncWorkflow:
    @workflow.run
    async def sync_config(self, branch: str):
        # Pull latest from Git
        subprocess.run(["git", "checkout", branch])
        subprocess.run(["git", "pull"])
        
        # Read configuration
        with open("configs/app.yaml") as f:
            config = yaml.safe_load(f)
        
        # Apply configuration
        return await apply_configuration(config)

4. Connect Git hooks to Temporal:

#!/bin/bash
# .git/hooks/post-receive
temporal workflow start \
  --task-queue config-queue \
  --type GitSyncWorkflow \
  --input '{"branch": "main"}'

Building Custom Systems for Your Organization

One of the most powerful aspects of the Git + Temporal combination is the ability to build systems that exactly match your organization’s unique requirements. Unlike off-the-shelf tools that force you to adapt your processes to their models, Git + Temporal provides the foundation to create custom infrastructure that reflects how your company actually operates.

Encoding Business Logic and Policies

Every organization has unique policies, approval workflows, and operational requirements. With Temporal, these become code:

@workflow.defn
class DeploymentWorkflow:
    @workflow.run
    async def deploy_with_company_policies(self, change_request: dict):
        # Your company's specific approval requirements
        if change_request['risk_level'] == 'high':
            approvals_needed = ['security_team', 'architecture_team', 'director']
        elif change_request['affects_payment_systems']:
            approvals_needed = ['payment_team', 'compliance_officer']
        else:
            approvals_needed = ['team_lead']
        
        # Collect approvals according to YOUR policies
        for approver in approvals_needed:
            approval = await workflow.execute_activity(
                request_approval,
                args=[approver, change_request],
                start_to_close_timeout=timedelta(hours=24)
            )
            if not approval.approved:
                await workflow.execute_activity(
                    notify_rejection,
                    args=[change_request, approval.reason]
                )
                return DeploymentResult(status='rejected')
        
        # Your company's specific deployment windows
        if not self.is_deployment_window_open():
            await workflow.sleep_until(self.next_deployment_window())
        
        # Deploy using your company's specific procedures
        result = await workflow.execute_activity(
            deploy_to_environment,
            args=[change_request],
            retry_policy=RetryPolicy(
                maximum_attempts=3,
                backoff_coefficient=2.0
            )
        )
        
        return result

Custom Workflows That Reflect Reality

Most companies have workflows that don’t fit neatly into standard tools. With Git + Temporal, you can model exactly how your organization works:

Example: Custom Release Process

@workflow.defn
class CustomReleaseWorkflow:
    @workflow.run
    async def company_release_process(self, release_version: str):
        # Your company's unique pre-release checks
        await self.run_integration_tests()
        await self.security_scan()
        await self.performance_baseline()
        
        # Your specific staging progression
        environments = ['dev', 'qa', 'staging-us', 'staging-eu', 'production']
        
        for env in environments:
            # Deploy to environment
            await self.deploy_to_environment(env, release_version)
            
            # Your company's specific validation for each environment
            if env == 'qa':
                await self.run_automated_qa_suite()
                await self.manual_qa_signoff()
            elif env.startswith('staging'):
                await self.run_regional_compliance_checks(env)
                await self.monitor_for_duration(hours=4)
            elif env == 'production':
                await self.gradual_rollout_with_monitoring()
        
        # Your company's post-release procedures
        await self.update_documentation()
        await self.notify_stakeholders()
        await self.schedule_retrospective()

Organization-Specific Data Models

Store your company’s unique configuration structures in Git:

# Your company's custom service definition
# configs/services/payment-processor.yaml
service:
  name: payment-processor
  owner: payments-team
  compliance:
    pci_dss: level_1
    sox: required
    gdpr: enabled
  deployment:
    regions: [us-east-1, eu-west-1]
    requires_dual_approval: true
    blackout_windows:
      - start: "2024-11-25"
        end: "2024-11-29"
        reason: "Black Friday freeze"
  monitoring:
    custom_metrics:
      - payment_processing_time
      - fraud_detection_rate
      - settlement_accuracy
  dependencies:
    internal:
      - user-service
      - audit-logger
    external:
      - stripe-api
      - banking-gateway

Your Temporal workflows can understand and enforce these custom structures, something generic tools cannot do.

Building Domain-Specific Languages

For complex organizations, you can create your own domain-specific language (DSL) stored in Git and executed by Temporal:

# Your company's custom deployment DSL
# deployments/q4-release.yaml
deployment:
  name: Q4-Feature-Release
  
  prerequisites:
    - database_migration: v2.3.0
    - feature_flags: 
        - payment_v2: enabled
        - new_ui: 10_percent
    
  stages:
    - name: canary
      target: 5%
      duration: 2h
      success_criteria:
        error_rate: <0.1%
        p99_latency: <200ms
      
    - name: partial
      target: 50%
      duration: 24h
      success_criteria:
        error_rate: <0.5%
        customer_complaints: <10
        
    - name: full
      target: 100%
      monitoring_period: 48h

Temporal workflows interpret and execute your DSL according to your rules:

@workflow.defn
class DSLExecutor:
    @workflow.run
    async def execute_deployment_dsl(self, dsl_path: str):
        # Load your custom DSL from Git
        dsl = await load_from_git(dsl_path)
        
        # Execute according to YOUR business rules
        for stage in dsl['stages']:
            await self.execute_stage(stage)
            
            # Your company's specific success evaluation
            if not await self.evaluate_success_criteria(stage):
                await self.rollback()
                return

Integrating with Your Existing Systems

Every company has legacy systems, internal tools, and specific integrations. Git + Temporal adapts to your environment:

@workflow.defn
class LegacyIntegrationWorkflow:
    @workflow.run
    async def integrate_with_mainframe(self, config_change: dict):
        # Your company's mainframe requires specific format
        mainframe_format = await self.convert_to_ebcdic(config_change)
        
        # Your internal change management system
        ticket_id = await workflow.execute_activity(
            create_servicenow_ticket,
            args=[config_change],
            start_to_close_timeout=timedelta(minutes=5)
        )
        
        # Your company's specific batch window
        await workflow.sleep_until(self.next_mainframe_batch_window())
        
        # Submit to your mainframe
        job_id = await workflow.execute_activity(
            submit_mainframe_job,
            args=[mainframe_format],
            start_to_close_timeout=timedelta(hours=1)
        )
        
        # Wait for your specific job completion signal
        await workflow.wait_condition(
            lambda: self.check_mainframe_job_complete(job_id)
        )
        
        # Update your internal systems
        await self.update_cmdb(config_change)
        await self.close_servicenow_ticket(ticket_id)

Custom Compliance and Governance

Encode your industry-specific and company-specific compliance requirements:

@workflow.defn
class ComplianceWorkflow:
    @workflow.run
    async def enforce_company_compliance(self, change: dict):
        # Your industry's specific requirements
        if self.is_financial_system(change['system']):
            # SOX compliance for your financial systems
            await self.ensure_sox_compliance(change)
            await self.dual_approval_process(change)
            await self.create_audit_trail(change)
            
        if self.handles_eu_data(change):
            # GDPR requirements for your EU operations
            await self.verify_data_residency(change)
            await self.update_privacy_impact_assessment(change)
            
        if self.is_healthcare_related(change):
            # HIPAA requirements for your healthcare data
            await self.verify_encryption_standards(change)
            await self.log_phi_access(change)
        
        # Your company's specific audit requirements
        audit_record = {
            'timestamp': workflow.now(),
            'change': change,
            'approvals': self.approvals,
            'compliance_checks': self.compliance_results,
            'business_justification': change.get('justification'),
            'risk_assessment': self.risk_score
        }
        
        # Store in Git for immutable audit trail
        await workflow.execute_activity(
            commit_to_git,
            args=['audit-log', audit_record],
            start_to_close_timeout=timedelta(seconds=10)
        )

The Power of Custom Systems

By building on Git + Temporal, you’re not constrained by what tool vendors think you need. You can:

• Model your actual business processes, not simplified approximations
• Enforce your specific policies and compliance requirements
• Integrate with your existing systems, no matter how unique
• Create workflows that understand your business domain
• Build abstractions that make sense to your teams
• Evolve your systems as your business changes

This is the fundamental advantage: instead of adapting your company to fit the tool, you build tools that fit your company.

Real-World Applications

Compliance and Audit

Financial services and healthcare organizations use this pattern for:

• Immutable audit trails via Git’s cryptographic hashes
• Complete workflow history in Temporal
• Cryptographic signatures on commits for non-repudiation
• Detailed tracking of who changed what, when, and why

Distributed System Coordination

Technology companies use Git + Temporal for:

• Service mesh configuration management
• Database schema migrations across clusters
• Coordinating deployments across regions
• Managing feature flags and rollouts

Batch Processing and ETL

Data engineering teams leverage this combination for:

• Storing pipeline definitions in Git
• Orchestrating data workflows with Temporal
• Version controlling data transformations
• Maintaining lineage and provenance

Cost Analysis

The investment in these tools by major technology companies is substantial:

Git Development Investment:

• 19 years of continuous development
• Thousands of contributors
• Major corporate sponsors (Google, Microsoft, Facebook)
• Estimated value: Hundreds of millions in development costs

Temporal Development Investment:

• Built on years of Uber’s Cadence development
• $120+ million in venture funding
• Production use at Netflix, Stripe, Datadog
• Enterprise-grade reliability and scale

Available for Free: Both tools are open source and available without licensing costs. Organizations benefit from billions in collective R&D investment.

Conclusion

Git and Temporal, when combined, provide infrastructure automation capabilities that many organizations build or purchase separately. This combination offers powerful patterns for configuration management, workflow orchestration, and audit trails—all available as open source software.

The approach requires thinking differently about infrastructure—using Git not just for code but as a configuration database, and Temporal not just for workflows but as an orchestration layer for all system operations. For organizations willing to adopt these patterns, the benefits include reduced complexity, eliminated vendor lock-in, and access to battle-tested tools developed by some of the world’s best engineering teams.

---

Marvin Tutt is the Chief Executive Officer of Caia Tech, where he focuses on innovative applications of open source technologies for infrastructure automation. He can be reached at [email protected]