Scott Obert

Performance Testing Strategies for Cloud Applications: Load Testing at Scale

Nov 28, 2021 | Tags: Performance Testing, Load Testing, AWS, Monitoring, Chaos Engineering, TypeScript

Introduction

In our Modern Development Practices series, we’ve explored test-driven development, code quality gates, API design patterns, microservices communication, and database design. Today, we’re focusing on performance testing strategies for cloud applications – a critical practice for ensuring your systems can handle real-world load and scale gracefully.

Cloud applications present unique challenges for performance testing: auto-scaling behaviors, distributed architectures, and pay-per-use pricing models all require specialized testing approaches. We’ll explore comprehensive strategies from unit-level performance tests to large-scale load testing and chaos engineering.

Performance Testing Pyramid

Unit Performance Tests

Start with fast, focused performance tests at the unit level:

import { performance } from 'perf_hooks';

interface PerformanceAssertion {
  maxExecutionTime: number;
  maxMemoryUsage?: number;
  iterations?: number;
}

function performanceTest(
  testName: string,
  testFunction: () => Promise<void> | void,
  assertion: PerformanceAssertion
) {
  return async () => {
    const iterations = assertion.iterations || 1000;
    const executionTimes: number[] = [];
    let maxMemoryUsed = 0;

    for (let i = 0; i < iterations; i++) {
      const memBefore = process.memoryUsage().heapUsed;
      const startTime = performance.now();
      
      await testFunction();
      
      const endTime = performance.now();
      const memAfter = process.memoryUsage().heapUsed;
      
      executionTimes.push(endTime - startTime);
      maxMemoryUsed = Math.max(maxMemoryUsed, memAfter - memBefore);
    }

    // Statistical analysis
    const avgTime = executionTimes.reduce((sum, time) => sum + time, 0) / iterations;
    const p95Time = executionTimes.sort()[Math.floor(iterations * 0.95)];
    const maxTime = Math.max(...executionTimes);

    console.log(`${testName} Performance Metrics:`);
    console.log(`  Average: ${avgTime.toFixed(2)}ms`);
    console.log(`  P95: ${p95Time.toFixed(2)}ms`);
    console.log(`  Max: ${maxTime.toFixed(2)}ms`);
    console.log(`  Max Memory: ${(maxMemoryUsed / 1024 / 1024).toFixed(2)}MB`);

    if (avgTime > assertion.maxExecutionTime) {
      throw new Error(`Performance test failed: Average execution time ${avgTime.toFixed(2)}ms exceeds limit ${assertion.maxExecutionTime}ms`);
    }

    if (assertion.maxMemoryUsage && maxMemoryUsed > assertion.maxMemoryUsage) {
      throw new Error(`Performance test failed: Memory usage ${maxMemoryUsed} exceeds limit ${assertion.maxMemoryUsage}`);
    }
  };
}

// Example usage
describe('UserService Performance', () => {
  const userService = new UserService();

  it('should validate user input efficiently', performanceTest(
    'User validation',
    async () => {
      await userService.validateUser({
        email: 'test@example.com',
        name: 'Test User',
        age: 25
      });
    },
    {
      maxExecutionTime: 1, // 1ms max average
      maxMemoryUsage: 1024 * 1024, // 1MB
      iterations: 10000
    }
  ));

  it('should process user queries efficiently', performanceTest(
    'User query processing',
    async () => {
      await userService.findUsers({ status: 'active', limit: 100 });
    },
    {
      maxExecutionTime: 5, // 5ms max average
      iterations: 1000
    }
  ));
});

API Performance Tests

Test API endpoints with realistic payloads and concurrency:

import axios, { AxiosResponse } from 'axios';

interface LoadTestConfig {
  concurrency: number;
  totalRequests: number;
  rampUpTime: number; // seconds
  endpoint: string;
  method: 'GET' | 'POST' | 'PUT' | 'DELETE';
  payload?: any;
  headers?: Record<string, string>;
}

interface LoadTestResult {
  totalRequests: number;
  successfulRequests: number;
  failedRequests: number;
  averageResponseTime: number;
  p95ResponseTime: number;
  p99ResponseTime: number;
  maxResponseTime: number;
  requestsPerSecond: number;
  errors: string[];
}

class ApiLoadTester {
  async runLoadTest(config: LoadTestConfig): Promise<LoadTestResult> {
    const responseTimes: number[] = [];
    const errors: string[] = [];
    let successfulRequests = 0;
    let failedRequests = 0;

    const requestsPerWorker = Math.ceil(config.totalRequests / config.concurrency);
    const rampUpDelay = (config.rampUpTime * 1000) / config.concurrency;

    const startTime = Date.now();

    // Create worker promises
    const workers = Array.from({ length: config.concurrency }, async (_, workerIndex) => {
      // Ramp up delay
      await this.sleep(workerIndex * rampUpDelay);

      for (let i = 0; i < requestsPerWorker && (successfulRequests + failedRequests) < config.totalRequests; i++) {
        try {
          const requestStart = performance.now();
          
          const response = await this.makeRequest(config);
          
          const requestEnd = performance.now();
          const responseTime = requestEnd - requestStart;
          
          responseTimes.push(responseTime);
          successfulRequests++;
          
          // Validate response
          if (response.status >= 400) {
            errors.push(`HTTP ${response.status}: ${response.statusText}`);
            failedRequests++;
            successfulRequests--;
          }
        } catch (error) {
          failedRequests++;
          errors.push(error instanceof Error ? error.message : 'Unknown error');
        }
      }
    });

    await Promise.all(workers);

    const totalTime = (Date.now() - startTime) / 1000; // seconds
    responseTimes.sort((a, b) => a - b);

    return {
      totalRequests: successfulRequests + failedRequests,
      successfulRequests,
      failedRequests,
      averageResponseTime: responseTimes.reduce((sum, time) => sum + time, 0) / responseTimes.length,
      p95ResponseTime: responseTimes[Math.floor(responseTimes.length * 0.95)] || 0,
      p99ResponseTime: responseTimes[Math.floor(responseTimes.length * 0.99)] || 0,
      maxResponseTime: Math.max(...responseTimes),
      requestsPerSecond: (successfulRequests + failedRequests) / totalTime,
      errors: [...new Set(errors)] // Unique errors
    };
  }

  private async makeRequest(config: LoadTestConfig): Promise<AxiosResponse> {
    const requestConfig = {
      method: config.method,
      url: config.endpoint,
      data: config.payload,
      headers: config.headers,
      timeout: 30000, // 30 second timeout
      validateStatus: () => true // Don't throw on HTTP errors
    };

    return axios(requestConfig);
  }

  private sleep(ms: number): Promise<void> {
    return new Promise(resolve => setTimeout(resolve, ms));
  }
}

// Usage example
describe('API Load Tests', () => {
  const loadTester = new ApiLoadTester();

  it('should handle user creation load', async () => {
    const result = await loadTester.runLoadTest({
      concurrency: 10,
      totalRequests: 1000,
      rampUpTime: 30,
      endpoint: 'https://api.example.com/users',
      method: 'POST',
      payload: {
        name: 'Load Test User',
        email: 'loadtest@example.com'
      },
      headers: {
        'Content-Type': 'application/json',
        'Authorization': 'Bearer test-token'
      }
    });

    console.log('Load Test Results:', result);

    // Assertions
    expect(result.successfulRequests).toBeGreaterThan(950); // 95% success rate
    expect(result.averageResponseTime).toBeLessThan(100); // 100ms average
    expect(result.p95ResponseTime).toBeLessThan(200); // 200ms P95
    expect(result.requestsPerSecond).toBeGreaterThan(30); // 30 RPS minimum
  }, 120000); // 2 minute timeout
});

AWS Load Testing with Artillery

Artillery Configuration

// artillery-config.yml generator
interface ArtilleryConfig {
  target: string;
  phases: Array<{
    duration: number;
    arrivalRate: number;
    rampTo?: number;
  }>;
  scenarios: Array<{
    name: string;
    weight: number;
    flow: Array<any>;
  }>;
}

class ArtilleryConfigGenerator {
  generateConfig(
    baseUrl: string,
    testScenarios: any[],
    loadPattern: 'spike' | 'gradual' | 'sustained'
  ): ArtilleryConfig {
    const phases = this.getLoadPhases(loadPattern);

    return {
      target: baseUrl,
      phases,
      scenarios: testScenarios
    };
  }

  private getLoadPhases(pattern: string) {
    switch (pattern) {
      case 'spike':
        return [
          { duration: 60, arrivalRate: 1 },      // Warm up
          { duration: 30, arrivalRate: 1, rampTo: 100 }, // Spike
          { duration: 60, arrivalRate: 100 },    // Sustain
          { duration: 30, arrivalRate: 100, rampTo: 1 }  // Cool down
        ];

      case 'gradual':
        return [
          { duration: 120, arrivalRate: 1, rampTo: 50 },  // Gradual ramp
          { duration: 300, arrivalRate: 50 },             // Sustain
          { duration: 60, arrivalRate: 50, rampTo: 1 }    // Cool down
        ];

      case 'sustained':
        return [
          { duration: 60, arrivalRate: 1, rampTo: 25 },   // Ramp up
          { duration: 600, arrivalRate: 25 },             // 10 min sustain
          { duration: 60, arrivalRate: 25, rampTo: 1 }    // Cool down
        ];

      default:
        throw new Error('Unknown load pattern');
    }
  }
}

// Artillery test scenarios
const userJourneyScenarios = [
  {
    name: 'User Registration Journey',
    weight: 30,
    flow: [
      {
        post: {
          url: '/api/users/register',
          json: {
            email: '{{ $randomEmail() }}',
            password: 'TestPassword123',
            name: '{{ $randomFirstName() }} {{ $randomLastName() }}'
          },
          capture: {
            header: 'location',
            as: 'userLocation'
          }
        }
      },
      {
        get: {
          url: '{{ userLocation }}',
          headers: {
            'Authorization': 'Bearer {{ authToken }}'
          }
        }
      }
    ]
  },
  {
    name: 'User Login and Browse',
    weight: 70,
    flow: [
      {
        post: {
          url: '/api/auth/login',
          json: {
            email: 'test@example.com',
            password: 'TestPassword123'
          },
          capture: {
            json: '$.token',
            as: 'authToken'
          }
        }
      },
      {
        get: {
          url: '/api/users/profile',
          headers: {
            'Authorization': 'Bearer {{ authToken }}'
          }
        }
      },
      {
        get: {
          url: '/api/products?page=1&limit=20',
          headers: {
            'Authorization': 'Bearer {{ authToken }}'
          }
        }
      }
    ]
  }
];

AWS Lambda Performance Testing

// Lambda performance test wrapper
import { LambdaClient, InvokeCommand } from '@aws-sdk/client-lambda';

interface LambdaPerformanceTest {
  functionName: string;
  payload: any;
  concurrency: number;
  iterations: number;
  coldStartAnalysis?: boolean;
}

class LambdaLoadTester {
  private lambdaClient: LambdaClient;

  constructor() {
    this.lambdaClient = new LambdaClient({ region: process.env.AWS_REGION });
  }

  async testLambdaPerformance(config: LambdaPerformanceTest): Promise<{
    coldStarts: number;
    warmStarts: number;
    averageDuration: number;
    averageBilledDuration: number;
    maxMemoryUsed: number;
    errorRate: number;
    timeouts: number;
  }> {
    const results: any[] = [];
    const errors: string[] = [];
    const timeouts = 0;

    // First, trigger cold starts if needed
    if (config.coldStartAnalysis) {
      await this.triggerColdStarts(config.functionName, config.concurrency);
      await this.sleep(30000); // Wait for functions to cool down
    }

    // Run the actual test
    const workers = Array.from({ length: config.concurrency }, async () => {
      for (let i = 0; i < Math.ceil(config.iterations / config.concurrency); i++) {
        try {
          const startTime = Date.now();
          
          const command = new InvokeCommand({
            FunctionName: config.functionName,
            Payload: JSON.stringify(config.payload),
            LogType: 'Tail'
          });

          const response = await this.lambdaClient.send(command);
          const endTime = Date.now();

          if (response.FunctionError) {
            errors.push(response.FunctionError);
            continue;
          }

          // Parse CloudWatch logs for metrics
          const logResult = response.LogResult 
            ? Buffer.from(response.LogResult, 'base64').toString('utf-8')
            : '';

          const metrics = this.parseCloudWatchLogs(logResult);
          
          results.push({
            ...metrics,
            totalTime: endTime - startTime,
            isColdStart: metrics.initDuration > 0
          });

        } catch (error) {
          errors.push(error instanceof Error ? error.message : 'Unknown error');
        }
      }
    });

    await Promise.all(workers);

    // Analyze results
    const coldStarts = results.filter(r => r.isColdStart).length;
    const warmStarts = results.filter(r => !r.isColdStart).length;
    const durations = results.map(r => r.duration);
    const billedDurations = results.map(r => r.billedDuration);
    const memoryUsages = results.map(r => r.maxMemoryUsed);

    return {
      coldStarts,
      warmStarts,
      averageDuration: durations.reduce((sum, d) => sum + d, 0) / durations.length,
      averageBilledDuration: billedDurations.reduce((sum, d) => sum + d, 0) / billedDurations.length,
      maxMemoryUsed: Math.max(...memoryUsages),
      errorRate: errors.length / (results.length + errors.length),
      timeouts
    };
  }

  private async triggerColdStarts(functionName: string, count: number): Promise<void> {
    // Invoke multiple instances to force cold starts
    const invocations = Array.from({ length: count }, () => 
      this.lambdaClient.send(new InvokeCommand({
        FunctionName: functionName,
        Payload: JSON.stringify({ warmup: true })
      }))
    );

    await Promise.all(invocations);
  }

  private parseCloudWatchLogs(logs: string): any {
    // Parse CloudWatch logs for Lambda metrics
    const reportMatch = logs.match(/REPORT RequestId: .* Duration: ([\d.]+) ms\s+Billed Duration: (\d+) ms\s+Memory Size: (\d+) MB\s+Max Memory Used: (\d+) MB/);
    const initMatch = logs.match(/Init Duration: ([\d.]+) ms/);

    return {
      duration: reportMatch ? parseFloat(reportMatch[1]) : 0,
      billedDuration: reportMatch ? parseInt(reportMatch[2]) : 0,
      memorySize: reportMatch ? parseInt(reportMatch[3]) : 0,
      maxMemoryUsed: reportMatch ? parseInt(reportMatch[4]) : 0,
      initDuration: initMatch ? parseFloat(initMatch[1]) : 0
    };
  }

  private sleep(ms: number): Promise<void> {
    return new Promise(resolve => setTimeout(resolve, ms));
  }
}

Database Performance Testing

DynamoDB Load Testing

import { DynamoDBClient } from '@aws-sdk/client-dynamodb';
import { DynamoDBDocumentClient, PutCommand, GetCommand, QueryCommand } from '@aws-sdk/lib-dynamodb';

interface DynamoDBLoadTestConfig {
  tableName: string;
  readConcurrency: number;
  writeConcurrency: number;
  testDuration: number; // seconds
  itemSize: number; // bytes
}

class DynamoDBLoadTester {
  private docClient: DynamoDBDocumentClient;

  constructor() {
    const client = new DynamoDBClient({ region: process.env.AWS_REGION });
    this.docClient = DynamoDBDocumentClient.from(client);
  }

  async runLoadTest(config: DynamoDBLoadTestConfig): Promise<{
    readMetrics: PerformanceMetrics;
    writeMetrics: PerformanceMetrics;
    throttledRequests: number;
  }> {
    const readMetrics: number[] = [];
    const writeMetrics: number[] = [];
    let throttledRequests = 0;

    const endTime = Date.now() + (config.testDuration * 1000);

    // Write load test
    const writeWorkers = Array.from({ length: config.writeConcurrency }, async () => {
      while (Date.now() < endTime) {
        try {
          const startTime = performance.now();
          
          await this.docClient.send(new PutCommand({
            TableName: config.tableName,
            Item: this.generateTestItem(config.itemSize)
          }));

          const duration = performance.now() - startTime;
          writeMetrics.push(duration);

        } catch (error: any) {
          if (error.name === 'ProvisionedThroughputExceededException') {
            throttledRequests++;
          }
          await this.sleep(100); // Back off on errors
        }
      }
    });

    // Read load test
    const readWorkers = Array.from({ length: config.readConcurrency }, async () => {
      while (Date.now() < endTime) {
        try {
          const startTime = performance.now();
          
          await this.docClient.send(new GetCommand({
            TableName: config.tableName,
            Key: {
              PK: this.generateRandomKey(),
              SK: this.generateRandomKey()
            }
          }));

          const duration = performance.now() - startTime;
          readMetrics.push(duration);

        } catch (error: any) {
          if (error.name === 'ProvisionedThroughputExceededException') {
            throttledRequests++;
          }
          await this.sleep(50); // Back off on errors
        }
      }
    });

    await Promise.all([...writeWorkers, ...readWorkers]);

    return {
      readMetrics: this.calculateMetrics(readMetrics),
      writeMetrics: this.calculateMetrics(writeMetrics),
      throttledRequests
    };
  }

  private generateTestItem(sizeInBytes: number): any {
    const baseItem = {
      PK: this.generateRandomKey(),
      SK: this.generateRandomKey(),
      timestamp: new Date().toISOString(),
      entityType: 'TEST_ITEM'
    };

    // Add padding to reach desired size
    const currentSize = JSON.stringify(baseItem).length;
    const paddingSize = Math.max(0, sizeInBytes - currentSize);
    
    return {
      ...baseItem,
      padding: 'x'.repeat(paddingSize)
    };
  }

  private generateRandomKey(): string {
    return Math.random().toString(36).substring(2, 15);
  }

  private calculateMetrics(data: number[]): PerformanceMetrics {
    data.sort((a, b) => a - b);
    
    return {
      count: data.length,
      average: data.reduce((sum, val) => sum + val, 0) / data.length,
      median: data[Math.floor(data.length / 2)],
      p95: data[Math.floor(data.length * 0.95)],
      p99: data[Math.floor(data.length * 0.99)],
      min: Math.min(...data),
      max: Math.max(...data)
    };
  }

  private sleep(ms: number): Promise<void> {
    return new Promise(resolve => setTimeout(resolve, ms));
  }
}

interface PerformanceMetrics {
  count: number;
  average: number;
  median: number;
  p95: number;
  p99: number;
  min: number;
  max: number;
}

Chaos Engineering

AWS Fault Injection Simulator Integration

import { FISClient, StartExperimentCommand, StopExperimentCommand, GetExperimentCommand } from '@aws-sdk/client-fis';

interface ChaosExperiment {
  name: string;
  templateId: string;
  duration: number; // minutes
  targets: Record<string, string>;
  actions: string[];
}

class ChaosEngineer {
  private fisClient: FISClient;

  constructor() {
    this.fisClient = new FISClient({ region: process.env.AWS_REGION });
  }

  async runChaosExperiment(experiment: ChaosExperiment): Promise<{
    experimentId: string;
    beforeMetrics: SystemMetrics;
    afterMetrics: SystemMetrics;
    recoveryTime: number;
  }> {
    console.log(`Starting chaos experiment: ${experiment.name}`);

    // Collect baseline metrics
    const beforeMetrics = await this.collectSystemMetrics();

    // Start FIS experiment
    const startCommand = new StartExperimentCommand({
      experimentTemplateId: experiment.templateId,
      tags: {
        'ChaosTest': experiment.name,
        'Environment': process.env.ENVIRONMENT || 'test'
      }
    });

    const startResponse = await this.fisClient.send(startCommand);
    const experimentId = startResponse.experiment?.id;

    if (!experimentId) {
      throw new Error('Failed to start chaos experiment');
    }

    // Monitor experiment
    await this.monitorExperiment(experimentId, experiment.duration);

    // Measure recovery time
    const recoveryStartTime = Date.now();
    await this.waitForSystemRecovery();
    const recoveryTime = Date.now() - recoveryStartTime;

    // Collect post-experiment metrics
    const afterMetrics = await this.collectSystemMetrics();

    console.log(`Chaos experiment ${experiment.name} completed. Recovery time: ${recoveryTime}ms`);

    return {
      experimentId,
      beforeMetrics,
      afterMetrics,
      recoveryTime
    };
  }

  private async monitorExperiment(experimentId: string, durationMinutes: number): Promise<void> {
    const endTime = Date.now() + (durationMinutes * 60 * 1000);

    while (Date.now() < endTime) {
      const getCommand = new GetExperimentCommand({ id: experimentId });
      const response = await this.fisClient.send(getCommand);

      const state = response.experiment?.state?.status;
      
      if (state === 'completed' || state === 'stopped' || state === 'failed') {
        console.log(`Experiment ${experimentId} finished with state: ${state}`);
        break;
      }

      console.log(`Experiment ${experimentId} status: ${state}`);
      await this.sleep(30000); // Check every 30 seconds
    }
  }

  private async waitForSystemRecovery(): Promise<void> {
    const maxWaitTime = 5 * 60 * 1000; // 5 minutes
    const startTime = Date.now();

    while (Date.now() - startTime < maxWaitTime) {
      const metrics = await this.collectSystemMetrics();
      
      if (this.isSystemHealthy(metrics)) {
        console.log('System has recovered');
        return;
      }

      console.log('Waiting for system recovery...');
      await this.sleep(10000); // Check every 10 seconds
    }

    throw new Error('System did not recover within expected time');
  }

  private async collectSystemMetrics(): Promise<SystemMetrics> {
    // Implement metrics collection from CloudWatch, application metrics, etc.
    return {
      errorRate: 0.01, // 1%
      responseTime: 150, // ms
      throughput: 1000, // requests/minute
      availability: 99.9 // percentage
    };
  }

  private isSystemHealthy(metrics: SystemMetrics): boolean {
    return (
      metrics.errorRate < 0.05 && // Less than 5% error rate
      metrics.responseTime < 500 && // Less than 500ms response time
      metrics.availability > 99.0 // Greater than 99% availability
    );
  }

  private sleep(ms: number): Promise<void> {
    return new Promise(resolve => setTimeout(resolve, ms));
  }
}

interface SystemMetrics {
  errorRate: number;
  responseTime: number;
  throughput: number;
  availability: number;
}

// Usage example
describe('Chaos Engineering Tests', () => {
  const chaosEngineer = new ChaosEngineer();

  it('should handle Lambda function failures gracefully', async () => {
    const experiment: ChaosExperiment = {
      name: 'Lambda Function Failure',
      templateId: 'EXT123456789', // FIS experiment template ID
      duration: 5, // 5 minutes
      targets: {
        'lambda-functions': 'user-service-function'
      },
      actions: ['stop-lambda-function']
    };

    const result = await chaosEngineer.runChaosExperiment(experiment);

    // Assertions
    expect(result.recoveryTime).toBeLessThan(60000); // Should recover within 1 minute
    expect(result.afterMetrics.availability).toBeGreaterThan(99.0); // 99%+ availability
    expect(result.afterMetrics.errorRate).toBeLessThan(0.05); // Less than 5% errors
  }, 600000); // 10 minute timeout
});

Monitoring and Observability

Custom Metrics Collection

import { CloudWatchClient, PutMetricDataCommand } from '@aws-sdk/client-cloudwatch';

class PerformanceMonitor {
  private cloudWatchClient: CloudWatchClient;
  private metrics: Map<string, number[]> = new Map();

  constructor() {
    this.cloudWatchClient = new CloudWatchClient({ region: process.env.AWS_REGION });
  }

  recordMetric(metricName: string, value: number, unit: string = 'Count'): void {
    if (!this.metrics.has(metricName)) {
      this.metrics.set(metricName, []);
    }
    this.metrics.get(metricName)!.push(value);

    // Send to CloudWatch
    this.sendToCloudWatch(metricName, value, unit);
  }

  startTimer(operationName: string): () => void {
    const startTime = performance.now();
    
    return () => {
      const duration = performance.now() - startTime;
      this.recordMetric(`${operationName}.Duration`, duration, 'Milliseconds');
    };
  }

  async measureAsyncOperation<T>(
    operationName: string,
    operation: () => Promise<T>
  ): Promise<T> {
    const stopTimer = this.startTimer(operationName);
    
    try {
      const result = await operation();
      this.recordMetric(`${operationName}.Success`, 1);
      return result;
    } catch (error) {
      this.recordMetric(`${operationName}.Error`, 1);
      throw error;
    } finally {
      stopTimer();
    }
  }

  getMetricStatistics(metricName: string): {
    count: number;
    average: number;
    min: number;
    max: number;
    p95: number;
    p99: number;
  } {
    const values = this.metrics.get(metricName) || [];
    
    if (values.length === 0) {
      return { count: 0, average: 0, min: 0, max: 0, p95: 0, p99: 0 };
    }

    values.sort((a, b) => a - b);
    
    return {
      count: values.length,
      average: values.reduce((sum, val) => sum + val, 0) / values.length,
      min: values[0],
      max: values[values.length - 1],
      p95: values[Math.floor(values.length * 0.95)],
      p99: values[Math.floor(values.length * 0.99)]
    };
  }

  private async sendToCloudWatch(metricName: string, value: number, unit: string): Promise<void> {
    try {
      const command = new PutMetricDataCommand({
        Namespace: 'Application/Performance',
        MetricData: [{
          MetricName: metricName,
          Value: value,
          Unit: unit,
          Timestamp: new Date(),
          Dimensions: [{
            Name: 'Environment',
            Value: process.env.ENVIRONMENT || 'development'
          }]
        }]
      });

      await this.cloudWatchClient.send(command);
    } catch (error) {
      console.error('Failed to send metric to CloudWatch:', error);
    }
  }
}

// Usage in application code
const monitor = new PerformanceMonitor();

class UserService {
  async createUser(userData: CreateUserRequest): Promise<User> {
    return monitor.measureAsyncOperation('UserService.CreateUser', async () => {
      const stopValidationTimer = monitor.startTimer('UserService.ValidateInput');
      await this.validateUserInput(userData);
      stopValidationTimer();

      const stopDbTimer = monitor.startTimer('UserService.DatabaseWrite');
      const user = await this.userRepository.create(userData);
      stopDbTimer();

      monitor.recordMetric('UserService.UserCreated', 1);
      return user;
    });
  }
}

Distributed Tracing for Performance

import { trace, SpanStatusCode, SpanKind } from '@opentelemetry/api';

class TracedPerformanceService {
  private tracer = trace.getTracer('performance-service');

  async processOrder(orderId: string): Promise<void> {
    const span = this.tracer.startSpan('processOrder', {
      kind: SpanKind.SERVER,
      attributes: {
        'order.id': orderId,
        'service.name': 'order-processor'
      }
    });

    try {
      // Step 1: Validate order
      const validationSpan = this.tracer.startSpan('validateOrder', {
        parent: span,
        attributes: { 'order.id': orderId }
      });

      const startTime = performance.now();
      await this.validateOrder(orderId);
      const validationTime = performance.now() - startTime;

      validationSpan.setAttributes({
        'validation.duration_ms': validationTime,
        'validation.status': 'success'
      });
      validationSpan.setStatus({ code: SpanStatusCode.OK });
      validationSpan.end();

      // Step 2: Process payment
      await this.tracePaymentProcessing(orderId, span);

      // Step 3: Update inventory
      await this.traceInventoryUpdate(orderId, span);

      span.setStatus({ code: SpanStatusCode.OK });
      span.setAttributes({
        'order.status': 'processed',
        'processing.completed_at': new Date().toISOString()
      });

    } catch (error) {
      span.setStatus({
        code: SpanStatusCode.ERROR,
        message: error instanceof Error ? error.message : 'Unknown error'
      });
      throw error;
    } finally {
      span.end();
    }
  }

  private async tracePaymentProcessing(orderId: string, parentSpan: any): Promise<void> {
    const span = this.tracer.startSpan('processPayment', {
      parent: parentSpan,
      attributes: {
        'order.id': orderId,
        'payment.processor': 'stripe'
      }
    });

    try {
      const startTime = performance.now();
      await this.processPayment(orderId);
      const processingTime = performance.now() - startTime;

      span.setAttributes({
        'payment.duration_ms': processingTime,
        'payment.status': 'completed'
      });
      span.setStatus({ code: SpanStatusCode.OK });
    } catch (error) {
      span.setStatus({
        code: SpanStatusCode.ERROR,
        message: error instanceof Error ? error.message : 'Payment failed'
      });
      span.setAttributes({
        'payment.status': 'failed',
        'payment.error': error instanceof Error ? error.message : 'Unknown error'
      });
      throw error;
    } finally {
      span.end();
    }
  }

  private async traceInventoryUpdate(orderId: string, parentSpan: any): Promise<void> {
    const span = this.tracer.startSpan('updateInventory', {
      parent: parentSpan,
      attributes: {
        'order.id': orderId
      }
    });

    try {
      const startTime = performance.now();
      const itemsUpdated = await this.updateInventory(orderId);
      const updateTime = performance.now() - startTime;

      span.setAttributes({
        'inventory.update_duration_ms': updateTime,
        'inventory.items_updated': itemsUpdated,
        'inventory.status': 'updated'
      });
      span.setStatus({ code: SpanStatusCode.OK });
    } catch (error) {
      span.setStatus({
        code: SpanStatusCode.ERROR,
        message: error instanceof Error ? error.message : 'Inventory update failed'
      });
      throw error;
    } finally {
      span.end();
    }
  }

  private async validateOrder(orderId: string): Promise<void> {
    // Simulate validation logic
    await this.sleep(50);
  }

  private async processPayment(orderId: string): Promise<void> {
    // Simulate payment processing
    await this.sleep(200);
  }

  private async updateInventory(orderId: string): Promise<number> {
    // Simulate inventory update
    await this.sleep(100);
    return 3; // Number of items updated
  }

  private sleep(ms: number): Promise<void> {
    return new Promise(resolve => setTimeout(resolve, ms));
  }
}

Performance Testing in CI/CD

GitHub Actions Performance Pipeline

# .github/workflows/performance-tests.yml
name: Performance Tests

on:
  push:
    branches: [ main, develop ]
  pull_request:
    branches: [ main ]
  schedule:
    - cron: '0 2 * * *' # Daily at 2 AM

jobs:
  performance-tests:
    runs-on: ubuntu-latest
    environment: performance-testing

    steps:
    - uses: actions/checkout@v3

    - name: Setup Node.js
      uses: actions/setup-node@v3
      with:
        node-version: '18'
        cache: 'npm'

    - name: Install dependencies
      run: npm ci

    - name: Run unit performance tests
      run: npm run test:performance:unit

    - name: Setup test environment
      run: |
        aws cloudformation deploy \
          --template-file infrastructure/test-stack.yml \
          --stack-name perf-test-stack-${{ github.run_id }} \
          --capabilities CAPABILITY_IAM \
          --parameter-overrides \
            Environment=perf-test \
            BranchName=${{ github.ref_name }}

    - name: Run API performance tests
      run: |
        export API_BASE_URL=$(aws cloudformation describe-stacks \
          --stack-name perf-test-stack-${{ github.run_id }} \
          --query 'Stacks[0].Outputs[?OutputKey==`ApiUrl`].OutputValue' \
          --output text)
        npm run test:performance:api
      env:
        AWS_REGION: us-east-1

    - name: Run load tests with Artillery
      run: |
        npx artillery run artillery-config.yml \
          --target $API_BASE_URL \
          --output performance-results.json

    - name: Analyze performance results
      run: node scripts/analyze-performance.js

    - name: Upload performance report
      uses: actions/upload-artifact@v3
      with:
        name: performance-report
        path: performance-report.html

    - name: Comment PR with results
      if: github.event_name == 'pull_request'
      uses: actions/github-script@v6
      with:
        script: |
          const fs = require('fs');
          const results = JSON.parse(fs.readFileSync('performance-summary.json', 'utf8'));
          
          const comment = `
          ## Performance Test Results
          
          | Metric | Value | Threshold | Status |
          |--------|-------|-----------|--------|
          | Average Response Time | ${results.avgResponseTime}ms | <100ms | ${results.avgResponseTime < 100 ? '✅' : '❌'} |
          | P95 Response Time | ${results.p95ResponseTime}ms | <200ms | ${results.p95ResponseTime < 200 ? '✅' : '❌'} |
          | Error Rate | ${results.errorRate}% | <1% | ${results.errorRate < 1 ? '✅' : '❌'} |
          | Throughput | ${results.throughput} RPS | >100 RPS | ${results.throughput > 100 ? '✅' : '❌'} |
          `;
          
          github.rest.issues.createComment({
            issue_number: context.issue.number,
            owner: context.repo.owner,
            repo: context.repo.repo,
            body: comment
          });

    - name: Cleanup test environment
      if: always()
      run: |
        aws cloudformation delete-stack \
          --stack-name perf-test-stack-${{ github.run_id }}

Conclusion

Performance testing in cloud applications requires a multi-layered approach that spans from unit-level micro-benchmarks to large-scale chaos engineering experiments. Key strategies include:

  • Start with unit performance tests for quick feedback
  • Implement comprehensive load testing with realistic user journeys
  • Test serverless-specific patterns like cold starts and auto-scaling
  • Practice chaos engineering to validate resilience
  • Monitor continuously with distributed tracing and custom metrics
  • Integrate performance testing into your CI/CD pipeline

The patterns and tools we’ve explored provide a foundation for building robust, scalable cloud applications that perform well under real-world conditions.

In our final post of this series, “Technical Debt Management in Growing Codebases,” we’ll explore strategies for maintaining code quality and performance as your applications evolve and scale.

Further Reading

Comments