Skip to content
mcmqtt - AI Coordination Platform

Agent Swarms

Agent Swarms

Agent swarms are collections of coordinated AI agents that work together to accomplish complex tasks through parallel execution and real-time coordination. mcmqtt’s swarm architecture enables you to deploy hundreds of agents safely and efficiently.

What are Agent Swarms?

An agent swarm consists of:

  • Multiple agent instances running in isolated containers
  • Shared coordination layer via MQTT messaging
  • Centralized task distribution and result aggregation
  • Health monitoring and automatic recovery
  • Resource management with configurable limits
graph LR
    A[Swarm Controller] --> B[Task Queue]
    B --> C[Agent 1]
    B --> D[Agent 2]
    B --> E[Agent N]
    C --> F[Results Aggregator]
    D --> F
    E --> F
    F --> G[Final Output]


    style A fill:#3b82f6,color:#fff
    style F fill:#10b981,color:#fff

Basic Swarm Deployment

Deploy Your First Swarm

Terminal window
# Deploy a simple 5-agent swarm
uvx mcmqtt swarm deploy \
  --name my-first-swarm \
  --agents generic-worker \
  --count 5 \
  --isolation container

Monitor Swarm Status

Terminal window
# Check swarm health
uvx mcmqtt swarm status my-first-swarm


# Output:
# 🟢 Swarm: my-first-swarm (5/5 agents healthy)
# 📊 Tasks Completed: 1,247
# ⏱️  Average Response Time: 234ms
# 💾 Memory Usage: 45% (2.3GB/5GB)
# 🔄 Uptime: 2h 34m 18s

Scale the Swarm

Terminal window
# Scale up to 10 agents
uvx mcmqtt swarm scale my-first-swarm --count 10


# Scale down to 3 agents
uvx mcmqtt swarm scale my-first-swarm --count 3

Advanced Swarm Configurations

High-Performance Browser Testing

Deploy a swarm optimized for web application testing:

Terminal window
uvx mcmqtt swarm deploy \
  --name browser-test-swarm \
  --agents browser-test \
  --count 25 \
  --target https://my-app.com \
  --config '{
    "browser": "chrome",
    "headless": true,
    "viewport": "1920x1080",
    "timeout": 30000,
    "screenshots": true,
    "performance_metrics": true
  }' \
  --resources '{
    "memory": "512MB",
    "cpu": "0.5",
    "disk": "2GB"
  }'

API Load Testing Swarm

Create a distributed load testing network:

Terminal window
uvx mcmqtt swarm deploy \
  --name api-load-test \
  --agents load-tester \
  --count 50 \
  --config '{
    "target_url": "https://api.example.com",
    "requests_per_second": 100,
    "test_duration": "10m",
    "endpoints": [
      "/api/users",
      "/api/products",
      "/api/orders"
    ]
  }' \
  --coordination '{
    "ramp_up_time": "30s",
    "cooldown_time": "15s",
    "failure_threshold": 5
  }'

Security Assessment Swarm

Deploy specialized security testing agents:

Terminal window
uvx mcmqtt swarm deploy \
  --name security-swarm \
  --agents security-scanner \
  --count 10 \
  --target my-app.com \
  --config '{
    "scan_types": ["owasp", "ssl", "headers", "ports"],
    "intensity": "medium",
    "compliance": ["pci", "gdpr"],
    "exclude_paths": ["/admin", "/internal"]
  }' \
  --isolation strict \
  --network isolated

Swarm Coordination Patterns

Leader-Follower Pattern

One agent coordinates the others:

# Deploy with leader coordination
swarm_config = {
  "pattern": "leader-follower",
  "leader_selection": "random",  # or "performance", "manual"
  "failover": "automatic",
  "coordination_topics": {
    "leader_heartbeat": "swarm/{swarm_id}/leader/heartbeat",
    "task_assignment": "swarm/{swarm_id}/tasks/assign",
    "result_collection": "swarm/{swarm_id}/results/collect"
  }
}

Worker Pool Pattern

Shared task queue with worker agents:

# Deploy worker pool
swarm_config = {
  "pattern": "worker-pool",
  "queue_size": 1000,
  "task_timeout": 30,
  "retry_attempts": 3,
  "load_balancing": "round-robin"  # or "least-connections", "random"
}

Pipeline Pattern

Sequential processing across agent stages:

# Deploy processing pipeline
swarm_config = {
  "pattern": "pipeline",
  "stages": [
    {"name": "data-collection", "agents": 5},
    {"name": "data-processing", "agents": 10},
    {"name": "data-output", "agents": 3}
  ],
  "stage_coordination": "sequential",
  "buffer_size": 100
}

Swarm Management

Real-time Monitoring

Terminal window
# Watch swarm metrics in real-time
uvx mcmqtt swarm watch my-swarm


# Output streaming:
# 2024-01-15 10:30:15 | Agent-7  | Task completed: user-journey-checkout
# 2024-01-15 10:30:16 | Agent-12 | Task started: api-stress-test
# 2024-01-15 10:30:17 | Agent-3  | Warning: High memory usage (85%)
# 2024-01-15 10:30:18 | Agent-15 | Task completed: security-scan-headers

Task Distribution

# Distribute tasks to swarm
mqtt_publish({
  "topic": f"swarm/{swarm_id}/tasks/new",
  "payload": {
    "task_id": "task_001",
    "task_type": "browser_test",
    "target": "https://example.com/checkout",
    "config": {
      "user_scenario": "purchase_flow",
      "test_data": "test_user_001"
    },
    "priority": "high",
    "timeout": 60
  }
})

Result Aggregation

# Collect results from all agents
mqtt_subscribe({
  "topic": f"swarm/{swarm_id}/results/+",
  "qos": 1
})


# Results format:
{
  "task_id": "task_001",
  "agent_id": "agent-7",
  "status": "completed",
  "result": {
    "success": true,
    "response_time": 245,
    "screenshot": "base64_image_data",
    "performance_metrics": {...}
  },
  "timestamp": "2024-01-15T10:30:00Z"
}

Swarm Safety Features

Container Isolation

Each agent runs in a secure container:

# Agent container security
security_context:
  read_only_root_filesystem: true
  run_as_non_root: true
  run_as_user: 1000
  capabilities:
    drop: ["ALL"]
  seccomp_profile: "default"

Resource Limits

Prevent resource exhaustion:

# Per-agent resource limits
resources:
  limits:
    memory: "512Mi"
    cpu: "500m"
    ephemeral-storage: "1Gi"
  requests:
    memory: "256Mi"
    cpu: "250m"

Health Monitoring

Automatic health checks and recovery:

Terminal window
# Health check configuration
health_checks:
  liveness_probe:
    http_get:
      path: /health
      port: 8080
    initial_delay_seconds: 30
    period_seconds: 10


  readiness_probe:
    http_get:
      path: /ready
      port: 8080
    initial_delay_seconds: 5
    period_seconds: 5

Runaway Detection

Automatic detection of problematic agents:

# Consciousness monitoring alerts
⚠️  Agent-12: CPU usage > 90% for 5 minutes
🔴 Agent-23: Memory leak detected (growth rate: 50MB/min)
🛑 Agent-31: Unresponsive to heartbeat - terminating container
🔄 Agent-31: Clean restart initiated
✅ Agent-31: Health check passed - rejoining swarm

Performance Optimization

Batch Processing

Process multiple tasks per agent cycle:

# Configure batch processing
batch_config = {
  "batch_size": 10,
  "batch_timeout": 30,
  "parallel_execution": True,
  "result_aggregation": "immediate"
}

Connection Pooling

Reuse connections across tasks:

# HTTP connection pooling
http_config = {
  "pool_connections": 10,
  "pool_maxsize": 100,
  "max_retries": 3,
  "backoff_factor": 0.3
}

Memory Optimization

Optimize memory usage for large swarms:

# Memory optimization settings
memory_config = {
  "garbage_collection": "aggressive",
  "object_pooling": True,
  "cache_size_limit": "100MB",
  "result_streaming": True  # Don't keep all results in memory
}

Troubleshooting Swarms

Agent Not Starting

Terminal window
# Check agent logs
uvx mcmqtt swarm logs my-swarm --agent agent-7


# Check resource availability
uvx mcmqtt swarm resources --available


# Verify container image
docker images | grep mcmqtt-agent

Poor Performance

Terminal window
# Analyze swarm performance
uvx mcmqtt swarm analyze my-swarm --metrics performance


# Check for bottlenecks
uvx mcmqtt swarm bottlenecks my-swarm


# Optimize configuration
uvx mcmqtt swarm optimize my-swarm --auto-tune

Communication Issues

Terminal window
# Test MQTT connectivity
uvx mcmqtt swarm test-connectivity my-swarm


# Check message routing
uvx mcmqtt swarm trace-messages my-swarm --task-id task_001


# Verify topic permissions
uvx mcmqtt broker check-permissions --swarm my-swarm

Best Practices

1. Start Small, Scale Gradually

Terminal window
# Begin with small swarms
uvx mcmqtt swarm deploy --count 3
# Monitor performance, then scale
uvx mcmqtt swarm scale --count 10

2. Use Resource Limits

Terminal window
# Always set resource limits
--resources '{"memory": "512MB", "cpu": "0.5"}'

3. Monitor Health Continuously

Terminal window
# Set up monitoring
uvx mcmqtt swarm monitor my-swarm --alerts enabled

4. Plan for Failures

Terminal window
# Configure automatic recovery
--recovery '{"max_retries": 3, "backoff": "exponential"}'

Next Steps

Now that you understand agent swarms, explore specific use cases:

Agent swarms transform individual AI capabilities into coordinated intelligence networks. Start with simple task distribution, scale to enterprise-grade orchestration. 🚀