Multi-Node Clustering

Documentation Index

Fetch the complete documentation index at: https://docs.lasso.sh/llms.txt

Use this file to discover all available pages before exploring further.

​

Overview

​

What Clustering Provides

• Dashboard aggregation: View metrics across all nodes in a single interface
• Per-region drill-down: Compare provider performance by geographic region
• Cluster health monitoring: Node status, region discovery, and topology visualization
• Circuit breaker visibility: See breaker states across all nodes and regions

​

What Clustering Does NOT Affect

• Routing decisions: Each node routes independently based on local latency
• Request hot path: No cross-node coordination during request handling
• Circuit breakers: Per-node state, no shared breaker coordination
• Provider selection: Based on local measurements only

​

Architecture

┌─────────────────────────────────────────────────────────────┐
│ Application (US-East)                                       │
│ └─> Lasso Node (US-East)                                   │
│     ├─> Routes based on local latency measurements         │
│     └─> Shares metrics with cluster via BEAM distribution  │
├─────────────────────────────────────────────────────────────┤
│ Application (EU-West)                                       │
│ └─> Lasso Node (EU-West)                                   │
│     ├─> Routes based on local latency measurements         │
│     └─> Shares metrics with cluster via BEAM distribution  │
├─────────────────────────────────────────────────────────────┤
│ Cluster Aggregation                                         │
│ ├─> Topology monitoring (node health across regions)       │
│ ├─> Regional metrics aggregation for dashboard             │
│ └─> No impact on routing hot path                          │
└─────────────────────────────────────────────────────────────┘

​

Configuration

​

Required Environment Variables

​

Configuration in runtime.exs

# config/runtime.exs
with dns_query when is_binary(dns_query) <- System.get_env("CLUSTER_DNS_QUERY"),
     node_basename when is_binary(node_basename) <- System.get_env("CLUSTER_NODE_BASENAME") do
  config :libcluster,
    topologies: [
      dns: [
        strategy: Cluster.Strategy.DNSPoll,
        config: [
          polling_interval: 5_000,
          query: dns_query,
          node_basename: node_basename
        ]
      ]
    ]
end

​

DNS Service Discovery

• Kubernetes: Headless service (returns all pod IPs)
• Consul: Service discovery with DNS interface
• Internal DNS: Custom DNS server resolving to node IPs
• Cloud DNS: AWS Route 53, GCP Cloud DNS, etc.

​

DNS Requirements

1. Multiple A records: DNS query must return all node IPs
2. Internal network: Nodes must reach each other on EPMD port (4369) and distribution ports
3. TTL: Low TTL for fast node discovery (recommended: 5-30 seconds)

​

Port Requirements

​

Example Configurations

​

Kubernetes

apiVersion: v1
kind: Service
metadata:
  name: lasso
spec:
  clusterIP: None  # Headless service
  selector:
    app: lasso
  ports:
    - port: 4000
      name: http
    - port: 4369
      name: epmd

apiVersion: apps/v1
kind: Deployment
metadata:
  name: lasso
spec:
  replicas: 3
  selector:
    matchLabels:
      app: lasso
  template:
    metadata:
      labels:
        app: lasso
    spec:
      containers:
        - name: lasso
          image: myregistry.com/lasso-rpc:latest
          env:
            - name: SECRET_KEY_BASE
              valueFrom:
                secretKeyRef:
                  name: lasso-secrets
                  key: secret-key-base
            - name: PHX_HOST
              value: "rpc.example.com"
            - name: PHX_SERVER
              value: "true"
            - name: LASSO_NODE_ID
              valueFrom:
                fieldRef:
                  fieldPath: metadata.name  # pod-name-0, pod-name-1, etc.
            - name: CLUSTER_DNS_QUERY
              value: "lasso.default.svc.cluster.local"
            - name: CLUSTER_NODE_BASENAME
              value: "lasso"
          ports:
            - containerPort: 4000
              name: http
            - containerPort: 4369
              name: epmd

​

Docker Compose

version: '3.8'

services:
  lasso-us-east:
    build: .
    environment:
      SECRET_KEY_BASE: ${SECRET_KEY_BASE}
      PHX_HOST: rpc.example.com
      PHX_SERVER: "true"
      LASSO_NODE_ID: us-east-1
      CLUSTER_DNS_QUERY: lasso-cluster
      CLUSTER_NODE_BASENAME: lasso
    networks:
      - lasso-cluster
    ports:
      - "4001:4000"

  lasso-eu-west:
    build: .
    environment:
      SECRET_KEY_BASE: ${SECRET_KEY_BASE}
      PHX_HOST: rpc.example.com
      PHX_SERVER: "true"
      LASSO_NODE_ID: eu-west-1
      CLUSTER_DNS_QUERY: lasso-cluster
      CLUSTER_NODE_BASENAME: lasso
    networks:
      - lasso-cluster
    ports:
      - "4002:4000"

networks:
  lasso-cluster:
    driver: bridge

​

VM/Bare Metal with Consul

# On us-east-1 node
curl -X PUT -d '{"Name": "lasso", "Address": "10.0.1.10"}' \
  http://localhost:8500/v1/agent/service/register

# On eu-west-1 node
curl -X PUT -d '{"Name": "lasso", "Address": "10.0.2.10"}' \
  http://localhost:8500/v1/agent/service/register

# us-east-1
export CLUSTER_DNS_QUERY="lasso.service.consul"
export CLUSTER_NODE_BASENAME="lasso"
export LASSO_NODE_ID="us-east-1"

# eu-west-1
export CLUSTER_DNS_QUERY="lasso.service.consul"
export CLUSTER_NODE_BASENAME="lasso"
export LASSO_NODE_ID="eu-west-1"

​

Node Identity

​

Recommended Naming

​

Why Node ID Matters

• Metrics partitioning: State is keyed by {provider_id, node_id}
• Regional comparison: Dashboard groups metrics by region
• Circuit breaker visibility: See which regions have open breakers
• Traffic analysis: Understand request distribution across nodes

​

Cluster Topology

​

Node States

​

Health Checks

• Interval: 15 seconds
• Timeout: 5 seconds
• Failure threshold: 3 consecutive failures → :unresponsive
• Method: :rpc.multicall/4 to all connected nodes

​

Topology Events

# Subscribe to cluster events
Phoenix.PubSub.subscribe(Lasso.PubSub, "cluster:topology")

# Receive events
{:node_connected, %{node: :'lasso@us-east-1', region: "us-east-1"}}
{:node_disconnected, %{node: :'lasso@us-east-1'}}
{:node_state_change, %{node: :'lasso@us-east-1', from: :discovering, to: :responding}}

​

Dashboard Integration

​

MetricsStore

# Get provider leaderboard across all nodes
MetricsStore.get_provider_leaderboard("default", "ethereum")
# => %{
#   data: [...],
#   coverage: %{responding: 3, total: 3},
#   stale: false
# }

• TTL: 15 seconds
• RPC timeout: 5 seconds
• Invalidation: Automatic on node connect/disconnect
• Aggregation: Weighted averages by call volume

​

Regional Drill-Down

• View aggregate performance across all regions
• Drill into specific regions to identify geographic issues
• Compare provider performance region-by-region
• See which regions have circuit breakers open

​

Troubleshooting

​

Nodes Not Connecting

# Test DNS query
dig lasso.internal

# Should return multiple A records
;; ANSWER SECTION:
lasso.internal. 30 IN A 10.0.1.10
lasso.internal. 30 IN A 10.0.2.10

# Test EPMD port from another node
telnet 10.0.1.10 4369

# Check configuration
env | grep CLUSTER
# CLUSTER_DNS_QUERY=lasso.internal
# CLUSTER_NODE_BASENAME=lasso

​

Nodes Becoming Unresponsive

# View cluster topology in dashboard
# Navigate to: http://localhost:4000/dashboard
# Look for nodes in :unresponsive state

• Network partitions
• High CPU/memory usage preventing health check responses
• Firewall blocking distribution ports

​

Best Practices

1. Use stable node IDs: Don’t change LASSO_NODE_ID after deployment
2. Monitor cluster health: Watch for nodes in :unresponsive state
3. Plan for network partitions: Nodes gracefully degrade to standalone mode
4. Use internal DNS: Don’t expose Erlang distribution to public internet
5. Test failover: Verify dashboard still works when nodes disconnect

​

Next Steps

• Geo-Distributed Deployment - Multi-region architecture
• Production Checklist - Pre-launch verification
• Architecture - Understand cluster design