AI Services

RankFlow AI — AI Architecture: LLM Gateway, Router & Multi-Model Strategy

Version: 2.0.0

docs/specs/ai/ai-services-01-architecture.md

On this page

1. System Architecture
2. LLM Gateway
2.1 Core Interface
2.2 Gateway Flow
2.3 Rate Limiting
2.4 Budget Enforcement
3. Multi-Model Router
3.1 Model Registry
3.2 Task-to-Model Mapping
3.3 Model Selection Decision Tree
4. Model Selection Strategy
4.1 Cost-Quality Trade-off Matrix
4.2 Dynamic Model Downgrade
4.3 Urgent vs. Batch Priority
5. Cost Management & Budgeting
5.1 Cost Calculation
5.2 Budget Tracking (Redis)
5.3 Cost Alert Thresholds
5.4 Cost Optimization Strategies
6. Fallback & Retry Strategy
6.1 Fallback Chain
6.2 Retry Policy
6.3 Dead Letter Queue (DLQ)
7. Circuit Breaker Pattern
7.1 Circuit Breaker Configuration
7.2 Circuit Breaker State Machine
7.3 Circuit Breaker Implementation
8. Provider Integrations
8.1 Anthropic (Claude) Adapter
8.2 OpenAI (GPT-4o) Adapter
8.3 Google (Gemini) Adapter
8.4 Provider Selection Strategy
9. Streaming & Real-Time Inference
9.1 Streaming Use Cases
9.2 Streaming Implementation
9.3 Non-Streaming (Background Jobs)
10. Batch Processing
10.1 Batch Use Cases
10.2 Batch Prompt Optimization
11. Monitoring & Observability
11.1 Metrics Dashboard
11.2 Structured Logging
11.3 Health Check Endpoint
12. Configuration Schema
12.1 Environment Variables
12.2 Database Schema

Version: 2.0.0
Date: 2026-06-16
Scope: LLM Gateway architecture, model routing, cost management, fallback chains, circuit breaker, provider integrations, streaming, and real-time inference
Target Audience: AI Engineers, Backend Engineers, DevOps
Service Path: src/server/services/ai/

1. System Architecture#

┌─────────────────────────────────────────────────────────────────────────────┐
│                           CLIENT REQUESTS                                    │
│  (Dashboard UI, Onboarding, Scheduled Jobs, Admin Actions, Webhooks)       │
└─────────────────────────────────────────────────────────────────────────────┘
                                    │
                                    ▼
┌─────────────────────────────────────────────────────────────────────────────┐
│                         LLM GATEWAY (src/server/services/ai/gateway.ts)      │
│                                                                              │
│  ┌─────────────────┐  ┌─────────────────┐  ┌─────────────────┐            │
│  │  Request Queue   │  │  Auth & Rate      │  │  Cost Budget    │            │
│  │  (BullMQ /      │  │  Limiting         │  │  Check          │            │
│  │   Inngest)       │  │  (Redis)          │  │  (per client)   │            │
│  └────────┬────────┘  └────────┬────────┘  └────────┬────────┘            │
│           │                    │                    │                       │
│           └────────────────────┴────────────────────┘                       │
│                              │                                               │
│                              ▼                                               │
│  ┌─────────────────────────────────────────────────────────────────┐        │
│  │                    MULTI-MODEL ROUTER                            │        │
│  │  ┌─────────────┐  ┌─────────────┐  ┌─────────────┐  ┌────────┐ │        │
│  │  │ Task-to-    │  │ Practice-   │  │ Client      │  │ Budget │ │        │
│  │  │ Model Map   │→ │ Specific    │→ │ Override    │→ │ Cap    │ │        │
│  │  │ (default)   │  │ Template    │  │ (optional)  │  │ (hard) │ │        │
│  │  └─────────────┘  └─────────────┘  └─────────────┘  └────────┘ │        │
│  │                              │                                     │        │
│  │                              ▼                                     │        │
│  │  ┌─────────────────────────────────────────────────────────┐    │        │
│  │  │              SELECTED MODEL + CONFIGURATION                │    │        │
│  │  │  { provider, model, temperature, maxTokens, systemPrompt }│    │        │
│  │  └─────────────────────────────────────────────────────────┘    │        │
│  └─────────────────────────────────────────────────────────────────┘        │
│                              │                                               │
│                              ▼                                               │
│  ┌─────────────────────────────────────────────────────────────────┐        │
│  │              PROVIDER ADAPTER LAYER                              │        │
│  │  ┌─────────────┐  ┌─────────────┐  ┌─────────────┐            │        │
│  │  │  Anthropic  │  │   OpenAI    │  │  Google     │  Future:    │        │
│  │  │  (Claude)   │  │  (GPT-4o)   │  │  (Gemini)   │  DeepSeek,  │        │
│  │  │             │  │             │  │             │  Mistral     │        │
│  │  └─────────────┘  └─────────────┘  └─────────────┘            │        │
│  └─────────────────────────────────────────────────────────────────┘        │
│                              │                                               │
│                              ▼                                               │
│  ┌─────────────────────────────────────────────────────────────────┐        │
│  │              POST-PROCESSING PIPELINE                          │        │
│  │  ┌─────────────┐  ┌─────────────┐  ┌─────────────┐  ┌────────┐ │        │
│  │  │  Medical    │  │  Quality    │  │  Format     │  │  Cache │ │        │
│  │  │  Compliance │  │  Score      │  │  Validation │  │  Store │ │        │
│  │  │  Filter     │  │  (auto)     │  │  (JSON/etc) │  │        │ │        │
│  │  └─────────────┘  └─────────────┘  └─────────────┘  └────────┘ │        │
│  └─────────────────────────────────────────────────────────────────┘        │
│                              │                                               │
│                              ▼                                               │
│  ┌─────────────────────────────────────────────────────────────────┐        │
│  │              AUDIT & LOGGING LAYER                               │        │
│  │  ┌─────────────┐  ┌─────────────┐  ┌─────────────┐            │        │
│  │  │  Cost Log   │  │  Quality    │  │  Prompt     │  Latency    │        │
│  │  │  (per req)  │  │  Score      │  │  Version    │  Metrics    │        │
│  │  └─────────────┘  └─────────────┘  └─────────────┘            │        │
│  └─────────────────────────────────────────────────────────────────┘        │
└─────────────────────────────────────────────────────────────────────────────┘
                                    │
                                    ▼
┌─────────────────────────────────────────────────────────────────────────────┐
│                         OUTPUT: Generated Content                            │
│  + Metadata: model used, cost, latency, quality score, compliance status     │
└─────────────────────────────────────────────────────────────────────────────┘

2. LLM Gateway#

2.1 Core Interface#

The LLM Gateway is the single entry point for ALL AI generation in the platform. No code outside the gateway calls provider SDKs directly.

// src/server/services/ai/gateway.ts

interface GenerateRequest {
  task: TaskType;                    // e.g., "gbp_post", "landing_page_article"
  practiceId: string;                // For client personalization & budget tracking
  variables: Record<string, unknown>; // Template variables (practice name, city, etc.)
  
  // Optional overrides
  model?: ModelKey;                  // Override default model selection
  systemPrompt?: string;             // Override system prompt
  temperature?: number;              // Override temperature (default: 0.7)
  maxTokens?: number;                  // Override max tokens (default: 1024)
  jsonMode?: boolean;                  // Force JSON output
  stream?: boolean;                    // Enable streaming (for UI)
  
  // Context
  requestId: string;                 // UUID for tracing
  userId?: string;                     // For audit trail
  impersonating?: boolean;             // If true, read-only context
}

interface GenerateResponse {
  content: string;                   // Generated text
  metadata: {
    model: ModelKey;                 // Actual model used (after routing)
    provider: ProviderKey;            // "anthropic" | "openai" | "google"
    tokensUsed: {
      input: number;
      output: number;
      total: number;
    };
    costUsd: number;                  // Calculated cost
    latencyMs: number;                // Total generation time
    qualityScore?: number;             // 0-100, auto-evaluated
    complianceStatus: "PASS" | "WARN" | "FAIL";
    promptVersion: string;            // Template version used
    fallbackUsed?: boolean;           // True if fallback model was used
    fallbackFrom?: ModelKey;         // Original model that failed
    cacheHit?: boolean;              // True if served from cache
  };
}

// The gateway function
declare function generate(request: GenerateRequest): Promise<GenerateResponse>;

2.2 Gateway Flow#

┌────────────────────────────────────────────────────────────┐
│  Step 1: Request Validation                                │
│  - Validate task type exists in TASK_REGISTRY              │
│  - Validate practiceId exists and is active                │
│  - Check rate limit (Redis: `rate_limit:{practiceId}`)     │
│  - Check budget (Redis: `ai_budget:{practiceId}`)         │
│  - If impersonating, apply read-only restrictions         │
└────────────────────────────────────────────────────────────┘
                              │
                              ▼
┌────────────────────────────────────────────────────────────┐
│  Step 2: Template Resolution                                 │
│  - Query DB for practice-specific template (latest version)│
│  - If not found: query global default template               │
│  - If not found: use hardcoded fallback prompt               │
│  - Substitute variables into template                        │
└────────────────────────────────────────────────────────────┘
                              │
                              ▼
┌────────────────────────────────────────────────────────────┐
│  Step 3: Model Selection                                     │
│  - Get default model for task from TASK_MODEL_MAP           │
│  - Apply practice-level override from template              │
│  - Apply client-level override from request                 │
│  - Check if model is available (circuit breaker status)      │
│  - If model unavailable, use fallback model                 │
└────────────────────────────────────────────────────────────┘
                              │
                              ▼
┌────────────────────────────────────────────────────────────┐
│  Step 4: Cache Check                                         │
│  - Generate cache key: hash(task + practiceId + variables)   │
│  - Check Redis cache (`ai_cache:{hash}`)                   │
│  - If cache hit and < 24h old: return cached response        │
│  - If cache hit and > 24h old: re-generate (stale)          │
└────────────────────────────────────────────────────────────┘
                              │
                              ▼
┌────────────────────────────────────────────────────────────┐
│  Step 5: Provider Call                                       │
│  - Call selected provider adapter                           │
│  - Set timeout: 30s for Haiku/Mini, 60s for Sonnet/GPT-4o  │
│  - Track start time for latency                             │
│  - If timeout: trigger fallback                             │
└────────────────────────────────────────────────────────────┘
                              │
                              ▼
┌────────────────────────────────────────────────────────────┐
│  Step 6: Post-Processing                                     │
│  - Medical compliance filter (regex + keyword scan)         │
│  - Quality scoring (automated evaluation)                    │
│  - Format validation (JSON mode, length checks)              │
│  - If compliance FAIL: reject + log + alert                 │
│  - If quality < 60: trigger regeneration (max 3 attempts)   │
└────────────────────────────────────────────────────────────┘
                              │
                              ▼
┌────────────────────────────────────────────────────────────┐
│  Step 7: Audit & Logging                                     │
│  - Log to `ai_usage` table (DB)                             │
│  - Log to structured logger (Pino)                          │
│  - Update Redis counters (daily/monthly spend)              │
│  - Check cost thresholds (alert if exceeded)                │
│  - Store in cache (if not rejected)                          │
└────────────────────────────────────────────────────────────┘
                              │
                              ▼
                    GenerateResponse

2.3 Rate Limiting#

// Per-practice rate limits (enforced at gateway level)
const RATE_LIMITS = {
  // Per minute
  perMinute: {
    "gbp_post": 10,
    "social_caption": 10,
    "review_reply": 20,
    "landing_page_article": 5,
    "citation_description": 50,
    "schema_markup": 5,
    "content_refresh": 3,
    "blog_post": 5,
    "default": 10,
  },
  // Per day
  perDay: {
    "gbp_post": 100,
    "social_caption": 100,
    "review_reply": 200,
    "landing_page_article": 20,
    "citation_description": 500,
    "schema_markup": 20,
    "content_refresh": 10,
    "blog_post": 30,
    "default": 100,
  },
};

// Redis key: `rate_limit:{practiceId}:{task}:{window}`
// Window: `minute` or `day`
// Value: incrementing counter with TTL

2.4 Budget Enforcement#

// Per-practice AI budget (enforced at gateway level)
const BUDGETS = {
  starter: { daily: 2.00, monthly: 20.00 },    // USD
  growth: { daily: 5.00, monthly: 50.00 },
  pro: { daily: 15.00, monthly: 150.00 },
  enterprise: { daily: 50.00, monthly: 500.00 },
};

// Budget enforcement logic:
// 1. Check `ai_budget:daily:{practiceId}` in Redis
// 2. If daily budget exceeded: reject with "AI budget exceeded" error
// 3. If monthly budget exceeded: reject + alert admin + flag for review
// 4. Budget resets at UTC midnight (daily) and UTC 1st of month (monthly)

// Non-essential tasks can be throttled when budget is close:
// - Non-essential: blog_post, content_refresh, site_evolution, aeo_content
// - Essential: review_reply, gbp_post (already scheduled), schema_markup

3. Multi-Model Router#

3.1 Model Registry#

All supported models are registered in a central registry with capabilities, cost, and context window.

// src/server/services/ai/models/registry.ts

interface ModelConfig {
  key: ModelKey;
  provider: ProviderKey;
  modelId: string;                    // Provider-specific model ID
  name: string;                       // Human-readable name
  
  // Capabilities
  capabilities: {
    reasoning: 1 | 2 | 3 | 4 | 5;     // Complex reasoning ability
    creativity: 1 | 2 | 3 | 4 | 5;    // Creative writing ability
    instructionFollowing: 1 | 2 | 3 | 4 | 5;
    jsonMode: boolean;                 // Supports structured JSON output
    vision: boolean;                   // Supports image input
    multilingual: boolean;             // Supports non-English languages
    longContext: boolean;              // Supports > 32K context
    streaming: boolean;                // Supports streaming
    systemPrompts: boolean;            // Supports system prompts
  };
  
  // Cost (per million tokens)
  cost: {
    input: number;                   // USD per 1M input tokens
    output: number;                  // USD per 1M output tokens
    cachedInput?: number;            // USD per 1M cached input tokens (if supported)
  };
  
  // Limits
  limits: {
    contextWindow: number;             // Max tokens in context
    maxOutputTokens: number;         // Max tokens in response
    maxRequestsPerMinute: number;     // Rate limit for this model
  };
  
  // Medical compliance
  medicalCompliance: {
    accuracy: 1 | 2 | 3 | 4 | 5;     // Medical accuracy rating
    hallucinationRisk: "low" | "medium" | "high";
    recommendedFor: TaskType[];     // Tasks this model is best for
  };
}

const MODEL_REGISTRY: Record<ModelKey, ModelConfig> = {
  "claude-sonnet": {
    key: "claude-sonnet",
    provider: "anthropic",
    modelId: "claude-sonnet-4-20250514",
    name: "Claude 4 Sonnet",
    capabilities: {
      reasoning: 5, creativity: 4, instructionFollowing: 5,
      jsonMode: true, vision: true, multilingual: true,
      longContext: true, streaming: true, systemPrompts: true,
    },
    cost: { input: 3.00, output: 15.00, cachedInput: 1.50 },
    limits: { contextWindow: 200_000, maxOutputTokens: 8192, maxRequestsPerMinute: 1000 },
    medicalCompliance: { accuracy: 5, hallucinationRisk: "low", recommendedFor: [
      "landing_page_article", "schema_markup", "content_refresh", "seo_audit", "blog_post", "translation"
    ]},
  },
  
  "claude-haiku": {
    key: "claude-haiku",
    provider: "anthropic",
    modelId: "claude-3-5-haiku-20241022",
    name: "Claude 3.5 Haiku",
    capabilities: {
      reasoning: 3, creativity: 3, instructionFollowing: 4,
      jsonMode: true, vision: false, multilingual: true,
      longContext: true, streaming: true, systemPrompts: true,
    },
    cost: { input: 0.25, output: 1.25 },
    limits: { contextWindow: 200_000, maxOutputTokens: 4096, maxRequestsPerMinute: 2000 },
    medicalCompliance: { accuracy: 3, hallucinationRisk: "low", recommendedFor: [
      "gbp_post", "faq", "review_reply", "social_caption", "hashtag_generation", "alt_text", "digest_summary"
    ]},
  },
  
  "gpt-4o": {
    key: "gpt-4o",
    provider: "openai",
    modelId: "gpt-4o",
    name: "GPT-4o",
    capabilities: {
      reasoning: 4, creativity: 4, instructionFollowing: 4,
      jsonMode: true, vision: true, multilingual: true,
      longContext: true, streaming: true, systemPrompts: true,
    },
    cost: { input: 2.50, output: 10.00, cachedInput: 1.25 },
    limits: { contextWindow: 128_000, maxOutputTokens: 4096, maxRequestsPerMinute: 1000 },
    medicalCompliance: { accuracy: 4, hallucinationRisk: "low", recommendedFor: [
      "keyword_research", "schema_markup", "content_edit", "report_summary"
    ]},
  },
  
  "gpt-4o-mini": {
    key: "gpt-4o-mini",
    provider: "openai",
    modelId: "gpt-4o-mini",
    name: "GPT-4o Mini",
    capabilities: {
      reasoning: 3, creativity: 2, instructionFollowing: 3,
      jsonMode: true, vision: false, multilingual: true,
      longContext: true, streaming: true, systemPrompts: true,
    },
    cost: { input: 0.15, output: 0.60 },
    limits: { contextWindow: 128_000, maxOutputTokens: 4096, maxRequestsPerMinute: 3000 },
    medicalCompliance: { accuracy: 3, hallucinationRisk: "medium", recommendedFor: [
      "citation_description", "social_caption", "meta_description", "hashtag_generation",
      "alt_text", "platform_adapt", "sentiment_analysis", "image_prompt"
    ]},
  },
  
  "gemini-flash": {
    key: "gemini-flash",
    provider: "google",
    modelId: "gemini-2.5-flash",
    name: "Gemini 2.5 Flash",
    capabilities: {
      reasoning: 3, creativity: 3, instructionFollowing: 3,
      jsonMode: true, vision: true, multilingual: true,
      longContext: true, streaming: true, systemPrompts: true,
    },
    cost: { input: 0.15, output: 0.60 },
    limits: { contextWindow: 1_000_000, maxOutputTokens: 8192, maxRequestsPerMinute: 2000 },
    medicalCompliance: { accuracy: 3, hallucinationRisk: "medium", recommendedFor: [
      "image_prompt", "alt_text", "faq", "hashtag_generation"
    ]},
  },
};

3.2 Task-to-Model Mapping#

The default mapping is the starting point. All three override layers can modify it.

// src/server/services/ai/models/task-mapping.ts

// Layer 1: Hardcoded defaults (fallback of last resort)
const DEFAULT_TASK_MODELS: Record<TaskType, ModelKey> = {
  // High-quality content (expensive but necessary)
  "landing_page_hero": "claude-sonnet",
  "landing_page_about": "claude-sonnet",
  "landing_page_article": "claude-sonnet",
  "blog_post": "claude-sonnet",
  "content_refresh": "claude-sonnet",
  "site_evolution": "claude-sonnet",
  "seo_audit": "claude-sonnet",
  "translation": "claude-sonnet",
  "aeo_content": "claude-sonnet",
  "schema_markup": "claude-sonnet",
  "content_edit": "claude-sonnet",
  "template_copy": "claude-sonnet",
  "ab_variant": "claude-sonnet",
  
  // Medium-quality content (balanced cost/quality)
  "gbp_post": "claude-haiku",
  "faq": "claude-haiku",
  "faq_update": "claude-haiku",
  "review_reply": "claude-haiku",
  "support_reply": "claude-haiku",
  "lead_email": "claude-haiku",
  "save_offer": "claude-haiku",
  "upgrade_email": "claude-haiku",
  "digest_summary": "claude-haiku",
  "nap_summary": "claude-haiku",
  "report_summary": "claude-sonnet",  // Reports need high quality
  "image_prompt": "claude-haiku",
  
  // Bulk / simple content (cheap, fast)
  "citation_description": "gpt-4o-mini",
  "social_caption": "gpt-4o-mini",
  "meta_description": "gpt-4o-mini",
  "hashtag_generation": "gpt-4o-mini",
  "alt_text": "gpt-4o-mini",
  "platform_adapt": "gpt-4o-mini",
  "sentiment_analysis": "gpt-4o-mini",
  "keyword_research": "gpt-4o",
  "section_regenerate": "claude-sonnet",
  "citation_resubmit": "gpt-4o-mini",
  "custom_report": "claude-sonnet",
};

// Layer 2: Practice-specific overrides (from DB PromptTemplate)
// SELECT modelConfig FROM PromptTemplate WHERE practiceId = ? AND taskType = ? AND isDefault = true
// Returns: { model: "claude-sonnet", temperature: 0.7, maxTokens: 2048 }

// Layer 3: Client-request overrides (from API call)
// request.model = "claude-sonnet" → overrides everything

// Resolution order (highest priority wins):
// 1. Client override (request.model)
// 2. Practice template override (PromptTemplate.modelConfig.model)
// 3. Default task mapping (DEFAULT_TASK_MODELS)

3.3 Model Selection Decision Tree#

Client requests content generation
│
├─ Is client override specified? ──→ YES → Use client's model
│                                     (validate against registry first)
│
├─ Is practice template override? ──→ YES → Use template's model
│                                     (from PromptTemplate.modelConfig)
│
├─ Use default task mapping
│
├─ Is selected model available? (circuit breaker check)
│   ├─ NO → Use fallback model (see Fallback Chain)
│   └─ YES → Proceed
│
├─ Is model within budget?
│   ├─ NO → Downgrade to cheaper model (if task allows)
│   └─ YES → Proceed
│
└─ Return final model + configuration

4. Model Selection Strategy#

4.1 Cost-Quality Trade-off Matrix#

Task	Quality Need	Volume	Default Model	Cost/Gen	Monthly Cost
Landing page hero	Very High	1x	Claude Sonnet	~$0.005	~$0.005
Landing page article	Very High	3-5x	Claude Sonnet	~$0.015	~$0.06
GBP post	Medium	12x/mo	Claude Haiku	~$0.003	~$0.036
Social caption	Low	16x/mo	GPT-4o Mini	~$0.001	~$0.016
Citation description	Low	30x	GPT-4o Mini	~$0.002	~$0.002
Review reply	Medium	~30x/mo	Claude Haiku	~$0.002	~$0.06
Blog post	High	8x/mo	Claude Sonnet	~$0.012	~$0.096
Content refresh	Very High	1x/mo	Claude Sonnet	~$0.010	~$0.010
Schema markup	Very High	1x/mo	Claude Sonnet	~$0.008	~$0.008
FAQ	Medium	1x/mo	Claude Haiku	~$0.003	~$0.003
Translation	Very High	1x/qtr	Claude Sonnet	~$0.008	~$0.003
TOTAL per client/mo					~$0.30

4.2 Dynamic Model Downgrade#

When a client is approaching their daily budget limit, non-essential tasks can be downgraded to cheaper models:

// Dynamic downgrade rules
const DOWNGRADE_MAP: Record<ModelKey, ModelKey | null> = {
  "claude-sonnet": "claude-haiku",    // Sonnet → Haiku (saves ~80%)
  "claude-haiku": "gpt-4o-mini",      // Haiku → Mini (saves ~60%)
  "gpt-4o": "gpt-4o-mini",            // GPT-4o → Mini (saves ~90%)
  "gpt-4o-mini": null,                // Can't downgrade further
};

// Downgrade triggers:
// - Daily budget > 80%: Downgrade non-essential tasks
// - Daily budget > 95%: Downgrade all tasks except essential
// - Essential tasks (never downgraded): review_reply, schema_markup, gbp_post (scheduled)
// - Non-essential tasks (downgradable): blog_post, content_refresh, site_evolution, aeo_content

4.3 Urgent vs. Batch Priority#

// Priority levels for queue ordering
enum GenerationPriority {
  CRITICAL = 1,   // Review replies (customer-facing, time-sensitive)
  HIGH = 2,        // GBP posts, social posts (scheduled content)
  NORMAL = 3,      // Landing page content, blog posts
  LOW = 4,         // Content refresh, SEO audits, reports
  BATCH = 5,       // Citation descriptions (30x bulk), alt text
}

// Queue implementation: BullMQ priority queue
// Higher priority jobs are processed first
// Batch jobs are processed during off-peak hours (2 AM - 6 AM IST)

5. Cost Management & Budgeting#

5.1 Cost Calculation#

// src/server/services/ai/cost/calculator.ts

function calculateCost(
  modelKey: ModelKey,
  inputTokens: number,
  outputTokens: number,
  cachedTokens?: number
): number {
  const config = MODEL_REGISTRY[modelKey];
  if (!config) throw new Error(`Unknown model: ${modelKey}`);
  
  const inputCost = (inputTokens / 1_000_000) * config.cost.input;
  const outputCost = (outputTokens / 1_000_000) * config.cost.output;
  const cachedCost = cachedTokens 
    ? (cachedTokens / 1_000_000) * (config.cost.cachedInput || config.cost.input)
    : 0;
  
  return roundTo(inputCost + outputCost - cachedCost, 6); // 6 decimal places
}

// Example: 1000 input + 500 output tokens on Claude Sonnet
// cost = (1000/1M)*3.00 + (500/1M)*15.00 = 0.003 + 0.0075 = $0.0105

// Example: 2000 input + 1000 output tokens on GPT-4o Mini
// cost = (2000/1M)*0.15 + (1000/1M)*0.60 = 0.0003 + 0.0006 = $0.0009

5.2 Budget Tracking (Redis)#

// Redis keys for budget tracking
// ai_budget:daily:{practiceId}  → counter (resets at UTC midnight)
// ai_budget:monthly:{practiceId} → counter (resets at UTC 1st of month)
// ai_budget:daily:system         → system-wide daily counter

// Budget check logic:
async function checkBudget(practiceId: string, plan: PlanType): Promise<BudgetStatus> {
  const dailyKey = `ai_budget:daily:${practiceId}`;
  const monthlyKey = `ai_budget:monthly:${practiceId}`;
  const systemKey = `ai_budget:daily:system`;
  
  const [dailySpent, monthlySpent, systemSpent] = await Promise.all([
    redis.get(dailyKey),
    redis.get(monthlyKey),
    redis.get(systemKey),
  ]);
  
  const limits = BUDGETS[plan];
  const dailyRemaining = limits.daily - (parseFloat(dailySpent || "0"));
  const monthlyRemaining = limits.monthly - (parseFloat(monthlySpent || "0"));
  
  return {
    dailyRemaining,
    monthlyRemaining,
    dailyPercent: (parseFloat(dailySpent || "0") / limits.daily) * 100,
    monthlyPercent: (parseFloat(monthlySpent || "0") / limits.monthly) * 100,
    systemDailyPercent: (parseFloat(systemSpent || "0") / SYSTEM_DAILY_LIMIT) * 100,
    canProceed: dailyRemaining > 0 && monthlyRemaining > 0,
    shouldDowngrade: dailyRemaining < limits.daily * 0.2, // < 20% remaining
  };
}

5.3 Cost Alert Thresholds#

Threshold	Action	Recipients	Channel
Practice > $50/day	Flag for review	Admin	Slack + Email
Practice > $500/month	Throttle non-essential	Admin + Client	Email
Practice > $1000/month	Hard stop (all tasks)	Admin + Client	Email + Slack
System > $1000/day	Throttle non-essential system-wide	Admin	Slack
System > $2000/day	Emergency stop	Admin + On-call	PagerDuty + Slack
Per-generation > $1.00	Log as expensive generation	Admin	Log only

5.4 Cost Optimization Strategies#

Prompt caching: Reuse system prompts across similar requests. Claude supports prompt caching at 50% discount.
Batch processing: Group 30 citation descriptions into a single batch call (if provider supports it).
Model downgrade: Use cheaper models for low-quality tasks (GPT-4o Mini for captions).
Response trimming: Set maxTokens aggressively. If a GBP post only needs 300 chars, don't allow 4000 tokens.
Cache TTL: Cache generated content for 24 hours. If the same request comes in again, serve from cache.
Off-peak processing: Run batch jobs (citations, SEO audits) during off-peak hours when API costs are sometimes lower.
Streaming for UI: Use streaming for real-time UI feedback, but don't stream for background jobs.

6. Fallback & Retry Strategy#

6.1 Fallback Chain#

Every model has a defined fallback. If the primary fails, the fallback is tried automatically.

// src/server/services/ai/fallback/chain.ts

const FALLBACK_CHAIN: Record<ModelKey, ModelKey | null> = {
  "claude-sonnet": "gpt-4o",          // Sonnet → GPT-4o (similar quality)
  "claude-haiku": "gpt-4o-mini",      // Haiku → GPT-4o Mini (similar speed/cost)
  "gpt-4o": "claude-sonnet",          // GPT-4o → Claude Sonnet (cross-provider)
  "gpt-4o-mini": "claude-haiku",      // Mini → Haiku (cross-provider)
  "gemini-flash": "gpt-4o-mini",      // Gemini → Mini (if Google fails)
};

// Fallback triggers:
// 1. Provider API error (5xx, 4xx)
// 2. Request timeout (>30s for fast models, >60s for slow)
// 3. Rate limit (429) from provider
// 4. Empty or malformed response
// 5. Circuit breaker open

// Fallback logic:
async function generateWithFallback(
  request: GenerateRequest,
  primaryModel: ModelKey
): Promise<GenerateResponse> {
  const models = [primaryModel];
  let fallback = FALLBACK_CHAIN[primaryModel];
  while (fallback) {
    models.push(fallback);
    fallback = FALLBACK_CHAIN[fallback];
  }
  
  for (const model of models) {
    try {
      return await generateWithModel(request, model);
    } catch (error) {
      logger.warn({ model, error: error.message }, "Model failed, trying fallback");
      // Continue to next model
    }
  }
  
  // All models failed
  throw new FallbackExhaustedError(`All models failed: ${models.join(" → ")}`);
}

6.2 Retry Policy#

// Retry configuration per model
const RETRY_CONFIG: Record<ModelKey, RetryConfig> = {
  "claude-sonnet": { maxAttempts: 3, backoffMs: 1000, maxBackoffMs: 10000, retryableErrors: ["timeout", "5xx", "429"] },
  "claude-haiku": { maxAttempts: 3, backoffMs: 500, maxBackoffMs: 5000, retryableErrors: ["timeout", "5xx", "429"] },
  "gpt-4o": { maxAttempts: 3, backoffMs: 1000, maxBackoffMs: 10000, retryableErrors: ["timeout", "5xx", "429"] },
  "gpt-4o-mini": { maxAttempts: 3, backoffMs: 500, maxBackoffMs: 5000, retryableErrors: ["timeout", "5xx", "429"] },
};

// Exponential backoff with jitter:
// delay = min(backoffMs * 2^(attempt-1) + random(0, 1000), maxBackoffMs)

6.3 Dead Letter Queue (DLQ)#

If all models fail after retries, the request goes to the Dead Letter Queue:

// DLQ handling:
// 1. Store failed request in `ai_dlq` table with:
//    - request, error messages, attempted models, timestamp
// 2. Alert admin via Slack: "AI generation failed for {task} / {practiceId}"
// 3. For customer-facing tasks: queue for manual content creation
// 4. For background tasks: retry from DLQ after 1 hour (up to 3x)
// 5. Admin can retry individual items from `/admin/ai-evaluation` DLQ tab

7. Circuit Breaker Pattern#

7.1 Circuit Breaker Configuration#

// src/server/services/ai/circuit-breaker.ts

interface CircuitBreakerConfig {
  failureThreshold: number;      // Failures before opening
  failureWindowMs: number;         // Time window for counting failures
  openDurationMs: number;        // How long to stay open
  halfOpenMaxCalls: number;      // Max test calls in half-open state
  halfOpenSuccessThreshold: number; // Successes needed to close
}

const CIRCUIT_BREAKERS: Record<ProviderKey, CircuitBreakerConfig> = {
  "anthropic": {
    failureThreshold: 5,
    failureWindowMs: 60_000,      // 5 failures in 1 minute
    openDurationMs: 30_000,        // Open for 30 seconds
    halfOpenMaxCalls: 1,           // 1 test call
    halfOpenSuccessThreshold: 1,   // 1 success to close
  },
  "openai": {
    failureThreshold: 5,
    failureWindowMs: 60_000,
    openDurationMs: 30_000,
    halfOpenMaxCalls: 1,
    halfOpenSuccessThreshold: 1,
  },
  "google": {
    failureThreshold: 3,
    failureWindowMs: 60_000,
    openDurationMs: 60_000,        // Longer for Google (sometimes flaky)
    halfOpenMaxCalls: 1,
    halfOpenSuccessThreshold: 1,
  },
};

// States:
// CLOSED → Normal operation
// OPEN → Block all requests, return error immediately (save cost)
// HALF_OPEN → Allow 1 test request, monitor result

7.2 Circuit Breaker State Machine#

┌─────────┐     5 failures in 60s      ┌─────────┐
│ CLOSED  │ ────────────────────────→ │  OPEN   │
│ (normal)│                           │ (block) │
└─────────┘                           └────┬────┘
     ▲                                       │
     │  1 success in half-open                │  30s timeout
     │                                       │
     │      ┌─────────┐                      │
     └──────│HALF_OPEN│ ←────────────────────┘
            │ (test)  │
            └────┬────┘
                 │
                 │  1 failure in half-open
                 ▼
            ┌─────────┐
            │  OPEN   │ (reset timer)
            └─────────┘

7.3 Circuit Breaker Implementation#

class CircuitBreaker {
  private state: "CLOSED" | "OPEN" | "HALF_OPEN" = "CLOSED";
  private failures: number[] = []; // timestamps of failures
  private halfOpenCalls = 0;
  private halfOpenSuccesses = 0;
  private openTimer: NodeJS.Timeout | null = null;
  
  constructor(private config: CircuitBreakerConfig) {}
  
  async call<T>(fn: () => Promise<T>): Promise<T> {
    if (this.state === "OPEN") {
      throw new CircuitOpenError("Circuit breaker is OPEN");
    }
    
    if (this.state === "HALF_OPEN" && this.halfOpenCalls >= this.config.halfOpenMaxCalls) {
      throw new CircuitOpenError("Circuit breaker is HALF_OPEN, max test calls reached");
    }
    
    if (this.state === "HALF_OPEN") {
      this.halfOpenCalls++;
    }
    
    try {
      const result = await fn();
      this.onSuccess();
      return result;
    } catch (error) {
      this.onFailure();
      throw error;
    }
  }
  
  private onSuccess() {
    if (this.state === "HALF_OPEN") {
      this.halfOpenSuccesses++;
      if (this.halfOpenSuccesses >= this.config.halfOpenSuccessThreshold) {
        this.close();
      }
    }
  }
  
  private onFailure() {
    const now = Date.now();
    this.failures = this.failures.filter(t => now - t < this.config.failureWindowMs);
    this.failures.push(now);
    
    if (this.failures.length >= this.config.failureThreshold) {
      this.open();
    }
  }
  
  private open() {
    this.state = "OPEN";
    this.openTimer = setTimeout(() => this.halfOpen(), this.config.openDurationMs);
    logger.warn({ provider: this.config.provider }, "Circuit breaker OPENED");
  }
  
  private halfOpen() {
    this.state = "HALF_OPEN";
    this.halfOpenCalls = 0;
    this.halfOpenSuccesses = 0;
    logger.info({ provider: this.config.provider }, "Circuit breaker HALF_OPEN");
  }
  
  private close() {
    this.state = "CLOSED";
    this.failures = [];
    if (this.openTimer) clearTimeout(this.openTimer);
    logger.info({ provider: this.config.provider }, "Circuit breaker CLOSED");
  }
}

8. Provider Integrations#

8.1 Anthropic (Claude) Adapter#

// src/server/services/ai/providers/anthropic.ts

import { createAnthropic } from "@ai-sdk/anthropic";
import { generateText, streamText } from "ai";

const anthropic = createAnthropic({
  apiKey: process.env.ANTHROPIC_API_KEY,
});

export async function generateWithAnthropic(
  modelId: string,
  options: GenerateOptions
): Promise<ProviderResult> {
  const startTime = Date.now();
  
  const result = await generateText({
    model: anthropic(modelId),
    system: options.systemPrompt,
    prompt: options.prompt,
    maxTokens: options.maxTokens || 1024,
    temperature: options.temperature || 0.7,
    ...(options.jsonMode ? { responseFormat: { type: "json_object" } } : {}),
  });
  
  return {
    text: result.text,
    usage: {
      inputTokens: result.usage.promptTokens,
      outputTokens: result.usage.completionTokens,
      totalTokens: result.usage.promptTokens + result.usage.completionTokens,
    },
    latencyMs: Date.now() - startTime,
  };
}

export async function streamWithAnthropic(
  modelId: string,
  options: GenerateOptions
) {
  return streamText({
    model: anthropic(modelId),
    system: options.systemPrompt,
    prompt: options.prompt,
    maxTokens: options.maxTokens || 1024,
    temperature: options.temperature || 0.7,
  });
}

8.2 OpenAI (GPT-4o) Adapter#

// src/server/services/ai/providers/openai.ts

import { createOpenAI } from "@ai-sdk/openai";
import { generateText, streamText } from "ai";

const openai = createOpenAI({
  apiKey: process.env.OPENAI_API_KEY,
});

export async function generateWithOpenAI(
  modelId: string,
  options: GenerateOptions
): Promise<ProviderResult> {
  const startTime = Date.now();
  
  const result = await generateText({
    model: openai(modelId),
    system: options.systemPrompt,
    prompt: options.prompt,
    maxTokens: options.maxTokens || 1024,
    temperature: options.temperature || 0.7,
    ...(options.jsonMode ? { responseFormat: { type: "json_object" } } : {}),
  });
  
  return {
    text: result.text,
    usage: {
      inputTokens: result.usage.promptTokens,
      outputTokens: result.usage.completionTokens,
      totalTokens: result.usage.totalTokens,
    },
    latencyMs: Date.now() - startTime,
  };
}

export async function streamWithOpenAI(
  modelId: string,
  options: GenerateOptions
) {
  return streamText({
    model: openai(modelId),
    system: options.systemPrompt,
    prompt: options.prompt,
    maxTokens: options.maxTokens || 1024,
    temperature: options.temperature || 0.7,
  });
}

8.3 Google (Gemini) Adapter#

// src/server/services/ai/providers/google.ts

import { createGoogleGenerativeAI } from "@ai-sdk/google";
import { generateText, streamText } from "ai";

const google = createGoogleGenerativeAI({
  apiKey: process.env.GOOGLE_API_KEY,
});

export async function generateWithGoogle(
  modelId: string,
  options: GenerateOptions
): Promise<ProviderResult> {
  const startTime = Date.now();
  
  const result = await generateText({
    model: google(modelId),
    system: options.systemPrompt,
    prompt: options.prompt,
    maxTokens: options.maxTokens || 1024,
    temperature: options.temperature || 0.7,
  });
  
  return {
    text: result.text,
    usage: {
      inputTokens: result.usage.promptTokens,
      outputTokens: result.usage.completionTokens,
      totalTokens: result.usage.totalTokens,
    },
    latencyMs: Date.now() - startTime,
  };
}

8.4 Provider Selection Strategy#

Provider	Strengths	Weaknesses	Best For
Anthropic (Claude)	Excellent reasoning, low hallucination, great system prompt adherence, medical accuracy	Slightly more expensive, slower for bulk	Landing pages, medical content, complex reasoning, JSON-LD
OpenAI (GPT-4o)	Fast, great JSON mode, good vision, reliable	Can be verbose, occasionally off-topic	Structured data, keyword research, quick captions
Google (Gemini)	Very long context (1M tokens), cheap, good vision	Less reliable for complex reasoning, newer	Image prompts, alt text, long-context summarization

9. Streaming & Real-Time Inference#

9.1 Streaming Use Cases#

Streaming is used for UI interactions where the user wants to see content appear in real-time:

Use Case	Task	Model	Why Streaming
Content Editor AI Assist	`content_edit`	Claude Sonnet	User sees text appear as they type
Landing Page Live Preview	`section_regenerate`	Claude Sonnet	Admin sees preview update in real-time
Blog Post Draft	`blog_post`	Claude Sonnet	Writer sees content flow in
Review Reply Suggest	`review_reply`	Claude Haiku	User sees reply suggestion appear
Chat Support	`support_reply`	Claude Haiku	Conversational feel

9.2 Streaming Implementation#

// src/server/services/ai/streaming.ts

import { streamText } from "ai";

export async function streamGeneration(
  request: GenerateRequest
): Promise<ReadableStream> {
  const model = await selectModel(request);
  const provider = MODEL_REGISTRY[model].provider;
  const template = await resolveTemplate(request);
  const prompt = substituteVariables(template, request.variables);
  
  const stream = await streamText({
    model: getProviderModel(provider, model),
    system: template.systemPrompt,
    prompt,
    maxTokens: request.maxTokens || 1024,
    temperature: request.temperature || 0.7,
  });
  
  // Log start of stream
  logger.info({
    event: "ai_stream_start",
    task: request.task,
    model,
    practiceId: request.practiceId,
  });
  
  return stream.toDataStreamResponse();
}

9.3 Non-Streaming (Background Jobs)#

All background jobs use non-streaming for efficiency:

GBP post generation
Social caption generation
Citation descriptions
Review replies
Content refresh
SEO audits
Monthly reports

10. Batch Processing#

10.1 Batch Use Cases#

Batch	Size	Task	Optimization
Citation descriptions	30	`citation_description`	Single prompt with "Generate 30 unique descriptions"
Alt text generation	5-10	`alt_text`	Single prompt with all images
Hashtag sets	10-15	`hashtag_generation`	Single prompt per post
Review replies	10-20	`review_reply`	Individual prompts (each reply is unique)
Social captions (weekly)	4-8	`social_caption`	Individual prompts per platform

10.2 Batch Prompt Optimization#

// For citation descriptions (30x in one call):
// Instead of 30 individual API calls:
// "Generate 30 unique business descriptions for the following directories...
//  Directory 1: Justdial - tone: professional, length: 150 words
//  Directory 2: Practo - tone: medical, length: 200 words
//  ..."
//
// This reduces API calls from 30 to 1, but increases token count.
// Cost comparison:
// 30 individual calls: 30 * (500 input + 200 output) = 15K input + 6K output
// 1 batch call: 1 * (8000 input + 6000 output) = 8K input + 6K output
// Savings: ~45% on input tokens (fewer system prompts)
// Risk: If one description fails, all fail. Mitigation: validate each output.

11. Monitoring & Observability#

11.1 Metrics Dashboard#

The admin dashboard (/admin/ai-evaluation) shows real-time AI metrics:

Metric	Source	Refresh	Alert Threshold
Generation rate	`ai_usage` table	10s	> 100/min
Average latency	`ai_usage` table	10s	> 5000ms
Error rate	`ai_usage` table	10s	> 5%
Cost per minute	`ai_usage` table	10s	> $10/min
Cost per client (top 10)	`ai_usage` table	1min	> $50/day
Model distribution	`ai_usage` table	1min	—
Quality score distribution	`ai_usage` table	5min	< 60% avg
Compliance failure rate	`ai_usage` table	5min	> 1%
Cache hit rate	Redis	1min	< 50%
Circuit breaker status	In-memory	Real-time	Any OPEN

11.2 Structured Logging#

// Every generation is logged with structured data:
logger.info({
  event: "ai_generation",
  task: request.task,
  model: response.metadata.model,
  provider: response.metadata.provider,
  tokensUsed: response.metadata.tokensUsed.total,
  inputTokens: response.metadata.tokensUsed.input,
  outputTokens: response.metadata.tokensUsed.output,
  costUsd: response.metadata.costUsd,
  latencyMs: response.metadata.latencyMs,
  qualityScore: response.metadata.qualityScore,
  complianceStatus: response.metadata.complianceStatus,
  promptVersion: response.metadata.promptVersion,
  fallbackUsed: response.metadata.fallbackUsed,
  cacheHit: response.metadata.cacheHit,
  practiceId: request.practiceId,
  userId: request.userId,
  requestId: request.requestId,
}, "AI generation completed");

11.3 Health Check Endpoint#

// GET /api/health/ai
// Returns:
{
  "status": "ok",
  "providers": {
    "anthropic": { "status": "ok", "circuitBreaker": "CLOSED", "lastError": null },
    "openai": { "status": "ok", "circuitBreaker": "CLOSED", "lastError": null },
    "google": { "status": "ok", "circuitBreaker": "CLOSED", "lastError": null }
  },
  "models": {
    "claude-sonnet": { "available": true, "avgLatencyMs": 2500 },
    "claude-haiku": { "available": true, "avgLatencyMs": 800 },
    "gpt-4o": { "available": true, "avgLatencyMs": 1800 },
    "gpt-4o-mini": { "available": true, "avgLatencyMs": 500 }
  },
  "queues": {
    "ai_generation": { "waiting": 12, "active": 3, "completed": 1450, "failed": 2 }
  },
  "costs": {
    "today": 45.20,
    "thisMonth": 890.50,
    "topClient": { "practiceId": "xxx", "cost": 12.50 }
  }
}

12. Configuration Schema#

12.1 Environment Variables#

# Provider API Keys
ANTHROPIC_API_KEY=sk-ant-...
OPENAI_API_KEY=sk-...
GOOGLE_API_KEY=AIza...

# Cost Limits
AI_SYSTEM_DAILY_LIMIT=2000           # USD per day system-wide
AI_PRACTICE_DAILY_LIMIT_DEFAULT=50   # USD per practice per day (can be overridden by plan)

# Circuit Breaker
AI_CIRCUIT_FAILURE_THRESHOLD=5
AI_CIRCUIT_OPEN_DURATION_MS=30000

# Cache
AI_CACHE_TTL_SECONDS=86400           # 24 hours
AI_CACHE_MAX_SIZE=10000              # Max cached entries

# Rate Limits
AI_RATE_LIMIT_PER_MINUTE_DEFAULT=100
AI_RATE_LIMIT_PER_DAY_DEFAULT=1000

# Fallback
AI_FALLBACK_ENABLED=true
AI_MAX_FALLBACK_ATTEMPTS=2

# Timeouts
AI_TIMEOUT_FAST_MS=30000             # Haiku, Mini
AI_TIMEOUT_SLOW_MS=60000             # Sonnet, GPT-4o

12.2 Database Schema#

-- AI usage tracking table
CREATE TABLE ai_usage (
  id UUID PRIMARY KEY DEFAULT gen_random_uuid(),
  request_id UUID NOT NULL,
  practice_id UUID REFERENCES practices(id),
  user_id UUID REFERENCES users(id),
  task_type VARCHAR(50) NOT NULL,
  model VARCHAR(50) NOT NULL,
  provider VARCHAR(20) NOT NULL,
  input_tokens INTEGER NOT NULL,
  output_tokens INTEGER NOT NULL,
  cost_usd DECIMAL(10, 6) NOT NULL,
  latency_ms INTEGER NOT NULL,
  quality_score INTEGER,
  compliance_status VARCHAR(10) NOT NULL,
  prompt_version VARCHAR(20) NOT NULL,
  fallback_used BOOLEAN DEFAULT FALSE,
  fallback_from VARCHAR(50),
  cache_hit BOOLEAN DEFAULT FALSE,
  content_preview TEXT,                -- First 200 chars of output (for debugging)
  created_at TIMESTAMP DEFAULT NOW(),
  
  INDEX idx_practice_task (practice_id, task_type),
  INDEX idx_created_at (created_at),
  INDEX idx_model (model),
  INDEX idx_compliance (compliance_status)
);

-- AI dead letter queue
CREATE TABLE ai_dlq (
  id UUID PRIMARY KEY DEFAULT gen_random_uuid(),
  request JSONB NOT NULL,
  error_messages JSONB NOT NULL,
  attempted_models JSONB NOT NULL,
  retry_count INTEGER DEFAULT 0,
  max_retries INTEGER DEFAULT 3,
  status VARCHAR(20) DEFAULT "pending",
  created_at TIMESTAMP DEFAULT NOW(),
  resolved_at TIMESTAMP
);

End of AI Architecture Documentation — RankFlow AI v2.0.0