RankFlow AI — AI Services Test Specification

Version: 1.0.0
Date: 2026-06-13
Spec: docs/specs/ai-services.md
Scope: src/server/services/ai/
Providers: Anthropic (Claude), OpenAI (GPT-4o)

---

1. Multi-LLM Router#

Test Plan & Verification#

Unit Tests#

Integration Tests#

Success Criteria (Binary)#

• Router resolves the correct default model for every supported task type (12 tasks).
• Client model override bypasses default mapping and is validated against supported models.
• Unknown task types throw RouterError with a clear message before any provider call.
• GenerateResult always contains all 6 required fields with correct types.
• latencyMs is a positive integer measured from just before the provider call to just after.
• jsonMode: true forces JSON output for providers that support it; fails loudly for unsupported.
• tRPC content.generate mutation returns a full content record with AI metadata populated.

Agent Context (Pre-conditions)#

• Required DB state: PromptTemplate rows for at least one global default and one practice-specific override per task type.
• Required env vars: ANTHROPIC_API_KEY (mocked), OPENAI_API_KEY (mocked), DATABASE_URL (test DB).
• Required external mocks: Anthropic generateText and OpenAI generateText mocked via msw or vitest spies.

Verification Commands#

# Run router tests
pnpm test:unit -- src/__tests__/unit/ai/router.test.ts
pnpm test:integration -- src/__tests__/integration/ai/router.test.ts

---

2. Model Configuration#

Test Plan & Verification#

Unit Tests#

Integration Tests#

Success Criteria (Binary)#

• Every supported model key (claude-sonnet, claude-haiku, gpt-4o, gpt-4o-mini) resolves to correct provider, model ID, cost rates, and context window.
• Unsupported model keys throw ConfigError immediately; no fallback or silent defaulting.
• Cost calculation uses the exact formula: (inputTokens / 1_000_000) * inputCost + (outputTokens / 1_000_000) * outputCost.
• Cost output is a non-negative finite number, rounded to 6 decimal places.
• maxTokens is validated against the model's context window; requests exceeding it are rejected before the provider call.
• Missing provider API keys cause initialization failures with explicit error messages.

Agent Context (Pre-conditions)#

• Required DB state: None (pure config lookup).
• Required env vars: ANTHROPIC_API_KEY and OPENAI_API_KEY present or explicitly unset for negative tests.
• Required external mocks: None.

Verification Commands#

# Run model config tests
pnpm test:unit -- src/__tests__/unit/ai/model-config.test.ts
pnpm test:integration -- src/__tests__/integration/ai/model-config.test.ts

---

3. Task-to-Model Mapping#

Test Plan & Verification#

Unit Tests#

Integration Tests#

Success Criteria (Binary)#

• All 12 supported task types map to their documented default models exactly.
• Practice-level template overrides (modelConfig in DB) take precedence over global defaults.
• Client-level model option in GenerateOptions takes precedence over all DB-level defaults.
• Missing templates do not break model selection; hardcoded defaults are used.
• Model selection resolution order is: client override → practice default → global default → hardcoded fallback.

Agent Context (Pre-conditions)#

• Required DB state: PromptTemplate rows with modelConfig overrides for at least one practice and one global default.
• Required env vars: DATABASE_URL (test DB).
• Required external mocks: Provider SDKs stubbed to verify which model is actually called.

Verification Commands#

# Run task mapping tests
pnpm test:unit -- src/__tests__/unit/ai/task-mapping.test.ts
pnpm test:integration -- src/__tests__/integration/ai/task-mapping.test.ts

---

4. Prompt Template System#

Test Plan & Verification#

Unit Tests#

Integration Tests#

Success Criteria (Binary)#

• Template resolution order is strictly: practice-specific (latest version, isDefault=true) → global default (latest version, isDefault=true) → hardcoded fallback.
• Variable substitution replaces all {{key}} occurrences with the corresponding value; unmatched keys are left unchanged or raise an error based on config.
• Template variables are treated as plain text; no injection of HTML, SQL, or script tags into prompts.
• A/B promotion auto-promotes a variant only when its performanceScore exceeds the control by ≥ 0.3.
• outputSchema is forwarded to the provider when jsonMode is enabled; invalid JSON output triggers a retry or error.
• modelConfig from the template is applied to the generation call, overriding global defaults but not client overrides.

Agent Context (Pre-conditions)#

• Required DB state: PromptTemplate rows with practice-specific, global, A/B variants, and versioned records.
• Required env vars: DATABASE_URL (test DB).
• Required external mocks: Provider SDKs to inspect the exact prompt and options sent.

Verification Commands#

# Run template system tests
pnpm test:unit -- src/__tests__/unit/ai/templates.test.ts
pnpm test:integration -- src/__tests__/integration/ai/templates.test.ts

---

5. Cost Tracking#

Test Plan & Verification#

Unit Tests#

Integration Tests#

Success Criteria (Binary)#

• Per-request log contains all 11 required fields with correct data types.
• Cost calculation is exact to 6 decimal places and uses the correct per-model rates.
• Daily aggregation by practice is accurate and queryable.
• Monthly aggregation by practice is accurate and queryable.
• System-wide daily aggregation is accurate and triggers throttling when > $1000.
• Alerts fire once per threshold per day; no duplicate noise.
• Throttling blocks non-essential tasks but never blocks essential tasks (review replies, audit reports).
• Cost metadata is persisted to the Content DB record and returned in the tRPC response.

Agent Context (Pre-conditions)#

• Required DB state: Content records with costUsd, aiTokensUsed, aiProvider, aiModel, createdAt for aggregation queries.
• Required env vars: ADMIN_ALERT_WEBHOOK_URL (mocked), DATABASE_URL (test DB).
• Required external mocks: Alert delivery endpoint (webhook/Resend) mocked.

Verification Commands#

# Run cost tracking tests
pnpm test:unit -- src/__tests__/unit/ai/cost-tracking.test.ts
pnpm test:integration -- src/__tests__/integration/ai/cost-tracking.test.ts

---

6. Fallback Chain#

Test Plan & Verification#

Unit Tests#

Integration Tests#

Success Criteria (Binary)#

• Every primary model has exactly one documented fallback model; fallback rules are bi-directional and exhaustive.
• Primary failure triggers fallback within < 100ms of error detection (no retry loops on primary).
• Fallback success returns a valid GenerateResult with model and provider reflecting the fallback, not the primary.
• Both primary and fallback failure throws FallbackExhaustedError with both errors, logs an incident, and notifies admin.
• Circuit breaker opens after 5 failures in 60 seconds; blocks all requests for 30 seconds.
• Circuit breaker transitions to half-open after 30 seconds or 1 success; allows 1 test request.
• Circuit breaker closes after 3 consecutive successes in half-open state.
• Open circuit rejects requests immediately without calling any provider (saves cost and latency).

Agent Context (Pre-conditions)#

• Required DB state: Minimal PromptTemplate rows so generation can proceed.
• Required env vars: ANTHROPIC_API_KEY, OPENAI_API_KEY (both mocked).
• Required external mocks: Anthropic and OpenAI generateText mocked with configurable failure modes (timeout, 500, 429, success).

Verification Commands#

# Run fallback chain tests
pnpm test:unit -- src/__tests__/unit/ai/fallback.test.ts
pnpm test:integration -- src/__tests__/integration/ai/fallback.test.ts

---

7. Provider Integrations#

Test Plan & Verification#

Unit Tests#

Integration Tests#

Mock Responses#

Anthropic Success Mock

{
  "id": "msg_01XgYxVjCJ8Kz3vRq9LmNpQ2",
  "type": "message",
  "role": "assistant",
  "content": [
    { "type": "text", "text": "Discover the latest in dental implant technology at Dr. Smith Dental Clinic. Schedule your consultation today!" }
  ],
  "model": "claude-sonnet-4-20250514",
  "usage": {
    "input_tokens": 120,
    "output_tokens": 45
  }
}

Anthropic Error Mock (500)

{
  "type": "error",
  "error": {
    "type": "api_error",
    "message": "Internal server error"
  }
}

Anthropic Error Mock (429)

{
  "type": "error",
  "error": {
    "type": "rate_limit_error",
    "message": "Rate limit exceeded"
  }
}

OpenAI Success Mock

{
  "id": "chatcmpl-abc123",
  "object": "chat.completion",
  "model": "gpt-4o",
  "choices": [
    {
      "index": 0,
      "message": {
        "role": "assistant",
        "content": "Best dentist in Kochi | Dr. Smith Dental Clinic offers top-rated teeth whitening, root canal, and braces treatments."
      },
      "finish_reason": "stop"
    }
  ],
  "usage": {
    "prompt_tokens": 85,
    "completion_tokens": 32,
    "total_tokens": 117
  }
}

OpenAI Error Mock (500)

{
  "error": {
    "message": "The server had an error while processing your request.",
    "type": "server_error",
    "param": null,
    "code": null
  }
}

OpenAI Error Mock (429)

{
  "error": {
    "message": "Rate limit reached",
    "type": "requests",
    "param": null,
    "code": "rate_limit_exceeded"
  }
}

Success Criteria (Binary)#

• Anthropic provider initializes with createAnthropic and calls generateText with all options passed through.
• OpenAI provider initializes with createOpenAI and calls generateText with response_format when jsonMode is true.
• Both providers return GenerateResult with text, model, provider, tokensUsed, costUsd, and latencyMs.
• Token usage is extracted from usage.promptTokens + usage.completionTokens (Anthropic) or usage.totalTokens (OpenAI).
• Unsupported providers throw ProviderError immediately; no silent fallback to a default provider.
• Malformed API responses (missing usage, missing text) throw ProviderError with a clear message.
• Streaming mode returns a valid data stream with text chunks for both providers.
• Mock responses for success, 500, and 429 cases are documented and used in contract tests.

Agent Context (Pre-conditions)#

• Required DB state: None (provider-level tests).
• Required env vars: ANTHROPIC_API_KEY (mocked), OPENAI_API_KEY (mocked).
• Required external mocks: msw handlers for https://api.anthropic.com/v1/messages and https://api.openai.com/v1/chat/completions returning the mock responses above.

Verification Commands#

# Run provider integration tests
pnpm test:unit -- src/__tests__/unit/ai/providers.test.ts
pnpm test:integration -- src/__tests__/integration/ai/providers.test.ts
pnpm test:contract -- src/__tests__/contract/ai/providers.test.ts

---

8. Content Types#

Test Plan & Verification#

Unit Tests#

Integration Tests#

Compliance & Medical Safety Tests#

Success Criteria (Binary)#

• GBP post output is 150–300 characters and contains a call-to-action.
• Directory profile article is 500–1000 words with keyword density < 2%.
• Review replies are empathetic for negative reviews and never contain PHI (patient names, contact info, dates).
• Citation descriptions are 100–200 words and unique per directory (no duplicates for the same practice).
• Social captions respect platform-specific character limits and hashtag rules.
• Schema markup is valid JSON-LD with @context: "https://schema.org" and a valid @type.
• FAQ outputs contain 5–10 Q&A pairs with accurate medical information.
• Medical safety filter rejects content with banned phrases (100% cure, guaranteed results, etc.) and triggers regeneration.
• Drug claim detection flags content with drug names + unverified claims for human review.
• HIPAA filter strips PHI from review replies; if stripping fails twice, the content is rejected.
• Medical advice content either includes a disclaimer or is tagged needs_disclaimer before publish.
• AI content approval triggers a ConsentLog record with IP, timestamp, and user ID.

Agent Context (Pre-conditions)#

• Required DB state: PromptTemplate rows for each content type; ConsentLog table empty before approval tests.
• Required env vars: ANTHROPIC_API_KEY, OPENAI_API_KEY (mocked); DATABASE_URL (test DB).
• Required external mocks: Provider SDKs returning content that passes and fails compliance checks.
• Required fixtures: Pre-defined mock responses that contain banned phrases and PHI for negative tests.

Verification Commands#

# Run content type tests
pnpm test:unit -- src/__tests__/unit/ai/content-types.test.ts
pnpm test:integration -- src/__tests__/integration/ai/content-types.test.ts
pnpm test:compliance -- src/__tests__/compliance/ai-safety.test.ts

---

9. Verification Commands#

Full AI Services Test Suite#

# ───────────────────────────────────────────────
# 1. Unit Tests (fast, isolated)
# ───────────────────────────────────────────────
pnpm test:unit -- src/__tests__/unit/ai/

# Individual unit test files
pnpm test:unit -- src/__tests__/unit/ai/router.test.ts
pnpm test:unit -- src/__tests__/unit/ai/model-config.test.ts
pnpm test:unit -- src/__tests__/unit/ai/task-mapping.test.ts
pnpm test:unit -- src/__tests__/unit/ai/templates.test.ts
pnpm test:unit -- src/__tests__/unit/ai/cost-tracking.test.ts
pnpm test:unit -- src/__tests__/unit/ai/fallback.test.ts
pnpm test:unit -- src/__tests__/unit/ai/providers.test.ts
pnpm test:unit -- src/__tests__/unit/ai/content-types.test.ts

# ───────────────────────────────────────────────
# 2. Integration Tests (DB + mocked providers)
# ───────────────────────────────────────────────
pnpm test:integration -- src/__tests__/integration/ai/

# Individual integration test files
pnpm test:integration -- src/__tests__/integration/ai/router.test.ts
pnpm test:integration -- src/__tests__/integration/ai/model-config.test.ts
pnpm test:integration -- src/__tests__/integration/ai/task-mapping.test.ts
pnpm test:integration -- src/__tests__/integration/ai/templates.test.ts
pnpm test:integration -- src/__tests__/integration/ai/cost-tracking.test.ts
pnpm test:integration -- src/__tests__/integration/ai/fallback.test.ts
pnpm test:integration -- src/__tests__/integration/ai/providers.test.ts
pnpm test:integration -- src/__tests__/integration/ai/content-types.test.ts

# ───────────────────────────────────────────────
# 3. Contract Tests (provider API shapes)
# ───────────────────────────────────────────────
pnpm test:contract -- src/__tests__/contract/ai/providers.test.ts

# ───────────────────────────────────────────────
# 4. Compliance Tests (medical safety)
# ───────────────────────────────────────────────
pnpm test:compliance -- src/__tests__/compliance/ai-safety.test.ts

# ───────────────────────────────────────────────
# 5. Full AI Services Suite (all categories)
# ───────────────────────────────────────────────
pnpm test -- src/__tests__/unit/ai/ src/__tests__/integration/ai/ src/__tests__/compliance/ai-safety.test.ts

Coverage Target#

# Run with coverage report
pnpm test:unit --coverage -- src/__tests__/unit/ai/

Pre-commit Gate#

# All AI tests must pass before any commit touching src/server/services/ai/
pnpm test:unit -- src/__tests__/unit/ai/ && pnpm test:integration -- src/__tests__/integration/ai/

---

End of AI Services Test Specification — RankFlow AI v1.0.0