RankFlow AI — Job Queue & Workflow Engine Test Specification

Version: 1.0.0
Date: 2026-06-13
Target Spec: docs/specs/job-queue-workflows.md
Systems Under Test: Inngest (durable workflows) + BullMQ (background jobs) + Redis (queue + cache)
Test Tooling: Vitest, msw, ioredis-mock, bullmq test helpers, inngest test utils

---

1. Inngest Workflow Engine Tests#

1.1 Unit Tests — Job Processor Logic#

1.2 Integration Tests — Inngest Step Functions#

1.3 Failure/Recovery Tests — Inngest#

1.4 Concurrency Limit Tests — Inngest#

1.5 Success Criteria (Binary) — Inngest#

• All 8 onboarding steps execute in correct order when triggered with valid input
• OAuth wait step resolves within 7-day timeout when callback is received
• OAuth wait step fails gracefully (no infinite hang) when callback never arrives
• Citation builder submits to all active directories and stores results with correct status mapping
• Parallel directory submissions respect per-workflow concurrency limit (3 for citation)
• Cron-triggered workflows (profile-refresh, monthly-report, token-refresh) execute on schedule
• Inngest retries failed step.run up to the configured retry count (3) with exponential backoff
• Non-retriable conditions (missing practice) fail immediately without retry waste
• Workflow concurrency limits are enforced and excess flows queue correctly
• Sleep steps resume accurately after simulated time advance

1.6 Agent Context (Pre-conditions) — Inngest#

• Required DB state: practice, location, citationDirectory tables with at least 1 mock record each; citationDirectory.isActive = true for test directories
• Required env vars: INNGEST_EVENT_KEY, INNGEST_API_URL (mocked in test)
• Required external mocks: profile.generate, profile.publish, email.send, gbpAuth.getAuthUrl, ai.generate, submitViaApi, submitViaBrowser — all mocked via msw or vi.fn()
• Mock Redis state: ioredis-mock instance initialized with empty queue state; no prior jobs lingering
• Inngest test harness: inngest.test() or equivalent test runner configured to simulate events and step execution

1.7 Verification Commands — Inngest#

# Unit: step logic
pnpm test:unit -- src/__tests__/unit/inngest/

# Integration: full workflows
pnpm test:integration -- src/__tests__/integration/inngest/

# Retry & failure scenarios
pnpm test:integration -- src/__tests__/integration/inngest/onboarding.retry.test.ts
pnpm test:integration -- src/__tests__/integration/inngest/citation.retry.test.ts

# Concurrency
pnpm test:integration -- src/__tests__/integration/inngest/concurrency.test.ts

# Sleep/resume durability
pnpm test:integration -- src/__tests__/integration/inngest/sleep-resume.test.ts

---

2. BullMQ Job Queue Tests#

2.1 Unit Tests — Job Processor Logic#

2.2 Integration Tests — BullMQ Queue Operations#

2.3 Failure/Recovery Tests — BullMQ#

2.4 Concurrency Limit Tests — BullMQ#

2.5 Scheduling Tests — BullMQ Cron#

2.6 Success Criteria (Binary) — BullMQ#

• All 8 BullMQ queue types accept and store jobs with correct payload shape
• Workers process jobs from their assigned queue with zero cross-queue leakage
• Job processors execute the correct business logic and return expected result shapes
• Exponential backoff retries function correctly (delay doubles per attempt)
• Retry exhaustion moves job to Dead Letter Queue with full context preserved
• Admin retry from DLQ re-queues job to original queue with same originalJobId
• Worker concurrency limits are strictly enforced (3 GBP, 5 social, 1 token-refresh, etc.)
• Rate limiters throttle job throughput within configured windows (30/min GBP, 25/min social)
• All 6 cron schedules produce correct next-run times for their expressions
• Cron jobs execute on time within ±1 minute tolerance under normal load
• Graceful shutdown pauses workers, lets active jobs complete, and closes cleanly
• Worker crash recovery detects stalled jobs and re-queues them without data loss
• Delayed jobs remain in delayed state until expiration, then process correctly
• Repeatable jobs can be added, queried, and removed correctly

2.7 Agent Context (Pre-conditions) — BullMQ#

• Required DB state: practice, location, oauthToken, citation, review tables with mock records
• Required env vars: REDIS_URL pointing to ioredis-mock instance; BULLMQ_PREFIX (optional)
• Required external mocks: GBP API (msw), social platform APIs (msw), email provider (msw), Firecrawl/NAP scraper (msw), PDF generator (vi.fn())
• Mock Redis state: Fresh ioredis-mock instance per test; no pre-existing jobs; flushall before each test suite
• Queue state: All 8 queues initialized but empty; worker instances created but not started until test action
• Time control: vi.useFakeTimers() or bullmq test helpers for advancing cron/delayed job time

2.8 Verification Commands — BullMQ#

# Unit: processor logic
pnpm test:unit -- src/__tests__/unit/bullmq/

# Integration: queue operations + scheduling
pnpm test:integration -- src/__tests__/integration/bullmq/queue.ops.test.ts
pnpm test:integration -- src/__tests__/integration/bullmq/scheduler.test.ts

# Retry, DLQ, recovery
pnpm test:integration -- src/__tests__/integration/bullmq/retry.test.ts
pnpm test:integration -- src/__tests__/integration/bullmq/dlq.test.ts
pnpm test:integration -- src/__tests__/integration/bullmq/recovery.test.ts

# Concurrency & rate limits
pnpm test:integration -- src/__tests__/integration/bullmq/concurrency.test.ts
pnpm test:integration -- src/__tests__/integration/bullmq/limiter.test.ts

# Graceful shutdown
pnpm test:integration -- src/__tests__/integration/bullmq/shutdown.test.ts

# Full BullMQ suite
pnpm test:integration -- src/__tests__/integration/bullmq/

---

3. Retry Policies & Dead Letter Queue Tests#

3.1 Unit Tests — Retry Calculator#

3.2 Integration Tests — Retry → DLQ → Admin Retry Flow#

3.3 Success Criteria (Binary) — Retry & DLQ#

• Exponential backoff delay is calculated correctly: delay * 2^(attempt-1)
• Retry delay is capped at a maximum value (e.g., 24 hours)
• Job-type-specific retry configs are looked up correctly; unknown types fall back to default
• shouldRetry returns true when attemptsMade < config.attempts, false otherwise
• After exhausting retries, job is automatically moved to Dead Letter Queue with all metadata preserved
• DLQ notification email is sent to admin@rankflow.in with job name, ID, attempts, and error message
• Admin retry function re-queues job to original queue with same originalJobId and removes from DLQ
• Admin retry on missing job throws clear Job not found in dead letter queue error
• Admin retry on unknown queue name fails safely without dropping the DLQ entry
• Batch retry can re-queue multiple DLQ jobs atomically
• Auto-retry (if configured) re-queues eligible DLQ jobs after a cooldown period
• No job is retried more than its configured maximum attempts under any circumstance

3.4 Agent Context (Pre-conditions) — Retry & DLQ#

• Required DB state: No specific DB state required for retry logic; DLQ state is Redis-only
• Required env vars: ADMIN_EMAIL=admin@rankflow.in (or mocked in test); REDIS_URL for ioredis-mock
• Required external mocks: email.send (mocked) to capture admin notification; logger (mocked) to verify error logs
• Mock Redis state: dead-letter queue empty; all original queues empty; flushall before suite
• DLQ state: deadLetterQueue instance initialized with connection: redisMock

3.5 Verification Commands — Retry & DLQ#

# Unit: retry calculator
pnpm test:unit -- src/__tests__/unit/retry/calculator.test.ts

# Integration: full retry → DLQ → admin retry flow
pnpm test:integration -- src/__tests__/integration/dlq/full-flow.test.ts
pnpm test:integration -- src/__tests__/integration/dlq/admin-retry.test.ts
pnpm test:integration -- src/__tests__/integration/dlq/notification.test.ts

# Full DLQ suite
pnpm test:integration -- src/__tests__/integration/dlq/

---

4. Observability & Monitoring Tests#

4.1 Unit Tests — Structured Logging#

4.2 Integration Tests — Health Check & Metrics#

4.3 Success Criteria (Binary) — Observability#

• Every job start emits a job_started log with jobId, jobName, practiceId, attempt, and queue
• Every job completion emits a job_completed log with durationMs > 0
• Every job failure emits a job_failed log with error message and attempt count
• Every DLQ move emits a job_dlq_moved log with error and attempts
• Health endpoint /api/health/jobs returns 200 with accurate queue counts for all 8 queues + DLQ
• Health endpoint returns correct totalWaiting, totalActive, totalFailed aggregations
• Queue depth alert triggers when any single queue has > 100 waiting jobs
• Failure rate alert triggers when > 5% of jobs fail within any 1-hour window
• Duration alert triggers when average job duration exceeds 2x the baseline
• DLQ count alert triggers when > 10 jobs enter DLQ within 24 hours
• Inngest latency alert triggers when any step takes > 5 minutes
• Worker CPU alert triggers when CPU > 80% for > 5 minutes
• Admin dashboard renders all queue states accurately with color-coded health status
• Admin dashboard job detail view shows full job history, payload, error, and action buttons

4.4 Agent Context (Pre-conditions) — Observability#

• Required DB state: No specific DB state; metrics are queue-state derived
• Required env vars: REDIS_URL for ioredis-mock; ADMIN_API_KEY (if health endpoint is protected)
• Required external mocks: logger (mocked Pino or equivalent) to capture and assert log objects; email.send for alert notifications
• Mock Redis state: Queues populated with known job counts (waiting/active/completed/failed) for predictable health responses
• Queue state: At least 3 queues have non-zero counts for dashboard rendering tests
• Metrics baseline: Baseline durations stored in mock config for alert threshold calculations

4.5 Verification Commands — Observability#

# Unit: logger assertions
pnpm test:unit -- src/__tests__/unit/observability/logger.test.ts

# Integration: health endpoint + metrics
pnpm test:integration -- src/__tests__/integration/observability/health.test.ts
pnpm test:integration -- src/__tests__/integration/observability/metrics.test.ts

# Dashboard UI
pnpm test:integration -- src/__tests__/integration/observability/dashboard.test.ts

# Full observability suite
pnpm test:integration -- src/__tests__/integration/observability/

# Health endpoint direct curl (smoke test after deployment)
curl -s http://localhost:3000/api/health/jobs | jq '.queues | length'

---

5. Idempotency & Safety Tests#

5.1 Unit Tests — Idempotency Checks#

5.2 Integration Tests — Safety Checks#

5.3 Success Criteria (Binary) — Idempotency & Safety#

• generateJobId produces deterministic, unique keys in format practiceId:jobType:date
• Duplicate job with same jobId is rejected or de-duplicated; queue depth does not increase
• Re-entrant job processor detects duplicate execution and returns cached result without side effects
• GBP suspension guard prevents > 3 posts per week per location
• Social rate limiter prevents exceeding platform API limits per time window
• Token refresh only runs when expiry is within 24 hours (not prematurely, not too late)
• Cost cap prevents AI jobs when monthly budget is exceeded; returns clear COST_CAP reason
• All multi-table DB writes in job processors use transactions; partial writes are impossible
• Graceful shutdown completes without losing active jobs; Redis state remains consistent
• Duplicate Inngest events with same payload do not trigger duplicate workflow side effects
• Idempotency holds across Redis restarts (job IDs are deterministic, not session-based)

5.4 Agent Context (Pre-conditions) — Idempotency & Safety#

• Required DB state: practice with monthlyAiSpend and monthlyAiBudget fields; location with weeklyPostCount field; oauthToken with expiresAt field; citation and directorySubmission tables
• Required env vars: MONTHLY_AI_BUDGET_DEFAULT=50; GBP_MAX_POSTS_PER_WEEK=3; TOKEN_REFRESH_WINDOW_HOURS=24
• Required external mocks: GBP API (msw), social APIs (msw), AI provider (msw), email provider (msw)
• Mock Redis state: Fresh ioredis-mock; queue state must be inspectable for duplicate detection
• Queue state: All queues empty before idempotency tests; flushall between test suites

5.5 Verification Commands — Idempotency & Safety#

# Unit: idempotency key generation + duplicate detection
pnpm test:unit -- src/__tests__/unit/idempotency/

# Integration: safety guards
pnpm test:integration -- src/__tests__/integration/safety/gbp-guard.test.ts
pnpm test:integration -- src/__tests__/integration/safety/social-rate.test.ts
pnpm test:integration -- src/__tests__/integration/safety/token-refresh.test.ts
pnpm test:integration -- src/__tests__/integration/safety/cost-cap.test.ts
pnpm test:integration -- src/__tests__/integration/safety/transaction.test.ts
pnpm test:integration -- src/__tests__/integration/safety/shutdown.test.ts
pnpm test:integration -- src/__tests__/integration/safety/workflow-idempotency.test.ts

# Full safety suite
pnpm test:integration -- src/__tests__/integration/safety/

---

6. Cross-Cutting Integration Tests#

6.1 End-to-End Flow Tests#

6.2 Mock Redis & Queue State#

6.3 Success Criteria (Binary) — Cross-Cutting#

• Full onboarding E2E flow completes all 8 steps and produces expected artifacts (site, post, email, schedule)
• Citation submit-to-verify flow spans 7 days with durable sleep and correct event chaining
• GBP cron flow triggers on correct days (Mon/Wed/Fri) and respects idempotency per day
• Monthly report cron triggers on 1st of month and generates report + email
• Mock Redis accurately simulates BullMQ key structures and queue operations
• flushall reliably resets all queue state between test suites
• Queue snapshots capture and restore state accurately for reproducible test scenarios
• No test suite leaks queue state into subsequent suites (complete isolation)

6.4 Agent Context (Pre-conditions) — Cross-Cutting#

• Required DB state: Full mock practice with location, owner email, and all related tables populated via factories
• Required env vars: All production env vars set to mock values; NODE_ENV=test; INNGEST_TEST_MODE=true
• Required external mocks: All external APIs mocked (msw handlers for Google, social platforms, email, AI, Firecrawl)
• Mock Redis state: ioredis-mock with keepData: false; flushall in beforeEach and afterAll
• Queue state: All 8 queues + DLQ initialized empty; workers created but not started until test action
• Time control: vi.useFakeTimers() with Date.now() mockable for cron and sleep tests
• Test factories: createMockPractice(), createMockLocation(), createMockDirectory() available from src/__tests__/factories/

6.5 Verification Commands — Cross-Cutting#

# E2E flows
pnpm test:e2e -- e2e/onboarding.spec.ts
pnpm test:e2e -- e2e/citation.spec.ts
pnpm test:e2e -- e2e/gbp-post.spec.ts
pnpm test:e2e -- e2e/monthly-report.spec.ts

# Mock Redis + queue state
pnpm test:integration -- src/__tests__/integration/mock/

# Full cross-cutting suite
pnpm test:e2e
pnpm test:integration -- src/__tests__/integration/mock/

---

7. Execution Plan & Agent Assignments#

Phase 1: Mock Infrastructure (Day 1)#

Phase 2: Unit Tests (Days 2-3)#

Phase 3: Integration Tests (Days 4-5)#

Phase 4: E2E & Cross-Cutting (Day 6)#

Phase 5: Verification & Compliance (Day 7)#

---

Appendix A: Mock Redis Configuration#

// src/__tests__/mocks/redis.ts
import Redis from "ioredis-mock";

export const redisMock = new Redis({
  data: {}, // Empty initial state
  keyPrefix: "bull:", // Match BullMQ default
});

// BullMQ-compatible key helpers for assertions
export async function getQueueKeys(queueName: string): Promise<string[]> {
  const keys = await redisMock.keys(`bull:${queueName}:*`);
  return keys;
}

export async function getJobCount(queueName: string, state: string): Promise<number> {
  return redisMock.zcard(`bull:${queueName}:${state}`);
}

export async function flushAll(): Promise<void> {
  await redisMock.flushall();
}

Appendix B: Mock Factory Examples#

// src/__tests__/factories/job.ts
import { faker } from "@faker-js/faker";

export const createMockJobPayload = (overrides?: Record<string, unknown>) => ({
  practiceId: `prac_${faker.string.nanoid(8)}`,
  locationId: `loc_${faker.string.nanoid(8)}`,
  content: faker.lorem.paragraph(),
  ...overrides,
});

export const createMockRetryConfig = (overrides?: Partial<{ attempts: number; delay: number }>) => ({
  attempts: 3,
  delay: 5000,
  ...overrides,
});

Appendix C: Test Data Constants#

---

End of Job Queue & Workflow Engine Test Specification — RankFlow AI v1.0.0