Research
RankFlow AI — Refined Technical Specification
Every capability of RankFlow is defined by a skill file. Skills are markdown documents with structured frontmatter that serve as:
docs/research_technical_refined.mdOn this page
- Agent-Centric, Skill-Based Local SEO Automation Platform
- 1. Core Philosophy
- 1.1 Skills Are the Source of Truth
- 1.2 Agent-Centric Design
- 1.3 Deterministic Workflows
- 1.4 API-First Everything
- 1.5 Lean and Observable
- 2. Infrastructure (AWS + Dokploy)
- 2.1 Why Dokploy?
- 2.2 AWS Architecture
- 2.3 Dokploy Services
- 2.4 Environment Variables (Dokploy)
- 3. System Architecture
- 3.1 High-Level Diagram
- 3.2 Request Flow
- 4. The Skill System
- 4.1 Skill File Format
- API Endpoint
- Restate Workflow
- Acceptance Criteria
- 4.4 Skill Execution
- 5. Agent Harness
- 5.1 What Is the Harness?
- 5.2 Harness Architecture
- 5.3 Observability in the Harness
- 5.4 Self-Healing
- 6. Database Schema (Lean)
- 6.1 Core Models (~15 tables)
- 6.2 Schema Principles
- 7. FastAPI Backend
- 7.1 Project Structure
- 7.2 FastAPI App Setup
- 7.3 Skill Execution Endpoint
- 8. Next.js Frontend
- 8.1 Project Structure
- 8.2 API Client
- 8.3 Site Renderer (Dynamic Landing Pages)
- 9. Workflow Engine (Restate)
- 9.1 Why Restate?
- 9.2 Restate Setup
- 9.3 Example Workflows
- 9.4 Restate Deployment
- 10. Landing Page Scale Plan
- 10.1 Phase 1: Path-Based Routing (MVP)
- 10.2 Phase 2: Subdomain Routing
- 10.3 Phase 3: Custom Domains
- 10.4 The "Living Site" Architecture
- 10.5 Site Templates
- 10.6 Asset Management
- 11. SEO Data Integration (DataForSEO)
- 11.1 DataForSEO API
- 11.2 DataForSEO Client
- 11.3 Rank Tracking Workflow
- 12. Browser Automation (API-First)
- 12.1 Hyperbrowser
- 12.2 Firecrawl
- 12.3 Citation Submission Workflow
- 13. Social Media Integration
- 13.1 Composio
- 13.2 Zernio Scheduling
- 13.3 Social Post Workflow
- 14. Observability & Auditability
- 14.1 Structured Logging
- 14.2 Health Checks
- 14.3 Audit Logging
- 14.4 Admin Dashboard Observability
- 15. Security & Compliance
- 15.1 Token Encryption
- 15.2 Rate Limiting
- 15.3 DPDPA Compliance
- 16. Project Structure
- 16.1 Monorepo Layout
- 17. Deployment Guide
- 17.1 Local Development
- 17.2 Dokploy Deployment
- 17.3 CI/CD (GitHub Actions)
- Appendix: Skill Registry
- Core Skills (V1)
- Skill Dependency Graph
Agent-Centric, Skill-Based Local SEO Automation Platform#
Version: 2.0.0
Date: 2025
Status: Architecture Blueprint — Agent-Buildable
Infrastructure: AWS + Dokploy
Philosophy: Skills are the single source of truth. Code is generated from skills. Runtime behavior is driven by skills.
1. Core Philosophy#
1.1 Skills Are the Source of Truth#
Every capability of RankFlow is defined by a skill file. Skills are markdown documents with structured frontmatter that serve as:
- Declarative specifications — what the feature does, inputs, outputs, errors
- Implementation guides — how coding agents should build it
- Runtime contracts — how the harness executes and validates the feature
- Observability schemas — what to log, measure, and alert on
Principle: If you want to change how something works, you modify the skill. A coding agent reads the skill and updates the code. A runtime agent reads the skill and executes the workflow.
1.2 Agent-Centric Design#
The platform is built by agents, for agents:
- Coding agents read skills → generate/modify code → test against skill acceptance criteria
- Runtime agents (the harness) read skills → plan execution → dispatch workflows → observe results
- Self-healing: When a skill fails, the harness logs the failure, attempts fallback skills, and escalates to human if needed
1.3 Deterministic Workflows#
Every business process is a durable workflow managed by Restate:
- Workflows survive crashes, restarts, and deployments
- Steps are retried with exponential backoff
- External events (OAuth callbacks, webhooks) are awaited natively
- State is transparent — you can inspect exactly where any workflow is
1.4 API-First Everything#
No local binaries, no Puppeteer on the server, no Chrome in Docker:
- Browser automation → Hyperbrowser API, Firecrawl API, ScrapeGraph API
- Social media → Composio API, Zernio API, direct platform APIs
- SEO data → DataForSEO API
- AI → Claude/GPT-4o APIs
1.5 Lean and Observable#
- One deploy target: AWS + Dokploy (not Vercel + VPS split)
- One queue system: Redis Streams (not BullMQ)
- One workflow engine: Restate (not custom job tables)
- One logger: Structured JSON logs (Pino + Python structlog)
- One schema: ~15 core models (not 30+)
2. Infrastructure (AWS + Dokploy)#
2.1 Why Dokploy?#
Dokploy is a self-hosted PaaS that runs on your AWS infrastructure. It gives you:
- Heroku-like experience on your own servers
- Git-based deployments with auto-build
- Database provisioning (Postgres, Redis, MySQL)
- SSL certificates (Let's Encrypt)
- Environment variables management
- Monitoring (basic logs, resource usage)
- Multi-server support (scale later)
You own the infrastructure. No Vercel lock-in. No surprise bills.
2.2 AWS Architecture#
AWS Account (ap-south-1 / Mumbai)
│
├── VPC
│ ├── Public Subnet
│ │ └── EC2 (t3.large / t3.xlarge) — Dokploy + Apps
│ ├── Private Subnet (future)
│ │ └── RDS PostgreSQL (future — start with Dokploy Postgres)
│ └── Security Groups
│
├── S3 Bucket
│ ├── assets.rankflow.ai/ # Doctor photos, logos
│ ├── reports.rankflow.ai/ # Generated PDFs (if needed later)
│ └── backups.rankflow.ai/ # DB backups
│
├── Route 53
│ ├── rankflow.ai # Main domain
│ ├── *.rankflow.ai # Wildcard for subdomains
│ └── api.rankflow.ai # API subdomain
│
├── CloudFront (optional, future)
│ └── CDN for static assets
│
└── IAM Roles
├── DokployEC2Role # S3 access, CloudWatch logs
└── RestateRole # If using Restate Cloud
2.3 Dokploy Services#
| Service | Type | Resource | Purpose |
|---|---|---|---|
rankflow-web |
App (Docker) | 1 vCPU, 2GB RAM | Next.js frontend |
rankflow-api |
App (Docker) | 1 vCPU, 2GB RAM | FastAPI backend |
rankflow-worker |
App (Docker) | 1 vCPU, 2GB RAM | Restate workers |
rankflow-db |
Database | 1 vCPU, 2GB RAM | PostgreSQL (Dokploy managed) |
rankflow-cache |
Database | 0.5 vCPU, 1GB RAM | Redis (Dokploy managed) |
Total estimated cost: ~$60-80/month for the EC2 instance + Dokploy services.
2.4 Environment Variables (Dokploy)#
# App
APP_ENV=production
APP_URL=https://rankflow.ai
API_URL=https://api.rankflow.ai
# Database
DATABASE_URL=postgresql://rankflow:xxx@rankflow-db:5432/rankflow
# Redis
REDIS_URL=redis://rankflow-cache:6379
# Auth (Better Auth)
BETTER_AUTH_SECRET=xxx
BETTER_AUTH_URL=https://rankflow.ai
# Google OAuth + GBP
GOOGLE_CLIENT_ID=xxx
GOOGLE_CLIENT_SECRET=xxx
# AI
ANTHROPIC_API_KEY=xxx
OPENAI_API_KEY=xxx
# SEO Data
DATAFORSEO_LOGIN=xxx
DATAFORSEO_PASSWORD=xxx
# Browser Automation
HYPERBROWSER_API_KEY=xxx
FIRECRAWL_API_KEY=xxx
SCRAPEGRAPH_API_KEY=xxx
# Social
COMPOSIO_API_KEY=xxx
ZERNIO_API_KEY=xxx
# Storage (S3)
AWS_ACCESS_KEY_ID=xxx
AWS_SECRET_ACCESS_KEY=xxx
AWS_S3_BUCKET=rankflow-assets
AWS_REGION=ap-south-1
# Restate
RESTATE_ENDPOINT=http://rankflow-worker:8080
# Email
RESEND_API_KEY=xxx
# Monitoring
SENTRY_DSN=xxx
3. System Architecture#
3.1 High-Level Diagram#
┌─────────────────────────────────────────────────────────────────────┐
│ CLIENT LAYER │
│ ┌──────────────┐ ┌──────────────┐ ┌──────────────┐ ┌──────────┐ │
│ │ Doctor │ │ Doctor │ │ Public │ │ Admin │ │
│ │ Dashboard │ │ Site │ │ Site │ │ Panel │ │
│ │ (Next.js) │ │ (Next.js) │ │ (Next.js) │ │(Next.js) │ │
│ └──────────────┘ └──────────────┘ └──────────────┘ └──────────┘ │
└─────────────────────────────────────────────────────────────────────┘
│
▼ HTTP / WebSocket
┌─────────────────────────────────────────────────────────────────────┐
│ API GATEWAY (FastAPI) │
│ ┌──────────────┐ ┌──────────────┐ ┌──────────────┐ │
│ │ REST API │ │ Webhooks │ │ Health │ │
│ │ (Routers) │ │ (Stripe, │ │ /metrics │ │
│ │ │ │ Composio, │ │ │ │
│ │ │ │ Google) │ │ │ │
│ └──────────────┘ └──────────────┘ └──────────────┘ │
│ │
│ Middleware: Auth (Better Auth), Rate Limit, Request ID, Logging │
└─────────────────────────────────────────────────────────────────────┘
│
▼ Internal API
┌─────────────────────────────────────────────────────────────────────┐
│ SKILL HARNESS (Python) │
│ ┌──────────────┐ ┌──────────────┐ ┌──────────────┐ │
│ │ Skill │ │ Workflow │ │ State │ │
│ │ Registry │ │ Planner │ │ Manager │ │
│ │ (loads .md) │ │ (plans exec)│ │ (Restate) │ │
│ └──────────────┘ └──────────────┘ └──────────────┘ │
│ │
│ ┌──────────────┐ ┌──────────────┐ ┌──────────────┐ │
│ │ Execution │ │ Fallback │ │ Audit │ │
│ │ Engine │ │ Handler │ │ Logger │ │
│ │ (dispatches)│ │ (retries) │ │ (structured)│ │
│ └──────────────┘ └──────────────┘ └──────────────┘ │
└─────────────────────────────────────────────────────────────────────┘
│
▼ API Calls
┌─────────────────────────────────────────────────────────────────────┐
│ EXTERNAL SERVICES │
│ ┌──────────┐ ┌──────────┐ ┌──────────┐ ┌──────────┐ ┌──────────┐ │
│ │ Claude │ │Hyper- │ │DataForSEO│ │ Composio │ │ Resend │ │
│ │ Sonnet │ │browser │ │ │ │ │ │ │ │
│ │ GPT-4o │ │Firecrawl │ │ │ │ Zernio │ │ S3 │ │
│ │ (AI) │ │(Browser) │ │ (SEO) │ │ (Social) │ │(Storage) │ │
│ └──────────┘ └──────────┘ └──────────┘ └──────────┘ └──────────┘ │
└─────────────────────────────────────────────────────────────────────┘
│
▼
┌─────────────────────────────────────────────────────────────────────┐
│ DATA LAYER │
│ ┌──────────┐ ┌──────────┐ ┌──────────┐ ┌──────────┐ │
│ │PostgreSQL│ │ Redis │ │ S3 │ │ Restate │ │
│ │ (Prisma) │ │(Streams,│ │ (Assets) │ │ State │ │
│ │ │ │ Cache) │ │ │ │ Store │ │
│ └──────────┘ └──────────┘ └──────────┘ └──────────┘ │
└─────────────────────────────────────────────────────────────────────┘
3.2 Request Flow#
- Doctor clicks "Generate GBP Post" in Next.js dashboard
- Next.js calls FastAPI
/api/v1/skills/gbp-post-createwith practice_id, content - FastAPI validates auth via Better Auth session
- FastAPI calls Harness
execute_skill("gbp-post-create", payload) - Harness loads skill
skills/04-gbp-post-create.md - Harness plans execution: validate → generate → publish → log
- Harness dispatches to Restate workflow
GbpPostCreateWorkflow - Restate executes steps with automatic retries and state persistence
- Restate calls external APIs (Claude for content, GBP API for publish)
- Restate stores result in Postgres, updates job status
- Harness logs structured audit entry
- FastAPI returns result to Next.js
- Next.js shows success + audit trail
4. The Skill System#
4.1 Skill File Format#
Every skill is a markdown file with YAML frontmatter:
---
id: 04-gbp-post-create
name: Create GBP Post
category: gbp
version: 1.0.0
autonomous: true # Can run without human approval
approval_required: false # If false, runs immediately
triggers:
- manual
- scheduled
- ai-generated
inputs:
practice_id:
type: string
required: true
description: "The practice ID"
location_id:
type: string
required: true
description: "The GBP location ID"
content:
type: string
required: false
description: "Post content (if not provided, AI generates)"
max_length: 1500
media_urls:
type: array[string]
required: false
description: "URLs of images/videos to attach"
max_items: 10
cta_type:
type: enum[BOOK, CALL, LEARN_MORE, SIGN_UP, ORDER]
required: false
default: BOOK
outputs:
post_id:
type: string
description: "The created post ID"
status:
type: enum[SCHEDULED, PUBLISHED, FAILED]
published_url:
type: string
required: false
dependencies:
- 03-gbp-connect
- 21-content-generate
external_apis:
- google-business-profile
- claude-sonnet
cost_estimate_usd: 0.05
time_estimate_seconds: 10
retry_policy:
max_attempts: 3
backoff: exponential
initial_delay_seconds: 5
observability:
metrics:
- gbp_posts_created_total
- gbp_post_create_duration_seconds
- gbp_post_create_cost_usd
alerts:
- condition: "status == FAILED"
severity: warning
message: "GBP post creation failed"
- condition: "duration_seconds > 30"
severity: warning
message: "GBP post creation slow"
---
## Purpose
Create and publish a Google Business Profile post for a medical practice.
## Business Context
GBP posts appear in Google Search and Maps results. They drive patient engagement
and improve local SEO. Medical practices should post 1-2x per week.
## Procedure
### Step 1: Validate Prerequisites
- Check practice has active GBP connection (skill 03)
- Check GBP API quota (max 500 posts/day per location)
- Verify location_id belongs to practice
### Step 2: Generate or Validate Content
- If content not provided, call skill 21-content-generate
- Validate content length ≤ 1500 characters
- Ensure content complies with medical advertising guidelines
- Add CTA if not present
### Step 3: Prepare Media
- If media_urls provided, validate each URL is accessible
- Check file size ≤ 10MB per image, ≤ 100MB per video
- Verify media format (JPEG, PNG, MP4, MOV)
### Step 4: Call GBP API
- Format request per Google Business Profile API spec
- Set languageCode: "en-IN"
- Include searchTerms for SEO
- Handle API errors:
- 429 (quota exceeded) → queue for next day
- 401 (auth expired) → trigger skill 03 re-auth
- 400 (bad request) → flag for human review
### Step 5: Store Result
- Save post to database with status
- Link to content_piece if AI-generated
- Update practice metrics
### Step 6: Audit Log
- Log: practice_id, location_id, content_length, status, cost, duration
- If failed, log error message and stack trace
## Error Handling
| Error | Cause | Action |
|-------|-------|--------|
| AUTH_EXPIRED | Refresh token invalid | Trigger skill 03, notify user |
| QUOTA_EXCEEDED | Daily limit reached | Queue for next day, notify user |
| CONTENT_REJECTED | Violates GBP policy | Flag for human review |
| MEDIA_TOO_LARGE | File exceeds limit | Compress or reject |
| NETWORK_ERROR | Temporary failure | Retry with backoff |
## Implementation Notes for Coding Agents
### Database Model
```prisma
model GbpPost {
id String @id @default(cuid())
practiceId String
locationId String
gbpLocationId String
content String @db.Text
mediaUrls String[]
ctaType String?
status String // SCHEDULED, PUBLISHED, FAILED
gbpPostId String? // Google's ID
publishedAt DateTime?
failedReason String?
costUsd Decimal? @db.Decimal(10, 6)
createdAt DateTime @default(now())
updatedAt DateTime @updatedAt
}
API Endpoint#
@router.post("/skills/gbp-post-create")
async def create_gbp_post(
payload: GbpPostCreateInput,
user: User = Depends(get_current_user)
):
# Validate user owns practice
# Call harness.execute_skill("04-gbp-post-create", payload)
# Return result
Restate Workflow#
@restate.workflow
async def gbp_post_create_workflow(ctx: restate.Context, payload: dict):
# Step 1: Validate
practice = await ctx.run("validate", lambda: validate_practice(payload["practice_id"]))
# Step 2: Generate content if needed
if not payload.get("content"):
content = await ctx.run("generate", lambda: ai.generate_gbp_post(practice))
else:
content = payload["content"]
# Step 3: Publish
result = await ctx.run("publish", lambda: gbp_api.create_post(practice, content))
# Step 4: Store
await ctx.run("store", lambda: db.gbp_post.create({...}))
return result
Acceptance Criteria#
- Can create a GBP post with provided content
- Can generate AI content if none provided
- Respects 1500 character limit
- Handles auth errors gracefully
- Handles quota exceeded gracefully
- Stores audit log entry
- Updates practice metrics
- Returns within 30 seconds
### 4.2 Skill Categories
| Category | Description | Example Skills |
|----------|-------------|----------------|
| `onboarding` | Getting doctors set up | practice-setup, gbp-connect, asset-upload |
| `gbp` | Google Business Profile | post-create, post-schedule, review-reply, insights-fetch |
| `social` | Social media | connect, post-create, post-schedule |
| `citation` | Directory listings | submit, verify-nap, update |
| `site` | Landing pages | generate, update, deploy, evolve |
| `seo` | SEO operations | audit, keyword-track, competitor-analyze |
| `content` | AI content | generate, approve, publish |
| `report` | Analytics & reports | generate, email, dashboard |
| `lead` | Lead management | capture, nurture, notify |
| `billing` | Payments | subscribe, invoice, upgrade |
| `admin` | Platform operations | moderate, support, health-check |
| `platform` | System capabilities | auth, notification, dns-configure |
### 4.3 Skill Registry
The harness maintains a skill registry loaded at startup:
```python
class SkillRegistry:
def __init__(self, skills_dir: str):
self.skills: dict[str, Skill] = {}
self._load_all(skills_dir)
def _load_all(self, directory: str):
for path in Path(directory).glob("**/*.md"):
skill = self._parse_skill(path)
self.skills[skill.id] = skill
def _parse_skill(self, path: Path) -> Skill:
content = path.read_text()
# Parse YAML frontmatter
# Parse markdown body
# Extract code blocks for implementation hints
return Skill(...)
def get(self, skill_id: str) -> Skill | None:
return self.skills.get(skill_id)
def list_by_category(self, category: str) -> list[Skill]:
return [s for s in self.skills.values() if s.category == category]
def get_dependencies(self, skill_id: str) -> list[Skill]:
skill = self.skills.get(skill_id)
if not skill:
return []
return [self.skills[d] for d in skill.dependencies if d in self.skills]
4.4 Skill Execution#
class SkillHarness:
def __init__(self, registry: SkillRegistry, restate: RestateClient):
self.registry = registry
self.restate = restate
self.logger = structlog.get_logger()
async def execute(self, skill_id: str, payload: dict, context: ExecutionContext) -> ExecutionResult:
skill = self.registry.get(skill_id)
if not skill:
raise SkillNotFoundError(skill_id)
# Check prerequisites
for dep_id in skill.dependencies:
dep = self.registry.get(dep_id)
if dep and not await self._check_prerequisite(dep, payload):
return ExecutionResult(
status="FAILED",
error=f"Prerequisite not met: {dep_id}",
skill_id=skill_id
)
# Log start
self.logger.info(
"skill_execution_started",
skill_id=skill_id,
practice_id=payload.get("practice_id"),
user_id=context.user_id,
trigger=context.trigger
)
start_time = time.time()
try:
# Dispatch to Restate workflow
if skill.workflow_name:
result = await self.restate.workflow(
skill.workflow_name,
payload,
retry_policy=skill.retry_policy
)
else:
# Direct execution for simple skills
result = await self._execute_direct(skill, payload)
duration = time.time() - start_time
# Log success
self.logger.info(
"skill_execution_completed",
skill_id=skill_id,
status="SUCCESS",
duration_seconds=duration,
cost_usd=result.get("cost_usd", 0)
)
return ExecutionResult(
status="SUCCESS",
data=result,
skill_id=skill_id,
duration_seconds=duration
)
except Exception as e:
duration = time.time() - start_time
# Log failure
self.logger.error(
"skill_execution_failed",
skill_id=skill_id,
error=str(e),
error_type=type(e).__name__,
duration_seconds=duration
)
# Attempt fallback if defined
if skill.fallback_skill:
return await self.execute(skill.fallback_skill, payload, context)
return ExecutionResult(
status="FAILED",
error=str(e),
skill_id=skill_id,
duration_seconds=duration
)
5. Agent Harness#
5.1 What Is the Harness?#
The harness is the runtime brain of RankFlow. It is not just a workflow engine — it is an agentic orchestrator that:
- Understands skills — Reads skill files, knows what the platform can do
- Plans execution — Given a goal, finds the right skills, orders them, handles dependencies
- Dispatches work — Calls Restate workflows, external APIs, or direct functions
- Observes everything — Logs every action, measures every metric, traces every request
- Self-corrects — Retries, falls back, escalates, learns from failures
- Reports back — Tells the user (and the admin) what happened and why
5.2 Harness Architecture#
# src/harness/main.py
class RankFlowHarness:
"""
The central agent harness for RankFlow AI.
Responsibilities:
- Skill registry management
- Execution planning and dispatch
- Observability and audit logging
- Self-healing and escalation
"""
def __init__(self):
self.skills = SkillRegistry("/app/skills")
self.restate = RestateClient()
self.audit = AuditLogger()
self.metrics = MetricsCollector()
self.escalation = EscalationManager()
self.llm = LLMRouter() # Claude → GPT-4o fallback
async def handle_goal(self, goal: str, context: dict) -> HarnessResult:
"""
High-level goal handler.
Example goals:
- "Onboard Dr. Sharma's cardiology practice"
- "Create this week's GBP posts for all active practices"
- "Update landing pages based on new keyword rankings"
"""
# Step 1: Plan — Use LLM to break goal into skill executions
plan = await self._plan_goal(goal, context)
# Step 2: Execute — Run each step via Restate
results = []
for step in plan.steps:
result = await self.execute_skill(step.skill_id, step.payload, context)
results.append(result)
if result.status == "FAILED" and not step.can_continue_on_failure:
break
# Step 3: Report — Summarize what happened
summary = await self._generate_summary(goal, plan, results)
return HarnessResult(
goal=goal,
plan=plan,
results=results,
summary=summary,
status="SUCCESS" if all(r.status == "SUCCESS" for r in results) else "PARTIAL"
)
async def _plan_goal(self, goal: str, context: dict) -> ExecutionPlan:
"""Use LLM to plan which skills to execute."""
# Load relevant skills as context
all_skills = self.skills.list_all()
skill_descriptions = "\n".join([
f"- {s.id}: {s.name} (category: {s.category}, autonomous: {s.autonomous})"
for s in all_skills
])
prompt = f"""
You are the RankFlow AI planning engine. Given a user goal and available skills,
break the goal into a sequence of skill executions.
Available Skills:
{skill_descriptions}
User Goal: {goal}
Context: {json.dumps(context)}
Return a JSON execution plan:
{{
"steps": [
{{
"skill_id": "skill-id",
"payload": {{...}},
"can_continue_on_failure": false,
"reason": "why this step"
}}
]
}}
"""
response = await self.llm.generate(prompt, json_mode=True)
return ExecutionPlan.parse_raw(response)
5.3 Observability in the Harness#
Every execution produces a trace:
{
"trace_id": "abc-123-def",
"goal": "Onboard Dr. Sharma",
"practice_id": "practice_123",
"user_id": "user_456",
"started_at": "2025-01-15T09:00:00Z",
"completed_at": "2025-01-15T09:02:30Z",
"status": "SUCCESS",
"steps": [
{
"step_id": 1,
"skill_id": "01-practice-setup",
"status": "SUCCESS",
"started_at": "2025-01-15T09:00:00Z",
"completed_at": "2025-01-15T09:00:05Z",
"duration_ms": 5000,
"cost_usd": 0.0,
"inputs": {"name": "Dr. Sharma Cardiology", "city": "Mumbai"},
"outputs": {"practice_id": "practice_123"},
"logs": [...]
},
{
"step_id": 2,
"skill_id": "03-gbp-connect",
"status": "WAITING",
"started_at": "2025-01-15T09:00:05Z",
"waiting_for": "oauth_callback",
"auth_url": "https://accounts.google.com/..."
}
],
"total_cost_usd": 0.05,
"total_duration_ms": 150000
}
This trace is:
- Stored in PostgreSQL (for long-term audit)
- Sent to structured logs (for real-time debugging)
- Displayed in the admin dashboard (for human oversight)
- Used by the harness for self-correction (learn from patterns)
5.4 Self-Healing#
class SelfHealingManager:
async def handle_failure(self, trace: ExecutionTrace, step: ExecutionStep):
"""Decide what to do when a skill fails."""
skill = self.skills.get(step.skill_id)
# Strategy 1: Retry with modified parameters
if step.attempt < skill.retry_policy.max_attempts:
return HealingAction.RETRY_WITH_BACKOFF
# Strategy 2: Use fallback skill
if skill.fallback_skill:
return HealingAction.FALLBACK(skill.fallback_skill)
# Strategy 3: Use LLM to generate workaround
workaround = await self._generate_workaround(trace, step)
if workaround.confidence > 0.8:
return HealingAction.WORKAROUND(workaround)
# Strategy 4: Escalate to human
return HealingAction.ESCALATE(
reason="All automated recovery failed",
context=trace
)
6. Database Schema (Lean)#
6.1 Core Models (~15 tables)#
// prisma/schema.prisma
// ─── Auth & Users ─────────────────────────────────────────
model User {
id String @id @default(cuid())
email String @unique
name String?
image String?
role String @default("OWNER") // OWNER, ADMIN, EDITOR, VIEWER
createdAt DateTime @default(now())
updatedAt DateTime @updatedAt
practices PracticeMember[]
auditLogs AuditLog[]
@@map("users")
}
// ─── Practice (Tenant) ────────────────────────────────────
model Practice {
id String @id @default(cuid())
name String
slug String @unique
type String @default("CLINIC") // CLINIC, HOSPITAL, DENTAL, etc.
status String @default("active") // active, trial, suspended, cancelled
ownerId String
// Subscription
tier String @default("FREE") // FREE, STARTER, PRO, ENTERPRISE
trialEndsAt DateTime?
subscriptionEndsAt DateTime?
// Branding
logoUrl String?
primaryColor String? @default("#2563eb")
// Site
subdomain String @unique @default(cuid())
customDomain String? @unique
siteTemplate String @default("medical-modern")
sitePublished Boolean @default(false)
// Settings JSON
settings Json @default("{}")
createdAt DateTime @default(now())
updatedAt DateTime @updatedAt
// Relations
members PracticeMember[]
locations Location[]
gbpAccounts GbpAccount[]
socialAccounts SocialAccount[]
contentPieces ContentPiece[]
siteSections SiteSection[]
jobs Job[]
reports Report[]
leads Lead[]
auditLogs AuditLog[]
@@map("practices")
}
model PracticeMember {
id String @id @default(cuid())
practiceId String
userId String
role String @default("EDITOR")
createdAt DateTime @default(now())
practice Practice @relation(fields: [practiceId], references: [id], onDelete: Cascade)
user User @relation(fields: [userId], references: [id], onDelete: Cascade)
@@unique([practiceId, userId])
@@map("practice_members")
}
// ─── Location (GBP Entity) ────────────────────────────────
model Location {
id String @id @default(cuid())
practiceId String
name String
// NAP
businessName String
address String
city String
state String
postalCode String
country String @default("IN")
phone String
email String?
website String?
// Geo
latitude Decimal? @db.Decimal(10, 8)
longitude Decimal? @db.Decimal(11, 8)
// Business
category String @default("Medical Clinic")
services String[] @default([])
businessHours Json @default("{}")
// SEO
targetKeywords String[] @default([])
languages String[] @default(["English", "Hindi"])
isPrimary Boolean @default(false)
createdAt DateTime @default(now())
updatedAt DateTime @updatedAt
practice Practice @relation(fields: [practiceId], references: [id], onDelete: Cascade)
gbpLocation GbpLocation?
@@map("locations")
}
// ─── GBP Integration ──────────────────────────────────────
model GbpAccount {
id String @id @default(cuid())
practiceId String
accountEmail String
// OAuth tokens (encrypted at application layer)
accessToken String @db.Text
refreshToken String @db.Text
tokenExpiresAt DateTime
isActive Boolean @default(true)
lastSyncedAt DateTime?
createdAt DateTime @default(now())
updatedAt DateTime @updatedAt
practice Practice @relation(fields: [practiceId], references: [id], onDelete: Cascade)
locations GbpLocation[]
@@map("gbp_accounts")
}
model GbpLocation {
id String @id @default(cuid())
gbpAccountId String
locationId String // Internal location ID
gbpLocationId String // Google's location ID
name String
status String @default("ACTIVE")
lastApiSyncAt DateTime?
createdAt DateTime @default(now())
updatedAt DateTime @updatedAt
gbpAccount GbpAccount @relation(fields: [gbpAccountId], references: [id], onDelete: Cascade)
location Location @relation(fields: [locationId], references: [id], onDelete: Cascade)
posts GbpPost[]
reviews Review[]
insights GbpInsight[]
@@unique([gbpLocationId])
@@map("gbp_locations")
}
model GbpPost {
id String @id @default(cuid())
gbpLocationId String
contentPieceId String?
content String @db.Text
mediaUrls String[] @default([])
ctaType String?
status String @default("SCHEDULED") // SCHEDULED, PUBLISHED, FAILED
gbpPostId String? // Google's post ID
scheduledFor DateTime?
publishedAt DateTime?
failedReason String?
costUsd Decimal? @db.Decimal(10, 6)
createdAt DateTime @default(now())
updatedAt DateTime @updatedAt
gbpLocation GbpLocation @relation(fields: [gbpLocationId], references: [id], onDelete: Cascade)
@@map("gbp_posts")
}
model GbpInsight {
id String @id @default(cuid())
gbpLocationId String
date DateTime @db.Date
viewsSearch Int @default(0)
viewsMaps Int @default(0)
websiteClicks Int @default(0)
phoneClicks Int @default(0)
drivingDirections Int @default(0)
searchQueries Json? @default("[]")
createdAt DateTime @default(now())
gbpLocation GbpLocation @relation(fields: [gbpLocationId], references: [id], onDelete: Cascade)
@@unique([gbpLocationId, date])
@@map("gbp_insights")
}
// ─── Reviews ──────────────────────────────────────────────
model Review {
id String @id @default(cuid())
locationId String
gbpLocationId String?
gbpReviewId String? @unique
reviewerName String?
rating Int @db.SmallInt
comment String? @db.Text
replyText String? @db.Text
replyBy String?
replyGeneratedByAI Boolean @default(false)
replyPublished Boolean @default(false)
repliedAt DateTime?
reviewDate DateTime
status String @default("NEW") // NEW, REPLIED, FLAGGED
createdAt DateTime @default(now())
updatedAt DateTime @updatedAt
location Location @relation(fields: [locationId], references: [id], onDelete: Cascade)
gbpLocation GbpLocation? @relation(fields: [gbpLocationId], references: [id])
@@map("reviews")
}
// ─── Social Media ─────────────────────────────────────────
model SocialAccount {
id String @id @default(cuid())
practiceId String
platform String // FACEBOOK, INSTAGRAM, LINKEDIN, TWITTER
accountName String
accountId String? // Platform's ID
profileUrl String?
// Tokens (encrypted)
accessToken String @db.Text
refreshToken String? @db.Text
tokenExpiresAt DateTime?
isActive Boolean @default(true)
createdAt DateTime @default(now())
updatedAt DateTime @updatedAt
practice Practice @relation(fields: [practiceId], references: [id], onDelete: Cascade)
posts SocialPost[]
@@unique([practiceId, platform])
@@map("social_accounts")
}
model SocialPost {
id String @id @default(cuid())
socialAccountId String
contentPieceId String?
content String @db.Text
mediaUrls String[] @default([])
scheduledFor DateTime?
publishedAt DateTime?
status String @default("SCHEDULED") // SCHEDULED, PUBLISHED, FAILED
externalPostId String?
failedReason String?
createdAt DateTime @default(now())
updatedAt DateTime @updatedAt
socialAccount SocialAccount @relation(fields: [socialAccountId], references: [id], onDelete: Cascade)
@@map("social_posts")
}
// ─── Content (AI-Generated) ───────────────────────────────
model ContentPiece {
id String @id @default(cuid())
practiceId String
locationId String?
type String // GBP_POST, SOCIAL_POST, SITE_CONTENT, FAQ, SCHEMA
status String @default("DRAFT") // DRAFT, PENDING_REVIEW, APPROVED, PUBLISHED
title String?
content String @db.Text
excerpt String? @db.Text
seoTitle String?
seoDescription String?
focusKeywords String[] @default([])
aiGenerated Boolean @default(false)
aiModel String?
aiTokensUsed Int?
costUsd Decimal? @db.Decimal(10, 6)
humanEdited Boolean @default(false)
editedBy String?
editedAt DateTime?
createdAt DateTime @default(now())
updatedAt DateTime @updatedAt
practice Practice @relation(fields: [practiceId], references: [id], onDelete: Cascade)
location Location? @relation(fields: [locationId], references: [id])
@@map("content_pieces")
}
// ─── Site Sections (Landing Page) ─────────────────────────
model SiteSection {
id String @id @default(cuid())
practiceId String
sectionKey String // hero, about, services, testimonials, contact, faq
sortOrder Int @default(0)
content String @db.Text
mediaUrls String[] @default([])
config Json @default("{}") // Section-specific settings
isVisible Boolean @default(true)
createdAt DateTime @default(now())
updatedAt DateTime @updatedAt
practice Practice @relation(fields: [practiceId], references: [id], onDelete: Cascade)
@@unique([practiceId, sectionKey])
@@map("site_sections")
}
// ─── Jobs / Workflows ─────────────────────────────────────
model Job {
id String @id @default(cuid())
practiceId String
skillId String // Which skill was executed
status String @default("PENDING") // PENDING, RUNNING, COMPLETED, FAILED, WAITING
payload Json // Input data
result Json? // Output data
error String?
restateId String? // Restate workflow ID
startedAt DateTime?
completedAt DateTime?
costUsd Decimal? @db.Decimal(10, 6)
createdAt DateTime @default(now())
updatedAt DateTime @updatedAt
practice Practice @relation(fields: [practiceId], references: [id], onDelete: Cascade)
@@index([practiceId, status])
@@index([skillId, status])
@@map("jobs")
}
// ─── Reports ──────────────────────────────────────────────
model Report {
id String @id @default(cuid())
practiceId String
name String
periodStart DateTime
periodEnd DateTime
status String @default("GENERATING") // GENERATING, READY, FAILED
sections Json // Report data
scoreOverall Int? @default(0)
scoreGbp Int? @default(0)
scoreSeo Int? @default(0)
scoreReviews Int? @default(0)
emailedAt DateTime?
viewedAt DateTime?
createdAt DateTime @default(now())
updatedAt DateTime @updatedAt
practice Practice @relation(fields: [practiceId], references: [id], onDelete: Cascade)
@@map("reports")
}
// ─── Leads ────────────────────────────────────────────────
model Lead {
id String @id @default(cuid())
practiceId String
name String?
phone String?
email String?
message String? @db.Text
source String // site, gbp, social, direct
status String @default("NEW") // NEW, CONTACTED, CONVERTED, LOST
createdAt DateTime @default(now())
updatedAt DateTime @updatedAt
practice Practice @relation(fields: [practiceId], references: [id], onDelete: Cascade)
@@map("leads")
}
// ─── Audit Log ────────────────────────────────────────────
model AuditLog {
id String @id @default(cuid())
practiceId String?
userId String?
action String // CREATE, UPDATE, DELETE, LOGIN, SKILL_EXECUTE, etc.
entityType String // table name or resource type
entityId String?
oldValue Json?
newValue Json?
metadata Json @default("{}")
createdAt DateTime @default(now())
practice Practice? @relation(fields: [practiceId], references: [id], onDelete: SetNull)
user User? @relation(fields: [userId], references: [id], onDelete: SetNull)
@@index([practiceId, createdAt])
@@index([userId, createdAt])
@@map("audit_logs")
}
6.2 Schema Principles#
- No soft deletes — hard delete with audit log (simpler for agents)
- No complex enums — use
Stringwith validation (easier to modify) - JSON for flexible config —
settings,config,metadatafields - Encrypted at app layer — tokens stored as strings, encrypted before DB
- No separate billing tables for MVP — store subscription info on Practice
- No separate AI cost tracking table — store cost on Job and ContentPiece
7. FastAPI Backend#
7.1 Project Structure#
rankflow-api/
├── src/
│ ├── main.py # FastAPI app, middleware, lifespan
│ ├── config.py # Settings, env vars
│ ├── dependencies.py # Auth, DB, Redis deps
│ │
│ ├── api/
│ │ ├── v1/
│ │ │ ├── __init__.py
│ │ │ ├── router.py # Aggregate all v1 routers
│ │ │ ├── auth.py # Better Auth integration
│ │ │ ├── practices.py # Practice CRUD
│ │ │ ├── locations.py # Location CRUD
│ │ │ ├── gbp.py # GBP operations
│ │ │ ├── social.py # Social operations
│ │ │ ├── content.py # Content management
│ │ │ ├── site.py # Site/landing page
│ │ │ ├── skills.py # Skill execution endpoints
│ │ │ ├── reports.py # Report generation
│ │ │ ├── leads.py # Lead management
│ │ │ ├── webhooks.py # Stripe, Composio, Google
│ │ │ └── health.py # Health checks
│ │ └── __init__.py
│ │
│ ├── harness/
│ │ ├── __init__.py
│ │ ├── main.py # RankFlowHarness class
│ │ ├── registry.py # SkillRegistry
│ │ ├── planner.py # Goal → ExecutionPlan
│ │ ├── executor.py # Skill execution engine
│ │ ├── audit.py # Audit logging
│ │ ├── metrics.py # Metrics collection
│ │ └── healing.py # Self-healing logic
│ │
│ ├── workflows/
│ │ ├── __init__.py
│ │ ├── restate_client.py # Restate connection
│ │ ├── gbp.py # GBP workflows
│ │ ├── social.py # Social workflows
│ │ ├── site.py # Site generation workflows
│ │ ├── content.py # Content generation workflows
│ │ ├── report.py # Report workflows
│ │ └── onboarding.py # Onboarding workflows
│ │
│ ├── services/
│ │ ├── __init__.py
│ │ ├── ai/
│ │ │ ├── __init__.py
│ │ │ ├── router.py # Claude → GPT-4o fallback
│ │ │ ├── prompts.py # Prompt templates
│ │ │ └── models.py # Pydantic models
│ │ ├── gbp/
│ │ │ ├── __init__.py
│ │ │ ├── client.py # GBP API client
│ │ │ ├── auth.py # OAuth flow
│ │ │ └── types.py # Pydantic models
│ │ ├── social/
│ │ │ ├── __init__.py
│ │ │ ├── composio.py # Composio integration
│ │ │ ├── zernio.py # Zernio scheduling
│ │ │ └── platforms.py # Platform-specific logic
│ │ ├── seo/
│ │ │ ├── __init__.py
│ │ │ ├── dataforseo.py # DataForSEO client
│ │ │ ├── analyzer.py # SEO analysis
│ │ │ └── schema.py # Schema.org generation
│ │ ├── browser/
│ │ │ ├── __init__.py
│ │ │ ├── hyperbrowser.py # Hyperbrowser API
│ │ │ ├── firecrawl.py # Firecrawl API
│ │ │ └── scrapegraph.py # ScrapeGraph API
│ │ ├── storage/
│ │ │ ├── __init__.py
│ │ │ └── s3.py # S3 upload/download
│ │ └── email/
│ │ ├── __init__.py
│ │ └── resend.py # Resend client
│ │
│ ├── db/
│ │ ├── __init__.py
│ │ ├── prisma.py # Prisma client singleton
│ │ └── models.py # Pydantic models for DB
│ │
│ ├── core/
│ │ ├── __init__.py
│ │ ├── auth.py # Better Auth integration
│ │ ├── cache.py # Redis cache wrapper
│ │ ├── crypto.py # Encryption/decryption
│ │ ├── logging.py # Structured logging setup
│ │ └── exceptions.py # Custom exceptions
│ │
│ └── skills/ # Skill definitions (loaded at runtime)
│ ├── 00-platform-overview.md
│ ├── 01-practice-setup.md
│ ├── 02-asset-upload.md
│ ├── 03-gbp-connect.md
│ ├── 04-gbp-post-create.md
│ ├── 05-gbp-post-schedule.md
│ ├── 06-gbp-review-reply.md
│ ├── 07-gbp-insights-fetch.md
│ ├── 08-social-connect.md
│ ├── 09-social-post-create.md
│ ├── 10-social-post-schedule.md
│ ├── 11-citation-submit.md
│ ├── 12-citation-verify.md
│ ├── 13-site-generate.md
│ ├── 14-site-update.md
│ ├── 15-site-evolve.md
│ ├── 16-seo-audit.md
│ ├── 17-keyword-track.md
│ ├── 18-competitor-analyze.md
│ ├── 19-report-generate.md
│ ├── 20-report-email.md
│ ├── 21-content-generate.md
│ ├── 22-content-approve.md
│ ├── 23-lead-capture.md
│ ├── 24-lead-nurture.md
│ ├── 25-dns-configure.md
│ ├── 26-billing-subscribe.md
│ ├── 27-user-invite.md
│ ├── 28-notification-send.md
│ ├── 29-error-handle.md
│ └── 30-fallback-execute.md
│
├── prisma/
│ └── schema.prisma
│
├── tests/
│ ├── unit/
│ ├── integration/
│ └── skills/ # Skill acceptance tests
│
├── Dockerfile
├── docker-compose.yml
├── pyproject.toml
└── .env.example
7.2 FastAPI App Setup#
# src/main.py
from contextlib import asynccontextmanager
from fastapi import FastAPI
from fastapi.middleware.cors import CORSMiddleware
from src.config import settings
from src.core.logging import setup_logging
from src.db.prisma import prisma
from src.harness.main import RankFlowHarness
from src.workflows.restate_client import RestateClient
from src.api.v1.router import api_router
@asynccontextmanager
async def lifespan(app: FastAPI):
# Startup
setup_logging()
await prisma.connect()
app.state.harness = RankFlowHarness()
app.state.restate = RestateClient()
await app.state.harness.initialize()
yield
# Shutdown
await prisma.disconnect()
app = FastAPI(
title="RankFlow AI API",
description="Agent-centric local SEO automation platform",
version="2.0.0",
lifespan=lifespan
)
# Middleware
app.add_middleware(
CORSMiddleware,
allow_origins=[settings.APP_URL],
allow_credentials=True,
allow_methods=["*"],
allow_headers=["*"],
)
# Request ID injection
@app.middleware("http")
async def request_id_middleware(request, call_next):
request_id = str(uuid.uuid4())
request.state.request_id = request_id
response = await call_next(request)
response.headers["X-Request-ID"] = request_id
return response
# Routes
app.include_router(api_router, prefix="/api/v1")
# Health check
@app.get("/health")
async def health_check():
return {
"status": "healthy",
"version": "2.0.0",
"timestamp": datetime.utcnow().isoformat()
}
7.3 Skill Execution Endpoint#
# src/api/v1/skills.py
from fastapi import APIRouter, Depends, HTTPException
from pydantic import BaseModel
from src.dependencies import get_current_user, get_harness
from src.harness.main import RankFlowHarness
router = APIRouter(prefix="/skills", tags=["skills"])
class ExecuteSkillRequest(BaseModel):
skill_id: str
payload: dict
async_execution: bool = False # If true, return job ID immediately
class ExecuteSkillResponse(BaseModel):
status: str
data: dict | None = None
job_id: str | None = None
error: str | None = None
@router.post("/execute", response_model=ExecuteSkillResponse)
async def execute_skill(
request: ExecuteSkillRequest,
user: User = Depends(get_current_user),
harness: RankFlowHarness = Depends(get_harness)
):
"""Execute a skill by ID."""
context = ExecutionContext(
user_id=user.id,
practice_id=request.payload.get("practice_id"),
trigger="manual",
request_id=getattr(request, "state", {}).get("request_id")
)
if request.async_execution:
# Queue for background execution
job = await harness.queue_skill(request.skill_id, request.payload, context)
return ExecuteSkillResponse(status="QUEUED", job_id=job.id)
# Synchronous execution
result = await harness.execute(request.skill_id, request.payload, context)
return ExecuteSkillResponse(
status=result.status,
data=result.data,
error=result.error
)
@router.get("/list")
async def list_skills(harness: RankFlowHarness = Depends(get_harness)):
"""List all available skills."""
skills = harness.registry.list_all()
return [
{
"id": s.id,
"name": s.name,
"category": s.category,
"autonomous": s.autonomous,
"description": s.description
}
for s in skills
]
@router.get("/{skill_id}")
async def get_skill(skill_id: str, harness: RankFlowHarness = Depends(get_harness)):
"""Get skill details."""
skill = harness.registry.get(skill_id)
if not skill:
raise HTTPException(status_code=404, detail="Skill not found")
return skill.to_dict()
8. Next.js Frontend#
8.1 Project Structure#
rankflow-web/
├── src/
│ ├── app/
│ │ ├── layout.tsx # Root layout with providers
│ │ ├── page.tsx # Marketing landing page
│ │ │
│ │ ├── (dashboard)/ # Authenticated dashboard
│ │ │ ├── layout.tsx # Dashboard shell
│ │ │ ├── page.tsx # Dashboard home
│ │ │ ├── practice/
│ │ │ ├── locations/
│ │ │ ├── gbp/
│ │ │ ├── social/
│ │ │ ├── content/
│ │ │ ├── site/
│ │ │ ├── reports/
│ │ │ ├── leads/
│ │ │ ├── settings/
│ │ │ └── billing/
│ │ │
│ │ ├── site/ # Public client sites
│ │ │ └── [practiceSlug]/
│ │ │ └── page.tsx # Dynamic site renderer
│ │ │
│ │ ├── auth/
│ │ │ ├── signin/
│ │ │ ├── signup/
│ │ │ └── callback/
│ │ │
│ │ └── api/ # Next.js API routes (minimal)
│ │ └── webhooks/
│ │
│ ├── components/
│ │ ├── ui/ # shadcn/ui components
│ │ ├── dashboard/ # Dashboard-specific
│ │ ├── site/ # Site/landing page sections
│ │ └── forms/ # Reusable forms
│ │
│ ├── lib/
│ │ ├── api.ts # FastAPI client (fetch wrapper)
│ │ ├── auth.ts # Better Auth client
│ │ ├── utils.ts # Utilities
│ │ └── hooks.ts # React hooks
│ │
│ └── types/
│ └── index.ts # TypeScript types
│
├── public/
│ └── templates/ # Site template assets
│
├── next.config.js
├── tailwind.config.ts
├── tsconfig.json
└── package.json
8.2 API Client#
// src/lib/api.ts
const API_URL = process.env.NEXT_PUBLIC_API_URL;
class RankFlowAPI {
private async request<T>(
endpoint: string,
options: RequestInit = {}
): Promise<T> {
const session = await authClient.getSession();
const response = await fetch(`${API_URL}${endpoint}`, {
...options,
headers: {
"Content-Type": "application/json",
...(session?.token ? { Authorization: `Bearer ${session.token}` } : {}),
...options.headers,
},
});
if (!response.ok) {
const error = await response.json();
throw new Error(error.message || "API request failed");
}
return response.json();
}
// Skills
async executeSkill(skillId: string, payload: object, async = false) {
return this.request("/api/v1/skills/execute", {
method: "POST",
body: JSON.stringify({ skill_id: skillId, payload, async_execution: async }),
});
}
async listSkills() {
return this.request("/api/v1/skills/list");
}
// Practices
async getPractices() {
return this.request("/api/v1/practices");
}
async getPractice(id: string) {
return this.request(`/api/v1/practices/${id}`);
}
// GBP
async getGbpAccounts(practiceId: string) {
return this.request(`/api/v1/gbp/accounts?practice_id=${practiceId}`);
}
async createGbpPost(practiceId: string, data: object) {
return this.request("/api/v1/gbp/posts", {
method: "POST",
body: JSON.stringify({ practice_id: practiceId, ...data }),
});
}
// Site
async getSiteSections(practiceId: string) {
return this.request(`/api/v1/site/sections?practice_id=${practiceId}`);
}
async updateSiteSection(practiceId: string, sectionKey: string, data: object) {
return this.request(`/api/v1/site/sections/${sectionKey}`, {
method: "PUT",
body: JSON.stringify({ practice_id: practiceId, ...data }),
});
}
// Reports
async getReports(practiceId: string) {
return this.request(`/api/v1/reports?practice_id=${practiceId}`);
}
// Leads
async getLeads(practiceId: string) {
return this.request(`/api/v1/leads?practice_id=${practiceId}`);
}
}
export const api = new RankFlowAPI();
8.3 Site Renderer (Dynamic Landing Pages)#
// src/app/site/[practiceSlug]/page.tsx
import { notFound } from "next/navigation";
import { Metadata } from "next";
interface Props {
params: { practiceSlug: string };
}
export async function generateMetadata({ params }: Props): Promise<Metadata> {
const practice = await getPracticeBySlug(params.practiceSlug);
if (!practice) return { title: "Not Found" };
return {
title: practice.seoTitle || practice.name,
description: practice.seoDescription,
robots: practice.sitePublished ? "index, follow" : "noindex, nofollow",
};
}
export default async function SitePage({ params }: Props) {
const practice = await getPracticeBySlug(params.practiceSlug);
if (!practice || !practice.sitePublished) {
notFound();
}
const sections = await getSiteSections(practice.id);
return (
<div style={{ "--primary": practice.primaryColor } as React.CSSProperties}>
<SchemaInjector practice={practice} />
{sections
.filter((s) => s.isVisible)
.sort((a, b) => a.sortOrder - b.sortOrder)
.map((section) => (
<SiteSection
key={section.sectionKey}
section={section}
practice={practice}
/>
))}
</div>
);
}
// Site sections map
const sectionComponents: Record<string, React.FC<any>> = {
hero: HeroSection,
about: AboutSection,
services: ServicesSection,
testimonials: TestimonialsSection,
contact: ContactSection,
faq: FAQSection,
cta: CTASection,
};
function SiteSection({ section, practice }: { section: any; practice: any }) {
const Component = sectionComponents[section.sectionKey];
if (!Component) return null;
return (
<section id={section.sectionKey} className={`section-${section.sectionKey}`}>
<Component content={section.content} mediaUrls={section.mediaUrls} config={section.config} practice={practice} />
</section>
);
}
9. Workflow Engine (Restate)#
9.1 Why Restate?#
Restate provides durable execution — the ability to write code that survives crashes, retries, and long waits. For an agent-centric platform, this is critical:
- No job status tables — Restate tracks state internally
- No polling — Workflows sleep and wake up on events
- No timeout anxiety — Workflows can run for days
- Transparent — You can inspect exactly where any workflow is
- Type-safe — Python SDK with full type hints
9.2 Restate Setup#
# src/workflows/restate_client.py
import restate
from restate import Context, WorkflowContext
# Restate server runs as a separate service (or embedded)
RESTATE_ENDPOINT = os.getenv("RESTATE_ENDPOINT", "http://localhost:8080")
class RestateClient:
def __init__(self):
self.endpoint = RESTATE_ENDPOINT
async def workflow(self, name: str, payload: dict, retry_policy: dict = None):
"""Invoke a Restate workflow."""
# Use Restate SDK to invoke
pass
async def send_event(self, workflow_id: str, event_name: str, payload: dict):
"""Send an event to a waiting workflow."""
pass
9.3 Example Workflows#
# src/workflows/gbp.py
import restate
from restate import WorkflowContext
@restate.workflow
async def gbp_post_create_workflow(ctx: WorkflowContext, request: dict):
"""
Durable workflow for creating a GBP post.
Survives crashes, retries failed steps, waits for external events.
"""
practice_id = request["practice_id"]
location_id = request["location_id"]
# Step 1: Validate practice and GBP connection (retried automatically)
practice = await ctx.run("validate_practice", lambda: validate_practice(practice_id))
gbp_account = await ctx.run("get_gbp_account", lambda: get_gbp_account(practice_id))
if not gbp_account:
raise restate.TerminalError("No active GBP account found")
# Step 2: Generate content if not provided
content = request.get("content")
if not content:
content = await ctx.run("generate_content", lambda: ai.generate_gbp_post(
practice=practice,
location=location_id,
tone="professional"
))
# Step 3: Publish to GBP (with retry)
result = await ctx.run("publish_post", lambda: gbp_api.create_post(
account=gbp_account,
location_id=location_id,
content=content,
media_urls=request.get("media_urls", []),
cta_type=request.get("cta_type", "BOOK")
))
# Step 4: Store in database
post = await ctx.run("store_post", lambda: db.gbp_post.create({
"practice_id": practice_id,
"gbp_location_id": location_id,
"content": content,
"status": "PUBLISHED",
"gbp_post_id": result["name"],
"published_at": datetime.utcnow(),
}))
# Step 5: Audit log
await ctx.run("audit", lambda: audit.log({
"action": "GBP_POST_CREATED",
"practice_id": practice_id,
"entity_id": post.id,
"metadata": {"gbp_post_id": result["name"]}
}))
return {"post_id": post.id, "gbp_post_id": result["name"]}
@restate.workflow
async def gbp_oauth_workflow(ctx: WorkflowContext, request: dict):
"""
Workflow for GBP OAuth connection.
Can wait for days for the user to complete OAuth.
"""
practice_id = request["practice_id"]
# Step 1: Generate OAuth URL
auth_data = await ctx.run("generate_auth_url", lambda: gbp_auth.get_auth_url(
practice_id=practice_id,
redirect_uri=f"{APP_URL}/api/webhooks/gbp/oauth"
))
# Step 2: Wait for OAuth callback (can sleep for days)
callback = await ctx.promise("oauth_callback")
# Step 3: Exchange code for tokens
tokens = await ctx.run("exchange_tokens", lambda: gbp_auth.exchange_code(
code=callback["code"],
state=callback["state"]
))
# Step 4: Store tokens
await ctx.run("store_tokens", lambda: db.gbp_account.create({
"practice_id": practice_id,
"account_email": tokens["email"],
"access_token": encrypt(tokens["access_token"]),
"refresh_token": encrypt(tokens["refresh_token"]),
"token_expires_at": tokens["expires_at"],
}))
# Step 5: Sync locations
locations = await ctx.run("sync_locations", lambda: gbp_api.list_locations(tokens["access_token"]))
for loc in locations:
await ctx.run("store_location", lambda: db.gbp_location.create({...}))
return {"account_id": account.id, "locations_count": len(locations)}
@restate.workflow
async def onboarding_workflow(ctx: WorkflowContext, request: dict):
"""
Complete practice onboarding workflow.
Orchestrates multiple skills over time.
"""
practice_id = request["practice_id"]
# Step 1: Create practice record
practice = await ctx.run("create_practice", lambda: db.practice.create(request))
# Step 2: Generate initial site content
await ctx.run("generate_site", lambda: site.generate(practice_id))
# Step 3: Wait for GBP connection (user must complete OAuth)
await ctx.run("notify_gbp_auth", lambda: notify_user(practice_id, "Please connect your Google Business Profile"))
# Wait for GBP connection event
await ctx.promise("gbp_connected")
# Step 4: Generate first GBP post
await ctx.run("create_first_post", lambda: harness.execute("04-gbp-post-create", {
"practice_id": practice_id,
"content": "Welcome to our practice! We are now accepting new patients."
}))
# Step 5: Schedule weekly posts
await ctx.run("schedule_posts", lambda: schedule_weekly_posts(practice_id))
# Step 6: Wait 1 day, then send welcome report
await ctx.sleep(timedelta(days=1))
await ctx.run("send_welcome_report", lambda: report.generate_welcome(practice_id))
return {"practice_id": practice_id, "status": "onboarded"}
9.4 Restate Deployment#
Restate can be deployed:
- Self-hosted (Docker container alongside the app)
- Restate Cloud (managed, pay per invocation)
For MVP, self-hosted on the same EC2 instance:
# docker-compose.yml (Dokploy services)
services:
restate:
image: restatedev/restate:latest
ports:
- "8080:8080"
- "9070:9070" # Admin port
environment:
- RESTATE_OBSERVABILITY__LOG__FORMAT=json
volumes:
- restate-data:/restate-data
10. Landing Page Scale Plan#
10.1 Phase 1: Path-Based Routing (MVP)#
https://rankflow.ai/site/dr-sharma-cardiology-mumbai
- Single Next.js app
- Dynamic route
/site/[practiceSlug] - Content from database
- No DNS complexity
- No SSL complexity
10.2 Phase 2: Subdomain Routing#
https://dr-sharma.rankflow.ai
- Wildcard DNS:
*.rankflow.ai → EC2 IP - Next.js middleware checks
Hostheader - Same app, same database
- SSL via Dokploy (Let's Encrypt wildcard)
// src/middleware.ts
export function middleware(request: NextRequest) {
const host = request.headers.get("host") || "";
// Check if subdomain request
if (host.endsWith("rankflow.ai") && host !== "rankflow.ai") {
const subdomain = host.replace(".rankflow.ai", "");
// Rewrite to site renderer
return NextResponse.rewrite(
new URL(`/site/${subdomain}${request.nextUrl.pathname}`, request.url)
);
}
return NextResponse.next();
}
10.3 Phase 3: Custom Domains#
https://drsharma.com → CNAME to dr-sharma.rankflow.ai
- Doctor adds CNAME record
- Cloudflare API verifies DNS
- Dokploy provisions SSL certificate
- Next.js middleware handles custom domain
10.4 The "Living Site" Architecture#
The site is not static — it evolves based on data:
@restate.cron(every="7d")
async def evolve_site_workflow(ctx: WorkflowContext):
"""Weekly site evolution — updates content based on SEO data."""
practices = await ctx.run("get_active_practices", lambda: db.practice.find_many(
site_published=True,
auto_update_enabled=True
))
for practice in practices:
# 1. Get latest SEO data
rankings = await ctx.run("get_rankings", lambda: dataforseo.get_rankings(practice.id))
# 2. Get GBP insights
insights = await ctx.run("get_insights", lambda: gbp_api.get_insights(practice.id))
# 3. AI plans updates
update_plan = await ctx.run("plan_updates", lambda: ai.plan_site_updates(
practice=practice,
rankings=rankings,
insights=insights
))
# 4. Apply updates
for update in update_plan.updates:
await ctx.run(f"update_{update.section}", lambda: db.site_section.update(
practice_id=practice.id,
section_key=update.section,
content=update.new_content
))
# 5. Revalidate site
await ctx.run("revalidate", lambda: revalidate_site(practice.subdomain))
# 6. Notify doctor
await ctx.run("notify", lambda: email.send_update_summary(
practice_id=practice.id,
changes=update_plan.updates
))
10.5 Site Templates#
Templates are React components + config:
// src/lib/site-templates.ts
export interface SiteTemplate {
id: string;
name: string;
sections: string[]; // hero, about, services, testimonials, contact, faq
defaultConfig: {
primaryColor: string;
fontFamily: string;
borderRadius: string;
};
}
export const templates: Record<string, SiteTemplate> = {
"medical-modern": {
id: "medical-modern",
name: "Medical Modern",
sections: ["hero", "about", "services", "testimonials", "faq", "contact"],
defaultConfig: {
primaryColor: "#2563eb",
fontFamily: "Inter, system-ui, sans-serif",
borderRadius: "0.5rem",
},
},
"dental-clean": {
id: "dental-clean",
name: "Dental Clean",
sections: ["hero", "about", "services", "gallery", "testimonials", "contact"],
defaultConfig: {
primaryColor: "#06b6d4",
fontFamily: "DM Sans, system-ui, sans-serif",
borderRadius: "9999px",
},
},
};
10.6 Asset Management#
Doctors upload assets via dashboard:
- Logo → S3, optimized to WebP
- Doctor photo → S3, face detection, optimized
- Clinic photos → S3, gallery generation
- Certificates → S3, displayed in about section
AI can also generate missing assets:
- No logo? → Generate simple text-based logo
- No photos? → Use GBP photos or stock images
- No doctor photo? → Prompt user or use placeholder
11. SEO Data Integration (DataForSEO)#
11.1 DataForSEO API#
DataForSEO provides comprehensive SEO data via REST API:
| API | Purpose | Use Case |
|---|---|---|
| SERP API | Search results for keywords | Rank tracking |
| Keywords Data API | Search volume, difficulty, CPC | Keyword research |
| Competitor API | Competitor domain metrics | Competitor analysis |
| Backlinks API | Backlink profile | Link building |
| On-Page API | Page analysis | SEO audit |
| Business Data API | GBP data, reviews | Local SEO |
11.2 DataForSEO Client#
# src/services/seo/dataforseo.py
import httpx
class DataForSEOClient:
def __init__(self):
self.login = os.getenv("DATAFORSEO_LOGIN")
self.password = os.getenv("DATAFORSEO_PASSWORD")
self.base_url = "https://api.dataforseo.com/v3"
async def _request(self, endpoint: str, data: dict) -> dict:
async with httpx.AsyncClient() as client:
response = await client.post(
f"{self.base_url}/{endpoint}",
json=data,
auth=(self.login, self.password)
)
response.raise_for_status()
return response.json()
async def get_serp(self, keyword: str, location: str, language: str = "en"):
"""Get SERP results for a keyword."""
return await self._request("serp/google/organic/live/advanced", {
"keyword": keyword,
"location_code": location,
"language_code": language,
"device": "desktop",
"os": "windows"
})
async def get_rankings(self, keywords: list[str], domain: str, location: str):
"""Check where a domain ranks for keywords."""
results = []
for keyword in keywords:
serp = await self.get_serp(keyword, location)
# Find domain in results
position = None
for i, result in enumerate(serp["tasks"][0]["result"][0]["items"]):
if domain in result.get("domain", ""):
position = i + 1
break
results.append({
"keyword": keyword,
"position": position,
"date": datetime.utcnow().isoformat()
})
return results
async def get_keyword_data(self, keyword: str, location: str):
"""Get search volume, difficulty, CPC."""
return await self._request("keywords_data/google/search_volume/live", {
"keywords": [keyword],
"location_code": location,
"language_code": "en"
})
async def get_competitors(self, domain: str, location: str):
"""Get competitor domains."""
return await self._request("domain_analytics/competitors/live", {
"target": domain,
"location_code": location,
"language_code": "en"
})
async def get_backlinks(self, domain: str):
"""Get backlink profile."""
return await self._request("backlinks/backlinks/live", {
"target": domain,
"mode": "as_is"
})
11.3 Rank Tracking Workflow#
@restate.cron(every="1d")
async def rank_tracking_workflow(ctx: WorkflowContext):
"""Daily rank tracking for all active practices."""
practices = await ctx.run("get_practices", lambda: db.practice.find_many(
status="active",
tier={"in": ["STARTER", "PRO", "ENTERPRISE"]}
))
for practice in practices:
locations = await ctx.run("get_locations", lambda: db.location.find_many(
practice_id=practice.id
))
for location in locations:
keywords = location.target_keywords
if not keywords:
continue
# Get rankings
rankings = await ctx.run("fetch_rankings", lambda: dataforseo.get_rankings(
keywords=keywords,
domain=practice.custom_domain or f"{practice.subdomain}.rankflow.ai",
location=get_location_code(location.city)
))
# Store results
await ctx.run("store_rankings", lambda: db.rank_tracking.create_many(
data=[{
"practice_id": practice.id,
"location_id": location.id,
"keyword": r["keyword"],
"position": r["position"],
"date": datetime.utcnow()
} for r in rankings]
))
# Check for significant changes
for ranking in rankings:
if ranking["position"] and ranking["position"] <= 3:
await ctx.run("notify_top3", lambda: notification.send(
practice_id=practice.id,
title="🎉 Top 3 Ranking!",
message=f"Your practice is now in the top 3 for '{ranking['keyword']}'"
))
12. Browser Automation (API-First)#
12.1 Hyperbrowser#
Hyperbrowser provides remote browser sessions via API:
# src/services/browser/hyperbrowser.py
import httpx
class HyperbrowserClient:
def __init__(self):
self.api_key = os.getenv("HYPERBROWSER_API_KEY")
self.base_url = "https://app.hyperbrowser.ai/api/v1"
async def create_session(self):
"""Create a new browser session."""
async with httpx.AsyncClient() as client:
response = await client.post(
f"{self.base_url}/sessions",
headers={"Authorization": f"Bearer {self.api_key}"},
json={"browser": "chrome", "headless": True}
)
return response.json()["session_id"]
async def navigate(self, session_id: str, url: str):
async with httpx.AsyncClient() as client:
response = await client.post(
f"{self.base_url}/sessions/{session_id}/navigate",
headers={"Authorization": f"Bearer {self.api_key}"},
json={"url": url}
)
return response.json()
async def fill_form(self, session_id: str, selector: str, value: str):
async with httpx.AsyncClient() as client:
response = await client.post(
f"{self.base_url}/sessions/{session_id}/fill",
headers={"Authorization": f"Bearer {self.api_key}"},
json={"selector": selector, "value": value}
)
return response.json()
async def click(self, session_id: str, selector: str):
async with httpx.AsyncClient() as client:
response = await client.post(
f"{self.base_url}/sessions/{session_id}/click",
headers={"Authorization": f"Bearer {self.api_key}"},
json={"selector": selector}
)
return response.json()
async def screenshot(self, session_id: str):
async with httpx.AsyncClient() as client:
response = await client.get(
f"{self.base_url}/sessions/{session_id}/screenshot",
headers={"Authorization": f"Bearer {self.api_key}"}
)
return response.content
async def close_session(self, session_id: str):
async with httpx.AsyncClient() as client:
await client.delete(
f"{self.base_url}/sessions/{session_id}",
headers={"Authorization": f"Bearer {self.api_key}"}
)
12.2 Firecrawl#
Firecrawl scrapes and extracts structured data:
# src/services/browser/firecrawl.py
import httpx
class FirecrawlClient:
def __init__(self):
self.api_key = os.getenv("FIRECRAWL_API_KEY")
self.base_url = "https://api.firecrawl.dev/v1"
async def scrape(self, url: str, extract_schema: dict = None):
"""Scrape a URL and optionally extract structured data."""
async with httpx.AsyncClient() as client:
response = await client.post(
f"{self.base_url}/scrape",
headers={"Authorization": f"Bearer {self.api_key}"},
json={
"url": url,
"formats": ["markdown", "html"],
"onlyMainContent": True,
"extract": extract_schema
}
)
return response.json()
async def verify_nap(self, url: str, expected_nap: dict):
"""Verify NAP consistency on a directory listing."""
schema = {
"type": "object",
"properties": {
"business_name": {"type": "string"},
"address": {"type": "string"},
"phone": {"type": "string"}
}
}
result = await self.scrape(url, extract_schema=schema)
extracted = result["data"]["extract"]
return {
"name_match": self._fuzzy_match(expected_nap["name"], extracted.get("business_name", "")),
"address_match": self._fuzzy_match(expected_nap["address"], extracted.get("address", "")),
"phone_match": self._normalize_phone(expected_nap["phone"]) in self._normalize_phone(extracted.get("phone", ""))
}
12.3 Citation Submission Workflow#
@restate.workflow
async def citation_submit_workflow(ctx: WorkflowContext, request: dict):
"""Submit practice to a directory using browser automation APIs."""
practice_id = request["practice_id"]
directory = request["directory"] # justdial, practo, etc.
practice = await ctx.run("get_practice", lambda: db.practice.find_unique(practice_id))
location = await ctx.run("get_location", lambda: db.location.find_first(practice_id))
# Use Hyperbrowser for form submission
session_id = await ctx.run("create_session", lambda: hyperbrowser.create_session())
try:
if directory == "justdial":
await ctx.run("navigate", lambda: hyperbrowser.navigate(session_id, "https://www.justdial.com/Free-Listing"))
await ctx.run("fill_name", lambda: hyperbrowser.fill_form(session_id, "input[name='cnm']", location.business_name))
await ctx.run("fill_phone", lambda: hyperbrowser.fill_form(session_id, "input[name='mn']", location.phone))
await ctx.run("fill_city", lambda: hyperbrowser.fill_form(session_id, "input[name='city']", location.city))
await ctx.run("submit", lambda: hyperbrowser.click(session_id, "button[type='submit']"))
# Capture result
screenshot = await ctx.run("screenshot", lambda: hyperbrowser.screenshot(session_id))
# Store result
await ctx.run("store", lambda: db.citation.create({
"practice_id": practice_id,
"directory": directory,
"status": "SUBMITTED",
"screenshot_url": await s3.upload(screenshot)
}))
elif directory == "practo":
# Similar flow for Practo
pass
finally:
await ctx.run("close_session", lambda: hyperbrowser.close_session(session_id))
13. Social Media Integration#
13.1 Composio#
Composio handles OAuth and action execution for social platforms:
# src/services/social/composio.py
import httpx
class ComposioClient:
def __init__(self):
self.api_key = os.getenv("COMPOSIO_API_KEY")
self.base_url = "https://backend.composio.dev/api/v1"
async def initiate_connection(self, app: str, redirect_uri: str, metadata: dict):
"""Start OAuth flow for a social platform."""
async with httpx.AsyncClient() as client:
response = await client.post(
f"{self.base_url}/connectedAccounts",
headers={
"x-api-key": self.api_key,
"Content-Type": "application/json"
},
json={
"appName": app,
"redirectUri": redirect_uri,
"metadata": metadata
}
)
return response.json()
async def execute_action(self, action: str, params: dict):
"""Execute a Composio action."""
async with httpx.AsyncClient() as client:
response = await client.post(
f"{self.base_url}/actions/{action}/execute",
headers={
"x-api-key": self.api_key,
"Content-Type": "application/json"
},
json={"input": params}
)
return response.json()
async def publish_facebook_post(self, connection_id: str, message: str, media_urls: list = None):
return await self.execute_action("FACEBOOK_PUBLISH_POST", {
"connected_account_id": connection_id,
"message": message,
"photos": media_urls or []
})
async def publish_instagram_post(self, connection_id: str, caption: str, image_url: str):
return await self.execute_action("INSTAGRAM_PUBLISH_POST", {
"connected_account_id": connection_id,
"caption": caption,
"image_url": image_url
})
13.2 Zernio Scheduling#
Zernio handles post scheduling:
# src/services/social/zernio.py
import httpx
class ZernioClient:
def __init__(self):
self.api_key = os.getenv("ZERNIO_API_KEY")
self.base_url = "https://api.zernio.com/v1"
async def schedule_posts(self, posts: list[dict]):
"""Schedule multiple posts."""
async with httpx.AsyncClient() as client:
response = await client.post(
f"{self.base_url}/schedule/batch",
headers={"Authorization": f"Bearer {self.api_key}"},
json={
"posts": posts,
"webhook_url": f"{APP_URL}/api/webhooks/zernio"
}
)
return response.json()
13.3 Social Post Workflow#
@restate.workflow
async def social_post_workflow(ctx: WorkflowContext, request: dict):
"""Create and publish/schedule social posts."""
practice_id = request["practice_id"]
platforms = request["platforms"] # ["FACEBOOK", "INSTAGRAM"]
content = request["content"]
scheduled_for = request.get("scheduled_for")
# Get connected accounts
accounts = await ctx.run("get_accounts", lambda: db.social_account.find_many(
practice_id=practice_id,
platform={"in": platforms},
is_active=True
))
if not accounts:
raise restate.TerminalError("No active social accounts found")
posts = []
for account in accounts:
if scheduled_for:
# Schedule via Zernio
post = await ctx.run("schedule", lambda: zernio.schedule_posts([{
"platform": account.platform,
"account_id": account.composio_connection_id,
"content": content,
"scheduled_for": scheduled_for.isoformat()
}]))
else:
# Publish immediately via Composio
if account.platform == "FACEBOOK":
post = await ctx.run("publish_fb", lambda: composio.publish_facebook_post(
connection_id=account.composio_connection_id,
message=content
))
elif account.platform == "INSTAGRAM":
post = await ctx.run("publish_ig", lambda: composio.publish_instagram_post(
connection_id=account.composio_connection_id,
caption=content,
image_url=request.get("image_url")
))
posts.append({
"platform": account.platform,
"external_post_id": post.get("post_id"),
"status": "SCHEDULED" if scheduled_for else "PUBLISHED"
})
# Store in database
await ctx.run("store_posts", lambda: db.social_post.create_many(
data=[{
"practice_id": practice_id,
"social_account_id": account.id,
"content": content,
"status": p["status"],
"external_post_id": p["external_post_id"],
"scheduled_for": scheduled_for
} for account, p in zip(accounts, posts)]
))
return {"posts": posts}
14. Observability & Auditability#
14.1 Structured Logging#
# src/core/logging.py
import structlog
import logging
import sys
def setup_logging():
structlog.configure(
processors=[
structlog.stdlib.filter_by_level,
structlog.stdlib.add_logger_name,
structlog.stdlib.add_log_level,
structlog.stdlib.PositionalArgumentsFormatter(),
structlog.processors.TimeStamper(fmt="iso"),
structlog.processors.StackInfoRenderer(),
structlog.processors.format_exc_info,
structlog.processors.UnicodeDecoder(),
structlog.processors.JSONRenderer()
],
context_class=dict,
logger_factory=structlog.stdlib.LoggerFactory(),
wrapper_class=structlog.stdlib.BoundLogger,
cache_logger_on_first_use=True,
)
# Configure standard library logging
logging.basicConfig(
format="%(message)s",
stream=sys.stdout,
level=logging.INFO,
)
Every log entry includes:
{
"timestamp": "2025-01-15T09:00:00Z",
"level": "info",
"logger": "rankflow.harness",
"event": "skill_execution_started",
"skill_id": "04-gbp-post-create",
"practice_id": "practice_123",
"user_id": "user_456",
"request_id": "req-789",
"trace_id": "trace-abc",
"duration_ms": null,
"cost_usd": null
}
14.2 Health Checks#
# src/api/v1/health.py
from fastapi import APIRouter
from src.db.prisma import prisma
from src.core.cache import redis
router = APIRouter()
@router.get("/health")
async def health_check():
checks = {}
status = "healthy"
# Database
try:
await prisma.$query_raw("SELECT 1")
checks["database"] = "pass"
except Exception as e:
checks["database"] = f"fail: {str(e)}"
status = "unhealthy"
# Redis
try:
await redis.ping()
checks["redis"] = "pass"
except Exception as e:
checks["redis"] = f"fail: {str(e)}"
status = "unhealthy"
# Restate
try:
# Check Restate connectivity
checks["restate"] = "pass"
except Exception as e:
checks["restate"] = f"fail: {str(e)}"
status = "degraded"
# External APIs (lightweight checks)
try:
# Check Claude API health
checks["claude_api"] = "pass"
except:
checks["claude_api"] = "fail"
if status == "healthy":
status = "degraded"
return {
"status": status,
"checks": checks,
"timestamp": datetime.utcnow().isoformat(),
"version": "2.0.0"
}
@router.get("/metrics")
async def metrics():
"""Prometheus-compatible metrics endpoint."""
# Return metrics for scraping
pass
14.3 Audit Logging#
Every significant action is logged to the AuditLog table:
# src/harness/audit.py
class AuditLogger:
async def log(self, event: AuditEvent):
"""Log an audit event."""
# Store in database
await db.audit_log.create({
"practice_id": event.practice_id,
"user_id": event.user_id,
"action": event.action,
"entity_type": event.entity_type,
"entity_id": event.entity_id,
"old_value": event.old_value,
"new_value": event.new_value,
"metadata": event.metadata
})
# Also log to structured logger
structlog.get_logger().info(
"audit_event",
practice_id=event.practice_id,
user_id=event.user_id,
action=event.action,
entity_type=event.entity_type,
entity_id=event.entity_id
)
14.4 Admin Dashboard Observability#
The admin panel shows:
| View | What It Shows |
|---|---|
| System Health | DB, Redis, Restate, external API status |
| Active Workflows | Running workflows, waiting workflows, failed workflows |
| Skill Execution Log | Recent skill executions, success/failure rates, average duration |
| Queue Depth | Pending jobs by skill category |
| Cost Tracker | AI spend per practice, per skill, per day |
| Error Feed | Recent errors, grouped by type, with stack traces |
| Practice Health Grid | All practices, color-coded by issues |
| GBP Status Board | Connected, suspended, needs attention |
| Site Evolution Log | What changed on each site this week |
15. Security & Compliance#
15.1 Token Encryption#
All OAuth tokens encrypted at application layer before DB storage:
# src/core/crypto.py
from cryptography.fernet import Fernet
import os
fernet = Fernet(os.getenv("ENCRYPTION_KEY"))
def encrypt(value: str) -> str:
return fernet.encrypt(value.encode()).decode()
def decrypt(encrypted: str) -> str:
return fernet.decrypt(encrypted.encode()).decode()
15.2 Rate Limiting#
# src/core/ratelimit.py
from redis.asyncio import Redis
redis = Redis.from_url(os.getenv("REDIS_URL"))
async def check_rate_limit(key: str, limit: int, window: int) -> bool:
"""Simple Redis-based rate limiting."""
current = await redis.incr(key)
if current == 1:
await redis.expire(key, window)
return current <= limit
# Usage in endpoints
@app.post("/api/v1/skills/execute")
async def execute_skill(request: Request, ...):
user_id = request.state.user.id
if not await check_rate_limit(f"ratelimit:skills:{user_id}", 20, 60):
raise HTTPException(429, "Rate limit exceeded")
...
15.3 DPDPA Compliance#
# src/core/compliance.py
async def export_user_data(practice_id: str) -> dict:
"""Export all data for a practice (DPDPA right to access)."""
practice = await db.practice.find_unique(practice_id)
locations = await db.location.find_many(practice_id=practice_id)
posts = await db.gbp_post.find_many(practice_id=practice_id)
reviews = await db.review.find_many(where={"location": {"practice_id": practice_id}})
return {
"practice": practice,
"locations": locations,
"posts": posts,
"reviews": reviews,
"exported_at": datetime.utcnow().isoformat()
}
async def delete_user_data(practice_id: str):
"""Delete/anonymize practice data (DPDPA right to erasure)."""
await db.practice.update(practice_id, {
"name": "[DELETED]",
"status": "deleted",
"custom_domain": None,
"subdomain": f"deleted-{datetime.utcnow().timestamp()}"
})
# Delete tokens
await db.gbp_account.update_many(
{"practice_id": practice_id},
{"access_token": "[DELETED]", "refresh_token": "[DELETED]", "is_active": False}
)
await db.social_account.update_many(
{"practice_id": practice_id},
{"access_token": "[DELETED]", "refresh_token": "[DELETED]", "is_active": False}
)
16. Project Structure#
16.1 Monorepo Layout#
rankflow/
├── apps/
│ ├── web/ # Next.js frontend
│ │ ├── src/
│ │ ├── public/
│ │ ├── next.config.js
│ │ └── package.json
│ │
│ └── api/ # FastAPI backend
│ ├── src/
│ ├── prisma/
│ ├── skills/
│ ├── tests/
│ ├── Dockerfile
│ └── pyproject.toml
│
├── packages/
│ ├── shared-types/ # Shared TypeScript/Python types
│ ├── ui-components/ # Shared React components
│ └── skill-schema/ # Skill file validation
│
├── infra/
│ ├── docker-compose.yml # Local development
│ ├── dokploy-config.json # Dokploy deployment config
│ └── aws/ # Terraform/CDK (future)
│
├── docs/
│ ├── skills/ # Skill definitions (source of truth)
│ ├── architecture.md
│ └── api-reference.md
│
├── scripts/
│ ├── deploy.sh
│ └── seed.sh
│
├── .github/
│ └── workflows/
│ ├── ci.yml
│ └── deploy.yml
│
├── README.md
├── Makefile
└── .env.example
17. Deployment Guide#
17.1 Local Development#
# 1. Clone repo
git clone https://github.com/rankflow/rankflow.git
cd rankflow
# 2. Start infrastructure
docker-compose up -d postgres redis restate
# 3. Setup API
cd apps/api
python -m venv .venv
source .venv/bin/activate
pip install -r requirements.txt
prisma generate
prisma migrate dev
python -m src.main
# 4. Setup Web (new terminal)
cd apps/web
npm install
npm run dev
# 5. Access
# API: http://localhost:8000
# Web: http://localhost:3000
# Restate UI: http://localhost:9070
17.2 Dokploy Deployment#
# 1. Provision EC2 instance (t3.large, Ubuntu 22.04)
# 2. Install Dokploy
curl -fsSL https://dokploy.com/install.sh | bash
# 3. Configure Dokploy via UI
# - Add GitHub integration
# - Create project "rankflow"
# - Add services:
# - rankflow-web (Next.js)
# - rankflow-api (FastAPI)
# - rankflow-worker (Restate)
# - postgres (database)
# - redis (cache)
# - restate (workflow engine)
# 4. Set environment variables in Dokploy UI
# 5. Deploy
# Dokploy auto-deploys on git push to main branch
17.3 CI/CD (GitHub Actions)#
# .github/workflows/deploy.yml
name: Deploy
on:
push:
branches: [main]
jobs:
test:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v4
- name: Setup Python
uses: actions/setup-python@v5
with:
python-version: "3.11"
- name: Setup Node
uses: actions/setup-node@v4
with:
node-version: "20"
- name: Test API
run: |
cd apps/api
pip install -r requirements.txt
pytest
- name: Test Web
run: |
cd apps/web
npm ci
npm run test
- name: Test Skills
run: |
cd apps/api
python -m pytest tests/skills/
deploy:
needs: test
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v4
- name: Deploy to Dokploy
run: |
# Trigger Dokploy deployment via webhook
curl -X POST ${{ secrets.DOKPLOY_WEBHOOK_URL }}
Appendix: Skill Registry#
Core Skills (V1)#
| ID | Name | Category | Autonomous | External APIs |
|---|---|---|---|---|
| 01 | Practice Setup | onboarding | false | — |
| 02 | Asset Upload | onboarding | false | S3 |
| 03 | GBP Connect | gbp | false | Google OAuth |
| 04 | GBP Post Create | gbp | true | GBP API, Claude |
| 05 | GBP Post Schedule | gbp | true | GBP API |
| 06 | GBP Review Reply | gbp | true | GBP API, Claude |
| 07 | GBP Insights Fetch | gbp | true | GBP API |
| 08 | Social Connect | social | false | Composio |
| 09 | Social Post Create | social | true | Composio, Claude |
| 10 | Social Post Schedule | social | true | Zernio |
| 11 | Citation Submit | citation | true | Hyperbrowser |
| 12 | Citation Verify | citation | true | Firecrawl |
| 13 | Site Generate | site | false | Claude, S3 |
| 14 | Site Update | site | true | Claude, S3 |
| 15 | Site Evolve | site | true | DataForSEO, Claude |
| 16 | SEO Audit | seo | true | DataForSEO, Claude |
| 17 | Keyword Track | seo | true | DataForSEO |
| 18 | Competitor Analyze | seo | true | DataForSEO |
| 19 | Report Generate | report | true | DataForSEO, Claude |
| 20 | Report Email | report | true | Resend |
| 21 | Content Generate | content | true | Claude |
| 22 | Content Approve | content | false | — |
| 23 | Lead Capture | lead | true | — |
| 24 | Lead Nurture | lead | true | Resend, Claude |
| 25 | DNS Configure | platform | false | Cloudflare |
| 26 | Billing Subscribe | billing | false | Stripe/Razorpay |
| 27 | User Invite | platform | false | Resend |
| 28 | Notification Send | platform | true | Resend |
| 29 | Error Handle | platform | true | — |
| 30 | Fallback Execute | platform | true | — |
Skill Dependency Graph#
01-practice-setup
└── 03-gbp-connect
└── 04-gbp-post-create
└── 05-gbp-post-schedule
└── 06-gbp-review-reply
└── 07-gbp-insights-fetch
└── 19-report-generate
└── 20-report-email
└── 08-social-connect
└── 09-social-post-create
└── 10-social-post-schedule
└── 13-site-generate
└── 14-site-update
└── 15-site-evolve
└── 16-seo-audit
└── 17-keyword-track
└── 18-competitor-analyze
└── 21-content-generate
└── 22-content-approve
└── 04-gbp-post-create
└── 09-social-post-create
└── 14-site-update
└── 23-lead-capture
└── 24-lead-nurture
End of Refined Technical Specification