Skip to main content

03 - Backend Architecture Design

This document details the underlying design of the FastAPI backend engine, multi-tier databases, multi-model routing integration, and defensive gateway for the Taimoe Enterprise AI Gateway.

1. Technology Stack​

The backend architecture is built entirely on the Python ecosystem, maintaining strict type safety and high concurrency performance:

  • Gateway Engine: FastAPI (Python 3.10+). Delivers outstanding asynchronous performance (async/await), leveraging Pydantic for strict input/output schema validation, and natively generating OpenAPI specifications. Ideal for handling heavy asynchronous LLM API proxy requests.
  • Main Database (RDBMS): PostgreSQL. Paired with SQLModel (a modern ORM based on SQLAlchemy and Pydantic) to manage relational entities (multi-tenant structures, IAM roles, Project attributes, Policy settings, etc.). Uses Alembic for structured schema migrations.
  • Telemetry & Observability Store: ClickHouse. Used to store high-throughput API traces and telemetry metrics (latency, token usage, cost) for sub-second analytical aggregations.
  • Caching & Rate Limiting: Redis. Manages session state, RBAC cache, Virtual Key validation, and distributed rate limiting (Token Bucket algorithm).
  • Object Storage: Google Cloud Storage (GCS) / AWS S3. Stores large conversation JSON payloads and historical audit logs exceeding ClickHouse storage retention policies.
  • Background Tasks: APScheduler / Celery. Executes periodic cache syncs, expired log archiving, and billing data aggregation.

2. Database Schema & Entities​

The database utilizes a built-in multi-tenant architecture where all core entities have explicit isolation boundaries:

  • Organizations: Top-level tenant container representing an enterprise entity or conglomerate, providing full data and billing isolation.
  • Projects: Project workspaces within an organization. API keys, model quotas, and access policies are scoped and rate-limited at this level.
  • Users: Maps to external OIDC / SAML SSO Subject IDs. Users are assigned roles.
  • Roles & RoleBindings: Role-Based Access Control (RBAC) supporting granular permission inheritance.
  • ApiKeys (VirtualKeys): Stores hashed API access credentials used by external clients and application runtimes to invoke the Gateway.
  • ProviderCredentials: Platform-managed model provider credentials (e.g., OpenAI API Keys or GCP Service Account JSON keys).
  • PolicyConfigs: Stores security policy rules bound to projects or Virtual Keys (Quota Guard, PII Redaction, Model Armor, etc.).

3. Multiplexing Gateway Engine​

As the single conduit for all LLM traffic, the Taimoe platform delivers efficient model routing and protection:

3.1 Northbound API (Northbound API Governance)​

When enterprise applications invoke LLMs, they directly call:

POST /v1/organizations/{org_slug}/projects/{project_slug}/models/{model_alias}:predict

Or OpenAI-compatible endpoints:

POST /v1/chat/completions
  • Interception & Verification Pipeline:
    1. Validate Virtual Key (Redis)
    2. Verify Project Quotas (Redis Rate Limiter)
    3. Resolve Alias (model_alias -> Real Provider & Model Version)
    4. Apply Pre-call Policies (PII / Guardrails)
  • Payload Format: Natively compatible with OpenAI API or Google Vertex AI API specifications.

3.2 Southbound Adapters (Southbound Model Adapters)​

Based on the resolved Provider, the Gateway dynamically loads the corresponding Adapter:

  • Vertex AI Adapter: Initializes Google Cloud connection and securely proxies requests to models like gemini-1.5-pro.
  • OpenAI Adapter: Attaches real OpenAI credentials and proxies requests to gpt-4o.
  • Local Model Adapter: Proxies requests to enterprise self-hosted open-source models (vLLM, Ollama, etc.).
sequenceDiagram
    autonumber
    actor Client as Client App / Agent
    participant GW as FastAPI Gateway
    participant Redis as Redis Cache
    participant DB as PostgreSQL
    participant Provider as LLM Provider (Vertex/OpenAI)
    participant Worker as Telemetry Worker (ClickHouse/GCS)

    Client->>GW: POST /predict (Virtual Key + Alias)
    GW->>Redis: 1. Validate Virtual Key & Quotas
    Redis-->>GW: Key Valid & Quota OK
    GW->>DB: 2. Resolve Model Alias & Fetch Provider Credentials
    DB-->>GW: Real Provider Config & Decrypted Credentials
    GW->>GW: 3. Apply Pre-call Policies (PII Redaction & Guardrails)
    GW->>Provider: 4. Dispatch Request via Southbound Adapter
    Provider-->>GW: Stream / Full Response Payload
    GW->>GW: 5. Execute Post-call Interceptors (Token Count & Audit)
    GW-->>Client: Return Sanitized Response
    GW-)Worker: Async Log Ingestion (Spans to ClickHouse, Payloads to GCS)

4. Defensive Interceptors & Policy Pipeline​

Every API call passing through the Gateway traverses a strict pipeline:

  • Pre-call Interceptors: Verifies whether quotas are exhausted before requests are dispatched to Providers. Supports PII redaction by replacing sensitive strings (SSNs, credit cards) with [REDACTED].
  • Post-call Interceptors: Inspects Provider response payloads to prevent malicious scripts and logs accurate Prompt / Completion token usage.

5. Telemetry & Audit Logs​

To prevent blocking real-time LLM API requests, the backend processes telemetry via asynchronous worker queues:

  • Metrics Ingestion: API metrics such as latency, tokens, and cost are asynchronously batch-written to ClickHouse.
  • Payload Archive: Complete Request / Response payloads are compressed and uploaded to Object Storage (GCS) for forensic audits.
  • Admin Audit Logging (app/services/audit.py): All modifications to Virtual Keys, Policies, or RBAC roles are automatically recorded in immutable relational database audit logs.