Skip to main content

Observability & Auditing Architecture

As a core pillar of the Taimoe Enterprise AI Gateway, we deliver comprehensive traffic observability and immutable audit logging to help enterprises satisfy stringent regulatory standards.

1. Design Goals​

One of the biggest pain points for enterprise LLM adoption is the "black box" nature of API calls. The Taimoe AI Gateway addresses this by centrally funneling all LLM traffic to achieve:

  • Real-Time Operational Monitoring (Metrics): Enables IT teams to track outbound LLM traffic distribution, API error rates, P50/P95 latencies, and token consumption costs across projects.
  • Immutable Audit Logging: Records every prompt input and LLM output in full, satisfying security and compliance audit requirements.

To support high-throughput log volume and real-time analytical queries, the platform employs a cloud-native ClickHouse + GCS + Fluent Bit architecture.

2. Data Sources & Storage Layer Design​

To balance real-time aggregation query performance with large text storage costs, we separate index metadata from raw payload storage:

Storage TierPurpose & Stored Content
ClickHouseAnalytics & Indexing (Metadata): Stores structured metrics including trace_id, project_id, api_key_id, cost, latency, and tokens.
Snippet Mechanism: Stores the initial 500 characters of the User Prompt for rapid search.
GCS / AWS S3Full Payloads: Stores complete JSON payloads (including large prompts) for every request. ClickHouse stores the corresponding gcs_url.

Log collection and persistence are managed by Kubernetes Fluent Bit (DaemonSet), streaming from Gateway stdout asynchronously to ClickHouse and Object Storage without impacting request processing latency.

flowchart LR
    subgraph DataPlane["API Traffic"]
        GW["FastAPI Gateway Engine"]
    end

    subgraph Collection["Log Processing Layer"]
        Stdout["Container stdout / JSON Logs"]
        FluentBit["Fluent Bit DaemonSet"]
    end

    subgraph StorageTier["Hybrid Storage Tier"]
        CH[("ClickHouse<br/>(Structured Metrics & Traces)")]
        GCS[("GCS / S3 Object Storage<br/>(Full Raw JSON Payloads)")]
        PG[("PostgreSQL<br/>(Admin Audit Logs)")]
    end

    subgraph Consumers["Visualization & Analysis"]
        Grafana["Grafana Dashboards"]
        Console["Taimoe Console Audit Viewer"]
    end

    GW --> Stdout
    Stdout --> FluentBit
    FluentBit -->|Metadata & Spans| CH
    FluentBit -->|Compressed Payload Archive| GCS
    GW -->|Admin Config Events| PG

    CH --> Grafana
    CH --> Console
    GCS --> Console
    PG --> Console

3. Full-Stack Observability (Metrics & Tracing)​

3.1 Platform Dashboards (Grafana Integration)​

Taimoe supports exporting telemetry to mainstream observability platforms such as Grafana, querying ClickHouse directly for sub-second dashboard reporting.

Built-in Analytical Dimensions:

  • Traffic & Performance Trends: QPS, Latency, Error Rate curves.
  • Cost Analysis: Token consumption and spending split by Model (GPT-4 vs Gemini) or by Project (Project A vs Project B).
  • Top-N Rankings: Top 10 Virtual Keys with the highest cost consumption.

3.2 Tracing Mechanism​

For complex asynchronous or multi-step agent invocations, the Taimoe Gateway parses the X-Taimoe-Trace-Id header. If internal enterprise services maintain distributed tracing spans, they can pass trace IDs into the Gateway to seamlessly correlate Taimoe logs with enterprise APM tools (e.g., Datadog, Jaeger).

4. Audit Logging​

Every transaction passing through the Taimoe Gateway generates an immutable historical audit record, serving forensic investigations and Data Loss Prevention (DLP).

4.1 Audit Record Schema​

A complete API call audit log entry includes:

  • Timestamp & Identity: Timestamp, Project ID, Virtual Key Alias
  • Request Details: Model Name, Client IP, Target Provider
  • Response Data: Status Code, Latency, Prompt Tokens, Completion Tokens, Cost (USD)
  • Security Policy Flags: Rate Limit triggers, PII masking flags
  • Full Payload Link: Object storage payload URL

4.2 Console Query Interface​

Administrators can use multi-dimensional filters in the Taimoe Console:

  • Filter by Time Range + Project Name + Status Code.
  • Expand individual entries to dynamically fetch complete Prompt and Completion JSON payloads from Object Storage for security reviews.

5. Architectural Roadmap​

  1. Data Lifecycle Management: Automatically archive audit logs older than 90 days from standard GCS buckets to Coldline/Archive storage classes.
  2. PII Masking & Sanitization: When Model Armor and PII interception policies are enabled, Audit Logs can persist sanitized prompts to prevent sensitive PII from reaching log storage.
  3. Proactive Alerting: Automatically trigger Slack or email notifications to IT teams when error rates (5xx) spike or spending anomalies occur.