Tumeryk — AI Safety Infrastructure for Production LLMs
At Tumeryk I build the systems that decide whether an LLM response is safe enough to ship. Backend infrastructure, evaluation pipelines, and observability tooling for production AI deployments.
My Contribution
- Designed and implemented the guardrail testing agent end-to-end
- Architected the base policy enforcement layer — the foundation all other safety rails depend on
- Built the automated LLM evaluation workflow pipeline
- Implemented the PII detection and sensitive-data validation rail
- Designed and shipped the observability dashboard — React frontend + backend metrics API
Feature Deep-Dives
Guardrail Testing Agent
Automatically evaluates LLM policy compliance across risk categories. Input: policy definitions + model endpoint + prompt set. Output: structured compliance report + numeric trust score per response. Replaces manual safety review cycles.
Base Policy Enforcement Layer
Middleware-style layer that every custom policy routes through before execution. Validates against organizational safety constraints, defined risk thresholds, and compliance rules. The core primitive — every downstream guardrail builds on it.
Automated LLM Evaluation Workflow
Takes adversarial prompts grouped by risk category, fires them at the target model or private endpoint, collects responses, passes them to a judge model configured for safety/accuracy/compliance scoring. Results aggregated into per-category reports with pass/fail thresholds.
PII / Sensitive-Data Validation Rail
Real-time scanning of LLM inputs and outputs for PII patterns (names, emails, identifiers, financial data). Intercepts before data reaches the model or the end user. Configurable sensitivity levels per deployment.
Observability Dashboard
React frontend showing live guardrail metrics, evaluation run outcomes, system logs, and policy evaluation traces. Backend metrics API aggregates across guardrail runs and surfaces trend data.