Work / Case Study
Enterprise LLMOps Platform
Multi-Model Routing & AI Cost Governance
Production LLM infrastructure that served enterprise workloads at 200K+ requests/day under SLA commitments
作品集 / 案例
企業級 LLMOps 平台
多模型路由與 AI 成本治理
承載企業工作負載、200K+ requests/day 並維持 SLA 的生產級 LLM 基礎設施
Overview總覽
Product manager for Vext's LLMOps SaaS platform: prompt lifecycle management, RAG workflows, multi-model routing, and agent orchestration. Worked with a six-person development team, serving enterprise clients in government, education, and software.
The routing and governance layer spanned OpenAI, Gemini, Azure AI, AWS, and OpenRouter. It balanced reliability, latency, and inference cost for each use case, and it carried 200K+ requests a day under SLA.
在 Vext Technologies 主導生產級 LLMOps SaaS 平台的產品策略與端到端交付,與 6 人開發團隊協作,涵蓋 prompt 生命週期管理、RAG 工作流、多模型路由與 agent 編排,服務政府、教育、軟體業的企業客戶。
平台的路由與治理層橫跨 OpenAI、Gemini、Azure AI、AWS、OpenRouter 五家供應商,依用例平衡可靠性、延遲與推論成本,承載 200K+ requests/day 並維持既定 SLA 承諾。
Challenge & Solution挑戰與解法
Challenge
At production scale, inference costs swing between providers and models. High traffic trips provider rate limits. Depending on one provider means their outage becomes your client's outage. Clients under SLA expect predictable cost and availability regardless.
Solution
A routing and governance layer sits between client workloads and the providers. Traffic is allocated by inference cost. Tasks go to models that fit them. When a provider degrades or rate-limits, fallbacks reroute around it. Governance controls (token optimization, rate limit management, vector search tuning) keep production stable as load grows.
挑戰
企業 LLM 工作負載進入生產規模後,推論成本隨供應商與模型波動、高流量時段會觸發供應商 rate limit、依賴單一供應商時上游故障會直接影響客戶服務。客戶在 SLA 承諾下,要求可預測的成本與可用性。
解法
在客戶工作負載與模型供應商之間建立平台級的路由與治理層:流量依推論成本分配、任務依模型能力分派,fallback 機制在供應商劣化或限流時自動改道。治理控制涵蓋 token 優化、rate limit 管理與向量搜尋調校,讓生產行為隨負載成長保持穩定。
System Architecture系統架構
Illustrative concept diagram, recreated for portfolio purposes; contains no proprietary information.
概念示意圖,為作品集重繪,不含專有資訊。
Key Capabilities核心能力
- Multi-model routingOne layer across OpenAI, Gemini, Azure AI, AWS, and OpenRouter; no workload depends on a single provider
- Cost-based allocationRequests go to providers by inference cost per use case
- Capability-based routingEach task goes to a model suited for it
- FallbacksAutomatic rerouting when a provider degrades or rate-limits, so SLAs hold
- RAG and prompt lifecycleEnterprise retrieval pipelines on LangChain, Pinecone, and pgvector
- Cost governanceToken optimization, rate limit management, auto-scaling constraints, and vector search tuning, built into the platform
- 多模型路由單一路由層橫跨 OpenAI、Gemini、Azure AI、AWS、OpenRouter,客戶工作負載不依賴單一供應商
- 成本導向流量分配請求依各用例的推論成本分流到對應供應商
- 能力導向任務分派依照任務選擇適合的模型
- Fallback 機制供應商劣化或限流時自動改道,維持 SLA 承諾
- RAG 工作流與 prompt 生命週期管理企業檢索管線建於 LangChain、Pinecone、pgvector
- AI 成本治理token 優化、rate limit 管理、auto-scaling 限制、向量搜尋調校,作為平台級控制
Technology Stack技術棧
My Contribution我的貢獻
- Product strategy and end-to-end delivery of the platform, with a six-person development team
- Designed the routing strategy: cost-based allocation, capability-based routing, fallbacks across five providers
- Delivered enterprise LLM workflows for government, education, and software clients at 200K+ requests a day under SLA
- Ran cost governance and production stability: token optimization, rate limits, auto-scaling constraints, vector search tuning
- Primary technical contact for enterprise clients; turned business use cases into deployable workflows and validated them through structured testing and real-world use
- 主導 LLMOps 平台的產品策略與端到端交付:prompt 生命週期管理、RAG 工作流、多模型路由、agent 編排
- 設計多模型路由策略:成本導向流量分配、能力導向任務分派,與跨五家供應商的 fallback 機制
- 為政府、教育、軟體業的企業客戶交付 LLM 工作流,承載 200K+ requests/day 並維持 SLA
- 推動 AI 成本治理與生產穩定:token 優化、rate limit 管理、auto-scaling 限制、向量搜尋調校
- 擔任企業客戶的主要技術窗口:將商業用例轉譯為可部署的 AI 工作流,經結構化測試與實際驗證迭代