← All experience

SIA Partners · Internal platform and R&D

Turning internal research into reusable platform capability

Inference benchmark design, internal platform engineering, mentoring

Continuous, alongside client engagements

This work ran alongside client engagements and reduced the cost of learning new infrastructure repeatedly. It produced a reproducible inference benchmark, controlled access to shared GPUs, and internal experiments that later supported client deployments.

A benchmark that records the complete configuration

Model leaderboards do not answer when latency degrades under concurrency, what a specific GPU delivers, or whether a faster serving configuration preserves output quality. I led the design of a Go command-line tool that validates a configuration, deploys an open-source model, runs the benchmark, and records latency percentiles, throughput, quality, runtime, and parameters.

MLflow stores each run so results remain comparable. Metabase exposes the results to people who do not use MLflow directly. Most implementation work targets vLLM, with lighter support for Ollama and SGLang. AWS, NVIDIA, and Scaleway teams participated in the work.

Shared GPUs with attributable access

The benchmark required shared GPU capacity without a shared access key. I placed the internal GPU machines behind Teleport, giving users short-lived, identity-based, audited access. Resource availability was then managed for the internal lab so experiments could be scheduled rather than competing without coordination.

Research that crossed into client delivery

Internal AWS AgentCore work contributed to two production client deployments, for Cegos and Icade, and to SIA obtaining its AWS Agentic certification. A separate DeepStream and Triton proof of concept validated Jetson hardware before the later Nhood edge-vision deployment.

Making the capability transferable

Through the Open Squad initiative, I mentored junior developers working on research prototypes. The focus was decision quality: identifying the cost of a design choice and documenting the evidence needed to revisit it.

• Go
• vLLM
• Ollama
• SGLang
• MLflow
• Metabase
• Terraform
• Teleport
• AWS AgentCore
• Kubernetes
• GitLab CI