Case 01 · Enterprise AI · Product development
Enterprise AI application platform
Every model call metered and priced, and a failed task that re-runs only the part that failed. The platform is the team's work; this is the part I did.
All workSummary
The case in a minute
Problem
Usage checks, quota and failure handling were scattered across services. Quota was counted in tokens although prices differ by model and token type, and failed background tasks retried without limit, re-running finished work and its cost.
My part
As part of the product team: usage metering and organisation quotas at the model-call gateway, model pricing, and the reliability of background AI tasks.
Stage
Merged and running in the development environment. Production receives it with later versioned releases.
Key changes
- Metering, quota checks and error handling moved into the gateway, with a local usage record so a slow metering service never holds a request.
- Quota counted in money: input, cached and output tokens each carry a cost and a sale price per model, checked before deployment configuration is generated.
- Long tasks run in saved segments. A retry re-runs only the failed segment, within a retry limit and with growing waits.
- Duplicate submissions blocked, queued tasks cancelled with their parent, read-only roles enforced in the front end, and the Windows development environment supported.
Result
≈80%
fewer tokens on the same job
Measured on one job in the test environment. Not a platform-wide, production or cost figure.
Verified
Each change was reproduced locally, covered by a test, fixed, and then checked against logs from the Kubernetes environment.
How it worksLong jobs save their progress segment by segment; a retry re-runs only the segment that failed.
- Segment 1: Done, saved
- Segment 2: Done, saved
- Segment 3: Failed
- Segment 4: Not yet run
- Segment 5: Not yet run
Segments 1 and 2 are not re-run
The retry re-runs segment 3 only
A longer wait before each retry.Stops at the retry limit.
Team, stage and limits
- Development environment
- Merged and running
- Production
- With later versioned releases
- Charging at the sale price
- Switched on in production only after real cloud billing costs are reconciled
- Team
- The platform is built by the team. Data isolation between organisations is part of the product, not my work.
- The number
- The 80% describes one job in the test environment, and nothing else.
01Background
The company develops an enterprise AI application platform: teams use large language models to process data and tasks, with data kept apart between organisations and AI usage and cost managed. I joined the team in October 2026.
02Key decisions
Four decisions, each with the problem, what was decided, and what it was traded against.
-
Decision 01
When metering slows down, users do not wait
- Problem
- Quota checks, usage calculation and failure handling were scattered across services. If every call had to wait for the metering service, a slow one would hold every user up.
- Decision
- All three moved into the model-call gateway, which meters each organisation in real time. The gateway keeps a local usage record, so a slow metering service does not hold requests up.
- Trade-off
- The gateway now holds state it has to look after. In return, the rules sit at the one point every request passes, and a slow dependency does not become a user's wait.
Failures are told apart there too: quota exhausted, model busy and connection failed each return their own clear error response. Each of the three is handled differently; folded into one error, they leave the user guessing.
What it showsI put rules where every request has to pass, and decide in advance who waits when a dependency slows down.
Schematic, not a production screen Task
Model-call gateway
- Quota check
- Real-time metering per organisation
- Local usage record
Model
Metering service
When it is slow, requests do not wait
Errors returned, kept apart
- Quota exhausted
- Model busy
- Connection failed
-
Decision 02
Out of quota: block new tasks, let running ones finish
- Problem
- When the monthly quota runs out, stopping everything is the simplest rule. It also throws away tasks halfway through, along with the usage already spent on them.
- Decision
- Once the quota is exhausted, new tasks cannot start; tasks already running can finish within a limited grace.
- Trade-off
- Usage can run slightly past the quota, which is why the grace is limited. A small, bounded overrun was preferred to discarding work already paid for.
What it showsI treat stopping new spending and protecting work already under way as two separate problems.
Schematic, not a production screen Quota exhausted
Running task: Finishes within the grace
New task: Cannot start
Limited grace
Time
-
Decision 03
A retry redoes only what failed
- Problem
- Background AI tasks that failed retried forever, re-ran work that was already done, and kept generating cost.
- Decision
- Long tasks run in segments, with progress saved after each one. A retry re-runs only the failed segment, up to a retry limit, and waits longer before each attempt than before the last.
- Trade-off
- Segments and saved progress add complexity. In return, a failure no longer pays again for finished work, and it cannot repeat without end.
The limit and the growing waits leave a passing failure room to recover, while one that persists stops instead of piling up cost.
ResultIn the test environment, on the same job, token usage fell by about 80% compared with before the fix. That is one test job's measurement, not the platform's cost.
What it showsI count what a failure costs afterwards as part of the design, not only whether the task succeeds.
-
Decision 04
A pricing rule, checked before deployment
- Problem
- Quota was counted in tokens. But prices differ by model and by type of token, so a token count does not say how much was spent.
- Decision
- Quota is now counted in money: for each model, input, cached and output tokens each have a cost price and a sale price. Before deployment configuration is generated, a check requires the sale price to be above cost; if it is not, no configuration is generated.
- Trade-off
- Every change to the price list has one more gate to pass. In exchange, a price below cost is stopped before any configuration exists, not found after billing.
Cost is recorded separately, so margins can be analysed later.
What it showsI can turn a commercial rule into a check the system enforces on its own.
Schematic, not a production screen For each modelQuota counted in money
- Input tokens
- Cost price Sale price
- Cached tokens
- Cost price Sale price
- Output tokens
- Cost price Sale price
Before deployment configuration is generated
Is every sale price above cost?
- Yes Generate the deployment configuration
- No Stop: no configuration is generated
Cost recorded separately, for later margin analysis
03How it was verified
Each change was reproduced locally first, covered by a test, fixed, and then checked against logs from the Kubernetes environment. What can be verified is mostly behaviour:
- Once the quota is exhausted, new tasks do not start, and running tasks finish within the grace.
- Quota exhausted, model busy and connection failed each return a different error response.
- If the sale price is not above cost, no deployment configuration is generated.
- A retry re-runs only the failed segment, and stops at the retry limit.
- A duplicate submission is blocked; stopping or deleting a task also cancels its queued tasks; read-only roles are restricted in the front end as well.
- The relevant inter-process communication moved from a Unix socket to a named pipe, so the Windows development environment is compatible.