Our AI stack had no front door, so we built one

At LayerX, we run our own intelligence. Five GPU machines in our own stack serve the models behind our internal tools, our experiments and the products we ship.

For a long time, that was the whole setup. Five machines. Five URLs. No front door.

Anyone who had the address could use the GPU. Nobody had to log in or hold a key, and there was no limit. When a machine slowed down, the first question was always "who's on it?" Most of the time, nobody could answer.

We knew better. We did it anyway, because shipping came first.

How we got here

The stack didn't start messy. It got messy one machine at a time.

A new project needed a model, so we brought up a server. Another project needed a different model, so we brought up another. Each one was set up by whoever needed it, for whatever they were building that week. Every client hard-coded the URL of the box it talked to. Moving a model meant hunting down every place that pointed to the old one.

Then there were the models our GPUs couldn't serve. When a team needed a bigger model or more capacity, they got their own OpenAI or OpenRouter key. Those keys lived in .env files, in CI secrets and in someone's terminal history. Spend showed up on invoices nobody was watching.

Machines, people and permissions lived in three separate worlds, and nothing connected them.

The questions we couldn't answer

Once we looked at it honestly, it came down to five questions we couldn't answer:

• Who is allowed in? Inference servers ship with weak or no key management. The URL was the password.
• How much can they use? We had no budgets and no rate limits, so one runaway script could drain a machine for everyone.
• Who used what? Nothing recorded which person ran which model, when, or how many tokens.
• What happens when a machine goes down? Every client pointed straight at it just failed.
• What is it costing us? Third-party spend was spread across keys and invoices, with no single view.

None of these are hard problems on their own. Together, they meant we were running our most expensive infrastructure on trust and guesswork.

It turned out we weren't the only ones

We started building a fix for ourselves. Then we talked to our closest partners and heard the same story, almost word for word.

They had GPUs, or vendor accounts, or both. They had teams that wanted access. And they wanted three things:

1. One central place to manage who can use intelligence.
2. Real control over what it costs.
3. Routing that sends each request to the right place, using their own hardware first and paying a vendor only when they have to.

That's the moment an internal tool turns into a product. We started building it for ourselves. Then we started building it with them.

It's also a concrete piece of something we've written about before: the layer between AI and your data. Governance starts with knowing who's at the door.

What AI Gateway is

AI Gateway is the single front door to your intelligence: your own GPUs and third-party models, behind one API.

Here's what that means in practice.

It's a drop-in. The gateway speaks the OpenAI API. Point your existing SDK at it, swap in an AI Gateway key, and your code keeps working:

1const client = new OpenAI({
2  baseURL: "https://<your-gateway>/v1",
3  apiKey: process.env.AIG_KEY, // aig_...
4});

Chat completions (streaming included), embeddings and model listing all pass straight through.

Every person gets their own keys. Keys are shown once, stored hashed, and can be revoked or set to expire. Each user has a list of models they're allowed to call, and their keys inherit it.

Every person gets a budget. Monthly and daily token quotas are set per user and shared across all their keys. Each key also has a rate limit. Users get an email when they reach 80% and 100% of their quota, so nobody hits the wall by surprise.

Routing is smart about where requests go. Each model maps to one or more providers, arranged in priority tiers. Our own GPUs sit in the first tier. Within a tier, the gateway sends each request to the least busy machine. If that whole tier is down, saturated or rate-limited, it falls back to the next one, for example OpenRouter. Clients never notice. They just get an answer.

Every request has a price. Each provider mapping can carry token prices, so every request is recorded with its USD cost. You can see spend by user, key, model and provider in one place.

Everyone has a portal. Admins invite users, set quotas and model access, register providers and models, and watch usage across the whole fleet. Users sign in with an emailed code (no passwords), create their own keys, and see their own usage and remaining quota.

It serves decision models too. Not every question needs a paragraph back. Decision models like Jev or Clef answer typed questions (yes or no, which option, how urgent) with a probability for every answer. They get their own endpoint, /v1/systemone, and the same keys, quotas, routing and usage tracking as everything else.

The life of one request

Every call through the gateway takes the same path:

auth → rate limit → quota check → model routing → provider selection → stream → usage recorded

The key is checked, the limits are enforced, the gateway picks the best provider for that model right now, the response streams back, and the tokens are counted. All of that happens before the client knows anything happened.

What it doesn't do (on purpose)

We'd rather be clear about this up front.

• We never log or store prompts or responses. Not in logs, not in the database, not in queues, not in caches. We record that a request happened, who made it, which model and provider served it, and how many tokens it used. What was said stays between the user and the model.
• It doesn't lock you in. Your code talks to the standard OpenAI API, not to anything proprietary of ours. Behind the gateway, providers are just entries in a list: add a vendor, drop one, or move a model from OpenAI to your own GPUs, and your clients never change a line. You keep your own vendor accounts, and if you ever stop using the gateway, you point your SDK back at the vendor and carry on.
• It doesn't bill or invoice anyone. Costs are recorded for visibility, and quotas are counted in tokens.
• It doesn't host or schedule your inference servers. You run the machines; the gateway decides who gets to use them.
• It doesn't translate between vendor API formats. If you need Anthropic or Gemini models, route them through OpenRouter.

Where we are now

All of our machines now sit behind AI Gateway. Every internal tool and every engineer goes through the same front door. Our close partners are running it on their own stacks.

Now we're opening it up to a small group of teams who have the same problem we had: GPUs or vendor accounts, people who need access, and no clear picture of who's using what.

There's more on the way. Tighter network controls are next on our list, and our partners keep adding to it.

Intelligence needs a front door.

So here's the question: can you tell me, right now, who used your GPUs yesterday and what it cost? If that takes more than a minute, take a look at AI Gateway. We'd like to hear how your team is handling it.