Nemocore puts NVIDIA Switchyard on Fastly's global edge, accelerated by BlueField DPUs in the fleet. It classifies, triages, caches, secures, and routes AI traffic before a single token is spent at the model, without keeping what it sees. Everything around inference runs at the edge, privately. The inference itself does not.
NVIDIA Switchyard is an open-source router that sends LLM traffic across models and providers while keeping native OpenAI and Anthropic API compatibility. Nemocore runs that routing logic inside Fastly's edge platform, so the decision happens next to the user instead of at a central gateway.
Any application that already speaks the OpenAI or Anthropic API. No SDK change. Point the base URL at Nemocore.
TLS termination, request screening, bot fingerprinting, rate and token quotas, and the intent and context caches. Same network, same purge, same logs as the rest of your Fastly traffic.
Classifies each request, picks a model and provider by capability, cost, latency, and data sensitivity, and can switch models mid-task. Runs on the edge fleet as a software router.
Line-rate packet and payload inspection, crypto and compression offload, and an isolated security domain on the data path. The hardware that makes inline AI security affordable at scale.
Frontier-model vendors, self-hosted Nemotron, or your own endpoints. GPUs live here. Nemocore decides which origin gets the request, and whether it needs to go at all.
No GPU-driven inference at the edge. Fastly's fleet does not carry GPUs, and Nemocore doesn't need them. Classification, triage, caching, policy, and security are CPU- and DPU-class workloads. The edge's job is to make sure only the requests that truly need a GPU ever reach one, and that they carry as few tokens as possible.
AI traffic is large, long-lived, and sensitive. Inspecting every prompt and response inline on host CPUs gets expensive fast. A DPU moves that work off the host onto dedicated silicon on the same card as the network port.
Payload inspection, injection and exfiltration screening, and pattern matching run on the DPU's Arm cores and accelerators, so deep inspection doesn't steal cycles from request serving.
Security policy executes on the DPU, outside the host OS. A compromised workload on the host cannot quietly switch off the controls that watch it.
Long-lived streaming connections (SSE and WebSocket) are expensive on host CPUs. Offloading TLS and transport work frees compute for caching, classification, and routing.
Cheap decisions such as block, rate-limit, cache-hit, and tier tagging can be made on the card, ahead of the host stack. Rejected traffic never costs a request slot, let alone a token.
Per-request signals (token counts, tier, route, cache outcome, bot score) are produced on the data path and streamed through Fastly's real-time logging. Signals describe the request, never its content or sender.
Policy ships as software to every DPU-equipped point of presence, the same way config ships to the rest of the network. No per-site appliances.
Triage and routing mean the edge has to understand a request. Nemocore is designed so that understanding is momentary: content is processed in flight, never written to logs, and never joined to who sent it.
Requests arrive through a privacy relay, using the same standards behind Private Relay: MASQUE and Oblivious HTTP. Nemocore can classify, screen, and route the prompt, but it never sees the client's IP address or account identity.
For the most sensitive workloads, the edge acts as a double-blind intermediary, as in Private AI Compute. It sees identity metadata needed to meter and secure the connection, and nothing about what is inside the encrypted request.
Prompts and responses are inspected in memory on the request path. Nothing from the content of a request is written to logs, traces, or analytics.
What gets recorded is operational: token counts, sensitivity tier, route, cache outcome, and timing. No prompt text, no response text, no PII, and no client identifiers.
Intent and context caches are opt-in, tenant-isolated, encrypted, and short-lived. Restricted and confidential tiers default to off. Anything cached can be purged globally in seconds, and is never linked to a person.
Detection exists to strip or mask personal data before it travels to a model or a cache, so what leaves the edge carries less personal information than what arrived.
Inspection and policy run on BlueField DPUs, outside the host OS. The goal is that privacy controls can't be quietly switched off by a workload running next to them, and that the setup can be attested.
Nemocore does not use customer prompts or responses to train or tune models, and classifiers run on content in flight, not on stored corpora.
Fastly has spent years building infrastructure for privacy-first companies, and helping write the standards they rely on. Nemocore extends that record to AI traffic.
Fastly's platform is used as part of Private AI Compute's privacy-preserving request routing, in a double-blind model where no single party has access to both identifying metadata and request content. The goal is privacy enforced by design, so misuse is structurally hard rather than merely against policy.
fastly.com/blog →Fastly runs a second-hop relay in Private Relay, which can see the destination site but not the user's original IP address, so no single party can link identity to browsing.
fastly.com/blog →Fastly contributes to the privacy protocols these systems depend on, including MASQUE, Oblivious HTTP, and Oblivious DNS over HTTPS, built on QUIC and HTTP/3.
Fastly's acceptable use policy prohibits harassing and threatening content and material that violates third-party privacy rights. Fastly will not be the infrastructure for campaigns that expose people.
fastly.com/acceptable-use →Nine jobs the edge does well, applied to prompts, responses, agents, and the bots that crawl for training data.
Prompt-injection and jailbreak screening, payload smuggling, secrets and PII leakage on the way out, and abuse of model endpoints. Built on Next-Gen WAF and shared threat intelligence.
Tell good agents from scrapers, then charge for access. Allow, meter, challenge, or require payment per crawl, per page, or per token. Turn training-data traffic into revenue instead of cost.
Send fewer tokens for the same answer: strip duplicated context, trim boilerplate, cap runaway agent loops, and enforce per-key and per-team token budgets.
Serve repeat answers from the edge and send only what changed. Cache hits cost zero model tokens and return in edge latency.
Decide which class of model a request needs: small and cheap, mid-tier, or frontier. Match by task difficulty, modality, context length, and required tools.
Identify what the caller is trying to do (lookup, summarize, code, act) and route accordingly. Simple intents get templates or cached answers; risky intents get extra scrutiny.
Cache by meaning, not exact text. Semantically equivalent questions resolve to one answer, opt-in per tenant, with short TTLs and scopes that respect who is allowed to see it. Never tied to a user.
Agents resend the same repo, policy, and record every turn. Hold large context once in a tenant-owned, encrypted, short-lived cache, so repeated material doesn't cross the network again. Keeping stable content first also helps providers' own prompt caches hit.
Restricted data stays on approved origins and regions. Confidential data goes only to vendors under zero-retention terms. Fail over across providers without the client noticing.
Click a stage to see what happens before a request ever reaches a GPU.
The request is checked for bot identity, abuse, injection patterns, and quota before anything else. Obvious bad traffic is dropped on the DPU. Known good agents are tagged and allowed. Crawlers that should pay are challenged or metered.
A small CPU-class classifier tags the request with an intent, a difficulty estimate, and a data-sensitivity tier (public, internal, confidential, restricted). This tag drives every later decision and is logged with the request.
Exact match first, then semantic match on intent, scoped to what this caller may see. A hit returns immediately from the edge and costs no model tokens. A near-hit supplies stored context so only the new part is sent.
Switchyard picks the cheapest model and provider that satisfies capability, latency, region, and policy. Restricted data is limited to approved origins. If a provider is slow or down, traffic moves without the client changing anything.
Before forwarding, the edge removes duplicated context and arranges stable content first, so provider prompt caches hit more often. Where context selection is enabled, it also trims text the task doesn't need, with answer quality measured alongside the savings.
Tokens in and out, cache outcome, route, and tier are recorded per request and streamed in real time. This metadata powers chargeback, bot billing, and audit questions like "which tier of traffic went where?" Prompt and response content is never logged.
Customers already put Fastly in front of their applications. That makes the edge a controlled place to introduce AI policy and context handling, with explicit, versioned rules instead of changes to every application.
An administrator approves a transformation. It gets a version number.
Every cached artifact the rule touches is tagged with the rule version and tenant. Never with a user.
The new version is previewed and evaluated before it goes live.
Only the entries built under the old version are invalidated, in seconds, not the whole cache.
Compressing a request shrinks what crosses the network. The model still sees the same text. Nemocore keeps the two apart and reports each on its own.
Transfer savings are measured in bytes, including dictionary delivery and refresh, and are not claimed as token or GPU savings. Dictionaries are tenant-scoped, since sharing compression state across users can leak information. Streaming latency, meaning time to first token, is measured alongside size. A benchmark on realistic multi-turn agent sessions is planned.
Proposal. This is the intended deployment model for Nemocore, building on Fastly's existing rule-based edge configuration and surrogate-key purging. It is not a currently available feature. Any rule that modifies content would be explicit, administrator-controlled, and logged by version, and would respect the customer's consent and privacy settings.
The same origin-selection logic Fastly runs for HTTP, applied to inference. Cost and latency optimize within the limits that policy sets.
Approved origins only, such as self-hosted Nemotron inside your network. No third-party vendor is ever offered. Cached only where policy explicitly allows it.
Vendors with signed data-processing terms and zero data retention. Nearest compliant region wins on latency. Purged on a short TTL.
Full cost and latency optimization across contracted providers and any eligible region. Longest cache lifetimes.
Nemocore adopts incrementally on top of the Fastly services you already run.
Put AI endpoints behind Fastly. Turn on WAF, bot scoring, token metering, and logs. See where tokens go.
Enable exact, intent, and context caches. Immediate token and latency savings on repeat traffic.
Turn on intent triage and Switchyard routing across models and providers, with residency policy.
Apply access rules and pricing to AI crawlers and agents hitting your content and APIs.
Nemocore is a prototype in development. If you're a Fastly or NVIDIA partner, customer, or teammate and want an early look, we'd like to hear what you'd use it for.
Request a briefing