Integrating Large Language Models with ASP.NET Core

Integrating Large Language Models (LLMs) with ASP.NET Core: A Developer Guide

Introduction

Adding an LLM to an ASP.NET Core application is more than sending a prompt to an API.

A production integration needs a clear boundary between the HTTP API, application logic, model provider, prompt construction, retrieval, security, observability, and failure handling.

A useful architecture is:

Angular / Client
       ↓
ASP.NET Core API
       ↓
Application Service
       ↓
LLM / AI Abstraction
       ↓
Model Provider

When retrieval is required:

Request
  ↓
API
  ↓
Embedding
  ↓
Vector Search
  ↓
Relevant Context
  ↓
LLM
  ↓
Response

Keep the Model Provider Behind an Application Boundary

Avoid calling a model SDK directly from controllers.

Instead:

public interface IChatService
{
    Task<string> CompleteAsync(
        string prompt,
        CancellationToken cancellationToken);
}

The infrastructure implementation can use the selected provider.

This keeps provider-specific details away from business workflows.

Request Flow

A typical endpoint might look like:

app.MapPost("/api/chat",
    async (
        ChatRequest request,
        IChatService chatService,
        CancellationToken cancellationToken) =>
    {
        var response = await chatService.CompleteAsync(
            request.Message,
            cancellationToken);

        return Results.Ok(new ChatResponse(response));
    });

The endpoint remains responsible for HTTP concerns, while the service owns the AI interaction.

Prompt Construction

Do not concatenate untrusted input into a large system prompt without considering injection and data-boundary risks.

Separate:

system instructions
application context
retrieved context
user input

Treat retrieved documents and user-provided text as data, not trusted instructions.

Structured Output

When an application needs machine-readable results, prefer structured responses supported by the selected model/provider rather than asking the model to “please return valid JSON” and hoping for the best.

Validate model output before using it as application data.

For example:

public sealed record ClassificationResult(
    string Category,
    double Confidence);

The application should validate the resulting structure before taking business actions.

Semantic Kernel

Semantic Kernel can provide abstractions for AI services, prompts, plugins, and orchestration.

Use it when the application benefits from those abstractions.

Do not introduce an AI orchestration framework simply to call one model once.

The architectural principle remains:

Business Workflow
       ↓
AI Abstraction
       ↓
Provider

Retrieval-Augmented Generation

For domain-specific answers, an LLM often needs external knowledge.

A RAG pipeline can be:

User Question
     ↓
Embedding
     ↓
Vector Search
     ↓
Top Relevant Chunks
     ↓
Prompt Context
     ↓
LLM
     ↓
Grounded Answer

The vector database is responsible for retrieval; it is not the LLM itself.

For .NET applications, PostgreSQL with pgvector and managed vector services are both viable depending on operational requirements.

Conversation History

Do not send unlimited conversation history on every request.

Define:

maximum history
summarization strategy
relevant context
token budget

Store conversation state deliberately if users need conversations to persist.

Security

AI endpoints should be treated like any other sensitive API.

Consider:

authentication
authorization
rate limiting
input size limits
output validation
secret management
logging controls
prompt injection defenses
tenant isolation

Never expose provider API keys to Angular or other untrusted clients.

Streaming

For chat interfaces, streaming can improve perceived responsiveness.

Possible architectures include:

LLM
 ↓
ASP.NET Core streaming endpoint
 ↓
Angular

Depending on requirements, use an appropriate streaming mechanism such as Server-Sent Events or SignalR.

Do not assume that streaming automatically improves overall backend throughput. It changes how results are delivered to the client.

Observability

Track operational information such as:

request duration
provider latency
failures
token usage when available
model/provider
retrieval latency
number of retrieved chunks

Avoid logging raw prompts or responses when they may contain sensitive information.

Resilience

Model providers can fail, throttle, or become temporarily unavailable.

Use appropriate:

timeouts
cancellation
retry policies for transient failures
circuit breakers where justified
fallback behavior

Do not blindly retry every model request because retries can increase cost and duplicate side effects.

Testing

Test deterministic application behavior independently of the model.

Use abstractions such as IChatService so application tests can provide a fake implementation.

For model behavior, use curated evaluation cases rather than relying only on traditional unit tests.

Conclusion

ASP.NET Core provides a strong boundary for integrating LLM capabilities into enterprise applications.

Keep AI provider details behind application abstractions, validate model output, protect the endpoint, control context and token usage, and treat retrieval and model orchestration as explicit application concerns.

The goal is not simply to “add AI.” The goal is to integrate model capabilities without allowing unpredictable model behavior or provider-specific code to leak throughout the application.