Building an Enterprise RAG Application in C# with Google Gemini and SQL Server 2025 Vector Engine

While in-memory vector stores and lightweight prototyping tools are excellent for local proofs-of-concept, production enterprise AI demands durability, transaction consistency (ACID), enterprise-grade security, and relational data proximity. Storing vector embeddings in a separate, isolated database introduces synchronization complexity, network latency, distributed transaction headaches, and bifurcated access control.

With the release of Microsoft SQL Server 2025, developers now have native vector search capabilities built directly into the relational database engine. By introducing the VECTOR data type and hardware-accelerated similarity functions like VECTOR_DISTANCE(), SQL Server enables unified querying over both structured relational records and dense unstructured vector embeddings in a single database.

In this comprehensive guide, we build a production-ready Retrieval-Augmented Generation (RAG) pipeline in C# (.NET 10). We integrate Google Gemini (gemini-3.8-flash and gemini-embedding-001) with SQL Server 2025 using Microsoft's new standardized Microsoft.Extensions.VectorData abstractions via CommunityToolkit.VectorData.SqlServer and Microsoft.Data.SqlClient.

Why SQL Server 2025 vs External Vector DBs?

Under the Hood: SQL Server 2025 Vector Engine

SQL Server 2025 introduces the native VECTOR(dimensions, data_type) type:

• float32 (Single Precision): Supports up to 1,998 dimensions. Storage is computed as dimensions × 4 bytes + 8 bytes header (for 768 dimensions: 3,080 bytes).
• float16 (Half Precision): Supports up to 3,996 dimensions for models requiring higher dimension counts.

Calculations utilize SIMD hardware acceleration via the VECTOR_DISTANCE T-SQL function:

SELECT TOP(1)
    Id, Title, Content,
    VECTOR_DISTANCE('cosine', Vector, @QueryVector) AS DistanceScore
FROM [dbo].[gemini_rag_docs]
ORDER BY DistanceScore ASC;

The Gemini Matryoshka Representation Learning (MRL) Factor

Google's gemini-embedding-001 produces 3,072 dimensions by default. However, SQL Server's float32 vector limit is 1,998 dimensions. Passing raw 3,072-dimensional vectors causes SQL Server to reject the parameter with:

Msg 2717, Level 15, State 3: The size (3072) given to the type 'vector' exceeds the maximum allowed (1998).

Because Google trained the model using Matryoshka Representation Learning (MRL), the embedding space is nested. The front dimensions encapsulate the core semantic density. By truncating the vector to 768 dimensions and recalculating the Euclidean L2 norm, we maintain high semantic quality while staying well within SQL Server's limits:

private static ReadOnlyMemory<float> TruncateAndNormalize(ReadOnlyMemory<float> vector, int dimensions)
{
    var span = vector.Span.Slice(0, dimensions);
    float sumSq = 0f;
    for (int i = 0; i < span.Length; i++) sumSq += span[i] * span[i];
    float norm = MathF.Sqrt(sumSq);

    var result = new float[dimensions];
    for (int i = 0; i < dimensions; i++) result[i] = span[i] / norm;
    return result;
}

Complete C# RAG Implementation

Project Configuration (RAGDemo.csproj)

<Project Sdk="Microsoft.NET.Sdk">

  <PropertyGroup>
    <OutputType>Exe</OutputType>
    <TargetFramework>net10.0</TargetFramework>
    <ImplicitUsings>enable</ImplicitUsings>
    <Nullable>enable</Nullable>
  </PropertyGroup>

  <ItemGroup>
    <PackageReference Include="CommunityToolkit.VectorData.SqlServer" Version="1.0.0" />
    <PackageReference Include="Microsoft.Data.SqlClient" Version="7.1.1" />
    <PackageReference Include="Microsoft.SemanticKernel" Version="1.74.0" />
    <PackageReference Include="Microsoft.SemanticKernel.Connectors.Google" Version="1.74.0-alpha" />
  </ItemGroup>

</Project>

Application Code (Program.cs)

using CommunityToolkit.VectorData.SqlServer;
using Microsoft.Extensions.VectorData;
using Microsoft.SemanticKernel.ChatCompletion;
using Microsoft.SemanticKernel.Connectors.Google;
using Microsoft.SemanticKernel.Embeddings;

namespace RAGDemo
{
    public class Program
    {
        static async Task Main(string[] args)
        {
            Console.WriteLine("=== VoidGeeks: C# RAG with Google Gemini & SQL Server 2025 Vector ===\n");

            // 1. Configure Gemini API credentials
            string apiKey = Environment.GetEnvironmentVariable("GEMINI_API_KEY") 
                ?? "YOUR_GEMINI_API_KEY";
            string chatModelId = "gemini-3.8-flash";
            string embeddingModelId = "gemini-embedding-001";

            // 2. Initialize Gemini chat and embedding services
            var textEmbeddingService = new GoogleAITextEmbeddingGenerationService(embeddingModelId, apiKey);
            var chatCompletionService = new GoogleAIGeminiChatCompletionService(chatModelId, apiKey);

            // 3. Connect to SQL Server 2025 Vector Store
            string connectionString = "Server=MSSQLSERVER01;Database=RAGDemoDB;Integrated Security=True;TrustServerCertificate=True;Encrypt=True;";
            var vectorStore = new SqlServerVectorStore(connectionString);
            var collection = vectorStore.GetCollection<long, DocumentChunk>("gemini_rag_docs");
            
            // Auto-creates [dbo].[gemini_rag_docs] with VECTOR(768, float32) if not exists
            await collection.EnsureCollectionExistsAsync();
            Console.WriteLine("Connected to SQL Server 2025 and verified table 'gemini_rag_docs'.");

            // 4. Ingest sample documents into SQL Server
            var documents = new[]
            {
                new DocumentChunk { Id = 1, Title = "Remote Work Policy", Content = "Employees are allowed up to 3 days of remote work per week. Requests must be approved by the department manager." },
                new DocumentChunk { Id = 2, Title = "Equipment Reimbursement", Content = "The company reimburses home office peripherals up to $500 annually. Receipts must be submitted by December 15." },
                new DocumentChunk { Id = 3, Title = "Wellness Program", Content = "All full-time staff receive a complimentary gym pass or a $60 monthly fitness allowance via the benefits portal." }
            };

            Console.WriteLine("\nGenerating embeddings via Gemini and ingesting knowledge base into SQL Server...");
            foreach (var doc in documents)
            {
                var fullVector = await textEmbeddingService.GenerateEmbeddingAsync(doc.Content);
                // Truncate to 768 dims for SQL Server & L2-normalize
                doc.Vector = TruncateAndNormalize(fullVector, 768);
                await collection.UpsertAsync(doc);
                Console.WriteLine($"Ingested document chunk #{doc.Id}: '{doc.Title}' into SQL Server.");
            }

            // 5. Execute Vector Search
            string userQuery = "How much can I expense for my home monitor setup, and what is the deadline?";
            Console.WriteLine($"\nUser Query: {userQuery}");

            var fullQueryVector = await textEmbeddingService.GenerateEmbeddingAsync(userQuery);
            var queryVector = TruncateAndNormalize(fullQueryVector, 768);

            Console.WriteLine("Searching for matching documents in SQL Server using vector similarity...");
            var searchResults = collection.SearchAsync(queryVector, top: 1);

            string retrievedContext = "";
            await foreach (var record in searchResults)
            {
                retrievedContext += record.Record.Content + "\n";
                Console.WriteLine($"Retrieved Context: \"{record.Record.Content}\" (Cosine Distance Score: {record.Score:F4})");
            }

            // 6. Augment Prompt with Grounded Context
            var chatHistory = new ChatHistory();
            chatHistory.AddSystemMessage("""
                You are an internal corporate assistant. Answer the user question strictly using the provided context. 
                If the answer cannot be found in the context, state that you do not know.
                """);

            chatHistory.AddUserMessage($"""
                Context:
                {retrievedContext}

                Question:
                {userQuery}
                """);

            // 7. Generate Response via Gemini
            Console.WriteLine("\nGemini Response:");
            for (int attempt = 1; attempt <= 3; attempt++)
            {
                try
                {
                    var response = await chatCompletionService.GetChatMessageContentAsync(chatHistory);
                    Console.WriteLine(response.Content);
                    break;
                }
                catch (Exception ex) when (attempt < 3)
                {
                    Console.WriteLine($"Gemini API retry attempt #{attempt}: {ex.Message}");
                    await Task.Delay(2000);
                }
            }
        }

        private static ReadOnlyMemory<float> TruncateAndNormalize(ReadOnlyMemory<float> vector, int dimensions)
        {
            var span = vector.Span.Slice(0, dimensions);
            float sumSq = 0f;
            for (int i = 0; i < span.Length; i++) sumSq += span[i] * span[i];
            float norm = MathF.Sqrt(sumSq);
            var result = new float[dimensions];
            for (int i = 0; i < dimensions; i++) result[i] = span[i] / norm;
            return result;
        }
    }

    public class DocumentChunk
    {
        [VectorStoreKey]
        public long Id { get; set; }

        [VectorStoreData]
        public string Title { get; set; } = string.Empty;

        [VectorStoreData]
        public string Content { get; set; } = string.Empty;

        [VectorStoreVector(768, DistanceFunction = DistanceFunction.CosineDistance)]
        public ReadOnlyMemory<float> Vector { get; set; }
    }
}

Execution Trace & Database Verification

Running the application yields:

Connected to SQL Server 2025 and verified table 'gemini_rag_docs'.

Generating embeddings via Gemini and ingesting knowledge base into SQL Server...
Ingested document chunk #1: 'Remote Work Policy' into SQL Server.
Ingested document chunk #2: 'Equipment Reimbursement' into SQL Server.
Ingested document chunk #3: 'Wellness Program' into SQL Server.

User Query: How much can I expense for my home monitor setup, and what is the deadline?
Searching for matching documents in SQL Server using vector similarity...
Retrieved Context: "The company reimburses home office peripherals up to $500 annually. Receipts must be submitted by December 15." (Score: 0.2380)

Gemini Response:
Based on the provided context, the company reimburses up to $500 annually for home office peripherals (though a "home monitor setup" is not specifically named), and receipts must be submitted by December 15.

Querying the database directly via SQL:

SELECT Id, Title, LEFT(Content, 35) AS ContentSnippet, LEFT(CAST(Vector AS NVARCHAR(MAX)), 35) AS VectorSnippet 
FROM [dbo].[gemini_rag_docs];

Engineering Lessons Learned & Field Gotchas