37 posts tagged with "AI Engineering"

Building a GPT-style large language model yourself is the fastest way to stop treating foundation models as black boxes. This playbook walks through the full path used in educational GPT implementations: prepare text, implement attention, assemble a decoder-only stack, pretrain (or load weights), then fine-tune for classification or instruction following.

The conceptual sequence mirrors production LLM development at smaller scale. You can run the educational path on a laptop; frontier training still needs datacenter compute. Use this article as an engineering map, then implement with LLMs-from-scratch and Sebastian Raschka’s Build a Large Language Model (From Scratch) (Manning).

Enterprise generative AI engineering is no longer limited to writing prompts and connecting an application to a large language model. A production AI system must combine software engineering, governed data access, model routing, retrieval, tool execution, evaluation, monitoring, security, cost control and continuous delivery.

Databricks addresses these requirements through an integrated set of capabilities covering the complete GenAI lifecycle: querying foundation models and agents, building custom and low-code agents, connecting agents to governed tools, preparing structured and unstructured data, implementing retrieval with AI Search, deploying agents and applications, governing traffic through Unity AI Gateway, tracing with MLflow, evaluating and monitoring quality, operationalising through LLMOps, and delivering governed experiences through Genie One.

Diffusion models are generative machine-learning models that learn to create data by reversing a gradual corruption process. During training, clean data—such as an image—is progressively disturbed with Gaussian noise. A neural network then learns how to predict and remove that noise. During generation, the model begins with random noise and repeatedly denoises it until a coherent image, video, audio clip, three-dimensional object, molecular structure, or other output emerges.

Although early diffusion systems operated directly on image pixels, modern systems often work in a compressed latent space and use either a convolutional U-Net or a transformer-based denoising network. Text encoders, cross-attention, classifier-free guidance, ControlNet, LoRA adapters, sophisticated numerical solvers, quantisation, distillation, and flow-matching techniques have made diffusion models more controllable and computationally practical.

Hugging Face Diffusers provides a modular implementation of this ecosystem. It separates pretrained denoising models, schedulers, text encoders, autoencoders, adapters, and pipelines so developers can combine and optimise them independently. The library supports image, video, audio, editing, restoration, super-resolution, depth estimation, and other diffusion-related workflows.

Twenty-two thousand five hundred and eighty. That is roughly how many GPT-2-scale models (≈124M parameters) fit inside Kimi K3 (≈2.8T parameters, 2026). Scale mattered—but it is not the whole story.

A parallel evolution happened inside the architecture itself:

AI models gradually changed from systems that repeatedly search every previous token into systems that maintain, update, erase, and selectively retrieve structured internal memories.

This article combines that memory-systems interpretation with the concrete architectural worklog 22580: From GPT2 to Kimi3, Explained by ali (@waterloo_intern), including the original diagrams, plus the research path from full softmax attention → linear attention → DeltaNet → Gated DeltaNet → Kimi Delta Attention → Kimi Linear / Kimi K3.

An LLM architecture defines how a model converts language into numbers, moves information between tokens, stores short-term contextual memory, transforms representations through neural layers, predicts or generates tokens, and scales parameters and computation.

Most modern language models belong to one or more of these families:

• Recurrent architectures: RNN, LSTM, GRU and RWKV
• Encoder-only Transformers: BERT-style models
• Decoder-only Transformers: GPT, Llama and Mistral-style models
• Encoder–decoder Transformers: T5 and BART-style models
• Sparse Mixture-of-Experts models
• Local, sparse and linear-attention models
• Retention and delta-rule architectures
• State-space models such as Mamba
• Convolutional sequence models such as Hyena
• Hybrid attention–recurrent architectures
• Diffusion language models
• Multimodal language models
• Retrieval-augmented language-model systems

LLM inference is expensive because the model repeatedly recomputes attention state over the same tokens. Caching attacks that redundancy at every layer of the stack: inside one decode loop (KV cache), across concurrent requests on a serving GPU (PagedAttention / prefix caching), across API calls for stable prompt prefixes (prompt caching), and finally at the application layer when the answer itself can be reused (exact / semantic response caching).

This guide synthesizes the mechanics from Sebastian Raschka’s KV-cache and architecture work, Sankalp’s PagedAttention deep dive, provider prompt-caching docs, AWS caching patterns, and application-layer guides from Machine Learning Mastery, IBM, Latitude, ngrok, and related sources into one engineering playbook.

Generative AI cost management is more complicated than ordinary cloud cost management. A conventional application might be charged according to CPU hours, memory, storage, and network usage. An LLM application can accumulate costs through input tokens, output tokens, cached tokens, reasoning tokens, embeddings, retrieval, reranking, guardrails, model evaluation, agent tool calls, retries, observability, and the infrastructure supporting the application.

The main FinOps challenge is therefore not simply:

How can we use the cheapest model?

The correct question is:

What is the lowest sustainable cost at which the application can meet its quality, latency, reliability, security, and business-outcome requirements?

Caching in a conventional application usually means storing the result of a database query or API request. In an LLM application, there are many more opportunities: repeated questions, equivalent paraphrases, identical embeddings, repeated retrieval and reranking, repeated tool calls, stable system prompts, and recurring agent planning steps.

The strongest architecture therefore does not implement one “LLM cache.” It implements a hierarchy of specialised caches across the RAG and agent pipeline.

Evaluating a conventional machine-learning model is often straightforward. A classification model can be measured using accuracy, precision, recall, F1 score, or area under the curve. A regression model can be measured using mean absolute error or root mean squared error.

Large language model applications are more difficult to evaluate because they are:

• Non-deterministic
• Generative rather than strictly predictive
• Capable of producing several valid answers
• Often composed of multiple models, tools, retrievers, databases, prompts, and agents
• Able to modify external environments
• Sensitive to prompt wording, model versions, context ordering, retrieved documents, and tool responses
• Expected to satisfy qualitative requirements such as helpfulness, clarity, faithfulness, tone, safety, and policy compliance

A production-grade evaluation strategy therefore cannot rely on one benchmark or one quality score. It must evaluate the system at several levels:

1. The underlying model
2. Individual LLM responses
3. Retrieval and generation components
4. Tool calls and agent trajectories
5. Multi-agent coordination
6. End-to-end business outcomes
7. Operational performance
8. Safety, compliance, and governance
9. Real production behaviour

Hugging Face Evaluate provides useful building blocks for metrics, comparisons, and measurements. However, Hugging Face currently presents LightEval as the more actively maintained toolkit for modern LLM benchmarking, while agentic applications require additional trace, tool, environment, and outcome evaluation capabilities.

Artificial intelligence architecture is moving beyond simple chatbot development. Modern enterprise AI systems must connect to business applications, retrieve organisational knowledge, call external tools, follow security policies, manage long-running tasks and produce outputs that other software systems can reliably process.

The Claude Certified Architect – Foundations, commonly known as CCA-F, focuses on the architectural knowledge needed to design these systems using Anthropic’s Claude platform.