AI & LLM Engineering Glossary

Clear technical definitions of 30 essential terms across artificial intelligence, large language models, quantization, RAG, and agentic systems.

표시 중 30 / 30

AI Agent: An autonomous software entity powered by an LLM that plans multi-step goals, uses external tools (such as web search, file editors, and terminal execution), inspects feedback, and completes tasks without human micromanagement.
Attention Mechanism: A mathematical algorithm in transformer models that allows every token to calculate dynamic importance weights against every other token in the context window. This enables the model to connect related concepts across thousands of lines of text.
Benchmark: A standardized evaluation suite used to objectively measure and rank an AI model's capabilities across specific tasks — such as SWE-bench Verified for software engineering, AIME for competitive math, or MMLU for general knowledge.
Cached Input: A cost and latency optimization where inference providers store pre-computed attention states for repeated prompt prefixes. Reusing cached tokens typically slashes API input costs by 50% to 90% and cuts TTFT substantially.
Chain-of-Thought (CoT): A step-by-step reasoning technique where an AI model generates intermediate problem-solving thoughts before arriving at a final answer. Native reasoning models like OpenAI o3 and DeepSeek R1 use CoT to solve complex math and coding challenges.
Context Window: The maximum number of tokens a model can consider at once — both the input it reads and the output it writes. A 1,000,000-token context window can hold roughly 750,000 English words in a single request.
Embeddings: High-dimensional numerical vector representations of words, sentences, or documents that capture their semantic meaning. Closer vector distances indicate closer conceptual similarity, powering vector search and RAG systems.
Fine-Tuning: The process of taking a pre-trained foundation model and further training it on a specialized, domain-specific dataset. This tailors the model for specialized roles such as medical diagnosis, legal contract analysis, or company coding conventions.
Hallucination: When a model confidently states something that is factually wrong or invented. Lower hallucination rates are a key quality signal, and techniques like RAG, ground-truth citations, and reasoning chains help minimize them.
Inference: The operational process of running live user inputs through an already-trained AI model to generate answers, code, or predictions. API token pricing covers the compute cost incurred during this inference phase.
KV Cache: A GPU memory optimization that stores previously calculated Key and Value attention matrices so the model does not recompute attention for existing tokens at each generation step. KV cache is vital for fast streaming generation and prompt caching.
Large Language Model (LLM): A deep-learning AI model trained on massive text corpora to understand, summarize, and generate human-like language. Modern examples include GPT-4o, Claude 3.7 Sonnet, and Gemini 2.0 Flash.
Latency: The total duration required for an AI system to process an input and deliver a complete output response, typically encompassing network transmission, TTFT, and generation time.
Mixture-of-Experts (MoE): An architecture where a large model is split into many specialist "expert" sub-networks and only a few activate per token. This keeps inference cheap while allowing a huge total parameter count — for example DeepSeek-V3 has 671B total parameters but activates only 37B per token.
Model Context Protocol (MCP): An open standard created by Anthropic that allows AI applications and agents to connect seamlessly to local filesystems, development environments, and third-party APIs via standardized server interfaces.
Multimodal: An AI model capable of natively understanding and generating multiple data types — such as text, images, audio, video, and computer code — within a unified neural network without requiring separate external pipelines.
Parameters: The internal variables or weights that a neural network adjusts during training to store patterns and knowledge. For instance, Llama 3.3 70B has 70 billion parameters, while frontier models range into hundreds of billions.
Quantization: The technique of reducing the numerical precision of model weights — such as converting 16-bit floating points (FP16) to 8-bit or 4-bit integers (INT8/INT4). This dramatically cuts GPU VRAM requirements and speeds up local model inference with negligible loss in accuracy.
RAG (Retrieval-Augmented Generation): An architecture that retrieves relevant document chunks from an external vector database or search engine and injects them into the model's prompt. This grounds answers in fresh, proprietary facts without requiring expensive model retraining.
Reasoning Model: An AI model trained via reinforcement learning to spend dynamic compute thinking through complex logic, math, and architectural problems before outputting a solution. Examples include OpenAI o1, o3, and DeepSeek R1.
RLHF (Reinforcement Learning from Human Feedback): A training technique that fine-tunes AI models using human rankings and preferences to align responses with human values, enhance helpfulness, and eliminate toxic or misleading answers.
Speculative Decoding: An inference acceleration technique that uses a smaller, ultra-fast draft model to predict multiple candidate tokens in parallel, which are then validated in a single forward pass by the large target model, boosting generation speed without losing accuracy.
System Prompt: A persistent, high-priority instruction set given to an AI model before user interaction that defines its persona, tone, guardrails, constraints, and operational context.
Temperature: A sampling parameter that controls the randomness and creativity of an AI model's output. A low temperature (e.g., 0.1 to 0.3) makes responses focused and deterministic for math or code, while a higher temperature (e.g., 0.7 to 1.0) produces varied, creative responses.
Time to First Token (TTFT): The latency in milliseconds between sending a prompt to an AI model and receiving the very first token of the output. TTFT measures prompt processing time and server responsiveness.
Token: The basic unit of text that a language model processes, typically representing about 3 to 4 English characters or roughly 0.75 words. For example, the sentence "Artificial intelligence is fast" breaks down into about 5 tokens.
Tokens Per Second (TPS): A metric quantifying the streaming inference throughput of an AI model, indicating how rapidly generated text streams back to the client. Higher TPS is critical for real-time interactive coding and conversational voice assistants.
Tool Calling: The ability of a language model to recognize when an external capability is needed and output structured JSON arguments to invoke external APIs, databases, or terminal commands. The runtime executes the tool and passes results back to the model.
Transformer: The neural network architecture introduced in 2017 that powers modern LLMs, using self-attention mechanisms to process all tokens in a sequence concurrently rather than sequentially.
Vision-Language Model (VLM): A multimodal neural network engineered to jointly process, reason over, and synthesize both visual content (images, diagrams, UI screenshots, video) and natural language text within a single framework.

자주 묻는 질문 (FAQ)

Can DeepSeek R1 run locally on a 24GB VRAM GPU like RTX 4090?+

The full 671B DeepSeek R1 requires multi-GPU clusters, but distilled versions (such as DeepSeek-R1-Distill-Qwen-14B or 32B with 4-bit quantization) run smoothly on a single 24GB RTX 4090.

Which model has lower latency for real-time voice streaming: Gemini 2.0 Flash or GPT-4o-mini?+

Gemini 2.0 Flash leads with native multimodal audio-to-audio streaming and sub-200ms time-to-first-token (TTFT), making it faster than text-pipelined GPT-4o-mini setups.

How does Gemini 2.0 Pro 2 million token context window process entire code repositories?+

Gemini 2.0 Pro ingests up to 1.5 million words of code in a single prompt with near 100% needle-in-a-haystack retrieval accuracy, eliminating the need for complex RAG chunking pipelines.

Is self-hosting Llama 3.3 70B cheaper than using GPT-4o API?+

Self-hosting Llama 3.3 70B on cloud GPUs becomes cheaper than GPT-4o when processing over 50 million tokens per month, offering complete data privacy with zero licensing fees.

What is the best open-source LLM to replace proprietary APIs in 2026?+

DeepSeek V3 (671B MoE) and Meta Llama 3.3 70B are the top open-weight models matching closed frontier APIs in coding, tool execution, and multilingual reasoning.

How does EAGLE-3 speculative decoding achieve 2x to 4x inference speedup?+

EAGLE-3 generates draft tokens using lightweight autoregressive draft heads trained on target model intermediate representations, verifying candidate tokens in a single target forward pass to accelerate bandwidth-bound generation.

What is Parallel-EAGLE (P-EAGLE) in vLLM and how does it eliminate drafting bottlenecks?+

Parallel-EAGLE (P-EAGLE) drafts all candidate tokens in a single parallel draft step instead of sequential autoregressive iterations, delivering an additional 20% to 30% latency reduction over standard EAGLE in vLLM.

When does speculative decoding decrease LLM throughput instead of speeding it up?+

Speculative decoding decreases throughput when the draft model acceptance rate falls below 50%—such as highly creative writing or mismatched temperature distributions—causing drafting and verification overhead to exceed direct generation costs.

How to select the best draft model for speculative decoding in vLLM or SGLang?+

The optimal draft model should share the same vocabulary and tokenizer, feature a high distribution alignment (>70% acceptance rate), and execute draft steps in under 15% of the target model single-step forward pass time.