LLMOps blueprint for closed-source large language models

• A vendor of a software platform for financial advisors needs to build a knowledge management system that allows the advisors to ask questions about the platform, financial products, and tax policies. Under the hood, the system searches through an extensive collection of documents and other unstructured data. 
• A wholesaler of industrial goods needs to build a knowledge management system that allows customers to ask questions about products and browse product details. Internally, the system queries product specifications submitted by the supplier in PDF format. 
• A banking company needs to provide its associates with a user-friendly business intelligence (BI) solution for querying stock data using natural language questions like “What were the daily trading volumes for Alphabet stock over the last two weeks?”. Internally, the application uses an LLM to generate SQL and query a relational database (which we view as a tool). 

• Document storing and searching: RAG architecture requires a vector database for indexing knowledge base documents and their components, while enabling nearest neighbor search. This capability is commonly implemented using embedding-based search which, in turn, requires a model or service for computing text embeddings.
• Document preprocessing: The documents should typically be preprocessed to extract text from structured formats like PDF or mask-sensitive information such as PII data. 
• Document indexing: The preprocessed documents are indexed in the vector database, which can be a relatively complex process that requires tracking the removed/updated/deleted documents in the knowledge base, enriching documents with metadata such as document type and access groups, splitting documents into chunks, and computing embeddings.
• LLM gateway: The platform should support integrations with various LLM providers and models, including text generation and embedding computing models. The interaction gateway can also provide features like rate limiting, autoretry, etc. 
• Model management: Providers of closed-source LLMs commonly provide the ability to fine-tune models on proprietary data, necessitating proper management of the corresponding datasets and collections of fine-tuned models.
• Prompt management: LLM-based applications tend to use a large number of prompts that directly or indirectly control system behavior and user experience. The applications should use a unified framework for managing and versioning prompts.
• Observability: The production instance of the application needs to be continuously monitored to detect technical and functional issues in early stages. In particular, the problem of text generation quality should be continuously evaluated. 
• Cost efficiency: The platform should provide caching, cost tracking, and other services that optimize the cost of LLM usage. 
• Performance and latency: Text generation using LLMs is not a particularly fast operation, necessitating means for improving latency and other performance properties of the solution. 
• Safety and compliance: The platform should provide a mechanism for detecting and preventing LLM hallucinations, toxic responses, and other threats to a safe and consistent user experience. The platform should also provide protection for tools/systems that are controlled by LLMs.   
• Protection against bad actors: Security and data privacy are very important aspects of LLM-based solutions, and the platform should provide protection on both the LLM and user sides.

1. Document preprocessing such as PII data masking.
2. Splitting the documents into manageable chunks that fit the LLM context, and attributing metadata to each chunk.
3. Computing embeddings for the chunks.
4. Saving the chunks indexed by embeddings to the vector database.

1. The application creates a query context (e.g. by concatenating all messages in the dialog history including the latest input from the user) and converts it into a standalone question using the text-generating LLM. 
2. The embedding for this standalone question is computed using an embedding model.
3. Relevant document chunks are searched in the vector database based on the question embedding (nearest neighbor search).
4. The relevant chunks can be additionally filtered based on the metadata.
5. The chunks are retrieved, combined, and injected as the context, enriching the LLM’s ability to produce a relevant and well-grounded response.

Example technology stack

– Data indexing:Spark, LangChain, Airflow
– Vector database:ChromaDB, Qdrant, Milvus, Pinecone, or FAISS
– Application stack:Python, LangChain, LlamaIndex, FastAPI

• Caching: The LLM responses can be cached to handle frequent queries/contexts more efficiently. The most basic solution is to use standard caching components like Redis that perform an exact match between a new query and a cached query. However, this approach might be inefficient in LLM applications because of the high variability of queries. A more sophisticated solution lies in semantic caching, which employs embedding-based search to find cached queries that are similar to the new one. Internally, a semantic cache can rely on a vector database. 
• Streaming: User experience can be improved by printing out the generated text token by token (similar to the approach employed in the ChatGPT web interface) instead of waiting for a complete response in a blocking mode. This technique is commonly referred to as streaming. Streaming APIs are offered by most LLM providers, but creating a streaming application requires propagating near real-time token streams through all layers including backend services and frontend UI. Consequently, streaming needs to be incorporated in the solution design and appropriate orchestration and UI frameworks should be selected.

Example technology stack

– Exact match caching:Redis
– Semantic caching:GPTCache + vector database
– Streaming:OpenAI Streaming, LangChain (supports streaming), Streamlit Chat (supports streaming)

• Tailored for low-latency transactions: Feature stores are purpose-built to provide low-latency transactional access to large collections of features including basic attributes and derived values such as propensity scores. This design effectively mitigates scalability bottlenecks, ensuring swift and efficient data retrieval.
• Interoperability with ML models: Feature stores provide interoperability between in-house ML models (e.g. personalization or recommendation models) and LLM applications. If the company already uses a feature store, it is easy for LLM applications to tap into this infrastructure.
• Continuous feature updates: Feature stores are typically connected with supplementary infrastructure elements like data collection pipelines and scoring models, ensuring that the features remain consistently updated.

Example technology stack

– Feast, Vertex AI Feature Store, or Databricks Feature Store

Example technology stack

– PromptHub

• Inherent complexity of generated text evaluation: Assessing the quality, usefulness, compliance, and safety of generated text is inherently difficult to evaluate in general.
• Versatility of conversational systems: Conversational systems are extremely versatile and flexible, making it difficult or impossible to perform comprehensive testing.
• Integration with vector search: Vector search used in RAG adds an additional layer of complexity and uncertainty.
• Continuous LLM updates: LLM providers continuously update their products, causing shifts in application behavior that can prove unpredictable.
• Testability of complex LLM chains: Given their intricate interdependencies, complex LLM chains are difficult to test and debug.

• Toxicity assessment: This is a component that evaluates text for the presence of violence, harmful, or offensive content, generating scores (e.g. hate_speech = 0.28) that trigger mitigating actions. The toxicity checks can be performed for both user input and LLM output.
• Topic bans: This is a component that detects certain topics and takes mitigating actions. For example, it can detect politics-related topics in the user input (e.g. “What do you think of the president?”) and return a blocking response (e.g. “I’m a shopping assistant, I don’t like to talk about politics”). This check can be applied to both user input and LLM output.  
• Relevance validation: This is a guardrail that validates the generated response as relevant to the input question and context, often measured through similarity scores between the prompt and response. For example, the score will be low for the question “What is the current sales tax in California?” when the answer is “Yangtze River is the longest river in both China and Asia”. 
• Contradiction assessment: This is a component that verifies the absence of LLM output self-contradictions, contradictions to the input, or contradictions with established facts. 
• Hallucinations and factuality detection: This guardrail detects hallucinations and non-factual statements in the LLM output. Detecting hallucinations is generally a challenging task that can be approached in many different ways. One practical approach for closed-source LLMs is to generate multiple responses for the same query and validate their agreement.

Example technology stack

– Standard guardrails:LLM-Guard, OpenAI Moderation APIs
– Custom flows:NVIDIA NeMo, custom guards using Hugging Face model

• Anonymization/deanonymization: This guardrail consists of two parts. The first one is the input interceptor that detects sensitive data elements such as names, addresses, and other PII data, and replaces them with surrogate tokens. The second is the output interceptor that performs the reverse mapping. In some cases, the anonymization operation can be made irreversible.  
• Prompt injection: This is a component that detects malicious prompts that attempt to override system prompts, hijack the conversation context, or instruct the LLM to perform unintended actions (jailbreak). This guardrail is particularly important for LLM agents that control internal or external systems via API.
• Secret detection: This is a scanner that detects login, password, credit card numbers and other sensitive data. This check can be applied to both user input and LLM output.  
• URL detection: This scanner detects malicious URLs in user input and LLM output.
• Rate limiting: This guardrail throttles input querying and LLM invocation rates with the goal of preventing outages, DoS attacks, and excessive (and costly) LLM calls due to defects.

1. The PII data can be replaced with surrogate tokens as a part of the preprocessing stage.
2. The mapping between actual PII values and tokens is saved in a database.
3. Reverse mapping is performed in query time to produce the final response for the user.

Example technology stack

– LLM-Guard, Guardrail ML, custom guards using Hugging Face model

• Prompt analytics: The prompts received from users or external systems should be logged and analyzed. In particular, the prompts can be clustered and visualized to better understand the application usage.
• Problematic prompts: Prompts that deviate from the typical clusters can be flagged as outliers.
• User feedback capturing: As we discussed earlier, the quality of generated text can be challenging to evaluate. The quality analysis and detection of problematic prompts can be facilitated by capturing implicit or explicit user feedback (e.g. thumbs up/down).  
• Automatic checks: Continuous LLM updates can be countered with automatic quality, compliance, and safety checks. 
• Monitoring: LLM-backed applications require tracking metrics such as cache hit ratio and throughputs/latencies at different stages of the LLM chains.
• Logging: Logging requests, responses, prompts, and automatic checks enable auditability, and support optimization and troubleshooting.

Example technology stack

– Arize

• Training dataset management: Fine-tuning hinges on the availability of training datasets, which demand data cataloging and tracking, mirroring the conventional practices applied to datasets in traditional MLOps. 
• Model registry: As the number of fine-tuned models grows, maintaining a registry of such models with the associated metadata, such as training parameters, becomes an important concern. This is also a well-understood problem in traditional MLOps. 

Example technology stack

– Training dataset management:DataHub
– Model repository:Vertex AI, MLflow

• Auxiliary functions such as embedding computing, PII data detection and masking, and some other operations that are performed by guardrails do not necessarily require cutting-edge LLMs and can be implemented using small models. Consequently, these functions are good candidates for implementing open-source, privately deployed, and fine-tuned models.
• Text generation and reasoning capabilities typically require state-of-the-art LLMs which makes them more difficult to implement using the open-source approach. Using privately deployed, closed-source LLMs or open-source LLMs for text generation is a major strategic decision that is informed by functional, security, budgeting, operational, and business considerations.