From Theory to Practice: Building Effective AI Systems with RAG

Over the past year, our journey with artificial intelligence has revealed crucial insights about implementing AI systems in practical settings. One fundamental lesson stands out above all others: data is the cornerstone of any effective AI implementation. The challenge isn’t just having data – it’s about transforming unstructured information into a format that AI models can meaningfully process.

Organizations can spend considerable time developing sophisticated ETL (Extract, Transform, Load) workflows to prepare and maintain their data. This preparation phase is often more complex and time-consuming than the actual AI implementation itself. The process requires careful orchestration of data processing functions, including advanced ETL flows that prepare data for both efficient query and storage.

PRACTICAL APPLICATIONS OF RAG

When it comes to implementation strategies, we’ve found that Retrieval- Augmented Generation (RAG) solves most use cases, making fine-tuning often unnecessary. RAG deserves special attention as it encompasses various techniques including contextual retrieval, keyword retrieval, and reranking – a full paper all on its own. These methods, combined with proper document processing, form the backbone of practical AI applications. While RAG might seem nebulous at first, its practical applications have proven invaluable in real-world scenarios.

Document question-answering is perhaps the most common, where RAG enables precise responses from technical documentation, policy manuals, and research papers. Customer service applications leverage RAG to ground responses in product information and policies, while legal teams use it to analyze contracts and regulatory documents with proper citations. Internal knowledge management becomes more dynamic with RAG, allowing natural language queries against company documentation. We’re also exploring promising applications in analytics analysis and project estimation, where RAG can process historical project data and metrics to provide insights on timelines and resource requirements. These applications succeed because RAG combines the fluency of large language models with the reliability of retrieving specific information from verified sources.

ENTER BR INSIGHT: A COMPREHENSIVE AI FRAMEWORK

Our internal workflow orchestration framework, BR Insight, represents a practical implementation of these principles. The framework is designed to handle the complexities of AI implementation in a modular and scalable way, processing and analyzing data through a series of well-defined workflows that primarily focus on Large Language Model (LLM) inference work. We’ve built the system with containerization in mind, ensuring flexibility across different cloud providers and computing environments, whether serverless or dedicated instances.

The framework’s architecture consists of three primary backend components. The Document Processor, built in Python and leveraging Unstructured.io, serves as the foundation for handling diverse document types. This component addresses several critical challenges in document processing, including variations in document size and type, context preservation across document sections, and special content handling. When encountering images and tables, the system employs LLM summarization before embedding, ensuring comprehensive content processing.

The Orchestrator component, also Python-based, manages the workflow through a DAG (Directed Acyclic Graph) structure. It standardizes all interactions through JSON-based input and output, maintaining actions in separate repositories. This design ensures that each action remains isolated and portable, enabling flexible deployment across different environments. The self- contained nature of these operations supports future scaling through distributed workload processing.

Our API layer, built on Node.js, serves as the system’s interface. It manages database connections, handles credential security, and controls task orchestration. This layer also integrates embedding services, providing flexibility in choosing between dedicated models or external services based on specific needs and requirements.

5 TIPS FOR AI IMPLEMENTATION

1. MAINTAINING AGILITY IN MODEL SELECTION IS CRUCIAL

The AI landscape evolves rapidly, and today’s optimal model might not be tomorrow’s best choice. We currently use Claude 3.5 Sonnet for most applications, but our architecture, built on abstraction frameworks like LangChain, allows us to switch models easily as better options emerge.

2. CLOUD INFRASTRUCTURE AND RELIABILITY

Unless AI is central to your product offering, SaaS solutions might be more appropriate. AI stack maintenance demands constant attention to model updates, infrastructure management, data leakage prevention, and conversation auditing. While we chose to build our own tools at BR for internal use and experimentation, we still rely on Amazon Bedrock for model hosting rather than maintaining our own model infrastructure.

3. DATA QUALITY DIRECTLY IMPACTS LLM PERFORMANCE

Implement robust systems for source citation, verification, and regular quality assessments. Tools like LangSmith can help monitor performance and maintain feedback loops, ensuring consistent quality in AI outputs.

4. MAINTAINING VIGILANT SUPERVISION OF AI SYSTEMS

While RAG significantly improves model grounding, these systems remain statistical in nature and can produce errors. Implement explicit guardrails in prompts or dedicated audit steps, regularly adjust based on model behavior, and exercise particular caution in high-risk domains like healthcare and financial transactions.

5. GENERATIVE AI ISN’T ALWAYS THE ANSWER

While LLMs are powerful, they’re not always the best solution. Traditional AI models might better suit specific tasks, often offering better performance at lower cost. Task-specific solutions for vision or categorization might be more appropriate, and computational resources should factor into model selection.

HOW TO START

Originally published in EXP. Magazine