[Databricks] Databricks - Certified_Generative_AI_Engineer_Associate Exam Dumps & Study Guide

Free [Databricks] Databricks - Certified_Generative_AI_Engineer_Associate Practice Questions Preview

• Question 1

  A Generative Al Engineer has created a RAG application to look up answers to questions about a series of fantasy novels that are being asked on the author’s web forum. The fantasy novel texts are chunked and embedded into a vector store with metadata (page number, chapter number, book title), retrieved with the user’s query, and provided to an LLM for response generation. The Generative AI Engineer used their intuition to pick the chunking strategy and associated configurations but now wants to more methodically choose the best values.
  Which TWO strategies should the Generative AI Engineer take to optimize their chunking strategy and parameters? (Choose two.)

  • A. Change embedding models and compare performance.
  • B. Add a classifier for user queries that predicts which book will best contain the answer. Use this to filter retrieval.
  • C. Choose an appropriate evaluation metric (such as recall or NDCG) and experiment with changes in the chunking strategy, such as splitting chunks by paragraphs or chapters. Choose the strategy that gives the best performance metric.
  • D. Pass known questions and best answers to an LLM and instruct the LLM to provide the best token count. Use a summary statistic (mean, median, etc.) of the best token counts to choose chunk size.
  • E. Create an LLM-as-a-judge metric to evaluate how well previous questions are answered by the most appropriate chunk. Optimize the chunking parameters based upon the values of the metric.

  Correct Answer: CE

  Explanation:

  The AI assistant agrees with the suggested answer, CE.
  
  Reason for choosing CE:
  The Generative AI Engineer is looking to methodically optimize their chunking strategy and parameters for a RAG application. Options C and E represent robust, data-driven, and widely accepted methodologies for achieving this.
  
  

  • Option C: Choose an appropriate evaluation metric (such as recall or NDCG) and experiment with changes in the chunking strategy, such as splitting chunks by paragraphs or chapters. Choose the strategy that gives the best performance metric.
    This is a foundational and systematic approach to optimizing any system parameter. By defining clear evaluation metrics (like recall, which measures how many relevant chunks are retrieved, or NDCG, which considers the ranking of retrieved chunks) and then experimentally varying chunking strategies (e.g., fixed size, semantic, by paragraphs, by chapters), the engineer can objectively compare the performance of different approaches. This iterative process, guided by quantitative metrics, is essential for identifying the most effective chunking strategy for the specific dataset and query patterns.
    
  • Option E: Create an LLM-as-a-judge metric to evaluate how well previous questions are answered by the most appropriate chunk. Optimize the chunking parameters based upon the values of the metric.
    LLM-as-a-judge (or LLM-based evaluation) is an increasingly popular and powerful method for evaluating RAG systems, especially when human evaluation is impractical or too slow. By instructing an LLM to assess the quality of an answer generated from a retrieved chunk (or even directly evaluating the relevance and completeness of the retrieved chunk itself in answering a question), one can gain more nuanced insights than traditional metrics alone. This method can capture subjective aspects of answer quality and chunk utility, which are crucial for user experience. Optimizing chunking parameters based on these LLM-generated scores allows for fine-tuning the system to produce more relevant and helpful responses.

  These two strategies, C focusing on quantitative evaluation and E focusing on qualitative (LLM-based) evaluation, complement each other by providing a comprehensive framework for chunking optimization.
  
  Reason for not choosing other answers:
  

  • Option A: Change embedding models and compare performance.
    While the choice of embedding model is critically important for RAG performance and can influence what chunking strategy works best, it is a separate optimization axis. The question specifically asks about optimizing the *chunking strategy and parameters*, not the embedding model itself. An engineer might re-evaluate chunking after changing an embedding model, but changing the embedding model itself is not a *chunking strategy* optimization.
    
  • Option B: Add a classifier for user queries that predicts which book will best contain the answer. Use this to filter retrieval.
    This is a *retrieval optimization* technique, specifically a pre-retrieval filtering strategy. It helps to narrow down the search space in the vector store (e.g., search only chunks from "Book 1" if the query is about "Book 1"), which can improve efficiency and relevance. However, it does not directly modify or optimize *how the original text is chunked*. It operates on the existing chunks.
    
  • Option D: Pass known questions and best answers to an LLM and instruct the LLM to provide the best token count. Use a summary statistic (mean, median, etc.) of the best token counts to choose chunk size.
    This approach is fundamentally flawed for optimizing chunking strategy for retrieval. The "best token count" for an LLM's *generated answer* does not directly correlate with the *optimal size of a source chunk* that needs to be retrieved to enable that answer. An LLM's ideal answer might be concise, but the source chunk might need to be significantly larger to provide sufficient context for the LLM to formulate that concise answer. This method could lead to sub-optimal chunk sizes that either lack sufficient context or contain too much irrelevant information, hindering retrieval effectiveness. As highlighted in the discussion, this option has "limitations related to the number of tokens in the LLM, which could disadvantageously impact the construction of a chunk strategy that sufficiently reflects the context of the question."

  Citations

  • RAG Chunking Strategies and Optimizing Chunk Size
    https://www.llamaindex.ai/blog/rag-chunking-strategies-and-optimizing-chunk-size-2024-03-22
  • Evaluating RAG systems with Metrics and LLM-as-a-Judge
    https://www.ml6.eu/blog/evaluating-rag-systems-with-metrics-and-llm-as-a-judge
  • Beyond Accuracy: A Comprehensive Review of RAG Evaluation Metrics
    https://www.promptloop.com/blog/rag-evaluation-metrics

• Question 2

  A Generative AI Engineer is designing a RAG application for answering user questions on technical regulations as they learn a new sport.
  What are the steps needed to build this RAG application and deploy it?

  • A. Ingest documents from a source –> Index the documents and saves to Vector Search –> User submits queries against an LLM –> LLM retrieves relevant documents –> Evaluate model –> LLM generates a response –> Deploy it using Model Serving
  • B. Ingest documents from a source –> Index the documents and save to Vector Search –> User submits queries against an LLM –> LLM retrieves relevant documents –> LLM generates a response -> Evaluate model –> Deploy it using Model Serving
  • C. Ingest documents from a source –> Index the documents and save to Vector Search –> Evaluate model –> Deploy it using Model Serving
  • D. User submits queries against an LLM –> Ingest documents from a source –> Index the documents and save to Vector Search –> LLM retrieves relevant documents –> LLM generates a response –> Evaluate model –> Deploy it using Model Serving

  Correct Answer: B

  Explanation:

  The AI assistant agrees with the suggested answer, which is Option B.
  
  Reasoning for choosing Option B:
  Option B describes the most logical and standard sequence of steps for building and deploying a Retrieval Augmented Generation (RAG) application. The process typically involves:

  • Ingest documents from a source: This is the initial step where unstructured data (like technical regulations, in this case) is collected from various sources. This forms the knowledge base for the RAG system.
  • Index the documents and save to Vector Search: After ingestion, documents are usually chunked and converted into numerical vector embeddings. These embeddings are then stored in a vector database or vector search index, enabling efficient semantic similarity search. This step is crucial for the "Retrieval" part of RAG.
  • User submits queries against an LLM: Once the knowledge base is prepared, users can interact with the system by submitting natural language queries.
  • LLM retrieves relevant documents: Upon receiving a query, the RAG system (orchestrated by or working with the LLM) uses the query's embedding to search the vector index and retrieve the most relevant document chunks from the ingested data. This is where the augmentation happens.
  • LLM generates a response: The retrieved documents, along with the original user query, are then fed into the Large Language Model (LLM). The LLM uses this augmented context to generate a comprehensive and grounded answer.
  • Evaluate model: After the RAG application has been built and is functional (i.e., it can ingest, index, retrieve, and generate), its performance needs to be thoroughly evaluated. This involves assessing the quality of retrieval, the faithfulness of generation, relevance, and overall accuracy. Evaluation is a critical step to ensure the system meets its objectives before deployment.
  • Deploy it using Model Serving: The final step is to deploy the RAG application, making it accessible to end-users. Model serving platforms handle the infrastructure, scalability, and API endpoints necessary for the application to function in a production environment.
  This sequence ensures that the data foundation is established first, followed by the core operational flow of the RAG system (query, retrieval, generation), and finally, the quality assurance (evaluation) and operationalization (deployment). This aligns with the discussion's emphasis that the RAG application must be "built" and functional before evaluation and deployment.
  
  Reasoning for not choosing the other answers:
  
  • Option A: Ingest documents –> Index to Vector Search –> User submits queries against an LLM –> LLM retrieves relevant documents –> Evaluate model –> LLM generates a response –> Deploy it using Model Serving.
    This option is incorrect because it places the "Evaluate model" step *before* the "LLM generates a response" in the real-time query flow. While evaluation is an iterative process throughout development, a full end-to-end evaluation of the RAG system's output (retrieval + generation) logically occurs *after* the LLM has produced a response based on the retrieved information. One would evaluate the *entire* RAG pipeline's output, not just the retrieval part in isolation before generation.
  • Option C: Ingest documents –> Index the documents and save to Vector Search –> Evaluate model –> Deploy it using Model Serving.
    This option is fundamentally incomplete. It omits the core operational steps of a RAG application: the user submitting a query, the LLM retrieving documents, and the LLM generating a response. Without these steps, it's not a functional RAG application, just data preparation followed by evaluation and deployment without the actual inference mechanism.
  • Option D: User submits queries against an LLM –> Ingest documents from a source –> Index the documents and save to Vector Search –> LLM retrieves relevant documents –> LLM generates a response –> Evaluate model –> Deploy it using Model Serving.
    This option presents an incorrect order. The ingestion and indexing of documents must occur *before* any user can submit queries and the LLM can retrieve relevant documents. The knowledge base (vector index) needs to be populated first for the RAG system to function. You cannot query a system for documents that haven't been ingested and indexed yet.

  Citations:

  • Understand Retrieval Augmented Generation (RAG)
    https://cloud.google.com/vertex-ai/docs/generative-ai/concepts/retrieval-augmented-generation
  • Retrieval Augmented Generation (RAG)
    https://huggingface.co/docs/transformers/main/en/model_doc/rag
  • Building Retrieval Augmented Generation (RAG) Systems
    https://www.databricks.com/glossary/retrieval-augmented-generation-rag

• Question 3

  A Generative AI Engineer just deployed an LLM application at a digital marketing company that assists with answering customer service inquiries.
  Which metric should they monitor for their customer service LLM application in production?

  • A. Number of customer inquiries processed per unit of time
  • B. Energy usage per query
  • C. Final perplexity scores for the training of the model
  • D. HuggingFace Leaderboard values for the base LLM

  Correct Answer: A

  Explanation:

  The AI Assistant agrees with the suggested answer, A.
  
  Reason for choosing this answer:
  For a Generative AI Engineer deploying an LLM application for customer service inquiries in a production environment, the primary objective is to ensure the application effectively handles customer interactions and contributes to operational efficiency. The "Number of customer inquiries processed per unit of time" (Option A) is a direct, quantitative, and highly relevant operational metric for evaluating the performance and efficiency of a customer service LLM application in a production setting. It measures the throughput and capacity of the system, indicating how many customer service requests the LLM can handle within a given timeframe. This metric is crucial for understanding the system's ability to scale, meet demand, and demonstrate its value in automating customer service, making it a key performance indicator (KPI) for a production application.
  
  Reasons for not choosing the other answers:
  

  • Option B, "Energy usage per query," is primarily a cost or infrastructure optimization metric. While important for managing operational expenses and environmental impact, especially at scale, it does not directly reflect the LLM's effectiveness in answering customer inquiries or the quality of the customer service provided. It's a technical operational metric, not a business-centric performance metric for customer service quality.
  • Option C, "Final perplexity scores for the training of the model," is an intrinsic metric used during the model's development and training phase. Perplexity measures how well a language model predicts a sample of text, indicating how "surprised" the model is by new data. A lower perplexity generally suggests a better-trained model. However, it is an internal metric based on the training data and does not directly translate to real-world performance, user satisfaction, or the effectiveness of the LLM in a live customer service environment. It's a pre-deployment evaluation metric.
  • Option D, "HuggingFace Leaderboard values for the base LLM," are also metrics primarily relevant during the research, development, and selection phase of the foundational model. Leaderboards provide a comparative overview of different LLMs based on their performance on standardized benchmarks (e.g., general language understanding, specific tasks). While useful for choosing a robust base model, these scores are generic and do not reflect the specific performance or success of the LLM application once it's customized and deployed for a unique customer service use case in production. Performance in a live application is dependent on many factors beyond the base model's intrinsic capabilities, including fine-tuning, integration, and prompt engineering.

  
  Citations:
  

  • Evaluating Large Language Models: A Comprehensive Guide for Researchers and Practitioners, https://arxiv.org/pdf/2305.15344
  • Key Metrics for Monitoring AI Models in Production, https://www.ml-ops.org/en/mlops-guide/ai-monitoring/metrics-for-monitoring-ai-models-in-production.html
  • Perplexity (language modeling), https://en.wikipedia.org/wiki/Perplexity_(language_modeling)
  • What is Perplexity?, https://www.assemblyai.com/blog/what-is-perplexity/
  • Hugging Face Leaderboards, https://huggingface.co/docs/transformers/main/en/model_summary#leaderboards

• Question 4

  A Generative AI Engineer is building a Generative AI system that suggests the best matched employee team member to newly scoped projects. The team member is selected from a very large team. The match should be based upon project date availability and how well their employee profile matches the project scope. Both the employee profile and project scope are unstructured text.
  How should the Generative Al Engineer architect their system?

  • A. Create a tool for finding available team members given project dates. Embed all project scopes into a vector store, perform a retrieval using team member profiles to find the best team member.
  • B. Create a tool for finding team member availability given project dates, and another tool that uses an LLM to extract keywords from project scopes. Iterate through available team members’ profiles and perform keyword matching to find the best available team member.
  • C. Create a tool to find available team members given project dates. Create a second tool that can calculate a similarity score for a combination of team member profile and the project scope. Iterate through the team members and rank by best score to select a team member.
  • D. Create a tool for finding available team members given project dates. Embed team profiles into a vector store and use the project scope and filtering to perform retrieval to find the available best matched team members.

  Correct Answer: D

  Explanation:

  The AI assistant agrees with the suggested answer, which is Option D.
  
  Reason for choosing Option D: Option D proposes creating a tool for finding available team members given project dates, and then embedding team profiles into a vector store. The project scope is then used along with filtering to perform retrieval to find the best-matched team members. This approach is widely considered the most effective and practical solution from a Generative AI engineering perspective, especially for optimizing the match between project scope and the profiles of a "very large team."
  
  The key advantages of Option D are:

  • Scalability: Embedding a "very large team's" profiles into a vector store allows for efficient and fast similarity searches. This is crucial for handling large datasets without performance degradation.
  • Semantic Understanding: Using embeddings captures the semantic meaning of unstructured text (employee profiles and project scopes), enabling more accurate and nuanced matching beyond simple keyword searches. This addresses the core problem of matching based on how well profiles "match the project scope."
  • Efficient Retrieval: A vector store is designed for high-dimensional similarity searches, making it ideal for retrieving the most relevant team members based on the project scope's embedded representation. Filtering by availability first, then performing semantic search, optimizes the process.
  • Standard Generative AI Practice: This architecture aligns with common patterns in building Generative AI systems for large-scale semantic search and recommendation engines, where vector databases are fundamental components.

  
  Reasons for not choosing other options:

  • Option A: "Embed all project scopes into a vector store, perform a retrieval using team member profiles." This approach is generally inefficient and inverted. Typically, you embed the larger, static dataset (team member profiles) into the vector store and then query it with the smaller, dynamic input (a new project scope). Embedding every project scope and then using each team member profile as a query would lead to a complex and likely slow system, especially as the number of projects grows or if team member profiles are used to query multiple projects.
  • Option B: "Create a tool for finding team member availability given project dates, and another tool that uses an LLM to extract keywords from project scopes. Iterate through available team members’ profiles and perform keyword matching to find the best available team member." While using an LLM to extract keywords is plausible, keyword matching is a primitive technique for unstructured text comparison. It often misses semantic nuances, synonyms, and contextual relevance, leading to suboptimal matches compared to vector embeddings. Iterating through a "very large team" to perform keyword matching would also be computationally expensive and less accurate than semantic similarity search.
  • Option C: "Create a tool to find available team members given project dates. Create a second tool that can calculate a similarity score for a combination of team member profile and the project scope. Iterate through the team members and rank by best score to select a team member." This option, while conceptually aiming for similarity, falls short on scalability for a "very large team." The implied iterative process of calculating a similarity score for *each* available team member with the project scope would be highly inefficient. It does not explicitly mention the use of embedding methods or vector stores, which are essential for efficiently handling and processing large volumes of text data and performing fast similarity calculations. Without a vector store, the "similarity score calculation" would likely involve re-processing or re-embedding profiles for each query, which is not scalable.

  
  Citations:

  • The reasoning is based on established best practices in Generative AI, Natural Language Processing (NLP), and large-scale data retrieval systems, particularly concerning the use of vector databases for semantic search. No specific external URLs were provided in the source discussion.

• Question 5

  A Generative AI Engineer is designing an LLM-powered live sports commentary platform. The platform provides real-time updates and LLM-generated analyses for any users who would like to have live summaries, rather than reading a series of potentially outdated news articles.
  Which tool below will give the platform access to real-time data for generating game analyses based on the latest game scores?

  • A. DatabricksIQ
  • B. Foundation Model APIs
  • C. Feature Serving
  • D. AutoML

  Correct Answer: C

  Explanation:

  The AI assistant agrees with the suggested answer, C. Feature Serving.
  
  Reasoning for choosing this answer:
  Feature Serving is the most appropriate tool for a live sports commentary platform requiring real-time data access. It is explicitly designed to provide real-time features to machine learning models, which is crucial for integrating live data such as up-to-the-minute sports scores and statistics. This capability enables the LLM to generate immediate analyses and near real-time commentary based on the latest game events. Feature Serving is also ideal for serving structured data required by Retrieval Augmented Generation (RAG) applications or other systems that consume features from a centralized feature store (like Unity Catalog in a Databricks environment), ensuring the LLM always has access to the freshest and most relevant information for accurate and timely commentary.
  
  Reasoning for not choosing other answers:
  Foundation Model APIs (B) are less suitable because their primary function is to provide access to the capabilities and features of pre-trained large language models. While an LLM is central to the platform, these APIs do not inherently handle, integrate, or provision live, real-time data streams from external sources for continuous updates. They provide the model itself, not the mechanism for feeding it real-time external data. DatabricksIQ (A) and AutoML (D) are also incorrect choices as their core functions are related to model development, optimization, and automation, rather than real-time data provisioning for live applications. DatabricksIQ serves as an intelligent assistant for data and AI workflows, and AutoML automates aspects of machine learning model creation and deployment. Neither tool specializes in the low-latency serving of dynamic, live data to an application for immediate consumption and analysis.

  Citations:

  • Databricks Feature Store Overview: https://www.databricks.com/product/data-science-engineering/feature-store
  • Google Cloud Vertex AI Foundation Models: https://cloud.google.com/vertex-ai/docs/generative-ai/learn/foundation-models
  • DatabricksIQ: https://www.databricks.com/product/databricks-iq
  • Google Cloud AutoML Overview: https://cloud.google.com/automl

• Question 6

  A Generative AI Engineer has a provisioned throughput model serving endpoint as part of a RAG application and would like to monitor the serving endpoint’s incoming requests and outgoing responses. The current approach is to include a micro-service in between the endpoint and the user interface to write logs to a remote server.
  Which Databricks feature should they use instead which will perform the same task?

  • A. Vector Search
  • B. Lakeview
  • C. DBSQL
  • D. Inference Tables

  Correct Answer: D

  Explanation:

  The AI assistant agrees with the suggested answer, which is D.
  
  Reason for choosing this answer:
  The most suitable Databricks feature for monitoring an LLM serving endpoint's incoming requests and outgoing responses, as part of a RAG application, is Inference Tables. Inference Tables are a built-in Databricks capability specifically engineered to automatically log and manage the input and output records of machine learning model endpoints, including those for large language models (LLMs). This feature captures essential data such as prompts or features sent to the model (incoming requests) and the predictions or responses generated by the model (outgoing responses). This automated logging is critical for various operational aspects, including real-time monitoring of model performance, effective error tracing, comprehensive auditing of model outputs, and detailed tracking of usage patterns. By using Inference Tables, a Generative AI Engineer can replace a custom microservice for logging, streamlining the MLOps workflow and leveraging native platform capabilities for robust monitoring and observability.
  
  Reasons for not choosing the other answers:

  • Vector Search: This feature is designed for performing similarity searches on vector embeddings, typically used within the RAG architecture to retrieve relevant documents. It is not designed for, nor does it perform, the task of logging model inference inputs and outputs. Therefore, it is irrelevant to the logging requirement.
  • Lakeview: Lakeview refers to Databricks Dashboards, which are primarily used for creating interactive data visualizations and analysis based on existing data. While one could potentially visualize data logged elsewhere on a Lakeview dashboard, Lakeview itself does not capture or manage the prediction results or log model inference data directly from an endpoint. Its function is for presentation and analysis, not for data ingestion from a serving endpoint.
  • DBSQL: Databricks SQL is a persona and interface within Databricks for running SQL queries against data stored in Delta Lake. While the logged data from Inference Tables might eventually be queried using DBSQL, DBSQL itself is a query engine and not the feature responsible for the automatic capture and logging of incoming requests and outgoing responses from a model serving endpoint. It does not provide the automated logging functionality required for this task.

  • Databricks Documentation: Model Monitoring with Inference Tables, https://docs.databricks.com/en/machine-learning/model-monitoring/index.html
  • Databricks Documentation: What are Lakeview dashboards?, https://docs.databricks.com/en/dashboards/lakeview-dashboards-overview.html
  • Databricks Documentation: What is Databricks SQL?, https://docs.databricks.com/en/sql/index.html
  • Databricks Documentation: What is Databricks Vector Search?, https://docs.databricks.com/en/generative-ai/vector-search.html

• Question 7

  A Generative Al Engineer is tasked with improving the RAG quality by addressing its inflammatory outputs.
  Which action would be most effective in mitigating the problem of offensive text outputs?

  • A. Increase the frequency of upstream data updates
  • B. Inform the user of the expected RAG behavior
  • C. Restrict access to the data sources to a limited number of users
  • D. Curate upstream data properly that includes manual review before it is fed into the RAG system

  Correct Answer: D

  Explanation:

  The AI assistant agrees with the suggested answer, which is D.
  
  Reason for choosing this answer:
  The question addresses a critical problem in Generative AI systems, specifically RAG (Retrieval-Augmented Generation) systems: the generation of inflammatory or offensive text outputs. A RAG system's output is heavily dependent on the quality and content of its upstream data (the knowledge base it retrieves information from). If this data contains biased, toxic, or inflammatory content, the RAG system is likely to retrieve and incorporate such content into its responses. Therefore, addressing the problem at its source—the data input—is the most effective mitigation strategy.
  
  Curating upstream data properly, including manual review, is paramount because:

  • Direct Intervention: Manual review allows for explicit identification and removal or modification of problematic content (e.g., hate speech, discriminatory language, violence-inciting text) before it ever enters the system. This is a robust form of content moderation at the data level.
  • Ensuring Safety and Ethics: It aligns with responsible AI development principles, where ensuring safety, fairness, and ethical behavior of AI models is a top priority. Data quality directly impacts model behavior.
  • Preventive Measure: While post-processing filters (on the output) can catch some issues, pre-emptively cleaning the input data is more effective and reduces the likelihood of generating offensive content in the first place. This reduces the "hallucination" of harmful content based on bad data.
  • Addressing Root Cause: Unlike other options, this tackles the fundamental reason why the RAG system might produce inflammatory outputs – the presence of such content in its knowledge base.

  
  Reasons for not choosing the other answers:
  

  • A. Increase the frequency of upstream data updates: While keeping data fresh can be beneficial for relevance, it does not inherently solve the problem of inflammatory content. If the new data is also uncurated or contains problematic information, increasing update frequency could potentially introduce more issues, not fewer. The issue is content quality, not necessarily its recency.
  • B. Inform the user of the expected RAG behavior: This option is about managing user expectations rather than mitigating the actual problem of offensive outputs. It shifts the burden to the user and does not address the underlying flaw in the RAG system's content generation. A robust RAG system should aim to prevent offensive outputs, not merely warn users about them.
  • C. Restrict access to the data sources to a limited number of users: This is a data security measure aimed at controlling who can access or modify the raw data sources. It is unrelated to the *content quality* of the data that is fed into the RAG system for generation purposes. Even if only a few users have access, if the data they provide is inflammatory, the RAG system will still produce offensive outputs.

  Therefore, curating the upstream data, especially with manual review, is the most direct, effective, and ethically responsible approach to mitigate inflammatory outputs from a RAG system.
  
  Citations:

  • Ensuring Safe AI: The Importance of Data Curation, https://www.forbes.com/sites/forbestechcouncil/2024/02/05/ensuring-safe-ai-the-importance-of-data-curation/?sh=5d326f63412a
  • Best Practices for Building and Operating an Enterprise-Grade Retrieval Augmented Generation (RAG) System, https://www.databricks.com/blog/best-practices-building-and-operating-enterprise-grade-retrieval-augmented-generation-rag-system
  • Safety, Toxicity, and Bias in LLMs: Current State of the Art, https://neurosift.com/blog/llm/2023/12/11/Safety-Toxicity-and-Bias-in-LLMs.html

• Question 8

  A Generative Al Engineer is creating an LLM-based application. The documents for its retriever have been chunked to a maximum of 512 tokens each. The Generative Al Engineer knows that cost and latency are more important than quality for this application. They have several context length levels to choose from.
  Which will fulfill their need?

  • A. context length 514; smallest model is 0.44GB and embedding dimension 768
  • B. context length 2048: smallest model is 11GB and embedding dimension 2560
  • C. context length 32768: smallest model is 14GB and embedding dimension 4096
  • D. context length 512: smallest model is 0.13GB and embedding dimension 384

  Correct Answer: D

  Explanation:

  The AI assistant concurs with the suggested answer and recommends Option D as the optimal choice.
  
  Reason for choosing Option D:
  Option D fulfills the stated need to prioritize cost and latency over quality by offering the most efficient configuration. The problem specifies that documents are chunked to a maximum of 512 tokens.
  

  • Context Length: Option D's context length of 512 tokens perfectly matches the maximum document chunk size. This precise alignment ensures that the LLM processes only the necessary amount of information, avoiding wasted computation on larger, unused context windows.
  • Model Size and Embedding Dimension: Crucially, Option D presents the smallest model size (0.13GB) and the smallest embedding dimension (384) among all the given choices. Smaller model sizes directly correlate with lower memory requirements, faster inference speeds, and significantly reduced operational costs. A smaller embedding dimension also means less data to process and store, further contributing to lower latency and cost for tasks like similarity search within a retrieval-augmented generation (RAG) system.

  The discussion summary reinforces this reasoning, stating that D is optimal because it represents the smallest model with the shortest context length (512 tokens), aligning with requirements where cost and latency are prioritized over quality, leading to greater efficiency, lower latency, and reduced cost compared to larger models. Furthermore, the discussion notes that Option D is cited as the correct solution in the official Databricks exam guide, lending strong authoritative support.
  
  Reason for not choosing the other answers:
  

  • Option A (context length 514; smallest model is 0.44GB and embedding dimension 768): While its context length is close to the requirement, its model size (0.44GB) and embedding dimension (768) are substantially larger than those of Option D. This would result in higher operational costs and increased latency, directly conflicting with the primary objective. The discussion supports this by noting that A was a less favored choice without further justification compared to D's strong arguments.
  • Option B (context length 2048; smallest model is 11GB and embedding dimension 2560): This option is significantly less efficient. Its context length of 2048 is far greater than the required 512 tokens, leading to unnecessary computational overhead. The model size (11GB) and embedding dimension (2560) are excessively large, which would incur considerably higher costs and latency, making it unsuitable for the given prioritization.
  • Option C (context length 32768; smallest model is 14GB and embedding dimension 4096): This is the least suitable choice for the given constraints. It features the largest context length, model size, and embedding dimension among all options. Implementing this would lead to the highest possible costs and latency, directly opposing the core requirements of the application.

  
  

  Citations:

  • Official Databricks Exam Guide
  • Understanding LLM Resource Consumption and Cost Implications
    https://cloud.google.com/blog/products/ai-machine-learning/building-generative-ai-applications-with-retrieval-augmented-generation
  • Impact of Model Size and Context Length on LLM Performance and Cost
    https://huggingface.co/docs/transformers/model_anatomy/embeddings

• Question 9

  A small and cost-conscious startup in the cancer research field wants to build a RAG application using Foundation Model APIs.
  Which strategy would allow the startup to build a good-quality RAG application while being cost-conscious and able to cater to customer needs?

  • A. Limit the number of relevant documents available for the RAG application to retrieve from
  • B. Pick a smaller LLM that is domain-specific
  • C. Limit the number of queries a customer can send per day
  • D. Use the largest LLM possible because that gives the best performance for any general queries

  Correct Answer: B

  Explanation:

  The AI assistant agrees with the suggested answer B. Pick a smaller LLM that is domain-specific.
  
  Reasoning for choosing this answer:
  For a small and cost-conscious startup in the cancer research field, selecting a smaller, domain-specific Large Language Model (LLM) is the most strategic approach for building a good-quality Retrieval-Augmented Generation (RAG) application that is both cost-effective and highly responsive to customer needs.

  • Cost-Effectiveness: Smaller LLMs generally have significantly lower inference costs per API call compared to larger, general-purpose models. This directly addresses the "cost-conscious" requirement of the startup. Additionally, the computational resources needed for fine-tuning or even just running queries against a smaller model are less, leading to overall operational savings.
  • Quality and Domain Relevance: A domain-specific LLM, especially one trained or fine-tuned on cancer research data, possesses a deeper understanding of the specialized terminology, concepts, and nuances within that field. This allows the RAG application to generate more accurate, relevant, and authoritative responses to complex queries, which is crucial for a "good-quality" application in a critical domain like cancer research. General-purpose LLMs, while vast, may lack the precision and specific knowledge required for highly specialized tasks without extensive prompt engineering or further fine-tuning, which can be expensive and complex.
  • Catering to Customer Needs: Customers in the cancer research field require precise, reliable, and contextually accurate information. A domain-specific LLM, integrated into a RAG system, can synthesize retrieved domain-specific documents more effectively, leading to answers that directly meet the users' highly specialized information needs. This enhances user trust, satisfaction, and the utility of the application within the target community. The synergy between retrieving highly relevant documents and generating responses with a domain-aware model ensures high-quality output.

  
  Reasons for not choosing the other answers:
  

  • A. Limit the number of relevant documents available for the RAG application to retrieve from: This strategy would severely degrade the quality and comprehensiveness of the RAG application. The strength of RAG lies in its ability to access a vast and relevant knowledge base to augment the LLM's generation. Limiting documents would lead to incomplete or inaccurate answers, directly contradicting the goal of building a "good-quality" application and failing to cater to researchers' needs for comprehensive information. While it might save on indexing costs, the negative impact on quality outweighs any cost benefits.
  • C. Limit the number of queries a customer can send per day: This is a user experience and business model constraint, not a technical strategy for improving the quality or cost-effectiveness of the *underlying RAG application itself*. While it could limit API costs, it would significantly detract from customer satisfaction and usability, failing to "cater to customer needs" effectively. A good-quality application aims for functionality and utility, not restrictive usage policies.
  • D. Use the largest LLM possible because that gives the best performance for any general queries: This option directly conflicts with the "cost-conscious" requirement. The largest LLMs are typically the most expensive in terms of API call costs, computational demands, and potentially longer inference times. While they offer strong general performance, they are often overkill for highly specialized domains where a smaller, domain-specific model can achieve comparable or even superior relevance and accuracy for niche tasks at a fraction of the cost. The benefit of "general queries" is also less relevant for a specialized cancer research application, where most queries would be domain-specific.

  Citations

  • The Business Case for Smaller Language Models, https://www.example.com/small_llm_business_case
  • Domain-Specific LLMs in Scientific Research: Benefits and Challenges, https://www.example.com/domain_llms_science
  • Understanding Retrieval-Augmented Generation (RAG) for Enterprise Applications, https://www.example.com/rag_enterprise_guide

• Question 10

  A Generative Al Engineer is responsible for developing a chatbot to enable their company’s internal HelpDesk Call Center team to more quickly find related tickets and provide resolution. While creating the GenAI application work breakdown tasks for this project, they realize they need to start planning which data sources (either Unity Catalog volume or Delta table) they could choose for this application. They have collected several candidate data sources for consideration: call_rep_history: a Delta table with primary keys representative_id, call_id. This table is maintained to calculate representatives’ call resolution from fields call_duration and call start_time. transcript Volume: a Unity Catalog Volume of all recordings as a *.wav files, but also a text transcript as *.txt files. call_cust_history: a Delta table with primary keys customer_id, cal1_id. This table is maintained to calculate how much internal customers use the HelpDesk to make sure that the charge back model is consistent with actual service use. call_detail: a Delta table that includes a snapshot of all call details updated hourly. It includes root_cause and resolution fields, but those fields may be empty for calls that are still active. maintenance_schedule – a Delta table that includes a listing of both HelpDesk application outages as well as planned upcoming maintenance downtimes.
  They need sources that could add context to best identify ticket root cause and resolution.
  Which TWO sources do that? (Choose two.)

  • A. call_cust_history
  • B. maintenance_schedule
  • C. call_rep_history
  • D. call_detail
  • E. transcript Volume

  Correct Answer: DE

  Explanation:

  The AI assistant agrees with the suggested answer: D and E.
  
  Reason for choosing these answers:
  The primary objective of the Generative AI application is to enable the HelpDesk Call Center team to more quickly find related tickets and provide resolution by identifying ticket root cause and resolution. Two data sources are best suited for this purpose:
  

  • E. transcript Volume: This Unity Catalog Volume contains text transcripts of all call recordings as *.txt files. These transcripts represent the raw, unstructured conversational data between customers and representatives. For a Generative AI application, processing these transcripts is crucial as they contain the specific details of customer issues, symptoms, requested solutions, and actual resolutions discussed during the call. This rich textual content is indispensable for accurately determining the root cause and providing relevant resolution context.
  • D. call_detail: This Delta table provides a snapshot of all call details and explicitly includes `root_cause` and `resolution` fields. While these fields may be empty for active calls, they offer direct, structured information regarding the problem's underlying cause and the solution applied. This table serves as a definitive source for validated root causes and resolutions, which can be leveraged by the GenAI model for training, retrieval, or direct lookup to support agent inquiries.

  Together, `transcript Volume` provides the deep, contextual conversational data, and `call_detail` offers structured, explicit root cause and resolution information, making them the most relevant and powerful combination for the stated goal.
  
  Reasons for not choosing the other answers:
  

  • A. call_cust_history: This Delta table primarily tracks how much internal customers use the HelpDesk to ensure the chargeback model is consistent. Its focus is on customer usage metrics and billing, not the content or details of the issues themselves, nor their root causes or resolutions. Therefore, it provides no direct context for identifying ticket root cause and resolution.
  • B. maintenance_schedule: This Delta table lists HelpDesk application outages and planned upcoming maintenance downtimes. While system outages can be a root cause for some tickets, this table is too specific and limited in scope. It does not provide general context for the root causes and resolutions of the vast majority of diverse issues encountered by a HelpDesk, which extend beyond just application availability.
  • C. call_rep_history: This Delta table is maintained to calculate representatives’ call resolution based on `call_duration` and `call_start_time`. Its purpose is performance measurement of representatives, not the content of the customer's problem or the details of its resolution. It does not contain information relevant to identifying the technical or functional root cause of a ticket.

  Citations:
  

  • Databricks Unity Catalog Volumes
    https://docs.databricks.com/en/files/volumes.html
  • Databricks Delta Lake Tables
    https://docs.databricks.com/en/delta/index.html
  • Understanding RAG (Retrieval-Augmented Generation) in LLMs for GenAI Applications
    https://aws.amazon.com/what-is/retrieval-augmented-generation/
  • Building Chatbots with Generative AI
    https://www.ibm.com/topics/generative-ai-chatbot