Research project · +16 pp SAQ accuracy · 2025 - 2026
A quantized Llama-3-8B cultural QA pipeline using LoRA fine-tuning and dynamic few-shot retrieval, improving short-answer accuracy by 16 percentage points.
A course research project on cultural question answering with Llama-3-8B. I simplified brittle output prompts, converted multiple-choice training examples into direct question-answer pairs, fine-tuned the model with LoRA, and retrieved the three closest examples for each query. Short-answer accuracy rose from 0.49 to 0.65, while multiple-choice accuracy remained unchanged.
Large language models struggle with geographic and cultural nuance because of representation gaps in their pre-training corpora. On top of that, smaller open-weights models around 8B parameters drift away from complex zero-shot formatting instructions, which breaks automated evaluation.
Replaced the brittle formatting constraints with natural instruction layouts and built a dynamic few-shot retrieval framework. The reference corpus is restructured into clean query-response pairs, and at inference time the pipeline retrieves the three most semantically relevant exemplars and injects them into the prompt.
The final pipeline improved short-answer accuracy from 0.49 to 0.65. Dynamic retrieval delivered the largest gain, while external web search reduced accuracy by three to five percentage points and was dropped from the final system.
Llama-3-8B with 4-bit quantization for efficiency, MiniLM-L6-v2 for semantic embedding and retrieval. A data augmentation stage strips multiple-choice options from the training data to create direct query-response pairs. Inference uses greedy search for deterministic, concise outputs that satisfy the evaluation script.
Embeds each query with MiniLM and injects the three closest examples into the inference prompt.
Automated transformation of multiple-choice datasets into direct query-response pairs, doubling the retrieval corpus.
LoRA plus 4-bit quantization made fine-tuning the 8B model feasible on limited hardware.
Compared greedy, beam, and self-consistent decoding, then tested web retrieval against the curated internal corpus.
Set up the model environment with parameter-efficient adapters and ran the baselines.
Implemented the few-shot retrieval structures and compiled the retrieval database.
Tested decoding parameters and search API dependencies.
Wrote up the analysis documenting the 16% benchmark improvement.
Swipe or open to viewScroll, drag, or open to view
