Research project · +16% accuracy · 2025 - 2026
Fine-tuning plus dynamic RAG on a quantized Llama-3-8B for cultural question answering: 16% more accurate than the zero-shot baseline.
A research project on improving Llama-3-8B's cultural reasoning. LLMs tend to carry cultural alignment biases from Western-dominated training corpora. I built a pipeline that combines dynamic retrieval-augmented generation (RAG) with targeted LoRA fine-tuning, improving short-answer accuracy by 16% over the zero-shot baseline.
Large language models struggle with geographic and cultural nuance because of representation gaps in their pre-training corpora. On top of that, smaller open-weights models around 8B parameters drift away from complex zero-shot formatting instructions, which breaks automated evaluation.
Replaced the brittle formatting constraints with natural instruction layouts and built a dynamic few-shot retrieval framework. The reference corpus is restructured into clean query-response pairs, and at inference time the pipeline retrieves the three most semantically relevant exemplars and injects them into the prompt.
The combination of LoRA fine-tuning and dynamic RAG beat the baseline models on the strict verification metrics, with a 16% absolute accuracy improvement on the short-answer evaluation suite.
Llama-3-8B with 4-bit quantization for efficiency, MiniLM-L6-v2 for semantic embedding and retrieval. A data augmentation stage strips multiple-choice options from the training data to create direct query-response pairs. Inference uses greedy search for deterministic, concise outputs that satisfy the evaluation script.
Injects the most semantically relevant in-context examples per query, driving the 16% short-answer accuracy gain.
Automated transformation of multiple-choice datasets into direct query-response pairs, doubling the retrieval corpus.
LoRA plus 4-bit quantization made fine-tuning the 8B model feasible on limited hardware.
Tested external search integration and found that clean internal data beats noisy web results for this domain.
Set up the model environment with parameter-efficient adapters and ran the baselines.
Implemented the few-shot retrieval structures and compiled the retrieval database.
Tested decoding parameters and search API dependencies.
Wrote up the analysis documenting the 16% benchmark improvement.
