Gambia Legal Aid

Gambians have almost no access to legal information. Lawyers are scarce and expensive, a one-hour consultation is more than a week's wage for most. The laws themselves are scattered across PDFs that most people will never find or be able to read. Hallucinated legal advice from a chatbot would be actively harmful: a person told they have rights they don't, or vice versa, in matters of arrest, dismissal, eviction, domestic violence. So the bar wasn't 'build a chatbot'. The bar was: build a chatbot that REFUSES to lie, even at the cost of being less useful.

• Read every Act we ingested (Constitution, Criminal Code, Labour Act, Children's Act, Sexual Offences Act, Domestic Violence Act, Immigration Act, Rent Act 2014/2017/2024, etc.) end-to-end before writing the system prompt. Could not enforce grounding without knowing what grounded actually meant.
• Studied how Westlaw and LexisNexis structure citations. They cite by section number, sub-section, and Act. Realised my retrieval had to surface those exact metadata pieces or the citations would be unverifiable.
• Read the OpenAI 'Constitutional AI' paper and the provider's RAG literature. The pattern that stuck: validate the answer against the retrieved context BEFORE returning it. If the model hallucinated, reject and retry, don't ship and apologise.
• Studied curly-quote vs straight-quote behaviour across browsers and PDFs because the model would generate the curly version of "section 14" when the source text had the straight version, and my substring validator initially missed this and let hallucinations through.
• Read Gambian legal cases on unfair dismissal, domestic violence prosecutions, and immigration appeals to make sure the topic anchors I was building actually retrieved the sections that mattered to real disputes.

• Hallucinated legal advice is worse than no legal advice, could literally harm people who acted on it.
• No budget for managed vector DB. Had to build retrieval cheaply on Supabase pgvector + a TF-IDF layer.
• Statute PDFs are messy: scanned text, inconsistent formatting, mixed heading styles, footnotes baked into body text.
• Mobile-first UI, the audience is mostly phone users on slow connections. Streaming had to feel natural, not like a stuck page.
• Gambian English has its own register; the system had to switch tones depending on whether the user asked a legal question or just chatted.

1. Decision 01

   Groq Llama 3.3 70B as the generation model, not the model or GPT-4.

   Legal Q&A is high-volume (every Gambian who has the link uses it for free) and the cost math has to work. Groq's Llama 3.3 70B is fast (sub-second first token), generous on free tier, and quality is high enough for the format I need (one tight paragraph with citations). The strict system prompt + multi-layer hallucination guard does the heavy lifting; the model itself just needs to be coherent and follow instructions. Cost-engineered for a public-good product.

2. Decision 02

   Multi-layer hallucination guard: section allowlist + verbatim quote validation + banned phrases.

   Standard RAG guards against hallucination by retrieving good context. Mine goes further. After retrieval I build a per-Act allowlist mapping section number to section title. The prompt instructs the model to cite ONLY numbers from this allowlist, AND to quote any direct text VERBATIM as a substring of the retrieved chunks. Post-generation validator: every cited section number is checked against the allowlist; every quoted span is checked as a substring of context. Anything that fails is hard-rejected, the system retries with explicit feedback ('you cited section 15 but only sections 14, 18, 22 are valid; quoted text was not in context'). If the retry also fails, the system refuses to answer rather than ship a bad citation.

3. Decision 03

   Section title allowlist with claim-matching, not just number allowlist.

   An early bug had the model citing the right number but for the wrong reason, 'Section 15 (Powers of tribunal) for a claim about notice periods' when notice was actually Section 14. Numbers matched, semantics didn't. Fixed by passing the full allowlist as `15. Powers of tribunal\n14. Notice of termination\n…` so the model could see WHY each number existed and match the claim to the right title (commit 2102d0e).

4. Decision 04

   TF-IDF ranking + topic anchors + low temperature, layered.

   Pure semantic search returned the chunks closest in meaning to the query. But legal queries often involve specific terms that semantic search smooths over (e.g. 'section 130' as a phrase). I added TF-IDF ranking on top to surface chunks containing the rare terms, plus topic anchors (hand-curated keyword maps like `unfair dismissal, then [Sections 130, 132, 139, 140 of Labour Act]`) to force-include sections that had to be in scope. Low temperature (0.1-0.2) eliminated the model's tendency to creatively combine unrelated chunks. Three orthogonal lenses, layered (commit 88d0e72).

5. Decision 05

   Multi-word anchor filtering when building the SQL OR clause.

   The retrieval query was being built dynamically as `text ILIKE '%anchor1%' OR text ILIKE '%anchor2%'…`. Multi-word anchors like 'unfair dismissal' were getting passed in but Postgres ILIKE treated them as literal strings, fine. The problem: low-signal terms ('the', 'and') were polluting the OR clause and matching everything. Filtered to drop those when high-signal terms were present (commit 05a9f64).

6. Decision 06

   Stream the answer word-by-word AFTER validation, not during generation.

   Standard SSE streaming would write tokens as the model produces them. But if a hallucinated citation appears mid-stream, you can't pull it back, the user already read it. Instead I generate the full answer, validate it, THEN stream it word-by-word for the typewriter effect. Slightly slower start (the user waits ~3s for first token instead of ~1s), but every word that appears is already validated. Trust over speed (commit dc4ae3f).

7. Decision 07

   Banned-phrase list: no 'consult a lawyer', no 'review your contract'.

   These phrases are noise. They're what useless lawyer-bots say to dodge accountability. The user is here BECAUSE they don't have a lawyer. Banned them at the prompt level AND at the post-processing level. If the model slipped them in, the post-processor stripped them. The system has to act AS the lawyer or it fails the brief (commits 5f68053, 61a93b1).

8. Decision 08

   Section number extraction via 'NUMBER. Title' inline regex, not block parsing.

   Statute PDFs vary wildly in heading style. Some have section numbers as `Section 14.`, others as `14.`, others embed them mid-paragraph. Block-based parsing missed half of them. Switched to an inline regex pattern that matches `NUMBER. Title` wherever it appears in the text. Caught more sections, including ones that were headings inside paragraphs (commit f7133d7).

9. Decision 09

   Diagnostic response headers in production, `X-Search-Error`, build IDs.

   When something failed in production, I had no easy way to see why. Vercel logs were behind a paywall I couldn't afford. I added response headers: `X-Search-Error` for retrieval failures, `X-Build-Id` for caching debugging. Every response carries enough metadata that I can debug a user's issue from their browser dev tools alone (commits a361856, c13ecd9).

• First version retrieved top-5 chunks unconditionally and let the model write whatever. Tested it on questions outside the corpus, it confabulated convincingly, citing fake section numbers with fake quotes. Built the citation-anchor + verbatim-quote check as the gate. Pass rate dropped sharply, trust went up. Ship the system that says 'I don't know' over the one that confidently lies.
• Initial chunking was naive 1000-char windows. A user asked about marriage law and got back chunks from a tax statute that happened to share a phrase. Switched to section-aware chunking: parse the statute's section markers, chunk on them, each chunk is a self-contained legal unit (commit 22f5c6f).
• Curly quotes vs straight quotes broke the validator silently. the model would output the curly version of a quoted span and the substring check against straight-quote source text would fail to match. Tightened the regex to normalise both flavours and added 3-word span detection (commit 8cb717a). The bug was invisible in dev (where I copy-pasted source) and only appeared in production where the model's outputs differed from my inputs.
• Added a hard-fail validation that REFUSES to ship the answer if the retry also produces hallucinated citations. Logging shows the user 'I couldn't find this in the statutes I have' instead of returning a confidently wrong answer. The metric I optimise is 'lies shipped per million queries', hard fail is the only way to drive that to zero (commit 5d5a1e4).
• Surfaced 400-status messages in the UI. Originally the client showed 'Something went wrong' for every failure. Now legal-validation rejection is shown as 'I couldn't ground this in the statutes I have' which is a different message from network failure. Honest UX is harder than generic UX (commit 5f68053).

• I did not understand TF-IDF deeply enough at the start. Pure semantic retrieval was failing on rare-but-critical terms (specific section numbers, specific named provisions). Spent a weekend reading sklearn's TF-IDF implementation, then wrote my own minimal version that combined cosine similarity from embeddings with sparse TF-IDF scores. The hybrid retrieves better than either alone.
• I had never written a hallucination guard before. First version was just 'check each cited section number against the source'. That caught 30% of bugs. Iteratively added: verbatim quote check, section title matching, banned phrase post-processing, hard-fail on retry, diagnostic headers. Each layer caught a class of failure the previous layers missed.
• I didn't know how messy real statute PDFs would be. Footnotes inline in body text, OCR errors ('s.l4' instead of 's.14'), inconsistent section numbering across acts. Wrote a per-Act ingestion script that normalised each one's quirks before chunking. It's not generalisable, every act needed its own preprocessing rules. The honest answer is: legal-doc ingestion is a manual job pretending to be an automated one.
• I underestimated how aggressively users would test the system. Within a week of deploying, someone asked 'what does the constitution say about [made up topic]?', the model used to confabulate. Adding the hard-fail was the response to that. The lesson: every confident answer is implicitly a contract. Break it once and the whole system loses trust.

• Live RAG chatbot answering Gambian legal questions
• 12+ Acts ingested (Constitution, Criminal Code, Labour Act, Children's Act, Sexual Offences, Domestic Violence, Immigration, Rent 2014/2017/2024)
• Multi-layer hallucination guard: section allowlist + verbatim quote validation + banned phrases + hard-fail on retry
• TF-IDF ranking + topic anchors + low temperature, layered with semantic search
• Streamed answers, validated before display
• Diagnostic response headers in production for debugging without paid logs
• Mobile-optimised, works on slow connections
• Refuses to answer rather than hallucinate, measurable: zero shipped citations to non-existent sections

I'd add multilingual support, Wolof, Mandinka, Fula, Jola. Most of my target users speak those before English. The retrieval layer can stay English (the source statutes are English) but input/output translation around it would dramatically expand the audience. Especially for women in rural areas dealing with the Domestic Violence Act, who often don't have written English literacy.