---
title: "Prompting Strategies for AI Agents"
description: "Practical lessons from writing prompts for an agentic legal research tool."
date: "2026-08-23"
tags: ["ai", "agents", "prompting", "langgraph"]
---

# Prompting Strategies for AI Agents

The workflow for the agentic legal research tool I'm working on uses six separate LLM calls. They route questions, rewrite follow-ups, retrieve sources, draft answers, check their grounding, and respond to non-legal conversations.

Each LLM call has its own prompt, and I learned quite a few practical strategies from building and debugging those prompts.

## 1. Give each prompt one job

I find that it is so much easier to test and fix a simple prompt that does one thing rather than one that does a million things. In our case, it is easier to deal with a prompt that only performs query classification than with a more verbose prompt that not only performs classification but also handles searches, citations, answer checking, etc. A simpler prompt is more focused, and the description is also easier to read. Just like us humans, LLMs can become overwhelmed when handling too many things, which might cause some tasks to be handled poorly.

For example, my contextualization step turns a follow-up question such as “What about criminal cases?” into a standalone question using the chat history. That is its whole job, and the next LLM call takes it from there.

```python
_SYSTEM = """You rewrite a follow-up question from a Malaysian legal research chat
into a self-contained search query."""
```

*Excerpt — additional prompt rules follow in the source.*

*Source: [contextualization prompt](https://github.com/aishahsofea/ai-legal-tool/blob/4c96571b56c8c66bbb7d08f8e96ce85f81c44e25/agent/nodes/contextualize.py#L35-L52)*

This pattern is called [prompt chaining](https://www.anthropic.com/research/building-effective-agents): it breaks a complex task into smaller calls, with each call using the previous one’s output.

## 2. Return structured data when code needs the answer

The LLM, as we know it, is notoriously non-deterministic. One way around this is to make it more predictable, and we can do that by defining the structure the LLM should use for its response. According to OpenAI, schema-enforced [Structured Outputs](https://developers.openai.com/api/docs/guides/structured-outputs) are more reliable than good old plain JSON.

My router, for example, returns a defined category and supporting fields. This way, responses are more deterministic and can be broken down more systematically, compared with sentence-based responses, where parsing the prose is more prone to mistakes or misinterpretation.

```python
class _RouterOutput(BaseModel):
    reasoning: str
    query_type: Literal[
        "statute_lookup", "topical", "provision_extraction",
        "conversational", "clarify",
    ]
    response_language: Literal["en", "bm", "mixed"]
    clarifying_question: str = ""

_structured_llm = _llm.with_structured_output(_RouterOutput)
```

*Source: [router output schema](https://github.com/aishahsofea/ai-legal-tool/blob/4c96571b56c8c66bbb7d08f8e96ce85f81c44e25/agent/nodes/router.py#L30-L39)*

## 3. Turn chain of thought into structured evidence

Next, we can take it up a notch by organizing the output fields so that the response follows a clear chain of thought. This prompting strategy guides the model through a task by giving it a set of reasoning steps to follow before producing an answer. This can actually be free-form too. For example, a prompt might say, “Think carefully step by step, then give the answer.” The model then writes an open-ended explanation before its conclusion. OpenAI’s [GPT-4.1 prompting guide](https://developers.openai.com/cookbook/examples/gpt4-1_prompting_guide#3-chain-of-thought) shows this pattern.

My grounding checker uses the same idea but in a more constrained way. Instead of asking for a free-form reasoning paragraph, it turns each thought into a field in the structured output:

```python
class _GroundingClaim(BaseModel):
    claim: str
    cited_act_number: str
    cited_section_number: str
    quote: str
    reason: str
    support: Literal["supported", "partial", "unsupported"]
```

The full schema also gives its fields descriptions, which clarify what each value must establish.

The fields create an evidence-first reasoning chain:

1. `claim`: What exact legal claim is being checked?
2. `cited_act_number` and `cited_section_number`: Which source is supposed to support it?
3. `quote`: What short, verbatim passage in that source carries the claim?
4. `reason`: What does the passage cover, and what does it leave unsupported?
5. `support`: Given that evidence, is the claim supported, partially supported, or unsupported?

The structured chain works in the system’s favor. Every result has the same shape. Code can validate the label, confirm that the quote appears in the retrieved source, and attach accepted evidence to the citation receipt. The output is also shorter and easier to audit than an unrestricted reasoning transcript. It captures the reasoning needed to verify the decision without depending on a long free-form account of the model’s hidden thought process.

*Source: [grounding schema and prompt](https://github.com/aishahsofea/ai-legal-tool/blob/4c96571b56c8c66bbb7d08f8e96ce85f81c44e25/agent/nodes/grounding_check.py#L30-L86)*

## 4. Treat tool descriptions as prompts

How does a model decide which tool to trigger? You guessed it: by reading its description. Therefore, it is equally important to ensure that the tool's description is as clear and precise as possible. Anthropic’s guide to [writing tools for agents](https://www.anthropic.com/engineering/writing-tools-for-agents) shows how precise descriptions improve tool selection.

The description for my `follow_references` tool says when it is allowed, when it is unnecessary, and how often it may run. Keeping those rules next to the tool helps the model apply them at the moment of choice.

```python
"""Follow direct, published statutory references from one exact retrieved anchor.

Use this only for explicit reference intent (what a provision refers to, is
subject to/notwithstanding, what refers to it, or a definition explicitly
located elsewhere), and only after ``lookup_section`` or ``search_statutes``
has returned the anchor section. Never use it for an ordinary exact lookup,
a topical/broad question, or as a default second search. One call is allowed
per retrieval run; results are one hop and at most five published edges.
"""
```

*Source: [full tool description](https://github.com/aishahsofea/ai-legal-tool/blob/4c96571b56c8c66bbb7d08f8e96ce85f81c44e25/agent/retrieval/tools.py#L117-L144)*

## 5. Define a stopping point

An agent that can search needs to know when searching is no longer useful. My retrieval prompt allows one reformulated search, then tells the agent to stop. A recursion limit in code provides a final safety net.

```python
RECURSION_LIMIT = int(os.getenv("RETRIEVAL_RECURSION_LIMIT", "6"))
```

```text
If a search returns no sections or the results look off-topic, call
`search_statutes` again ONCE with a reformulated query (broader wording or
different keywords). Do not keep searching indefinitely.

Stop as soon as you have relevant sections.
```

*Source: [retrieval limits](https://github.com/aishahsofea/ai-legal-tool/blob/4c96571b56c8c66bbb7d08f8e96ce85f81c44e25/agent/retrieval/agent.py#L36-L55)*

## 6. Resolve ambiguous cases explicitly

If two labels could fit, state which one should win.

Without a tie-break, the model must invent its own policy, and its choice may change between similar requests. This is why advice to [be direct and explicit](https://platform.claude.com/docs/en/build-with-claude/prompt-engineering/claude-prompting-best-practices#be-clear-and-direct) should include concrete decision rules.

```text
IMPORTANT tie-break: only use conversational when the message is UNAMBIGUOUSLY
social or meta. When in doubt — if the message has any legal substance at all —
classify it as one of the three legal types, not conversational.
```

*Source: [router tie-break](https://github.com/aishahsofea/ai-legal-tool/blob/4c96571b56c8c66bbb7d08f8e96ce85f81c44e25/agent/nodes/router.py#L59-L61)*

## 7. Describe what good output looks like

If we instruct the model not to do something, it only knows what to avoid, but it doesn't know exactly what it should do instead, so it can come up with any output format. Anthropic's guidance on [controlling response formats](https://platform.claude.com/docs/en/build-with-claude/prompt-engineering/claude-prompting-best-practices#control-the-format-of-responses) recommends describing the desired result instead.

```text
Write about the provision, not about the reader:
"Section 90A requires...", "the Act provides that...".

Never address the reader's own situation — no "you should", "you must",
"you need to", "you are advised", "in your case", or "I recommend".
```

*Source: [answer-writing rules](https://github.com/aishahsofea/ai-legal-tool/blob/4c96571b56c8c66bbb7d08f8e96ce85f81c44e25/agent/nodes/synthesiser.py#L48-L60)*

The rule is backed by a deterministic supervisor, not left to the writing prompt alone. Its third rule makes the enforcement explicit: “A response containing any of those phrases is rejected and re-drafted, so the reframing has to happen here.”

*Source: [supervisor validation](https://github.com/aishahsofea/ai-legal-tool/blob/4c96571b56c8c66bbb7d08f8e96ce85f81c44e25/agent/nodes/supervisor.py#L14)*

Lead with the desired result, then use restrictions to mark its limits.

## 8. Put long context before the question

This is similar to how human attention works. If you are asked a question and given the information afterwards, you need to keep the question in mind and then recall it after digesting the information. That feels a bit all over the place. But if you're given the information first and understand the context, the query that comes afterwards feels easier to answer.

It's the same with AI. When a prompt contains several retrieved documents and one short question, place the documents first and the question near the final instruction. [Long-context guidance](https://platform.claude.com/docs/en/build-with-claude/prompt-engineering/claude-prompting-best-practices#long-context-prompting) reports better results from this order.

My answer-writing prompt provides the retrieved sections, chat history, and preferences before the query. The query then sits beside the instruction that acts on it.

```python
user_message = f"""Retrieved statute sections:
{context}

Conversation history:
{history_text or '(none)'}
{preferences_block(recalled_memory)}
Query: {state['query']}

Answer the query using only the sections above. Cite each section you rely on."""
```

*Source: [prompt assembly](https://github.com/aishahsofea/ai-legal-tool/blob/4c96571b56c8c66bbb7d08f8e96ce85f81c44e25/agent/nodes/synthesiser.py#L115-L131)*

## Treat these strategies as hypotheses

Most of these lessons began as failures. A vague tool description caused poor choices. An unclear tie-break produced inconsistent routing. A partial ban let unwanted phrasing slip through.

Fixing those problems made the system easier to understand and test. It does not prove that every technique will improve every agent. I still need to compare prompt versions against a varied set of real legal questions.

Treat these strategies as testable design choices, not universal laws. Start with a clear task, define what success looks like, and measure whether each change helps.