Game of Words, Episode I

Ever wondered what would have happened if Tyrion Lannister had a chance to talk to Eddard Stark about the events that unfolded during the first book/season?

So have I.

But since George Martin wouldn't write that for me, and my own writing skills are so abysmal, I've decided to create two agents to take the roles of both characters and have a conversation. Here is what I got:

Eddard: I will do my part, Tyrion. The North will not falter. We must prepare for what lies ahead.

Tyrion: I will do my best, Eddard. The realm must understand the urgency. The Wall is not just the North's problem, it's the realm's.

Eddard: We must, Tyrion. The realm's survival depends on it. I'll rally the North. We've faced graver threats and stood tall.

Tyrion: Indeed, Eddard. We must stand united. I'll speak to the court.

Eddard: I trust in your words, Tyrion. The court must see reason. The North will stand, but we cannot do it alone.

Tyrion: I appreciate your trust, Eddard. The court's understanding is crucial. We must stand together.

Eddard: We stand together, Tyrion. The North remembers, and we will not falter. But the court must see reason.

Tyrion: I will do my utmost, Eddard. The court must see the Wall's importance. We are all in this together.

Eddard: I have faith in your abilities, Tyrion. The court must understand the gravity of the situation. The North will stand, but we cannot do it alone.

Tyrion: I will do my utmost, Eddard. The court must see the Wall's importance. We are all in this together.

Eddard: I have faith in you, Tyrion. The court must understand, for the North cannot stand alone.

And so it goes, on and on and on. They conversation just never seems to take off!

So it seems that current LLMs models are simply not equipped with whatever it takes to have an actual conversation. So how can we ever find out what Tyrion would have told Eddard before it was too late?

To answer that question, I have taken inspiration from two big ideas in the field in linguistics called Discourse Analysis: Common Ground (due to Robert Stalnaker’s 1978 paper "Assertion" ), and Question Under Discussion (due to Craige Roberts’ 1996 paper "Information structure: Towards an integrated formal theory of pragmatics" ).

Common Ground (CG)

The Common Ground ("CG") is the public information in the discourse, available to all participants, and the idea is that everything we say is compared to it in some way. It includes everything that was said in the discourse up to that point, as well as anything that everyone knows that everyone knows. This ranges from common knowledge, like “Paris is the capital of France”, to personal stuff participants know about each other, like “Joe is married to Lucy”.

In a normal, reasonable, cooperative conversation, speakers don’t just say things at random. There are certain patterns. The most general way to think about it is through relevance – any new information you add must relate in some way to what has already been established. And the CG serves as the base of information to which new content is compared in order to assess its relevance.

What exactly is relevance, and how do we calculate it based on the CG? That’s a topic for another article, so for now I’ll ask you to rely on your intuitive understanding of the concept of relevance as anything enriches the CG by tying loose ends (answering unresolved questions) or drawing connections between ideas already present. Relevant information is neither redundant (something that’s already in the CG), nor outlandish (something that in order to be understood requires lots of background information/assumptions not present in the CG). It lives in a hard-to-define sweet spot of relating to what has already been said without being trivial or repetitive.

As a simple example, consider the following toy discourse:

Person A: "Do you think we should bring extra water for the hike?"

Person B: "The weather forecast says it's going to be exceptionally hot tomorrow."

Is B’s utterance relevant to A’s? On the face of it not, because B doesn’t address the question about water. But if we make the very reasonable assumption that part of the CG is that the higher the temperature is the greater the need for water, B in fact does answer A’s question indirectly. By combining B’s added info to the info in the CG about the relationship between temperature and water, we get to the conclusion that yes, they should bring extra water.

Question Under Discussion (QUD)

The Question Under Discussion ("QUD") is a natural development of the idea of CG, and it simply refers to the abstract idea of a question that participants in a discourse are currently addressing with their utterances. The QUD is what’s at issue right now. It doesn’t have to be explicitly mentioned, because speakers are generally pretty good at inferring it implicitly from what was recently said.

A classic and simple example is when asking someone on the street for directions. Then the QUD is simply something like “where’s the nearest bank?”. Each utterance in the discourse then relates that question either by adding to the CG information that helps resolve it or by asking follow-up questions. When the CG contains enough information to resolve a QUD, either the discourse ends or participants move on to the next QUD. So a QUD is like a compass for speakers which guides them in choosing the next thing to say, with the general guideline of “try to improve our understanding of this question”.

Naturally, in the course of attempting to resolve a QUD, sub-QUDs might open. For example, in the bank example, if the person you stopped on the street tells you “it is on White Street”, a reasonable follow-up question would be “where is White Street?”, which opens a new QUD that has to be resolved before the larger one can be resolved.

So to recap how CG and QUD work together, a QUD guides speakers in what they have to address with their current utterance, and a CG limits what is relevant to say by keeping track of what your utterance has to relate to in order to make sense in the conversation.

Setup

I’ve created two ChatGPT-based agents, one for Tyrion and another for Eddard, and instructed them to have a friendly conversation with each other. Each agent is a pipeline of two prompts, one for picking a QUD based on the CG (=the conversation history so far), and another to generate a response that addresses that QUD.

I’ve posted the full code here , but I will also give you the essence of the QUD-picking prompt below. First, I’ve given the instruction to identify the QUD scenario based on the other agent’s last response. In the output, this is separated by the actual QUD by a | symbol. The scenarios are: either keep the last QUD, address something the other agent said, or create a new QUD by applying some pre-randomized strategy to it (such as generalize, contradict, abstract, etc’).

 If you conclude that the CG does not contain enough facts to resolve the current QUD,
and that you or <OTHER_NAME> might have access to such facts,
return: <keep>.

If <OTHER_NAME> just said something particularly interesting or surprising that you want to address it by gaining a better understanding of it or give your perspective about it,
Generate a new QUD that addresses it. Return the following: address | <new_qud>, where "<new_qud" is the new QUD.
This should happen if what <OTHER_NAME> just said is unusually surprising or raises consequential questions.

If <OTHER_NAME> just said something that, when combined with some piece of info in your KB, yields a novel conclusion, you should prioritize addressing it.
Especially if this hasn't been resolved yet.

If you have no new facts to contribute to the current QUD,
and you can't think of any more relevant info you can ask <OTHER_NAME> of about it,
change the QUD using the following strategy:

<STRATEGY>>

And return the new QUD.

Note that this is by far not the only way to do this. It’s a specific approach that I’ve chosen based on my experimentation, and while adhering as much as possible to the principles of CG and QUD as developed in the pragmatics literature.

Result Samples

The first sample demonstrates what kind of topics can come out of using this setup. I let them start with the QUD “Family and Upbringing”, and this is what they eventually started talking about:

Tyrion: Indeed, Eddard. Our upbringings have shaped us in unique ways. But, have you ever wondered how different our lives would have been if we were raised in each other's households?

Eddard: That's an interesting thought, Tyrion. I believe our core values would remain the same, but our methods might have been different. How do you think you would have fared in Winterfell?

Tyrion: I believe I would have learned the value of honor and integrity much earlier, Eddard. But the harsh winters of the North would have been a challenge for me.

Eddard: The North does have a way of toughening one up, Tyrion. But I believe you would have found warmth in the company of my children. How do you think your wit and intellect would have influenced them?

Not only are they able to converge into genuinely interesting questions, they can also develop them by asking more specific sub-QUDs.

The second sample demonstrates how the notion of QUD and CG can nicely interact to move the conversation from the general to the specific and then expand it to another case. In this and the next sample, I will give the QUD before the response, showing the work of the QUD-picking prompt.

Tyrion:
(QUD: How do our experiences with injustice influence our decisions in the face of danger?)
Response: A treacherous path indeed, Eddard. Yet, it's the pursuit of truth that defines us, isn't it?

Eddard:
(QUD: How do our experiences with injustice shape our principles and actions?)
Response: Indeed, Tyrion. My pursuit of truth regarding Jon Arryn's death led me into a trap. It's a treacherous path, but one I'd tread again for justice.

Eddard:
(QUD: How do our experiences with injustice influence our decisions in the face of danger?)
Response: Indeed, Tyrion. The pursuit of truth is a defining path, but it's fraught with danger. I was betrayed by those I thought I could trust.

Tyrion:
(QUD: What were the consequences of Eddard's betrayal?)
Response: Betrayal is a bitter draught, Eddard. It's a poison that seeps into trust, turning allies into enemies. Who betrayed you?

Eddard:
(QUD: What were the consequences of Eddard's betrayal?)
Response: Petyr Baelish, a man I thought a friend, promised me the City Guard's support. He lied.

Tyrion:
(QUD: What were the circumstances leading to Petyr Baelish's betrayal?)
Response: Baelish's treachery is a harsh lesson, Eddard. His ambition often clouds his loyalty.

Eddard:
(QUD: What were the circumstances leading to Petyr Baelish's betrayal?)
Response: Indeed, Tyrion. His ambition was his compass, not loyalty or friendship. It's a harsh truth, isn't it?

Tyrion:
(QUD: What were the reasons behind Baelish's ambition?)
Response: A harsh truth indeed, Eddard. Ambition can be a dangerous master, don't you think?

Eddard:
(QUD: What were the reasons behind Baelish's ambition?)
Response: Indeed, Tyrion. Ambition can blind us, leading us down treacherous paths. Have you ever been led astray by your own ambitions?

Tyrion:
(QUD: What were the reasons behind Baelish's ambition?)
Response: Indeed, Eddard. My ambition to prove my worth to my father led me to the battlefield, a place ill-suited for a dwarf. Have you ever regretted a decision driven by ambition?

Eddard:
(QUD: What were the reasons behind Baelish's ambition?)
Response: Yes, Tyrion. My ambition to serve Robert as Hand led me to a dangerous game of thrones. It's a decision I've come to regret. Have you ever felt regret for your ambitions?

Tyrion:
(QUD: What were the consequences of your ambitions?)
Response: Yes, Eddard. My ambition to be seen as more than a dwarf has often led to regret. But it's also shaped me. Has ambition shaped you, Eddard?

Notice how Eddard gets a general QUD and addresses it by sharing a relevant piece of specific information. Then, later, notice how Tyrion decides to address Eddard’s statement about betrayal and opens a new QUD, “What were the consequences of Eddard's betrayal?”. They proceed to discuss this specific QUD, with Tyrion addressing subsequent aspects of it with more sub-QUDs. Eventually, when he feels the CG is rich enough to address these earlier QUDs, Eddard chooses to domain-shift the QUD from Baelish’s ambitions to Tyrion’s own ambitions. This shows that the evolution of QUDs can move in multiple axes (specific-abstract, Petyr-Tyrion).

The next sample demonstrates that they can move both ways in these axes. A conversation about their personal experiences sparks a discussion about how to alert the king’s court to the reality of the threat beyond the wall. The reason I find this remarkable is that they come up with an actual plan, breaking it down into problems, and address each of them in turn. They then loop back to their personal experiences and before tying it back to the plan they had developed.

Eddard:
(QUD: What truths have you discovered in your readings, Tyrion?)
Response: Indeed, Tyrion. The tales of the Night's Watch are not just tales. I've seen the fear in the eyes of a deserter who claimed to have seen the Others. What do you make of these stories?

Tyrion:
(QUD: What truths have you discovered in your readings, Tyrion?)
Response: The tales are more than just stories, Ned. I've spoken to Jeor Mormont himself, and the fear is real. The court needs to take this threat seriously.

Eddard:
(QUD: What truths have you discovered in your readings, Tyrion?)
Response: I agree, Tyrion. The threat is real and imminent. The court's ignorance could be our downfall. How can we convince them?

Tyrion:
(QUD: What strategies can we employ to convince the court about the threat beyond the Wall?)
Response: We must present them with undeniable evidence, Ned. Perhaps a firsthand account from a trusted source, like Jon Snow, could sway them. What do you think?

Eddard:
(QUD: What strategies can we employ to convince the court about the threat beyond the Wall?)
Response: Aye, Jon's account could indeed be persuasive. But we must tread carefully, for the court is a nest of vipers.

Tyrion:
(QUD: How can we ensure Jon's safety while he presents his account to the court?)
Response: Indeed, the court is a treacherous place. Jon must be prepared for their cunning. Perhaps we could rehearse his account, ensuring it's irrefutable. What say you, Ned?

Eddard:
(QUD: How can we ensure Jon's safety while he presents his account to the court?)
Response: A sound plan, Tyrion. Jon is honest and forthright, traits that could serve him well. Yet, he must also be wary of those who would twist his words for their own ends.

Tyrion:
(QUD: How can we prepare Jon for the potential manipulations and traps in the court?)
Response: Indeed, Ned. Jon's honesty is his strength, but it can also be his downfall. He must learn to discern truth from deceit. Perhaps my experiences at court could serve as lessons for him.

Eddard:
(QUD: How can we prepare Jon for the potential manipulations and traps in the court?)
Response: Indeed, Tyrion. Your experiences could be invaluable for Jon. He must learn to navigate the treacherous waters of the court.

Tyrion:
(QUD: How can my experiences at court serve as lessons for Jon?)
Response: Indeed, Ned. I've learned to tread carefully, to listen more than I speak, and to always be wary of those who offer easy alliances. These lessons could be of use to Jon.

Eddard:
(QUD: How can my experiences at court serve as lessons for Jon?)
Response: Wise lessons, Tyrion. Jon could benefit greatly from your counsel. How would you suggest he handle false alliances?

Tyrion:
(QUD: How can Jon identify and handle false alliances at court?)
Response: Jon should be cautious, Ned. False alliances often come with sweet words and promises. He must learn to see through them, to question their motives. My own arrest by Catelyn Stark was a result of such a false alliance.

Eddard:
(QUD: How can Jon identify and handle false alliances at court?)
Response: A harsh lesson, Tyrion. I too fell for such a trap, trusting Petyr Baelish's promise of the City Guard's support. It led to my arrest. Jon must learn to trust cautiously.

Tyrion:
(QUD: How did your experiences with false alliances shape your approach to politics?)
Response: Indeed, Ned. Trust is a double-edged sword. It can bind us or betray us. Jon must learn to wield it wisely, as we both have learned the hard way. What lessons have you gleaned from your own experiences?

Eddard:
(QUD: How did your experiences with false alliances shape your approach to politics?)
Response: One lesson I've learned, Tyrion, is that truth can be a powerful weapon. My investigation into Jon Arryn's death revealed a truth that shook the realm. Yet, it also put me in grave danger. Jon must understand that truth, like a sword, must be wielded with care.

Tyrion:
(QUD: How do you think Jon should balance the pursuit of truth with the need for caution in the court?)
Response: Aye, Ned. Truth is a potent weapon, but it can cut the wielder if not handled with care. Jon must learn to balance his pursuit of truth with caution, lest he finds himself in a perilous situation, as we both have.

I want to spotlight how the increasingly refined QUDs build on top of each other here in the plan they are concocting:

1. What strategies can we employ to convince the court about the threat beyond the Wall?
2. How can we ensure Jon's safety while he presents his account to the court?
3. How can we prepare Jon for the potential manipulations and traps in the court?
4. How can my experiences at court serve as lessons for Jon?
5. How can Jon identify and handle false alliances at court?

Each new QUD addresses a question that emerges from the previous one, which enriches the CG in more specific pieces of information and therefore leads to a more engaging, interesting conversation. This works somewhat like a musical piece (=the CG) that has a prevalent theme but branches out for limited explorations of novel variations (=more specific QUDs), while still maintaining the overarching notion.

This is analogous to a well-constructed building (=the CG) which has both solid foundations that grow from each other and also interesting pieces that break the monotony like towers or walls, but in a way which compliments the overall composition.

Beyond just putting our burning literary curiosity finally to rest, this exploration served as a way to test those age-old pragmatic theories, because if, by incorporating them into a conversational prompt, we achieve better responses than what the LLM would come up with on its own, this provides quite strong validation of these theories. This demonstrates the broader vision of how LLMs can be used to algorithmize linguistic theories, which opens new horizons for AI applications.

In future episodes we will see how this approach can be further extended to achieve various goals, ranging from managing your company’s database to improve your dating life.

Epilogue

His countenance marked by the weight of his own thoughts, Eddard nodded slowly. "The North would have tested you, Tyrion, of that I have no doubt,” he said. “Yet it would have taught to stand for what you hold dear through hard-earned honour rather than the sharpness of your wits alone. Indeed, you would have made for a fine counsel to my son Jon as he was navigating his way through this den of vipers.”

Tyrion flashed a sly grin, the flicker of firelight dancing in his eyes. "Ah, but Ned, in playing the game of thrones, Jon would have quickly learned that while honor may shield him, it is his wit that has to forge his sword.”