· 7 min read
Can you figure out where people's talking points come from with LLMs?
🥳
This week on citymeetings.nyc I published all 203 testimonies from a 15-hour public hearing on a big and contentious zoning proposal in New York City titled City of Yes for Housing Opportunity.
Conversations about this work piqued my curiosity: is it possible to approximate where people’s talking points come from using language models?
I took a crack at answering this question with a basic approach and found that it is:
- Mostly correct when I can verify where someone’s talking points come from.
- Difficult to evaluate otherwise.
- Useful in others ways.
My approach here was to:
- Grab talking points for an organization.
- Prompt an LLM to determine how aligned testimonies are with the talking points.
Many organizations in support have published talking points on City of Yes for Housing Opportunity, but I didn’t have any from Paul Graziano, a vocal opponent of the proposal who has organized a lot of the opposition in NYC.
Many of Graziano’s presentations to local community boards are on YouTube, so I used Claude to extract his talking points from 4 of their transcripts. This took ~5 minutes and the results are great.
This prompt and code on Github is what I used to determine the the similarity between an individual’s testimony and a set of talking points.
My code uses instructor to extract structured data and I ran it using Claude Sonnet 3.5.
Most of the output from the prompt is designed to force chain-of-thought and make it easy to verify the output, but what I’m primarily seeking is this label (this is an excerpt from the code linked to above):
analysis: str = Field(
description="""An analysis of how closely the testimony is related to the reference talking points.
You must classify the testimony as one of the following:
- HIGH_ALIGNMENT: The testimony is very closely related to the reference talking points, often making the exact same claims and arguments. The language may be similar and it is plausible that the person is using the reference talking points in their testimony.
- SUPPORTS_TALKING_POINTS: The testimony is supporting the same claims and arguments as the talking points, but it does not make the same ones or use very similar language.
- NOT_ALIGNED: The testimony is not closely related to the reference talking points.
"""
)
similarity: Literal[
"HIGH_ALIGNMENT",
"SUPPORTS_TALKING_POINTS",
"NOT_ALIGNED",
] = Field(
description="""The level of similarity between the testimony and the reference talking points, given your analysis."""
)
You can find raw output for each set of talking points here, but it might be easier to poke around the final reports instead, where I massage all the output into easier-to-read Markdown.
Let’s take a look at what’s in these reports.
The Open Plans report leads with an overview, followed by their talking points, then a section comparing each testimony to their talking points.
There were 16 testimonies with high alignment to the reference talking points -- largely similar language, quotes, claims, and arguments.
There were 69 testimonies that were supporting the reference talking points. These were similar in spirit to the reference talking points.
There were 28 testimonies that were not aligned to the reference talking points. These did not address any of the specific issues raised in them.
# Table of Contents
- [Reference Talking Points](#reference-talking-points)
- [High Alignment](#high-alignment) (16 testimonies)
- [Supports Talking Points](#supports-talking-points) (69 testimonies)
- [Not Aligned](#not-aligned) (28 testimonies)
Let’s take a look at output for one testimony. Here is a comparison of testimony by Jackson Chabot, Director of Advocacy at Open Plans, against Open Plans’ talking points.
The model correctly labels his testimony as HIGH_ALIGNMENT.
## Jackson Chabot
[https://citymeetings.nyc/city-planning-commission/2024-07-10-city-of-yes-public-hearing/jackson-chabot-from-open-plans-on-lifting-parking](https://citymeetings.nyc/city-planning-commission/2024-07-10-city-of-yes-public-hearing/jackson-chabot-from-open-plans-on-lifting-parking)
- **Similarity**: HIGH_ALIGNMENT
- **For/Against COYHO:** For
- **Stated Affiliations:** Open Plans
- **Borough:** -
- **Neighborhood:** -
### Quotes
> Cars contribute to the climate change crisis They create unsafe streets, and overall, they make our city less livable.
- **Closeness to Reference Talking Point:** SUPPORTS_POINT
- **Analysis:** This quote directly supports the talking point about transportation being a leading cause of climate change-causing emissions and the need to disincentivize driving. It also aligns with the point about making streets safer.
> By requiring parking to be built in new developments, we're encouraging residents to build or sorry to buy and own vehicles.
- **Closeness to Reference Talking Point:** IDENTICAL_POINT
- **Analysis:** This quote closely aligns with the talking point about parking mandates incentivizing driving and the need to break this cycle to combat climate change.
> Instead of continuing this cycle, lifting mandates breaks it by encouraging the use of public and active modes of transportation.
- **Closeness to Reference Talking Point:** SUPPORTS_POINT
- **Analysis:** This quote directly supports the talking point about lifting parking mandates to prioritize active and public transportation.
> On a ground floor basis and surface parking lots negatively impact the vibrancy of neighborhood because they create dead zones without community space or vibrant retail or other things that neighbors can use.
- **Closeness to Reference Talking Point:** IDENTICAL_POINT
- **Analysis:** This quote is very similar to the talking point about parking mandates harming street life and making it more likely for developers to build street-level parking instead of stores or restaurants.
> Instead of prioritizing parking, we can begin to prioritize people, we can build public space, and we create a more livable city by lifting parking mandates.
- **Closeness to Reference Talking Point:** SUPPORTS_POINT
- **Analysis:** This quote aligns closely with the talking point about planning for the city we want, not the city we have, and prioritizing good public transportation and livable spaces over car dependence.
### Analysis
The testimony by Jackson Chabot demonstrates a very close relationship to the reference talking points. The speaker presents arguments that are highly aligned with the key messages in the talking points, often making the same claims and using similar reasoning.
Key similarities include:
1. Emphasizing the outdated nature of parking mandates
2. Highlighting the connection between parking requirements and climate change
3. Discussing the cycle of car dependency created by parking mandates
4. Advocating for prioritizing public and active transportation
5. Emphasizing the negative impact of parking on neighborhood vibrancy and livability
While the testimony doesn't use identical language to the talking points, it consistently supports the same core arguments and perspective. The speaker doesn't mention some specific details found in the talking points (like statistics on parking costs or examples from other cities), but the overall message and reasoning are very closely aligned.
The testimony also adds some additional context and examples that aren't in the talking points, such as the reference to a specific developer's plans and a personal appeal about future housing needs, but these additions still support the main arguments present in the talking points.
Given the high degree of alignment in both content and perspective, it is plausible that the speaker is familiar with or using the reference talking points as a basis for their testimony, even if not quoting them directly.
Pretty cool! The analysis is on-point and the label is correct. Here is Jackson Chabot’s testimony on citymeetings.nyc.
Here are the reports I ran:
- Open Plans (Supportive)
- Open New York (Supportive)
- Paul Graziano (Opposed)
The encouraging result here is that most of the verifiable results are correct. When someone is known to be a member of an organization, their testimony almost always has high alignment with their org’s talking points.
But it’s the unverifiable results that I’m interested in and it’s hard to evaluate them.
I was hoping there would be cases of obvious alignment, where I’d feel comfortable saying “it’s entirely plausible that this person was using organization X’s talking points”.
Unfortunately, almost all testimonies are somewhere in between.
While it’s tough to determine if a testimony used a set of talking points, organizations can use this information in other useful ways:
- They can find enthusiastic people to recruit.
- They can measure how popular their message is among an audience.
- They can adapt their messaging to address more people supportive of their cause.
And they can do this for any niche audience that they care about. Iterating on my prompts and running this analysis cost ~$40 using Claude Sonnet 3.5.
(Yes, I can build software. But give it a year and we’ll see tools that anyone who can use a spreadsheet can wield.)
I’m currently working on improving chapter generation for citymeetings.nyc, but I’m keen to explore making purpose-built tools for similar analyses for:
- Journalists
- Advocacy orgs
- Policy researchers
- Government affairs
- Government
If you’d like to chat about any of the above, say hello!