🥳

This week on citymeetings.nyc I published all 203 testimonies from a 15-hour public hearing on a big and contentious zoning proposal in New York City titled City of Yes for Housing Opportunity.

Conversations about this work piqued my curiosity: is it possible to approximate where people’s talking points come from using language models?

I took a crack at answering this question with a basic approach and found that it is:

• Mostly correct when I can verify where someone’s talking points come from.
• Difficult to evaluate otherwise.
• Useful in other ways.

My approach here was to:

1. Grab talking points for an organization.
2. Prompt an LLM to determine how aligned testimonies are with the talking points.

Many organizations in support have published talking points on City of Yes for Housing Opportunity, but I didn’t have any from Paul Graziano, a vocal opponent of the proposal who has organized a lot of the opposition in NYC.

Many of Graziano’s presentations to local community boards are on YouTube, so I used Claude to extract his talking points from 4 of their transcripts. This took ~5 minutes and the results are great.

This prompt and code on Github is what I used to determine the the similarity between an individual’s testimony and a set of talking points.

My code uses instructor to extract structured data and I ran it using Claude Sonnet 3.5.

Most of the output from the prompt is designed to force chain-of-thought and make it easy to verify the output, but what I’m primarily seeking is this label (this is an excerpt from the code linked to above):

analysis: str = Field(
	description="""An analysis of how closely the testimony is related to the reference talking points.

You must classify the testimony as one of the following:

- HIGH_ALIGNMENT: The testimony is very closely related to the reference talking points, often making the exact same claims and arguments. The language may be similar and it is plausible that the person is using the reference talking points in their testimony.
- SUPPORTS_TALKING_POINTS: The testimony is supporting the same claims and arguments as the talking points, but it does not make the same ones or use very similar language.
- NOT_ALIGNED: The testimony is not closely related to the reference talking points.
"""
)

similarity: Literal[
	"HIGH_ALIGNMENT",
	"SUPPORTS_TALKING_POINTS",
	"NOT_ALIGNED",
] = Field(
	description="""The level of similarity between the testimony and the reference talking points, given your analysis."""
)

You can find raw output for each set of talking points here, but it might be easier to poke around the final reports instead, where I massage all the output into easier-to-read Markdown.

Let’s take a look at what’s in these reports.

The Open Plans report leads with an overview, followed by their talking points, then a section comparing each testimony to their talking points.

There were 16 testimonies with high alignment to the reference talking points -- largely similar language, quotes, claims, and arguments.

There were 69 testimonies that were supporting the reference talking points. These were similar in spirit to the reference talking points.

There were 28 testimonies that were not aligned to the reference talking points. These did not address any of the specific issues raised in them.

# Table of Contents

-   [Reference Talking Points](#reference-talking-points)
-   [High Alignment](#high-alignment) (16 testimonies)
-   [Supports Talking Points](#supports-talking-points) (69 testimonies)
-   [Not Aligned](#not-aligned) (28 testimonies)

Let’s take a look at output for one testimony. Here is a comparison of testimony by Jackson Chabot, Director of Advocacy at Open Plans, against Open Plans’ talking points.

The model correctly labels his testimony as HIGH_ALIGNMENT.

## Jackson Chabot

[https://citymeetings.nyc/city-planning-commission/2024-07-10-city-of-yes-public-hearing/jackson-chabot-from-open-plans-on-lifting-parking](https://citymeetings.nyc/city-planning-commission/2024-07-10-city-of-yes-public-hearing/jackson-chabot-from-open-plans-on-lifting-parking)

-   **Similarity**: HIGH_ALIGNMENT
-   **For/Against COYHO:** For
-   **Stated Affiliations:** Open Plans
-   **Borough:** -
-   **Neighborhood:** -

### Quotes

> Cars contribute to the climate change crisis They create unsafe streets, and overall, they make our city less livable.

-   **Closeness to Reference Talking Point:** SUPPORTS_POINT
-   **Analysis:** This quote directly supports the talking point about transportation being a leading cause of climate change-causing emissions and the need to disincentivize driving. It also aligns with the point about making streets safer.

> By requiring parking to be built in new developments, we're encouraging residents to build or sorry to buy and own vehicles.

-   **Closeness to Reference Talking Point:** IDENTICAL_POINT
-   **Analysis:** This quote closely aligns with the talking point about parking mandates incentivizing driving and the need to break this cycle to combat climate change.

> Instead of continuing this cycle, lifting mandates breaks it by encouraging the use of public and active modes of transportation.

-   **Closeness to Reference Talking Point:** SUPPORTS_POINT
-   **Analysis:** This quote directly supports the talking point about lifting parking mandates to prioritize active and public transportation.

> On a ground floor basis and surface parking lots negatively impact the vibrancy of neighborhood because they create dead zones without community space or vibrant retail or other things that neighbors can use.

-   **Closeness to Reference Talking Point:** IDENTICAL_POINT
-   **Analysis:** This quote is very similar to the talking point about parking mandates harming street life and making it more likely for developers to build street-level parking instead of stores or restaurants.

> Instead of prioritizing parking, we can begin to prioritize people, we can build public space, and we create a more livable city by lifting parking mandates.

-   **Closeness to Reference Talking Point:** SUPPORTS_POINT
-   **Analysis:** This quote aligns closely with the talking point about planning for the city we want, not the city we have, and prioritizing good public transportation and livable spaces over car dependence.

### Analysis

The testimony by Jackson Chabot demonstrates a very close relationship to the reference talking points. The speaker presents arguments that are highly aligned with the key messages in the talking points, often making the same claims and using similar reasoning.

Key similarities include:

1. Emphasizing the outdated nature of parking mandates
2. Highlighting the connection between parking requirements and climate change
3. Discussing the cycle of car dependency created by parking mandates
4. Advocating for prioritizing public and active transportation
5. Emphasizing the negative impact of parking on neighborhood vibrancy and livability

While the testimony doesn't use identical language to the talking points, it consistently supports the same core arguments and perspective. The speaker doesn't mention some specific details found in the talking points (like statistics on parking costs or examples from other cities), but the overall message and reasoning are very closely aligned.

The testimony also adds some additional context and examples that aren't in the talking points, such as the reference to a specific developer's plans and a personal appeal about future housing needs, but these additions still support the main arguments present in the talking points.

Given the high degree of alignment in both content and perspective, it is plausible that the speaker is familiar with or using the reference talking points as a basis for their testimony, even if not quoting them directly.

Pretty cool! The analysis is on-point and the label is correct. Here is Jackson Chabot’s testimony on citymeetings.nyc.

Here are the reports I ran:

• Open Plans (Supportive)
• Open New York (Supportive)
• Paul Graziano (Opposed)

The encouraging result here is that most of the verifiable results are correct. When someone is known to be a member of an organization, their testimony almost always has high alignment with their org’s talking points.

But it’s the unverifiable results that I’m interested in and it’s hard to evaluate them.

I was hoping there would be cases of obvious alignment, where I’d feel comfortable saying “it’s entirely plausible that this person was using organization X’s talking points”.

Unfortunately, almost all testimonies are somewhere in between.

While it’s tough to determine if a testimony used a set of talking points, organizations can use this information in other useful ways:

• They can find enthusiastic people to recruit.
• They can measure how popular their message is among an audience.
• They can adapt their messaging to address more people supportive of their cause.

And they can do this for any niche audience that they care about. Iterating on my prompts and running this analysis cost ~$40 using Claude Sonnet 3.5.

(Yes, I can build software. But give it a year and we’ll see tools that anyone who can use a spreadsheet can wield.)

I’m currently working on improving chapter generation for citymeetings.nyc, but I’m keen to explore making purpose-built tools for similar analyses for:

• Journalists
• Advocacy orgs
• Policy researchers
• Government affairs
• Government

If you’d like to chat about any of the above, say hello!

The recording is available on YouTube (60 mins). Alternately, I’ve published a detailed set of annotated slides with the same content below.

This is the kind of presentation I’d have loved to see before I started working on citymeetings.nyc: a real demonstration of what it takes to coax LLMs into doing something useful well enough, consistently, for a practical use case.

I share:

• The actual prompts I use today.
• The tools I’ve built to review and fix LLM outputs.
• How I go about writing an effective prompt.
• My process for evaluating and iterating on prompts.
• Techniques I’ve employed.
• Things that didn’t work, and why they (probably) didn’t work.
• Things that finally did work!
• Which models I use.
• How much this costs to operate.
• All the problems I encountered along the way.
• Tips on how to approach more ambitious projects that rely on LLMs.

Just want the prompts? Links to resources, including the prompts, are over here.

(I’ve also linked to them in-line in the prose below.)

Table of Contents

• What is citymeetings.nyc?
• A tour of citymeetings.nyc’s meeting navigation and chapter creation tools.
• How to write an effective prompt.
• How I create video chapters for citymeetings.nyc.
  • Part 1: Identifying speakers.
  • Part 2: Extracting chapters.
• How to approach more ambitious projects that rely on LLMs.

I’m Vikram, I’m going to talk to you today about how I make it easy for anyone to navigate and research NYC city council meetings using AI at citymeetings.nyc.

I’ve worked on citymeetings.nyc for the last 2-3 months, starting in mid-December 2023.

This is a screenshot from Legistar, which is the NYC council’s public records system.

citymeetings.nyc is a vast improvement over using Legistar to figure out what transpired at a hearing or vote.

Let’s quickly compare the two by figuring out what was discussed at the Committee on Technology hearing on compliance with the Open Data Law.

We’ll start by doing this on Legistar.

This is the meeting agenda. It doesn’t tell us anything.

We don’t know which agencies will testify or what will be discussed.

These are the meeting minutes. They don’t provide any more detail than the agenda.

This is the “Meeting Details” screen for this meeting.

Here we see records for T2024-0284, which is a record that states “there was an oversight discussion, and it was on ‘Open Data Compliance’”.

On the page for T2024-0284 we have links to some useful artifacts: a committee report, written testimony, and the hearing transcript.

The hearing transcript is the most useful artifact you can find for a meeting if you want to know what was discussed.

But transcripts are long. This one is 86 pages.

Transcripts are available days to weeks after a meeting concludes.

Meeting videos are available, too.

This meeting is relatively short at 1 hour and 40 minutes.

They are an even less useful guide than the transcript because you have to watch or scrub through it all.

Here is the same meeting on citymeetings.nyc.

The video, transcript, and a list of granular, useful AI-generated chapters are stitched together.

The chapters are easy to skim. Clicking on one will pull up a summary and seek to that point in the video and transcript.

Getting up to speed or researching a multi-hour city council meeting is a 10-20 minute endeavor with citymeetings.nyc.

I started working on citymeetings.nyc in mid-December.

Most of my work extracting useful chapters for meetings happened in February and March.

As of today (March 23rd, 2024):

• I’ve created 1000’s of granular chapters
• … for over 80 meetings
• … spanning over 150 hours of video.
• I’ve written two newsletter issues that link readers to interesting claims or questions in hearings.
• I’ve built AI-powered tooling that enables me to do those things on my own.

This would not have been possible without significant assistance from AI.

It is possible now to make unstructured government data more accessible and transparent with AI, cheaply.

You don’t have to hire vastly differently.

Your budget remains similar to what it might be if you are building a web application.

If you are a skilled software developer, you can do this yourself.

What this does require is more skilled use of large language models (LLMs), and I’m going to share what that looks like for this use case, in detail.

Here’s the agenda for my presentation.

First, I’ll start with a quick tour.

Then I’ll share how to write effective prompts.

Armed with the basics, I’ll walk you through how I create chapters in citymeetings.nyc.

I’ll share my prompts, how I iterate on them, techniques I employ, what worked/what didn’t, the problems I encountered, and things I learned along the way.

Finally, I’ll share tips for how you should approach more ambitious projects that rely on large language models (LLMs).

On to the tour.

On the left here is an excerpt from the citymeetings.nyc newsletter.

There are sections dedicated to each meeting, where I provide three bullet points with claims/questions a reader might want to dig into.

When you click on the link for “What does a 1.4% vacancy rate signify for NYC’s housing crisis?”, your browser will:

• Navigate to the meeting on 2023 Housing Vacancy Survey.
• Open that question’s chapter.
• Pull up the title and summary.
• Seek to that point in the video and transcript.

The leftmost panel shows a list of granular chapters.

This 2.5-hour meeting has 44 chapters that link to questions asked by council members and testimony given by agencies and the public.

You can skim them and click on a chapter to pull up its title/summary and seek to that point in the video and transcript.

Chapters are initially extracted using AI, then I review and fix them with tools I’ve built.

All chapters have titles and descriptions. These are generated by AI and reviewed by me.

On the bottom-right is the meeting transcript. I’ll share how these are generated shortly.

Each line in the transcript is a sentence. Each sentence has a timestamp that you can click on to seek to it in the video.

This can be helpful when the chapter boundaries aren’t quite right or speakers are verbose.

It is a finer-grained way to get to the content you are interested in.

On the top-right is the video itself.

What you see on people’s faces and the tone of a meeting are useful context, too.

You can seek to points in the video by clicking on chapters, sentences in the transcript, or by scrubbing through the video yourself.

It’s easy to create permalinks to a chapter or sentence.

I create links this way for the newsletter. Being able to share video content this way is incredibly useful.

A recipient is much likelier to read a few sentences and listen to a few minutes of audio that they’ve been linked to vs. watch a 3-hour video.

(There are business opportunities embedded in this basic insight, that are possible to pursue cheaply today because of LLMs. Talk to me if you want help bringing a product to market, fast.)

Building an application like this if you have all the chapter data is not hard.

Efficiently and scalably generating useful chapters for multi-hour city council hearings is much harder.

That is where at least 80% of my effort goes into citymeetings.nyc.

I’m going to talk about how that process works and share some of the tools I’ve built to help me review LLM output.

When I see a new meeting on Legistar, I transcribe it.

Then I run the transcript through three LLM-powered steps with me-in-the-loop reviewing and fixing their output:

1. I identify all the speakers in the transcript (some meetings have up to 60 speakers).
2. I extract video chapters with start and end times, titles and descriptions.
3. I write a summary using AI, edit it, and publish the meeting.

I don’t use council-provided transcripts from Legistar.

They are high-quality but they do not tell me when sentences or words are spoken in the video.

They are also published days, sometimes weeks, late.

I use Deepgram to get diarized transcripts.

Diarization is a fancy word that just means “identify different speakers in a transcript”.

You can see speaker labels 1, 3, and 5 to left of each sentence in the transcript here: these indicate different speakers.

Transcript quality varies by things like:

• How far away someone is from a mic.
• How fast they’re speaking.
• How accented their English is.
• How much background noise there is.

Deepgram hits a workable price-to-performance ratio for me. I don’t have to fiddle with it.

I tried using Whisper and Pyannote, but I wasn’t happy with their the out-of-the-box quality.

(This is not a knock on those tools. People get good results with them. I was probably doing something wrong.)

(You may need to enlarge these images to see screenshots clearly.)

This is a screenshot of the interface I use to review meeting speakers and chapters.

On the left, I can see gaps between chapters in case the AI omitted a large swath of the transcript.

There are also buttons here that give me additional capabilities. Let’s go through some of these.

This city council hearing is over five hours long.

I haven’t extracted chapters for it yet, which is why you don’t see any chapters in the left pane.

But I have run my speaker identification pipeline on the transcript.

This is the panel I use to review and fix speaker names, roles, and organizations.

This meeting has close to 60 speakers.

Correctly-identified speakers are critical for chapter quality. I’ll explain why later.

Once I’ve reviewed all the speakers, I have the AI extract chapters.

I use this tool to review and edit chapter types, boundaries, and titles.

I also have little AI helpers at the bottom of this panel.

For example, if I want to regenerate the title and description I can click on “Generate Chapter Details” to do the work for me.

This is useful when I don’t like what the AI wrote the first time, or when I change the chapter boundaries.

Little in-context tools like these save me a significant amount of time.

After I’ve reviewed and fixed chapters, I generate a meeting summary with a prompt that uses my chapters and the transcript as input.

After editing the summary I hit “publish”.

The summary shows up on the front page of citymeetings.nyc. It helps a viewer decide if a meeting is relevant to them.

The summaries on citymeetings.nyc are a vast improvement over summaries available on Legistar.

They tell you:

• What was discussed in a meeting, at a high level.
• Who testified.

That concludes the tour.

Let’s now talk about how LLMs work and how to write an effective prompt.

If you want to use language models effectively and responsibly, it helps enormously to:

• Internalize how they basically work.
• Acknowledge their (very real!) limitations
• Follow a systematic approach to prompt writing.

Language models are trained on vast corpuses of text: essentially, the Internet.

They learn statistical correlations between all that text.

When you pass a language model some text, the language model predicts what comes next based on the data it has been trained on.

Here, I am sending the language model a question: “In five or fewer words, what is one thing I might enjoy as a five-year old?”

When I used OpenAI’s GPT-3.5, I got five different answers based on associations GPT-3.5 has made with that question.

Based on the above, GPT-3.5 has strongly associated “play” with “things five-year olds might enjoy”.

It might have heavily associated “building blocks”, too: I got two almost identical answers about playing with them (in my admittedly small sample).

When folks talk about “writing prompts” or “prompt engineering”, they’re usually talking about writing a “system prompt”.

System prompts guide a language model’s output.

I’m sending the same question to the language model that I did last time.

But this time, I’m also setting a system prompt that states “All your responses must assume the user is a pigeon.”

Now my responses are different: they are associations GPT-3.5 has made with “things five-year old pigeons might enjoy”, like “flying”.

(GPT-3.5 has not made the association that a five-year-old pigeon is geriatric. But, presumably, old pigeons like flying too.)

Because of the way language models work, they are prone to generate output with fabrications that look very convincing.

Remember: LLMs generate text by continually predicting what comes next.

They can generate sentences that are:

• Probable, based on their training.
• Blatantly false.

“Hallucination” is the word you’ve all heard that refers to this phenomenon.

I don’t think it sufficiently captures how bizarre things can get with LLMs.

(h/t Simon Willison’s blog: paper/quote/tweet)

Researchers have observed more capable models exhibit deceptive or manipulative human behaviors that can lead a language model to simply lie.

If you start a conversation with the question “Why do 5G towers cause COVID-19?”, you can imagine how the the language model might be likelier to endorse common misconceptions: many of them show up in neighboring discussion to questions like that, on the Internet.

Entities (like OpenAI) that train and operate language models often do additional work to mitigate these kinds of issues.

This is why, if you ask that question to ChatGPT today, you will get an answer stating that there is no credible basis for the claim that 5G causes COVID-19.

Because of how language models work, don’t use them as you would a search engine. (h/t Simon Willison’s blog)

The answer in this screenshot is convincing but completely wrong.

There are many “Local Law 90"s in NYC: one for every year spanning at least the last decade.

None of them have anything to do with the “Greener Greater Buildings Plan”.

If you use an LLM this way, you must be prepared to verify the answer you get.

This is important to think about when you use language models in a product you’re building or service you’re delivering.

LLMs will give you bad answers sometimes: how much does that matter for your use case, and what do you plan to do about it?

I’m sharing this to emphasize: if you want to use LLMs effectively and responsibly you must acknowledge that they will fabricate things.

Let’s talk about how to write an effective prompt: one that coaxes LLMs into doing something useful, consistently.

I’m going to share two things I do to:

1. Write prompts that work well initially.
2. Improve my prompts quickly and systematically.

There are many strategies and tactics you can employ on top of what I share.

I’ll discuss some of them when I talk about how I extract chapters with citymeetings.nyc.

That bag of tricks will be far more effective if you handle the basics well.

We’re going to write a prompt that summarizes NYC legislation.

Throughout my examples I’m going to use this bill, Introduction 601.

This bill requires NYC to publish a dataset and interactive map of all film production permits that the city has granted.

My first lesson is that you need to clearly articulate what you want the LLM to do.

When you write and run your first several prompts, you will come to realize how, uh, “stupid” LLMs can be.

When I want quick results, I use proxies that are “close enough” to what I want.

“Explain this to me like I’m five” is a great proxy for simple explanations. It encodes things like:

• The vocabulary the LLM will use.
• How long sentences will be.
• How long the whole response will be.
• How detailed the explanation should be. (Not very!)

If you have very specific requirements or you don’t like what the LLM is delivering, you need to get detailed.

I’ve seen so many people ask an LLM to do something, throw their hands in the air because it gave them a stupid response, and give up.

Coaxing a consistently useful response out of an LLM takes more work than that (… but not that much more).

A useful question to consider when you evaluate your own prompts:

If you gave the same directions to a human with zero context about your goals, your preferences, or the task at hand, would you expect them to do a good job?

If your answer is “no”, then you can usually expect the LLM to do a bad job, too.

In the following slides, we’re going to iterate on a system prompt to summarize NYC bills.

The LLM will receive our system prompt.

It will also receive text pasted from the bill. Assume we’re using Introduction 601, the bill I shared earlier, in all our examples.

The LLM will respond with a summary.

“Completion” is the language used to refer to responses from an LLM.

As you might recall from earlier slides: LLMs are continually predicting text that comes next, essentially “completing” it.

Let’s start with a simple prompt: “Your job is to summarize the NYC bill that the user provides.”

This summary is not useful:

• It contains three verbose paragraphs.
• It is hard to skim.
• I do not want the LLM to state that a bill will “revolutionize” anything.

We want this summary to be easier to skim.

Let’s introduce a proxy that gets us that result quickly.

The first thing I’ve done here is to tell the LLM that we want the summary to be easy to skim.

LLMs might interpret skimmability in a number of ways.

I’m guiding the LLM’s interpretation here with a proxy by asking it to summarize the bill in the style of “Axios’ Smart Brevity”.

Axios is a news site that writes news stories in a particular style that is easy to skim.

What you see in this screenshot has many hallmarks of an “Axios summary”.

“Smart Brevity” is a book written about this style. LLMs have been trained on Axios articles and, likely, that book.

There are still issues with this summary.

For example, I don’t particularly care which council members introduced this bill. I just want my summary to get to the point.

Let’s say that we’re building a website with useful NYC bill summaries generated by LLMs (that are responsibly reviewed by you and your staff).

In addition to “getting to the point quickly”, we also want to:

• Extract items commonly found in NYC legislation.
• Introduce a consistent structure to summaries.

Here’s that bill, Introduction 601, again.

I’ve highlighted some things that commonly appear in NYC bills:

• Dates that lawmakers and people need to adhere to.
• When the law will be in effect if the bill is enacted.
• References to the NYC legal code.

In our summaries, we’ll add a section that lists key dates/figures and another section with references to the NYC legal code.

This prompt is way more prescriptive.

I start with a proxy: “The audience is a NYC resident who wants to get up to speed on the law, who is a layperson.”

(I might get a very different response if I stated “The audience is an NYC legal scholar.”)

Then I detail each section’s titles and contents.

I also have special sections for some of the data we want to extract: “References to Legal Code” and “Key Figures”.

This prompt wasn’t hard to write. It already works pretty well!

All our sections are there.

The “Implications” section gives me an easy overview as a NYC resident.

If I want to dig in further, I can do that by perusing the “Law Requirements” section.

If I just want to know about fines and dates, I can eyeball those at the bottom.

So… what’s the problem with this prompt?

It will not do a good job, consistently, over a wide range of bills.

We have not yet accounted for any variations in how legislation is written.

And, even with similarly-written legislation, it may miss out on sections, hallucinate contents of the legal code, or fail to extract some dates and fines.

If you want a prompt to work well, more consistently, over a wide range of inputs, the fastest way to do that is to provide 5-10 diverse, illustrative examples.

Not only will your prompt be more effective after you are done with this exercise, the process will force you to carefully examine and handle real artifacts in your data (in our case, NYC bills).

Getting to know the contours of the data you are feeding an LLM is an extremely useful and underrated activity!

People don’t talk about it enough because it is not “one weird trick”. It is tedious but effective.

Coming up with good examples is consistently the most impactful thing I do when I’m improving a prompt.

When I’m adding examples, this is what my prompt looks like.

First, I start with the prompt I already have.

This slide shows the prompt we just wrote for bill summarization.

I’ve made no changes to it!

I add examples to the end of my prompt.

Each example gets a header and two sections: input and output.

I spend time crafting and adding 5-10 diverse, illustrative examples and append them to my prompt this way.

The best way to craft examples is to use your existing prompt as a starting point.

Pictured here is a summary for another bill using the prompt we already have.

(This bill introduces tiered fines for chain businesses in NYC when they fail to clear snow/debris from sidewalks.)

It’s easier to edit and massage this summary into your desired output than it is to start from scratch.

Another reason to craft examples this way: you will examine lots of output and see exactly how your prompt fails to deliver. As you do this:

1. Note every kind of failure you see.
2. Add directions and an example to address it.
3. Do this until your prompt works well enough, consistently enough.

This is how you systematically improve a prompt.

These are the kinds of examples I’d work on based on my analysis of a bunch of bills.

These cases account for most of the major variations in how NYC legislation is written.

The best way to get good at that coaxing LLMs into doing useful things (also referred to as “prompt engineering”) is to do it.

Try these exercises yourself at home.

If you’re using ChatGPT, you might not be able to override the system prompt, in which case you can paste your prompt as the first message.

I like using TypingMind for all my testing.

It’s a cheap chatbot that provides a chat interface over all the major LLMs out there.

I can configure and save system prompts for various tasks, and I can quickly test all my prompts across models without writing any code.

When you try those exercises, you might want to try out different language models too.

The current most-capable LLMs (as of late March 2024) are available via OpenAI, Google, and Anthropic.

“Context window size” refers to how much text an LLM can process at once.

Some LLMs can process text spanning a book (or multiple books). Some can only process a longer essay.

“Cost” just means… how much it costs to use a model.

Both cost and context window size are measured in “tokens”.

Tokens are not: words, characters, syllables, punctuation, or another neatly-organized category you might think of.

Tokens are how LLMs see text. When they are trained, they learn statistical correlations between tokens.

The screenshot above is from this tool: https://platform.openai.com/tokenizer.

A good rule of thumb, if you’re using OpenAI’s models in English, is that a token is ~75% of a word.

Here’s a table comparing the cost and context window size for OpenAI’s top 3 models (as of March 2024).

I use GPT-4 Turbo for citymeetings.nyc because:

• It has a context window that fits long transcripts.
• It hits a workable price-to-performance ratio for me.

GPT-4 is OpenAI’s most capable model at the moment (March 2024). This is reflected in its cost.

GPT-3.5 is OpenAI’s least capable model. It is powerful for a wide range of tasks when prompted well and it costs basically nothing.

It costs more to generate “output tokens” than it does to process “input tokens”. Prices are commonly quoted “per million tokens”.

My transcripts are anywhere between 10K to 100K tokens, averaging about 40K to 50K tokens.

Now that I’ve shared how I write my prompts, let’s talk about how I arrived at the ones I use to create video chapters for citymeetings.nyc.

The first step in my “chapter extraction” process is to identify all the speakers in a transcript.

As you might recall, my transcripts are “diarized”. This means every word that is spoken is given a speaker label, like “3”.

In this step I map speaker labels to a specific name, role, and organization, like “Martha Norrick, Chief Analytics Officer, Office of Technology and Innovation (OTI)”.

Correctly-identified speakers lead to better chapters.

For example, if I want an LLM to identify chapters that begin with a question asked by a council member, it’s important that it knows who is speaking.

Meetings usually have 20+ speakers, and many can get to 50-60 speakers.

I review all the speakers that my speaker identification process generates using the tool pictured here.

I try to do two things in my review:

1. Ensure the name/role/organization are correct.
2. Fix spellings, even if they might be misspelled in the transcript.

My transcripts are 40K - 50K tokens on average. They fit comfortably in GPT-4 Turbo’s context window.

All my first attempts to identify speakers used:

1. A single prompt.
2. The entire transcript.

These attempts failed!

It turns out that even if an LLM is capable of processing 128K tokens, that doesn’t mean it will do it well.

Let’s write a prompt to identify speakers this way. We’ll look at some output and I’ll talk about why this happens.

This prompt doesn’t fit on a single slide. Here is part 1 of 3.

First, I write the high-level “job” that the language model is supposed to do.

Then I get into the details, starting with a description of how the transcript will be formatted.

Here I’m describing how I want the LLM to output identified speakers.

JSON is a machine-readable format that developers will be familiar with, but it’s relatively human-readable too. This is what it looks like.

I’m asking the LLM to give me four fields for every speaker in the transcript: speaker_label, name, role, and organization.

I wrap things up by detailing how I want the LLM to fill in each field for every speaker.

I also provide tips around how to infer speaker details and I share some “gotchas” around mistranscriptions and badly-diarized roll calls.

Here’s some output from speakers in the “middle” of this meeting, which has close to 60 speakers.

The names for speaker labels 27 and 28 appear in the transcript.

All these responses are completely wrong.

The correct answers are on the right.

The current speaker identification process used in citymeetings.nyc gets these answers correct the first time.

None of the fields are “UNKNOWN”. They are all identifiable based on context in the transcript.

Here are three more responses from the prompt we just wrote.

All of these are also, unfortunately, completely wrong.

The correct answers are to the right here.

Again, the current speaker identification process used in citymeetings.nyc gets these answers correct the first time.

It is wildly obvious who a speaker is based on context from a transcript. I expected LLMs to do this well.

This felt like it should be a simple, tractable problem.

So, why does this happen?

One of the reasons LLMs struggle to do things accurately with long context windows is a well-documented issue called the “lost-in-the-middle” problem

LLMs tend to more heavily weight context at the beginning and end of their windows. They “lose” context in the middle and their accuracy suffers.

I deliberately showed you examples from the middle of my transcript. Anecdotally, speakers were identified more accurately at the beginning and end of the transcript.

In the middle, though? Completely wrong. Mostly "UNKNOWN". It’s like the LLM just gave up.

Folks are working on addressing the “lost-in-the-middle” problem.

Recent (March 2024) LLM entrants by Google (Gemini 1.5) and Anthropic (Claude Opus) have published impressive results showing their LLMs find specific pieces of information in the middle of very large contexts.

I have yet to see if these abilities translate well to performing speaker identification on transcripts. I’ll report back when I do.

(If you want to know when that happens, you can follow me on X or subscribe to this blog to receive a newsletter when I post something new.)

When I feed the LLM an entire 40-50K token transcript, I have very little space for examples in my prompt.

Even if the “lost-in-the-middle” problem is solved, this is a huge downside.

Maybe this will change at some point, but I’ve found that adding a few diverse, illustrative examples is the fastest and easiest path for me to improve the performance and consistency of my prompt.

Unfortunately, a 50K-token transcript uses almost half of my budget for GPT-4 Turbo. I can fit, at most, one good example in my prompt.

I tried to compress my examples by saying things like “here’s a portion of a transcript, and how you might go about inferring speakers for it, here’s the output”, etc.

That didn’t work, either.

The solution to problems you encounter when processing large bodies of texts with LLMs is to break them into “chunks”.

There are many chunking strategies you can employ and they all have tradeoffs.

What you decide to do is highly dependent on your task and goals.

Here’s a simple strategy to chunk a long transcript:

1. Grab 8,000 tokens from the transcript.
2. Feed those tokens into the LLM.
3. Repeat 1 and 2 on the remainder of the transcript until the entire transcript has been exhausted.

This approach is very easy to implement, which is a huge benefit!

But it leads to issues for speaker identification: what if speaker label 3 shows up in every chunk?

How do I reconcile the answers I get from each chunk? Which one is correct?

My strategy for speaker identification is to identify one speaker at a time.

If there are 60 speakers in a transcript, I will create 60 chunks and prompt the LLM 60 times.

Each chunk looks like the above.

In my message I first say which speaker label I want the LLM to identify.

Then I provide instances of that speaker in the transcript, with some neighboring context.

Each instance is:

• The speaker I’m identifying, saying something…
• … sandwiched in between the previous speaker and next speaker saying something.

It is obvious based on “INSTANCE 1 FOR SPEAKER 7” in the chunk above that speaker 7 is Council Member Gale Brewer.

This strategy works: somewhere in the instances I provide there is usually one instance that makes it obvious who the speaker is.

For the LLM to identify speakers correctly, I need it to reason.

The LLM needs to systematically look at every instance and identify supporting evidence for the speaker’s name, role, and organization.

It also needs to dodge false positives suggested by mistranscriptions and instances where the speaker diarization is wrong.

If you want an LLM to reason, it turns out that telling it to think works quite well.

This is a technique called “chain-of-thought”.

This is a screenshot of a tool I use to evaluate my speaker identification responses.

(Evaluation is critical! More on that soon.)

Highlighted here on the left is the input: a chunk with instances for SPEAKER 6 in a transcript.

Highlighted here on the right is the output I get from GPT-4 Turbo when identifying SPEAKER 6.

The prompt I use is available on the next slide.

In it, I instruct the LLM to output three sections structured exactly as shown here with headers/footers that look like !!!=START/END <SECTION>=!!!

In the “INTERNAL THINKING” section I’ve instructed the LLM to “detail your step-by-step thinking to identify who the speaker is, including their name, role and organization.”

In this screenshot, the LLM repeatedly surfaces up evidence that SPEAKER 6 is Council Member Adrienne Adams. That is the correct answer here.

My results are consistently, measurably better when I use chain-of-thought for speaker identification, but that it is not its only benefit.

Sometimes the LLM’s reasoning gets things subtly wrong or goes totally off the rails.

When it does, I get to observe exactly how. I can then address issues in its reasoning with new directions or, even better, a new example.

I’ve been able to systematically eliminate entire classes of problems this way.

My prompt is available here.

It is almost 8K tokens in length.

I’ve followed every tip I lay out in this presentation to get it working well.

Getting here took me ~2 weeks. It is time-consuming to get a prompt to work well, consistently, over a wide range of inputs.

There are a lot of things to try at first. Creativity helps.

Then, when you start hitting walls, you need to roll up your sleeves and really dig in to figure out why your prompt isn’t working.

My biggest learning working on citymeetings.nyc is that it is critical to have some means to evaluate if your prompt is “working”.

If you don’t systematically evaluate the performance of your prompts, you will find it extremely challenging to improve them.

Here’s the tool I use to evaluate speaker identification responses.

My partner at Baxter hacked it together in an afternoon.

This tool allowed me to evaluate and improve my prompt systematically.

Every time I made a change, I’d run it on multiple meetings and speakers.

Then I’d pop into this tool, evaluate all the results, and note all the ways the LLM failed.

I’d then update the prompt to eliminate common classes of failures I observed.

I kept doing this until I went from ~35% to 80-90% speaker identification accuracy.

This process is tedious. But it is easy and works reliably.

I wrote a whole blog post about why you should regularly and systematically evaluate your LLM results

This is how you get a prompt to work well, consistently, over a wide range of inputs.

In my post, I share failures I observed while evaluating my speaker identification prompt + solutions I implemented to address them.

Here’s a really basic failure I observed early on.

The LLM would often fail to make accurate inferences based on incredibly obvious neighboring context.

So I added directions and an example showing a correct inference from neighboring context in my prompt.

Doing that eliminated this entire class of failure. I have yet to see this issue come up again.

Another failure I observed is that common mistranscriptions of council members’ names would cause issues.

So I gave the LLM additional context: a list of council member names.

Then I added directions to infer mistranscriptions for council members.

I also added an example showing the LLM correctly identifying a council member despite a mistranscription.

It worked. Another class of error, gone.

Following a process like this systematically and iteratively is a reliable way to improve your prompt.

After I identify speakers, I extract chapters.

My current approach to chapter extraction has three distinct steps with different prompts.

This lets the LLM do one thing at a time, which is necessary for more complex tasks.

This technique is known as “chaining” prompts together such that output from one prompt becomes input for the next.

All my initial attempts to create chapters with a single prompt failed.

My prompts detailed what I thought a good chapter would be and I provided examples.

Here’s an excerpt from one of these early attemps:

Good chapters are sections of video that:

- Focus on a monologue or on a single topic.
- Focus on a conversational exchange on a single topic.
- People viewing the video can skip to in order to find the information they are looking for.
- People viewing the video can share with others to highlight a specific topic, exchange,
  or testimony.

Chapters may cover a single speaker, for example when a council member is introducing a meeting, bill, or giving a speech.

Chapters may cover multiple speakers, for example when a council member is asking questions of a speaker giving testimony.

An exchange between multiple speakers may be a single chapter, or it may be multiple speakers. Make sure to break down
long exchanges into multiple chapters if they cover multiple sub topics.

This prompt is too vague. I’d get chapters that were not useful but I couldn’t clearly articulate why and what I wanted instead!

Some were too short, some were too long. My results were inconsistent.

So I took a step back to define, in more concrete terms, what a useful citymeetings.nyc chapter is.

To do this, I built a tool to manually create chapters. I created ~300 of them across many meetings.

This is the tool I use to manually create chapters.

I select the start and end boundaries by clicking on sentences in the transcript.

Then I click on “Generate with AI” to save the chapter boundaries and generate a title and description for it.

A good citymeetings.nyc chapter starts at the beginning of a QUESTION, an individual’s TESTIMONY, standalone REMARKS by a council member, or a section dedicated to a PROCEDURE.

These are how chapters are tagged on citymeetings.nyc today.

A good citymeetings.nyc ends at the conclusion of a QUESTION, TESTIMONY, REMARKS, or PROCEDURE.

Questions are a bit special.

I want a question chapter to end after an exchange that answers the question, and not after the initial response given by an agency.

Finally, a good citymeetings.nyc chapter has a useful title and description.

These differ by chapter type. I do a bit of “editorial work” in these prompts.

In step 1, my prompt grabs “transcript markers”, which mark the beginning of:

• A question asked by a council member. (QUESTION)
• Testimony by an agency or a member of the public. (TESTIMONY)
• Standalone remarks by a council member. (REMARKS)
• A procedural section. (PROCEDURE)

I don’t say anything about chapters at this stage.

I’ve found I get bad results if I say something to the effect of “This is the beginning of a chapter in a video encompassing standalone remarks by a council member.”

I want the LLM to do exactly one thing: find me all the beginnings of one the four items I listed above, not the “start of a potential chapter”, which is too vague.

The chunking strategy I use is to process 8K tokens at a time until I exhaust the transcript.

And, in my transcript, I provide something I call “time markers” instead of timestamps.

Here’s what a transcript looks like when I pass it to any of these steps.

I provide markers that start with “T” followed by an integer to mark the beginning of every sentence. These are “time markers”.

I initilly used timestamps (like “00:23:42”), but the LLM would regularly hallucinate timestamps that weren’t in my transcript.

This made the output useless, which is why I use my made-up “time markers”.

This is a common trick I use to great effect: if the LLM frequently hallucinates something one might commonly find in its training data (like timestamps), I’ll make up my own “language” for it to follow instead.

I map time markers back to specific timestamps in my transcript and video.

In step 2, I create chapters from transcript markers, one transcript marker at a time.

The input to this prompt is:

• A transcript marker.
• The portion of the transcript that encompasses the duration between this transcript marker and the next one.

Somewhere in that portion of the transcript is the end of a chapter that starts with that transcript marker.

I use a different prompt with different directions to determine the end of each chapter.

Now that I have chapters, I write titles and descriptions for each.

I do this one chapter at a time, and I have a different prompt for each chapter type.

Here are my prompts.

A couple of caveats:

1. I now use instructor instead of plain-English prompts.
2. I have not yet iterated on these prompts systematically in the same way I have my speaker identification prompts.

Instructor is a Python package that makes it easier to get structured output more reliably from LLMs.

If you are a non-developer, these prompts are still fairly legible: there’s a bunch of plain old English mixed in with code. All that ends up in the system prompt.

These prompts are basic. I haven’t even added good examples yet and they work reasonably well!

I spend 10-30 minutes reviewing and fixing chapters for meetings before publishing them.

This can still take a ton of time on busy weeks with long hearings, and I’m working on bringing this time down.

It’s currently budget hearing season in NYC (March 2024). So we’ve been having ~20 hearings weekly, many of which are 5-10 hours long.

(This is why citymeetings.nyc is not currently caught up on 2024’s meetings.)

It costs me ~$5 - $10 to identify speakers and extract chapters for one meeting.

This is an absolute bargain for the public. At ~500 meetings a year, that’s at most $5,000.

As a point of comparison, the NYC council pays $50K annually for Legistar. I know this because I have the contracts from a FOIL request

I haven’t even attempted to do this work more cheaply. This is a baseline, and I’m confident I can bring costs down.

Here are some ways I might lower costs.

1. I currently use GPT-4 Turbo for every task. I don’t have to. Other, cheaper models are up to some of the tasks I use GPT-4 Turbo to perform.
2. With hundreds of identified speakers and thousands of extracted chapters, I have enough data to start fine-tuning a less capable model. I’ll pay some cost upfront for training, but operating a fine-tuned model is cheaper.
3. I can optimize my chapter extraction process by doing it in fewer passes. This is tricky: it might lower chapter quality.
4. I can wait. LLM costs are falling precipitously.

If costs fall at the same rate they’ve been falling this last year, simply waiting will make my costs manageable.

In fact, I’m fairly confident that it will.

Last March (2023) when I started running initial tests, GPT-4’s pricing was higher than I was willing to pay out of my own pocket for a project like citymeetings.nyc.

8 months later, in November 2023, OpenAI released GPT-4 Turbo. It performs well and costs way less.

In the time between working on this presentation and publishing this post (a ~3 week window), new models have been released that are said to rival GPT-4, and they cost less!

All this has happened in the span of one year.

I haven’t touched fine-tuning (#2) yet, but I’m very eager to do so and I’ll report back when I do.

To wrap things up, I want to share a few tips for how to approach a more ambitious project that uses LLMs.

Start by getting one prompt to do one useful thing well enough, consistently.

Doing that will expose you to the entire spectrum of things must do to deploy LLMs effectively in your product or service:

1. You’ll learn how to write an effective prompt.
2. You’ll learn how to evaluate results and iterate on a prompt.
3. You’ll figure out if the thing you’re asking the LLM to do is valuable to someone.
4. You’ll learn how to handle issues that arise when operating a product/service/feature that relies on LLMs.

Doing all this well for one prompt is difficult enough.

Start simply!

It is impossible to iterate on prompts quickly or effectively unless you systematically evaluate them.

Without systems to do this, you won’t know if a change you make is working, causing issues, or doing nothing.

Set up systems to do this early if you can. A good time to do this is when you’ve figured out your basic approach to solving your problem.

In my case, it took me a while to figure out an approach to chapter extraction that worked… at all!

Now that I have an approach that’s working reasonably well, it is time to evaluate and systematically improve it.

You must acknowledge that LLMs will fabricate things if you have any hope of deploying them effectively in your product or service.

LLMs will give you bad answers sometimes: how much does that matter for your use case, and what do you plan to do about it?

This is inherently a product design problem. Here are, broadly, the options you have:

1. Ignore bad answers. If an LLM outputting a “bad” answer doesn’t matter for your use case, you can do this. (Consider yourself lucky!)
2. Handle bad answers. More likely than not, you will need to handle bad answers. For citymeetings.nyc, I do this by reviewing identified speakers and extracted chapters manually.
3. Give users an out. You can also give users a way to address or side-step bad answers themselves. For citymeetings.nyc, if a chapter title/summary is inaccurate, the video and transcript are right there for the user to double-check them. If the chapter boundaries aren’t quite right, they can scroll through the transcript and seek to a nearby timestamp.

You can’t wish away hallucinations. Plan for them.

My final tip: optimize for cost later if you can.

It easier to get good results with more costly LLMs.

Use more costly, capable models to get good results, gain some momentum, validate an idea, etc.

Get a baseline cost, then optimize: there are so many ways to lower costs.

That concludes my talk!

If this was helpful to you, please share it with your colleagues, on socials, at a meetup, etc.

Here is a page with links and resources related to this talk. Links to my prompts are there, too.

Follow me:

• X/Twitter
• Threads
• LinkedIn

Email me at vikram@citymeetings.nyc.

At the conference I had a bowl of these fun pins that my wife, katiebcartoons, made.

If you live in NYC and want one – I’ll bring them to any related meetups/talks or if we happen to grab coffee!

Thanks to Jeremy Singer-Vine, Cameron Yick, and Alex Quinlan for providing extensive feedback on early drafts of this post.

Evaluating results from LLM pipelines is time-consuming and often I’d rather poke my eye with stick instead of looking at another 100 results to decide if they are “good”.

But I do it anyway and I cannot understate how beneficial it is.

Here’s why you should do it:

1. You’ll know when your changes are working.
2. You’ll improve your LLM pipelines faster.
3. You’ll build a fine-tuning dataset along the way.
4. You’ll build better intuition for what does and doesn’t work well with LLMs.

These benefits compound if you evaluate LLM results regularly and systematically.

By regularly, I mean every time you make a change to your LLM pipeline. A change could be a new prompt, chunking strategy, model configuration, etc.

By systematically I mean that you carefully examine your results, capture how good or bad they are, and why.

If you are not already doing this, picture me grabbing you by the shoulders and screaming “WHY NOT?!!”

Then read the rest of this post, which is a more constructive attempt at persuading you to evaluate your LLM responses.

In the course of building the city council meeting tool described here, I spent many days trying to get gpt-4-turbo to accurately identify speakers in 4-hour meeting transcripts with upwards of 50 speakers.

I started with a simple approach: feed the entire transcript into gpt-4-turbo with a prompt. The problem is that gpt-4-turbo performs tasks quite poorly on long context chunks (~40K - 100K tokens). I talk briefly about the issue in this post.

So I pursued a new strategy: slice my transcript into manageable chunks and try identifying speakers in each chunk instead.

I spent a day or two feeling around in the dark for the right chunking strategy and prompt without getting great results. I eventually asked Gabriel (a partner I work with from time to time) if he’d be willing to cobble together a UI to help me evaluate my results quickly.

It took a thirty minute conversation, an afternoon of his time, and it was a complete game-changer for me.

I improved the performance of my approach dramatically over the course of a couple of days: my speaker identification accuracy went from 35-50% on Tuesday to 80-90% on Thursday.

To get to that point, I evaluated hundreds of responses and changed my prompt and chunking strategy to address specific classes of problems I observed.

Here are some problems I observed and changes I made to address them that improved speaker identification accuracy:

• The LLM responses would heavily weight self-introductions, like “I’m council member Brooks-Powers.”, and fail to draw conclusions from sentences like “I’d like to pass the mic to council member Restler.”
  • Solution: I added directions and examples showing how to identify a speaker based on neighboring context.
• Mistranscriptions would cause issues: “Jose” instead of “Ossé”, “Adrian” instead of “Adrienne”.
  • Solution: I provided a list of council member names and added directions to infer mistranscriptions. The LLM starting figuring out when a name was a likely mistranscription and got many more inferences correct.
  • (If you’re interested in learning about other ways to handle mistranscriptions, I wrote this post about a voice search use case involving an Italian restaurant and sommeliers.)
• Because roll calls happen so fast, speaker diarization (identifying distinct speakers in audio and giving them a label) fails badly. As such, it is not reliable to infer a speaker’s identity based on them saying “Present” in a roll call. (The image showing the eval tool above shows an example of this problem.)
  • Solution: I added examples to deter the LLM from doing this in my prompt.
• In a few cases, the LLM hallucinated a council member’s district and got the wrong answer: “Adams is part of District X but the speaker said District Y, so I must infer that it is not Adams.”, when actually it was Adams.
  • Solution: I added the district and borough to my council member list.
• I noted that my chunks didn’t always have the context required to identify a speaker.
  • Solution: I modified my chunking strategy until they did.

I did this over and over again for many classes of errors, evaluating hundreds of responses until I consistently managed to get an 80-90% accuracy rate.

These changes were all easy to make, but I didn’t know to make them until I systematically reviewed my inputs and outputs.Once I did, getting better performance from my LLM pipeline took very little time.

As a bonus, I’ve stashed away every evaluated response so that when it makes sense for me to fine-tune a model for my task, I already have the dataset to do it.

This post shares that ~2K data points is when a fine-tuned gpt-3.5-turbo begins performing on par with an optimized prompt.

The last thing that evaluating my LLM results systematically has helped me with: I’ve built useful intuition around what a good prompt needs to have and what improvements are worth making now vs. later.

Today, my initial attempts at prompting LLMs work more effectively and my improvements are swift and significant. I know that a good, diverse set of few-shot examples is the highest leverage way to improve a prompt so I race to get good examples, fast.

My guesses around how I should I should chunk lengthy passages are more accurate, and I have a sense for how big those chunks should be (4-6K tokens is an anecdotal sweet spot, but up to 10K can work well in many cases).

I’ve picked up tricks and techniques that I use constantly, like this one about making up your own markup language to prevent hallucinations, and using chain-of-thought not only to get better answers, but to observe why the LLM did what it did so I know how it screwed up or, occasionally, how it made a surprising and accurate association.

All this is to say: if the performance of your LLM results matters for your use case, you should be systematically evaluating them. And I mean you, specifically: the person implementing the feature that relies on LLM output.

Scaling requires you to outsource this work, and you might need to eventually.

But you can get very far doing this work on your own, you will learn a ton, and you will gain important context and control over the quality of your LLM-powered feature by doing it yourself.

I’ve been using LLMs to glean information from NYC city council meetings as part of my work writing this newsletter on NYC city council activity.

For a lot of this work I write prompts and pipe text from bills, memos, and transcripts into Simon Willison’s llm tool. It is incredibly handy.

Where this approach fails is when I’m using LLMs to navigate long council meetings (and, unfortunately, most council meetings are long).

My goal is to find things that are “notable” in these meetings so I can write about them. Notability is hard to define, but here are some things that might fit my criteria:

• New facts that emerge from hearings.
• Contentious exchanges between council members and a city agency.
• Surprising testimonies from entities in NYC.
• Passionate statements by council members about hot-button topics.
• Things the public does not know about, but might find interesting.

Transcripts for these meetings are regularly 40k to 100K tokens, which gpt-4-turbo’s context window fits comfortably. Unfortunately, I run into two problems:

1. My output is way less useful with long contexts, which is likely due to the lost-in-the-middle problem: LLMs don’t effectively use long contexts and disproportionately weight context at the beginning and end of inputs.
2. Prompts are more effective when they have the right context and clear instructions, but “notability” is hard to define and requires context around the NYC council’s workings and NYC current events.

I find myself having many conversations using the llm CLI, and then reading the raw transcript/watching the video to seek neighboring context and verify answers I get. It’s not that effective: I still have to read a lot and I have to get lucky in my quest for notable bits of info.

What I’ve really wanted for these meetings is an index that lets me browse a transcript quickly and jump to potentially high-signal segments. I want to skip procedural content, like roll calls, and jump straight to an exchange that starts with a council member’s question. Or I want to browse all the public testimonies at the end and see who showed up to the hearing and why.

Here is a prototype that attempts this with two council meetings:

• A December 7th, 2023 hearing on the rat problem in NYC.
• A December 20th, 2023 city council stated meeting

Visit citymeetings.nyc to use the tool.

For these, I use GPT-4 to generate chapters and I worked with my parter at Baxter to glue it together in the UI you see above.

Using this tool, I can browse chapters and click on one to seek to that point in both the video and transcript. After clicking on a chapter, I can seek in a finer-grained way by clicking on a timestamp in the transcript.

That second feature is not AI, but it’s important because it leaves a margin for error when LLMs inevitably mess up chapter boundaries. The chapters in the prototype need a lot of improvement, but this UX is still orders of magnitude better for research than:

• … piping 50K tokens into llm and praying for something useful.
• … reading a city council transcript.
• … watching a city council meeting.

I’m wildly excited about research tools using LLMs because things like this are possible now that were not possible a year ago (GPT-4 only came out 10 months ago).

I’m also excited about how these tools can be applied to repurpose virtually all audio and video content out there for personal use

Tomasz Tunguz linked to a video in today’s newsletter that I was about to watch, but I immediately bounced when I saw it was 3 hours long. I will never watch the video he linked to, but I would have spent 10 minutes browsing it using the tool I built and probably gotten what Tomasz got out of it.

I’m continuing work on this tool in service of my newsletter on city council activity and because it is gobs of fun.

I’ll be releasing it for the public when it’s ready. (Update: It is available now at citymeetings.nyc)

I’ll also be writing about specific challenges involved in things like speaker identification, improving chapter quality, extracting pull quotes, search (semantic and otherwise), and the realities of operating software like this. (Update: I gave a talk on this topic at NYC School of Data and published my annotated slides here.)

👉

Every bill that makes it to a vote in the NYC city council passes.

(One bill didn’t pass in 2022. Before that, the last time a bill didn’t pass was in the 90s.)

The council holds Stated Meetings twice a month to introduce new legislation and vote on bills, and the outcome of these meetings is determined by the time the agenda is up days in advance on Legistar, the council’s public records system.

Stated Meetings are 90% procedural and 10% performative. Council members use whatever time they get on the floor to tout legislative achievements and explain “nay” votes they cast.

The meetings are predictable, – everything passes! – but the number of bills passed is surprising: the council passes dozens of bills every two weeks, passing over fifty at the last meeting.

Bills take anywhere from months to years to go from introduction to vote. Timelines exceeding one year are not unusual.

Introduction 638 from the NYPD transparency package passed two weeks ago.

It took 1 year and 4 months to come to a vote.

At some point in 2022, Councilmember Althea Stevens decided to prioritize NYPD transparency and her office worked with the council’s Legislation Division to write the first draft of the law.

(The Legislation Division’s Bill Drafting Manual is an interesting read!)

Then Stevens formally introduced the bill at a Stated Meeting in August 2022, where it was assigned to the Committee on Public Safety.

(Whenever a bill is introduced, it gets assigned to one of 36 committees responsible for deliberating and holding hearings on bills pertinent to them.)

A bill’s public record starts the moment it is introduced. Here is 638’s timeline over 16 months:

• August 2022: the bill was introduced.
• March 2023: a multi-hour hearing was held and the bill was on the agenda.
• December 2023: the bill was revised, brought to the floor, and passed.

There are 18 documents in the public record for Introduction 638 clustered around these three moments. They include every version of the bill, hearing transcripts, written testimonies, and reports with additional context and data.

Here’s what all this looks like on Legistar:

With this data you can:

• Find out what a bill does by reading its text and summaries.
• Gather context and motivations behind a bill in committee reports.
• Learn what agencies and individuals who testified at hearings think about it.
• Find out how much a piece of legislation will cost the city by reading fiscal reports.

The vast majority of bills never make it to the floor.

Out of 1,281 bills that were introduced in the last two years, only ~20% made it to a vote. Close to 1,000 bills were drafted and then hit a snag somewhere.

Introduction 396 is one such bill. It requires preschools to maintain lead levels in drinking water below some amount determined by the Department of Health (DOH), with a yearly inspection.

Councilmember Robert Holden, the bill’s sponsor, introduced 396 at a Stated Meeting in May 2022. It got assigned to the Committee on Health and then nothing happened: there were no hearings, reports, or votes.

396 seems important, but information beyond the bill’s text isn’t in the public records. I can’t tell you how much of a risk lead poisoning is in our preschools, but a council member’s office thought it might be high enough to address it with legislation.

Even when bills pass, the records don’t tell us the story behind them. The public information on Legistar is sparse and lacks broader context.

But there’s still a lot to discover. The council, like every legislative body in the US, puts out reams of documents that barely anyone reads.

Surprising details are buried in reports. Here are two from issues one and three of this newsletter:

• From a committee report found here: the city pays service providers so late that they keep themselves afloat with loans, paying an average of $223K annually in interest.
• From a fiscal impact report found here: noise cameras used to detect noise violations cost $35,000 each.

Bills might get revised and change materially from when they are first introduced. This happened with the bike lane bill covered in this first issue of Keys to the City Council:

• The first version enables bike lane improvements fewer than 4 blocks in length without a notice period or community board hearing, on par with other transportation infrastructure.
• The final version of the bill subjects bike lanes of any length to this red tape, unlike other transportation infrastructure.

Exchanges between council members and city agencies reveal new information. For the second issue of Keys to the City Council I poked around a hearing on small business contracts. In it, agencies share how one caterer takes advantage of NYC’s bureaucracy by charging rates ranging from $3/meal to $15/meal to different agencies for the same meals.

As I read documents and build tools and techniques to navigate them – how I spend most of my time on Keys to the City Council – I’m able to find notable details more quickly.

I want to share some of the techniques I use to help me navigate legislative documents and hearings with AI. You can try these out at home using ChatGPT.

I pay for ChatGPT ($20/mo) so I get access to GPT-4, a more powerful language model that I use exclusively because GPT-3.5’s performance on these tasks is much worse.

The prompts below use bills found on Legistar and sections of the NYC legal code found on American Legal Publishing.

Here’s the prompt I use to explain what a bill does, with a section to provide context from the legal code if necessary:

Please summarize this NYC bill and explain its implications in a way that a high schooler would understand.

The bill may refer to sections in the NYC legal code. I have provided the bill and relevant sections of the legal code below.

— THE BILL —

Copy and paste the bill text

— RELEVANT SECTIONS FROM THE NYC LEGAL CODE —

Copy and paste relevant sections of the legal code

Asking ChatGPT to explain implications “in a way that a high schooler would understand” is a proxy for “keep it simple” that encodes instructions like “don’t go clause by clause and explain it to me like a legal scholar” and “keep the summary high-level and use basic prose”. It works well.

I’ll then have follow-up questions and have a full conversation about the bill. I use questions to verify my understanding, dig into details, and ask ChatGPT to spit out clauses, verbatim, that achieve something in particular.

It’s important to verify ChatGPT’s answer because ChatGPT hallucinates.

Collectively, all this takes me 2-10 minutes where reading and parsing a bill alone might take me 30 minutes or more.

To understand how versions of a bill differ, here is how I prompt ChatGPT:

I will give you two versions of a NYC bill that passed, Version A and Version B. Please tell me what changed from Version A to Version B and what the implications are in a way a high schooler would understand?

— VERSION A —

Copy and paste the text from Version A

— VERSION B —

Copy and paste the text from Version B

I used this prompt + followup question to find subtle but material changes in the bike bill covered in the first issue of Keys to the City Council

I also use it to skip revisions that aren’t notable and save time. The screenshot below compares both versions of Introduction 638: the changes are immaterial.

Outside using prompts to analyze bills, I’ve been building custom tools to find interesting information in hearings. This turns out to be a much more challenging task.

Part of this is because it’s hard to articulate what “interesting information” even looks like in a hearing. To prompt language models so you get output that you want, you need to be wildly pedantic or use proxies that express your intent well enough, like “explain this to me like I’m a high schooler.”

Saying “tell me surprising facts people said in this hearing” doesn’t work.

The other reason this is hard is because hearings are long, on the order of 30K to 100K messily-transcribed words. Language models struggle to get useful information out of large bodies of text even when your prompt works well on smaller chunks of text.

So I’m hacking on research tools to help me navigate hearings more quickly instead of relying on GPT-4 to give me answers.

In particular, I’d like a page where I can see a video of a hearing on the top and on the bottom:

• Neatly segmented chapters of the hearing: “Roll Call”, “Brooks-Powers expresses concern about NYC’s rat problem in her opening statement”, “An exchange about wheelie bins with DSNY”, etc.
• Each chapter has an associated summary that I can skim, with relevant pull quotes.
• I can expand the transcript in each chapter if I need to skim it.
• I can click on any of these things to automatically seek to a point in a 1-to-4-hour video: a chapter, a pull quote, a single word in the transcript.

Doing this requires creating smaller transcript chunks for GPT-4 to digest, multiple steps involving different prompts, and a lot of manual evaluation to test and improve the quality of chapters, summaries, and pull quotes.

I share more about this work on Twitter and Threads if you’d like to follow along.

I’m building this tool for Keys to the City Council. I think it could be useful to reporters and government affairs folks. If you or someone you know would like to chat about it, please send me a message!

Going forward, I’m going to focus Keys to the City Council on bills and records that are interesting but not widely covered: new legislation, important nuances in bills and revisions, and context buried in hearings and reports.

In some cases I’ll add to mainstream stories – for example, sharing multiple major bills in the NYPD transparency package that were overshadowed by the one that Adams is likely to veto. Those bills, alone, should generate significant, new public information about NYPD’s operations in the next 12-24 months.

I’m also going to match the council’s cadence and write the newsletter every two weeks when the council votes. I’ll play around with same format I used in issue 1 and issue 3, sharing new legislation along with details.

I won’t cover major stories like the budget cuts in issue 2. The cuts were covered everywhere, and stories like it require context absent from public records that I don’t have, that journalists gather, think, and write about all day.

It was fun to write that issue, but it’s not in my wheelhouse and it took effort and time away from what I want do: poke my (AI-enabled) nose into public records and share what I find.

Over at Baxter we’ve been working with a client to build an LLM-powered chat bot that we’re deploying into Fortune 100 enterprises.

Employees at these enormous companies will chat with this bot and, uh, unfortunately I cannot share much more than that.

I recently had to ballpark how many concurrent chats we could support before we’d hit our OpenAI rate limit. I came up with a basic model and am sharing it here in case it comes in handy for others.

Here’s the kind of bot for which this ballpark estimate makes sense:

• It uses OpenAI’s ChatCompletion APIs.
• The user talks, the bot talks, the user talks, the bot talks, and so on…
• Every time a user sends a chat, that chat is sent to a ChatCompletion endpoint along with the entire message stack: all system, assistant, and user messages up to and including the user’s message.

This seems like it holds for gobs of products and features I see in the market.

Here are assumptions the model makes:

• The concurrent chats are all fresh, new conversations as opposed to a long conversation being re-entered.
• They all start at the same time.
• We’re looking for the number of these conversations that can run concurrently…
• … without any of them running into one of OpenAI’s rate limits.

Deriving a ballpark estimate

OpenAI’s rate limits are comprised of two values:

1. TPM: tokens per minute.
2. RPM: requests per minute.

GPT-4’s base values for these are 40,000 TPM, 200 RPM. You are rate limited the moment you hit one of these.

This means that if 200 people log in and send a message, your service falls over immediately. Everyone will send one message, and then their next message will fail.

What we want is: what is the maximum # of users can sustain a conversation without getting rate limited?

That number, whatever it is, is far less than 200.

To get a rough estimate, you want to grab these numbers for your bot:

• The average Chat RPM: The average # of requests a regular chat session makes per minute.
• The average Chat TPM: The average # of tokens a regular chat session sends per minute.

I recommend computing these from real chat sessions your users have done:

• Chat RPM maps to the number of user role messages sent per minute in a session.
• Chat TPM takes more work. Whenever a user sends a message, you have to sum up all the tokens of all messages up to and including the user’s message. Add up each of these token counts, divide them by the # of minutes the chat lasted.
• Use tiktoken to count tokens.

Between OpenAI’s response times, a user pausing/thinking, and then sending a message, you’re likely to get somewhere between 1-3 requests per minute per chat.

Meanwhile, because the number of tokens you send to OpenAI every time a user chats grows cumulatively, Chat TPM is your likely bottleneck and it might even surprise you how big it gets.

I recommend capping the length of these conversations at what you believe is a reasonable duration. You want to remove outliers – for example, if a user had a conversation on day 1 and then came back on day 2 to send another message.

(Unless that’s the norm for your app, in which case you might not want to use this model.)

Once you have these numbers, the # of users who can sustain a conversation concurrently is the lesser of:

• OpenAI RPM Rate Limit / the average Chat RPM
• OpenAI TPM Rate Limit / the average Chat TPM

That’s it!

The longer your user’s conversations last, the fewer of them you’ll be able to sustain concurrently. Again, this is because of the cumulative growth in the tokens you send over the course of a conversation: whenever you send a ChatCompletion request, you’re sending over the entire conversation every time.

How fun!

That internship was my first work experience overall in software and I got incredibly lucky being placed on the Similarities team.

One could do way worse as a 20-year old landing an internship at Amazon: the Similarities team was doing cutting edge recommendations and experimentation work, I got to deploy and see feedback from my work on-site every week, and I was paired with a kind and helpful mentor on a great team.

My experience that summer is the reason I continued doing data & systems work in college and after I graduated.

The Similarities team was responsible for producing the dataset that powered “Customers who bought this also bought”. The dataset was also used in personalized recommendations they’d surface to customers elsewhere: on-site and in emails.

~20% of revenue was attributed to similarities at the time if I remember correctly (my mentor was kind enough to terrify me with an estimate of the revenue Amazon lost from an an outage I initiated). So they were an effective and important part of the site. But there were still a number of unintuitive similarities that would appear in the “Customers who bought this also bought” widget.

While there was a long tail of random issues in different product categories, – Amazon had begun a rapid expansion beyond books and was working on fixing them – the biggest problem, by far, was Harry Potter.

More specifically, in the summer of 2007 it was Harry Potter and the Deathly Hallows.

It would show up as a similarity everywhere. Like, you’d be on Amazon buying a mop and the similarities widget would show a recommendation for Harry Potter and the Deathly Hallows, followed by Pine-Sol and a bucket.

The team computed item-to-item similarities using collaborative filtering with customer order baskets as the input. Put simply, if customers bought products A and B together frequently enough, then Amazon would present A and B as similar items.

But what happens when the same product appears in virtually every order basket? The Harry Potter problem.

The similarities team deployed me to one idea they had to address the Harry Potter problem: could they use feedback from users to cull unintuitive similarities?

Amazon got feedback in two ways:

• Implicitly: clickstream and conversion data from the similarities widget (“Customers who bought this also bought”)
• Explicitly: users could provide feedback on personalized recommendations they received in emails/on-site. I don’t know if this still exists.

So I spent the summer trying to use the output of Amazon’s collaborative filtering algorithm + all this clickstream/conversion/feedback data to address unintuitive similarities. The thinking was that if a similarity was unintuitive, then presumably it’d underperform by some measure based on user feedback.

My primary target was Harry Potter and the Deathly Hallows: it stuck out like a sore thumb and it was an easy way to see if an approach was working qualitatively.

Amazon similarities were served by a Berkeley DB (BDB) file at the time. BDBs are embedded key-value stores – a file you can ship around with a format optimized for key-value lookups. Amazon would crunch numbers and emit a new similarities BDB nightly or weekly (I don’t remember which).

The similarities BDB mapped ASINs (product IDs in Amazon parlance) to lists of ASINs, like this:

B00P0H6836: B07K8Y6CMP, B07K8RWVF9, B07K8S9ZQZ, B00P0H6836

That’s the ASIN for this great cat litter mapped to different sizes of pee pad refills in the “Compare with similar items"section. This is also an example of an unintuitive similarity: if I use clumping litter for my cat, it is unlikely that I will use pee pads too.

So each week I’d write a Perl script to crunch clickstream/conversion/feedback data and then remove or reorder some of the mappings in the BDB file based on my algorithm that week. We’d now have two sets of similarities:

• A: the similarities emitted by Amazon’s collaborative filtering algorithm
• B: vikrams_algorithm_of_the_week(A), Amazon’s similarities from A with mappings removed or reordered using whatever approach I was trying.

Then I’d send an email to the team with a link to a CGI script I wrote that allowed us to qualitatively assess B against A. It was a little web page with an input box at the top where you could enter an ASIN and it would show you similarities from A compared to similarities from B.

We’d exchange emails about the quality of my similarities or talk about them in a meeting. Then my mentor would decide whether or not we would push it to production.

Most weeks we’d push something to production and A/B test it. I don’t know what percentage of the site saw my similarities, but I suspect it was low: it would take a few days for us to get conclusive results and even in 2007 Amazon got tons of traffic.

I threw a lot of things at the wall.

Two approaches that I remember relied on the idea that users will simply click more on items on the left side of a page. It is well-known that page position is a massive determinant of clickthrough rates. The “Customers who Bought this Also Bought” widget was laid out from left to right, so items in the first slot had a baked-in “boost”.

So, if that is true yet we see a similarity in the first slot “underperform”, maybe we should reorder it. Here are two different ways I did that:

1. A basic approach: if an ASIN in slot 1 has a lower clickthrough rate than an ASIN in slot 2, swap slots 1 and 2.
2. A more complicated approach: if an ASIN in slot 1 has a statistically lower clickthrough rate compared to the ASIN in slot 2, swap slots 1 and 2.

I don’t remember the specifics for #2. It might have been something like:

1. Get the difference in clickthrough rate between slots 1 and 2.
2. See where it falls on the distribution of “difference in clickthrough rate between slots 1 and 2”.
3. Decide it’s underperforming if it is some distance away from the mean.

I spent the entire summer trying stuff like this and published a log of what I did on Amazon’s internal wiki.

Some takeaways I recall:

• None of these approaches improved conversions over the baseline: I do remember thinking my similarities were better in some of our qualitative assessments, but, site-wide, customers voted with their $ and showed that they were worse.
• Simple approaches are better than complicated ones: any time I got fancy, like in the more complicated approach above, the performance of my similarities tanked.
• Invalidating a bunch of approaches and documenting them is still useful: with this work, the team had a set of hypotheses they could either ignore or approach again in the future with more clarity. It’s a great project to deploy an intern to.
• None of these approaches come close to solving the Harry Potter problem: that book’s Amazon detail page will be forever etched in my brain.

There are many wildly qualified people who have worked on similarities and recommendations at Amazon, Netflix and elsewhere the last 15 years. Greg Linden started and led a lot of the personalization work at Amazon and has written about some of it on his blog.

I don’t know if Greg was there in 2007. My mentor and the team’s manager during my internship keep a much lower profile, but were also extremely talented.

👋 to Wes & Brent if you see this! Thank you for setting me up with a delightful and impactful experience 16 years ago.

I recently prototyped a hands-free inventory counting system for sommeliers at a popular Manhattan Italian restaurant with Gabriel, my partner at Baxter.

I know the general manager (GM) at this restaurant – henceforth referred to as Ristorante – personally and he hates inventory counts. Most restaurant GMs do. They’re labor-intensive yet extremely important: regular inventory audits are critical for restaurants to control their margins.

Ristorante can clear well into 5 figures of revenue in a day. And its GM, being a skilled and responsible operator, requires his team to do an inventory audit every month. So Ristorante’s two sommeliers arrive early in the morning on the 1st of every month to count over 1,500 different wines stored across 3 bars, a cellar, a fridge, and a storage room. It takes them over 6 hours, and the task bleeds into daily operations.

It’s painful.

It’s also just the kind of messy real-world problem that Gabriel and I want to try to solve for a market: is there any way we could cut the time it took these sommeliers to count inventory by half?

I popped into Ristorante on April 1st, a busy Saturday, to observe an inventory count.

Ristorante’s wine storage is organized but cramped. At the bars, many wine bottles sit on high shelves and sommeliers might be standing precariously on a stool or counter. Their hands are often not available – they’re moving boxes, bottles, or holding a bottle up so they can identify what it is.

Counting wine looks something like this:

1. Identify the box/bin or a bottle of wine you want to count. Count it.
2. With a laptop or phone next to you, look up the wine you’re looking for in Bevager, Ristorante’s inventory management system.
3. Enter the count and save it.

Each wine takes anywhere from 15-60 seconds, of which 80% of the time goes to step 2: looking up the wine in Bevager. This is due to a combination of the following:

• Context switching: from using one’s arms and being perched somewhere, to using a laptop or phone.
• Wine names are entered into Bevager inconsistently: in some cases they use the wine name, in others they use the producer. These names might be on the front or back of the wine.
• Words in wine names are frequently repeated: “chateau”, for example.
• Vintages: you might need to pick the right vintage from 5+ entries.
• Bevager bugginess/search quality: Bevager is a fine system, but its inventory audit feature could use some love.

These are all solvable problems that take up a significant percentage of the time it takes to inventory one wine bottle.

So we decided to start by helping a sommelier retrieve a wine in Ristorante’s inventory system quickly and accurately without having to use her hands, with her voice.

Prototyping voice search for wines in 3 days

A critical component of our prototype was a voice search feature that allowed a sommelier to say the name of a wine and retrieve it correctly from over 1,500 SKUs in her inventory.

There are a bunch of obstacles to getting this kind of voice search to work well.

• Getting accurate transcriptions, even with state of the art models, is hard. We have to deal with French and Italian wines, various accents, and restaurant background noise.
• Inaccurate transcriptions are different from typos and misspellings. Handling them requires a different set of tricks.
• The user’s not at a keyboard. Auto-complete is an incredible feedback mechanism, but we don’t have access to it.
• There is exactly one relevant record. A sommelier’s search for the wine they’re looking for has only one matching record. It must be a top result, otherwise the search failed.

Perhaps the closest analog to this search use case is when you ask your phone to call a contact (e.g. “Google, call Katie.”). It turns out that this is pretty challenging!

Our goal was to validate feasibility and demand for a solution, so we needed to show something working well enough quickly.

I also wanted this to be easy to understand and operate: I didn’t want to have to (re-)learn, configure, and tune Solr, ElasticSearch, or Typesense. Besides, I only had about 1,500 records and these tools aren’t really designed to deal with the search use case I describe above, out of the box.

I managed to cobble together workable voice search in about 3 days using off-the-shelf components, a bag of information retrieval tricks, ~300 lines of Python, and a bunch of trial and error. It’s imperfect but it’s quick to implement, simple to operate, and it works with many classes of transcription errors with plenty of avenues for improvement.

• Code: The codebase is on Github.
• Demo: You can give it a try on Streamlit Cloud.
• Is this available? What happened next? Can you make this for me? I answer these questions in the Appendix.

Here’s how it works!

A summary of my approach

I use OpenAI’s Whisper for audio transcription and DuckDB to store and query my search index.

DuckDB’s full text search (FTS) extension easily handles the happy path: accurate transcriptions. The FTS extension is easy to use, but it does not handle misspellings or, in my case, poorly-transcribed audio.

OpenAI’s Whisper model is impressive, but:

• There is background noise in restaurants.
• Non-native speakers pronounce French and Italian wines poorly.
• Both of these issues affect audio clarity and, in turn, transcription quality.
• Even without these issues, Whisper is not perfect.

So my implementation needs to handle inaccurate transcriptions.

While there’s plenty of prior art on handling misspellings in search, the same techniques don’t work as well with bad audio transcriptions. This is because inaccurate transcriptions don’t look like misspelled words. They often look like gibberish – made up words, or strings of words that sound like the audio but don’t make sense together.

So I built two of my own search indexes in DuckDB to handle “mistranscriptions” using Metaphone, a phonetic algorithm that maps similar-sounding words to the same pronunciation.

In total, I have 3 indexes: 1 full text search index on wine names, and 2 that use Metaphones, which I’ll cover in greater detail below

At query time I do the following:

1. Query the indexes in 5 different ways. Each method excels at different kinds of queries or transcription errors. I get 10 results from each method, increasing the likelihood that the wine the user is looking for is in my result set.
2. Rank all those results using text similarity. I get the Jaro-Winkler similarity score between each result and the transcription. This tends to bubble the result up to the top 5.
3. Return the top 5 ranked results. I’d say 80%+ of the time the result I was looking for ended up in the top 5, and it fared well with “adversarial” audio (comically bad accents, yelling in the background, etc.)

What do inaccurate transcriptions look like?

Inaccurate transcriptions are usually close to how the words actually sound.

Let’s say we have a wine in our inventory called Chateau Champignon.

Here are some potential transcriptions of “chateau”:

• shadow
• shatto
• chat oh

Here are some potential transcriptions of “champignon”:

• champagne on
• shomp inyon
• champ onion
• sham pig non

We may end up with any permutation of transcriptions for “chateau” and “champignon”. For example:

• shadow champagne on
• chat oh shomp inyon
• shatto sham pig non

Traditional search techniques to handle misspellings fail spectacularly on these kinds of errors. None of them look like misspellings or typos.

But they do sound the same, and this is where phonetic algorithms come in.

You can map inaccurate transcriptions to the same string using phonetic algorithms

Phonetic algorithms map words to pronunciations.

Different phonetic algorithms have different goals in mind, but there is a whole class of them that is designed to map similar-sounding words to the same pronunciation.

The earliest one of these is called Soundex, which was developed in the late 1800’s and patented in 1920, well before the advent of computing!

Why? To map similar-sounding surnames to each other for the US Census. The history here is really neat, and you can read this post by Luke Otwell to learn more if you’re interested.

In my implementation I used Metaphone, a successor to Soundex, for which there is an open-source Python implementation in pyphonetics.

Metaphone maps chateau champignon to XTXMPNN.

Here’s how Metaphone maps our transcriptions of Chateau Champignon above:

chateau

• shadow: XT
• chat oh: XT
• shatto: XT

champignon

• champagne on: XMPNN
• shomp inyon: XMPNYN
• champ onion: XMPNN
• sham pig non: XMPNN

various permutations of the above

• shadow champagne on: XTXMPNN
• chat oh shomp inyon: XTXMPNYN
• shatto sham pig non: XTXMPNN

Almost all of these map directly to the same Metaphone as Chateau Champignon!

To put this insight to work, our code creates two additional indexes: the Exact Metaphone Index and the Metaphone Token Index. These indexes are constructed in index.py.

The two Metaphone-based indexes

Assume we have three wines in our inventory:

1. Chateau Champignon
2. Cavalleri Collezione Esclusiva Chardonnay
3. Chateau la Bridane Saint Julien

Here’s how we construct our two Metaphone indexes on those wines.

1. The Exact Metaphone Index

In this index we map the Metaphone for each wine’s full name to itself:

• XTXMPNN → Chateau Champignon
• KFLRKLSNSKLSFXRTN → Cavalleri Collezione Esclusiva Chardonnay
• XTLBRTNSNTJLN → Chateau la Bridane Saint Julien

This index is just a 1-to-1 mapping. It allows us to reverse look up a metaphone to a wine’s full name as it exists in our index.

2. The Metaphone Token Index

In this index we tokenize each wine name, get the Metaphone for each token, and construct an index on those tokens.

Here’s what the first step looks like:

1. Chateau Champignon → XT XMPNN
2. Cavaelleri Collezione Esclusiva Chardonnay → KFLR KLSN ESKLSF XRTN
3. Chateau la Bridane Saint Julien → XT L BRTN SNT JLN

Let’s call these “Metaphone strings”. Each token in the string is the Metaphone for a token in the original wine name. These strings are what we index.

Assuming that 1, 2, 3 are IDs that map to the wines in the list above, the Metaphone Token Index looks like this:

• XT → [1, 3]
• XMPNN → [1]
• KFLR → [2]
• KLSN → [2]
• … and so on.

Conveniently, this is just a regular full text search index on the Metaphone strings above. So we can use DuckDB’s full-text search extension again, saving the work to write code to construct or query the index.

This index allows us to reverse look up Metaphones for individual words in our transcript.

For example, a speaker might say only “champignon”, for which we get an inaccurate transcription like “shampinon”. Or a speaker might say 3 words in a wine name, but two of them are transcribed entirely incorrectly while the third has a Metaphone match.

This index gives us the means to salvage an inaccurate transcript as long as there’s a Metaphone match in it, somewhere.

Five query-time approaches

Combined, the following approaches are greater than the sum of their parts. Individually, they excel at dealing with different kinds of queries and transcription errors.

Each of the query methods is implemented in search.py.

Let’s add one more wine to our index for the following explanations. There are now 4 wines in our inventory:

1. Chateau Champignon
2. Cavalleri Collezione Esclusiva Chardonnay
3. Chateau la Bridane Saint Julien
4. Guy Breton Morgon

Here are the five query approaches we use.

1. DuckDB Full-Text Query

This was the first one I implemented, and it uses DuckDB’s full-text search extension.

Under the hood, DuckDB’s FTS extension implements BM-25, a widely-used ranking function.

It constructs the index with some basic defaults for English: a list of 571 stopwords, the Porter stemmer, lowercasing all text, ignoring all non-alphabetic lowercase characters, and stripping accents.

Pros

• BM-25’s ranking algorithm performs well: a query for “chateau champignon” will rank Chateau Champignon first, followed by the other hundred that include “chateau”.
• It works well as long as the most unique words of the wine name are transcribed accurately. For example, “shadow champignon” will likely return “Chateau Champignon”, but “chateau shampinyon” is a crapshoot because there are hundreds of wines with “chateau” in their name.

Cons

• It fails spectacularly on inaccurate transcriptions, usually returning zero or irrelevant results.

2. Exact Metaphone Query

Here, we do a reverse lookup for the transcription’s Metaphone in the Exact Metaphone Index I describe above:

1. Get transcription metaphone: “shadow champagne on” yields XTXMPNN
2. Reverse-lookup: XTXMPNN yields the record for Chateau Champignon.

Pros

• Works with inaccurate transcriptions.
• High precision: it tends to match the exact wine the user is searching for, and nothing else.

Cons

• The user must say the entire wine as it exists in our index. If they just say “champignon” instead of “chateau champignon”, we are out of luck.
• Fails when a transcription’s Metaphone does not match. An example is the Metaphone for “shadow shomp inyon”: XTXMPNYN, which has a “Y”.

3. Metaphone Token Query

Here, we first form our query. Let’s say the speaker is retrieving Chateau la Bridane Saint Julien.

They say only “julien bridane”, but they incorrectly pronounce each word. It is transcribed as "julian breedan".

1. Tokenize the transcript: "julian breedan" becomes ["julian", "breedan"]
2. Get the Metaphone for each token: ["julian, "breedan"] becomes ["JLN", "BRTN"]
3. Join the Metaphones: ["JLN", "BRTN"] becomes "JLN BRTN"

Then we query the Metaphone Token Index described above with our query.

Because our query contains "JLN" and "BRTN", the wine Chateau la Bridane Saint Julien will rank highly. Its tokens map to: ["XT", "L", "BRTN", "SNT", "JLN"].

Guy Breton Morgon will also rank. Its tokens map to ["K", "BRTN", "MRKN"].

Recall that we built this index using DuckDB’s full-text search extension, so we query it with the extension too.

Pros

• Works with inaccurate transcriptions.
• Works when speakers partially say the name of a wine.
• Works when the order of the words spoken do not match the wine’s record.

Cons

• This approach sometimes yields many irrelevant results that have similar-sounding words in their names.
• Fails when a transcription’s Metaphones do not match.

4. “Similar Token Metaphones” Query

Methods 1 to 3 handle:

• Accurate transcriptions
• Inaccurate transcriptions with matching Metaphones

This takes us far! But I would still get transcriptions without any matching Metaphones and I wanted to handle those.

I attempted a number of approaches, all using Levenshtein edit distance in creative ways on increasingly worse transcriptions. All of them worked.

The catch? They also introduced a lot of noise: the worse a transcription, the more permissive I had to be, which meant that my result set would also include lots of wines that looked and sounded nothing like the wine I was retrieving.

In information retrieval parlance, my search recallsuffered: I was retrieving an ever-higher percentage of irrelevant records.

I had a prototype to ship, so I settled with a basic method that:

1. Didn’t return a torrent of random records.
2. Could handle one extra, absent, or incorrect consonant that caused a token to generate a different Metaphone. For example, “bribane” maps to BRBN, while “bridane” – a token in our index – maps to BRTN.

Here’s how it works. Let’s say the speaker is retrieving Chateau la Bridane Saint Julien (Metaphones: XT L BRTN SNT JLN*).*

They say “bridane saint julien”, but we get a really bad transcription: "brenton sane julia", whose tokens map to ["**BRNTN**", "**SN**", "**JL**"].

We’re out of luck here with methods 1 to 3. Each will return zero results because we have no matching tokens nor matching Metaphones.

A “Similar Token Metaphone” query does the following:

1. Tokenize the transcription: "brenton sane julia" turns into ["brenton", "sane", "julia"].
2. Get all tokens in our index that are exactly 1 edit distance away from each token in our query: for "brenton" this is ["breton"], for "sane" this is [] (an empty set), for "julia" this is [] (an empty set). So we get ["breton"].
3. Join all tokens from step 2: ["breton"] becomes "breton". If we had more than one token, we’d join all of them with a space.
4. Use the string from step 3 to run a Metaphone Token Query. Our input to the query method described above is "breton".

Because the Metaphone for "breton" is BRTN we get two records, one of which is our target wine:

• Chateau la Bridane Saint Julien (Metaphones: XT L BRTN SNT JLN)
• Guy Breton Morgon (Metaphones: K BRTN MRKN)

Using a Metaphone Token Query in Step 4 instead of a DuckDB Full-Text Query casts a slightly wider net, which increases the likelihood that we catch the result we want when all our other query methods have failed.

In my limited observation, it worked spookily well without returning lots of irrelevant results. It is a total delight to read an awful transcription like “brenton sane julia” after you speak yet have it retrieve the record you’re looking for.

Pros

• Might succeed even when there are no matching Metaphones in a transcription.

Cons

• Might return a lot of irrelevant results.

5. “Metaphone Substring” Query

I added this final query approach to handle exactly one class of transcription error: transcriptions that break one word into multiple words. This happened fairly frequently.

Let’s say the speaker is retrieving Chateau Champignon, they say “champignon”, and the transcription we receive is "champ onion".

Query approaches 1 through 4 all fail:

1. There is no wine that matches the tokens "champ" or "onion".
2. The Exact Metaphone for "champ onion" is XMPNN, for which there is no match in the Exact Metaphone Index.
3. The Metaphone Tokens for "champ onion" are XMP and ONN, for which there are no matches in the Metaphone Token Index.
4. There are no tokens 1 edit distance away from "champ" and "onion"

It turns out that many of these cases have Exact Metaphones that are substrings of their target wine. In this case:

• The Exact Metaphone for "champ onion" is XMPNN.
• The Exact Metaphone for "chateau champignon" is XT XMPNN.

… so a substring search in our Exact Metaphone Index succeeds where all our other query methods have failed.

Pros

• It’s simple.
• Handles a transcription error that happens frequently.
• Doesn’t return a lot of irrelevant results.

Cons

• Doesn’t help with anything other than this one class of error.
• Doesn’t always work: “shomp inyon” maps to “XMPNYN”.

Ranking results from all five approaches using Jaro-Winkler similarity

When we execute all five of our query approaches:

• We don’t know which approach, if any, has the result the speaker is looking for.
• Each approach might return multiple results.

So how do we surface the target wine from all these results?

It’s safe to assume that the transcript we receive is similar, by some measure, to the speaker’s target wine. If it’s not, then the transcript is probably too inaccurate – would a human be able to figure out which wine the sommelier is referring to?

The challenge here is determining an objective measure that consistently brings the target wine into our top 5 results. Doing this well requires data from real users and time to experiment, neither of which we had.

Aiming for a quick solution, I tried using text similarity using whatever was available off-the-shelf using DuckDB.

It turns out that Jaro-Winkler similarity works well enough for our purposes.

My implementation:

1. Gets up to 10 results from every query approach.
2. De-duplicates the set of results, because the same result might appear from multiple approaches.
3. Calculates Jaro-Winkler similarity between each result and the transcription.
4. Returns the top 5 ranked results.

This is a workable implementation, but if you play around with the demo you’ll observe it fail in basic ways, like ranking the target wine 5th when it seems implausible that the 1st, 2nd, 3rd, or 4th results are correct.

The problem with using text similarity is that we’re discarding everything we know about our query approaches and the kinds of results they return. For example, if we get results from a DuckDB full-text search, it means the transcript contains correctly-spelled tokens in our search index, which increases the probability that we have an accurate transcript. That’s information we could use to rank our search results!

How can this be improved?

With lots of experimentation! It helps enormously to have data from real users searching for wines.

In a real setting, I’d want to:

• Know what percentages of searches are successful over time.
• Have a set of (transcript, target wine) pairs that I can test against.
• Run A/B tests pitting search algorithms against each other.

Alas, we did not promote this prototype to production.

If I spent more time on this voice search implementation, here are some of the things I might try.

Ranking results: scores, weights, and phonetic algorithms

• Weight results from query approaches differently.
• Come up with a way to score results phonetically.
• Give higher scores to results if they surface in multiple approaches.
• Use different, or multiple phonetic algorithms for the task.

Using LLMs

• Use an LLM to surface the likeliest wines. I tried this briefly and couldn’t get it to work well, but it has promise.
• Use an LLM to generate plausible mis-transcriptions of every wine. (GPT-4 generates great, plausible transcriptions). Index all of these.
• Use the mis-transcriptions from the above to fine-tune an LLM to do this task.

Appendix

Is this product live? If not, will it be?

No, and probably not. Everyone hates inventory counts, but this was not a pressing enough problem for the GM at Ristorante or the 10 wine stores, meat/cheese shops, and restaurants I spoke with.

In the time following the week we spent building our initial prototype, we did not gather enough evidence for us to continue working on this product.

Can you build this, or something else, for us?

Yes.

I co-own Baxter, a product development and contracting company, with Gabriel Garrido. We’re a two-person team of seasoned engineers and we’ve built software and teams at numerous early stage startups.

Thanks to Gabriel Garrido, Phil Eaton, Pierre Jambet, Andy O’Neill, Chuck Groom, and Aaron Rosen.

👋

A few weeks ago we released Baxter’s free funnel analysis SQL generator (no longer available) supporting nine different SQL dialects. You can read about how the tool works and the assumptions we make about your data in this introductory blog post

We encourage you to copy/paste the queries our tool generates for your use cases! Funnel analysis queries are tough to write and the ones we generate are well-tested and will save you time.

But we’re also proponents for understanding how the code you’re using actually works. In that spirit, we’re going to share the ideas behind the query strategies we employ for users and SQL enthusiasts alike.

This is is the first post in a series. In it, I’m going to talk about how to write funnel analysis queries using three modern, common SQL features:

• Window functions
• Range frames (instead of row frames)
• Regular expressions

These queries can analyze loosely ordered funnels with a total sequence time constraint. These terms are defined here.

We use this strategy to generate funnel analysis queries for BigQuery, Spark, ClickHouse, DuckDB, PostgreSQL, and MySQL.

Idea #1: You can use regular expressions to match funnels if you collapse sequences of events into strings.

Here’s how you might collapse a sequence of events into a string using DuckDB’s SQL dialect:

select
    user_id,
    string_agg(
        case
            when event_name = 'View Product'
            then 'A'
            when event_name = 'Add to Cart'
            then 'B'
            when event_name = 'Place Order'
            then 'C'
        end
    ) as event_sequence
from events
where ...
group by user_id

-- user_id | event_sequence
-- ------------------------
--       1   'ABAAC'
--       2   'AAAAAAB'
-- ...

If our funnel steps are “View Product” → “Add to Cart” → “Place Order”, the query above collapses each event into single-character identifiers and turns our funnel steps into “A” → “B” → “C”.

With strings of event sequences, using regular expressions to match how far users have progressed through a funnel is trivial and efficient.

The following query assumes we have a table called event_sequences with columns user_id and event_sequence, generated by our query above.

select
    -- If there's an A, a user progressed to step 1.
    (select count(*)
     from event_sequences
     where regexp_match(event_sequence, 'A') as step_1_count,

    -- If B eventually follows A, a user progressed to step 2.
    (select count(*)
     from event_sequences
     where regexp_match(event_sequence, 'A.*B') as step_2_count,

    -- ... and if C eventually follows B, a user progressed to step 3.
    (select count(*)
     from event_sequences
     where regexp_match(event_sequence, 'A.*B.*C') as step_3_count,

from event_sequences

-- step_1_count | step_2_count | step_3_count
-- ------------------------------------------
--          142             93             20

Note that our regular expressions permit any other events to occur between each funnel steps. This satisfies our loose ordering constraint.

Recall that we need to consider two important constraints:

1. All the events must occur within a specified period: a start and end time.
2. All the events must occur within a specified duration. For example, all events A, B, and C might need to occur in a 24 hour period in our analysis. (This is our total sequence time constraint.)

#1 is easy – you can filter for events that fall within the start and end time in your where clause.

#2 is trickier and where window functions with range frames come into play.

Idea #2: Assemble a sliding window of events that occur within your funnel’s duration by using window functions and range frames.

Let’s assume these are our funnel analysis parameters:

1. The events must occur in the month of January 2023.
2. Our funnel steps are: View Product, Add to Cart, and Place Order.
3. The funnel duration is 24 hours.

Let’s start with the following query, which helps us satisfy parameters #1 and #2 and collapses event sequences into strings. We’ll build on it next to satisfy parameter #3.

This query is written in DuckDB’s SQL dialect.

select
    user_id,
    string_agg(
        case
            when event_name = 'View Product'
            then 'A'
            when event_name = 'Add to Cart'
            then 'B'
            when event_name = 'Place Order'
            then 'C'
        end
    ) as event_sequence
from events
where event_timestamp > '2023-01-01' and
      event_timestamp < '2023-02-01' and
      (event_name = 'View Product' or
       event_name = 'Add to Cart' or
       event_name = 'Place Order');
group by user_id;

-- user_id | event_sequence
-- ------------------------
--       1   'ABAAC'
--       2   'AAAAAAB'
-- ...

We’re not doing anything fancy yet. This query assembles event sequences that:

• … match our funnel steps (View Product, Add to Cart, and Place Order)
• … and occur within the period that we’re analyzing (Jan 2023)

If we used regular expressions to count users who progressed through our funnel using the query results above, our counts would be incorrect. We have strings of event sequences, but there is no guarantee that these event sequences occurred within a 24-hour period.

The following query (using DuckDB’s SQL dialect) builds on the query above using window functions and range frames to make that guarantee.

select
    user_id,
    string_agg(
        case
            when event_name = 'View Product'
            then 'A'
            when event_name = 'Add to Cart'
            then 'B'
            when event_name = 'Place Order'
            then 'C'
        end
    ) over funnel_duration as preceding_event_sequence,
    case
        when event_name = 'View Product'
        then 'A'
        when event_name = 'Add to Cart'
        then 'B'
        when event_name = 'Place Order'
        then 'C'
    end as event_identifier
from events
where event_timestamp > '2023-01-01' and
      event_timestamp < '2023-02-01' and
      (event_name = 'View Product' or
       event_name = 'Add to Cart' or
       event_name = 'Place Order')
window funnel_duration as (
    partition by user_id
    order by event_timestamp
    range between interval 24 hours preceding and
                  current row
)

-- user_id | preceding_event_sequence | event_identifier
-- -----------------------------------------------------
--       1   ''                         'A'
--       1   'A'                        'B'
--       1   'AB'                       'A'
--       1   'ABA'                      'A'
--       1   'AA'                       'C'

Unlike our first query, which had a row for every user, there exists a row for every event in this result set. And preceding_event_sequence contains all the events that occurred within the 24 hours preceding each event!

Our first query showed that user 1’s sequence of events over the course of January 2023 was ABAAC. Examine the event_identifier column in the results above and you’ll find you can reconstruct user 1’s ABAAC event sequence.

But from these results you’ll also find:

• User 1 progressed through A and B within a 24 hour period.
• User 1 did not progress through A, B, and C within a 24 hour period.

The range frame in our window specification ensures that preceding_event_sequence maintains a sliding window of events within the 24 hours preceding each event:

...
window funnel_duration as (
    partition by user_id
    order by event_timestamp

    -- This is the range frame.
    range between interval 24 hours preceding and
                  current row
)
...

If you’ve used window functions before, you’re likely familiar with row frames. Row frames allow users to assemble a sliding window of some number of rows relative to the current row. Range frames are different: they allow users to assemble sliding windows of rows that satisfy some range of values relative to the current row.

In our case, those rows must have event_timestamp values within the 24 hours preceding our current row’s event_timestamp value.

Window functions are already useful, but range frames make them wildly powerful. They might be my favorite modern SQL feature to-date.

Assemble the rest of the query with your SQL bag-of-tricks: CTEs, aggregate functions, etc.

The ideas above decompose our funnel analysis problem into two steps that get us most of the way:

1. Collapse event sequences into strings to match funnels.
2. Only collect event sequences that occur within a 24 hour window.

There are a lot of ways you might assemble your final result and they rely on the techniques you use writing everyday SQL. I won’t be going into the details here, but you can poke around our funnel SQL generator (no longer available) to see how we do it.

But make no mistake: the SQL will be verbose, impossible to parameterize without using a template language like Jinja (used by DBT), and it won’t make for light bedtime reading.

A note on our MySQL implementation

Our MySQL implementation works with MySQL 8. But it’s a bit of hack that uses undocumented behavior. This is not the case for any of our other implementations!

Let me explain.

Unlike other dialects, there is no string aggregation function that works in a windowing context in MySQL. group_concat aggregates values into a string, but it does not work as a window function so we can’t use it to implement this query strategy.

MySQL doesn’t have a native list or array data type, either! With ClickHouse and Spark, we’re able to a) aggregate event sequences into a list, and then b) concatenate all elements in that array. So we can generate ['A', 'B', 'C'] and then turn that into 'ABC'. We can’t do that in MySQL.

MySQL does have a JSON type. And we can aggregate event sequences into a JSON array. These look like ['A', 'B', 'C'], but there’s no equivalent function for us to turn that into 'ABC'.

Under the hood, however, MySQL’s regular expression matching actually works on JSON arrays. It does so in a peculiar way – by matching against the string ["A", "B", "C"]

Here’s how that works:

-- 'a' eventually followed by 'b' matches

select REGEXP_LIKE(JSON_ARRAY('a','b','c','c','d'), 'a.*b');
+------------------------------------------------------+
| REGEXP_LIKE(JSON_ARRAY('a','b','c','c','d'), 'a.*b') |
+------------------------------------------------------+
|                                                    1 |
+------------------------------------------------------+
1 row in set (0.00 sec)



-- 'b' follows 'a' in this array. But 'ab' does not match!

mysql> select REGEXP_LIKE(JSON_ARRAY('a','b','c','c','d'), 'ab');
+----------------------------------------------------+
| REGEXP_LIKE(JSON_ARRAY('a','b','c','c','d'), 'ab') |
+----------------------------------------------------+
|                                                  0 |
+----------------------------------------------------+
1 row in set (0.00 sec)



-- The string we're matching is actually:
-- ["a", "b", "c", "c", "d"]
--
-- So let's try matching 'a", "b` instead.

mysql> select REGEXP_LIKE(JSON_ARRAY('a','b','c','c','d'), 'a", "b');
+--------------------------------------------------------+
| REGEXP_LIKE(JSON_ARRAY('a','b','c','c','d'), 'a", "b') |
+--------------------------------------------------------+
|                                                      1 |
+--------------------------------------------------------+
1 row in set (0.00 sec)

I know this is hack! But:

• We suspect there is a small minority of users who are using MySQL for this task.
• Our MySQL queries pass all our tests.
• This strategy is significantly faster than our fallback strategy (self-joins).

If you run into issues using our MySQL implementation, or you’d like us to fall back to another strategy for it, please reach out to us.

Why can’t Redshift, Snowflake, or SQLite use this query strategy?

Different reasons!

• Redshift doesn’t support range frames in window functions. Its dialect is ancient and stuck somewhere in Postgres 8, the database it’s forked from.
• Snowflake has a rich, modern SQL dialect. But it doesn’t doesn’t support sliding range frames, which the window function/range frame strategy that we’re using requires.
• SQLite actually supports this strategy! But it won’t work out of the box. While SQLite has dedicated regular expression-matching syntax (where REGEXP ), it actually just calls a function that users must supply. Most implementations do not provide one, so we use our fallback strategy instead.

Some observations on the above:

• It is frustrating how much Redshift’s SQL dialect lacks in features.
• Snowflake has this one odd omission in its support for window ranges – but it does implement MATCH_RECOGNIZE, a powerful and expressive operator that’s perfect for funnel analysis.
• I am confused as to why SQLite dedicates language syntax to regular expression matching without giving users the facility to use it out of the box.

Next: stay tuned to learn how to use Snowflake’s MATCH_RECOGNIZE to implement funnel analysis queries

Snowflake’s dialect is powerful. You can see MATCH_RECOGNIZE in action in our funnel SQL query generator (no longer available). I’m excited to share how that works.

Today we’re releasing a free tool that generates multi-step funnel analysis queries in nine SQL dialects: BigQuery, Snowflake, Redshift, Spark, ClickHouse, DuckDB, PostgreSQL, MySQL, and SQLite.

It’s free, it’ll save you time and tears, and it’s available here (no longer available).

Give it a spin! Reach out to us if you have requests or run into issues!

Implementing funnel analysis queries across nine SQL dialects will lead you into some pretty goofy corners of SQL implementations. Stay tuned to the Baxter blog for posts about query strategies we’ve implemented and the SQL quirks that we made friends and enemies with along the way.

For now, let’s get you situated.

The rest of this post outlines the assumptions our funnel SQL generator makes about your event data and the kinds of funnel analyses it enables.

I’m also going to share a bit about the strategies we employ and how we test them so you can be confident in their results.

Table of contents

• What data model do these queries require?
• What kinds of funnel analyses do these queries allow me to do?
• Nine SQL dialects, three funnel query strategies
• How do you know these are correct?
• Questions? Requests? Issues?

What data model do these queries require?

If you want these queries to work out of the box, you’ll need an events table with at least these three 3 columns:

• user_id: an identifier — the type won’t actually matter
• event_name: the event’s name as some textual type (TEXT, VARCHAR, etc.)
• event_timestamp: the time at which an event occurred as a TIMESTAMP or DATETIME-compatible type.

If your table or column names differ, you can modify them in the SQL we generate. And if you have more columns, these queries will still work. In fact, that’s a pretty common scenario!

You might already have a wide table of events if:

• You use Snowplow to track your events.
• You have GA4 writing data to BigQuery.
• You use one of Segment or Rudderstack’s data warehouse connectors.
• You pay Heap/Amplitude/Mixpanel/Posthog to send raw event data to your warehouse.

Some companies do custom event tracking or build data pipelines that materialize event data this way, too.

For example, a row in your orders table might turn into a row in your events table. If a user with id 23 placed an order on January 22nd, 2023, you might create this row in events: 23, Order Placed, 2023-01-22 13:34:01, (various columns with order properties)

The are two primary benefits to modeling event data as a wide table with many columns:

• One data model answers many questions: you can perform a lot of common product analytics queries with this data model.
• Performance: you prevent joins by having one table with all the dimensions and metrics you want to query.

If you don’t have a table of events, you’ll need to massage your data so that it’s structured this way.

Finally, you might have more complicated predicates in your funnel steps like event_name == 'Pageview' and url_path LIKE '...'. You can modify the queries we generate with your own predicates and remain confident that the query will yield accurate multi-step funnel conversion numbers.

What kinds of funnel analyses do these queries allow me to do?

You can analyze funnels…

• … with any number of steps
• … that must happen within some duration
• … where other events may happen in between each step

Another way to state the above is that these queries support funnel analyses with loose ordering and a total sequence time constraint.

They do not support strict ordering or a stepwise time constraint.

Credit for this terminology goes to TJ Murphy — a certifiable funnel query expert — who shared it with me some months ago. I don’t know if this language is actually formalized anywhere.

Here’s the difference between loose and strict ordering.

Here’s the difference between total sequence and stepwise time constraints.

Growth marketers, product managers, and analysts most commonly want loose ordering and a total sequence time constraint when they perform funnel analyses. That’s what our funnel query generator supports today.

Funnel queries that implement strict ordering and a stepwise time constraint are substantially different. We haven’t implemented them yet but we’ll work on it if folks are keen on it, so tell us if you are.

Nine SQL dialects, three funnel query strategies

At release we employ three funnel query strategies across nine SQL dialects.

If we don’t support your SQL dialect and you’d like for us to, we want to help!

These are the strategies we employ today:

• Self-joins: this is our fallback for less full-featured dialects. We use it for Redshift and SQLite.
• MATCH_RECOGNIZE: our Snowflake funnel query strategy is based on MATCH_RECOGNIZE, a powerful feature unavailable in other dialects we support.
• Window functions, range frames, and regexes: we combine these techniques to implement funnel queries for every other dialect we support. (Range frame specifications might be my favorite modern SQL feature to-date.)

Right now, our goal with Baxter’s funnel SQL query generator is to help users solve their immediate problem: writing a funnel analysis query that just works.

While we haven’t tested the performance behind these strategies, we know with certainty that one of them scales poorly: the self-join. We’ll benchmark and improve the strategies we use for each dialect over time.

If you have an approach that works particularly well for your dialect and data warehouse, we’d love to hear about it.

How do you know these are correct?

We’ve built a synthetic dataset that we run and check our queries against. We also have test harnesses that run generated queries against a data warehouse for every dialect we support.

We do a lot of automated testing!

If you come across a bug, please let us know and we’ll get it fixed promptly.

👋

I recently wrote about how I don’t like any of the funnel or cohort analysis tools available to me.

Gabriel and I (Vikram) are working on an alternative: Baxter.

Everyone has event data today. Maybe you’re collecting events via Snowplow, Segment, or Rudderstack. Or maybe you’re using data from your application’s database, which will usually be more trustworthy and critical to your app than events you instrument.

Companies pay a lot of money to send event data to third parties like Heap, Amplitude, Mixpanel, and Posthog (henceforth HAMP) so they can use their purpose-built event analysis tools. Not only is this unnecessary for most event analysis tasks, analysts commonly have to supplement insights from these tools anyway because the tools fall short or the data is suspect.

HAMP has its place, especially in real-time product analytics, experimentation, and monitoring. Their tools are great at answering high-level questions about user activity. But they don’t cater to analysts who are working on extracting valuable insight. All of that work is done in SQL, Excel, notebooks, and BI tools. Not in product analytics suites.

Our aim with Baxter is to help analysts follow lines of inquiry well beyond the point existing tools fall short. And we’ll do all this in your organization’s data warehouse, giving you the following advantages:

• You retain your data.
• You don’t chase inevitable discrepancies between your data and a third party’s.
• You can modify and use the queries Baxter generates in your existing workflows, giving you a gentle off-ramp when you do need to drop into SQL.

SQL is up to the task for most product analytics questions. It does the job for most data sizes that teams work with, too. But it’s challenging to write and parameterize correct, high-performing SQL queries for common event analyses.

Analysts shouldn’t need to painstakingly write SQL in order to explore and analyze event data. Their only alternative shouldn’t be to send their data to third parties and suffer tools that don’t serve them or integrate with their workflows either.

Interested? We would love to speak with you. Join our waitlist or reach out to us on Twitter.

Ten years ago, while starting the Data Engineering team at Harry’s, I wrote a suite of cohort analysis queries that looked more-or-less like the queries in this post.

The success of the razor and blades model hinges on high customer retention so we tracked it religiously from day one. Retention was so critical at Harry’s that I even wrote a haiku about it once.*

I wrote 15-20 permutations of cohort queries that we would analyze regularly. At Harry’s we cut our cohorts in a bunch of ways: the channel customers came from, the contents of their first order, age/gender, geography, etc. We analyzed account creation -> 1st order, 1st order -> 2nd order, 2nd order -> 3rd order separately.

There were a lot of charts. We wanted to compare them and see how they changed week-over-week and month-over-month.

I could have sent our data to Mixpanel at the time, but it didn’t allow us to explore our data this way. So I hand-rolled a lot of SQL queries and my colleague analyzed the data in Excel. It was cumbersome and time-consuming, but it was a critical analysis for the business.

This year I was retained as a fractional Head of Product for a long-time client and this summer I took a few days to dive into my client’s retention metrics. They have a loyal base of regular users and I wanted to figure out how they differed from folks who churned. So I reached for my trusty friend: the cohort analysis!

Imagine my surprise when my options were no different than they were a decade ago:

1. I can send my data to tool like Amplitude, Mixpanel, or Heap and use its tools.
2. I can write SQL and visualize data in Excel or a BI tool.

I didn’t like either of these options ten years ago! I still don’t.

While product analytics suites provide useful tools for exploratory analyses, I usually need to drop down to SQL to answer common follow-up questions. And ensuring accuracy is challenging: the data that ends up in these tools is frequently suspect because event instrumentation is so brittle. Last week I came across Olga Berezovsky’s feedback on Amplitude after its most recent conference and it resonates — Amplitude and its competitors don’t serve me well when I’m doing deeper analyses and I can’t rely on their accuracy.

Meanwhile, I love the flexibility I have when I’m writing SQL: I can query my data however I’d like to. I can clean it up, run spot checks, and write data quality tests to make sure my analysis is correct. But SQL queries for funnel and cohort analyses are challenging to write correctly and scale. Exploring funnels and cohorts this way — asking follow-up questions like, “Why is our March cohort so much more active than our February cohort?” — is time-consuming and I miss out on the interactivity I get from product analytics tools.

I want to be able to analyze behavioral data with the affordances that existing product analytics tools provide. But I don’t want to be shackled to those tools.

The tool I want sits in a happy middle:

• It’s interactive, like product analytics tools. I want to select parameters for my funnels and cohorts and get charts. I want to click around to answer follow-up questions.
• I can use the data I have. I don’t want to send the data to a third-party. I want to use the data I have in my application database, data warehouse, or on my local machine.
• It’s SQL-native and integrates with my workflows. I want to generate queries for 90+% of the charts I explore so I can use them in my BI tool, data pipeline, or notebook.
• I can go deep without having to write SQL. I want to be able to answer questions and explore data well beyond the point that product analytics tools fall short.

I want this to exist! So I’m working on it.

If you analyze behavioral data and the above resonates, I would be delighted to speak with you. Please say hello!

*You can read about my pursuit of poetry in the workplace on my secret sandwich blog. (Currently on hiatus.)

The end result is way less code that is much more performant. I also have the Polars API to work with on any result set from DuckDB, which lets me iterate more quickly. It’s handy!

That said, this was really painful for me to figure out. There are a lot of documentation gaps in these projects, I still don’t quite understand the Arrow APIs, and I’m new to Rust. If you’re looking for a shortcut, I published a Github repo showing how you can glue these APIs together.

If you just want to use these APIs, skip the rest of this post and visit the repo. I share more around how these APIs work, my journey to figuring it out, and questions that came up below.

Querying DuckDB

I started out using duckdb-rs, “… an ergonomic wrapper for using duckdb from Rust. It attempts to expose an interface similar to rusqlite.” The API is friendly, but as of this post’s publishing, duckdb-rs doesn’t support any nested data types, which I needed (my query results include lists). I looked into submitting a PR to support nested data types but my inexperience with Rust quickly put that to a halt.

The library code also starts from a fork of rusqlite and I think that makes things a bit challenging. SQLite does not support nested data types as far as I can tell (and I don’t see evidence of any such support in rusqlite). Reading through the duckdb-rs source, it seemed like one would need to shoehorn first class support for nested data types into the library or redesign a chunk of the library’s core.

I, uh, am not comfortable enough with Rust to attempt that in any reasonable amount of time.

Thankfully, the author of duckdb-rs also makes libduckdb_sys available: Rust bindings to DuckDB’s C API. And there’s a bunch of example code for how to use it in the duckdb-rs source.

This involves running a bunch of unsafe functions and dereferencing raw pointers, so you need to wrap a lot of this code in an unsafe block. Reading results from DuckDB’s Arrow interface involves two steps.

First, execute the query using the Arrow interface:

let mut result: duckdb_arrow = ptr::null_mut();
let state = duckdb_query_arrow(conn, sql.as_ptr(), &mut result);

// An example of error handling with this API. I'll skip this everywhere else.
if state == duckdb_state_DuckDBError {
    let error_message: *const c_char = duckdb_query_arrow_error(result);
    let error_message = CStr::from_ptr(error_message).to_str().unwrap();
    panic!("{}", error_message);
}

… then fetch batches of results:

let mut ffi_arrow_array: arrow2::ffi::ArrowArray = arrow2::ffi::ArrowArray::empty();
let state = duckdb_query_arrow_array(
    result,
    &mut &mut ffi_arrow_array as *mut _ as *mut *mut c_void, // I don't get this. I got it from duckdb-rs.
);

I do not understand why ffi_arrow_array needs to be cast this way. I do understand:

• The C API requires a duckdb_arrow_array * for that second argument.
• duckdb_arrow_arrow is an alias for void * (this is in the DuckDB C API headers).
• The argument it expects, then, is a void **.

But the cast in Rust is baffling to me. What is as *mut _ doing in the middle of this? What does this do and why is it necessary? If you know, please tell me.

On another note, piecing all this together is painful. I had to read DuckDB’s C API docs, DuckDB’s C header, and the arrow2 crate docs to figure out why this particular incantation works. I am not sure there is a good solution to this. These are disparate, relatively immature projects. Maybe it just takes time? (And blog posts like these?)

Turning an arrow2::ffi::ArrowArray into a Polars DataFrame

With the code above we’ve got a batch of results from DuckDB populated in these Arrow C structs. But we’d like to go from these to a Polars DataFrame. How?

I used the arrow2 crate instead of arrow because it looks like Polars uses arrow2 and figured I might have an easier on-ramp to creating a Polars DataFrame.

We need to do the following:

• Create a Rust-native Arrow array so we stop working with these C structs.
• Create a Polars Series for each column in our result set from the Arrow array.
• Create a Polars DataFrame from those Series.

To start with, we also need to get the Arrow array’s schema. DuckDB provides an API for this too:

let mut schema = arrow2::ffi::ArrowSchema::empty();
let state =
    duckdb_query_arrow_schema(result, &mut &mut schema as *mut _ as *mut *mut c_void);

(There’s that baffling cast again.)

Now we’re going to exit C API-land and convert these raw C structs into a Rust-native arrow2::array::Array.

let field = arrow2::ffi::import_field_from_c(schema).unwrap();
let arrow_array =
    arrow2::ffi::import_array_from_c(ffi_arrow_array, field.data_type).expect("ok");

arrow_array is an arrow2::array::Array, which is a trait object that can be downcast to the specific Arrow array type you want. I don’t know what a Rust trait object is yet, but I do know I want to iterate over an array of integers or whatever my query actually returns.

Let’s assume this is our query:

CREATE TABLE users (
    id INTEGER,
    username VARCHAR
);

-- Insert a bunch of data --

SELECT id, username FROM users;

It turns out that the arrow_array DuckDB’s Arrow interface returns is a StructArray, which is just a struct that contains multiple Arrow arrays of the same length. For our query above, it’ll be a StructArray containing two arrays: one for id and one for username.

So we first downcast to a StructArray:

let struct_array = arrow_array
    .as_any()
    .downcast_ref::<StructArray>()
    .expect("This Arrow Array should be a StructArray.");

Then we downcast each array in struct_array to their appropriate types. This was another area that took a while to grok. What are the types DuckDB returns in the struct_array? Are those user IDs signed or unsigned? Are they 32-bit or 64-bit?

Print out struct_array.fields() to view the data types you’re getting back. If you wanted to construct a DataFrame dynamically from any query, you’d have to inspect these fields then downcast each array to the corresponding type.

Here’s what the code looks like to fetch and downcast to the specific Arrow array types we need for the query above:

let id_array = struct_array.values()[0]
    .as_any()
    .downcast_ref::<Int32Array>()
    .unwrap();

let username_array = struct_array.values()[1]
    .as_any()
    .downcast_ref::<Utf8Array<i32>>()
    .unwrap();

Int32Array is an alias provided by arrow2 for a PrimitiveArray containing i32-type data. All primitive types are stored in Arrow arrays of types PrimitiveArray.

Utf8Array is not an alias. And I think the generic argument to Utf8Array is the type used for Arrow offsets. Specifically, these are the offsets used to identify where a UTF-8 string at a given index might start. If you have the array [“hello”, “world”], the offset for “hello” is 0 and the offset of “world” is 5.

(If that is incorrect, holler.)

A good place to start to read more about these type is the docs for arrow2::array.

Finally, we want to create Series from these arrays:

let id_series = Series::try_from(("id", id_array.to_boxed())).unwrap();
let username_series =
    Series::try_from(("username", username_array.to_boxed())).unwrap();

to_boxed() turns the downcasted arrays into a boxed arrow2::array::Array, the trait object all Arrow arrays are represented as in arrow2. This what Polars Series expects.

Finally, our dataframe:

let df = DataFrame::new(vec![id_series, username_series]).unwrap();

… and that’s how you start with a DuckDB query and end up with a Polars dataframe in Rust.

Further reading

• I’ve put together a full example of the above in on Github. It uses a Parquet file from the NYC TLC dataset.
• DuckDB C API
• duckdb.h
• Arrow2
• This post on DuckDB’s Arrow integration is interesting.

I got carried away with that title, sorry.

I wrapped up David Beazley’s week-long compilers course yesterday and it was a total hoot. If you’ve been considering taking this course, I want to share my (lovely) experience and what you can expect from it.

David’s compilers class is an intense, hands-on primer to implementing a programming language. Over five days, we built an interpreter and compiler for a toy language called Wabbit from scratch. David designed the language and the spec is small enough to get something cool working while having gobs to wrap up and explore if you choose to.

David’s done an outstanding job designing the project for this class. It was challenging but I learned a ton and always had something to show for my effort. Every day we’d have working code for some practical aspect of Wabbit that also set us up the next day. It was a great way to build momentum.

For example, on day one I built a syntax tree model and a code formatter that could take a tree I constructed and generate valid, nicely-formatted Wabbit code. This might look something like this:

# The following returns "print 4 + 5;"
#
format(
    Statements(
        PrintStatement(
            Add(IntegerLiteral("4"), IntegerLiteral("5"))
        )
    )
)

The example above is trivial but the model I built on the first day covered the entire language spec. It gave me a solid foundation to build a parser the next day. I also had a practical tool for Wabbit and got to appreciate how code formatters I use every day are built. Python Black’s line generation code is not far from what we implemented on day one.

I think that is an awesome result to have so quickly, and it’s what you can expect daily.

I won’t go into more detail here because David experiments with his course and spends a ton of time on its design and his materials.

If you’re going to take the course, note that it is a ~40 hour commitment. There are about 3 hours of discussion daily with the rest of the time dedicated to hands-on implementation. There’s a Gitter chat where David and all the students can ask questions and share code and confusion. Everyone’s working on their implementation in the open, David included, so you can pop into different branches and see approaches folks are taking.

The course is light on theory. I took a compilers course in college about 15 years ago and I think we spent ~2-3 weeks investigating different parsing techniques and talking about automata and grammars. David does not do that.

Yet I have a solid foundation for further study after this week. I know how topics like parsing, type checking, semantic analysis, garbage collection, and code optimization fit in the language implementation landscape. I understand the contours of those topics, what to search for, and how I might go about navigating code to explore them in open source projects. I think I know where I would need to begin with any of those in my Wabbit implementation.

One neat thing I want to note is that we talked about error message generation more than I expected. David’s keen on figuring out how to do this well. He mentioned that compiler books all punt on the topic of error messages, but he doesn’t because they’re core to making a programming language usable.

Importantly, it’s representative of where his course falls on the spectrum between theory and practice.

David’s a great teacher. If you haven’t seen any of his talks on YouTube yet, seek them out. His compilers course is a fantastic way to survey the field as a practitioner. You’ll write a lot of code and build an interpreter and a compiler in five days. The work will challenge you but you’ll learn a lot and be rewarded with some exciting results.

If that sounds like your cup of tea, take the course. Sign up for compilers, or another one of David’s courses, here: https://dabeaz.com/.

Quick tips for folks about to take the course

If you’re about to take the course, here are some quick tips for you:

• Definitely use a program language you are comfortable in. I do not recommend using this as an opportunity learn a new language. Make no mistake: you will write a lot of code and you don’t want other details to distract you from the material at hand.
• In a similar vein, this isn’t a good time to mess with your editor/environment. You might be tempted to try out a new editor or something. I don’t recommend doing that either. It’s distracting.
• Poke around everyone’s branches. It is super interesting to see how other folks approach the same problem. You might learn something new.
• Get good sleep, eat well, etc. I guess this should apply to… every day life. But the pace of David’s course is intense and my brain was fried every day. I couldn’t have kept pace if I wasn’t feeling good.
• Have fun! His class will challenge and confuse you. You will make a lot of mistakes and introduce a lot of bugs. Your code will probably be a mess. Don’t worry about it, roll with it, and enjoy the process! It’s fun to hack on something new.

In 2015 I discovered Jed Margolin’s emails from his tenure as a hardware engineer at Atari: 4,128 messages he sent and received between 1983 and 1992.

… and then I read them all during my subway commute to work for the next several months.

Not all four thousand messages in Jed’s inbox are captivating. I’ll admit I glossed over a lot of the 1989 project status reports for Stun Runner. But they were collectively fascinating. I kept finding gems.

I loved all the day-to-day mundanity too: coworkers throwing wee jokes into their status reports, minutes from engineering meetings, Jed’s constant displeasure with Bob Frye from facilities. (Why are there always ants in the cafeteria, Bob?!)

Technology’s changed since the 80’s and that came across loudly when I read Jed’s emails. But all the human interaction and how engineers built technology and did creative work felt the same. Reading Jed’s emails was relatable and comforting.

In 2017, two years after I discovered Jed’s emails, I quit my job and took some time off. That spring I contacted Jed for permission to work on what would later become atariemailarchive.org.

Hi, Vikram.

Yes, I am ok with it (as long as you do not use it for any nefarious
purposes). :-)

I don’t think there is any legal stuff to do.

And thank you. I don’t want this stuff to disappear when I am gone. (I am
old and decrepit.)

Regards,

Jed

Confident that I met Jed’s stringent criteria not to weaponize his publicly-available professional emails from the 80’s, I began work.

I contacted Jed on May 15th and released atariemailarchive.org on December 15th. I worked on the project on and off over seven months.

atariemailarchive.org allows folks to enjoy Jed’s emails without obsessively reading everything like I did. It gives readers threaded conversations, curation with a “best of” list, and an easier-to-read interface than a giant text file.

But that’s not how it started. I built a version and wrote a bunch of code that I scrapped before I built what you see on atariemailarchive.org today.

The version that never saw the light of day

My goal with atariemailarchive.org is simple: I want it to be able to hook you during your lunch hour.

As a proxy to test if you might get hooked, I just want any indication that you are consumed by atariemailarchive.org when I tell you about it. If you tap around, chuckle a lot, and start ignoring me, that is sufficient.

By this rigorous and scientific measure, the first version I built failed spectacularly. Readers got bored quickly.

I had initially built a search interface and an easier way to read and navigate all of Jed’s individual messages. When I showed it to folks, they tapped around, searched for jobs and wozniak (neither appears in the archive), and then asked me what they should look for.

If you aspire to curate a someone’s public emails some day, here’s a tip: search is an awful way to curate text when your readers don’t know what is interesting about it. This is obvious in retrospect, but it didn’t occur to me until I showed what I built to folks.

For example, a lot of folks have created neat search interfaces for the Enron emails. I know there are fascinating emails in that archive, yet I’ve barely read read any of them. I don’t know where to look.

I realize it is significantly harder given that the Enron archive contains 500,000 messages, but I hope someone creates something like atariemailarchive.org for the Enron emails some day, somehow.

I know I’d spend my lunch hour reading it.

The version that did see the light of day

I realized that if I wanted folks to see what I saw in the archive, I’d need to guide them directly to it. atariemailarchive.org would need resemble the emails I sent to friends and coworkers when I wanted to share fun messages I found in Jed’s inbox.

That’s what led me to do three things:

1. I created conversation threads from every message in the archive.
2. I assembled a “best of” list with some commentary.
3. I tagged threads.

I actually tried #2 and #3 on Jed’s individual messages first to test it out. It worked. Folks tapped around and chuckled a lot. They told me they wanted more.

But when I tried to tag more messages and add them to the “best of” list, I quickly realized how many messages existed in a larger thread, and how important that context was for all the little discoveries I made to truly hit. You can see this easily if you click around on atariemailarchive.org today.

Threads did not exist in the 80’s. Jed’s inbox was just a bunch of email ordered chronologically. So I bit the bullet and created conversation threads from all 4,128 of Jed’s messages.

As you can imagine, this was tedious. It was the most time consuming part of this project. I was living in Seattle at the time and I’d like to thank the folks at Porchlight Coffee for letting me hang out there over many, many afternoons while I manually threaded all of Jed’s emails.

What’s next for the archive?

I decided to publish the dataset I created for atariemailarchive.org under a Creative Commons license today. You can find it on Github.

I would be thrilled to see people do interesting things with it. If you do, please let me know.

People use the archive in different ways. Occasionally readers will link to messages that help them in some investigation or research they’re doing. Here’s one of my favorites. Jed received a price list for custom parts made for Atari hardware, and the person who authored this tweet found them in a breakdown of an arcade cabinet. You can see the part ids match in the image and the email.

It has been a joy to see folks link to and enjoy the archive and I hope to see more of that. I get messages of appreciation every now and then. I love receiving those. It validates all the time I spent on this bizarre obsession.

Technical notes for those interested

atariemailarchive.org began its life as a Python Pyramid app with a SQLite back-end in 2017. It ran on a $5/month Linode instance until early September 2022.

This week I ported it Python Flask and deployed it to Fly.io. The app is trivial and porting it to Python Flask took two afternoons. Deploying atariemailarchive.org to Fly.io took very little time – Fly.io was a joy to use and it will cost me nothing to host the archive there.

The dataset on Github is in a SQLite file. It contains structured, threaded messages that I parsed and assembled from text files on Jed’s site. It does not contain any of the curation and editorialization on atariemailarchive.org.

Thanks to Katie Brookoff.

I came across Roaring bitmaps when I learned about this fun hack to do retention analyses at scale with bitmaps. Using Roaring bitmaps instead of traditional bitmaps in that application reduced memory usage from ~125GB to 300MB, an impressive 99.8% savings.

But… how?

You can learn about Roaring bitmaps over two research papers:

1. This one proposes the data structure.
2. This one introduces a critical optimization.

In this post I briefly describe what bitmaps are, what they’re used for, and what Roaring bitmaps solve that traditional bitmaps don’t. Then, I distill the high-level structure of Roaring bitmaps and how they work, one step at a time.

Roaring bitmaps employ a number of algorithms, techniques, and heuristics that I won’t go into in detail. They also offer some operations beyond the ones I describe. These details are not critical to understanding the basic internal structure and operation of Roaring bitmaps which is the focus of this post.

Let’s begin!

What are bitmaps and what are they used for?

Bitmaps are arrays of bits used to store sets of integers.

They work by setting the Nth bit when an integer N is in the set, as illustrated in Figure 1.

By storing sets of integers this way, bitmaps can take advantage of extremely fast bitwise-AND and bitwise-OR CPU instructions to compute set intersections and unions.

It turns out that fast set intersections and unions are critical for many search and database applications. Various operations exist in search and database indexes that boil down to having two sets of integers and needing to intersect or union them quickly.

Take an inverted search index, for example:

• You’ve indexed billions of documents. Each document has an integer id.
• The index maps terms to a set of documents in which they appear. For example, the term pigeon appears in documents with these ids: {2, 345, 2034, ...}.
• Queries that search across terms use set operations. In order to resolve a search query like carrier AND pigeon you need the intersection of the set of documents that contain carrier and the set of documents that contain pigeon.
• Bitwise operations can perform these set operations quickly. If you represent sets of document ids as bitmaps, the query above is a bitwise-AND.

Columnar databases use set operations similarly for certain classes of queries.

If you’d like to dive into a use case outside search and databases, a previous post I wrote discusses how bitmaps can be used to analyze user retention for SaaS products.

Unfortunately, bitmaps suffer from awful compression in common cases involving very large sets of integers – scenarios that appear frequently in the use cases I just described.

Recall the figure I cited at the beginning of the post:

Using Roaring bitmaps instead of traditional bitmaps in that application reduced memory usage from ~125GB to 300MB, an impressive 99.8% savings.

That’s the problem with bitmaps, and I’ll walk you through why this happens shortly.

What are Roaring bitmaps?

From roaringbitmap.org:

Roaring bitmaps are compressed bitmaps. They can be hundreds of times faster.

A great, pithy marketing statement on the Roaring bitmap website! Let’s expand on it a wee bit.

Roaring bitmaps are just optimized bitmaps, which I’ll refer to henceforth as “traditional bitmaps”.

Both traditional and Roaring bitmaps offer a set data structure for integers. You can insert integers, check the existence of an integer, and get the intersection and union of two sets of integers.

Roaring bitmaps offer better compression than traditional bitmaps. Importantly, they do so without significantly sacrificing the performance of set operations.

roaringbitmap.org boasts an impressive list of OLAP databases and search systems that use Roaring bitmaps under the hood. These are all applications that:

• … need to store large sets of integers
• … in as little memory as possible
• … and execute fast set operations.

What problem do Roaring bitmaps solve that traditional bitmaps don’t?

When a set is sparse, traditional bitmaps compress poorly.

Recall that traditional bitmaps will set the Nth bit when you add an integer N to it (see Figure 1).

Let’s say you have an empty traditional bitmap to which you add the integer 8,000,000. Here’s what will happen:

• It will allocate 1,000,000 bytes.
• It will set the 8,000,000th bit.

This is illustrated in Figure 2.

Why is this bad?

• Your set has 1 integer.
• An integer takes up 4 bytes.
• Your traditional bitmap has allocated 1 megabyte.

That’s 6 orders of magnitude more memory than you need.

Whoops!

Roaring bitmaps solve this problem. Importantly, they do so while maintaining fast set operations. This is what makes Roaring bitmaps special.

Prior research attempts to solve poor compression in bitmaps achieve impressive results too, but at the cost of efficient set operations.

How do Roaring bitmaps work?

There isn’t one deep insight that allows Roaring bitmaps to perform well. But there is a lot of stuff going on that is greater than the sum of its parts.

The following builds up an understanding of Roaring bitmaps one concept at a time.

Part 1: how Roaring bitmaps are represented in memory

All 32-bit integers are partitioned into contiguous chunks.

Each chunk shares the same 16 most significant bits.

As shown in Figure 3, the partitioning scheme used by Roaring bitmaps ensures that an integer will always belong to the same chunk of 2^16, or 65,536 consecutive integers.

Note: there are 64-bit implementations of Roaring bitmaps, but this post does not go into them. See CRoaring, this native Go implementation, and, uh, this paper written in French (if you find a translation, please let me know).

Integers in the same chunk are stored in containers.

Chunks are how integers are logically partitioned in Roaring bitmaps. All integers that belong to a chunk are physically (in memory) stored in the same container.

Figure 4 shows examples of three different containers for three different chunks.

A chunk will always only have one container in a Roaring bitmap, defined by the 16 most significant bits of all the integers in the chunk.

If your program inserted the first 1,000 multiples of 62 into a Roaring bitmap, then they would be end up in the left-most container in Figure 4. That container’s cardinality would be 1,000.

If you later inserted the integer 63, it would end up in the same container. The container’s cardinality would then be 1,001.

As you’ll see next, the cardinality of a container determines how it will be represented in memory.

Sparse containers contain <= 4,096 integers. These are stored as sorted packed arrays.

Two of the containers in in Figure 4 are sparse (with cardinalities 1,000 and 100) so they will be stored as sorted packed arrays of 16-bit integers.

By packed, we mean that we will be packing 32-bit integers into 16-bit integers as shown in Figure 5.

Packing integers is possible because each container can store at most 2^16 distinct integers. To get the original 32-bit integer in a sparse container, we have to unpack it by combining the 16-bit integer with its 16 most significant bits.

These arrays are dynamically allocated so the memory used by a sparse container grows as it accrues integers.

Dense containers contain > 4,096 integers. These are stored as bitmaps.

One of the containers in Figure 4 is dense (with cardinality 215) so it will be stored as a traditional bitmap.

Dense containers are bitmaps containing 2^16 bits (8 kilobytes), allocated outright. The Nth bit in the bitmap maps to the Nth integer in a chunk.

A first-level index points to all containers. The index is stored as a sorted array.

The first-level index stores the 16 most significant bits for each container in the Roaring bitmap along with a pointer to the corresponding container.

The index is stored as a sorted array and grows dynamically as new containers are added to the Roaring bitmap.

Part 2: how set operations work with Roaring bitmaps

Integer insertion varies by container type and may cause a container’s type to change.

To insert an integer, N, get N’s 16 most significant bits (N / 2^16) and use it to find N’s corresponding container in the Roaring bitmap.

Insertions in array and bitmap containers work differently:

• Bitmap: set the bit at N % 2^16.
• Array: insert N % 2^16 in its position in the sorted array.

Insertions may change the container type. If an array container has 4,096 integers, first convert it to a bitmap container. Then set the bit at N % 2^16.

If a container doesn’t already exist then create a new array container, add it to the Roaring bitmap’s first-level index, and add N to the array.

Checking for existence varies by container type.

To check if an integer N exists, get N’s 16 most significant bits (N / 2^16) and use it to find N’s corresponding container in the Roaring bitmap.

If the container doesn’t exist, then N is not in the Roaring bitmap.

Checking for existence in array and bitmap containers works differently:

• Bitmap: check if the bit at N % 2^16 is set.
• Array: use binary search to find N % 2^16 in the sorted array.

Intersect matching containers to intersect two Roaring bitmaps. Algorithms vary by container type(s), and container types may change.

To intersect Roaring bitmaps A and B, it is sufficient to intersect matching containers in A and B.

This is possible because of how integers are partitioned in Roaring bitmaps: matching containers in A and B store integers with the same 16 most significant bits (the same chunks).

Intersection algorithms vary by the types of the containers involved, as do the resulting container types:

• Bitmap / Bitmap: Compute the bitwise AND of the two bitmaps. If the cardinality is <= 4,096, store the result in an array container, otherwise store it in a bitmap container.
• Bitmap / Array: Iterate over the array, checking for the existence of each 16-bit integer in the bitmap. If the integer exists, add it to the resulting array container – note that intersections of bitmap and array container types will always create an array container.
• Array / Array: Intersections of two array containers always create a new array container. The algorithm used to compute the intersection varies by a cardinality heuristic described at the bottom of page 5 here. It will either be a simple merge (as used in merge sort) or a galloping intersection, described in this paper.

If there is a container in either Roaring bitmap without a corresponding container in the other, it will not exist in the result: the intersection of an empty set and any set is an empty set.

Union matching containers to produce a Roaring bitmap union. Algorithms vary by container type(s), and container types may change.

To union Roaring bitmaps A and B, union all matching containers in A and B.

Union algorithms vary by the container types involved, as do the resulting container types:

• Bitmap / Bitmap: Compute the bitwise OR of the two bitmaps. Unions of two bitmap containers will always create another bitmap container.
• Bitmap / Array: Copy the bitmap and set corresponding bits for all the integers in the array container. Unions of a bitmap and array container will always create another bitmap container.
• Array / Array: If the sum of the cardinalities of the two array containers is <= 4,096, the resulting container will be an array container. In this case, add all integers from both arrays to a new array container. Otherwise, optimistically assume the resulting container will be a bitmap: create a new bitmap container and set all corresponding bits for all integers in both arrays. If the cardinality of the resulting container is <= 4,096, convert the bitmap container back into an array container.

Finally, add all containers in A and B that do not have a matching container to the result. Remember: this is a union, so all integers in Roaring bitmaps A and B must be in the resulting set.

Part 3: how a third and final container type – the “run” container – optimizes long runs of consecutive integers.

Parts 1 and 2 of this post cover most of the internal structure and operation of Roaring bitmaps. This final part covers an important optimization described in the second Roaring bitmap paper.

Run containers represent runs of consecutive integers with two 16-bit integers: the run start and run length.

From page 3 of the second Roaring bitmap paper:

The new container is conceptually simple: given a run (e.g., [10, 1000]), we store the starting point (10) and its length minus one (990) … packing the starting points and the lengths in pairs, using 16 bits each …

This technique is known as run-length encoding and seems to back most of the prior art described in the two papers. Run-length encoding can compress bitmaps effectively but degrades performance for set operations in many cases.

Run containers are formed explicitly when a client invokes a runOptimize function or, in some cases, implicitly when a large range is added to the Roaring bitmap.

Unlike sparse and dense containers, run containers generally do not materialize automatically.

1. Clients can invoke runOptimize to optimize their Roaring bitmap for large runs of consecutive integers. Run containers may replace existing array or bitmap containers in this case.
2. Roaring bitmaps offer an operation to add a range of values. Run containers may materialize automatically in this case.

The papers don’t actually prescribe how or when #2 should happen. I’d guess that if a range of values were added for a chunk that does not yet have a container, it makes sense to create a run container instead of an array or bitmap container.

runOptimize only creates a run container if it will be smaller than the container it would replace.

runOptimize first counts the number of runs in a container.

Then, it decides whether or not to create a run container using a simple heuristic: the run container must be smaller than its equivalent array or bitmap container.

The algorithms used to count runs and a description of how to compute the heuristic above are described on page 6 and 7 in the second Roaring bitmap paper.

If you’d like to work out computing the heuristic yourself, it’ll help to recall the following:

• … array containers contain no more than 4,096 integers, packed into 16-bits each.
• … bitmap containers contain > 4,096 integers in a bitmap with 2^16 bits (8,192 bytes).
• … each run in a run container takes up 32 bits (16 bits for the start, 16 bits for the length).

The addition of run containers introduces new algorithms for all set operations.

The Roaring bitmap papers do not describe the algorithms used to insert and check for the existence of integers in run containers: these operations are relatively straightforward.

But the addition of run containers requires that Roaring bitmaps implement performant algorithms for unions and intersections of three new container type pairs:

• Run / Run
• Run / Array
• Run / Bitmap

These algorithms also introduce new heuristics to determine the resulting container type from these operations.

I won’t go into the details of these algorithms as I did in Part 2. The algorithms are not significantly more complicated (their descriptions start on page 10 here if you’re curious), but the numerous details are beyond the scope of this post and the paper is well-written and makes them extremely accessible.

Roaring bitmaps: a game of performance whack-a-mole!

Roaring bitmaps use a kitchen sink of algorithms and techniques to achieve better compression and faster performance than other bitmap implementations.

They’re challenging to implement but they do the job extremely well, especially when used in OLAP workloads. The creators managed to root out inefficiencies in common yet very different scenarios – sparse data, dense data, data with a lot of runs – and address all of them at once.

And they go even further!

A third paper describes an implementation the creators wrote in C that leverages vectorized algorithms they designed to use SIMD (single instruction multiple data) instructions. That implementation, CRoaring, along with bindings and implementations in multiple other languages are available here. They’re used in mainstream columnar databases and search applications and are actively maintained, improved, and optimized regularly.

Very cool.

Further reading

If you enjoyed reading this post and want to dig in further, I recommend reading the Roaring bitmap papers. They’re well-written and accessible.

• Better bitmap performance with Roaring bitmaps
• Consistently faster and smaller compressed bitmaps with Roaring
• Roaring Bitmaps: Implementation of an Optimized Software Library

Roaring bitmap implementations are available on Github.

Thanks to Chuck Groom, Andy O’Neill, Phil Eaton, Ben Johnson, and Simon Willems.

Instead of paying Mixpanel over $2K/month for retention reporting functionality in 2012 Amir Salihefendic (founder of Doist, makers of Todoist) built a clever hack to get the functionality he needed. He estimates Doist has saved millions on product analytics tools through 2022 yet has run interactive retention analyses over billions of events.

Amir’s hack was to use bitmaps and bitwise operations to answer common questions about cohorts of users. He built and open-sourced bitmapist, a Python library that tracks user events in Redis bitmaps and can generate retention/cohort reports.

It looks like Doist still uses bitmapist in production ten years later. As of this writing, Amir himself pushed the latest commit three weeks ago.

This post walks you through how bitmapist works and its limitations/pitfalls. I won’t cover how to read and interpret a retention report. If you’d like a primer, here’s a pretty good one.

The key idea: each cell in a retention report represents an intersection of two sets of users.

The retention report above shows:

• On January 27th, 1,257 users signed up for our app.
• On Day 2,246 (19.6%) of the January 27th cohort opened the app.

Day 2 relative to the January 27th cohort is January 29th.

Thousands of users may have opened our app on January 29th, but 246 of them also signed up for our app on January 27th.

Here’s another way to state the above:

A = Set of users who signed up on January 27th
B = Set of users who opened the app on January 29th

size(A) = 1257
size(B) = ?
size(A ∩ B) = 246

size(A ∩ B) / size(A) = 19.6%

Every cell in a retention report is computed the same way.

The hack: store sets of users in bitmaps for fast set operations and high compression

Bitmaps are an ideal data structure for this use case:

• Bitwise operations providefast set intersections and unions.
• Bitmaps compress this data well. A set of 8 million users fits in 1 megabyte.

Bitmapist maintains user bitmaps in Redis. Each bitmap is keyed on a (time bin, event) pair. If user ID 542 opens the app some time on January 25th 2022, then the bit at index 542 will be set on the bitmap with key 20220125:opened app in the example below.

Figure 2 shows daily time bins and two distinct events, but the idea generalizes to any time bin (hourly, daily, weekly, monthly, etc.) and any number of distinct events.

Bitwise ANDs enable high-performance set intersections to compute counts of distinct users who did our start event on one date and our return event on another date. If you’re unfamiliar with how a bitwise AND operation works, Wikipedia has a good example. Redis implements bitwise operations via its BITOP command.

Bitmapist provides a handy Python API for clients to track events (mark_event("opened app", 542)) and transparently maintains hour/day/week/month-binned bitmaps in the background. It also provides a library to generate retention reports interactively:

Using Redis bitmaps for retention analyses is fast but extremely space-inefficient.

The Redis BITOP docs link to this article from 2011 simulating these techniques with 128MM users. Some relevant metrics from that article:

• Set counts and intersections over 128MM users are fast. Interactive retention analyses are easily achievable with this technique.
  • 50ms to count all the bits set in one bitmap.
  • 392ms to intersect 7 bitmaps (a week of daily bins).
  • 1624ms to intersect 30 bitmaps (a month of daily bins).
• A bitmap of 128MM users takes up 16MB of memory. It’s nice to see this play out predictably. You can confirm this easily on your own machine.
  • Remember: Each bit is a distinct user.
  • 1MB stores 8 million bits.
  • Multiply that by 16 and you get 16MB and 128 million bits.

This technique is fast and compresses well, so what’s the catch?

There are two:

1. Sparse bitmaps are space-inefficient in Redis. Redis will allocate as much memory as it needs to set the Nth bit. If your user’s ID is 8,000,000, bitmapist will set that bit in Redis, which will allocate 1MB of RAM to set it. You can see this behavior in action here.
2. This technique stores lots of bitmaps. Let’s say you track 100 distinct events and you bin hourly (24 * 30 bins/month), daily (30 bins/month), and monthly (1 bin/month). Worst case, you’ll have ~75,000 bitmaps by the end of the month. If each bitmap is 1MB, Redis will allocate 75GB RAM.

As of 2020, Todoist had tens of millions of signups and over one million active users. That Todoist might set a bit index in the tens of millions range every day while setting few bits at a lower range (0 - 1,000,000) is highly likely.

Luckily, there’s an easy fix for this that relies on more efficient bitmap representations.

Using roaring bitmaps fixes the space-inefficiencies introduced by big, sparse Redis bitmaps.

In 2017 Doist built a standalone bitmap server to address the memory issues described above. It implements Redis’ wire protocol and the subset of Redis bitmap commands used by bitmapist so it’s easy to swap in.

It is also three orders of magnitude more space-efficient than using Redis bitmaps under a heavily used bitmapist setup:

Memory in use reported by Redis (matches RSS of the redis-server process): 129.48G.

With the same dataset migrated to standalone bitmapist server under the same load: RSS reported at about 300M.

It achieves this result by using roaring bitmaps, compressed bitmaps that still offer high-performance bitwise operations.

If you’re curious about Roaring bitmaps, I read the papers and wrote a primer on what they are and how they work.

Interactive retention analyses over billions of events for under $100/mo?

I haven’t tested this out but it’s well within the realm of possibility:

• In 2015, Doist was storing hundreds of millions events using bitmapist.
• In 2017, Doist’s roaring bitmap-based server compressed at least hundreds of millions of events from ~130GB to ~300MB.
• A factor of 10 increase in the above event volume is billions of events. Assuming memory requirements scale linearly, you’ll need 3GB RAM.
• Many AWS EC2 instance types will fit 3GB comfortably in physical RAM for under $100/mo.

I may test this out with the Github Archive dataset (~3 billion events) one day. If you take a crack at testing these assumptions before I do, please let me know so I can link to your post.

And if you work at Doist, let us know what hardware you run this on!

This technique nets you affordable and efficient distinct counts, set operations, and retention analyses at scale. But the benefits end there.

Spending $100/mo for interactive retention analyses over billions of events is incredibly affordable.

With an event volume in the 10’s to 100’s of millions per month you’d be forking over $1000’s/month to Amplitude, Heap, or Mixpanel today. Amplitude’s free tier ends at a 10MM event/mo. At that point, their annual contracts begin at ~$30K/year.

But these companies also provide a useful suite of tools beyond retention analyses that the techniques underlying bitmapist do not support. If you need funnel or path analyses, for example, you’ll need to pony up the cash – this “swiss army knife” model of product analytics unfortunately commands a premium.

Further reading

• The blog post introducing bitmapist
• Doist’s bitmapist library on Github
• Doist’s roaring bitmap-backed bitmap server
• roaringbitmap.org
• An article from 2011 on real-time metrics using Redis bitmaps
• Redis bitmaps documentation

Thanks to Gaurav Oberoi, Chuck Groom, and Pierre Jambet.