How accurate
are the answers?
Each version changes one thing about the agent. All of them answer the same 100 hand-written questions, and every answer is checked against the result of a gold SQL query.
Accuracy went from 77% with v1 to 82% with v4. The biggest jump (+5 points) came from v4: 3 worked examples in the system prompt.
By category
| Version | What changed | Overall | Single table | Joins | Aggregation | Name ambiguity | Trends over time | Tool calls | Median latency | Median cost |
|---|---|---|---|---|---|---|---|---|---|---|
| v1 | list_tables, describe_table, run_query only | 77% | 80% | 65% | 80% | 80% | 80% | 4.3 | 5.3 s | $0.001 |
| v2 | Column comments and sample rows in describe_table | 79% | 85% | 70% | 80% | 80% | 80% | 4.0 | 5.0 s | $0.001 |
| v3 | Add search_people and search_titles | 77% | 90% | 55% | 85% | 80% | 75% | 3.7 | 4.8 s | $0.001 |
| v4 | 3 worked examples in the system prompt | 82% | 90% | 70% | 85% | 85% | 80% | 2.9 | 4.6 s | $0.001 |
Every question, every version
| Id | Question | v1 | v2 | v3 | v4 |
|---|---|---|---|---|---|
| s01 | What is the highest-rated movie released in 2019 with at least 100,000 votes? | pass | pass | pass | pass |
| s02 | How many movies in the database were released in 1994? | pass | pass | pass | pass |
| s03 | What is the longest movie by runtime that has at least 50,000 votes? | pass | pass | pass | pass |
| s04 | How many movies have an average IMDb rating of 9.0 or higher? | pass | pass | pass | pass |
| s05 | Which movie has the most IMDb votes? | fail | fail | pass | pass |
| s06 | List the 5 most-voted movies released in 2010, from most to fewest votes. | pass | pass | pass | pass |
| s07 | What is the average runtime in minutes of movies released in 2020, ignoring movies with unknown runtime? Round to 2 decimal places. | pass | pass | pass | pass |
| s08 | How many movies are tagged with the Western genre? | pass | pass | pass | pass |
| s09 | What is the lowest-rated movie with at least 50,000 votes? | fail | pass | pass | pass |
| s10 | How many movies released from 2000 through 2009 have at least 500,000 votes? | pass | pass | pass | pass |
| s11 | What are the 10 highest-rated movies with at least 1,000,000 votes? Break rating ties by vote count. | pass | pass | pass | pass |
| s12 | In what year was the earliest movie in the database released? | pass | pass | pass | pass |
| s13 | How many distinct genres appear across all movies? | pass | pass | pass | pass |
| s14 | What is the average IMDb rating of movies with at least 100,000 votes? Round to 2 decimal places. | pass | pass | pass | pass |
| s15 | Which movie released in 1999 has the most votes? | fail | fail | fail | fail |
| s16 | How many movies run longer than 180 minutes? | pass | pass | pass | pass |
| s17 | What is the shortest movie by runtime with at least 100,000 votes? | pass | pass | pass | pass |
| s18 | How many Animation movies are rated 8.0 or higher? | pass | pass | pass | pass |
| s19 | Which release year has the most movies in the database? | fail | fail | fail | fail |
| s20 | Which horror movies with at least 100,000 votes are rated 8.0 or higher? | pass | pass | pass | pass |
| j01 | Which actor or actress appears in the most movies rated 8.0 or higher? | pass | fail | fail | pass |
| j02 | Who directed the most movies with at least 100,000 votes? | pass | fail | fail | fail |
| j03 | Among directors with at least 7 movies that have 50,000 or more votes, who has the highest average rating across those movies? | fail | pass | fail | pass |
| j04 | Which director born in 1975 or later has directed the most movies with at least 50,000 votes? | pass | pass | pass | pass |
| j05 | Which actors and actresses appear in the cast of both The Godfather (1972) and The Godfather Part II (1974)? | pass | pass | pass | pass |
| j06 | Which 5 actresses have appeared in the most movies with at least 100,000 votes? | pass | pass | pass | pass |
| j07 | How many different people have directed at least one movie rated 8.5 or higher with at least 100,000 votes? | pass | pass | pass | pass |
| j08 | Who has the most writing credits on movies rated 8.0 or higher with at least 25,000 votes? | fail | fail | pass | fail |
| j09 | Which movies did Quentin Tarantino both write and direct? | pass | pass | pass | pass |
| j10 | Which actor or actress has appeared in the most movies directed by Steven Spielberg? | fail | fail | fail | fail |
| j11 | What is the average rating of the movies Leonardo DiCaprio acted in? Round to 2 decimal places. | fail | pass | pass | pass |
| j12 | Which director has directed Robert De Niro in the most movies? | pass | pass | fail | fail |
| j13 | Who directed the highest-rated movie of 1972 that has at least 100,000 votes? | pass | pass | fail | pass |
| j14 | How many movies feature both Tom Hanks and Meg Ryan in the cast? | fail | pass | fail | pass |
| j15 | Who directed each of the 5 most-voted movies? Return each movie's title and its director. | pass | pass | pass | pass |
| j16 | Among actresses with at least 10 movies that have 50,000 or more votes, who has the highest average rating across those movies? | fail | fail | fail | pass |
| j17 | Which Christopher Nolan film has the highest IMDb rating? | pass | pass | pass | fail |
| j18 | Which directors have made at least 6 films rated 8.0 or higher with at least 25,000 votes? | pass | pass | pass | pass |
| j19 | What is the most common genre among movies directed by Alfred Hitchcock? | fail | fail | fail | fail |
| j20 | Which 3 people have directed the most movies with at least 10,000 votes? | pass | pass | pass | pass |
| a01 | What is the average rating of horror movies by decade, starting with the 1960s? Round to 2 decimal places. | pass | pass | pass | pass |
| a02 | How many movies are there in each genre? | pass | pass | pass | pass |
| a03 | Which genre has the highest average rating among genres with at least 500 movies? | fail | fail | fail | fail |
| a04 | What is the average number of votes for movies released in 2010? Round to 2 decimal places. | pass | pass | pass | pass |
| a05 | What is the average rating of each genre for movies with at least 100,000 votes? | pass | pass | pass | pass |
| a06 | How many movies fall in each whole-number rating band, where a band is the rating rounded down (so 7.8 is in band 7)? | pass | pass | pass | pass |
| a07 | What is the median movie runtime in minutes, ignoring unknown runtimes? | pass | pass | pass | pass |
| a08 | How many people have directed 20 or more movies in the database? | pass | pass | pass | pass |
| a09 | Which decade has the highest average movie rating, among decades with at least 1,000 movies? | fail | fail | fail | fail |
| a10 | What percentage of movies with at least 100,000 votes are tagged Drama? Round to 2 decimal places. | pass | pass | pass | pass |
| a11 | What is the average number of genres per movie? Round to 2 decimal places. | pass | pass | pass | pass |
| a12 | How many movies have exactly one genre? | fail | pass | pass | pass |
| a13 | What is the total number of IMDb votes across all movies released in 2008? | pass | pass | pass | pass |
| a14 | What are the 3 most common genres among movies rated 8.0 or higher with at least 50,000 votes, and how many such movies does each have? | pass | pass | pass | pass |
| a15 | How many people in the database list both director and actor among their primary professions? | pass | fail | pass | pass |
| a16 | What is the average birth year of the directors of movies with at least 100,000 votes, counting each director once and ignoring unknown birth years? Round to 2 decimal places. | pass | pass | pass | pass |
| a17 | How many movies have more than 1,000,000 votes? | pass | pass | pass | pass |
| a18 | Which genre has the longest average runtime among genres with at least 100 movies? | fail | fail | fail | fail |
| a19 | What is the standard deviation of ratings for movies with at least 10,000 votes? Round to 2 decimal places. | pass | pass | pass | pass |
| a20 | How many movies were released in each decade from the 1920s through the 2020s? | pass | pass | pass | pass |
| n01 | How many movies has Nolan directed? | pass | pass | pass | pass |
| n02 | How many movies has Scorsese directed? | pass | pass | pass | pass |
| n03 | What is the average rating of the movies Kubrick directed? Round to 2 decimal places. | pass | pass | pass | pass |
| n04 | Which Tarantino-directed movie has the most votes? | pass | fail | fail | fail |
| n05 | How many movies did Hitchcock direct? | pass | pass | pass | pass |
| n06 | How many movies has Streep acted in? | pass | pass | pass | pass |
| n07 | Which movie in The Godfather trilogy has the highest rating? | fail | fail | fail | fail |
| n08 | What year was the movie Alien released? | pass | pass | pass | pass |
| n09 | Who directed Heat? | pass | pass | pass | pass |
| n10 | How many movies has Hanks acted in that Spielberg directed? | fail | fail | fail | pass |
| n11 | What is the highest-rated movie DiCaprio has acted in? | pass | pass | fail | pass |
| n12 | How many movies has Miyazaki directed? | fail | pass | pass | pass |
| n13 | What is the average rating of Fincher's movies as director? Round to 2 decimal places. | pass | pass | pass | pass |
| n14 | How many movies has Nicholas Cage acted in? | pass | pass | pass | pass |
| n15 | What is the highest-rated movie Villeneuve directed? | pass | pass | pass | fail |
| n16 | Who are the top 3 billed cast members of The Matrix? | pass | pass | pass | pass |
| n17 | What year did Parasite come out? | fail | fail | pass | pass |
| n18 | How many movies has Wes Anderson directed? | pass | pass | pass | pass |
| n19 | What is the highest-rated movie with Star Wars in its title? | pass | pass | pass | pass |
| n20 | How many movies did Eastwood direct? | pass | pass | pass | pass |
| t01 | How has the average movie runtime changed by decade since 1950? | pass | pass | pass | pass |
| t02 | What was the average runtime of movies released in 1980, and of movies released in 2020? Round to 2 decimal places. | pass | pass | pass | pass |
| t03 | How many movies were released each year from 2000 to 2020? | pass | pass | pass | pass |
| t04 | How has the average rating of movies with at least 10,000 votes changed by decade since the 1950s? Round to 2 decimal places. | pass | pass | pass | pass |
| t05 | Between 1990 and 2020, in which year was the average rating of horror movies the lowest, counting only years with at least 50 horror movies? | fail | fail | fail | pass |
| t06 | What percentage of movies in each decade since the 1960s are tagged Comedy? Round to 2 decimal places. | pass | pass | pass | pass |
| t07 | How many movies with at least 100,000 votes were released in each decade since the 1950s? | pass | pass | pass | pass |
| t08 | Which year from 1980 through 2019 has the most movies rated 8.0 or higher? | pass | pass | pass | fail |
| t09 | How has the average number of genres per movie changed by decade since the 1950s? Round to 2 decimal places. | pass | pass | fail | pass |
| t10 | How many Animation movies were released in each decade from the 1980s through the 2010s? | pass | pass | pass | pass |
| t11 | List every movie Christopher Nolan directed with its release year and rating, in release order. | pass | pass | pass | pass |
| t12 | How has the average age of directors at release changed by decade since the 1960s? Use release year minus birth year, only for directors with a known birth year. Round to 2 decimal places. | pass | pass | pass | pass |
| t13 | Which decade produced the most movies rated 8.0 or higher with at least 25,000 votes? | fail | fail | fail | fail |
| t14 | What is the total number of IMDb votes for movies released in each decade since the 1970s? | pass | pass | pass | pass |
| t15 | For each year from 2010 to 2019, which movie got the most votes? | pass | pass | pass | pass |
| t16 | What is the average rating of the movies Tom Hanks acted in, by decade of release? Round to 2 decimal places. | pass | pass | pass | pass |
| t17 | How many movies longer than 150 minutes were released in each decade since the 1960s? | pass | pass | pass | pass |
| t18 | Which year in the 1990s had the highest average rating for movies with at least 50,000 votes? | fail | fail | fail | fail |
| t19 | How many different people directed at least one movie released in each decade since the 1950s? | fail | fail | fail | fail |
| t20 | What was the average rating of Drama movies for each year from 2010 to 2020? Round to 2 decimal places. | pass | pass | pass | pass |
How scoring works
The gold SQL for each question runs against the same database the agent uses. The agent passes when the rows from its final query hold the same values, ignoring column names. Extra columns are fine, numbers match within 0.01, and row order counts only when the question asks for an order.
Every question names its vote threshold and, where needed, its tiebreak, so each one has exactly one correct answer. A version that scores worse than the one before it stays in the table.