Accuracy / Agent evaluations

How accurate
are the answers?

Each version changes one thing about the agent. All of them answer the same 100 hand-written questions, and every answer is checked against the result of a gold SQL query.

Accuracy went from 77% with v1 to 82% with v4. The biggest jump (+5 points) came from v4: 3 worked examples in the system prompt.

By category

VersionWhat changedOverallSingle tableJoinsAggregationName ambiguityTrends over timeTool callsMedian latencyMedian cost
v1list_tables, describe_table, run_query only
77%
80%
65%
80%
80%
80%
4.35.3 s$0.001
v2Column comments and sample rows in describe_table
79%
85%
70%
80%
80%
80%
4.05.0 s$0.001
v3Add search_people and search_titles
77%
90%
55%
85%
80%
75%
3.74.8 s$0.001
v43 worked examples in the system prompt
82%
90%
70%
85%
85%
80%
2.94.6 s$0.001
Every question, every version
IdQuestionv1v2v3v4
s01What is the highest-rated movie released in 2019 with at least 100,000 votes?passpasspasspass
s02How many movies in the database were released in 1994?passpasspasspass
s03What is the longest movie by runtime that has at least 50,000 votes?passpasspasspass
s04How many movies have an average IMDb rating of 9.0 or higher?passpasspasspass
s05Which movie has the most IMDb votes?failfailpasspass
s06List the 5 most-voted movies released in 2010, from most to fewest votes.passpasspasspass
s07What is the average runtime in minutes of movies released in 2020, ignoring movies with unknown runtime? Round to 2 decimal places.passpasspasspass
s08How many movies are tagged with the Western genre?passpasspasspass
s09What is the lowest-rated movie with at least 50,000 votes?failpasspasspass
s10How many movies released from 2000 through 2009 have at least 500,000 votes?passpasspasspass
s11What are the 10 highest-rated movies with at least 1,000,000 votes? Break rating ties by vote count.passpasspasspass
s12In what year was the earliest movie in the database released?passpasspasspass
s13How many distinct genres appear across all movies?passpasspasspass
s14What is the average IMDb rating of movies with at least 100,000 votes? Round to 2 decimal places.passpasspasspass
s15Which movie released in 1999 has the most votes?failfailfailfail
s16How many movies run longer than 180 minutes?passpasspasspass
s17What is the shortest movie by runtime with at least 100,000 votes?passpasspasspass
s18How many Animation movies are rated 8.0 or higher?passpasspasspass
s19Which release year has the most movies in the database?failfailfailfail
s20Which horror movies with at least 100,000 votes are rated 8.0 or higher?passpasspasspass
j01Which actor or actress appears in the most movies rated 8.0 or higher?passfailfailpass
j02Who directed the most movies with at least 100,000 votes?passfailfailfail
j03Among directors with at least 7 movies that have 50,000 or more votes, who has the highest average rating across those movies?failpassfailpass
j04Which director born in 1975 or later has directed the most movies with at least 50,000 votes?passpasspasspass
j05Which actors and actresses appear in the cast of both The Godfather (1972) and The Godfather Part II (1974)?passpasspasspass
j06Which 5 actresses have appeared in the most movies with at least 100,000 votes?passpasspasspass
j07How many different people have directed at least one movie rated 8.5 or higher with at least 100,000 votes?passpasspasspass
j08Who has the most writing credits on movies rated 8.0 or higher with at least 25,000 votes?failfailpassfail
j09Which movies did Quentin Tarantino both write and direct?passpasspasspass
j10Which actor or actress has appeared in the most movies directed by Steven Spielberg?failfailfailfail
j11What is the average rating of the movies Leonardo DiCaprio acted in? Round to 2 decimal places.failpasspasspass
j12Which director has directed Robert De Niro in the most movies?passpassfailfail
j13Who directed the highest-rated movie of 1972 that has at least 100,000 votes?passpassfailpass
j14How many movies feature both Tom Hanks and Meg Ryan in the cast?failpassfailpass
j15Who directed each of the 5 most-voted movies? Return each movie's title and its director.passpasspasspass
j16Among actresses with at least 10 movies that have 50,000 or more votes, who has the highest average rating across those movies?failfailfailpass
j17Which Christopher Nolan film has the highest IMDb rating?passpasspassfail
j18Which directors have made at least 6 films rated 8.0 or higher with at least 25,000 votes?passpasspasspass
j19What is the most common genre among movies directed by Alfred Hitchcock?failfailfailfail
j20Which 3 people have directed the most movies with at least 10,000 votes?passpasspasspass
a01What is the average rating of horror movies by decade, starting with the 1960s? Round to 2 decimal places.passpasspasspass
a02How many movies are there in each genre?passpasspasspass
a03Which genre has the highest average rating among genres with at least 500 movies?failfailfailfail
a04What is the average number of votes for movies released in 2010? Round to 2 decimal places.passpasspasspass
a05What is the average rating of each genre for movies with at least 100,000 votes?passpasspasspass
a06How many movies fall in each whole-number rating band, where a band is the rating rounded down (so 7.8 is in band 7)?passpasspasspass
a07What is the median movie runtime in minutes, ignoring unknown runtimes?passpasspasspass
a08How many people have directed 20 or more movies in the database?passpasspasspass
a09Which decade has the highest average movie rating, among decades with at least 1,000 movies?failfailfailfail
a10What percentage of movies with at least 100,000 votes are tagged Drama? Round to 2 decimal places.passpasspasspass
a11What is the average number of genres per movie? Round to 2 decimal places.passpasspasspass
a12How many movies have exactly one genre?failpasspasspass
a13What is the total number of IMDb votes across all movies released in 2008?passpasspasspass
a14What are the 3 most common genres among movies rated 8.0 or higher with at least 50,000 votes, and how many such movies does each have?passpasspasspass
a15How many people in the database list both director and actor among their primary professions?passfailpasspass
a16What is the average birth year of the directors of movies with at least 100,000 votes, counting each director once and ignoring unknown birth years? Round to 2 decimal places.passpasspasspass
a17How many movies have more than 1,000,000 votes?passpasspasspass
a18Which genre has the longest average runtime among genres with at least 100 movies?failfailfailfail
a19What is the standard deviation of ratings for movies with at least 10,000 votes? Round to 2 decimal places.passpasspasspass
a20How many movies were released in each decade from the 1920s through the 2020s?passpasspasspass
n01How many movies has Nolan directed?passpasspasspass
n02How many movies has Scorsese directed?passpasspasspass
n03What is the average rating of the movies Kubrick directed? Round to 2 decimal places.passpasspasspass
n04Which Tarantino-directed movie has the most votes?passfailfailfail
n05How many movies did Hitchcock direct?passpasspasspass
n06How many movies has Streep acted in?passpasspasspass
n07Which movie in The Godfather trilogy has the highest rating?failfailfailfail
n08What year was the movie Alien released?passpasspasspass
n09Who directed Heat?passpasspasspass
n10How many movies has Hanks acted in that Spielberg directed?failfailfailpass
n11What is the highest-rated movie DiCaprio has acted in?passpassfailpass
n12How many movies has Miyazaki directed?failpasspasspass
n13What is the average rating of Fincher's movies as director? Round to 2 decimal places.passpasspasspass
n14How many movies has Nicholas Cage acted in?passpasspasspass
n15What is the highest-rated movie Villeneuve directed?passpasspassfail
n16Who are the top 3 billed cast members of The Matrix?passpasspasspass
n17What year did Parasite come out?failfailpasspass
n18How many movies has Wes Anderson directed?passpasspasspass
n19What is the highest-rated movie with Star Wars in its title?passpasspasspass
n20How many movies did Eastwood direct?passpasspasspass
t01How has the average movie runtime changed by decade since 1950?passpasspasspass
t02What was the average runtime of movies released in 1980, and of movies released in 2020? Round to 2 decimal places.passpasspasspass
t03How many movies were released each year from 2000 to 2020?passpasspasspass
t04How has the average rating of movies with at least 10,000 votes changed by decade since the 1950s? Round to 2 decimal places.passpasspasspass
t05Between 1990 and 2020, in which year was the average rating of horror movies the lowest, counting only years with at least 50 horror movies?failfailfailpass
t06What percentage of movies in each decade since the 1960s are tagged Comedy? Round to 2 decimal places.passpasspasspass
t07How many movies with at least 100,000 votes were released in each decade since the 1950s?passpasspasspass
t08Which year from 1980 through 2019 has the most movies rated 8.0 or higher?passpasspassfail
t09How has the average number of genres per movie changed by decade since the 1950s? Round to 2 decimal places.passpassfailpass
t10How many Animation movies were released in each decade from the 1980s through the 2010s?passpasspasspass
t11List every movie Christopher Nolan directed with its release year and rating, in release order.passpasspasspass
t12How has the average age of directors at release changed by decade since the 1960s? Use release year minus birth year, only for directors with a known birth year. Round to 2 decimal places.passpasspasspass
t13Which decade produced the most movies rated 8.0 or higher with at least 25,000 votes?failfailfailfail
t14What is the total number of IMDb votes for movies released in each decade since the 1970s?passpasspasspass
t15For each year from 2010 to 2019, which movie got the most votes?passpasspasspass
t16What is the average rating of the movies Tom Hanks acted in, by decade of release? Round to 2 decimal places.passpasspasspass
t17How many movies longer than 150 minutes were released in each decade since the 1960s?passpasspasspass
t18Which year in the 1990s had the highest average rating for movies with at least 50,000 votes?failfailfailfail
t19How many different people directed at least one movie released in each decade since the 1950s?failfailfailfail
t20What was the average rating of Drama movies for each year from 2010 to 2020? Round to 2 decimal places.passpasspasspass

How scoring works

The gold SQL for each question runs against the same database the agent uses. The agent passes when the rows from its final query hold the same values, ignoring column names. Extra columns are fine, numbers match within 0.01, and row order counts only when the question asks for an order.

Every question names its vote threshold and, where needed, its tiebreak, so each one has exactly one correct answer. A version that scores worse than the one before it stays in the table.