Moneyball, but for Mario—the data behind Super Mario Maker popularity

(Note: all data in this piece was taken from the game on the morning of September 30 unless otherwise noted)

In our initial review of Super Mario Maker (which we played on pre-release servers accessible only to developers and critics), we noted that the game was sure to change quite a bit "once the public starts making and uploading its own levels in significant numbers... Who knows what kind of level design trends and counter-trends will rise to the top once the game's own internal social network effects come into play."

Now that over one million players have uploaded over 2.2 million distinct levels in the game over a matter of weeks, we can finally begin to answer that question. After 30 years of "Super Mario" games, Super Mario Maker gives us a rare opportunity to survey what kinds of Mario levels are generally most loved by the public... or at least the portion of the public that gives "stars" to its favorite levels.

We've been keeping an eye on the Top 50 "Most Starred" levels listed in the game's "Course World" section to see if the cream of the level design crop was rising to the top. Our analysis shows that, to the contrary, the levels that seem to be getting the lion's share of public adoration are ones that use some cheap (if occasionally clever) design tricks to attract attention. That's thanks in part to some significant issues with the way Super Mario Maker surfaces levels and measures their popularity.

Auto-completion is the new black

The most starred levels in Super Mario Maker overwhelmingly fall into a few distinct categories. While many of the top levels fall into more than one of these (the second-most popular level is currently an auto-completed stage that uses an Amiibo mushroom to construct an homage to Super Mario Kart, for instance), very few can't be matched to at least one of the following six archetypes:

Auto-complete levels

By far the easiest path to achieving Super Mario Maker popularity is to make a level that plays itself. In total, 19 of the top 50 levels can be completed without hitting a single button (or by simply running to the right). That includes 11 of the top 13 levels, which clog up the very top of the game's Most Starred chart with levels that don't require any active input from the player. Six of these levels were part of a significant subset of auto-complete levels that use pink music-note blocks to play a song as Mario moves. All six of these levels played theme songs from famous Nintendo franchises, of course.

The overwhelming popularity of these levels makes some sense. Auto-complete levels are fun to watch, often bouncing Mario around frenetically and putting him in constant death-defying close calls. They're obviously very easy to finish, too, which gives most players a prominent opportunity to grant a star using the button shown at the end of the level. Still, it's a bit sad that so many Super Mario Maker players apparently like watching the game play itself at least as much as taking an active part in the level.

Ultra-hard levels

On the opposite end of the spectrum, 17 super-popular stages were completed under 15 percent of the time (measured by number of total lives, not individual players). Eight of these levels had completion rates under 10 percent if you want to be stricter with where you cut off the "ultra-hard" moniker. Either way, it seems clear the most popular levels you actually have to play are ones that make players work a little bit to get to the end.

The sadistic "Super Meat Bros." level (an homage to ultra-hard indie darling Super Meat Boy) takes the cake in this category, with a completion rate of just 0.44 percent. That means the average player will die over 230 times before getting to that goal (or just giving up, we suppose). Surprisingly, "Super Meat Bros." has managed to attract over 25,000 stars, even though fewer than 20,000 people have actually beaten it (in over 4.6 million attempts!).

Amiibo costume levels

People seem to love changing Mario into other popular characters. A full 16 of the top 50 levels transformed Mario's appearance using the "mystery mushroom" item, which creators can unlock by tapping the appropriate Amiibo to the Wii U gamepad (or beating an in-game challenge). One popular level—titled "Amiibo Maniac!!"—has attracted over 19,000 stars just by offering a long row of question blocks to let Mario try on every single Amiibo costume, in order, without the threat of any enemies. Somebody obviously has too much money to spend on Amiibos...

Homage levels

A full 20 percent of the Top 50 levels were explicitly inspired by some other game. Some of these levels were downright clever, from Metroid and Zelda-themed dungeons to levels that accurately recreate Super Mario 64 stages in two dimensions. Others were downright confusing, like a "SONIC 1 MEMORY" level that bore only a passing resemblance to the original game's Green Hill Zone. Whatever the quality, players seem to respond strongly to seeing familiar games reinterpreted with Mario Maker's palette.

"Gimmick" levels

I loosely defined this category as "levels that have gameplay that's significantly different from what you'd usually see in a Mario game" (auto-complete and homage levels excluded). One gimmicked level requires players to avoid a phalanx of power-up mushrooms to fit into a small tunnel. Another puts a fire-flower powered Mario in a clown-copter to dodge flying spinies, shmup-style. Two more put Mario in a race with enemies that will cut off his path if he's too slow. There are levels that force Mario through narrow mazes and levels that ask Mario to just dance around with Yoshi in this category, too.

These levels get bonus points for cleverness, but they seem hard to imitate without getting tiresome. With gimmick levels, once you've seen the basic idea, you generally don't need to see it again.

"Real" levels

These are levels that I subjectively judged would fit in an actual Super Mario Bros. game—not too hard, not too gimmicky, and not too derivative of other games. Real levels need a good mix of enemies and power ups and calm and exciting sections. Even being generous on all of these counts, the "real" levels were still easily outnumbered by auto-complete tomfoolery, gimmicky tricks, or ultra-hard challenges (though two "ultra-hard" levels also made it onto the "real" list).