2021 NFL analytics survey – Most and least analytically inclined teams, future GM candidates, more

The Cleveland Browns have taken over the title of the NFL’s analytically advanced franchise from a division rival.

When we surveyed analytics staffers across the league a year ago, they voted the Baltimore Ravens as the most analytically advanced — defined by a combination of the level of data-driven work produced and the degree to which that work is implemented in decision-making. But in 2021, the Browns were runaway winners, as voted by their peers in an ESPN poll.

This was the second year we’ve conducted a 13-question survey of the league’s analytics minds to gain a better understanding of the league’s quantitative landscape and how it is evolving. We sent the survey to an analytics person with each team, and 22 responded.

Some staffers left additional comments, and others were called by ESPN for contextual follow-ups. Participants were allowed to select their own team where applicable, and questions were asked with the understanding that they don’t have perfect information about other teams. All were granted anonymity so they could speak freely, and though there were 22 respondents, a few abstained from some questions. Let’s begin with the teams that seek and use analytics the most in today’s game.

Jump to:
Most advanced | Least advanced
Most useful metrics | GM candidates
Obstacles to more buy-in

Which NFL team is the most analytically advanced?

1. Cleveland Browns (17)
2. Baltimore Ravens (4)
3. Miami Dolphins (1)

Which team produces the highest level of analytics work?

1. Cleveland Browns (14)
2. Baltimore Ravens (3)
3. Buffalo Bills (2)
T4. Dallas Cowboys (1)
T4. Indianapolis Colts (1)
T4. Philadelphia Eagles (1)

Which team most incorporates analytics into its decision-making?

1. Cleveland Browns (11)
2. Baltimore Ravens (6)
3. Philadelphia Eagles (2)
T4. Green Bay Packers (1)
T4. New York Giants (1)
T4. Indianapolis Colts (1)

Cleveland received the most votes for highest level of work last year, but now it’s a clean sweep of these three categories in the second season of the Andrew Berry-Kevin Stefanski regime. The Browns are “making best use of resources and making every effort to gain a competitive advantage through analytics, so hearing it was a runaway was not too surprising,” said an NFC analytics staffer.

An AFC analytics employee added that Berry has “a very unique combination of people skills, general intelligence — obviously he’s got a strong IT and data science background — and he’s trained as a scout in the league. He’s got a unique blend of all those things.”

The Browns’ integration of analytics into their fabric is hardly a secret. It was first strongly embraced when Sashi Brown ran the team in 2016, then went dormant under John Dorsey and then came back to the forefront again under Berry. The Browns now have the largest analytics staff in the league, with a group that includes chief strategy officer Paul DePodesta — who began with Cleveland in 2016 at the start of the Brown era — and three vice presidents in analytics or with an analytics background.

Multiple analytics staffers posited that the Browns probably should have received more credit previously and that votes swung their way now that they are having success on the field. (Cleveland is 3-1 through Week 4.)

“They got a lot of criticism for a number of years, I guess a lot of traditional scouts would say they were so married to the idea of winning draft-day trades and acquiring future years’ draft capital,” a veteran analytics director said. “And they smartly did that and they’re kind of reaping the rewards now. Their roster is pretty much as talented as any you’ll find in the league top to bottom, and I think that’s directly a product of their ability to acquire so many valuable draft assets.”

The Ravens still earned a few votes for most analytically advanced and a few more for the degree they incorporate data into decision-making. The most obvious place this shows up is in their fourth-down aggression, which was on display in their win over the Chiefs in Week 2, when QB Lamar Jackson sealed the win with a fourth-down conversion.

“Andrew Berry leads the NFL in terms of acceptance and adoption of analytics. On the coaching side, John Harbaugh has the most buy-in,” wrote one survey-taker.

Which teams are among the top five most analytically inclined?

Cleveland Browns (22), Baltimore Ravens (22), Philadelphia Eagles (14), Buffalo Bills (12), Indianapolis Colts (8), Los Angeles Rams (6), Minnesota Vikings (4), San Francisco 49ers (4), Jacksonville Jaguars (3), Atlanta Falcons (2), Green Bay Packers (2), New England Patriots (2), Dallas Cowboys (1), Denver Broncos (1), Detroit Lions (1) and New York Jets (1)

The Eagles remained the No. 3 team on this list, despite an offseason report that their analytics department was at the center of an organizational conflict during the Doug Pederson era. A veteran director said, “It kind of comes from Howie Roseman down. He’s always been a proponent of analytics. I think, at least on the scouting side, they’ve always been pretty advanced.”

One front-office member was particularly impressed with the Rams’ ability to go against the grain, citing examples from limiting staff pro day and combine attendance to the pass-first approach they took to defense in 2020: “I love that they just seem that they challenge themselves in terms of their thought process.” Even if Los Angeles’ contrarian angles weren’t all analytically based, they felt that their openness to being different indicated they were likely open to using data to find an edge, too. Prioritizing the pass is certainly an analytical tenet.

“I’m inferring a little bit, but everything they did on defense last year, even what McVay was looking for when he was trying to find a defensive coach,” the senior staffer said. “The pass-focused part of it, which still is not the norm in the NFL. The norm is we have to stop the run first even though everybody knows it’s a passing league — I think it was progressive for them to act that way.” The staffer added that those who think outside the box are often the ones who find temporary advantages.

Indianapolis got some attention here, too. An AFC staffer said, “I hear the Colts do some interesting things with game management. I hear [Frank] Reich is really into it and those guys are really involved.”

Staying in the AFC South, the Jaguars have a large analytics staff, though many members have hybrid roles split between football and business responsibilities. This past offseason, the team hired senior vice president of football operations and strategy Karim Kassam, but he then left in what the team described as a mutual agreement just a few months later.

play

Stephen A. Smith and Tim Tebow disagree on whether Urban Meyer has the mindset to handle the Jaguars’ 0-4 start.

“I considered Jacksonville for another team but having people in the area doesn’t mean much if it’s Urban Meyer’s show,” wrote one front-office member. “See that as a mess where they don’t seem to know what they’re doing unfortunately. … They clearly have a divide in the building. I don’t understand why anybody puts a big investment in this stuff and then also entrust the most key roles with folks who don’t want to have anything to do with it.”

Lastly, in follow-up conversations, a survey-taker named the Jets and Lions as under-the-radar teams here, and another pointed to the Giants and Seahawks as potential top-10 clubs.

Which team in the NFL is least analytically advanced?

1. Tennessee Titans (8)
T2. Cincinnati Bengals (4)
T2. Washington Football Team (4)
4. Las Vegas Raiders (2)
T5. New Orleans Saints (1)
T5. New York Giants (1)
T5. New York Jets (1)

One voter abstained.

This end of the spectrum actually seems tougher for staffers to decipher. After all, being the most analytically arcane might be hard to spot relative to the competition. But there’s one way a team can stand out in this area: staffing. And that seemed to be a factor for why the Titans took the category.

Until recently, the Titans were the only team, to our knowledge, without a full-time analytics worker in their football operations department. That changed when Tennessee hired Matt Iammarino as assistant developer of analytical football research in August. Not having a defined analytics department doesn’t definitively mean an eschewing of analytics, but it is a strong clue. Plus, a veteran analytics employee noted that the Titans’ infamous 4th-and-2 punt from the Ravens’ 40-yard line in the fourth quarter of the team’s wild-card loss last year was a red flag.

Still, Tennessee was far from an unanimous vote. Cincinnati and Washington were runners-up here, and each have only one known full-time analytics worker.

“I mean, the Bengals are an easy target,” said an AFC analytics staffer. “I just know how their scouting department works, and you look at their directory online. They have one guy? Their decision-making isn’t quite there yet. I don’t know for a fact that they’re worse than anyone else, but they’re an easy target, and I’m probably right.”

Which player-level metric in the public sphere is most useful for player evaluation?

EPA-based metrics/Total QBR (6)
Pro Football Focus grades/WAR (3)
Pressure statistics (2)
Approximate Value (2)
Target rate (1)
Yards per attempt (1)

Seven voters abstained.

Expected points added (EPA) is a longstanding staple of football analytics work. The idea is to view the game through the lens of points as opposed to yards — meaning it includes the context of down, distance, yard line and clock.

“EPA is probably the most available thing and is definitely something we use on a regular basis,” an NFC analyst said. “Because I think it just further allows you to delve into player contribution — what true value is within a play that may not be as easily decipherable given your generic box scores and production.”

Among current NFL analytics staffers, who will first become an NFL GM, if any?

Kwesi Adofo-Mensah, Browns VP of football operations (8)
Dave Giuliani, Browns director, research & strategy (2)
Alec Halaby, Eagles VP of football operations & strategy (1)
Brian Hampton, 49ers VP of football administration (1)
Dennis Lock, Bills director of football research & strategy (1)
Andrew Healy, Browns VP of research & strategy (0.5)
Ken Kovash, Browns VP of player personnel process & development (0.5)

Seven voters abstained. An additional vote was cast for “someone from Cleveland.”

No one has ascended from an analytics role to the general manager chair yet. How quickly will that happen and who will do it first?

“It’s going to happen soon,” a senior staffer said. “I think you’re going to have owners wanting GMs who have it all. There’s really no reason you can’t have a football person who is data fluent and data interested. It’s not an either/or.”

Adofo-Mensah, viewed as Berry’s right-hand man in Cleveland, does appear to be the most likely candidate. He’s a top lieutenant on the most analytically advanced team in the league, previously was the director of football research and development with the 49ers and interviewed for the Panthers’ GM job last offseason.

He was one of four members of the Browns’ front office to be mentioned on a ballot (one voter split between Healy and Kovash), another sign of their stature among teams analytically. How quickly the next data-first GM is hired may, the senior staffer noted, depend a lot on how well the Browns perform. An AFC staffer, however, was somewhat skeptical that any Cleveland copycats would necessarily work the same without Berry.

“If the Browns can show they can sustain success for the next two, three years, somebody’s going to try and recreate that model,” they said. “Whether or not that’s successful, it’s like anything else: You can’t just take something and pick it up and move it somewhere else and expect it to work the same way.”

The Bills being a contender similarly worked in Lock’s favor, thought the analytics worker who voted for the current Buffalo director, saying “I’ve always just held him in high regard. … PhD in statistics, as bright as they come.”

The voters were asked specifically for the first person they thought would ascend to GM, but an NFC analyst added that down the line Rams manager of football analytics Sarah Bailey might one day reach that level, too.

Which areas of football operations do you or your analytics team work on?

Coaching (22)
Pro personnel (20)
Draft (20)
Game management (20)
Sport science (18)

The 22 voters were asked to check all that apply.

For me, the breadth of analytics departments really stood out here. But for those around the league, it was actually an expected result.

“It doesn’t surprise me at all. Obviously there’s differing levels of involvement,” an AFC staffer said. “Anytime you’re going to need any advanced data analysis, that’s what the analytics departments is for. So you’re going to do projects for everybody in those departments. How often you do them or the level of actual involvement will vary. But you’re going to do a project for everybody on that list at some point.”

An NFC front-office member noted that hitting low-hanging fruit across all five areas was a mechanism to get the analytics team in the door with different groups. From there, it could build upon that work for larger projects.

Does your team have an analytics staffer on the coaching headset during games?

Yes (15)
No (6)

One voter abstained.

Perhaps no analytics work is more visible than fourth-down and two-point conversion in-game decisions, and roughly two-thirds of the teams surveyed indicated they have an analytics employee on headset. What determines if this is the case?

“Honestly, either the head coach has to want it or the owner has to want it and make the head coach agree to it,” an AFC analytics staffer said. Others noted that relationships play a key role, as the head coach is going to want a level of comfort with whoever is delivering the information. At least one team surveyed said they have an analytics staffer in the booth but not on the headset.

Does your team use raw player-tracking data to create proprietary in-house metrics?

Yes (19)
No (2)

Last year’s survey asked if player-tracking data was used at all, and almost every respondent said yes. So this year, we tried to be a little more specific — but the answers were still overwhelmingly positive.

“That’s actually great to hear,” said a senior staffer of the results. “I wouldn’t have guessed [19] of 32 even would have been creating in-house metrics from raw [NFL Next Gen Stats] data.”

But others thought the result made sense.

“Just given how long we’ve had access to this data and the rate that teams are hiring, I would be very surprised if even on the lower department sizes if they don’t have one person that’s at least starting to play around with the player tracking and incorporating it in-house,” an NFC analyst said. “Just because that’s the direction the game is heading in.”

The analyst added that when digging in to the player tracking themselves, they started by recreating public player-tracking work first — from the Big Data Bowl (e.g. quantifying defensive pass coverage) and ESPN (e.g. pass block win rate) — and then worked to progress beyond that. Asked for an example of player-tracking usage, an AFC staffer said their team uses a completion probability model, though they created their own in-house model rather than the one provided by NFL Next Gen Stats.

When it comes to analytics, the average NFL team is ____ years behind the average MLB team.

Average response: 9.8 years
Range of responses: 5-15 years

“Honestly how many teams are data-first in baseball? It’s certainly a healthy number and it’s an essential part for the vast majority,” said a senior staffer. “We’re just a really long way off that. The game, baseball, has fundamentally been changed in so many ways — the shift, how teams use their pitchers. [In football], fourth down, things are moving, but that train just started in the last few years. I think in 10 years, I think you’ll see football in a very, very different place.”

Others felt like there wasn’t as large a gap as it seemed, and that the added difficulty of quantifying a more fluid game in football was a key separator.

“Fifteen years is an eternity!” said a director when told of the range of votes. “I think it’s harder to do in the NFL because there are just so many people that are coming from more of a traditional football background. Because I think the knowledge that’s required is a lot more specialized. The coaches at the highest level are really kind of PhD’s in the game of football.”

The same director noted that the variety of both skills and body types in football were far more varied than in baseball, where specialized analysis of, for example, pitching mechanics can apply to half of the roster.

If you could wave a magic wand to improve analytics usage at your team through one of the following, which would you choose?

Increased buy-in from decision-makers (8)
More staff (8)
Higher quality raw data (5)
More third-party tools (0)

One voter abstained.

“We see the ability that we have with, say, five individuals in this department,” said someone with an NFC team who voted for increased buy-in. “What’s going to be that multiplicative factor if we add more people, or we allow them access to better data. As long as you’re getting buy-in from decision-makers, the rest of it falls in place.”

But another staffer argued that a larger staff could make a world of difference. “There’s a need for the skills on analytics teams to be a little more specialized and a little less general purpose. When your staff is small, you have to be more of a generalist,” they said. “You can get so bogged down with the routine of the NFL and responding to those [ad-hoc] requests. You need folks to clear their schedule and they can work on longer term projects.”

A senior front-office member who voted for higher quality data said they viewed the question a little more philosophically and imagined a theoretical dataset that contained the assignments for every player on every play. That would go a long way toward quantifying individual player’s value.

What typically prevents NFL teams from further adopting analytics in football operations?

This final question was left open-ended and generated a variety of responses. The most common theme was communication with decision-makers.

“I don’t think it’s tough to explain to them what the model is supposed to do. I think it’s tough when the model doesn’t match up with what they see,” an AFC staffer said. “It’s in the cases where they’re clearly right about what they see and it’s just some type of unique case where the model isn’t strong in a certain area because it’s not accounting for something.”

They noted that in, say, a draft meeting you often don’t have much time and typically are just giving a quick breakdown, but that in those edge cases, that staffer tries to take the time to explain why the model is producing a surprising result. Otherwise, they risk coaches quickly losing trust in the model.

“I don’t think every team does that,” said the AFC staffer. “[The decision-makers are] just like ‘Oh, this model’s f—ed up’ and they move on.”

Some responses placed an emphasis on the analytics workers’ ability to communicate, while others placed the onus on decision-makers who aren’t invested in data-based approaches or just have an old-school mentality in general.

“Mistrust, lack of communication, arrogance, fear, short-sightedness, lack of resources, novelty all play a factor. Breaking down the wall between coaches/scouts and quants takes time, trust and collaboration,” wrote one survey-taker.

“Decision inertia,” another respondent said. “Teams are slow to adopt new methods and then need to see immediate success or risk going back to the ‘traditional’ way of doing things.”

Lastly, one survey-taker laid out an issue that plagues NFL teams in analytics and beyond: “Excellent processes don’t guarantee excellent results without a good QB.”