Defensive Metrics: How Objective Data Can Support Subjective Evaluations

By Joe Hamrahi - Guest Columnist, and staff writer for BaseballDigestDaily.com
12/12/2006

Each year the anticipation grows building up to the Gold Glove (GG) awards presentation. And each year, it seems, the awards announcement is shrouded in controversy and debate. You are almost certain to hear "Derek Jeter won the Gold Glove AGAIN?" as you walk past your company's water cooler. But why have we become so upset about who gets awarded the hardware? Once upon a time, the Gold Glove just meant a little trophy on your favorite player's Topps baseball card. Not anymore!

Times have changed, and so has the dissemination of information.  We have gone from compiling simple statistics such as putouts, assists, and errors to tracking the velocity of the ball, the path it has taken, and whether or not the average player could make a play on it. Enhanced replay and video analysis has enabled companies like Baseball Info Solutions (BIS) to provide comprehensive details about a player's defensive ability. BIS provides a wide variety of data that others can use to analyze and rank baseball defenders.

For example, David Pinto of Baseball Musings has developed the Probabilistic Model of Range (PMR). PMR attempts to measure range based on the ease or difficulty in fielding a specific ball in play.  Easy plays made shouldn’t count much toward determining a fielder’s range.  Difficult plays made should.  What the system does is use examples of balls in play to determine what plays are difficult and what plays are easy. PMR uses six factors; the batted ball type (fly, ground, liner, etc), how hard the ball was hit, the direction the ball was hit, the handedness of the batter, the handedness of the pitcher, and the park.

Then there's UZR by Mitchel Lichtman (MGL). UZR uses a STATS Inc. database, rather than BIS, although the databases should be very similar. MGL breaks the field down into “grids” or zones, much smaller than the zones that STATS uses for its zone ratings, based upon distance and location. The location is a “pie slice” from foul line to foul line, with 22 slices across the field, each one around 4 degrees wide of course. MGL does essentially the same thing with the data and the baseline probabilities as John Dewan does with The Fielding Bible. If a ball is turned into an out, that fielder gets credit for that portion of a “ball” that he normally does not convert into an out. For example, if he normally converts that ball (with all the concurrent parameters) into an out 60% of the time, he gets credit for .4 outs.

Speaking of The Fielding Bible, BIS founder John Dewan has come up with his own ranking system...the plus/minus system. As John describes in The Bill James Handbook 2007, BIS uses in-depth data to try to figure out how each player compares to his peers at his position. A player gets credit (or a +) if he makes a play that at least one other player at his position missed during the season, and he loses credit (or a -) if he misses a play that at least one player made. This may sound simple, but the attention to detail is astounding. And BIS and Dewan are always looking to make the system better. "This year we added fliners (a cross between fly balls and liners), and we completely redid the outfield system [to accommodate the change] ," says John. "Since there was more data to deal with, I had to segment it into compartments a little better. We counted anything hit within 5 feet as a location. Last year everything was counted by foot. But with the addition of fliners, we had more buckets and were able to improve accuracy."

With all the new defensive information available and objective measurements flying around, it's no wonder the fans are up in arms over the Gold Glove awards. Sure, the study of defense is not a perfect science...yet. But voters can certainly rely on the new objective methods to help support their subjective votes. The question is...will they?

John Dewan and his staff introduced the Fielding Bible Awards this year. John formed a panel of experts ranging from Bill James to Mat Olkin to Rob Neyer. He then distributed the defensive numbers that were available, at the time, to the panel. "We told them they could use the reports if they wanted to, or they could use them in their fireplaces to keep them warm," says Dewan jokingly. "The whole purpose of the poll was to incorporate a subjective element (into the voting) because I feel that even with the progress we have made putting together our metrics, visual observation is still a huge part of evaluating defense."

Putting together a panel of "Who's Who" sure lent credibility to the results of the poll. Dewan feels that the group came out "with a tremendous set of awards that we feel truly identifies the best defensive players in baseball." While we don't know who relied on defensive metrics, we do know that a sufficient cross-section of viewers is certainly represented. Not only did the panel include the likes of Bill James, Hal Richman, Rob Neyer, and John Dewan, it also included two groups. The first group came at the urging of Tom Tango. Instead of voting himself, Tom graciously suggested that his fans poll represent "his" vote. Then there were the BIS Video Scouts who, as Dewan puts it, "are watching video on every player in every game."

The results of the Fielding Bible Awards were quite impressive. While the rankings for a particular player would sometimes vary greatly from one voter to the next, the overall results were more credible than the Gold Glove awards voted on by the managers and coaches. Here's the list of winners from the Fielding Bible Awards (which, by the way, were completed before the GG winners were announced) compared to the Gold Glove award winners:

One can infer from the results, that those who relied on defensive metrics, or at least had the numbers at their disposal, made more informed decisions. Does this mean that the actual voting process will change anytime soon? I asked John Dewan if he felt the voting might be based on more objective data in the near future. "It's going to be a slow process...even slower than I thought," says John. "I was pleased to see Albert Pujols recognized. We recognized Albert Pujols last year...a year before the voters. And we recognized Orlando Hudson a couple of years ago before the voters. But, I was shocked to see that Adam Everett was not selected as the best SS in the National League."

It's pretty obvious that the GG voters don't study fielding metrics. Nor does it seem they have read The Fielding Bible. If they did, they would have realized that Everett leads all shortstop in defensive ability...and it's not even close! Still, Dewan makes it clear that "he has tremendous respect for the voters." Aside from Derek Jeter, he doesn't have many problems with the Gold Glove choices!

When it comes to Jeter, Dewan makes a keen observation. "Derek Jeter is the best shortstop in baseball. But...Derek Jeter is NOT the best defensive shortstop in baseball. He wins this [Gold Glove] award because he's the best shortstop. It's easy to vote for Derek Jeter. He excels at every part of the game except fielding ground balls. So it's a bit understandable why he's getting votes." This really makes a lot of sense. When managers and coaches only see an opponent about 20 times a year, it makes it that much more difficult to cast a vote for someone who isn't on the ESPN highlight reel every other night. It would seem that votes are cast based upon reputation and exposure. So when it comes down to voting for Adam Everett or Omar Vizquel (who's been in the league for 18 years), who do you think the voters are most apt to choose?

It would seem that a problem like this could be easily rectified. Why not just distribute the available defensive data to the managers and coaches before they cast their votes? Use the work of multiple authors and different systems if it makes Bud Selig and the league feel less biased or more comfortable. If we have the information, why not use it? Better information leads to more informed decisions. Nobody is telling the voters to eliminate subjective analysis. All we're asking is that they make use of the available objective tools. If they do, it's a good bet that the Gold Glove awards will regain respectability.

Additional Notes

- Alfonso Soriano finished 6th in the Fielding Bible Awards. As Dewan says, "This guy can field! He led the league in baserunner kills. Now he's a defensive asset in LF as opposed to the liability he was at second base."

- John feels that Adrian Gonzalez, Yuniesky Betancourt, and Willy Taveras are among the outstanding up-and-coming defensive players. "I really like Taveras," he says. "He can really move up the list if his bat can keep him in the lineup. He's one of the best defensive outfielders in the league."

- Hal Richman of Strat-O-Matic fame was on the panel of experts for the Fielding Bible Awards. Hal ranked the following players as the best at their respective positions:

Hal cautions Strat-O-Matic players that his vote may or may not reflect the eventual 2006 fielding ratings for players in the game!

 Joe’s work can also be found at http://www.baseballdigestdaily.com/bullpen/index.php?option=com_content&task=blogsection&id=1&Itemid=26