Rankings Awry

The now AWOL rankings “Top 50 Business Schools for Getting Hired”  showcases the weaknesses and flaws in rankings, real and potential  But first a little background:

Last week CNN Money.com published Fortune magazine’s ranking of the “50 Best Business Schools for Getting Hired.” I chose not to blog it, because I had a lot of other material to write about, and it seemed this ranking wasn’t adding a lot of new insight given the WSJ, FT, and Forbes rankings that focus on employment and ROI. But if you want to see the basic list, without the background data, ClearAdmit has the top 20 listed.

Late last night I received an email from Paul Bodine, author and fellow blogger, suggesting that I look at his post “Forgetting North Carolina.” In a nutshell CNNMoney had published potentially bad data and left out at least two schools, UNC’s Kenan-Flagler and Boston University. CNNMoney and Fortune had based their rankings on data provided by Quacquarelli Symonds Ltd .

Well Kenan-Flagler and Kenan-Flagler alumni did not like being omitted. In fact they were outraged. UNC protested to QS and CNNMoney.  QS’s response alluded to confusion between UNC and North Carolina State. 

Kenan Flagler alumni posted background on the whole mess revealing that UNC was never contacted about the survey and claiming that the published data contained significant errors, in addition to omitted schools. UNC also protested to Fortune and CNNMoney.  According to a letter from UNC:

“UNC Kenan-Flagler did not appear anywhere on the list. This was clearly a mistake, which was acknowledged by QS on Feb. 22. Click the following link for background about what we uncovered as we researched the error. Other schools, though not omitted from the list, have made similar complaints that the data is inaccurate.

“We requested that CNNMoney.com take down the ranking and make a public correction. We have worked with all three organizations to ensure that there will be no further distribution of the ranking.

“The site was taken down on Feb. 26. On Feb. 27, a correction was posted, and we received assurances from Fortune, CNNMoney.com and QS that they will not distribute the ranking in any form.

“We will be sure that all key constituencies — including recruiters, alumni, admitted students and applicants — are informed about the error and how we have addressed it. Career opportunities for UNC MBAs continue to be outstanding and place us among the very best schools in the world.”

CNNMoney’s correction/apology acknowledges that “flaws in methodology may have resulted in University of North Carolina’s Kenan-Flagler Business School and Boston University being omitted from the list.”

It’s safe to say that this mess is a nightmare for QS, CNNMoney, and Fortune. Providing and publishing rankings so sloppily slapped together is beneath criticism for an industry that even when the data is accurate has more than its share of critics and is deserving of skepticism. The CNNMoney/QS fiasco is about as bad as it gets for rankings.

But how should you use rankings when the data collection is professional and thorough and neither the schools nor the publications manipulate the results? Is it possible for rankings to really rank and measure educational quality? Should you simply choose to apply to the Top Ten? Ranking X – Y?

No.

Applicants are misguided and lazy if they mindlessly attempt to apply large category rankings to themselves as individuals. Individuals have different needs and different values. It is virtually impossible for a published ranking to mirror exactly an individual’s needs, goals, and educational values. The rankings, be they for college, medical, law, business, or other graduate programs, simply lack the granularity necessary to provide significant  guidance to individual applicants.

So do they have any value? Why do applicants pay so much attention to them?

Applicants pay too much attention to them. Period. But unlike many of their critics, I do feel they have value:

• When accurate, unlike the removed QS/CNNMoney version, they are sources of information. Sometimes valuable information. Databanks. I use the data, and so should you. If you want to know the average salaries of graduates from particular schools or their average entering test scores, the rankings will have that information compiled in one place. Like a library, they are sources of information. They are not an excuse for decision-making; using them mindlessly could be the equivalent of a lobotomy. And an expensive one at that.
• Rankings are surveys and reflect commonly held beliefs about institutions. Reputation and brand can be factors in choosing where to apply and which schools to accept offers from. They should never be the sole reason you apply or accept an offer of admission. After you research a school’s strengths and weaknesses, culture, admitted student profiles, and educational and professional opportunities, then you can consider brand.