OVERVIEW of Criticisms of the U.S. News & World Report rankings of ìAmerica's Best Collegesî.

1. The Peer Evaluation is a glorified opinion survey. It does not measure the quality of undergraduate education.
2. Bias and inaccuracy in the data. 
3. The rankings really measure high school achievements.
4. The rankings do not measure academic quality of colleges.
5. The books are cooked by manipulating the weighting factors.
6. Real measures of academic quality - learning outcomes - are totally absent!
7. The premise of a "BEST" college or university is deceptive and wrong.
8. What is good about the USN&WR "America's Best Colleges" issue.

The U.S. News & World Report (USN&WR) rankings of ìAmerica's Best Collegesî are highly controversial. The best that has been said of them is that the issue is a useful starting point for a college search. The worst that has been said of them is that they are simply a ìbeauty contestî with the primary purpose of making money for the publisher. A short review of some of the most prominent criticisms is presented below.

OVERVIEW:  What are the criticisms of the U.S. News & World Report rankings?

The U.S. News & World Report rankings of ìAmerica's Best Collegesî incorporates a number of categories. However some categories are given much more weight than others; for example the ìPeer Evaluationî (really an opinion survey) accounts for 25% of the score, while Student:Faculty ratio accounts for only 1%. See the categories and weights.
 

There are many problems with the methodology used by USN&WR. Some of these problems have been summarized in "Grading on the Curve" by Glenn Kersten. More than this, it is questionable that a single rank number can accurately represent the quality of an education for several thousand individual students at a college.  The entire concept may be fundamentally flawed.

The scores assigned by Peer Evaluation will frequently have more to do with the reputation in research of a college or university, the amount of money it garners in federal funds and the publication record of its faculty. These things are both more visible and much easier to quantify. This score contributes the largest proportion of the total score, yet it may have very little to do with the quality of teaching and learning for undergraduates!

The fact that the survey probably measures research reputation rather than teaching quality, suggests that a "good" score in this category may not be good for many students. Unfortunately, faculty who spend more time on research, may have less time for teaching and undergraduates.

Another problems with the "evaluation" is that USN&WR will not report what questions are asked, so nobody knows what a score of "3.5", or a score of "4.0" actually means!

 There are two sources of bias - or even inaccuracy - which are of concern:  1.) strategic ranking;  2.) manipulation and misrepresentation of the data which schools submit.

Strategic Ranking.  The administrators who are asked to complete the surveys are generally from similar colleges and universities in similar regions. Thus they are asked to rank the very schools with whom they compete for students. In such a case, administrators will be tempted to rank a competitor lower in order to make their own school appear better. 

Manipulation and Misrepresentation of Data.  There is good evidence that some colleges and universities inflate the SAT scores of their students and manipulate other critical data to improve their ranking.

3.  The rankings really measure high school achievements. 

USN&WR reports that they use many categories in ranking colleges and universities. In reality however, the rankings are largely determined by only five factors. Moreover the determining factors measure what the students did in high school - not what they achieved at their college! In other words it is the students who are making the college, not the college which makes the students!

In order to determine the real importance of each of the categories (as opposed to the weight that USN&WR says they carry), data from the first tier of Masters Universities in the northeast  has been analyzed. Correlation Coefficients of Rank vs. Category were calculated. The categories which most influence the rating are:

1. Opinion survey
2. Freshman retention rate (% of freshmen who return for the sophomore year).
3. Graduation rate (% of students who graduate within 6 years).
4. % of freshman in the top 25% of their High School graduating class.
5. SAT range of the middle half of the freshman class (25th - 75th percentiles).

While it might be presumed that the 2 remaining categories, Freshman Retention Rate and Graduation Rate, are measures of academic quality in the college, this is not necessarily true either. In fact, both of these categories are closely related to SAT score and High School Rank, and the college itself seems to make little difference, for better or worse. Good students stay in college and graduate, regardless of the academic quality provided by the university.  Consequently Freshman Retention Rate and Graduation Rate really count freshman ability twice, while seeming to be something else!

On the other hand, categories which might have something to do with educational quality (% of classes <20; student:faculty ratio) are far less important, and a high proportion of committed full-time faculty counts for nothing! Even worse, the rankings actually reward colleges for having large classes!

4.  The rankings do not measure academic quality of colleges.

It is also instructive to note what categories do not have much influence on rank. If the relationship between these categories and Rank is plotted, it becomes clear that they have no influence, because there is no difference in academic quality between high and low ranked schools!

1. % of Full-Time Faculty -- there is very little difference, if any, between the top colleges and the bottom ones.

 
2. % of classes less than 20  -- at first glance it appears that highly ranked universities offer smaller classes than the lower ones. However there is a large amount of scatter about the trend line. Moreover, if the data are inspected carefully, the only reason for the trend is that the top university is very high, and the bottom college is very low. Otherwise there is no difference between #2, #3, #4 and #5, and #33, #34 and #35.

 
3. Student:Faculty ratio -- there is very little difference, if any, between the top colleges and the bottom ones.

 
4. % of classes greater than 50 -- there is very little difference, if any, between the top colleges and the bottom ones. Moreover the % of large classes at Comprhensive Universities is so small that it is unlikely to have a meaningful effect on the college experience for students.

5.  The books are cooked by manipulating the weighting factors.

One of the criticisms of the USN&WR rankings is that they have not been designed to rank colleges on the basis of academic quality at all. Rather they start with the presumption that elite private research universities such as Harvard, Yale, Princeton and Stanford are the "best" for everyone, and have simply cobbled together a formula which makes them come out on top year after year.

A chart of the importance of the various categories in determining a college's ranking shows the basis for this charge. Only 20% of the ranking is based on factors related to teaching quality. 10% of the ranking is based on the spending per student,, (even though many would argue that throwing money at schools does not necessarily improve education). 38% of the score is based on measures of student ability when they enter from high school, and 32% is based on the survey of opinion. But why does USN&WR do this?

In fact there is no justification for the weighting factors used and they have been heavily criticized by experts on higher education. If for example, small class size contributors 6% to the rank, why do large class sizes contribute only 2%. And why is the % of full-time faculty and the student:faculty ratio worth almost nothing? In fact a rigorous mathematical study of the weighting factors concluded that they were baseless and arbitrary.

The significance of this is that by choosing biased weights, schools can be moved up or down to manipulate the ratings as desired. To illustrate this point, a simplified ranking system could be constructed, which used only measures of real academic quality. With the weights chosen by USN&WR (% of classes less than 20 is 6 times more important than % of full-time faculty; % of classes larger than 50 is twice as important as % of full-time faculty; ) Harvard ranks #1 as shown in the first column. However it would be just as reasonable to say that a low number of large classes is 6 times more important than having small classes or a full-time faculty. In this case Harvard moves from the top of the rankings to the middle of the pack, and Fordham moves from the bottom to the top, as shown in the last column!

The conclusion is that the rankings are not really measures of the "best" colleges at all. They are really just a list of the universities which are most similar to Harvard, Yale and Princeton - old, famous, rich, private research universities, exclusively for those who have played the high school game the best, and don't need good teaching anyway! Since they are not measures of academic quality, whatever the rankings say about colleges or universities not like the Ivy League - such as smaller, regional, liberal-arts colleges and universities - is meaningless at best and deceptive at worst.

6. Real measures of educational quality - learning outcomes - are totally absent!

As shown above, the USN&WR rankings have been criticized because the categories, and the weighting factors, do not measure academic quality. The rankings have also been criticized because they ignore factors which domeasure academic quality.

Some of the vital considerations which USN&WR ignores are as follows:

1. Career Development of Undergraduates.
2. Skills and Commitment of the Faculty in Mentioning Undergraduates.
3. Career Outcomes.

7.  The premise of a "BEST" college or university is deceptive and wrong.

 

Not all colleges and universities have the same goals and missions. Consequently the question isn't ìWhich college or university is ìbest?î, but rather ìWhich college or university is best for what?î  Moreover, not all college students are alike or have the same needs either.

Dr. Gerhard Caspar, former President of Stanford University has publicly questioned the basic premise of ranking universities.
 

8.  What is good about the USN&WR "America's Best Colleges" issue.

While the rankings themselves are not to be taken seriously, the issue itself contains many articles with solid advice. On the whole there is nothing new here, but it is an excellent compendium and exposition of the conventional wisdom.

The issue is also valuable because it lists most of the colleges and universities in the U.S. by region and category. This makes it possible for students and their families to identify all the options they have, and to consider some good schools which they may not have known about.

Finally the issue provides some hard data about colleges and universities. It is unfortunate that USN&WR fails to provide much of the information which would be most useful, and misuses the data it does provide, but that does not mean the data itself is without value.