Professor Richard G. Heck (Brown) - Critique of Philosophical Gourmet and Advice on Graduate School

Richard G. Heck, Jr, currently Professor of Philosophy at Brown University critiques The Philosophical Gourmet Report (PGR), particularly the problems in method. Secondly, he gives advice on how to choose a graduate program in philosophy without using the PGR. He points out some interesting flaws, but arguably might have a bias against the report.

About the Philosophical Gourmet Report

"[T]here are still serious problems with the Philosophical Gourmet Report. For example:

1. Just as it has for years, the Report ranks "graduate programs" on the basis of a single factor—the quality of the faculty's research—whose correlation with the quality of a student's graduate education is, though surely positive, arguably small. Factors that are arguably more significant, such as how devoted the faculty are to graduate teaching and whether they are any good at it, are ignored.⁴
2. There is no reason even to believe that the survey accurately measures the quality of the research being done by a given institution's faculty. It asks respondents to rate entire departments, although individual respondents cannot be expected to have first-hand knowledge of the work of more than a few people in any department. An individual's ranking of a "department" must therefore either reflect the work of only a few of its members or, more likely, be based, to a significant extent, upon reputation, not the quality of current research. What the gourmet report measures directly is thus faculty reputation.
3. As one prominent philosopher once mentioned to me, one would certainly hope that there was some significant positive correlation between someone's reputation and the quality of h'er research. But it is not obvious how strong the correlation is, especially with younger people or people who work in more technical (or simply less popular) areas. More importantly, to defend the Philosophical Gourmet Report on such grounds is to commit a simple statistical fallacy: the product of significant positive correlations—that of the quality of a program with the quality of research, and that of the latter with reputation—need not be a significant positive correlation.
4. That said, there is presumably some positive correlation between the Gourmet Report's rankings and the quality of a given department's graduate program. But it remains an open question how well the Report's rankings track anything that should matter to a potential graduate student. What attempts have been made to correlate the Report's rankings with placement results, say, have been inconclusive, but they offer no support for anything but a very weak correlation. And, in purely practical terms, aren't placement results what matter to prospective graduate students?
   To mention placement results is to expose oneself to the ridicule on the ground that one fails to recognize that the Report is supposed to reflect current strength, whereas placement records reflect past strength. But that is a very silly criticism, the obvious reply being that past placement records may nonetheless be a very good (although, of course, imperfect) predictor of future placement success, possibly a much better predictor than the results of a survey. Moreover, the research people are being asked to rank was also done in the past. How well the quality of past research reflects future placement success would seem a very open question.⁵
5. The Report has a built-in bias towards large departments.⁶ For some time, Leiter himself tried to correct for this bias by awarding smaller departments extra points.⁷ That practice, which was absurd on its face, has since stopped, so the bias towards large departments is directly reflected in the rankings.
6. For the 2004–06 survey, "More than half those surveyed were philosophers who had filled out the surveys in previous years; the remainder were nominated by members of the Advisory Board, who picked research-active faculty in their fields." The risks of relying upon a self-selecting group should be obvious.^8"

http://frege.brown.edu/heck/philosophy/aboutpgr.php

Advice on Choosing a Graduate Program

"So you can't trust the Philosophical Gourmet Report. How are you supposed to find out to which programs you should apply, then? How are you supposed to decide in which program you should actually enroll, once you get all those wonderful offers? The short answer, of course, is that you should talk to your advisors, surf the web, and ask a lot of questions. (A good place to start your search is Keith DeRose's list of PhD programs.) But before doing that, you need to have some sense of what questions need asking and what exactly it is that you're trying to find out."

http://frege.brown.edu/heck/philosophy/choosing.php