On tests, merit, and firefighting

I have a proposal. Let's do away with long and costly presidential campaigns and the expensive and complicated voting process and just appoint as president the candidate with the highest SAT scores. The many people who supported the Supreme Court's decision in the case of Ricci v. DeStefano should agree with this change.

In Ricci, the Supreme Court decided that the City of New Haven, Connecticut, whose mayor is John DeStefano, had discriminated against white firefighters by not promoting them based on the results of a test. The City of New Haven argued that it refused to promote anyone because of flaws in the test. The flaws, the City's attorneys argued, led to low rates of eligibility for promotion among black and Hispanic firefighters. Since no one was promoted, the City maintained that there was no discrimination. Also, since the test was biased against blacks and Hispanics, New Haven claimed that it would have been sued for racial discrimination if it had promoted firefighters based on the test. Five of the Supreme Court Justices, the majority, were not convinced by the City's arguments.

Supporters of the Supreme Court's Ricci decision argue that firefighting is a serious job where lives are on the line, and it is necessary to select the absolute best person for the job. Being president of the United States is also a very serious job. The president is commander-in-chief of the military so the president also clearly makes life and death decisions. Since the presidency is so important, why don't we have a standardized test where we can objectively see which candidate is most qualified? I propose the SAT.

One thing Ricci-decision fans should note is that the SAT may be biased against Republican candidates. Barack Obama, a Democrat, excelled at Harvard Law School, and taught at the University of Chicago Law School. These achievements suggest to me that he probably has high SAT scores. John McCain, Obama's Republican rival for the presidency, graduated 894th out of 899 from the Naval Academy. In contrast, Jimmy Carter, a Democrat, also attended the Naval Academy and graduated 59th out of 820. If I had to guess, I would assume that Carter would have better SAT scores than McCain. To my knowledge, the Republican darlings, Sarah Palin and Ronald Reagan did not have distinguished academic careers, but Bill Clinton, a Democrat, was elected to the Phi Beta Kappa college honor society. Again, my best guess is that Clinton would beat Palin and Reagan in SAT scores. It looks like the SAT selection method would mean fewer Republican presidents going forward.

Is it fair to use a method of selecting the president that might be biased against one political party? Is it fair to use a promotion test that is biased against one or more racial group? I find both of these biases worrisome, but the actual civil rights law, as I understand it (I'm not a lawyer), is that biases of this sort, or "disparate impact," are allowable if (1) the selection method relates to skills that are absolutely necessary for the job in question, and (2) if there is no equally good alternative that would produce a more democratic, inclusive or racially diverse result.

It should be obvious that among the many problems with using the SAT to select the president, the SAT does not measure the required job skills of being president. It is not a presidential test. What about the New Haven promotion exam? Does it measure the required skills for being a captain or lieutenant in the New Haven fire department? I would argue that in at least one important way it does not.

All parties in the Ricci case agree that having a leadership position in the fire department requires leadership skills. While a test of the sort administered in New Haven might be adequate for assessing firefighting knowledge, it is not likely a good test for assessing leadership skills.

There are two types of leadership skills needed. One is the ability to get along well with, motivate and direct other firefighters. The second is the ability to make good scientific and tactical decisions in a real fire or emergency situation. It is one thing to know the correct answer in a sedate testing situation; it is another thing to be able to think clearly and act decisively in an emergency. Neither of these abilities can be measured with an artificial test. Only real-life interactions can make clear who has and who does not have these abilities. I am not aware of any attempt to assess these leadership abilities by the City of New Haven. So, in my view, the New Haven test does not determine who can best do the job.

Aside from leadership skills, how good was the test at assessing the scientific and technical knowledge that can be measured with an exam? I do not know, but the Supreme Court documents suggest that small changes would have made the test fairer to blacks and Hispanics. In other words, it appears that there was an equally good alternative assessment that would have produced a more racially inclusive result. The result would have been equally good because the changes would have left most of the test questions exactly the same.

Some of the minority firefighters claimed that they had difficulty obtaining the test preparation materials because of cost and because materials were temporarily out-of-stock. This problem can be rectified by making all of the study books available for free to all candidates at the same time. All candidates would then have an equal amount of time to prepare for the test. "Banding"--treating scores such as, for example, 93.5, 93.7 and 93.9 all as 94 and all of the same "rank"--"would have made four black and one Hispanic candidates eligible for then-open lieutenant and captain positions," according to the majority opinion of the Court. This small change would have led to a much more racially inclusive result. Because no New Haven firefighters were allowed to check the appropriateness of the test for New Haven specifically, there may have been a few questions that were bad for a New Haven test. Procedures should be established for one or more New Haven fire department officials or proxies, if necessary, to make sure that all of the questions are appropriate for New Haven. Implementing these small changes--none of which require major changes to the test--would likely have made more black and Hispanic firefighters eligible for promotion.

An additional change that could have been made was to change the relative importance placed on the written and oral parts of the test. According to the dissenting Justices and a firefighter from Bridgeport, Connecticut, changing the weight of the written portion from 60 percent to 30 percent would produce a less racially disparate result without having any negative effects on public safety. The dissenting Justices reported that the average city assigned a 30 percent weight to the written portion of the test in 1996 (the most recent data available). According to the firefighter from Bridgeport, that city's fire department became more racially inclusive once it placed more weight on the oral part of the exam. The firefighter also claimed that the oral portion of the exam deals more with real-life scenarios. It is possible that changing the weights assigned to the written and oral parts of the exam would not only produce a smaller racial disparity, but also improve the overall quality of the assessment.

The larger point here is that standardized tests have to be critically examined. Just because a test is being used does not mean that the test provides a fair assessment of required skills. If we were to begin using the SAT to select the president, I would argue that it is not fair and not an assessment of required job skills. In Ricci, also, it seems that there were changes that could have been made to make the test better and fairer.