Legal Affairs

Current Issue
printer friendly
email this article

space space space
Debate Club

Judges In Lab Coats?

Michael L. Martinez and Jay P. Kesan debate.

This Week's Entries: Monday | Tuesday | Wednesday | Thursday

In 1993, the Supreme Court decision in the Daubert case changed how scientific evidence is presented in federal court. Under Daubert, the trial judge serves as a gatekeeper for the types of expert evidence that may be admitted, instead of the jury evaluating the evidence on its own. Many legal academics and tort lawyers have claimed that, since then, there has been a perverse incentive for defendants (chemical and pharamaceutical companies are often cited) to fund "scientific" studies to be used in litigation, studies that often come to conclusions opposite those reached in peer-reviewed research. As a result, drugs have been allowed that increase risks of cancer and pose other safety problems, according to critics. Even if this mercenary research can't disprove the peer-reviewed studies, it can create enough of an appearance of uncertainty about key evidence to sway a courtroom.
Science in the Courts

Don't Knock Daubert Until We've Tried It
An arms race of scientific studies may be good for both science and the law.

Should evidence from commissioned scientific studies be banned? If not, do trial judges following Daubert need more guidance about how to evaluate mercenary research and, if so, what kind?

Michael L. Martinez is a partner in the Washington, D.C., office of the law firm Crowell & Moring LLP. Jay P. Kesan is Professor & Director of the Program in Intellectual Property and Technology Law at the University of Illinois.

Martinez: 4/3/06, 07:44 AM
This is a very interesting question, particularly as it is framed. There should not be any blanket bar on commissioned scientific studies that are developed for possible use in litigation. Rather, such studies should be analyzed and subjected to the same rigorous analysis applied to other scientific and technical testimony. If such studies pass muster then courts should permit their admission into evidence.

The Daubert decision is commonly viewed as the Supreme Court's effort to keep junk science out of the courtroom—a fair interpretation I think of what the Court was trying to do. In trying to accomplish that goal, the Court identified factors the lower courts should apply before admitting into evidence a scientific or technical opinion. The Court explained that the scientific or specialized knowledge must assist the trier of fact, must be relevant, and must "fit" the particular facts of the case.

Now in assessing how to apply that test, the Court explained that various factors should be considered. For example, a judge should consider things like whether the methodology at issue has been published and peer reviewed, whether a rate of error is known, and whether the methodology is generally accepted as valid in the scientific or technical field. This is not an exclusive list, and importantly, the Supreme Court emphasized that the ultimate test for the admissibility of a scientific or technical opinion should be flexible. One size does not fit all.

With that as guidance, it seems that there is ample room for commissioned scientific studies to be admitted as expert evidence in appropriate cases. So long as the studies use a valid methodology and meet the other criteria of Daubert there is no reason to conclude otherwise.

I think the question as framed is interesting because I think the bigger problem with litigation funded studies lies more with opportunistic plaintiffs than with corporate defendants. For a variety of reasons—not the least of which is to protect their good names—corporate defendants have an incentive to fund studies that lead to a correct conclusion. By contrast, plaintiffs have more incentives to fund studies that support their claims, whether or not they are scientifically correct.

There is little doubt that judging the difference between what is a valid study and what is not can be a daunting task, particularly for a trial judge who lacks any expertise in a complex subject area. The Supreme Court recognized that issue in Daubert and expressed confidence that trial judges are competent to solve this problem, whether or not they have the particularized expertise.

Kesan: 4/3/06, 09:10 PM
Many thanks, Mike, for kicking off this discussion about scientific evidence in the courts and the Daubert standard.

Mike notes that Daubert was the Supreme Court's effort to keep junk science out of the courtroom. I agree that Daubert and its progeny, Joiner and Kumho Tire, attempt to police the admissibility of scientific evidence in the courts.

Junk science is, unfortunately, not merely the creation or exclusive province of "opportunistic plaintiffs," as Mike states; rather it is at least equally the province of "corporate defendants," too.

In the context of defendants, let us make sure we understand what we are talking about. There are numerous well-documented studies that show that there are systematic attempts to generate favorable scientific results in anticipation of or to preempt litigation. Outcome-driven science (ODS) is junk science. In a sound-bite driven world, let me add one of my own. ODS is odious, if you need a slogan.

Mike, I find that you are a bit too eager to conclude that "corporate defendants have an incentive to fund studies that lead to a correct conclusion." What is the basis for this statement? What are these incentives you are referring to? Any there any studies showing this? Is it anything more than personal opinion? Where is the empirical or other evidence to support this conclusion?

The Daubert opinion outlined several factors that judge should consider when deciding whether the proffered evidence amounted to "scientific knowledge" vel non. I would add that complete information about the entities funding a particular study and the amount of annual funding provided by each entity should be another important factor considered by the trial judge. This information should be completely disclosed. Trial judges may perhaps be competent to assess whether the evidence being presented amounts to scientific knowledge, but only if there are presented with full and complete information. Unfortunately, in the past, scientific studies have been funded by outside counsel in law firms acting as intermediaries between private firms and scientists to ensure that the funding and other relevant information pertaining to the study remain privileged, and therefore, not discoverable in civil litigation.

In short, Mike, I agree with you about the overall goals and objectives of Daubert. I am considerably less sanguine about the means to get there without additional policing and appropriate legal mechanisms to ensure that the substantive policies animating the Daubert decision are thoroughly vindicated.

There are several other points worth making—what should these legal regimes and mechansims look like?—but I will wait until another day, and I look forward to reading your response. Please stay tuned.

This Week's Entries: Monday | Tuesday | Wednesday | Thursday

Martinez: 4/4/06, 12:33 PM
Good morning, Jay. I'll just dive in: I do not buy into your term "outcome-driven science" because I think the term improperly suggests the conclusion you seek to reach. Let's look at it another way. Parties are engaged in complex scientific or technical litigation that requires unique expertise. One of the parties (let's say a defendant corporation) decides that scientific tests of some kind might be useful to bolster its case. So, the defendant seeks out and retains some experts to conduct these tests.

There is nothing inherently wrong (or odious, to use your word) with that. Critically, the judge in his role as "gatekeeper," ultimately will decide whether the science or tests at issue meet all the Daubert criteria of reliability, relevance, etc. that I discussed yesterday. So whatever the study concludes, there is always a judge at the gate deciding whether the results are scientifically valid and not "junk." If they are valid results that aid the factfinder in his or her resolution of the case, does it really matter who funded the study?

In fact, as my law partner Bill Anderson and I explained a few months ago in an article published in Legal Times, the decision to conduct such tests in the context of litigation is very risky. First, what happens if the tests do not yield the result the defendant desires? That fact likely will be discovered by the plaintiffs at some point. Second, the reality of a test gone bad, depending on the situation, may be fatal to the defendant's case, as well as harmful to the corporation's product (if a product is at issue) and good name. Third, even if these defense overcomes these hurdles, the plaintiffs will try to taint the results as not credible because they were funded by the defendant. Fourth, regardless of the results, the process of how the tests were established and conducted in litigation is subject to rigorous review via the discovery process. These are powerful incentives to ensure that any tests conducted reach a correct result as opposed to a merely litigation-friendly result.

So Jay, I think there are more than adequate safeguards to offset your concerns about odious science. Clearly, not every plaintiff plays by the rules and not every defendant plays by the rules. We do not have a perfect system. But we do have a system that is designed—where complex expertise is involved—to get at the correct answers to aid the trier of fact. If getting at the correct answers involves litigant-funded science or tests, there is nothing per se wrong with that.

Kesan: 4/4/06, 09:15 PM
Mike, you are right. The hypothetical situation you describe involves a defendant corporation that decides that "scientific tests of some kind might be useful to bolster its case" and then proceeds to perhaps conduct such tests and present the results to the court.

Normally, this situation permits the opposing side to examine the data, analyze it, and perhaps even reproduce the same tests using experts of their own. In addition, they may be able to depose the other side's experts who conducted the tests, question them about the samples, the conditions, and the like, and thoroughly subject them to detailed questioning.

Alas, if these norms and conditions prevailed in every situation, there would be no need for this debate!

Indeed, parties recognize the perils that you so correctly pointed out in your Legal Times article. So they protect themselves from having to face the level of scrutiny that you extoll. How do they do that?

They employ some of the following measures. They fund studies in anticipation of future litigation. These studies can be extensive and not conducted within the limitations or the robustness of the full adverserial process. For example, the scientists conducting the tests are not hired experts who can be questioned in a deposition. The studies can also be funded through intermediaries such as other law firms hired by the firms to hide behind the protection afforded by the attorney-client privilege, as I pointed out yesterday. Hence, the same transparency and the possibility of complete information that is available to both sides—that we both find very satisfying—is completely absent in these circumstances. The utopian scenario that you describe and that I started with at the beginning of this post is short-lived.

Unfortunately, it gets worse. Not only do parties, who fear that they may be held liable for their wrongful conduct, seek to generate countervailing favorable evidence, at times, they make sure that data that might be unfavorable to them is never even gathered. The July / August 2005 issue of Health Affairs carries a fascinating study conducted by a highly qualified group of researchers, led by Daniel M. Cook, at the University of California at San Francisco. That paper showed how one corporation managed to successfully thwart for more than 10 years the collection of data regarding the deleterious effects of secondhand smoke.

In short, Bill Anderson and you are right. It is not smart to wait until you get sued to conduct any scientific tests. You need to be ready for the day when a complaint may be filed against you armed with scientific data favorable to you, and, if possible, even prevent the collection of data unfavorable to your interests. How do we apply Daubert and its underlying concerns regarding junk science in these circumstances? What can we do then to ensure that the evidence being proferred amounts to scientifc knowledge? Any thoughts?

This Week's Entries: Monday | Tuesday | Wednesday | Thursday

Martinez: 4/5/06, 02:28 PM
Well, Jay, my college debate professor was correct. When your opponent starts by saying "Mike you are right," watch the argument you just made get turned on its head. And indeed that is what you have done. My hypothetical yesterday was designed to demonstrate how a defendant corporation faces many risks when deciding whether or not to conduct litigation-related tests. It was not intended to be a launching pad for assaults on big corporations and how they sometimes act badly.

But since you have pointed us to a study about a tobacco company hiding study results from the public, let's take a look at that. The study you cite proves my point. As you note, the study examines how information about harmful second-hand smoke was hidden from the public by one corporation. But that information, via litigation and other means, is now in the public domain and that corporation is paying a heavy price to plaintiffs, state attorneys general, and the public at large for its duplicity. The reputation of the company is harmed, perhaps irreparably. As I noted the first day, corporate defendants have incentives to fund studies that lead to a correct result. Conversely, as the study you cite proves, there is a huge disincentive for corporations to fund litigation-driven junk science or worse, to hide study results that harm them.

I could just as easily have used a hypothetical plaintiff in yesterday's posting using the following situation. An issue in a plaintiff's claim is how much of a substance got on a person's skin, got into the body through the skin, and once in the body caused harmful effects. The plaintiff decides to commission a dermal absorption study. When the study does not yield the desired result, plaintiff shelves the study and does not even show it to the expert hired to assist with the case. Why? Because such a case is usually a "one-shot deal" for the plaintiff, and that, coupled with a contingency fee motivation, does not create incentives for correct scientific studies. Rather, the incentive for plaintiffs is usually an outcome-oriented study.

Having said all that, I think we do have some common ground. We both agree that some parties on both sides of the "v." try to get a leg up in litigation by funding studies that they hope will help their side. My view, again, is that there is nothing wrong with that given the safeguard criteria and gatekeeper functions built into the Daubert process. For more on this topic I would commend you and the readers to a recent article written by Professor William Childs at Western New England School of Law. Professor Childs' ultimate conclusion is that the intersection of law and science on these highly technical issues strengthens both fields and is not something that should be avoided. It is an interesting read, and I see that Legal Affairs has published a summary of it.

Kesan: 4/5/06, 09:32 PM
Hi, Mike, I appreciate your points and they are well-taken. In tort litigation, plaintiffs and/or their attorneys may conduct studies to prove their case, and they are certainly looking to bolster their case. But, as you pointed out, tests conducted during litigation can be scrutinized thoroughly if relied on by a party to make their case. We have good discovery rules that are helpful. However, plaintiffs (or their attorneys) do not know the subject matter of litigation they may be involved in prior to taking a case—they do not know where their next case is going to come from. Hence, they are unable and they have no incentive to, shall we say, "cook the lab notebooks" in anticipation of litigation. On the other hand, defendants in tort litigation know the business they are in and their liability exposure. They are in a good position to pre-empt possible future litigation with favorable data.

I hope I have made my point clearly. I am not "dead-set against such studies in the context of litigation." And I do not want to use your hypothetical as " a launching pad for assaults on big corporations and how they sometimes act badly." Readers familiar with my writings know that this ad hominem charge does not stick. It is the integrity of the scientific process that is at stake here, and as a scientist who spent many years in research and development in the world of information technology, I am justly concerned about this.

That said, I am in agreement with you that the parties in a litigation are there to win, not to adhere to good scientific norms. Hence, there is a need to craft additional rules to reduce opportunistic conduct by both sides with full disclosure rules and complete transparency. For instance, I would propose that there should be full disclosure of any direct or indirect involvement by any party in any study that is being relied on by a party in a litigation. Such disclosure should not be restricted to financial support and should include disclosure of other support for a study in the form of materials, supplies, performance of testing, and the like. These rules should be accompanied by significant judicial sanctions. If a party's involvement in a study is not disclosed in a timely fashion and is discovered later, the studies at issue should be excluded by the court and perhaps costs to the other side related to analyzing and responding to that study should be awarded.

Your example about the dermal absorption study brings up several other issues that are ignored by Daubert. Daubert operates at a conceptual level for the most part, and it is not sufficiently operationalized by the circuit courts or district courts. I see four parts to analyzing scientific evidence that is presented in any case and Daubert addressed only the first two. We have: (a) the scientific technique that is being relied on, e.g., a testing technique to measure dermal absorption; (b) the applicability of the scientific technique to the facts of the case (the so-called "fit" requirement); (c) the execution of the scientific technique in the case at hand, and the fidelity of the particular execution of the tests in the case to the particular scientific technique; and (d) the scope of the conclusions that are being presented in court, and whether they are commensurate with the technique being employed.

For example, can a given technique accurately measure dermal absorption of a substance at the levels being presented to the court? All four points need to be analyzed carefully in order to determine whether the evidence being presented amounts to "scientific, technical or specialized knowledge" under Rule 702. This is the central analytical concern of Daubert. There are numerous cases in the past few years that expose the lack of attention to items (c) and (d) by trial courts within the context of a Daubert hearing. Some of these errors are corrected on appeal, and there are others where they are not. Again, my focus is on ensuring that all parties live up to Daubert's ideals, and this task is far from trivial in an adversarial setting.

I am pleased to see your reference to Prof. Bill Childs' work, and I enjoyed reading it when it was posted on SSRN. It is insightful at a theoretical level, and there is much to commend about his proposals. Based on this post, it is easy to see that I am in agreement with his rules seeking greater disclosure about funding sources in scientific journals and in the courtroom. I also embrace his suggestion about greater reliance on court-appointed experts, subject, of course, to appropriate cross-examination.

I am sorry that this has turned out to be a rather long response. I very much look forward to your post.

This Week's Entries: Monday | Tuesday | Wednesday | Thursday

Martinez: 4/6/06, 10:25 AM
Good afternoon, Jay. Your latest posting stakes out a good bit of common ground. I never said Daubert and the process it has created is perfect. In fact, I think I indicated the opposite on Tuesday. So I agree with you there probably are ways to improve the process, and I have no problem with some of your suggestions. For example, your proposal about fuller disclosure of who is funding or contributing to a study is a good one. In fact it is akin to and consistent with some court disclosure rules. For example, the Supreme Court of the United States requires in its Rule 37(6) that parties filing amicus briefs with the Court disclose in the brief who wrote it in whole or in part, as well as the identity of any person or entity who made a monetary contribution to the brief's preparation or submission. Obviously the motivation behind the Supreme Court's rule is to ensure full disclosure of what entities or persons are pushing one argument over another. So yes, I think there can be useful improvements made to the process.

But I am not entirely clear what point you are making with respect to your four-part analysis of Daubert in the context of the dermal absorption example I gave yesterday. To recap, you thought that Daubert does not adequately address two issues: (1) the execution of the scientific technique as well as the fidelity of the execution; and (2) the scope of the conclusions presented to the court and their relation to the technique being employed. Yet you note that to the extent trial courts are not devoting adequate attention to these points, some cases nevertheless get corrected on appeal.

I agree with you that both of these points should be analyzed under Daubert. But I also think the Daubert analysis is defined in a sufficiently broad and flexible way that they are covered. For example, I think the question of the execution of a test and whether it is being done with fidelity to the science at issue is a question that seems squarely within the Daubert relevance and reliability inquiries. If a test is not done with fidelity to proper science and execution requirements the results would likely be flawed and thus not relevant or reliable. A vigorous inquiry in discovery and via Daubert filings and a hearing should resolve such issues. Similarly, the scope of the conclusions issue seems to me to be well within the bounds of a "fit" inquiry, and perhaps relevance as well. So I go back to my original point on Monday—I believe Daubert as currently operating, adequately deals with these issues, even if there are some modest improvements that might be made. Jay, it has been a pleasure debating with you.

Kesan: 4/6/06, 03:30 PM
Mike, I agree there is some common ground in our positions with respect to some aspects of Daubert. Nevertheless, there are some issues from my previous post that require some expatiating to better clarify my areas of concern.

I am glad that you can see the merit in having disclosure rules that require all parties to disclosure direct and/or indirect interest in any scientific study together with an appropriate sanctions regime for violations. Two additional points are relevant. At this point, any disclosure rules are not part of the standard Daubert factors being employed by the courts, in the same category as whether a study is peer-reviewed, the error rate of the technique at issue, and so on. Hence, there is no common law overlay through judicial decisions that requires such an inquiry as part of the routine Daubert gatekeeping function. Second, disclosure of any monetary interest by amici in Supreme Court briefs is not an entirely appropriate analogy. In the case of scientific studies, biases can be very hard to determine (e.g., the real power in a scientific study is not what data you choose to keep, but rather what data you choose to ignore), and it not simply a matter of knowing where someone is coming from, as in an amicus brief. Only through careful policing of disclosure rules over a period of time, can we create a culture of complete transparency under Daubert.

As for the four-part analysis of Daubert, let me clarify why the four questions are distinct. The first two parts—the choice of scientific technique and the issue of the "fit" between the underlying technique and the issue in the case—are not specific to the particular case at hand. Rather, they are issues that are common to that category of cases. For example, using DNA profiling is a appropriate scientific technique and using this technique to identify a person uniquely is a question of fit since I may be using DNA profiling to uniquely identify or reject a person. Note how these questions are relevant in all cases using DNA profiling for identifying individuals.

The third and fourth parts of my Daubert analysis are unique to the particular facts of the case at hand. The third part goes to the execution of the methodology in a particular case: How did the DNA samples get taken, how were they preserved, how were they tested, and so on. The fourth part goes to whether the conclusions that are drawn from the DNA profiling are commensurate with the execution of the methodology. For example, did the manner in which the testing was conducted in this case permit one to uniquely identify a person at say the probability of 1 in 10 billion, 100 million or 10 million. Notice, that the third and fourth parts of my the analysis go beyond the "fit" inquiry in the second part and any relevance inquiry under Rule 401. This entire four-part inquiry is essential if we are to fully vindicate the goals of Daubert.

As currently constituted, the Rule 702 inquiry and the judicial overlay by Daubert focus only on the first and second parts and do not contain the case-specific analysis. At times, I have noticed a few cases where appellate courts appreciate issues related to the execution of the methodology in a case or point out where overbroad conclusions that are not supportable by the underlying methodology, but again, these exceptions prove the point that the current Daubert analytical framework is poorly specified.

There are other significant complications. The nature of the scientific evidence being presented in different causes of action are dramatically different. The scientific evidence in a toxic tort case is different from a medical malpractice case or a patent infringement case. All of them have to be analyzed under Daubert, but judicial clarification in each of these substantive areas is very limited.

I believe there is a real need for more judicial guidance and/or legislative enactment in this arena to make substantial progress and vigilantly weed out junk science from the courtroom. Mike, I want to thank you for the patience, courtesy, and intellectual commitment that you have extended to me.

This Week's Entries: Monday | Tuesday | Wednesday | Thursday

printer friendly email this article letter to the editor
space space space

Contact Us