You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
<td style="text-align: left;">Information experiments are typically used for causal inference, not descriptive inference, whether or not they are delivered through a survey. In some cases surveydelivered information experiments are almost indistinguishable from field experiments — for instance if information is delivered in a way similar to treatments of interest and if outcomes are measured outside of the survey, for instance through measures of subsequent behaviors. The key difficulty with embedding an information experiment in a survey is with respect to external validity—whether the effects of information delivered in this way are similar to effects of information delivered in the wild, and so lots of good work in this vein tries to address that headon.</td>
2463
-
<td style="text-align: left;">Randomized response surveys in which people are randomly assigned to answer either a sensitive question or a non-sensitive question are typically used for descriptive inference. The goal is to estimate the prevalence of some property of subjects, such as whether people have engaged in illegal behavior. The randomization makes it possible to make inferences about the prevalence of the sensitive behavior while protecting individual privacy. Here the randomization is a tool to make measurement possible, not the focus of interest itself. There is a causal effect of the question on the answer, but the purpose is to make descriptive inferences about something else.</td>
2462
+
<td style="text-align: left;">Information experiments are typically used for causal inference, not descriptive inference, whether or not they are delivered through a survey. In some cases survey-delivered information experiments are almost indistinguishable from field experiments — for instance if information is delivered in a way similar to treatments of interest and if outcomes are measured outside of the survey, through measures of subsequent behaviors. The key difficulty with embedding an information experiment in a survey is with respect to external validity—whether the effects of information delivered in this way are similar to effects of information delivered in the wild, and so lots of good work in this vein tries to address that head-on.</td>
2463
+
<td style="text-align: left;">Randomized response surveys, in which people are randomly assigned to answer either a sensitive question or a non-sensitive question are typically used for descriptive inference <span class="citation" data-cites="blair2015design">(<a href="#ref-blair2015design" role="doc-biblioref">Blair, Imai, and Zhou 2015</a>)</span>. The goal is to estimate the prevalence of some property of subjects, such as whether people have engaged in illegal behavior. The randomization makes it possible to make inferences about the prevalence of the sensitive behavior while protecting individual privacy. Here the randomization is a tool to make measurement possible, not the focus of interest itself. There is a causal effect of the procedure on the answer, but the purpose is to make descriptive inferences about something else.</td>
2464
2464
</tr>
2465
2465
</tbody>
2466
2466
</table>
2467
2467
</div>
2468
2468
</figure>
2469
2469
</div>
2470
-
<p>I think these two cases show a sharp difference between the two goals. In other cases however the purpose is not always so clear. The next sections illustrate these differences for common types of survey experiment.</p>
2470
+
<p>I think these two cases show a sharp difference between the two goals. The purpose is not always so clear, however. The next sections illustrate different purposes for common types of survey experiment.</p>
<p><span class="citation" data-cites="de2022improving">De la Cuesta, Egami, and Imai (<a href="#ref-de2022improving" role="doc-biblioref">2022</a>)</span> describe conjoints as “a factorial survey experiment that is designed to measure multidimensional preferences”. Note the emphasis on measurement. I think the remit of conjoints is a little broader though. For example they might also be used to study how people make classifications or understand concepts. But they might also be used when the estimand is causal.</p>
2488
+
<p><span class="citation" data-cites="de2022improving">De la Cuesta, Egami, and Imai (<a href="#ref-de2022improving" role="doc-biblioref">2022</a>)</span> describe conjoints as “a factorial survey experiment that is designed to measure multidimensional preferences”. Note the emphasis on measurement. Arguably, the remit of conjoints for descriptive inference is a little broader. For example they might also be used to study how people make classifications or understand concepts. But, arguably, conjoints might sometimes also be used when the estimand is causal.</p>
<h3 data-number="2.2.1" class="anchored" data-anchor-id="conjoints-for-descriptive-inference."><span class="header-section-number">2.2.1</span> Conjoints for descriptive inference.</h3>
2491
2491
<p>In the many cases in which the goal is to measure preferences, interpretations, or classification rules, conjoint experiments may be best thought of as focused on descriptive inference and using causal inference to make those descriptive inferences.</p>
<h3 data-number="2.2.2" class="anchored" data-anchor-id="conjoints-for-causal-inference"><span class="header-section-number">2.2.2</span> Conjoints for causal inference</h3>
2502
-
<p>Even still, conjoints can be used also when the primary target is a causal estimand. Say you really are interested in whether the presence of a given feature on a list of features makes it more likely that an outcome will be selected from the list.</p>
2502
+
<p>Even still, conjoints can also be used when the primary target is a causal estimand. Say you really are interested in whether the presence of a given feature on a list of features makes it more likely that an outcome will be selected from the list.</p>
2503
2503
<p>You might have an application where people are electing candidates and know nothing about the candidates other than what they get in a flyer. You want to know how features of the flyer affect the choice. Then you are pretty close to the conjoint. You are interested in the effect of the feature on behavior. You have to worry about external validity (is there too much control and all that) but these are common worries for any experiment.</p>
2504
2504
<p>This is the sort of setting discussed in <span class="citation" data-cites="bansak2023using">Bansak et al. (<a href="#ref-bansak2023using" role="doc-biblioref">2023</a>)</span>.</p>
2505
2505
<p>The risk above remains: the effect you are getting is the effect of the attribute on the list, not the average (total) effect of the attribute itself on the outcomes. For example you might find that a powerful candidate does well <em>given</em> different values of corruption (even for different distributions of corruption), but this does not give you the effect of power itself, since, after all, power corrupts.</p>
Blair, Graeme, Kosuke Imai, and Yang-Yang Zhou. 2015. <span>“Design and Analysis of the Randomized Response Technique.”</span> <em>Journal of the American Statistical Association</em> 110 (511): 1304–19.
De la Cuesta, Brandon, Naoki Egami, and Kosuke Imai. 2022. <span>“Improving the External Validity of Conjoint Analysis: The Essential Role of Profile Distribution.”</span> <em>Political Analysis</em> 30 (1): 19–45. <a href="https://doi.org/10.1017/pan.2020.40">https://doi.org/10.1017/pan.2020.40</a>.
| Information experiments are typically used for causal inference, not descriptive inference, whether or not they are delivered through a survey. In some cases survey delivered information experiments are almost indistinguishable from field experiments --- for instance if information is delivered in a way similar to treatments of interest and if outcomes are measured outside of the survey, for instance through measures of subsequent behaviors. The key difficulty with embedding an information experiment in a survey is with respect to external validity---whether the effects of information delivered in this way are similar to effects of information delivered in the wild, and so lots of good work in this vein tries to address that head on. | Randomized response surveys in which people are randomly assigned to answer either a sensitive question or a non-sensitive question are typically used for descriptive inference. The goal is to estimate the prevalence of some property of subjects, such as whether people have engaged in illegal behavior. The randomization makes it possible to make inferences about the prevalence of the sensitive behavior while protecting individual privacy. Here the randomization is a tool to make measurement possible, not the focus of interest itself. There is a causal effect of the question on the answer, but the purpose is to make descriptive inferences about something else. |
76
+
| Information experiments are typically used for causal inference, not descriptive inference, whether or not they are delivered through a survey. In some cases survey-delivered information experiments are almost indistinguishable from field experiments --- for instance if information is delivered in a way similar to treatments of interest and if outcomes are measured outside of the survey, through measures of subsequent behaviors. The key difficulty with embedding an information experiment in a survey is with respect to external validity---whether the effects of information delivered in this way are similar to effects of information delivered in the wild, and so lots of good work in this vein tries to address that head-on. | Randomized response surveys, in which people are randomly assigned to answer either a sensitive question or a non-sensitive question are typically used for descriptive inference [@blair2015design]. The goal is to estimate the prevalence of some property of subjects, such as whether people have engaged in illegal behavior. The randomization makes it possible to make inferences about the prevalence of the sensitive behavior while protecting individual privacy. Here the randomization is a tool to make measurement possible, not the focus of interest itself. There is a causal effect of the procedure on the answer, but the purpose is to make descriptive inferences about something else. |
77
77
78
78
: Ideal types, experiment types used clearly for causal inference and for descriptive inference {#tbl-ideals}
79
79
80
80
81
-
I think these two cases show a sharp difference between the two goals. In other cases however the purpose is not always so clear. The next sections illustrate these differences for common types of survey experiment.
81
+
I think these two cases show a sharp difference between the two goals. The purpose is not always so clear, however. The next sections illustrate different purposes for common types of survey experiment.
82
82
83
83
84
84
## Priming experiments
@@ -103,7 +103,7 @@ That makes lots of sense in principle. In practice I think sometimes we see peop
103
103
## Conjoints
104
104
105
105
@de2022improving describe conjoints as "a factorial survey experiment that is designed to measure multidimensional preferences".
106
-
Note the emphasis on measurement. I think the remit of conjoints is a little broader though. For example they might also be used to study how people make classifications or understand concepts. But they might also be used when the estimand is causal.
106
+
Note the emphasis on measurement. Arguably, the remit of conjoints for descriptive inference is a little broader. For example they might also be used to study how people make classifications or understand concepts. But, arguably, conjoints might sometimes also be used when the estimand is causal.
107
107
108
108
### Conjoints for descriptive inference.
109
109
@@ -121,7 +121,7 @@ For another example, in @hartmann2024trading, we use a conjoint to measure polic
121
121
122
122
### Conjoints for causal inference
123
123
124
-
Even still, conjoints can be used also when the primary target is a causal estimand. Say you really are interested in whether the presence of a given feature on a list of features makes it more likely that an outcome will be selected from the list.
124
+
Even still, conjoints can also be used when the primary target is a causal estimand. Say you really are interested in whether the presence of a given feature on a list of features makes it more likely that an outcome will be selected from the list.
125
125
126
126
You might have an application where people are electing candidates and know nothing about the candidates other than what they get in a flyer. You want to know how features of the flyer affect the choice. Then you are pretty close to the conjoint. You are interested in the effect of the feature on behavior. You have to worry about external validity (is there too much control and all that) but these are common worries for any experiment.
0 commit comments