Dr. Ed Chi gave a very interesting and inspiral talk in the HCII 2009 at San Diego, titled as "Living laboratories: Rethinking Ecological Designs and Experimentation in Human Computer Interaction".
I knew Dr. Chi first from his paper of the data state reference model taxonomy for infovis, which is his thesis study. This taxonomy has inspired many research and formed guidelines for design infovis tool, such as prefuse toolkit.
In Dr. Chi's presentation, he post a table, which contains four different HCI laboraty scenarios, depending on 1) whether the system under control, and 2) where the evaluation is located, in lab or in the wild. See the figure 1.
Figure 1 The four evaluation scenarios in HCI testing
