Measures, models, and methods for macroevaluation of impact of information systems within their environment and for microevaluation of performance of systems components. Experimental methods and experiments in information retrieval, information seeking, and library processes.

The goals are to provide students with an understanding of critical issues, problems, and accomplishments in evaluation and experimentation in information science and librarianship, in order to be able to undertake own research in this area.

While there are no courses required as direct prerequisites for this seminar, the students are expected to have a basic knowledge related to:

The course consists of:

1. Required readings and summaries: for each topic there will be several required readings. For each reading the student will prepare a short summary and critical review. The summaries will be submitted according to a schedule provided at the beginning of the semester.

The goal of the summaries is not only for a student to reinforce learning the content of the reading by writing, but even more so to critically evaluate and/or relate the content (or part thereof) to own context, experiences, and other readings and learning. Summaries are not intended to be mere abstracts. Thus, the emphasis is not on the summary by and for itself but on critical evaluation and/or drawing of relations. Higher grades in summaries relate to the extent to which contents are critically evaluated or to which relations are drawn, and not to mere repetitions of contents.

2. Semester project: each student will identify a research problem related to one or more topics covered in the course and either

• Conduct an experimental or observational study, which means complete design, and pilot data collection, analysis, and presentation, or
• Undertake an exhaustive, critical literature review on that topic.

The narrative style of the summary is left to the student. However, each summary MUST have as a heading (i.e. on top or the cover page):

1. student’s name,

2. course number,

3. assignment number from the schedule, and

Summaries should be handed in per assigned schedule. Summaries that do not have headings and citation form as prescribed will have five points deducted.

The summaries and term paper should follow the standardized format as suggested by:

It is strongly suggested that all students purchase this book. The Manual can be obtained from Rutgers and other bookstores. It serves as a manual for other Ph.D. courses and the dissertation, thus it is a valuable tool above and beyond this course. In addition to providing standards for formats, references and citations, the Manual also provides useful suggestions for writing and organizing of reports and articles.

OUTLINE OF TOPICS

1. Different approaches to evaluation. Essential elements in the systems approach and in experimentation. Levels of evaluation.
2. Experimental method. Application to information systems and libraries. What to measure in an experiment? How to measure?
3. Criteria for evaluation. Measures based on criteria, for different components selected for evaluation. Metrics for characterization of services.
4. Measuring instruments for evaluation and experimentation. Construction of surveys. Measurement scales. Data as base for metrics.
5. Information seeking studies; user studies in various environments.
6. Value and impact studies related to users, organizations, and environment.
7. TREC studies in information retrieval.
8. Evaluation and experimentation in digital libraries. Electronic services, databases, and networks.
9. Evaluation of human-human and human-computer interaction. Searching, reference, mediation. Online Public Access Catalogs (OPACs).
10. Economic studies and assessments. Measuring costs and cost benefits.

Allen, B. (1994). Cognitive abilities and information system usability. Information Processing & Management, 30 (2) 177-191.

Auster, E. and Choo, C.W. (1993). Environmental scanning by CEOs in two Canadian industries. Journal of the American Society for Information Science, 44 (4) 194-203,

Borgman, C. and Rice, R.E. (1992). The convergence of information science and communication: a bibliometric analysis. Jounal of the American Society for Information Science, 43 (6) 397-411.

Borgman, Christine. (1996). Why Are Online Catalogs Still Hard to Use? Journal of the American Society for Information Science, 47 (7), 493-503.

Boyce, B. B., Meadow, C. D., & Kraft, D. H. (1994). Measurement in Information Science. San Diago CA: Academic Press

Brinkley, M (1999) The EQUINOX Project: Library Performance Measurement and Quality

Chatman, E.A. (1991). Life in a small world: Applicability of gratification theory to information-seeking behavior. Journal of the American Society for Information Science, 42 (6) 438-449.

Chatman, E.A. (1996). The impoverished life-world of outsiders. Journal of the American Society for Information Science, 47 (3) 193-206.

Chen, H., Danowitz, A.K., Lynch, K.J., Goodman, S.E., and McHenry, W.K. (1994). Explaining and alleviating information management indeterminism: a knowledge-based framework. Information Processing & Management, 30 (4) 55-577.

Clarke, Z. (1999). Performance indicators for the electronic library: the EQUINOX Project. Multimedia Information & Technology, 25 235-237.

Cobbledick, S. (1996). The information-seeking behavior of artists: Exploratory interviews. Library Quarterly, 66 (4) 343-372.

Connell, T.H. (1995). Subject searching in online catalogs: Metaknowledge used by experienced searchers. Journal of the American Society for Information Science, 46 (7) 506-518.

Dow, R. F. (1999). Using assessment criteria to determine library quality. Journal of Academic Librarianship, 24 (4) 277-281.

Drabenstott, Karen and Weller, Marjorie. (1996). Failure Analysis of Subject Searches in a Test of a New Design for Subject Access to Online Catalogs. Journal of the American Society for Information Science ,47 (7), 519-537.

Egan, D.E., Remde, J.R., Gomez, L.M., Landauer, T.K., Eberhardt, J., & Lochbaum, C.C. (1989). Formative design-evaluation of SuperBook. ACM Transactions on Information Systems, 7 (1) 30-57.

Ellis, D., Furner, J., and Willett, P. (1996), On the creation of hypertext links in full-text documents: Measurement of retrieval effectiveness. Journal of the American Society for Information Science, 47 (4) 287-300.

Fidel, R. (1991). Searchers’ selection of search keys. Parts I&II. Journal of the American Society for Information Science, 42 (7) 490-514.

Forgionne, G.A., and Kohli, R. (1995). Integrated MSS effects: an empirical health care investigation. Information Processing & Management, 31 (6) 879-896.

Franz, C.R., and Jin, K.J. (1995). The structure of group conflict in a collaborative work group during information systems development. Journal of Applied Communication Research, 23 108-127.

Gordon, M. D.; Pathak, P. (1999). Finding information on the World Wide Web: the retrieval effectiveness of search engines. Information Processing & Management, 35 (2)141-180.

Harman, Donna. (1995). Overview of the Second Text Retrieval (TREC-2). Information Processing and Management, 31, (3), 271-289.

Harman, Donna. (1995). The Second Text Retrieval Conference (TREC-2). Information Processing and Management, 31, (3), 269-270.

Harter, S.P. (1996). Variations in relevance assessments and the measurement of retrieval effectiveness. Journal of the American Society for Information Science, 47 (1) 37-49.

Hayes, R.M. (1993). Measurement of information. Information Processing & Management, 29 (1) 1-11.

Heath, S.B. Questioning at home and at school: a comparative study. From: Spindler, G. (ed.). Doing the Ethnography of Schooling: Educational Anthropology in Action. Prospect Heights, Illinois: Waveland Press, Inc., pp 105-127.

Hersh, W.R., MD, and Hickam, D.H., MD, MPH. (1998). How well do physicians us electronic information retrieval systems? A framework for investigation and systematic review. Journal of the American Medical Association, 280 (15) 1347-1352.

Hert, C.A. (1996). User Goals on an online public access catalog. Journal of the American Society for Information Science, 47 (7) 504-518.

Hert, Carol. (1996). User Goals on an Online Public Access Catalog. Journal of the American Society for Information Science, 47 (7), 504-518.

Kao, C.; Lin, Y.-C. (1999). Comparing university libraries of different university size. LIbri, 49 , 150-158.

Katzer, J., Cook, K.H., & Crouch, W.W. (1965). Evaluating information: a guide for users of social science research. Chapters 16 and 17. pp.197-207. New York:McGraw Hill.

Klobas, J.E., and McGill, T. (1995). Identification of technological gatekeepers in the information technology profession. Journal of the American Society for Information Science, 46 (8) 581-589.

Krantz, D.H., Luce, R.D., Suppes, P. & Tversky, A. (1971) Foundations of measuremet (vol. 1)San Diego CA: Academic Press.

Kuhlthau, Carol. (1991). Inside the Search Process: Information Seeking.from the User’s Perspective. Journal of the American Society for Information Science, 42 (6), 361-371.

Kuhlthau, Carol. (1993). Seeking Meaning:A Process Approach to Library and Information Services. Ablex,

Lancaster, F.W. (1993). If You Want to Evaluate Your Library... Univ. of Illinois, Grad. School of Library and Information Science. 352 pp.

Losee, R. M.; Paris, L. A. H.(1999). Measuring search-engine quality and query difficulty ranking with Target and Freestyle. Journal of the American Society for Information Science, 50 (10) 882-889.

Markey, K. (1981). Levels of question formulation in negotiation of information need during the online research interview: a proposed model. Information Processing & Management, 17 (5) 215-225.

Marshall, J.G. (1992). The Impact of Hospital Libraries on Clerical Decision-Making:The Rochester Study. Bulletin of the Medical Library Association, 80, (2), 169-78.

McCain, K.W., White, H.D., and Griffith, B.C. (1987). Comparing retrieval performance in online data bases. Information Processing & Management, 23 (5) 539-553.

Nahl, D., and Tenopir, C. (1996). Affective and cognitive searching behavior or novice end-users of a full-text database. Journal of the American Society for Information Science, 47 (4) 276-286.

Park, Hongseok. (1997). Relevance of Science Information: Origins and Dimensions of Relevance and their Implications to Information Retrieval. Information Processing and Management, 33, (3), 339-352. (Dissertation).

Robertson, S.E., and Hancock-Beaulieu, M.M. (1992). On the evaluation of IR systems. Information Processing & Managemet, 28, (4) 457-466.

Salasin, J, and Cedar, T. (1985). Information-seeking behavior in an applied research/service setting. Journal of the American Society for Information Science, 36 (2) 94-102,

Salton, G., Allan, J., & Buckley, C. (1994). Automatic structuring and retrieval of large text files. Communications of the ACM, 37 (2) 97-108.

Saracevic, T. & Kantor, P. (1997). Studying the value of library and information services. I. Establishing a theoretical framework. II. Methodology and Taxonomy. Journal of the American Society for Information Science, 48 (6), 527-542, 543-563.

Saracevic, T. Relevance, A. (1975). Review of and a Framework for the Thinking on the Notion in Information Science. Journal of the American Society for Information Science, 29 321-343.

Saracevic, T., and Kantor, P. (1988). A study of information seeking and retrieving. Parts I-III. Journal of the American Society for Information Science, 39 (3) 161-216.

Saracevic, T., Kantor. P., Chamis, A. Y., & Trivison, D. (1988). A study of information seeking and retrieving. I. Background and methodology. II. Users, questions and effectiveness. III. Searchers, searches and overlap. Journal of the American Society for Information Science, 39 (3), 161-176, 177-196, 197-216.

Shute, S.J., and Smith, P.J. (1993). Knowlegde-based search tactics. Information Processing & Management, 29 (1) 29-45.

Sonnenwald, D.H. (1995). Contested collaboration: a descriptive model of intergroup communication in information system design. Information Processing & Management, 31 (6) 849-877.

Sparck-Jones, Karen. (1995), Reflections on TREC. Information Processing and Management, 31, (3), 291-314.

Spink, A. & Saracevic, T. (1997). Interactive information retrieval: Sources and effectiveness of search terms during mediated online searching. Journal of the American Society for Information Science, 48, (8), 741-761.

Su, L.T. (1994). The relevance of recall and precision in user evaluation. Journal of the American Society for Information Science, 45 (3) 207-217.

Tague-Sutcliffe, J. (1995). Measuring information: an information services perspective. Chapters 1 & 2, pp 1-56. San Diego: Academic Press.

Tague-Sutcliffe, Jean. (1992). The Pragmatics of Information Retrieval Experimentation, Revisited. Information Processing and Management, 28, (4), 467-490.

Van House, Nancy. (1990). Output Measures and the Evaluation Process. In Betty J. Turock, Editor, Evaluating Federally Funded Library Programs, U.S. Department of Education, 5-13.

Vorhees, E.M. (2000). Special issue: The sixth Text REtrieval Conference (TREC). Information Processing & Management, 36 (1) 1-204. Especially see articles by Vorhees, E. M. & Harman, D. Overview of the sixth Text Retrieval Conference (TREC-6), 1- 35), and Sparck Jones, K. Further reflections on TREC, 37-85.

Yuan, W. (1997). End-user searching behavior in information retrieval: a longitudinal study. Journal of the American Society for Information Science, 48 (3) 218-234.

Yuan, W.; Meadow, C. T. (1999). A study of the use of variables in information retrieval user studies. Journal of the American Society for Information Science 50 (2)140-150

Weeks 1 & 2—Choose own readings. What is science? What is experimentation in science?

Week 3:

Pettigrew, K. and McKechnie, L. (2001). "The use of theory in information science research." Journal of the American Society for Information Science and Technology 52:11. p. 62-73.

Saracevic, T.; Kantor, P.; Chamis, A.; and Trivison, D. (1988). "A Study of Information Seeking and Retrieving: I. Background and Methodology; II. Users, Questions, and Effectiveness; III. Searchers, Searches, and Overlap. Journal of the American Society for Information Science. 39:3, pp. 161-216.

Su, L. (1994). "The Relevance of Recall and Precision in User Evaluation." Journal of the American Society for Information Science 44:3, pp. 207-217.

Week 4:

Blair, D. (1996). "Stairs Redux: Thoughts on the STAIRS Evaluation, Ten Years after." Journal of the American Society for Information Science. 47 (1): 4-22.

Blair, D. and Maron, M. (1985). "An Evaluation of Retrieval Effectiveness for a Full-Text Document-Retrieval System." Communications of the ACM. 28 (3): 289-300.

Harter, S. and Chen, Y-R. (1996). "Colinked Descriptors: Improving Vocabulary Selection for End-User Searching". Journal of the American Society for Information Science 47 (4): 311-325.

Salton, G. (1986). "Another Look at Automatic Text-Retrieval Systems." Communications of the ACM. 29 (7): 648-656.

Salton, G.: Allan, J. & Buckley, C. (1994). "Automatic Structuring and Retrieval of Large Text Files." Communications of the ACM 37 (2): 97-108.

Week 5:

Belkin, N. (1990). "The Cognitive Viewpoint in Information Science." Journal of Information Science. 16: 11-25.

Connell, T. (1995). "Subject Searching in Online Catalogs: Metaknowledge Used by Experienced Searchers." Journal of the American Society for Information Science. 46 (7): 506-518.

Fidel, R. (1991). "Searchers’ Selection of Search Keys: I. The Selection Routine."

Journal of the American Society for Information Science. 42(7):490-500.

Shute, S. and Smith, P. (1993). "Knowledge-Based Search Tactics." Information Processing and Management. 29 (1): 29-45.

Spink, A. and Saracevic, T. (1997). "Interaction in Information Retrieval: Selection and Effectiveness of Search Terms." Journal of the American Society for Information Science. 48 (8(: 741-761.

Week 6:

Belkin, N.J.; Cool, C.; Kelly, D.; Lin, S-J; Park, S.Y.; Perez-Carballo, J.; & Sikora, C. (2001). "Iterative exploration, design and evaluation of support for query reformulation in interactive information retrieval." Information Processing & Management 37:3.

Hert, C., Jacob, E.K., Dawson, P. (2000). "A usability assessment of online indexing structure in the networked environment." Journal of the American Society for Information Science 51:11, pp. 971-988.

Week 7:

Allan, J.; Leuski, A.; Swan, R.; & Byrd, D. (2001). "Evaluating combinations of ranked lists and visualizations of inter-document similarity." Information Processing & Management 37:3.

Bodner, R.; Chignell, M.; Charoenkitkarn, N.; Golovchinsky, G.; & Kopak, R. (2001). "The impact of text browsing of text retrieval performance." Information Processing & Management 37:3.

Hersh, W. & Over, P. (2001). "Editorial: Interactivity at the Text Retrieval Conference." Information Processing & Management 37:3.

Hersh, W.; Turpin, A.; Price, S.; Kraemer, D.; Olson, D.; Chan, B.; & Sacherek, L. (2001). "Challenging conventional assumptions of automated information retrieval with real users: Boolean searching and batch retrieval evaluation." Information Processing & Management 37:3.

Larson, R. (2001). "TREC interactive with Cheshire II." Information Processing & Management 37:3.

Over, P. (2001). "The TREC interactive track: An annotated bibliography." Information Processing & Management 37:3.

Wu, M.; Fuller, M. & Wilkinson, R. (2001). "Using clustering and classification approaches in interactive retrieval." Information Processing & Management 37:3.

Yang, K.; Maglaughlin, K.; & Newby, G. (2001). " Passage feedback with IRIS." Information Processing & Management 37:3.

Week 8:

Carter, D. & Janes, J. (2000). Unobtrusive data analysis of digital reference questions and service at the internet public library: An exploratory study. Library Trends 49(2), 251-265.

Egan, D. et al. (1989). Formative design-evaluation of SuperBook. ACM 7(1), 30-57.

Hert, C. (1996). User goals on an online public access catalog. Journal of the American Society for Information Science 47(7), 504-518.

Marchionini, G. (2000). Evaluating digital libraries: A longitudinal and multifaceted view. Library Trends 49(2), 304-333.

Week 9:

Auster, E. & Choo, C. (1993). "Environmental scanning by CEO’s in Two Canadian Industries." Journal of the American Society for Information Science 44(4), 194-203.

Chen, A.; Danowitz, A.; Lynch, A.; & Goodman, S. (1994). "Explaining and alleviating information management indeterminism: A knowledge-based framework." Information, Processing & Management 30 (4), 557-577.

Forgionne, G. & Kohli, R. (1995). "Integrated MSS effects: An empirical health care investigation." Information, Processing & Management 31 (6), 879-896.

Kling, Rob. (1999). What is social informatics and why does it matter? D-Lib Magazine 5 (1).

Kulthau, C. (1999). "The role of experience in the information search process of an early career information worker: Perceptions of uncertainty, complexity, construction, and sources. Journal of the American Society for Information Science 50 (5), 399-412.

Sonnenwald, D. (1995). "Contested collaboration: A descriptive model of intergroup communication in information system design." Information, Processing & Management 31 (6), 859-877.

Week 10:

Chatman, E. (1991). "Life in a small world: Applicability of gratification theory to information seeking behavior." Journal of the American Society for Information Science 42(6), 438-449.

Chatman, E. (1996). "The impoverished life-world of outsiders." Journal of the American Society for Information Science 47(3), 193-206.

Heath, S. (NA). "Questioning at home and at school: A comparative study." (Ed. by G. Spindler). In Doing Ethnography of Schooling. Prospect Heights, IL: Waveland Press, Inc., 103-127.

Klobas, J. & McGill, T. (1995). "Identification of technological gatekeepers in the information technology profession. Journal of the American Society for Information Science 46(8), 581-589.