Discovery systems – testing known item searching

Many libraries have responded to user demand for Google-style searching by implementing web scale discovery services such as EDS, Primo, Summon, and WorldCat Local.   By exporting their local catalogs to the meta-aggregate article indexes provided by these vendors, libraries create “single search” across articles and books.

One frustration with this approach has been that local collections get “lost” in the overwhelming amount of subscription article content.  For this post I wanted to measure  discoverability of local collections in the four major systems.  I conducted a test of known item print book searches in eight different implementations.

The test

I chose eight libraries with large local collections that had implemented one of the four major discovery services.  All eight combined local and subscription content in a single result list. For comparison I also tested Google.

I searched using the keyword default search box and did not facet or sort.  In each system I ran the following types of searches:

  • Single word titles (e.g. 1984)
  • Titles with “stop words” (e.g To have and have not)
  • Title/author keyword (e.g. Smith and On beauty)
  • Book citation (copied from bibliographies)
  • ISBN  

All of my searches were for print items held by the libraries.

The results

Search comparison results graph

Results by type of search
  • WorldCat Local won, ranking highest in all search types
  • Google came in a close second
  • Summon scored well in all categories, but did not excel in citation searches
  • WorldCat Local and Google were the only systems to rank highly in citation searches
  • Primo scored low on author/title searches and failed citation searches
  • EDS did the worst: stop words generated errors, and no results were returned for citations or ISBN searches

Title vs. keyword matching

Some systems ranked keyword relevance higher than an exact title match.  This could make single-word titles hard to find. For example, a user looking for the novel “V” by Pynchon might see this first result in EDS:   EDS title result

Conclusion

This test only measured known item searching for print materials.  In that category it is not surprising that WorldCat Local, a discovery system built around OCLC’s union book catalog, excelled.  It is also not surprising that local print content tends to get “lost” in systems focused more on subscription article content.

EDS and Primo were the low performers, but it should be noted that these tools may perform significantly better in different implementations:  I tested only two implementations of each system and did not take into account any customized “boosting” of local content that may be applied by libraries.

This test, while cursory, does suggest that library discovery systems continue to have problems with known item searching.  Our users come to these systems with expectations formed by Google: they  expect to be able to search by author/title keyword, ISBN, and to be able to copy and paste formatted citations.  And we are failing to meet those expectations.

Library sites tested

Smith College/5 Colleges Consortium (EDS)

University of Georgia (EDS)

Oxford University, UK (Primo)

University of Minnesota (Primo)

University of Texas Libraries (Summon)

Duke University (Summon)

University of California Berkeley (WorldCat Local)

University of Washington (WorldCat Local)

9 thoughts on “Discovery systems – testing known item searching

  1. As I was setting up our Primo installation, I purposefully broke things up between “our local collections” (stuff on the shelves, our DSpace instance, LUNA) and “everything” which had all that stuff and added the Primo Central Index. You mention that all these libraries combined everything. Do you have an opinion on how breaking out locally resident stuff might or might not improve things?

    http://gml-primo.hosted.exlibrisgroup.com/primo_library/libweb/action/search.do?menuitem=0

    1. Hi Wally,

      Thanks for the comment! I like the flexibility Primo has to scope collections, and I think for large libraries this can be helpful. For example, Oxford University (one of the sites I looked at) has their default search set to just local collections, and an “everything” tab to include PCI. I think the jury is still out on this, though, and more user testing needs to be done. Do users find those other tabs? Or do they just start typing into the first search box they see?

      Emily

  2. Emily

    Interesting test. I am looking into such issues too, I suspect though based on the actual searches I see in our system – Summon, some of the tests you do are more important than others.

    Eg Title/Author is much more common here then tossing in the whole citation. Though the system itself might shape user behavior, ie people realise tossing in the whole citation in Summon never works, so you don’t see as many such searches.

    The Titles with “stop words” examples of 1984 and V I worry could give misleadingly good results because those are common ones to tune to. Just as everyone tunes to “Nature” pulling up the the journal. Still shocking some WDS fail 1984.

    Based on actual searches users are struggling with I would say try

    1) Journal titles with 1-2 common words

    2) “classic” text + author search (try any famous management/sociology text with multiple editions, classic philosophy text, english lit novels etc)

    3) Partial titles of classic works with or without author – eg

    Still good test, I may try to duplicate your testing methodology.

    1. The whole idea of testing known item searching in different environments with a set of items is worth a lot more discussion. I share many of Aaron’s concerns with this particular set (common titles that are easy to tweak, full citations weighed the same as other, more typical types of searches). Determining user intent (which known item were they actually searching for?) is tricky, as is figuring out which aspect of the catalog and discovery layer is responsible for an undesirable result.

      The University of Minnesota retired Aleph (and the Aleph OPAC) last fall and Primo/Alma + selected Primo Central databases is now our default search interface. We were concerned enough about how retiring the traditional OPAC would impact our users that we did quite a bit of Known Item testing at that time. We summarized the results for staff and interested users here: http://www.youtube.com/watch?v=ois6IH5SnnI

      We’ve run more tests with different data sets since this initial comparison last fall and plan to repeat them when we update to the latest version of Primo and make some tweaks to the normalization rules that translate our MARC records into the PCX markup used in Primo. In the meantime, we’d love to discuss what makes a good test with anyone else doing similar work.

      And thanks, Emily, for starting this discussion and posting your spreadsheet!

      1. Hi Jan,

        Thanks for the great response! And congrats on UMN’s thoughtful and effective implementation of Primo. I agree that there are big challenges around understanding user intent – and my test is certainly not conclusive, due to the concerns you’ve mentioned. Thanks for sharing your process and experience – this is especially helpful as Harvard begins our implementation of discovery. Best, Emily

  3. Thanks for making the effort to perform these concrete examples and sharing your results and reasoning (also in the previous posts).

    I find it amazing that Google manages to get that much right – which has to be paid for or painstakingly developed at every single library. Let’s hope more libraries will make use of it.

Leave a Reply

Your email address will not be published. Required fields are marked *