Do online reviews *really* matter?

It is some time since I started being interesting in evaluating and in improving the “quality” of information that review and rating sites make available to their users. The word “quality” may be an indefinite buzz word, but behind it I imagine some less abstract concept like “usefulness” and “persuasiveness”, and some preliminary research questions as: “how much impact has that information on users when they take a decision?” or “how can I help the user in exploiting at the best the online reviews data?”. Thousands of reviews (despite ordered by some importance criterion) still force the user to a peer reading, and it may not be uncommon that users read what they already expect to read.

A part from my thoughts, I have recently stepped into a paper that brings light to the topic. “Do online reviews matter? An empirical investigation of panel data” by W. Duan, B. Gu and A. B. Whinston.

It is an interesting study that examines the persuasive effect and the awareness effect of online reviews on movie’s box office revenues. Persuasiveness is the quality of making someone believes to do something (by giving him good reasons to do it), in this case, in buying a movie ticket. Awareness is the quality of being informed of something, in this case, of the existence of a movie. The study’s outcome is, in the very synthesis, surprising. On line reviews have no effect on box office revenues; their pervasive efficacy is almost zero. On the contrary the volume of online posting, which makes more users be aware of the existence of a movie, clearly has an impact on box office revenues. This result has also another interesting consequence; the word-of-mouth (which understands the production of online reviews) seems to have more importance than the quality of the reviews themselves in moving users’ decision to purchase a ticket.

It is difficult, for me, to understand whether the efficacy of the word-of-mouth versus the quality of the information is something inherent to the human way of coping with information or it is not. But and limited to the movie context, the paper shows that the information brought by online reviews is currently not useful as we would like. I see here very promising opportunities for future work, and we researcher should try to investigate further into this direction.

Gabriele Lenzini

Tags: , ,

4 Responses to “Do online reviews *really* matter?”

  1. Gabriele, very interesting post! It reminds me of the problem of how to best evaluate the effectiveness of recommender systems. Researchers often resort to predictive accuracy. At RecSys 2007 (ACM conference on collaborative filtering), industrial practitioners pointed out that predictive accuracy does not often reflect user experience. Recent findings support that view and researchers are consequently coming round to the idea of proposing new metrics that actually reflect user experience. One of those metrics may well persuasion – the degree to which a recommender system is able to change its users’ behavior, right? ;-) Daniele.

  2. G. Lenzini says:

    Thanks Daniele for your comment.

    Yes I think that “persuasion” could be a good metric. On the very intuitive level, it can be measured by considering the increment of business (and its derivatives, i.e. the “inclination” and the “acceleration” of that curve) in response to the posting of a piece of information. Perhaps something related to the measures used to evaluate the impact (and the echos) of news in the media?

  3. Neal Lathia says:

    The problem with a generic measure of persuasion is that it lacks objectivity: how can you compare two recommender system algorithms side by side? How can you measure the effect of a single component of a recommender system (and not the content that is being recommended, the algorithm that decides what to recommend, or the interface recommending it at the same time?)

    On the other hand, if you are measuring things like ticket sales- isn’t that the same as predicting a binary profile- thus giving mean error measures a reason to persist?

  4. [...] the question is “what measure should we be optimising toward?” This was discussed in a previous post; and here there are mentions of the confidence in seeing the best alternatives, optimising for [...]