Measurement: Impact, trends and developments

A pop-up pizza restaurant hidden in Fitzroy aimed to show the power of the Yellow Pages as a tool for businesses to be found. The initial components of the campaign drew many an opinion. Using Media Monitors proprietary tools and analysis, it is apparent that offline advertising campaigns have now become online ones.

When businesses use social media for their marketing, brand representation or public relations initiatives, assessing their success is a vital step in the process. The recent Sensis Yellow Pages Hidden Pizza Restaurant campaign, which ran from 12–25 April, provides an opportunity to quantify the impact of such an initiative. 
 

The campaign began with an in-book Yellow Pages ad as well as a mail-out in Fitzroy. With the cross over to the online sphere in full-swing by 16 April, we began our analysis. Over the two weeks from 16 to 29 April there were 376 tweets that mentioned the Sensis Yellow Pages Hidden Pizza Restaurant campaign. The campaign aimed to show that Yellow Pages remains a powerful search tool despite a shift in search behaviour increasingly towards the online environment. Discussion of the restaurant, which plied hipsters in Fitzroy with free pizza and lemonade, was spread by participants through word-of-mouth and social media interactions. 
 

Steering clear of a debate on the merits of the campaign itself, it’s interesting to take a look at the social media interaction that occurred – specifically how conversation evolved on Twitter. The covert nature of the campaign resulted in limited discussion in traditional offline media – just one mention in broadcast (on Melbourne’s 3CR) and three mentions in press (one article each in The Age and City Weekly and in a letter in MX). 
 

As a shortcut we can gauge from the data that: 

• The pre-emptive leak of the business behind the “covert” Hidden Pizza campaign resulted in greater overall exposure through social media channels; 
• The nature of the conversation as a result of that leak led to meta-discussion about the campaign more often than participation in the campaign. Tweets more often mentioned a magazine article assessing the campaign than free pizza; and 
• Limiting social media interaction (such as disabling comments on Facebook), while a useful tool to measure the effectiveness of Yellow Pages as an online search provider, worked against raising the profile of both the Hidden Pizza Restaurant and Yellow Pages.

An article that appeared on the Anthill Magazine website by Lachy Wharton that discussed the Yellow Pages social media strategy and the response by Stephen Ronchi, Strategic Communications Manager at Sensis, presents a starting point to help to contextualise some of the data.
 

The conversation that occurred in those 370-odd tweets was highly varied, with many tweeters willingly engaged with the campaign: 

• “Ooh Hidden Pizza restaurant somewhere in Melbourne. Can you find it? The pizza is free for you as a reward! http://hiddenpizza.com.au/” (jcstreettweet, 20/04/2010 16:22) 
• “enjoyed delicious free pizza and homemade lemonade Hidden Pizza Restaurant!” (georgie189, 23/04/2010 22:26) 

However, other participants quickly raised the conversation to a meta-level and commented on the campaign itself: 

• “@hritchie Isn't Hidden Pizza just advertising for Yellow Pages?” (Puck_, 20/04/2010 12:00) 
• “Great example of relationship marketing and hidden marketing with hidden pizza in Fitzroy! Very clever. http://bit.ly/aFdIpE” (dtailnoisemaker, 20/04/2010 17:03)

For arguments sake, it’s worthwhile to include both kinds of conversation (the active participant and the meta-tweets that commented about the campaign) when looking at the numbers.
 

So what does the data say?

Despite the covert nature of the campaign, many people on Twitter knew exactly who was picking up the tab for all of that free pizza. Utilising Media Monitors microblogging tracking service on Mediaportal, we can chart the number of tweets mentioning Yellow Pages, Sensis and the first Anthill Magazine article by Wharton in the context of the key words “Hidden Pizza” across the period, and assign the number of followers that correspond to those tweets to give the following:

More than a third of the discussion that mentioned Hidden Pizza on Twitter also mentioned either the Yellow Pages or Sensis brands. Well before the official confirmation through mainstream media outlets participants in the social media conversation were well aware of the campaign. What is important to note here is that the leak of the campaign’s backer worked both for and against Sensis. The early reveal resulted in greater discussion of the campaign, but it also quickly circulated that another “big business” was attempting to leverage social media interaction to generate interest in its service. As a demonstration, the following is a timeline of all mentions of Hidden Pizza across the period, and the corresponding number of followers that each tweet reached:

The huge spike in followers to 350,000 on 25April – and a lack of a corresponding rise in tweets – can be attributed to just one tweeter (@aichasoebandono), which also closely corresponds to when the campaign was picked up outside of Australia, and was featured on several non-English blogs (e.g. Update or Die). The numbers look nice for the campaign but given that the tweeter is clearly outside of the target audience it’s probably best to remove the skew to better reflect the rest of the data (which would be fairly meaningless with a hugely inflated outlier). Adjusting for the skew, we have the following:

There are a few things worthy of comment here:

• The high number of followers (and tweets) on 16 April can be attributed, in part, to the first wave of interest in the Anthill Magazine article; 
• Following the flat-spot over the weekend of 17-18 April, interest increased over the first half of the next week, possibly as a result of a blog post that appeared on Fairfax’s National Times site by The Vulture: Hijacking friendships for a pizza the action, which discussed the campaign; and 
• The rebound in tweets and followers on 27 April can be attributed to two completely different sources: half the volume of followers were the result of three tweets by blogger raxraxrax (raxlakhani)on the blog post Hidden Pizza: a slice of marketing genius, while most of the remaining conversation marked the shift to mentions outside of Australia on non-English blogs.

The proportion of retweets can also be tracked as a measure of a campaign's spread. A recently article in Technology Review, Why Twitter is the Future of News, makes a fairly strong case for the importance of retweets in reaching a larger audience. Direct retweets formed a significant share of the distribution across the sample for mentions of Hidden Pizza (23%): 

A large portion of retweets that mentioned the keywords “Hidden Pizza” raised the discussion led by Anthill Magazine on the campaign; however, the article spread more quickly and to a larger audience when supported by interraction and commentary. Retweets of the initial @AnthillMagazine tweet, such as “RT @AnthillMagazine Hidden Pizza Restaurant reveals not-so-hidden flaws in Yellow Pages' digital s.. http://bit.ly/9BBNLU*” numbered 16 and reached 6,063 followers. Retweets of JulianCole’s comment on the article (“Cracking article on the Hidden Pizza Restaurant's (Yellow Pages) flawed digital strategy by @Lachyw http://bit.ly/9CZN4V*”) generated greater interest with 23 retweets and reached 15,553 followers. At the time of the tweet, @JulianCole had a little more than 2,600 followers, while AnthillMagazine, which originally tweeted about the article, had more than 11,000. 
* bit.ly links no longer active

Conclusions

While the brand exposure on Twitter as a result of the campaign may be taken as a positive for Sensis an enlightening metric is the overall tone of that discussion. There were an abundance of tweets flying around that ascribed positive or negative sentiments to the campaign (“a slice of marketing genius”, “it was good”, “Hidden Pizza is slow” etc), but perhaps the most basic level of engagement was whether tweets mentioned that there was something for “free”, or whether the campaign was “flawed” (particularly where tweets quoted the headline of the first Anthill Magazine article). This split is fairly important because it demonstrates that the audience for the campaign were either engaged with the hook employed by Sensis (the free pizza), or interested in meta-discussion – that Hidden Pizza represented a marketing campaign. The breakdown is as follows:

Of 376 tweets mentioning Hidden Pizza over the period analysed from 16 – 29 April, 151 either mentioned that there was something for “free”, or that the campaign was “flawed”. Slightly more tweets discussed the Anthill Magazine article than free food, which is a fairly important point. While the number of tweets is fairly low on both sides, it does demonstrate something that all businesses that engage with guerrilla social media campaigns should be aware of – that participants are quick to raise discussion to a meta-level when they sense covert marketing. This occurrence is certainly not new, and was widely discussed following the fallout from the Witchery viral video “scandal” in 2009.

Where Sensis differs in this instance, however, is in the execution of the campaign – it largely focused on providing something for free as a demonstration of the relevance of Yellow Pages, rather than to outright sell a product. This point was not lost on the collective unconscious, as the campaign progressed and the negative sentiment diminished. 

Stephen Ronchi’s response to Wharton’s Anthill Magazine article made the point fairly convincingly that the campaign was intentionally designed to restrict social media interaction to better measure the use of Yellow Pages’ services. Ronchi has also noted that the campaign continues through May with video and photos to come. However, the conversation in the social media space, and the rate at which participants raised the topic to meta-discussion, is perhaps indicative of a need to track social media campaigns with longitudinal studies.

Limiting the spread of the campaign across social media provided a measuring tool for Sensis to gauge the use of Yellow Pages to find the Hidden Pizza Restaurant. However, it also worked against the campaign and limited the social media interaction and, as a result, the traditional media coverage, which resulted in a smaller audience overall.

It will be interesting to see how the remainder of the current TV campaign pans out and whether Sensis decides to move forward with more hidden pop up stores in different states in the coming months.