 |
     |
||
 |
||
 
| ||
|
|
||
|
||
|
| ||
In Depth
eCE's unique analytical power comes from the quantification of each and every highlighting action, thereby generating a wealth of data at an incredibly fine-grained level. The key to leveraging this comes from the eCE report tools that we provide in two forms: an online portal and a desktop app. Both the online and desktop version have almost identical interfaces. The report layout consists of two main areas. There is a control panel on the left and a color map on the right. The control panel allows you to determine important settings that impact the color map. Let's begin with the control panel settings ...
Color Map Settings ... There are four map options: Like, Dislike, Net Like, and Frequency. These settings determine what results the color map represents. For instance, the default setting is Net Like. This means that the colors of the map reflect the net difference between like and dislike highlighting. If the Like setting is selected, then the map colors are based only on the like highlighting results. Same goes for the Dislike setting. Finally, the Frequency setting shows which items in the eCE media received the most highlights regardless of whether they were like or dislike highlights. The frequency map is a measure of the intensity of attention garnered by items in an eCE media.
Subgroup Analysis ... If no subgroups are defined before the report is generated, then "All Respondents" is the sole selection in the Subgroup Analysis dropdown. If subgroups are defined, then clicking on the dropdown box enables the selection of any of the subgroups. Upon selection, the color map, Overall Performance scores, and Item Rank Orderings will update to reflect the data for the selected group.
Data Transform ... The data that the color map is based on can be transformed using the "Data Transform" and "Results Based On" settings. The defaults are for the color map data to be normalized and calculated based on the total sample size (N). Using normalized results allows you to see how items (e.g., words and/or images) within an eCE media (i.e., print ad, message statement, etc.) perform relative to one another. So, when you look at a normalized color map, you will virtually always see a mix of red and green. This does not mean that the red items are necessarily disliked by the respondents. Rather, it means that these items are liked less than other items in the same eCE media by the respondents. It is common in eCE studies for non-normalized color maps to be mostly or entirely green, reflecting the fact that on balance most eCE media receive more like highlights than dislike highlights. While sometimes helpful, we frequently have found that color maps based on raw data are less illuminating than normalized maps. A color map based on normalized data provides direct insight into which specific sections of the ad/message need improvement.
Results Based On ... This setting allows you to change the denominator used for calculation of the percentages (e.g., percent like highlighting actions and percent dislike highlighting actions). When Results Based On is set to "% of respondents", all percentages are calculated based on the total sample size of respondents. By contrast, when Results Based On is set to "% of highlights", percentages are based on the total number of highlighting actions (i.e., both like and dislike) for a specific item. An important difference between these settings is that "% of respondents" uses a single number (i.e., N) as the denominator for all percentage calculations for all items in the eCE media. This is different from the "% of highlights" setting where the denominator for percentage calculations can be distinct for each item. In practice, we have sometimes found that the Results Based On setting has a substantial effect on the color map, and sometimes not. There is no statistically or theoretically correct data setting.
Legend ... The color map legend indicates how differing normalized results are assigned differing shades of green or red. There are different legends for the Like, Dislike, Net Like, and Frequency color maps, but they all can be interpreted in a similar fashion. The darker the shade of green or red, the more liked/disliked is an item in the eCE media. You will probably see items that are “shaded” in white. For the Net Like color map, this indicates that the item’s net likeability is in the middle of the overall distribution (40th-60th percentile). For the Like, Dislike, or Frequency color maps, this indicates is in the 0-20th percentile of the relevant scale.
Overall Performance ... The eCE report provides two metrics of the general reaction of a respondent population to an eCE media. First, the Net eCE rating is calculated as the total number of positively highlighted items minus the total number of negatively highlighted ones. Second, the Normalized Net eCE rating is a transformation that allows for performance comparisons across ads. The latter is especially useful when testing multiple ads or constructing a benchmark database. The Net eCE Rating should generally not be used to compare eCE media with each other because this rating generally increases as the number of items within an eCE media increases. For instance, if you were to compare a long message statement with a short message statement, it is likely that the Net eCE Rating for the former would be higher. By contrast, the Normalized Net eCE Rating takes into account both the number of items in an eCE media and also the total respondent sample size. In this way, multiple eCE media can be compared not only within a single study but across studies as well.
Color Map ... The right side of an eCE report displays the color map. The map is designed to facilitate detailed insights regarding which parts of an eCE media are over-performing (i.e., more liked) and which are under-performing (i.e., less liked). There are four basic variants of color maps. The default is the Net Like map. This is a holistic view of the likeability of an eCE media since it is based on the difference between the like and dislike highlighting results. Items with the highest net likeability are colored dark green; those with the lowest net likeability are in dark red. There are four shades of green/red to indicate where an item falls on the normalized net likeability distribution. White shading indicates that an item lies in the middle of the distribution (i.e., the 41st - 60th percentiles). Importantly, the amount of green vs. red shading that is displayed is a function of the shape of the normalized distribution, and the proportion of green vs. red will vary from media to media. So, it is not correct to infer that a color map with more green than red items shaded is more liked. Rather, in this example, it suggests that there are a larger number of items in the higher range of the net likeability distribution. The clustering of items with similar net likeability can yield the appearance that It is also helpful to examine the like and dislike results separately by examining these different color maps. Finally, the Frequency color map provides a perspective on what
Summary Stats Boxes ... Each item (i.e., word, image, sentence, or paragraph) has its own quantitative profile that includes: positive and negative highlighting counts/percentages, Net Like rating (i.e., like ratings minus dislike ratings), and Normalized Net Like rating and percentile ranking. To display these summary statistics for an item, you simply place the cursor over the item or, for the desktop app, click on the item. In addition, you can show a summary stats box by clicking on the item in one of the Item Rank Orderings boxes. These boxes contain complete rank ordered lists of all of the items in an eCE media. For instance, the #1 item in the Like list has the highest proportion of like highlights of all items. At the end of the Like list is the item with the lowest proportion. Using these boxes, you can get a sense of where an item is ranked relative to other items along the various dimensions of likeability and frequency.
Combined Summary Stats ... With the click of a button, you can display essential stats for any item, but what if you are interested in the statistical profile of multiple items (e.g., a sentence or a paragraph)? Simply click on an item and hold your mouse button down while moving your cursor over additional items. Once your cursor has moved over the final item, then release your mouse button. A summary stats box will display with the numbers reflecting the averages across all of the selected items.
Desktop App Buttons ... While the online report portal and the desktop app are almost identical in features and appearance, there are some minor differences. For the desktop app, if you would like to import a new eCE report data file (in XML format), then just click the "IMPORT" button in the top right. If you would like to clear any summary stats box that is currently displaying on screen, then click the "Clear Selections" button toward the bottom left. Finally, if you would like to save a screenshot of any variant of an eCE report (including the control panel and color map), click on the "JPG" or "PDF" button in the top right corner. Either of these image formats are easy to embed in a word processing or presentation document.
Strategies for Analysis
eCE is a new approach to evaluating static message media, and thus there are few established protocols for analyzing eCE results. We have developed some useful analytical strategies to help researchers create compelling narratives using eCE data. After loading the eCE data file into the desktop app or online portal, begin by examining the color map at the initial default settings. What is shaded in darker/lighter green vs. what is in darker/lighter red? Click on a few of the particularly salient words/images/items on the color map in order to view the summary stats for these items. Consider the total sample size and compare this to the number of like and dislike highlights that an item receives. Possibly use the Combined Summary Stats feature to see how an entire sentence or groups of words performed. While doing these things, remember that the default color map display uses normalized data, and thus the mix of green and red does not necessarily indicate that respondents liked (in an absolute sense) certain aspects and disliked others. The color shadings are relative. So the color map is a guide to what respondents liked more/less in an eCE media relative to other items in the same media. At this point, it can be useful to examine whether the raw (i.e., non-normalized) results have an interesting story to tell. Usually, the answer is no. But if an item(s) have more dislike highlights relative to like highlights, then this is a striking finding that should be noted and taken into account in the researcher's overall narrative.
After absorbing the normalized net like perspective (and doing a a check of the non-normalized view), then it can be helpful to examine the components that comprise the net like map. That is, the like ratings and the dislike ratings separately. The researcher can quickly do this by selecting the different options from the Color Map radio buttons. Think of this as analogous to conducting univariate analyses on normal survey data before doing more complex multivariate analyses. So, the like and dislike color maps will help flesh out the story of the net like map. To further ground the narrative, also look at the frequency color map. The frequency map simply represents the combined number of like and dislike highlights - as opposed to the net like map which is based on the difference between likes and dislikes. Thus, the frequency map is a useful proxy for where in the message media respondents were more/less focused on (i.e., what items drew their attention the most and thus led them to highlight it either positively or negatively).
It is sometimes suggestive to use the Item Rank Ordering lists beneath the color map. These are simply another way to help develop an explanatory story about an eCE. Quickly viewing the top ten items that rank highest in terms of likes, dislikes, and frequency can increase a researcher's understanding of the overall color map.
In comparing multiple eCE media with one another, use the normalized net like score in the Overall Performance box. This score ranges from -1 to 1, with a higher positive score indicating the greater likeability of the media. The normalized scores are comparable across eCE media within the same study and across studies with different sample sizes.
Once a researcher has resolved on a narrative explaining the full sample eCE results, then s/he will probably be interested to view subgroup results. Use the Subgroup Analysis dropdown to display color map, overall performance stats, and item rank ordering lists for each of the groups. When examining subgroups, we recommend that researchers follow the same steps as we advocate above for full sample analysis.
