Multivariate testing in marketing

Last updated

In marketing, multivariate testing or multi-variable testing techniques apply statistical hypothesis testing on multi-variable systems, typically consumers on websites. Techniques of multivariate statistics are used.

Contents

In internet marketing

In internet marketing, multivariate testing is a process by which more than one component of a website may be tested in a live environment. [1] It can be thought of in simple terms as numerous A/B tests performed on one page at the same time. A/B tests are usually performed to determine the better of two content variations; multivariate testing uses multiple variables to find the ideal combination. [2] The only limits on the number of combinations and the number of variables in a multivariate test are the amount of time it will take to get a statistically valid sample of visitors and computational power.

Multivariate testing is usually employed in order to ascertain which content or creative variation produces the best improvement in the defined goals of a website, whether that be user registrations or successful completion of a checkout process (that is, conversion rate). [3] Dramatic increases can be seen through testing different copy text, form layouts and even landing page images and background colours. However, not all elements produce the same increase in conversions, and by looking at the results from different tests, it is possible to identify those elements that consistently tend to produce the greatest increase in conversions. [4]

Testing can be carried out on a dynamically generated website by setting up the server to display the different variations of content in equal proportions to incoming visitors. Statistics on how each visitor went on to behave after seeing the content under test must then be gathered and presented. Outsourced services can also be used to provide multivariate testing on websites with minor changes to page coding. These services insert their content to predefined areas of a site and monitor user behavior.

In a nutshell, multivariate testing can be seen as allowing website visitors to vote with their clicks for which content they prefer and will stand the most chance of their proceeding to a defined goal. The testing is transparent to the visitor with all commercial solutions capable of ensuring that each visitor is shown the same content on every visit.

Some websites benefit from constant 24/7 continuous optimization as visitor response to creatives and layouts differ by time of day/week or even season.

Multivariate testing is currently an area of high growth in internet marketing as it helps website owners to ensure that they are getting the most from the visitors arriving at their site. Areas such as search engine optimization and pay per click advertising bring visitors to a site and have been extensively used by many organisations but multivariate testing allows internet marketeers to ensure that visitors are being shown the right offers, content and layout to convert them to sale, registration or the desired action once they arrive at the website.

There are two principal approaches used to achieve multivariate testing on websites. One being Page Tagging; a process where the website creator inserts JavaScript into the site to inject content variants and monitor visitor response. Page tagging typically tracks what a visitor viewed on the website and for how long that visitor remained on the site together with any click or conversion related actions performed. Page tagging is often done by a technical team rather than the online marketer who designs the test and interprets the results in the light of usability analysis. [5] Later refinements on this method allow for a single common tag to be deployed across all pages, reducing deployment time and removing the need for re-deployment between tests.

The second principal approach used does not require page tagging. By establishing a DNS-proxy or hosting within a website's own datacenter, it is possible to intercept and process all web traffic to and from the site undergoing testing, insert variants and monitor visitor response. In this case, all logic sits on the server rather than browser-side, and after initial DNS changes are made, no further technical involvement is required from the website point of view. SiteSpect is known to employ this method of implementation.

Multivariate testing can also be applied to email body content and mobile web pages.

In addition to testing the efficacy of various creative/content executions on a website, the principles of multivariate testing can and often are used to test various offer combinations. Examples of this are testing various price points, purchase incentives, premiums, trial periods or other similar purchase incentives both individually and in combination with each other. The value of this is that marketers (both traditional and online) can use multivariate testing principles online to quickly ascertain and predict the effectiveness of offers without going through the more traditional multivariate testing methods which take significantly more time and money (focus groups, telephone surveys, etc.).

Design of experiments

Statistical testing relies on design of experiments. Several methods in use for multivariate testing include:

  1. Full factorial the most straightforward method whereby all possible combinations of content variants are served with equal probability.
  2. Discrete choice and what has mutated to become choice modeling is the complex technique that won Daniel McFadden the Nobel Prize in Economics in 2000. Choice modeling models how people make tradeoffs in the context of a purchase decision. By systematically varying the attributes or content elements, one can quantify their impact on outcome, such as a purchase decision. What is most important are the interaction effects uncovered, which neither the Taguchi methods nor optimal design discern. [6]
  3. Optimal design involves iterations and waves of testings. Optimal design allows marketers the ability not only to test the maximum number of creative permutations in the shortest period of time but also to take into account relationships, interactions, and constraints across content elements on a website.[ citation needed ] This allows one to find the optimal solution unencumbered by limitations.
  4. Taguchi methods: with multiple variations of content in multiple locations on a website, a large number of combinations need to be tested, and medium/low traffic websites can take a long time to get a large enough sample to find statistically significant differences in performance if differences really exist. For example, if three different images are to be tested in each of three locations, there are nine combinations to test. Taguchi methods (namely Taguchi orthogonal arrays) can be used in the design of experiments in order to reduce the variations but still give statistically valid results on individual content elements. [7] Taguchi uses fractional factorial designs.

See also

Related Research Articles

Engineering statistics combines engineering and statistics using scientific methods for analyzing data. Engineering statistics involves data concerning manufacturing processes such as: component dimensions, tolerances, type of material, and fabrication process control. There are many methods used in engineering analysis and they are often displayed as histograms to give a visual of the data as opposed to being just numerical. Examples of methods are:

  1. Design of Experiments (DOE) is a methodology for formulating scientific and engineering problems using statistical models. The protocol specifies a randomization procedure for the experiment and specifies the primary data-analysis, particularly in hypothesis testing. In a secondary analysis, the statistical analyst further examines the data to suggest other questions and to help plan future experiments. In engineering applications, the goal is often to optimize a process or product, rather than to subject a scientific hypothesis to test of its predictive adequacy. The use of optimal designs reduces the cost of experimentation.
  2. Quality control and process control use statistics as a tool to manage conformance to specifications of manufacturing processes and their products.
  3. Time and methods engineering use statistics to study repetitive operations in manufacturing in order to set standards and find optimum manufacturing procedures.
  4. Reliability engineering which measures the ability of a system to perform for its intended function and has tools for improving performance.
  5. Probabilistic design involving the use of probability in product and system design
  6. System identification uses statistical methods to build mathematical models of dynamical systems from measured data. System identification also includes the optimal design of experiments for efficiently generating informative data for fitting such models.

Search engine optimization (SEO) is the process of improving the quality and quantity of website traffic to a website or a web page from search engines. SEO targets unpaid traffic rather than direct traffic or paid traffic. Unpaid traffic may originate from different kinds of searches, including image search, video search, academic search, news search, and industry-specific vertical search engines.

Taguchi methods are statistical methods, sometimes called robust design methods, developed by Genichi Taguchi to improve the quality of manufactured goods, and more recently also applied to engineering, biotechnology, marketing and advertising. Professional statisticians have welcomed the goals and improvements brought about by Taguchi methods, particularly by Taguchi's development of designs for studying variation, but have criticized the inefficiency of some of Taguchi's proposals.

Ad serving describes the technology and service that places advertisements on Web sites, mobile Mobile Apps, and Connected TVs. Ad serving technology companies provide software to Web sites and advertisers to serve ads, count them, choose the ads that will make the Web site or advertiser the most money, and monitor the progress of different advertising campaigns. Ad servers are divided into two types—publisher ad servers and advertiser ad servers.

Search engine marketing (SEM) is a form of Internet marketing that involves the promotion of websites by increasing their visibility in search engine results pages (SERPs) primarily through paid advertising. SEM may incorporate search engine optimization (SEO), which adjusts or rewrites website content and site architecture to achieve a higher ranking in search engine results pages to enhance pay per click (PPC) listings and increase the Call to action (CTA) on the website.

Web analytics is the measurement, collection, analysis, and reporting of web data to understand and optimize web usage. Web analytics is not just a process for measuring web traffic but can be used as a tool for business and market research and assess and improve website effectiveness. Web analytics applications can also help companies measure the results of traditional print or broadcast advertising campaigns. It can be used to estimate how traffic to a website changes after launching a new advertising campaign. Web analytics provides information about the number of visitors to a website and the number of page views, or create user behavior profiles. It helps gauge traffic and popularity trends, which is useful for market research.

In online marketing, a landing page, sometimes known as a "lead capture page", "single property page", "static page", "squeeze page" or a "destination page", is a single web page that appears in response to clicking on a search engine optimized search result, marketing promotion, marketing email or an online advertisement. The landing page will usually display directed sales copy that is a logical extension of the advertisement, search result or link. Landing pages are used for lead generation. The actions that a visitor takes on a landing page is what determines an advertiser's conversion rate. A landing page may be part of a microsite or a single page within an organization's main web site.

Google Analytics Web analytics service from Google

Google Analytics is a web analytics service offered by Google that tracks and reports website traffic, currently as a platform inside the Google Marketing Platform brand. Google launched the service in November 2005 after acquiring Urchin.

Conversion rate optimization

Conversion rate optimization (CRO) is the process of increasing the percentage of users or website visitors to take a desired action.

Webtrends is a private company headquartered in Portland, Oregon, United States. It provides digital analytics, optimization and software related to digital marketing and e-commerce. It provides services to approximately 2,000 companies.

A/B testing Experiment methodology

A/B testing is a user experience research methodology. A/B tests consist of a randomized experiment with two variants, A and B. It includes application of statistical hypothesis testing or "two-sample hypothesis testing" as used in the field of statistics. A/B testing is a way to compare two versions of a single variable, typically by testing a subject's response to variant A against variant B, and determining which of the two variants is more effective.

Rule-developing experimentation or RDE is a systematized solution-oriented business process of experimentation that designs, tests, and modifies alternative ideas, packages, products, or services in a disciplined way using experimental design, so that the developer and marketer discover what appeals to the customer, even if the customer can't articulate the need, much less the solution.

Google Optimize, (GO) formerly called Google Website Optimizer, is a freemium web analytics and testing tool by Google. It allows running some experiments that are aimed to help online marketers and webmasters to increase visitor conversion rates and overall visitor satisfaction.

Uplift modelling, also known as incremental modelling, true lift modelling, or net modelling is a predictive modelling technique that directly models the incremental impact of a treatment on an individual's behaviour.

Multivariate landing page optimization (MVLPO) is a specific form of landing page optimization where multiple variations of visual elements on a webpage are evaluated. For example, a given page may have k choices for the title, m choices for the featured image or graphic, and n choices for the company logo. This example yields k×m×n landing page configurations.

In internet marketing, cultural multivariate testing is multivariate testing performed on an international website in each geographically distinct market for the purpose of website localisation, i.e. finding the best culturally tailored design for the users in that specific location.

Optimus is a Process Integration and Design Optimization (PIDO) platform developed by Noesis Solutions. Noesis Solutions takes part in key research projects, such as PHAROS and MATRIX.

In electronic commerce, conversion marketing is marketing with the intention of increasing conversions—that is, site visitors who are paying customers. The process of improving the conversion rate is called conversion rate optimization. However, different sites may consider a "conversion" to be a result other than a sale. Say a customer were to abandon an online shopping cart. The company could market a special offer, like free shipping, to convert the visitor into a paying customer. A company may also try to recover the customer through an online engagement method, such as proactive chat, to attempt to assist the customer through the purchase process.

App store optimization (ASO) is the process of increasing an app or game’s visibility in an app store, with the objective of increasing organic app downloads. Apps are more visible when they rank highly on a wide variety of search terms, maintain a high position in the top charts, or get featured on the store. Additionally, app store optimization encompasses activities that aim to increase the conversion of app impressions into downloads.

Dynamic creative optimization (DCO), is a form of programmatic advertising that allows advertisers to optimize the performance of their creative using real-time technology.

References

  1. ↑ Josef A. Mazanec and Helmut Strasser (2000). A Nonparametric Approach to Perceptions-Based Market Segmentation: Foundations. Springer. ISBN   3-211-83473-7.
  2. ↑ "What is multivariate testing?". www.campaignmonitor.com.
  3. ↑ "Experimentation & Testing: A Primer". Avinash Kaushik. 2006-05-22.
  4. ↑ "WilsonWeb.com, Conversion/Testing: 10 Factors to Test that Could Increase the Conversion Rate of your Landing Pages, by Sumantra Roy, 06/05/2007".
  5. ↑ "Archived copy". Archived from the original on 2008-08-28. Retrieved 2008-08-20.{{cite web}}: CS1 maint: archived copy as title (link) "Web Analytics Demystified", "Web Analytics and Data Collection: The Page Tag", By Judah Phillips
  6. ↑ MarketingNPV, 3 Ways to Accelerate Your Learning Process
  7. ↑ Webpronews.com, Scientific Web Site Optimization using AB Split Testing, Multi Variable Testing, and The Taguchi Method, by Matthew Roche, 07/26/2004 Archived 2007-11-04 at the Wayback Machine
This page is based on this Wikipedia article
Text is available under the CC BY-SA 4.0 license; additional terms may apply.
Images, videos and audio are available under their respective licenses.