And like category pages, it’s not just about look-and-feel, but also your merchandising strategy — click-through and conversion rates can be improved more by surfacing relevant and desirable products than how many products are served per results page, or how large product images are. A/B testing the site search experience should not be conducted without first optimizing site search tuning logic.
Key metrics to track
When conducting A/B tests for site search, know which success metric you aim to improve:
- Search results exit rate
- Search query refinement rate
- Engagement with autosuggested terms and products
- Engagement with sort and filter tools
- Engagement with pagination links
- Product page click-through rate (indicating search results are relevant and sort and filter tools are usable)
- “Add to Cart” rate
- Conversion rate
- Revenue per visitor
- Search-to-browse rate (the percentage that abandon site search to use category browsing in a single visit)
Because site searches can be very specific, items per order and average order value may not raise significantly from changes to the site search experience. Customers who search are often “spearfishing” for something specific, and may be less likely than category browsers to build their carts.
What you don’t need to test
While you can literally test anything in your experience, certain design elements have less influence over customer behavior, and will provide you less return on your testing investment:
- Default sort option (this should always be Relevance)
- Repeating the search term at the top of results (no brainer, just do it)
- Showing the number of results returned (just do it!)
- Anything you’ve already tested on category pages regarding product information like showing QuickLook vs not, showing star ratings or product image sizes (keep search tests focused on what makes search experience unique)
Instead, focus on formulating hypotheses that address identified experience blockers that may be affecting key metrics on site search results pages.
Autosuggest usability requires both back-end tuning and front-end optimization. When designing your front end, you want to think through what design pattern is most helpful in guiding your customer to the most relevant set of results.
Does the “paradox of choice” apply to autosuggest menus? You could be throwing too much at your searcher with more than “X” suggestions. Use A/B testing to solve for X.
Should you show suggestions for categories, search terms or individual product matches in your menus, or a mix of these “types”?
If you show a mix of types, should categories be presented above or below search terms or individual products?
What’s the best way to federate results (group by suggestion type)?
Does a two-column flyout outperform a single-column design?
Visual emphasis on departments
Does text color increase engagement with category scope refinements? Does it increase click-through on search results pages (indicating selecting a department at the autosuggest level serves better results than un-scoped searches?)
Does indenting category suggestions increase engagement or lead to more clicks on refined searches? Can you measure click-through and conversion rates on category scoped search results against results from search term suggestions?
Showing featured products
Do featured product listings (with images, titles and prices) increase click-through and conversion against not using them? Do they visually compete with and reduce engagement with search term suggestions?
Does the number of featured products impact engagement?
Can you increase engagement with suggested search terms by making them equally prominent as product results? Does labeling suggestions “Top Sellers” convert higher than labeling them “Featured Products” or “Best Matches”?
Regardless of how you design your autosuggest menu, don’t forget to include a test version without it to validate whether it helps or hurts the search experience. Make sure to segment results on desktop and mobile devices, as autosuggest can be much more difficult to navigate on mobile devices, as a device’s keyboard typically cuts screen space in half. You may discover your best approach is to include it on desktop only.
A popular tactic is to redirect search queries that exact-match categories and sub-categories to their respective product list pages.
For example, a search for “memory foam dog beds” directs to a category page, which is laid out quite differently than its search results pages.
Orvis’ category page layout
Orvis’ search results page layout
Notice that the search page displays smaller product images, more filtered navigation options and omits the hero banner. It’s important to validate if category redirection is a better experience. This is a trickier A/B test to set up, as it only applies to search terms that are redirected to category pages, and may not be feasible with all testing tools.
Visitors who use your search box are in task mode. Are your promotional banners distracting from their task?
Does showing promotions tailored to the search term increase click through, conversion and revenue? Does it increase urgency and reduce days to purchase?
Do banners that include department filters get more engagement than left-side menus? Does this lead to better search refinements, higher click through and higher conversion and revenue?
Do promotional banners add value at the bottom of search results, or do they compete with navigation?
Header design and refinement tools
Search results headers ideally repeat the search query, display the number of product matches, suggest related terms, display filter and sort tools and make it easy to move forward and back between results pages. If your search header omits any of these search results best practices, it’s worth testing your existing design against a revised one.
A compilation of search results page headers from top online retailers
Your analytics overlays may also reveal which elements are not attracting engagement. It’s very important that customers notice, understand and interact with filters, facets, sort tools and page navigation. If it appears any of these tools are not getting the love they should, consider alternative designs that focus on bringing more visibility to these elements.
For example, user testing has shown site users prefer using Next buttons to clicking numbered paginated links. A redesign of this banner to include large Prev and Next buttons could be a winner.
When filter and sort don’t appear in the white space above product results, they can be easy to overlook. This example’s Shop By and Sort By labels are hidden in a gray field and don’t look interactive. Their companion icons aren’t necessarily recognizable, while the sunglass icons draw more attention.
“You may also like” suggestions are typically the domain of product detail pages, but occasionally appear on site search pages. If you currently show them, test them against not showing them on search pages. Product recommendations are a call-to-action away from search results. Pulling the customer’s mindset out of search mode (which is typically closer to conversion) into discovery mode may lower conversion rate.
Search success is all about relevance. Do you really want to show items unrelated to the search term?
No results found
No matter how well you’ve tuned your site search back end, you will have searches for which no good matches are found. Good design can salvage the situation, poor design can cause site abandonment.
There are three high-level guided approaches you can A/B test to determine which is the most helpful for disappointed searchers: suggest tips for refining the search term, show suggested products, or encourage category browsing.
You’re likely already using one of these three approaches. Before further tweaking the approach you’re already using, test it against the two approaches you’re not using. Consider this a radical redesign.
Suggest tips for refining the search term
Show suggested products
Encourage category browsing
Combined approaches: search tips and recommended products
Maximizing site search results performance
Because site search success hinges more on the relevance of returned results than product list layout, A/B testing search pages without first addressing back-end optimization is putting the test before the horse.
And because search result list pages function much like category pages, consider testing category page elements like filter and sort design, product thumbnail size and pagination across both site search and category pages. Reserve site search tests to elements specific to site search results.
Keep in mind that mobile search context and experience can be very different than desktop, and these experiences should be tested and measured separately. Don’t be afraid to implement different design patterns for mobile than desktop, such as the number of results shown per page, autosuggest menu design and behavior, and header design.
Need help with your ecommerce A/B testing strategy? Drop me a line.
Ecommerce Illustrated is a project of Edgacent, an ecommerce advisory group.