Overview

Constructor’s conversion optimization and discovery benefits are only as good as our uptime and performance. For this reason, we have a robust process of performance validation and monitoring. During the holiday season and the peak demand period of Black Friday and Cyber Monday, we increase our standards in all of these areas out of recognition that it is the most important selling period for many of our customers.

Survey of peak demand for 2020

In planning our preparations in the run-up to the 2021 holiday season, we looked back to daily and peak demand changes during the holiday season and also reviewed how our baseline traffic has increased in the time since then.

Last Black Friday our overall traffic increased 200% over daily baseline levels, and peaked at 500% of baseline. Not only did we maintain our 100% uptime, but performance during the peak demand periods actually improved relative to equivalent periods (due to changes in traffic patterns). Since then the system has already scaled without interruption or degradation for average daily traffic by over 383%.

Performance improvements over the past year

Over the past year we have worked continuously to drive even better performance and scalability, contributing to improved latencies and zero downtime. Some example projects and outcomes include:

• Optimized scaleout policies
• Introduced stand-by server pools
• Improved instance boot and data download time by ~300%
• Increased performance of personalization service
• Doubled performance of underlying search & browse servers
• Decreased index update delivery times
• Increased database read capacity

Scale-out performance testing

We have tested scaling to 2000% of current average daily traffic volume, while validating the continued performance of the following:

• Database connections 
• Monitoring infrastructure
• Networking infrastructure
• Response latencies @ median, 90th percentile, 95th percentile, 99th percentile
• Response latencies for each customer, and each product used by each customer
• Data ingestion SLA times

Chaos and anti-fragility testing

We also use chaos testing to validate that catastrophic failure of the following supporting infrastructure does not impact critical features (primarily search, autosuggest, browse, recommendations, collections request/response times):

• Disabled MySQL
• Disabled index builders
• Disabled personalization queues
• Disabled supplemental ranking engines
• Availability zone and data center failures

All of the above is in addition to the rigorous performance test and rollout plan we use for every release:

• Full test suite on every pull request (incremental code change).
• Production traffic replay for all deployment builds (multiple times a week).
• Rolling, risk-adjusted deployment procedures across worldwide data centers.
• Canary deployment for deploys touching critical path request/response lifecycle.
• Automatic build failures if sensitive thresholds on result quality, latency, memory consumption, CPU consumption and more are breached at any of these levels.

Standard on-call procedures

At all times we have multiple on-call schedules for the following teams:

• Front-end and client teams
• Data science and result quality teams
• Core platform and response performance  teams
• Each of these have multiple fallbacks and tiered escalation policies

Automated alerting

Alerting is automated across dozens of metrics to ensure we are aware of incidents within seconds. A few representative examples:

• Queuing times
• Per-service latencies
• Memory and CPU consumption

Special holiday on-call procedures

In addition, we take special precautions during peak holiday shopping periods:

• We will over-provision all infrastructure above and beyond typical scale-out policy.
• We double on-call rotation utilizing the above-mentioned automatic notification and escalation policies.
• The entire account and product team will be monitoring throughout the Black Friday / Cyber Monday period, with elevated focus for other holiday periods (such as Boxing Day).

Conclusion

At Constructor, we take uptime, performance, and service stability very seriously because the best conversion optimization and ML are moot if we don’t deliver fast and stable service consistently. The goal of this document is to provide our customers with a broad overview of our site reliability practices, as well as a specific view of our holiday readiness procedures. As always, please feel free to reach out to your Customer Success Manager if you have any further questions.

Facets are key to great search and browse experiences because they allow users to narrow their results to find the right product. To provide better insights into facet engagement, we recently released analytics for facet usage. Building on this recent release, customers can now take action based on those insights with Category Facet Configuration. 

For example, in your online grocery store you may observe usage of the “Price” facet significantly exceeds the "Brand" facet in the Beverages category, or that few users interact with the “Fat Free” Nutrition facet. Using Category Facet Configuration you can now move “Price” to the highest position and hide the “Fat Free” facet option in the Beverage category, completely independent of the existing global facet configuration. Seeing different trends in Frozen Food? No problem, you can set a custom facet configuration there too.

These new facet configuration capabilities supplement our existing dynamic facet ranking options, which rank facets based on those that are most common among the top ranking products for a given category. For instance on a cosmetics site, the Foundation page would have the facets associated with actual foundation products at the top of the page, such as “Formulation”, “Brand” and “Color”. In contrast, “Bristle Type” would be ranked lower because this is a facet associated with foundation brushes, which behavioral data indicate are not as appealing for users on the Foundation page.

Check out category facet configurations here.

We’ve delivered this feature for Category Browse based on traffic and customer demand, but we’ll be evaluating expanding to Collections and/or Search in the future. Reach out to feedback@constructor.io or your Customer Success Manager if you’d like to see this more broadly!

Across most eCommerce verticals, we often see users who engage with facet refinements convert at a higher rate than users who do not. 

For this reason, we've long supported the ability to rank facets not just alphabetically, or by the number of matches, but also by automatically increasing the rank of facets and options corresponding to product ranking, itself optimized for business KPIs. To this feature we added global facet ranking overrides in 2020.

We'll be rolling out new features for facet ranking in the coming weeks, but want to share a first release for facets: Browse facet analytics! Check browse analytics here!

Category browse facet analytics

We now report facet engagement for each category page both in the percentage of requests that include facet refinements, as well as for engagement with individual facets. 

In this case, we see that Gender and Age are particularly important to users. Because Constructor will associate users' ultimate clicks, add to carts and purchase events with the original category intent (a generic 'Jackets' landing page in this case), all the products users arrived at after faceting will be appropriately ranked for the principal category page. The analytics here tell us about the product attributes users typically consider when they land at a broad page like 'Jackets' - Brand, Gender, Age and Size in this case. This information can help inform overrides - currently these are supported at the global level through gobal facet configuration, but we'll soon allow these at the query- or category-level.

Category browse analytics

Incidentally, the release of category facet analytics also marks our first release of category detail analytics as a whole. Previously we provided an overview of all category pages and their clickthrough rates, but we'll now surface detailed analytics for any category page!

Finally, we've improved our interface in the category analytics overview, allowing users to hover on a category to jump to interact, analytics or searchandizing.

One of our motivating goals at Constructor is to give customers superpowers. Ranking out of the box should optimize for customer business metrics like RPV and conversion, and when customer product or merchant teams want to adjust ranking for business goals, the experience should be delightful. 

We've recently released two new features to improve the searchandizing experience: Search-in-searchandizing and improved numeric range selection.

Search-in-searchandizing

Customers may want to boost the ranking of owned (store) brands, or prioritize healthier food to align with brand objectives. 

Previously this could be frustrating when the number of categories or facet options climbed into the hundreds (or thousands). Now, our search will allow merchant and product teams to search for the right option, whether in search, browse, collections or global searchandizing contexts!

Numeric range selection

Searchandizing range values can be extremely powerful, especially when paired with the ability to define hidden or protected facets shown to admin users, but not to other users. This allows prioritizing clearance of high inventory items, optimizing for margin or newer products. 

With our new range selection UX, it is now possible to make fine grained and specific extents for searchandizing, as well as allowing explicit selection of all values above or below a given point, as well as setting thresholds even if products on the page don't currently have a given value (such as when the product with greatest inventory is 100, but the goal is to boost products only once they have 1,000 items in inventory). The new selection UX is available in Search, Browse, Collections and Global Searchandizing!

Patch delta catalog format 

Customers integrating with our catalog feed methods (HTTP or FTP) have the ability to provide product data for products and variations through full feeds (sync) as well as partial or update files (deltas). This makes it easy for anyone - integration engineers, merchants or product managers - to download and validate the catalog from the dashboard (available here with admin access) in order to answer questions about what's been sent to Constructor.

While deltas allowed customers to update just those items with changes, there is an assumption that all of the data about the items (products or variations) being updated is sent whenever any of the item data changed. For customers implementing complicated BOPIS (buy online, pick up in store) store availability data with feeds, this was a challenge. 

As a result, we released a new catalog capability - patchdelta. This catalog format allows updating some of the data for some of the products and is covered in more depth here.

New features

Single Sign on (SSO) Early Access We support Single Sign-On for our admin dashboard! This is in early access with a limited set of customers, but will become available more broadly in the future depending on plan subscription. Initial support is targeted towards Okta as the identity provider, but will expand based on demand. Reach out to your account or CS manager if you'd like to be added to the waitlist!

Top search, zero result and browse report export! Reports can now be generated for the top search terms, top zero result terms and top categories to facilitate outside analysis and stakeholder reporting! These reports include key metrics like views, clicks and click-through rates, joining our existing reports for top purchases.

Advanced Japanese language support Similar to our release of Chinese language support in March, we have added advanced Japanese language search support. Since Japanese has no (or very few) spaces between words, splitting product names and user queries into searchable components (tokens) is non-trivial. We have added special dedicated models to our search algorithms to handle this process (known as 'tokenization')

We are expanding aggressively around the world and our cutting-edge language modeling for languages that present unique challenges, such as Chinese and Japanese (tokenization) and Dutch and German (compounding / decompounding) is an important component of this strategy. If you're interested in expanding language support, reach out to your customer success manager!

Self-service for API values

• Displayable field configuration We now allow dashboard users to configure which fields will be returned by the API. Navigate to the index configuration and choose "Manage Searchability & Displayability"

• Remove searchandizing options that no longer exist in results Searchandizing rules may be set in a category or search result page, then products with these attributes go out of stock or otherwise leave the catalog. We want to preserve these rules so if products matching the rules return to the catalog, we continue to map them as expected, but we now make it more clear when there are rules which no longer correspond to products in the search or browse page -- and make it easy to remove these rules.
  
• Populate names and images in recommendations merchandizing We now show product names and images in the recommendations merchandizing interface to simplify the process of editing manual recommendations merchandizing rules.
• Pointer consistency Use pointer for clickable items consistently in dashboard.
• Filter padding in collections edit interface Filters no longer overflow in the collections edit interface when editing rules at the bottom of the page.

Dashboard performance

• Optimized performance for collections page In certain cases large numbers of facets could slow down the collections page. We now lazy-load facets and options to accelerate the collection creation experience.

Product research

With our Q2 ending July 31st, we're hard at work on some major initiatives closing out the quarter as well as research to inform priorities for Q3! In order to ensure we deliver the greatest value to our customers, we're conducting some focused product research in a few areas of current and potential investment.

If you're interested in participating in this product research, please use let us know with this brief 3 question survey.

New in collections

Items added manually to a collection can now be deleted or replaced in bulk!

Documentation updates

• Updated catalog documentation to reflect that Json metadata can be submitted with a CSV catalog upload.
• Added documentation for our item-based recommendation merchandizing capabilities released in late May.

Client updates

• Added bindings to our Javascript client library to retrieve groups (categories) without retrieving items. This allows customers to easily build category hierarchies in their site without retrieving a full result set for a particular category page.

Result and API fixes and improvements

• Fixed an issue that could lead to different results depending on type of apostrophes found in query and/or catalog: ' or ’.
• Fixed an issue that could lead to partial recall of matches in edge cases where stemming, synonyms and keywords not found in the catalog could lead to partial result recall.
• When filtering the catalog ingestion status dashboard by a particular index, the summary counts for tasks in progress and queued is now accurate.

New in Analytics 

• Purchase & conversion report We have added the ability to export purchase and conversion analytics data! To export a conversion report, head over to your conversion analytics page and select the export icon - the report will be sent to you via email! 

API updates

• JSON serialized query strings We observed some customers using extensive query-time parameters, which could result in duplication in request parameters. To improve usability, we now allow serializing search parameters, documented below. See full documentation for the qs parameter here (bottom of page)
  
• Metadata return default Items added via API and catalog will no longer be automatically returned in search, browse and recommendation responses. We've made this change as we observed that customers could add large amounts of data about products via API or feed, which would result in unexpected performance degradation. We also want to avoid a scenario when sensitive data like margin is inadvertently exposed before fields are hidden. Allowing field displayability can be accomplished via, API or by reaching out to support. It will be possible in a forthcoming dashboard update as well.

Result improvements

• Store- or region-specific search suggestions We can now tailor search suggestions more specifically for customers with buy online / pick up in store (BOPIS) or regional availability represented in one index. Search suggestions will represent searches with results in the context of a user's store. Reach out to support to learn more about implementing this functionaity.
• Avoid rules reducing recall Avoid a rare scenario where several rules could reduce an items rank below zero, suppressing it's display.

Recommendations

• Filtered items preview Filtered items strategy can now be previewed in the dashboard, see your recommendations section for more. 
• Bestseller preview We now allow previewing bestseller recommendations without selecting a user profile to view unpersonalized bestseller recommendations

Documentation

• Searchandizing docs Updated search searchandizing docs to document all current capabilities.
• Documentation reorganization Reorganized API documentation around key goals: Returning Results, Managing Catalogs, Defining Rules and Logging User Events.

Dashboard improvements & fixes

• All machine learning rules applied to a query are correctly displayed in the dashboard regardless of query capitalization.  
• Fixed an issue that could cause the current dashboard tab highlighting to go missing.
• Clarified wording in interact modals.
• Updated styles of login and password reset to match application UX.
• Removed unnecessary notification upon login, confirming login.
• You can now navigate directly form a collection definition to the collection searchandizing page.

Recommendations merchandizing!

• We now support recommendation merchandizing in the dashboard for alternative or complementary strategies!
  
• Our initial release here focuses on hiding specific items when requesting recommendations for another item, but these capabilities will expand in the future.
• Fundamentally, we are committed to algorithmic optimization of KPIs across search, browse, recommendations and the entire product suite. That said, we recognize that customers want to intervene if they see an opportunity to improve and want to and that some customers may not have as much behavioral data as others (though we are working here as well - see below!). 

Recommendations UX improvements

• Additional recommendation strategy preview support We expanded preview capabilities in the dashboard to include query and bestseller recommendation strategies. This leaves just filtered items strategy without dashboard preview support (look for this soon).
  
• Prevent recommendation pod deletion We have restricted the deletion of recommendation pods with usage to admin users.
• Improved warning on recommendation pod deletion We now check for requests in the last 7 days for recommendation pods and present a warning before deletion.
  
  

Performance

• Collections dashboard performance imporvements We improved performance of collections edit page for customers with larger catalogs by over 50%. This is the first of a few dashboard performance improvements for customers with larger catalogs.

API

• The REST batch items update endpoint now accepts items without the item_name parameter if id is provided.
• Hide range buckets We now permit hiding a particular range bucket from numeric range facet fields in the facet configuration API. This allows customers to just show facet selection options for 3+, 4+ and 5 star reviews, hiding 1+ and 2+ star filters.
• Field presence summary & searchability API We launched a searchability API, which allows requesting all fields associated with items in the catalog, configuring which of these should be searchable and how these should be searched (exact or fuzzy). A forthcoming release will provide 

Result Fixes

• Facet count consistency In very rare cases, selecting a facet to narrow the results could lead to an increased number of results as spell correction was applied more aggressively for the narrower result set. This is now avoided in all cases.
• Trailing spaces in facets Fixed an issue that could cause facet with trailing spaces uploaded in certain situations to be parsed incorrectly.

Display Fixes

• Fixed a UX display bug in date selector.
• Fixed facet range display in the facet configuration dashboard overflowing if there were too many options.
• Filter selections are now maintained while paginating in interact.
• Align icons in interface when item is not found in catalog.

Merchandizing

• Item-level blacklisting We now allow removing specific items in browse, search and collections merchandizing! If there's a particular item you want to exclude from a browse or collections page, but you don't want to update your catalog or can't target it with a category-based rule, you can now remove just the one item.
  
  
  

Quality of life

• Navigating away from the homepage and returning will preserve state of last-selected tabs.
• Interact left and right scrollbars are now independent, simplifying facet interaction and result previewing!
• Clicking on product names anywhere in the app will now display our new sidebar UX (with in-depth data and API data) everywhere items are shown to users.
  
  
  

User accounts

• Send an email to users when they are deleted, noting the company from which they were deleted and the user who deleted them.
• Improved discoverability of user deletion (accessible only to admins).

API changes

• Allow sending both item name and item id in v2 behavioral endpoints to simplify troubleshooting.

Fixes

• Fixed a display glitch in collections confirmation modal warning of unsaved changes.
• Allow editing of facet options when there are more than 1,000 options for a given facet.