Retailers on Fb and Instagram: Understanding relationships between merchandise to enhance purchaser and vendor expertise

This research in letter summarizes various projects carried out by co-authors Yaniv Sheena and Oren Sar Shalom together with their colleagues on the Relevance Foundations team at Meta.

What the research is:

In 2020 we launched shops on Facebook and Instagram to make it easier for businesses to set up a digital storefront and sell online. Shops currently have a huge inventory of products from different industries and different sellers, where the data provided is usually unstructured and multilingual and in some cases important information is missing.

Understanding the core features of these products and coding their relationships can help unlock a wide variety of ecommerce experiences, whether by recommending similar or complementary products on the product page or diversifying shopping feeds to avoid the same product is displayed multiple times. To open up these possibilities, we have set up a team of researchers and engineers in Tel-Aviv with the aim of creating a product diagram that takes different product relationships into account. The team has already rolled out features that are built into various Meta products.

Our research focuses on capturing and embedding different ideas about relationships between products. These methods are based on signals from the content of the products (text, image, etc.) as well as from past user interactions (e.g. collaborative filtering).

First, let’s tackle the problem of product deduplication, where we group duplicates or variants of the same product. Finding duplicates, or near duplicates, among billions of items is like looking for a needle in a haystack. For example, if a local store in Israel and a major brand in Australia sell the exact same shirt or variations of the same shirt (e.g. different colors), we group those products together. This is a challenge with the order of billions of products with varying images (some of poor quality), descriptions, and languages.

Next, we introduce Frequently Bought Together (FBT), an approach to product recommendation based on products that people buy together or with which they interact.

How it works:

Product clustering

We have developed a clustering platform that groups similar elements in real time. For each new item that is listed in the shops catalog, our algorithm assigns either an existing cluster or a new cluster.

This process consists of the following steps:

  • Product retrieval: We use an image index based on GrokNet visual embedding and text search based on an internal search backend powered by unicorn. We call up up to 100 similar products from an index of representative articles, which can be imagined as cluster focus.
  • Similarity in pairs: We compare the new item with each representative item using a paired model that predicts a similarity score given two products.
  • Element-to-cluster assignment: We choose the most similar product and apply a static threshold. When the threshold is reached, we assign the item. Otherwise we will create a new singleton cluster.

We specify two types of clustering rooms based on the business objectives:

  • Exact duplicates: Grouping instances of the same product
  • Product variants: Grouping variants of the same product (e.g. shirts in different colors or iPhones with different storage space)

For each type of clustering, we train a model tailored to the task at hand. The model is based on Gradient Boosted Decision Trees (GBDT) with a binary loss and uses both dense and sparse features. Among the functions we use GrokNet embedding cosine distance (image distance), LASER embedding distance (cross-language text display), text functions such as the Jaccard index and a tree-based distance between product taxonomies. This enables us to capture both visual and textual similarities while using signals such as brand and category. In addition, we also experimented with the SparseNN model, a deep model originally developed at Meta for personalization. It is designed to combine dense and sparse features to train a network end-to-end together by learning semantic representations for the sparse features. However, this model did not outperform the GBDT model, which is much lighter in terms of training time and resources.

Our models require training data sets for both clustering tasks: we send product pairs to human evaluators to put together sets for training, validation, and evaluation. Also, to get more relevant pairs with hard negatives, we use an active learning approach based on our existing retrieval mechanisms, followed by Uncertainty and Density Sampling (SUD).

To evaluate our approach, we have put together a set consisting of around 100,000 product pairs from the clothing & accessories, health & beauty and household sectors. Each pair was commented on by people who marked whether the two products were different, exact duplicates or variants. Then we measure precision and recall by using the steps above to infer whether the products would be in the same cluster. End results are determined by verticals, which usually have different characteristics.

Performance of pairwise similarity models: GBDT vs. SparseNN

System-level performance clustering by industry

Since combining different products can lead to an unsatisfactory user experience, we have coordinated our models with a focus on precision. The results suggest that we could solve a large part of the problem, but we still need to focus on improving the memory. Additionally, we found that health and beauty products were more challenging and required better understanding of the text.

Frequently Bought Together (FBT)

Analysis of past purchases shows that customers often search for several items in a short period of time, so that together they have a synergistic benefit. A notable example is a pair of jeans, along with a belt and possibly a matching shirt. When a customer is looking at a certain product (so-called seed product), our job is to help them find complementary products.

Arguably the most common way to find matching products is to simply count co-purchases. This means that we observe the (normalized) number of customers who bought the seed items and another candidate product shortly afterwards. If this amount exceeds a certain threshold, we say the candidate product is a good FBT recommendation for the seed product. With the ever-growing variety of products in the shops on Facebook and Instagram, however, there is always an abundance of new products that have not yet been bought in large quantities. Reducing the referral threshold creates an overwhelming amount of noise – and especially replacement items that are muddled by complementary items.

To fix this, we’re applying a two-step solution. First we work at the category level (more like the product level) to identify pairs of categories that fit together. This aggregation solves the problem of scarcity of purchases, and its output has been further verified by skilled taxonomists. Then we can resort to a simple number-based approach that sets a low threshold but only takes into account pairs that belong to related categories.

However, even with a low threshold, there are many products that are not covered by this method. To increase the coverage, we apply the following steps:

  • First we use the variant model and copy the recommendations of a product into its variants.
  • Second, we use a model that predicts the extent to which a pair of elements will complement each other based on their visual appearance.

As a training kit for this model, we need a list of products that go together. To do this, we go through fashion images and extract the products that appear, provided that products that appear in the same image make a good FBT recommendation.

To evaluate the performance of our approach, we conducted an experiment (A / B test) in which we suggested a number of complementary items (product page) to buyers considering a product. We compared our approach to a baseline (control) made up of suggestions handpicked by sellers. The FBT recommendation resulted in a relative improvement in click rate of 12 percent, demonstrating the feasibility and effectiveness of this approach.

Why it matters:

Our methods of integrating product similarities have improved various consumer-facing applications in shops. First, we introduced cluster-based post-ranking logic that diversifies product search results. We also showed that similarities based on deliberate user actions resulted in better recommendations compared to suggestions chosen by sellers. After all, we are constantly working with different teams in all shops to use our signals and improve relevance. Through intensive A / B testing, we’ve learned that capturing relationships between products is an important step in creating better user experiences.

What’s next:

We are currently developing a holistic model that simultaneously takes into account behavioral data such as co-views, co-purchases (different users viewing or buying the same product) and the preferences of the users who interacted with each item along with product information such as image, text description, Price and brand. These two types of modalities, buyer loyalty and product information, are learned in a mutually reinforcing manner, with one type of modality acting as a label for the other. Specifically, the behavior modality for a given seed product allows us to find two products, one of which makes a better recommendation than the other, whereby the side information can be learned using triplet loss. Likewise, the side information modality generates triplets, which make it possible to improve the behavioral characteristics.

Comments are closed.