Complete Technical Paper Content — Ai Hybrid Recommendation
1. Executive Abstract, Problem Formulation & Background
A recommender system, also called a recommendation engine or content discovery platform is a type of information filtering system that aims to suggest items most relevant for some input or to a particular user. In the context of social media, search engines, and other online services, a recommender system for a given service might sometimes informally but erroneously be referred to as the service's "algorithm". The use of recommender systems is pervasive, with commonly recognised examples including the generation of playlists for video and music streaming services, product recommendations for e-commerce platforms, and topics and individual items posted on social media platforms and the open web. Online services employ machine learning or specialized deep learning recommendation models (DLRMs) that analyze user behavior and preferences to generate personalized content feeds for millions of users at any moment. The value of these systems becomes particularly evident in scenarios where users must select from a large number of options, such as products, media, or content. Suggestions are typically designed to shorten or work as a substitute for decision-making processes, including the selection of a product, musical selection, or online news source to read.
Last.fm creates a "station" of recommended songs by observing what bands and individual tracks the user has listened to on a regular basis and comparing those against the listening behavior of other users. Last.fm will play tracks that do not appear in the user's library, but are often played by other users with similar interests. As this approach leverages the behavior of users, it is an example of a collaborative filtering technique. Pandora uses the properties of a song or artist (a subset of the 450 attributes provided by the Music Genome Project) to seed a "station" that plays music with similar properties. User feedback is used to refine the station's results, deemphasizing certain attributes when a user "dislikes" a particular song and emphasizing other attributes when a user "likes" a song. This is an example of a content-based approach. Each type of system has its strengths and weaknesses. In the above example, Last.fm requires a large amount of information about a user to make accurate recommendations. This is an example of the cold start problem, and is common in collaborative filtering systems. Whereas Pandora needs very little information to start, it is far more limited in scope (for example, it can only make recommendations that are similar to the original seed).
and implemented at scale and worked through in technical reports and publications from 1994 onwards by Jussi Karlgren, then at SICS, and research groups led by Pattie Maes at MIT, Will Hill at Bellcore, and Paul Resnick, also at MIT, whose work with GroupLens was awarded the 2010 ACM Software Systems Award. Montaner provided the first overview of recommender systems from an intelligent agent perspective. Gediminas Adomavicius provided a new, alternate overview of recommender systems. Herlocker provides an additional overview of evaluation techniques for recommender systems, and Beel et al. discussed the problems of offline evaluations. Beel et al. have also provided literature surveys on available research paper recommender systems and existing challenges.
One approach to the design of recommender systems that has wide use is collaborative filtering. Collaborative filtering is based on the assumption that people who agreed in the past will agree in the future, and that they will like similar kinds of items as they liked in the past. The system generates recommendations using only information about rating profiles for different users or items. By locating peer users/items with a rating history similar to the current user or item, they generate recommendations using this neighborhood. The collaborative filtering approach does not rely on machine analyzable content and therefore it is capable of accurately recommending complex items such as movies without requiring an "understanding" of the item itself. Many algorithms have been used in measuring user similarity or item similarity in recommender systems. For example, the k-nearest neighbor (k-NN) approach and the Pearson Correlation as first implemented by Allen. Collaborative filtering approaches often suffer from three problems: cold start, scalability, and sparsity.
Cold start: For a new user or item, there is not enough data to make accurate recommendations. Note: one commonly implemented solution to this problem is the multi-armed bandit algorithm. Scalability: There are millions of users and products in many of the environments in which these systems make recommendations. Thus, a large amount of computation power is often necessary to calculate recommendations. Sparsity: The number of items sold on major e-commerce sites is extremely large. The most active users will only have rated a small subset of the overall database. Thus, even the most popular items have very few ratings. Social networking services have used collaborative filtering to recommend new friends, groups, and other social connections by examining the network of connections between a user and their friends.
2. Related Work, Literature Survey & Algorithmic Foundations
A model of the user's preference. A history of the user's interaction with the recommender system. Basically, these methods use an item profile (i.e., a set of discrete attributes and features) characterizing the item within the system. To abstract the features of the items in the system, an item presentation algorithm is applied. A widely used algorithm is the tf–idf representation (also called vector space representation). The system creates a content-based profile of users based on a weighted vector of item features. The weights denote the importance of each feature to the user and can be computed from individually rated content vectors using a variety of techniques. Simple approaches use the average values of the rated item vector while other sophisticated methods use machine learning techniques such as Bayesian Classifiers, cluster analysis, decision trees, and artificial neural networks in order to estimate the probability that the user is going to like the item. Watch time can be used as a primary indicator of whether the user likes a post, and the posts can be clustered and associated in several ways via social graphs and techniques similar to the concept of embeddings in machine learning. Posts can be ranked with a dedicated model, and ranking and recommendation models can be used together in a ranking and recommendation system. Recommendation systems can have two towers: one with the user profile as an embedding and another for items or posts also as embeddings. Both types of embeddings can be generated from users and posts by specialized embedding models and then the differences between the two types of embeddings can be computed with cosine difference to match users to items. Then a ranking model can be used to show a feed that is diverse as it is scrolled that does not cluster together similar posts. A key issue with content-based filtering is whether the system can learn user preferences from users' actions regarding one content source and use them across other content types. When the system is limited to recommending content of the same type as the user is already using, the value from the recommendation system is significantly less than when other content types from other services can be recommended. For example, recommending news articles based on news browsing is useful. Still, it would be much more useful when music, videos, products, discussions, etc., from different services, can be recommended based on news browsing. To overcome this, most content-based recommender systems now use some form of the hybrid system. Content-based recommender systems can also include opinion-based recommender systems. In some cases, users are allowed to leave text reviews or feedback on the items. These user-generated texts are implicit data for the recommender system because they are potentially rich resources of both feature/aspects of the item and users' evaluation/sentiment to the item. Features extracted from the user-generated reviews are improved metadata of items, because as they also reflect aspects of the item like metadata, extracted features are widely concerned by the users. Sentiments extracted from the reviews can be seen as users' rating scores on the corresponding features. Popular approaches of opinion-based recommender system utilize various techniques including text mining, information retrieval, sentiment analysis (see also Multimodal sentiment analysis) and deep learning.
Weighted: Combining the score of different recommendation components numerically. Switching: Choosing among recommendation components and applying the selected one. Mixed: Recommendations from different recommenders are presented together to give the recommendation. Cascade: Recommenders are given strict priority, with the lower priority ones breaking ties in the scoring of the higher ones. Meta-level: One recommendation technique is applied and produces some sort of model, which is then the input used by the next technique.
Mobile recommender systems make use of internet-accessing smartphones to offer personalized, context-sensitive recommendations. This is a particularly difficult area of research as mobile data is more complex than data that recommender systems often have to deal with. It is heterogeneous, noisy, requires spatial and temporal auto-correlation, and has validation and generality problems. There are three factors that could affect the mobile recommender systems and the accuracy of prediction results: the context, the recommendation method and privacy. Additionally, mobile recommender systems suffer from a transplantation problem – recommendations may not apply in all regions (for instance, it would be unwise to recommend a recipe in an area where all of the ingredients may not be available). One example of a mobile recommender system are the approaches taken by companies such as Uber and Lyft to generate driving routes for taxi drivers in a city. This system uses GPS data of the routes that taxi drivers take while working, which includes location (latitude and longitude), time stamps, and operational status (with or without passengers). It uses this data to recommend a list of pickup points along a route, with the goal of optimizing occupancy times and profits.
One of the events that energized research in recommender systems was the Netflix Prize. From 2006 to 2009, Netflix sponsored a competition, offering a grand prize of $1,000,000 to the team that could take an offered dataset of over 100 million movie ratings and return recommendations that were 10% more accurate than those offered by the company's existing recommender system. This competition energized the search for new and more accurate algorithms. On September 21, 2009, the grand prize of US$1,000,000 was given to the BellKor's Pragmatic Chaos team using tiebreaking rules. The most accurate algorithm in 2007 used an ensemble method of 107 different algorithmic approaches, blended into a single prediction. As stated by the winners, Bell et al.:
Predictive accuracy is substantially improved when blending multiple predictors. Our experience is that most efforts should be concentrated in deriving substantially different approaches, rather than refining a single technique. Consequently, our solution is an ensemble of many methods. Many benefits accrued to the web due to the Netflix project. Some teams have taken their technology and applied it to other markets. Some members from the team that finished second place founded Gravity R&D, a recommendation engine that's active in the RecSys community. 4-Tell, Inc. created a Netflix project–derived solution for ecommerce websites. A number of privacy issues arose around the dataset offered by Netflix for the Netflix Prize competition. Although the data sets were anonymized in order to preserve customer privacy, in 2007 two researchers from the University of Texas were able to identify individual users by matching the data sets with film ratings on the Internet Movie Database (IMDb). As a result, in December 2009, an anonymous Netflix user sued Netflix in Doe v. Netflix, alleging that Netflix had violated United States fair trade laws and the Video Privacy Protection Act by releasing the datasets. This, as well as concerns from the Federal Trade Commission, led to the cancellation of a second Netflix Prize competition in 2010.
3. Dataset Ingestion, Preprocessing & Stochastic Augmentations
Diversity – Users tend to be more satisfied with recommendations when there is a higher intra-list diversity, e.g. items from different artists. Recommender persistence – In some situations, it is more effective to re-show recommendations, or let users re-rate items, than showing new items. There are several reasons for this. Users may ignore items when they are shown for the first time, for instance, because they had no time to inspect the recommendations carefully. Privacy – Recommender systems usually have to deal with privacy concerns because users have to reveal sensitive information. Building user profiles using collaborative filtering can be problematic from a privacy point of view. Many European countries have a strong culture of data privacy, and every attempt to introduce any level of user profiling can result in a negative customer response. Much research has been conducted on ongoing privacy issues in this space. The Netflix Prize is particularly notable for the detailed personal information released in its dataset. Ramakrishnan et al. have conducted an extensive overview of the trade-offs between personalization and privacy and found that the combination of weak ties (an unexpected connection that provides serendipitous recommendations) and other data sources can be used to uncover identities of users in an anonymized dataset. User demographics – Beel et al. found that user demographics may influence how satisfied users are with recommendations. In their paper they show that elderly users tend to be more interested in recommendations than younger users. Robustness – When users can participate in the recommender system, the issue of fraud must be addressed. Serendipity – Serendipity is a measure of "how surprising the recommendations are". For instance, a recommender system that recommends milk to a customer in a grocery store might be perfectly accurate, but it is not a good recommendation because it is an obvious item for the customer to buy. "[Serendipity] serves two purposes: First, the chance that users lose interest because the choice set is too uniform decreases. Second, these items are needed for algorithms to learn and improve themselves". Trust – A recommender system is of little value for a user if the user does not trust the system. Trust can be built by a recommender system by explaining how it generates recommendations, and why it recommends an item. Labelling – User satisfaction with recommendations may be influenced by the labeling of the recommendations. For instance, in the cited study click-through rate (CTR) for recommendations labeled as "Sponsored" were lower (CTR=5.93%) than CTR for identical recommendations labeled as "Organic" (CTR=8.86%). Recommendations with no label performed best (CTR=9.87%) in that study.
Artificial intelligence (AI) applications in recommendation systems are the advanced methodologies that leverage AI technologies, to enhance the performance recommendation engines. The AI-based recommender can analyze complex data sets, learning from user behavior, preferences, and interactions to generate highly accurate and personalized content or product suggestions. The integration of AI in recommendation systems has marked a significant evolution from traditional recommendation methods. Traditional methods often relied on inflexible algorithms that could suggest items based on general user trends or apparent similarities in content. In comparison, AI-powered systems have the capability to detect patterns and subtle distinctions that may be overlooked by traditional methods. These systems can adapt to specific individual preferences, thereby offering recommendations that are more aligned with individual user needs. This approach marks a shift towards more personalized, user-centric suggestions. Recommendation systems widely adopt AI techniques such as machine learning, deep learning, and natural language processing. These advanced methods enhance system capabilities to predict user preferences and deliver personalized content more accurately. Each technique contributes uniquely. The following sections will introduce specific AI models utilized by a recommendation system by illustrating their theories and functionalities.
Data Representation: Create a n-dimensional space where each axis represents a user's trait (ratings, purchases, etc.). Represent the user as a point in that space. Statistical Distance: 'Distance' measures how far apart users are in this space. See statistical distance for computational details Identifying Neighbors: Based on the computed distances, find k nearest neighbors of the user to which we want to make recommendations Forming Predictive Recommendations: The system will analyze the similar preference of the k neighbors. The system will make recommendations based on that similarity
Time and Seasonality: what specify time and date or a season that a user interacts with the platform User Navigation Patterns: sequence of pages visited, time spent on different parts of a website, mouse movement, etc. External Social Trends: information from outer social media
User Tower: Encodes user-specific features, such as interaction history or demographic data. Item Tower: Encodes item-specific features, such as metadata or content embeddings. The outputs of the two towers are fixed-length embeddings that represent users and items in a shared vector space. A similarity metric, such as dot product or cosine similarity, is used to measure relevance between a user and an item. This model is highly efficient for large datasets as embeddings can be pre-computed for items, allowing rapid retrieval during inference. It is often used in conjunction with ranking models for end-to-end recommendation pipelines.
4. Neural Network Backbone, Layer Specifications & Loss Formulations
Multimodal learning is a type of deep learning that integrates and processes multiple types of data, referred to as modalities, such as text, audio, images, or video. This integration allows for a more holistic understanding of complex data, improving model performance in tasks like visual question answering, cross-modal retrieval, text-to-image generation, aesthetic ranking, and image captioning. Multimodal learning was proposed in 2011 at the beginning of the deep learning period. Large multimodal models, such as Google Gemini and GPT-4o, have become increasingly popular since 2023, enabling increased versatility and a broader understanding of real-world phenomena.
Models such as CLIP (Contrastive Language–Image Pretraining) learn joint representations of images and text by optimizing contrastive objectives, allowing the model to match images with their corresponding textual descriptions.
Cross-modal retrieval: cross-modal retrieval allows users to search for data across different modalities (e.g., retrieving images based on text descriptions), improving multimedia search engines and content recommendation systems. Classification and missing data retrieval: multimodal Deep Boltzmann Machines outperform traditional models like support vector machines and latent Dirichlet allocation in classification tasks and can predict missing data in multimodal datasets, such as images and text. Healthcare diagnostics: multimodal models integrate medical imaging, genomic data, and patient records to improve diagnostic accuracy and early disease detection, especially in cancer screening. Content generation: models like DALL·E generate images from textual descriptions, benefiting creative industries, while cross-modal retrieval enables dynamic multimedia searches. Robotics and human-computer interaction: multimodal learning improves interaction in robotics and AI by integrating sensory inputs like speech, vision, and touch, aiding autonomous systems and human-computer interaction. Emotion recognition: combining visual, audio, and text data, multimodal systems enhance sentiment analysis and emotion recognition, applied in customer service, social media, and marketing. Code search: multimodal representations combining source code structure (e.g., abstract syntax trees) with natural language descriptions improve retrieval of relevant code snippets for a given query. Source code has also been rendered as a two-dimensional pixel image, for instance by converting a method's control-flow graph into a black-and-white image processed with a convolutional neural network to search for malwares.
Hybrid Recommendation System Overview The objective of this notebook is to create news_recommend_dataset and build a hybrid recommedation system to recommend news for Austrian news website Kurier.at . Given the information of the current news article, the goal is to predict the next news article the visitor would like to read. First, use stadard SQL to query dataset and extract content-based features from Google analytics data in BigQuery. Next, extract embeddings from neural collaborative filtering to get user and item latent factors. After combining content-based features with latent factors, we employ wide & deep architecture to build a hybrid recommendation system and recommend the next news article for readers. The hybrid model is trained in gcloud ai-platform, and a gcloud model is created to allow batch prediction or online prediction with REST API. (This project is implemented in python 3 and tensorflow 2 and is only for self-practice purpose.) Dependencies python 3.6.9 tensorflow 2.1.0 BigQuery Google Coud Platform Dataset cloud-training-demos.GA360_test.ga_sessions_sample : Public BigQuery dataset which contains Google analytics data from Austrian news website Kurier.at news_recommend_dataset : Dataset combining preprocessed content-based features with user and item latent factors from neural collaborative filtering. How to run Run preprocess.ipynb to preprocess dataset in BigQuery Run neural_colaborate_filter_train.ipynb to extract user and item latent factors using neural collaborative filtering Run hybrid_recsys_train.ipynb for creating hybrid model and perform batch and online predictions Paper The implementation is related to the following papers: Neural Collaborative Filtering Wide & Deep Learning for Recommender Systems Implementation The neural collaborative filtering is implemented using the below network architecture. It takes user id and item id to predict rating for the item. The hybrid model is implemented using the below network architecture. It takes content-based features and latent factors to predict the probability of each next items which the user may like to read. Result The traing job log file is shown below. The final test result is loss: 4.471430, acc: 9.715167, top_10_acc: 42.097153. The top 10 accuracy is around 42.3%, which means our model has 42.3% chance to correcly predict the next news article the visitor would like to view if our hybrid recommendation model recommend 10 items. While randomly picking 10 items from total 2421 news articles, the top 10 accuracy would only be 0.413%. Our modle is 100 times better in top 10 accuracy than random picking. The tensorboard of the training process is shown below Slides hybrid_recsys_slides Reference This project is inspired by the following resources: Neural Collaborative Filtering Wide & Deep Learning for Recommender Systems Recommendation Systems with TensorFlow on GCP End-to-end Machine Learning with TensorFlow on GCP Collaborative Filtering using Deep Neural Networks (in Tensorflow) Get started with TensorBoard Deploying models Method: projects.predict
The objective of this notebook is to create news_recommend_dataset and build a hybrid recommedation system to recommend news for Austrian news website Kurier.at . Given the information of the current news article, the goal is to predict the next news article the visitor would like to read. First, use stadard SQL to query dataset and extract content-based features from Google analytics data in BigQuery. Next, extract embeddings from neural collaborative filtering to get user and item latent factors. After combining content-based features with latent factors, we employ wide & deep architecture to build a hybrid recommendation system and recommend the next news article for readers. The hybrid model is trained in gcloud ai-platform, and a gcloud model is created to allow batch prediction or online prediction with REST API.
5. Training Protocol, Convergence Behavior & Experimental Evaluation
(This project is implemented in python 3 and tensorflow 2 and is only for self-practice purpose.)
cloud-training-demos.GA360_test.ga_sessions_sample : Public BigQuery dataset which contains Google analytics data from Austrian news website Kurier.at
news_recommend_dataset : Dataset combining preprocessed content-based features with user and item latent factors from neural collaborative filtering.
recommender-system machine-learning ensemble-learning vertex-ai google-cloud mlops hybrid-model collaborative-filtering content-based-filtering deep-learning
Intelligent Shopper's Assistant: A Hybrid Product Recommendation System Overview This project is a comprehensive implementation of an advanced product recommendation system for e-commerce. It demonstrates a multi-layered approach, starting from simple baseline models and progressing to a sophisticated, explainable hybrid engine. The system leverages user ratings, product metadata, and the sentiment of review text to provide relevant, diverse, and transparent recommendations. This project is designed to showcase proficiency in core machine learning concepts, natural language processing (NLP), and MLOps principles like creating reproducible environments and serving models via an API. 🚀 Features Popularity-Based Model: A baseline model that recommends the most reviewed items. This is effective for new users (solving the "cold start" problem). Sentiment-Aware Collaborative Filtering: A powerful "Users who liked this also liked..." model. It goes beyond simple ratings by analyzing the sentiment of review text using NLTK's VADER , creating a more nuanced "adjusted rating" that better reflects user satisfaction. Content-Based Filtering: Recommends items based on their attributes ("Because you liked a product with these features..."). It uses TF-IDF to vectorize product descriptions and Cosine Similarity to find similar items. Advanced Hybrid Engine: Intelligently combines the recommendations from all models. It prioritizes personalized results from collaborative and content-based filtering and uses the popularity model as a robust fallback, ensuring a diverse and relevant final list. Explainable AI (XAI): A key feature that builds user trust. Every recommendation is delivered with a human-readable explanation, detailing why the item is being suggested (e.g., "Recommended because it shares features like {'coffee', 'organic'}..." or "Recommended because users who liked product X also liked this item."). 🛠️ Tech Stack Language: Python Core Libraries: Data Manipulation: Pandas, NumPy Machine Learning: Scikit-learn (for TF-IDF, Cosine Similarity) Scientific Computing: SciPy (for Sparse Matrices) Natural Language Processing (NLP): NLTK (for VADER Sentiment Analysis) Deployment: Flask (for creating a simple REST API endpoint) Development Environment: VS Code, Virtual Environments (venv) 📂 Project Structure /product-recommendation-system |-- recsys_env/ # Virtual environment |-- app.py # Main application, hybrid logic, and Flask API |-- collaborative_filtering_model.py |-- content_based_model.py |-- popularity_model.py |-- item2vec_model.py # Conceptual code for a deep learning approach |-- Reviews.csv # The dataset |-- README.md # This documentation |-- requirements.txt # Project dependencies 📈 Future Work Implement Item2Vec: The item2vec_model.py contains the conceptual code for a deep learning approach using gensim . This would involve training a Word2Vec model on user purchase sequences to learn latent product embeddings, potentially capturing more complex relationships. Full-Fledged UI: Develop a front-end interface (e.g., using React or Streamlit) to interact with the recommendation API in a user-friendly way. Scalability: For larger datasets, migrate the data processing and model training to a distributed computing framework like Apache Spark.
Model Figures & Evaluation Curves (4 Images)
Deep Learning Execution Pipeline
(Preprocessing)
(Feature Extraction)
(Self-Attention)
(Loss Optimization)