Building recommender Systems using contextual bandit methods to address cold-start issue and online real-time learning
Thompson Sampling, Single-user Multi-product Simulation, Multi-armed Bandit
The objective of this app is to apply the bandit algorithms to recommendation problem under a simulated envrionment. Although in practice we would also use the real data, the complexity of the recommendation problem and the associated algorithmic challenges can already be revealed even in this simple setting.
Inspired by the following works:
Multi-user Multi-product Contextual Simulation, Contextual Bandit, Vowpal Wabbit
The objective of this app is to apply the contextual bandit algorithms to recommendation problem under a simulated envrionment. The recommender agent is able to quickly adapt the changing bahavior of users and change the recommendation strategy accordingly.
Image Embeddings, Offline Learning
The objective is to recommend products and adapt the model in real-time using user's feedback using Actor-critic algorithm. Suppose, we observed users’ behavior and acquired some products they clicked on. It is fed into the Actor Network which decides what we would like to read next. It produces an ideal product embedding. It can be compared with other product embeddings to find similarities. The most matching one will be recommended to the user. The Critic helps to judge the Actor and help it find out what is wrong.
Inspired by the following works:
Offline Learning
The core intuition is that we couldn't just blindly apply RL algorithms in a production system out of the box. The learning period would be too costly. Instead, we need to leverage the vast amounts of offline training examples to make the algorithm perform as good as the current system before releasing into the online production environment. An agent is first given access to many offline training examples produced from a fixed policy. Then, they have access to the online system where they choose the actions.
Inspired by the following works:
Traditionally, the recommendation problem was considered as a simple classification or prediction problem; however, the sequential nature of the recommendation problem has been shown. Accordingly, it can be formulated as a Markov decision process (MDP) and reinforcement learning (RL) methods can be employed to solve it. In fact, recent advances in combining deep learning with traditional RL methods, i.e. deep reinforcement learning (DRL), has made it possible to apply RL to the recommendation problem with massive state and action spaces.
Goal: Quickly help users find products they would like to buy
In e-commerce and other digital domains, companies frequently want to offer personalised product recommendations to users. This is hard when you don’t yet know a lot about the customer, or you don’t understand what features of a product are pertinent. With limited information about what actions to take, what their payoffs will be, and limited resources to explore the competing actions that you can take, it is hard to know what to do.
Goal: Compare and find the best performing recommender model
Goal: Bring the most relevant option to the top
Goal: Recommend a never-ending feed of items (news, products, images, music)
- LinUCB Contextual News Recommendation
- Experiment with Bandits
- n-armed Bandit Recommender
- Bandit Algorithms for Website Optimization [eBook O’reilly] [GitHub] [Colab]
- MAB Ranking PyPi
- RecSim GitHub, Video, Medium
- https://vowpalwabbit.org/tutorials/contextual_bandits.html
- https://github.com/sadighian/recommendation-gym
- https://learning.oreilly.com/library/view/reinforcement-learning-pocket/9781098101527/ch02.html
- https://github.com/awarebayes/RecNN/
- https://vowpalwabbit.org/neurips2019/
- https://github.com/criteo-research/reco-gym
- https://pypi.org/project/SMPyBandits/
- https://github.com/bgalbraith/bandits
- https://pypi.org/project/mab-ranking/
- https://www.optimizely.com/optimization-glossary/multi-armed-bandit/
- https://abhishek-maheshwarappa.medium.com/multi-arm-bandits-for-recommendations-and-a-b-testing-on-amazon-ratings-data-set-9f802f2c4073