Automating WhatsApp with NLP: Complete guide
We all know that coding is a superpower and can achieve numerous things with it. In this article we briefly review one of its application by ‘Building a chat bot using python’. Well, actually not only a chat bot but you can do a lot of cool stuffs with it. All you need is basic knowledge in python programming. P.S. This tutorial is only for educational purpose to demonstrate another application of Natural Language Processing (NLP) in Machine Learning using Python. Don’t blame us if your friends block you on whatsapp 😜.
We started this as a summer project during the quarantine period and now the source code along with a downloadable version of a pre release named ChatBot v0.1.0-alpha is available on GitHub. For training the bot we have used an NLP model based on word-2-vec and a Bayesian approach for updating the probabilities, originally developed by my roommate Aritra Banerjee a.k.a Arolive (find it on Arolive’s GitHub). The bot is still under development and it needs to learn a lot, but it can do the following things right now:
- Sending Automated replies on behalf of you — Let the ChatBot reply to your chats when you are too busy. Our algorithm will train the ChatBot to talk just like you do (both user & chat specific 😎 ).
- Sending Scheduled text messages to multiple chats — Never forget to wish your loved ones on their birthdays/anniversaries. Let the ChatBot send a scheduled text.
- Sending text messages iteratively — The ChatBot becomes handy in case if you wish to apologize to someone by sending ‘I am sorry’ a 100 times (couldn’t think of a better example than this! 😅)
- Clone a bot — You can create a customized ChatBot with a secondary account and chat with it, train it to talk like anyone!!
- Tell us if you can think of something else to do! After all necessity is the mother of all inventions.
Now if you are interested in the technical part and the ML implementation here is the thing you are searching for. We summarize the steps as follows:
Pr-requisites we have considered
Platform : Python 3.6
Packages/dependencies : Selenium 3.141.0 , beautifulsoup4 4.7.1, numpy 1.18.2, gensim 3.7.3, scipy 1.1.0 etc.
OS : linux/windows (simply because neither of us use a Mac)
Step 1 : Configuring the setup
First you need a browser. Our ChatBot works on both Firefox and Chromium (Firefox is recommended since Chromium does not seem to support BMP file emojis). Once you decide upon the browser you need to install the web driver for that,
For Firefox : Download Here
For Chromium : Download Here
Note that you will be needing the path of this web driver for accessing the browser using python. Once you have the web driver and browser ready, the Selenium package in python bridges the gap between the your local machine’s compiler and the website you wish to work on (for our case we will be using WhatsApp Web).
Step 2 : Building the NLP model
The main purpose of this article is to give the reader an overall idea about our proposed Natural Language Processing model. Our algorithm works on the following structure:
- The problem : Suppose we have a text corpus involving chats of 2 individuals U₁ and U₂. Further we want to automate the replies from U₂ corresponding to a particular text message(s) from U₁.
- Data processing : Let T₁ : {T₁₀, T₁₁, .. ,T₁ₘ} be m text messages from U₁ which is followed by n text messages T₂ : {T₂₀, T₂₁, .. ,T₂ₙ} from U₂. Our proposed algorithm combines the individual texts messages and establishes a One-2-One mapping between T₁ and T₂. This step also includes some pre-processing of the raw text corpus (for e.g. converting into lowercase, removing digits, removing blank spaces, removing selective punctuation etc). After processing we have One-2-One mapped set of strings, ready to feed the model.
Trending Bot Articles:
1. Chatbot Best Practices for designing a Conversational Experience
3. Chatbot software: Dominate the market (The Ultimate Guide)
4. How To Grow Your Business With The Help Of Artificial Intelligence
The main challenge with the model as well as the most interesting part was to work with a corpus that includes both English and local languages (Bengali in our case), where essentially the context of a word plays a major role. Hence, to obtain the vectorized string embedding, we have used the standard word-2-vec model, in order to take the semantics into consideration as well. Empirically the best results were obtained with window size 5.
- Predictive model : Let V₁ & V₂ be the corresponding word-2-vec embedding for T₁ & T₂. Also let mapper(.) defines the 1–1 function which maps V₁ to corresponding V₂, more precisely for a given input vector V₁ corresponding to U₁, mapper(.) returns the unique V₂ corresponding to U₂. Now for a new test input T₁* from U₁ we predict the corresponding T₂* as,
V₂* = mapper(argmin {cosine-distance (V₁*, V₁)})
Note that, the above algorithm ensures that the predicted reply should have occurred at least once in the corpus, and in this way it eliminates the chance of meaningless replies.
Although the actual implementation includes a Bayesian approach of updating the weights for each string embedding. We start with trivial prior assumption of equally probable replies of the texts, in other words each vector Vᵢ has a probability of 1/count(Vᵢ) of occurring in the corpus. Next each Vᵢ is multiplied with an appropriate weight given by α/(α + p(Vᵢ)) to emphasize the conditional posterior distribution of the texts over the whole corpus. The hyper-parameter value α is set to its empirically optimal choice 0.001. The function p() computes the empirical probabilities of occurrences of the text replies.
Step 3 : Lets code it
The above algorithm is pretty straight forward to code, thanks to Arolive for this (Take a peek at Arolive’s GitHub repository). The ChatBot is built with a thin wrapper around this model which is easily customizable.
Apart from the algorithm the website scrapping part requires a little familiarity with HTML and our life is made easy when we have “beautiful” parsers (beautifulsoup4 4.7.1). Note that scrapping a webpage is not always a legal act under certain terms and one should be careful about its uses (but I guess everything for fair in educational purposes 😛 ).
Future work & references:
- Till now the launcher function whatsapp.py partially has the above mentioned functionalities only, but there are numerous things that can be done with. Feel free to fork the repository and continue this project. Also we have listed several issues which can be good starting point for potential future developments.
- Please don’t hesitate if you have any comments/suggestions 😃 specially on the NLP model. Feel free to share your thoughts/views either by creating a pull request on the GitHub repository or contact me (homepage).
Don’t forget to give us your 👏 !
