How to Build CHATBOT Project using PYTHON?

• What is this project all about?
  In this project with source code, I built a ChatBot using Deep Learning techniques. This ChatBot was trained on a dataset which contains categories (intents), pattern and responses. You can use a special recurrent neural network (LSTM) to classify which category the user’s message belongs to and then give a random response from the list of responses.
• My DataSet
  I used the ‘intents.json’ dataset. This is a JSON file that contains the patterns I needed to find and the responses I wanted to return to the user. You can also download the dataset: Chatbot Project dataset

STEP NO. 01Importing and Loading the Data File

Firstly, I made a file named as train_chatbot.py. I imported the necessary packages for my ChatBot and initialized the variables that I will use later in the project.

The data file is in JSON format so I used the JSON package to parse the JSON file into Python.

1. import nltk
2. from nltk.stem import WordNetLemmatizer
3. lemmatizer = WordNetLemmatizer()
4. import json
5. import pickle
6. import numpy as np
7. from keras.models import Sequential
8. from keras.layers import Dense, Activation, Dropout
9. from keras.optimizers import SGD
10. import random
11. words=[]
12. classes = []
13. documents = []
14. ignore_words = ['?', '!']
15. data_file = open('intents.json').read()
16. intents = json.loads(data_file)

Here’s a snap of the intent.json file.

.

STEP NO. 02Pre-processing of Data

When working with text data, I performed various preprocessing on the data before I made a Machine Learning or a Deep Learning model. Tokenizing is the most basic and first thing you can do on text data. Tokenizing is the process of breaking the whole text into small parts like words.

.

So I iterated through the patterns and tokenized the sentence using nltk.word_tokenize() function and append each word in the words list. I also created a list of classes for my tags.

1. for intent in intents['intents']:
2. for pattern in intent['patterns']:
3. #tokenize each word
4. w = nltk.word_tokenize(pattern)
5. words.extend(w)
6. #add documents in the corpus
7. documents.append((w, intent['tag']))
8. # add to our classes list
9. if intent['tag'] not in classes:
10. classes.append(intent['tag'])

.

After this, I lemmatized each word and remove duplicate words from the list. Lemmatizing is the process of converting a word into its lemma form and then creating a pickle file to store the Python objects which I used while predicting.

1. # lemmatize, lower each word and remove duplicates
2. words = [lemmatizer.lemmatize(w.lower()) for w in words if w not in ignore_words]
3. words = sorted(list(set(words)))
4. # sort classes
5. classes = sorted(list(set(classes)))
6. # documents = combination between patterns and intents
7. print (len(documents), "documents")
8. # classes = intents
9. print (len(classes), "classes", classes)
10. # words = all words, vocabulary
11. print (len(words), "unique lemmatized words", words)
12. pickle.dump(words,open('words.pkl','wb'))
13. pickle.dump(classes,open('classes.pkl','wb'))

So, it was time to create the training data in which I provided the input and the output. My input will be the pattern and output will be the class my input pattern belongs to. But the computer doesn’t understand the text so I converted text into numbers.

1. # create our training data
2. training = []
3. # create an empty array for our output
4. output_empty = [0] * len(classes)
5. # training set, bag of words for each sentence
6. for doc in documents:
7. # initialize our bag of words
8. bag = []
9. # list of tokenized words for the pattern
10. pattern_words = doc[0]
11. # lemmatize each word - create base word, in attempt to represent related words
12. pattern_words = [lemmatizer.lemmatize(word.lower()) for word in pattern_words]
13. # create our bag of words array with 1, if word match found in current pattern
14. for w in words:
15. bag.append(1) if w in pattern_words else bag.append(0)
16. # output is a '0' for each tag and '1' for current tag (for each pattern)
17. output_row = list(output_empty)
18. output_row[classes.index(doc[1])] = 1
19. training.append([bag, output_row])
20. # shuffle our features and turn into np.array
21. random.shuffle(training)
22. training = np.array(training)
23. # create train and test lists. X - patterns, Y - intents
24. train_x = list(training[:,0])
25. train_y = list(training[:,1])
26. print("Training data created")




STEP NO. 04Building the Model
So my data is ready, then I built a Deep Neural Network that had 3 layers. I used the Keras sequential API for this. After training the model for 200 epochs, I achieved 100% accuracy on my model.
This is how I saved the model as ‘chatbot_model.h5’.
1. # Create model - 3 layers. First layer 128 neurons, second layer 64 neurons and 3rd output layer contains number of neurons
2. # equal to number of intents to predict output intent with softmax
3. model = Sequential()
4. model.add(Dense(128, input_shape=(len(train_x[0]),), activation='relu'))
5. model.add(Dropout(0.5))
6. model.add(Dense(64, activation='relu'))
7. model.add(Dropout(0.5))
8. model.add(Dense(len(train_y[0]), activation='softmax'))
9. # Compile model. Stochastic gradient descent with Nesterov accelerated gradient gives good results for this model
10. sgd = SGD(lr=0.01, decay=1e-6, momentum=0.9, nesterov=True)
11. model.compile(loss='categorical_crossentropy', optimizer=sgd, metrics=['accuracy'])
12. #fitting and saving the model
13. hist = model.fit(np.array(train_x), np.array(train_y), epochs=200, batch_size=5, verbose=1)
14. model.save('chatbot_model.h5', hist)
15. print("model created")
.
STEP NO. 05Predicting the Response
I loaded the trained model and then used a Graphical User Interface (GUI) that predicted the response from the bot. The model will only tell me the class it belongs to, so I implemented some functions which will identify the class and then retrieve me a random response from the list of responses.
Now I imported the necessary packages and loaded the ‘words.pkl’ and ‘classes.pkl’ pickle files which I created when I trained my model.
1. import nltk
2. from nltk.stem import WordNetLemmatizer
3. lemmatizer = WordNetLemmatizer()
4. import pickle
5. import numpy as np
6. from keras.models import load_model
7. model = load_model('chatbot_model.h5')
8. import json
9. import random
10. intents = json.loads(open('intents.json').read())
11. words = pickle.load(open('words.pkl','rb'))
12. classes = pickle.load(open('classes.pkl','rb'))
.
To predict the class, I provided input in the same way as I did while training. So I created some functions that will perform text preprocessing and then predict the class.
1. def clean_up_sentence(sentence):
2. # tokenize the pattern - split words into array
3. sentence_words = nltk.word_tokenize(sentence)
4. # stem each word - create short form for word
5. sentence_words = [lemmatizer.lemmatize(word.lower()) for word in sentence_words]
6. return sentence_words
7. # return bag of words array: 0 or 1 for each word in the bag that exists in the sentence
8. def bow(sentence, words, show_details=True):
9. # tokenize the pattern
10. # bag of words - matrix of N words, vocabulary matrix
11. bag = [0]*len(words)
12. for s in sentence_words:
13. for i,w in enumerate(words):
14. if w == s:
15. # assign 1 if current word is in the vocabulary position
16. bag[i] = 1
17. if show_details:
18. print ("found in bag: %s" % w)
19. return(np.array(bag))
20. def predict_class(sentence, model):
21. # filter out predictions below a threshold
22. p = bow(sentence, words,show_details=False)
23. res = model.predict(np.array([p]))[0]
24. ERROR_THRESHOLD = 0.25
25. results = [[i,r] for i,r in enumerate(res) if r>ERROR_THRESHOLD]
26. # sort by strength of probability
27. results.sort(key=lambda x: x[1], reverse=True)
28. return_list = []
29. for r in results:
30. return_list.append({"intent": classes[r[0]], "probability": str(r[1])})
31. return return_list

After predicting the class, I got a random response from the list of intents.

1. def getResponse(ints, intents_json):
2. tag = ints[0]['intent']
3. list_of_intents = intents_json['intents']
4. for i in list_of_intents:
5. if(i['tag']== tag):
6. result = random.choice(i['responses'])
7. break
8. return result
9. def chatbot_response(text):
10. ints = predict_class(text, model)
11. res = getResponse(ints, intents)
12. return res

Finally, I coded a GUI. For this, I used the Tkinter library which already comes in Python. I took the input message from the user and then used the helper functions I created to get the response from the bot and display it on the GUI. You will also get the same response provided no error code is been run.

Here’s the response.

1. #Creating GUI with tkinter
2. import tkinter
3. from tkinter import *
4. def send():
5. msg = EntryBox.get("1.0",'end-1c').strip()
6. EntryBox.delete("0.0",END)
7. if msg != '':
8. ChatLog.config(state=NORMAL)
9. ChatLog.insert(END, "You: " + msg + '\n\n')
10. ChatLog.config(foreground="#442265", font=("Verdana", 12 ))
11. res = chatbot_response(msg)
12. ChatLog.insert(END, "Bot: " + res + '\n\n')
13. ChatLog.config(state=DISABLED)
14. ChatLog.yview(END)
15. base = Tk()
16. base.title("Hello")
17. base.geometry("400x500")
18. base.resizable(width=FALSE, height=FALSE)
19. #Create Chat window
20. ChatLog = Text(base, bd=0, bg="white", height="8", width="50", font="Arial",)
21. ChatLog.config(state=DISABLED)
22. #Bind scrollbar to Chat window
23. scrollbar = Scrollbar(base, command=ChatLog.yview, cursor="heart")
24. ChatLog['yscrollcommand'] = scrollbar.set
25. #Create Button to send message
26. SendButton = Button(base, font=("Verdana",12,'bold'), text="Send", width="12", height=5,
27. bd=0, bg="#32de97", activebackground="#3c9d9b",fg='#ffffff',
28. command= send )
29. #Create the box to enter message
30. EntryBox = Text(base, bd=0, bg="white",width="29", height="5", font="Arial")
31. #EntryBox.bind("<Return>", send)
32. #Place all components on the screen
33. scrollbar.place(x=376,y=6, height=386)
34. ChatLog.place(x=6,y=6, height=386, width=370)
35. EntryBox.place(x=128, y=401, height=90, width=265)
36. SendButton.place(x=6, y=401, height=90)
37. base.mainloop()

STEP NO. 06Finally running the ChatBot

  First, I trained the model using the command in the terminal:

python train_chatbot.py

.

Fortunately, I didn’t get any error. Same will be the result for you if you follow me exactly. So to run the app, I run the second file.

python chatgui.py

.

The program opened up a GUI window within a few seconds. With the GUI I easily chatted with the bot.

.

SCREENSHOTS

Thanks for giving your valuable time. If you find my content useful, then please follow this blog.

Any kind of suggestion for further improvement will be highly appreciated:)