Ribbit
What goes into a chatbot architecture? Well, here’s a description of the one running Fraser the Bot.
User -> Instant message server -> fraserthebot.com
Fraser the Bot currently works with five different instant messaging applications. Each instant messaging app has its own API, and they all have to be configured with the URL where they are expected to send messages. The URL must be HTTPS. My servers are running on Amazon Web Services.
Load Balancer -> Core server
The first stop on AWS is at the load balancer. The load balancer hands the request (the message) off to one of Fraser’s core servers, each of which is a Linux virtual machine instance. There can be any number of core servers up and running as necessary to deal with increasing user load.
Core server
Fraser is using a Python web server framework called Tornado for handling requests. The web request handlers and Fraser’s core functions sit within a running Tornado instance. The core server functions are all written in Python.
Request properties
Each instant messaging app sends requests to its configured URL on fraserthebot.com, and each URL is handled by a different request handler function. Each of those request handlers has to parse the arguments out of the request (sometimes HTTP POST, sometimes HTTP GET). Usually the request body is JSON, and the request arguments always include an ID for the chat group, an ID for the individual sender (one user within the chat group), and the sender’s message text. Each instant messaging app has its own additional message properties, but those three properties are common to all.
Core server -> Database server
With the chat group ID and sender ID, the core server has enough information to call a PostgreSQL database server instance, which is also running on Amazon Web Services. This database query goes to a server side function, which deals with creating records for new chats and gathering chat session information from a few different tables. There may be many core servers, but there is only one database server.
Core server -> Network file system server
Most messages to Fraser require a bit of natural language processing. Each Wikipedia article has been processed and the data results from that processing stored on a network filesystem server on AWS. As much as possible, the article data is cached in memory in the core server, but when there is a cache miss, the data is fetched from the network file system. There may be many core servers, but there is only one network filesystem server.
Core server -> Instant message server
Each user message to Fraser may result in one or more instant messages being sent from Fraser to the user or chat group.
Core server -> Database server
Another call to the database server is made, to store the updated chat session information. This is how the chat session state (the game play state) is preserved from one message to the next. The state is stored in the one database server, rather than on the core servers. That way, the load balancer may send an individual player’s messages to an arbitrary core server and game play works as expected.
Asynchronous I/O
Calls from the core server to the database, to the network file system, and to the instant messaging server are all done in an asynchronous fashion. That is, the I/O call is launched without sitting and waiting for the response. The core server immediately gets back to work dealing with other requests. Once the I/O call finishes, the core server returns to the task which triggered the I/O call in the first place.
What’s missing?
For one, a message queue. Some of the instant message servers expect the bot server to respond within five seconds. Fraser’s server doesn’t take anywhere near that long to respond, but if things got bogged down, some instant message server requests might time out. A message queue would allow the core server to respond to the HTTP requests immediately by storing the message on the queue rather than processing it right then. What else? An in-memory database like Redis, running on another AWS server, would allow the core servers to fetch all the article data from that fast, additional layer of cache rather than from the network file server.
Good grief. There must be an easier way.
Certainly! Have a look at Microsoft’s Bot Framework. It has come a very long way since I started this project. In other words, if I were starting a new project today I might do things differently. 🙂