A Socratic epistemology for verbal emotional intelligence

Introduction

Epistemology is the branch of philosophy that deals with knowledge and belief. According to basic results in epistemology, knowledge is defined as true, justified belief. This paper was inspired by reflecting on how humans justify their beliefs about emotions. This reflection led to an experimental method for collecting human knowledge about emotions and a computational model that uses the collected knowledge in an automated dialog agent.

The logician Charles S. Peirce identified three types of thought processes by which a person can justify their beliefs and thereby acquire knowledge: induction, deduction, and hypothesis (Peirce, 1868). Whereas induction is primarily involved with observational data, deduction and hypothesis have a linguistic, propositional component. The third of these, hypothesis (also known as abduction (Eco & Sebeok, 1988)), has been compared with the Socratic method of question-asking dialogs (Hintikka, 2007). The Socratic method was named after the ancient Greek philosopher Socrates who applied his method of inquiry to examine concepts that seem to lack any concrete definition, in particular some of the complex moral and psychological concepts of his time like “justice”, “knowledge”, “piety”, “temperance”, and “love”. We claim that this method of inquiry can shed light on how people justify beliefs about emotional concepts, which also seem to defy concrete definition.

Question-asking allows people to learn about things without directly experiencing them. Since a computer agent cannot directly experience emotions as a human would, question-asking can be leveraged for the computer agent to learn about emotional concepts. Question-asking has also been proposed as a stage in child development responsible for rapid learning and language acquisition (Frazier, Gelman & Wellman, 2009). Likewise, a computer agent can use question-asking to acquire knowledge and vocabulary. We call the approach of using question-asking to interactively acquire linguistic knowledge about emotion by a computer dialog agent a Socratic epistemology for verbal emotional intelligence.

The knowledge acquired by the Socratic epistemology for verbal emotional intelligence is an informal, social type of knowledge. This informal knowledge about emotions is important because although there has been much recent progress toward understanding the underlying biological basis for emotion, humans have been able to understand emotions informally since ancient times. We call this informal, language-based understanding of emotions natural language description of emotion (Kazemzadeh, 2013). Natural language descriptions of emotion are utterances that refer to emotions, as opposed to utterances that express emotions. This phenomenon can be seen as a specific subset of the larger phenomenon of emotional language, which also includes emotion or sentiment expressed towards some object, vocal modulation due to emotion, and persuasion and pragmatics. Studying language that deals with referential statements about emotions is a novel focus that is distinct from the prevailing trends of studying the expressive characteristics of emotional language.

The framework we present also differs from other computational theories of emotion in that it aims to study how people describe emotions, rather than how emotions should be described. As such, it can be seen as a descriptive, rather than prescriptive, theory, and hence has commonalities with sociological studies of emotions (King, 1989; Shaver, Murdaya & Fraley, 2001; Mihalcea & Liu, 2006; Kamvar & Harris, 2009; Kazemzadeh et al., 2012) and crowd-sourcing. Although there is clearly a biological basis for emotions, the way that people describe emotions in natural language varies over time from culture to culture and person to person. A descriptive theory of how people describe emotions in natural language should reflect this variation. Thus, a particular instantiation of a theory of natural language description of emotion should be relative to a particular agent or set of agents at a given time who hold similar beliefs and communicate using the same language. Rather than trying to establish single, particular definitions for a predetermined set of emotion words, we study how shared meaning can be established through natural language dialog behavior even when people and agents may have different ways of experiencing and describing an undetermined set of emotion words.

The key goals of this paper are to use question-asking to observe natural language descriptions of emotion in the context of interactive dialogs and to computationally model the communicative behavior that supports the referential link between language and emotions. To do this, we use an experimental game called emotion twenty questions (EMO20Q). EMO20Q is played like the traditional game of twenty questions, except that one player, the answerer, must choose a word that names an emotion. The other player, the questioner, must try to guess the emotion that the answerer picked by posing a series of twenty or fewer questions. Importantly, we do not restrict the set of emotion words that the players can choose nor the questions they can ask: the set of emotion words and questions is not limited except by the players’ judgment and cooperation. An example of a human–computer EMO20Q game is given below:

Agent: Let me know when you have picked an emotion word.

User: Ok

Agent: Is it very intense?

User: No

Agent: Is it positive?

User: Yes

Agent: Is it an emotion that just pops up spontaneously (vs. being triggered by something)?

User: Depends

Agent: Is it caused by anything in particular

User: Well, yes, you need the right conditions

Agent: Is it relaxation?

User: Yes

Agent: So did I get it right?

User: Yes, you got it.

Agent: Awesome! Would you like to play again?

User: No, thanks.

Agent: Thank you for playing

The data from the human–human EMO20Q games allows us to observe what words people consider to denote emotions, what questions people ask about emotions, what factors determine the length of question/answer exchanges, and what is the probability that the emotion words will be correctly guessed. These results are used to create the computer agent and test its performance with respect to human performance. For the automated agent presented in this paper, we focused on the case where the computer agent plays the role of the questioner while future work for an answerer agent is discussed in ‘Discussion’.

The paper is organized as follows. ‘Background’ discusses the motivations and theory behind our work. ‘Model’ describes the computational model and algorithm we used to create an EMO20Q questioner agent. ‘Experiments’ discusses experiments we conducted of humans and computers playing EMO20Q. ‘Results’ describes the results of testing the agent. Finally ‘Discussion’ and ‘Conclusion’ propose future work and provide discussion and links to open source software implementations.

Background

Model

Bayesian models have been successfully applied to a wide range of human cognitive abilities (Griffiths, Kemp & Tenenbaum, 2008), including inductive inference of word meaning from corpora (Steyvers, Griffiths & Dennis, 2006) and experimental stimuli (Xu & Tenenbaum, 2005) and powering affective dialog agents (Carofiglio, De Rosis & Novielli, 2009). To our knowledge, this work is the first application of Bayesian cognitive models for learning emotion words from dialog interaction.

The model we use for the EMO20Q questioner agent is a sequential Bayesian belief update algorithm. This model fits the framework of Socratic epistemology, as described in the introduction, because it combines the notion of belief and question-asking. Intuitively, this algorithm instantiates an agent whose semantic knowledge is based on data from previous EMO20Q games. The agent begins a new game of EMO20Q with a uniform belief about the emotion word to be guessed. Based on the previous semantic knowledge, the agent asks questions and updates its belief based on each observation of the user’s answers to the questions. While the EMO20Q game is played, the observations are stored in the agent’s episodic buffer (Baddeley, 2000), also known as working memory. After the game, the agent updates its semantic knowledge using the results of the game, clears its episodic buffer, and is then ready to play again. The words in italics are high-level abstractions used to create a cognitive model for the agent, which is underlyingly implemented as a sequential Bayesian statistical model.

The semantic knowledge described above is implemented as the conditional probability of observing a set of question–answer pairs given a hidden variable ranging over emotion words. This conditional probability distribution is estimated from the corpus of past human–human and human–computer EMO20Q games as follows. Let E be the set of emotion words and let ε ∈ E be this categorical, Bayesian (i.e., unobserved) random variable distributed over the set E. The cardinality of set E grows as the agent encounters new emotion words, from 71 words after the human–human experiments to 180 after the human–computer experiments. The probability of ε, P(ε) is the belief about the emotion word to be guessed. Each question–answer pair from the game of EMO20Q is considered as an observation or feature of the emotion being predicted. Thus if Q is the set of questions and A is the set of answers, then a question q ∈ Q and an answer a ∈ A together compose the feature value f = (q, a), i.e., f ∈ Q × A. The conditional probability distribution, P(f|ε), which represents semantic knowledge, is estimated from the past dialogs by counting the frequency that a question q was answered with answer a for a given emotion word.

The estimation of the conditional distribution P(f|ε) used a smoothing factor of 0.5 to deal with sparsity and to prevent probabilities from becoming zero if the players answered in a nonstandard way. Besides the theoretical motivation of using a Bayesian model to represent the agent’s belief state, the smoothing had the practical benefit of preventing the agent from getting stuck on an incorrect line of questions, for example, if we were to have used a decision tree model as an alternative baseline to the sequential formulation of naïve Bayes.

In this model we stipulate that the questions are classified into discrete classes that are specified through a semantic annotation of the human–human data, as described in ‘Human-human EMO20Q’. For example, the question “is it a positive emotion?” is represented as the semantic expression `e.valence==positive'. If the answer to this question was “maybe”, the resulting feature dimension would be represented as (`e.valence==positive',`other'). More examples of the question features can be seen in Table 4. This set of question classes Q, of cardinality 727, was held constant in the human–computer experiments.

Similarly, we stipulate that the set of answers A are four discrete cases: “yes”, “no”, “other”, and “none”. Initially, the feature vector consists of “none” values corresponding to each question. After each turn, the feature dimension corresponding to the turn’s question is updated with that turn’s answer, either “yes”, “no”, or “other”. When the answer either contains “yes” or “no”, it is labeled accordingly. Otherwise it is labeled “other”. The feature value “none” remains assigned to all the questions that were not asked in a given dialog. “None” can be seen as a missing feature when the absence of a feature may be important. For example, the fact that a certain question was not asked about a particular emotion may be due to the fact that that question was not relevant at a given point in a dialog.

Using Bayes rule and the independence assumption of the naïve Bayes model, we can formulate the agent’s belief about the emotion vector ε after observing features f₁…f_t, in one single batch, as opposed to sequentially (which will be formulated next): (1)${\displaystyle P\left(\epsilon |f_1,\dots ,f_t\right)=\frac{\prod _{i=1}^t\left[P\left(f_i|\epsilon \right)\right]P\left(\epsilon \right)}{\prod _{i=1}^{t}P\left(f_i\right)}.}$ This is simply the formulation of naïve Bayes, where in this case P(ε) is the prior probability of a player choosing a specific emotion word, $\prod _{i=1}^t\left[P\left(f_i|\epsilon \right)\right]$ is the likelihood of seeing question–answer pairs given specific emotion words, and $\prod _{i=1}^{t}P\left(f_i\right)$ is the probability of observing a set of question–answer pairs in general.

In terms of the high-level cognitive model, the set of observational feature vectors f₁…f_t is what was described as the agent’s episodic buffer. P(f|ε) is the agent’s semantic knowledge that relates question–answer features to emotion words. p(ε) and P(ε|f₁, …, f_t) are the agent’s initial/prior and final/posterior beliefs, respectively.

In Eq. (1), the posterior belief of the agent about emotion e_k at time t, P(ε = e_k|f₁, …, f_t) is computed only after the agent has asked all t questions. This model is known as naïve Bayes. In contrast the sequential Bayes model that we use is dynamic: the agent updates its belief at each time point based on the posterior probability of the previous step, i.e., at time t ${\displaystyle P\left(\epsilon |f_1,\dots ,f_t\right)=\frac{P\left(f_t|\epsilon \right)P\left(\epsilon |f_1,\dots ,f_{t-1}\right)}{P\left(f_1,\dots ,f_t\right)}.}$ When the game begins, the agent can start with a uniform prior on its belief of which emotion is likely or it can use information obtained in previously played games. In the experiments of this paper, we use a uniform prior, P(ε = e_k) = 1/|E|, ∀k = 1…|E|. We chose to use the uniform prior to initialize the agent because our training data contains many single count training instances and because we want to examine how the system performs with fewer constraints.

We introduce a new variable β_t,k = P(ε = e_k|f₁, …, f_t) for the agent’s belief about emotion k at time t and postulate that the agent’s prior belief at a given time is the posterior belief of the previous step. Then, the agent’s belief unfolds according to the formula: ${\displaystyle {\beta }_{0,k}=P\left(\epsilon =e_k\right)=1/\left|E\right|}$ ${\displaystyle {\beta }_{1,k}=\frac{P\left(f_1|\epsilon =e_k\right)}{P\left(f_1\right)}{\beta }_{0,k}}$ (2)${\displaystyle {\beta }_{t,k}=\frac{P\left(f_t|\epsilon =e_k\right)}{P\left(f_t\right)}{\beta }_{t-1,k}.}$ Decomposing the computation of the posterior belief allows the agent to choose the best question to ask the user at each turn, rather than having a fixed battery of questions.

We define “the best question” at time t to be the question class that is most likely to have a “yes” answer given the posterior belief at time t − 1, P(ε|f₁, …, f_t−1): ${\displaystyle {\text{argmax}}_{q\in Q}P\left(\left(q,\text{‘yes’}\right)\right)|\epsilon \left.\right)P\left(\epsilon |f_1,\dots ,f_{i-1}\right).}$ This next-question criterion is a heuristic motivated by considering “yes” answers to be positive feedback that the agent is on the right track. While this heuristic worked well in practice, other next-question criteria may be possible in future research. After the question class has been chosen it is converted from its logical form (e.g., `e.valence==positive') to a concrete realization from the human–human data (e.g., “is it positive?”) by choosing a random question that had been annotated with the logical form. In this way, the questions are more varied and might even include slang or typos that occurred in the human–human data. In terms of natural language generation, the choice of question class can be seen as pragmatic planning and the choice of surface form can be seen as a form of lexical access.

At time t the agent asks the best question and takes the user’s response as input. It then parses the input to classify it into one of {“yes”, “no”, “other”}. This information is then used to update the agent’s posterior belief β_t+1,k about each emotion e_k ∈ E, which will then be used as the prior probability in the following step. The unfolding of variable β in Eq. (2) models the update of belief as it is justified by the agent’s question-asking and the user’s answers. It is this computational model of question-asking and belief update that represents the Socratic epistemology for verbal emotional intelligence in a software agent. Table 1 shows an example interaction between the automated EMO20Q questioner agent and a human user, along with a trace of the agent’s belief state that shows the justification of beliefs by question-asking.

Identity questions are a special type of question where the agent makes a guess about the emotion that the other player has picked. Identity questions are chosen with the same best question criteria as other questions and the belief state is initially updated in the usual way. However, after the belief update there is a transition to a different dialog state. An affirmative answer to an identity question (e.g., “is it happy?”) means that the agent successfully identified the user’s chosen emotion and the game is then terminated successfully. Any other answer to an identity question will set the posterior probability of that emotion to zero because the agent can be sure it is not the emotion of interest. The usual belief state update is performed even in the case of identity questions for two reasons. First, identity questions can be ambiguous: the example “is it happy?” can be interpreted as “Did you pick the emotion ‘happiness’?” (i.e., an identity question) or “Did you pick a happy emotion?” (i.e., an ordinary question). In this case, the usual Bayesian belief update deals with the former interpretation. Second, identity questions, even when answered negatively, contain information that can be useful for updating the belief state beyond setting the posterior probability of the negatively identified emotion to zero.

The pseudo-code for the main loop of the adaptive Bayesian agent is shown in Algorithm 1. This automated, data-driven component was framed within a manually designed dialog graph, as shown in Fig. 1. The dialog graph is implemented as a generalized pushdown transducer. Recall that a pushdown transducer is a transducer that determines its output symbol and next state based on its current state, the input symbol, and the top of its stack (Allauzen & Riley, 2012). A generalized pushdown transducer is a pushdown transducer that is not limited to only the top of the stack when determining the output and next state. This aspect is important in the question-asking loop because the stack represents the episodic memory, which stores the question–answer observations. Otherwise, the agent could be implemented as a plain pushdown transducer.

Experiments

The EMO20Q experiments we conducted can be partitioned into human–human and human–computer experiments. ‘Human-human EMO20Q’ examines the data from human–human experiments, which was the initial corpus used to train the EMO20Q question-asking agent and which was the source of the agent’s set of questions. ‘Human-computer EMO20Q’ focuses on experiments with the question-asking agent described in ‘Model’.

Results

The results of our pilot experiments on fifteen subjects are summarized in Table 5. To compare the agent’s performance with human performance, we used two objective measures and one subjective measure. The success rate, shown in column two of Table 5, is an objective measure of how often the EMO20Q games ended with the agent successfully guessing the user’s emotion. The number of turns it took for the agent to guess the emotion is the other objective measure. The last column, naturalness, is a subjective measure where users rated how human-like the agent was, on a 0–10 scale.

In the pilot study, the agent obtained a performance of 44% successful outcomes (where the emotion word was correctly guessed). This performance was much less than in the human–human experiments, where successful outcomes occurred in 85% of EMO20Q games. However, the results indicated that this performance was due to sparsity of data. The emotion words chosen by the subjects as “easy” were recognized by the agent with similar success rate and number of required turns as human–human games. Some examples of “easy” emotions are anger, happiness, and sadness. However, successful outcomes were fewer in emotions chosen as “medium” and “difficult”. Some examples of “medium” emotions are contentment, curiosity, love, and tiredness. Pride, frustration, vindication, and zealousness are examples of “difficult” emotions. Overall, 28 new emotion words were encountered in the pilot study.

The results in terms of successful outcomes and number of turns required to guess the emotion word are roughly reflected in the percent of words that are in-vocabulary. Despite the low performance on emotion words rated “medium” and “difficult”, there was not a corresponding decrease in the perceived naturalness of the questioner agent. This led us to believe that the model could reproduce somewhat natural behavior, but that the data we had was insufficient due to the amount of out-of-vocabulary words in the medium and difficult classes, which motivated us to perform the second, larger-scale experiment with 100 players from Mechanical Turk.

In the larger scale Mechanical Turk experiment, we aimed to improve performance by retraining the model after each batch of 10 subjects. This strategy did in fact increase the successful outcome rate and reduced the length of the EMO20Q dialogs (number of questions), as can be seen from comparing Tables 5 and 6, which are visualized in Fig. 2. Across all three difficulty classes, the successful outcome rate improved. The “difficult” class had the largest relative improvement in successful outcomes, increasing from 13% to 25%, and the overall successful outcome increased from 44% to 57%. The increase in successful outcomes was significant at p < .05 as measured by a two-sided proportion test. The lengths of the EMO20Q dialogs decreased most for the medium difficulty class, resulting in an average of 1.6 less turns for this class, and overall the dialog length decreased from 15.6 to 14.8 turns. However, the observed differences in dialog length and in-vocabulary rate were not statistically significant. Also, the improvement from the beginning to the end of the Mechanical Turk experiment was not significant. Overall, 81 new emotion words were encountered in the Mechanical Turk study.

One surprising result was that even after collecting data from 300 EMO20Q dialogs (more than doubling the earlier human–human data), the out-of-vocabulary rate stayed nearly the same. We had expected out-of vocabulary-words to become fewer as more data had been seen. However, with each round of the Mechanical Turk experiment, we continued to receive new emotion words rather than converging to a closed vocabulary. For the Mechanical Turk experiment, we did not ask subjects about the perceived naturalness of the agent in order to save on time, and hence costs to pay the Turkers, so unfortuntately we cannot say whether the perceived naturalness increased.

Of the 101 subjects, only one was rejected, due to misunderstanding the task by choosing the words “easy”, “medium”, and “difficult” instead of emotion words. This level of acceptance, approximately 99% is rather high for Mechanical Turk, showing a high degree of cooperation. Several users commented that we could have paid less because the task was fun.

A listing of the words chosen by the subjects of the Mechanical Turk experiment is given in Table 7. It can be seen that there are a wide variety of words. A few (those marked by “?”) were questionable in the authors’ intuitions, but otherwise the words showed a high level of understanding and cooperation by the Mechanical Turkers. The three difficulty classes of words were not disjoint: some words like anger, disgust, love, and confusion spanned several categories. It can be concluded that these three difficulty levels do not form precise, natural classes of emotion words, but the levels do show a trend toward a smaller basic vocabulary and a wider open vocabulary. The difficulty levels also served as a method to elicit a diverse set of words. The original human–human dialogs identified 71 unique emotion words, after the pilot study there were 99 unique emotion words, and after the large-scale Mechanical Turk experiment there were 180 unique emotion words.

Discussion

The human–human EMO20Q data abounds in highly nuanced natural language descriptions of emotion. For example, one human–human EMO20Q game ended with a discussion of whether “pride” and “proud” refer to the same emotion:

[regarding “proud” vs. “pride”] because my intuition was that they’re different… you know pride sometimes has a negative connotation

In another human–human EMO20Q dialog, a player had difficulty answering whether “anger” was a negative emotion:

[questioner:] so is it a negative emotion?

[answerer:] sort of, but it can be righteous

In one human–computer game, one player differentiated the emotion of loving from the emotion of being loved and another player picked the emotion “maudlin”, which the authors needed to look up in a dictionary.

Given the highly nuanced, idiosyncratic descriptions in the human–human data, we were surprised at the amount of successful outcomes in the human–human EMO20Q games and we were initially unsure whether devising an automated agent would be feasible. Although analyzing this level of detail is beyond the scope of many current systems, we saw that it is a task that humans can do with high success rates. In fact the successful outcome rates in the human–human EMO20Q games are comparable to agreement rates on emotional annotations at a much coarser level, such as labeling data with nine basic emotion labels (Busso et al., 2008).

The human–computer results showed us that it is possible for a computer agent to perform well at the questioner role of EMO20Q and moreover that the agent can learn new vocabulary items and improve its performance past the pilot study, which only used annotated human–human bootstrap data, by using unannotated human–computer data. The fully trained agent successfully completed 57% of the EMO20Q games, which is 67% of human–human performance and 30% better than the bootstrapped agent of the pilot study. The agent’s emotion word vocabulary nearly doubled after the Mechanical Turk experiment. Normally larger emotion vocabularies would result in less agreement in annotation tasks but this showed that in the EMO20Q dialog task, vocabulary size is not a weakness but rather a strength. Even when the agent fails to guess the human opponent’s emotion word in the EMO20Q game, the agent’s behavior of searching for knowledge makes it appear human-like, which enables the agent maintain user engagement and learn from new, out-of-vocabulary words.

Other methodologies could be used for the EMO20Q agent and decision trees would seem to have been a natural choice. However, we saw the potential pitfall of decision trees, in their default formulation, to get stuck on the wrong branch of a tree if a subject answered a question in a non-standard way. In contrast, as our formulation of sequential Bayes extends naïve Bayes to deal with sequential observation of features, decision trees would similarly need to be updated to deal with sequentially observed features. We dealt with this issue in our methodology by smoothing the counts of the sequential Bayes model and by dynamically choosing the next question after each dialog turn. The sequential Bayes model proved capable of dealing with variability in users’ answers and providing natural behavior in the case of out of vocabulary words. Other methodologies for an EMO20Q agent would need to deal with these challenges.

The ground truth issue involved in annotating recorded data with descriptive labels is a challenge that the Socratic epistemology can shed light upon. The traditional annotation task seeks to have human annotators assign one of a number of labels to data. In the case of emotion research, usually the labels are a controlled vocabulary of several emotion descriptors, like “angry”, “happy”, “sad”, “disgusted”, “fearful”, “surprised”, and “neutral”. The problem with this approach is that these labels often do not fit realistic emotional data. Theoretically, our approach addresses the issue of ground truth in the annotation task with the notion of epistemology, which frames the issue as justification of belief rather than ground truth. Practically, our approach addresses the issue of non-prototypical emotions by enabling a more nuanced representation where the description is not a small, closed set of alternatives but rather an interactive process of communication over a large, open set of natural language descriptions. Though this more nuanced view brings with it new challenges, we have shown that the design of an intelligent dialog agent is a feasible way of dealing with these challenges.

We plan to continue this research in several ways. First, we hope to see the effect of modality on how people describe emotions in natural language. The current work was limited to text-based chat, so the paralinguistic data that may help to convey emotional information was minimized. Including audio and video data may allow greater convergence of the players to agree upon the unknown emotion in EMO20Q. Another area of future research will be to model the answerer role. The current research focused on the questioner role, but the answerer role will offer additional challenges and insights. In particular, automating the answerer role will require more robust natural language understanding because it will need to process new, unseen questions from users, whereas the questioner used a fixed set of questions and only had to process answers to yes/no questions. The answerer would also likely require a different model than the Socratic, question-asking model presented in this paper. A successful answerer agent would allow a pleasing closed-loop simulation where both roles of EMO20Q are played by computer. There are also further areas to explore for the questioner agent, in particular, the criterion for choosing each question. We also think that this approach can improve emotion annotation and other annotation tasks, such as coding behavioral data for psychological assessment. In these tasks human annotators are asked to label data using a controlled vocabulary of words and agreement is established statistically between isolated annotators. However, we have shown that humans are able to communicate with high accuracy using a large, subjective vocabulary and we think that allowing natural language descriptions in an interactive, question-asking setting will allow for more accurate and less constrained annotations. Moreover, using a similar agent to elicit and identify items in open-ended vocabularies in other domains is another area for future work. We have found that in the case of emotion words, the EMO20Q agent was able to elicit a growing vocabulary and successfully identify items from the vocabulary, which could be possible for other domains. For example, eliciting “things users are passionate about” or medical illnesses are possible open-ended domains that may be hard to enumerate in an a priori exhaustive list, but which might be identified by a similar dialog agent.

Conclusion

The main goals of this paper were to formulate a theoretical and computational model for a subset of human emotional language. We called this model the Socratic epistemology for verbal emotional intelligence because it uses question-asking to justify beliefs about emotions in a natural language dialog context. We presented the emotion twenty questions (EMO20Q) game and showed that the level of human performance was high despite not limiting the players to any predefined emotion vocabulary. We also presented an automated agent that can play the question-asking role of EMO20Q. This agent uses a sequential Bayesian belief update algorithm to simulate a cognitive process by which the agent updates its belief state of candidate emotion words over time. This framework was inspired by a method of question-asking that was proposed by the ancient philosopher Socrates and the field of epistemology:

[Gorgias:] Just as different drugs draw forth different humors from the body—some putting a stop to disease, others to life—so too with words: some cause pain, others joy, some strike fear, some stir the audience to boldness, some benumb and bewitch the soul with evil persuasion” (Gorgias, Encomium of Helen, c.415 B.C.).

Socrates: You, Gorgias, like myself, have had great experience of disputations, and you must have observed, I think, that they do not always terminate in mutual edification, or in the definition by either party of the subjects which they are discussing;…Now if you are one of my sort, I should like to cross-examine you, but if not I will let you alone. And what is my sort? you will ask. I am one of those who are very willing to be refuted if I say anything which is not true, and very willing to refute any one else who says what is not true, and quite as ready to be refuted as to refute (Plato, Gorgias, 380 B.C.).

In the first quote above, Gorgias, a Sophist rhetorician, describes the effects of words on a person’s emotions. Gorgias describes emotions by making reference to the theory of physiological humors. Humankind’s conception of emotions has changed since the time of the ancients, who believed that emotions were generated from bodily “humors”, which in turn were derived from alchemical elements, but our conception of emotion is still largely expressible through language.

In the second quote, Socrates (as quoted by Plato) cross-examines Gorgias to determine Gorgias’ beliefs. Socrates applied his method of question-asking to understand beliefs about complex abstract concepts that were disputed in ancient times. Two millennia later we have used a computational implementation of this method to make a dialog agent better understand human beliefs about emotional concepts.

We have provided an anonymized version of data we gathered from EMO20Q, source code for the experiments, demos, and other resources at http://sail.usc.edu/emo20q.