Dialog Context Language Modeling With Recurrent Neural Networks

Dialog, NLG Comments

Paper Link



However all these methods focused on applying context by encoding preceding text without considering interactions in dialogs.


Context Dependent RNNLM

Let D = \((U_1, U_2, ... U_K)\) be a dialog with K turns and involve 2 speakers. In this case, turn is just the utterance of a single speaker and could involve multiple messages. The kth turn \(U_k = (w_1, w_2, ..., w_{T_K})\) is represented as a sequence of T_k words.

\[\begin{align*} P (U_k | U_{less_than_k}) = \prod_{t=1}^{T_k} P ({w_t}^{U_k} | {w_{less_than_t}}^{U_k}, U_{less_than_k}) \end{align*}\]
Dialog RNN

Context Representations

However, the above 2 methods treat dialog history as a sequence of inputs, without modeling dialog interactions. In order to deal with this, the paper proposes 2 different architectures -

Interactive Dialog Context LM

In the above model, we define the context and initial hidden state as follows - \(c = {h_{T_{k-1}}}^{U_{k-1}}\) and \({h_0}^{U_k} = {h_{T_{k-2}}}^{U_{k-2}}\)

External State Interactive Dialog Context LM

In the above model, we have an external RNN to encode the respresentation from 1 turn to another.


Model K=1 K=2 K=3 K=5
RNNLM 60.4 - - -
DRNNLM - 60.1 58.6 59.1
CCDCLM - 63.9 61.4 62.2
IDCLM - - 58.8 58.6
ESIDCLM - - 58.4 58.5

Kaushik Rangadurai

Code. Learn. Explore

Share this post