A truly astounding exemplar of how coverage of AI goes horribly wrong.

What Happened

The paper that originated all this terror has the fun seemingly un-scary title “Deal or No Deal? End-to-End Learning for Negotiation Dialogues” and was done by researchers from Facebook and the Georgia Institute of Technology. As the title implies, the problem being addressed is the creation of AI models for human-like negotation through natural language. To tackle this, the researchers first collected a brand new dataset of 5808 negotations between plain ol’ humans with the data-collection powerhorse Mechanical Turk.

These negotiations involve two people agreeing on a split of some set of items, which have different values for each negotiator. This is a nicer problem to tackle than generic chatbot conversation, because there is a straighforward evaluation metric (how much value each negotiator manages to secure) and the language needed is less broad than generic speech. The final approach used a combination of supervised learning (just emulating the dialogues from the dataset) and reinforcement learning (learning to generate speech based on maximizing negotiation outcomes). This is a pretty hefty paper, and all this drama came about because of one tiny section:

During reinforcement learning, an agent A attempts to improve its parameters from conversations with another agent B. While the other agent B could be a human, in our experiments we used our fixed supervised model that was trained to imitate humans. The second model is fixed as we found that updating the parameters of both agents led to divergence from human language.

To be clear, this is not all that surprising, since the optimization criterion here is much more specific that developing a robust generic language to communicate about the world. And even if the criterion were that broad, there is no reason for the optimization to converge upon English without some supervised loss to guide it there. I am no professor, but even with my meager knowledge of AI I am fairly confident in saying this is a truly, utterly unremarkble outcome.

The Reactions

It started with a tweet:

And then, things developed swiftly:

Our Perspective

As the multiple hype-debunking articles just above imply, this small aspect of the research is really not a big deal. The only big deal about all this is how crazy coverage of this story got, given how mundane the AI development was. After all, few media stories have ever gotten AI researchers this heated:

In fact, the coverage has gotten so out of hand that the researcher responsible for it was driven to write a long post addressing the nonsense:

And that right there is all that needs to be said.


AI models optimizing to use nonsensical communication is not surprising nor impressive, which makes the extremely hyperbolic media coverage of this story downright impressive.

We love hearing from you! Feel free to provide feedback, suggest coverage, or express interest in helping. Or, comment below!
comments powered by Disqus