Yes, We Need Statistical Significance Testing

A rule of thumb may yield correct results but can’t be scientifically credible

A rule of thumb may yield correct results but can’t be scientifically credible.

Illustration by the author.

Take any research paper or blog post presenting a new method for AI, you’ll very probably find a statement similar to this:

[…] a significant improvement over previous work.

If this is a method applied to language generation tasks (automatic summariza…