Written out of the story: the robots capable of making the news

In January 2016, Fan Hui, three-time champion of the east Asian board game Go, lost to a computer program developed by Google’s subsidiary DeepMind. The game is far more difficult to master than chess, so it was a historic victory for artificial intelligence (AI) which has seen huge advances in recent years. But could AI one day perform more creative tasks? A small tech firm in the US is proving that even journalists aren’t safe from the robot revolution with a product that automates the writing process.

Wordsmith – an artificial writer developed by the North Carolina-based company Automated Insights – cherrypicks elements from a dataset and uses them to structure a “human sounding” article. As well as being able to use more emotive language, it varies diction and syntax to make its work more readable.

Founder Robbie Allen describes the program as a “natural language generation platform” which can be used to churn out data-driven articles faster than even the most nimble-fingered hack. Last year the company produced more than 1.5 billion pieces of content, up from 300 million in 2013.

It already has several high-profile customers, including Gannett, which publishes USA Today and Yahoo News. Yahoo uses Wordsmith to draft texts for Fantasy Sport, a game in which players create their dream football team using the professional profiles of real athletes, then compete in fictitious games with virtual teams fielded by other players.

In June 2014, Associated Press announced it would use technology from Automated Insights to produce most of its US corporate earnings stories, claiming it would free up time for its reporters to concentrate on meatier articles. Wordsmith also knows how to draft reports on marketing, business activity and financial results.

“Although I have a background in IT, I used to write a fair number of recaps about college basketball games,” Allen explains. “If you look at a recap for a sporting event, a significant portion of it is just analysing and describing those numbers. So I thought what I could do is write algorithms that encapsulated that. That was the early genesis behind what is now Automated Insights.”

In October 2015, the company launched its first self-service version of the platform, allowing users to upload their own data and create story templates. Previously customers would provide the data and Automated Insights would produce the content.

There are limitations to what the software can write, however. Content has to be data-driven – so in a sports report, for example, you can automate baseball recaps all day long, but the story would not be able to capture more qualitative, descriptive statements because the data is not available to provide more colourful information.

Allen claims that Wordsmith is revolutionising how journalists write. “If you look at technology over the last 20 or 30 years, every facet of our daily lives has been changed by it in some way,” he says. “But the way you write today is probably not a whole lot different than what you would have done a few decades ago. We have word processing software and machines but there hasn’t been much innovation.”

However, doing something different isn’t without challenges. Allen says many people are unaware that technology is capable of writing coherent, readable prose – so educating the market about the software’s potential has been key to the development of the business.

“As a small company that is something that we have to wrestle with,” he admits. “People aren’t always accepting of new things and it feels like we are trying to sell a car to people who have only seen horses before. But that said, there is a lot of promise in what we are doing.”

Allen adds that he has been surprised by the positive reception to the technology, which could be seen as a threat to real journalists. He recalls being worried, at the product’s launch in 2011 that it would be grouped with so-called content farms, websites producing low-quality content generated solely to ensure that they appear high on the list of results returned by a search engine.

He says: “There was this notion that if you can automate a story then no one needs to write another version. It’s ludicrous because for any given story there are lots and lots of perspectives; there are many ways to write about something.”

Feedback has been largely enthusiastic and Allen says he is now excited to see how this technology will evolve.

But while the technology is interesting, automated writing still faces huge challenges. “If all journalism was formulaic, then automation would be the future,” says editor, writer and digital media consultant Jon Bernstein. “But obviously not all writing is like this. Journalism is not just about providing information – it’s about putting it into the context, it’s about telling people what it means.”

He adds that the traditional structure of a news story is the who, what, why, when and how. It’s the crucial ‘why’ of the story that is the missing piece of the puzzle for AI.

Bernstein explains: “The ‘why’ requires the ability to contextualise, the ability to analyse and to infer. And I think that skill is incredibly tricky for a piece of technology to learn. It can do it to an extent but it can’t do it in the way that somebody who has huge experience about the story is able to.”

Sign up to become a member of the Guardian Small Business Network here for more advice, insight and best practice direct to your inbox.