Google, democracy and the truth about internet search

I’m pretty sure I read this article in 2016 when it first came out, but I’m reading it again now with the eyes of someone thinking hard about why and how progressives should extract themselves from social media. I’m realising that search engine optimisation – fighting back against the forces that seed Google with horrifying answers to questions like “Are Jews Evil?” – needs to be a part of that plan.

A few days later, I talk to Danny Sullivan, the founding editor of SearchEngineLand.com. He’s been recommended to me by several academics as one of the most knowledgeable experts on search. Am I just being naive, I ask him? Should I have known this was out there? “No, you’re not being naive,” he says. “This is awful. It’s horrible. It’s the equivalent of going into a library and asking a librarian about Judaism and being handed 10 books of hate. Google is doing a horrible, horrible job of delivering answers here. It can and should do better.”

He’s surprised too. “I thought they stopped offering autocomplete suggestions for religions in 2011.” And then he types “are women” into his own computer. “Good lord! That answer at the top. It’s a featured result. It’s called a “direct answer”. This is supposed to be indisputable. It’s Google’s highest endorsement.” That every women has some degree of prostitute in her? “Yes. This is Google’s algorithm going terribly wrong.”

It’s not “going terribly wrong” though, it’s working exactly as it’s meant to. The original PageRank algorithm looks to find the most-linked to pages using those terms, and the far right have made huge networks of pages all linking to each other on these topics.

Last week Jonathan Albright, an assistant professor of communications at Elon University in North Carolina, published the first detailed research on how rightwing websites had spread their message. “I took a list of these fake news sites that was circulating, I had an initial list of 306 of them and I used a tool – like the one Google uses – to scrape them for links and then I mapped them. So I looked at where the links went – into YouTube and Facebook, and between each other, millions of them… and I just couldn’t believe what I was seeing.

“They have created a web that is bleeding through on to our web. This isn’t a conspiracy. There isn’t one person who’s created this. It’s a vast system of hundreds of different sites that are using all the same tricks that all websites use. They’re sending out thousands of links to other sites and together this has created a vast satellite system of rightwing news and propaganda that has completely surrounded the mainstream media system.”

Albright’s map also provides a clue to understanding the Google search results I found. What these rightwing news sites have done, he explains, is what most commercial websites try to do. They try to find the tricks that will move them up Google’s PageRank system. They try and “game” the algorithm. And what his map shows is how well they’re doing that.

That’s what my searches are showing too. That the right has colonised the digital space around these subjects – Muslims, women, Jews, the Holocaust, black people – far more effectively than the liberal left.

SEO makes a lot of people I know pull a face like “ewww,” and rightly so, with its reputation for scamminess and, at best, the way you wind up doing what Big Evil Google want you to do. But at its core what’s going on here – and what’s at the core of Google’s PageRank algorithm – is a lot of compelling articles getting a lot of links from other sites. If we turn up our noses at doing that effectively (and ethically), are we leaving the search results in the hands of the bad guys?