Are the results as ‘relevant’ and ‘trustworthy’ as Google promises? We did our own experiment.[1]
By Natalia Stanusch
While many consider human rights violations in Qatar as a pivotal issue of this year’s World Cup, to what extent does the global gatekeeper of information, Google, consider human rights violations relevant?
What many of us can learn about Qatar in relation to the FIFA World Cup comes from the internet, specifically search engines, the most trusted news source globally according to a 2021 survey.[2] We turn to search engines in our browsers to search for most important news or to search for a relevant information. But most often, we do not search – we google.
Owning over 90 percent of the market share of search engines worldwide,[3] Google is the search engine of the world. Yet Google does not only provide information online, but also orients and controls the information flow.[4] “Our mission is to organize the world’s information and make it universally accessible and useful,”[5]Google claims on its “About” page. Google declares that its search engine is “the world’s largest library.”[6] But this metaphor is misleading unless every library is continuously “improving your experience”[7] by presenting to you “the most relevant, useful results,”[8] deciding on what your ‘experience’ ought to be, and what ‘relevant’ should mean to you – and does it all for profit. Imagine a library where books constantly change their location, where each person who enters sees different books in a different order, and is not aware of the tailored order of the books she can and cannot see. In Google Search, the websites are not ‘cataloged’ as in a library, but rather filtered, sorted, and prioritized, directly affecting the content diversity that users experience.[9] Rather than a library, Google Search is an opaque, apparently efficient, constantly morphing ranking system.
Tarleton Gillespie, a Principal Researcher at Microsoft Research and a Professor at Cornell University, argues in The Relevance of Algorithms that the order of results that Google algorithms display within SERP (Search Engine Result Page) “represents a particular knowledge logic,”[10] a logic based upon an undefined notion of the most ‘relevant’ information which steers the content that users receive. Google is thus, to borrow from Gillespie, “producing and certifying knowledge.”[11] Google produces and certifies knowledge as what Google deems most relevant indeed becomes most relevant. The recommendation algorithms which run Google Results Pages allow some sources to be visible while burying other sources deep in the depths of the web, leaving many users unaware of the reasons (and existence) behind some sources being ‘pushed down’ within Google’s hierarchy of relevance.
Google Results In 18 Countries
So, what do I see if I google ‘Qatar World Cup,’ and what does my friend in Brazil see? Is it different from what I would see if I were in Greece, or in the UK? — the short answer is, yes. I asked my friends from several countries to search for ‘Qatar World Cup’ using Google. The countries (or what Google defines as ‘Google Region’) my friends searched from were, in alphabetical order, Argentina, Brazil, Canada, Denmark, France, Germany, Greece, India, Italy, the Netherlands, New Zealand, Norway, Poland, Qatar, Spain, the UK, the USA, and the UAE.
As most users tend to search in their native language using short and simple queries,[12] I asked my friends to search for ‘Qatar World Cup’ in the official language of the country they are in (if they spoke it). In the query, I did not include any words related to human rights issues so that the query contains ‘neutral’ keywords, especially given the several reports of Google showing biases over politically sensitive or controversial queries.[13] To minimize personalization in the results, I asked my friends to either use a new browser, or to use incognito mode while searching. I collected the results between the 18th and 20th of November 2022, within a timed spawn of about 35 hours. I looked at the results on the first five pages for each query.
I marked in red the placements of results related to Qatar World Cup human rights violations, its dark side, controversy, boycott, etc. in each of the five SERPs for each query. Unrelated results were marked in grey. The order of the results shown below represents the order of the results presented within SERPs, starting from the top (what appears first, or ‘above the fold’). Google Features (inorganic results such as Top News or Films) were counted as singular results even if they contained several hyperlinks within them. Each organic result was counted as one. The number of results in the SERPs differed from 8 to 12 results per page, hence the white (blank) spaces were included to account for those differences. The results are as follows.
This reverse engineering is an attempt to reveal the inner logic and workings of Google search, recommendation, and ranking algorithms. Yet whatever it reveals is temporal, to say the least. Google search results are intrinsically unstable due to continuous readjustment of Google’s algorithms and recommending system.[14] Safiya Umoja Noble, a professor at UCLA and the author of Algorithms of Oppression: How Search Engines Reinforce Racism points out that it is often impossible to know how and why Google results change over time,[15] leading to the limitation of any research to be representative of a singular capture of constantly shifting and changing search results. The number of results differs in each of the query captures, depending on the layout that Google presents for a specific query, in a specific location, and in a specific time.
Google argues that its “systems understand that up-to-date information might be more useful than older pages. This means that you’ll see the latest information when you’re searching for sports scores (…) or anything related that’s especially new.”[16] The most ‘useful’ information appears on the first page of the results, the page which many of my friends found curious to ‘abandon’ as they were asked to click on the second, third, fourth, and even the fifth page of the results. But their curiosity and surprise are understandable – most users stop their search at the first page of Google results,[17] while more than 80 percent of users stop looking after the first three results on the first page.[18] Looking at the first five pages of Google results accounts for cases in which no human rights related results are presented within the first page (as in the case of the search query conducted in Qatar), as well as allows for a deeper dive into what ‘is relevant’ based on a language and ‘Google region.’
Correlation or causation, or both? Google does not allow for a clear answer. Could we say that queries conducted in the countries which qualified to the World Cup (Argentina, Brazil, Canada, Denmark, France, Germany, Netherlands, Poland, Qatar, Spain, UK, USA) tend to deem the human rights issues less relevant as football-oriented news ‘score higher’ for Google algorithms? Maybe, as could be the case for Canada, Spain, or the Netherlands, but not necessarily, looking at the case of Denmark, Germany, and Brazil. Are searches conducted in countries which did not qualify to the World Cup (Greece, India, Italy, New Zealand, Norway, and the UAE) less occupied with football, thus filtering through more human rights related results? Maybe in the case of Norway, but not in the case of India. How to account for the search conducted in Poland, where three results concerning human rights were ranked high on the first page, yet there are no such results present on other four pages, or in Italy, which illustrates the opposite case? Is there no logic? There is, an algorithmic logic and a human logic, neither of which Google really wants you to know about.
The Almighty Algorithm, the All-knowing Google
Google is not an information company, but an advertisement company, biased towards its own properties and sponsored content,[19] which explains why the first thing that appeared in each of twenty queries was a Google Feature, a Google-generated ‘box’ about Qatar World Cup information and ‘forthcoming’ scores. In his book, Platform Capitalism, scholar Nick Srnicek discusses Google as an example of an “advertising platform,”[20] a platform that profits off advertisements and sponsored content. The logic of sponsored content shows not only in cases of results that are not ‘relevant’ but paid for but also in tricking the Google search system. In the case of the query conducted in Denmark, the first result on the second page was an ad paid for by Danish Amnesty International, promoting its report on human rights violations in Qatar. Following Noble’s argument, “Google creates advertising algorithms, not information algorithms.”[21]
The notion of ‘relevance’ that Google’s hierarchy of results reflects is thus highly questionable, to begin with. While Google does mention its ‘systems,’ which are algorithms which are designed by human engineers, Google is significantly less outspoken about its human content moderators. As described in a recent study, Google employs region-based human moderators who ‘judge’ whether the results are relevant for a specific cultural and regional context.[22] Those moderators have to operate based on strict guidelines, often working more like machines than humans, yet their input is critical for Google results pages.[23] Google’s complete lack of transparency clashes with Google’s strategy to appear ‘objective.’[24]
What happens within Google’s SERP is not only a negotiation of what is ‘most relevant,’ but also a competition among websites for gaining the most visibility. Search Engine Optimization (SEO) is a set of practices to put websites ‘higher’ in Google results, a ‘gaming’ of a system. Hence, Google constantly changes algorithms that rank pages, while pages are being re-optimized trying to appear as more ‘relevant’ and ‘trustworthy’ for Google. While looking at the results of ‘Qatar World Cup’ query, we need to consider that many of the results (both concerning human issues and not) are at least playing (if not gaming) Google’s game of being ‘relevant’ and ‘trustworthy,’ hence they rank high within Google’s SERP.
Should we then consider Google as ‘relevant’ and ‘trustworthy’? While it is somewhat known that personalization influences what users see online, many are still unaware of the invisible editing of Google’s recommendation and ranking algorithms. Most users are not aware that they are only given a ‘slice’ of the content, or a different slice compared to what they could stumble upon if they searched from a different geographical location or in a different language. Google’s monopoly on information is exercised upon billions of users across the globe in their daily searches. Those users often trust Google with showing what matters. The Google search ‘Qatar World Cup’ across twenty queries, in different languages and locations, is a good example of it. How ‘relevant’ human issues appear depends on Google. Should we trust an advertising company with the global curation of news and knowledge?
About the author: Natalia Stanusch is interested in studying digital media artifacts by way of critical theory. Stanusch’s research foci include datafication, critical algorithm studies, and digital visual culture, as she combines them in a continuous attempt to explore the power infrastructures behind new technologies. In her practice, Stanusch often brings together twentieth-century avant-garde legacies and media studies. She wrote and presented on memes, AI art, and the ideology of the Metaverse. Stanusch is currently based at the University of Amsterdam where she is finishing her MA in Media Studies, specialization New Media and Digital Culture.
[1] This text was inspired by a project ‘Google Auditing,’ initiated in the class Digital Research Seminar, on which I worked alongside Kefeng Cao and Çağla Onaran, under the supervision of Professor Richard Rogers at the University of Amsterdam, in November 2022.
[2] Statista, Jaunary 2022
[3] Statista, 2022
[4] Latzer et al., 2016; Steiner et al, 2022, p. 217
[5] Google, ‘About Google’
[6] Google, ‘How Search Works’
[7] Google, ‘How Search Works’
[8] Google, ‘Ranking Results’
[9] Steiner et al, 2022, p. 219
[10] Gillespie, 2014, p. 168
[11] Gillespie, 2014, p. 168
[12] Noble, 2018, p. 37
[13] e.g. Bogers & Loes, et al, 2020; Houli, et al. 2021
[14] Gillespie, 2014, p. 178
[15] Noble, 2018, p. 4; p. 16
[16] Google, ‘Ranking Results’
[17] Jansen & Spink, 2006; Steiner et al, 2022, p. 219
[18] Jasen & Spink, 2006
[19] Rogers 2019, p. 26
[20] Srnicek, 2017, p. 49
[21] Noble 2018, p. 38
[22] Meisner, C., Duffy, B. E., & Ziewitz, M., 2022, p. 3
[23] Meisner, C., Duffy, B. E., & Ziewitz, M. 2022, p. 2
[24] Noble, 2018, p. 45
References
Bogers, L., Niederer, S., Bardelli, F., & De Gaetano, C. (2020). ”Confronting Bias in the Online Representation of Pregnancy.” Convergence, 26(5–6), 1037–1059. https://doi.org/10.1177/1354856520938606
Gillespie, Tarleton. (2014. “The Relevance of Algorithms.” In T. Gillespie, P. J. Boczkowski, & K. A. Foot (Eds.), Media Technologies: Essays on Communication, Materiality, and Society (pp. 167–194). Cambridge, MA, USA: MIT Press.
Google. About Google. Accessed 21 Nov, 2022. https://about.google
Google. How the Search Works. Accessed 21 Nov, 2022. https://www.google.com/intl/en_us/search/howsearchworks/how-search-works/
Google. Ranking Results. Accessed 21 Nov, 2022. https://www.google.com/intl/en_us/search/howsearchworks/how-search-works/ranking-results/
Houli D, Radford ML, Singh VK. (2021) “‘COVID19 is_’: The Perpetuation of Coronavirus Conspiracy Theories via Google Autocomplete.” Proc Assoc Inf Sci Technol. 2021;58(1):218-229. doi: 10.1002/pra2.450.
Jansen, J. & Spink, A. (2006). “How Are We Searching the World Wide Web? A Comparison of Nine Search Engine Transaction Logs.” Information Processing & Management. 42. 248-263. 10.1016/j.ipm.2004.10.007.
Jürgens, P., & Stark, B. (2017). “The Power of Default on Reddit: A General Model to Measure the Influence of Information Intermediaries.” Policy & Internet, 9(4), 395–419. https://doi.org/10. 1002/poi3.166.
Latzer, M., Hollnbuchner, K., Just, N., & Saurwein, F. (2016). “The Economics of Algorithmic Selection on the Internet.” In J. M. Bauer & M. Latzer (Eds.), Handbook on the economics of the Internet (pp. 395–425). Edward Elgar.
Meisner, C., Duffy, B. E., & Ziewitz, M. (2022). “The Labor of Search Engine Evaluation: Making Algorithms More Human or Humans More Algorithmic?” New Media & Society, 0(0). https://doi.org/10.1177/14614448211063860.
Noble, Safiya U. (2018). Algorithms of Oppression: How Search Engines Reinforce Racism. New York: NYU Press.
Pan, B., Hembrooke, H., Joachims, T., Lorigo, L., Gay, G., & Granka, L. (2007). “In Google We Trust: Users’ Decisions on Rank, Position, and Relevance.” Journal of Computer-Mediated Communication, 12(3), 801–823. https://doi.org/10.1111/j.1083-6101.2007.0035.x.
Rogers, Richard. (2019/2nd edition forthcoming). Doing Digital Methods. Sage Publishing. London, United Kingdom.
Statista (January 2022). Most trusted sources of general news and information worldwide from 2011 to 2021. https://www.statista.com/statistics/381455/most-trusted-sources-of-news-and-info-worldwide/
Statista. (2022). Worldwide desktop market share of leading search engines from January 2010 to July 2022. Statista Research Department. https://www.statista.com/statistics/216573/worldwide-market-share-of-search-engines/.
Steiner, M., Magin, M., Stark, B. & Geiß, S. (2022) “Seek and You Shall Find? A Content Analysis on the Diversity of Five Search Engines’ Results on Political Queries,” Information, Communication & Society, 25:2, 217-241, DOI: 10.1080/1369118X.2020.1776367.
