Post-Data Update!
(Blog #2)

(Slideshow of my 3-minute presentation with more context!)

COntext

To provide context to my research over the past few months, my study seeks to investigate the effect of political polarization on the use of dark patterns within online news websites. At the beginning of my research, I hypothesized that the political lean of websites had an impact on the prevalence and frequency of dark patterns in news websites.

As a refresher, dark patterns refer to design elements within websites that are intentionally included to attempt to influence a user's actions into doing something that could benefit the website.

Contribution/Gap

My research and data are significant in contributing to the field of dark pattern research and UX (user experience) design as a whole since political polarization is a sparsely examined topic within the field. Most existing research (Zeng et al., 2020 and Soe et al., 2020) emphasizes the in-depth study of single patterns across all types of websites instead of factoring in external factors (such as political affiliation) as a possible influence on how dark patterns are used.

I am thereby contributing to the existing research conversation as my study examines whether the political lean of online news websites correlates with the number of patterns included within the website. As a result, this research will help provide insight into the impact of political polarization and vested interest within a website to researchers studying dark patterns and UX design as a whole.

Methodology & Data Collection

Since the first blog post (here), my research has not deviated significantly from my original methodology as I continued to use the websites categorized by Bakshy, et al. 2015 to select websites from each political lean (Left, Moderate, Right) using a Python program (included below). I then continued gathering my data using the methodology defined by Di Geronimo, et al. 2020 to analyze websites using 10-minute recordings and analyzing it by the dark pattern categories defined by (Gray, et al. 2018) (Nagging, Obstruction, Sneaking, Interface Interference, and Forced Action).

However, as compared to my original research, I included sub-categories for some patterns in my research data which appeared more frequently than others. This trend mainly culminated around the "Hidden Ad" Dark Pattern (essentially interface elements that are disguised to not look like ads), which dominated the interface interference category.

I chose to separate these sub-patterns into a different sub-category within my research data to allow for a more in-depth analysis of pattern trends, especially due to how significant of an impact it has had on my final data.

(Some examples of dark patterns (including "Hidden Ad") that I encountered during data collection are provided on the right with the website I found them on.)

(www.rawstory.com) - (Nagging)

(www.thefederalistpapers.org) - (Hidden Ad)

(www.thefederalistpapers.org) - (Sneaking)

Progress

As of this blog post, I have finished my data collection with all my 60 websites analyzed for dark patterns. I have also finished analyzing trends within my recorded data (as shown below), and I am currently constructing my final presentation slideshow with my thirty-minute presentation seemingly only over the horizon.

(Grand total of 470 patterns recorded!)

(Small example snippet of my data spreadsheet)

(The python program used to select websites)

Roadblocks

Throughout data collection, I experienced significant roadblocks in gathering my data, primarily regarding the websites I analyzed.

One more major roadblock came from specific websites that implemented procedurally generating dark patterns as the user kept scrolling. Originally, I did not account for this possibility as all the academic articles that I analyzed for this project did not mention such an instance. This became an issue as I was unable to record all the possible patterns within the website due to the 10-minute time limit for the videos. Additionally, these patterns (specifically Interface Interference) also make up a majority of my data at 297 of 470 recorded patterns. As excluding this pattern would essentially be deleting data, I decided to include it within my data analysis and will list it as a limitation of my results.

An additional roadblock I also encountered was with the "Forced Action" dark pattern, which I had to exclude from my statistical tests as I only got 3 total recorded patterns across all 60 websites, which caused it to fail the large counts condition (where the number of patterns in a category across the table had to exceed a certain number that was recorded) to conduct a Chi-Square test.

(Fig. A- Segmented Bar Graph of data separated by political lean)

(Fig. B - Frequency Table of each pattern and political lean)

(Fig. C - Contribution Table of each pattern and political lean)

(Fig. D - Expected Counts Table)

(Fig. E - Chi-Square Results)

Results/Analysis

To analyze my data, I conducted a Chi-Square test for Independence to determine whether my hypotheses hold true or not and if there is a significant difference between the data I obtained from a statistical lens. My hypotheses are listed below:

Null Hypothesis (Ho) = Political lean has no impact on the frequency of dark patterns in online news websites.

Alternate Hypothesis (Ha) = Political lean has an impact on the frequency of dark patterns in online news websites.

Starting with the topmost visual (Fig. A), it is of a segmented bar graph which represents how much a specific dark pattern is represented in each political lean as a percentage. It is essentially a graphed version of Fig. B, which also provides the precise number of patterns recorded for each dark pattern category for each political lean.

Fig. C represents a table that lists the contribution of each category, which is how much my data deviates from the calculated expected value (Fig. D) which are calculated values that assume no correlation between political lean and dark patterns. A higher contribution represents a more significant difference between the expected and actual data value. (This influences the end result)

Fig. E represents the results of the chi-square test (x^2), with the p-value being the probability of getting a result that conforms to the null hypothesis listed above..

Discussion

When analyzing my data, I determined that my trends were significant at a p-value of < 0.05 (when α = 0.05). which means that there is moderately convincing evidence that political lean does have an impact in the number of dark patterns.

This conclusion is also reiterated by the result of the Chi-Square test (X^2), as the value from the test (15.866) exceeds the critical value (12.592, which is the cutoff for significance) by a notable amount, which also represents that political lean has an impact on the frequency of patterns.

On a pattern-by-pattern basis, it appears that Interface interference (II) was the most commonly found within the moderate-leaning websites as it accounted for 44.6% of II patterns as opposed to left (28.8%) or right-leaning (26.6%) websites. Both left and moderate-leaning contributions for this category were also high, indicating a value that deviated more significantly than expected (right was only 0.04!).

Interestingly, contribution values for Nagging were significantly more prevalent within moderate and left-leaning websites. This trend contrasts other patterns such as Obstruction (more prominent in left/right leaning) and Sneaking (primarily left-leaning).

Overall, this portrays that left-leaning news websites tend to deviate much more significantly compared to their right and moderate-leaning counterparts for most patterns other than nagging, of which moderate and right-leaning tend to deviate more from than left-leaning websites.

While I have yet to find concrete research to help back up my conclusion, I have inferences regarding some of the trends I have discovered within my data.

- Moderate websites within my research tended to be more well-known websites that often serve a local community or city as opposed to a more specific political ideology. This could have contributed to my overwhelming Interface Interference category as "Hidden Ads" were the largest factor of the category and such websites likely have high traffic from the community causing them to implement more patterns to target the increased local traffic.

limitations

While I was able to collect a significant amount of data, there are multiple limitations regarding my research.

The most major limitation is the fact that I am the only one personally classifying and coding each website for dark patterns. Due to the scope of the research, I can't precisely follow the methodology provided by Di Geronimo, et al. 2020 to address this issue as they used two researchers to classify patterns to remove bias.

Additionally, the previously mentioned issue with infinitely generating patterns is another limitation as it is the primary cause of such a high Interface Interference category which may have influenced my final results.

Finally, due to the sheer number of websites that I had to record (60 videos, meaning 10 hours of recording), I was unable to record them all within a reasonable timeframe, which could lead to differences in the overall appearance of the websites as I did not record them all at once.

Conclusion

Nonetheless, my research still bears significant implications within the realm of dark patterns and UX (user experience) design research. My results can help become a starting point to incite more widespread research on the implications of political polarization online, especially how polarized the US has become on the advent of the 2024 election which faces many new and recurring political issues.

Even to the average everyday user, my results can also help inform people about how online news websites that advocate for a specific political lean may be designed to manipulate people; and possibly how to avoid them.

Overall, this update serves as the last and final stepping stone before my final presentation on April 3rd, and I would like to thank you for reading this lengthy and admittedly verbose blog post.

References

Bakshy, Eytan, et al. “Exposure to Ideologically Diverse News and Opinion on Facebook.” Science, vol. 348, no. 6239, May 2015, pp. 1130–1132, https://doi.org/10.1126/science.aaa1160.

Di Geronimo, L., Braz, L., Fregnan, E., Palomba, F., & Bacchelli, A. (2020). UI Dark Patterns and Where to Find Them: A Study on Mobile Applications and User Perception. Proceedings of the 2020 CHI Conference on Human Factors in Computing Systems, 1–14. https://doi.org/10.1145/3313831.3376600

Gray, C. M., Kou, Y., Battles, B., Hoggatt, J., & Toombs, A. L. (2018). The Dark (Patterns) Side of UX Design. CHI ’18: Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems, 1–14. https://doi.org/10.1145/3173574.3174108

Soe, T. H., Nordberg, O. E., Guribye, F., & Slavkovik, M. (2020). Circumvention by design - dark patterns in cookie consent for online news outlets. NordiCHI ‘20: Proceedings of the 11th Nordic Conference on Human-Computer Interaction: Shaping Experiences, Shaping Society, 1–12. https://doi.org/10.1145/3419249.3420132

Zeng, E., Kohno, T., & Roesner, F. (2020, May). Bad news: Clickbait and deceptive ads on news and misinformation websites. ConPro ‘20: Workshop on Technology and Consumer Protection, 1-11. https://badads.cs.washington.edu/files/Zeng-ConPro2020-BadNews.pdf

Post-Data Update! (Blog #2)