A privacy harm arises when damage is done to a person’s health, property, relationships, or opportunities in life. Their credit card information was stolen. Their medical information was used to discriminate against them in a job search process. Their personal life details were leaked on social media, damaging their reputation and net worth.
Data privacy harms occur in an adversarial setting. There is a malicious actor trying to thwart the environmental assumptions of the engineer. Their motivations are varying, ranging anywhere from academic research to monetary reward.
The engineers themselves often have some legitimate purpose for the data. The purpose may be selfish (like targeted advertising), but it does not cause undue harm to their users. Protecting user data from malicious purposes, on the other hand, is a software security problem. The data should be stored safely, anonymized, encrypted, noise-calibrated, aggregated whenever possible, secured, and deleted after its retention period has expired.
In the past, as discussed in our brief history of data privacy law, privacy harms were thought of in the context of individuals, civil harms, the press, insurance assessment, and hiring practices. Now, with the Internet, we have to consider more kinds of privacy harms and more kinds of adversaries.
A malicious computer user now has the power to inflict privacy harms on large populations all at once. For example, in the 2018 Facebook-Cambridge Analytica scandal, Cambridge-Analytica used an API created by Facebook to scrape millions of user profiles to aid in the political advertising campaigns of Donald Trump and Ted Cruz. While some of the blame lies with Cambridge-Analytica, the third-party, part of the blame lies with the Facebook engineers who designed the API for one use case, but failed to consider the use cases of a potential adversary.
Engineers not only have to protect against external adversaries; they also have to safeguard against adversaries from within their own engineering organization. In a 2014 study conducted by Facebook and Cornell data scientists, researchers discovered they could manipulate the feelings of Facebook users by altering what kinds of content they saw. In the study, there was no informed consent, no harm tracking, no effort to follow safety precedents for human subjects research or go through the Institutional Review Boards (IRBs). It is unknown how many users suffered mental health problems following the experiments. Facebook, in this case, failed to protect their users from privacy harms caused by their own enterprising employees.
Just as a bank needs to protect their users’ money vaults, engineers need to protect their users’ data from malicious and harmful use. This mean using the latest encryption schemes and differential privacy algorithms, but also consideration and monitoring of the kinds of things adversaries can do once they have access to user data. The past two examples were not a security breach in the classic sense; the “bankers” here simply forgot to place a lock on the vault. This was not a result of a lack in security technology; this was a result of an engineer’s failure of foresight and prioritization of functionality over safety.
Read more
Privacy Harms (this article)
Covert Surveillance (TBD)