We’ve had several decades of platforms and apps collecting data about us – and in a heavyweight panel on Sunday at SXSW, the debate turned to how that data is being used to both make assumptions about us and alter the products and services we’re offered.
In the introduction, Ashkan Soldani referenced IBM’s development of software that was designed to ascertain whether individuals arriving into Europe from Syria were terrorists or refugees. Using multiple data sources the software creates a ‘terrorist score’ which determines the likelihood that someone is involved in terrorism activity.
And while the motivation behind this kind of tool is understandable, the potential for misuse or mistakes is clear.
Nicole Wong, former U.S. Deputy Chief Technology Officer for the Obama administration, introduced the concept of negative selection algorithms, citing the case of a major university in the U.S. that changed its procedures for recruitment to its highly regarded computer science degree when it became clear the initial screening process had inadvertently discriminated against women.;