{"id":761,"date":"2018-05-20T06:53:38","date_gmt":"2018-05-20T06:53:38","guid":{"rendered":"http:\/\/achangeiscoming.net\/?p=761"},"modified":"2024-01-20T05:13:10","modified_gmt":"2024-01-20T05:13:10","slug":"trip-memory-lane-part-2-social-threat-modeling","status":"publish","type":"post","link":"https:\/\/2024.thenexus.today\/index.php\/2018\/05\/20\/trip-memory-lane-part-2-social-threat-modeling\/","title":{"rendered":"A trip down memory lane: part 2 of &#8220;Social threat modeling&#8221; &#8212; DRAFT!"},"content":{"rendered":"<p>\t\t\t\t<em>Note: as Shireen Mitchell and others are discussing on Twitter, <a href=\"https:\/\/twitter.com\/digitalsista\/status\/996859628179816448\">&#8220;social threat modeling&#8221; isn&#8217;t necessarily a great name<\/a>.\u00a0 Suggestions welcome!<\/em><\/p>\n<p>Just as I was finishing <a href=\"https:\/\/medium.com\/a-change-is-coming\/social-threat-modeling-the-winds-of-change-are-in-the-air-8dc330479a50\">The winds of change are in the air<\/a>, Twitter helpfully provided an excellent opportunity to illustrate the value of applying threat modeling techniques to social problems.\u00a0 VP of Trust and Security Del Harvey&#8217;s <a href=\"https:\/\/blog.twitter.com\/official\/en_us\/topics\/product\/2018\/Serving_Healthy_Conversation.html\">Serving Healthy Conversation<\/a> describes their latest attempt to improve the <a href=\"https:\/\/www.amnesty.org\/en\/latest\/research\/2018\/03\/online-violence-against-women-chapter-1\/\">toxic environment on Twitter<\/a>: use behavioral algorithms to detect the small number of users that &#8220;negatively impact the health of the conversation.&#8221;\u00a0 What could possibly go wrong?<\/p>\n<p>Before we get to that, let&#8217;s take a stroll down memory lane &#8230;<\/p>\n<blockquote class=\"twitter-tweet\" data-lang=\"en\">\n<p dir=\"ltr\" lang=\"en\">We&#8217;re reverting the changes to block functionality. <a href=\"https:\/\/t.co\/H3W3V27rKN\">https:\/\/t.co\/H3W3V27rKN<\/a><\/p>\n<p>\u2014 Twitter (@Twitter) <a href=\"https:\/\/twitter.com\/Twitter\/status\/411340741864980480?ref_src=twsrc%5Etfw\">December 13, 2013<\/a><\/p><\/blockquote>\n<p><script async src=\"https:\/\/platform.twitter.com\/widgets.js\" charset=\"utf-8\"><\/script><\/p>\n<p>Leigh Honeywell&#8217;s <a class=\"hoverable\" href=\"https:\/\/modelviewculture.com\/pieces\/another-six-weeks-muting-vs-blocking-and-the-wolf-whistles-of-the-internet\" data-cke-saved-href=\"https:\/\/modelviewculture.com\/pieces\/another-six-weeks-muting-vs-blocking-and-the-wolf-whistles-of-the-internet\">Another Six Weeks: Muting vs. Blocking and the Wolf Whistles of the Internet<\/a> on <em>Model View Culture <\/em>has a good summary of what went wrong here:<\/p>\n<blockquote><p>In attempting to solve the problem of users being retaliated against for blocking, Twitter missed other ways that harassers operate on their service.\u00a0 Retweeting, in particular, is often used by harassers to expose the target\u2019s content to the friends of the harasser \u2013 potentially subjecting the target to a new wave of harassment. With the blocking functionality changed to work as \u201cmute\u201d, targets lost the ability to stop their harassers from retweeting them.<\/p><\/blockquote>\n<p>One reason that computer security is so complex is that there are so many different threats that it&#8217;s very easy for a change to trigger unexpected consequences &#8211; fixing one problem creates another worse. Threat modeling, done well, gives a structured way to analyze this.\u00a0\u00a0\u00a0 Let&#8217;s start with the simplified threat model for harassment Shireen Mitchell and I sketched out for <a href=\"https:\/\/medium.com\/a-change-is-coming\/diversity-friendly-software-at-sxsw-2017-references-c0ca05a191a6\">our March 2017 SXSW talk<\/a> (although we wound up not presenting it), and then Kelly Ireland and I refined as part of my talk at <a href=\"https:\/\/medium.com\/a-change-is-coming\/transforming-tech-with-diversity-friendly-software-338f56d91df\">TRANSform Tech later that month<\/a>.<\/p>\n<p><img loading=\"lazy\" decoding=\"async\" class=\"aligncenter size-large wp-image-750\" src=\"http:\/\/achangeiscoming.net\/wp-content\/uploads\/sites\/7\/2018\/05\/example-threat-model-1024x551.png\" alt=\"Threat model for different ways of harrassing people\" width=\"1024\" height=\"551\" \/><\/p>\n<p>One of the ways harassers attack people is to flood them with messages; one of the ways to do that is to get people to help.\u00a0 There are several different ways to do that, of course, one of which is retweeting.\u00a0 Taking the analysis down to the next level:<\/p>\n<p><img loading=\"lazy\" decoding=\"async\" class=\"aligncenter size-large wp-image-765\" src=\"http:\/\/achangeiscoming.net\/wp-content\/uploads\/sites\/7\/2018\/05\/get-people-to-help-1024x379.png\" alt=\"Get people to help by: (1) sending them a link (2) retweet so followers see it (3) ...\" width=\"1024\" height=\"379\" \/><\/p>\n<p>Before Twitter&#8217;s changes, there was an easy way for the target to close off this avenue of attack: block the harasser so they can&#8217;t see the tweet.<\/p>\n<p><img loading=\"lazy\" decoding=\"async\" class=\"aligncenter size-large wp-image-766\" src=\"http:\/\/achangeiscoming.net\/wp-content\/uploads\/sites\/7\/2018\/05\/block-harasser-1024x408.png\" alt=\"Similar to the previous diagram but with a big circle &quot;Block harasser&quot; covering up the box saying &quot;retweet so followers see it&quot;\" width=\"1024\" height=\"408\" \/><\/p>\n<p>When Twitter changed blocking functionality, it re-opened up that avenue of harassment.<\/p>\n<p>Of course, the harasser has other options as well.\u00a0\u00a0 But as Leigh Honeywell points out:<\/p>\n<blockquote><p>When the unannounced change was noticed, users and commentators argued that a determined harasser could have always copied-and-pasted a target\u2019s tweets, set up new accounts, or otherwise worked around the existing blocking functionality, and that the original blocking functionality represented a false sense of security. These arguments ignored the value of that functionality for dealing with unmotivated, low-grade and opportunistic harassers.<\/p><\/blockquote>\n<p>If Twitter had done a good job of threat modeling, they would have considered the variety of threats, everything from organized alt-right campaigns to the kinds of opportunistic-but-relatively-lazy people who will retweet because it&#8217;s an easy way of poking somebody while showing off to their buddies, but won&#8217;t devote a lot of effort to it.\u00a0 Targets of harassment understand these differences, of course.\u00a0\u00a0 But as Leigh Honeywell says<\/p>\n<blockquote><p>While I do not know what consultation Twitter did in deciding how this feature change would impact their users, the magnitude of the response suggests that it wasn\u2019t enough. Engaging users who are directly impacted by harassment must be central to any platform\u2019s efforts at combating abuse.<\/p><\/blockquote>\n<p>Another useful things about applying threat modeling to harassment is that it encourages you to think from the targets&#8217; points of view.\u00a0\u00a0 Still, &#8220;find out what people using the software want&#8221; is software engineering 101 whether or not you&#8217;re doing thread modeling. Twitter has no excuse for not doing that here.<\/p>\n<p>Why didn&#8217;t it happen?\u00a0\u00a0 Leigh Honeywell that the rapid and intense criticism of &#8220;emerged in large part from marginalized communities, who are\u00a0<a href=\"http:\/\/cdm16064.contentdm.oclc.org\/utils\/getfile\/collection\/p266901coll4\/id\/655\/filename\/617.pdf\">disproportionately affected by online abuse.<\/a>&#8221;\u00a0 Hold that thought, as we flash forward a few years.<\/p>\n<blockquote class=\"twitter-tweet\" data-lang=\"en\">\n<p dir=\"ltr\" lang=\"en\">We want you to get notifications that matter. Starting today, you won&#8217;t get notified when you are added to a list. <a href=\"https:\/\/t.co\/82xiaZbg1W\">https:\/\/t.co\/82xiaZbg1W<\/a><\/p>\n<p>\u2014 Twitter Safety (@TwitterSafety) <a href=\"https:\/\/twitter.com\/TwitterSafety\/status\/831247282544599040?ref_src=twsrc%5Etfw\">February 13, 2017<\/a><\/p><\/blockquote>\n<p><script async src=\"https:\/\/platform.twitter.com\/widgets.js\" charset=\"utf-8\"><\/script><\/p>\n<blockquote class=\"twitter-tweet\" data-lang=\"en\">\n<p dir=\"ltr\" lang=\"en\">Are you serious? <a href=\"https:\/\/twitter.com\/jack?ref_src=twsrc%5Etfw\">@jack<\/a> <a href=\"https:\/\/twitter.com\/mrdonut?ref_src=twsrc%5Etfw\">@mrdonut<\/a>? We wanted to be able to report abusive users adding us to lists not stop being alerted when it happened! <a href=\"https:\/\/t.co\/r3p2Zmb0Ew\">https:\/\/t.co\/r3p2Zmb0Ew<\/a><\/p>\n<p>\u2014 \ud83c\udd78\ud83c\udd89\ud83c\udd89\ud83c\udd88 (@iglvzx) <a href=\"https:\/\/twitter.com\/iglvzx\/status\/831263589667217408?ref_src=twsrc%5Etfw\">February 13, 2017<\/a><\/p><\/blockquote>\n<p><script async src=\"https:\/\/platform.twitter.com\/widgets.js\" charset=\"utf-8\"><\/script><\/p>\n<blockquote class=\"twitter-tweet\" data-lang=\"en\">\n<p dir=\"ltr\" lang=\"en\">We heard your feedback &#8211; this was a misstep. We\u2019re rolling back the change and we\u2019ll keep listening. <a href=\"https:\/\/t.co\/9ozPS4XlVI\">https:\/\/t.co\/9ozPS4XlVI<\/a><\/p>\n<p>\u2014 Twitter Safety (@TwitterSafety) <a href=\"https:\/\/twitter.com\/TwitterSafety\/status\/831281441082265600?ref_src=twsrc%5Etfw\">February 13, 2017<\/a><\/p><\/blockquote>\n<p><script async src=\"https:\/\/platform.twitter.com\/widgets.js\" charset=\"utf-8\"><\/script><\/p>\n<p>Hey wait a second, I&#8217;m noticing a pattern here!<\/p>\n<p>Sarah Perez&#8217; <a class=\"hoverable\" href=\"https:\/\/techcrunch.com\/2017\/02\/14\/twitter-quickly-kills-a-poorly-thought-out-anti-abuse-measure\/\" data-cke-saved-href=\"https:\/\/techcrunch.com\/2017\/02\/14\/twitter-quickly-kills-a-poorly-thought-out-anti-abuse-measure\/\">Twitter quickly kills a poorly thought out anti-abuse measure<\/a> on <em>TechCrunch <\/em>goes into more detail.\u00a0 Before Twitter&#8217;s changes, adding people to lists meant that they got a notification.\u00a0 Harassers would create a list with an offensive name and use this to bombard people with offensive notifications.\u00a0\u00a0 Removing notifications when people are added to lists got rid of this avenue of harassment &#8211; but overlooked the fact that they were other important reasons for the notifications.<\/p>\n<p>Once again, Twitter ignored the perspective of the targets of harassment.<\/p>\n<blockquote class=\"twitter-tweet\" data-lang=\"en\">\n<p dir=\"ltr\" lang=\"en\">literally 10 (paid) Black Women with &gt;5K followers would head all this crap off at the pass \ud83d\udc85\ud83c\udffd<\/p>\n<p>\u2014 LeslieMac (@LeslieMac) <a href=\"https:\/\/twitter.com\/LeslieMac\/status\/831508266949689344?ref_src=twsrc%5Etfw\">February 14, 2017<\/a><\/p><\/blockquote>\n<p><script async src=\"https:\/\/platform.twitter.com\/widgets.js\" charset=\"utf-8\"><\/script><\/p>\n<p>Indeed.\u00a0 <a href=\"https:\/\/medium.com\/a-change-is-coming\/listen-to-black-women-262cd10fbe44\">Listen to black women!<\/a><\/p>\n<p>And more generally: marginalized communities are\u00a0disproportionately affected by online abuse.\u00a0\u00a0 Even by the low standards of the tech industry, Twitter&#8217;s diversity numbers are pretty bad.*\u00a0 As former Twitter engineer Leslie Miley said in Charlie Warzel&#8217;s <a class=\"hoverable\" href=\"https:\/\/www.buzzfeed.com\/charliewarzel\/a-honeypot-for-assholes-inside-twitters-10-year-failure-to-s?utm_term=.bi4p0yxao#.xoo128mKk\" data-cke-saved-href=\"https:\/\/www.buzzfeed.com\/charliewarzel\/a-honeypot-for-assholes-inside-twitters-10-year-failure-to-s?utm_term=.bi4p0yxao#.xoo128mKk\">&#8220;A Honeypot For Assholes&#8221;: Inside Twitter\u2019s 10-Year Failure To Stop Harassment<\/a><\/p>\n<blockquote><p>The decision-makers were not people who got abuse and didn&#8217;t understand that it\u2019s not about content, it&#8217;s about context. If Twitter had people in the room who&#8217;d been abused on the internet \u2014 meaning not just straight, white males \u2014 when they were creating the company, I can assure you the service would be different.<\/p><\/blockquote>\n<p>Just to be crystal clear: software engineering techniques do not substitute for having a diverse team, inclusive culture, and equitable power distribution and compensation.\u00a0\u00a0 Sure, &#8220;social threat modeling&#8221; can be useful even for relatively-homogeneous product development teams, as long as they can work with (and listen to) other voices like social scientists and marginalized people in their community.\u00a0\u00a0 But the technical aspects aren&#8217;t enough by themselves.\u00a0 As Shireen Mitchell says, &#8220;The solution is multifaceted. Those that chose just one path will fail. We need all of it.&#8221;<\/p>\n<p>Instead of doing that, though, Twitter&#8217;s trying to throw technology at the problem. \u00a0 Machine learning!\u00a0 Artificial intelligence! \u00a0\u00a0Behavioral algorithms!\u00a0 In part 3 of the series, we&#8217;ll use &#8220;social threat modeling&#8221; to explore some of the reasons this won&#8217;t work.\u00a0 A teaser:<\/p>\n<blockquote class=\"twitter-tweet\" data-lang=\"en\">\n<p dir=\"ltr\" lang=\"en\">This is a recipe for disaster and DISCRIMINATION. &#8220;its algorithm will mute certain users so tweets fall behind a \u201csee more replies\u201d link and content will no longer be searchable. The primary factors will be complaints, blocks, and mutes an account receives.&#8221; <a href=\"https:\/\/t.co\/3RQoi0G9dn\">https:\/\/t.co\/3RQoi0G9dn<\/a><\/p>\n<p>\u2014 Erin Biba (@erinbiba) <a href=\"https:\/\/twitter.com\/erinbiba\/status\/996477855243612161?ref_src=twsrc%5Etfw\">May 15, 2018<\/a><\/p><\/blockquote>\n<p><script async src=\"https:\/\/platform.twitter.com\/widgets.js\" charset=\"utf-8\"><\/script><\/p>\n<p>&nbsp;<\/p>\n<p>&nbsp;<\/p>\n<hr \/>\n<p>* According to their 2017 report, <a href=\"https:\/\/www.usatoday.com\/story\/tech\/news\/2017\/01\/19\/twitter-diversity-2016\/96749454\/\">only 14% of the people in technical roles are women<\/a>, so there probably weren&#8217;t a lot of women of color involved on the product development team.\u00a0 And only 2% of the company&#8217;s employees are African American.\t\t<\/p>\n","protected":false},"excerpt":{"rendered":"<p>Note: as Shireen Mitchell and others are discussing on Twitter, &#8220;social threat modeling&#8221; isn&#8217;t necessarily a great name.\u00a0 Suggestions welcome! Just as I was finishing The winds of change are in the air, Twitter helpfully provided an excellent opportunity to illustrate the value of applying threat modeling techniques to social problems.\u00a0 VP of Trust and [&hellip;]<\/p>\n","protected":false},"author":1,"featured_media":0,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[420,14,421,16],"tags":[378],"class_list":["post-761","post","type-post","status-publish","format-standard","hentry","category-drafts","category-social-sciences","category-software","category-tales-from-the-net","tag-twitter"],"_links":{"self":[{"href":"https:\/\/2024.thenexus.today\/index.php\/wp-json\/wp\/v2\/posts\/761","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/2024.thenexus.today\/index.php\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/2024.thenexus.today\/index.php\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/2024.thenexus.today\/index.php\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/2024.thenexus.today\/index.php\/wp-json\/wp\/v2\/comments?post=761"}],"version-history":[{"count":1,"href":"https:\/\/2024.thenexus.today\/index.php\/wp-json\/wp\/v2\/posts\/761\/revisions"}],"predecessor-version":[{"id":4332,"href":"https:\/\/2024.thenexus.today\/index.php\/wp-json\/wp\/v2\/posts\/761\/revisions\/4332"}],"wp:attachment":[{"href":"https:\/\/2024.thenexus.today\/index.php\/wp-json\/wp\/v2\/media?parent=761"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/2024.thenexus.today\/index.php\/wp-json\/wp\/v2\/categories?post=761"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/2024.thenexus.today\/index.php\/wp-json\/wp\/v2\/tags?post=761"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}