There are a lot of reasons people might not want their posts on a social network to be indexed by search engines. \u00a0One of the most important is personal safety. Harassers often use search engines to find out information about the people they’re targeting \u2013 or to find new people to target. \u00a0So Mastodon gives you the option (on the Preferences\/Other settings page) to opt out of search engine indexing.<\/p>\n
Unfortunately, as we’ll discuss below, selecting this option doesn’t actually fully opt you out of search engine indexing. \u00a0<\/p>\n
If you’re familiar with Mastodon’s cavalier approach to privacy and security, this probably doesn’t come as a surprise to you. \u00a0As hachyderm.io admin Kris Nova says in Operating Mastodon, Privacy, and Content<\/a><\/p>\n My immediate advice is to treat everything on Mastodon as if it is public data!<\/strong><\/p><\/blockquote>\n However, a lot of people think Mastodon’s more private and secure than it really is. For one thing, Mastodon has long positioned itself as reducing harassment by learning from Twitter\u2019s mistakes<\/a>. \u00a0And making it harder for harassers to search for past messages is often used as an example of this. \u00a0As EFF’s Bill Buddington says in \u00a0Is Mastodon Private and Secure?<\/a> <\/p>\n “This cuts down on harassment, because abusive accounts will have a harder time discovering posts and accounts using key words typically used by the population they\u2019re targeting (a technique frequently used by trolls and harassers)”<\/p><\/blockquote>\n So having this option that doesn’t actually work gives people a false sense of security \u2013 but leaves them exposed if they’re at risk of harassment.<\/p>\n Understanding why this option doesn’t work as expected requires a bit of digging into how “noindex” rule works in search engines and how Mastodon treats the opt-out setting.<\/p>\n As Google’s Block Search Indexing with \u2018noindex’<\/a> describes, when an HTML page has a <meta name=”robots” content=”noindex”> tag, Google and other search engines that support the noindex rule won’t index the page. \u00a0Of course, badly-behaved search engines can<\/em> ignore the tag, so this won’t stop people who write their own crawlers. \u00a0Still, writing a crawler and storing all your own data is a pretty significant investment, so this is a useful level of protection.<\/p>\n Mastodon version 4.0 always puts a noindex tag on most pages<\/a>, but whether or not it’s on your profile depends on the “opt-out of search engine indexing” option. \u00a0The option is also used to determine whether there’s a noindex tag on web pages for your posts. \u00a0My indieweb.social account has that option turned on, so if you look at the HTML for this post<\/a>, you’ll see the noindex rule. \u00a0So far so good.<\/p>\n But if I do a search for “@jdp23@indieweb.social<\/a>“, I’ll find pages with my posts in them. \u00a0In fact, I can even do a search for specific text in one of my posts \u2013 here’s one for “jdp23 gh*st<\/a>” that brings up a thread Anil Dash started that I replied to. \u00a0In this particular case I don’t care, but imagine a situation where instead of gh*st I had used a term that attracts harassers. \u00a0<\/p>\n That’d be very bad. \u00a0<\/p>\n And as Darius Kazemi verified<\/a>, this applies to unlisted posts as well as public posts. <\/p>\n From a software perspective, the bug here is that Mastodon is only checking the “opt out of search engines” setting for the original author. \u00a0Anil, like many others, doesn’t mind if his posts are indexed by search engines. \u00a0When I reply to him, that means that my post will be indexed by search engines as well \u2013 even though I’ve opted out.<\/p>\n I filed a bug report<\/a> on this and it’ll be interesting to see what the response is.<\/p>\n This isn’t the only way your public and unlisted Mastodon posts can wind up in a search engine even if you’ve opted out. \u00a0If somebody from another instance is following you, there’s no guarantee that the software they’re running will pay attention to the “opt-out from search engines” setting. \u00a0As long as the other instances are running Mastodon software, this isn’t an issue unless admins have intentionally disabled this functionality. \u00a0However, other software that’s compatible with Mastodon may not know about this setting. \u00a0<\/p>\n In fact, because of the way Mastodon implements federation, even posts that have been deleted have copies on other instances that can still be found by search engines. \u00a0Yikes!<\/p>\n It’s worth noting that “local-only posts<\/a>,” supported by Mastodon forks (variants) like Glitch and Hometown, provide significant protection here. \u00a0Local-only posts aren’t included in externally-accesses pages, so search engines never see them. As Hometown maintainer Kazemi points out, if you’re on an instance where you trust the admins and the other members, local-only posts give you the ability to ensure that your stuff only goes to actors you personally trust. Unfortunately, Mastodon’s BDFL (benevolent dictator for life) has rejcted this valuable anti-harassment technology from the main line of code, so most instances don’t have this functionality.<\/p>\n Of course, Mastodon’s not the only social network site where you don’t have any privacy. \u00a0Twitter allows you to delete your tweets and direct messages, but doesn’t actually commit to deleting them from their internal databases or backups. \u00a0And since there’s currently more organized harassment on Twitter than Mastodon, and their investors (including Larry Ellison of Oracle, Prince Alwaleed bin Talal bin Abdulaziz of Saudi Arabia, and the Qatar Investment Authority) get special rights to your personal data, the risks are likely higher there. \u00a0<\/p>\n Still, don’t kid yourself: Mastodon’s security and privacy story is not good. Lenin Alevski recently found a system misconfiguration vulnerability making content and videos from supposedly-private direct messages open to the world<\/a>; as well as infosec.exchange’s 33,000 users, Alevski reports this affected several other high-profile sites. \u00a0The lack of end-to-end encryption means that admins can read supposedly-private direct messages \u2013 and if you’re DM’ing with somebody on another instance, their admins can read it as well. \u00a0 <\/p>\nWhy doesn’t opting out really opt you out?<\/h2>\n
But wait, there’s more<\/h3>\n