<story><title>Facebook Stored Hundreds of Millions of User Passwords in Plain Text for Years</title><url>https://krebsonsecurity.com/2019/03/facebook-stored-hundreds-of-millions-of-user-passwords-in-plain-text-for-years/</url></story><parent_chain><item><author>the_duke</author><text>(Edit: reworded a bit to make it clear I don&amp;#x27;t think this is acceptable)&lt;p&gt;Sounds a lot like some service was logging the full body of a signup&amp;#x2F;login request, which then was readable for anyone with access to the logging&amp;#x2F;tracing infrastructure.&lt;p&gt;Dumb mistake but it&amp;#x27;s not hard to imagine this happening,  considering that FB probably has a bunch of services involved in the login&amp;#x2F;signup flow to prevent bots&amp;#x2F;spam, abuse, etc.&lt;p&gt;Not to imply this is acceptable, especially at a IT company like FB with vast resources and know how. Raw passwords are an especially big screw-up.
There are a lot of failures here, from actually logging something so sensitive over giving access to so many employees to not noticing this for years. (Assuming this was actually log data).&lt;p&gt;BUT if we are honest, anonymizing log data is rarely a priority. Even if it is, leaking sensitive data can happen easily in a lot of different points in the infrastructure. In actual application code, client + server exception tracing (just imagine a deserialization exception which contains part of the input) , web server, load balancer, proxies, service mesh... 
There is a lot of interesting stuff hiding in the logs at pretty much every company.&lt;p&gt;This is a good time to look in the mirror and audit your logging and tracing data. Unless you are in a highly regulated field like finance&amp;#x2F;healthcare or  there is a strong company-wide culture for security&amp;#x2F;privacy with regular audits already, I can almost guarantee you will find at least one data point that should not be where it is.&lt;p&gt;Protecting sensitive data needs to be a big consideration for every dev, ops and especially management, which has to allocate enough time for security reviews and audits.</text></item></parent_chain><comment><author>mk89</author><text>&amp;gt; BUT if we are honest, anonymizing log data is rarely a priority. Even if it is, leaking sensitive data can happen easily in a lot of different points in the infrastructure. In actual application code, client + server exception tracing (just imagine a deserialization exception which contains part of the input) , web server, load balancer, proxies, service mesh... There is a lot of interesting stuff hiding in the logs at pretty much every company.&lt;p&gt;I think so too, and I agree with you.&lt;p&gt;A couple of months ago we had to discuss with a colleague that wanted to log the request&amp;#x2F;response body because &amp;quot;without it, it makes debugging almost impossible&amp;quot;. We certainly don&amp;#x27;t log, and we made sure that even exceptions don&amp;#x27;t spit internal data, and if it happens, it&amp;#x27;s only an exceptional case, a bug (so far, I didn&amp;#x27;t see). It&amp;#x27;s tough, because of course when you have access to plain data you can do anything you want to, but we owe our customers this additional step. Laziness can&amp;#x27;t be justified in this context.&lt;p&gt;However, generally speaking, I think it&amp;#x27;s a common practice in many companies, it&amp;#x27;s not the first time I see this happen, and I bet it&amp;#x27;s not gonna be the last.&lt;p&gt;It&amp;#x27;s a matter of mindset. The more I read such articles, the more I lose trust in even large companies. Typically smaller companies don&amp;#x27;t care, in order to move &amp;quot;fast&amp;quot;. However, when this happens to large corporations, where does it end? Today it&amp;#x27;s facebook with the passwords, tomorrow it might be amazon with tons of credit card numbers because of a legacy system not anymore maintained...</text></comment>
<story><title>Facebook Stored Hundreds of Millions of User Passwords in Plain Text for Years</title><url>https://krebsonsecurity.com/2019/03/facebook-stored-hundreds-of-millions-of-user-passwords-in-plain-text-for-years/</url></story><parent_chain><item><author>the_duke</author><text>(Edit: reworded a bit to make it clear I don&amp;#x27;t think this is acceptable)&lt;p&gt;Sounds a lot like some service was logging the full body of a signup&amp;#x2F;login request, which then was readable for anyone with access to the logging&amp;#x2F;tracing infrastructure.&lt;p&gt;Dumb mistake but it&amp;#x27;s not hard to imagine this happening,  considering that FB probably has a bunch of services involved in the login&amp;#x2F;signup flow to prevent bots&amp;#x2F;spam, abuse, etc.&lt;p&gt;Not to imply this is acceptable, especially at a IT company like FB with vast resources and know how. Raw passwords are an especially big screw-up.
There are a lot of failures here, from actually logging something so sensitive over giving access to so many employees to not noticing this for years. (Assuming this was actually log data).&lt;p&gt;BUT if we are honest, anonymizing log data is rarely a priority. Even if it is, leaking sensitive data can happen easily in a lot of different points in the infrastructure. In actual application code, client + server exception tracing (just imagine a deserialization exception which contains part of the input) , web server, load balancer, proxies, service mesh... 
There is a lot of interesting stuff hiding in the logs at pretty much every company.&lt;p&gt;This is a good time to look in the mirror and audit your logging and tracing data. Unless you are in a highly regulated field like finance&amp;#x2F;healthcare or  there is a strong company-wide culture for security&amp;#x2F;privacy with regular audits already, I can almost guarantee you will find at least one data point that should not be where it is.&lt;p&gt;Protecting sensitive data needs to be a big consideration for every dev, ops and especially management, which has to allocate enough time for security reviews and audits.</text></item></parent_chain><comment><author>kissickas</author><text>&amp;gt; readable for anyone with access to the logging&amp;#x2F;tracing infrastructure&lt;p&gt;The article says more than 20,000 Facebook employees. A quick search shows that they have around 35k now - do 60% of employees at Facebook really need access to all of those logs?</text></comment>