Archive for the ‘Policy’ Category

What Kickstarter Did Right

Monday, February 17th, 2014

Only a few details have emerged about the recent breach at Kickstarter, but it appears that this one will be a case study in doing things right both before and after the breach.

What Kickstarter has done right:

  • Timely notification
  • Clear messaging
  • Limited sensitive data retention
  • Proper password handling

Timely notification

The hours and days after a breach is discovered are incredibly hectic, and there will be powerful voices both attempting to delay public announcement and attempting to rush it. When users’ information may be at risk beyond the immediate breach, organizations should strive to make an announcement as soon as it will do more good than harm. An initial public announcement doesn’t have to have all the answers, it just needs to be able to give users an idea of how they are affected, and what they can do about it. While it may be tempting to wait for full details, an organization that shows transparency in the early stages of a developing story is going to have more credibility as it goes on.

Clear messaging

Kickstarter explained in clear terms what was and was not affected, and gave straightforward actions for users to follow as a result. The logging and access control groundwork for making these strong, clear statements at the time of a breach needs to be laid far in advance and thoroughly tested. Live penetration testing exercises with detailed post mortems can help companies decide if their systems will be able to capture this critical data.

Limited sensitive data retention

One of the first questions in any breach is “what did they get?”, and data handling policies in place before a breach are going to have a huge impact on the answer. Thinking far in advance about how we would like to be able to answer that question can be a driver for getting those policies in place. Kickstarter reported that they do not store full credit card numbers, a choice that is certainly saving them some headaches right now. Not all businesses have quite that luxury, but thinking in general about how to reduce the retention of sensitive data that’s not actively used can reduce costs in protecting it and chances of exposure over the long term.

Proper password handling (mostly)

Kickstarter appears to have done a pretty good job in handling user passwords, though not perfect. Password reuse across different websites continues to be one of the most significant threats to users, and a breach like this can often lead to ripple effects against users if attackers are able to obtain account passwords.

In order to protect against this, user passwords should always be stored in a hashed form, a representation that allows a server to verify that a correct password has been provided without ever actually storing the plaintext password. Kickstarter reported that their “passwords were uniquely salted and digested with SHA-1 multiple times. More recent passwords are hashed with bcrypt.” When reading breach reports, the level of detail shared by the organization is often telling and these details show that Kickstarter did their homework beforehand.

A strong password hashing scheme must protect against the two main approaches that attackers can use: hash cracking, and rainbow tables. The details of these approaches have been well-covered elsewhere, so we can focus on what Kickstarter used to make their users’ hashes more resistant to these attacks.

To resist hash cracking, defenders want to massively increase the amount of work an attacker has to do to check each possible password. The problem with hash algorithms like SHA1 and MD5 is that they are too efficient; they were designed to be completed in as few CPU cycles as possible. We want the opposite from a password hash function, so that it is reasonable to check a few possible passwords in normal use but computationally ridiculous to try out large numbers of possible passwords during cracking. Kickstarter indicated that they used “multiple” iterations of the SHA1 hash, which multiplies the attacker effort required for each guess (so 5 iterations of hashing means 5 times more effort). Ideally we like to see a hashing attempt take at least 100 ms, which is a trivial delay during a legitimate login but makes large scale hash cracking essentially infeasible. Unfortunately, SHA1 is so efficient that it would take more than 100,000 iterations to raise the effort to that level. While Kickstarter probably didn’t get to that level (it’s safe to assume they would have said so if they did), their use of multiple iterations of SHA1 is an improvement over many practices we see.

To resist rainbow tables, it is important to use a long, random, unique salt for each password. Salting passwords removes the ability of attackers to simply look up hashes in a precomputed rainbow tables. Using a random, unique salt on each password also means that an attacker has to perform cracking on each password individually; even if two users have an identical password, it would be impossible to tell from the hashes. There’s no word yet on the length of the salt, but Kickstarter appears to have gotten the random and unique parts right.

Finally, Kickstarter’s move to bcrypt for more recent passwords is particularly encouraging. Bcrypt is a modern key derivation function specifically designed for storing password representations. It builds in the idea of strong unique salts and a scalable work factor, so that defenders can easily dial up the amount computation required to try out a hash as computers get faster. Bcrypt and similar functions such as PBKDF2 and the newer scrypt (which adds memory requirements) are purpose built make it easy to get password handling right; they should be the go-to approach for all new development, and a high-priority change for any codebases still using MD5 or SHA1.

Humble Helps

Sunday, May 23rd, 2010

I just ran across a post at PreachSecurity about a recent CSRF discovered in OpenCart, and a blog post by the discoverer about his interactions with the maintainer. I share Rafal’s (and Ben’s) frustration with the interaction, but I think there’s an additional lesson to be learned here. Clearly this was a loss for the cause of security, but I think it was a loss in more ways than many of the participants see.

Ben didn’t just lose the battle to get one bug fixed, he also lost an ally in the fight. Daniel (of OpenCart) was and still is in the best position to make small changes to vastly improve the security of OpenCart for all users, but after a public browbeating what are the chances he’ll be willing to get help from security folks? That’s the critical part.

The bug by bug approach to information security is trench warfare: it will take us too many years and too many lives (or at least careers) to gain ten feet of mutilated ground. We need people and practices on our side, and “wins” in security should be measured in those terms, not just in bug counts.

Yes, we as security people often have to tell people that some code or process or idea is absolutely wrong. But to say those things without destroying the human capital we desperately need, we must do it with an overabundance of patience, humility, goodwill, and tact.

I think Ben did a reasonable job of trying to be respectful and helpful, but there was something jarring in it for me.

From: “Ben”
Sent: Friday, January 22, 2010 8:06 PM
To: < *******>
Subject: OpenCart – Enquiry


I recently installed OpenCart and I noticed that it is vulnerable to CSRF attacks. I have created a sample page that is capable of inserting a rouge user (the page currently prompts you but could be done silently if the attacker knows the url of the site).*********.html

Please let know that you are looking into the security issue and are going to release an update with a fix otherwise I will make the issue public.

If you need any help fixing the problem please let me know.


See it? “…otherwise I will make the issue public.” Now, as a security person that may not seem hugely disconcerting; we know that when the carrot of coordinated disclosure fails, the stick of public disclosure can get results.  However, put yourself in Daniel’s position: he just got threatened. Reread the email, and think honestly about whether anything else constructive that Ben said will have as much of an influence on Daniel as the mere feeling that he’s being bullied.

So, Daniel gets a lot of emails from people that misunderstand some code, and he sends back a quick response (on a Friday evening no less):

On 2010-01-22, at 4:50 PM, Daniel Kerr wrote:
Ben you seem to be very clever to come up with this. But! you need to be logged in for this to happen.

So, clearly he didn’t get it. Amazingly, he still seems to be receptive to Ben. On the technical side, he may not even have checked out the PoC Ben provided. Ben responds with a lot of good information. It’s a technically accurate and helpful response for someone who is ready to learn about CSRF, but this is how he leads off:

HI Daniel,
That is the whole point of a CSRF attack. Please read for an explanation on the attack.

Ouch. I’m sure it wasn’t Ben’s intent (or maybe he was just frustrated; understandable), but that line right there is going to put Daniel in defense mode. It’s subtle, but it’s an “I’m right, you’re wrong” moment. Even if Ben is right (he is), anyone’s ego would step in and interrupt rational thought right there.

Imagine if the second email had been even more patient and humble.

Hi Daniel,
Yes, you’re right that it requires the OpenCart to be logged in, but CSRF really is a commonly used attack, and it can be very dangerous <<insert Ben’s other paragrahs here; they’re a great description of how CSRF could be exploited.>>

There’s more good information on wikipedia [link], and there’s actually a pretty straightforward fix that can eliminate CSRF vulnerabilities  [link to owasp CSRF page, or whatever you like]. I’ve attached some files that fix these vulns, and added some anti-CSRF functionality to the URL class to make it easy to clean up any other cases.

Instead, things kinda spiral downward. It ends with:

On 2010-01-22, at 8:05 PM, Daniel Kerr wrote:
what protection do you recommend?

followed immediately by:

On 2010-01-22, at 8:05 PM, Daniel Kerr wrote:
“…your [sic] just wasting my time.”

Communication is no longer occurring. We as security people must take the responsibility to prevent that. What if, at the moment that things were spiraling downhill, Ben had sent an email like.

Hi Daniel,
I hear your frustration with trying to protect users from doing dumb stuff, and I agree there’s no way to fix all the stupid things they could do, but at least CSRF is one type of attack that we can stop cold. If you have ten minutes, I’d love to talk about why I think it’s really important, and how some protections could be added to OpenCart without too much effort.

My phone number is 555-555-1212: call me any time, or let me know if there’s a good time to call you.

Yes, I know it’s crazy. It’s over the top, it’s above and beyond the call of duty, and it’s kinda weird: use the telephone… really? But that might, just might, have helped Ben win not just the battle over a single bug, but might have won an ally in the security of an entire application, and gained the cause of security goodwill in the bargain.

I’m not saying that there aren’t complete, incorrigible, asshat people out there, or that killing them with kindness is any kind of panacea. Ultimately, Ben may not have been able to make progress even if he was a genetic hybrid of Mother Teresa and Bruce Schneier. What I am saying is that we too often forget that real security comes from the people, not just the code.

So, when you find yourself in a situation like Ben, please consider digging deep into your well of patience and giving a bit more when you’re tempted to give less.

Your Password Policy Tells Me That You Don’t Understand Security

Monday, March 22nd, 2010

How many times have you tried to create a password and discovered that it was too secure for the site?

“Please enter a password consisting of letters and numbers.” What?! This is 2010; there’s no reason any software out there shouldn’t support quotes, backslashes, exclamation points, or even kanji characters in a password. A site that fails to support at least all the printable characters on a modern keyboard is saying something very, very scary about their backend software: “We don’t know how it will react.” Think about that. The developers are so unsure about the quality of the implementation of this standard, well-trodden area of code that they have to have their customers help to make sure it doesn’t break.

Do you really want to do business with a company that isn’t sure it can handle textual passwords? This problem isn’t limited to podunk websites; bank and brokerage sites are implicated particularly frequently in this.

So, next time this happens to you, don’t just click away or grumble and pick a weaker password; send a message to customer service, tell them it’s a problem, and send this article.

If you are a website affected by this:

First of all, thanks for reading. Whether you’re with PR or Engineering, the fact that you’re actually looking into a complaint puts you head and shoulders above a lot of sites.

Here is the question you should ask internally: “Is this a policy decision or a technical one?

If it’s a policy decision, you might be okay. Some organizations decide that special characters are too much for their users to handle: perhaps they result in too much customer support work doing password resets. If that’s the case, please take the time to look at the numbers, gauge your current customer base, and revisit that decision. The typical computer user has matured a lot in the last 5 years and with this change you can easily upgrade your users’ security (and confidence).

If it’s a technical decision, it’s time to start worrying. The user registration and login process is the most clear-cut example of security related code in any system. That entire code path, from the front end GUI to the back end database should have been thoroughly checked, tested, and made absolutely bulletproof. If there’s any doubt in anyone’s mind that the code on this path doesn’t do exactly what it should, that calls the entire rest of the development process into question. The key issue here is input validation, sanitization, and proper practice in isolating code (how your software makes decisions) from data (what it bases those decisions on). If this is done correctly, then there is no technical reason that special characters (indeed, any characters!) cannot be used in passwords. If it’s not done correctly, then the software is vulnerable to a whole class of potent attacks. Specifics of those attacks have been well covered elsewhere but the takeaway needs to be that this single detail may indicate much deeper problems. Looking hard at this issue now may save a great deal of pain and expense later.

Be aware that it might actually be a technical problem even if it’s also codified in policy. This is the most insidious case. What started as a technical limitation “way back when” got papered over with user documentation, and a well-intentioned inquiry gets nowhere. Make sure you push for an answer. If you’re a non-technical person trying to get a clear answer, ask what happens if international users try to use non-English characters in their username or password (Internationalization and Unicode support raise some of the same issues of character sets and separation of code and data). If it wouldn’t work, you’ve got an opportunity to kill a security bird and an internationalization bird with one stone by addressing this issue

Getting Help:

The Open Web Application Security (OWASP) project is a phenomenal resource for anyone wanting to get serious about security. Check out the OWASP Top 10 for a quick view of the kinds of vulnerabilities that exist in software, and what practices fix them (a short overview video; poor sound, good explanation).

Wall Of Shame:

(This is a starter collection based on polling a few friends, I’m sure there are hundreds more. Leave a comment if you have others, or if any of these have fixed it)

Capital One
Smith Barney
Wells Fargo (toLowerCase()?) (Only special chars are @.-_)
GoGo Inflight Wifi