<story><title>Let&apos;s Encrypt is down</title><url>https://letsencrypt.status.io/?170519</url></story><parent_chain></parent_chain><comment><author>jaas</author><text>Josh from Let&amp;#x27;s Encrypt here. First, my apologies for the trouble this has caused.&lt;p&gt;I want to offer people here an early root cause analysis. I say early because we have not entirely completed our investigation or a post-mortem.&lt;p&gt;OCSP requests that use the GET method use standard base64 encoding, which can contain two slashes one after another. While debugging why a small number of OCSP requests consistently failed our engineers observed a rather odd, but standard, web server behavior. When a server receives a request with multiple slashes one after another they will collapse them into a single slash. This caused our OCSP responder to consider requests that had this unusual encoding quirk invalid and would respond to with a &amp;#x27;400 Bad Request&amp;#x27; response. The fix seemed quite simple: disable the slash collapsing behavior.&lt;p&gt;Unfortunately, stopping this behavior surfaced a more serious issue. The AIA extension that we include in certificates we issue contains a URI for our OCSP server. This URI contains a trailing slash. According to RFC 6960 Appendix 1 an OCSP request using the GET method is constructed as follows &amp;#x27;GET {url}&amp;#x2F;{url-encoding of base-64 encoding of the DER encoding of the OCSPRequest}&amp;#x27; where the url &amp;#x27;may be derived from the value of the authority information access extension in the certificate being checked for revocation&amp;#x27;. A number of user agents take this quite literally and will construct the URL without inspecting the contents of the AIA extension meaning that they ended up with a double slash between the host name and the base64 encoded OCSP request. Before we disabled slash collapsing this was fine as the web server was silently fixing this problem. Once we stopped collapsing slashes we started seeing problems.&lt;p&gt;From our OCSP server&amp;#x27;s perspective a majority of the OCSP requests we were receiving were prepended with a slash and we were unable to decode them so we&amp;#x27;d respond with a &amp;#x27;400 Bad Request&amp;#x27; response and move on. This coincided with a large number of previously cached responses on our CDN expiring, causing us to start getting hit with a large number of requests. Because we were responding with &amp;#x27;400 Bad Request&amp;#x27; responses we were setting explicit no-cache headers which meant we had a near 0% cache (CDN) offload rate and were hit with the full brunt of our OCSP request load at our origin servers. This caused our whole infrastructure to get bogged down.</text></comment>
<story><title>Let&apos;s Encrypt is down</title><url>https://letsencrypt.status.io/?170519</url></story><parent_chain></parent_chain><comment><author>hannob</author><text>So in case this helps anyone, I had people complaining about strange OCSP errors all over the morning coming from my server (using apache httpd).&lt;p&gt;It turns out apache does practically everything to behave as dumb as possible in case of OCSP downtimes.&lt;p&gt;If the OCSP sends an error it will send the error as a stapled OCSP reply (instead of using an old, still valid OCSP reply). You can&amp;#x27;t make it behave sane here, but you can at least tell it to not return the error with SSLStaplingReturnResponderErrors set to off.&lt;p&gt;However if the OCSP isn&amp;#x27;t available at all apache will fake its own OCSP error (sic!) and send it. This is controlled by the option SSLStaplingFakeTryLater, which defaults to on. So if your firefox users get strange OCSP errors, it&amp;#x27;s most likely this. The doc for SSLStaplingFakeTryLater claims that this option is only effective if SSLStaplingFakeTryLater is set to on, however that&amp;#x27;s wrong.&lt;p&gt;tl;dr set both of these options to &amp;quot;off&amp;quot;, then at least apache won&amp;#x27;t staple any garbage in your TLS connection, firefox will try to reach the ocsp on its own and fail and still accept the connection. Yes, that&amp;#x27;s all pretty fucked up.</text></comment>