Thursday, 13 November 2008

Fixing the IE7/MOSS/HTTP 413/Client SSL/View State White Screen of Death

Content on the Tourism WA partner sites like Australia's Golden Outback needs to be accessible by content editors over the internet. To facilitate this, we use client certificates over SSL to authenticate users. Each certificate is additionally mapped to an Active Directory account to support granular control over specific assets and provide standard "auditing" (in the MOSS version history sense) of user activity. Users additionally benefit from a single sign-on experience since IE hands off the client certificate in a relatively seamless manner.
In the image below the first box describes how client certifcates are created and provided to the content editors; the second box describes how IE and IIS authenticate the user to MOSS via AD.

This approach looks pretty nice on paper apart from the manual task of creating the client certificate and the potential distribution security. Our external content editor user base is fairly small and well-known, however.

Following the launch of the first partner site using this security model, we came across a painful situation I'll describe here but hope you never encounter in the field. If you have come across this problem, hopefully this post will steer you in the right direction. 

So the problem: 

When attempting to save, check in, or publish a page, IE content editors were presented with a white screen (a page with nothing in it, more specifically) in a relatively short timeframe. The problem appeared to be sporadic and definitely affect some pages but not others.

Inspecting the content editor's machine revealed no hardware or software issues and nothing was logged to the client's event logs. The HTML source was nothing special with the IE Dev Toolbar telling the same story:

<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.0 Transitional//EN">
<HTML><HEAD>
<META http-equiv=Content-Type content="text/html; charset=windows-1252"></HEAD>
<BODY></BODY></HTML>

Although some basic IE security and JS settings were tweaked, friendly error messages were already turned off. As our initial user base was regionally-based and we mistakenly believed we couldn't reproduce the problem on the local network, we initially ascribed the problem to a lousy, congested regional network connection. I say we mistakenly believed we couldn't reproduce the problem because we generally use the non-SSL-secured internal site at head office due to an unrelated issue. 

This wasn't a lot to go on and we struggled to diagnose the problem while waiting for the installation of a WAN accellerator upgrade to our own regional office to prove or disprove our theory about the network infrastructure. Meanwhile, our third partner site came online and the content editors logged the same issue; this time, those content editors were sitting in Perth on metropolitain ADSL 2+ wires.

With the installation of a broadband connection at home I decided to see what I could find out outside the corporate environment. I logged into to one of the secure sites and got lucky--I was finally able to reproduce the problem on one of every five or so attempts to save a rather large page. 

With that achievement under my hat I fired up my trusty friend Fiddler 2 and reloaded the site. As I mentioned above, the partner sites are delivered to content editors over an SSL-secured (443) connection; in other words, that means inspecting the HTML request/response data outside the browser would normally be a bit tricky. Luckily Fiddler 2 makes this easy--you simply need to check the Decrypt HTTPS traffic checkbox in the Fiddler settings (under Proxy Settings) and Fiddler will present you with the goods. 

In my case that was step one but step 2 meant I'd also have to hand over a valid client certificate since Fiddler essentially does the man in the middle thing to pull off the SSL decryption bit. Fiddler's documentation is pretty good on this subject so after exporting my existing certificate with a .cer extension I dropped it into My Documents\Fiddler2\ClientCertificate.cer (named exactly as shown). At long last I could not only reproduce the problem but I could even see under the covers of the HTTP protocol!

And it was that simple: hitting the same problem page a few times and Fiddler told me it was all about a lousy HTTP 413 error--all this for a bloody error code IE should have been showing me in the first place. 

Here's the request:

POST /en/ThisPageIsTroubba.aspx HTTP/1.1
Accept: image/gif, image/x-xbitmap, image/jpeg, image/pjpeg, application/vnd.ms-excel, application/vnd.ms-powerpoint, application/msword, application/xaml+xml, application/vnd.ms-xpsdocument, application/x-ms-xbap, application/x-ms-application, application/x-shockwave-flash, application/x-silverlight, application/x-silverlight-2-b2, */*
Referer: https://edit.problemsite.com/en/ThisPageIsTroubba.aspx
Accept-Language: en-au,en-GB;q=0.5
Content-Type: application/x-www-form-urlencoded
UA-CPU: x86
Accept-Encoding: gzip, deflate
User-Agent: Mozilla/4.0 (compatible; MSIE 7.0; Windows NT 5.1; .NET CLR 1.1.4322; .NET CLR 2.0.50727; InfoPath.2; .NET CLR 3.0.04506.30; .NET CLR 3.0.04506.648; .NET CLR 3.5.21022)
Host: edit.problemsite.com
Content-Length: 71069
Connection: Keep-Alive
Cache-Control: no-cache
Cookie: WSS_KeepSessionAuthenticated=443; MSOWebPartPage_AnonymousAccessCookie=443; s_cc=true; s_sq=%5B%5BB%5D%5D
 
HTTP/1.1 413 Request Entity Too Large
Content-Length: 0
Server: Microsoft-IIS/6.0
X-Powered-By: ASP.NET
Date: Sun, 02 Nov 2008 10:47:22 GMT
Connection: close

"What is an HTTP 413 error," I asked, just like you may be doing now? It means the server rejected the request because it considered it too big. Weird, considering the request was only just over 71KB but at least we could rule out the network--a good thing because a lot of our future content editors are very much regional.

The w3c tells us the following about HTTP 413:

"The server is refusing to process a request because the request entity is larger than the server is willing or able to process. The server MAY close the connection to prevent the client from continuing the request.
 
If the condition is temporary, the server SHOULD include a Retry- After header field to indicate that it is temporary and after what time the client MAY try again."

I mentioned page weight previously and the title of this posting also ties in viewstate. Why? I've already suggested the white screen problem seemed to affect some pages but not others. While investigating this problem, I noted the viewstate on most of the anonymous partner site pages is very low at a <2KB.

Still with me? 

We're currently running on Windows 2003 and IIS 6.0. When IIS has to deal with client certificates, the server uses the UploadReadAheadSize metabase property during the SSL preload process to determine the maximum buffer size for the incoming request. The default buffer size is 48KB to prevent denial of service attacks and for that reason Microsoft doesn't recommend setting it too high. In our case, the 71KB request was considered too large and IIS wouldn't let it through. Why the save/check-in/publish worked sporadically I can't say, but increasing the value assigned to this property instantly fixed the problem. We increased this setting on every relevant IIS web site using this command:

cscript adsutil.vbs set w3svc/1/uploadreadaheadsize 204800

Note the setting represents bytes, not kilobytes, and the TechNet article on changing this is incorrectly specifies a value of 200.

In theory the HTTP 413 should have come up in the server logs but if it did, we never caught it. 

For more information about the metabase changes, the following blog posts may be of use:

Custom-Built Microsoft Office SharePoint Server 2007 Branded Sites and Webpart Development - info@mediawole.com

5 comments:

  1. Great Post! boy, Cant wait till I get this error :S

    ReplyDelete
  2. We seems to get this error all the time. We change the meta values. it works then suddenly for no apparent reason it stops working until I run the batch file again. Seriously considering reserting to a linux server for all my uploading needs. Heck I may just change everything to Linux and PHP

    ReplyDelete
  3. Does this thing work if server is accesing some other server on client's request? please provided more information over it. Provide links to related topics if possible.

    ReplyDelete
  4. @r4 ds:

    I assume you're talking about the client SSL certificate setup for authorising users and the impersonation of those users... I can't answer your first question with any certainty, unfortunately, since we don't do that ourselves.

    In terms of links I've only seen a very small number of articles/posts on the subject and we actually inherited this configuration from an MCMS 2002 setup. I haven't read these in detail but try: http://www.windowsecurity.com/articles/Client-Certificate-Authentication-IIS6.html

    and

    http://www.iis.net/ConfigReference/system.webServer/security/authentication/clientCertificateMappingAuthentication

    ReplyDelete
  5. so we seemed to be having this same issue - windows server 2008 and iis7. strangely enough we did NOT see this problem on our 2003 IIS6 environment. I just applied this change and hope we no longer see this blank page issue.

    I did see the 403 in our iis log files which is actually what led me to your site!

    See below:
    2010-05-07 20:03:40 111.111.111.11 POST /url/Lists/ourlist/CustomEditForm.aspx ID=8953&List=5da71b51-2111-4d88-a462-e184a7c13a9e&Source=https%3a%2f%2fwebapp.oururl.com%2fots%2flists%2fourlist%2fpage.aspx 443 - 11.11.11.11 Mozilla/4.0+(compatible;+MSIE+7.0;+Windows+NT+5.1;+InfoPath.2;+.NET+CLR+2.0.50727;+.NET+CLR+1.1.4322;+.NET+CLR+3.0.04506.30;+.NET+CLR+3.0.4506.2152;+.NET+CLR+3.5.30729) 413 0 0 405

    Note the 413 at the end...that is what lead me to your site.

    Thanks so much for the help!

    ReplyDelete

Spam comments will be deleted

Note: only a member of this blog may post a comment.