Server 2003 Upgrade
|
Server 2003 Upgrade |
|
|
||
|
Home « Computers « DevBlog | |
Back to: Development Blog Contents
The first week of October 2003 -- Many strange things happen after upgrading the Nancy Street server to Windows 2003. I have documented the 10 days of suffering in the hope that I can help others who follow me. Some problems are solved, some are bypassed and others are self-inflicted accidentally, but most are probably caused by the Microsoft security "lockdown" in 2003. I present a summary first, then if you find anything interesting you can read further.
Summary
2003 Server Enterprise Upgrade
 |
All ASP and ASP.NET applications become inaccessible. ASP responds with 404 Not Found errors. ASP.NET responds with Server Application Unavailable and code 500-2. Web and news searches produce hundreds and hundreds of hits from people describing the same problems, especially the ASP.NET one which has people tearing their hair out, just like me. Despite hours of searching and experimenting with every conceivable aspect of security and configuration no answer is found. |
|
The CVS for NT Server is inaccessible from outside the LAN. I am quite sure all of the ports and services are working correctly and the router is forwarding correctly. The only response from the CVS client is "the connection is refused". |
|
A .NET application called DRS (Daily Runsheet System) being developed at Nancy Street produces bizarre errors. Both the server and client components (based upon .NET Remoting) are inaccessible to-and-from the server. Exceptions are thrown in the Remoting code out of control of the user code. Despite hours of debugging and testing, the problem remains unchanged. |
|
Debugging the DRS Remoting application is complicated by the fact that a .NET 1.1 security lockdown prevents clients from getting detailed exception error messages from the server. The default of customErrors=RemoteOnly causes a bland generic exception error message to be returned to clients. Hours of experiments try to find a way of setting customErrors=Off, both in code and in config files. |
By late Saturday evening (04-Oct-2003) I was so fed-up and frustrated that I was about to trash 2003 Server and revert to 2000 Advanced Server. Luckily I decided to run some experiments on the test PC sitting next to the server which was running 2003 Server Standard Edition. I found that CVS Server and the DRS application worked perfectly (it was too difficult to test CVS Server). Bingo! There is something fundamentally different in the behaviour of the applications on 2003 Server Enterprise and Standard Editions. I search the detailed feature documentation for the products but I can't find anything obvious that would cause such problems.
Reinstall of 2003 Server Standard No.1
After 2 hours of tedious work I format the server and install 2003 Standard. I find immediately that CVS and DRS are working as expected, as they did on the test PC. ASP and ASP.NET applications fail with the original errors and more hours of work fail to coax them into life. I go to bed frustrated at 2am.
 |
The CVS Server works perfectly on 2003 Server Standard. |
|
The DRS Remoting application works perfectly on 2003 Server Standard. |
|
ASP and ASP.NET application continue to fail with the original errors. |
Reinstall of 2003 Server Standard No.2
The next morning I continue to search the web and newsgroups for answers to Server Application Unavailable but find none. The only clue in one post is that the problem seems vaguely related to installing or activating IIS after the operating system is installed. I did accidentally forget to install IIS during the first reinstall, so perhaps this was a fatal mistake. My only option is to start again. I carefully install Windows 2003 Server Standard Edition again from scratch. This time I select and activate ASP.NET in the early stages of the setup.
|
ASP.NET applications start working after IIS is installed with a fresh Windows 2003 Server Standard. It remains a mystery why this worked. |
|
Parts of the now working ASP and ASP.NET applications fail to access databases and other support files in the file system even though permission is granted. |
Another hour is wasted searching for solutions to the file access problems. I eventually get a clue from the IIS logs where I see the message Disallowed_Parent_Path. I stumble across an option in the IIS Home Directory -> Configuration -> Options dialog to Enable Parent Paths. The help warns that this is not a recommended option, but I have to leave it on. Did this used to be on by default in previous editions?
|
Turning on Enable Parent Paths lets my ASP and ASP.NET apps access files outside of the web directory. But does this leave me vulnerable to Nimda style overflow attacks? |
Share Security
I share the folders containing utilities and music for use on the home LAN. All access to the shares is read-only despite carefully setting the permissions. After another hour or frustration I stumble across the fact that the default share permission is now read-only. The help tells me something I didn't know: The share and file system permissions are independent of each other and the lowest permission of their union is the resulting permission. The default share used to be read-write.
|
Set default share permissions to higher than the default of read-only or you'll never get higher access in the file system permissions. |
16-Bit Applications
After spending a week on the previous problems I felt confident that I was over the worst and I could now make the 2003 Server (Standard Edition) available publicly as the new server. I make a few tweaks to customise the server, including putting the old Windows 3.1 vintage clock.exe program in the start menu. This error pops-up:
Config.nt The system file is not suitable for running
MS-DOS and Microsoft Windows Applications.
What the hell is this?!?! I've never seen such an error before in my life. I do indeed find that all 16-bit apps will not run. I run web and news searches and see hundreds of other reports of this problem, some going back to NT4. Clues point towards corrupt or missing autoexec.nt and config.nt files in the system32 directory. I have two 2003 Standard Edition servers sitting next to each other so I compare the system files and find they are quite different. The test PC has dozens of hidden files in the C:\ root directory and server has only a handful. The system32 directory on the test PC has more files than the server. Both of these PCs were installed with the same version of Windows from the same DVD. I know I used slightly differently setup options for each system, but how is it possible to finish-up with such wildly different results.
I run sfc /scannow to look for missing files, but nothing changes. I forcibly load the latest app compatibility updates, but nothing happens. In desperation I manually copy autoexec.nt and config.nt from the system32 directory on the test PC over to the server. This fixes the problem and now 16-bit apps start to work. Two nearly identical PCs with the same operating system have different system directory contents and I get bizarre errors as a result. Now I'm left wondering what the hell else might be wrong with my live 2003 server.
|
autoexec.nt and config.nt are copied from the system32 folder on a test PC to the live server to allow 16-bit apps to run. |
Host Headers
After a few days I noticed that the only hits on my web site were from a handful of search engines, none from the public. I set my proxy to an external cache so I could see myself from the outside world and found that when I accessed my dyndns.org hosted domains I received Invalid Host Header. Suddenly I realised I had put the two host header entries into IIS without the www prefix.
|
Set your IIS host headers correctly, which in my case was a www prefix on the domain names. |
ISAPI Filter Permissions
My ISAPI filter written in C++ that writes custom logging information to files ceased to work. I received popup errors from IIS 6 that an fprintf call had failed (and then the filter is deactivated). Debugging ISAPI filters is notoriously difficult, so I had to write debugging lines to a file in my c:\temp directory. After some delicate work I found that the fopen calls were failing with code 13 (EACCESS permission denied). This was clearly caused by yet another security lockdown default change. I was unable to quickly determine which security account IIS 6 runs under (not IUSR or IWAM) so I had to set modify permission for everyone on the logging output folder to temporarily get the ISAPI filter working. It was purely a fluke that I had permission to write logging information to the c:\temp directory, otherwise I would have had absolutely no clues as to what was going wrong.
|
IIS 6 ISAPI filters have a default owner of NETWORK SERVICE. Permissions must be granted accordingly for access to the file system. Ownership can be customised in IIS 6 Application Pool properties. |
Setting customErrors=Off
In the Remoting based client-server app I was developing, no detailed exception messages were being returned to the clients. Only a generic "server error" message was returned no matter what went wrong and it told me to set customErrors off to see details. Many hours of searching and experiments fail to set customErrors=Off. Creating a config file didn't work (initially) and no way could be found code to set the property. A few days later I visit my colleague at his home and we run more failed experiments. I finally decide to try the config file again, and suddenly it works! I take the project changes home and now they work there also. It is a mystery why the config file did not initially work. A file named 'server.config' with these contents was placed in the server runtime directory:
| <configuration> <system.runtime.remoting> <customErrors mode="off" /> </system.runtime.remoting> </configuration> |
The parent server application that hosts the remoting object (a singleton in our case) needs to make this call before the channels are configured in code:
| RemotingConfiguration.Configure(filename); |
It remains a mystery how to set this property in code, as our Remoting application performs all Remoting setup and calls in code. A config file in the app setup is inconvenient.
|
A config file containing customErrors="Off" is required to allow detailed exception error messages to be return to Remoting clients. |
Afterthought -- Why does maintenance of Windows have to be so difficult? As I keep telling my colleagues: In the late 80s it took less time to install and configure the operating system for a dozen huge payroll company mainframes with hundreds of users than it does to upgrade a single Windows server at home. What has gone wrong?
Back to: Development Blog Contents