I just wasted 2 days (one of which was on my weekend), and a lot of sleep, trying to work out why I couldn’t upgrade the Windows 2000 server which looks after my domain, DHCP, RIS, SUS and a whole load of other bits at home.
Every time I tried to run Windows Server 2003 setup it seemed to hang – and everything else was pretty slow too. I had to launch control panel applets using their .cpl filenames (e.g. appwiz.cpl for the Add or Remove Programs applet) and services would not stop cleanly.
I decided that my system was badly broken and quickly built a virtual machine on another piece of hardware, promoting that to a domain controller to provide a live backup of Active Directory. As in-place upgrades weren’t working, I resigned myself to the fact that I was going to have to migrate everything to the virtual server, then rebuild the original box but I wanted to cleanly remove the original domain controller from the directory.
Every time I ran the Active Directory installation wizard (dcpromo.exe) it failed – usually with the following error.
Active Directory Installation Failed
The operation failed because:
Failed to prepare for or remove the sysvol replication “The file replication service cannot be stopped.”
(Even though logged events with IDs 13502 and 13503 suggested that the FRS had indeed stopped).
Microsoft knowledge base article 332199 led me to try the dcpromo /forceremoval command but that failed in exactly the same way. I ran dcdiag /s:localhost on each server to look for any issues, checked that each server could ping the other one, that net view \\servername returned a list of shares, and all required DNS entries were present. I checked the DNS settings (to make sure that each server was using itself as the primary DNS server and the other domain controller as a secondary) and restarted just to be sure but all to no avail.
To cut a long story short, I found the answer purely by fluke. I couldn’t get the DHCP server service to stop cleanly (to let me migrate the database to my virtual machine) so I did a Google search for “windows services hang on stop”. This turned up a TechRepublic thread titled APC Java issues cause services to hang. I realised that I do have an APC UPS attached to the server, and that I was using a version of PowerChute Business Edition (PBE) that had been sitting there happily for a couple of years (v6.2.2) – I hadn’t upgraded to 7.x as recommended by APC knowledge base article 7202 because APC had never e-mailed me to notify me of a problem and services that aren’t broken (and that don’t have an inbuilt patching mechanism) generally get left well alone on my systems!
Lo and behold, the APC services had hung on startup and there were various events logged with ID 7022 (the APC PBE Agent service hung on starting). I disabled both the APC PBE client and server services, using the registry (as the services console was inoperable) to locate HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services\servicename\ and set Start to 0x00000004 for disabled (0x00000002 is automatic and 0x00000003 is manual), restarted the server and had the fastest boot sequence in days! My Windows installation was responsive again and I was able to remove the offending applications in a few short clicks.
My problems were nothing to do with Active Directory, DNS or even Windows – they all boiled down to an expired Sun Java Runtime Environment (JRE) certificate and sloppy coding from APC which meant that if their services hung, then so did all subsequent ones. I’ve never been a fan of Java applications on Windows – generally they are slow and have a poor user interface – and this experience has done nothing to change my mind.
Once the APC PBE agent, client and server had been removed, I was able to successfully (and cleanly) demote the original domain controller (avoiding having to follow the steps in Microsoft knowledge base article 216498 to remove data left in the directory after an unsuccessful demotion) but having migrated all the services to my virtual machine, I decided to go ahead and perform a clean installation of Windows on the original hardware anyway. I’m currently mid-way through patching the rebuilt server but I’m so glad that P McGrath from Rocky Mount, VA posted his experience on TechRepublic and Google did it’s thing.
Remind me again – how did we ever manage to find things out before we had the web?
Hey Mark, Thanks for the blog/post. I WAS aware of the APC Java issue, but I thought it only caused a very very slow start up. I was working on a Windows 2000 domain controller and I was not able to demote it. A simple search with google using “failed to prepare for or remove the sysvol replication” let me right to your blog. I too have had the fastest boot on this server in quite some time, and I was able to demote it without a problem.
Wow. I have to agree with the other post. I have been under the gun trying to repair some shifty behavior with a domain controller demotion and your post seems to have hit the nail on the head. Thanks a million.
Mark
Thanks for the blog. It seems that we had a slow starting server caused by the APC software. I knew it was APC and your blog helped me identify the solution to being able to stop and start services again.
Cheers
I just went through a week of heck with a 3rd party mail scanning app with it’s services stuck at “stopping”. Had to reboot server to get it working. Went nuts with their tech support until I also did a Google search on Windows 2003 service hangs and saw a post on APN Powerchute 6 problems. Yep, I had it too. Did the registry key hack to disable both the peserver and peageant serices and rebooted. Voila!
I experienced the SAME EXACT PROBLEM recently and it has left me extremely frustated. This has caused such a slowdown on some of my servers that I cannot connect via remote desktop to any of our Domain Controllers in our remote offices. This means I will have to drive all over the great state of Indiana and manually remove the offending software from at least 4 different locations. Thanks a lot APC and Sun!
I also had this exact same problem although making the registry change didn’t resolve my problem. I had to actually remove all the APC stuff from the registry. Once I did that my server started acting normally again. Thanks so much – I thought I was going to have to reinstall the OS on this server, but now I might not have to that.
All I can say is “Ditto!” Thanks for the post, Mark! You saved me from a very long week! Beer’s on me if you’re ever in San Fran!
-neal
neal@neal.org
Thank you so much for posting this! I can’t believe what a difference this has made! I was finally able to remove Active Directory and demote this server to a member server. Thanks again! Kudos!
Thanks a bunch Mark – just gone through the same no-can-do dcpromo /forceremoval – your blog was right on the money.
Thanks a million Mark – your Blog has solved my issue – I would never have suspected the APC software because it was on all our servers before I even started working here, and given that it is software that would nearly always be installed on mission critical servers one might be forgiven for assuming that it would have been stable and well tested! Anyhow when I tried to remove AD from a Win2k server and got the same error, Google lead me to your Blog, i removed the APC software and finally I was able to demote the old server. Thanks Mark! :-)
Just want to ditto what everyone else had said… I was trying to dcpromo down a server from a remote office and it kept hanging on shutting down FRS… Never noticed the issues with the APC software before (since the server used to be in a remote office, I never knew how long it took to boot)…
Sigh…. Thanks again.
Mark,
I’m glad my post on TechRepublic helped. The whole point of posting was to try to get the word out on this issue. Thanks for the reference. WhoooHooo! I’ve been blogged!
Pat McGrath
Thanks Mark! its nearly 3am and the ‘failed to stop frs’ had nearly beaten us until i found your blog! cheers v.much ;)
Thanks! You’ve saved me a very frustrating afternoon, my last ditch google led me here!
Thank you sooooooooooooooooooooo much, the amount of time I have wasted chasing my tail on this.
After spending about 12 hours trying to demote a Win2k DC, I read your blog and sure enough those two APC services were haning up the file replication service every time. Disabled them and it worked right away. Thanks for your blog post.
APC now offers an ‘opt in’ only electronic magazine to notify users of Software and firmware updates. You can find this on the right-hand side of this page http://bit.ly/7XPOdN
Alternatively the product RSS feed provides much more comprohensive updates on all new products releases http://bit.ly/6e42bF
Hi Jane – really glad to hear about APC’s improved customer communications. I certainly wasn’t aware of anything like that when I wrote this post back in 2005!