I am chasing down some very strange issues in a customers SharePoint 2013 SP1 farm
Behavior
At some point there was an issue when the customer was creating new web applications where the web applications listed in SharePoint were not the same as the sites that were deployed in IIS. In Central Admin there were 22 web applications listed, but in IIS
in one of the WFE servers there were only 18 IIS sites.
The customer originally had 4 WFE servers which showed a different number of sites on each WFE server. We went ahead and removed all but one WFE from the farm by removing via the config wizard. The other 3 WFE servers removed from the farm with no issue.
We then proceeded to delete the web applications via central admin that did not match with what was on the last remaining WFE server. There were 4 that had to be deleted. When we checked the WFE the 4 app pools were still present on the server so we manually
deleted these.
All seemed to be fine as we then able to deploy the customers web app scoped solutions with no further errors about missing or corrupted web.config files.
ISSUE
Now we need to add the additional wfe servers back to the farm and are unable to do so.
We try to join the server via Connect-spconfigurationdatabase command, it runs for 5 mins or so and then fails with the error
Connect-SPConfigurationDatabase : Object reference not set to an instance of
an object.
At line:1 char:1
+ Connect-SPConfigurationDatabase -DatabaseServer "WDVSQLSP1301" -DatabaseName
"Sh ...
+ ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
While the server is trying to join I am watching IIS and I see all the current application pools created on the server and one of the IIS sites. Within a minute or so the number of application pools begins to reduce, the single site is deleted and the command
fails with the error above.
What is strange is that there are 4 application pools left over on the server and two of them are application pools that were associated with the web applications that had been deleted from central admin.
I am trying to figure out where these old application pools could be being pulled from in the farm as they are not listed on any WFE servers or other app servers in the farm.
Any suggestions on things to look at is greatly appreciated.