Hi Team,
We have an issue with Sharepoint integrated with TFS. We run a large on-premise instance of TFS, 23 collection dbs totalling > 4TB in data, with > 5000 users.
We have 2 TFS App Tiers (primary/secondary) with Sharepoint installed on both. Sharepoint URL is behind a load balancer and uses HTTPS (as is TFS url). Sharepoint SQL databases are installed on a separate SQL instance to TFS collection databases.
Issue:
We had a port down alert on the Sharepoint SQL instance on 3rd September, which lasted 10 minutes. After which, sharepoint stopped working. After trying/checking various things we are now stuck and getting errors when trying to repair the TFS -> Sharepoint access grants via the TFS Admin console.
Our support team can access the Sharepoint Central Admin site on the primary app tier server only. However the portal sites are not available, and users receive an HTTP 500 error. Further, we get HTTP 200 errors when trying to repair the connection with TFS and the sharepoint sites.
Subset of log entries below:
TFS Admin Console, trying to reconfigure Sharepoint Extensions:
[Info @17:44:15.794] Validating auto-configure for SharePoint Web application
[Info @17:44:15.872] Validation Pending: The user is not a farm administrator. Therefore, there must be an existing access grant between Team Foundation Server and SharePoint Products.
[Warning@17:44:15.872]
Exception Message: TF400324: Team Foundation services are not available from the server.
Technical information (for administrator):
The underlying connection was closed: An unexpected error occurred on a send. (type TeamFoundationServiceUnavailableException)
Exception Stack Trace: at Microsoft.TeamFoundation.Client.Channels.TfsHttpWebRequest.ReadResponse(HttpWebResponse webResponse, WebException webException)
at Microsoft.TeamFoundation.Client.Channels.TfsHttpWebRequest.SendRequest()
at Microsoft.TeamFoundation.Client.Channels.TfsHttpRequestChannel.Request(TfsMessage message, TimeSpan timeout)
at Microsoft.TeamFoundation.Client.Channels.TfsHttpClientBase.Invoke(TfsClientOperation operation, Object[] parameters, TimeSpan timeout, Object[]& outputs)
at Microsoft.TeamFoundation.Client.SharePoint.SharePointTeamFoundationIntegrationService.CheckIsFarmAdministrator()
at Microsoft.TeamFoundation.Admin.SharePointConfigurationHelper.ValidateAutoconfigureWebApplication(Uri adminUrl, Uri siteUrl, ITFLogger logger)
TF255329: The following site could not be accessed: <site>. The server that you specified did not return the expected response.
Either you have not installed the Team Foundation Server Extensions for SharePoint Products on this server, or a firewall is blocking access to the specified site or the SharePoint Central Administration site.
TF255439: You do not have the permissions required to integrate with the following SharePoint Web application: <servername>. The
validation check failed for the following user account: domain\service account. This account must be a direct member of the Farm Administrators group on the web application.
Sharepoint ULS Log:
09/10/2019 15:37:39.67 w3wp.exe (0x2844) 0x1854 SharePoint Foundation Claims Authentication fsq7 High SPSecurityContext: Request for security token failed with exception: System.IO.FileLoadException: Loading this assembly would produce a different grant set from other instances. (Exception from HRESULT:
0x80131401) Server stack trace: at System.Net.HttpWebRequest.GetRequestStream() at System.ServiceModel.Channels.HttpOutput.WebRequestHttpOutput.GetOutputStream() at System.ServiceModel.Channels.HttpOutput.Send(TimeSpan timeout) at System.ServiceModel.Channels.HttpChannelFactory`1.HttpRequestChannel.HttpChannelRequest.SendRequest(Message message, TimeSpan timeout) at System.ServiceModel.Channels.RequestChannel.Request(Message message, TimeSpan timeout) at System.ServiceModel.Dispatcher.RequestChannelBinder.Request(Message message, TimeSpan timeout) at System.S... 89dc029f-17f6-507b-b038-f8471ce19ad1
Event Log:
Event ID 8306
An exception occurred when trying to issue a security token: Loading this assembly would produce a different grant set from other instances. (Exception from HRESULT:
0x80131401).
What we have tried:
- Confirmed support user is a member of the Farm Administrators group.
- Confirmed Sharepoint service account is a member of the Farm Administrators group.
- Checked connectivity between Sharepoint SQL Server and TFS App Tiers (where Sharepoint is installed) for service account and support user account.
- Checked SQL permissions on Sharepoint for service account and support user account.
- Rebooted all servers.
- Various IIS resets.
- Checked SecurityTokenServiceApplicationPool is running.
- Confirmed 'Enable Kernel-mode authentication' is checked against 'Sharepoint Web Services' site.
- Database Status on Sharepoint Central Admin looks fine.
- Our UAT test environment has the same stack, and does not have this issue.
Our TFS Stack:
- SharePoint Foundation 2013 with SP1
- SQL Server 2012 Enterprise SP4 (sharepoint)
- SQL Server 2016 Enterprise SP2 (TFS) in Always On configuration
Please assist as many users are affected.
Many thanks,
Scott.