We ended up opening a case with Microsoft SharePoint support to troubleshoot a strange SP2013 NavNode problem. I'll share the problem and the solution here because it's good information that probably quite a few others might be able to use. And then I will
ask my question about which SQL maintenance jobs should be run for SP2013 databases.
///////////VISIBLE SYMPTOMS///////////
For some of our sharepoint sites we noticed performance problems when adding, removing, or rearranging navigation nodes. Sometimes the change we would make would timeout but ultimately succeed. Sometimes it would timeout and give a correlation ID and
not succeed. Sometimes we would end up with duplicate navnodes. It was all rather bizarre. And if we used powershell/stsadm to back the site up and restore it to another database, the problem would go away. So we knew it was some kind of corruption in
the database.
///////////ULS ERRORS///////////
• 06/01/2015 12:07:21.79 w3wp.exe (0x1E20) 0x3348 SharePoint Foundation Database
ad194 High ExecuteQuery failed with original error 0x80131530 e9230c9d-5fc0-40e3-1719-199edcb83b88
• 06/01/2015 12:07:21.79 w3wp.exe (0x1E20) 0x3348 SharePoint Foundation Database
8z23 Unexpected Unexpected query execution failure in navigation query, HResult -2146233040. Query text (if available): "BEGIN TRAN DECLARE @abort int SET @abort = 0 DECLARE @EidBase int,@EidHome int SET
@EidBase = 0 SET @EidHome = NULL IF @abort = 0 BEGIN EXEC @abort = proc_NavStructAllocateEidBlockWebId @wssp0, @wssp1, @wssp2, @wssp3, @EidBase OUTPUT SELECT @wssp4 = @EidBase, @wssp5 = @abort END IF @abort = 0 BEGIN EXEC @abort = proc_NavStructMoveNode '4583A9E7-E1B6-44BA-BBFF-9DD5EE4A50B9','0036EE07-798A-4F8B-8241-1F33F506CCA0',2052,1025,-2,N'20150601
17:00:38',@EidBase,@EidHome, @wssp6 OUTPUT SELECT @wssp7 = @abort END IF @abort = 0 BEGIN EXEC proc_NavStructLogChangesAndUpdateSiteChangedTime @wssp8, @wssp9, NULL END IF @abort <> 0 BEGIN ROLLBACK TRAN END ELSE BEGIN COMMIT TRAN END IF @abort = 0
BEGIN EXEC proc_UpdateDiskUsed '4583A9E7-E1B6-44BA-BBFF-9DD5EE4A50B9' END " e9230c9d-5fc0-40e3-1719-199edcb83b88
• 06/01/2015 12:07:21.79 w3wp.exe (0x1E20) 0x3348 SharePoint Foundation General
8kh7 High An unexpected error occurred while manipulating the navigational structure of this Web. e9230c9d-5fc0-40e3-1719-199edcb83b88
09/17/2015 15:28:19.30 w3wp.exe (0x01F0) 0x2C10 SharePoint Foundation Database fa44 High
Slow Query StackTrace-Managed: at Microsoft.SharePoint.Utilities.SqlSession.OnPostExecuteCommand(SqlCommand command, SqlQueryData monitoringData) at Microsoft.SharePoint.Utilities.SqlSession.ExecuteReader(SqlCommand
command, CommandBehavior behavior, SqlQueryData monitoringData, Boolean retryForDeadLock) at Microsoft.SharePoint.SPSqlClient.ExecuteQueryInternal(Boolean retryfordeadlock) at Microsoft.SharePoint.SPSqlClient.ExecuteQuery(Boolean
retryfordeadlock) at Microsoft.SharePoint.Library.SPRequestInternalClass.MoveNavigationNode(String bstrUrl, Int32 lNodeId, DateTime dateParented, Int32 lParentId, Int32 lPreviousSiblingId) at Microsoft.SharePoint.Library.SPRequestInternalClass.MoveNavigationNode(String
bstrUrl, Int32 lNodeId, DateTime dateParented, Int32 lParentId, Int32 lPreviousSiblingId) at Microsoft.SharePoint.Library.SPRequest.MoveNavigationNode(String bstrUrl, Int32 lNodeId, DateTime dateParented, Int32 lParentId, Int32 lPreviousSiblingId)
at
09/17/2015 15:28:19.31 w3wp.exe (0x01F0) 0x2C10 SharePoint Foundation Database 880i High
System.Data.SqlClient.SqlException (0x80131904): Transaction (Process ID 502) was deadlocked on lock resources with another process and has been chosen as the deadlock victim. Rerun the transaction. at System.Data.SqlClient.SqlConnection.OnError(SqlException
exception, Boolean breakConnection, Action`1 wrapCloseInAction) at System.Data.SqlClient.TdsParser.ThrowExceptionAndWarning(TdsParserStateObject stateObj, Boolean callerHasConnectionLock, Boolean asyncClose)
at System.Data.SqlClient.TdsParser.TryRun
09/17/2015 15:28:19.31 w3wp.exe (0x01F0) 0x2C10 SharePoint Foundation Database 880k High
at Microsoft.SharePoint.SPSqlClient.ExecuteQueryInternal(Boolean retryfordeadlock) at Microsoft.SharePoint.SPSqlClient.ExecuteQuery(Boolean retryfordeadlock) at Microsoft.SharePoint.Library.SPRequestInternalClass.MoveNavigationNode(String
bstrUrl, Int32 lNodeId, DateTime dateParented, Int32 lParentId, Int32 lPreviousSiblingId) at
09/17/2015 15:28:19.31 w3wp.exe (0x01F0) 0x2C10 SharePoint Foundation Database 880j High
SqlError: 'Transaction (Process ID 502) was deadlocked on lock resources with another process and has been chosen as the deadlock victim. Rerun the transaction.' Source: '.Net SqlClient Data Provider' Number: 1205 State: 51 Class: 13 Procedure:
'proc_NavStructDirtyPageDependents' LineNumber: 35 Server: 'CSRSQLSPAD1\SPI2013PROD' acf22e9d-5f9e-40e3-a394-cb756064ccad
09/17/2015 15:28:19.31 w3wp.exe (0x01F0) 0x2C10 SharePoint Foundation Database 5586 Critical
Unknown SQL Exception 1205 occurred. Additional error information from SQL Server is included below. Transaction (Process ID 502) was deadlocked on lock resources with another process and has been chosen as the deadlock victim. Rerun the transaction.
acf22e9d-5f9e-40e3-a394-cb756064ccad
09/17/2015 15:28:19.31 w3wp.exe (0x01F0) 0x2C10 SharePoint Foundation Database d0d6 High
System.Data.SqlClient.SqlException (0x80131904): Transaction (Process ID 502) was deadlocked on lock resources with another process and has been chosen as the deadlock victim. Rerun the transaction. at System.Data.SqlClient.SqlConnection.OnError(SqlException
exception, Boolean breakConnection, Action`1 wrapCloseInAction) at System.Data.SqlClient.TdsParser.ThrowExceptionAndWarning(
09/17/2015 15:28:19.31 w3wp.exe (0x01F0) 0x2C10 SharePoint Foundation General ai1wu Medium
System.Runtime.InteropServices.COMException: An unexpected error occurred while manipulating the navigational structure of this Web., StackTrace: at Microsoft.SharePoint.Navigation.SPNavigationNode.MoveInternal(SPNavigationNodeCollection
collection, Int32 iPreviousNodeId) at Microsoft.SharePoint.Publishing.Navigation.SPNavigationSiteMapNode.HandleJavaScriptNavigationNode
09/17/2015 15:28:19.31 w3wp.exe (0x01F0) 0x2C10 SharePoint Foundation General ajlz0
High Getting Error Message for Exception System.Web.HttpUnhandledException (0x80004005): Exception of type 'System.Web.HttpUnhandledException' was thrown. ---> Microsoft.SharePoint.SPException:
An unexpected error occurred while manipulating the navigational structure of this Web. ---> System.Runtime.InteropServices.COMException: An unexpected error occurred while manipulating the navigational structure of this Web. at
Microsoft.SharePoint.Library.SPRequestInternalClass.MoveNavigationNode
With debug diagnostics I was able to dump the w3wp.exe while the repro was going on and the only problematic thing I saw was one worker thread “waiting on data to be returned from the database server.” We could see that the “current executing command
is: proc_GetTpWebMetaDataAndListMetaData.”
///////////RESEARCH///////////
Research brought me to this sql query that detects duplicate entries in the navnodes:
http://h20564.www2.hp.com/hpsc/doc/public/display?docId=mmr_kc-0107890-5
SELECT TOP(20) Count(nav.Eid) AS 'DuplicateCount', nav.DocId, ad.Dirname, ad.Leafname FROM NavNodes AS nav with(nolock) INNER JOIN AllDocs AS ad with(nolock) ON nav.Docid = ad.Id WHERE nav.EidParent = 1025 AND DocID IS NOT NULL GROUP BY nav.DocId, ad.DirName,
ad.LeafName ORDER BY 'DuplicateCount' DESC
This was an interesting detour but it didn't really help. The advice at the following sites did NOT help, surprisingly:
http://h20564.www2.hp.com/hpsc/doc/public/display?docId=mmr_kc-0107890-5
http://sp-thornet.blogspot.ae/2013/05/sharepoint-2010-corrupt-navigation-nodes.html
http://sharepointbabe.blogspot.com/2013/12/unexpected-error-when-changing-top.html
Also my research pointed out that there was a NavNode duplication that was reintroduced as a regression by the August 2012 CU for SP2010 (note 2010, not 2013) that also had a sql deadlock condition as its root cause, if I remember correctly.
It was solved by a hotfix that was rolled up into the April 2013 CU for SP2010. I have never heard of this bug showing up in SP2013 (which we are on) but the similarities between what we’re seeing and this bug are palpable. I was wondering if it
was a bug. So we opened a case with Microsoft support.
We set ULS logging to VerboseEX level and reproduced the error. Here is what stood out to the Microsoft support engineer:
DataAnalysis:
Start of the correlation:
09/28/2015 14:33:46.72 w3wp.exe (0x18A8) 0x1E90 SharePoint Foundation Logging Correlation
Data 77a3 Verbose Starting correlation. f379329d-3406-105e-a72a-ed7b53a2606e
09/28/2015 14:33:46.72 w3wp.exe (0x18A8) 0x1E90 SharePoint Foundation Logging Correlation
Data xmnv Medium Name=Request (POST:http://websitename.qa:80/can/_layouts/15/AreaNavigationSettings.aspx)
f379329d-3406-105e-a72a-ed7b53a2606e
Sample of slow queries
09/28/2015 14:34:15.06 w3wp.exe (0x18A8) 0x1E90 SharePoint Foundation Database
fa43 High Slow Query Duration: 13167.302827594 f379329d-3406-105e-a72a-ed7b53a2606e
09/28/2015 14:36:07.11 w3wp.exe (0x18A8) 0x1E90 SharePoint Foundation Database
fa43 High Slow Query Duration: 14080.2901625765 f379329d-3406-105e-a72a-ed7b53a2606e
09/28/2015 14:38:38.24 w3wp.exe (0x18A8) 0x1E90 SharePoint Foundation Database
fa43 High Slow Query Duration: 13421.683342436 f379329d-3406-105e-a72a-ed7b53a2606e
09/28/2015 14:39:02.41 w3wp.exe (0x18A8) 0x1E90 SharePoint Foundation Database
fa43 High Slow Query Duration: 12039.466538345 f379329d-3406-105e-a72a-ed7b53a2606e
09/28/2015 14:39:54.75 w3wp.exe (0x18A8) 0x1E90 SharePoint Foundation Database
fa43 High Slow Query Duration: 13868.4964150472 f379329d-3406-105e-a72a-ed7b53a2606e
Logical reads:
09/28/2015 14:33:46.97 w3wp.exe (0x18A8) 0x1E90 SharePoint Foundation Database
f7h9 VerboseEx SQL IO Estimate: Procedure proc_FetchDocForHttpGet, Total Logical Reads 242. f379329d-3406-105e-a72a-ed7b53a2606e
09/28/2015 14:33:46.97 w3wp.exe (0x18A8) 0x1E90 SharePoint Foundation Database
f7h9 VerboseEx SQL IO Estimate: Procedure proc_FetchDocForHttpGet, Total Logical Reads 242. f379329d-3406-105e-a72a-ed7b53a2606e
Connections 32 queries
09/28/2015 14:33:46.88 w3wp.exe (0x18A8) 0x1E90 SharePoint Foundation Database
tzku Verbose ConnectionString: 'Data Source=SQLSPI2013QA;Initial Catalog=SPI_Prod_DealerServices_CanadianDealerServices_Content;Integrated
Security=True;Enlist=False;Pooling=True;Min Pool Size=0;Max Pool Size=100;Asynchronous Processing=True;Connect Timeout=15' Partition: f237ae7f-3cf0-4a5c-b2b1-0c24204fdb1e ConnectionState: Closed ConnectionTimeout: 15
f379329d-3406-105e-a72a-ed7b53a2606e
All execution times -> all SQL requests ranging from 11 – 18 seconds
09/28/2015 14:34:01.50 w3wp.exe (0x18A8) 0x1E90 SharePoint Foundation Monitoring
b4ly Verbose Leaving Monitored Scope (SPSqlClient). Execution Time=13932.4626453921 f379329d-3406-105e-a72a-ed7b53a2606e
09/28/2015 14:34:15.06 w3wp.exe (0x18A8) 0x1E90 SharePoint Foundation Monitoring
b4ly Verbose Leaving Monitored Scope (SPSqlClient). Execution Time=13170.3747263968 f379329d-3406-105e-a72a-ed7b53a2606e
09/28/2015 14:34:27.64 w3wp.exe (0x18A8) 0x1E90 SharePoint Foundation Monitoring
b4ly Verbose Leaving Monitored Scope (SPSqlClient). Execution Time=12583.333915344
f379329d-3406-105e-a72a-ed7b53a2606e
09/28/2015 14:34:38.99 w3wp.exe (0x18A8) 0x1E90 SharePoint Foundation Monitoring
b4ly Verbose Leaving Monitored Scope (SPSqlClient). Execution Time=11349.5634920081 f379329d-3406-105e-a72a-ed7b53a2606e
09/28/2015 14:34:52.29 w3wp.exe (0x18A8) 0x1E90 SharePoint Foundation Monitoring
b4ly Verbose Leaving Monitored Scope (SPSqlClient). Execution Time=13291.4156687512 f379329d-3406-105e-a72a-ed7b53a2606e
09/28/2015 14:35:04.29 w3wp.exe (0x18A8) 0x1E90 SharePoint Foundation Monitoring
b4ly Verbose Leaving Monitored Scope (SPSqlClient). Execution Time=12000.987066792
f379329d-3406-105e-a72a-ed7b53a2606e
09/28/2015 14:35:16.80 w3wp.exe (0x18A8) 0x1E90 SharePoint Foundation Monitoring
b4ly Verbose Leaving Monitored Scope (SPSqlClient). Execution Time=12511.4372776428 f379329d-3406-105e-a72a-ed7b53a2606e
09/28/2015 14:35:34.83 w3wp.exe (0x18A8) 0x1E90 SharePoint Foundation Monitoring
b4ly Verbose Leaving Monitored Scope (SPSqlClient). Execution Time=18016.6065100453 f379329d-3406-105e-a72a-ed7b53a2606e
09/28/2015 14:36:07.12 w3wp.exe (0x18A8) 0x1E90 SharePoint Foundation Monitoring
b4ly Verbose Leaving Monitored Scope (SPSqlClient). Execution Time=14083.1648994495 f379329d-3406-105e-a72a-ed7b53a2606e
09/28/2015 14:36:31.59 w3wp.exe (0x18A8) 0x1E90 SharePoint Foundation Monitoring
b4ly Verbose Leaving Monitored Scope (SPSqlClient). Execution Time=12470.2326692359 f379329d-3406-105e-a72a-ed7b53a2606e
09/28/2015 14:36:43.48 w3wp.exe (0x18A8) 0x1E90 SharePoint Foundation Monitoring
b4ly Verbose Leaving Monitored Scope (SPSqlClient). Execution Time=11900.7612699379 f379329d-3406-105e-a72a-ed7b53a2606e
09/28/2015 14:36:55.00 w3wp.exe (0x18A8) 0x1E90 SharePoint Foundation Monitoring
b4ly Verbose Leaving Monitored Scope (SPSqlClient). Execution Time=11517.2457672693 f379329d-3406-105e-a72a-ed7b53a2606e
09/28/2015 14:37:09.18 w3wp.exe (0x18A8) 0x1E90 SharePoint Foundation Monitoring
b4ly Verbose Leaving Monitored Scope (SPSqlClient). Execution Time=14178.7410830147 f379329d-3406-105e-a72a-ed7b53a2606e
09/28/2015 14:37:21.15 w3wp.exe (0x18A8) 0x1E90 SharePoint Foundation Monitoring
b4ly Verbose Leaving Monitored Scope (SPSqlClient). Execution Time=11971.7309043468 f379329d-3406-105e-a72a-ed7b53a2606e
09/28/2015 14:37:45.17 w3wp.exe (0x18A8) 0x1E90 SharePoint Foundation Monitoring
b4ly Verbose Leaving Monitored Scope (SPSqlClient). Execution Time=11461.8488522983 f379329d-3406-105e-a72a-ed7b53a2606e
09/28/2015 14:37:57.31 w3wp.exe (0x18A8) 0x1E90 SharePoint Foundation Monitoring
b4ly Verbose Leaving Monitored Scope (SPSqlClient). Execution Time=12151.2991874666 f379329d-3406-105e-a72a-ed7b53a2606e
09/28/2015 14:38:24.82 w3wp.exe (0x18A8) 0x1E90 SharePoint Foundation Monitoring
b4ly Verbose Leaving Monitored Scope (SPSqlClient). Execution Time=13845.709929614
f379329d-3406-105e-a72a-ed7b53a2606e
09/28/2015 14:38:38.24 w3wp.exe (0x18A8) 0x1E90 SharePoint Foundation Monitoring
b4ly Verbose Leaving Monitored Scope (SPSqlClient). Execution Time=13424.6617936078 f379329d-3406-105e-a72a-ed7b53a2606e
09/28/2015 14:39:02.41 w3wp.exe (0x18A8) 0x1E90 SharePoint Foundation Monitoring
b4ly Verbose Leaving Monitored Scope (SPSqlClient). Execution Time=12043.0871102333 f379329d-3406-105e-a72a-ed7b53a2606e
09/28/2015 14:39:15.80 w3wp.exe (0x18A8) 0x1E90 SharePoint Foundation Monitoring
b4ly Verbose Leaving Monitored Scope (SPSqlClient). Execution Time=13395.1089454107 f379329d-3406-105e-a72a-ed7b53a2606e
09/28/2015 14:39:27.52 w3wp.exe (0x18A8) 0x1E90 SharePoint Foundation Monitoring
b4ly Verbose Leaving Monitored Scope (SPSqlClient). Execution Time=11719.5715516916 f379329d-3406-105e-a72a-ed7b53a2606e
09/28/2015 14:39:55.19 w3wp.exe (0x18A8) 0x1E90 SharePoint Foundation Monitoring
b4ly Verbose Leaving Monitored Scope (SPSqlClient). Execution Time=14313.012757208
f379329d-3406-105e-a72a-ed7b53a2606e
Error: Getting data from SQL
09/28/2015 14:39:54.77 w3wp.exe (0x18A8) 0x1E90 SharePoint Foundation Database
fa45 High System.Threading.ThreadAbortException: Thread was being aborted. at SNIReadSyncOverAsync(SNI_ConnWrapper*
, SNI_Packet** , Int32 ) at SNINativeMethodWrapper.SNIReadSyncOverAsync(SafeHandle pConn, IntPtr& packet, Int32 timeout) at System.Data.SqlClient.TdsParserStateObject.ReadSniSyncOverAsync()
at System.Data.SqlClient.TdsParserStateObject.TryReadNetworkPacket() at System.Data.SqlClient.TdsParserStateObject.TryPrepareBuffer() at System.Data.SqlClient.TdsParserStateObject.TryReadByteArray(Byte[] buff,
Int32 offset, Int32 len, Int32& totalRead) at System.Data.SqlClient.TdsParserStateObject.TryReadString(Int32 length, String& value) at System.Data.SqlClient.TdsParser.TryProcessError(Byte token, TdsParserStateObject
stateObj, SqlError& error) at System.Data.SqlClient.TdsParser.TryRun(RunBehavior runBehavior, SqlCommand cmdHandler, SqlDataReader dataStream, BulkCopySimpleResultSet bulkCopyHandler, TdsParserStateObject stateObj, Boolean&
dataReady) at System.Data.SqlClient.SqlDataReader.TryConsumeMetaData() at System.Data.SqlClient.SqlDataReader.get_MetaData() at System.Data.SqlClient.SqlCommand.FinishExecuteReader(SqlDataReader
ds, RunBehavior runBehavior, String resetOptionsString) at System.Data.SqlClient.SqlCommand.RunExecuteReaderTds(CommandBehavior cmdBehavior, RunBehavior runBehavior, Boolean returnStream, Boolean async, Int32 timeout, Task& task,
Boolean asyncWrite, SqlDataReader ds) at System.Data.SqlClient.SqlCommand.RunExecuteReader(CommandBehavior cmdBehavior, RunBehavior runBehavior, Boolean returnStream, String method, TaskCompletionSource`1 completion, Int32 timeout,
Task& task, Boolean asyncWrite) at System.Data.SqlClient.SqlCommand.RunExecuteReader(CommandBehavior cmdBehavior, RunBehavior runBehavior, Boolean returnStream, String method) at System.Data.SqlClient.SqlCommand.ExecuteReader(CommandBehavior
behavior, String method) at System.Data.SqlClient.SqlCommand.ExecuteReader(CommandBehavior behavior) at Microsoft.SharePoint.Utilities.SqlSession.ExecuteReader(SqlCommand command, CommandBehavior behavior, SqlQueryData
monitoringData, Boolean retryForDeadLock) f379329d-3406-105e-a72a-ed7b53a2606e
09/28/2015 14:39:54.79 w3wp.exe (0x18A8) 0x1E90 SharePoint Foundation Database
fa46 High at Microsoft.SharePoint.SPSqlClient.ExecuteQueryInternal(Boolean retryfordeadlock) at Microsoft.SharePoint.SPSqlClient.ExecuteQuery(Boolean
retryfordeadlock) at Microsoft.SharePoint.Library.SPRequestInternalClass.MoveNavigationNode(String bstrUrl, Int32 lNodeId, DateTime dateParented, Int32 lParentId, Int32 lPreviousSiblingId) at Microsoft.SharePoint.Library.SPRequestInternalClass.MoveNavigationNode(String
bstrUrl, Int32 lNodeId, DateTime dateParented, Int32 lParentId, Int32 lPreviousSiblingId) at Microsoft.SharePoint.Library.SPRequest.MoveNavigationNode(String bstrUrl, Int32 lNodeId, DateTime dateParented, Int32 lParentId, Int32 lPreviousSiblingId)
at Microsoft.SharePoint.Navigation.SPNavigationNode.MoveInternal(SPNavigationNodeCollection collection, Int32 iPreviousNodeId) at Microsoft.SharePoint.Publishing.Navigation.SPNavigationSiteMapNode.HandleJavaScriptNavigationNode(Guid
objId, Int32 nodeId, String name, String url, String description, NodeTypes nodeType, String target, String audience, SPNavigationNodeCollection collection, Dictionary`2 oldDictionary) at Microsoft.SharePoint.Publishing.Internal.CodeBehind.AreaNavigationSettingsPage.OKButton_Click(Object
sender, EventArgs e) at System.Web.UI.WebControls.Button.RaisePostBackEvent(String eventArgument) at System.Web.UI.Page.ProcessRequestMain(Boolean includeStagesBeforeAsyncPoint, Boolean includeStagesAfterAsyncPoint)
at System.Web.UI.Page.ProcessRequest(Boolean includeStagesBeforeAsyncPoint, Boolean includeStagesAfterAsyncPoint) at System.Web.UI.Page.ProcessRequest() at System.Web.UI.Page.ProcessRequest(HttpContext context)
at System.Web.HttpApplication.CallHandlerExecutionStep.System.Web.HttpApplication.IExecutionStep.Execute() at System.Web.HttpApplication.ExecuteStep(IExecutionStep step, Boolean& completedSynchronously) at
System.Web.HttpApplication.PipelineStepManager.ResumeSteps(Exception error) at System.Web.HttpApplication.BeginProcessRequestNotification(HttpContext context, AsyncCallback cb) at System.Web.HttpRuntime.ProcessRequestNotificationPrivate(IIS7WorkerRequest
wr, HttpContext context) at System.Web.Hosting.PipelineRuntime.ProcessRequestNotificationHelper(IntPtr rootedObjectsPointer, IntPtr nativeRequestContext, IntPtr moduleData, Int32 flags) at System.Web.Hosting.PipelineRuntime.ProcessRequestNotification(IntPtr
rootedObjectsPointer, IntPtr nativeRequestContext, IntPtr moduleData, Int32 flags) at System.Web.Hosting.UnsafeIISMethods.MgdIndicateCompletion(IntPtr pHandler, RequestNotificationStatus& notificationStatus)
at System.Web.Hosting.UnsafeIISMethods.MgdIndicateCompletion(IntPtr pHandler, RequestNotificationStatus& notificationStatus) at System.Web.Hosting.PipelineRuntime.ProcessRequestNotificationHelper(IntPtr rootedObjectsPointer, IntPtr
nativeRequestContext, IntPtr moduleData, Int32 flags) at System.Web.Hosting.PipelineRuntime.ProcessRequestNotification(IntPtr rootedObjectsPointer, IntPtr nativeRequestContext, IntPtr moduleData, Int32 flags) f379329d-3406-105e-a72a-ed7b53a2606e
09/28/2015 14:39:54.80 w3wp.exe (0x18A8) 0x1E90 SharePoint Foundation Database
tzku High ConnectionString: 'Data Source=SQLSPI2013QA;Initial Catalog=SPI_Prod_DealerServices_CanadianDealerServices_Content;Integrated
Security=True;Enlist=False;Pooling=True;Min Pool Size=0;Max Pool Size=100;Asynchronous Processing=True;Connect Timeout=15' Partition: f237ae7f-3cf0-4a5c-b2b1-0c24204fdb1e ConnectionState: Closed ConnectionTimeout: 15
f379329d-3406-105e-a72a-ed7b53a2606e
09/28/2015 14:39:54.80 w3wp.exe (0x18A8) 0x1E90 SharePoint Foundation Database
tzkv High SqlCommand: 'BEGIN TRAN DECLARE @abort int SET @abort = 0 DECLARE @EidBase int,@EidHome int SET @EidBase = 0
SET @EidHome = NULL IF @abort = 0 BEGIN EXEC @abort = proc_NavStructAllocateEidBlockWebId @wssp0, @wssp1, @wssp2, @wssp3, @EidBase OUTPUT SELECT @wssp4 = @EidBase, @wssp5 = @abort END IF @abort = 0 BEGIN EXEC @abort = proc_NavStructMoveNode 'F237AE7F-3CF0-4A5C-B2B1-0C24204FDB1E','E7FB0369-03DA-4F9F-BA62-451DC53FDB9D',2136,1002,-2,N'20150917
22:07:34',@EidBase,@EidHome, @wssp6 OUTPUT SELECT @wssp7 = @abort END IF @abort = 0 BEGIN EXEC proc_NavStructLogChangesAndUpdateSiteChangedTime @wssp8, @wssp9, NULL END IF @abort <> 0 BEGIN ROLLBACK TRAN END ELSE BEGIN COMMIT TRAN END IF @abort = 0
BEGIN EXEC proc_UpdateDiskUsed 'F237AE7F-3CF0-4A5C-B2B1-0C24204FDB1E' END ' CommandType: Text CommandTimeout: 0 Parameter: '@wssp0' Type: UniqueIdentifier Size: 0 Direction: Input Value: 'f237ae7f-3cf0-4a5c-b2b1-0c24204fdb1e'
Parameter: '@wssp1' Type: UniqueIdentifier Size: 0 Direction: Input Value: 'e7fb0369-03da-4f9f-ba62-451dc53fdb9d' Parameter: '@wssp2' Type: Int Size: 0 Direction: Input Value: '0' Parameter: '@wssp3' Type: Int
Size: 0 Direction: Input Value: '2136' Parameter: '@wssp4' Type: Int Size: 0 Direction: Output Value: '' Parameter: '@wssp5' Type: Int Size: 0 Direction: Output Value: '' Parameter: '@wssp6'
Type: DateTime Size: 0 Direction: Output Value: '' Parameter: '@wssp7' Type: Int Size: 0 Direction: Output Value: '' Parameter: '@wssp8' Type: UniqueIdentifier Size: 0 Direction: Input Value: 'f237ae7f-3cf0-4a5c-b2b1-0c24204fdb1e'
Parameter: '@wssp9' Type: UniqueIdentifier Size: 0 Direction: Input Value: 'e7fb0369-03da-4f9f-ba62-451dc53fdb9d' f379329d-3406-105e-a72a-ed7b53a2606e
09/28/2015 14:39:54.80 w3wp.exe (0x18A8) 0x1E90 SharePoint Foundation Database
aek90 High SecurityOnOperationCheck = True f379329d-3406-105e-a72a-ed7b53a2606e
09/28/2015 14:39:54.80 w3wp.exe (0x18A8) 0x1E90 SharePoint Foundation Database
btq8 High System.Threading.ThreadAbortException: Thread was being aborted. at SNIReadSyncOverAsync(SNI_ConnWrapper*
, SNI_Packet** , Int32 ) at SNINativeMethodWrapper.SNIReadSyncOverAsync(SafeHandle pConn, IntPtr& packet, Int32 timeout) at System.Data.SqlClient.TdsParserStateObject.ReadSniSyncOverAsync()
at System.Data.SqlClient.TdsParserStateObject.TryReadNetworkPacket() at System.Data.SqlClient.TdsParserStateObject.TryPrepareBuffer() at System.Data.SqlClient.TdsParserStateObject.TryReadByteArray(Byte[] buff,
Int32 offset, Int32 len, Int32& totalRead) at System.Data.SqlClient.TdsParserStateObject.TryReadString(Int32 length, String& value) at System.Data.SqlClient.TdsParser.TryProcessError(Byte token, TdsParserStateObject
stateObj, SqlError& error) at System.Data.SqlClient.TdsParser.TryRun(RunBehavior runBehavior, SqlCommand cmdHandler, SqlDataReader dataStream, BulkCopySimpleResultSet bulkCopyHandler, TdsParserStateObject stateObj, Boolean&
dataReady) at System.Data.SqlClient.SqlDataReader.TryConsumeMetaData() at System.Data.SqlClient.SqlDataReader.get_MetaData() at System.Data.SqlClient.SqlCommand.FinishExecuteReader(SqlDataReader
ds, RunBehavior runBehavior, String resetOptionsString) at System.Data.SqlClient.SqlCommand.RunExecuteReaderTds(CommandBehavior cmdBehavior, RunBehavior runBehavior, Boolean returnStream, Boolean async, Int32 timeout, Task& task,
Boolean asyncWrite, SqlDataReader ds) at System.Data.SqlClient.SqlCommand.RunExecuteReader(CommandBehavior cmdBehavior, RunBehavior runBehavior, Boolean returnStream, String method, TaskCompletionSource`1 completion, Int32 timeout,
Task& task, Boolean asyncWrite) at System.Data.SqlClient.SqlCommand.RunExecuteReader(CommandBehavior cmdBehavior, RunBehavior runBehavior, Boolean returnStream, String method) at System.Data.SqlClient.SqlCommand.ExecuteReader(CommandBehavior
behavior, String method) at System.Data.SqlClient.SqlCommand.ExecuteReader(CommandBehavior behavior) at Microsoft.SharePoint.Utilities.SqlSession.ExecuteReader(SqlCommand command, CommandBehavior behavior, SqlQueryData
monitoringData, Boolean retryForDeadLock) at Microsoft.SharePoint.SPSqlClient.ExecuteQueryInternal(Boolean retryfordeadlock) at Microsoft.SharePoint.SPSqlClient.ExecuteQuery(Boolean retryfordeadlock)
f379329d-3406-105e-a72a-ed7b53a2606e
09/28/2015 14:39:54.80 w3wp.exe (0x18A8) 0x1E90 SharePoint Foundation Database
ad194 High ExecuteQuery failed with original error 0x80131530 f379329d-3406-105e-a72a-ed7b53a2606e
09/28/2015 14:39:54.80 w3wp.exe (0x18A8) 0x1E90 SharePoint Foundation Database
8z23 Unexpected Unexpected query execution failure in navigation query, HResult -2146233040. Query text (if available): "BEGIN
TRAN DECLARE @abort int SET @abort = 0 DECLARE @EidBase int,@EidHome int SET @EidBase = 0 SET @EidHome = NULL IF @abort = 0 BEGIN EXEC @abort = proc_NavStructAllocateEidBlockWebId @wssp0, @wssp1, @wssp2, @wssp3, @EidBase OUTPUT SELECT @wssp4 = @EidBase, @wssp5
= @abort END IF @abort = 0 BEGIN EXEC @abort = proc_NavStructMoveNode 'F237AE7F-3CF0-4A5C-B2B1-0C24204FDB1E','E7FB0369-03DA-4F9F-BA62-451DC53FDB9D',2136,1002,-2,N'20150917 22:07:34',@EidBase,@EidHome, @wssp6 OUTPUT SELECT @wssp7 = @abort END IF @abort = 0
BEGIN EXEC proc_NavStructLogChangesAndUpdateSiteChangedTime @wssp8, @wssp9, NULL END IF @abort <> 0 BEGIN ROLLBACK TRAN END ELSE BEGIN COMMIT TRAN END IF @abort = 0 BEGIN EXEC proc_UpdateDiskUsed 'F237AE7F-3CF0-4A5C-B2B1-0C24204FDB1E' END "
f379329d-3406-105e-a72a-ed7b53a2606e
09/28/2015 14:39:54.82 w3wp.exe (0x18A8) 0x1E90 SharePoint Foundation Stack Trace Collection
0000 Verbose Returning hr=0x8107140D: owssvr.dll: (unresolved symbol, module offset=00000000000DB7B5) at 0x00007FFB866FB7B5 Microsoft.SharePoint.Library.ni.dll: (unresolved
symbol, module offset=0000000000104417) at 0x00007FFB87694417 f379329d-3406-105e-a72a-ed7b53a2606e
09/28/2015 14:39:54.82 w3wp.exe (0x18A8) 0x1E90 SharePoint Foundation General
8kh7 High An unexpected error occurred while manipulating the navigational structure of this Web. f379329d-3406-105e-a72a-ed7b53a2606e
09/28/2015 14:39:54.83 w3wp.exe (0x18A8) 0x1E90 SharePoint Foundation General
8kh7 High Stack trace: onetutil.dll: (unresolved symbol, module offset=00000000000A215F) at 0x00007FFB8643215F onetutil.dll: (unresolved symbol, module offset=00000000000A20A9) at
0x00007FFB864320A9 owssvr.dll: (unresolved symbol, module offset=00000000000549DC) at 0x00007FFB866749DC owssvr.dll: (unresolved symbol, module offset=00000000000DBA15) at 0x00007FFB866FBA15 Microsoft.SharePoint.Library.ni.dll: (unresolved symbol, module offset=0000000000104417)
at 0x00007FFB87694417 f379329d-3406-105e-a72a-ed7b53a2606e
09/28/2015 14:39:54.85 w3wp.exe (0x18A8) 0x1E90 SharePoint Foundation General
8nca Medium Application error when access /_layouts/15/AreaNavigationSettings.aspx, Error=Request timed out. f379329d-3406-105e-a72a-ed7b53a2606e
09/28/2015 14:39:54.86 w3wp.exe (0x18A8) 0x1E90 SharePoint Foundation Runtime tkau
Unexpected System.Web.HttpException: Request timed out. f379329d-3406-105e-a72a-ed7b53a2606e
09/28/2015 14:39:54.88 w3wp.exe (0x18A8) 0x1E90 SharePoint Foundation General
ajlz0 High Getting Error Message for Exception System.Web.HttpException (0x80004005): Request timed out. f379329d-3406-105e-a72a-ed7b53a2606e
////////SOLUTION//////////
Here is the solution the engineer gave us and which worked for us in all our SP2013 farms for all the affected content databases:
On the SQL server, you need to run update statistics and update execution plan for all SharePointDBs
Exec sp_MSForEachtable 'UPDATE STATISTICS WITH FULLSCAN'
DBCC FREEPROCCACHE
Use at your own risk. I really didn't think it would help us but it totally solved the problem.
We were also told:
"After some discussion with others on the support team we would recommend running updating statistics with full scan (possible defragging indexes) on a daily basis. If the statistics remain updated then the execution plan will remain up
to date. If you delete thousands of documents/files/large site collection. Then you will also see performance issue and might need to update statistics in the middle of the day."
/////////QUESTION//////////
Why should we need to run “update statistics” once in a while when
https://support.microsoft.com/en-us/kb/932744 says that “update statistics” is “unnecessary because the SharePoint Timer service performs this task automatically.” (Maybe it’s not the exact same kind of updating?)
I checked some of the older sources I could find. . .
http://technet.microsoft.com/en-us/library/hh292622.aspx - Best practices for SQL Server in a SharePoint Server farm
http://www.microsoft.com/download/en/details.aspx?id=24282 - Database Maintenance for Microsoft SharePoint 2010 Products
http://blogs.msdn.com/b/ronalg/archive/2008/08/27/sql-maintenance-for-sharepoint-databases.aspx
http://technet.microsoft.com/en-us/library/cc990273.aspx - SQL Server 2008 R2 and SharePoint 2010 Products: Better Together (white paper) (SharePoint Server
2010)
All I see there is, "By using maintenance plans in SQL Server 2008 or SQL Server 2005, an administrator can schedule such operations as running database consistency checks, and reorganizing or rebuilding indexes."
And so I find myself wanting to communicate this to the Microsoft SharePoint Product group:
Since SharePoint timer jobs are supposed to do sql maintenance, and since the right sql maintenance wasn’t done by timer jobs to our sql databases, and since there is zero documentation out there on what maintenance dbas should be doing on sharepoint databases,
can you file a bug on this, escalate it to the product group, get a fix for it (a new timer job perhaps)?
It makes me wonder what sql maintenance jobs are running on the SQL servers that host the sharepoint databases for o365's sharepoint online.
So what do you forum readers know that I don't about this? Did I miss some documentation? Which SQL maintenance jobs should be set up for SP2013 databases on SQL2012?
Thanks for considering it!