Background: the SharePoint Server 2013 Standard SP1 on a multiple server farm, 2 WFEs, and two ASs.
The Usage and Health Data Collection Service Application was set up using a PowerShell script. The Health report part is working since the beginning. However, when accessing any usage reports, namely "popularity trends", under Search Service Application, Site Collections, Sites, and Libraries, it always returns "Error: Sorry, something went wrong. An unexpected error has occured" when one try to download a report file. A Correlation ID is always provided, but offer little information for exactly where we should look up. (See the end of this message.)
So far we have checked the following items:
1. The Usage and Health Service Application and its proxy has been created and started.
2. The associated Usage and Health database is running and has a healthy size, several GB.
3. Under CA->Monitoring->Configure usage adn health data collection, "Enable usage data collection" checked. Log File location exists in all servers. But only has an ".ini" file under each subfolders here.
4. Under CA->Monitoring->Review job definations, "Microsoft SharePoint Foundation Usage Data Import" was set "Minutes", "Microsoft SharePoint Foundation Usage Data Processing" was set "Disabled". Has since updated to "Hourly".
5. Under CA->Monitoring->Check job status, "Microsoft SharePoint Foundation Usage Data Import" was reported "suceeded" for all servers. but for "Microsoft SharePoint Foundation Usage Data Processing" only repored suceeded from two Applications Servers, not from the WFEs. (Although under "Scheduled", the WFEs are also listed with Scheduled job.")
6. Under Management Shell, "Get-SPUsageDefinition" shows there is no "Receivers" set for any entries; so "EnableReceivers" are all set to "False". Based on the scripted provided in http ://geekswithblogs.net/bjackett/archive/2013/08/26/powershell-script-to-workaround-no-data-in-sharepoint-2013-usage.aspx, we have turned on the receiver for the entries of "Analytics Usage" and "Page Requests". But not for anything else yet.
Here is a typical segment of the error log obtained using "Merge-SPlogFiles" with corellation id provided:
09/25/2014 11:36:26.13 w3wp.exe (XXXXXWEB02:0x0D60) 0x1E78 SharePoint Server Search Query
dka1 High SearchServiceApplicationProxy::GetAnalyticsEventTypeDefinitions--Proxy Name:Search Service Application Proxy EndPoint: http ://xxxxxweb01:32843/024fc5f105e444a8b4b8e23d63aa2012/SearchService.svc 1afabb9c-cd3b-0074-0f8c-79621a092583
09/25/2014 11:36:26.13 w3wp.exe (XXXXXWEB02:0x0D60) 0x1E78 SharePoint Server Search Query
dk8z High SearchServiceApplicationProxy::GetChannel--Channel Creation time: 0 1afabb9c-cd3b-0074-0f8c-79621a092583
09/25/2014 11:36:26.13 w3wp.exe (XXXXXWEB02:0x0D60) 0x1E78 SharePoint Foundation Topology
e5mc Medium WcfSendRequest: RemoteAddress: 'http://xxxxxweb01:32843/024fc5f105e444a8b4b8e23d63aa2012/SearchService.svc' Channel: 'Microsoft.Office.Server.Search.Administration.ISearchServiceApplication' Action: 'http://tempuri.org/IAnalyticsServiceApplication/GetAnalyticsEventTypeDefinitions'
MessageId: 'urn:uuid:be3e199e-3970-4834-9a9b-1ed6379c39fe' 1afabb9c-cd3b-0074-0f8c-79621a092583
09/25/2014 11:36:26.15 w3wp.exe (XXXXXWEB01:0x170C) 0x1730 SharePoint Server Database
abihi High The scale out database mappings cache has been refreshed for service id 024fc5f1-05e4-44a8-b4b8-e23d63aa2012. The new mappings are as follows: 1afabb9c-cd3b-0074-0f8c-79621a092583
09/25/2014 11:36:26.15 w3wp.exe (XXXXXWEB01:0x170C) 0x1730 SharePoint Server Database
abihh High The scale out database mappings cache is getting refreshed for service id 024fc5f1-05e4-44a8-b4b8-e23d63aa2012. 1afabb9c-cd3b-0074-0f8c-79621a092583
09/25/2014 11:36:26.15 w3wp.exe (XXXXXWEB01:0x170C) 0x1730 SharePoint Server Search Analytics
ad4bl High SearchServiceApplication (SearchAnalyticsReportingDatabase) GetScaleOutDatabases 1afabb9c-cd3b-0074-0f8c-79621a092583
09/25/2014 11:36:26.15 w3wp.exe (XXXXXWEB01:0x170C) 0x1730 SharePoint Server Database
abihi High The scale out database mappings cache has been refreshed for service id 024fc5f1-05e4-44a8-b4b8-e23d63aa2012. The new mappings are as follows: 1afabb9c-cd3b-0074-0f8c-79621a092583
09/25/2014 11:36:26.15 w3wp.exe (XXXXXWEB01:0x170C) 0x1730 SharePoint Server Database
abihf High A database range mapping cannot be found for composite key: 2B-85-37-0C-D0-34-8E-41-91-C6-2A-C2-5A-F4-BE-5B-00-00-00-00-00-00-00-00-00-00-00-00-00-00-00-00 1afabb9c-cd3b-0074-0f8c-79621a092583
09/25/2014 11:36:26.15 w3wp.exe (XXXXXWEB01:0x170C) 0x1730 SharePoint Foundation Micro Trace
uls4 Medium Micro Trace Tags: 0 nasq,0 e5mb,0 abihh,0 ad4bl,10 abihi,0 abihh,0 ad4bl,0 abihi,0 abihf 1afabb9c-cd3b-0074-0f8c-79621a092583
09/25/2014 11:36:26.15 w3wp.exe (XXXXXWEB01:0x170C) 0x1730 SharePoint Foundation Monitoring
b4ly Medium Leaving Monitored Scope (ExecuteWcfServerOperation). Execution Time=12.2741158443322 1afabb9c-cd3b-0074-0f8c-79621a092583
09/25/2014 11:36:26.21 w3wp.exe (XXXXXWEB02:0x0D60) 0x1E78 SharePoint Server Search Query
dka5 High SearchServiceApplicationProxy::GetAnalyticsEventTypeDefinitions--Error occured: System.ServiceModel.FaultException`1[System.ServiceModel.ExceptionDetail]: We're sorry, we weren't able to complete the operation, please try again in a few minutes.
If you see this message repeatedly, contact your administrator. (Fault Detail is equal to An ExceptionDetail, likely created by IncludeExceptionDetailInFaults=true, whose value is: Microsoft.SharePoint.SPException: We're sorry, we weren't able to complete
the operation, please try again in a few minutes. If you see this message repeatedly, contact your administrator. at Microsoft.Office.Server.SPScaleOutDatabaseMap.CreateSqlSession(ISPScaleOutDatabaseMapProvider mapProvider, Byte[] compositeKey,
Guid forceRefreshVersion, Guid& version) at Mi... 1afabb9c-cd3b-0074-0f8c-79621a092583
09/25/2014 11:36:26.21* w3wp.exe (XXXXXWEB02:0x0D60) 0x1E78 SharePoint Server Search Query
dka5 High ...crosoft.Office.Server.SPScaleOutDatabaseCommandExecutor.Execute(ExecuteDelegate operation, ISPScaleOutDatabaseMapProvider mapProvider, SPSqlCommand command, Byte[] compositeKey) at Microsoft.Office.Server.Search.Analytics.AnalyticsReportingApi.GetTenantSettings(Guid
tenantId) at Microsoft.Office.Server.Search.Analytics.AnalyticsServiceApplication.GetTenantConfig(Guid tenantId, Boolean throwOnMissing) at Microsoft.Office.Server.Search.Analytics.AnalyticsServiceApplication.GetTenantEventTypeDefinitions(Guid
tenantId, Boolean throwOnMissing) ...). 1afabb9c-cd3b-0074-0f8c-79621a092583
09/25/2014 11:36:26.24 w3wp.exe (XXXXXWEB02:0x0D60) 0x1E78 SharePoint Foundation Topology
8313 Critical A failure was reported when trying to invoke a service application: EndpointFailure Process Name: w3wp Process ID: 3424 AppDomain Name: /LM/W3SVC/1049122689/ROOT-1-130560999036408585 AppDomain ID: 2 Service Application Uri: urn:schemas-microsoft-com:sharepoint:service:024fc5f105e444a8b4b8e23d63aa2012#authority=urn:uuid:2aab9ca7aa6d4e6fb06e4d8d2168ca59&authority=https://XXXXXapp04:32844/Topology/topology.svc
Active Endpoints: 1 Failed Endpoints:1 Affected Endpoint: http ://XXXXXweb01:32843/024fc5f105e444a8b4b8e23d63aa2012/SearchService.svc 1afabb9c-cd3b-0074-0f8c-79621a092583
09/25/2014 11:36:26.25 w3wp.exe (XXXXXWEB02:0x0D60) 0x1E78 SharePoint Foundation General
8nca Medium Application error when access /_layouts/15/topreport.aspx, Error=We're sorry, we weren't able to complete the operation, please try again in a few minutes. If you see this message repeatedly, contact your administrator. Server stack trace:
at System.ServiceModel.Channels.ServiceChannel.ThrowIfFaultUnderstood(Message reply, MessageFault fault, String action, MessageVersion version, FaultConverter faultConverter) at System.ServiceModel.Channels.ServiceChannel.HandleReply(ProxyOperationRuntime
operation, ProxyRpc& rpc) at System.ServiceModel.Channels.ServiceChannel.Call(String action, Boolean oneway, ProxyOperationRuntime operation, Object[] ins, Object[] outs, TimeSpan timeout) at System.ServiceModel.Channels.ServiceChannelProxy.InvokeService(IMethodCallMessage
methodCall, ... 1afabb9c-cd3b-0074-0f8c-79621a092583
09/25/2014 11:36:26.25* w3wp.exe (XXXXXWEB02:0x0D60) 0x1E78 SharePoint Foundation General
8nca Medium ...ProxyOperationRuntime operation) at System.ServiceModel.Channels.ServiceChannelProxy.Invoke(IMessage message) Exception rethrown at [0]: at System.Runtime.Remoting.Proxies.RealProxy.HandleReturnMessage(IMessage
reqMsg, IMessage retMsg) at System.Runtime.Remoting.Proxies.RealProxy.PrivateInvoke(MessageData& msgData, Int32 type) at Microsoft.Office.Server.Search.Analytics.IAnalyticsServiceApplication.GetAnalyticsEventTypeDefinitions(Guid
tenantId, AnalyticsEventTypes filter) at Microsoft.Office.Server.Search.Administration.SearchServiceApplicationProxy.<>c__DisplayClass2f9.<GetAnalyticsEventTypeDefinitions>b__2f8(ISearchServiceApplication serviceApplication)
at Microsoft.Office.Server.Search.Administration.SearchServiceApplicationProxy.DoSpLoadBalancedUriWsOp[T](W... 1afabb9c-cd3b-0074-0f8c-79621a092583
09/25/2014 11:36:26.25* w3wp.exe (XXXXXWEB02:0x0D60) 0x1E78 SharePoint Foundation General
8nca Medium ...ebServiceBackedOperation`1 webServiceCall, Int32 timeoutInMilliseconds, Int32 wcfTimeoutInMilliseconds, String operationName) at Microsoft.Office.Server.Search.Administration.SearchServiceApplicationProxy.DoWebServiceBackedOperation[T](String
operationName, Int32 timeoutInMilliseconds, Int32 wcfTimeoutInMilliseconds, WebServiceBackedOperation`1 webServiceCall) at Microsoft.Office.Server.Search.Administration.SearchServiceApplicationProxy.DoWebServiceBackedOperation[T](String
operationName, WebServiceBackedOperation`1 webServiceCall) at Microsoft.Office.Server.Analytics.Reporting.AnalyticsReport.DetermineEventTypes(ISearchServiceApplication ssa, Guid tenantId, Boolean multiItemQuery)
at Microsoft.Office.DocumentManagement.Pages.TopReportPage.GetTopReportEventTypes(SPSite sit... 1afabb9c-cd3b-0074-0f8c-79621a092583
09/25/2014 11:36:26.25* w3wp.exe (XXXXXWEB02:0x0D60) 0x1E78 SharePoint Foundation General
8nca Medium ...e) at Microsoft.Office.DocumentManagement.Pages.TopReportPage.SetupTopReport(SPWeb web, SPList list) at Microsoft.Office.DocumentManagement.Pages.TopReportPage.OnInit(EventArgs e)
at System.Web.UI.Control.InitRecursive(Control namingContainer) at System.Web.UI.Page.ProcessRequestMain(Boolean includeStagesBeforeAsyncPoint, Boolean includeStagesAfterAsyncPoint) 1afabb9c-cd3b-0074-0f8c-79621a092583
09/25/2014 11:36:26.27 w3wp.exe (XXXXXWEB02:0x0D60) 0x1E78 SharePoint Foundation Runtime
tkau Unexpected System.ServiceModel.FaultException`1[[System.ServiceModel.ExceptionDetail, System.ServiceModel, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089]]: We're sorry, we weren't able to complete the operation, please try again in
a few minutes. If you see this message repeatedly, contact your administrator. Server stack trace: at System.ServiceModel.Channels.ServiceChannel.ThrowIfFaultUnderstood(Message reply, MessageFault fault, String action,
MessageVersion version, FaultConverter faultConverter) at System.ServiceModel.Channels.ServiceChannel.HandleReply(ProxyOperationRuntime operation, ProxyRpc& rpc) at System.ServiceModel.Channels.ServiceChannel.Call(String
action, Boolean oneway, ProxyOperationRuntime operation, Object[] ins, Object[] outs, TimeSpan timeout) ... 1afabb9c-cd3b-0074-0f8c-79621a092583
09/25/2014 11:36:26.27* w3wp.exe (XXXXXWEB02:0x0D60) 0x1E78 SharePoint Foundation Runtime
tkau Unexpected ... at System.ServiceModel.Channels.ServiceChannelProxy.InvokeService(IMethodCallMessage methodCall, ProxyOperationRuntime operation) at System.ServiceModel.Channels.ServiceChannelProxy.Invoke(IMessage
message) Exception rethrown at [0]: at System.Runtime.Remoting.Proxies.RealProxy.HandleReturnMessage(IMessage reqMsg, IMessage retMsg) at System.Runtime.Remoting.Proxies.RealProxy.PrivateInvoke(MessageData&
msgData, Int32 type) at Microsoft.Office.Server.Search.Analytics.IAnalyticsServiceApplication.GetAnalyticsEventTypeDefinitions(Guid tenantId, AnalyticsEventTypes filter) at Microsoft.Office.Server.Search.Administration.SearchServiceApplicationProxy.<>c__DisplayClass2f9.<GetAnalyticsEventTypeDefinitions>b__2f8(ISearchServiceApplication
serviceApplication) at Mic... 1afabb9c-cd3b-0074-0f8c-79621a092583
09/25/2014 11:36:26.27* w3wp.exe (XXXXXWEB02:0x0D60) 0x1E78 SharePoint Foundation Runtime
tkau Unexpected ...rosoft.Office.Server.Search.Administration.SearchServiceApplicationProxy.DoSpLoadBalancedUriWsOp[T](WebServiceBackedOperation`1 webServiceCall, Int32 timeoutInMilliseconds, Int32 wcfTimeoutInMilliseconds, String operationName)
at Microsoft.Office.Server.Search.Administration.SearchServiceApplicationProxy.DoWebServiceBackedOperation[T](String operationName, Int32 timeoutInMilliseconds, Int32 wcfTimeoutInMilliseconds, WebServiceBackedOperation`1 webServiceCall)
at Microsoft.Office.Server.Search.Administration.SearchServiceApplicationProxy.DoWebServiceBackedOperation[T](String operationName, WebServiceBackedOperation`1 webServiceCall) at Microsoft.Office.Server.Analytics.Reporting.AnalyticsReport.DetermineEventTypes(ISearchServiceApplication
ssa, Guid tenantId, Boolean multiItemQu... 1afabb9c-cd3b-0074-0f8c-79621a092583
09/25/2014 11:36:26.27* w3wp.exe (XXXXXWEB02:0x0D60) 0x1E78 SharePoint Foundation Runtime
tkau Unexpected ...ery) at Microsoft.Office.DocumentManagement.Pages.TopReportPage.GetTopReportEventTypes(SPSite site) at Microsoft.Office.DocumentManagement.Pages.TopReportPage.SetupTopReport(SPWeb web, SPList
list) at Microsoft.Office.DocumentManagement.Pages.TopReportPage.OnInit(EventArgs e) at System.Web.UI.Control.InitRecursive(Control namingContainer) at System.Web.UI.Page.ProcessRequestMain(Boolean
includeStagesBeforeAsyncPoint, Boolean includeStagesAfterAsyncPoint) 1afabb9c-cd3b-0074-0f8c-79621a092583
09/25/2014 11:36:26.27 w3wp.exe (XXXXXWEB02:0x0D60) 0x1E78 SharePoint Foundation General
ajlz0 High Getting Error Message for Exception System.Web.HttpUnhandledException (0x80004005): Exception of type 'System.Web.HttpUnhandledException' was thrown. ---> System.ServiceModel.FaultException`1[System.ServiceModel.ExceptionDetail]: We're sorry,
we weren't able to complete the operation, please try again in a few minutes. If you see this message repeatedly, contact your administrator. (Fault Detail is equal to An ExceptionDetail, likely created by IncludeExceptionDetailInFaults=true, whose value is:
Microsoft.SharePoint.SPException: We're sorry, we weren't able to complete the operation, please try again in a few minutes. If you see this message repeatedly, contact your administrator. at Microsoft.Office.Server.SPScaleOutDatabaseMap.CreateSqlSession(ISPScaleOutDatabaseMapProvider
mapProvid... 1afabb9c-cd3b-0074-0f8c-79621a092583
09/25/2014 11:36:26.27* w3wp.exe (XXXXXWEB02:0x0D60) 0x1E78 SharePoint Foundation General
ajlz0 High ...er, Byte[] compositeKey, Guid forceRefreshVersion, Guid& version) at Microsoft.Office.Server.SPScaleOutDatabaseCommandExecutor.Execute(ExecuteDelegate operation, ISPScaleOutDatabaseMapProvider mapProvider, SPSqlCommand
command, Byte[] compositeKey) at Microsoft.Office.Server.Search.Analytics.AnalyticsReportingApi.GetTenantSettings(Guid tenantId) at Microsoft.Office.Server.Search.Analytics.AnalyticsServiceApplication.GetTenantConfig(Guid
tenantId, Boolean throwOnMissing) at Microsoft.Office.Server.Search.Analytics.AnalyticsServiceApplication.GetTenantEventTypeDefinitions(Guid tenantId, Boolean throwOnMissing) ...). at System.Web.UI.Page.HandleError(Exception
e) at System.Web.UI.Page.ProcessRequestMain(Boolean includeStagesBeforeAsyncPoint, Boolean includeStagesAfterA... 1afabb9c-cd3b-0074-0f8c-79621a092583
09/25/2014 11:36:26.27* w3wp.exe (XXXXXWEB02:0x0D60) 0x1E78 SharePoint Foundation General
ajlz0 High ...syncPoint) at System.Web.UI.Page.ProcessRequest(Boolean includeStagesBeforeAsyncPoint, Boolean includeStagesAfterAsyncPoint) at System.Web.UI.Page.ProcessRequest() at System.Web.UI.Page.ProcessRequest(HttpContext
context) at System.Web.HttpApplication.CallHandlerExecutionStep.System.Web.HttpApplication.IExecutionStep.Execute() at System.Web.HttpApplication.ExecuteStep(IExecutionStep step, Boolean& completedSynchronously)
1afabb9c-cd3b-0074-0f8c-79621a092583
09/25/2014 11:36:26.36 w3wp.exe (XXXXXWEB02:0x0D60) 0x1E78 SharePoint Foundation Upgrade
aiaih High [Forced due to logging gap, Original Level: Verbose] desiredVersion: {0} 1afabb9c-cd3b-0074-0f8c-79621a092583
Please help to identify the problem and the proper way to fix it. Thanks.
(Sorry I have to disable all hyper links in the message in order to post this message.)