Analytics using Application Insights on SharePoint classic sites

I will cover in detail how to add App Insights to a SharePoint classic site to track page views, but also touch on performance tracking and why this might be important.

There is guidance on the Microsoft Docs page on how to setup app insights, but it is not mentioned another possible scenario to add application insights on a site page and this is with JavaScript link (custom action). Adding JS link for the site will not require master page update. I do not recommend modifying the master page, so I will go with custom action for this post.

Create App Insights instance

To use App Insights monitoring and analytics capabilities we will need App Insights instance deployed to Azure. This is a good document to start with. Once the instance is created, we can get the instrumental key and use it with the script bellow.

var appInsights = window.appInsights || function (a) {

}({

instrumentationKey: "<YOUR_INSTRUMENTATION_KEY_HERE>"

});

window.appInsights = appInsights, appInsights.queue && 0 === appInsights.queue.length && appInsights.trackPageView();

This will give us the key and the script we need to use app insights for monitoring, but how to load that script from SharePoint page? I have created a file with the above script called appInsightsTracking.js for my case.

How load the appInsightsTracking.js on SharePoint classic site page

Custom action can be used to load the script from every page in the site collection, this is efficient way to do it in SharePoint online since the script will be loaded from CDN. SharePoint does some magic under the hood, if we add a js file to Site Assets or Style Library and use it for custom action JS link. It re-writes the URL so the script is loaded on a SharePoint page from CDN. Few lines of PnP Powershell code are needed to create custom action in case the site feature DenyAddAndCustomizePages is disabled.

Connect-PnPOnline https://xxx.sharepoint.com/sites/mysite -Credentials creds

# use Add-PnPFile to upload your appInsightsTracking.js file to SiteAssets to fully automate the process

Add-PnPFile -Path .\<YOUR_PATH>\appInsightsTracking.js -Folder SiteAssets/appInsightsTracking.js

# use Add-PnPJavaScriptLink to create new js link custom action to load the javascript with every page of the SharePoint classic site

Add-PnPJavaScriptLink -Name 'AppInsightsPerformance' -Url SiteAssets/appInsightsTracking.js -Scope Site -Sequence 100

If the "Scope" parameter in Add-PnPJavaScriptLink cmdlet is Site that means that the app insights will monitor all the sites and sub-sites in the site collection. If changed to 'Web' it will monitor only the current web where the custom action is installed in e.g. 'https://xxx.sharepoint.com/sites/mysite '. The "Sequence" parameter will determine what is the order that JavaScript file will load on the page. Since we do not want that to be a performance issue we can set it to high number to ensure other scripts are not waiting this one to load. In the SharePoint Online scenario, the script will be loaded from https://privatecdn.sharepointonline.com/xxx.sharepoint.com/sites/mysite/siteassets/appinsightstracking.js which will give good performance and load times since the script has been loaded from CDN.

If the site has the "DenyAddAndCustomizePages" feature enabled, a tenant administrator will have to disable the feature, so the JavaScript custom action can be successfully added to the classic SharePoint site. This can be done with the following PnP Powershell script:

Connect-PnPOnline https://xxx.sharepoint.com/sites/mysite -Credentials creds

# set DenyAddAndCustomizePages to 1 #Disabled

Set-PnPTenantSite https://xxx.sharepoint.com/sites/mysite -NoScriptSite:$false

How disable app insights from the SharePoint classic site pages

The following PnP Powershell will disable the custom action and app insights would not log information about the site collection anymore.

# get all custom actions

# find the id of the AppInsightsPerformance from the list

Get-PnPCustomAction -Scope Site

# remove the custom action

Remove-PnPCustomAction -Identity cc6da0c1-b344-446d-b483-5d53a4e6d996 -Scope Site -Force

Monitor SharePoint page load times

App Insights logs data by default on the performance of the page and can be used in reports to give overview how a SharePoint classic page performs. This is the performance metrics payload sent to app insights out of the box:

// This is what App Insights sends out of the box

// 7:{time: "2018-07-12T22:43:23.122Z", iKey: "d63d4ac8-d904-4577-91a7-fbd62f6ed92a",…}

// data: {baseType: "PageviewPerformanceData",…}

// baseData: {ver: 2, name: "Search: -", url: "https://prufinancial.sharepoint.com/sites/velinsearch",…}

// domProcessing:"00:00:00.817"

// duration:"00:00:03.056"

// name:"Search: -"

// networkConnect:"00:00:00.001"

// perfTotal:"00:00:03.056"

// receivedResponse:"00:00:00.124"

// sentRequest:"00:00:02.087"

// url:"https://prufinancial.sharepoint.com/sites/velinsearch"

// ver:2

// baseType:"PageviewPerformanceData"

// iKey:"d63d4ac8-d904-4577-91a7-fbd62f6ed92a"

// name:"Microsoft.ApplicationInsights.d63d4ac8d904457791a7fbd62f6ed92a.PageviewPerformance"

Navigation timing API and extending the performance metrics by sending app insights custom event

What probably app insights uses for the performance payload under the hood is the browser Navigation timing API. We can utilize it further and send custom event with the datа we would like to receive if we are not satisfied by the OOTB payload that is sent. Here is a code sniped that sends app insights custom event with all the metrics we get from the Navigation timing API.

function performanceMetrics() {

if(maxRetry === retryCount) {

console.log('Max retry reached.')

retrun;

}

retryCount++;

if(window.performance.timing.loadEventEnd) {

console.log('performanceMetrics start');

// https://developer.mozilla.org/en-US/docs/Web/API/Navigation_timing_API

var perfData = window.performance.timing;

// Calculate the total page load time

// This subtracts the time at which navigation began (navigationStart) from the time at which the load event handler returns (loadEventEnd). This gives you the perceived page load time.

var pageLoadTime = perfData.loadEventEnd - perfData.navigationStart;

// Calculate request response time

// You can calculate the time elapsed between the beginning of a request and the completion of getting the response using code like this:

var connectTime = perfData.responseEnd - perfData.requestStart;

// Calculate page render time

// This is obtained by starting with the time at which loading of the DOM and its dependencies is complete (domComplete) and subtracting from it the time at which parsing of the DOM began (domLoading).

var renderTime = perfData.domComplete - perfData.domLoading;

// track metrics

appInsights.trackMetric("pageLoadTime", pageLoadTime);

appInsights.trackMetric("connectTime", connectTime);

appInsights.trackMetric("renderTime", renderTime);

// track as event

appInsights.trackEvent("performanceMetrics", {

"correlationId": "someid",

"pageLoadTime": pageLoadTime,

"connectTime": connectTime,

"renderTime": renderTime,

"performance.timing": JSON.stringify(perfData.toJSON())

});

console.log('performanceMetrics sent');

return;

}

setTimeout(performanceMetrics, 2000);

}

performanceMetrics();

Please note I am waiting with setTimeoout to populate the timings and then send the data. Some properties like window.performance.timing.loadEventEnd are populated with values after the "DOMContentLoaded" or window.onload fired so I have to wait using setTimeout.

The app insights trackEvent custom payload can look like

// This is my custom data sent via trackEvent:

// 6:{time: "2018-07-12T22:43:24.557Z", iKey: "d63d4ac8-d904-4577-91a7-fbd62f6ed92a",…}

// data:{baseType: "EventData", baseData: {ver: 2, name: "performanceMetrics",…}}

// baseData:{ver: 2, name: "performanceMetrics",…}

// name:"performanceMetrics"

// properties:

// {correlationId: "someid", pageLoadTime: "3056", connectTime: "2211", renderTime: "845",…}

// connectTime:"2211"

// correlationId:"someid"

// pageLoadTime:"3056"

// performance.timing:"{"navigationStart":1531435399966,"unloadEventStart":1531435402151,"unloadEventEnd":1531435402156,"redirectStart":0,"redirectEnd":0,"fetchStart":1531435399967,"domainLookupStart":1531435399967,"domainLookupEnd":1531435399967,"connectStart":1531435399967,"connectEnd":1531435399967,"secureConnectionStart":0,"requestStart":1531435399994,"responseStart":1531435402081,"responseEnd":1531435402205,"domLoading":1531435402174,"domInteractive":1531435402660,"domContentLoadedEventStart":1531435402660,"domContentLoadedEventEnd":1531435402710,"domComplete":1531435403019,"loadEventStart":1531435403019,"loadEventEnd":1531435403022}"

// renderTime:"845"

// ver:2

// baseType:"EventData"

// iKey:"d63d4ac8-d904-4577-91a7-fbd62f6ed92a"

// name:"Microsoft.ApplicationInsights.d63d4ac8d904457791a7fbd62f6ed92a.Event"

Conclusion

The SharePoint site usage page provided by Microsoft is still not feature rich and it does not provide with page hits information (but only document hits) for example. Until that is improved we can use app insights to track page views or create custom heat maps since we can send custom events. The traditional application load testing or stress testing where you measure the performance of a page or site is not permitted in SharePoint Online, but we can use application insights to gather page performance matrices directly from user's browser. This can give us enough information if a page or site is slow for user or users in specific region.

App Insights external reference might be a problem

The app insights spinet above will try reach the app insights servers to download the app insights JavaScript library. If your organization does not allow external links, you will have to bundle the app insights library together with the script.

Useful URLs

https://www.w3.org/TR/navigation-timing/#sec-window.performance-attribute

https://github.com/Microsoft/ApplicationInsights-JS

https://developer.mozilla.org/en-US/docs/Web/API/Performance

https://developer.mozilla.org/en-US/docs/Web/API/Navigation_timing_API

https://docs.microsoft.com/en-us/azure/application-insights/app-insights-api-custom-events-metrics

https://developer.mozilla.org/en-US/docs/tag/Navigation%20Timing

https://docs.microsoft.com/en-us/powershell/module/sharepoint-pnp/add-pnpcustomaction?view=sharepoint-ps