Archiving Azure Application Insights Data for Deep Analytics

A solution for exporting Insights Data for correlated analysis and archival.

  • Did you have an incident on your Insights data that you want to analyze offline?
  • Does your current plan doesn’t allow you to store Insights data for more than a specified period, usually 3 months?
  • Does turning on the “Continuous Export” option in Azure tell you it’s not for the old analytics data but only the newly generated future data only?
  • Do you want to store Aggregate Insights data for Usage Analytics over a large period of time on a local server?
  • Do you want to correlate your App Insights with some Inhouse Data?
    Here is a simpler cost-effective way to accomplish this!

Currently, Azure Applications like Azure Functions, AppServices, or VM hosted Dotnet/ Java/ NodeJs applications use Application Insights for Application Performance Monitoring(APM) and Logging. By default, Azure Application Insights will save the data/metrics for the application without any additional cost for 90 days. After that, we have to pay extra to increase the retention of application insight data, but still, we can keep the data only up to 730 days at the maximum.

If you are looking for long term storage, you have to enable the “Continuous Export” option in the application insight, but this will work only for the new data but not for old data.

You may choose to export from Azure Dashboard also, but there are some constraints in this method. You may be constrained either by storage or bandwidth or other constraints. Please refer to the constraints before choosing your approach.

Nevertheless having a script would be handy and provides convenience. You may choose to delete archive Analytics data to save on costs after exporting through this script.

This simple python script will help you to export the data in the following scenarios :

  1. When you want to export between 2 fixed time ranges.
  2. When you have a large amount of data in Application Insights and you have constraints to export via Azure Dashboard.
  3. When you don’t want to enable the Continuous Export option.

Requirement:

1. Python 3 with PIP

2. Application Insights Application ID and Read-Only API Key. (https://dev.applicationinsights.io/documentation/Authorization/API-key-and-App-ID)

3. PowerBI Desktop or Other tools to view JSON or CSV files.

(https://powerbi.microsoft.com/en-us/desktop/)

Steps:

1. Install requests library using PIP.

$ sudo pip install requests

2. Download the script from GitHub Gist to your local machine.

https://gist.github.com/satheeshpayoda/92065d9fbaf5b0158728a8537d79af0e

3. Run the python script with — help parameter. So you can see all the available options.

$ python export-data-ai.py — help

Glossary for the parameters:

4. Once you have the necessary things, you can call a python script with the required parameters.

Example 1:

If you want to export the data between 2021/01/01 12 AM to 2021/01/02 12 AM

$ python export-data-ai.py ‘app-id’ ‘api-key’ ‘2020–12–01T00:00:00Z’ ‘2020–12–02T01:00:00Z’

Example 2:

If you want to export the data between 2021/01/01 12 AM to 2021/01/02 12 AM in 6 intervals.

$ python export-data-ai.py ‘app-id’ ‘api-key’ ‘2020–12–01T00:00:00Z’ ‘2020–12–02T01:00:00Z’ — split_range=21600

Example 3:

If you want to export the data in CSV format

$ python export-data-ai.py ‘app-id’ ‘api-key’ ‘2020–12–01T00:00:00Z’ ‘2020–12–02T01:00:00Z’ — toCSV true

Example 4:

If you faced any API error due to Azure limitation, try changing these two values:

— split_range — For more than 500,000 records or 64 MB limit, decrease the split_range interval so the script will not reach the limit.

— sleep — To fix the Rate Limit issue, increase the sleep seconds

$ python export-data-ai.py ‘app-id’ ‘api-key’ ‘2020–12–01T00:00:00Z’ ‘2020–12–02T01:00:00Z’ — toCSV true

5. You can use the PowerBI Desktop tool to view or visualize the exported data. Alternatively, you can also use Google Sheet/Excel to view the exported CSV file.

Tips:

  • If you are trying to export large amounts of data from an Application Insight, it is advised to use Azure Virtual Machine with sufficient storage before running this python script because It will be faster and easier.
  • If export fails because of any Application Insight API limits, try again by changing the following two parameters: — split_range and — sleep
  • If you don’t want to export all the Application Insight data, you can choose which tables to export, by using the variable name: tables in the python script.
  • Feel free to improve the script from Github Gist.

Further References

Author: Satheeshkumar Ravichandran

Your Digital Transformation partner. We are here to share knowledge on varied technologies, updates; and to stay in touch with the tech-space.

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store