How Can I Use Computer Vision To Read Charts? Graph Interpretation AI & Chart Analysis

Do you struggle with analyzing charts & graphs? Computer vision can help! Learn how it can quickly & accurately extract insights from charts in our blog post.

Using computer vision or Artificial Intelligence (AI) algorithms to interpret charts is a fairly complex problem, which is precisely why the AI job market is almost always on the lookout for talent that not only has a deep understanding of machine learning, deep learning, natural language processing, data structures and graph neural networks in addition to the basics of computer science.

The automotive and transportation industry is expected to be the largest end-user segment for computer vision during the forecast period from 2021 to 2028, due to the increasing demand for autonomous vehicles and advanced driver assistance systems. (Source: Grand View Research)

‍

Getting back to the chart!

We believe you will agree that this is a two-step process - reading the chart and interpreting the values - both of which are full of complexity.

Let us explain it with an example. Take a look at this chart:
For computer vision to work on this chart, it needs to do the following:

Recognize that is a chart, and that is not a big problem. You can do that easily by training your algorithm with charts of different types as training data. In fact, by deploying basic machine learning techniques, it will also tell you what type of chart this is - perhaps a time series chart.
Understand that there are two axes for this chart and their values vary from 0 to 100 for the Y axis and Dec 2007 to October 2008 for the X axis. This is fairly complex - first, you need to employ OCR in some capacity to efficiently accomplish the data mining and figure out which data aligns to which axis.

Understand that there are two cities whose values are plotted on the graph - New York (NY) and San Francisco (SF). You may need to refer to external sources to figure out NY is New York and SF is San Francisco but this is fairly easy too.

Now, the most complex bit - see and calculate values of all variables at a particular point in a graph. For instance, in the given circle, it needs to figure out that:

In August 2008, the average temperature in NY was 80 degrees and for San Francisco, it was 60 degrees. This is not trivial, even for a human being. This is a relatively simple chart, here is a little more complex chart:

This is where all your fundamental vision algorithms come into play - pixel interpretation, border, edge detection, etc. Assuming you can do that, then the second half of the problem is to answer questions based on the interpretation of this data. They can get abstract very easily. Here is a sample set of questions in increasing order of complexity for the temperatures chart:

Which city is hotter in May - June?

This is relatively easy if you have solved the reading of the chart

Was this a harsh winter for New York?

The algorithm cannot answer this question without access to external data. It needs to figure out what temperatures NY has been over the past 10, 20 years and then match these values to arrive at an answer

When is the most comfortable time to visit New York?

Now, we are talking about stuff that is not even there on the chart. The algorithm first needs to figure out what comfortable time means in the context of this chart and then figure out an answer.

As you can see, it can get out of hand very quickly.

How to Enhance Data Interpretation Efficiency?

You can do three things to improve your success with chart interpretation problems:

Start with the end

A better way to solve this problem is to approach it from the other end - with the question that you want the computer vision algorithm to answer and model your recognition system around that.

Data & Interpretation Constraints

It also helps tremendously if you know the type of graph that you need to interpret. It is a lot easier to build relevant AI Index and algorithms that have to read weather charts with two cities' information in them rather than a generic time series chart. Try to create a solution with some constraints to improve accuracy.

Pre-Processing of External Data

Once you have figured out #2, you may need to get external data and train your system to have the context around the data it needs to interpret. For instance - basic elaboration or dictionary to understand NY=New York & SF=San Francisco, the knowledge of average highs and lows of temperature across different cities over a period of time, etc.

"Computer vision and machine learning are going to be key drivers in how we develop and implement technology in the future." - Sundar Pichai, CEO, Google.

The good news is that AI is advancing fast and it is just a matter of time before computer vision algorithms will start interpreting complex charts with ease.

‍

About this blog

AI can be a game-changer, but only if you know how to play the game, something that global private investment firms are slowly but steadily getting better at!

If anything, we have all learned the growing importance of Machine Learning and Artificial Intelligence in the past decade.

This blog is a practical guide to turning AI into real business value. For instance, predicting future growth prospects on the basis of the compound annual growth rate of the past few years. Learn how to:

• Make sense of complex documents and images.
• Extract the data you need to drive intelligent process automation.
• Apply AI to gain insights and knowledge from your business documents.

FAQs on Computer Vision

What is the future of computer vision?

The future of computer vision is bright with advancements in deep learning, real-time and edge computing, and 3D vision. These technologies improve object recognition, scene understanding, and image classification. Real-time and edge computing enable faster processing and analysis of visual data. 3D vision enhances depth perception. These advancements have applications in autonomous vehicles, robotics, healthcare, augmented reality, and industrial automation, shaping a more intelligent and intuitive visual world.

How is computer vision used in virtual reality?

Computer vision plays a crucial role in enhancing the virtual reality (VR) experience through various applications. Gesture recognition enables users to interact with the virtual environment using hand gestures, enhancing immersion and user interaction. Environment reconstruction techniques use computer vision algorithms to create realistic virtual environments by capturing and analyzing real-world scenes. Object recognition and tracking enable the virtual environment to interact with real-world objects, allowing users to manipulate and interact with virtual objects in a more natural and intuitive way. These applications in computer vision contribute to a more immersive and realistic VR experience, opening up possibilities for entertainment, training, simulations, and various other domains.

In which industries and domains is computer vision being applied?

Computer vision is being applied to a wide range of industries and domain.
Healthcare: diagnosis and treatment planning
Agriculture: crop monitoring, yield estimation, disease detection
Manufacturing: quality control and process optimization
Retail: inventory management, customer analytics, augmented reality shopping experiences

What are the challenges in using computer vision to read charts?

Some challenges in using computer vision to read charts include:
- Chart complexity: some charts may contain a large number of data points or multiple data series, making it difficult to identify patterns & trends
- Data quality: charts may contain incomplete or inaccurate data, which can affect the accuracy of the analysis
- Chart type: different types of charts may require different analysis techniques & some chart types may be more difficult to analyze than others
- Interpretation: even with accurate analysis, the computer may not be able to interpret the data in the same way a human would & human interpretation may be necessary to fully understand the insights provided by the computer.