In today’s world, everything changes all the time and the truth is very subjective. In the Market Intelligence profession, one would use VUCA to describe such a word. It means volatile, uncertain, complex and ambiguous. We should strive to decrease VUCA to make better decisions. In today’s post, I’ll speak about one activity that can help - how to start monitoring competitors online.
Market Intelligence Professionals spend hours on routine tasks, and monitoring competitors’ websites are no exclusion. On average, companies track from 5 to 7 competitors on an ongoing basis. Collecting stats for each of them from various sources sounds like a lot of work. So let’s make out of it in four simple steps.
Step 1: Planning
As with everything in business, it’s important to gather our thoughts first and understand what we want to do.
Who do we want to monitor?
This is easy to answer at first. Of course, our competitors. However, this question requires a bit more thinking. Our competitors usually have mothers and daughters, subsidiaries, joint ventures, etc. It’s important to map our competitors and understand who is who first (if you haven’t already done so).
What what do we want from Competitor monitoring?
Most probably, one or both of the following things: (1) extract some kind of data and (2) get notified on significant changes.
Data: We should plan what kind of values we want to monitor. Do we want to monitor data represented by a number (e.g. the team size), money-value (price), text (positioning statement), date (latest blog post update) or do we want to monitor an array of data (all partners, all blog posts, all job vacancies)
Notifications: Not all changes are equal. If your big competitor opens one position or not, that’s probably not relevant. However, if they suddenly start looking for 100 new employees your project country, or they publish an annual report, add a new business partner or increase their prices by 30 %, you would probably like to know.
Step 2: Research
It’s time to start digging a bit deeper. Let’s find websites and social media accounts of your competitors.
How to find competitors’ websites?
A good start is to analyse the domain via a service such as Phonebook.cz
Once you’ve gathered a list of interesting websites, it’s time to move to the next question.
How to find competitors’ social media accounts?
Most companies will have their social media links right on their website, not just once, but on as many internal pages as they can put them. There is also online software that will give you information about off-site links incoming to a website that should give you what you’re looking for but it’s faster to go through your list and do it by hand.
In addition, it is possible to search Google using word strings that you devise to pull up the social media sites of a business, such as “midesk.co (inurl:“linkedin.com/company” OR inurl:“twitter.com/” OR inurl:“https://www.youtube.com/channel”)”)
What exactly to monitor?
Once you’ve built a basic understanding of which competitors are where and what you could want to monitor, it’s time to start analysing what you’ve gathered until this moment and standardise it. I will demonstrate this standardisation in the following two examples (plus one advanced bonus example):
Example 1: Website research
Let’s use the example of our homepage (https://midesk.co).
Our website’s most prominent element is the headline “Effortlessly Track Competitors & Periodically Scrape Their Data” which is the most important thing new visitors see and makes 80 cents on the dollar. We also display a positioning statement “Midesk is a Market intelligence platform that allows you to automate your Competitor monitoring and reporting starting at € 59 / month.” Next, we showcase our business partners and list the major benefits of our platform. Let’s call these variables “data types”.
Now it’s time to click through the website and identify other “data types”. You will most definitely find some of the following:
- prices
- departments,
- team members,
- stock price information,
- documents like annual reports,
- offices,
- blog posts,
- case studies,
- job openings,
- press releases
- and so on.
It’s important to realise that Data Types don’t necessarily have to be what you see. You can quantify it. For example, a competitor may list all of their job openings and you just want to calculate a number of job openings not download all of the titles and descriptions. Or data types can be also “boolean”. For example, whether or not your competitor has published a blog post about “your topic” in the last 30 days.
Example 2: Social Media
You can do the same for social media. Go through your list and think about what Data Types you would like to monitor. Is it the number of followers, latest tweets, total watch time on Youtube or number of LinkedIn profiles that belong to an organisation? Go through your list of social media accounts and define your Data Types.
Bonus example: Website Advanced research
Websites offer so many hidden “gold nuggets” as one of our clients likes to say. Feel free to skip this step, as it requires a bit of technical knowledge.
Most websites use JavaScript and fetch data from the backend not to have to reload the whole page. It is possible to intercept this data. Let’s have a look at the example of Emmy Scooters, a German e-mobility company. Let’s review the following website: Emmy Scooters
At a first glance, once you open the link, you will see a map. But what you don’t see is that the website offers all of their scooter data publicly. Right-click on the page, click on “Inspect” (in Chrome), open the tab “Network” and filter for “Fetch/XHR”. You’ll discover that one of the resources includes data on all of their 2607 scooters in Germany. Each scooters item includes its city, exact location (Address, ZIP code), fuel level, price, vehicle type, etc. It’s only up to you how far you want to go with it. You can continuously provide a lot of insight about the company if you monitor this particular background process.
There is so much more that’s possible with the Inspector, and we encourage you to review your competitors’ website having the panel open. Who knows what “gold nuggets” you may find.
Step 3: Traditional Implementation
So you have a list of companies, target websites and social media, Data Types and now what?
It’s time to build an excel file. A sheet is your competitor’s name, columns are dates when you check and rows are your data types. Now you need to “only” check periodically and you’ve successfully created simple Competitor monitoring. It’s a good practice to prepare a “standard deliverable”, such as an alert template you can email to your stakeholders on significant changes.
Step 4: Automatic implementation
Of course, you can do it by hand yourself. But you can also use an online service (like Midesk) to automate it all. In principle our system connects four cornerstones of such Market Intelligence work.
1. Data Repositories
Before we can start monitoring something, it is important (as we pointed out in this blog post) to plan ahead. In our system, a key ingredient is what we call “Data repositories”. They are special databases where data is saved and from where you can further work with in the a structured (+ Market Intelligence) way.
Key features:
- Data storage
- Currency management
- Source management
- Calculation management
- Period management
- Export to Excel
2. Monitoring Jobs
Monitoring jobs take care of managing the monitoring work. They take care of the scheduling, proxy management, order and failure. They are the manager. Starting a monitoring job is as simple as clicking on a button “Add job”.
Key features:
- Schedule
- Manage Proxy
- Extract Data
- Run Tasks
- Manage failure
3. Monitoring Tasks
Monitoring tasks are the bread and butter of the extraction. They consist of logic that’s carried out every single time a monitoring job is run. They tell what data are extracted, from where, how are they transformed and to which data repositories the data gets saved. They also take care of notifications.
Key features:
- Extract data
- One or multiple datapoint
- Methods: json, xPath, regex
- Automatic recognition of value
- Transform data (e.g. automatic multiplication)
- Save data to data repositories
- Create notifications
4. Notifications
Notifications are the last part of the process. It’s important to be notified on significant changes. Difference to previous found results is also very important. That’s where our system provides nice numeric, textual and structural difference reports.
Key features:
- Identify significance
- Identify differences to previous result (number, text, lists)
- Manage email distribution to multiple recipients
- Create sharable & commendable article
