New Monitoring Recipes, JSON Scraping & Article Downloader

Jakub
4 min read

We rebuilt Web & API monitoring from the ground up and added three super-powerful features we’ve been asked about for a long time. Introducing Monitoring Recipes, list-based scraping and something we call “Forward Download” for article scraping.

Apply Monitoring Recipes to kick-start Market Intelligence Monitoring

We’ve also heard your feedback that setting up monitoring jobs was sometimes difficult. Truth to speak, it can be a lengthy process even for us. Some of you probably remember a demo where we had to take a homework and come back to you with a working solution after the meeting. After all, we’re all in the same boat — using industry-standards which can be complicated. Unfortunately, the problem lays in the diversity of the web rather than on the extraction techniques themselves.

Now imagine a World where we could skip the part where you’re fiddling around with xPath and just click a button and all is ready for you. This is what a Monitoring Recipe brings to the table.

Let’s apply a Monitoring Recipe

Monitoring recipes are blueprints and user interfaces that set up everything for your with a couple of clicks. Let me illustrate it on the example of the monitoring and extracting data from ifo Institute for Economic Research, one of the leading economic research institutes in Europe.

The website of ifo Institute for Economic Research

What market data are we after?

The ifo Institute analyses economic policy and is widely known for its monthly Ifo Business Climate Index for Germany index. Other important indexes include the ifo Export Expectations, ifo Employment Barometer and locally-known ifo Business Climate in Eastern Germany and ifo Business Climate Index for Saxony.

ifo’s data are important but time-consuming to monitor and extract

The institute publishes their data monthly in a textual way. The website doesn’t offer a comprehensive view on past data. It also publishes the data with a loose precision of ± a couple of days which makes companies that utilise the index visit the website frequently to know the predictions as soon as possible. But waiting for the results and its extraction can be digitised.

Solution: Let’s use a Midesk Recipe to monitor the website, create notifications and store data to Repositories.

Once you open the user interface, you will be able to select you recipe. Click on a recipe called ifo Institute and it’s ready. Midesk will scaffold a Monitoring Job, create a new Data Repository, prepare related Data Types and set up basic in-system notifications.

Selecting recipes

Dynamic scraping with based on lists makes monitoring much easier to maintain and uncorks a mass-monitoring capability

We’ve learnt that you usually want to scale your monitoring beyond a handful of demo examples we showcase during our demo-tour. You usually have lists of data you on which want to replicate the functionality. So we’ve added a possibility add variables to the URL and to your Monitoring Tasks.

Add variables to your URL ${ keywords } and specify keywords to dynamically monitor websites

You can enable keywords by clicking on “Enable Keywords” button and placing a ${ keywords } placeholder in your URL like so: https://midesk.co/${ keywords }. Midesk will now run your Monitoring Job for every keyword you specify. Imagination doesn’t have limits. You can for example extract all titles from all subpages on a site, get search results for a list of keywords or just switch language and download your content automatically in all available languages.

“Forward Download”: Download News Articles from identified URLs

Lately multiple clients have asked us if it were possible to download content of scraped results. The use cases have been to download new blog posts and search results that have a link to a different page.

It’s natural that you would like to consequently download such content or at least make the system handle the content accordingly. So we’ve added this feature which you can enable in your Monitoring Tasks as shown on the image. If you are subscribed to our premium package, you can use this feature.

Download & categorize scraped content as news articles automatically

Midesk will then download new article (preventing duplicates) into your News section. You can work with the new content as you would work with other news articles - you can attach it to emails, share it directly, comment it and search and embed it in your Report builder (PowerPoints presentations).

See our solution in action

Let's discuss your particular Market & Competitive Intelligence needs and see how Midesk can address them.

© 2019 - 2023 Midesk UG (haftungsbeschränkt)