- Web scraping can create tools to improve department or even employee productivity.
A lot has been written about web scraping, mostly focusing on how corporations can use it to generate more revenue and produce better services.
Some use cases have been developed for smaller businesses, which are becoming more popular as automated data collection becomes more accessible.
Web scraping can often be (only partially correctly) seen as something directly tied to revenue. It either improves operational efficiency or creates a product or service.
Little has been written about how web scraping can create tools to improve department or even employee productivity.
Benefits of internal data scraping
It might seem that internal data (i.e., information collected from one’s website) is easily accessible and that there would be no need to use scraping.
At best, fringe cases, such as searching for 404 hyperlinks or anchor text are mentioned. Even then, SEO tools can often perform such tasks, making internal scrapers an undertaking not worth undergoing.
Internal scraping, however, does have the benefit of being unlikely to trigger any issues that are usually associated with external data. After all, it’s your website, so there’s no need to worry about copyright infringement or producing a negative user experience unknowingly.
Additionally, there’s no need to work around anti-bot solutions or wonky website structures.
So, such data collection has none of the drawbacks usually associated with web scraping, reducing the overhead required to initiate such tasks.
Data for content management
Writing is something all businesses have to do nowadays. Landing pages and blog posts drive organic traffic, especially with the help of SEO activities.
There’s often a call to create “good content”. Although no one seems to quite grasp what makes a piece of writing good, most of us seem to understand what it is once we see it.
Getting there, however, is tough. Writing is an ephemeral skill that’s hard to pass down as there are fairly few hard and fast rules. As anyone’s experience might dictate – grammar and syntax aren’t enough for good writing.
Additionally, copywriters will often have wildly different weak points. Some may have smaller vocabularies, resulting in less eloquent pieces of content.
Others may use parasite sentences or words that impart no value to the reader. Building a one-size-fits-all training programme is significantly harder than in some other areas of expertise.
Internal web scraping, however, can unveil potential areas for improvement. There are some prerequisites:
- Articles, blog posts, and landing pages should have a known author assigned to them. Such data has to be managed properly to ensure that authors always match the content they produce.
- There has to be a significant amount of content already published to generate a large enough dataset. A dozen, at the least, would be a good starting point.
- Writing has to be somewhat consistent in topics and quality.
Building plans for improvement
We need the above prerequisites to create an author-based dataset, which can be constantly updated whenever new content appears.
Once such preparations are in place, data analysis can begin, and improvement plans can be drafted.
A common pitfall of many writers is the overuse of certain idioms or words. While not a major issue, it can ruin the flow of text and stifle more creative approaches to writing.
With internal scraping, in-depth statistics on the overall vocabulary and frequency of use can be collected.
Prepositions, pronouns, conjunctions, and other lexical categories should be removed outright to give a better overview. Such a dataset, however, shows how wide a writer’s vocabulary is and if they opt for repetitive use of words, leading to clear avenues for improvement.
Additionally, sentence and paragraph length can be analysed. There seems to be a trend and expectation that both should be short, especially for online publications.
Little hard data on that subject exists. Internal scraping provides us with a window into the potential truthfulness of such statements.
In isolation, these datasets can prove to be useful tools for writer self-improvement. In combination, however, they can be used to analyse what works from a business perspective.
Some writers will have better performance for reading times, scroll depth, etc., all of which are directly tied to the quality of the work.
Such data won’t be visible through internal scraping itself, however. But popular tracking tools such as Google Analytics give us enough data to enrich writer datasets to make performance analysis easier.
It’s important to note, however, that the data points from Google Analytics should be selected carefully. Not all metrics are a testament to the writer’s skill.
Views, a seemingly intuitive metric, are far detached from the quality of the work.
Without internal scraping, finding out why certain writers build better pieces of content would be tough.
Additionally, it may be easier to be led astray as the metrics the business is concerned with (views, conversions, etc.) don’t always reflect the quality of the writing. It may reflect the quality of SEO research or a multitude of other factors.
Conclusion
Scraping is uniquely beneficial because its main product is the creation of data. While it has been mostly associated with improving business performance, it can be used in so many ways that focusing on that side of the equation limits scraping’s true potential.
Building an internal database to be used for the improvement of copywriting is just one such unusual use of scraping. In general, it can be used to customise data-driven practices and help build up teams where one-size-fits-all training might be more difficult to produce.
- Aleksandras Šulženko is the Product Owner at Oxylabs.io, a company specialising in web data gathering.