The last twenty years have seen an exponential rise in price comparison sites offering a plethora of different services. From holiday and flight companies to specialist finance and credit card services, there’s almost no limit to the number of goods and services you can find and compare online these days.
While most price comparison sites have their own internal processes and deals agreed with their preferred suppliers, almost all use one of the methods outlined below (or, sometimes, a combination of them) to scour the web and find the best deals for their prospective clients.
Data models used by the biggest price comparison services
Experts suggest data is now the world’s most valuable commodity – worth even more than traditional heavyweights like gold and oil – and with price comparison sites, it’s easy to see why. The collation of data from disparate sources drives these platforms and, in turn, allows the comparison platform to offer their services and turn a profit.
Whether you’re looking to change your firm’s energy provider on a site like businesscomparison.com or are searching to find the best deal on your next foreign trip, chances are good that the service you use will employ one of the models listed below to grab you the best, most up-to-date prices:
Direct merchant feeds: It’s very common for price comparison sites to partner with preferred suppliers and take feeds directly from them to populate their search results. This method of finding results typically sees the comparison site using the service company’s Application Programming Interface (API) or sometimes FTP access to generate results. Often, the service provider charges the price comparison platform for this service. Also, there is a major drawback in this model for the comparison site – namely, that the supply firms all tend to store their data in different formats, which can make the collation of results very tricky.
Third-party API feeds: Rather than working with direct feeds, many of the comparison sites leave the headache of collecting pricing data to third-party firms. This makes data collation work considerably easier for the comparison site as the problems with data formatting are handled by the third-party provider. In addition, going this route provides the fastest development pathway since the data supplier does the majority of the work – although they can charge a substantial fee for the service, which, somewhat obviously, will cut into the comparison site’s overall profit margins.
Web scraping: For smaller comparison providers (or those just starting out), it often isn’t practical or financially viable to use the two data collection methods outlined above. Often larger supply companies won’t consider it worth their while to hand their direct feeds to start-ups, while the costs involved in both methods frequently prove to be prohibitively expensive for smaller comparison companies. In this instance, a comparison firm will typically use another form of data collection called web scraping. Web scraping requires automated bots to make regular visits to supplier sites and automatically gather results from their dataset. The programming for these bots is usually done in-house by the comparison site – although it is also quite common to use a specialist Data-as-a-Service (DaaS) provider.