Optimizing Web Scraping from 15 Hours to 3 minutes.

Web Scraping is a pool to learn about full-stack development. How an API works? What happens when I hit a URL? How does content on my webpage gets loaded?

FUN FACT:
There’s no way to programmatically determine if a page is being scraped. (Image Generated by Aayush Ostwal)

In a coding interview, while solving a problem, always one follow-up question that the interviewers ask, Can you further optimize the logic? And supposedly, you answered and got the job and BOOOM!! Nobody talks about the optimization of code during product development. And, all you have is a task, you nailed it, and the job is done. Then what is the point of learning all the stuff because in end you are just implementing a no-optimized, brute-forced algorithm?

Well, let me get it straight! If you think what I told you above is true, you are naive and new to the industry. It usually does not happen. Optimization is the key approach to making your product stand in this competitive world. Maybe during Proof of concept, we can implement a simple solution. But when it comes to production, optimizations, and scaling do matter.

Web Scraping is one such domain that, I find, is a pool to learn about full-stack development. How does an API work? What happens when I hit a URL? How does content on my webpage gets loaded? And answers to these questions can take you…