Welcome to the world of web scraping, where data meets technology in the vast expanse of the internet. Web scraping, simply put, is the art of extracting information from websites automatically. Imagine being able to sift through countless web pages, collecting valuable data points with the precision of a digital archaeologist. Whether you're a researcher seeking insights, a business analyst tracking market trends, or an enthusiast exploring the digital landscape, web scraping opens doors to a wealth of information waiting to be discovered. Join us on this journey as we delve into the fundamentals, techniques, and ethical considerations of web scraping, unlocking the power of data at your fingertips.
I have embarked on the web scraping project using one of the most Python libraries BeautifulSoup which is a versatile and powerful tool for web scraping. It has intuitive interface and powerful features, BeautifulSoup simplifies the process of parsing HTML and XML documents, making it an ideal choice as it extracts data effortlessly and navigates through web pages with ease.
For my for my project, I decided to do web scraping for a books site which had 50 pages of information pertaining to books, authors, ratings, pricing and much more and parse the data into a .csv file for further analysis. Here is the roadmap I followed.
Started with importing the libraries:
Pandas is a powerful data manipulation and analysis library in Python. While it's not specifically designed for web scraping, Pandas is often used to process and analyze data extracted from websites.
Requests is a Python library used for making HTTP requests. It allows you to fetch web pages from URLs, which is an essential step in web scraping. While you can use other libraries for making HTTP requests, Requests is popular due to its simplicity and ease of use.
Pandas and Requests are libraries that are frequently used in conjunction with web scraping tools to handle different aspects of the scraping process, such as fetching web pages and processing the extracted data.
Followed by importing BeautifulSoup which a Python library designed for quick and easy parsing of HTML and XLM documents. It provides tools for navigating, searching and modifying the parse tree making extracting specific data from web pages easy.
bs4 package from where BeautifulSoup class is imported, allowing the use of Python code to parse HTML and XML documents for web scraping.
Moving to the code that fetched the HTML content from the web pages and extracted the list of books by initializing a empty list to store book data
followed by a loop that iterates over the range of 1 to 50 pages(inclusive) by executing the for loop for each value of 'i' looping through the URL using a f-string to dynamically generate the URL based value for 'i'.
response sends request to the URL and assigns the response to the variable 'response', using the function 'requests.get(url)' to fetch the content of the webpage.
Further creating a BeautifulSoup object names 'soup', which represents the HTML content. Here BeautifulSoup() is used to parse the HTML content using 'html.parser'. BeautifulSoup's .find() method was used to find the HTML elements and assign it to variables like <ol> and <articles>
For article starts the loop iterating over each 'article' element in the 'articles' list that is present on the webpage. The loop moves through collecting data like title, image, rating, pricing and finally appending the data into columns.
In summary the code loops over 50 pages of the website, extracts book data and stores it in the 'books' list.
After the loop is completely executed, the 'books' list is stored into pandas DataFrame where each row of the DataFrame represents a single book entry with its corresponding information.
The DataFrame then saves the data to .csv file using the given path to get the following result
I hope you all enjoyed my journey into unveiling the digital goldmine! please feel free to give feed back and comments as that will help me grow to become a better analyst.
credits: @TheLinguists
Comentários