Setting up the Environment

Before diving into web scraping with Python, set up your environment by installing the necessary libraries.

First, install the following libraries: requests, BeautifulSoup, and pandas. These packages play a crucial role in web scraping, each serving different purposes.

To install these libraries, click on the previously provided links for a full guide (including troubleshooting) or simply run the following commands:

pip install requests
pip install beautifulsoup4
pip install pandas

The requests library will be used to make HTTP requests to websites and download the HTML content. It simplifies the process of fetching web content in Python.

BeautifulSoup is a fantastic library that helps extract data from the HTML content fetched from websites. It makes navigating, searching, and modifying HTML easy, making web scraping straightforward and convenient.

Pandas will be helpful in data manipulation and organizing the scraped data into a CSV file. It provides powerful tools for working with structured data, making it popular among data scientists and web scraping enthusiasts.

Fetching and Parsing URL

Next, you’ll learn how to fetch and parse URLs using Python to scrape data and save it as a CSV file. We will cover sending HTTP requests, handling errors, and utilizing libraries to make the process efficient and smooth.

Sending HTTP Requests

When fetching content from a URL, Python offers a powerful library known as the requests library. It allows users to send HTTP requests, such as GET or POST, to a specific URL, obtain a response, and parse it for information.

We will use the requests library to help us fetch data from our desired URL.

For example:

import requests

response = requests.get('https://example.com/data.csv')

The variable response will store the server’s response, including the data we want to scrape. From here, we can access the content using response.content, which will return the raw data in bytes format.

Handling HTTP Errors

Handling HTTP errors while fetching data from URLs ensures a smooth experience and prevents unexpected issues. The requests library makes error handling easy by providing methods to check whether the request was successful.

Here’s a simple example:

import requests

response = requests.get('https://example.com/data.csv')

response.raise_for_status()

The raise_for_status() method will raise an exception if there’s an HTTP error, such as a 404 Not Found or 500 Internal Server Error. This helps us ensure that our script doesn’t continue to process erroneous data, allowing us to gracefully handle any issues that may arise.

With these tools, you are now better equipped to fetch and parse URLs using Python. This will enable you to effectively scrape data and save it as a CSV file.

Extracting Data from HTML

In this section, we’ll discuss extracting data from HTML using Python. The focus will be on utilizing the BeautifulSoup library, locating elements by their tags, and attributes.

Using BeautifulSoup

BeautifulSoup is a popular Python library that simplifies web scraping tasks by making it easy to parse and navigate through HTML. To get started, import the library and request the page content you want to scrape, then create a BeautifulSoup object to parse the data:

from bs4 import BeautifulSoup

import requests url = "example_website"

response = requests.get(url)

soup = BeautifulSoup(response.text, "html.parser")

Now you have a BeautifulSoup object and can start extracting data from the HTML.

Locating Elements by Tags and Attributes

BeautifulSoup provides various methods to locate elements by their tags and attributes. Some common methods include find(), find_all(), select(), and select_one().

Let’s see these methods in action:

# Find the first <span> tag

span_tag = soup.find("span") # Find all <span> tags

all_span_tags = soup.find_all("span") # Locate elements using CSS selectors

title = soup.select_one("title") # Find all <a> tags with the "href" attribute

links = soup.find_all("a", {"href": True})

These methods allow you to easily navigate and extract data from an HTML structure.

Once you have located the HTML elements containing the needed data, you can extract the text and attributes.

Here’s how:

# Extract text from a tag

text = span_tag.text # Extract an attribute value

url = links[0]["href"]

Finally, to save the extracted data into a CSV file, you can use Python’s built-in csv module.

import csv # Writing extracted data to a CSV file

with open("output.csv", "w", newline="") as csvfile: writer = csv.writer(csvfile) writer.writerow(["Index", "Title"]) for index, link in enumerate(links, start=1): writer.writerow([index, link.text])

Following these steps, you can successfully extract data from HTML using Python and BeautifulSoup, and save it as a CSV file.

Recommended: Basketball Statistics – Page Scraping Using Python and BeautifulSoup

Organizing Data

This section explains how to create a dictionary to store the scraped data and how to write the organized data into a CSV file.

Creating a Dictionary

Begin by defining an empty dictionary that will store the extracted data elements.

In this case, the focus is on quotes, authors, and any associated tags. Each extracted element should have its key, and the value should be a list that contains individual instances of that element.

For example:

data = { "quotes": [], "authors": [], "tags": []

}

As you scrape the data, append each item to its respective list. This approach makes the information easy to index and retrieve when needed.

Working with DataFrames and Pandas

Once the data is stored in a dictionary, it’s time to convert it into a dataframe. Using the Pandas library, it’s easy to transform the dictionary into a dataframe where the keys become the column names and the respective lists become the rows.

Simply use the following command:

import pandas as pd df = pd.DataFrame(data)

Exporting Data to a CSV File

With the dataframe prepared, it’s time to write it to a CSV file. Thankfully, Pandas comes to the rescue once again. Using the dataframe’s built-in .to_csv() method, it’s possible to create a CSV file from the dataframe, like this:

df.to_csv('scraped_data.csv', index=False)

This command will generate a CSV file called 'scraped_data.csv' containing the organized data with columns for quotes, authors, and tags. The index=False parameter ensures that the dataframe’s index isn’t added as an additional column.

Recommended: 17 Ways to Read a CSV File to a Pandas DataFrame

And there you have it—a neat, organized CSV file containing your scraped data!

Handling Pagination

This section will discuss how to handle pagination while scraping data from multiple URLs using Python to save the extracted content in a CSV format. It is essential to manage pagination effectively because most websites display their content across several pages.

Looping Through Web Pages

Looping through web pages requires the developer to identify a pattern in the URLs, which can assist in iterating over them seamlessly. Typically, this pattern would include the page number as a variable, making it easy to adjust during the scraping process.

Once the pattern is identified, you can use a for loop to iterate over a range of page numbers. For each iteration, update the URL with the page number and then proceed with the scraping process. This method allows you to extract data from multiple pages systematically.

For instance, let’s consider that the base URL for every page is "https://www.example.com/listing?page=", where the page number is appended to the end.

Here is a Python example that demonstrates handling pagination when working with such URLs:

import requests

from bs4 import BeautifulSoup

import csv base_url = "https://www.example.com/listing?page=" with open("scraped_data.csv", "w", newline="") as csvfile: csv_writer = csv.writer(csvfile) csv_writer.writerow(["Data_Title", "Data_Content"]) # Header row for page_number in range(1, 6): # Loop through page numbers 1 to 5 url = base_url + str(page_number) response = requests.get(url) soup = BeautifulSoup(response.text, "html.parser") # TODO: Add scraping logic here and write the content to CSV file. 

In this example, the script iterates through the first five pages of the website and writes the scraped content to a CSV file. Note that you will need to implement the actual scraping logic (e.g., extracting the desired content using Beautiful Soup) based on the website’s structure.

Handling pagination with Python allows you to collect more comprehensive data sets, improving the overall success of your web scraping efforts. Make sure to respect the website’s robots.txt rules and rate limits to ensure responsible data collection.

Exporting Data to CSV

You can export web scraping data to a CSV file in Python using the Python CSV module and the Pandas to_csv function. Both approaches are widely used and efficiently handle large amounts of data.

Python CSV Module

The Python CSV module is a built-in library that offers functionalities to read from and write to CSV files. It is simple and easy to use. To begin with, first, import the csv module.

import csv

To write the scraped data to a CSV file, open the file in write mode ('w') with a specified file name, create a CSV writer object, and write the data using the writerow() or writerows() methods as required.

with open('data.csv', 'w', newline='') as file: writer = csv.writer(file) writer.writerow(["header1", "header2", "header3"]) writer.writerows(scraped_data)

In this example, the header row is written first, followed by the rows of data obtained through web scraping.

Using Pandas to_csv()

Another alternative is the powerful library Pandas, often used in data manipulation and analysis. To use it, start by importing the Pandas library.

import pandas as pd

Pandas offers the to_csv() method, which can be applied to a DataFrame. If you have web-scraped data and stored it in a DataFrame, you can easily export it to a CSV file with the to_csv() method, as shown below:

dataframe.to_csv('data.csv', index=False)

In this example, the index parameter is set to False to exclude the DataFrame index from the CSV file.

The Pandas library also provides options for handling missing values, date formatting, and customizing separators and delimiters, making it a versatile choice for data export.

10 Minutes to Pandas in 5 Minutes

If you’re just getting started with Pandas, I’d recommend you check out our free blog guide (it’s only 5 minutes!):

Recommended: 5 Minutes to Pandas — A Simple Helpful Guide to the Most Important Pandas Concepts (+ Cheat Sheet)

Are you tired of feeling boxed in by your Python plots and ready to break free from the constraints of traditional legend placement?

In this guide, I’ll show you how to put legends outside your plot for (click to jump):

• Matplotlib
• Seaborn
• Pandas
• Bokeh

Let’s start with the first!

Matplotlib Put Legend Outside Plot

Let’s start with various ways to position the legend outside for better visualization and presentation.

Matplotlib Set Legend Outside Plot (General)

First, let’s adjust the legend’s position outside the plot in general. To do this, use the bbox_to_anchor parameter in the legend function like this: matplotlib.pyplot.legend(bbox_to_anchor=(x, y)) . Here, adjust the values of x and y to control the legend’s position.

import matplotlib.pyplot as plt # Sample data for plotting

x = [1, 2, 3, 4, 5]

y1 = [1, 4, 9, 16, 25]

y2 = [1, 8, 27, 64, 125] # Create the plot

plt.plot(x, y1, label='y = x^2')

plt.plot(x, y2, label='y = x^3') # Set the legend's position outside the plot using bbox_to_anchor

plt.legend(bbox_to_anchor=(1.05, 1)) # Add axis labels and title

plt.xlabel('x')

plt.ylabel('y')

plt.title('Plotting with External Legend') # Display the plot

plt.show()

In this example, the bbox_to_anchor parameter is set to (1.05, 1), which moves the legend slightly to the right of the plot.

Info: The bbox_to_anchor parameter in matplotlib.pyplot.legend() uses a tuple of two values, x and y, to control the position of the legend. x=0/1 controls the left/right and y=0/1 controls the bottom/top legend placement.

Generally, in axes coordinates:

• (0, 0) represents the left-bottom corner of the axes.
• (0, 1) represents the left-top corner of the axes.
• (1, 0) represents the right-bottom corner of the axes.
• (1, 1) represents the right-top corner of the axes.

However, the values of x and y are not limited to the range [0, 1]. You can use values outside of this range to place the legend beyond the axes’ boundaries.

For example:

• (1.05, 1) places the legend slightly to the right of the top-right corner of the axes.
• (0, 1.1) places the legend slightly above the top-left corner of the axes.

Using negative values is also allowed. For example:

• (-0.3, 0) places the legend to the left of the bottom-left corner of the axes.
• (1, -0.2) places the legend below the bottom-right corner of the axes.

The range of x and y depends on the desired position of the legend relative to the plot. By adjusting these values, you can fine-tune the legend’s position to create the perfect visualization.

Matplotlib Set Legend Below or Above Plot

To place the legend below the plot, you can set the loc parameter as ‘upper center’ and use bbox_to_anchor like this: plt.legend(loc='upper center', bbox_to_anchor=(0.5, -0.1)). For placing the legend above the plot, use bbox_to_anchor=(0.5, 1.1) instead .

Matplotlib Set Legend Left of Plot (Upper, Center, Lower Left)

For positioning the legend to the left of the plot, use the following examples:

• Upper left: plt.legend(loc='center left', bbox_to_anchor=(-0.2, 0.5))
• Center left: plt.legend(loc='center left', bbox_to_anchor=(-0.1, 0.5))
• Lower left: plt.legend(loc='lower left', bbox_to_anchor=(-0.2, 0))

Matplotlib Set Legend Right of Plot (Upper, Center, Lower Right)

To position the legend to the right of the plot, you can try the following:

• Upper right: plt.legend(loc='upper right', bbox_to_anchor=(1.1, 1))
• Center right: plt.legend(loc='center right', bbox_to_anchor=(1.1, 0.5))
• Lower right: plt.legend(loc='lower right', bbox_to_anchor=(1.1, 0))

Matplotlib Set Subplots Legend Outside Plot

When working with subplots, you can place a single, unified legend outside the plot by iterating over the axes and using the legend() method on the last axis (source) . Remember to use the bbox_to_anchor parameter to control the legend’s position.

Here’s an example that does two things, i.e., (1) placing the legend on the right and (2) adjusting the layout to accommodate the external legend:

import numpy as np

import matplotlib.pyplot as plt # Sample data for plotting

x = np.linspace(0, 10, 100)

y1 = np.sin(x)

y2 = np.cos(x) # Create subplots

fig, axes = plt.subplots(nrows=2, ncols=1, sharex=True) # Plot data on the first subplot

axes[0].plot(x, y1, label='sin(x)')

axes[0].set_title('Sine Wave') # Plot data on the second subplot

axes[1].plot(x, y2, label='cos(x)', color='orange')

axes[1].set_title('Cosine Wave') # Set a common x-label for both subplots

fig.text(0.5, 0.04, 'x', ha='center') # Set y-labels for individual subplots

axes[0].set_ylabel('sin(x)')

axes[1].set_ylabel('cos(x)') # Create a unified legend for both subplots

handles, labels = axes[-1].get_legend_handles_labels()

for ax in axes[:-1]: h, l = ax.get_legend_handles_labels() handles += h labels += l # Place the unified legend outside the plot using bbox_to_anchor

fig.legend(handles, labels, loc='upper right', bbox_to_anchor=(1, 0.75)) # Adjust the layout to accommodate the external legend

fig.subplots_adjust(right=0.7) # Display the subplots

plt.show()

Legend Outside Plot Is Cut Off

If your legend is cut off, you can adjust the saved figure’s dimensions using plt.savefig('filename.ext', bbox_inches='tight'). The bbox_inches parameter with the value ‘tight’ will ensure that the whole legend is visible on the saved figure .

Add Legend Outside a Scatter Plot

For a scatter plot, you can use the same approach as mentioned earlier by adding the loc and bbox_to_anchor parameters to position the legend outside the plot. For instance, plt.legend(loc='upper right', bbox_to_anchor=(1.1, 1)) will place the legend in the upper right corner outside the scatter plot .

If your legend is cut off when placing it outside the plot in Python’s Matplotlib, you can adjust the layout and save or display the entire figure, including the external legend, by following these steps:

1. Use the bbox_to_anchor parameter in the legend() function to control the position of the legend.
2. Adjust the layout of the figure using the subplots_adjust() or tight_layout() function to make room for the legend.
3. Save or display the entire figure, including the external legend.

Here’s an example demonstrating these steps:

import matplotlib.pyplot as plt # Sample data for plotting

x = [1, 2, 3, 4, 5]

y1 = [1, 4, 9, 16, 25]

y2 = [1, 8, 27, 64, 125] # Create the plot

plt.plot(x, y1, label='y = x^2')

plt.plot(x, y2, label='y = x^3') # Set the legend's position outside the plot using bbox_to_anchor

plt.legend(bbox_to_anchor=(1.05, 1), loc='upper left') # Add axis labels and title

plt.xlabel('x')

plt.ylabel('y')

plt.title('Plotting with External Legend') # Adjust the layout to accommodate the external legend

plt.subplots_adjust(right=0.7) # Display the plot

plt.show()

In this example, you use the subplots_adjust() function to adjust the layout of the figure and make room for the legend.

You can also use the tight_layout() function, which automatically adjusts the layout based on the elements in the figure:

# ...

# Set the legend's position outside the plot using bbox_to_anchor

plt.legend(bbox_to_anchor=(1.05, 1), loc='upper left') # Add axis labels and title

plt.xlabel('x')

plt.ylabel('y')

plt.title('Plotting with External Legend') # Automatically adjust the layout to accommodate the external legend

plt.tight_layout(rect=[0, 0, 0.7, 1]) # Display the plot

plt.show()

In this case, the rect parameter is a list [left, bottom, right, top], which specifies the normalized figure coordinates of the new bounding box for the subplots. Adjust the values in the rect parameter as needed to ensure the legend is not cut off.

Additional Legend Configurations

In this section, you’ll learn about some additional ways to customize the legend for your plots in Python using Matplotlib. This will help you create more meaningful and visually appealing visualizations. However, feel free to skip ahead to the following sections on plotting the legend outside the figure in Seaborn, Pandas, and Bokeh.

Python Set Legend Position

As you already know by now, you can place the legend at a specific position in the plot by using the bbox_to_anchor parameter. For example, you could place the legend outside the plot to the right by passing (1.05, 0.5) as the argument:

import matplotlib.pyplot as plt

plt.legend(bbox_to_anchor=(1.05, 0.5))

This will place the legend slightly outside the right border of the plot, with its vertical center aligned with the plot center.

Python Set Legend Location

You can easily change the location of the legend by using the loc parameter. Matplotlib allows you to use predefined locations like 'upper right', 'lower left', etc., or use a specific coordinate by passing a tuple:

plt.legend(loc='upper left')

This will place the legend in the upper-left corner of the plot.

Python Set Legend Font Size

To change the font size of the legend, you can use the fontsize parameter. You can pass a numeric value or a string like 'small', 'medium', or 'large' to set the font size:

plt.legend(fontsize='large')

This will increase the font size of the legend text.

Python Set Legend Title

If you want to add a title to your legend, you can use the title parameter. Just pass a string as the argument to set the title:

plt.legend(title='My Legend Title')

This will add the title “My Legend Title” above your legend.

Python Set Legend Labels

If you’d like to customize the legend labels, you can pass the labels parameter. It takes a list of strings as the argument:

plt.legend(labels=['Label 1', 'Label 2'])

Your legend will now display the custom labels “Label 1” and “Label 2” for the corresponding plot elements.

Python Set Legend Color

Changing the color of the legend text and lines can be achieved by using the labelcolor parameter. Just pass a color string or a list of colors:

plt.legend(labelcolor='red')

This will change the color of the legend text and lines to red.

Seaborn Put Legend Outside Plot

In this section, I’ll show you how to move the legend outside the plot using Seaborn. Let’s dive into various ways of setting the legend position, like below, above, left or right of the plot.

Sns Set Legend Outside Plot (General)

First, let’s talk about the general approach:

You can use the legend() function and the bbox_to_anchor parameter from Matplotlib to move the legend. You can combine this with the loc parameter to fine-tune the legend’s position.

Here’s a quick example:

import seaborn as sns

import matplotlib.pyplot as plt # Sample data for plotting

tips = sns.load_dataset("tips") # Create a Seaborn plot with axis variable 'ax'

fig, ax = plt.subplots()

sns_plot = sns.scatterplot(x="total_bill", y="tip", hue="day", data=tips, ax=ax) # Set the legend's position outside the plot using bbox_to_anchor on 'ax'

ax.legend(bbox_to_anchor=(1.05, 1), loc='upper left', borderaxespad=0.) # Add axis labels and title

ax.set_xlabel('Total Bill')

ax.set_ylabel('Tip')

ax.set_title('Tips by Total Bill and Day') # Display the plot

plt.show()

Sns Set Legend Below or Above Plot

Now let’s move the legend below or above the plot. Simply adjust the bbox_to_anchor parameter accordingly. For example, to place the legend below the plot, you can use bbox_to_anchor=(0.5, -0.1):

ax.legend(loc='upper center', bbox_to_anchor=(0.5, -0.1))

And to place the legend above the plot, use bbox_to_anchor=(0.5, 1.1):

ax.legend(loc='lower center', bbox_to_anchor=(0.5, 1.1))

Sns Set Legend Left of Plot (Upper, Center, Lower Left)

Similarly, to position the legend on the left side of the plot, you can use the following code snippets:

Upper left:

ax.legend(loc='upper right', bbox_to_anchor=(-0.15, 1))

Center left:

ax.legend(loc='center right', bbox_to_anchor=(-0.15, 0.5))

Lower left:

ax.legend(loc='lower right', bbox_to_anchor=(-0.15, 0))

Sns Set Legend Right of Plot (Upper, Center, Lower Right)

Lastly, to place the legend on the right side of the plot, adjust the bbox_to_anchor parameter like so:

Upper right:

ax.legend(loc='upper left', bbox_to_anchor=(1.05, 1))

Center right:

ax.legend(loc='center left', bbox_to_anchor=(1.05, 0.5))

Lower right:

ax.legend(loc='lower left', bbox_to_anchor=(1.05, 0))

With these techniques, you can easily position the legend outside the plot using Seaborn! Happy plotting!

Pandas Put Legend Outside Plot

When working with Pandas and Matplotlib, you often want to move the legend outside the plot to improve readability. No worries! You can do this by taking advantage of the fact that .plot() returns a Matplotlib axis, enabling you to add .legend(bbox_to_anchor=(x, y)) to your code.

Here’s how:

First, import the necessary libraries like Pandas and Matplotlib:

import pandas as pd

import matplotlib.pyplot as plt

Create your DataFrame and plot it using the plot() function, like this:

data = {'A': [1, 2, 3], 'B': [4, 5, 6]}

df = pd.DataFrame(data)

ax = df.plot()

Next, you’ll want to place the legend outside the plot. Adjust the coordinates parameter, bbox_to_anchor, to position it according to your preferences. For example, if you want to place the legend to the right of the plot, use:

ax.legend(bbox_to_anchor=(1.05, 1)) plt.tight_layout()

plt.show()

This code will place the legend to the right of the plot, at the top .

Bokeh Put Legend Outside Plot

Placing a legend outside the plot in Bokeh can be easily done by using the add_layout method. You’ll need to create a Legend object manually and then add it to your plot using add_layout. This gives you the flexibility to position the legend anywhere on your plot, which is particularly helpful when you have many curves that might be obscured by an overlapping legend.

Here’s a short example to help you move the legend outside the plot in Bokeh:

from bokeh.plotting import figure, show

from bokeh.models import Legend, LegendItem

from bokeh.io import output_notebook

output_notebook() # Example data

x = list(range(10))

y1 = [i**2 for i in x]

y2 = [i**0.5 for i in x] # Create a figure

p = figure(title="Example Plot") # Add line glyphs and store their renderer

r1 = p.line(x, y1, line_color="blue", legend_label="Line 1")

r2 = p.line(x, y2, line_color="red", legend_label="Line 2") # Create Legend object

legend = Legend(items=[ LegendItem(label="Line 1", renderers=[r1]), LegendItem(label="Line 2", renderers=[r2])

], location="top_left") # Add legend to plot

p.add_layout(legend, 'right') # Show plot

show(p)

To ensure your plot is as clear as possible, we recommend experimenting with different legend positions and layouts. By customizing your plot, you can maintain a clean and organized display of your curves even when there’s a lot of information to convey.

With Bokeh, you have full control over your plot’s appearance, so enjoy exploring different options to find the perfect fit for your data!

---

Keep Learning With Your Python Cheat Sheet!

Feel free to download our free cheat sheet and join our tech and coding academy by downloading the free Finxter cheat sheets — it’s fun!