4. HTML Scrapping

4.1. BeautifulSoup

4.1.2. Install

$ pip install beautifulsoup4

4.1.3. Parser

Parser

Typical usage

Advantages

Disadvantages

Python’s html.parser

BeautifulSoup(markup, "html.parser")

  • Batteries included

  • Decent speed

  • tolerant (as of Python 2.7.3 and 3.2.)

  • Not very tolerant (before Python 2.7.3 or 3.2.2)

lxml’s HTML parser

BeautifulSoup(markup, "lxml")

  • Very fast

  • Tolerant

  • External C dependency

lxml’s XML parser

BeautifulSoup(markup, "lxml-xml") BeautifulSoup(markup, "xml")

  • Very fast

  • The only currently supported XML parser

  • External C dependency

html5lib

BeautifulSoup(markup, "html5lib")

  • Extremely tolerant

  • Parses pages the same way a web browser does

  • Creates valid HTML5

  • Very slow

  • External Python dependency

4.1.4. Open

from bs4 import BeautifulSoup

with open("index.html") as file:
    html = BeautifulSoup(file, 'html.parser')

html.find(id='menubox').decompose()

4.1.5. Basic Usage

from bs4 import BeautifulSoup


html_doc = """
    <html><head><title>The Dormouse's story</title></head>
    <body>
    <p class="title"><b>The Dormouse's story</b></p>

    <p class="story">Once upon a time there were three little sisters; and their names were
    <a href="http://example.com/elsie" class="sister" id="link1">Elsie</a>,
    <a href="http://example.com/lacie" class="sister" id="link2">Lacie</a> and
    <a href="http://example.com/tillie" class="sister" id="link3">Tillie</a>;
    and they lived at the bottom of a well.</p>

    <p class="story">...</p>
"""

html = BeautifulSoup(html_doc, 'html.parser')

print(html.prettify())
# <html>
#  <head>
#   <title>
#    The Dormouse's story
#   </title>
#  </head>
#  <body>
#   <p class="title">
#    <b>
#     The Dormouse's story
#    </b>
#   </p>
#   <p class="story">
#    Once upon a time there were three little sisters; and their names were
#    <a class="sister" href="http://example.com/elsie" id="link1">
#     Elsie
#    </a>
#    ,
#    <a class="sister" href="http://example.com/lacie" id="link2">
#     Lacie
#    </a>
#    and
#    <a class="sister" href="http://example.com/tillie" id="link2">
#     Tillie
#    </a>
#    ; and they lived at the bottom of a well.
#   </p>
#   <p class="story">
#    ...
#   </p>
#  </body>
# </html>
html.title              # <title>The Dormouse's story</title>
html.title.name         # 'title'
html.title.string       # 'The Dormouse's story'
html.title.parent.name  # 'head'
html.p                  # <p class="title"><b>The Dormouse's story</b></p>
html.p['class']         # 'title'
html.a                  # <a class="sister" href="http://example.com/elsie" id="link1">Elsie</a>

html.find_all('a')
# [<a class="sister" href="http://example.com/elsie" id="link1">Elsie</a>,
#  <a class="sister" href="http://example.com/lacie" id="link2">Lacie</a>,
#  <a class="sister" href="http://example.com/tillie" id="link3">Tillie</a>]

html.find(id="link3")
# <a class="sister" href="http://example.com/tillie" id="link3">Tillie</a>

4.1.6. Iterating over items

for link in html.find_all('a'):
    print(link.get('href'))

# http://example.com/elsie
# http://example.com/lacie
# http://example.com/tillie

4.1.7. Getting Page Text

html.get_text()
# The Dormouse's story
#
# The Dormouse's story
#
# Once upon a time there were three little sisters; and their names were
# Elsie,
# Lacie and
# Tillie;
# and they lived at the bottom of a well.
#
# ...

4.2. Assignments

4.2.1. Iris

  • Filename: scrapping_iris.py

  • Lines of code to write: 20 lines

  • Estimated time of completion: 30 min

  1. Za pomocą beautifulsoup4 ze strony https://github.com/AstroMatt/book-python/blob/master/numerical-analysis/data/iris-dirty.csv pobierz dane zbióru Irysów.

  2. Parsując kod HTML oczyść dane.

  3. Skasuj pierwszy wiersz nagłówkowy.

  4. Kolumny nazwij: Sepal length, Sepal width, Petal length, Petal width, Species

  5. Wyświetl dane w formacie listy dictów, kluczami mają być nazwy kolumn.

4.2.2. EVA

  • Filename: scrapping_eva.py

  • Lines of code to write: 100 lines

  • Estimated time of completion: 45 min

  1. Na podstawie podanych URL:

  2. Scrappuj stronę wykorzystując beautifulsoup4

  3. Przygotuj plik CSV z danymi dotyczącymi spacerów kosmicznych

  4. Spróbuj to samo zrobić za pomocą pandas.read_html():

    • Podając jako parametr czwarty URL

    • Dla częściowo sparsowanej strony, np. wyciągniętej tabelki

The whys and wherefores
  • Komunikacja HTTP (request, response)

  • Parsowanie odpowiedzi HTTP

  • Sprawdzanie stanu połączenia

  • Serializacja i parsowanie HTML

  • Korzystanie z Web Inspectora w przeglądarce

  • Używanie bibliotek zewnętrznych