6. HTML Scrapping

6.1. BeautifulSoup

6.1.2. Install

$ pip install beautifulsoup4

6.1.3. Parser

Parser Typical usage Advantages Disadvantages
Python’s html.parser BeautifulSoup(markup, "html.parser")
  • Batteries included
  • Decent speed
  • tolerant (as of Python 2.7.3 and 3.2.)
  • Not very tolerant (before Python 2.7.3 or 3.2.2)
lxml’s HTML parser BeautifulSoup(markup, "lxml")
  • Very fast
  • Tolerant
  • External C dependency
lxml’s XML parser BeautifulSoup(markup, "lxml-xml") BeautifulSoup(markup, "xml")
  • Very fast
  • The only currently supported XML parser
  • External C dependency
html5lib BeautifulSoup(markup, "html5lib")
  • Extremely tolerant
  • Parses pages the same way a web browser does
  • Creates valid HTML5
  • Very slow
  • External Python dependency

6.1.4. Open

from bs4 import BeautifulSoup

with open("index.html") as file:
    html = BeautifulSoup(file, 'html.parser')

html.find(id='menubox').decompose()

6.1.5. Basic Usage

from bs4 import BeautifulSoup


html_doc = """
    <html><head><title>The Dormouse's story</title></head>
    <body>
    <p class="title"><b>The Dormouse's story</b></p>

    <p class="story">Once upon a time there were three little sisters; and their names were
    <a href="http://example.com/elsie" class="sister" id="link1">Elsie</a>,
    <a href="http://example.com/lacie" class="sister" id="link2">Lacie</a> and
    <a href="http://example.com/tillie" class="sister" id="link3">Tillie</a>;
    and they lived at the bottom of a well.</p>

    <p class="story">...</p>
"""

html = BeautifulSoup(html_doc, 'html.parser')

print(html.prettify())
# <html>
#  <head>
#   <title>
#    The Dormouse's story
#   </title>
#  </head>
#  <body>
#   <p class="title">
#    <b>
#     The Dormouse's story
#    </b>
#   </p>
#   <p class="story">
#    Once upon a time there were three little sisters; and their names were
#    <a class="sister" href="http://example.com/elsie" id="link1">
#     Elsie
#    </a>
#    ,
#    <a class="sister" href="http://example.com/lacie" id="link2">
#     Lacie
#    </a>
#    and
#    <a class="sister" href="http://example.com/tillie" id="link2">
#     Tillie
#    </a>
#    ; and they lived at the bottom of a well.
#   </p>
#   <p class="story">
#    ...
#   </p>
#  </body>
# </html>
html.title              # <title>The Dormouse's story</title>
html.title.name         # 'title'
html.title.string       # 'The Dormouse's story'
html.title.parent.name  # 'head'
html.p                  # <p class="title"><b>The Dormouse's story</b></p>
html.p['class']         # 'title'
html.a                  # <a class="sister" href="http://example.com/elsie" id="link1">Elsie</a>

html.find_all('a')
# [<a class="sister" href="http://example.com/elsie" id="link1">Elsie</a>,
#  <a class="sister" href="http://example.com/lacie" id="link2">Lacie</a>,
#  <a class="sister" href="http://example.com/tillie" id="link3">Tillie</a>]

html.find(id="link3")
# <a class="sister" href="http://example.com/tillie" id="link3">Tillie</a>

6.1.6. Iterating over items

for link in html.find_all('a'):
    print(link.get('href'))

# http://example.com/elsie
# http://example.com/lacie
# http://example.com/tillie

6.1.7. Getting Page Text

html.get_text()
# The Dormouse's story
#
# The Dormouse's story
#
# Once upon a time there were three little sisters; and their names were
# Elsie,
# Lacie and
# Tillie;
# and they lived at the bottom of a well.
#
# ...

6.2. Assignments

6.2.1. EVA

  1. Na podstawie podanych URL:

  2. Scrappuj stronę wykorzystując beautifulsoup4

  3. Przygotuj plik CSV z danymi dotyczącymi spacerów kosmicznych

  4. Spróbuj to samo zrobić za pomocą pandas.read_html():

    • Podając jako parametr czwarty URL
    • Dla częściowo sparsowanej strony, np. wyciągniętej tabelki
About:
  • Filename: scrapping_eva.py
  • Lines of code to write: 100 lines
  • Estimated time of completion: 45 min
The whys and wherefores:
 
  • Komunikacja HTTP (request, response)
  • Parsowanie odpowiedzi HTTP
  • Sprawdzanie stanu połączenia
  • Serializacja i parsowanie HTML
  • Korzystanie z Web Inspectora w przeglądarce
  • Używanie bibliotek zewnętrznych