04_TIDE_session.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Master TIDE - Session 4\n",
    "\n",
    "1. **Web scraping et parsing HTML**\n",
    "3. **API**"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 1. Web Scraping\n",
    "\n",
    "- Extraction d'informations d'un site web.\n",
    "- A utiliser en l'absence de données ouvertes ou d'API.\n",
    "- Technique fragile car le site web peut changer du jour au lendemain.\n",
    "- Problématique juridique...\n",
    "\n",
    "**Avec requests**\n",
    "\n",
    "Doc :\n",
    "- requests : https://requests.readthedocs.io/en/master/"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "200"
      ]
     },
     "execution_count": 1,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "import requests\n",
    "\n",
    "r = requests.get('https://kim.fspot.org/cours/page2.html')\n",
    "r.status_code"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "bytes"
      ]
     },
     "execution_count": 2,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# type\n",
    "type(r.content)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "str"
      ]
     },
     "execution_count": 4,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# str en précisant un encodage\n",
    "content = r.content.decode('utf-8')\n",
    "type(content)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "5993"
      ]
     },
     "execution_count": 30,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "len(content)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "\n",
       "<!DOCTYPE html>\n",
       "\n",
       "<html lang=\"en\">\n",
       "<head>\n",
       "<meta charset=\"utf-8\"/>\n",
       "<meta content=\"width=device-width, initial-scale=1.0\" name=\"viewport\"/>\n",
       "<link crossorigin=\"anonymous\" href=\"https://unpkg.com/purecss@1.0.1/build/pure-min.css\" integrity=\"sha384-oAOxQR6DkCoMliIh8yFnu25d7Eq/PHS21PClpwjOTeU2jRSq11vu66rf90/cZr47\" rel=\"stylesheet\"/>\n",
       "<link href=\"https://unpkg.com/purecss@1.0.1/build/grids-responsive-min.css\" rel=\"stylesheet\"/>\n",
       "<link href=\"page2.css\" rel=\"stylesheet\"/>\n",
       "</head>\n",
       "<body>\n",
       "<div class=\"pure-menu pure-menu-horizontal\">\n",
       "<a class=\"pure-menu-heading\" href=\"#\">Your Logo</a>\n",
       "<ul class=\"pure-menu-list\">\n",
       "<li class=\"pure-menu-item\"><a class=\"pure-menu-link\" href=\"#\">Home</a></li>\n",
       "<li class=\"pure-menu-item pure-menu-selected\"><a class=\"pure-menu-link\" href=\"#\">Pricing</a></li>\n",
       "<li class=\"pure-menu-item\"><a class=\"pure-menu-link\" href=\"#\">Contact</a></li>\n",
       "</ul>\n",
       "</div>\n",
       "<div class=\"banner\">\n",
       "<h1 class=\"banner-head\">\n",
       "        Simple Pricing.<br/>\n",
       "        Try before you buy.\n",
       "    </h1>\n",
       "</div>\n",
       "<div class=\"l-content\">\n",
       "<div class=\"pricing-tables pure-g\">\n",
       "<div class=\"pure-u-1 pure-u-md-1-3\">\n",
       "<div class=\"pricing-table pricing-table-free\">\n",
       "<div class=\"pricing-table-header\">\n",
       "<h2>Personal</h2>\n",
       "<span class=\"pricing-table-price\">\n",
       "                        $5 <span>per month</span>\n",
       "</span>\n",
       "</div>\n",
       "<ul class=\"pricing-table-list\">\n",
       "<li>Free setup</li>\n",
       "<li>Custom sub-domain</li>\n",
       "<li>Standard customer support</li>\n",
       "<li>1GB file storage</li>\n",
       "<li>1 database</li>\n",
       "<li>Unlimited bandwidth</li>\n",
       "</ul>\n",
       "<button class=\"button-choose pure-button\">Choose</button>\n",
       "</div>\n",
       "</div>\n",
       "<div class=\"pure-u-1 pure-u-md-1-3\">\n",
       "<div class=\"pricing-table pricing-table-biz pricing-table-selected\">\n",
       "<div class=\"pricing-table-header\">\n",
       "<h2>Small Business</h2>\n",
       "<span class=\"pricing-table-price\">\n",
       "                        $25 <span>per month</span>\n",
       "</span>\n",
       "</div>\n",
       "<ul class=\"pricing-table-list\">\n",
       "<li>Free setup</li>\n",
       "<li>Use your own domain</li>\n",
       "<li>Standard customer support</li>\n",
       "<li>10GB file storage</li>\n",
       "<li>5 databases</li>\n",
       "<li>Unlimited bandwidth</li>\n",
       "</ul>\n",
       "<button class=\"button-choose pure-button\">Choose</button>\n",
       "</div>\n",
       "</div>\n",
       "<div class=\"pure-u-1 pure-u-md-1-3\">\n",
       "<div class=\"pricing-table pricing-table-enterprise\">\n",
       "<div class=\"pricing-table-header\">\n",
       "<h2>Enterprise</h2>\n",
       "<span class=\"pricing-table-price\">\n",
       "                        $45 <span>per month</span>\n",
       "</span>\n",
       "</div>\n",
       "<ul class=\"pricing-table-list\">\n",
       "<li>Free setup</li>\n",
       "<li>Use your own domain</li>\n",
       "<li>Premium customer support</li>\n",
       "<li>100GB file storage</li>\n",
       "<li>25 databases</li>\n",
       "<li>Unlimited bandwidth</li>\n",
       "</ul>\n",
       "<button class=\"button-choose pure-button\">Choose</button>\n",
       "</div>\n",
       "</div>\n",
       "</div> <!-- end pricing-tables -->\n",
       "<div class=\"information pure-g\">\n",
       "<div class=\"pure-u-1 pure-u-md-1-2\">\n",
       "<div class=\"l-box\">\n",
       "<h3 class=\"information-head\">Get started today</h3>\n",
       "<p>\n",
       "                    Lorem ipsum dolor sit amet, consectetur adipisicing elit, sed do eiusmod tempor incididunt ut labore et dolore magna aliqua. Lorem ipsum dolor sit amet, consectetur adipisicing elit, sed do eiusmod tempor incididunt ut labore et dolore magna aliqua. Ut enim ad minim veniam, quis nostrud exercitation.\n",
       "                </p>\n",
       "</div>\n",
       "</div>\n",
       "<div class=\"pure-u-1 pure-u-md-1-2\">\n",
       "<div class=\"l-box\">\n",
       "<h3 class=\"information-head\">Pay monthly or annually</h3>\n",
       "<p>\n",
       "                    Lorem ipsum dolor sit amet, consectetur adipisicing elit, sed do eiusmod tempor incididunt ut labore et dolore magna aliqua. Ullamco laboris nisi ut aliquip ex ea commodo\n",
       "                    consequat. Duis aute irure dolor in reprehenderit in voluptate velit esse.\n",
       "                </p>\n",
       "</div>\n",
       "</div>\n",
       "<div class=\"pure-u-1 pure-u-md-1-2\">\n",
       "<div class=\"l-box\">\n",
       "<h3 class=\"information-head\">24/7 customer support</h3>\n",
       "<p>\n",
       "                    Cillum dolore eu fugiat nulla pariatur. Excepteur sint occaecat cupidatat non proident, sunt in culpa qui officia deserunt mollit anim id est laborum. Lorem ipsum dolor sit amet, consectetur adipisicing elit, sed do eiusmod tempor incididunt ut labore et dolore magna aliqua.\n",
       "                </p>\n",
       "</div>\n",
       "</div>\n",
       "<div class=\"pure-u-1 pure-u-md-1-2\">\n",
       "<div class=\"l-box\">\n",
       "<h3 class=\"information-head\">Cancel your plan anytime</h3>\n",
       "<p>\n",
       "                    Duis aute irure dolor in reprehenderit in voluptate velit esse\n",
       "                    cillum dolore eu fugiat nulla pariatur. Excepteur sint occaecat cupidatat non proident, sunt in culpa qui officia deserunt mollit anim id est laborum.\n",
       "                </p>\n",
       "</div>\n",
       "</div>\n",
       "</div> <!-- end information -->\n",
       "</div> <!-- end l-content -->\n",
       "<div class=\"footer l-box\">\n",
       "<p>\n",
       "<a href=\"#\">Try now</a> for 14 days. No credit card required. Header image courtesy of <a href=\"http://unsplash.com/\">Unsplash</a>.\n",
       "    </p>\n",
       "</div>\n",
       "</body>\n",
       "</html>"
      ]
     },
     "execution_count": 6,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# BeautifulSoup\n",
    "\n",
    "from bs4 import BeautifulSoup\n",
    "\n",
    "soup = BeautifulSoup(content, \"html.parser\")\n",
    "soup"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "On utilise différentes méthodes :\n",
    "    \n",
    "- `find(tag, attrs)` : trouve le premier tag avec les attributs spécifiés\n",
    "- `findAll(tag, attrs)` : trouve tous les tags avec les attributs spécifiés\n",
    "- `select(css)` : trouve tous les tags avec les css spécifiées\n",
    "- etc."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<a class=\"pure-menu-heading\" href=\"#\">Your Logo</a>"
      ]
     },
     "execution_count": 7,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "soup.find(\"a\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<div class=\"pure-menu pure-menu-horizontal\">\n",
       "<a class=\"pure-menu-heading\" href=\"#\">Your Logo</a>\n",
       "<ul class=\"pure-menu-list\">\n",
       "<li class=\"pure-menu-item\"><a class=\"pure-menu-link\" href=\"#\">Home</a></li>\n",
       "<li class=\"pure-menu-item pure-menu-selected\"><a class=\"pure-menu-link\" href=\"#\">Pricing</a></li>\n",
       "<li class=\"pure-menu-item\"><a class=\"pure-menu-link\" href=\"#\">Contact</a></li>\n",
       "</ul>\n",
       "</div>"
      ]
     },
     "execution_count": 8,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "soup.find(\"div\", attrs={'class': 'pure-menu'})"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<div class=\"banner\">\n",
       "<h1 class=\"banner-head\">\n",
       "        Simple Pricing.<br/>\n",
       "        Try before you buy.\n",
       "    </h1>\n",
       "</div>"
      ]
     },
     "execution_count": 9,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "soup.find(\"div\", attrs={'class': 'banner'})"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "b'<html lang=\"en\"><head><title>leboncoin.fr</title><style>#cmsg{animation: A 1.5s;}@keyframes A{0%{opacity:0;}99%{opacity:0;}100%{opacity:1;}}</style></head><body style=\"margin:0\"><p id=\"cmsg\">Please enable JS and disable any ad blocker</p><script data-cfasync=\"false\">var dd={\\'rt\\':\\'c\\',\\'cid\\':\\'AHrlqAAAAAMA_D8PhF9nZKgAw1i1NQ==\\',\\'hsh\\':\\'05B30BD9055986BD2EE8F5A199D973\\',\\'t\\':\\'bv\\',\\'s\\':2089,\\'e\\':\\'63b77a0412735184ad7c4e2cc5d4d7e4692f24d45f401bda59d8e91cd7bff0fe\\',\\'host\\':\\'geo.captcha-delivery.com\\',\\'cookie\\':\\'j1ITeCcWY9v5GD0lPa0bDVEkc9xpNhHftTBnraIeBApKuuifEfnh80abXTbxLnSb8wPrM73JLq~IDEKksDMHdMxIMIMotSzj61Zmj1kUu4Jx8pDXxsvCYKKH8PnAmC8P\\'}</script><script data-cfasync=\"false\" src=\"https://ct.captcha-delivery.com/c.js\"></script></body></html>'"
      ]
     },
     "execution_count": 11,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "requests.get(\"https://leboncoin.fr\").content"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "La technique consiste par exemple à alimenter un dictionnaires avec les valeurs trouvées :\n",
    "- soit en utilisant `tag.attrs['attr']` pour collecter la valeur attr du tag\n",
    "- soit en utilisant `tag.text` pour collecter la valeur `text`\n",
    "- on peut aussi utiliser d'autres méthodes comme `select()` qui joue sur la CSS \n",
    "- éventuellement en recherchant dans un nouveau tag à l'intérieur d'un tag donné"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<div class=\"pricing-table pricing-table-free\">\n",
       "<div class=\"pricing-table-header\">\n",
       "<h2>Personal</h2>\n",
       "<span class=\"pricing-table-price\">\n",
       "                        $5 <span>per month</span>\n",
       "</span>\n",
       "</div>\n",
       "<ul class=\"pricing-table-list\">\n",
       "<li>Free setup</li>\n",
       "<li>Custom sub-domain</li>\n",
       "<li>Standard customer support</li>\n",
       "<li>1GB file storage</li>\n",
       "<li>1 database</li>\n",
       "<li>Unlimited bandwidth</li>\n",
       "</ul>\n",
       "<button class=\"button-choose pure-button\">Choose</button>\n",
       "</div>"
      ]
     },
     "execution_count": 15,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "div_tag = soup.find('div', attrs={'class': 'pricing-table'})\n",
    "div_tag"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{'type': 'Personal', 'price': '$5', 'storage': '1GB', 'database': '1'}"
      ]
     },
     "execution_count": 20,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# exploration du HTML\n",
    "row = {}\n",
    "div_tag = soup.find('div', attrs={'class': 'pricing-table'})\n",
    "h2_tag = div_tag.find(\"h2\")\n",
    "row[\"type\"] = h2_tag.text\n",
    "span_tag = div_tag.find(\"span\", attrs={'class': 'pricing-table-price'})\n",
    "row[\"price\"] = span_tag.text.strip().split()[0]\n",
    "lis = div_tag.select(\"li\")\n",
    "row[\"storage\"] = lis[3].text.split()[0]\n",
    "row[\"database\"] = lis[4].text.split()[0]\n",
    "row"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[{'type': 'Personal', 'price': '$5', 'storage': '1GB', 'database': '1'},\n",
       " {'type': 'Small Business',\n",
       "  'price': '$25',\n",
       "  'storage': '10GB',\n",
       "  'database': '5'},\n",
       " {'type': 'Enterprise', 'price': '$45', 'storage': '100GB', 'database': '25'}]"
      ]
     },
     "execution_count": 21,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# exploration du HTML\n",
    "rows = []\n",
    "for div_tag in soup.find_all('div', attrs={'class': 'pricing-table'}):\n",
    "    row = {}\n",
    "    h2_tag = div_tag.find(\"h2\")\n",
    "    row[\"type\"] = h2_tag.text\n",
    "    span_tag = div_tag.find(\"span\", attrs={'class': 'pricing-table-price'})\n",
    "    row[\"price\"] = span_tag.text.strip().split()[0]\n",
    "    lis = div_tag.select(\"li\")\n",
    "    row[\"storage\"] = lis[3].text.split()[0]\n",
    "    row[\"database\"] = lis[4].text.split()[0]\n",
    "    rows.append(row)\n",
    "\n",
    "rows"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "metadata": {},
   "outputs": [],
   "source": [
    "# /html/body/div[3]/div[1]/div[1]/div/div/span\n",
    "# /html/body/div[3]/div[1]/div[2]/div/div/span\n",
    "# /html/body/div[3]/div[1]/div[1]/div/ul/li[5]\n",
    "# /html/body/div[3]/div[1]/div[4]/div/div/span"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Méthode :\n",
    " 1. On collecte les informations d'un  tag pour un seul enregistrement sous la forme d'un dictionaire.\n",
    " 2. On généralise en bouclant sur tous les tags de même nature et on rassemble les dictionnaires dans une liste.\n",
    " 3. On transforme la liste de dictionaires en DataFrame."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>type</th>\n",
       "      <th>price</th>\n",
       "      <th>storage</th>\n",
       "      <th>database</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>Personal</td>\n",
       "      <td>$5</td>\n",
       "      <td>1GB</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>Small Business</td>\n",
       "      <td>$25</td>\n",
       "      <td>10GB</td>\n",
       "      <td>5</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>Enterprise</td>\n",
       "      <td>$45</td>\n",
       "      <td>100GB</td>\n",
       "      <td>25</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "             type price storage database\n",
       "0        Personal    $5     1GB        1\n",
       "1  Small Business   $25    10GB        5\n",
       "2      Enterprise   $45   100GB       25"
      ]
     },
     "execution_count": 22,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Transformation en DataFrame\n",
    "import pandas as pd\n",
    "df = pd.DataFrame(rows)\n",
    "df"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>type</th>\n",
       "      <th>price</th>\n",
       "      <th>storage</th>\n",
       "      <th>database</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>Personal</td>\n",
       "      <td>$5</td>\n",
       "      <td>1GB</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>Small Business</td>\n",
       "      <td>$25</td>\n",
       "      <td>10GB</td>\n",
       "      <td>5</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>Enterprise</td>\n",
       "      <td>$45</td>\n",
       "      <td>100GB</td>\n",
       "      <td>25</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>Privilege</td>\n",
       "      <td>$99</td>\n",
       "      <td>1TB</td>\n",
       "      <td>100</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "             type price storage  database\n",
       "0        Personal    $5     1GB         1\n",
       "1  Small Business   $25    10GB         5\n",
       "2      Enterprise   $45   100GB        25\n",
       "3       Privilege   $99     1TB       100"
      ]
     },
     "execution_count": 26,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "r = requests.get('https://kim.fspot.org/cours/page3.html')\n",
    "\n",
    "soup = BeautifulSoup(r.content, \"html.parser\")\n",
    "\n",
    "rows = []\n",
    "\n",
    "for div_tag in soup.find_all('div', attrs={'class': 'pricing-table'}):\n",
    "    row = {}\n",
    "    h2_tag = div_tag.find(\"h2\")\n",
    "    row[\"type\"] = h2_tag.text\n",
    "    span_tag = div_tag.find(\"span\", attrs={'class': 'pricing-table-price'})\n",
    "    row[\"price\"] = span_tag.text.strip().split()[0]\n",
    "    lis = div_tag.select(\"li\")\n",
    "    row[\"storage\"] = lis[3].text.split()[0]\n",
    "    row[\"database\"] = lis[4].text.split()[0]\n",
    "    rows.append(row)\n",
    "\n",
    "df = pd.DataFrame(rows)\n",
    "df = df.astype({\"database\": int})\n",
    "df"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "<class 'pandas.core.frame.DataFrame'>\n",
      "RangeIndex: 4 entries, 0 to 3\n",
      "Data columns (total 4 columns):\n",
      " #   Column    Non-Null Count  Dtype \n",
      "---  ------    --------------  ----- \n",
      " 0   type      4 non-null      object\n",
      " 1   price     4 non-null      object\n",
      " 2   storage   4 non-null      object\n",
      " 3   database  4 non-null      int64 \n",
      "dtypes: int64(1), object(3)\n",
      "memory usage: 260.0+ bytes\n"
     ]
    }
   ],
   "source": [
    "df.info()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Inconvénients du web scraping:\n",
    "- plutôt lent (car on parse potentiellement beaucoup de HTML inutile)\n",
    "- ne donne pas les résultats attendus si une partie du contenu est intégré dynamiquement à la page via javascript\n",
    "- un changement dans l'architecture du html ou du css (e.g: refonte du design du site) oblige à réécrire entièrement le programme"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### API\n",
    "\n",
    "Exemple: Deezer\n",
    "\n",
    "Artiste : https://www.deezer.com/fr/artist/3037\n",
    "\n",
    "Récupérer le nombre de fans d'un artiste avec requests (cherchez le tag div avec l'id `\"naboo_artist_social_small\"`)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 31,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "1415152"
      ]
     },
     "execution_count": 31,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "import requests\n",
    "from bs4 import BeautifulSoup\n",
    "# request\n",
    "artist = 3037\n",
    "response = requests.get(f'https://www.deezer.com/fr/artist/{artist}')\n",
    "len(response.content)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# /html/body/div[1]/div/div[6]/div/div[1]/div[1]/div[2]/ul/li"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Récupérer le nombre de fans d'un artiste avec l'API :\n",
    "\n",
    "Doc:\n",
    "- https://pypi.org/project/deezer-python/\n",
    "\n",
    "Installation :\n",
    "- *pip install deezer-python*"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Le terme \"API\" est très générique et peut désigner bien des choses, mais dans le jargon on l'utilise souvent pour désigner un service web qui renvoie non pas:\n",
    "> des pages web au format HTML (destinées à être lues par un humain dans son navigateur)\n",
    "\n",
    "mais:\n",
    "> des données au format JSON (destinées à être traitées par un programme)\n",
    "\n",
    "![img](https://miro.medium.com/max/4238/1*OcmVkcsM5BWRHrg8GC17iw.png)\n",
    "\n",
    "Puisque les API sont dédiées à l'usage via des programmes, elles disposent en général d'une bonne documentation, et sont fiables et stables dans le temps. Tandis que sur des pages web HTML classiques, le design peut par exemple changer du jour au lendemain et rendre votre programme BeautifulSoup obsolète."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 32,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "b'{\"id\":3037,\"name\":\"Genesis\",\"link\":\"https:\\\\/\\\\/www.deezer.com\\\\/artist\\\\/3037\",\"share\":\"https:\\\\/\\\\/www.deezer.com\\\\/artist\\\\/3037?utm_source=deezer&utm_content=artist-3037&utm_term=0_1734621629&utm_medium=web\",\"picture\":\"https:\\\\/\\\\/api.deezer.com\\\\/artist\\\\/3037\\\\/image\",\"picture_small\":\"https:\\\\/\\\\/cdn-images.dzcdn.net\\\\/images\\\\/artist\\\\/c0ed2cf09c7bf072111d614fc4557b66\\\\/56x56-000000-80-0-0.jpg\",\"picture_medium\":\"https:\\\\/\\\\/cdn-images.dzcdn.net\\\\/images\\\\/artist\\\\/c0ed2cf09c7bf072111d614fc4557b66\\\\/250x250-000000-80-0-0.jpg\",\"picture_big\":\"https:\\\\/\\\\/cdn-images.dzcdn.net\\\\/images\\\\/artist\\\\/c0ed2cf09c7bf072111d614fc4557b66\\\\/500x500-000000-80-0-0.jpg\",\"picture_xl\":\"https:\\\\/\\\\/cdn-images.dzcdn.net\\\\/images\\\\/artist\\\\/c0ed2cf09c7bf072111d614fc4557b66\\\\/1000x1000-000000-80-0-0.jpg\",\"nb_album\":32,\"nb_fan\":1024432,\"radio\":true,\"tracklist\":\"https:\\\\/\\\\/api.deezer.com\\\\/artist\\\\/3037\\\\/top?limit=50\",\"type\":\"artist\"}'"
      ]
     },
     "execution_count": 32,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# API JSON\n",
    "response = requests.get(f'https://api.deezer.com/artist/{artist}')\n",
    "response.content"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 36,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "dict"
      ]
     },
     "execution_count": 36,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "import json\n",
    "d = json.loads(response.content)\n",
    "type(d)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 33,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "889"
      ]
     },
     "execution_count": 33,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "len(response.content)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 34,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "1591.8470191226097"
      ]
     },
     "execution_count": 34,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "1415152/889"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 37,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "requests.models.Response"
      ]
     },
     "execution_count": 37,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "type(response)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 38,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{'id': 3037,\n",
       " 'name': 'Genesis',\n",
       " 'link': 'https://www.deezer.com/artist/3037',\n",
       " 'share': 'https://www.deezer.com/artist/3037?utm_source=deezer&utm_content=artist-3037&utm_term=0_1734621629&utm_medium=web',\n",
       " 'picture': 'https://api.deezer.com/artist/3037/image',\n",
       " 'picture_small': 'https://cdn-images.dzcdn.net/images/artist/c0ed2cf09c7bf072111d614fc4557b66/56x56-000000-80-0-0.jpg',\n",
       " 'picture_medium': 'https://cdn-images.dzcdn.net/images/artist/c0ed2cf09c7bf072111d614fc4557b66/250x250-000000-80-0-0.jpg',\n",
       " 'picture_big': 'https://cdn-images.dzcdn.net/images/artist/c0ed2cf09c7bf072111d614fc4557b66/500x500-000000-80-0-0.jpg',\n",
       " 'picture_xl': 'https://cdn-images.dzcdn.net/images/artist/c0ed2cf09c7bf072111d614fc4557b66/1000x1000-000000-80-0-0.jpg',\n",
       " 'nb_album': 32,\n",
       " 'nb_fan': 1024432,\n",
       " 'radio': True,\n",
       " 'tracklist': 'https://api.deezer.com/artist/3037/top?limit=50',\n",
       " 'type': 'artist'}"
      ]
     },
     "execution_count": 38,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "response.json()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 39,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "1024432"
      ]
     },
     "execution_count": 39,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# nb_fan\n",
    "data = response.json()\n",
    "data['nb_fan']"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 40,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/jpeg": "/9j/4AAQSkZJRgABAQIAHAAcAAD/4gzYSUNDX1BST0ZJTEUAAQEAAAzIYXBwbAIQAABtbnRyUkdCIFhZWiAH4AABAAQACQAcACxhY3NwQVBQTAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAQAA9tYAAQAAAADTLWFwcGwAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAABFkZXNjAAABUAAAAGJkc2NtAAABtAAAAbBjcHJ0AAADZAAAACN3dHB0AAADiAAAABRyWFlaAAADnAAAABRnWFlaAAADsAAAABRiWFlaAAADxAAAABRyVFJDAAAD2AAACAxhYXJnAAAL5AAAACB2Y2d0AAAMBAAAADBuZGluAAAMNAAAAD5jaGFkAAAMdAAAACxtbW9kAAAMoAAAAChiVFJDAAAD2AAACAxnVFJDAAAD2AAACAxhYWJnAAAL5AAAACBhYWdnAAAL5AAAACBkZXNjAAAAAAAAAAhEaXNwbGF5AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAbWx1YwAAAAAAAAAiAAAADGhySFIAAAAIAAABqGtvS1IAAAAIAAABqG5iTk8AAAAIAAABqGlkAAAAAAAIAAABqGh1SFUAAAAIAAABqGNzQ1oAAAAIAAABqGRhREsAAAAIAAABqHVrVUEAAAAIAAABqGFyAAAAAAAIAAABqGl0SVQAAAAIAAABqHJvUk8AAAAIAAABqGVzRVMAAAAIAAABqGhlSUwAAAAIAAABqG5sTkwAAAAIAAABqGZpRkkAAAAIAAABqHpoVFcAAAAIAAABqHZpVk4AAAAIAAABqHNrU0sAAAAIAAABqHpoQ04AAAAIAAABqHJ1UlUAAAAIAAABqGZyRlIAAAAIAAABqG1zAAAAAAAIAAABqGNhRVMAAAAIAAABqHRoVEgAAAAIAAABqGVzWEwAAAAIAAABqGRlREUAAAAIAAABqGVuVVMAAAAIAAABqHB0QlIAAAAIAAABqHBsUEwAAAAIAAABqGVsR1IAAAAIAAABqHN2U0UAAAAIAAABqHRyVFIAAAAIAAABqGphSlAAAAAIAAABqHB0UFQAAAAIAAABqABpAE0AYQBjdGV4dAAAAABDb3B5cmlnaHQgQXBwbGUgSW5jLiwgMjAxNgAAWFlaIAAAAAAAAPMWAAEAAAABFspYWVogAAAAAAAAccAAADmKAAABZ1hZWiAAAAAAAABhIwAAueYAABP2WFlaIAAAAAAAACPyAAAMkAAAvdBjdXJ2AAAAAAAABAAAAAAFAAoADwAUABkAHgAjACgALQAyADYAOwBAAEUASgBPAFQAWQBeAGMAaABtAHIAdwB8AIEAhgCLAJAAlQCaAJ8AowCoAK0AsgC3ALwAwQDGAMsA0ADVANsA4ADlAOsA8AD2APsBAQEHAQ0BEwEZAR8BJQErATIBOAE+AUUBTAFSAVkBYAFnAW4BdQF8AYMBiwGSAZoBoQGpAbEBuQHBAckB0QHZAeEB6QHyAfoCAwIMAhQCHQImAi8COAJBAksCVAJdAmcCcQJ6AoQCjgKYAqICrAK2AsECywLVAuAC6wL1AwADCwMWAyEDLQM4A0MDTwNaA2YDcgN+A4oDlgOiA64DugPHA9MD4APsA/kEBgQTBCAELQQ7BEgEVQRjBHEEfgSMBJoEqAS2BMQE0wThBPAE/gUNBRwFKwU6BUkFWAVnBXcFhgWWBaYFtQXFBdUF5QX2BgYGFgYnBjcGSAZZBmoGewaMBp0GrwbABtEG4wb1BwcHGQcrBz0HTwdhB3QHhgeZB6wHvwfSB+UH+AgLCB8IMghGCFoIbgiCCJYIqgi+CNII5wj7CRAJJQk6CU8JZAl5CY8JpAm6Cc8J5Qn7ChEKJwo9ClQKagqBCpgKrgrFCtwK8wsLCyILOQtRC2kLgAuYC7ALyAvhC/kMEgwqDEMMXAx1DI4MpwzADNkM8w0NDSYNQA1aDXQNjg2pDcMN3g34DhMOLg5JDmQOfw6bDrYO0g7uDwkPJQ9BD14Peg+WD7MPzw/sEAkQJhBDEGEQfhCbELkQ1xD1ERMRMRFPEW0RjBGqEckR6BIHEiYSRRJkEoQSoxLDEuMTAxMjE0MTYxODE6QTxRPlFAYUJxRJFGoUixStFM4U8BUSFTQVVhV4FZsVvRXgFgMWJhZJFmwWjxayFtYW+hcdF0EXZReJF64X0hf3GBsYQBhlGIoYrxjVGPoZIBlFGWsZkRm3Gd0aBBoqGlEadxqeGsUa7BsUGzsbYxuKG7Ib2hwCHCocUhx7HKMczBz1HR4dRx1wHZkdwx3sHhYeQB5qHpQevh7pHxMfPh9pH5Qfvx/qIBUgQSBsIJggxCDwIRwhSCF1IaEhziH7IiciVSKCIq8i3SMKIzgjZiOUI8Ij8CQfJE0kfCSrJNolCSU4JWgllyXHJfcmJyZXJocmtyboJxgnSSd6J6sn3CgNKD8ocSiiKNQpBik4KWspnSnQKgIqNSpoKpsqzysCKzYraSudK9EsBSw5LG4soizXLQwtQS12Last4S4WLkwugi63Lu4vJC9aL5Evxy/+MDUwbDCkMNsxEjFKMYIxujHyMioyYzKbMtQzDTNGM38zuDPxNCs0ZTSeNNg1EzVNNYc1wjX9Njc2cjauNuk3JDdgN5w31zgUOFA4jDjIOQU5Qjl/Obw5+To2OnQ6sjrvOy07azuqO+g8JzxlPKQ84z0iPWE9oT3gPiA+YD6gPuA/IT9hP6I/4kAjQGRApkDnQSlBakGsQe5CMEJyQrVC90M6Q31DwEQDREdEikTORRJFVUWaRd5GIkZnRqtG8Ec1R3tHwEgFSEtIkUjXSR1JY0mpSfBKN0p9SsRLDEtTS5pL4kwqTHJMuk0CTUpNk03cTiVObk63TwBPSU+TT91QJ1BxULtRBlFQUZtR5lIxUnxSx1MTU19TqlP2VEJUj1TbVShVdVXCVg9WXFapVvdXRFeSV+BYL1h9WMtZGllpWbhaB1pWWqZa9VtFW5Vb5Vw1XIZc1l0nXXhdyV4aXmxevV8PX2Ffs2AFYFdgqmD8YU9homH1YklinGLwY0Njl2PrZEBklGTpZT1lkmXnZj1mkmboZz1nk2fpaD9olmjsaUNpmmnxakhqn2r3a09rp2v/bFdsr20IbWBtuW4SbmtuxG8eb3hv0XArcIZw4HE6cZVx8HJLcqZzAXNdc7h0FHRwdMx1KHWFdeF2Pnabdvh3VnezeBF4bnjMeSp5iXnnekZ6pXsEe2N7wnwhfIF84X1BfaF+AX5ifsJ/I3+Ef+WAR4CogQqBa4HNgjCCkoL0g1eDuoQdhICE44VHhauGDoZyhteHO4efiASIaYjOiTOJmYn+imSKyoswi5aL/IxjjMqNMY2Yjf+OZo7OjzaPnpAGkG6Q1pE/kaiSEZJ6kuOTTZO2lCCUipT0lV+VyZY0lp+XCpd1l+CYTJi4mSSZkJn8mmia1ZtCm6+cHJyJnPedZJ3SnkCerp8dn4uf+qBpoNihR6G2oiailqMGo3aj5qRWpMelOKWpphqmi6b9p26n4KhSqMSpN6mpqhyqj6sCq3Wr6axcrNCtRK24ri2uoa8Wr4uwALB1sOqxYLHWskuywrM4s660JbSctRO1irYBtnm28Ldot+C4WbjRuUq5wro7urW7LrunvCG8m70VvY++Cr6Evv+/er/1wHDA7MFnwePCX8Lbw1jD1MRRxM7FS8XIxkbGw8dBx7/IPci8yTrJuco4yrfLNsu2zDXMtc01zbXONs62zzfPuNA50LrRPNG+0j/SwdNE08bUSdTL1U7V0dZV1tjXXNfg2GTY6Nls2fHadtr724DcBdyK3RDdlt4c3qLfKd+v4DbgveFE4cziU+Lb42Pj6+Rz5PzlhOYN5pbnH+ep6DLovOlG6dDqW+rl63Dr++yG7RHtnO4o7rTvQO/M8Fjw5fFy8f/yjPMZ86f0NPTC9VD13vZt9vv3ivgZ+Kj5OPnH+lf65/t3/Af8mP0p/br+S/7c/23//3BhcmEAAAAAAAMAAAACZmYAAPKnAAANWQAAE9AAAAoOdmNndAAAAAAAAAABAAEAAAAAAAAAAQAAAAEAAAAAAAAAAQAAAAEAAAAAAAAAAQAAbmRpbgAAAAAAAAA2AACnQAAAVYAAAEzAAACewAAAJYAAAAzAAABQAAAAVEAAAjMzAAIzMwACMzMAAAAAAAAAAHNmMzIAAAAAAAEMcgAABfj///MdAAAHugAA/XL///ud///9pAAAA9kAAMBxbW1vZAAAAAAAAAYQAACgEgAAAADMHn4AAAAAAAAAAAAAAAAAAAAAAP/bAEMABgQFBgUEBgYFBgcHBggKEAoKCQkKFA4PDBAXFBgYFxQWFhodJR8aGyMcFhYgLCAjJicpKikZHy0wLSgwJSgpKP/CAAsIAHgAeAEBEQD/xAAcAAABBAMBAAAAAAAAAAAAAAAABAUGBwEDCAL/2gAIAQEAAAABtQAxlMoyjapCANmxfDkdeWPOdDS77gImwJUfpueHxK7+paAgiS+EzVz2p2CRLVwEWe8ROdxN3dQAAICxWrXLjNWp59ABHq7sRBUt4M2qTOQBjmtrsWBpuiKzT3htANPL+twb821UpfcuAEPNGtcgzZdaFx2MAQOmn6V1bmXtTLZNwAFbU/Ip/TpIXmDTi7/QEBpP30/RkQnMTRyy63IMQakDpqO0PYrhVSnprcGjnvy29NU7XtrxiH56eVH/xAAoEAABBAEEAgEDBQAAAAAAAAAEAQIDBQYAEBMgERQSISQ0FTAxMjP/2gAIAQEAAQUC7yTxRLsZO0UamMmNG6lkrCoRCFDaySxjEFxy1nMiQyGaXEjHkgakexsVWWpQ7JY5OpjDW2TZSB8bAuOGZqJYZaOih5JVmwg6mvk8HAn+oWRG3G4hIyaWrlSMzowljiqmOeauZXETSVC/DLLGo9myMqwzHi14wiTBkSas6lW1CV8UoIosArelnU+4QEMwMYsuERgwgRdto+ymjdXl+1F+1cwe6VUCLBazO+MUciqFRczjppGxR+8vlF8p2tLUevQa75CKiL7fILOAUUctOKqFQUXJXuRjHSLNW/P0u1nOpBusQIeyYmZ08+qOX51GR27S5a60ZzRqis6zf5P/AJ0GS4Z+3KrMO2xcjnqupsqQCO+uxIzx96qXmxjbCvwuuYTfCugbyS21YgLcpjSOr2xzkWWyrpQF1hUngnrmrPttPsllr8kikKp9qP5IaVYDGh6xqGVhKL5TpmTfNbpv9o/rHkoCBHaxKB77AuF8BGq2z4mAfj9MunjZW7BefUymBk1XrC3qheZCeW6Fk4iIlRY95pGQxXJzjjEr3/pYkazEtTwmV2fPLrE2MGAu7d5c+mp5UZnwH2//xAAyEAACAQIDBgQFBAMBAAAAAAABAgMAEQQSMRATICEiQTJCUWEUI1JxgQWRocEwQ2Jy/9oACAEBAAY/AuMCSREJ9Ttkme+VBflRlliEYv02Oo4lSOPezNol7Usqi19R6HYYrtv5B05e3vU/xIU7oXz1ip8fE8rSAiNuy0Y5LkxG1/bY7ORkUc6MxiEMPk56iuh1b7Hhz4VEIkjCbxv9dA4NSZc7Z2Go561GpxEs6NydZFsy+9EPzRG0+1YnDaJPdf30qbBfqcGeMNfS9jQw36Nh8t+QOX+qwwik3uXnKubLnP3rErDG0LKcjRtqDWFw7yZOgMLU+EYQGQLfeRC1x78MuH86ANROFn3TiZ9RcHnQbHzRsB2jS1/zUoPdnFQYtJchS1xbW1Z54QX9dKPw0YjP1d65Y+VR7KKfD4QPLK8gZix5moIMUgfdqBVsPEqD24VnineCW2Ule4pIYvCtZsQ4QV8dhcTdhzKDYRhYN6FNib0SVyuvJl9P8brJn3cS+WozETYEqwtTt6C9I8cilbE9X1VOzMMmVcwHrRdzZRQzQsFNXHH805pDogrEM65TKRYCllMgkdu4qSEm8zrYKKjhbpGl6C5g5PPN61g1U6zDlTdaEX09qi3nitxzSk3zNsxCFvlBM9qeRzdmN9mGY/RSx4cdMZ8frQGMvY8s69qBTw25cT29Dtcr5kKH87bxmxy5f52pfWPo4pZGNgqmr7Ii+kiZxtxsPmjB2zf+/wCuJYx52pYywXMdTSESl83/ADWBFua8r/jbiIltkkjKtek3hVg3cbJo76re3FA/o1tnw8y5mFsrVh5IluE62/baSniVGP8AFMJemXt99i4kNlUcrfUOJD6SbBS/arxi0MnNfbYZAPlqpDfmnjkUqQe+wRy9tCKHTY8O7bm7np2w5vFkF6csQrR9Sk7J07FL1FilGnQ2yNyAQraGlK+G3LgaSRsqLqaaTyaKPanxj8luAvvUUY8zAVavhYT8tD1e52TYuYhVY6n0FSLA7fDEWynvsAqNPpUDb//EACgQAQABAwMDBAIDAQAAAAAAAAERACExEEFRIGFxgZGh4cHRMLHw8f/aAAgBAQABPyHrxwwDJoZJNBrUgZNMrLnJ1DnpKWAyrsVn2QrKFk0B2IVDRFUTBZeyelSiTFc2/FQGmL32aOEId5Bmn9hvR5XimEfMxPS5ksC0FW29NGRAm5kFJGZSc8HcoDZ6FiD+6IrHLiyaBSI3TYqVKX1AfmpR7q2mvhU8BGStvfel7NnHHbcvVjSuoDgmHplpDw5GgPJyy9jX4Wkfco7UFioFKTLlSuV2gtVPtQk9IgT7mp4T7TfeKlUCkTlaQAZRIYhhKVMWYXfL0mAD4tVPdHlyu7QRpYJ3ofOfOsROmybdheCimTSN38Yw+JIZWp/wAsReagzL/Cnp5Tky5qRyvENQvjegef2RQ7lACSNzrNiAnJf1QTbhkxBFAAhmaTxSI2siTu8VNGBlCSJz5omsZnn9VuItLIVYoDCzyrAU5OJt1LArtSc0kdjbR4M9NsJvTPk0ukw4iCva34rN65vXbtUYzd5KN4KCjEdUuda9qy8zpsb07DUNZcE7xdCnLnPoMdU95U05d2hwbBeR1DIqR2b/AL/h8zCG/wCChKADaKTEKNh70dxYPZrY6pYY4a2Aw5eNJWt85D99WbMpHyfWj+avQ81DtQG4PDV4jOcoqEVikJPodM7RGZM0QIj46ewh/Tpb5aBOJE/1VtMxGFuaTKUriFqdwew20IGxZneKvdDZgz36YFwcJxF51jnz7lFK3iQi/HrpGO2Twj91mU9jtoAIhQkSskIPh0GCBKbUrlh4VCycV/N8VlL+dqDGAiu43T/dtIVaP9G9GFEiWhvohmViv+VAa//aAAgBAQAAABAEkGEAv0DrQtABJga4E3gOIAQgO0FagxGIix6kf//EACUQAQABAwMFAQEBAQEAAAAAAAERACExQVFhECBxgZGhscEw8P/aAAgBAQABPxDtWOmZvXvgLQAgjcToopZE7AeWKFMJApnTRMc9wvrEt0lgkOVClkWQEilNRHoIFMm2IidCTGtRcJgZwsHhXKFEKIQ2AWxOGNr06GpmRInVITxHRvkMwALL5WjjTKtFi7Sr4oQqeYe3VHtEJ1pkg3KnqyaPliQ8TSwhQ1shsLkdKElkBVsRs3e6N0CMXbeliizlEYN1nRAuVINUgCLrrOaNaZVqJXBglAk0oMoWIWhSshWaQ1HGMgZVaS4bUYFgORBguZiYaOwnFGHV3PCftJSAybVjbWSl/wB4yEyqyQSM0B8BObp+FK69ZhkIRsxajQ0QmLZUT7qfv5/dlYH1QwcCRRMzLyd2QAFTa4YgTIBi1f8AutvRfb2pJ8TOW65e7QBWkWVMpyt6mciQquAvVrbnWnSTcHUjPROchgBkMsb0zoli8A/L/wDNQXfJtFIcuMbUEuBhIkknJxNJNIFGZE0qKWbxMQjVhIpAGIltKHOP7VzQCiV2A1VtFZGQDvUGPEtEwEIaj32+NFS3dBy1MhgGBIS64xFNOw4xMTIykRTgWLshJtH61nKdwVO3Zc5IphyTvA/gpEg4sCLDQSRpqDiMuKyYUpkSQ+R3E4gEtIaTa4VAcAFCmGso7cGLDS0zSTUIlusHgLdCjwbli9K0iIKUu5jpYXy08DMmAOqR6Y0p9viyoWTiO5Fv9ZKKlGqhTO89L4ZpooPxh9dCmdCUXBfwX7S1lzUXUsd4FfE+dxSkFbwgfUq0MNfetFRcTWW0XkbPUXgIriL9Piua1pEM3Bb07oThENTl/UrC56srEtQ+FiwtyieKaksJcAknZTq5AjgkiFuy/JoUHWSoDKnXo64BOhBf53JA8RqST5soYbVNw9OSC+mLTzS9CasNJMmFv1Qt2LIgTUmi6hpIAs7DiK1o2iZgtpDYLM7lEyEmVJ2sab3yHRSti/tERSwSRHClE2PED8Nzh6CpeFLoBGbw0C3C0bmE4TXoh+wNYbuKsNSbA7JaUhLjFblHUMe6csXomSM1BGCY6wzQ5YwY4JdxYN46DxOkSBP6q+0sgapW+5PZ0g9g4JuJqJNRIWQxIt+diqsvwCibFX6ax7cvmjXn3LyX0og3aFySHsFFhiHgqfmSsINPH9eOkMwFIDM+0nqi5DRkiWLhXDnoIciBysUzDKr5A/zr/9k=",
      "text/plain": [
       "<IPython.core.display.Image object>"
      ]
     },
     "execution_count": 40,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# picture\n",
    "from IPython.display import Image\n",
    "url = data['picture']\n",
    "r = requests.get(url)\n",
    "Image(data=r.content)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 49,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[\"I Can't Dance\",\n",
       " 'Jesus He Knows Me (2007 Remaster)',\n",
       " \"That's All (Platinum Collection Version)\",\n",
       " 'Mama (2007 Remaster)',\n",
       " 'Invisible Touch (2007 Remaster)',\n",
       " 'Abacab (2007 Remaster)',\n",
       " 'Hold on My Heart (Platinum Collection Version)',\n",
       " \"That's All (2007 Remaster)\",\n",
       " 'Follow You Follow Me (2007 Remaster)',\n",
       " 'No Son of Mine (2007 Remaster)',\n",
       " 'Land of Confusion (2007 Remaster)',\n",
       " 'In Too Deep (Platinum Collection Version)',\n",
       " 'Invisible Touch (Platinum Collection Version)',\n",
       " 'Land of Confusion (Platinum Collection Version)',\n",
       " 'Home by the Sea (2007 Remaster)',\n",
       " 'Firth of Fifth (2007 Stereo Mix)',\n",
       " 'No Son of Mine (Platinum Collection Version)',\n",
       " 'Tonight, Tonight, Tonight (Platinum Collection Version)',\n",
       " 'In Too Deep (2007 Remaster)',\n",
       " 'Dancing with the Moonlit Knight (2007 Stereo Mix)',\n",
       " 'Tell Me Why (2007 Remaster)',\n",
       " 'The Musical Box (2007 Stereo Mix)',\n",
       " 'Turn It on Again (2007 Remaster)',\n",
       " 'Entangled (2007 Remaster)',\n",
       " 'I Know What I Like (In Your Wardrobe) (2007 Stereo Mix)',\n",
       " 'Driving the Last Spike (2007 Remaster)',\n",
       " 'Second Home by the Sea (2007 Remaster)',\n",
       " 'Throwing It All Away (2007 Remaster)',\n",
       " 'Dance on a Volcano (2007 Remaster)',\n",
       " 'The Carpet Crawlers (2007 Stereo Mix)',\n",
       " 'The Cinema Show (2007 Stereo Mix)',\n",
       " \"Supper's Ready (2007 Stereo Mix)\",\n",
       " 'Duchess (2007 Remaster)',\n",
       " 'Home by the Sea (Platinum Collection Version)',\n",
       " 'Looking for someone',\n",
       " 'The Lamb Lies Down on Broadway (2007 Stereo Mix)',\n",
       " 'Illegal Alien (2007 Remaster)',\n",
       " 'Horizons (2007 Stereo Mix)',\n",
       " 'Mama (Live in Frankfurt)',\n",
       " 'Behind the Lines (2007 Remaster)',\n",
       " 'Dreaming While You Sleep (2007 Remaster)',\n",
       " 'Mama (Live)',\n",
       " 'Eleventh Earl of Mar (2007 Remaster)',\n",
       " 'A Trick of the Tail (2007 Remaster)',\n",
       " 'Throwing It All Away (Platinum Collection Version)',\n",
       " 'The Battle of Epping Forest (2007 Stereo Mix)',\n",
       " 'Watcher of the Skies (2007 Stereo Mix)',\n",
       " 'Never a Time (2007 Remaster)',\n",
       " 'Fading Lights (2007 Remaster)',\n",
       " 'No Reply at All (2007 Remaster)']"
      ]
     },
     "execution_count": 49,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "r = requests.get(data[\"tracklist\"])\n",
    "d = r.json()\n",
    "list_music = [dico['title'] for dico in d['data']]\n",
    "list_music"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 46,
   "metadata": {},
   "outputs": [],
   "source": [
    "from IPython import display\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 50,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<video src=\"https://cdnt-preview.dzcdn.net/api/1/1/1/d/0/0/1d0c4a760dac43444a36faf203415e03.mp3?hdnea=exp=1734623279~acl=/api/1/1/1/d/0/0/1d0c4a760dac43444a36faf203415e03.mp3*~data=user_id=0,application_id=42~hmac=473ae090a9c6bf073d22838bb2c565f88c7677a6c9eed46f24a7d1c0d60fe77c\" controls  >\n",
       "      Your browser does not support the <code>video</code> element.\n",
       "    </video>"
      ],
      "text/plain": [
       "<IPython.core.display.Video object>"
      ]
     },
     "execution_count": 50,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "display.Video(url=d[\"data\"][list_music.index('Duchess (2007 Remaster)')]['preview'])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Avantages d'une API\n",
    "- renvoie du format JSON, facile et rapide à traiter\n",
    "- renvoie un format stable et documenté (voire versionné)\n",
    "- exemple : https://developer.twitter.com/en/docs/twitter-api/api-reference-index\n",
    "- la documentation indique comment interagir avec l'API:\n",
    "    - quelle url\n",
    "    - quelle méthode http (GET, POST, ...)\n",
    "    - quels paramètres\n",
    "    - ...\n",
    "→ idéal pour les développeurs"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Quel intérêt pour le fournisseur d'API ?\n",
    "\n",
    "En général il met en place des quotas de requêtes ou d'autres limitations afin de proposer un service payant qui dispose de possibilités avancées / d'un meilleur support / etc.\n",
    "\n",
    "C'est pourquoi de nombreux services nécessitent de se connecter avec son compte client pour utiliser une API (e.g. https://openweathermap.org/api)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### **Basic Auth**\n",
    "\n",
    "Exemple: accéder à https://kim.fspot.org/private/\n",
    "\n",
    "Pour y accéder il est nécessaire d'utiliser les credentials suivant:\n",
    "- login: admin\n",
    "- password: secret\n",
    "\n",
    "Si on ne les passe pas (ou si on ne passe pas les bons), on a une erreur 401 (= unauthorized)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 51,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<Response [401]>"
      ]
     },
     "execution_count": 51,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# sans login/password\n",
    "res = requests.get('https://kim.fspot.org/private')\n",
    "res"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 52,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<Response [200]>"
      ]
     },
     "execution_count": 52,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# avec login password\n",
    "res = requests.get('https://kim.fspot.org/private', auth=('admin', 'secret'))\n",
    "res"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 53,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'<!doctype html>\\n<html>\\n<head>\\n<title>Success</title>\\n</head>\\n<body>\\n\\t<h1>Well done!</h1>\\n</body>\\n</html>\\n'"
      ]
     },
     "execution_count": 53,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# contenu\n",
    "res.content.decode('utf-8')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# YAML pour masquer auth crendentials"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Auth par token\n",
    "\n",
    "Exemple sur openweathermap :\n",
    "- documentation: https://openweathermap.org/appid\n",
    "- mes tokens: https://home.openweathermap.org/api_keys"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 55,
   "metadata": {},
   "outputs": [],
   "source": [
    "# requête avec un token\n",
    "token = ''"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Avantage des tokens:\n",
    "- évite que les requêtes HTTP contiennent le mot de passe - à la place elles contiennent un token\n",
    "- si je me fais \"voler\" un token, je peux le supprimer de mon compte"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 60,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{'coord': {'lon': 2.3488, 'lat': 48.8534},\n",
       " 'weather': [{'id': 801,\n",
       "   'main': 'Clouds',\n",
       "   'description': 'few clouds',\n",
       "   'icon': '02d'}],\n",
       " 'base': 'stations',\n",
       " 'main': {'temp': 279.84,\n",
       "  'feels_like': 276.13,\n",
       "  'temp_min': 278.85,\n",
       "  'temp_max': 280.58,\n",
       "  'pressure': 1013,\n",
       "  'humidity': 82,\n",
       "  'sea_level': 1013,\n",
       "  'grnd_level': 1003},\n",
       " 'visibility': 10000,\n",
       " 'wind': {'speed': 6.17, 'deg': 280},\n",
       " 'clouds': {'all': 20},\n",
       " 'dt': 1734622415,\n",
       " 'sys': {'type': 2,\n",
       "  'id': 2012208,\n",
       "  'country': 'FR',\n",
       "  'sunrise': 1734594027,\n",
       "  'sunset': 1734623716},\n",
       " 'timezone': 3600,\n",
       " 'id': 2988507,\n",
       " 'name': 'Paris',\n",
       " 'cod': 200}"
      ]
     },
     "execution_count": 60,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# requête avec un token\n",
    "url = f'http://api.openweathermap.org/data/2.5/weather?APPID={token}&q=Paris'\n",
    "res = requests.get(url)\n",
    "meteo = res.json()\n",
    "meteo"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 57,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "dict"
      ]
     },
     "execution_count": 57,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# type\n",
    "type(meteo)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 58,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{'city': 'Paris',\n",
       " 'country': 'FR',\n",
       " 'date': 1734622415,\n",
       " 'temp': 6.689999999999998}"
      ]
     },
     "execution_count": 58,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# extractions\n",
    "{'city': meteo['name'],\n",
    "'country': meteo['sys']['country'],\n",
    "'date': meteo['dt'],\n",
    "'temp': meteo['main']['temp'] - 273.15,}"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 64,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{'city': 'Paris',\n",
       " 'country': 'FR',\n",
       " 'date': '2024-12-19 16:33:35 December Thursday',\n",
       " 'sunrise': '2024-12-19 08:40:27',\n",
       " 'sunset': '2024-12-19 16:55:16',\n",
       " 'temp': 6.689999999999998}"
      ]
     },
     "execution_count": 64,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# extractions\n",
    "import time\n",
    "\n",
    "{'city': meteo['name'],\n",
    "'country': meteo['sys']['country'],\n",
    "'date': time.strftime('%Y-%m-%d %H:%M:%S %B %A', time.localtime(meteo['dt'])),\n",
    "'sunrise': time.strftime('%Y-%m-%d %H:%M:%S', time.localtime(meteo['sys']['sunrise'])),\n",
    "'sunset': time.strftime('%Y-%m-%d %H:%M:%S', time.localtime(meteo['sys']['sunset'])),\n",
    "'temp': meteo['main']['temp'] - 273.15,}"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 67,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'fr_FR.UTF-8'"
      ]
     },
     "execution_count": 67,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "import locale\n",
    "\n",
    "# mettre parfois fr_FR ou fr_FR.UTF-8 au lieu de fr, etc.\\n\",\n",
    "locale.setlocale(locale.LC_ALL, 'fr_FR.UTF-8') # français"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 68,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{'city': 'Paris',\n",
       " 'country': 'FR',\n",
       " 'date': '2024-12-19 16:33:35 décembre jeudi',\n",
       " 'sunrise': '2024-12-19 08:40:27',\n",
       " 'sunset': '2024-12-19 16:55:16',\n",
       " 'temp': 6.689999999999998}"
      ]
     },
     "execution_count": 68,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "{'city': meteo['name'],\n",
    "'country': meteo['sys']['country'],\n",
    "'date': time.strftime('%Y-%m-%d %H:%M:%S %B %A', time.localtime(meteo['dt'])),\n",
    "'sunrise': time.strftime('%Y-%m-%d %H:%M:%S', time.localtime(meteo['sys']['sunrise'])),\n",
    "'sunset': time.strftime('%Y-%m-%d %H:%M:%S', time.localtime(meteo['sys']['sunset'])),\n",
    "'temp': meteo['main']['temp'] - 273.15,}"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 75,
   "metadata": {},
   "outputs": [],
   "source": [
    "import datetime"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 78,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "datetime.datetime(2024, 12, 19, 16, 51, 3, 31613)"
      ]
     },
     "execution_count": 78,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "datetime.datetime.now()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 80,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "datetime.date(2024, 12, 19)"
      ]
     },
     "execution_count": 80,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "datetime.date.today()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 81,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "datetime.date(2025, 3, 29)"
      ]
     },
     "execution_count": 81,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "datetime.date.today() + datetime.timedelta(days=100)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 82,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "datetime.date(2024, 9, 10)"
      ]
     },
     "execution_count": 82,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "datetime.date.today() - datetime.timedelta(days=100)"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.11.9"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}