<commit_before>{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 練習\n",
    "\n",
    "- 模擬 google search 流程\n",
    "- https://www.google.com.tw/\n",
    "- 搜尋「人工智慧」\n",
    "- 紀錄前兩頁搜尋結果的連結"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "import os\n",
    "\n",
    "from selenium import webdriver\n",
    "from selenium.webdriver.common.keys import Keys\n",
    "from selenium.webdriver.common.by import By\n",
    "from pprint import pprint\n",
    "\n",
    "url = 'https://google.com'"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "======================================================================================= Page 0\n",
      "title: 人工智能- 维基百科，自由的百科全书\n",
      "url: https://zh.wikipedia.org/zh-tw/%E4%BA%BA%E5%B7%A5%E6%99%BA%E8%83%BD\n",
      "---\n",
      "title: AI 人工智慧| TechNews 科技新報\n",
      "url: https://technews.tw/category/cutting-edge/ai/\n",
      "---\n",
      "title: 人工智慧三大關鍵技術 - 數位時代\n",
      "url: https://www.bnext.com.tw/article/41534/3-key-techniques-of-ai\n",
      "---\n",
      "title: 人工智慧不可能超越人類，原因居然是這樣的……（上） ｜智慧機器人網 ...\n",
      "url: https://www.limitlessiq.com/news/post/view/id/3596/\n",
      "---\n",
      "title: 人工智慧AI – CASE報科學 - 國立臺灣大學科學教育發展中心\n",
      "url: https://case.ntu.edu.tw/blog/?cat=3772\n",
      "---\n",
      "title: AI人工智慧來了! 你的未來在哪裡？｜深度專題｜天下雜誌\n",
      "url: https://www.cw.com.tw/special/2073\n",
      "---\n",
      "title: 台灣人工智慧學校| Taiwan AI Academy\n",
      "url: http://aiacademy.tw/\n",
      "---\n",
      "title: 人工智慧：搜尋方法與邏輯推論(Artificial Intelligence - Search & Logic ...\n",
      "url: https://www.coursera.org/learn/rengong-zhineng\n",
      "---\n",
      "======================================================================================= Page 1\n",
      "title: 什麼是人工智慧？︱《三分鐘財經教室》#01 - YouTube\n",
      "url: https://www.youtube.com/watch?v=nKcsu4JierI\n",
      "---\n",
      "title: 人工智慧：搜尋方法與邏輯推論(Artificial Intelligence - Search & Logic ...\n",
      "url: https://www.coursera.org/learn/rengong-zhineng\n",
      "---\n",
      "title: 博客來-中文書>電腦資訊>概論/科技趨勢>人工智慧/機器學習\n",
      "url: https://www.books.com.tw/web/sys_bbotm/books/190102\n",
      "---\n",
      "title: 人工智能| 大紀元\n",
      "url: http://www.epochtimes.com/b5/tag/%E4%BA%BA%E5%B7%A5%E6%99%BA%E8%83%BD.html\n",
      "---\n",
      "title: 人工智慧對勞動就業的影響- STPI Research Portal - 科技政策觀點\n",
      "url: https://portal.stpi.narl.org.tw/index/article/10401\n",
      "---\n",
      "title: 關鍵字(人工智慧) - 104人力銀行工作列表\n",
      "url: https://www.104.com.tw/jobs/search/?ro=0&kwop=7&keyword=%E4%BA%BA%E5%B7%A5%E6%99%BA%E6%85%A7&order=1&asc=0&mode=s&page=1\n",
      "---\n",
      "title: 如何對面AI時代的孩子？人工智慧博士教年輕父母強化個人實力- Yahoo ...\n",
      "url: https://tw.news.yahoo.com/%E5%A6%82%E4%BD%95%E5%B0%8D%E9%9D%A2ai%E6%99%82%E4%BB%A3%E7%9A%84%E5%AD%A9%E5%AD%90-%E4%BA%BA%E5%B7%A5%E6%99%BA%E6%85%A7%E5%8D%9A%E5%A3%AB%E6%95%99%E5%B9%B4%E8%BC%95%E7%88%B6%E6%AF%8D%E5%BC%B7%E5%8C%96%E5%80%8B%E4%BA%BA%E5%AF%A6%E5%8A%9B-010012301.html\n",
      "---\n",
      "title: 為什麼人工智慧公司，要開始像瘋子一樣狂砸玻璃？ | TechOrange\n",
      "url: https://buzzorange.com/techorange/2018/10/17/artificial-audio-intelligence/\n",
      "---\n",
      "title: 人工智慧應用新趨勢與展望—學生與機器人共同學習-臺北產經資訊網\n",
      "url: https://www.taipeiecon.taipei/article_cont.aspx?MmmID=1201&MSid=1001302007727155764\n",
      "---\n",
      "title: 台灣國際人工智慧暨物聯網展| 展覽活動一覽| 台北南港展覽館1館\n",
      "url: http://www.twtcnangang.com.tw/zh-tw/calendar/detail/281\n",
      "---\n"
     ]
    }
   ],
   "source": [
    "try:\n",
    "    driver = webdriver.Chrome()\n",
    "    driver.get(url)\n",
    "    driver.maximize_window()\n",
    "    driver.implicitly_wait(10)\n",
    "    \n",
    "    search_input  = driver.find_element(By.ID, 'lst-ib')\n",
    "    search_input.send_keys(u'人工智慧')\n",
    "    search_input.send_keys(Keys.ENTER)\n",
    "    \n",
    "    for i in range(2):\n",
    "        print('='*87, 'Page {}'.format(i))\n",
    "\n",
    "        links = driver.find_elements(By.XPATH, '//div[@class=\"r\"]/a[@href]')\n",
    "\n",
    "        for link in links:\n",
    "            page_title = link.find_element(By.TAG_NAME, 'h3').text\n",
    "            page_url = ''\n",
    "\n",
    "            if link.get_attribute('href'):\n",
    "                page_url = link.get_attribute('href')\n",
    "\n",
    "            print('title: {}\\nurl: {}\\n---'.format(page_title, page_url))\n",
    "\n",
    "        next_page = driver.find_element(By.XPATH, '//*[@id=\"pnnext\"]/span[2]').click()\n",
    "\n",
    "except Exception as e:\n",
    "    print(e)\n",
    "finally:\n",
    "    driver.quit()"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.6"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}
<commit_msg>update google search page<commit_after>{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 練習\n",
    "\n",
    "- 模擬 google search 流程\n",
    "- https://www.google.com.tw/\n",
    "- 搜尋「人工智慧」\n",
    "- 紀錄前兩頁搜尋結果的連結"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "import os\n",
    "\n",
    "from selenium import webdriver\n",
    "from selenium.webdriver.common.keys import Keys\n",
    "from selenium.webdriver.common.by import By\n",
    "from pprint import pprint\n",
    "\n",
    "url = 'https://google.com'"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "======================================================================================= Page 0\n",
      "title: 人工智能- 维基百科，自由的百科全书\n",
      "url: https://zh.wikipedia.org/zh-tw/%E4%BA%BA%E5%B7%A5%E6%99%BA%E8%83%BD\n",
      "---\n",
      "title: 人工智慧三大關鍵技術 - 數位時代\n",
      "url: https://www.bnext.com.tw/article/41534/3-key-techniques-of-ai\n",
      "---\n",
      "title: AI 人工智慧| TechNews 科技新報\n",
      "url: https://technews.tw/category/cutting-edge/ai/\n",
      "---\n",
      "title: 人工智慧不可能超越人類，原因居然是這樣的……（上） ｜智慧機器人網 ...\n",
      "url: https://www.limitlessiq.com/news/post/view/id/3596/\n",
      "---\n",
      "title: 人工智慧AI – CASE報科學 - 國立臺灣大學科學教育發展中心\n",
      "url: https://case.ntu.edu.tw/blog/?cat=3772\n",
      "---\n",
      "title: AI人工智慧來了! 你的未來在哪裡？｜深度專題｜天下雜誌\n",
      "url: https://www.cw.com.tw/special/2073\n",
      "---\n",
      "title: 台灣人工智慧學校| Taiwan AI Academy\n",
      "url: http://aiacademy.tw/\n",
      "---\n",
      "title: 人工智慧- MBA智库百科\n",
      "url: https://wiki.mbalib.com/zh-tw/%E4%BA%BA%E5%B7%A5%E6%99%BA%E8%83%BD\n",
      "---\n",
      "======================================================================================= Page 1\n",
      "title: 什麼是人工智慧？︱《三分鐘財經教室》#01 - YouTube\n",
      "url: https://www.youtube.com/watch?v=nKcsu4JierI\n",
      "---\n",
      "title: 『AI人工智慧！機器學習& 突如其來的危機』芬特克FinTech EP3 - YouTube\n",
      "url: https://www.youtube.com/watch?v=i0UxYDqlX6o\n",
      "---\n",
      "title: 人工智慧：搜尋方法與邏輯推論(Artificial Intelligence - Search & Logic ...\n",
      "url: https://www.coursera.org/learn/rengong-zhineng\n",
      "---\n",
      "title: 博客來-中文書>電腦資訊>概論/科技趨勢>人工智慧/機器學習\n",
      "url: https://www.books.com.tw/web/sys_bbotm/books/190102\n",
      "---\n",
      "title: 人工智能| 大紀元\n",
      "url: http://www.epochtimes.com/b5/tag/%E4%BA%BA%E5%B7%A5%E6%99%BA%E8%83%BD.html\n",
      "---\n",
      "title: 如何對面AI時代的孩子？人工智慧博士教年輕父母強化個人實力- Yahoo ...\n",
      "url: https://tw.news.yahoo.com/%E5%A6%82%E4%BD%95%E5%B0%8D%E9%9D%A2ai%E6%99%82%E4%BB%A3%E7%9A%84%E5%AD%A9%E5%AD%90-%E4%BA%BA%E5%B7%A5%E6%99%BA%E6%85%A7%E5%8D%9A%E5%A3%AB%E6%95%99%E5%B9%B4%E8%BC%95%E7%88%B6%E6%AF%8D%E5%BC%B7%E5%8C%96%E5%80%8B%E4%BA%BA%E5%AF%A6%E5%8A%9B-010012301.html\n",
      "---\n",
      "title: 人工智慧對勞動就業的影響- STPI Research Portal - 科技政策觀點\n",
      "url: https://portal.stpi.narl.org.tw/index/article/10401\n",
      "---\n",
      "title: AI人工智慧時代來臨- 中時電子報\n",
      "url: https://www.chinatimes.com/newspapers/20180907000541-260204\n",
      "---\n",
      "title: 人工智慧應用新趨勢與展望—學生與機器人共同學習-臺北產經資訊網\n",
      "url: https://www.taipeiecon.taipei/article_cont.aspx?MmmID=1201&MSid=1001302007727155764\n",
      "---\n",
      "title: 人工智慧技術的下一波研發核心 - Digitimes\n",
      "url: https://www.digitimes.com.tw/col/article.asp?id=944\n",
      "---\n"
     ]
    }
   ],
   "source": [
    "try:\n",
    "    driver = webdriver.Chrome('/home/afun/Downloads/chromedriver')\n",
    "    driver.get(url)\n",
    "    driver.maximize_window()\n",
    "    driver.implicitly_wait(10)\n",
    "    \n",
    "    search_input  = driver.find_element(By.ID, 'lst-ib')\n",
    "    search_input.send_keys(u'人工智慧')\n",
    "    search_input.send_keys(Keys.ENTER)\n",
    "    \n",
    "    for i in range(2):\n",
    "        print('='*87, 'Page {}'.format(i))\n",
    "\n",
    "        links = driver.find_elements(By.XPATH, '//div[@class=\"r\"]/a[@href]')\n",
    "\n",
    "        for link in links:\n",
    "            page_title = link.find_element(By.TAG_NAME, 'h3').text\n",
    "            page_url = ''\n",
    "\n",
    "            if link.get_attribute('href'):\n",
    "                page_url = link.get_attribute('href')\n",
    "\n",
    "            print('title: {}\\nurl: {}\\n---'.format(page_title, page_url))\n",
    "\n",
    "        next_page = driver.find_element(By.XPATH, '//*[@id=\"pnnext\"]/span[2]').click()\n",
    "\n",
    "except Exception as e:\n",
    "    print(e)\n",
    "finally:\n",
    "    driver.quit()"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.6"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}

{'fraction_non_alphanumeric': 0.2632077258095058, 'fraction_numerical': 0.04128006059458436, 'mean_word_length': 2.488441215323646, 'pattern_counts': {'":': 68, '<': 3, '<?xml version=': 0, '>': 9, 'https://': 35, 'lorem ipsum': 0, 'www.': 24, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 1, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '4812646', 'n_tokens_mistral': 5376, 'n_tokens_neox': 5348, 'n_words': 543}