Iphone project

Marinskiy/01. Data Collection.ipynb

@@ -0,0 +1,331 @@
+{
+ "cells": [
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "# iPhone Project"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "*by Alexander Marinskiy*"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "## Part 1. Data Collection"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "For data collection, I decided to use the site avito.ru. Currently, over a million ads are published in the \"phones\" category, and each ad can contain up to 10 photos. Thus, the amount of dataset available is larger than we can theoretically process. Moreover, these are photos taken by the users themselves, which corresponds to the data on which the model will be tested."
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "### Step 1. Build functions to collect images"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "# import libraries\n",
+    "import pandas as pd\n",
+    "import requests\n",
+    "import time\n",
+    "from bs4 import BeautifulSoup\n",
+    "import os\n",
+    "import urllib.request "
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "# function to get web page from url\n",
+    "def get_html(url):\n",
+    "    \n",
+    "    # set user agent\n",
+    "    user_agent = 'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_9_3) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/35.0.1916.47 Safari/537.36'\n",
+    "\n",
+    "    # get web page from url\n",
+    "    r = requests.get(url, headers={'User-Agent': user_agent})#, proxies=proxy)\n",
+    "  \n",
+    "    # return text of web page\n",
+    "    return r.text"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "# function to get links to all the product images on the page \n",
+    "def get_links_from_page(html):\n",
+    "    \n",
+    "    # create soup\n",
+    "    soup = BeautifulSoup(html, 'lxml')\n",
+    "\n",
+    "    # get links to all the images\n",
+    "    images = [x['src'] for x in soup.findAll('img', {'class': 'large-picture-img'})]\n",
+    "    \n",
+    "    # filter only the images we need\n",
+    "    product_photo = []\n",
+    "    for i in images:\n",
+    "        if i[:9] != 'https://w':\n",
+    "            product_photo.append(i)\n",
+    "\n",
+    "    # return list with links to images\n",
+    "    return product_photo"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "# function to get list of links to pictures of iphone\n",
+    "def get_all_links(name, base_url, n_pages=10, query=''):\n",
+    "    \n",
+    "    # Construct url adress. \n",
+    "    # iPhones: https://www.avito.ru/rossiya/telefony/iphone?p=1&q=iphone+x\n",
+    "    # Other: https://www.avito.ru/rossiya/telefony/alcatel?p=1\n",
+    "    \n",
+    "    # if downloading iphones we specify model\n",
+    "    if query != '':\n",
+    "        query = '&q=iphone+' + query\n",
+    "    \n",
+    "    # get links to images from all the pages\n",
+    "    links_list = []\n",
+    "    for i in range(1, n_pages+1):\n",
+    "        url_gen = base_url + 'p=' + str(i) + query\n",
+    "        print(url_gen)\n",
+    "        page_html = get_html(url_gen)\n",
+    "        links_list += get_links_from_page(page_html)\n",
+    "        \n",
+    "        # wait for 5 second in order to avoid block from avito\n",
+    "        time.sleep(5)\n",
+    "\n",
+    "    # save list of links to csv\n",
+    "    df = pd.DataFrame()\n",
+    "    df['links'] = links_list\n",
+    "    df.to_csv(name+'.csv')"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "# get images\n",
+    "def get_images(subfolder, models):\n",
+    "    for model in models[:]:\n",
+    "        # create folder\n",
+    "        if not os.path.exists('dataset/' + subfolder + '/' + model):\n",
+    "            os.makedirs('dataset/' + subfolder + '/' + model)\n",
+    "\n",
+    "        # read list of links\n",
+    "        df = pd.read_csv(model+'.csv')\n",
+    "\n",
+    "        # print info\n",
+    "        print('Downloading ' + model + '. Total number of photos: ' + str(len(df['links'])))\n",
+    "\n",
+    "        # getting photos\n",
+    "        count = 0\n",
+    "        for i in df['links']:         \n",
+    "            count+= 1\n",
+    "            \n",
+    "            # print information massege every 100 photos\n",
+    "            if count % 100 == 0: \n",
+    "                print(str(count) + ' done')\n",
+    "\n",
+    "            # get the image\n",
+    "            try:\n",
+    "                urllib.request.urlretrieve(i, 'dataset/' + subfolder + '/' + model + '/' + str(count) + '.jpg')\n",
+    "            except:\n",
+    "                print('Skip photo ' + str(count) + ' due to error')\n",
+    "\n",
+    "            # wait for 0.5 second to avoid ban\n",
+    "            time.sleep(0.5)"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "### Step 2. Collect images of iPhones"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "Since we need to learn to recognize all the iPhone models that exist on the market, we will upload photos of these iPhones in equal proportions."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "# list of models\n",
+    "iphone_models = ['XR', 'XS', 'X', '8', '7', 'SE', '6S', '6', '5S', '5C', '5', '4S', '4', '3GS', '3G']"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "# create lists of links for all te models\n",
+    "for model in iphone_models:\n",
+    "    print('Getting links for model', model)\n",
+    "    get_all_links(name=model, base_url='https://www.avito.ru/rossiya/telefony/iphone?', n_pages=12, query=model)"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "get_images('iphone', iphone_models)"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "### Step 3. Get images of non-iphones"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "Since there are only 15 iPhone models, it was reasonable to upload the same number of photos for each model. In the case of non-iPhones, there are much more models, so another strategy was applied. I looked at the number of ads for each of the manufacturers on Avito and decided to upload photos in appropriate proportions. Thus, our dataset will reflect as closely as possible the conditions in which the model will be tested. The proportions have been saved in the non-iphones.xlsx file."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "# check what number of avito pages we need to download\n",
+    "df_non_iphones = pd.read_excel('non-iphones.xlsx')\n",
+    "df_non_iphones['n_pages'] = df_non_iphones['n_pages'].apply(int)\n",
+    "df_non_iphones['brand'] = df_non_iphones['brand'].apply(str)\n",
+    "df_non_iphones"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "# create lists of links for all te models\n",
+    "for i in range(len(df_non_iphones['brand'])):\n",
+    "    print('Getting links for model', df_non_iphones['brand'][i])\n",
+    "    get_all_links(name=df_non_iphones['brand'][i],\n",
+    "                  base_url='https://www.avito.ru/rossiya/telefony/' + df_non_iphones['brand'][i] + '?', \n",
+    "                  n_pages=df_non_iphones['n_pages'][i])"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "get_images('other', other_models)"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "Thus, I managed to collect a balanced dataset from 50,000 images of iPhones and phones from other manufacturers. Then this dataset was merged with the dataset collected by Anton Anisimov. The model was trained in a combined dataset.\n",
+    "\n",
+    "It is important to note that more than 1,000,000 phone advertisements are available on Avito, and in each ad there are 2-4 photos. Thus, our dataset can be painlessly increased by a factor of over sixty, which could significantly improve the accuracy of the model, but also would require more computational resources and time for training."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": []
+  }
+ ],
+ "metadata": {
+  "kernelspec": {
+   "display_name": "Python 3",
+   "language": "python",
+   "name": "python3"
+  },
+  "language_info": {
+   "codemirror_mode": {
+    "name": "ipython",
+    "version": 3
+   },
+   "file_extension": ".py",
+   "mimetype": "text/x-python",
+   "name": "python",
+   "nbconvert_exporter": "python",
+   "pygments_lexer": "ipython3",
+   "version": "3.6.8"
+  },
+  "varInspector": {
+   "cols": {
+    "lenName": 16,
+    "lenType": 16,
+    "lenVar": 40
+   },
+   "kernels_config": {
+    "python": {
+     "delete_cmd_postfix": "",
+     "delete_cmd_prefix": "del ",
+     "library": "var_list.py",
+     "varRefreshCmd": "print(var_dic_list())"
+    },
+    "r": {
+     "delete_cmd_postfix": ") ",
+     "delete_cmd_prefix": "rm(",
+     "library": "var_list.r",
+     "varRefreshCmd": "cat(var_dic_list()) "
+    }
+   },
+   "types_to_exclude": [
+    "module",
+    "function",
+    "builtin_function_or_method",
+    "instance",
+    "_Feature"
+   ],
+   "window_display": false
+  }
+ },
+ "nbformat": 4,
+ "nbformat_minor": 2
+}