Web Scraper - Completed

Gemfile

@@ -0,0 +1,8 @@
+# frozen_string_literal: true
+# A sample Gemfile
+source "https://rubygems.org"
+
+# gem "rails"
+  gem 'mechanize', '~>2.7.2', '<2.7.3'
+  gem 'pry' # because you'll probably need it
+  gem 'nokogiri', '~>1.6.7.2'

Gemfile.lock

@@ -0,0 +1,45 @@
+GEM
+  remote: https://rubygems.org/
+  specs:
+    coderay (1.1.1)
+    domain_name (0.5.20161021)
+      unf (>= 0.0.5, < 1.0.0)
+    http-cookie (1.0.3)
+      domain_name (~> 0.5)
+    mechanize (2.7.2)
+      domain_name (~> 0.5, >= 0.5.1)
+      http-cookie (~> 1.0.0)
+      mime-types (~> 1.17, >= 1.17.2)
+      net-http-digest_auth (~> 1.1, >= 1.1.1)
+      net-http-persistent (~> 2.5, >= 2.5.2)
+      nokogiri (~> 1.4)
+      ntlm-http (~> 0.1, >= 0.1.1)
+      webrobots (>= 0.0.9, < 0.2)
+    method_source (0.8.2)
+    mime-types (1.25.1)
+    mini_portile2 (2.0.0)
+    net-http-digest_auth (1.4)
+    net-http-persistent (2.9.4)
+    nokogiri (1.6.7.2)
+      mini_portile2 (~> 2.0.0.rc2)
+    ntlm-http (0.1.1)
+    pry (0.10.4)
+      coderay (~> 1.1.0)
+      method_source (~> 0.8.1)
+      slop (~> 3.4)
+    slop (3.6.0)
+    unf (0.1.4)
+      unf_ext
+    unf_ext (0.0.7.2)
+    webrobots (0.1.2)
+
+PLATFORMS
+  ruby
+
+DEPENDENCIES
+  mechanize (~> 2.7.2, < 2.7.3)
+  nokogiri (~> 1.6.7.2)
+  pry
+
+BUNDLED WITH
+   1.12.5

README.md

@@ -2,3 +2,5 @@
 Bottom feeding your way across the web
 
 [A Ruby-based web scraping and parsing project which uses NokoGiri, Mechanize, Open-uri.  From the Viking Code School.](http://www.vikingcodeschool.com)
+
+Completed by Pramod Jacob and Luke Schleicher

lib/csv_file.csv

@@ -0,0 +1,30 @@
+Ruby Developer,Kane Partners Staffing,https://www.dice.com/jobs/detail/Ruby-Developer-Kane-Partners-Staffing-Philadelphia-PA-19103/10195207/887420?icid=sr1-1p&q=ruby developer&l=philadelphia,"Philadelphia, PA",Posted 6 days ago
+Ruby Developer (Rubyist) - W2,"Computer Enterprises, Inc.",https://www.dice.com/jobs/detail/Ruby-Developer-%28Rubyist%29-%26%2345-W2-Computer-Enterprises%2C-Inc.-Philadelphia-PA-19103/ceiam/AKJO037524?icid=sr2-1p&q=ruby developer&l=philadelphia,"Philadelphia, PA",Posted 9 hours ago
+QA Automation Tester(iOS/Android/Ruby),Cygnus Professionals,https://www.dice.com/jobs/detail/QA-Automation-Tester%28iOS%26%2347Android%26%2347Ruby%29-Cygnus-Professionals-Philadelphia-PA-19092/10426508/QAIAR-MWK?icid=sr3-1p&q=ruby developer&l=philadelphia,"Philadelphia, PA",Posted 9 hours ago
+"Web Developer (Assoc/Mid-level/Senior), School of Medicine",Penn Medicine,https://www.dice.com/jobs/detail/Web-Developer-%28Assoc%26%2347Mid%26%2345level%26%2347Senior%29%2C-School-of-Medicine-Penn-Medicine-Philadelphia-PA-19019/10303860/60567?icid=sr4-1p&q=ruby developer&l=philadelphia,"Philadelphia, PA",Posted 10 hours ago
+Data Science Software Engineer,Penn Medicine,https://www.dice.com/jobs/detail/Data-Science-Software-Engineer-Penn-Medicine-Philadelphia-PA-19019/10303860/62577?icid=sr5-1p&q=ruby developer&l=philadelphia,"Philadelphia, PA",Posted 10 hours ago
+Software Engineer - Python / Django - Civic Apps,Bebee Affinity Social Network,https://www.dice.com/jobs/detail/Software-Engineer-%26%2345-Python-%26%2347-Django-%26%2345-Civic-Apps-Bebee-Affinity-Social-Network-Philadelphia-PA-19019/90940991/BBA000013042768?icid=sr6-1p&q=ruby developer&l=philadelphia,"Philadelphia, PA",Posted 13 hours ago
+Software Developer,Volt,https://www.dice.com/jobs/detail/Software-Developer-Volt-Philadelphia-PA-19019/cxvolt/J3L14X6W50Q8TYNLPKP?icid=sr7-1p&q=ruby developer&l=philadelphia,"Philadelphia, PA",Posted 11 hours ago
+Front End Developer - JavaScript without Frameworks,"Computer Enterprises, Inc.",https://www.dice.com/jobs/detail/Front-End-Developer-%26%2345-JavaScript-without-Frameworks-Computer-Enterprises%2C-Inc.-Philadelphia-PA-19103/ceiam/AKJO037843?icid=sr8-1p&q=ruby developer&l=philadelphia,"Philadelphia, PA",Posted 6 days ago
+"Principal Engineer, Software Development & Engineering",Bebee Affinity Social Network,https://www.dice.com/jobs/detail/Principal-Engineer%2C-Software-Development-%26-Engineering-Bebee-Affinity-Social-Network-Philadelphia-PA-19019/90940991/BBA000013042811?icid=sr9-1p&q=ruby developer&l=philadelphia,"Philadelphia, PA",Posted 13 hours ago
+Infrastructure - Technical Software Support Engineer III,Business One Consulting,https://www.dice.com/jobs/detail/Infrastructure-%26%2345-Technical-Software-Support-Engineer-III-Business-One-Consulting-Philadelphia-PA-19109/10195908/ZR_380_JOB?icid=sr10-1p&q=ruby developer&l=philadelphia,"Philadelphia, PA",Posted 4 weeks ago
+Mobile Automation Test Manager,Cognizant Technology Solutions,https://www.dice.com/jobs/detail/Mobile-Automation-Test-Manager-Cognizant-Technology-Solutions-Philadelphia-PA-19019/aces/00020097191?icid=sr11-1p&q=ruby developer&l=philadelphia,"Philadelphia, PA",Posted 9 hours ago
+DevOps Engineer,"e&e Technical Consultants, LLC",https://www.dice.com/jobs/detail/DevOps-Engineer-e%26e-Technical-Consultants%2C-LLC-Philadelphia-PA-19019/90835421/740848?icid=sr12-1p&q=ruby developer&l=philadelphia,"Philadelphia, PA",Posted 1 day ago
+IOS Automation Tester,Benvia,https://www.dice.com/jobs/detail/IOS-Automation-Tester-Benvia-Philadelphia-PA-19019/10119901/777973?icid=sr13-1p&q=ruby developer&l=philadelphia,"Philadelphia, PA",Posted 4 hours ago
+Infrastructure Automation Engineer II,MDT Technical,https://www.dice.com/jobs/detail/Infrastructure-Automation-Engineer-II-MDT-Technical-Philadelphia-PA-19103/10116487/9597?icid=sr14-1p&q=ruby developer&l=philadelphia,"Philadelphia, PA",Posted 1 week ago
+iOS Andriod Automation Engineeer,MDT Technical,https://www.dice.com/jobs/detail/iOS-Andriod-Automation-Engineeer-MDT-Technical-Philadelphia-PA-19102/10116487/9798?icid=sr15-1p&q=ruby developer&l=philadelphia,"Philadelphia, PA",Posted 1 week ago
+Senior Application Support Specialist Just in!,Access Staffing,https://www.dice.com/jobs/detail/Senior-Application-Support-Specialist-Just-in%21-Access-Staffing-Philadelphia-PA-19019/accperny/50808898?icid=sr16-1p&q=ruby developer&l=philadelphia,"Philadelphia, PA",Posted 3 hours ago
+PHP Back end Web Developer,RemX Specialty Staffing,https://www.dice.com/jobs/detail/PHP-Back-end-Web-Developer-RemX-Specialty-Staffing-Philadelphia-PA-19176/remca001/5814_2980635_1-954?icid=sr17-1p&q=ruby developer&l=philadelphia,"Philadelphia, PA",Posted 1 hour ago
+Software Developer,"Computer Enterprises, Inc.",https://www.dice.com/jobs/detail/Software-Developer-Computer-Enterprises%2C-Inc.-Philadelphia-PA-19103/ceiam/767368?icid=sr18-1p&q=ruby developer&l=philadelphia,"Philadelphia, PA",Posted 9 hours ago
+Java  Angular Developer,INENT Inc. (Innovative Enabling Technologies Inc),https://www.dice.com/jobs/detail/Java--Angular-Developer-INENT-Inc.-%28Innovative-Enabling-Technologies-Inc%29-Philadelphia%2C-PA-19019/10424591/773674?icid=sr19-1p&q=ruby developer&l=philadelphia,"Philadelphia,, PA",Posted 2 days ago
+UI Developer with angular.JS,Srimatrix Inc.,https://www.dice.com/jobs/detail/UI-Developer-with-angular.JS-Srimatrix-Inc.-Philadelphia-PA-19103/10526800/UI224?icid=sr20-1p&q=ruby developer&l=philadelphia,"Philadelphia, PA",Posted 1 week ago
+Web Developer,Kforce Inc.,https://www.dice.com/jobs/detail/Web-Developer-Kforce-Inc.-Philadelphia-PA-19104/kforcecx/ITAQG1577735?icid=sr21-1p&q=ruby developer&l=philadelphia,"Philadelphia, PA",Posted 6 hours ago
+Web 2.0 Software Engineer,Bebee Affinity Social Network,https://www.dice.com/jobs/detail/Web-2.0-Software-Engineer-Bebee-Affinity-Social-Network-Philadelphia-PA-19019/90940991/BBA000013447891?icid=sr22-1p&q=ruby developer&l=philadelphia,"Philadelphia, PA",Posted 13 hours ago
+PHP Developer,Bebee Affinity Social Network,https://www.dice.com/jobs/detail/PHP-Developer-Bebee-Affinity-Social-Network-Philadelphia-PA-19019/90940991/BBA000013042995?icid=sr23-1p&q=ruby developer&l=philadelphia,"Philadelphia, PA",Posted 13 hours ago
+Web Developer,Bebee Affinity Social Network,https://www.dice.com/jobs/detail/Web-Developer-Bebee-Affinity-Social-Network-Philadelphia-PA-19019/90940991/BBA000013447884?icid=sr24-1p&q=ruby developer&l=philadelphia,"Philadelphia, PA",Posted 13 hours ago
+"Senior Application Developer, Corporate",Penn Medicine,https://www.dice.com/jobs/detail/Senior-Application-Developer%2C-Corporate-Penn-Medicine-Philadelphia-PA-19019/10303860/61310?icid=sr25-1p&q=ruby developer&l=philadelphia,"Philadelphia, PA",Posted 10 hours ago
+Guidewire Configuration Developer,PWC,https://www.dice.com/jobs/detail/Guidewire-Configuration-Developer-PWC-Philadelphia-PA-19019/appblok/2392_89787BR?icid=sr26-1p&q=ruby developer&l=philadelphia,"Philadelphia, PA",Posted 8 hours ago
+Project Role - Developer (Temporary Assignment),Accenture,https://www.dice.com/jobs/detail/Project-Role-%26%2345-Developer-%28Temporary-Assignment%29-Accenture-Philadelphia-PA-19019/appblok/2863_1292?icid=sr27-1p&q=ruby developer&l=philadelphia,"Philadelphia, PA",Posted 8 hours ago
+Project Role - iOS Mobile Application Developer (Temporary Assignment),Accenture,https://www.dice.com/jobs/detail/Project-Role-%26%2345-iOS-Mobile-Application-Developer-%28Temporary-Assignment%29-Accenture-Philadelphia-PA-19019/appblok/2863_1263?icid=sr28-1p&q=ruby developer&l=philadelphia,"Philadelphia, PA",Posted 8 hours ago
+Guidewire Configuration Developer,PWC,https://www.dice.com/jobs/detail/Guidewire-Configuration-Developer-PWC-Philadelphia-PA-19019/appblok/2392_89811BR?icid=sr29-1p&q=ruby developer&l=philadelphia,"Philadelphia, PA",Posted 8 hours ago
+Project Role - Salesforce Developer (Temporary Assignment),Accenture,https://www.dice.com/jobs/detail/Project-Role-%26%2345-Salesforce-Developer-%28Temporary-Assignment%29-Accenture-Philadelphia-PA-19019/appblok/2863_1291?icid=sr30-1p&q=ruby developer&l=philadelphia,"Philadelphia, PA",Posted 8 hours ago

lib/csv_saver.rb

@@ -0,0 +1,26 @@
+require 'csv' 
+
+class CSVSaver
+
+  def initialize(jobs) # array of Job structs
+    @jobs = jobs
+  end
+
+  def save_csv
+    CSV.open('csv_file.csv', 'a') do |csv|
+      @jobs.each do |job|
+        csv << entry(job)
+      end
+    end
+  end
+
+  def clear_csv
+    CSV.open('csv_file.csv', 'w') { }
+  end
+
+  def entry(job)
+    [job.title, job.company_name, job.link, job.location, job.post_date]
+  end
+
+end
+

lib/execute.rb

@@ -0,0 +1,9 @@
+require_relative 'scraper' 
+
+scraper = Scraper.new("ruby developer", "philadelphia")
+jobs = scraper.get_jobs
+
+saver = CSVSaver.new(jobs)
+saver.save_csv
+
+# saver.clear_csv

lib/job.rb

@@ -0,0 +1,2 @@
+Job = Struct.new(:title, :company_name, :link, :location, :post_date)
+

lib/scraper.rb

@@ -0,0 +1,58 @@
+require 'rubygems'
+require 'bundler/setup'
+require 'mechanize'
+
+require_relative 'job'
+require_relative 'url_builder'
+require_relative 'csv_saver'
+
+# create a scraper that performs a search query and returns the first results page
+
+class Scraper
+
+  attr_reader :keyword, :location, :jobs
+
+  def initialize(keyword, location)
+    raise ArgumentError unless keyword.is_a?(String) && location.is_a?(String)
+    @keyword = keyword
+    @location = location
+    @jobs = []
+    @links = []
+  end
+
+  def get_jobs
+    scraper = Mechanize.new
+    scraper.history_added = Proc.new { sleep 0.5 }
+    url = build_url
+    page = scraper.get(url)
+    find_job_posts(page)
+    @jobs 
+  end
+
+  def build_url 
+    search = URLBuilder.new("https://www.dice.com/jobs?", q: @keyword, l: @location)
+    search.url
+  end
+
+  def find_job_posts(page)
+    page.links_with(:href => /detail/).each do |link|
+      next if @links.include?(link.href)
+      @links << link.href
+      post = link.click
+      @jobs << get_job_data(post, link)
+    end
+  end
+
+  def get_job_data(post, link)
+    job = Job.new 
+    job.title = post.search("h1.jobTitle").text
+    job.company_name = post.search("li.employer a.dice-btn-link").text
+    job.link = link.href
+    job.location = post.search("li.location").text
+    job.post_date = post.search("li.posted").text
+    job
+  end
+
+end
+
+

lib/url_builder.rb

@@ -0,0 +1,20 @@
+class URLBuilder 
+
+  attr_reader :url 
+
+  def initialize(base_uri, parameters = {}) 
+    @url = build_url(base_uri, parameters)
+  end
+
+  def build_url(base_uri, parameters)
+    string = base_uri
+
+    parameters.each do |key, value|
+      value = value.gsub(/\s/, "+")
+      string += key.to_s + "=" + value + "&"
+    end
+
+    string
+  end
+
+end