node-web-scraper

Simple web scraper to get a movie name, release year and community rating from IMDB. To run this example use the following commands:

$ npm install
$ node server.js

Then it will start up our node server, navigate to http://localhost:8081/ and see what happens.

Put the URL like "https://www.1mg.com/sitemap.xml"; it will be collecting nested xml from the root sitemap.xml its create copy of xml in local first...so you will see list of all xml found along with server.js. then we can read all the xml again to store into database. So it make an entry into mongodb database "one-mg", collection "urls" for each other links then xml.

which can be further used for crapping One by One.

what you will learn?

1. scrapXMLUrl(). scrap sitemap.xml and find all the other urls.
2. if url is a xml save xml file locally.
3. if url is other then xml save all the urls in mongo db collection "urls".
4. getList() GET /urls to enlist all the urls with pagination skip=0,limit=10 default and filter url by name if required.
5. set a key for auto process "auto".
6. store list 0-10 in items and display it in sidebar. add a scrap() method on onclick of each url.
7. scrap() will be fetching schema data and updating it along with already stored urls.
8. checkAutoHit(), if auto scrap is selected scrap next url from items. if list ends go to the next page. by raising skip by limit.
9. on success 200 change the pending to success and green in color. on 204 make it orange.