_ _
| |__ ___ _ __ | |_ __ _ _ __
| '_ \ / _ \| '_ \| __/ _` | '_ \
| |_) | (_) | | | | || (_| | | | |
|_.__/ \___/|_| |_|\__\__,_|_| |_|
Bontan is a simple scraper with specialized behavior for some sites (like Wikipedia) and smart fallbacks for others. This repository also includes other scraping utilities.
- bontan: Attempt to scrape all useful text content on the page.
- kinkan: Like Bontan, but as a summary - just grabs the first paragraph. ("kinkan" is Japanese for "kumquat".)
- sudachi: Simple utility to scrape using css selectors.
- mikan: A simple clone of import.io. Prints each item in the biggest list on the page.
Examples
First install it:
npm install -g bontan
bontan
For a full text version of the page:
bontan 'https://en.wikipedia.org/wiki/Pomelo'
Images will have their src
printed.
kinkan
Like Bontan, but with less output - typically it will use the page title, the first image, and the first p
element.
kinkan 'https://en.wikipedia.org/wiki/Pomelo'
Pomelo - Wikipedia, the free encyclopedia
/wikipedia/commons/thumb/1/1c/Citrus_grandis_-_Honey_White.jpg/220px-Citrus_grandis_-_Honey_White.jpg
Citrus maxima (or Citrus grandis), (Common names: shaddick,[1] pomelo, pummelo, pommelo, pamplemousse, or shaddok) is a natural (non-hybrid) citrus fruit, with the appearance of a big grapefruit, native to South and Southeast Asia.
sudachi
For grabbing things by css selectors. It uses a virtual dom (domino), which makes it comparatively fast but unable to handle contents generated by JS after page load. Let's try getting all the h3
elements:
sudachi 'https://en.wikipedia.org/wiki/Pomelo' h3
Possible non-hybrid pomelos[edit]
Hybrids[edit]
Personal tools
Namespaces
etc.
You can pass -r
to return innerHTML
instead of textContent
.
sudachi -r 'https://en.wikipedia.org/wiki/Pomelo' h3
<span class="mw-headline" id="Possible_non-hybrid_pomelos">Possible non-hybrid pomelos</span><span class="mw-editsection"><span class="mw-editsection-bracket">[</span><a href="/w/index.php?title=Pomelo&action=edit&section=5" title="Edit section: Possible non-hybrid pomelos">edit</a><span class="mw-editsection-bracket">]</span></span>
<span class="mw-headline" id="Hybrids">Hybrids</span><span class="mw-editsection"><span class="mw-editsection-bracket">[</span><a href="/w/index.php?title=Pomelo&action=edit&section=6" title="Edit section: Hybrids">edit</a><span class="mw-editsection-bracket">]</span></span>
Personal tools
Namespaces
etc.
mikan
This attempts to replicate some of the magic of import.io using a simple trick - usually, the most interesting list on a page is the longest one. Here's what happens when you point it at Stack Overflow:
mikan 'http://stackoverflow.com'
0 votes 1 answer 4 views Unique index not working ruby-on-rails unique-constraint database-indexes answered 1 min ago Thong Kuah 1,960
0 votes 0 answers 2 views fprintf giving me a blank .txt file in MATLAB matlab asked 1 min ago physicist82 1
0 votes 0 answers 11 views Node.js / Inheritance of variables and modules javascript node.js inheritance modified 1 min ago MiddleWare 138
5 votes 2 answers 30 views Global Events in Angular 2 angular2 modified 1 min ago pixelbits 14.5k
0 votes 1 answer 5 views Running “mvn test site” giving [ERROR] Failed to execute goal org.apache.maven.plugins:maven-site-plugin:3.3:site (default-site) on project maven selenium xslt modified 1 min ago Tunaki 29k
etc.
Or Hacker News:
mikan 'http://news.ycombinator.com'
1. The Trouble with the TPP, Day 5: Rights Holders “Shall” vs. Users “May” (michaelgeist.ca)
46 points by walterbell 3 hours ago | discuss
2. Tesla Model S can now park itself (techcrunch.com)
151 points by prostoalex 6 hours ago | 89 comments
3. Nvidia GPUs can break Chrome's incognito mode (charliehorse55.wordpress.com)
374 points by charliehorse55 11 hours ago | 120 comments
etc.
License
WTFPL, do as you please.
-POLM