symfony/dom-crawler
Symfony DomCrawler Component
Latest release v5.0.5 - Updated - 2.85K stars
Scrapy
A high-level Web Crawling and Web Scraping framework
Latest release 2.2.0 - Updated - 37.5K stars
jaybizzle/crawler-detect
CrawlerDetect is a PHP class for detecting bots/crawlers/spiders via the user agent
Latest release v1.2.95 - Updated - 1.17K stars
crawler
Crawler is a web spider written with Nodejs. It gives you the full power of jQuery on the server ...
Latest release 1.2.2 - Updated - 5.42K stars
simplecrawler
Very straightforward, event driven web crawler. Features a flexible queue interface and a basic c...
Latest release 1.1.9 - Updated - 2K stars
osmosis
Web scraper for NodeJS
Latest release 1.1.10 - Updated - 3.73K stars
puppeteer-extra-plugin-stealth
Stealth mode: Applies various techniques to make detection of headless puppeteer harder.
Latest release 2.4.9 - Updated - 1.39K stars
newspaper3k
Simplified python article discovery & extraction.
Latest release 0.2.8 - Updated - 9.5K stars
jaeger/querylist
Simple, elegant, extensible PHP Web Scraper (crawler/spider),Use the css3 dom selector,Based on p...
Latest release V4.2.5 - Updated - 1.97K stars
spatie/crawler
Crawl all internal links found on a website
Latest release 4.7.1 - Updated - 1.51K stars
wombat
Generic Web crawler with a DSL that parses structured data from web pages
Latest release 2.10.0 - Updated - 1.16K stars
wa72/htmlpagedom
jQuery-inspired DOM manipulation extension for Symfony's Crawler
Latest release v2.0.0 - Updated - 270 stars
us.codecraft:webmagic-core
A scalable web crawler framework for Java.
Latest release 0.7.3 - Updated - 8.67K stars
Abot
Abot is an open source C# web crawler built for speed and flexibility. It takes care of the low l...
Latest release 2.0.56 - Updated - 1.73K stars
google-play-scraper
scrapes app data from google play store
Latest release 7.1.3 - Updated - 1.25K stars
cheerio-httpcli
http client module with cheerio & iconv(-lite) & promise
Latest release 0.8.1 - Updated - 240 stars
spatie/robots-txt
Determine if a page may be crawled from robots.txt and robots meta tags
Latest release 1.0.6 - Updated - 101 stars
HtmlAgilityPack
This is an agile HTML parser that builds a read/write DOM and supports plain XPATH or XSLT (you a...
Latest release 1.11.24 - Updated
huginn_agent
Helpers for making new Huginn Agents
Latest release 0.6.1 - Updated - 28.4K stars
headless-chrome-crawler
Distributed web crawler powered by Headless Chrome
Latest release 1.8.0 - Updated - 4.56K stars
scrapy-redis
Redis-based components for Scrapy.
Latest release 0.6.8 - Updated - 4.3K stars
DotnetSpider
DotnetSpider, a .NET Standard web crawling library. It is lightweight, efficient and fast high-le...
Latest release 5.0.1-beta5 - Updated - 2.51K stars
apify
The scalable web crawling and scraping library for JavaScript/Node.js. Enables development of dat...
Latest release 0.21.1-beta.8 - Updated - 2.2K stars
zhihu-api
Unofficial API for zhihu (https://www.zhihu.com)
Latest release 3.0.0 - Updated - 255 stars
sitemap-generator
Easily create XML sitemaps for your website.
Latest release 8.5.0 - Updated - 220 stars
torrent-search-api
Yet another node torrent scraper based on x-ray. (Support iptorrents, torrentleech, torrent9, Yyg...
Latest release 2.1.3 - Updated - 182 stars
hltv
The unofficial HLTV Node.js API
Latest release 2.19.4 - Updated - 153 stars
scrapy
Scrapy is an open source and collaborative framework for extracting the data you need from websit...
Latest release 1.6.0 - Published - 37.5K stars
us.codecraft:webmagic-extension
A scalable web crawler framework for Java.
Latest release 0.7.3 - Updated - 8.67K stars
toapi
Every web site provides APIs.
Latest release 2.1.2 - Updated - 2.55K stars
License
Language
Keyword
Platform

Subscribe to an RSS feed of this search