Spoon - A package for building specific Proxy Pool for different Sites.

Spoon is a library for building Distributed Proxy Pool for each different sites as you assign.
Only running on python 3.

Install

Simply run: pip install spoonproxy or clone the repo and set it into your PYTHONPATH.

Run

Spoon-server

Please make sure the Redis is running. Default configuration is "host:localhost, port:6379". You can also modify the Redis connection.
Like example.py in spoon_server/example,
You can assign many different proxy providers.

from spoon_server.proxy.fetcher import Fetcher
from spoon_server.main.proxy_pipe import ProxyPipe
from spoon_server.proxy.kuai_provider import KuaiProvider
from spoon_server.proxy.xici_provider import XiciProvider
from spoon_server.database.redis_config import RedisConfig
from spoon_server.main.checker import CheckerBaidu

def main_run():
    redis = RedisConfig("127.0.0.1", 21009)
    p1 = ProxyPipe(url_prefix="https://www.baidu.com",
                   fetcher=Fetcher(use_default=False),
                   database=redis,
                   checker=CheckerBaidu()).set_fetcher([KuaiProvider()]).add_fetcher([XiciProvider()])
    p1.start()


if __name__ == '__main__':
    main_run()

Also, with different checker, you can validate the result precisely.

class CheckerBaidu(Checker):
    def checker_func(self, html=None):
        if isinstance(html, bytes):
            html = html.decode('utf-8')
        if re.search(r".*百度一下，你就知道.*", html):
            return True
        else:
            return False

Also, as the code shows in spoon_server/example/example_multi.py, by using multiprocess, you can get many queues to fetching & validating the proxies.
You can also assign different Providers for different url.
The default proxy providers are shown below, you can write your own providers.

Spoon-web

A Simple django web api demo. You could use any web server and write your own api.
Gently run python manager.py runserver **.**.**.**:*****
The simple apis include: