Fulmar is a distributed crawler system. By using non-blocking network I/O, Fulmar can handle hundreds of open connections at the same time. You can extractthe data you need from websites. In a fast, simple way.
You can save above code in a new file called baidu_spider.py and run command:
fulmar start_project baidu_spider.py
If you have installed redis, you will get:
Successfully start the project, project name: "baidu_spider".
Finally, start Fulmar:
fulmar all
Automatic installation:
pip install fulmar
Fulmar is listed in PyPI and can be installed with pip
or easy_install
.
Fulmar source code is hosted on GitHub.
Please visit Fulmar Docs.