downloaders
Python package to handle the download of multiple types of files.
How do I install this package?
As usual, just download it using pip:
pip install downloaders
Usage examples
from downloaders import BaseDownloader
downloader = BaseDownloader()
urls = [...]
downloader.download(urls)
Troubleshooting
MacOS multiprocessing nightmare fuel
Cupertino has a gift for us: somehow multiprocessing on MacOS in some specific astral configurations that I have yet to fully understand, it will crash with the following error:
The process has forked and you cannot use this CoreFoundation functionality safely. You MUST exec().
Break on __THE_PROCESS_HAS_FORKED_AND_YOU_CANNOT_USE_THIS_COREFOUNDATION_FUNCTIONALITY___YOU_MUST_EXEC__() to debug.
Apparently, this can be easily fixed by changing the way multiprocessing spawns processes, that is:
import platform, multiprocessing
if platform.system() == "Darwin":
multiprocessing.set_start_method('spawn')
The aforementioned solution was proposed on this StackOverflow question.