edi-energy.de scraper

The Python package edi_energy_scraper provides easy to use methods to mirror the website edi-energy.de.

Rationale / Why?

If you'd like to be informed about new regulations or data formats being published on edi-energy.de you can either

visit the site every day and hope that you see the changes if this is your favourite hobby,
or automate the task.

This repository helps you with the latter. It allows you to create an up-to-date copy of edi-energy.de on your local computer. Other than if you mirrored the files using wget or curl, you'll get a clean and intuitive directory structure.

From there you can e.g. commit the files into a VCS (like e.g. our edi_energy_mirror), scrape the PDF/Word files for later use...

We're all hoping for the day of true digitization on which this repository will become obsolete.

How to use the Package (as a user)

Install via pip:

pip install edi_energy_scraper

Create a directory in which you'd like to save the mirrored data:

mkdir edi_energy_de

Then import it and start the download:

import asyncio
from edi_energy_scraper import EdiEnergyScraper

# add the following lines to enable debug logging to stdout (CLI)
# import logging
# import sys
# logging.basicConfig(stream=sys.stdout, level=logging.DEBUG)

async def mirror():
    scraper = EdiEnergyScraper(path_to_mirror_directory="edi_energy_de")
    await scraper.mirror()


if __name__ == "__main__":
    loop = asyncio.new_event_loop()
    asyncio.set_event_loop(loop)
    asyncio.run(mirror())

This creates a directory structure:

-|-your_script_cwd.py
 |-edi_energy_de
    |- past (contains archived files)
        |- ahb.pdf
        |- ahb.docx
        |- ...
    |- current (contains files valid as of today)
        |- mig.pdf
        |- mig.docx
        |- ...
    |- future (contains files valid in the future)
        |- allgemeine_festlegungen.pdf
        |- schema.xsd
        |- ...