Site Map Parser
Script and library which reads urls and converts to objects, allows exporting as CSV or JSON.
Handle sitemaps according to: https://www.sitemaps.org/protocol.html
Installation
pip install site-map-parser
Usage
Script usage
smapper $url > /tmp/data.csv
Logs written to ~/sitemap_run.log
Arguments
| Argument | Options | Default | Information |
|---|---|---|---|
| -h | N/A | N/A | Outputs argument data |
| url | e.g. http://www.example.com - http://www.example.com/other_sitemap.xml
|
N/A | Required - sitemap data to retrieve |
| -l, --log |
CRITICAL or ERROR or WARNING or INFO or DEBUG
|
INFO |
logs to sitemapper_run.log in install folder |
| -e, --exporter |
csv or json
|
csv |
Export format of the data |
Library Usage
from sitemapparser import SiteMapParser
sm = SiteMapParser('http://www.example.com') # reads /sitemap.xml
if sm.has_sitemaps():
sitemaps = sm.get_sitemaps() # returns iterator of sitemapper.Sitemap instances
else:
urls = sm.get_urls() # returns iterator of sitemapper.Url instancesExporting
Two exporters are available: csv and json
CSV Exporter
from sitemapparser.exporters import CSVExporter
# sm set as per earlier library usage example
csv_exporter = CSVExporter(sm)
if sm.has_sitemaps():
print(csv_exporter.export_sitemaps())
elif sm.has_urls():
print(csv_exporter.export_urls())JSON Exporter
from sitemapparser.exporters import JSONExporter
# sm set as per earlier library usage example
json_exporter = JSONExporter(sm)
if sm.has_sitemaps():
print(json_exporter.export_sitemaps())
elif sm.has_urls():
print(json_exporter.export_urls())
![dependabot[bot]](https://avatars.githubusercontent.com/u/49699333?size=120)