An elegant way to search for urls with web scraping and javascript


Keywords
path searcher, path finder, url searcher, url finder, web scraping, data mining, path, url
License
MIT
Install
npm install patofinder@1.0.0

Documentation

Patofinder

Patofinder is a open-source node library (with CLI included) that resolve urls of a page from a get request. It doesnt just get all paths, but also filter and look for specific types or extensions.

Installation

Use the node package manager NPM to install it. You can install it locally to import the into your javascript/typescript code or install it globally and use it directly as a command:

npm i patofinder -g 
npm i patofinder

Usage

Import the find function into your js/ts code and pass the url target and the selected search type:

import * as patofinder from 'patofinder';

async function foo() {
    /*
     * Get all image paths from foobar.com
     */
    const images = await patofinder.find('http://www.foobar.com/', 'img');

    console.log(images); // ['http://www.foobar.com/logo.jpg', 'http://www.foobar.com/sample.png']

}

foo();

Or run directly from your prompt:

    patofinder --path http://www.foobar.com/ --type audio

Errors

The library just throw a error. If a request fails or the provided path is invalid, it will just return a empty array, but if a invalid type was provided, a error will be throw:

import * as patofinder from 'patofinder';

async function foo() {

    const valid = await patofinder.find('http://www.foobar.com/', 'css');
    const invalid = await patofinder.find('http://www.invalidpath.org/', 'php');

    console.log(valid); // ['http://www.foobar.com/styles/main.css']
    console.log(invalid); // []

    await patofinder.find('http://www.foobar.com/', 'foo'); // error

}

foo();

Examples

The library has few functions and commands, but you can see some use cases in the docs folder:

Live

On the release of version 1.0.0 a test sample api was created on the heroku servers. See here for more.

Contributing

Pull requests are welcome. For major changes or TypeSearches creation please open an issue first to discuss what you would like to change. You can also contribute by reporting errors or suggesting new features here.

License

MIT