hyperfiler

Release 1.1.0

Bundle web pages into single standalone HTML files.

Homepage Repository npm TypeScript Download

Keywords
hyperfiler
License
AGPL-3.0
Install
npm install hyperfiler@1.1.0

Documentation

HyperFiler Logo HyperFiler

HyperFiler Logo

Bundle web pages into single HTML files.

Table of Contents Icon Table of Contents

 •  Overview
 •  Basic Usage
 •  Installation
     ◦  NPM
     ◦  Windows
     ◦  Mac
     ◦  Linux
         ▪  Debian, Ubuntu, and Linux Mint
             ▻  .DEB File
             ▻  Package Repository
         ▪  Fedora
         ▪  Arch
     ◦  BSD
         ▪  FreeBSD
         ▪  OpenBSD
     ◦  Haiku
 •  How It Works
 •  Documentation
     ◦  Command Line API
     ◦  Programmatic API
         ▪  JavaScript
         ▪  TypeScript
         ▪  ClojureScript
     ◦  Creating Plugins
         ▪  JavaScript
         ▪  TypeScript
         ▪  ClojureScript
     ◦  Additional Documentation And Source Code Documentation
 •  Gallery
 •  Building From Source
 •  Known Limitations And Bug Reporting
 •  Upcoming Features
     ◦  Version 2.0.0
     ◦  No Specific Version
 •  License
     ◦  License Exceptions
 •  Donations
 •  Contributions
 •  Other Great Projects
 •  Similar Projects

Overview Icon Overview

HyperFiler bundles web pages and web apps into single HTML files. With HyperFiler you can:

  • Hyper Minify: HyperFiler includes minifiers for virtually every single resource in a web page (HTML, CSS, JS, images, audio, videos, favicons, cursors, fonts), and the experimental --hyper-minification argument may produce extremely small bundled pages.

  • Hyper Customize: There are over 100 command line arguments, allowing users to granularly choose which features and resources of a web page they want to keep in the bundle, and to modify these resources in various ways.

  • Hyper Sanitize: The --sanitize argument can be used to convert web pages into offline-only pages with no scripts, frames, or links to any external resources, allowing users to safely share pages that they’ve archived.

  • Hyper Transpile: HyperFiler includes a series of polyfills and transpilers that can make bundles viewable and usable on older browsers, and make older features available on modern browsers. You can even inject a WASM compiled Flash Player (using the --inject-flash-player argument) into the HTML file that will play flash content embedded into the page in a sandboxed environment.

  • Hyper Portable: The HyperFiler program is packed into a single JavaScript file, and is usable on any platform that supports Node.JS (however some optional functionality that requires external dependencies will be disabled, see the version 2.0.0 milestone and the documentation for more information).

Basic Usage Icon Basic Usage

To create a single HTML file from the main page of Wikipedia, run the following command in the terminal. The bundled file will be written to out.html in the same directory.

hyperfiler https://en.wikipedia.org/wiki/Main_Page -o out.html

Installation Icon Installation

NPM Icon NPM

HyperFiler can be installed generally on any platform that supports NPM through the NPM package. However, installation through NPM won’t include any of the optional dependencies needed to use all of the features, and NPM isn’t strictly needed to run HyperFiler. For better installation methods, see the below platform specific installation instructions. Run the following command to install HyperFiler globally using NPM.

npm install hyperfiler -g

If you want to use HyperFiler programmatically in your current project, use the following command:

npm install hyperfiler

Windows Icon Windows

For installation on Windows, first ensure that you have Node.JS and NPM installed. Follow the instructions for installing Node.JS and NPM here. Once installed, run the following command to install HyperFiler:

npm install hyperfiler -g

Mac Icon Mac

For installation on MacOSX, first ensure that you have Node.JS and NPM installed. Follow the instructions for installing Node.JS and NPM here. Once installed, run the following command to install HyperFiler:

npm install hyperfiler -g

Linux Icon Linux

For installation on Linux platforms, see the installation instructions for your specific distribution below.

Debian Icon Ubuntu Icon Linux Mint Icon Debian, Ubuntu, and Linux Mint

For Debian, Ubuntu, and Linux Mint, you can either install HyperFiler using the HyperFiler .DEB file available on the release page, or by using the APT package manager and wget.

.DEB File

After downloading a HyperFiler .DEB file from the release page, run the following command to install (make sure to update the version number in the command below):

sudo dpkg -i hyperfiler-<VERSION>.deb && sudo apt-get install -f
Package Repository

You can alternatively install HyperFiler and the dependencies directly from the releases and package repository using the following commands:

sudo apt-get install -y nodejs imagemagick pngcrush pngquant gifsicle libwebp6 ffmpeg fonttools mat2
wget -q -O - https://raw.githubusercontent.com/chowderman/hyperfiler/master/dist/debian/hyperfiler | sudo tee /usr/local/bin/hyperfiler > /dev/null
sudo chmod +x /usr/local/bin/hyperfiler

Fedora Icon Fedora

Run the following commands to install HyperFiler and all of the dependencies on Fedora:

sudo dnf install -y nodejs ImageMagick pngcrush pngquant gifsicle libwebp fonttools
wget -q -O - https://raw.githubusercontent.com/chowderman/hyperfiler/master/dist/fedora/hyperfiler | sudo tee /usr/local/bin/hyperfiler > /dev/null
sudo chmod +x /usr/local/bin/hyperfiler

Note that the current Fedora package repository is missing the following optional dependencies, and functions using these dependencies will be disabled:

FFMPEG
Mat2

Arch Icon Arch

Run the following commands to install HyperFiler and all of the dependencies on Arch Linux (these need to be run as root):

pacman -S --noconfirm which curl nodejs imagemagick pngcrush pngquant gifsicle libwebp ffmpeg python-fonttools mat2
curl https://raw.githubusercontent.com/chowderman/hyperfiler/master/dist/arch/hyperfiler > /usr/local/bin/hyperfiler
chmod +x /usr/local/bin/hyperfiler

BSD Icon BSD

For installation on BSD platforms, see the installation instructions for your specific distribution below.

FreeBSD Icon FreeBSD

Run the following commands to install HyperFiler and all of the dependencies on FreeBSD (these need to be run as root):

pkg install -y curl node ImageMagick7 pngcrush pngquant gifsicle webp ffmpeg py37-fonttools
curl https://raw.githubusercontent.com/chowderman/hyperfiler/master/dist/freebsd/hyperfiler > /usr/local/bin/hyperfiler
chmod +x /usr/local/bin/hyperfiler

Note that the current FreeBSD package repository is missing the following optional dependencies, and functions using these dependencies will be disabled:

Mat2

OpenBSD Icon OpenBSD

Run the following commands to install HyperFiler and all of the dependencies on OpenBSD (these need to be run as root):

pkg_add curl node ImageMagick pngcrush gifsicle libwebp ffmpeg
curl https://raw.githubusercontent.com/chowderman/hyperfiler/master/dist/openbsd/hyperfiler > /usr/local/bin/hyperfiler
chmod +x /usr/local/bin/hyperfiler

Note that the current OpenBSD package repository is missing the following optional dependencies, and functions using these dependencies will be disabled:

Fonttools
Pngquant

Haiku Icon Haiku

Run the following commands to install HyperFiler and all of the dependencies on HaikuOS:

pkgman install -y nodejs imagemagick pngcrush pngquant gifsicle libwebp ffmpeg fonttools
wget -q -O - https://raw.githubusercontent.com/chowderman/hyperfiler/master/dist/haiku/hyperfiler > /boot/system/non-packaged/bin/hyperfiler
chmod +x /boot/system/non-packaged/bin/hyperfiler

Note that the current HaikuOS package repository is missing the following optional dependencies, and functions using these dependencies will be disabled:

Mat2

How It Works Icon How It Works

How HyperFiler Works

Many modern browser support Data URIs, which can be used to convert binary files (such as a PNG image) into an encoded string of text (such as by using Base64 encoding). Since HTML is a text-only format, and since the browser can convert Data URIs into the underlying resources, binary page resources (such as images, audio, video, favicons, flash files, and many other binary resources) can be bundled into the HTML page itself. Web apps such as XP Paint use this technique to bundle the entire app into a single HTML file, making it very portable, easy to archive, and easy to share.

Documentation Icon Documentation

Command Line Icon Command Line API

Below is the help page for the HyperFiler CLI. Each command line argument (with the exception of a few such as --out and --silent) corresponds to a function of the same name that is called when the command line argument is provides. For example, the --remove-scripts argument corresponds to the removeScripts() function. Examples are provided for many commands in the HyperFiler Source Code Documentation

Programmatic API Icon Programmatic API

HyperFiler is written in TypeScript, and can be used programmatically in languages that can import JavaScript NPM libraries. Below is a basic example of importing and running HyperFiler programmatically.

JavaScript Icon JavaScript

const { HyperFiler } = require('hyperfiler');

(async () => {
  // Creating a new HyperFiler object. Each HyperFiler represent a single
  // bundled HTML page.
  const hyperFiler = new HyperFiler({
    url: 'https://en.wikipedia.org/wiki/Main_Page',
    out: './out.html',
  });

  // Running the bundling processing. All page resources will be fetched,
  // encoded, and bundled into the page.
  await hyperFiler.run();
})();

TypeScript Icon TypeScript

import { HyperFiler } from 'hyperfiler';

(async () => {
  // Creating a new HyperFiler object. Each HyperFiler represent a single
  // bundled HTML page.
  const hyperFiler: HyperFiler = new HyperFiler({
    url: 'https://en.wikipedia.org/wiki/Main_Page',
    out: './out.html',
  });

  // Running the bundling processing. All page resources will be fetched,
  // encoded, and bundled into the page.
  await hyperFiler.run();
})();

Clojure Icon ClojureScript

(ns app.main
  (:require ["hyperfiler" :refer (HyperFiler)]))

;; Creating a new HyperFiler object. Each HyperFiler represent a single
;; bundled HTML page.
(def hyper-filer (HyperFiler.
  #js{:url "https://en.wikipedia.org/wiki/Main_Page"
      :out "./out.html"}))

;; Running the bundling processing. All page resources will be fetched,
;; encoded, and bundled into the page.
(.then (.run hyper-filer))

Plugins Icon Creating Plugins

Internally, the architecture of HyperFiler is structured as a pipeline of plugins that are built and run at various stages of the bundling process. For example, when the --remove-scripts argument is provided when running the CLI command, the removeScripts() function will be wrapped into the removeScriptsPlugin(), and this plugin will be called at the stage before scripts are fetched to remove them from the bundled page. Since everything is a plugin in HyperFiler, developers can easily add their own custom functionality by building plugins and running them at various stages of the bundling process. Below are examples of creating and using custom plugins:

JavaScript Icon JavaScript

const { HyperFiler } = require('hyperfiler');

// Creating a HyperFiler plugin that will add a JavaScript alert that displays
// `Hello, World` when the page first loads.
async function addHelloWorldPlugin(hyperFiler) {
  hyperFiler.html = hyperFiler.html
    .split('</head>')
    .join(`
        <script>
          alert('Hello, World');
        </script>
      </head>
    `);
}

// Adding our new plugin to an array of HyperFiler plugins.
const beforeFinalHtmlCreationPlugins = [
  addHelloWorldPlugin,
];

(async () => {
  const hyperFiler = new HyperFiler({
    url: 'https://en.wikipedia.org/wiki/Main_Page',
    out: './out.html',

    // The plugin is added to the `beforeFinalHtmlCreation` stage of the
    // bundling process. Right before the final HTML page is created (but after
    // the all of the resource modification and inlining functions have been
    // run), the HyperFiler context will be passed to the plugin, and the state
    // of the HyperFiler object will be modified via the plugin call. Note
    // that plugins will be called sequentially in the order they are added to
    // the array.
    plugins: {
      beforeFinalHtmlCreation: beforeFinalHtmlCreationPlugins,
    }
  });

  await hyperFiler.run();
})();

TypeScript Icon TypeScript

import { HyperFiler, HyperFilerPlugin } from 'hyperfiler';

// Creating a HyperFiler plugin that will add a JavaScript alert that displays
// `Hello, World` when the page first loads.
async function addHelloWorldPlugin(
  hyperFiler: HyperFiler,
) : Promise<void> {
  hyperFiler.html = hyperFiler.html
    .split('</head>')
    .join(`
        <script>
          alert('Hello, World');
        </script>
      </head>
    `);
}

// Adding our new plugin to an array of HyperFiler plugins.
const beforeFinalHtmlCreationPlugins: HyperFilerPlugin[] = [
  addHelloWorldPlugin,
];

(async () => {
  const hyperFiler: HyperFiler = new HyperFiler({
    url: 'https://en.wikipedia.org/wiki/Main_Page',
    out: './out.html',

    // The plugin is added to the `beforeFinalHtmlCreation` stage of the
    // bundling process. Right before the final HTML page is created (but after
    // the all of the resource modification and inlining functions have been
    // run), the HyperFiler context will be passed to the plugin, and the state
    // of the HyperFiler object will be modified via the plugin call. Note
    // that plugins will be called sequentially in the order they are added to
    // the array.
    plugins: {
      beforeFinalHtmlCreation: beforeFinalHtmlCreationPlugins,
    }
  });

  await hyperFiler.run();
})();

Clojure Icon ClojureScript

(ns app.main
  (:require ["hyperfiler" :refer (HyperFiler)]))

;; Creating a HyperFiler plugin that will add a JavaScript alert that displays
;; Hello, World` when the page first loads.
(defn add-hello-world-plugin
  [hyper-filer]
  (do
    (-> (.-html hyper-filer)
        (.split "</head>")
        (.join "
            <script>
              alert('Hello, World');
            </script>
          </head>
        "))))

;; Adding our new plugin to an array of HyperFiler plugins.
(def before-final-html-creation-plugins #js[add-hello-world-plugin])

(def hyper-filer (HyperFiler.
  #js{:url "https://en.wikipedia.org/wiki/Main_Page"
      :out "./out.html"
      ;; The plugin is added to the `beforeFinalHtmlCreation` stage of the
      ;; bundling process. Right before the final HTML page is created (but after
      ;; the all of the resource modification and inlining functions have been
      ;; run), the HyperFiler context will be passed to the plugin, and the state
      ;; of the HyperFiler object will be modified via the plugin call. Note
      ;; that plugins will be called sequentially in the order they are added to
      ;; the array.
      :plugins #js{:beforeFinalHtmlCreation before-final-html-creation-plugins}}))

(.then (.run hyper-filer))

Below are all of the current stages available that plugins can be added to:

  • beforeInitialFetch

    • Plugins that will run during the stage of the hyper filing process before the entry HTML page has been fetched.

  • afterInitialFetch

    • Plugins that will run during the stage of the hyper filing process after the entry HTML page has been fetched.

  • beforePreCssInlineRemoval

    • Plugins that will run during the stage of the hyper filing process before built-in removal plugins have run prior to any CSS code being inlined in the document.

  • afterPreCssInlineRemoval

    • Plugins that will run during the stage of the hyper filing process after built-in removal plugins have run prior to any CSS code being inlined in the document.

  • beforeCssInlines

    • Plugins that will run during the stage of the hyper filing process before the CSS code is inlined in the document.

  • afterCssInlines

    • Plugins that will run during the stage of the hyper filing process after the CSS code is inlined in the document.

  • beforePostCssInlineRemoval

    • Plugins that will run during the stage of the hyper filing process before built-in removal plugins have run after the CSS code has been inlined in the document.

  • afterPostCssInlineRemoval

    • Plugins that will run during the stage of the hyper filing process after built-in removal plugins have run after the CSS code has been inlined in the document.

  • beforePostCssInlineResourceFetch

    • Plugins that will run during the stage of the hyper filing process after all of the CSS code has been inlined, but before the remaining document resources have been fetched.

  • afterPostCssInlineResourceFetch

    • Plugins that will run during the stage of the hyper filing process after all of the CSS code has been inlined and after the remaining document resources have been fetched.

  • beforeResourceModifications

    • Plugins that will run during the stage of the hyper filing process before any of the resources have been modified.

  • afterResourceModifications

    • Plugins that will run during the stage of the hyper filing process after the resources have been modified.

  • beforeResourceInlines

    • Plugins that will run during the stage of the hyper filing process before the remaining resources have been encoded and inlined.

  • afterResourceInlines

    • Plugins that will run during the stage of the hyper filing process after the remaining resources have been encoded and inlined.

  • beforeFinalInlineModifications

    • Plugins that will run during the stage of the hyper filing process before the final inline modifications are run.

  • afterFinalInlineModifications

    • Plugins that will run during the stage of the hyper filing process after the final inline modifications are run.

  • beforeFinalHtmlCreation

    • Plugins that will run during the stage of the hyper filing process before the final HTML page bundle is created.

  • afterFinalHtmlCreation

    • Plugins that will run during the stage of the hyper filing process after the final HTML page bundle is created.

Additional Documentation And Source Code Documentation

Additional documentation, examples, and source code documentation can be found on TypeDoc-generated HyperFiler Source Code Documentation page.

Gallery Icon Gallery

Below is a gallery of pages that have been bundled with HyperFiler. Try saving one of these pages using your browser (Ctrl+S in most browsers). You’ll notice that no CSS, scripts, or images are saved separately when saving the page

English Wikipedia article for Firefox

This is a snapshot of the english Wikipedia article for Firefox. Scripts and external links have been removed from the page using the --sanitize argument, making the page an offline-only page.

English Wikipedia article for Firefox In Grayscale

This is a snapshot of the english Wikipedia article for Firefox. Like the above page, scripts and external links have been removed from the page using the --sanitize argument, making the page an offline-only page. Additionally, the --grayscale argument has been used to grayscale all of the CSS, images, and videos on the page.

Unminified SVGO NPM Page

This page is a snapshot of the page for the SVGO library on NPM with no optimizations or minifications. The page includes documentation for how to use the SVGO library.

Hyper Minified SVGO NPM Page

Here is the same exact SVGO page, except that the --hyper-minification argument has been used. The is now over 99% smaller than the above page bundle. The high levels of minification were achieved through dead code elimination and feature elimination. All unused CSS, scripts, frames, hidden elements, fonts, non-display elements, redundant attributes, and non-display attributes, among other markup, has been removed, and all images have been minified using the HyperFiler minification algorithm. Additionally, all external links have been removed, and the entire page itself has been LZMA compressed, Z85 encoded, and bundled with a self-decoding and self-extracting script and a source map that will reassemble the page when it is opened in the browser. Through feature elimination, this page has essentially been converted into a read-only, offline documentation page for the SVGO library. If all you needed is the documentation for the SVGO library, this page will have 100% of the features needed at a small fraction of the size. (Note that browsers do a lot of caching and compressing, and the Base64 encoding of images in the above page increases its size, so the comparison is a bit fanciful, but it’s an interesting take on how far minification can be pushed given a defined feature set and current browser technologies).

Source Code Icon Building from Source

To build from source, install Node.JS and NPM and run the following commands:

git clone https://github.com/chowderman/hyperfiler.git
npm install --force
npm run build:all

After these commands are run, the HyperFiler library will be built in the out folder, and standalone HyperFiler programs for each platform will be built in the dist folder. See the HyperFiler package.json for all of the available commands for building HyperFiler.

Bug Icon Known Limitations And Bug Reporting

HTML pages are very complex, so there will likely be many edge cases when converting a page into a single HTML file. Resources may be missing, fetches may fail, and pages may require complex interactions with the server to build the page. If a page fails to assemble as expected, please open an issue on the issue tracker with the following information:

  • URL (the URL of the page fetched)

  • Command/Code (either the command used if using the CLI, or a source code snippet if running programmatically)

  • Description (brief description with any additional relevant information)

  • Console Output (the entire console output from running HyperFiler if possible)

Upcoming Features Icon Upcoming Features

Below are the planned features and release milestones.

Version 2.0.0

  • ❏ No external dependencies. All dependencies should be either WASM compiled and included in the bundle or replaced with pure JavaScript alternatives.

    • ❏ ImageMagick

    • ❏ Pngcrush

    • ❏ Pngquant

    • ❏ Gifsicle

    • ❏ Libwebp

    • ❏ FFMPEG

    • ❏ Fonttools

    • ❏ Mat2

No Specific Version

  • ❏ Program Features

    • ❏ Static HTML5 to HTML4 transpilation

    • ❏ Integration with Modernizr (detect features and then shim when necessary)

    • ❏ Integration with Google Closure Compiler (dead code elimination of common libraries, such as jQuery, using externs)

    • ❏ Creating HTML books from multiple pages (compile many HTML files into a single file with JS code to navigate between links)

    • ❏ HTML and CSS optimizer (since the entire HTML and CSS source code is in a single file, there is an opportunity to optimize the CSS properties and classes such that commonly used properties are consolidated into a single class and then applied to elements throughout the document that use that property)

    • ❏ Custom transport options (allow developer defined options for fetching resources)

    • ❏ A domain allowlist and denylist remover options (allows users and developers to specify domains that will not be fetched).

    • ❏ A 3rd party domain remover option.

    • ✓ Page grayscaler option.

    • ❏ Option to remove non-standard HTML tags.

    • ❏ Option to remove non-standard attributes on all HTML tags.

  • ❏ Image Types

    • ❏ AVIF support (likely through libavif)

    • ❏ JPEG XL support

  • ❏ Audio and Video Types

    • ❏ Support for additional codecs and functions for minifying more formats.

  • ❏ Legacy Plugins

    • ❏ Java Applet support (may be possible with a WASM compiled Java interpreter)

    • ❏ Silverlight support (projects like OpenSilver show some promise, but currently require recompilation as opposed to fallback injection. Any other solutions?)

  • ❏ Quality of Life Features

    • Inclusion in various package repositories.

      • ❏ Debian

      • ❏ Ubuntu

      • ❏ Linux Mint

      • ❏ Fedora

      • ❏ Arch

      • ❏ Windows

      • ❏ Mac

      • ❏ FreeBSD

      • ❏ OpenBSD

      • ❏ Haiku

    • Precompiled binary versions of HyperFiler (stand-alone single executable files bundling HyperFiler with a Node.JS runtime. Nexe may be an option for creating these stand-alone executable files).

      • ❏ Debian

      • ❏ Ubuntu

      • ❏ Linux Mint

      • ❏ Fedora

      • ❏ Arch

      • ❏ Windows

      • ❏ Mac

      • ❏ FreeBSD

      • ❏ OpenBSD

      • ❏ Haiku

    • ❏ Improved error handling and messages.

    • ❏ Improved documentation.

      • ❏ Additional examples on how to use plugins.

      • ❏ Additional information on the default options available.

    • ❏ Improved README.

      • ❏ More code examples.

  • ❏ Code Improvements

    • ❏ Unify the subprocess code into a single subprocess function.

    • ❏ Move dependency checking and conditional subprocess returns into separate functions.

License Icon License

AGPLv3 Logo

HyperFiler is licensed under the GNU Affero General Public License v3 or later (AGPLv3 or later). For additional license information, see the LICENSE file in this repository.

License Exceptions

Exceptions to the license may be granted by ChowderMan on a case-by-case basis. Please reach out to chowderman@protonmail.com for inquiries on license exceptions.

Donations Icon Donations

I’m a huge fan of the Internet Archive, a 501(c) non-profit institution that has been archiving web pages for over 20 years. If you like this project, please consider donating to that amazing organization to keep the preservation of the web going, they can do much more with the donations than I can.

Contributions Icon Contributions

If you would like to add a new feature to HyperFiler, feel free to make a pull request with your new feature. All pull requests are welcome!

Other Great Projects

  • HypnoSpace Outlaw: great game and a great source of inspiration, it’s an old internet simulator and puzzle game with many references to 1990s and 2000s internet and computer history.

Similar Projects

  • SingleFile: A JavaScript-based browser add-on and CLI for creating single file HTML pages.

  • Monolith: A Rust-based terminal program for creating single file HTML pages.

Stats

Dependencies
0
Dependent packages
0
Dependent repositories
0
Total releases
7
Latest release
First release
Stars
39
Forks
0
Watchers
1
Contributors
0
Repository size
33.6 MB
SourceRank
10

Development practices

Source repo 2FA enabled
TEXT!
Package manager 2FA enabled
TEXT!
Is security responsive
TEXT!
Dependencies are managed
TEXT!
Issue-free release available
TEXT!
Succession plan available
TEXT!
Package manager 2FA enabled
TEXT!

The Tidelift Subscription provides access to a continuously curated stream of human-researched and maintainer-verified data on open source packages and their licenses, releases, vulnerabilities, and development practices.

Learn more →

Releases

1.1.2
1.1.0
1.0.4
1.0.3
1.0.2
1.0.1
1.0.0

Something wrong with this page? Make a suggestion

Export .ABOUT file for this package

Last synced: 2021-08-19 01:24:52 UTC

Login to resync this project