@carbonenginejs/format-webgl

CarbonEngineJS WebGL format package reader/builder for .cewg packages, plus a DXBC to GLSL ES 3.00 emitter.


Keywords
carbonenginejs, format-webgl, webgl, webgl2, cewg, package, dxbc, glsl, shader
License
MIT
Install
npm install @carbonenginejs/format-webgl@0.2.0

Documentation

@carbonenginejs/format-webgl

CarbonEngineJS-facing reader/builder for .cewg WebGL shader packages, plus a pure-JavaScript DXBC (Direct3D compiled shader bytecode) to GLSL ES 3.00 emitter for the WebGL2 vertex/pixel/map-style-compute stages ccpwgl targets.

No native tooling, no build step; it runs in Node and the browser. This package has two independent capabilities:

  1. CEWG package read/write. CEWG is a CarbonEngineJS-invented container format - a flat sequence of four-byte-tagged chunks (INFO/META/GLSL/ ...), not a Microsoft or CCP one. CEWG is also the four-byte binary package magic.
  2. DXBC -> GLSL ES 3.00 emission, translating one DXBC vertex, pixel, or "map-style" compute stage (thread-per-fragment; real compute features like shared memory, barriers, atomics, and raw/structured/typed UAV reads are rejected) into GLSL ES 3.00 source. Depends on @carbonenginejs/format-dxbc for DXBC container/program/instruction decoding; this package adds no DXBC-parsing code of its own.

CarbonEngine and Fenris Creations (CCP Games) are named in this package because the WebGL profile targets Carbon/Trinity shader conventions and ccpwgl runtime needs. CEWG itself is a CarbonEngineJS package format. This package contains no CarbonEngine or Fenris Creations (CCP Games) source code unless NOTICE explicitly says otherwise, and it is not affiliated with or endorsed by CCP Games.

Install

npm install @carbonenginejs/format-webgl

Quick start

import { CjsFormatWebgl } from "@carbonenginejs/format-webgl";

const packageData = CjsFormatWebgl.read(cewgBytes);
const glsl = CjsFormatWebgl.emitGlsl(dxbcBytes);

Public API

The package root exports one public class: CjsFormatWebgl.

import CjsFormatWebgl from "@carbonenginejs/format-webgl";

// One-shot statics (camelCase by convention)
CjsFormatWebgl.isCewg(bytes);                       // cheap magic sniff
CjsFormatWebgl.inspect(bytes);                     // version + chunk + shader/stage-count summary
CjsFormatWebgl.read(bytes);                        // documented plain JSON package shape
CjsFormatWebgl.read(bytes, { emit: "raw" });       // internal CewgPackage instance (unstable)
CjsFormatWebgl.build([                             // assemble package bytes from chunk payloads
    [ "INFO", { format: "CEWG", formatVersion: 1 } ],
    [ "META", { effectName: "quadv5" } ],
    [ "GLSL", { format: "CEWG_GLSL_SET", formatVersion: 1, shaders: [ /* ... */ ] } ]
]);
CjsFormatWebgl.emitGlsl(dxbcBytes, { source: "Main.pass0.vertex" });
CjsFormatWebgl.buildEffect(effectBytes, {
    source: "res:/graphics/effect.dx11/managed/space/quadv5.sm_depth",
    allPermutations: true
});

// Reusable profile
const reader = new CjsFormatWebgl({ emit: "json", source: "myeffect.cewg" });
const result = reader.Read(bytes);

Named import is also available:

import { CjsFormatWebgl } from "@carbonenginejs/format-webgl";

The JSON graph (emit: "json", the default, for Read/read)

The JSON format value and the binary package magic are both CEWG.

Root
|- format ("CEWG"), version, sourcePath
|- chunks: { tag, size, offset }[]     // every chunk in the package, in file order
|- info: object | null                 // parsed INFO chunk (translator/package summary)
|- metadata: object | null             // parsed META chunk (caller-provided Carbon metadata)
|- permutationGraph: object | null     // complete PGRF source topology
|- reflection: object | null           // all-unique v15 portable reflection
|- reflectionBlobByteLength: number    // exact shared RBLB byte length
|- glsl: object | string | null        // parsed GLSL chunk JSON (whole-effect stage sets),
|                                      // or raw GLSL text for single-stage packages, or null
`- shaders: object[]                   // glsl.shaders when present, else []

emit: "raw"

Returns the internal CewgPackage instance directly - unstable, not schema-guaranteed, except for the validated portable-reflection accessor described below. Exposes pkg.GetChunk(tag), pkg.GetText(tag), pkg.GetJson(tag), and the info/metadata/permutationGraph/ reflection/reflectionBlobBytes/glsl/glslJson/dxbc getters. pkg.GetPortableEffectReflection(permutationIndex) returns one validated @carbonenginejs/format-hlsl/portable body with fresh owned byte arrays; omitting the index selects INFO.defaultPermutationIndex.

Inspect(bytes) / inspect(bytes)

A cheaper alternative to a full Read: package version, every chunk's tag/size/offset, GLSL shader/stage counts, source permutation/body counts, and reflection/blob counts.

Build(chunks) / build(chunks)

Wraps the CEWG builder: assembles package bytes from an ordered [tag, payload] list. Each payload may be a string (encoded as UTF-8), a plain object (JSON-encoded), or raw bytes (Uint8Array/ArrayBuffer/typed array view) - passed straight through to the same builder used by this project's packaging tooling.

EmitGlsl(dxbcBytes, options) / emitGlsl(dxbcBytes, options)

Translates one DXBC stage. options is a flat bag combining the emitter's ccpwgl-profile constructor bits with its per-call emit options - every key is optional and defaults are preserved exactly:

Option Default Meaning
constantBufferStyle "array" "array" emits uniform vec4 cbN[] (ccpwgl's uniform4fv path); "std140" emits layout(std140) uniform blocks.
pixelConstantBufferRemap { 0: 7 } Pixel-stage constant-buffer slot renames (ccpwgl keeps PS effect constants at cb7).
samplerName(register, stageName) vs${register} (vertex) / s${register} (pixel) Texture uniform naming per register and stage.
vertexStructuredCapacity 69 Element capacity for vertex-stage structured-buffer UBOs (bones; Carbon's max is 69 joints).
dataTextureWidth 2048 Row width for buffer/structured-resource data textures.
pairVaryings undefined Register list the paired pixel stage reads; any vertex output missing from this shader is declared and zero-filled.
source "memory" Name used in thrown error details.

Rejects non-vertex/pixel/compute DXBC stages with "Only vertex, pixel, and compute stages target WebGL2", and compute stages that need real compute-pipeline features (shared memory, barriers, atomics, raw/structured/typed UAV reads) with a message containing "not supported". The emitter only lowers WebGL2-safe register IO, texture sampling, and structured-memory patterns; DXBC instruction semantics come from @carbonenginejs/format-dxbc, while target-language lowering policy lives in this package.

BuildEffect(effectBytes, options) / buildEffect(effectBytes, options)

Converts a compiled .sm_* effect to CEWG using only caller-supplied bytes. The pipeline reads the effect, resolves its permutation bodies and stages, emits GLSL, assembles the package, strictly reads it back, and returns { bytes, info, metadata, permutationGraph, reflection, reflectionBlobs, glsl, inspection, qualification }.

The binary CEWG container remains version 1. New packages use:

  • INFO v2 plus PGRF v1 for version 8-14 input, explicitly without complete source reflection;
  • INFO v3 plus PGRF v1, all-unique WebGL RFLX v2, and shared exact RBLB bytes for version 15 input; and
  • the existing GLSL-set v1 backend graph.

PGRF preserves every Cartesian permutation and source alias. RFLX preserves every unique body's complete portable techniques, passes, stages, programs, exact constant-default bytes, resources, UAVs, samplers, signatures, annotations, states, and libraries. INFO binds PGRF/RFLX with SHA-256, while a caller-provided source SHA-256 is treated as an assertion against the exact input bytes.

Source completeness and backend translation qualification are separate. backendComplete remains false until the runtime binding/layout contract is packaged and validated, even when every GLSL program translates. Runtime completeness also remains false: CEWG carries immutable portable data, not live Tr2Shader objects or WebGL handles.

This API is browser-safe: it imports no filesystem, process, path, or native tool modules. A browser may fetch or select an .sm_* file, pass its bytes to buildEffect, and publish or download the returned CEWG bytes without a server-side conversion step.

Important options are source, sourceIdentity, allPermutations, technique, pass, stage, emitterOptions, includeSourceEffect, and the diagnostic-only allowFailures. Production conversion fails closed when any selected body, shader, or raster pass is incomplete.

The Node package:webgl command is an optional file adapter. Its normal JavaScript path delegates to this same public source-complete builder. It refuses to replace an existing output unless --overwrite or --force is supplied and never permits the output to replace the input effect. Its legacy native mode and specialized debug/resource-rewrite modes remain explicitly reflection-partial comparison tooling; tools-core builders use the browser-safe JavaScript API and never enable those modes.

The current runtime ownership boundary is documented in docs/effect-reflection.md. In short, runtime-resource owns package bytes, selection, per-index caching, and canonical Tr2Shader hydration; runtime-trinity consumes that graph through its mutable Tr2Effect/Tr2Material facade.

Indexed corpus builds

Do not use the format-local command to acquire or rebuild an indexed EVE or Frontier corpus. Agents producing packages for an engine, harness, build report, or persistent resource overlay must run the canonical tools-core builder from the @carbonenginejs/tools-core package:

npm.cmd run build:shader:webgl -- --shader-target eve-webgl2 --build latest --out <output>

Add --diagnostic to retain incomplete coverage for audit, or --force --no-reuse to transactionally replace and rebuild an existing output. The command writes build-report.json, durable JSONL progress, and a structured failure report. frontier-webgl2 uses the same command but its live protected index may require credentials.

The dependency direction is deliberately tools-core -> format-webgl. tools-core imports only this package's public root CjsFormatWebgl class; its transitive HLSL/DXBC imports are format-webgl's concern. Do not add tools-core as a format or browser dependency. Direct buildEffect use remains correct for browser conversion, library tests, and explicit one-file diagnostics.

Documentation

Reader Rules

  • Instance methods are PascalCase to avoid collisions with CarbonClass data.
  • Static one-shot methods are camelCase and live on CjsFormatWebgl.

Ported files

src/core/cewg/CewgPackage.js, CewgPackageBuilder.js, and src/core/glsl/DxbcGlslEmitter.js, DxbcGlslOperandFormatter.js, DxbcGlslHelpers.js are ported from this project's own prior-work repository, hlslreader (src/carbon/webgl/*.js, src/dxbc/glsl/*.js), with import paths and the emitter's DXBC-parsing prelude rewired onto @carbonenginejs/format-dxbc's public format (CjsFormatDxbc.read(bytes, { emit: "raw" })) instead of this package's own container/program/decoder classes, and error class renamed to this package's own CjsWebglReadError. Thrown messages are unchanged. See NOTICE for provenance.

Tests

npm test

Baseline tests are fully self-contained (synthetic DXBC bytes and synthetic CEWG chunk payloads assembled in-test) - no game assets, network access, or fixtures required. An optional corpus sweep emits GLSL for every vertex/pixel DXBC payload found under the directory supplied by WEBGL_CORPUS_DIR (raw magic scan, same approach as @carbonenginejs/format-dxbc's corpus test), and counts map-style compute successes/kill-list rejections without failing on them:

WEBGL_CORPUS_DIR=path/to/effect.dx11 npm test

License

MIT (see LICENSE and NOTICE).