Compare commits

..

66 Commits

Author SHA1 Message Date
Ingvar Stepanyan
b4c7792a93 Log decoding error 2020-11-03 12:52:13 +00:00
Ingvar Stepanyan
2a31a1bd84 Fix WebP2 signature 2020-11-03 12:52:13 +00:00
Ingvar Stepanyan
2c560659a7 Mark back as unstable 2020-11-03 12:52:13 +00:00
Ingvar Stepanyan
8155fc2f1f Rename to "WebP 2 (beta)" 2020-11-03 12:52:13 +00:00
Ingvar Stepanyan
c5446f234c Update WebP2 READMEs 2020-11-03 12:52:13 +00:00
Ingvar Stepanyan
bb6893e025 Update to newer memory management model 2020-11-03 12:52:13 +00:00
Ingvar Stepanyan
83d5357ac0 Wrap wp2 into timed 2020-11-03 12:52:13 +00:00
Ingvar Stepanyan
90706159fc Bind to WP2::EncoderConfig directly 2020-11-03 12:52:13 +00:00
Ingvar Stepanyan
c78f0c3747 Rebuild with new flags 2020-11-03 12:52:13 +00:00
Surma
0ba4b7cbd6 Update src/codecs/wp2/options.tsx 2020-11-03 12:52:13 +00:00
Surma
259597a70b Update src/codecs/processor-worker/index.ts 2020-11-03 12:52:13 +00:00
Surma
18b2bd1135 Add WebP2 support 2020-11-03 12:52:13 +00:00
Ingvar Stepanyan
e60ffddfd7 Delete .gitignore 2020-11-02 17:52:21 +00:00
Ingvar Stepanyan
d808c72798 Remove freeing todo 2020-11-02 17:52:17 +00:00
Ingvar Stepanyan
fcde9633ab Switch to -O3 for C++ codecs 2020-11-02 17:52:13 +00:00
Ingvar Stepanyan
e581075f64 Fix types after rebase onto latest dev 2020-11-02 17:52:10 +00:00
Ingvar Stepanyan
eb08d0049a Few more simplifications to JXL decoder 2020-11-02 17:52:08 +00:00
Ingvar Stepanyan
eed44dff97 Use make_unique; remove unnecessary cast 2020-11-02 17:52:06 +00:00
Ingvar Stepanyan
93474b025e Simplify JxlDecoder freeing
Use unique_ptr with custom deleter to enforce consistent deallocation, no matter how we leave the function.
2020-11-02 17:52:04 +00:00
Ingvar Stepanyan
5df81ad062 Remove unused free_result 2020-11-02 17:52:01 +00:00
Luca Versari
e00814ebe2 Update JXL to latest and do proper color management.
The JXL decoder now does colorspace conversion to sRGB correctly,
independently of the color space that the JPEG XL file was in.
2020-11-02 17:51:59 +00:00
Ingvar Stepanyan
4dab94686a XYB version 2020-11-02 17:51:56 +00:00
Ingvar Stepanyan
d9f65701e1 Switch decoder to C API 2020-11-02 17:51:53 +00:00
Ingvar Stepanyan
2c9f447399 Set default JXL encoder speed to 5 2020-11-02 17:51:50 +00:00
Ingvar Stepanyan
0e76dffaad Fix encoder colors 2020-11-02 17:51:47 +00:00
Ingvar Stepanyan
246d3ed6b3 Update jpeg-xl to latest upstream 2020-11-02 17:51:44 +00:00
Ingvar Stepanyan
df28fd434c Add timer to JPEG-XL 2020-11-02 17:51:40 +00:00
Surma
7e97b36249 Add alpha support 2020-11-02 17:51:37 +00:00
Surma
c2218bce90 Set default to 1 2020-11-02 17:51:33 +00:00
Surma
045cb14c54 Expose butteraugli distance 2020-11-02 17:51:29 +00:00
Surma
44a5b4921a Fix copy/paste error 2020-11-02 17:51:25 +00:00
Surma
18f7591c6a Update for encoder API changes 2020-11-02 17:51:22 +00:00
Surma
b8d0d5202d Update JXL library and build system 2020-11-02 17:51:17 +00:00
Surma
e26d8895a4 Exposed speed slider, but it's weird 2020-11-02 17:51:08 +00:00
Surma
b5ad52cfcf Fix JXL encoder 2020-11-02 17:51:04 +00:00
Surma
e1dd2796c1 Integrate encoder 2020-11-02 17:51:01 +00:00
Surma
98bc151907 Compile JXL encoder 2020-11-02 17:50:58 +00:00
Surma
36d23644aa Integrate JXL decoder 2020-11-02 17:50:55 +00:00
Surma
3697635bae Add test to JXL decoder 2020-11-02 17:50:50 +00:00
Surma
de1eb76226 Compile JXL decoder 2020-11-02 17:50:46 +00:00
Ingvar Stepanyan
fd87ae7d2a Force-rebuild codecs with -O3
Follow-up to https://github.com/GoogleChromeLabs/squoosh/pull/838.

I have no idea what the JS+Wasm diffs in the original PR even represented if they weren't proper rebuilds, but this time I just removed node_modules in all codecs to enforce a proper, clean rebuild for each C++ codec.

(Still think the speed-ups are worth it.)
2020-11-02 17:37:35 +00:00
Ingvar Stepanyan
5df7dd7590 Update helper.Makefile 2020-11-02 13:54:41 +00:00
Ingvar Stepanyan
013946b137 Pass CODEC_DIR and LIBAOM_DIR via export
Slightly simpler than passing them in HELPER_MAKEFLAGS.
2020-11-02 13:54:41 +00:00
Ingvar Stepanyan
81c183b0d6 Restructure the AVIF directories
Change the way AVIF finds AOM from default ([avif source]/ext/aom) to custom paths. This allows us to avoid unpacking same archives into duplicate folders, and instead make multiple builds from the same source.
2020-11-02 13:54:41 +00:00
Ingvar Stepanyan
f523db6403 Try out new flags for building only AVIF encoder/decoder
See the discussion in https://github.com/AOMediaCodec/libavif/issues/254 where this was implemented.

This allows us to avoid using ERROR_ON_UNDEFINED_SYMBOLS and build a truly separate encoder/decoder libs.
2020-11-02 13:54:41 +00:00
Ingvar Stepanyan
cc6ea9e11c Switch to -O3 for C++ codecs 2020-11-02 12:46:12 +00:00
Cătălin Mariș
bd4b67037b Further optimize logo.svg (#761)
Co-authored-by: Jake Archibald <jaffathecake@gmail.com>
2020-10-15 15:48:56 +01:00
Ingvar Stepanyan
8c5c97e106 Remove obsolete @ts-ignore 2020-10-07 20:42:48 +01:00
Ingvar Stepanyan
a9d3bd71b5 Bump oxipng
Integrating some upstream fixes from my branch.
2020-10-07 20:42:48 +01:00
Ingvar Stepanyan
0d0a9b4cdf Add COOP+COEP headers 2020-10-07 20:42:48 +01:00
Ingvar Stepanyan
f583770696 Explicitly disable HDR only for encoder 2020-10-07 20:42:48 +01:00
Ingvar Stepanyan
bae243ccdb Add feature detection to OxiPNG 2020-10-07 20:42:48 +01:00
Ingvar Stepanyan
02c113a68f Point oxipng to a patched version
Some upstream changes required for parallel build to work.
2020-10-07 20:42:48 +01:00
Ingvar Stepanyan
600eead007 Disable parallel feature for non-parallel OxiPNG 2020-10-07 20:42:48 +01:00
Ingvar Stepanyan
05416768d5 Update oxipng build system 2020-10-07 20:42:48 +01:00
Ingvar Stepanyan
35d31f2324 Add some comments to explain Rust thread glue 2020-10-07 20:42:48 +01:00
Ingvar Stepanyan
82fadac70e Fixup import.meta in OxiPNG 2020-10-07 20:42:48 +01:00
Ingvar Stepanyan
47f9d22dd8 Switch to crossbeam-channel
Still not perfect due to usage of a static global, but this is much cleaner and more efficient thanks to proper blocking of Workers that wait for new messages instead of a manual spin-loop.
2020-10-07 20:42:48 +01:00
Ingvar Stepanyan
9420dba3bc Parallel OxiPNG improvements
- Refactor to work around Chromium's issue with postMessage queuing. https://bugs.chromium.org/p/chromium/issues/detail?id=1075645
 - Convert codec code to TypeScript.
 - Make separate parallel and non-parallel builds.
 - Switch to nightly Rust for OxiPNG to allow parallel builds (but also reuse it for regular builds to avoid installing two toolchains).
2020-10-07 20:42:48 +01:00
Ingvar Stepanyan
e462875807 Type fix for gesturestart event 2020-10-07 20:42:48 +01:00
Ingvar Stepanyan
0747d2c419 Rework fallback for postMessage issue
Now initialise all workers with module+memory separately, and then instead of using postMessage to send thread pointers, push them into a crossbeam-deque on the Rust side.

Rayon already depends on crossbeam-dequeue, so we're not even adding another dependency, and this model allows us to push "tasks" (thread pointers) on the main thread and pop them on worker threads in arbitrary order without sacrificing correctness.
2020-10-07 20:42:48 +01:00
Ingvar Stepanyan
4c658b79ef OxiPNG + threads PoC 2020-10-07 20:42:48 +01:00
Ingvar Stepanyan
685558847f Multithread AVIF PoC 2020-10-07 20:42:48 +01:00
Trevor Manz
63ac34a662 Promisify emscripten modules & fix webp examples (#817) 2020-09-30 00:05:59 +01:00
Surma
42f9e4aed2 Merge pull request #828 from GoogleChromeLabs/create-dir
Ensure node_modules is created
2020-09-16 10:36:11 +01:00
Jake Archibald
e14790f0b9 Ensure node_modules is created 2020-09-16 10:24:20 +01:00
335 changed files with 20153 additions and 36987 deletions

9
.gitignore vendored
View File

@@ -1,11 +1,6 @@
.tmp
node_modules
/build
/*.log
*.scss.d.ts
*.css.d.ts
build
*.o
# Auto-generated by lib/feature-plugin.js
src/features-worker/index.ts
src/client/lazy-app/worker-bridge/meta.ts
src/client/lazy-app/feature-meta/index.ts

2
.nvmrc
View File

@@ -1 +1 @@
12.18.3
10.16.2

View File

@@ -1,4 +0,0 @@
{
"singleQuote": true,
"trailingComma": "all"
}

23
_headers.ejs Normal file
View File

@@ -0,0 +1,23 @@
# Long-term cache by default.
/*
Cache-Control: max-age=31536000
# And here are the exceptions:
/
Cache-Control: no-cache
/serviceworker.js
Cache-Control: no-cache
/manifest.json
Cache-Control: must-revalidate, max-age=3600
# URLs in /assets do not include a hash and are mutable.
# But it isn't a big deal if the user gets an old version.
/assets/*
Cache-Control: must-revalidate, max-age=3600
# COOP+COEP for WebAssembly threads.
/*
Cross-Origin-Embedder-Policy: require-corp
Cross-Origin-Opener-Policy: same-origin

View File

@@ -1,19 +0,0 @@
{
"extends": "./generic-tsconfig.json",
"compilerOptions": {
"lib": ["esnext", "dom", "dom.iterable"],
"types": []
},
"include": [
"src/features/**/client/**/*",
"src/features/**/shared/**/*",
"src/features/client-utils/**/*",
"src/shared/**/*",
"src/client/**/*",
// Not really clean, but we need these to access the type of the functions
// for comlink
"src/features/**/worker/**/*",
"src/features-worker/**/*",
"src/features/worker-utils/**/*"
]
}

View File

@@ -1,7 +1,6 @@
export interface AVIFModule extends EmscriptenWasm.Module {
interface AVIFModule extends EmscriptenWasm.Module {
decode(data: BufferSource): ImageData | null;
}
declare var moduleFactory: EmscriptenWasm.ModuleFactory<AVIFModule>;
export default function(opts: EmscriptenWasm.ModuleOpts): Promise<AVIFModule>;
export default moduleFactory;

File diff suppressed because it is too large Load Diff

Binary file not shown.

View File

@@ -1,6 +1,6 @@
#include <emscripten/bind.h>
#include <emscripten/threading.h>
#include <emscripten/val.h>
#include <emscripten/threading.h>
#include "avif/avif.h"
using namespace emscripten;
@@ -51,10 +51,13 @@ val encode(std::string buffer, int width, int height, AvifOptions options) {
avifImage* image = avifImageCreate(width, height, depth, format);
if (options.maxQuantizer == AVIF_QUANTIZER_LOSSLESS &&
options.minQuantizer == AVIF_QUANTIZER_LOSSLESS &&
options.minQuantizerAlpha == AVIF_QUANTIZER_LOSSLESS &&
options.maxQuantizerAlpha == AVIF_QUANTIZER_LOSSLESS && format == AVIF_PIXEL_FORMAT_YUV444) {
if (
options.maxQuantizer == AVIF_QUANTIZER_LOSSLESS &&
options.minQuantizer == AVIF_QUANTIZER_LOSSLESS &&
options.minQuantizerAlpha == AVIF_QUANTIZER_LOSSLESS &&
options.maxQuantizerAlpha == AVIF_QUANTIZER_LOSSLESS &&
format == AVIF_PIXEL_FORMAT_YUV444
) {
image->matrixCoefficients = AVIF_MATRIX_COEFFICIENTS_IDENTITY;
} else {
image->matrixCoefficients = AVIF_MATRIX_COEFFICIENTS_BT709;

View File

@@ -1,23 +1,7 @@
export interface EncodeOptions {
minQuantizer: number;
maxQuantizer: number;
minQuantizerAlpha: number;
maxQuantizerAlpha: number;
tileRowsLog2: number;
tileColsLog2: number;
speed: number;
subsample: number;
import { EncodeOptions } from '../../../src/codecs/avif/encoder-meta';
interface AVIFModule extends EmscriptenWasm.Module {
encode(data: BufferSource, width: number, height: number, options: EncodeOptions): Uint8Array | null;
}
export interface AVIFModule extends EmscriptenWasm.Module {
encode(
data: BufferSource,
width: number,
height: number,
options: EncodeOptions,
): Uint8Array | null;
}
declare var moduleFactory: EmscriptenWasm.ModuleFactory<AVIFModule>;
export default moduleFactory;
export default function(opts: EmscriptenWasm.ModuleOpts): Promise<AVIFModule>;

File diff suppressed because it is too large Load Diff

Binary file not shown.

View File

@@ -1,2 +0,0 @@
export * from './avif_enc';
export { default } from './avif_enc';

File diff suppressed because it is too large Load Diff

BIN
codecs/avif/enc/avif_enc_mt.wasm Executable file → Normal file

Binary file not shown.

View File

@@ -1,103 +1 @@
var threadInfoStruct = 0;
var selfThreadId = 0;
var parentThreadId = 0;
var initializedJS = false;
var Module = {};
function threadPrintErr() {
var text = Array.prototype.slice.call(arguments).join(' ');
console.error(text);
}
function threadAlert() {
var text = Array.prototype.slice.call(arguments).join(' ');
postMessage({ cmd: 'alert', text: text, threadId: selfThreadId });
}
var err = threadPrintErr;
this.alert = threadAlert;
Module['instantiateWasm'] = function (info, receiveInstance) {
var instance = new WebAssembly.Instance(Module['wasmModule'], info);
Module['wasmModule'] = null;
receiveInstance(instance);
return instance.exports;
};
this.onmessage = function (e) {
try {
if (e.data.cmd === 'load') {
Module['wasmModule'] = e.data.wasmModule;
Module['wasmMemory'] = e.data.wasmMemory;
Module['buffer'] = Module['wasmMemory'].buffer;
Module['ENVIRONMENT_IS_PTHREAD'] = true;
import(e.data.urlOrBlob)
.then(function (avif_enc_mt) {
return avif_enc_mt.default(Module);
})
.then(function (instance) {
Module = instance;
postMessage({ cmd: 'loaded' });
});
} else if (e.data.cmd === 'objectTransfer') {
Module['PThread'].receiveObjectTransfer(e.data);
} else if (e.data.cmd === 'run') {
Module['__performance_now_clock_drift'] = performance.now() - e.data.time;
threadInfoStruct = e.data.threadInfoStruct;
Module['registerPthreadPtr'](
threadInfoStruct,
/*isMainBrowserThread=*/ 0,
/*isMainRuntimeThread=*/ 0,
);
selfThreadId = e.data.selfThreadId;
parentThreadId = e.data.parentThreadId;
var max = e.data.stackBase;
var top = e.data.stackBase + e.data.stackSize;
Module['establishStackSpace'](top, max);
Module['_emscripten_tls_init']();
Module['PThread'].receiveObjectTransfer(e.data);
Module['PThread'].setThreadStatus(Module['_pthread_self'](), 1);
if (!initializedJS) {
Module['___embind_register_native_and_builtin_types']();
initializedJS = true;
}
try {
var result = Module['dynCall']('ii', e.data.start_routine, [
e.data.arg,
]);
if (!Module['getNoExitRuntime']()) Module['PThread'].threadExit(result);
} catch (ex) {
if (ex === 'Canceled!') {
Module['PThread'].threadCancel();
} else if (ex != 'unwind') {
Atomics.store(
Module['HEAPU32'],
(threadInfoStruct + 4) >> /*C_STRUCTS.pthread.threadExitCode*/ 2,
ex instanceof Module['ExitStatus'] ? ex.status : -2,
);
/*A custom entry specific to Emscripten denoting that the thread crashed.*/ Atomics.store(
Module['HEAPU32'],
(threadInfoStruct + 0) >> /*C_STRUCTS.pthread.threadStatus*/ 2,
1,
);
Module['_emscripten_futex_wake'](
threadInfoStruct + 0,
/*C_STRUCTS.pthread.threadStatus*/ 2147483647,
);
if (!(ex instanceof Module['ExitStatus'])) throw ex;
}
}
} else if (e.data.cmd === 'cancel') {
if (threadInfoStruct) {
Module['PThread'].threadCancel();
}
} else if (e.data.target === 'setimmediate') {
} else if (e.data.cmd === 'processThreadQueue') {
if (threadInfoStruct) {
Module['_emscripten_current_thread_process_queued_calls']();
}
} else {
err('worker.js received unknown command ' + e.data.cmd);
err(e.data);
}
} catch (ex) {
err('worker.js onmessage() captured an uncaught exception: ' + ex);
if (ex && ex.stack) err(ex.stack);
throw ex;
}
};
var threadInfoStruct=0;var selfThreadId=0;var parentThreadId=0;var Module={};function threadPrintErr(){var text=Array.prototype.slice.call(arguments).join(" ");console.error(text)}function threadAlert(){var text=Array.prototype.slice.call(arguments).join(" ");postMessage({cmd:"alert",text:text,threadId:selfThreadId})}var err=threadPrintErr;this.alert=threadAlert;Module["instantiateWasm"]=function(info,receiveInstance){var instance=new WebAssembly.Instance(Module["wasmModule"],info);Module["wasmModule"]=null;receiveInstance(instance);return instance.exports};this.onmessage=function(e){try{if(e.data.cmd==="load"){Module["DYNAMIC_BASE"]=e.data.DYNAMIC_BASE;Module["DYNAMICTOP_PTR"]=e.data.DYNAMICTOP_PTR;Module["wasmModule"]=e.data.wasmModule;Module["wasmMemory"]=e.data.wasmMemory;Module["buffer"]=Module["wasmMemory"].buffer;Module["ENVIRONMENT_IS_PTHREAD"]=true;if(typeof e.data.urlOrBlob==="string"){importScripts(e.data.urlOrBlob)}else{var objectUrl=URL.createObjectURL(e.data.urlOrBlob);importScripts(objectUrl);URL.revokeObjectURL(objectUrl)}avif_enc_mt(Module).then(function(instance){Module=instance;postMessage({"cmd":"loaded"})})}else if(e.data.cmd==="objectTransfer"){Module["PThread"].receiveObjectTransfer(e.data)}else if(e.data.cmd==="run"){Module["__performance_now_clock_drift"]=performance.now()-e.data.time;threadInfoStruct=e.data.threadInfoStruct;Module["registerPthreadPtr"](threadInfoStruct,/*isMainBrowserThread=*/0,/*isMainRuntimeThread=*/0);selfThreadId=e.data.selfThreadId;parentThreadId=e.data.parentThreadId;var max=e.data.stackBase;var top=e.data.stackBase+e.data.stackSize;Module["establishStackSpace"](top,max);Module["_emscripten_tls_init"]();Module["PThread"].receiveObjectTransfer(e.data);Module["PThread"].setThreadStatus(Module["_pthread_self"](),1);/*EM_THREAD_STATUS_RUNNING*/try{var result=Module["dynCall_ii"](e.data.start_routine,e.data.arg);if(!Module["getNoExitRuntime"]())Module["PThread"].threadExit(result)}catch(ex){if(ex==="Canceled!"){Module["PThread"].threadCancel()}else if(ex!="unwind"){Atomics.store(Module["HEAPU32"],(threadInfoStruct+4)>>/*C_STRUCTS.pthread.threadExitCode*/2,(ex instanceof Module["ExitStatus"])?ex.status:-2);/*A custom entry specific to Emscripten denoting that the thread crashed.*/Atomics.store(Module["HEAPU32"],(threadInfoStruct+0)>>/*C_STRUCTS.pthread.threadStatus*/2,1);Module["_emscripten_futex_wake"](threadInfoStruct+0,/*C_STRUCTS.pthread.threadStatus*/2147483647);if(!(ex instanceof Module["ExitStatus"]))throw ex}}}else if(e.data.cmd==="cancel"){if(threadInfoStruct){Module["PThread"].threadCancel()}}else if(e.data.target==="setimmediate"){}else if(e.data.cmd==="processThreadQueue"){if(threadInfoStruct){Module["_emscripten_current_thread_process_queued_calls"]()}}else{err("worker.js received unknown command "+e.data.cmd);err(e.data)}}catch(ex){err("worker.js onmessage() captured an uncaught exception: "+ex);if(ex.stack)err(ex.stack);throw ex}};if(typeof process==="object"&&typeof process.versions==="object"&&typeof process.versions.node==="string"){self={location:{href:__filename}};var onmessage=this.onmessage;var nodeWorkerThreads=require("worker_threads");global.Worker=nodeWorkerThreads.Worker;var parentPort=nodeWorkerThreads.parentPort;parentPort.on("message",function(data){onmessage({data:data})});var nodeFS=require("fs");var nodeRead=function(filename){return nodeFS.readFileSync(filename,"utf8")};function globalEval(x){global.require=require;global.Module=Module;eval.call(null,x)}importScripts=function(f){globalEval(nodeRead(f))};postMessage=function(msg){parentPort.postMessage(msg)};if(typeof performance==="undefined"){performance={now:function(){return Date.now()}}}}

View File

@@ -32,10 +32,7 @@ $(OUT_JS): $(OUT_CPP) $(LIBAOM_OUT) $(CODEC_OUT)
--closure 1 \
-s ALLOW_MEMORY_GROWTH=1 \
-s MODULARIZE=1 \
-s TEXTDECODER=2 \
-s ENVIRONMENT='worker' \
-s EXPORT_ES6=1 \
-s EXPORT_NAME="$(basename $(@F))" \
-s 'EXPORT_NAME="$(basename $(@F))"' \
-o $@ \
$+

View File

@@ -0,0 +1,4 @@
set -e
docker build -t squoosh-rust-nightly --build-arg RUST_IMG=rustlang/rust:nightly - < ../rust.Dockerfile
docker run --rm -v $PWD:/src squoosh-rust-nightly "$@"

View File

@@ -1,10 +1,4 @@
set -e
if [ ! -z "$RUST_IMG" ]
then
# Get part after ":" (https://stackoverflow.com/a/15149278/439965).
IMG_SUFFIX=-${RUST_IMG#*:}
fi
IMG_NAME=squoosh-rust$IMG_SUFFIX
docker build -t $IMG_NAME --build-arg RUST_IMG - < ../rust.Dockerfile
docker run --rm -v $PWD:/src $IMG_NAME "$@"
docker build -t squoosh-rust - < ../rust.Dockerfile
docker run --rm -v $PWD:/src squoosh-rust "$@"

View File

@@ -1,4 +1,4 @@
FROM emscripten/emsdk:2.0.8
FROM emscripten/emsdk:1.40.0
RUN apt-get update && apt-get install -qqy autoconf libtool pkg-config
ENV CFLAGS "-O3 -flto"
ENV CXXFLAGS "${CFLAGS} -std=c++17"

11
codecs/hqx/Dockerfile Normal file
View File

@@ -0,0 +1,11 @@
# This is intentionally an old version of Rust. Newer versions
# generate _significantly_ bigger Wasm binaries.
FROM rust:1.37
RUN rustup target add wasm32-unknown-unknown
RUN curl https://rustwasm.github.io/wasm-pack/installer/init.sh -sSf | sh
RUN mkdir /opt/binaryen && \
curl -L https://github.com/WebAssembly/binaryen/releases/download/1.38.32/binaryen-1.38.32-x86-linux.tar.gz | tar -xzf - -C /opt/binaryen --strip 1
ENV PATH="/opt/binaryen:${PATH}"
WORKDIR /src

21
codecs/hqx/build.sh Executable file
View File

@@ -0,0 +1,21 @@
#!/bin/bash
set -e
echo "============================================="
echo "Compiling wasm"
echo "============================================="
(
wasm-pack build -- --verbose --locked
rm pkg/.gitignore
)
echo "============================================="
echo "Compiling wasm done"
echo "============================================="
echo "!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!"
echo "Did you update your docker image?"
echo "Run \`docker pull ubuntu\`"
echo "Run \`docker pull rust\`"
echo "Run \`docker build -t squoosh-hqx .\`"
echo "!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!"

View File

@@ -1,6 +1,7 @@
{
"name": "hqx",
"scripts": {
"build": "RUST_IMG=rust:1.40 ../build-rust.sh"
"build:image": "docker build -t squoosh-hqx - < Dockerfile",
"build": "docker run --rm -v $(pwd):/src squoosh-hqx ./build.sh"
}
}

View File

@@ -1,5 +0,0 @@
# HQX
- Source: <https://github.com/CryZe/wasmboy-rs>
- Version: v0.1.2
- License: Apache 2.0

View File

@@ -1,48 +1,10 @@
/* tslint:disable */
/* eslint-disable */
/**
* @param {Uint32Array} input_image
* @param {number} input_width
* @param {number} input_height
* @param {number} factor
* @returns {Uint32Array}
*/
export function resize(
input_image: Uint32Array,
input_width: number,
input_height: number,
factor: number,
): Uint32Array;
export type InitInput =
| RequestInfo
| URL
| Response
| BufferSource
| WebAssembly.Module;
export interface InitOutput {
readonly memory: WebAssembly.Memory;
readonly resize: (
a: number,
b: number,
c: number,
d: number,
e: number,
f: number,
) => void;
readonly __wbindgen_malloc: (a: number) => number;
readonly __wbindgen_free: (a: number, b: number) => void;
}
/**
* If `module_or_path` is {RequestInfo} or {URL}, makes a request and
* for everything else, calls `WebAssembly.instantiate` directly.
*
* @param {InitInput | Promise<InitInput>} module_or_path
*
* @returns {Promise<InitOutput>}
*/
export default function init(
module_or_path?: InitInput | Promise<InitInput>,
): Promise<InitOutput>;
* @param {Uint32Array} input_image
* @param {number} input_width
* @param {number} input_height
* @param {number} factor
* @returns {Uint32Array}
*/
export function resize(input_image: Uint32Array, input_width: number, input_height: number, factor: number): Uint32Array;

View File

@@ -1,107 +1,2 @@
let wasm;
let cachegetUint32Memory0 = null;
function getUint32Memory0() {
if (
cachegetUint32Memory0 === null ||
cachegetUint32Memory0.buffer !== wasm.memory.buffer
) {
cachegetUint32Memory0 = new Uint32Array(wasm.memory.buffer);
}
return cachegetUint32Memory0;
}
let WASM_VECTOR_LEN = 0;
function passArray32ToWasm0(arg, malloc) {
const ptr = malloc(arg.length * 4);
getUint32Memory0().set(arg, ptr / 4);
WASM_VECTOR_LEN = arg.length;
return ptr;
}
let cachegetInt32Memory0 = null;
function getInt32Memory0() {
if (
cachegetInt32Memory0 === null ||
cachegetInt32Memory0.buffer !== wasm.memory.buffer
) {
cachegetInt32Memory0 = new Int32Array(wasm.memory.buffer);
}
return cachegetInt32Memory0;
}
function getArrayU32FromWasm0(ptr, len) {
return getUint32Memory0().subarray(ptr / 4, ptr / 4 + len);
}
/**
* @param {Uint32Array} input_image
* @param {number} input_width
* @param {number} input_height
* @param {number} factor
* @returns {Uint32Array}
*/
export function resize(input_image, input_width, input_height, factor) {
var ptr0 = passArray32ToWasm0(input_image, wasm.__wbindgen_malloc);
var len0 = WASM_VECTOR_LEN;
wasm.resize(8, ptr0, len0, input_width, input_height, factor);
var r0 = getInt32Memory0()[8 / 4 + 0];
var r1 = getInt32Memory0()[8 / 4 + 1];
var v1 = getArrayU32FromWasm0(r0, r1).slice();
wasm.__wbindgen_free(r0, r1 * 4);
return v1;
}
async function load(module, imports) {
if (typeof Response === 'function' && module instanceof Response) {
if (typeof WebAssembly.instantiateStreaming === 'function') {
try {
return await WebAssembly.instantiateStreaming(module, imports);
} catch (e) {
if (module.headers.get('Content-Type') != 'application/wasm') {
console.warn(
'`WebAssembly.instantiateStreaming` failed because your server does not serve wasm with `application/wasm` MIME type. Falling back to `WebAssembly.instantiate` which is slower. Original error:\n',
e,
);
} else {
throw e;
}
}
}
const bytes = await module.arrayBuffer();
return await WebAssembly.instantiate(bytes, imports);
} else {
const instance = await WebAssembly.instantiate(module, imports);
if (instance instanceof WebAssembly.Instance) {
return { instance, module };
} else {
return instance;
}
}
}
async function init(input) {
if (typeof input === 'undefined') {
input = import.meta.url.replace(/\.js$/, '_bg.wasm');
}
const imports = {};
if (
typeof input === 'string' ||
(typeof Request === 'function' && input instanceof Request) ||
(typeof URL === 'function' && input instanceof URL)
) {
input = fetch(input);
}
const { instance, module } = await load(await input, imports);
wasm = instance.exports;
init.__wbindgen_wasm_module = module;
return wasm;
}
export default init;
import * as wasm from "./squooshhqx_bg.wasm";
export * from "./squooshhqx_bg.js";

View File

@@ -0,0 +1,48 @@
import * as wasm from './squooshhqx_bg.wasm';
let cachegetUint32Memory0 = null;
function getUint32Memory0() {
if (cachegetUint32Memory0 === null || cachegetUint32Memory0.buffer !== wasm.memory.buffer) {
cachegetUint32Memory0 = new Uint32Array(wasm.memory.buffer);
}
return cachegetUint32Memory0;
}
let WASM_VECTOR_LEN = 0;
function passArray32ToWasm0(arg, malloc) {
const ptr = malloc(arg.length * 4);
getUint32Memory0().set(arg, ptr / 4);
WASM_VECTOR_LEN = arg.length;
return ptr;
}
let cachegetInt32Memory0 = null;
function getInt32Memory0() {
if (cachegetInt32Memory0 === null || cachegetInt32Memory0.buffer !== wasm.memory.buffer) {
cachegetInt32Memory0 = new Int32Array(wasm.memory.buffer);
}
return cachegetInt32Memory0;
}
function getArrayU32FromWasm0(ptr, len) {
return getUint32Memory0().subarray(ptr / 4, ptr / 4 + len);
}
/**
* @param {Uint32Array} input_image
* @param {number} input_width
* @param {number} input_height
* @param {number} factor
* @returns {Uint32Array}
*/
export function resize(input_image, input_width, input_height, factor) {
var ptr0 = passArray32ToWasm0(input_image, wasm.__wbindgen_malloc);
var len0 = WASM_VECTOR_LEN;
wasm.resize(8, ptr0, len0, input_width, input_height, factor);
var r0 = getInt32Memory0()[8 / 4 + 0];
var r1 = getInt32Memory0()[8 / 4 + 1];
var v1 = getArrayU32FromWasm0(r0, r1).slice();
wasm.__wbindgen_free(r0, r1 * 4);
return v1;
}

Binary file not shown.

View File

@@ -18,9 +18,7 @@ all: $(OUT_JS)
--closure 1 \
-s ALLOW_MEMORY_GROWTH=1 \
-s MODULARIZE=1 \
-s TEXTDECODER=2 \
-s ENVIRONMENT='worker' \
-s EXPORT_ES6=1 \
-s 'EXPORT_NAME="$(basename $(@F))"' \
-o $@ \
$+

View File

@@ -1,17 +1,18 @@
<!DOCTYPE html>
<!doctype html>
<style>
canvas {
image-rendering: pixelated;
}
</style>
<script type="module">
import imagequant from './imagequant.js';
<script src='imagequant.js'></script>
<script>
const Module = imagequant();
async function loadImage(src) {
// Load image
const img = document.createElement('img');
img.src = src;
await new Promise((resolve) => (img.onload = resolve));
await new Promise(resolve => img.onload = resolve);
// Make canvas same size as image
const canvas = document.createElement('canvas');
[canvas.width, canvas.height] = [img.width, img.height];
@@ -21,32 +22,19 @@
return ctx.getImageData(0, 0, img.width, img.height);
}
async function main() {
const module = await imagequant();
console.log('Version:', module.version().toString(16));
Module.onRuntimeInitialized = async _ => {
console.log('Version:', Module.version().toString(16));
const image = await loadImage('../example.png');
const rawImage = module.quantize(
image.data,
image.width,
image.height,
256,
1.0,
);
// const rawImage = Module.quantize(image.data, image.width, image.height, 256, 1.0);
const rawImage = Module.zx_quantize(image.data, image.width, image.height, 1.0);
console.log('done');
const imageData = new ImageData(
new Uint8ClampedArray(rawImage.buffer),
image.width,
image.height,
);
const imageData = new ImageData(new Uint8ClampedArray(rawImage.buffer), image.width, image.height);
const canvas = document.createElement('canvas');
canvas.width = image.width;
canvas.height = image.height;
const ctx = canvas.getContext('2d');
ctx.putImageData(imageData, 0, 0);
document.body.appendChild(canvas);
}
main();
};
</script>

View File

@@ -1,19 +1,6 @@
export interface QuantizerModule extends EmscriptenWasm.Module {
quantize(
data: BufferSource,
width: number,
height: number,
numColors: number,
dither: number,
): Uint8ClampedArray;
zx_quantize(
data: BufferSource,
width: number,
height: number,
dither: number,
): Uint8ClampedArray;
interface QuantizerModule extends EmscriptenWasm.Module {
quantize(data: BufferSource, width: number, height: number, numColors: number, dither: number): Uint8ClampedArray;
zx_quantize(data: BufferSource, width: number, height: number, dither: number): Uint8ClampedArray;
}
declare var moduleFactory: EmscriptenWasm.ModuleFactory<QuantizerModule>;
export default moduleFactory;
export default function(opts: EmscriptenWasm.ModuleOpts): Promise<QuantizerModule>;

File diff suppressed because it is too large Load Diff

BIN
codecs/imagequant/imagequant.wasm Executable file → Normal file

Binary file not shown.

View File

@@ -1,5 +1,5 @@
CODEC_URL = https://gitlab.com/wg1/jpeg-xl.git
CODEC_VERSION = v0.1
CODEC_VERSION = 4d70bd58fbcb758b61446aba447fedd9177a00c9
CODEC_DIR = node_modules/jxl
CODEC_BUILD_DIR := $(CODEC_DIR)/build
CODEC_OUT := $(CODEC_BUILD_DIR)/lib/libjxl.a
@@ -27,10 +27,7 @@ all: $(OUT_JS)
--closure 1 \
-s ALLOW_MEMORY_GROWTH=1 \
-s MODULARIZE=1 \
-s TEXTDECODER=2 \
-s ENVIRONMENT='worker' \
-s EXPORT_ES6=1 \
-s EXPORT_NAME="$(basename $(@F))" \
-s 'EXPORT_NAME="$(basename $(@F))"' \
-o $@ \
$+ \
$(CODEC_BUILD_DIR)/artifacts/libbrunslienc-static.bc \
@@ -51,8 +48,18 @@ $(CODEC_OUT): $(CODEC_DIR)/CMakeLists.txt
$(CODEC_DIR)/CMakeLists.txt: $(CODEC_DIR)
$(CODEC_DIR):
# The JXL repository doesnt have version tags or anything yet,
# so we have to pin to a specific commit for now. This implies we
# cant use --recursive, as we will change commit after checkout (it
# seems you cant clone a specific commit directly), and it also means
# we cant use --depth 1 because we want to change commits.
# The JXL code base also relies on submodules so we cant just download
# a .tar.gz from GitLab.
mkdir -p $@
git clone $(CODEC_URL) --recursive -j`nproc` --depth 1 --branch $(CODEC_VERSION) $@
git clone $(CODEC_URL) $@
cd $@ && \
git reset --hard $(CODEC_VERSION) && \
git submodule update --init --recursive
clean:
$(RM) $(OUT_JS) $(OUT_WASM)

View File

@@ -14,10 +14,9 @@ thread_local const val ImageData = val::global("ImageData");
// R, G, B, A
#define COMPONENTS_PER_PIXEL 4
#define EXPECT_TRUE(a) \
if (!(a)) { \
return val::null(); \
}
#define EXPECT_TRUE(a) \
if (!(a)) \
return val::null();
#define EXPECT_EQ(a, b) EXPECT_TRUE((a) == (b));
@@ -38,16 +37,16 @@ val decode(std::string data) {
size_t component_count = pixel_count * COMPONENTS_PER_PIXEL;
EXPECT_EQ(JXL_DEC_COLOR_ENCODING, JxlDecoderProcessInput(dec.get(), &next_in, &avail_in));
static const JxlPixelFormat format = {COMPONENTS_PER_PIXEL, JXL_TYPE_FLOAT, JXL_LITTLE_ENDIAN, 0};
size_t icc_size;
EXPECT_EQ(JXL_DEC_SUCCESS, JxlDecoderGetICCProfileSize(dec.get(), &format,
JXL_COLOR_PROFILE_TARGET_DATA, &icc_size));
EXPECT_EQ(JXL_DEC_SUCCESS,
JxlDecoderGetICCProfileSize(dec.get(), JXL_COLOR_PROFILE_TARGET_DATA, &icc_size));
std::vector<uint8_t> icc_profile(icc_size);
EXPECT_EQ(JXL_DEC_SUCCESS,
JxlDecoderGetColorAsICCProfile(dec.get(), &format, JXL_COLOR_PROFILE_TARGET_DATA,
JxlDecoderGetColorAsICCProfile(dec.get(), JXL_COLOR_PROFILE_TARGET_DATA,
icc_profile.data(), icc_profile.size()));
auto float_pixels = std::make_unique<float[]>(component_count);
static const JxlPixelFormat format = {COMPONENTS_PER_PIXEL, JXL_LITTLE_ENDIAN, JXL_TYPE_FLOAT};
EXPECT_EQ(JXL_DEC_SUCCESS, JxlDecoderSetImageOutBuffer(dec.get(), &format, float_pixels.get(),
component_count * sizeof(float)));
EXPECT_EQ(JXL_DEC_FULL_IMAGE, JxlDecoderProcessInput(dec.get(), &next_in, &avail_in));
@@ -55,7 +54,16 @@ val decode(std::string data) {
auto byte_pixels = std::make_unique<uint8_t[]>(component_count);
// Convert to sRGB.
skcms_ICCProfile jxl_profile;
EXPECT_TRUE(skcms_Parse(icc_profile.data(), icc_profile.size(), &jxl_profile));
// If the image is encoded in its original color space, the decoded data will be in the color
// space defined by the decoded ICC profile. Otherwise, it is in Linear sRGB. TODO: the decoded
// color profile should also be Linear sRGB if !uses_original_profile.
if (info.uses_original_profile) {
EXPECT_TRUE(skcms_Parse(icc_profile.data(), icc_profile.size(), &jxl_profile));
} else {
auto s = jxl::ColorEncoding::LinearSRGB(/*gray=*/false);
EXPECT_TRUE(s.CreateICC());
EXPECT_TRUE(skcms_Parse(s.ICC().data(), s.ICC().size(), &jxl_profile));
}
EXPECT_TRUE(skcms_Transform(
float_pixels.get(), skcms_PixelFormat_RGBA_ffff,
info.alpha_premultiplied ? skcms_AlphaFormat_PremulAsEncoded : skcms_AlphaFormat_Unpremul,

View File

@@ -1,7 +1,5 @@
export interface JXLModule extends EmscriptenWasm.Module {
interface JXLModule extends EmscriptenWasm.Module {
decode(data: BufferSource): ImageData | null;
}
declare var moduleFactory: EmscriptenWasm.ModuleFactory<JXLModule>;
export default moduleFactory;
export default function(opts: EmscriptenWasm.ModuleOpts): Promise<JXLModule>;

File diff suppressed because it is too large Load Diff

BIN
codecs/jxl/dec/jxl_dec.wasm Executable file → Normal file

Binary file not shown.

View File

@@ -13,10 +13,6 @@ struct JXLOptions {
// 7 = fastest
int speed;
float quality;
bool progressive;
int epf;
int nearLossless;
bool lossyPalette;
};
val encode(std::string image, int width, int height, JXLOptions options) {
@@ -26,61 +22,10 @@ val encode(std::string image, int width, int height, JXLOptions options) {
jxl::PaddedBytes bytes;
jxl::ImageBundle* main = &io.Main();
cparams.epf = options.epf;
cparams.speed_tier = static_cast<jxl::SpeedTier>(options.speed);
cparams.near_lossless = options.nearLossless;
cparams.butteraugli_distance = options.quality;
if (options.lossyPalette) {
cparams.lossy_palette = true;
cparams.palette_colors = 0;
cparams.options.predictor = jxl::Predictor::Zero;
}
// Reduce memory usage of tree learning for lossless data.
// TODO(veluca93): this is a mitigation for excessive memory usage in the JXL encoder.
cparams.options.nb_repeats = 0.1;
float quality = options.quality;
// Quality settings roughly match libjpeg qualities.
if (quality < 7 || quality == 100) {
cparams.modular_mode = true;
// Internal modular quality to roughly match VarDCT size.
cparams.quality_pair.first = cparams.quality_pair.second =
std::min(35 + (quality - 7) * 3.0f, 100.0f);
} else {
cparams.modular_mode = false;
if (quality >= 30) {
cparams.butteraugli_distance = 0.1 + (100 - quality) * 0.09;
} else {
cparams.butteraugli_distance = 6.4 + pow(2.5, (30 - quality) / 5.0f) / 6.25f;
}
}
if (options.progressive) {
cparams.qprogressive_mode = true;
cparams.progressive_dc = 1;
cparams.responsive = 1;
}
if (cparams.modular_mode) {
if (cparams.quality_pair.first != 100 || cparams.quality_pair.second != 100) {
cparams.color_transform = jxl::ColorTransform::kXYB;
} else {
cparams.color_transform = jxl::ColorTransform::kNone;
}
}
if (cparams.near_lossless) {
// Near-lossless assumes -R 0
cparams.responsive = 0;
cparams.modular_mode = true;
}
io.metadata.m.SetAlphaBits(8);
if (!io.metadata.size.Set(width, height)) {
return val::null();
}
io.metadata.SetAlphaBits(8);
uint8_t* inBuffer = (uint8_t*)image.c_str();
@@ -105,11 +50,7 @@ val encode(std::string image, int width, int height, JXLOptions options) {
EMSCRIPTEN_BINDINGS(my_module) {
value_object<JXLOptions>("JXLOptions")
.field("speed", &JXLOptions::speed)
.field("quality", &JXLOptions::quality)
.field("progressive", &JXLOptions::progressive)
.field("nearLossless", &JXLOptions::nearLossless)
.field("lossyPalette", &JXLOptions::lossyPalette)
.field("epf", &JXLOptions::epf);
.field("quality", &JXLOptions::quality);
function("encode", &encode);
}

View File

@@ -1,21 +1,7 @@
export interface EncodeOptions {
speed: number;
quality: number;
progressive: boolean;
epf: number;
nearLossless: number;
lossyPalette: boolean;
import { EncodeOptions } from '../../../src/codecs/jxl/encoder-meta';
interface JXLModule extends EmscriptenWasm.Module {
encode(data: BufferSource, width: number, height: number, options: EncodeOptions): Uint8Array | null;
}
export interface JXLModule extends EmscriptenWasm.Module {
encode(
data: BufferSource,
width: number,
height: number,
options: EncodeOptions,
): Uint8Array | null;
}
declare var moduleFactory: EmscriptenWasm.ModuleFactory<JXLModule>;
export default moduleFactory;
export default function(opts: EmscriptenWasm.ModuleOpts): Promise<JXLModule>;

File diff suppressed because it is too large Load Diff

BIN
codecs/jxl/enc/jxl_enc.wasm Executable file → Normal file

Binary file not shown.

View File

@@ -14,13 +14,11 @@ all: $(OUT_JS)
-I $(CODEC_DIR) \
${CXXFLAGS} \
${LDFLAGS} \
--closure 1 \
--bind \
--closure 1 \
-s ALLOW_MEMORY_GROWTH=1 \
-s MODULARIZE=1 \
-s TEXTDECODER=2 \
-s ENVIRONMENT='worker' \
-s EXPORT_ES6=1 \
-s 'EXPORT_NAME="$(basename $(@F))"' \
-o $@ \
$+

View File

@@ -1,12 +1,13 @@
<!DOCTYPE html>
<script type="module">
import mozjpeg_enc from './mozjpeg_enc.js';
<!doctype html>
<script src='mozjpeg_enc.js'></script>
<script>
const module = mozjpeg_enc();
async function loadImage(src) {
// Load image
const img = document.createElement('img');
img.src = src;
await new Promise((resolve) => (img.onload = resolve));
await new Promise(resolve => img.onload = resolve);
// Make canvas same size as image
const canvas = document.createElement('canvas');
[canvas.width, canvas.height] = [img.width, img.height];
@@ -16,9 +17,7 @@
return ctx.getImageData(0, 0, img.width, img.height);
}
async function main() {
const module = await mozjpeg_enc();
module.onRuntimeInitialized = async _ => {
console.log('Version:', module.version().toString(16));
const image = await loadImage('../example.png');
const result = module.encode(image.data, image.width, image.height, {
@@ -40,12 +39,10 @@
chroma_quality: 75,
});
const blob = new Blob([result], { type: 'image/jpeg' });
const blob = new Blob([result], {type: 'image/jpeg'});
const blobURL = URL.createObjectURL(blob);
const img = document.createElement('img');
img.src = blobURL;
document.body.appendChild(img);
}
main();
};
</script>

View File

@@ -1,37 +1,7 @@
export const enum MozJpegColorSpace {
GRAYSCALE = 1,
RGB,
YCbCr,
import { EncodeOptions } from '../../src/codecs/mozjpeg/encoder-meta';
interface MozJPEGModule extends EmscriptenWasm.Module {
encode(data: BufferSource, width: number, height: number, options: EncodeOptions): Uint8Array;
}
export interface EncodeOptions {
quality: number;
baseline: boolean;
arithmetic: boolean;
progressive: boolean;
optimize_coding: boolean;
smoothing: number;
color_space: MozJpegColorSpace;
quant_table: number;
trellis_multipass: boolean;
trellis_opt_zero: boolean;
trellis_opt_table: boolean;
trellis_loops: number;
auto_subsample: boolean;
chroma_subsample: number;
separate_chroma_quality: boolean;
chroma_quality: number;
}
export interface MozJPEGModule extends EmscriptenWasm.Module {
encode(
data: BufferSource,
width: number,
height: number,
options: EncodeOptions,
): Uint8Array;
}
declare var moduleFactory: EmscriptenWasm.ModuleFactory<MozJPEGModule>;
export default moduleFactory;
export default function(opts: EmscriptenWasm.ModuleOpts): Promise<MozJPEGModule>;

File diff suppressed because it is too large Load Diff

BIN
codecs/mozjpeg_enc/mozjpeg_enc.wasm Executable file → Normal file

Binary file not shown.

173
codecs/oxipng/Cargo.lock generated
View File

@@ -56,9 +56,9 @@ checksum = "08c48aae112d48ed9f069b33538ea9e3e90aa263cfa3d1c24309612b1f7472de"
[[package]]
name = "cc"
version = "1.0.62"
version = "1.0.60"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "f1770ced377336a88a67c473594ccc14eca6f4559217c34f64aac8f83d641b40"
checksum = "ef611cc68ff783f18535d77ddd080185275713d852c4f5cbb6122c462a7a825c"
[[package]]
name = "cfg-if"
@@ -66,12 +66,6 @@ version = "0.1.10"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "4785bdd1c96b2a846b2bd7cc02e86b6b3dbf14e7e53446c4f54c92a361040822"
[[package]]
name = "cfg-if"
version = "1.0.0"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "baf1de4339761588bc0619e3cbc0120ee582ebb74b53b4efbf79117bd2da40fd"
[[package]]
name = "cloudflare-zlib"
version = "0.2.5"
@@ -90,18 +84,6 @@ dependencies = [
"cc",
]
[[package]]
name = "color_quant"
version = "1.1.0"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "3d7b894f5411737b7867f4827955924d7c254fc9f4d91a6aad6b097804b1018b"
[[package]]
name = "const_fn"
version = "0.4.3"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "c478836e029dcef17fb47c89023448c64f781a046e0300e257ad8225ae59afab"
[[package]]
name = "crc"
version = "1.8.1"
@@ -113,57 +95,57 @@ dependencies = [
[[package]]
name = "crc32fast"
version = "1.2.1"
version = "1.2.0"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "81156fece84ab6a9f2afdb109ce3ae577e42b1228441eded99bd77f627953b1a"
checksum = "ba125de2af0df55319f41944744ad91c71113bf74a4646efff39afe1f6842db1"
dependencies = [
"cfg-if 1.0.0",
"cfg-if",
]
[[package]]
name = "crossbeam-channel"
version = "0.5.0"
version = "0.4.4"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "dca26ee1f8d361640700bde38b2c37d8c22b3ce2d360e1fc1c74ea4b0aa7d775"
checksum = "b153fe7cbef478c567df0f972e02e6d736db11affe43dfc9c56a9374d1adfb87"
dependencies = [
"cfg-if 1.0.0",
"crossbeam-utils",
"maybe-uninit",
]
[[package]]
name = "crossbeam-deque"
version = "0.8.0"
version = "0.7.3"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "94af6efb46fef72616855b036a624cf27ba656ffc9be1b9a3c931cfc7749a9a9"
checksum = "9f02af974daeee82218205558e51ec8768b48cf524bd01d550abe5573a608285"
dependencies = [
"cfg-if 1.0.0",
"crossbeam-epoch",
"crossbeam-utils",
"maybe-uninit",
]
[[package]]
name = "crossbeam-epoch"
version = "0.9.0"
version = "0.8.2"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "ec0f606a85340376eef0d6d8fec399e6d4a544d648386c6645eb6d0653b27d9f"
checksum = "058ed274caafc1f60c4997b5fc07bf7dc7cca454af7c6e81edffe5f33f70dace"
dependencies = [
"cfg-if 1.0.0",
"const_fn",
"autocfg",
"cfg-if",
"crossbeam-utils",
"lazy_static",
"maybe-uninit",
"memoffset",
"scopeguard",
]
[[package]]
name = "crossbeam-utils"
version = "0.8.0"
version = "0.7.2"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "ec91540d98355f690a86367e566ecad2e9e579f230230eb7c21398372be73ea5"
checksum = "c3c7c73a2d1e9fc0886a08b93e98eb643461230d5f1925e4036204d5f2e261a8"
dependencies = [
"autocfg",
"cfg-if 1.0.0",
"const_fn",
"cfg-if",
"lazy_static",
]
@@ -185,28 +167,27 @@ checksum = "e78d4f1cc4ae33bbfc157ed5d5a5ef3bc29227303d595861deb238fcec4e9457"
[[package]]
name = "hashbrown"
version = "0.9.1"
version = "0.9.0"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "d7afe4a420e3fe79967a00898cc1f4db7c8a49a9333a29f8a4bd76a253d5cd04"
checksum = "00d63df3d41950fb462ed38308eea019113ad1508da725bbedcd0fa5a85ef5f7"
[[package]]
name = "hermit-abi"
version = "0.1.17"
version = "0.1.16"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "5aca5565f760fb5b220e499d72710ed156fdb74e631659e99377d9ebfbd13ae8"
checksum = "4c30f6d0bc6b00693347368a67d41b58f2fb851215ff1da49e90fe2c5c667151"
dependencies = [
"libc",
]
[[package]]
name = "image"
version = "0.23.11"
version = "0.23.9"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "b4f0a8345b33b082aedec2f4d7d4a926b845cee184cbe78b703413066564431b"
checksum = "974e194911d1f7efe3cd8a8f9db3b767e43536327e899e8bc9a12ef5711b74d2"
dependencies = [
"bytemuck",
"byteorder",
"color_quant",
"num-iter",
"num-rational",
"num-traits",
@@ -241,9 +222,9 @@ checksum = "e2abad23fbc42b3700f2f279844dc832adb2b2eb069b2df918f455c4e18cc646"
[[package]]
name = "libc"
version = "0.2.80"
version = "0.2.77"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "4d58d1b70b004888f764dfbf6a26a3b0342a1632d33968e4a179d8011c760614"
checksum = "f2f96b10ec2560088a8e76961b00d47107b3a625fecb76dedb29ee7ccbf98235"
[[package]]
name = "libdeflate-sys"
@@ -269,9 +250,15 @@ version = "0.4.11"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "4fabed175da42fed1fa0746b0ea71f412aa9d35e76e95e59b192c64b9dc2bf8b"
dependencies = [
"cfg-if 0.1.10",
"cfg-if",
]
[[package]]
name = "maybe-uninit"
version = "2.0.0"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "60302e4db3a61da70c0cb7991976248362f30319e88850c487b9b95bbf059e00"
[[package]]
name = "memoffset"
version = "0.5.6"
@@ -292,9 +279,9 @@ dependencies = [
[[package]]
name = "miniz_oxide"
version = "0.4.3"
version = "0.4.2"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "0f2d26ec3309788e423cfbf68ad1800f061638098d76a83681af979dc4eda19d"
checksum = "c60c0dfe32c10b43a144bad8fc83538c52f58302c92300ea7ec7bf7b38d5a7b9"
dependencies = [
"adler",
"autocfg",
@@ -302,9 +289,9 @@ dependencies = [
[[package]]
name = "num-integer"
version = "0.1.44"
version = "0.1.43"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "d2cc698a63b549a70bc047073d2949cce27cd1c7b0a4a862d08a8031bc2801db"
checksum = "8d59457e662d541ba17869cf51cf177c0b5f0cbf476c66bdc90bf1edac4f875b"
dependencies = [
"autocfg",
"num-traits",
@@ -312,9 +299,9 @@ dependencies = [
[[package]]
name = "num-iter"
version = "0.1.42"
version = "0.1.41"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "b2021c8337a54d21aca0d59a92577a029af9431cb59b909b03252b9c164fad59"
checksum = "7a6e6b7c748f995c4c29c5f5ae0248536e04a5739927c74ec0fa564805094b9f"
dependencies = [
"autocfg",
"num-integer",
@@ -323,9 +310,9 @@ dependencies = [
[[package]]
name = "num-rational"
version = "0.3.2"
version = "0.3.0"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "12ac428b1cb17fce6f731001d307d351ec70a6d202fc2e60f7d4c5e42d8f4f07"
checksum = "a5b4d7360f362cfb50dde8143501e6940b22f644be75a4cc90b2d81968908138"
dependencies = [
"autocfg",
"num-integer",
@@ -334,9 +321,9 @@ dependencies = [
[[package]]
name = "num-traits"
version = "0.2.14"
version = "0.2.12"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "9a64b1ec5cda2586e284722486d802acf1f7dbdc623e2bfc57e65ca1cd099290"
checksum = "ac267bcc07f48ee5f8935ab0d24f316fb722d7a1292e2913f0cc196b29ffd611"
dependencies = [
"autocfg",
]
@@ -353,15 +340,14 @@ dependencies = [
[[package]]
name = "once_cell"
version = "1.5.2"
version = "1.4.1"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "13bd41f508810a131401606d54ac32a467c97172d74ba7662562ebba5ad07fa0"
checksum = "260e51e7efe62b592207e9e13a68e43692a7a279171d6ba57abd208bf23645ad"
[[package]]
name = "oxipng"
version = "4.0.0"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "ea40b366cecfce76ee3b082e7e6567b82cdef75644a22442ca8584bc666ff4eb"
version = "3.0.1"
source = "git+https://github.com/RReverser/oxipng?branch=crossbeam#0df4a32921f7b88d2be4649deb1b6fb7e5707ad8"
dependencies = [
"bit-vec",
"byteorder",
@@ -373,19 +359,11 @@ dependencies = [
"itertools",
"libdeflater",
"log",
"miniz_oxide 0.4.3",
"miniz_oxide 0.4.2",
"rayon",
"rgb",
"rustc_version",
]
[[package]]
name = "pest"
version = "2.1.3"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "10f4872ae94d7b90ae48754df22fd42ad52ce740b8f370b03da4835417403e53"
dependencies = [
"ucd-trie",
"zopfli",
]
[[package]]
@@ -402,9 +380,9 @@ dependencies = [
[[package]]
name = "proc-macro2"
version = "1.0.24"
version = "1.0.21"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "1e0704ee1a7e00d7bb417d0770ea303c1bccbabf0ef1667dae92b5967f5f8a71"
checksum = "36e28516df94f3dd551a587da5357459d9b36d945a7c37c3557928c1c2ff2a2c"
dependencies = [
"unicode-xid",
]
@@ -420,9 +398,9 @@ dependencies = [
[[package]]
name = "rayon"
version = "1.5.0"
version = "1.4.1"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "8b0d8e0819fadc20c74ea8373106ead0600e3a67ef1fe8da56e39b9ae7275674"
checksum = "dcf6960dc9a5b4ee8d3e4c5787b4a112a8818e0290a42ff664ad60692fdf2032"
dependencies = [
"autocfg",
"crossbeam-deque",
@@ -432,9 +410,9 @@ dependencies = [
[[package]]
name = "rayon-core"
version = "1.9.0"
version = "1.8.1"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "9ab346ac5921dc62ffa9f89b7a773907511cdfa5490c572ae9be1be33e8afa4a"
checksum = "e8c4fec834fb6e6d2dd5eece3c7b432a52f0ba887cf40e595190c4107edc08bf"
dependencies = [
"crossbeam-channel",
"crossbeam-deque",
@@ -454,9 +432,9 @@ dependencies = [
[[package]]
name = "rustc_version"
version = "0.3.0"
version = "0.2.3"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "65c94201b44764d6d1f7e37c15a8289ed55e546c1762c7f1d57f616966e0c181"
checksum = "138e3e0acb6c9fb258b19b67cb8abd63c00679d2851805ea151465464fe9030a"
dependencies = [
"semver",
]
@@ -469,21 +447,18 @@ checksum = "d29ab0c6d3fc0ee92fe66e2d99f700eab17a8d57d1c1d3b748380fb20baa78cd"
[[package]]
name = "semver"
version = "0.11.0"
version = "0.9.0"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "f301af10236f6df4160f7c3f04eec6dbc70ace82d23326abad5edee88801c6b6"
checksum = "1d7eb9ef2c18661902cc47e535f9bc51b78acd254da71d375c2f6720d9a40403"
dependencies = [
"semver-parser",
]
[[package]]
name = "semver-parser"
version = "0.10.1"
version = "0.7.0"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "42ef146c2ad5e5f4b037cd6ce2ebb775401729b19a82040c1beac9d36c7d1428"
dependencies = [
"pest",
]
checksum = "388a1df253eca08550bef6c72392cfe7c30914bf41df5269b68cbd6ff8f570a3"
[[package]]
name = "squoosh-oxipng"
@@ -499,9 +474,9 @@ dependencies = [
[[package]]
name = "syn"
version = "1.0.48"
version = "1.0.41"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "cc371affeffc477f42a221a1e4297aedcea33d47d19b61455588bd9d8f6b19ac"
checksum = "6690e3e9f692504b941dc6c3b188fd28df054f7fb8469ab40680df52fdcc842b"
dependencies = [
"proc-macro2",
"quote",
@@ -509,10 +484,10 @@ dependencies = [
]
[[package]]
name = "ucd-trie"
version = "0.1.3"
name = "typed-arena"
version = "1.7.0"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "56dee185309b50d1f11bfedef0fe6d036842e3fb77413abef29f8f8d1c5d4c1c"
checksum = "a9b2228007eba4120145f785df0f6c92ea538f5a3635a612ecf4e334c8c1446d"
[[package]]
name = "unicode-xid"
@@ -526,7 +501,7 @@ version = "0.2.68"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "1ac64ead5ea5f05873d7c12b545865ca2b8d28adfc50a49b84770a3a97265d42"
dependencies = [
"cfg-if 0.1.10",
"cfg-if",
"wasm-bindgen-macro",
]
@@ -573,3 +548,15 @@ name = "wasm-bindgen-shared"
version = "0.2.68"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "1d649a3145108d7d3fbcde896a468d1bd636791823c9921135218ad89be08307"
[[package]]
name = "zopfli"
version = "0.4.0"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "4079b79464426ade2a1b0177fb0ce8396ba6b4084267407e333573c666073964"
dependencies = [
"adler32",
"byteorder",
"crc",
"typed-arena",
]

View File

@@ -5,19 +5,20 @@ authors = ["Ingvar Stepanyan <me@rreverser.com>"]
edition = "2018"
publish = false
[package.metadata.wasm-pack.profile.release]
wasm-opt = ["-O", "--no-validation"]
[lib]
crate-type = ["cdylib"]
[dependencies]
oxipng = { version = "4.0.0", default-features = false, features = ["libdeflater"] }
wasm-bindgen = "0.2.68"
log = { version = "0.4.11", features = ["release_max_level_off"] }
rayon = { version = "1.5.0", optional = true }
once_cell = { version = "1.5.2", optional = true }
crossbeam-channel = { version = "0.5.0", optional = true }
oxipng = { version = "3.0.0", default-features = false }
wasm-bindgen = "0.2.64"
log = { version = "0.4", features = ["release_max_level_off"] }
rayon = { version = "1.3.0", optional = true }
once_cell = { version = "1.3.1", optional = true }
crossbeam-channel = { version = "0.4.2", optional = true }
[patch.crates-io]
# TODO: replace with upstream version once https://github.com/shssoichiro/oxipng/pull/327 is merged.
oxipng = { git = "https://github.com/RReverser/oxipng", branch = "crossbeam" }
[profile.release]
lto = true
@@ -25,3 +26,6 @@ opt-level = "s"
[features]
parallel = ["oxipng/parallel", "rayon", "crossbeam-channel", "once_cell"]
[package.metadata.wasm-pack.profile.release]
wasm-opt = ["-O", "--no-validation"]

3
codecs/oxipng/build.sh Executable file → Normal file
View File

@@ -3,6 +3,7 @@
set -e
rm -rf pkg,{-parallel}
wasm-pack build --target web
wasm-pack build
RUSTFLAGS='-C target-feature=+atomics,+bulk-memory' wasm-pack build -t web -d pkg-parallel -- -Z build-std=panic_abort,std --features=parallel
sed -i "s|input = import.meta.url.replace(/\\\.js$/, '_bg.wasm');||" pkg{,-parallel}/squoosh_oxipng.js
rm pkg{,-parallel}/.gitignore

10
codecs/oxipng/index.ts Normal file
View File

@@ -0,0 +1,10 @@
import { threads } from 'wasm-feature-detect';
async function init() {
if (await threads()) {
return (await import('./spawn')).default;
}
return import('./pkg');
}
export default init();

View File

@@ -1,6 +1,6 @@
{
"name": "oxipng",
"scripts": {
"build": "RUST_IMG=rustlang/rust:8bb115b1090d ../build-rust.sh ./build.sh"
"build": "../build-rust-nightly.sh ./build.sh"
}
}

View File

@@ -12,4 +12,4 @@
"module": "squoosh_oxipng.js",
"types": "squoosh_oxipng.d.ts",
"sideEffects": false
}
}

View File

@@ -1,37 +1,32 @@
/* tslint:disable */
/* eslint-disable */
/**
* @param {Uint8Array} data
* @param {number} level
* @returns {Uint8Array}
*/
export function optimise(data: Uint8Array, level: number): Uint8Array;
/**
* @param {number} num
* @returns {any}
*/
* @param {number} num
* @returns {any}
*/
export function worker_initializer(num: number): any;
/**
*/
*/
export function start_main_thread(): void;
/**
*/
*/
export function start_worker_thread(): void;
/**
* @param {Uint8Array} data
* @param {number} level
* @returns {Uint8Array}
*/
export function optimise(data: Uint8Array, level: number): Uint8Array;
export type InitInput =
| RequestInfo
| URL
| Response
| BufferSource
| WebAssembly.Module;
export type InitInput = RequestInfo | URL | Response | BufferSource | WebAssembly.Module;
export interface InitOutput {
readonly optimise: (a: number, b: number, c: number, d: number) => void;
readonly malloc: (a: number) => number;
readonly free: (a: number) => void;
readonly worker_initializer: (a: number) => number;
readonly start_main_thread: () => void;
readonly start_worker_thread: () => void;
readonly optimise: (a: number, b: number, c: number, d: number) => void;
readonly __wbindgen_export_0: WebAssembly.Memory;
readonly __wbindgen_malloc: (a: number) => number;
readonly __wbindgen_free: (a: number, b: number) => void;
@@ -39,15 +34,13 @@ export interface InitOutput {
}
/**
* If `module_or_path` is {RequestInfo} or {URL}, makes a request and
* for everything else, calls `WebAssembly.instantiate` directly.
*
* @param {InitInput | Promise<InitInput>} module_or_path
* @param {WebAssembly.Memory} maybe_memory
*
* @returns {Promise<InitOutput>}
*/
export default function init(
module_or_path?: InitInput | Promise<InitInput>,
maybe_memory?: WebAssembly.Memory,
): Promise<InitOutput>;
* If `module_or_path` is {RequestInfo} or {URL}, makes a request and
* for everything else, calls `WebAssembly.instantiate` directly.
*
* @param {InitInput | Promise<InitInput>} module_or_path
* @param {WebAssembly.Memory} maybe_memory
*
* @returns {Promise<InitOutput>}
*/
export default function init (module_or_path?: InitInput | Promise<InitInput>, maybe_memory: WebAssembly.Memory): Promise<InitOutput>;

View File

@@ -1,3 +1,4 @@
let wasm;
let memory;
@@ -8,189 +9,172 @@ heap.push(undefined, null, true, false);
let heap_next = heap.length;
function addHeapObject(obj) {
if (heap_next === heap.length) heap.push(heap.length + 1);
const idx = heap_next;
heap_next = heap[idx];
if (heap_next === heap.length) heap.push(heap.length + 1);
const idx = heap_next;
heap_next = heap[idx];
heap[idx] = obj;
return idx;
heap[idx] = obj;
return idx;
}
let cachedTextDecoder = new TextDecoder('utf-8', {
ignoreBOM: true,
fatal: true,
});
let cachedTextDecoder = new TextDecoder('utf-8', { ignoreBOM: true, fatal: true });
cachedTextDecoder.decode();
let cachegetUint8Memory0 = null;
function getUint8Memory0() {
if (
cachegetUint8Memory0 === null ||
cachegetUint8Memory0.buffer !== wasm.__wbindgen_export_0.buffer
) {
cachegetUint8Memory0 = new Uint8Array(wasm.__wbindgen_export_0.buffer);
}
return cachegetUint8Memory0;
if (cachegetUint8Memory0 === null || cachegetUint8Memory0.buffer !== wasm.__wbindgen_export_0.buffer) {
cachegetUint8Memory0 = new Uint8Array(wasm.__wbindgen_export_0.buffer);
}
return cachegetUint8Memory0;
}
function getStringFromWasm0(ptr, len) {
return cachedTextDecoder.decode(getUint8Memory0().slice(ptr, ptr + len));
return cachedTextDecoder.decode(getUint8Memory0().slice(ptr, ptr + len));
}
function getObject(idx) { return heap[idx]; }
function dropObject(idx) {
if (idx < 36) return;
heap[idx] = heap_next;
heap_next = idx;
}
function takeObject(idx) {
const ret = getObject(idx);
dropObject(idx);
return ret;
}
/**
* @param {number} num
* @returns {any}
*/
export function worker_initializer(num) {
var ret = wasm.worker_initializer(num);
return takeObject(ret);
}
/**
*/
export function start_main_thread() {
wasm.start_main_thread();
}
/**
*/
export function start_worker_thread() {
wasm.start_worker_thread();
}
let WASM_VECTOR_LEN = 0;
function passArray8ToWasm0(arg, malloc) {
const ptr = malloc(arg.length * 1);
getUint8Memory0().set(arg, ptr / 1);
WASM_VECTOR_LEN = arg.length;
return ptr;
const ptr = malloc(arg.length * 1);
getUint8Memory0().set(arg, ptr / 1);
WASM_VECTOR_LEN = arg.length;
return ptr;
}
let cachegetInt32Memory0 = null;
function getInt32Memory0() {
if (
cachegetInt32Memory0 === null ||
cachegetInt32Memory0.buffer !== wasm.__wbindgen_export_0.buffer
) {
cachegetInt32Memory0 = new Int32Array(wasm.__wbindgen_export_0.buffer);
}
return cachegetInt32Memory0;
if (cachegetInt32Memory0 === null || cachegetInt32Memory0.buffer !== wasm.__wbindgen_export_0.buffer) {
cachegetInt32Memory0 = new Int32Array(wasm.__wbindgen_export_0.buffer);
}
return cachegetInt32Memory0;
}
function getArrayU8FromWasm0(ptr, len) {
return getUint8Memory0().subarray(ptr / 1, ptr / 1 + len);
return getUint8Memory0().subarray(ptr / 1, ptr / 1 + len);
}
/**
* @param {Uint8Array} data
* @param {number} level
* @returns {Uint8Array}
*/
* @param {Uint8Array} data
* @param {number} level
* @returns {Uint8Array}
*/
export function optimise(data, level) {
try {
const retptr = wasm.__wbindgen_export_1.value - 16;
wasm.__wbindgen_export_1.value = retptr;
var ptr0 = passArray8ToWasm0(data, wasm.__wbindgen_malloc);
var len0 = WASM_VECTOR_LEN;
wasm.optimise(retptr, ptr0, len0, level);
var r0 = getInt32Memory0()[retptr / 4 + 0];
var r1 = getInt32Memory0()[retptr / 4 + 1];
var v1 = getArrayU8FromWasm0(r0, r1).slice();
wasm.__wbindgen_free(r0, r1 * 1);
return v1;
} finally {
wasm.__wbindgen_export_1.value += 16;
}
}
function getObject(idx) {
return heap[idx];
}
function dropObject(idx) {
if (idx < 36) return;
heap[idx] = heap_next;
heap_next = idx;
}
function takeObject(idx) {
const ret = getObject(idx);
dropObject(idx);
return ret;
}
/**
* @param {number} num
* @returns {any}
*/
export function worker_initializer(num) {
var ret = wasm.worker_initializer(num);
return takeObject(ret);
}
/**
*/
export function start_main_thread() {
wasm.start_main_thread();
}
/**
*/
export function start_worker_thread() {
wasm.start_worker_thread();
try {
const retptr = wasm.__wbindgen_export_1.value - 16;
wasm.__wbindgen_export_1.value = retptr;
var ptr0 = passArray8ToWasm0(data, wasm.__wbindgen_malloc);
var len0 = WASM_VECTOR_LEN;
wasm.optimise(retptr, ptr0, len0, level);
var r0 = getInt32Memory0()[retptr / 4 + 0];
var r1 = getInt32Memory0()[retptr / 4 + 1];
var v1 = getArrayU8FromWasm0(r0, r1).slice();
wasm.__wbindgen_free(r0, r1 * 1);
return v1;
} finally {
wasm.__wbindgen_export_1.value += 16;
}
}
async function load(module, imports, maybe_memory) {
if (typeof Response === 'function' && module instanceof Response) {
memory = imports.wbg.memory = new WebAssembly.Memory({
initial: 17,
maximum: 16384,
shared: true,
});
if (typeof WebAssembly.instantiateStreaming === 'function') {
try {
return await WebAssembly.instantiateStreaming(module, imports);
} catch (e) {
if (module.headers.get('Content-Type') != 'application/wasm') {
console.warn(
'`WebAssembly.instantiateStreaming` failed because your server does not serve wasm with `application/wasm` MIME type. Falling back to `WebAssembly.instantiate` which is slower. Original error:\n',
e,
);
} else {
throw e;
if (typeof Response === 'function' && module instanceof Response) {
memory = imports.wbg.memory = new WebAssembly.Memory({initial:17,maximum:16384,shared:true});
if (typeof WebAssembly.instantiateStreaming === 'function') {
try {
return await WebAssembly.instantiateStreaming(module, imports);
} catch (e) {
if (module.headers.get('Content-Type') != 'application/wasm') {
console.warn("`WebAssembly.instantiateStreaming` failed because your server does not serve wasm with `application/wasm` MIME type. Falling back to `WebAssembly.instantiate` which is slower. Original error:\n", e);
} else {
throw e;
}
}
}
}
}
const bytes = await module.arrayBuffer();
return await WebAssembly.instantiate(bytes, imports);
} else {
memory = imports.wbg.memory = maybe_memory;
const instance = await WebAssembly.instantiate(module, imports);
const bytes = await module.arrayBuffer();
return await WebAssembly.instantiate(bytes, imports);
if (instance instanceof WebAssembly.Instance) {
return { instance, module };
} else {
return instance;
memory = imports.wbg.memory = maybe_memory;
const instance = await WebAssembly.instantiate(module, imports);
if (instance instanceof WebAssembly.Instance) {
return { instance, module };
} else {
return instance;
}
}
}
}
async function init(input, maybe_memory) {
if (typeof input === 'undefined') {
input = import.meta.url.replace(/\.js$/, '_bg.wasm');
}
const imports = {};
imports.wbg = {};
imports.wbg.__wbindgen_module = function () {
var ret = init.__wbindgen_wasm_module;
return addHeapObject(ret);
};
imports.wbg.__wbindgen_memory = function () {
var ret = wasm.__wbindgen_export_0;
return addHeapObject(ret);
};
imports.wbg.__wbg_of_6510501edc06d65e = function (arg0, arg1) {
var ret = Array.of(takeObject(arg0), takeObject(arg1));
return addHeapObject(ret);
};
imports.wbg.__wbindgen_throw = function (arg0, arg1) {
throw new Error(getStringFromWasm0(arg0, arg1));
};
if (typeof input === 'undefined') {
}
const imports = {};
imports.wbg = {};
imports.wbg.__wbindgen_module = function() {
var ret = init.__wbindgen_wasm_module;
return addHeapObject(ret);
};
imports.wbg.__wbindgen_memory = function() {
var ret = wasm.__wbindgen_export_0;
return addHeapObject(ret);
};
imports.wbg.__wbg_of_6510501edc06d65e = function(arg0, arg1) {
var ret = Array.of(takeObject(arg0), takeObject(arg1));
return addHeapObject(ret);
};
imports.wbg.__wbindgen_throw = function(arg0, arg1) {
throw new Error(getStringFromWasm0(arg0, arg1));
};
if (
typeof input === 'string' ||
(typeof Request === 'function' && input instanceof Request) ||
(typeof URL === 'function' && input instanceof URL)
) {
input = fetch(input);
}
if (typeof input === 'string' || (typeof Request === 'function' && input instanceof Request) || (typeof URL === 'function' && input instanceof URL)) {
input = fetch(input);
}
const { instance, module } = await load(await input, imports, maybe_memory);
const { instance, module } = await load(await input, imports, maybe_memory);
wasm = instance.exports;
init.__wbindgen_wasm_module = module;
wasm.__wbindgen_start();
return wasm;
wasm = instance.exports;
init.__wbindgen_wasm_module = module;
wasm.__wbindgen_start();
return wasm;
}
export default init;

View File

@@ -1,11 +1,11 @@
/* tslint:disable */
/* eslint-disable */
export function optimise(a: number, b: number, c: number, d: number): void;
export function malloc(a: number): number;
export function free(a: number): void;
export function worker_initializer(a: number): number;
export function start_main_thread(): void;
export function start_worker_thread(): void;
export function optimise(a: number, b: number, c: number, d: number): void;
export const __wbindgen_export_0: WebAssembly.Memory;
export function __wbindgen_malloc(a: number): number;
export function __wbindgen_free(a: number, b: number): void;

View File

@@ -1,36 +1,8 @@
/* tslint:disable */
/* eslint-disable */
/**
* @param {Uint8Array} data
* @param {number} level
* @returns {Uint8Array}
*/
* @param {Uint8Array} data
* @param {number} level
* @returns {Uint8Array}
*/
export function optimise(data: Uint8Array, level: number): Uint8Array;
export type InitInput =
| RequestInfo
| URL
| Response
| BufferSource
| WebAssembly.Module;
export interface InitOutput {
readonly memory: WebAssembly.Memory;
readonly optimise: (a: number, b: number, c: number, d: number) => void;
readonly malloc: (a: number) => number;
readonly free: (a: number) => void;
readonly __wbindgen_malloc: (a: number) => number;
readonly __wbindgen_free: (a: number, b: number) => void;
}
/**
* If `module_or_path` is {RequestInfo} or {URL}, makes a request and
* for everything else, calls `WebAssembly.instantiate` directly.
*
* @param {InitInput | Promise<InitInput>} module_or_path
*
* @returns {Promise<InitOutput>}
*/
export default function init(
module_or_path?: InitInput | Promise<InitInput>,
): Promise<InitOutput>;

View File

@@ -1,126 +1,2 @@
let wasm;
let cachedTextDecoder = new TextDecoder('utf-8', {
ignoreBOM: true,
fatal: true,
});
cachedTextDecoder.decode();
let cachegetUint8Memory0 = null;
function getUint8Memory0() {
if (
cachegetUint8Memory0 === null ||
cachegetUint8Memory0.buffer !== wasm.memory.buffer
) {
cachegetUint8Memory0 = new Uint8Array(wasm.memory.buffer);
}
return cachegetUint8Memory0;
}
function getStringFromWasm0(ptr, len) {
return cachedTextDecoder.decode(getUint8Memory0().subarray(ptr, ptr + len));
}
let WASM_VECTOR_LEN = 0;
function passArray8ToWasm0(arg, malloc) {
const ptr = malloc(arg.length * 1);
getUint8Memory0().set(arg, ptr / 1);
WASM_VECTOR_LEN = arg.length;
return ptr;
}
let cachegetInt32Memory0 = null;
function getInt32Memory0() {
if (
cachegetInt32Memory0 === null ||
cachegetInt32Memory0.buffer !== wasm.memory.buffer
) {
cachegetInt32Memory0 = new Int32Array(wasm.memory.buffer);
}
return cachegetInt32Memory0;
}
function getArrayU8FromWasm0(ptr, len) {
return getUint8Memory0().subarray(ptr / 1, ptr / 1 + len);
}
/**
* @param {Uint8Array} data
* @param {number} level
* @returns {Uint8Array}
*/
export function optimise(data, level) {
try {
const retptr = wasm.__wbindgen_export_0.value - 16;
wasm.__wbindgen_export_0.value = retptr;
var ptr0 = passArray8ToWasm0(data, wasm.__wbindgen_malloc);
var len0 = WASM_VECTOR_LEN;
wasm.optimise(retptr, ptr0, len0, level);
var r0 = getInt32Memory0()[retptr / 4 + 0];
var r1 = getInt32Memory0()[retptr / 4 + 1];
var v1 = getArrayU8FromWasm0(r0, r1).slice();
wasm.__wbindgen_free(r0, r1 * 1);
return v1;
} finally {
wasm.__wbindgen_export_0.value += 16;
}
}
async function load(module, imports) {
if (typeof Response === 'function' && module instanceof Response) {
if (typeof WebAssembly.instantiateStreaming === 'function') {
try {
return await WebAssembly.instantiateStreaming(module, imports);
} catch (e) {
if (module.headers.get('Content-Type') != 'application/wasm') {
console.warn(
'`WebAssembly.instantiateStreaming` failed because your server does not serve wasm with `application/wasm` MIME type. Falling back to `WebAssembly.instantiate` which is slower. Original error:\n',
e,
);
} else {
throw e;
}
}
}
const bytes = await module.arrayBuffer();
return await WebAssembly.instantiate(bytes, imports);
} else {
const instance = await WebAssembly.instantiate(module, imports);
if (instance instanceof WebAssembly.Instance) {
return { instance, module };
} else {
return instance;
}
}
}
async function init(input) {
if (typeof input === 'undefined') {
input = import.meta.url.replace(/\.js$/, '_bg.wasm');
}
const imports = {};
imports.wbg = {};
imports.wbg.__wbindgen_throw = function (arg0, arg1) {
throw new Error(getStringFromWasm0(arg0, arg1));
};
if (
typeof input === 'string' ||
(typeof Request === 'function' && input instanceof Request) ||
(typeof URL === 'function' && input instanceof URL)
) {
input = fetch(input);
}
const { instance, module } = await load(await input, imports);
wasm = instance.exports;
init.__wbindgen_wasm_module = module;
return wasm;
}
export default init;
import * as wasm from "./squoosh_oxipng_bg.wasm";
export * from "./squoosh_oxipng_bg.js";

View File

@@ -0,0 +1,66 @@
import * as wasm from './squoosh_oxipng_bg.wasm';
const lTextDecoder = typeof TextDecoder === 'undefined' ? (0, module.require)('util').TextDecoder : TextDecoder;
let cachedTextDecoder = new lTextDecoder('utf-8', { ignoreBOM: true, fatal: true });
cachedTextDecoder.decode();
let cachegetUint8Memory0 = null;
function getUint8Memory0() {
if (cachegetUint8Memory0 === null || cachegetUint8Memory0.buffer !== wasm.memory.buffer) {
cachegetUint8Memory0 = new Uint8Array(wasm.memory.buffer);
}
return cachegetUint8Memory0;
}
function getStringFromWasm0(ptr, len) {
return cachedTextDecoder.decode(getUint8Memory0().subarray(ptr, ptr + len));
}
let WASM_VECTOR_LEN = 0;
function passArray8ToWasm0(arg, malloc) {
const ptr = malloc(arg.length * 1);
getUint8Memory0().set(arg, ptr / 1);
WASM_VECTOR_LEN = arg.length;
return ptr;
}
let cachegetInt32Memory0 = null;
function getInt32Memory0() {
if (cachegetInt32Memory0 === null || cachegetInt32Memory0.buffer !== wasm.memory.buffer) {
cachegetInt32Memory0 = new Int32Array(wasm.memory.buffer);
}
return cachegetInt32Memory0;
}
function getArrayU8FromWasm0(ptr, len) {
return getUint8Memory0().subarray(ptr / 1, ptr / 1 + len);
}
/**
* @param {Uint8Array} data
* @param {number} level
* @returns {Uint8Array}
*/
export function optimise(data, level) {
try {
const retptr = wasm.__wbindgen_export_0.value - 16;
wasm.__wbindgen_export_0.value = retptr;
var ptr0 = passArray8ToWasm0(data, wasm.__wbindgen_malloc);
var len0 = WASM_VECTOR_LEN;
wasm.optimise(retptr, ptr0, len0, level);
var r0 = getInt32Memory0()[retptr / 4 + 0];
var r1 = getInt32Memory0()[retptr / 4 + 1];
var v1 = getArrayU8FromWasm0(r0, r1).slice();
wasm.__wbindgen_free(r0, r1 * 1);
return v1;
} finally {
wasm.__wbindgen_export_0.value += 16;
}
}
export const __wbindgen_throw = function(arg0, arg1) {
throw new Error(getStringFromWasm0(arg0, arg1));
};

52
codecs/oxipng/spawn.ts Normal file
View File

@@ -0,0 +1,52 @@
import initOxiPNG, {
worker_initializer,
start_main_thread,
optimise,
} from './pkg-parallel';
// @ts-ignore
import wasmUrl from './pkg-parallel/squoosh_oxipng_bg.wasm';
import { WorkerInit } from './worker';
function initWorker(worker: Worker, workerInit: WorkerInit) {
return new Promise((resolve) => {
worker.postMessage(workerInit);
worker.addEventListener('message', () => resolve(), { once: true });
});
}
async function startMainThread() {
const num = navigator.hardwareConcurrency;
// First, let browser fetch and spawn Workers for our pool in the background.
// This is fairly expensive, so we want to start it as early as possible.
const workers = Array.from({ length: num }, () => new Worker('./worker', { type: 'module' }));
// Meanwhile, asynchronously compile, instantiate and initialise Wasm on our main thread.
await initOxiPNG(fetch(wasmUrl), undefined as any);
// Get module+memory from the Wasm instance.
//
// Ideally we wouldn't go via Wasm bindings here, since both are just JS variables, but memory is
// currently not exposed on the Wasm instance correctly by wasm-bindgen.
const workerInit: WorkerInit = worker_initializer(num);
// Once done, we want to send module+memory to each Worker so that they instantiate Wasm too.
// While doing so, we need to wait for Workers to acknowledge that they have received our message.
// Ideally this shouldn't be necessary, but Chromium currently doesn't conform to the spec:
// https://bugs.chromium.org/p/chromium/issues/detail?id=1075645
//
// If we didn't do this ping-pong game, the `start_main_thread` below would block the current
// thread on an atomic before even *sending* the `postMessage` containing memory,
// so Workers would never be able to unblock us back.
await Promise.all(workers.map(worker => initWorker(worker, workerInit)));
// Finally, instantiate rayon pool - this will use shared Wasm memory to send tasks to the
// Workers and then block until they're all ready.
start_main_thread();
return {
optimise,
};
}
export default startMainThread();

View File

@@ -1,6 +1,6 @@
import initOxiPNG, {
start_worker_thread,
} from 'codecs/oxipng/pkg-parallel/squoosh_oxipng';
/// <reference lib="webworker" />
import initOxiPNG, { start_worker_thread } from './pkg-parallel';
export type WorkerInit = [WebAssembly.Module, WebAssembly.Memory];
@@ -15,7 +15,6 @@ addEventListener(
// Note that we don't need to wait for Wasm instantiation here - it's
// better to start main thread as early as possible, and then it blocks
// on a shared atomic anyway until Worker is fully ready.
// @ts-ignore
postMessage(null);
await initOxiPNG(...(event.data as WorkerInit));

View File

@@ -1,60 +1,14 @@
/* tslint:disable */
/* eslint-disable */
/**
* @param {Uint8Array} input_image
* @param {number} input_width
* @param {number} input_height
* @param {number} output_width
* @param {number} output_height
* @param {number} typ_idx
* @param {boolean} premultiply
* @param {boolean} color_space_conversion
* @returns {Uint8Array}
*/
export function resize(
input_image: Uint8Array,
input_width: number,
input_height: number,
output_width: number,
output_height: number,
typ_idx: number,
premultiply: boolean,
color_space_conversion: boolean,
): Uint8Array;
export type InitInput =
| RequestInfo
| URL
| Response
| BufferSource
| WebAssembly.Module;
export interface InitOutput {
readonly memory: WebAssembly.Memory;
readonly resize: (
a: number,
b: number,
c: number,
d: number,
e: number,
f: number,
g: number,
h: number,
i: number,
j: number,
) => void;
readonly __wbindgen_malloc: (a: number) => number;
readonly __wbindgen_free: (a: number, b: number) => void;
}
/**
* If `module_or_path` is {RequestInfo} or {URL}, makes a request and
* for everything else, calls `WebAssembly.instantiate` directly.
*
* @param {InitInput | Promise<InitInput>} module_or_path
*
* @returns {Promise<InitOutput>}
*/
export default function init(
module_or_path?: InitInput | Promise<InitInput>,
): Promise<InitOutput>;
* @param {Uint8Array} input_image
* @param {number} input_width
* @param {number} input_height
* @param {number} output_width
* @param {number} output_height
* @param {number} typ_idx
* @param {boolean} premultiply
* @param {boolean} color_space_conversion
* @returns {Uint8Array}
*/
export function resize(input_image: Uint8Array, input_width: number, input_height: number, output_width: number, output_height: number, typ_idx: number, premultiply: boolean, color_space_conversion: boolean): Uint8Array;

View File

@@ -1,131 +1,2 @@
let wasm;
let cachegetUint8Memory0 = null;
function getUint8Memory0() {
if (
cachegetUint8Memory0 === null ||
cachegetUint8Memory0.buffer !== wasm.memory.buffer
) {
cachegetUint8Memory0 = new Uint8Array(wasm.memory.buffer);
}
return cachegetUint8Memory0;
}
let WASM_VECTOR_LEN = 0;
function passArray8ToWasm0(arg, malloc) {
const ptr = malloc(arg.length * 1);
getUint8Memory0().set(arg, ptr / 1);
WASM_VECTOR_LEN = arg.length;
return ptr;
}
let cachegetInt32Memory0 = null;
function getInt32Memory0() {
if (
cachegetInt32Memory0 === null ||
cachegetInt32Memory0.buffer !== wasm.memory.buffer
) {
cachegetInt32Memory0 = new Int32Array(wasm.memory.buffer);
}
return cachegetInt32Memory0;
}
function getArrayU8FromWasm0(ptr, len) {
return getUint8Memory0().subarray(ptr / 1, ptr / 1 + len);
}
/**
* @param {Uint8Array} input_image
* @param {number} input_width
* @param {number} input_height
* @param {number} output_width
* @param {number} output_height
* @param {number} typ_idx
* @param {boolean} premultiply
* @param {boolean} color_space_conversion
* @returns {Uint8Array}
*/
export function resize(
input_image,
input_width,
input_height,
output_width,
output_height,
typ_idx,
premultiply,
color_space_conversion,
) {
var ptr0 = passArray8ToWasm0(input_image, wasm.__wbindgen_malloc);
var len0 = WASM_VECTOR_LEN;
wasm.resize(
8,
ptr0,
len0,
input_width,
input_height,
output_width,
output_height,
typ_idx,
premultiply,
color_space_conversion,
);
var r0 = getInt32Memory0()[8 / 4 + 0];
var r1 = getInt32Memory0()[8 / 4 + 1];
var v1 = getArrayU8FromWasm0(r0, r1).slice();
wasm.__wbindgen_free(r0, r1 * 1);
return v1;
}
async function load(module, imports) {
if (typeof Response === 'function' && module instanceof Response) {
if (typeof WebAssembly.instantiateStreaming === 'function') {
try {
return await WebAssembly.instantiateStreaming(module, imports);
} catch (e) {
if (module.headers.get('Content-Type') != 'application/wasm') {
console.warn(
'`WebAssembly.instantiateStreaming` failed because your server does not serve wasm with `application/wasm` MIME type. Falling back to `WebAssembly.instantiate` which is slower. Original error:\n',
e,
);
} else {
throw e;
}
}
}
const bytes = await module.arrayBuffer();
return await WebAssembly.instantiate(bytes, imports);
} else {
const instance = await WebAssembly.instantiate(module, imports);
if (instance instanceof WebAssembly.Instance) {
return { instance, module };
} else {
return instance;
}
}
}
async function init(input) {
if (typeof input === 'undefined') {
input = import.meta.url.replace(/\.js$/, '_bg.wasm');
}
const imports = {};
if (
typeof input === 'string' ||
(typeof Request === 'function' && input instanceof Request) ||
(typeof URL === 'function' && input instanceof URL)
) {
input = fetch(input);
}
const { instance, module } = await load(await input, imports);
wasm = instance.exports;
init.__wbindgen_wasm_module = module;
return wasm;
}
export default init;
import * as wasm from "./squoosh_resize_bg.wasm";
export * from "./squoosh_resize_bg.js";

View File

@@ -0,0 +1,52 @@
import * as wasm from './squoosh_resize_bg.wasm';
let cachegetUint8Memory0 = null;
function getUint8Memory0() {
if (cachegetUint8Memory0 === null || cachegetUint8Memory0.buffer !== wasm.memory.buffer) {
cachegetUint8Memory0 = new Uint8Array(wasm.memory.buffer);
}
return cachegetUint8Memory0;
}
let WASM_VECTOR_LEN = 0;
function passArray8ToWasm0(arg, malloc) {
const ptr = malloc(arg.length * 1);
getUint8Memory0().set(arg, ptr / 1);
WASM_VECTOR_LEN = arg.length;
return ptr;
}
let cachegetInt32Memory0 = null;
function getInt32Memory0() {
if (cachegetInt32Memory0 === null || cachegetInt32Memory0.buffer !== wasm.memory.buffer) {
cachegetInt32Memory0 = new Int32Array(wasm.memory.buffer);
}
return cachegetInt32Memory0;
}
function getArrayU8FromWasm0(ptr, len) {
return getUint8Memory0().subarray(ptr / 1, ptr / 1 + len);
}
/**
* @param {Uint8Array} input_image
* @param {number} input_width
* @param {number} input_height
* @param {number} output_width
* @param {number} output_height
* @param {number} typ_idx
* @param {boolean} premultiply
* @param {boolean} color_space_conversion
* @returns {Uint8Array}
*/
export function resize(input_image, input_width, input_height, output_width, output_height, typ_idx, premultiply, color_space_conversion) {
var ptr0 = passArray8ToWasm0(input_image, wasm.__wbindgen_malloc);
var len0 = WASM_VECTOR_LEN;
wasm.resize(8, ptr0, len0, input_width, input_height, output_width, output_height, typ_idx, premultiply, color_space_conversion);
var r0 = getInt32Memory0()[8 / 4 + 0];
var r1 = getInt32Memory0()[8 / 4 + 1];
var v1 = getArrayU8FromWasm0(r0, r1).slice();
wasm.__wbindgen_free(r0, r1 * 1);
return v1;
}

Binary file not shown.

View File

@@ -1,13 +1,13 @@
ARG RUST_IMG=rust:1.47
ARG RUST_IMG=rust:1.44-stretch
FROM emscripten/emsdk:2.0.8 AS wasm-tools
FROM emscripten/emsdk:1.39.19 AS wasm-tools
WORKDIR /opt/wasm-tools
RUN wget -qO- https://github.com/rustwasm/wasm-pack/releases/download/v0.9.1/wasm-pack-v0.9.1-x86_64-unknown-linux-musl.tar.gz | tar -xzf - --strip 1
FROM $RUST_IMG AS rust
ARG RUST_IMG
RUN rustup target add wasm32-unknown-unknown
RUN if [[ $RUST_IMG = rustlang/rust:* ]] ; then rustup component add rust-src ; fi
RUN if [ "$RUST_IMG" = "rustlang/rust:nightly" ] ; then rustup component add rust-src ; fi
COPY --from=wasm-tools /emsdk/upstream/bin/wasm-opt /emsdk/upstream/bin/clang /usr/local/bin/
COPY --from=wasm-tools /emsdk/upstream/lib/ /usr/local/lib/
COPY --from=wasm-tools /emsdk/upstream/emscripten/system/include/libc/ /wasm32/include/
@@ -16,4 +16,4 @@ COPY --from=wasm-tools /opt/wasm-tools/wasm-pack /usr/local/cargo/bin/
ENV CPATH="/wasm32/include"
WORKDIR /src
CMD ["sh", "-c", "rm -rf pkg && wasm-pack build --target web -- --verbose --locked && rm pkg/.gitignore"]
CMD ["sh", "-c", "rm -rf pkg && wasm-pack build -- --verbose --locked && rm pkg/.gitignore"]

View File

@@ -1,4 +1,4 @@
CODEC_URL := https://storage.googleapis.com/downloads.webmproject.org/releases/webp/libwebp-1.1.0.tar.gz
CODEC_URL := https://storage.googleapis.com/downloads.webmproject.org/releases/webp/libwebp-1.0.2.tar.gz
CODEC_DIR = node_modules/libwebp
CODEC_OUT_RELATIVE = src/.libs/libwebp.a
CODEC_OUT := $(addprefix $(CODEC_DIR)/, $(CODEC_OUT_RELATIVE))
@@ -18,9 +18,7 @@ all: $(OUT_JS)
--closure 1 \
-s ALLOW_MEMORY_GROWTH=1 \
-s MODULARIZE=1 \
-s TEXTDECODER=2 \
-s ENVIRONMENT='worker' \
-s EXPORT_ES6=1 \
-s 'EXPORT_NAME="$(basename $(@F))"' \
-o $@ \
$+

View File

@@ -1,7 +1,5 @@
export interface WebPModule extends EmscriptenWasm.Module {
interface WebPModule extends EmscriptenWasm.Module {
decode(data: BufferSource): ImageData | null;
}
declare var moduleFactory: EmscriptenWasm.ModuleFactory<WebPModule>;
export default moduleFactory;
export default function(opts: EmscriptenWasm.ModuleOpts): Promise<WebPModule>;

File diff suppressed because it is too large Load Diff

BIN
codecs/webp/dec/webp_dec.wasm Executable file → Normal file

Binary file not shown.

View File

@@ -1,42 +1,7 @@
export interface EncodeOptions {
quality: number;
target_size: number;
target_PSNR: number;
method: number;
sns_strength: number;
filter_strength: number;
filter_sharpness: number;
filter_type: number;
partitions: number;
segments: number;
pass: number;
show_compressed: number;
preprocessing: number;
autofilter: number;
partition_limit: number;
alpha_compression: number;
alpha_filtering: number;
alpha_quality: number;
lossless: number;
exact: number;
image_hint: number;
emulate_jpeg_size: number;
thread_level: number;
low_memory: number;
near_lossless: number;
use_delta_palette: number;
use_sharp_yuv: number;
import { EncodeOptions } from '../../../src/codecs/webp/encoder-meta';
interface WebPModule extends EmscriptenWasm.Module {
encode(data: BufferSource, width: number, height: number, options: EncodeOptions): Uint8Array | null;
}
export interface WebPModule extends EmscriptenWasm.Module {
encode(
data: BufferSource,
width: number,
height: number,
options: EncodeOptions,
): Uint8Array | null;
}
declare var moduleFactory: EmscriptenWasm.ModuleFactory<WebPModule>;
export default moduleFactory;
export default function(opts: EmscriptenWasm.ModuleOpts): Promise<WebPModule>;

File diff suppressed because it is too large Load Diff

BIN
codecs/webp/enc/webp_enc.wasm Executable file → Normal file

Binary file not shown.

View File

@@ -19,10 +19,7 @@ all: $(OUT_JS)
--closure 1 \
-s ALLOW_MEMORY_GROWTH=1 \
-s MODULARIZE=1 \
-s TEXTDECODER=2 \
-s ENVIRONMENT='worker' \
-s EXPORT_ES6=1 \
-s EXPORT_NAME="$(basename $(@F))" \
-s 'EXPORT_NAME="$(basename $(@F))"' \
-o $@ \
$+

View File

@@ -1,7 +1,5 @@
export interface WP2Module extends EmscriptenWasm.Module {
interface WP2Module extends EmscriptenWasm.Module {
decode(data: BufferSource): ImageData | null;
}
declare var moduleFactory: EmscriptenWasm.ModuleFactory<WP2Module>;
export default moduleFactory;
export default function(opts: EmscriptenWasm.ModuleOpts): Promise<WP2Module>;

File diff suppressed because it is too large Load Diff

BIN
codecs/wp2/dec/wp2_dec.wasm Executable file → Normal file

Binary file not shown.

View File

@@ -1,20 +1,7 @@
export interface EncodeOptions {
quality: number;
alpha_quality: number;
speed: number;
pass: number;
sns: number;
import { EncodeOptions } from '../../../src/codecs/wp2/encoder-meta';
interface WP2Module extends EmscriptenWasm.Module {
encode(data: BufferSource, width: number, height: number, options: EncodeOptions): Uint8Array | null;
}
export interface WP2Module extends EmscriptenWasm.Module {
encode(
data: BufferSource,
width: number,
height: number,
options: EncodeOptions,
): Uint8Array | null;
}
declare var moduleFactory: EmscriptenWasm.ModuleFactory<WP2Module>;
export default moduleFactory;
export default function(opts: EmscriptenWasm.ModuleOpts): Promise<WP2Module>;

File diff suppressed because it is too large Load Diff

BIN
codecs/wp2/enc/wp2_enc.wasm Executable file → Normal file

Binary file not shown.

76
config/add-css-types.js Normal file
View File

@@ -0,0 +1,76 @@
const DtsCreator = require('typed-css-modules');
const chokidar = require('chokidar');
const util = require('util');
const sass = require('node-sass');
const normalizePath = require('normalize-path');
const sassRender = util.promisify(sass.render);
async function sassToCss(path) {
const result = await sassRender({ file: path });
return result.css;
}
/**
* @typedef {Object} Opts
* @property {boolean} watch Watch for changes
*/
/**
* Create typing files for CSS & SCSS.
*
* @param {string[]} rootPaths Paths to search within
* @param {Opts} [opts={}] Options.
*/
function addCssTypes(rootPaths, opts = {}) {
return new Promise((resolve) => {
const { watch = false } = opts;
const paths = [];
const preReadyPromises = [];
let ready = false;
for (const rootPath of rootPaths) {
const rootPathUnix = normalizePath(rootPath);
// Look for scss & css in each path.
paths.push(rootPathUnix + '/**/*.scss');
paths.push(rootPathUnix + '/**/*.css');
}
// For simplicity, the watcher is used even if we're not watching.
// If we're not watching, we stop the watcher after the initial files are found.
const watcher = chokidar.watch(paths, {
// Avoid processing already-processed files.
ignored: '*.d.*',
// Without this, travis and netlify builds never complete. I'm not sure why, but it might be
// related to https://github.com/paulmillr/chokidar/pull/758
persistent: watch,
});
function change(path) {
const promise = (async function() {
const creator = new DtsCreator({ camelCase: true });
const result = path.endsWith('.scss') ?
await creator.create(path, await sassToCss(path)) :
await creator.create(path);
await result.writeFile();
})();
if (!ready) preReadyPromises.push(promise);
}
watcher.on('change', change);
watcher.on('add', change);
// 'ready' is when events have been fired for file discovery.
watcher.on('ready', () => {
ready = true;
// Wait for the current set of processing to finish.
Promise.all(preReadyPromises).then(resolve);
// And if we're not watching, close the watcher.
if (!watch) watcher.close();
});
});
}
module.exports = addCssTypes;

View File

@@ -0,0 +1,47 @@
const path = require('path');
const fs = require('fs');
const ejs = require('ejs');
const AssetsPlugin = require('assets-webpack-plugin');
module.exports = class AssetTemplatePlugin extends AssetsPlugin {
constructor(options) {
options = options || {};
if (!options.template) throw Error('AssetTemplatePlugin: template option is required.');
super({
useCompilerPath: true,
filename: options.filename,
processOutput: files => this._processOutput(files)
});
this._template = path.resolve(process.cwd(), options.template);
const ignore = options.ignore || /(manifest\.json|\.DS_Store)$/;
this._ignore = typeof ignore === 'function' ? ({ test: ignore }) : ignore;
}
_processOutput(files) {
const mapping = {
all: [],
byType: {},
entries: {}
};
for (const entryName in files) {
// non-entry-point-derived assets are collected under an empty string key
// since that's a bit awkward, we'll call them "assets"
const name = entryName === '' ? 'assets' : entryName;
const listing = files[entryName];
const entry = mapping.entries[name] = {
all: [],
byType: {}
};
for (let type in listing) {
const list = [].concat(listing[type]).filter(file => !this._ignore.test(file));
if (!list.length) continue;
mapping.all = mapping.all.concat(list);
mapping.byType[type] = (mapping.byType[type] || []).concat(list);
entry.all = entry.all.concat(list);
entry.byType[type] = (entry.byType[type] || []).concat(list);
}
}
mapping.files = mapping.all;
return ejs.render(fs.readFileSync(this._template, 'utf8'), mapping);
}
};

View File

@@ -0,0 +1,29 @@
let loaderUtils = require('loader-utils');
let componentPath = require.resolve('./async-component');
module.exports = function () { };
module.exports.pitch = function (remainingRequest) {
this.cacheable && this.cacheable();
let query = loaderUtils.getOptions(this) || {};
let routeName = typeof query.name === 'function' ? query.name(this.resourcePath) : null;
let name;
if (routeName !== null) {
name = routeName;
}
else if ('name' in query) {
name = query.name;
}
else if ('formatName' in query) {
name = query.formatName(this.resourcePath);
}
return `
import async from ${JSON.stringify(componentPath)};
function load(cb) {
require.ensure([], function (require) {
cb( require(${loaderUtils.stringifyRequest(this, '!!' + remainingRequest)}) );
}${name ? (', ' + JSON.stringify(name)) : ''});
}
export default async(load);
`;
};

30
config/async-component.js Normal file
View File

@@ -0,0 +1,30 @@
import { h, Component } from 'preact';
export default function (req) {
function Async() {
Component.call(this);
let b, old;
this.componentWillMount = () => {
b = this.base = this.nextBase || this.__b; // short circuits 1st render
req(m => {
this.setState({ child: m.default || m });
});
};
this.shouldComponentUpdate = (_, nxt) => {
nxt = nxt.child === void 0;
if (nxt && old === void 0 && !!b) {
old = h(b.nodeName, { dangerouslySetInnerHTML: { __html: b.innerHTML } });
}
else {
old = ''; // dump it
}
return !nxt;
};
this.render = (p, s) => s.child ? h(s.child, p) : old;
}
(Async.prototype = new Component()).constructor = Async;
return Async;
}

158
config/auto-sw-plugin.js Normal file
View File

@@ -0,0 +1,158 @@
const util = require('util');
const minimatch = require('minimatch');
const SingleEntryPlugin = require('webpack/lib/SingleEntryPlugin');
const WebWorkerTemplatePlugin = require('webpack/lib/webworker/WebWorkerTemplatePlugin');
const ParserHelpers = require('webpack/lib/ParserHelpers');
const NAME = 'auto-sw-plugin';
const JS_TYPES = ['auto', 'esm', 'dynamic'];
/**
* Automatically finds and bundles Service Workers by looking for navigator.serviceWorker.register(..).
* An Array of webpack assets is injected into the Service Worker bundle as a `BUILD_ASSETS` global.
* Hidden and `.map` files are excluded by default, and this can be customized using the include & exclude options.
* @example
* // webpack config
* plugins: [
* new AutoSWPlugin({
* exclude: [
* '**\/.*', // don't expose hidden files (default)
* '**\/*.map', // don't precache sourcemaps (default)
* 'index.html' // don't cache the page itself
* ]
* })
* ]
* @param {Object} [options={}]
* @param {string[]} [options.exclude] Minimatch pattern(s) of which assets to omit from BUILD_ASSETS.
* @param {string[]} [options.include] Minimatch pattern(s) of assets to allow in BUILD_ASSETS.
*/
module.exports = class AutoSWPlugin {
constructor(options) {
this.options = Object.assign({
exclude: [
'**/*.map',
'**/.*'
]
}, options || {});
}
apply(compiler) {
const serviceWorkers = [];
compiler.hooks.emit.tapPromise(NAME, compilation => this.emit(compiler, compilation, serviceWorkers));
compiler.hooks.normalModuleFactory.tap(NAME, (factory) => {
for (const type of JS_TYPES) {
factory.hooks.parser.for(`javascript/${type}`).tap(NAME, parser => {
let counter = 0;
const processRegisterCall = expr => {
const dep = parser.evaluateExpression(expr.arguments[0]);
if (!dep.isString()) {
parser.state.module.warnings.push({
message: 'navigator.serviceWorker.register() will only be bundled if passed a String literal.'
});
return false;
}
const filename = dep.string;
const outputFilename = this.options.filename || 'serviceworker.js'
const context = parser.state.current.context;
serviceWorkers.push({
outputFilename,
filename,
context
});
const id = `__webpack__serviceworker__${++counter}`;
ParserHelpers.toConstantDependency(parser, id)(expr.arguments[0]);
return ParserHelpers.addParsedVariableToModule(parser, id, '__webpack_public_path__ + ' + JSON.stringify(outputFilename));
};
parser.hooks.call.for('navigator.serviceWorker.register').tap(NAME, processRegisterCall);
parser.hooks.call.for('self.navigator.serviceWorker.register').tap(NAME, processRegisterCall);
parser.hooks.call.for('window.navigator.serviceWorker.register').tap(NAME, processRegisterCall);
});
}
});
}
createFilter(list) {
const filters = [].concat(list);
for (let i=0; i<filters.length; i++) {
if (typeof filters[i] === 'string') {
filters[i] = minimatch.filter(filters[i]);
}
}
return filters;
}
async emit(compiler, compilation, serviceWorkers) {
let assetMapping = Object.keys(compilation.assets);
if (this.options.include) {
const filters = this.createFilter(this.options.include);
assetMapping = assetMapping.filter(filename => {
for (const filter of filters) {
if (filter(filename)) return true;
}
return false;
});
}
if (this.options.exclude) {
const filters = this.createFilter(this.options.exclude);
assetMapping = assetMapping.filter(filename => {
for (const filter of filters) {
if (filter(filename)) return false;
}
return true;
});
}
await Promise.all(serviceWorkers.map(
(serviceWorker, index) => this.compileServiceWorker(compiler, compilation, serviceWorker, index, assetMapping)
));
}
async compileServiceWorker(compiler, compilation, options, index, assetMapping) {
const entryFilename = options.filename;
const chunkFilename = compiler.options.output.chunkFilename.replace(/\.([a-z]+)$/i, '.serviceworker.$1');
const workerOptions = {
filename: options.outputFilename, // chunkFilename.replace(/\.?\[(?:chunkhash|contenthash|hash)(:\d+(?::\d+)?)?\]/g, ''),
chunkFilename: this.options.chunkFilename || chunkFilename,
globalObject: 'self'
};
const childCompiler = compilation.createChildCompiler(NAME, { filename: workerOptions.filename });
(new WebWorkerTemplatePlugin(workerOptions)).apply(childCompiler);
/* The duplication DefinePlugin ends up causing is problematic (it doesn't hoist injections), so we'll do it manually. */
// (new DefinePlugin({
// BUILD_ASSETS: JSON.stringify(assetMapping)
// })).apply(childCompiler);
(new SingleEntryPlugin(options.context, entryFilename, workerOptions.filename)).apply(childCompiler);
const subCache = `subcache ${__dirname} ${entryFilename} ${index}`;
let childCompilation;
childCompiler.hooks.compilation.tap(NAME, c => {
childCompilation = c;
if (childCompilation.cache) {
if (!childCompilation.cache[subCache]) childCompilation.cache[subCache] = {};
childCompilation.cache = childCompilation.cache[subCache];
}
});
await (util.promisify(childCompiler.runAsChild.bind(childCompiler)))();
const versionVar = this.options.version ?
`var VERSION = ${JSON.stringify(this.options.version)};` : '';
const original = childCompilation.assets[workerOptions.filename].source();
const source = `${versionVar}var BUILD_ASSETS=${JSON.stringify(assetMapping)};${original}`;
childCompilation.assets[workerOptions.filename] = {
source: () => source,
size: () => Buffer.byteLength(source, 'utf8')
};
Object.assign(compilation.assets, childCompilation.assets);
}
};

View File

@@ -0,0 +1,30 @@
const fs = require('fs');
/** A Webpack plugin to refresh file mtime values from disk before compiling.
* This is used in order to account for SCSS-generated .d.ts files written
* as part of compilation so they trigger only a single recompile per write.
*
* All credit for the technique and implementation goes to @reiv. See:
* https://github.com/Jimdo/typings-for-css-modules-loader/issues/48#issuecomment-347036461
*/
module.exports = class WatchTimestampsPlugin {
constructor(patterns) {
this.patterns = patterns;
}
apply(compiler) {
compiler.hooks.watchRun.tapAsync('watch-timestamps-plugin', (watch, callback) => {
const patterns = this.patterns;
const timestamps = watch.fileTimestamps;
for (const filepath of timestamps) {
if (patterns.some(pat => pat instanceof RegExp ? pat.test(filepath) : filepath.indexOf(pat) === 0)) {
let time = fs.statSync(filepath).mtime;
if (timestamps instanceof Map) timestamps.set(filepath, time);
else timestamps[filepath] = time;
}
}
callback();
});
}
};

View File

@@ -1,11 +1,7 @@
// These types roughly model the object that the JS files generated by Emscripten define. Copied from https://github.com/DefinitelyTyped/DefinitelyTyped/blob/master/types/emscripten/index.d.ts and turned into a type definition rather than a global to support our way of using Emscripten.
// TODO(@surma): Upstream this?
declare namespace EmscriptenWasm {
type ModuleFactory<T extends Module = Module> = (
moduleOverrides?: ModuleOpts,
) => Promise<T>;
type EnvironmentType = 'WEB' | 'NODE' | 'SHELL' | 'WORKER';
type EnvironmentType = "WEB" | "NODE" | "SHELL" | "WORKER";
// Options object for modularized Emscripten files. Shoe-horned by @surma.
// FIXME: This an incomplete definition!
@@ -21,9 +17,9 @@ declare namespace EmscriptenWasm {
printErr(str: string): void;
arguments: string[];
environment: EnvironmentType;
preInit: { (): void }[];
preRun: { (): void }[];
postRun: { (): void }[];
preInit: { (): void }[];
preRun: { (): void }[];
postRun: { (): void }[];
preinitializedWebGLContext: WebGLRenderingContext;
noInitialRun: boolean;
noExitRuntime: boolean;
@@ -32,25 +28,17 @@ declare namespace EmscriptenWasm {
wasmBinary: ArrayBuffer;
destroy(object: object): void;
getPreloadedPackage(
remotePackageName: string,
remotePackageSize: number,
): ArrayBuffer;
getPreloadedPackage(remotePackageName: string, remotePackageSize: number): ArrayBuffer;
instantiateWasm(
imports: WebAssembly.Imports,
successCallback: (module: WebAssembly.Module) => void,
imports: WebAssembly.Imports,
successCallback: (module: WebAssembly.Module) => void
): WebAssembly.Exports;
locateFile(url: string): string;
onCustomMessage(event: MessageEvent): void;
Runtime: any;
ccall(
ident: string,
returnType: string | null,
argTypes: string[],
args: any[],
): any;
ccall(ident: string, returnType: string | null, argTypes: string[], args: any[]): any;
cwrap(ident: string, returnType: string | null, argTypes: string[]): any;
setValue(ptr: number, value: any, type: string, noSafe?: boolean): void;
@@ -63,12 +51,7 @@ declare namespace EmscriptenWasm {
ALLOC_NONE: number;
allocate(slab: any, types: string, allocator: number, ptr: number): number;
allocate(
slab: any,
types: string[],
allocator: number,
ptr: number,
): number;
allocate(slab: any, types: string[], allocator: number, ptr: number): number;
Pointer_stringify(ptr: number, length?: number): string;
UTF16ToString(ptr: number): string;
@@ -85,7 +68,7 @@ declare namespace EmscriptenWasm {
HEAP8: Int8Array;
HEAP16: Int16Array;
HEAP32: Int32Array;
HEAPU8: Uint8Array;
HEAPU8: Uint8Array;
HEAPU16: Uint16Array;
HEAPU32: Uint32Array;
HEAPF32: Float32Array;
@@ -102,23 +85,16 @@ declare namespace EmscriptenWasm {
addOnPostRun(cb: () => any): void;
// Tools
intArrayFromString(
stringy: string,
dontAddNull?: boolean,
length?: number,
): number[];
intArrayFromString(stringy: string, dontAddNull?: boolean, length?: number): number[];
intArrayToString(array: number[]): string;
writeStringToMemory(
str: string,
buffer: number,
dontAddNull: boolean,
): void;
writeStringToMemory(str: string, buffer: number, dontAddNull: boolean): void;
writeArrayToMemory(array: number[], buffer: number): void;
writeAsciiToMemory(str: string, buffer: number, dontAddNull: boolean): void;
addRunDependency(id: any): void;
removeRunDependency(id: any): void;
preloadedImages: any;
preloadedAudios: any;
@@ -129,3 +105,4 @@ declare namespace EmscriptenWasm {
onRuntimeInitialized: () => void | null;
}
}

View File

@@ -1,23 +0,0 @@
{
"compilerOptions": {
"target": "ES2019",
"downlevelIteration": true,
"module": "esnext",
"jsx": "react",
"jsxFactory": "h",
"strict": true,
"moduleResolution": "node",
"composite": true,
"declarationMap": true,
"baseUrl": "./",
"rootDir": "./",
"outDir": ".tmp/ts",
"allowSyntheticDefaultImports": true,
"paths": {
"static-build/*": ["src/static-build/*"],
"client/*": ["src/client/*"],
"shared/*": ["src/shared/*"],
"features/*": ["src/features/*"]
}
}
}

23
global.d.ts vendored Normal file
View File

@@ -0,0 +1,23 @@
declare const __webpack_public_path__: string;
declare const PRERENDER: boolean;
declare interface NodeModule {
hot: any;
}
declare interface Window {
STATE: any;
ga: typeof ga;
}
declare namespace JSX {
interface Element { }
interface IntrinsicElements { }
interface HTMLAttributes {
decoding?: string;
}
}
declare module 'classnames' {
export default function classnames(...args: any[]): string;
}

View File

@@ -1,175 +0,0 @@
/**
* Copyright 2020 Google Inc. All Rights Reserved.
* Licensed under the Apache License, Version 2.0 (the "License");
* you may not use this file except in compliance with the License.
* You may obtain a copy of the License at
* http://www.apache.org/licenses/LICENSE-2.0
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
import rollup from 'rollup';
const prefix = 'client-bundle:';
const entryPathPlaceholder = 'CLIENT_BUNDLE_PLUGIN_ENTRY_PATH';
const importsPlaceholder = 'CLIENT_BUNDLE_PLUGIN_IMPORTS';
const allSrcPlaceholder = 'CLIENT_BUNDLE_PLUGIN_ALL_SRC';
export function getDependencies(clientOutput, item) {
const crawlDependencies = new Set([item.fileName]);
for (const fileName of crawlDependencies) {
const chunk = clientOutput.find((v) => v.fileName === fileName);
for (const dep of chunk.imports) {
crawlDependencies.add(dep);
}
}
// Don't add self as dependency
crawlDependencies.delete(item.fileName);
return [...crawlDependencies];
}
export default function (inputOptions, outputOptions, resolveFileUrl) {
let cache;
let entryPointPlaceholderMap;
let exportCounter;
let clientBundle;
let clientOutput;
return {
name: 'client-bundle',
buildStart() {
entryPointPlaceholderMap = new Map();
exportCounter = 0;
},
async resolveId(id, importer) {
if (!id.startsWith(prefix)) return null;
const realId = id.slice(prefix.length);
const resolveResult = await this.resolve(realId, importer);
// Add an additional .js to the end so it ends up with .js at the end in the _virtual folder.
if (resolveResult) return prefix + resolveResult.id + '.js';
// This Rollup couldn't resolve it, but maybe the inner one can.
return id + '.js';
},
load(id) {
if (!id.startsWith(prefix)) return;
const realId = id.slice(prefix.length, -'.js'.length);
exportCounter++;
entryPointPlaceholderMap.set(exportCounter, realId);
return [
`export default import.meta.${entryPathPlaceholder + exportCounter};`,
`export const imports = import.meta.${
importsPlaceholder + exportCounter
};`,
`export const allSrc = import.meta.${
allSrcPlaceholder + exportCounter
};`,
].join('\n');
},
async buildEnd(error) {
const entryPoints = [...entryPointPlaceholderMap.values()];
// The static-build is done, so now we can perform our client build.
// Exit early if there's nothing to build.
if (error || entryPoints.length === 0) return;
clientBundle = await rollup.rollup({
...inputOptions,
cache,
input: entryPoints,
});
cache = clientBundle.cache;
},
async renderStart(staticBuildOutputOpts) {
// The static-build has started generating output, so we can do the same for our client build.
// Exit early if there's nothing to build.
if (!clientBundle) return;
const copiedOutputOptions = {
assetFileNames: staticBuildOutputOpts.assetFileNames,
};
clientOutput = (
await clientBundle.generate({
...copiedOutputOptions,
...outputOptions,
})
).output;
},
resolveImportMeta(property, { moduleId, format }) {
// Pick up the placeholder exports we created earlier, and fill in the correct details.
let num = undefined;
if (property.startsWith(entryPathPlaceholder)) {
num = Number(property.slice(entryPathPlaceholder.length));
} else if (property.startsWith(importsPlaceholder)) {
num = Number(property.slice(importsPlaceholder.length));
} else if (property.startsWith(allSrcPlaceholder)) {
num = Number(property.slice(allSrcPlaceholder.length));
} else {
// This isn't one of our placeholders.
return;
}
const id = entryPointPlaceholderMap.get(num);
const clientEntry = clientOutput.find(
(item) => item.facadeModuleId === id,
);
if (property.startsWith(entryPathPlaceholder)) {
return resolveFileUrl({
fileName: clientEntry.fileName,
moduleId,
format,
});
}
const dependencies = getDependencies(clientOutput, clientEntry);
if (property.startsWith(allSrcPlaceholder)) {
return JSON.stringify(
[clientEntry.code, ...dependencies.map((d) => d.code)].join(';'),
);
}
return (
'[' +
dependencies
.map((item) => {
const entry = clientOutput.find((v) => v.fileName === item);
return resolveFileUrl({
fileName: entry.fileName,
moduleId,
format: outputOptions.format,
});
})
.join(',') +
']'
);
},
async generateBundle(options, bundle) {
// Exit early if there's nothing to build.
if (!clientOutput) return;
// Copy everything from the client bundle into the main bundle.
for (const clientEntry of clientOutput) {
// Skip if the file already exists
if (clientEntry.fileName in bundle) continue;
this.emitFile({
type: 'asset',
source: clientEntry.code || clientEntry.source,
fileName: clientEntry.fileName,
});
}
},
};
}

View File

@@ -1,191 +0,0 @@
/**
* Copyright 2020 Google Inc. All Rights Reserved.
* Licensed under the Apache License, Version 2.0 (the "License");
* you may not use this file except in compliance with the License.
* You may obtain a copy of the License at
* http://www.apache.org/licenses/LICENSE-2.0
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
import { promises as fsp, readFileSync } from 'fs';
import { createHash } from 'crypto';
import { promisify } from 'util';
import {
parse as parsePath,
resolve as resolvePath,
dirname,
normalize as nomalizePath,
} from 'path';
import postcss from 'postcss';
import postCSSNested from 'postcss-nested';
import postCSSUrl from 'postcss-url';
import postCSSModules from 'postcss-modules';
import postCSSSimpleVars from 'postcss-simple-vars';
import cssNano from 'cssnano';
import camelCase from 'lodash.camelcase';
import glob from 'glob';
const globP = promisify(glob);
const moduleSuffix = '.css';
const sourcePrefix = 'css:';
const addPrefix = 'add-css:';
const assetRe = new RegExp('/fake/path/to/asset/([^/]+)/', 'g');
const appendCssModule = '\0appendCss';
const appendCssSource = `
export default function appendCss(css) {
const style = document.createElement('style');
style.textContent = css;
document.head.append(style);
}
`;
export default function (resolveFileUrl) {
/** @type {string[]} */
let emittedCSSIds;
/** @type {Map<string, string>} */
let hashToId;
/** @type {Map<string, { module: string, css: string }>} */
let pathToResult;
return {
name: 'css',
async buildStart() {
emittedCSSIds = [];
hashToId = new Map();
pathToResult = new Map();
const cssPaths = (
await globP('src/**/*.css', {
nodir: true,
absolute: true,
})
).map((cssPath) =>
// glob() returns windows paths with a forward slash. Normalise it:
nomalizePath(cssPath),
);
await Promise.all(
cssPaths.map(async (path) => {
this.addWatchFile(path);
const file = await fsp.readFile(path);
let moduleJSON;
const cssResult = await postcss([
postCSSNested,
postCSSSimpleVars(),
postCSSModules({
getJSON(_, json) {
moduleJSON = json;
},
root: '',
}),
postCSSUrl({
url: ({ relativePath, url }) => {
if (/^(https?|data):/.test(url)) return url;
const parsedPath = parsePath(relativePath);
const source = readFileSync(
resolvePath(dirname(path), relativePath),
);
const fileId = this.emitFile({
type: 'asset',
name: parsedPath.base,
source,
});
const hash = createHash('md5');
hash.update(source);
const md5 = hash.digest('hex');
hashToId.set(md5, fileId);
return `/fake/path/to/asset/${md5}/`;
},
}),
cssNano,
]).process(file, {
from: path,
});
const cssClassExports = Object.entries(moduleJSON).map(
([key, val]) =>
`export const ${camelCase(key)} = ${JSON.stringify(val)};`,
);
const defs =
'// This file is autogenerated by lib/css-plugin.js\n' +
Object.keys(moduleJSON)
.map((key) => `export const ${camelCase(key)}: string;`)
.join('\n');
const defPath = path + '.d.ts';
const currentDefFileContent = await fsp
.readFile(defPath, { encoding: 'utf8' })
.catch(() => undefined);
// Only write the file if contents have changed, otherwise it causes a loop with
// TypeScript's file watcher.
if (defs !== currentDefFileContent) {
await fsp.writeFile(defPath, defs);
}
pathToResult.set(path, {
module: cssClassExports.join('\n'),
css: cssResult.css,
});
}),
);
},
async resolveId(id, importer) {
if (id === appendCssModule) return id;
const prefix = id.startsWith(sourcePrefix)
? sourcePrefix
: id.startsWith(addPrefix)
? addPrefix
: undefined;
if (!prefix) return;
const resolved = await this.resolve(id.slice(prefix.length), importer);
if (!resolved) throw Error(`Couldn't resolve ${id} from ${importer}`);
return prefix + resolved.id;
},
async load(id) {
if (id === appendCssModule) return appendCssSource;
if (id.startsWith(sourcePrefix)) {
const path = nomalizePath(id.slice(sourcePrefix.length));
if (!pathToResult.has(path)) {
throw Error(`Cannot find ${path} in pathToResult`);
}
const cssStr = JSON.stringify(pathToResult.get(path).css).replace(
assetRe,
(match, hash) =>
`" + import.meta.ROLLUP_FILE_URL_${hashToId.get(hash)} + "`,
);
return `export default ${cssStr};`;
}
if (id.startsWith(addPrefix)) {
const path = id.slice(addPrefix.length);
return (
`import css from 'css:${path}';\n` +
`import appendCss from '${appendCssModule}';\n` +
`appendCss(css);\n`
);
}
if (id.endsWith(moduleSuffix)) {
if (!pathToResult.has(id)) {
throw Error(`Cannot find ${id} in pathToResult`);
}
return pathToResult.get(id).module;
}
},
};
}

View File

@@ -1,41 +0,0 @@
/**
* Copyright 2020 Google Inc. All Rights Reserved.
* Licensed under the Apache License, Version 2.0 (the "License");
* you may not use this file except in compliance with the License.
* You may obtain a copy of the License at
* http://www.apache.org/licenses/LICENSE-2.0
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
import { promises as fs } from 'fs';
import { lookup as lookupMime } from 'mime-types';
const prefix = 'data-url:';
export default function dataURLPlugin() {
return {
name: 'data-url-plugin',
async resolveId(id, importer) {
if (!id.startsWith(prefix)) return;
return (
prefix + (await this.resolve(id.slice(prefix.length), importer)).id
);
},
async load(id) {
if (!id.startsWith(prefix)) return;
const realId = id.slice(prefix.length);
this.addWatchFile(realId);
const source = await fs.readFile(realId);
const type = lookupMime(realId) || 'text/plain';
return `export default 'data:${type};base64,${source.toString(
'base64',
)}';`;
},
};
}

View File

@@ -1,37 +0,0 @@
/**
* Copyright 2020 Google Inc. All Rights Reserved.
* Licensed under the Apache License, Version 2.0 (the "License");
* you may not use this file except in compliance with the License.
* You may obtain a copy of the License at
* http://www.apache.org/licenses/LICENSE-2.0
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
import * as path from 'path';
import { promises as fs } from 'fs';
import glob from 'glob';
import { promisify } from 'util';
const globP = promisify(glob);
export default function emitFiles({ root, include }) {
return {
name: 'emit-files-plugin',
async buildStart() {
const paths = await globP(include, { nodir: true, cwd: root });
await Promise.all(
paths.map(async (filePath) => {
return this.emitFile({
type: 'asset',
source: await fs.readFile(path.join(root, filePath)),
fileName: 'static/' + filePath,
});
}),
);
},
};
}

View File

@@ -1,270 +0,0 @@
/**
* Copyright 2020 Google Inc. All Rights Reserved.
* Licensed under the Apache License, Version 2.0 (the "License");
* you may not use this file except in compliance with the License.
* You may obtain a copy of the License at
* http://www.apache.org/licenses/LICENSE-2.0
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
import { promisify } from 'util';
import * as path from 'path';
import { posix } from 'path';
import glob from 'glob';
import { promises as fsp } from 'fs';
const globP = promisify(glob);
const autoGenComment =
'// This file is autogenerated by lib/feature-plugin.js\n';
export default function () {
let previousWorkerImports;
let previousJoinedMetas;
/**
* Generates the worker file & tsconfig for all features
*
* @param {string[]} workerImports
*/
async function generateWorkerFile(workerImports) {
const workerBasePath = path.join(process.cwd(), 'src', 'features-worker');
const featuresWorkerTsNames = workerImports.map((tsImport) => [
path.relative(workerBasePath, tsImport).split(path.sep).join(posix.sep),
path.basename(tsImport),
]);
const workerFile = [
autoGenComment,
`import { expose } from 'comlink';`,
`import { timed } from './util';`,
featuresWorkerTsNames.map(
([path, name]) => `import ${name} from './${path}';`,
),
`const exports = {`,
featuresWorkerTsNames.map(([_, name]) => [
` ${name}(`,
` ...args: Parameters<typeof ${name}>`,
` ): ReturnType<typeof ${name}> {`,
` return timed('${name}', () => ${name}(...args));`,
` },`,
]),
`};`,
`export type ProcessorWorkerApi = typeof exports;`,
`// 'as any' to work around the way our client code has insight into worker code`,
`expose(exports, self as any);`,
]
.flat(Infinity)
.join('\n');
await fsp.writeFile(path.join(workerBasePath, 'index.ts'), workerFile);
}
/**
* Generates the client JS to call worker methods.
*
* @param {string[]} workerImports
*/
async function generateWorkerBridge(workerImports) {
const workerBridgeBasePath = path.join(
process.cwd(),
'src',
'client',
'lazy-app',
'worker-bridge',
);
const featuresWorkerBridgeTsNames = workerImports.map((tsImport) => [
path
.relative(workerBridgeBasePath, tsImport)
.split(path.sep)
.join(posix.sep),
path.basename(tsImport),
]);
const bridgeMeta = [
autoGenComment,
featuresWorkerBridgeTsNames.map(
([path, name]) => `import type ${name} from '${path}';`,
),
`export const methodNames = ${JSON.stringify(
featuresWorkerBridgeTsNames.map(([_, name]) => name),
null,
' ',
)} as const;`,
`export interface BridgeMethods {`,
featuresWorkerBridgeTsNames.map(([_, name]) => [
` ${name}(`,
` signal: AbortSignal,`,
` ...args: Parameters<typeof ${name}>`,
` ): Promise<ReturnType<typeof ${name}>>;`,
]),
`}`,
]
.flat(Infinity)
.join('\n');
await fsp.writeFile(path.join(workerBridgeBasePath, 'meta.ts'), bridgeMeta);
}
async function generateWorkerFiles() {
const workerImports = (
await globP('src/features/*/**/worker/*.ts', {
absolute: true,
})
)
.filter((tsFile) => !tsFile.endsWith('.d.ts'))
.map((tsFile) => tsFile.slice(0, -'.ts'.length));
const joinedWorkerImports = workerImports.join();
// Avoid regenerating if nothing's changed.
// This also prevents an infinite loop in the watcher.
if (joinedWorkerImports === previousWorkerImports) return;
previousWorkerImports = joinedWorkerImports;
await Promise.all([
generateWorkerFile(workerImports),
generateWorkerBridge(workerImports),
]);
}
async function generateFeatureMeta() {
const getTsFiles = (glob) =>
globP(glob, {
absolute: true,
}).then((paths) =>
paths
.filter((tsFile) => !tsFile.endsWith('.d.ts'))
.map((tsFile) => tsFile.slice(0, -'.ts'.length)),
);
const metas = await Promise.all(
[
'src/features/encoders/*/shared/meta.ts',
'src/features/processors/*/shared/meta.ts',
'src/features/preprocessors/*/shared/meta.ts',
].map((glob) => getTsFiles(glob)),
);
const [encoderMetas, processorMetas, preprocessorMetas] = metas;
const featureMetaBasePath = path.join(
process.cwd(),
'src',
'client',
'lazy-app',
'feature-meta',
);
const joinedMetas = metas.flat().join();
// Avoid regenerating if nothing's changed.
// This also prevents an infinite loop in the watcher.
if (joinedMetas === previousJoinedMetas) return;
previousJoinedMetas = joinedMetas;
const getTsName = (tsImport) => [
path
.relative(featureMetaBasePath, tsImport)
.split(path.sep)
.join(posix.sep),
path.basename(tsImport.slice(0, -'/shared/meta'.length)),
];
const encoderMetaTsNames = encoderMetas.map((tsImport) =>
getTsName(tsImport),
);
const processorMetaTsNames = processorMetas.map((tsImport) =>
getTsName(tsImport),
);
const preprocessorMetaTsNames = preprocessorMetas.map((tsImport) =>
getTsName(tsImport),
);
const featureMeta = [
autoGenComment,
// Encoder stuff
encoderMetaTsNames.map(
([path, name]) => `import * as ${name}EncoderMeta from '${path}';`,
),
encoderMetaTsNames.map(
([path, name]) =>
`import * as ${name}EncoderEntry from '${path.replace(
/shared\/meta$/,
'client',
)}';`,
),
`export type EncoderState =`,
encoderMetaTsNames.map(
([_, name]) =>
` | { type: "${name}", options: ${name}EncoderMeta.EncodeOptions }`,
),
`;`,
`export type EncoderOptions =`,
encoderMetaTsNames.map(
([_, name]) => ` | ${name}EncoderMeta.EncodeOptions`,
),
`;`,
`export const encoderMap = {`,
encoderMetaTsNames.map(
([_, name]) =>
` ${name}: { meta: ${name}EncoderMeta, ...${name}EncoderEntry },`,
),
`};`,
`export type EncoderType = keyof typeof encoderMap`,
// Processor stuff
processorMetaTsNames.map(
([path, name]) => `import * as ${name}ProcessorMeta from '${path}';`,
),
`interface Enableable { enabled: boolean; }`,
`export interface ProcessorOptions {`,
processorMetaTsNames.map(
([_, name]) => ` ${name}: ${name}ProcessorMeta.Options;`,
),
`}`,
`export interface ProcessorState {`,
processorMetaTsNames.map(
([_, name]) => ` ${name}: Enableable & ${name}ProcessorMeta.Options;`,
),
`}`,
`export const defaultProcessorState: ProcessorState = {`,
processorMetaTsNames.map(
([_, name]) =>
` ${name}: { enabled: false, ...${name}ProcessorMeta.defaultOptions },`,
),
`}`,
// Preprocessor stuff
preprocessorMetaTsNames.map(
([path, name]) => `import * as ${name}PreprocessorMeta from '${path}';`,
),
`export interface PreprocessorState {`,
preprocessorMetaTsNames.map(
([_, name]) => ` ${name}: ${name}PreprocessorMeta.Options,`,
),
`}`,
`export const defaultPreprocessorState: PreprocessorState = {`,
preprocessorMetaTsNames.map(
([_, name]) => ` ${name}: ${name}PreprocessorMeta.defaultOptions,`,
),
`};`,
]
.flat(Infinity)
.join('\n');
await fsp.writeFile(
path.join(featureMetaBasePath, 'index.ts'),
featureMeta,
);
}
return {
name: 'feature-plugin',
async buildStart() {
await Promise.all([generateWorkerFiles(), generateFeatureMeta()]);
},
};
}

View File

@@ -1,54 +0,0 @@
/**
* Copyright 2020 Google Inc. All Rights Reserved.
* Licensed under the Apache License, Version 2.0 (the "License");
* you may not use this file except in compliance with the License.
* You may obtain a copy of the License at
* http://www.apache.org/licenses/LICENSE-2.0
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
import { promisify } from 'util';
import path from 'path';
import { posix } from 'path';
import glob from 'glob';
const globP = promisify(glob);
const moduleId = 'initial-css:';
export default function initialCssPlugin() {
return {
name: 'initial-css-plugin',
resolveId(id) {
if (id === moduleId) return moduleId;
},
async load(id) {
if (id !== moduleId) return;
const matches = await globP('shared/initial-app/**/*.css', {
nodir: true,
cwd: path.join(process.cwd(), 'src'),
});
// Sort the matches so the parentmost items appear first.
// This is a bit of a hack, but it means the util stuff appears in the cascade first.
const sortedMatches = matches
.map((match) => path.normalize(match).split(path.sep))
.sort((a, b) => a.length - b.length)
.map((match) => posix.join(...match));
const imports = sortedMatches
.map((id, i) => `import css${i} from 'css:${id}';\n`)
.join('');
return (
imports +
`export default ${sortedMatches.map((_, i) => `css${i}`).join(' + ')};`
);
},
};
}

View File

@@ -1,19 +0,0 @@
/**
* Copyright 2020 Google Inc. All Rights Reserved.
* Licensed under the Apache License, Version 2.0 (the "License");
* you may not use this file except in compliance with the License.
* You may obtain a copy of the License at
* http://www.apache.org/licenses/LICENSE-2.0
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
// Move .tmp/build/static to docs/
const fs = require('fs');
const del = require('del');
const path = require('path');
del.sync('build');
fs.renameSync(path.join('.tmp', 'build', 'static'), 'build');

Some files were not shown because too many files have changed in this diff Show More