Skinbase Enhance Setup Guide
===========================

This guide explains how to set up, enable, verify, and operate the Skinbase Enhance module end to end.

Use this document when you need to:

- enable Enhance in a local environment
- switch from stub mode to the external worker
- run the worker in Pillow mode or Real-ESRGAN mode
- configure queues, cleanup, and health checks
- understand what the module stores and how it behaves in production

What the module does
--------------------

Enhance accepts an uploaded or selected source image, creates an Enhance job, processes that job through the configured engine, stores the generated output on the Enhance storage disk, and keeps the original source file untouched.

Current supported engines on the Laravel side:

- `ENHANCE_ENGINE=stub`
- `ENHANCE_ENGINE=external_worker`

Current supported worker engines:

- `WORKER_ENGINE=pillow`
- `WORKER_ENGINE=realesrgan-ncnn`
- `WORKER_ENGINE=realesrgan`

`WORKER_ENGINE=realesrgan` currently aliases to `realesrgan-ncnn`.

Architecture
------------

The Enhance module is split into two layers.

Laravel application:

- accepts the Enhance request
- validates allowed file types, dimensions, scales, and modes
- stores job records
- dispatches the job to the queue
- owns permanent storage for Enhance sources and outputs
- exposes moderation, cleanup, and health commands

Optional external worker:

- downloads the copied Enhance source image
- upscales it with Pillow or Real-ESRGAN
- exposes a temporary internal result URL
- deletes its temporary result after Laravel confirms download

Laravel remains the source of truth. The worker is temporary processing only.

Default behavior and limits
---------------------------

Default config values from [config/enhance.php](config/enhance.php):

- disk: `ENHANCE_DISK` or the app filesystem default
- source prefix: `enhance/sources`
- output prefix: `enhance/outputs`
- preview prefix: `enhance/previews`
- default engine: `stub`
- allowed MIME types: `image/jpeg`, `image/png`, `image/webp`
- allowed modes: `standard`, `artwork`, `photo`, `illustration`
- allowed scales: `2`, `4`
- daily limit: `10`
- default queue: `default`

Lifecycle defaults:

- completed jobs expire after `30` days
- failed jobs expire after `7` days
- deleted job files get a `1` day grace period

Health defaults:

- processing jobs are considered stuck after `30` minutes
- queued jobs are considered stale after `60` minutes

Prerequisites
-------------

Laravel requirements:

- the application boots normally
- database migrations are current
- the configured filesystem disk is writable
- a queue worker or Horizon is running for the queue you assign to Enhance
- Laravel scheduler is enabled if you want automatic cleanup

Worker requirements for `external_worker` mode:

- Docker or a Python runtime for `services/enhance-worker`
- network access from Laravel to the worker URL
- a shared bearer token between Laravel and the worker
- for Real-ESRGAN mode, the `realesrgan-ncnn-vulkan` binary and model files

Setup paths
-----------

There are three practical ways to run Enhance.

1. Stub mode

- safest local starting point
- exercises the Laravel flow without a real AI runtime
- no worker needed

2. External worker with Pillow

- good for local integration testing and CI-like validation
- real HTTP worker contract
- deterministic fallback upscale path
- no Real-ESRGAN runtime files required

3. External worker with Real-ESRGAN

- production-oriented path
- uses the `realesrgan-ncnn-vulkan` CLI runtime
- requires runtime files and a verified worker host

Laravel setup
-------------

Minimum env for stub mode:

```env
ENHANCE_ENGINE=stub
ENHANCE_QUEUE=default
```

Recommended env for external worker mode:

```env
ENHANCE_ENGINE=external_worker
ENHANCE_QUEUE=enhance
ENHANCE_WORKER_URL=http://127.0.0.1:8095
ENHANCE_WORKER_TIMEOUT=900
ENHANCE_WORKER_TOKEN=change-this-token
ENHANCE_WORKER_MAX_DOWNLOAD_MB=60
```

Optional Laravel env keys you may tune:

```env
ENHANCE_DISK=public
ENHANCE_SOURCE_PREFIX=enhance/sources
ENHANCE_OUTPUT_PREFIX=enhance/outputs
ENHANCE_PREVIEW_PREFIX=enhance/previews

ENHANCE_MAX_UPLOAD_MB=20
ENHANCE_MAX_INPUT_WIDTH=4096
ENHANCE_MAX_INPUT_HEIGHT=4096
ENHANCE_MAX_OUTPUT_WIDTH=8192
ENHANCE_MAX_OUTPUT_HEIGHT=8192

ENHANCE_DAILY_LIMIT=10

ENHANCE_COMPLETED_EXPIRES_AFTER_DAYS=30
ENHANCE_FAILED_EXPIRES_AFTER_DAYS=7
ENHANCE_DELETED_FILE_GRACE_DAYS=1
ENHANCE_CLEANUP_CHUNK_SIZE=100

ENHANCE_STUCK_PROCESSING_AFTER_MINUTES=30
ENHANCE_STUCK_QUEUED_AFTER_MINUTES=60

ENHANCE_STUB_SHOW_WARNING=true
```

After changing env:

```bash
php artisan config:clear
```

Queue setup
-----------

Enhance jobs run on `ENHANCE_QUEUE`.

If you keep the default queue:

```bash
php artisan queue:work --queue=default
```

If you use a dedicated Enhance queue:

```bash
php artisan queue:work --queue=enhance,default
```

If Horizon or workers do not consume the configured queue, Enhance jobs will stay queued.

Scheduler and cleanup
---------------------

Enhance cleanup is scheduled from [routes/console.php](routes/console.php) and runs:

```bash
php artisan enhance:cleanup --force
```

Useful cleanup and health commands:

```bash
php artisan enhance:health
php artisan enhance:health --json
php artisan enhance:cleanup --dry-run
php artisan enhance:cleanup --force
```

Cleanup only removes files under the configured Enhance prefixes. It does not delete artwork originals or unrelated storage paths.

Enable stub mode
----------------

Use stub mode first if you want to validate the Laravel module without introducing worker runtime variables.

1. Set:

```env
ENHANCE_ENGINE=stub
ENHANCE_QUEUE=default
```

2. Clear config:

```bash
php artisan config:clear
```

3. Start a queue worker:

```bash
php artisan queue:work --queue=default
```

4. Check health:

```bash
php artisan enhance:health
```

5. Open `/enhance/create`, submit a small image, and verify the job completes.

Enable external worker mode with Pillow
---------------------------------------

This is the safest real integration path because it exercises the HTTP worker contract without requiring Real-ESRGAN files.

1. Start the worker:

```bash
cd services/enhance-worker
docker compose -f docker-compose.example.yml up --build
```

2. Confirm worker health:

```bash
curl http://127.0.0.1:8095/health
```

Expected result:

- `status: ok`
- `engine: pillow`

3. Set Laravel env:

```env
ENHANCE_ENGINE=external_worker
ENHANCE_QUEUE=enhance
ENHANCE_WORKER_URL=http://127.0.0.1:8095
ENHANCE_WORKER_TIMEOUT=600
ENHANCE_WORKER_TOKEN=change-this-token
ENHANCE_WORKER_MAX_DOWNLOAD_MB=60
```

4. Clear config and start queue workers:

```bash
php artisan config:clear
php artisan queue:work --queue=enhance,default
```

5. Verify the Laravel side:

```bash
php artisan enhance:health
php artisan test --filter=EnhanceExternalWorker
```

Enable external worker mode with Real-ESRGAN
--------------------------------------------

This is the production-oriented setup.

1. Install or mount the runtime files inside [services/enhance-worker](services/enhance-worker):

```bash
cd services/enhance-worker
bash scripts/download-realesrgan-ncnn.sh
bash scripts/verify-realesrgan.sh
```

Required file locations:

- binary: `bin/realesrgan-ncnn-vulkan`
- models: `models/*.param` and `models/*.bin`

2. Start the Real-ESRGAN worker:

```bash
docker compose -f docker-compose.realesrgan.example.yml up --build
```

3. Confirm worker health:

```bash
curl http://127.0.0.1:8095/health
```

Expected result when ready:

- `status: ok`
- `engine: realesrgan-ncnn`

Expected result when runtime files are missing or invalid:

- `status: degraded`

4. Set Laravel env:

```env
ENHANCE_ENGINE=external_worker
ENHANCE_QUEUE=enhance
ENHANCE_WORKER_URL=http://127.0.0.1:8095
ENHANCE_WORKER_TIMEOUT=900
ENHANCE_WORKER_TOKEN=change-this-token
ENHANCE_WORKER_MAX_DOWNLOAD_MB=60
```

5. Clear config and start the queue worker:

```bash
php artisan config:clear
php artisan queue:work --queue=enhance,default
```

6. Verify through the application by submitting a small image from `/enhance/create`.

Worker configuration
--------------------

Main worker env values:

```env
WORKER_HOST=0.0.0.0
WORKER_PORT=8095
WORKER_TOKEN=change-this-token

WORKER_ENGINE=pillow
WORKER_DEVICE=cpu

WORKER_MAX_UPLOAD_MB=20
WORKER_MAX_INPUT_WIDTH=4096
WORKER_MAX_INPUT_HEIGHT=4096
WORKER_MAX_OUTPUT_WIDTH=8192
WORKER_MAX_OUTPUT_HEIGHT=8192

WORKER_TMP_DIR=/app/storage/tmp
WORKER_OUTPUT_DIR=/app/storage/output
WORKER_RESULT_TTL_MINUTES=60
```

Real-ESRGAN-specific worker env values:

```env
WORKER_REALESRGAN_BIN=/app/bin/realesrgan-ncnn-vulkan
WORKER_REALESRGAN_MODEL_DIR=/app/models
WORKER_REALESRGAN_DEFAULT_MODEL=realesrgan-x4plus
WORKER_REALESRGAN_ANIME_MODEL=realesrgan-x4plus-anime
WORKER_REALESRGAN_TILE=0
WORKER_REALESRGAN_TTA=false
WORKER_REALESRGAN_VERBOSE=false
WORKER_REALESRGAN_TIMEOUT_SECONDS=900
WORKER_REALESRGAN_PREPROCESS_MAX_PIXELS=16777216
WORKER_REALESRGAN_OUTPUT_EXT=webp
WORKER_REALESRGAN_ALLOW_MODEL_FALLBACK=true
```

Compatibility values kept by the worker:

```env
WORKER_MODEL_DIR=/app/app/models
WORKER_DEFAULT_MODEL=realesrgan-x4plus
```

Worker behavior
---------------

Worker request contract:

- endpoint: `POST /v1/upscale`
- bearer auth required
- accepted output formats: `webp`, `png`, `jpg`
- allowed scales: `2`, `4`
- allowed modes: `standard`, `artwork`, `photo`, `illustration`

Health and temp file endpoints:

- `GET /health`
- `GET /v1/results/{filename}`
- `DELETE /v1/results/{filename}`

Mode and scale behavior
-----------------------

Real-ESRGAN mode mapping:

- `standard` -> default model
- `artwork` -> default model
- `photo` -> default model
- `illustration` -> anime model when available

Fallback behavior:

- if the requested model exists, it is used
- if it is missing and fallback is enabled, the default model is used
- if it is missing and fallback is disabled, processing fails safely

Scale behavior:

- `4x` returns native 4x output
- `2x` currently runs the 4x model and then downsamples to 2x

Storage and data flow
---------------------

Laravel stores Enhance files under the configured prefixes:

- sources under `enhance/sources`
- outputs under `enhance/outputs`
- previews under `enhance/previews`

The worker does not permanently store Enhance results.

When `ENHANCE_ENGINE=external_worker`:

1. Laravel prepares a copied Enhance source file.
2. Laravel sends the worker a temporary URL or a temporary signed internal route.
3. The worker downloads the source file.
4. The worker processes the image.
5. The worker exposes a temporary result URL.
6. Laravel downloads, validates, and stores the final output.
7. Laravel instructs the worker to delete the temporary result.

Verification checklist
----------------------

Laravel verification:

```bash
php artisan config:clear
php artisan enhance:health
php artisan enhance:health --json
```

Worker verification:

```bash
curl http://127.0.0.1:8095/health
```

Queue verification:

```bash
php artisan queue:work --queue=enhance,default
```

Application verification:

1. Open `/enhance/create`.
2. Upload or choose a small source image.
3. Select `2x` or `4x` and a valid mode.
4. Submit the job.
5. Confirm the job transitions from queued to completed.
6. Confirm output exists on the Enhance disk.
7. Confirm the source file and original artwork remain untouched.

Health states
-------------

Laravel health command helps identify:

- configured engine
- queue usage
- stuck jobs
- lifecycle status

Worker `/health` helps identify:

- current worker engine
- maximum input and output limits
- Real-ESRGAN binary availability
- Real-ESRGAN model directory readiness
- available Real-ESRGAN models

If the worker is in Real-ESRGAN mode and health returns `degraded`, do not treat the runtime as production-ready yet.

Troubleshooting
---------------

`Worker URL is missing.`

- set `ENHANCE_WORKER_URL`
- clear config

`Worker token is missing.`

- set `ENHANCE_WORKER_TOKEN`
- make sure the worker uses the same `WORKER_TOKEN`

`Worker is unavailable.`

- confirm the worker is reachable
- confirm the worker container is running
- confirm the URL points to the worker base URL

`Upscale engine is not available. Check model files and worker installation.`

- confirm `WORKER_ENGINE=realesrgan-ncnn`
- confirm `bin/realesrgan-ncnn-vulkan` exists and is executable
- confirm the required `.param` and `.bin` model files exist
- run `bash scripts/verify-realesrgan.sh`

Jobs stay queued

- confirm queue workers consume `ENHANCE_QUEUE`
- if using `enhance`, run workers with `--queue=enhance,default`

`status: degraded` from worker health

- verify binary and model directory paths
- verify runtime files are mounted inside the container
- fall back to `WORKER_ENGINE=pillow` until the runtime is fixed

Operations and safety notes
---------------------------

- Keep the worker bound to `127.0.0.1` or a private container network.
- Do not expose the worker publicly.
- Use a strong shared token.
- Keep Enhance on a dedicated queue when load increases.
- Keep cleanup enabled so stale outputs and failed files do not accumulate.
- Do not commit Real-ESRGAN binary or model weight files unless explicitly approved.
- The worker only serves generated files from its own output directory.
- The worker rejects unsupported source URLs and unsafe output paths.
- Original artwork files are never replaced by the Enhance flow.

Production rollout recommendation
---------------------------------

Recommended rollout sequence:

1. Enable `ENHANCE_ENGINE=stub` and verify the Laravel workflow.
2. Move to `ENHANCE_ENGINE=external_worker` with `WORKER_ENGINE=pillow`.
3. Verify queue, storage, cleanup, and health behavior.
4. Install Real-ESRGAN runtime files and switch the worker to `WORKER_ENGINE=realesrgan-ncnn`.
5. Confirm worker health is `ok` before calling the runtime production-ready.

Related docs
------------

- operational notes: [docs/enhance.md](docs/enhance.md)
- worker runtime docs: [services/enhance-worker/README.md](services/enhance-worker/README.md)