mirror of
https://github.com/ancsemi/Haven.git
synced 2026-10-25 16:20:42 +00:00
Kamden Bolhuis
6872049e71
* Stamp deleted attachments with their deletion time on every delete path The socket handlers carried their own copies of moveUploadToDeleted that skipped the utimes stamp, so an old attachment deleted from chat kept its upload time and was purged at the next 15 minute sweep instead of after the retention window. All callers now share the one implementation in src/uploadFiles.js. * Route every upload move and delete through a storage layer src/storage holds the driver for uploads/ (local disk for now) and keeps a new media_objects index in step with each write, so a bucket driver can sit behind the same calls later. Soft deletes, permanent deletes, purges and restores all go through it; src/uploadFiles.js is gone. Storage calls are asynchronous, and every caller now awaits them before it answers or emits: message, bulk, channel, sub-channel, DM and group DM deletes, kicks from a group, bot deletes, auto-cleanup, the encrypted file release, the uploads trim, self-destruct, and the sound, emoji, sticker and badge icon deletes. Self-destruct sweeps no longer overlap when a timer fires mid-sweep, and stop() keeps a running sweep from re-arming the timer. The storage layer accepts any path that stays inside uploads/, so names that keep an uploader's unusual extension still move and delete as before. * Serve /uploads from the storage layer instead of express.static src/storage/serve.js reads each file through the storage driver and streams it out, so the same safety headers will hold for files kept in a bucket. It answers as express.static did: the same weak ETag, Last-Modified, the 7 day immutable cache, conditional GETs and 304s, a single byte range with 206, HEAD, the dotfile refusal, and falling through to the app's 404, with a failed precondition or an unsatisfiable range still handed to the error handler. The guard that refuses deleted-attachments/ and bot-audio/ in every encoding moved into the same module unchanged. test/uploadsServe.test.js serves one folder through the old express.static setup and the new handler side by side and compares status, headers and body for each file type, conditional request, range and refused path. One difference: a folder path without a trailing slash (/uploads/stickers) now gets the 404 directly instead of a 301 to the same 404. * Admit uploads to storage only after their route has checked them A cleaned upload now waits in unprocessed/staging/ (never served) while its route checks the magic bytes and fixes the extension, and is then admitted to storage with storage.admitUpload before anything stores its URL. A failed admit answers 500 and leaves nothing behind; a restart sweeps any staged file a cut-short request left. Covers attachments and files, avatars, borders, bot and persona avatars, sounds, emoji and stickers (single and bulk), badge icons, the server icon and banner, role icons, sign-in avatars and the starter sticker pack. Bulk emoji and sticker uploads report a file that could not be stored as an error and carry on with the rest. A persona avatar refused as someone else's persona is now removed instead of left in uploads. * Read upload sizes from the media index instead of walking the folder The moderator usage report, the uploads trim, the Files & Media sizes, the check for files a bot delete may move, and the deleted-attachments purge now read media_objects rather than stat'ing or walking uploads/, so none of them needs a listing of the bucket once storage can be one. The report's numbers are unchanged; the trim still counts the temporary bot audio kept on disk, as before. The purge counts the retention window from the deletion time the index records, and the local driver still prunes the sub-folders it empties. The server only starts listening once storage.prepare has checked storage answers; a failure exits with a message rather than starting a server that cannot serve media. On the first start after this change it builds the index from what is already in uploads/ (src/storage/reconcile.js), once. scripts/reconcile-media.js runs the same repair by hand. The privacy sweep keeps the index in step when it rewrites, holds back or returns a file. * Add an S3-compatible storage driver for uploads HAVEN_EXTERNAL_STORAGE=true keeps media in a bucket (HAVEN_S3_ENDPOINT, _BUCKET, _REGION, _ACCESS_KEY, _SECRET_KEY, _PREFIX, _FORCE_PATH_STYLE) through src/storage/s3.js, behind the same calls as the local driver: put (multipart over 5 MB), head, ranged reads, soft delete and restore by copy and delete, and paged listing. Haven still streams every file itself, so the safety headers are unchanged. Missing settings or a bucket that does not answer stop the server at boot with the endpoint, bucket and reason. HAVEN_S3_LOCAL_FALLBACK=true serves files not yet moved from uploads/ on disk; deleting then removes the local copy too, so it cannot come back. Without it, files left on disk are counted in a warning at boot. Checksums are only sent and checked where the API needs them: the SDK's default CRC32 is rejected by stores such as Garage for files sent in parts. test/storageS3.integration.test.js runs against a real bucket when HAVEN_TEST_S3_ENDPOINT is set (passes on Garage 1.1.0) and is skipped otherwise. Not documented for operators yet: Ferry, badges, templates and backups still read uploads/ directly. * Clean photos in the bucket, release replaced profile pictures, reconcile on demand The privacy sweep has a pass of its own for media kept in a bucket: it goes by the media index, cleans each photo and video in unprocessed/ and writes it back only when cleaning changed it, and holds back a file it cannot clean the same way it does on disk. Its progress is kept per driver, so moving to a bucket checks everything there once. A replaced or removed avatar or border, and a persona's avatar when it is replaced or the persona is deleted, now go to deleted-attachments instead of staying in uploads forever, but only when the file is that person's own upload and nothing else still shows it. A persona can point at anyone's upload, so the ownership check is what keeps one person from taking another person's file. Border uploads now record their owner as avatars do. Admins can rebuild the media index from storage with POST /api/admin/media/reconcile, the same repair as scripts/reconcile-media.js. * Read uploads through storage everywhere, and move servers to a bucket The last readers of uploads/ on disk now go through src/storage: - Badge icons are keyed from storage (imageIconKey); built-in and emoji keys stay synchronous. - Ferry checks Discord's size limits against the media index, then reads each file's bytes once, so a retry no longer reads the files again. - Server templates read and write images through storage. exportTemplate and applyTemplate are now async; images still go in before the database changes and are removed if those fail. tools/template.js uses the target server's own storage, bucket included. - Backups with files stream them out of the bucket one at a time. A restore marks the restored database so the next start rebuilds the media index, and on a bucket first copies the restored files into it (media_restore_pending), so the restore no longer has to send gigabytes in the moment before it restarts. scripts/migrate-uploads-to-s3.js moves an existing server's files into the bucket while it stays up (with HAVEN_S3_LOCAL_FALLBACK=true), skips what is already there, and with --delete-local frees the disk only once the bucket holds everything. GUIDE.md, docker-compose.yml and .env.example now document external storage. Tested against Garage 1.1.0: a live move from disk to the bucket with nothing going missing, and a backup taken on the bucket restored back into it. * Let the media scripts read the server's .env migrate-uploads-to-s3.js and reconcile-media.js only saw HAVEN_S3_* when they were set in the shell, so on an install that keeps its settings in the data folder's .env, as a non-Docker install does, the migration refused to run. They now load that .env the way server.js does; anything set in the shell still wins. * More documentation in .env.example