Adds entries for transfer admin, account deletion, role reset and channel access, message deletes, pins and protection by moderators, channel settings, voice and channel removals, webhooks, whitelist, codes and tokens, emoji, stickers, sounds, tags, backups and restore, imports, updates and admin recovery. Settings record their previous value.
* Stamp deleted attachments with their deletion time on every delete path
The socket handlers carried their own copies of moveUploadToDeleted that
skipped the utimes stamp, so an old attachment deleted from chat kept its
upload time and was purged at the next 15 minute sweep instead of after the
retention window. All callers now share the one implementation in
src/uploadFiles.js.
* Route every upload move and delete through a storage layer
src/storage holds the driver for uploads/ (local disk for now) and keeps a
new media_objects index in step with each write, so a bucket driver can sit
behind the same calls later. Soft deletes, permanent deletes, purges and
restores all go through it; src/uploadFiles.js is gone.
Storage calls are asynchronous, and every caller now awaits them before it
answers or emits: message, bulk, channel, sub-channel, DM and group DM
deletes, kicks from a group, bot deletes, auto-cleanup, the encrypted file
release, the uploads trim, self-destruct, and the sound, emoji, sticker and
badge icon deletes. Self-destruct sweeps no longer overlap when a timer fires
mid-sweep, and stop() keeps a running sweep from re-arming the timer.
The storage layer accepts any path that stays inside uploads/, so names that
keep an uploader's unusual extension still move and delete as before.
* Serve /uploads from the storage layer instead of express.static
src/storage/serve.js reads each file through the storage driver and streams
it out, so the same safety headers will hold for files kept in a bucket. It
answers as express.static did: the same weak ETag, Last-Modified, the 7 day
immutable cache, conditional GETs and 304s, a single byte range with 206,
HEAD, the dotfile refusal, and falling through to the app's 404, with a
failed precondition or an unsatisfiable range still handed to the error
handler. The guard that refuses deleted-attachments/ and bot-audio/ in every
encoding moved into the same module unchanged.
test/uploadsServe.test.js serves one folder through the old express.static
setup and the new handler side by side and compares status, headers and
body for each file type, conditional request, range and refused path.
One difference: a folder path without a trailing slash (/uploads/stickers)
now gets the 404 directly instead of a 301 to the same 404.
* Admit uploads to storage only after their route has checked them
A cleaned upload now waits in unprocessed/staging/ (never served) while its
route checks the magic bytes and fixes the extension, and is then admitted
to storage with storage.admitUpload before anything stores its URL. A
failed admit answers 500 and leaves nothing behind; a restart sweeps any
staged file a cut-short request left.
Covers attachments and files, avatars, borders, bot and persona avatars,
sounds, emoji and stickers (single and bulk), badge icons, the server icon
and banner, role icons, sign-in avatars and the starter sticker pack. Bulk
emoji and sticker uploads report a file that could not be stored as an error
and carry on with the rest. A persona avatar refused as someone else's
persona is now removed instead of left in uploads.
* Read upload sizes from the media index instead of walking the folder
The moderator usage report, the uploads trim, the Files & Media sizes, the
check for files a bot delete may move, and the deleted-attachments purge now
read media_objects rather than stat'ing or walking uploads/, so none of them
needs a listing of the bucket once storage can be one. The report's numbers
are unchanged; the trim still counts the temporary bot audio kept on disk,
as before. The purge counts the retention window from the deletion time the
index records, and the local driver still prunes the sub-folders it empties.
The server only starts listening once storage.prepare has checked storage
answers; a failure exits with a message rather than starting a server that
cannot serve media. On the first start after this change it builds the index
from what is already in uploads/ (src/storage/reconcile.js), once.
scripts/reconcile-media.js runs the same repair by hand. The privacy sweep
keeps the index in step when it rewrites, holds back or returns a file.
* Add an S3-compatible storage driver for uploads
HAVEN_EXTERNAL_STORAGE=true keeps media in a bucket (HAVEN_S3_ENDPOINT,
_BUCKET, _REGION, _ACCESS_KEY, _SECRET_KEY, _PREFIX, _FORCE_PATH_STYLE)
through src/storage/s3.js, behind the same calls as the local driver: put
(multipart over 5 MB), head, ranged reads, soft delete and restore by copy
and delete, and paged listing. Haven still streams every file itself, so the
safety headers are unchanged. Missing settings or a bucket that does not
answer stop the server at boot with the endpoint, bucket and reason.
HAVEN_S3_LOCAL_FALLBACK=true serves files not yet moved from uploads/ on
disk; deleting then removes the local copy too, so it cannot come back.
Without it, files left on disk are counted in a warning at boot.
Checksums are only sent and checked where the API needs them: the SDK's
default CRC32 is rejected by stores such as Garage for files sent in parts.
test/storageS3.integration.test.js runs against a real bucket when
HAVEN_TEST_S3_ENDPOINT is set (passes on Garage 1.1.0) and is skipped
otherwise. Not documented for operators yet: Ferry, badges, templates and
backups still read uploads/ directly.
* Clean photos in the bucket, release replaced profile pictures, reconcile on demand
The privacy sweep has a pass of its own for media kept in a bucket: it goes
by the media index, cleans each photo and video in unprocessed/ and writes
it back only when cleaning changed it, and holds back a file it cannot clean
the same way it does on disk. Its progress is kept per driver, so moving to
a bucket checks everything there once.
A replaced or removed avatar or border, and a persona's avatar when it is
replaced or the persona is deleted, now go to deleted-attachments instead of
staying in uploads forever, but only when the file is that person's own
upload and nothing else still shows it. A persona can point at anyone's
upload, so the ownership check is what keeps one person from taking another
person's file. Border uploads now record their owner as avatars do.
Admins can rebuild the media index from storage with
POST /api/admin/media/reconcile, the same repair as
scripts/reconcile-media.js.
* Read uploads through storage everywhere, and move servers to a bucket
The last readers of uploads/ on disk now go through src/storage:
- Badge icons are keyed from storage (imageIconKey); built-in and emoji keys
stay synchronous.
- Ferry checks Discord's size limits against the media index, then reads each
file's bytes once, so a retry no longer reads the files again.
- Server templates read and write images through storage. exportTemplate and
applyTemplate are now async; images still go in before the database
changes and are removed if those fail. tools/template.js uses the target
server's own storage, bucket included.
- Backups with files stream them out of the bucket one at a time. A restore
marks the restored database so the next start rebuilds the media index,
and on a bucket first copies the restored files into it
(media_restore_pending), so the restore no longer has to send gigabytes
in the moment before it restarts.
scripts/migrate-uploads-to-s3.js moves an existing server's files into the
bucket while it stays up (with HAVEN_S3_LOCAL_FALLBACK=true), skips what is
already there, and with --delete-local frees the disk only once the bucket
holds everything. GUIDE.md, docker-compose.yml and .env.example now document
external storage.
Tested against Garage 1.1.0: a live move from disk to the bucket with
nothing going missing, and a backup taken on the bucket restored back into it.
* Let the media scripts read the server's .env
migrate-uploads-to-s3.js and reconcile-media.js only saw HAVEN_S3_* when
they were set in the shell, so on an install that keeps its settings in the
data folder's .env, as a non-Docker install does, the migration refused to
run. They now load that .env the way server.js does; anything set in the
shell still wins.
* More documentation in .env.example
* Remove location data from every uploaded photo and video
Photos and videos can carry the GPS coordinates of where they were
taken, and Haven kept whatever arrived: anyone who could see such a
file could find where it was taken. Not every upload has one. Android
zeroes the location for apps without its media location permission,
and a photo taken before the phone knows where it is has none. But
files uploaded from a computer, or through an app that has that
permission, keep their coordinates, and so may photos from an iPhone
(not tested). Every photo and video now has its location removed
before anyone else can see it, and the ones already on the server are
cleaned once. There is no setting to turn this off.
New dependencies: exiftool and ffmpeg
Both are now required. They are tried and true, kept up to date as
phones and cameras change their formats, and thorough, which is the
whole job here. exiftool knows every place a device hides a location
(EXIF, XMP, IPTC, maker notes, Apple and Android video tags, the
location tracks dashcams and action cameras record) and can read and
write all of them. ffmpeg rewrites a video keeping only its picture
and sound.
Haven looks for each in HAVEN_EXIFTOOL_PATH / HAVEN_FFMPEG_PATH, then
the data folder's tools/, then the PATH. When one is missing it
downloads a pinned version from GitHub into tools/ and checks it
against a fixed SHA-256 before using it: ExifTool 13.59 from its
author's repository (byte for byte the exiftool.org release; it needs
Perl, which Linux and macOS ship), ffmpeg from the ffmpeg-static
b6.1.1 release. ExifTool's Windows build is not on GitHub, so Windows
installs it by hand. HAVEN_NO_TOOL_DOWNLOAD=1 turns downloading off.
The Docker image installs both from Debian.
What happens to a file
1. exiftool reads it. The type comes from the contents, not the name.
Anything that is not a photo or video is left alone (PDFs, audio,
voice messages, encrypted DM files).
2. Nothing to remove (no location, XMP or IPTC) means no rewrite.
3. Photos: exiftool deletes the GPS tags, every location tag, all XMP
and all IPTC, without re-encoding, so pixels and rotation are
untouched. If anything survives, a second pass keeps only rotation
and the colour profile.
Videos: ffmpeg copies only the video and audio streams into a new
file with no metadata, without re-encoding. Dropping every other
stream is what removes the separate location tracks iPhones and
action cameras add.
4. exiftool reads the result again, embedded tracks included. Any
location left means the file is refused, never let through.
Files are rewritten under a temporary name and swapped in with a
rename, so a crash leaves the old file or the clean one, never both.
A rewrite keeps the file's original date, so retention and cleanup
still count from when it was uploaded.
New uploads
Uploads land in data/unprocessed/, which is not served, and only move
into uploads/ once clean, so a dirty file is never reachable, not
even for a moment. This covers every upload route: chat files and
pictures, avatars and borders, server icons and banners, role and
badge icons, emoji, stickers, and avatars fetched from a sign-in
provider.
- A photo or video whose extension lies (image.bin) gets its real
one, on disk and in the name shown in chat, so it displays inline.
- The progress bar fills once for the upload, then again as
"Removing location data..." following ffmpeg's progress on videos.
Photos take a moment, so they only change the label. Progress
reaches the uploader over the socket, so the upload endpoint and
the clients that use it are unchanged.
- Cancelling during that step stops the tool and discards the file.
- New uploads always go first: the sweep waits while one is cleaned.
At most two videos are cleaned at once, and the tools run at the
lowest CPU priority so chat and voice come first.
- A file that cannot be cleaned is refused, deleted, logged to the
console, and the uploader is told why:
"Your file contains GPS data that Haven failed to remove. Please
try again or remove the GPS data manually" (location still there)
"Haven couldn't process this file. Please try again or upload a
different file." (the tool itself failed)
- Without the tools, photo and video uploads are refused with "Photo/
Videos uploads are disabled due to a misconfiguration of the server";
chat uploads check this before sending anything. Other files still
upload. Startup logs the problem, and installing the tools reopens
uploads within five minutes, without a restart.
Files already on the server
A one-time sweep at startup cleans every photo and video in uploads/,
deleted-attachments included (those can be restored), where it is.
Old files keep being shown as normal while it runs, and it resumes
where it left off after a restart. A file it cannot clean is moved to
unprocessed/held/, out of reach, and retried after the sweep, up to
three more times, one file at a time. After that it is listed in
unprocessed/held/failed.json with the tool's error and the messages
that use it. Held files are never deleted automatically: a retry that
works puts the file back, otherwise an admin decides.
In chat, a held file is taken out of its message and the message gets
one note under whatever still shows, starting with a lock:
"This file is still being processed." while retries remain,
"This file couldn't be processed." after,
and counts when more than one file is held ("1 file is still being
processed. 2 files couldn't be processed."). The message text stays.
A picture group re-lays itself out around the files that remain; a
single picture left shows like a lone picture, a single video keeps
its tile so it still plays.
What this cannot reach
- Copies people already downloaded, copies Ferry already sent to
Discord, and the server's backups.
- Encrypted DMs: their files reach the server already encrypted. Each
DM now shows a dismissable notice saying so and asking people to
turn off location on their camera and check what they send.
Testing
Check a photo really has a location before uploading it:
exiftool -a -G1 -gps:all -c "%.6f" photo.jpg
Real coordinates show as [GPS] GPS Latitude / GPS Longitude lines.
Without -a, exiftool hides them: it keeps only its own calculated
copy of each, which -gps:all then filters out, so a photo with a
location looks like it has none.
Where uploads come from matters on Android. Since Android 10, an app
reads photos with the location zeroed out unless it declares
ACCESS_MEDIA_LOCATION, shown in its permission list as "read
locations from your media collection".
- Haven Mobile asks for photo and video access but not for that
permission, so on Android 10 and later every photo it sends
arrives with its GPS already zeroed by Android. Videos were not
tested but are assumed to be the same case.
- Firefox for Android does ask for it, and a photo uploaded through
Firefox was confirmed to reach Haven with its GPS intact. Test
from a browser like that, or from a computer.
- iPhone was not tested.
Then upload it, open the stored file in data/uploads/ (or download
it from chat), and run the same command: no location should come
back.
To see a held file, cut a phone photo that has GPS short, inside its
metadata (head -c 2000 usually is), and drop it in uploads/ before a
first start: exiftool can still read the location but cannot rewrite
the broken file, so it is held, retried and listed in failed.json.
test/privacyPass.test.js covers detection, renaming, and cleaning a
photo and a video. The cleaning tests and the image upload tests need
exiftool and ffmpeg installed; without them they skip or expect the
refusal, and test servers never download the tools.
Translations added for German, Persian, Japanese, Korean, Portuguese
and Chinese; Spanish, French, Polish and Russian get the progress label
(their other strings in these sections are still English).
* Leave Persian out of the location data strings
The translations validator fails any change to fa.json, which has no
_meta.flag, and choosing a flag for Persian is not this change's call.
Persian shows the new privacy pass text in English until that is
settled.
* Bump cache tags above main
---------
Co-authored-by: bo0sted <mc526+github@proton.me>
Co-authored-by: ancsemi <ancsemi@gmail.com>
Amnibro