mirror of
https://github.com/ancsemi/Haven.git
synced 2026-10-28 01:26:14 +00:00
5f8caf6bf8
* Remove location data from every uploaded photo and video
Photos and videos can carry the GPS coordinates of where they were
taken, and Haven kept whatever arrived: anyone who could see such a
file could find where it was taken. Not every upload has one. Android
zeroes the location for apps without its media location permission,
and a photo taken before the phone knows where it is has none. But
files uploaded from a computer, or through an app that has that
permission, keep their coordinates, and so may photos from an iPhone
(not tested). Every photo and video now has its location removed
before anyone else can see it, and the ones already on the server are
cleaned once. There is no setting to turn this off.
New dependencies: exiftool and ffmpeg
Both are now required. They are tried and true, kept up to date as
phones and cameras change their formats, and thorough, which is the
whole job here. exiftool knows every place a device hides a location
(EXIF, XMP, IPTC, maker notes, Apple and Android video tags, the
location tracks dashcams and action cameras record) and can read and
write all of them. ffmpeg rewrites a video keeping only its picture
and sound.
Haven looks for each in HAVEN_EXIFTOOL_PATH / HAVEN_FFMPEG_PATH, then
the data folder's tools/, then the PATH. When one is missing it
downloads a pinned version from GitHub into tools/ and checks it
against a fixed SHA-256 before using it: ExifTool 13.59 from its
author's repository (byte for byte the exiftool.org release; it needs
Perl, which Linux and macOS ship), ffmpeg from the ffmpeg-static
b6.1.1 release. ExifTool's Windows build is not on GitHub, so Windows
installs it by hand. HAVEN_NO_TOOL_DOWNLOAD=1 turns downloading off.
The Docker image installs both from Debian.
What happens to a file
1. exiftool reads it. The type comes from the contents, not the name.
Anything that is not a photo or video is left alone (PDFs, audio,
voice messages, encrypted DM files).
2. Nothing to remove (no location, XMP or IPTC) means no rewrite.
3. Photos: exiftool deletes the GPS tags, every location tag, all XMP
and all IPTC, without re-encoding, so pixels and rotation are
untouched. If anything survives, a second pass keeps only rotation
and the colour profile.
Videos: ffmpeg copies only the video and audio streams into a new
file with no metadata, without re-encoding. Dropping every other
stream is what removes the separate location tracks iPhones and
action cameras add.
4. exiftool reads the result again, embedded tracks included. Any
location left means the file is refused, never let through.
Files are rewritten under a temporary name and swapped in with a
rename, so a crash leaves the old file or the clean one, never both.
A rewrite keeps the file's original date, so retention and cleanup
still count from when it was uploaded.
New uploads
Uploads land in data/unprocessed/, which is not served, and only move
into uploads/ once clean, so a dirty file is never reachable, not
even for a moment. This covers every upload route: chat files and
pictures, avatars and borders, server icons and banners, role and
badge icons, emoji, stickers, and avatars fetched from a sign-in
provider.
- A photo or video whose extension lies (image.bin) gets its real
one, on disk and in the name shown in chat, so it displays inline.
- The progress bar fills once for the upload, then again as
"Removing location data..." following ffmpeg's progress on videos.
Photos take a moment, so they only change the label. Progress
reaches the uploader over the socket, so the upload endpoint and
the clients that use it are unchanged.
- Cancelling during that step stops the tool and discards the file.
- New uploads always go first: the sweep waits while one is cleaned.
At most two videos are cleaned at once, and the tools run at the
lowest CPU priority so chat and voice come first.
- A file that cannot be cleaned is refused, deleted, logged to the
console, and the uploader is told why:
"Your file contains GPS data that Haven failed to remove. Please
try again or remove the GPS data manually" (location still there)
"Haven couldn't process this file. Please try again or upload a
different file." (the tool itself failed)
- Without the tools, photo and video uploads are refused with "Photo/
Videos uploads are disabled due to a misconfiguration of the server";
chat uploads check this before sending anything. Other files still
upload. Startup logs the problem, and installing the tools reopens
uploads within five minutes, without a restart.
Files already on the server
A one-time sweep at startup cleans every photo and video in uploads/,
deleted-attachments included (those can be restored), where it is.
Old files keep being shown as normal while it runs, and it resumes
where it left off after a restart. A file it cannot clean is moved to
unprocessed/held/, out of reach, and retried after the sweep, up to
three more times, one file at a time. After that it is listed in
unprocessed/held/failed.json with the tool's error and the messages
that use it. Held files are never deleted automatically: a retry that
works puts the file back, otherwise an admin decides.
In chat, a held file is taken out of its message and the message gets
one note under whatever still shows, starting with a lock:
"This file is still being processed." while retries remain,
"This file couldn't be processed." after,
and counts when more than one file is held ("1 file is still being
processed. 2 files couldn't be processed."). The message text stays.
A picture group re-lays itself out around the files that remain; a
single picture left shows like a lone picture, a single video keeps
its tile so it still plays.
What this cannot reach
- Copies people already downloaded, copies Ferry already sent to
Discord, and the server's backups.
- Encrypted DMs: their files reach the server already encrypted. Each
DM now shows a dismissable notice saying so and asking people to
turn off location on their camera and check what they send.
Testing
Check a photo really has a location before uploading it:
exiftool -a -G1 -gps:all -c "%.6f" photo.jpg
Real coordinates show as [GPS] GPS Latitude / GPS Longitude lines.
Without -a, exiftool hides them: it keeps only its own calculated
copy of each, which -gps:all then filters out, so a photo with a
location looks like it has none.
Where uploads come from matters on Android. Since Android 10, an app
reads photos with the location zeroed out unless it declares
ACCESS_MEDIA_LOCATION, shown in its permission list as "read
locations from your media collection".
- Haven Mobile asks for photo and video access but not for that
permission, so on Android 10 and later every photo it sends
arrives with its GPS already zeroed by Android. Videos were not
tested but are assumed to be the same case.
- Firefox for Android does ask for it, and a photo uploaded through
Firefox was confirmed to reach Haven with its GPS intact. Test
from a browser like that, or from a computer.
- iPhone was not tested.
Then upload it, open the stored file in data/uploads/ (or download
it from chat), and run the same command: no location should come
back.
To see a held file, cut a phone photo that has GPS short, inside its
metadata (head -c 2000 usually is), and drop it in uploads/ before a
first start: exiftool can still read the location but cannot rewrite
the broken file, so it is held, retried and listed in failed.json.
test/privacyPass.test.js covers detection, renaming, and cleaning a
photo and a video. The cleaning tests and the image upload tests need
exiftool and ffmpeg installed; without them they skip or expect the
refusal, and test servers never download the tools.
Translations added for German, Persian, Japanese, Korean, Portuguese
and Chinese; Spanish, French, Polish and Russian get the progress label
(their other strings in these sections are still English).
* Leave Persian out of the location data strings
The translations validator fails any change to fa.json, which has no
_meta.flag, and choosing a flag for Persian is not this change's call.
Persian shows the new privacy pass text in English until that is
settled.
* Bump cache tags above main
---------
Co-authored-by: bo0sted <mc526+github@proton.me>
Co-authored-by: ancsemi <ancsemi@gmail.com>