This commit is contained in:
zand
2026-09-08 17:18:05 +02:00
commit 364544d77e
11 changed files with 704 additions and 0 deletions
+21
View File
@@ -0,0 +1,21 @@
MIT License
Copyright (c) 2026
Permission is hereby granted, free of charge, to any person obtaining a copy
of this software and associated documentation files (the "Software"), to deal
in the Software without restriction, including without limitation the rights
to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
copies of the Software, and to permit persons to whom the Software is
furnished to do so, subject to the following conditions:
The above copyright notice and this permission notice shall be included in all
copies or substantial portions of the Software.
THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
SOFTWARE.
+193
View File
@@ -0,0 +1,193 @@
# Mastodon Email Blocker
Small maintenance tool for Mastodon instances that keeps the local `EmailDomainBlock` table in sync with public disposable-email domain lists.
It supports two existing Mastodon layouts:
- a normal source installation where `bin/rails` and `bin/tootctl` are available on the host;
- an existing Docker Compose installation, where the commands are executed inside the Mastodon application container with `docker compose exec`.
The tool does not use the Mastodon Admin API, so importing a large list does not generate one HTTP request per domain and does not run into API rate limits.
## What it changes
The blocklists are downloaded, normalized and merged. Domains already present in Mastodon are skipped; missing domains are added with Mastodon's own `tootctl email-domain-blocks add` command.
By default the tool also turns existing `allow_with_approval` email-domain records into hard deny records. Use `--no-force-deny` if that is not wanted.
MX records are never blocked automatically. `--mx-mode report` can be used for occasional DNS analysis, but the result is informational only. Shared mail infrastructure is common and blocking an MX hostname can affect unrelated, legitimate domains.
## Sources
The default set is built from:
- mehrtat/disposable-email-domain
- FFraud-com/disposable-email-domains
- disposable/disposable-email-domains
If a source is temporarily unavailable, the other successful sources are still used. If every source fails, the run stops before changing Mastodon.
## Requirements
- Python 3.10 or newer
- access to the Mastodon administration commands
- `dnspython` only when MX reporting is enabled
Create a virtual environment:
```sh
python3 -m venv /opt/mastodon-email-blocker/venv
/opt/mastodon-email-blocker/venv/bin/pip install -r requirements.txt
```
Install the script:
```sh
install -d /opt/mastodon-email-blocker
install -m 0755 bin/mastodon-email-blocker.py /opt/mastodon-email-blocker/mastodon-email-blocker.py
install -d /etc/mastodon-email-blocker /var/lib/mastodon-email-blocker
install -m 0644 config/allowlist.example.txt /etc/mastodon-email-blocker/allowlist.txt
```
The account running the tool must be able to run the Mastodon commands and write to `/var/lib/mastodon-email-blocker`.
## Source installation
Run a dry run first:
```sh
/opt/mastodon-email-blocker/venv/bin/python \
/opt/mastodon-email-blocker/mastodon-email-blocker.py \
--backend native \
--mastodon-dir /path/to/mastodon/live \
--ruby-bin /path/to/ruby/bin \
--mx-mode off \
--dry-run
```
`--ruby-bin` can be omitted when the correct Ruby is already in `PATH`.
Apply the changes:
```sh
/opt/mastodon-email-blocker/venv/bin/python \
/opt/mastodon-email-blocker/mastodon-email-blocker.py \
--backend native \
--mastodon-dir /path/to/mastodon/live \
--ruby-bin /path/to/ruby/bin \
--mx-mode off
```
Do not source Mastodon's `.env.production` from this script. Rails should load the application's production configuration in the same way it does for normal Mastodon commands.
## Existing Docker Compose installation
Nothing is installed inside the Mastodon image. The Python tool runs on the Docker host and invokes Rails/tootctl in the existing application service.
For the standard Mastodon Compose layout the application service is normally named `web`. If the local Compose file uses another service name, pass it with `--compose-service`.
Dry run:
```sh
/opt/mastodon-email-blocker/venv/bin/python \
/opt/mastodon-email-blocker/mastodon-email-blocker.py \
--backend docker \
--compose-dir /path/to/mastodon-compose \
--compose-service web \
--mx-mode off \
--dry-run
```
Apply the changes:
```sh
/opt/mastodon-email-blocker/venv/bin/python \
/opt/mastodon-email-blocker/mastodon-email-blocker.py \
--backend docker \
--compose-dir /path/to/mastodon-compose \
--compose-service web \
--mx-mode off
```
For installations still using the old standalone binary, use:
```sh
--compose-command "docker-compose"
```
The Docker user running the tool must have permission to use Docker. No database password, Mastodon secret, `.env.production` file or API token is read by the Python program; the existing application container already has its normal environment.
## Allowlist
Put exceptions in `/etc/mastodon-email-blocker/allowlist.txt`, one domain per line:
```text
example.org
mail.example.net
```
The allowlist only prevents the tool from adding those domains. It does not delete a block that already exists in Mastodon.
## MX report
DNS/MX analysis is intentionally disabled for scheduled runs because resolving a large domain set is slow and MX providers are often shared.
Run it manually when useful:
```sh
/opt/mastodon-email-blocker/venv/bin/python \
/opt/mastodon-email-blocker/mastodon-email-blocker.py \
--backend native \
--mastodon-dir /path/to/mastodon/live \
--mx-mode report
```
or, for Compose:
```sh
/opt/mastodon-email-blocker/venv/bin/python \
/opt/mastodon-email-blocker/mastodon-email-blocker.py \
--backend docker \
--compose-dir /path/to/mastodon-compose \
--compose-service web \
--mx-mode report
```
The report is written to `/var/lib/mastodon-email-blocker/last-report.json`. MX entries in that file have `REPORT_ONLY` status and are not sent to Mastodon.
## systemd
Examples are in `systemd/`. Copy the matching service and the common timer, then create the corresponding environment file from `config/native.env.example` or `config/docker.env.example`.
For example:
```sh
install -m 0644 systemd/mastodon-email-blocker-docker.service /etc/systemd/system/mastodon-email-blocker.service
install -m 0644 systemd/mastodon-email-blocker.timer /etc/systemd/system/mastodon-email-blocker.timer
install -m 0644 config/docker.env.example /etc/mastodon-email-blocker/docker.env
```
Edit `/etc/mastodon-email-blocker/docker.env`, then enable the timer:
```sh
systemctl daemon-reload
systemctl enable --now mastodon-email-blocker.timer
```
A manual service run is useful before enabling the timer:
```sh
systemctl start mastodon-email-blocker.service
journalctl -u mastodon-email-blocker.service -n 100 --no-pager
```
## Interrupting and re-running
A run can be interrupted with `Ctrl+C`. Completed batches stay in Mastodon. On the next run the existing records are read again, so already-added domains are skipped and only the remaining ones are submitted.
## Files written
- `/var/lib/mastodon-email-blocker/last-report.json` — last run report
- `/var/lib/mastodon-email-blocker/state.json` — small state summary
The tool does not store Mastodon credentials or API tokens.
+421
View File
@@ -0,0 +1,421 @@
#!/usr/bin/env python3
import argparse
import concurrent.futures
import json
import os
import re
import shlex
import subprocess
import sys
import tempfile
import time
import urllib.request
from collections import defaultdict
from pathlib import Path
try:
import dns.resolver
except ImportError:
dns = None
VERSION = "1.0.0"
SOURCES = [
"https://raw.githubusercontent.com/mehrtat/disposable-email-domain/main/domains.txt",
"https://raw.githubusercontent.com/FFraud-com/disposable-email-domains/main/disposable-email-domains.txt",
"https://disposable.github.io/disposable-email-domains/domains.txt",
]
DOMAIN_RE = re.compile(
r"^(?=.{1,253}$)(?:[a-z0-9](?:[a-z0-9-]{0,61}[a-z0-9])?\.)+"
r"[a-z0-9](?:[a-z0-9-]{0,61}[a-z0-9])?$"
)
def log(message=""):
print(message, flush=True)
def normalize_domain(value):
if not value:
return None
value = value.strip().lower().rstrip(".")
if not value or value.startswith("#"):
return None
if value.startswith("@"):
value = value[1:]
if value.startswith("*."):
value = value[2:]
try:
value = value.encode("idna").decode("ascii")
except Exception:
return None
return value if DOMAIN_RE.match(value) else None
def read_domain_file(filename):
path = Path(filename)
if not path.exists():
return set()
result = set()
for line in path.read_text(encoding="utf-8", errors="ignore").splitlines():
domain = normalize_domain(line)
if domain:
result.add(domain)
return result
def download_source(url, timeout=120):
request = urllib.request.Request(
url,
headers={"User-Agent": f"mastodon-email-blocker/{VERSION}"},
)
with urllib.request.urlopen(request, timeout=timeout) as response:
content = response.read().decode("utf-8", errors="ignore")
result = set()
for line in content.splitlines():
domain = normalize_domain(line)
if domain:
result.add(domain)
return result
def chunks(values, size):
values = list(values)
for i in range(0, len(values), size):
yield values[i:i + size]
def save_json_atomic(filename, value):
filename = Path(filename)
filename.parent.mkdir(parents=True, exist_ok=True)
fd, temporary = tempfile.mkstemp(prefix=filename.name + ".", dir=str(filename.parent))
try:
with os.fdopen(fd, "w", encoding="utf-8") as handle:
json.dump(value, handle, indent=2, sort_keys=True)
handle.flush()
os.fsync(handle.fileno())
os.replace(temporary, filename)
finally:
if os.path.exists(temporary):
os.unlink(temporary)
class MastodonBackend:
def run_rails(self, code):
raise NotImplementedError
def run_tootctl_add(self, domains):
raise NotImplementedError
class NativeBackend(MastodonBackend):
def __init__(self, mastodon_dir, ruby_bin=None):
self.mastodon_dir = Path(mastodon_dir)
self.ruby_bin = ruby_bin
if not (self.mastodon_dir / "bin/rails").exists():
raise RuntimeError(f"bin/rails not found in {self.mastodon_dir}")
if not (self.mastodon_dir / "bin/tootctl").exists():
raise RuntimeError(f"bin/tootctl not found in {self.mastodon_dir}")
def _env(self):
env = os.environ.copy()
env["RAILS_ENV"] = "production"
if self.ruby_bin:
env["PATH"] = f"{self.ruby_bin}:{env.get('PATH', '/usr/local/bin:/usr/bin:/bin')}"
return env
def _run(self, command):
process = subprocess.run(
command,
cwd=str(self.mastodon_dir),
env=self._env(),
text=True,
stdout=subprocess.PIPE,
stderr=subprocess.PIPE,
)
if process.returncode != 0:
if process.stdout:
sys.stderr.write(process.stdout)
if process.stderr:
sys.stderr.write(process.stderr)
raise RuntimeError("Command failed: " + " ".join(command))
return process.stdout
def run_rails(self, code):
return self._run(["bin/rails", "runner", code])
def run_tootctl_add(self, domains):
return self._run(["bin/tootctl", "email-domain-blocks", "add", *domains])
class DockerBackend(MastodonBackend):
def __init__(self, compose_dir, compose_command="docker compose", service="web"):
self.compose_dir = Path(compose_dir)
self.compose_command = shlex.split(compose_command)
self.service = service
if not self.compose_dir.exists():
raise RuntimeError(f"Compose directory not found: {self.compose_dir}")
def _run(self, command):
full_command = [
*self.compose_command,
"exec",
"-T",
"-e", "RAILS_ENV=production",
self.service,
*command,
]
process = subprocess.run(
full_command,
cwd=str(self.compose_dir),
text=True,
stdout=subprocess.PIPE,
stderr=subprocess.PIPE,
)
if process.returncode != 0:
if process.stdout:
sys.stderr.write(process.stdout)
if process.stderr:
sys.stderr.write(process.stderr)
raise RuntimeError("Command failed: " + " ".join(full_command))
return process.stdout
def run_rails(self, code):
return self._run(["bin/rails", "runner", code])
def run_tootctl_add(self, domains):
return self._run(["bin/tootctl", "email-domain-blocks", "add", *domains])
def get_blocks(backend):
code = r'''
rows = EmailDomainBlock.order(:id).pluck(:id, :domain, :allow_with_approval, :parent_id)
STDOUT.write("__JSON__" + JSON.generate(rows))
'''
output = backend.run_rails(code)
marker = "__JSON__"
pos = output.rfind(marker)
if pos < 0:
raise RuntimeError("Could not read EmailDomainBlock from Rails")
return json.loads(output[pos + len(marker):])
def force_deny(backend):
code = r'''
count = EmailDomainBlock.where(allow_with_approval: true).update_all(
allow_with_approval: false,
updated_at: Time.current
)
STDOUT.write("__COUNT__#{count}")
'''
output = backend.run_rails(code)
marker = "__COUNT__"
pos = output.rfind(marker)
if pos < 0:
raise RuntimeError("Could not convert approval blocks to deny blocks")
return int(output[pos + len(marker):].strip())
def resolve_mx(domain, timeout):
resolver = dns.resolver.Resolver(configure=True)
resolver.timeout = timeout
resolver.lifetime = timeout
try:
answers = resolver.resolve(domain, "MX")
except Exception:
return domain, []
mx_records = set()
for answer in answers:
mx = normalize_domain(str(answer.exchange))
if mx:
mx_records.add(mx)
return domain, sorted(mx_records)
def build_backend(args):
if args.backend == "native":
if not args.mastodon_dir:
raise RuntimeError("--mastodon-dir is required for the native backend")
return NativeBackend(args.mastodon_dir, args.ruby_bin)
if not args.compose_dir:
raise RuntimeError("--compose-dir is required for the docker backend")
return DockerBackend(args.compose_dir, args.compose_command, args.compose_service)
def main():
parser = argparse.ArgumentParser(description="Maintain Mastodon disposable email domain blocks")
parser.add_argument("--backend", choices=["native", "docker"], required=True)
parser.add_argument("--mastodon-dir")
parser.add_argument("--ruby-bin", help="Directory containing ruby, only needed when Ruby is not already in PATH")
parser.add_argument("--compose-dir")
parser.add_argument("--compose-command", default="docker compose")
parser.add_argument("--compose-service", default="web")
parser.add_argument("--allowlist", default="/etc/mastodon-email-blocker/allowlist.txt")
parser.add_argument("--report", default="/var/lib/mastodon-email-blocker/last-report.json")
parser.add_argument("--state", default="/var/lib/mastodon-email-blocker/state.json")
parser.add_argument("--batch-size", type=int, default=250)
parser.add_argument("--mx-mode", choices=["off", "report"], default="off")
parser.add_argument("--mx-workers", type=int, default=40)
parser.add_argument("--mx-timeout", type=float, default=4.0)
parser.add_argument("--dry-run", action="store_true")
parser.add_argument("--no-force-deny", action="store_true")
args = parser.parse_args()
if args.mx_mode == "report" and dns is None:
raise RuntimeError("MX reporting requires dnspython: pip install dnspython")
backend = build_backend(args)
allowlist = read_domain_file(args.allowlist)
log(f"mastodon-email-blocker {VERSION}")
log(f"backend: {args.backend}")
log(f"mx mode: {args.mx_mode}")
log()
disposable = set()
source_stats = {}
successful_sources = 0
for url in SOURCES:
log(f"[source] {url}")
try:
domains = download_source(url)
source_stats[url] = len(domains)
successful_sources += 1
disposable |= domains
log(f" {len(domains):,} domains")
except Exception as exc:
source_stats[url] = f"ERROR: {exc}"
log(f" ERROR: {exc}")
if successful_sources == 0:
raise RuntimeError("All blocklist sources failed; refusing to change Mastodon")
disposable -= allowlist
log(f"[source] union: {len(disposable):,} disposable domains")
log("[mastodon] reading existing blocks")
rows = get_blocks(backend)
approvals = [row for row in rows if row[2] is True]
converted = 0
if approvals and not args.no_force_deny:
if args.dry_run:
log(f"[dry-run] would convert {len(approvals):,} approval blocks to deny")
else:
converted = force_deny(backend)
log(f"[mastodon] converted {converted:,} approval blocks to deny")
rows = get_blocks(backend)
existing = {normalize_domain(row[1]) for row in rows}
existing.discard(None)
missing = sorted(disposable - existing)
log(f"[domains] existing blocks: {len(existing):,}")
log(f"[domains] disposable domains to add: {len(missing):,}")
domain_batches = list(chunks(missing, args.batch_size))
for number, batch in enumerate(domain_batches, 1):
log(f"[domains] batch {number}/{len(domain_batches)} ({len(batch)})")
if args.dry_run:
continue
backend.run_tootctl_add(batch)
mx_to_domains = defaultdict(set)
domains_with_mx = 0
domains_without_mx = 0
if args.mx_mode == "report":
all_domains = sorted(disposable)
log(f"[mx] resolving {len(all_domains):,} domains; MX records are report-only")
with concurrent.futures.ThreadPoolExecutor(max_workers=args.mx_workers) as executor:
futures = {
executor.submit(resolve_mx, domain, args.mx_timeout): domain
for domain in all_domains
}
completed = 0
for future in concurrent.futures.as_completed(futures):
try:
domain, records = future.result()
except Exception:
domain, records = futures[future], []
if records:
domains_with_mx += 1
else:
domains_without_mx += 1
for mx in records:
mx_to_domains[mx].add(domain)
completed += 1
if completed % 5000 == 0 or completed == len(all_domains):
log(f"[mx] {completed:,}/{len(all_domains):,}")
top_mx = sorted(
((mx, len(domains)) for mx, domains in mx_to_domains.items()),
key=lambda item: (-item[1], item[0]),
)[:100]
final_rows = rows if args.dry_run else get_blocks(backend)
final_deny = sum(1 for row in final_rows if row[2] is False)
final_approval = sum(1 for row in final_rows if row[2] is True)
report = {
"version": VERSION,
"timestamp": int(time.time()),
"backend": args.backend,
"dry_run": args.dry_run,
"sources": source_stats,
"successful_sources": successful_sources,
"disposable_domains": len(disposable),
"allowlist_entries": len(allowlist),
"existing_before": len(rows),
"approval_before": len(approvals),
"approval_converted": converted,
"missing_domains": len(missing),
"domains_added_or_planned": len(missing),
"mx_mode": args.mx_mode,
"mx_autoblock": False,
"domains_with_mx": domains_with_mx,
"domains_without_mx": domains_without_mx,
"mx_unique": len(mx_to_domains),
"final_deny": final_deny,
"final_approval": final_approval,
"top_mx": [
{"mx": mx, "disposable_domains": count, "action": "REPORT_ONLY"}
for mx, count in top_mx
],
}
save_json_atomic(args.report, report)
save_json_atomic(args.state, {
"version": VERSION,
"last_run": int(time.time()),
"backend": args.backend,
"disposable_domains": len(disposable),
"mx_unique": len(mx_to_domains),
"mx_autoblock": False,
"last_report": args.report,
})
log()
log("Result")
log(f" disposable domains : {len(disposable):,}")
log(f" new domains : {len(missing):,}")
log(f" MX observed : {len(mx_to_domains):,}")
log(" MX blocked : 0")
log(f" deny blocks : {final_deny:,}")
log(f" approval blocks : {final_approval:,}")
log(f" report : {args.report}")
if __name__ == "__main__":
try:
main()
except KeyboardInterrupt:
log("\nInterrupted")
sys.exit(130)
except Exception as exc:
log(f"\nERROR: {exc}")
sys.exit(1)
+4
View File
@@ -0,0 +1,4 @@
# One domain per line.
# Entries here are removed from the aggregated disposable-domain set.
#
# example.org
+5
View File
@@ -0,0 +1,5 @@
# Used by the systemd example for an existing Docker Compose Mastodon installation.
# COMPOSE_DIR is the directory containing compose.yml/docker-compose.yml.
COMPOSE_DIR=/path/to/mastodon-compose
COMPOSE_SERVICE=web
COMPOSE_COMMAND=docker compose
+4
View File
@@ -0,0 +1,4 @@
# Used by the systemd example for a source/native Mastodon installation.
# Edit these paths for the local installation.
MASTODON_DIR=/path/to/mastodon/live
RUBY_BIN=/path/to/ruby/bin
+21
View File
@@ -0,0 +1,21 @@
# Docker Compose notes
This integration is for an existing Mastodon deployment. It does not create a Mastodon stack and does not add a new long-running application container.
The host-side script calls commands equivalent to:
```sh
docker compose exec -T -e RAILS_ENV=production web bin/rails runner '...'
docker compose exec -T -e RAILS_ENV=production web bin/tootctl email-domain-blocks add ...
```
The official Mastodon Compose file uses a `web` service for the Rails application. Other distributions and custom Compose files may use a different service name; set `--compose-service` accordingly.
Before running the blocker, these two checks should succeed from the Compose directory:
```sh
docker compose exec -T web bin/rails runner 'puts EmailDomainBlock.count'
docker compose exec -T web bin/tootctl email-domain-blocks list --help
```
If the local deployment uses a wrapper, a different Compose project directory, or the old `docker-compose` binary, adjust `--compose-dir` and `--compose-command`. The Python program does not need access to the database container and should not be given database credentials.
+1
View File
@@ -0,0 +1 @@
dnspython>=2.6,<3
@@ -0,0 +1,12 @@
[Unit]
Description=Update Mastodon disposable email domain blocks through Docker Compose
After=network-online.target docker.service
Wants=network-online.target docker.service
[Service]
Type=oneshot
EnvironmentFile=/etc/mastodon-email-blocker/docker.env
ExecStart=/opt/mastodon-email-blocker/venv/bin/python /opt/mastodon-email-blocker/mastodon-email-blocker.py --backend docker --compose-dir ${COMPOSE_DIR} --compose-command ${COMPOSE_COMMAND} --compose-service ${COMPOSE_SERVICE} --mx-mode off
ReadWritePaths=/var/lib/mastodon-email-blocker
ReadOnlyPaths=/etc/mastodon-email-blocker
TimeoutStartSec=infinity
@@ -0,0 +1,12 @@
[Unit]
Description=Update Mastodon disposable email domain blocks
After=network-online.target
Wants=network-online.target
[Service]
Type=oneshot
EnvironmentFile=/etc/mastodon-email-blocker/native.env
ExecStart=/opt/mastodon-email-blocker/venv/bin/python /opt/mastodon-email-blocker/mastodon-email-blocker.py --backend native --mastodon-dir ${MASTODON_DIR} --ruby-bin ${RUBY_BIN} --mx-mode off
ReadWritePaths=/var/lib/mastodon-email-blocker
ReadOnlyPaths=/etc/mastodon-email-blocker
TimeoutStartSec=infinity
+10
View File
@@ -0,0 +1,10 @@
[Unit]
Description=Run Mastodon disposable email blocker daily
[Timer]
OnCalendar=*-*-* 04:20:00
RandomizedDelaySec=30m
Persistent=true
[Install]
WantedBy=timers.target