nulldrm/Null-DRM-Official
1
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions

Initial commit: Null DRM Official

Browse source 
Capture, decrypt, and restream toolkit with compiled-in app modules
(RTE, TG4, BBC), on-device MITM proxy, streamd control plane, and www.
BBC module.yaml is published (clear streams); other module values stay local.
This commit is contained in:
404errordeveloper 2026-10-06 00:25:35 +02:00
commit 2fa8f2435f
121 changed files with 17802 additions and 0 deletions
Download patch file Download diff file Expand all files Collapse all files

40
docs/README.md Normal file
View file

@ -0,0 +1,40 @@
# Documentation
| Doc | Read it when |
|---|---|
| [quickstart.md](quickstart.md) | You just cloned the repo and want something working |
| [architecture.md](architecture.md) | You want to know how the pieces fit and why |
| [modules.md](modules.md) | You are adding support for a new streaming app |
| [capture.md](capture.md) | You need to discover an app's URLs, IDs and UI selectors |
| [streamd.md](streamd.md) | You are running the control plane + agent |
| [providers/rte.md](providers/rte.md) | You are filling in `apps/modules/rte/module.yaml` |
| [providers/tg4.md](providers/tg4.md) | You are filling in `apps/modules/tg4/module.yaml` |
## The one idea worth knowing first
The repo separates **logic** from **values**:
```text
apps/modules/tg4/*.go tracked. How to drive the app. No secrets, ever.
apps/modules/tg4/module.yaml gitignored. Account IDs, policy keys, video IDs, KIDs.
```
Both module packages have a test asserting an empty config yields empty
credentials, so the separation is checked rather than merely intended:
```bash
go -C apps/modules test ./... -run NoHardcoded -v
```
A fresh clone therefore builds and runs, but knows no channels until you supply
your own values. Each `providers/*.md` doc explains where to get them.
## Build
```bash
go -C apps/cli build -o ../../bin/drm . # Windows: -o ../../bin/drm.exe
./bin/drm proxy build # once, for phone capture
```
There is no build script and no Makefile — `go build` is the build system.
[quickstart.md](quickstart.md) lists every command.

241
docs/architecture.md Normal file
View file

@ -0,0 +1,241 @@
# Architecture
## What the system does
Given an Android app that streams DRM-protected live TV, it recovers the three
things a downloader needs — **manifest URL**, **Widevine content keys**, and any
**auth headers** — and keeps them fresh.
Two independent ways to get them:
1. **Catalog** (`drm catalog`) — the app's playback catalog is public, so a plain
HTTPS request plus a local CDM is enough. No phone.
2. **Phone capture** (`drm capture`) — launch the real app under an on-device
HTTPS MITM, drive its UI to the channel, and read the license exchange.
What it deliberately does not do: break DRM. The CDM used is a provisioned
Widevine device you supply (`.wvd`); the system only automates the request flow a
real client performs.
## One binary
```text
┌──────────────────────── bin/drm ────────────────────────┐
│ │
modules ───┤ capture catalog agent serve proxy │
linked in │ │
└─────────────────────────────────────────────────────────┘
```
`apps/cli` is the only `main`. It blank-imports `drmdecryption/modules/all`, which
is what makes `--app <name>` resolve. Each app module registers itself:
```go
func init() {
appreg.Register(Name, New) // --app rte
appreg.RegisterFlags(bindFlags) // --rte.<field> overrides
}
```
Adding a provider means adding a package and one import line. There is no plugin
loading, no scripting runtime, and nothing to deploy beside the binary.
## Logic vs values
This is the central constraint:
| `apps/modules/<name>/*.go` — **tracked** | `apps/modules/<name>/module.yaml` — **gitignored** (bbc published) |
|---|---|
| launch + auto-play sequence | android package id |
| navigation idioms (chip→Play, drawer→Live) | license URL, origin hostnames |
| manifest rewrite shape | channel KIDs, account id, policy key |
| catalog request flow | video ids, playback config URL |
| which capture fields are required | UI selector regexes, labels, aliases |
Module Go source contains no account id, policy key, video id, license URL, origin
host or KID. Values arrive by **flag → environment → module.yaml → Go default**,
and only mechanical things (timeouts, DASH timescales, card geometry) have
defaults in code.
`.gitignore` enforces the split:
```gitignore
apps/modules/**
!apps/modules/README.md
!apps/modules/go.mod
!apps/modules/go.sum
!apps/modules/*/
!apps/modules/*/*.go
!apps/modules/*/**/*.go
```
The directory re-include is required — git will not descend into an excluded
directory to find negated files.
## Go modules
Six modules, wired with `replace`:
```text
apps/pkg drmdecryption shared, provider-neutral
apps/modules drmdecryption/modules app modules -> pkg
apps/capture drmdecryption/apps/capture capture command -> pkg
apps/agent drmdecryption/apps/agent agent -> pkg
apps/streamd drmdecryption/apps/streamd control plane -> pkg
apps/proxy drmdecryption/apps/proxy MITM host CLI -> pkg
apps/cli drmdecryption/apps/cli the binary -> all of the above
apps/proxy/device appproxy standalone, cross-compiled for the phone
```
Command bodies live in non-`internal` packages (`capturecmd`, `agentcmd`,
`servecmd`, `proxyctlcmd`) because `internal/` is not importable across module
boundaries and `apps/cli` has to call them.
## Plugin contract
`apps/pkg/app` defines what a module must provide and what it may add.
Required (`app.App`):
| Method | Purpose |
|---|---|
| `Name`, `Package`, `LicenseURL` | identity |
| `Launch(c)` | cold-start the app, wait for its home UI |
| `AutoPlay(c, channel)` | navigate to a channel and confirm playback |
| `CaptureHints()` | which capture fields are required, plus URL scoring |
| `KeyMode()` | `"modulardrm"` or `"raw"` — never sniffed from the license URL |
| `MPDRewriter()` | manifest rewrite, or `mpd.Passthrough{}` |
| `Channels`, `HasChannel` | channel list and lookup |
| `ProxyBin`, `CAHash` | on-device MITM binary and CA hash |
Optional, detected by type assertion:
| Interface | Gives you |
|---|---|
| `app.Catalog` | `drm catalog --app <name>` works |
| `app.CatalogLister` | a richer `catalog --list` with catalog ids and provider flags |
| `app.StreamDefaults` | streamd downloader settings (selectors, rewriter, buffering) |
| `mpd.Rewriter` + `mpd.Register` | a named manifest rewrite streamd can resolve |
## Shared libraries (`apps/pkg`)
| Package | Role |
|---|---|
| `app` | plugin interface + registry + optional capability interfaces |
| `adb` | ADB client: shell, push/pull, UI dump, tap, swipe, playback probe |
| `uiflow` | UI primitives modules drive playback with — every threshold a parameter |
| `capture` | parse on-device capture JSON; score candidate manifest URLs |
| `phonecap` | one capture job end to end |
| `proxy` | push/start/stop the on-device MITM, CA install/reinject, pull captures |
| `wvkey` | shells out to `apps/wvkey/wvkey.py` |
| `brightcove` | Brightcove Playback API + HLS Widevine extraction — a *platform*, not a provider |
| `mpd` | rewriter interface + registry, synthetic live manifest builder, local server |
| `modcfg` | load a module's `module.yaml`, apply env overrides, resolve paths |
| `session` | write `outputs/<app>/<stamp>/` |
| `repo` | locate the repo root |
Nothing here names a streaming provider.
## Phone capture pipeline
```text
drm capture --app X --channel Y --auto-play
│
├─ push + start apps/proxy/device (appproxy) on the phone
├─ set the phone's global http_proxy to 127.0.0.1:8080
├─ Magisk-mount the MITM CA into the system trust store
├─ module.Launch() : force-stop → monkey launch → wait for a home marker
├─ module.AutoPlay() : app-specific navigation → wait for playback
├─ capture.Wait() : poll the device capture JSON + mirrored proxy log
│ until the module's required fields appear
├─ wvkey.py : PSSH + license → KID:KEY (mode from module.KeyMode())
└─ session.Write() : outputs/<app>/<stamp>/ + latest/
```
The MITM runs **on the device**, so it cannot read a module's values file. A
module's manifest-scoring hosts are passed to it as flags
(`capture.ScoreCfg.ProxyArgs()` → `-score-host`, `-score-deny`).
### Why URL scoring exists
A MITM sees EPG feeds, analytics beacons and config blobs alongside the real
manifest. `capture.ScoreCfg` ranks candidates: format-level signals (`.mpd`,
`.m3u8`, `playlist-hls`, `/manifest`, `.isml`) are generic and live in `apps/pkg`;
provider hosts and noise needles come from `module.yaml`.
## Manifest rewriting
Some providers serve an ad-stitched manifest whose session expires (HTTP 410)
while the underlying origin keeps running. `apps/pkg/mpd` can publish a
**synthetic** manifest on the origin encoder's own segment grid so a downloader
keeps fetching real segments:
```text
mpd.LearnGrid fetch the origin Smooth manifest once; derive segment
duration and phase offset per media type
mpd.BuildSynthetic emit a dynamic MPD on that grid, with a sliding window
mpd.StartLocal serve it on 127.0.0.1 and re-run the rewrite on every GET
mpd.Register/Lookup resolve a rewriter by name, so streamd needs no provider import
```
`SyntheticConfig` holds every value that could differ — timescales, stream names,
bandwidths, codecs, resolution, segment templates, live-edge offset, window size.
The RTE module supplies origins and KIDs; when the upstream manifest is gone, a
`mpd.KIDResolver` maps the KID from a previous capture back to its origin.
## Control plane (`drm serve`)
| Piece | Path |
|---|---|
| Command | `apps/streamd/servecmd` |
| HTTP API | `apps/streamd/internal/api` |
| SQLite | `apps/streamd/internal/db` |
| Dashboard | `apps/streamd/internal/ui` |
| Media supervisor | `apps/streamd/internal/supervisor` |
Port 8083 by default. Mutating routes need `Authorization: Bearer <token>`.
Streamd stores no provider knowledge. `GET /api/apps` reports the compiled-in
modules, and the supervisor resolves per-stream downloader settings at start time:
```text
settingsFor(stream):
stored row -> the stream's app module (app.StreamDefaults) -> neutral defaults
```
The rewriter is a registry lookup on the stream's `rewriter` column, and HLS is
detected from the manifest shape — not from an app name or a stream name.
## Agent (`drm agent run`)
```text
poll streamd ──► durable SQLite queue ──► claim a free ADB device
│
POST credentials ◄── phone capture (phonecap)
```
Per-stream claims expire via a websocket heartbeat, so a dead agent releases its
phones. The agent never installs apps and never runs a downloader. It drives
whichever module a stream names; it contains no provider logic.
## Where things land
| Path | Contents |
|---|---|
| `data/` | durable secrets: `.wvd`, CA — gitignored |
| `outputs/<app>/<stamp>/` | capture + catalog sessions, with `latest/` as a copy |
| `outputs/discover/<stamp>/` | MITM discover dumps |
| `.cache/streamd/` | streamd SQLite, work dirs, HLS output |
| `.cache/ui-<app>/` | UI dumps used by the automation |
## Honest status
| Area | State |
|---|---|
| Capture (catalog + phone), both modules | working, verified against live services |
| Compiled-in modules, one binary | done |
| streamd API + SQLite + dashboard + claims | done |
| Synthetic manifest rewrite | done, verified against a live origin |
| Agent queue + phone refresh | done |
| Real downloader/ffmpeg workers in streamd | partial — the supervisor shells out to external tools and is the least exercised path |

262
docs/capture.md Normal file
View file

@ -0,0 +1,262 @@
# Using the capture binary
`drm capture` runs an HTTPS MITM **on the phone**, drives the app, and records the
DRM exchange. It is both the production capture path and the tool you use to
discover everything a new module needs.
## Device setup (once)
```bash
adb devices # phone listed and authorised
./bin/drm proxy build # cross-compile the on-device MITM (linux/arm64)
./bin/drm proxy install-ca --reinject
```
The phone must be rooted (Magisk). `install-ca` pushes the MITM CA and bind-mounts
it over the system trust store, including into already-running zygote namespaces —
a user-store CA is not enough, modern apps ignore it.
Verify the mount:
```bash
adb shell 'ls /system/etc/security/cacerts/ | grep 6c3578b4'
adb shell 'su -c "ls /apex/com.android.conscrypt/cacerts/6c3578b4.0"'
```
If an app still reports `tls: unknown certificate`, re-run the reinject and
force-stop the app so it inherits the mount:
```bash
./bin/drm proxy reinject-ca
adb shell am force-stop <package>
```
Keep the screen on for the whole session — UI automation cannot tap a dark screen:
```bash
adb shell svc power stayon true
adb shell settings put system screen_off_timeout 1800000
```
Restore the timeout afterwards (default is usually 60000).
## Three modes
### Passive — no module needed
```bash
./bin/drm capture --wait 300
```
Starts the MITM and nothing else. You open any app and play something; it records
whatever license, PSSH and manifest traffic appears and writes
`outputs/capture/<stamp>/`. This is the first thing to run against an unknown app.
### Launch only
```bash
./bin/drm capture --app myapp --wvd data/device.wvd
```
Launches the app, then waits while you navigate by hand. Useful while you are still
working out the UI sequence.
### Full automation
```bash
./bin/drm capture --app myapp --channel main --auto-play --wvd data/device.wvd
```
Runs the module's `Launch` and `AutoPlay`, then waits for the required fields.
## Flags
| Flag | Default | Purpose |
|---|---|---|
| `--app` | _(empty)_ | module to drive; empty means passive |
| `--channel` | — | channel id, required by `--auto-play` |
| `--auto-play` | false | run the module's navigation |
| `--wait` | 180 | seconds to wait for the capture to complete |
| `--wvd` | — | Widevine device file; required to fetch keys |
| `--python` | `.venv` | interpreter for `wvkey.py` |
| `--key-mode` | from the module | force `modulardrm` or `raw` |
| `--user-agent` | — | UA for the license request |
| `--serial` | sole device | target a specific phone |
| `--close` | true | force-stop the app when done |
| `--<app>.<field>` | — | override any of that module's values |
## What a run looks like
```text
[*] Stopping any old appproxy...
[*] Pushing appproxy...
[+] appproxy pid=7604; capture -> /data/local/tmp/appproxy_cap.json
[*] Setting HTTP proxy -> 127.0.0.1:8080 (was "null")
[*] Re-injecting system CA (Magisk)...
[+] System CA present in conscrypt
[*] Launching rte (air.RTE.OSMF.Minimal)…
[*] Auto-play RTE 2…
[*] tap_ui "Live"/"Live tab, 2 out of 5" @ 324,2253
[*] tap chip @ 356,297
[*] tap Play @ 540,720
[+] air.RTE.OSMF.Minimal is PLAYING
[*] Waiting for license + PSSH + manifest…
[+] Capture:
pid: RKRPCEib8z9V
mpd: https://dai.google.com/linear/dash/pa/event/.../stream/...
license: https://widevine.entitlement.eu.theplatform.com/wv/web/ModularDrm?...
pssh: AAAASnBzc2gAAAAA7e+LqXnWSs6jyCfc1R0h7QAAACoiIGRmMTYzMzgyMWRk…
[+] key: 5490781a77572107ec251237e94e3b53:cee5fc45b70040ffea2b2576fb5abda5
[+] session: outputs/rte/20261005-220252/session.json
[*] Closing air.RTE.OSMF.Minimal…
[*] HTTP proxy cleared
```
### Output
```text
outputs/<app>/<stamp>/
session.json everything, machine-readable
mpd.txt manifest URL
key.txt KID:KEY (one per line for multi-key streams)
pssh.txt Widevine init data
auth.txt Authorization value, when the app uses one
pid.txt provider id, when the app uses one
outputs/<app>/latest/ a copy of the most recent run
```
Live log while a capture runs: `.cache/appproxy-<serial>.log`.
---
## Discovering a new app
### 1. Record everything
```bash
./bin/drm proxy discover --install-ca --reinject
# play the app on the phone; Ctrl+C when done
```
Writes `outputs/discover/<stamp>/`:
| File | Contents |
|---|---|
| `appproxy_traffic.jsonl` | every HTTP(S) request/response, one JSON object per line |
| `appproxy_cap.json` | the structured fields the proxy recognised |
| `appproxy.log` | tagged lines: `[MPD]`, `[LIC]`, `[PSSH]`, `[MAN]` |
### 2. Mine it
```bash
# the android package id
adb shell dumpsys window | grep mCurrentFocus
# license endpoint
grep -iE 'license|licence|widevine|/lic/' outputs/discover/*/appproxy.log
# manifest candidates
grep -oE 'https?://[^"]+\.(m3u8|mpd)' outputs/discover/*/appproxy_traffic.jsonl \
| sort -u
# the hosts involved, by frequency
grep -oE '"host":"[^"]+"' outputs/discover/*/appproxy_traffic.jsonl \
| sort | uniq -c | sort -rn | head -20
# config blobs that often carry ids and tokens
grep -oE 'https?://[^"]+(config|playback)[^"]*\.json' \
outputs/discover/*/appproxy_traffic.jsonl | sort -u
```
What to pull out, and where it goes in `module.yaml`:
| Found | Key |
|---|---|
| package id | `package` |
| license endpoint | `license_url` |
| CDN host serving the manifest | `score.hosts` |
| analytics / EPG hosts polluting the results | `score.deny` |
| account ids, policy keys, video ids | provider-specific keys |
### 3. Work out the UI
Dump the tree at each step and read off stable selectors:
```bash
adb shell uiautomator dump /sdcard/ui.xml
adb shell cat /sdcard/ui.xml > ui.xml
```
Extract the useful attributes:
```bash
grep -oE 'resource-id="[^"]*"' ui.xml | sort -u
grep -oE 'content-desc="[^"]*"' ui.xml | sort -u
```
Prefer `resource-id` over `content-desc` or `text` — ids survive translation and
copy changes. Check a candidate marker really exists before you rely on it; a
launch marker that never matches costs the full timeout on every capture and is
easy to miss because the pipeline recovers anyway.
Test taps by hand before writing Go:
```bash
adb shell input tap 540 732
adb shell input swipe 540 1700 540 700 350
```
### 4. Determine the key mode
| The app sends | Use |
|---|---|
| a JSON wrapper around the challenge, with an Authorization header | `modulardrm` |
| a raw binary challenge, `Content-Type: application/octet-stream` | `raw` |
Check the license request body in `appproxy_traffic.jsonl`. Then set `KeyMode()`
in the module — never sniff it from the URL at runtime.
### 5. Decide which fields are required
Look at what `appproxy_cap.json` actually contained after a successful play:
- DASH + JSON license → usually `auth`, `pid`, `pssh`, `mpd`
- HLS + binary license → usually `license_url`, `mpd` (the PSSH comes from the
playlist afterwards)
Pass exactly those to `a.Hints(...)`. Asking for a field the app never emits makes
every capture time out.
### 6. Verify
Where an app has a public catalog, capture the same channel both ways and compare —
independent paths agreeing on the PSSH and key set is strong evidence both are
right:
```bash
./bin/drm catalog --app myapp --channel main --keys --wvd data/device.wvd --json > a.json
./bin/drm capture --app myapp --channel main --auto-play --wvd data/device.wvd
diff <(jq -r '.keys[]' a.json | sort) <(sort outputs/myapp/latest/key.txt)
```
Also sanity-check that the KID you captured matches whatever you recorded in
`module.yaml` for that channel.
## Troubleshooting
| Symptom | Likely cause |
|---|---|
| `tls: unknown certificate` in the app | CA not mounted into conscrypt; reinject, then force-stop the app |
| `timed out waiting for capture ([auth pid pssh mpd])` | playback never started, or `Hints` asks for a field this app does not send |
| `launch: ui node not found within Ns` | launch marker does not match — dump the tree and pick a resource id |
| `launch: exit status 1` | monkey raced a force-stop; usually transient, retry |
| `auto-play failed: ... not found` | UI changed, or the phone is on a different screen than expected |
| manifest field holds an EPG or config URL | add that host to `score.deny` |
| `CA inject skipped/failed` | transient after repeated runs; re-run `proxy reinject-ca` |
| Screen asleep mid-run | `adb shell svc power stayon true` |
Clear the phone's proxy if a run is interrupted:
```bash
./bin/drm proxy clear-proxy
```

379
docs/modules.md Normal file
View file

@ -0,0 +1,379 @@
# Writing an app module
A module teaches the binary how to drive one streaming app. It is a Go package, it
is compiled in, and it contains no secrets.
Work through [capture.md](capture.md) first — you need the app's package id,
license URL, manifest host and UI selectors before you can write any of this.
## Anatomy
```text
apps/modules/myapp/
config.go Config struct + defaults + flag binding
myapp.go init(), the App type, Launch, AutoPlay, capability methods
steps.go navigation idioms specific to this app
catalog.go optional: resolve channels without a phone
mpd.go optional: manifest rewriting
module.yaml GITIGNORED - your values
```
Everything is registered from `init()`, and one line in `apps/modules/all/all.go`
links it into the binary.
## 1. Config
Embed `provider.Common` for the fields every module has, then add your own.
```go
package myapp
import (
"drmdecryption/modcfg"
"drmdecryption/modules/provider"
)
const Name = "myapp"
type Channel struct {
Label string `yaml:"label"`
LiveCardIndex int `yaml:"live_card_index"`
PlayDesc []string `yaml:"play_desc"`
}
type Config struct {
provider.Common `yaml:",inline"`
// Provider values. Never give these defaults in code.
APIBase string `yaml:"api_base"`
Channels map[string]Channel `yaml:"channels"`
Timeouts Timeouts `yaml:"timeouts"`
UI UI `yaml:"ui"`
}
type Timeouts struct {
// provider.Duration accepts "25s" and bare seconds; time.Duration alone
// would force raw nanoseconds in YAML.
LaunchWaitUI provider.Duration `yaml:"launch_wait_ui"`
Playback provider.Duration `yaml:"playback"`
}
type UI struct {
LaunchResourceContains []string `yaml:"launch_resource_contains"`
LaunchDescContains []string `yaml:"launch_desc_contains"`
}
func (c Config) withDefaults() Config {
t := &c.Timeouts
t.LaunchWaitUI = provider.Duration(t.LaunchWaitUI.D(25 * time.Second))
t.Playback = provider.Duration(t.Playback.D(45 * time.Second))
return c
}
```
`provider.Common` gives you `package`, `license_url`, `proxy_bin`, `ca_hash`,
`aliases` and `score`.
### Loading and overrides
```go
var flags struct {
packageName string
apiBase string
}
func loadConfig() (Config, modcfg.Result, error) {
var cfg Config
res, err := modcfg.Load(Name, &cfg) // reads apps/modules/myapp/module.yaml
if err != nil {
return cfg, res, err
}
cfg.Package = modcfg.Override(Name, "package", flags.packageName, cfg.Package)
cfg.APIBase = modcfg.Override(Name, "api_base", flags.apiBase, cfg.APIBase)
return cfg.withDefaults(), res, nil
}
```
`modcfg.Override` applies **flag → environment (`MYAPP_API_BASE`) → file**. A
missing `module.yaml` is not an error: the module must still construct so
`drm modules` can report it as unconfigured.
## 2. The App type
```go
func init() {
appreg.Register(Name, New)
appreg.RegisterFlags(bindFlags)
}
func bindFlags(fs *flag.FlagSet) {
fs.StringVar(&flags.packageName, Name+".package", "", "override "+Name+" package id")
fs.StringVar(&flags.apiBase, Name+".api-base", "", "override "+Name+" API base")
}
type App struct {
*provider.Base
cfg Config
}
func New() (appreg.App, error) {
cfg, res, err := loadConfig()
if err != nil {
return nil, err
}
labels := map[string]string{}
for id, ch := range cfg.Channels {
labels[id] = ch.Label
}
return &App{Base: provider.New(Name, cfg.Common, labels, res), cfg: cfg}, nil
}
```
Flag binding is a separate hook from construction on purpose: flags must be bound
to the config holder *before* any app is built, or overrides would be read too
late.
`provider.Base` supplies `Name`, `Package`, `LicenseURL`, `ProxyBin`, `CAHash`,
`HasChannel`, `Label`, `ChannelIDs`, `CacheDir`, alias-aware `Resolve`, and
`Hints(require…)` preloaded with your `score:` config.
### Required methods
```go
func (a *App) KeyMode() string { return "raw" } // or "modulardrm"
func (a *App) CaptureHints() capture.Hints {
return a.Hints("license_url", "mpd") // fields the MITM must yield
}
func (a *App) MPDRewriter() mpd.Rewriter { return mpd.Passthrough{} }
func (a *App) Channels() []appreg.Channel {
out := make([]appreg.Channel, 0, len(a.cfg.Channels))
for _, id := range a.ChannelIDs() {
out = append(out, appreg.Channel{ID: id, Label: a.Label(id)})
}
return out
}
```
`Hints(...)` names the JSON keys that must be present before a capture counts as
complete. Common sets:
| Field | Meaning |
|---|---|
| `mpd` | manifest URL (DASH or HLS — the field name is historical) |
| `license_url` | license endpoint seen on the wire |
| `pssh` | Widevine init data |
| `auth` | Authorization header value |
| `pid` | provider-specific id (e.g. a release/program id) |
Ask only for what the app actually produces. Requiring `pssh` from an HLS app that
puts it in the playlist will hang the capture.
## 3. Launch and AutoPlay
```go
func (a *App) Launch(c *adb.Client) error {
c.EnsureAwake()
c.ForceStop(a.Package())
time.Sleep(400 * time.Millisecond)
if err := uiflow.MonkeyLaunch(c, a.Package()); err != nil {
return err
}
time.Sleep(time.Duration(a.cfg.Timeouts.LaunchSettle))
c.DismissShadeIfFocused()
return uiflow.WaitUI(c, a.CacheDir(), uiflow.Match{
ResourceContains: a.cfg.UI.LaunchResourceContains,
DescContains: a.cfg.UI.LaunchDescContains,
}, time.Duration(a.cfg.Timeouts.LaunchWaitUI))
}
func (a *App) AutoPlay(c *adb.Client, channel string) error {
id, err := a.Base.Resolve(channel)
if err != nil {
return err
}
ch := a.cfg.Channels[id]
if err := a.openLivePage(c); err != nil {
return err
}
if err := uiflow.TapCard(c, a.CacheDir(), ch.LiveCardIndex,
time.Duration(a.cfg.Timeouts.Card), a.cfg.Cards); err != nil {
return err
}
return uiflow.WaitPlayback(c, a.CacheDir(), a.Package(), uiflow.PlaybackOpts{
Timeout: time.Duration(a.cfg.Timeouts.Playback),
SoftFail: true,
OrResourceContains: a.cfg.UI.PlaybackResourceContains,
})
}
```
A `Launch` error is logged and tolerated by the capture pipeline, so pick a launch
marker that genuinely exists — otherwise every run silently burns the full timeout
before `AutoPlay` recovers. **Prefer resource ids over descriptions**: ids are
stable and not localized. Dump the real tree to find them:
```bash
adb shell uiautomator dump /sdcard/ui.xml && adb shell cat /sdcard/ui.xml
```
### uiflow
| Function | Use |
|---|---|
| `WaitUI(c, cache, Match, timeout)` | block until a node exists |
| `TapUI(c, cache, Match, timeout)` | wait, then tap its centre |
| `Find(c, cache, Match, timeout)` | get the node for custom handling |
| `WaitPlayback(c, cache, pkg, PlaybackOpts)` | audio session, or a UI fallback signal |
| `Cards(nodes, CardOpts)` | content rows in a dump, top to bottom |
| `WaitCard` / `TapCard` | the nth content row, scrolling if needed |
| `MonkeyLaunch(c, pkg)` | launcher-intent start |
| `CompileRes([]string)` | case-insensitive regexes |
`Match` combines `TextRE`, `DescRE`, `TextContains`, `DescContains`,
`ResourceContains`, plus `Pick` (`largest`/`smallest`/`first`) and `MinArea`.
`WaitPlayback` normally confirms via the audio session. Players that start without
one need `SoftFail: true` plus a UI signal (`OrDescContains`,
`OrResourceContains`) so capture proceeds anyway.
`CardOpts` carries geometry (`MinY`, `MinH`, `MinW`) and denylists (`DescDeny`,
`TextDeny`) so chrome is not mistaken for content. Defaults suit a 1080-wide
phone; put real values in `module.yaml` if your app differs.
## 4. Optional: catalog
Implement `app.Catalog` and `drm catalog --app myapp` starts working.
```go
func (a *App) Resolve(channel string) (session.Stream, error) {
id, err := a.Base.Resolve(channel)
// ... hit the provider API, pick a manifest + license URL ...
return session.Stream{
Name: Name + "-" + id, App: Name, Channel: id,
MPD: manifestURL, LicenseURL: licenseURL,
HeadersJSON: "{}", Rewriter: "none", ManifestType: "hls",
}, nil
}
func (a *App) EnrichWithKeys(info *session.Stream, python, wvd string) error {
body, err := brightcove.Get(info.MPD)
// ... extract PSSH, then wvkey.FetchRawAll ...
}
```
For a nicer `--list`, also implement `app.CatalogLister` returning
`[]appreg.CatalogRow`.
Decode remote config generically. Do not model a provider's JSON field names as Go
struct fields — read into `map[string]any` and let `module.yaml` name the fields:
```yaml
channels:
main:
token_field: liveLoggedOutTokenIOI
stream_id_field: irelandStreamId
```
A renamed upstream field then needs an edit to your values file, not a rebuild.
## 5. Optional: manifest rewriting
Only needed when the captured manifest expires faster than the stream.
```go
func init() {
mpd.Register(Name, func() mpd.Rewriter { return NewRewriter() })
}
func (r *Rewriter) Name() string { return Name }
func (r *Rewriter) Rewrite(upstreamXML []byte, kidHint string) ([]byte, error) {
// identify the origin channel from the manifest or the KID, then:
return mpd.BuildSynthetic(r.cfg.Synthetic, channel, originBase, kid)
}
// Optional: recover the origin when upstream is gone and only the KID remains.
func (r *Rewriter) ResolveKID(kidHex string) (channel, originBase string, ok bool)
func (r *Rewriter) Prime(channel, originBase string)
```
Register a **factory**, not an instance — a rewriter may hold per-stream state.
## 6. Optional: streamd settings
```go
func (a *App) VideoSelect() string { return "for=best" }
func (a *App) AudioSelect() string { return "lang=en:for=best" }
func (a *App) RewriterName() string { return Name } // or "none"
func (a *App) LiveWaitSeconds() int { return 6 }
func (a *App) TSReadyBytes() int64 { return 2 << 20 }
```
Without this, streamd uses neutral defaults. This is what replaced guessing from
app and stream names.
## 7. Link it in
```go
// apps/modules/all/all.go
import (
_ "drmdecryption/modules/myapp"
_ "drmdecryption/modules/rte"
_ "drmdecryption/modules/tg4"
)
```
```bash
go -C apps/cli build -o ../../bin/drm .
./bin/drm modules
```
## 8. Test it
Assert the interfaces at compile time and prove no secrets leaked:
```go
var (
_ appreg.App = (*App)(nil)
_ appreg.Catalog = (*App)(nil)
_ appreg.StreamDefaults = (*App)(nil)
)
func TestNoHardcodedCredentials(t *testing.T) {
cfg := Config{}.withDefaults()
if cfg.Package != "" || cfg.APIBase != "" {
t.Fatal("provider values must come from module.yaml")
}
}
```
`apps/modules/all/all_test.go` already checks every registered module constructs
with no values file present, reports a valid key mode, and — if it declares a
rewriter name — actually registered one.
For anything needing the live service or your local values, use a build tag so the
normal run stays hermetic:
```go
//go:build live
```
```bash
go -C apps/modules test -tags live ./myapp/ -v
```
## Checklist
- [ ] No account id, key, token, video id, hostname or KID in any `.go` file
- [ ] `module.yaml` is gitignored (`git check-ignore -v apps/modules/myapp/module.yaml`)
- [ ] `New()` succeeds with no `module.yaml`
- [ ] Launch marker actually matches — watch for a silent full-timeout wait
- [ ] `CaptureHints` asks only for fields the app really produces
- [ ] `KeyMode` is correct (`raw` for binary challenges, `modulardrm` for JSON)
- [ ] Added to `all/all.go`
- [ ] `drm modules` lists it with the right channel count

58
docs/providers/bbc.md Normal file
View file

@ -0,0 +1,58 @@
# BBC module — filling in `module.yaml`
BBC iPlayer (Android). Streams from the `mobile-phone-main` mediaset are **clear**
(CDN signed URLs); capture only needs the **MPD/HLS master**. Transparent MITM keeps
NordVPN UK on the phone.
`apps/modules/bbc/module.yaml` is published in the repo (package + channel map only;
no secrets). Skeleton:
```yaml
package: bbc.iplayer.android
proxy_bin: bin/proxy-android-arm64
ca_hash: "6c3578b4"
score:
hosts:
- open.live.bbc.co.uk
- vs-cmaf-push-uk
- vod-dash-uk
- akamaized.net
- bbci.co.uk
- bidi.net.uk
deny:
- telemetry.api.bbci
- bag.api.bbc
- thumbnail
aliases:
one: bbcone
bbc1: bbcone
channels:
bbcone:
label: BBC One
vpid: bbc_one_london
bbctwo:
label: BBC Two
vpid: bbc_two_england
```
## Capture
```bash
# Leave NordVPN UK ON. Magisk CA should already be mounted (skip --reinject).
./bin/drm capture --app bbc --channel bbcone --wait 120
# Play BBC One on the phone when prompted.
```
Console prints the MPD in a banner; `outputs/bbc/<stamp>/session.json` holds the
same fields (`mpd` required; no `key` / `pssh` for clear streams).
Replay:
```powershell
ffplay -user_agent "Mozilla/5.0 (Windows NT 10.0; Win64; x64) Chrome/120.0.0.0 Safari/537.36" -i "<mpd url>"
```
PC needs a UK egress IP (VPN). Some CDNs reject HEAD; use GET / ffplay as above.

224
docs/providers/rte.md Normal file
View file

@ -0,0 +1,224 @@
# RTE module — filling in `module.yaml`
DASH + ModularDrm Widevine, captured from the phone. There is no public catalog, so
every channel needs one phone capture to bootstrap.
Create `apps/modules/rte/module.yaml`. It is gitignored. Nothing below is shipped
in the repo — you obtain it yourself from your own device.
## Skeleton
```yaml
package: <android package id>
license_url: "<Widevine ModularDrm endpoint, incl. its account query string>"
proxy_bin: bin/proxy-android-arm64
ca_hash: "6c3578b4"
score:
hosts:
- <manifest CDN host>
- <ad-stitching host, if the app uses one>
deny:
- <EPG / schedules host fragment>
aliases:
one: chanone
two: chantwo
channels:
chanone:
label: "Channel One"
chip_desc: ["^chanone$"]
play_desc: ["play\\s*chanone"]
chantwo:
label: "Channel Two"
chip_desc: ["^chantwo$"]
play_desc: ["play\\s*chantwo"]
# Origin encoders behind the live channels.
origins:
vc11: "https://<origin host>/live/<path>/vc11/vc11.isml/"
vc12: "https://<origin host>/live/<path>/vc12/vc12.isml/"
kids:
vc11: "<uuid-form KID>"
vc12: "<uuid-form KID>"
```
## Where each value comes from
### `package`
Open the app on the phone, then:
```bash
adb shell dumpsys window | grep mCurrentFocus
```
The part before `/` is the package id.
### `license_url`
Run a discover session and play a channel ([capture.md](capture.md)):
```bash
./bin/drm proxy discover --install-ca --reinject
grep -iE 'ModularDrm|widevine|license' outputs/discover/*/appproxy.log
```
Take the full URL including its query string — the account reference in it is part
of the endpoint. The module also accepts it per run:
```bash
./bin/drm capture --app rte --rte.license-url 'https://.../ModularDrm?...'
```
### `score.hosts` / `score.deny`
`hosts` are the hosts that legitimately serve manifests; `deny` are hosts or URL
fragments that must never be mistaken for one.
```bash
grep -oE '"host":"[^"]+"' outputs/discover/*/appproxy_traffic.jsonl \
| sort | uniq -c | sort -rn | head -20
```
Put the origin and any ad-stitching host in `hosts`. Put EPG/schedule feeds and
analytics in `deny` — this app's schedule feed otherwise scores as a manifest
because its URL looks plausible.
Format-level scoring (`.mpd`, `.m3u8`, `/manifest`, `.isml`) is built in; you only
supply the provider-specific parts.
### `channels.*.chip_desc` and `play_desc`
These are the regexes that locate the channel chip on the Live tab and the large
hero Play button. They match the node **content-description**, case-insensitively.
Navigate to the Live tab by hand and dump the tree:
```bash
adb shell uiautomator dump /sdcard/ui.xml
adb shell cat /sdcard/ui.xml | grep -oE 'content-desc="[^"]*"' | sort -u
```
You are looking for two things:
- a small chip per channel — its description is usually the bare channel name, so
anchor the regex (`^chanone$`) to avoid matching a longer label
- a large "play <channel>" button that appears after the chip is tapped
The module taps the hero Play directly when it is already on screen, otherwise it
taps the chip first and waits for Play to appear. A match is only accepted as the
hero button if its area is at least `hero_min_area` (default 200000), which keeps
small list-item play icons from winning.
### `origins` and `kids` — the important pair
These let the manifest rewriter keep working after the ad-stitched manifest session
expires. Both come from a capture.
**The KID** is in the capture session:
```bash
./bin/drm capture --app rte --channel chanone --auto-play --wvd data/device.wvd
jq -r '.kid' outputs/rte/latest/session.json
```
Convert it to dashed UUID form for the `kids:` map
(`8-4-4-4-12`), e.g. `df1633821dddfdd5bec9822c1ec0f052` becomes
`df163382-1ddd-fdd5-bec9-822c1ec0f052`. Either form is accepted, but UUID form is
easier to read.
**The origin** is the Smooth Streaming base URL. Find it in the discover traffic:
```bash
grep -oE 'https?://[^"]+\.isml[^"]*' outputs/discover/*/appproxy_traffic.jsonl \
| sed -E 's#(\.isml).*#\1/#' | sort -u
```
The last path element before `.isml` is the encoder channel id, which becomes the
map key. So `https://host/live/tc-1/vc11/vc11.isml/` is keyed `vc11`.
**Pair them up** by capturing each channel in turn and noting which KID appeared:
```bash
for ch in chanone chantwo; do
./bin/drm capture --app rte --channel $ch --auto-play --wvd data/device.wvd
echo "$ch -> $(jq -r .kid outputs/rte/latest/session.json)"
done
```
Then confirm the mapping resolves:
```bash
go -C apps/modules test -tags live ./rte/ -v
```
That test fetches each configured origin's manifest, checks the KID resolves back
to its channel, and verifies the synthetic manifest lands on the live encoder's
segment grid. If a KID does not resolve, `kids:` and `origins:` disagree.
### `ca_hash`
The subject hash of the MITM CA, which is what the Magisk mount installs as
`<hash>.0`. `6c3578b4` is the bundled CA; change it only if you generated your own.
```bash
adb shell 'ls /system/etc/security/cacerts/'
```
## Optional tuning
All of these have working defaults — set them only if the app changes.
```yaml
encoder_channel_re: '^(vc|channel)\d+$' # which origin ids use the encoder timeline
origin_marker: .isml # path element identifying an origin BaseURL
hero_min_area: 200000 # smallest node accepted as the hero Play
ui:
launch_desc_contains: ["live tab", "header logo"]
live_tab_desc_re: ["Live tab"]
timeouts:
launch_wait_ui: 25s
live_tab: 30s
chip_wait: 20s
play_wait: 15s
playback: 45s
synthetic:
video_timescale: 600
audio_timescale: 48000
video_name: video
audio_name: audio_128k
video_bandwidth: 6000000
audio_bandwidth: 128000
width: 1920
height: 1080
live_edge_offset_s: 12
window_segments: 6
```
The `synthetic` block describes the origin encoder. Its defaults match a common
Smooth-to-DASH setup; if your origin uses different stream names or bitrates, read
them off the origin manifest:
```bash
curl -s 'https://<origin>/live/<path>/vc11/vc11.isml/Manifest' | head -40
```
`StreamIndex Name="…"` gives `video_name` / `audio_name`, and `Bitrate="…"` gives
the bandwidths. Segment duration and phase are learned at runtime, not configured.
## Verify
```bash
./bin/drm modules # channel count should match your file
./bin/drm capture --app rte --channel chanone --auto-play --wvd data/device.wvd
go -C apps/modules test -tags live ./rte/ -v
```
A good capture yields `auth`, `pid`, `pssh` and `mpd`, and a single `KID:KEY` whose
KID matches the `kids:` entry for that channel's encoder.

288
docs/providers/tg4.md Normal file
View file

@ -0,0 +1,288 @@
# TG4 module — filling in `module.yaml`
Brightcove Live: HLS manifests and a raw Widevine license. The playback catalog is
public, so channels resolve **without a phone** once you have the account id and
policy key. Phone capture also works and is the only route for channels the catalog
does not list.
Create `apps/modules/tg4/module.yaml`. It is gitignored. None of these values ship
in the repo.
## The DRM stack
| Piece | Detail |
|---|---|
| Manifest | HLS `.m3u8` (prefer the DVR playlist) |
| License | `POST` a raw binary challenge to `.../lic/wv?token=<JWT>` |
| License auth | JWT in the **query string**, not a header |
| Key mode | `raw` (not the JSON ModularDrm wrapper) |
| Keys per stream | several — a multi-KID SAMPLE-AES/cbcs stream needs all of them |
| PSSH | from the HLS playlist's Widevine `KEYFORMAT` line, not from a manifest |
Also published on the same host: FairPlay (`/lic/fp`) and PlayReady (`/lic/pr`).
## Skeleton
```yaml
package: <android package id>
proxy_bin: bin/proxy-android-arm64
ca_hash: "6c3578b4"
account_id: "<brightcove account id>"
policy_key: "BCpkAD..."
playback_config_url: "https://<host>/<path>/playback_config.json"
score:
hosts:
- <live CDN host>
deny:
- <analytics host fragment>
aliases:
main: chanone
plus1: plus
channels:
chanone:
label: "Main"
video_id: "<brightcove video id>"
stream_id_field: <field in playback_config holding that id>
token_field: <field in playback_config holding the playback JWT>
live_card_index: 0
kids:
label: "Kids"
video_id: "<brightcove video id>"
token_field: <field name>
live_card_index: 1
sport:
label: "Sport"
phone_only: true # no catalog entry; capture from the phone
live_card_index: 3
ui:
launch_resource_contains:
- <a view id present on the home screen>
```
## Where each value comes from
### `package`
```bash
adb shell dumpsys window | grep mCurrentFocus
```
### `playback_config_url`
The app fetches a JSON config at startup that carries the live stream ids and
playback tokens. Find it in a discover session:
```bash
./bin/drm proxy discover --install-ca --reinject
grep -oE 'https?://[^"]+(config|playback)[^"]*\.json' \
outputs/discover/*/appproxy_traffic.jsonl | sort -u
```
Fetch it and look at the keys:
```bash
curl -s '<playback_config_url>' | jq 'keys'
```
You will see entries along the lines of `<region>StreamId` and
`liveLoggedOutToken<REGION>`. **Those names go in your channel entries**, not in Go
— the module decodes the config generically, so a renamed upstream field is a
one-line edit here.
### `account_id`
The Brightcove account id appears in the catalog request path and in the manifest
URL:
```bash
grep -oE 'edge\.api\.brightcove\.com/playback/v1/accounts/[0-9]+' \
outputs/discover/*/appproxy_traffic.jsonl | sort -u
```
### `policy_key`
A `BCpkAD…` string the app sends as `Accept: application/json;pk=<policyKey>` on
every catalog call. It is embedded in the APK and visible on the wire:
```bash
grep -oE 'pk=BCpkAD[A-Za-z0-9_-]+' outputs/discover/*/appproxy_traffic.jsonl \
| sort -u
```
Treat it as a secret: keep it in this gitignored file, or pass it per run.
```bash
./bin/drm catalog --app tg4 --channel chanone --tg4.policy-key 'BCpkAD...'
export TG4_POLICY_KEY='BCpkAD...'
```
### `channels.*.video_id` vs `stream_id_field`
Two ways to name the Brightcove video for a channel:
- `video_id` — pin it directly. Stable, and what you want once you know it.
- `stream_id_field` — read it from `playback_config.json` at run time. Survives the
provider rotating ids.
Set both and `video_id` wins. Find a channel's id by matching the config field to
the manifest URL you captured — the id appears as the first path segment:
```text
https://<cdn>/<videoId>/<region>/<accountId>/<jwt>/playlist-hls-dvr.m3u8
```
### `channels.*.token_field`
The field in `playback_config.json` holding the long-lived JWT used as
`livePlaybackToken` on the catalog call. Each channel/region has its own. Without
the right one the catalog returns no playable source.
### `channels.*.live_card_index`
Only used for phone capture. It is the channel's position in the app's Live page
card list, counting from 0, top to bottom.
Open the Live page by hand and dump the tree:
```bash
adb shell uiautomator dump /sdcard/ui.xml
adb shell cat /sdcard/ui.xml
```
Content cards are the clickable rows below the title area. The module finds them by
geometry and ignores chrome (Search, Profile, Cast, drawer). Confirm your indices by
running a capture and checking the tap y-coordinate changes between channels:
```text
[*] tap card #0 @ 540,732
[*] tap card #1 @ 540,1494
```
If the wanted index is off screen the module scrolls and re-counts, so a long list
still works.
### `channels.*.phone_only`
Set it for a channel that has no catalog entry. `drm catalog` then fails with a
message pointing at phone capture instead of a confusing "no video id" error.
### `ui.launch_resource_contains`
This app's home screen exposes almost no useful `content-desc` — only chrome like
Search, Profile, Cast and the drawer button — so readiness is keyed off **view
ids**, which are stable and not localized.
```bash
adb shell uiautomator dump /sdcard/ui.xml
adb shell cat /sdcard/ui.xml | grep -oE 'resource-id="[^"]*"' \
| sed 's/.*\///' | sort -u
```
Pick a container that only exists once the home screen has rendered, e.g. the
content rail's refresh layout or the drawer toolbar.
This matters: without a marker that actually matches, every capture burns the full
launch timeout before auto-play recovers. It still works, just 30 s slower each time.
### `score.hosts` / `score.deny`
`hosts` is the live CDN host that serves manifests. `deny` is for analytics — this
provider's metrics beacons carry `.m3u8` in a query string and otherwise score as
manifests.
```bash
grep -oE '"host":"[^"]+"' outputs/discover/*/appproxy_traffic.jsonl \
| sort | uniq -c | sort -rn | head -20
```
## Verify
Catalog path first — it needs no phone:
```bash
./bin/drm modules
./bin/drm catalog --app tg4 --list
```
```text
CHANNEL CATALOG ID LABEL
chanone 6383454185112 Main
kids 6383455473112 Kids
sport - Sport
provider flags: cula4Enabled=true liveEnabled=true ...
```
A `-` in CATALOG ID means neither `video_id` nor `stream_id_field` resolved.
`provider flags` are the booleans from `playback_config.json`, handy for spotting a
channel the provider has switched off.
Then resolve one for real:
```bash
./bin/drm catalog --app tg4 --channel chanone --keys --wvd data/device.wvd
```
Expect an HLS DVR manifest, a `/lic/wv?token=` license URL, a PSSH, and several
`KID:KEY` lines.
Finally, cross-check the phone path against it — independent routes agreeing on the
PSSH and the full key set is strong evidence both are configured correctly:
```bash
./bin/drm catalog --app tg4 --channel chanone --keys --wvd data/device.wvd --json > a.json
./bin/drm capture --app tg4 --channel chanone --auto-play --wvd data/device.wvd
diff <(jq -r '.keys[]' a.json | sort) <(sort outputs/tg4/latest/key.txt)
```
## Optional tuning
```yaml
stream_name_prefix: tg4 # prefix for generated stream names
cards:
min_y: 300 # ignore nodes above this (title area / chrome)
min_h: 250
min_w: 600
desc_deny: ["Search", "Profile", "Cast", "Open", "Closed"]
text_deny: ["catch-up"]
ui:
live_label: live # text identifying the Live menu entry and title
live_title_resource: <view id of the Live page title>
drawer_desc: ["Open", "Closed"]
drawer_fallback_x: 68 # tapped when the drawer button cannot be found
drawer_fallback_y: 183
menu_max_x: 700 # bounds for a drawer menu entry
menu_min_y: 250
menu_max_y: 700
playback_desc_contains: ["video player"]
playback_resource_contains: [<player view id>]
timeouts:
launch_wait_ui: 30s
live_nav: 25s
card: 25s
playback: 60s
```
The player often starts without a dumpsys audio session, so playback is confirmed
by `playback_resource_contains` / `playback_desc_contains` and the wait is
soft-failing: capture continues even if neither signal appears.
## Troubleshooting
| Symptom | Cause |
|---|---|
| `INVALID_POLICY_KEY` (HTTP 401) | wrong or rotated `policy_key` |
| `no playback token for channel "x" (field Y)` | `token_field` is not a key in `playback_config.json` |
| `no video id for channel "x"` | set `video_id`, or a `stream_id_field` that exists |
| `no HLS source in playback response` | the channel is off air, or the token is for a different region |
| `launch: ui node not found` | `ui.launch_resource_contains` does not match — re-dump the tree |
| `tap card #N` picks the wrong channel | card order changed; re-check `live_card_index` |
| Fewer keys than expected | a multi-KID stream needs every key; check `key.txt`, not just `key` |

252
docs/quickstart.md Normal file
View file

@ -0,0 +1,252 @@
# Quickstart
From an empty directory to a captured Widevine key.
## 1. Prerequisites
| Need | Version | Needed for |
|---|---|---|
| Go | 1.25+ | everything |
| Python | 3.10+ | the CDM helper (`apps/wvkey/wvkey.py`) |
| adb | any recent platform-tools, on `PATH` | phone capture only |
| `.wvd` device file | — | fetching content keys |
| Rooted Android phone (Magisk) | — | phone capture only |
Check Go:
```bash
go version
```
If Go is installed but not on `PATH`, set `GOROOT` and prepend its `bin`:
```bash
export GOROOT=/path/to/go
export PATH="$GOROOT/bin:$PATH"
```
## 2. Clone and build
```bash
git clone https://git.nulldrm.com/nulldrm/Null-DRM-Official.git
cd Null-DRM-Official
go -C apps/cli build -o ../../bin/drm . # Windows: -o ../../bin/drm.exe
```
That single command builds the whole toolchain: capture, catalog, agent, the
streamd control plane, the MITM host CLI, and every app module.
Verify:
```bash
./bin/drm help
./bin/drm modules
```
Expected on a fresh clone — modules exist, values do not:
```text
APP PACKAGE KEY MODE REWRITER CHANNELS VALUES
rte (not configured) modulardrm rte 0 missing: .../apps/modules/rte/module.yaml
tg4 (not configured) raw none 0 missing: .../apps/modules/tg4/module.yaml
```
## 3. The Python CDM helper
Only `wvkey.py` is Python; it turns a PSSH + license URL into `KID:KEY` lines.
```bash
python -m venv .venv
.venv/bin/pip install -r apps/wvkey/requirements.txt
```
On Windows:
```powershell
python -m venv .venv
.venv\Scripts\pip install -r apps\wvkey\requirements.txt
```
The binary finds `.venv` automatically; override with `--python /path/to/python`.
Place your Widevine device file under `data/` (gitignored):
```text
data/device.wvd
```
## 4. Add values for a module
Create the module's values file. It is gitignored, so it never leaves your machine.
```bash
$EDITOR apps/modules/tg4/module.yaml
```
Minimum for TG4 — see [providers/tg4.md](providers/tg4.md) for where each comes from:
```yaml
package: <android package id>
account_id: "<brightcove account id>"
policy_key: "BCpkAD..."
playback_config_url: "https://.../playback_config.json"
channels:
ioi:
label: "Main channel"
video_id: "<brightcove video id>"
token_field: <field name in the playback config>
live_card_index: 0
```
Confirm it loaded:
```bash
./bin/drm modules
./bin/drm catalog --app tg4 --list
```
Nothing stored? Pass values per run instead:
```bash
./bin/drm catalog --app tg4 --channel ioi \
--tg4.account-id 1234567890 --tg4.policy-key 'BCpkAD...'
```
Or through the environment (`<APP>_<FIELD>`, upper snake case):
```bash
export TG4_POLICY_KEY='BCpkAD...'
```
Precedence is **flag → environment → module.yaml → built-in default**. Only
mechanical defaults (timeouts, DASH timescales, card geometry) have built-ins.
## 5. Get keys without a phone
If the app has a public playback catalog:
```bash
./bin/drm catalog --app tg4 --channel ioi --keys --wvd data/device.wvd
```
```text
manifest: https://.../playlist-hls-dvr.m3u8
license_url: https://.../lic/wv?token=...
pssh: AAAAYHBzc2gAAAAA7e+LqXnWSs6j...
key: 27c38b955b983ac2827204f8d260d628:f31efbc58fc89526cce5e5dee50a7ebd
key[0]: 5b9d58edf81f3b5b9c3cc61f3e81845e:688236976dcf8cdde5cdfac4a698f256
...
```
Add `--json` for a machine-readable object, `--out` to write a session directory
without fetching keys.
## 6. Get keys from the phone
One-time device setup:
```bash
adb devices # your phone must be listed and authorised
./bin/drm proxy build # cross-compiles the on-device MITM (linux/arm64)
./bin/drm proxy install-ca --reinject
```
`install-ca` pushes the MITM CA and Magisk-mounts it into the system trust store.
Without it apps fail TLS with `unknown certificate`. See [capture.md](capture.md)
if the reinject does not report success.
Then capture:
```bash
./bin/drm capture --app rte --channel rteone --auto-play --wvd data/device.wvd
```
What it does: start the on-device MITM → point the phone's HTTP proxy at it →
reinject the CA → launch the app → drive its UI to the channel → wait for the
license/manifest traffic → run the CDM → write a session → close the app.
```text
[*] Launching rte (...)
[*] Auto-play RTE One…
[*] tap_ui "Live"/"Live tab, 2 out of 5" @ 324,2253
[*] tap Play @ 540,720
[+] ... is PLAYING
[+] Capture:
pid: 8T8Ohq7XT5KN
mpd: https://dai.google.com/linear/dash/pa/event/...
[+] key: df1633821dddfdd5bec9822c1ec0f052:b6324f76d2bb4dfe63e4272f803274f4
[+] session: outputs/rte/20261005-215448/session.json
```
Keep the screen on for the whole run:
```bash
adb shell svc power stayon true
```
Drop `--auto-play` to launch the app and navigate by hand, or drop `--app`
entirely for passive mode — the MITM records whatever you play.
## 7. Control plane
```bash
./bin/drm serve --bind 127.0.0.1:8083 --data .cache/streamd --token SECRET
```
Open <http://127.0.0.1:8083>. The app and channel pickers are populated from
`GET /api/apps`, i.e. from the modules compiled into the binary.
Create a stream from a capture:
```bash
./bin/drm catalog --app tg4 --channel ioi --keys --wvd data/device.wvd --json \
> /tmp/s.json
curl -X POST http://127.0.0.1:8083/api/streams \
-H 'Authorization: Bearer SECRET' -H 'Content-Type: application/json' \
--data @/tmp/s.json
```
## 8. Agent
The agent polls streamd, queues any enabled stream that is down or missing
credentials, claims a free phone, re-captures it and POSTs the result back.
```bash
export STREAMD_TOKEN=SECRET
./bin/drm agent run --config apps/agent/agent.yaml
./bin/drm agent devices # which phones it can use
./bin/drm agent status # recent jobs
./bin/drm agent enqueue --stream tg4-ioi --reason manual
```
Details in [streamd.md](streamd.md).
## 9. Tests
```bash
go -C apps/pkg test ./...
go -C apps/modules test ./...
go -C apps/streamd test ./...
```
Live-origin tests are tag-gated and need your local values:
```bash
go -C apps/modules test -tags live ./rte/ -v
```
## Troubleshooting
| Symptom | Cause |
|---|---|
| `unknown app "x"` | not compiled in — check `apps/modules/all/all.go`, then rebuild |
| `channels 0` in `drm modules` | no `module.yaml`, or `channels:` is empty |
| `unknown channel "y"` | id not in `channels:` and not in `aliases:` |
| `tls: unknown certificate` on the phone | CA not mounted — `drm proxy install-ca --reinject` |
| `wvkey requires --wvd` | pass `--wvd path/to/device.wvd` |
| `timed out waiting for capture` | playback never started, or the app used a path the MITM missed — see [capture.md](capture.md) |
| `INVALID_POLICY_KEY` | wrong or stale `policy_key` |

183
docs/streamd.md Normal file
View file

@ -0,0 +1,183 @@
# Control plane and agent
Two long-running pieces: `drm serve` holds the state and the dashboard, `drm agent
run` keeps credentials fresh using real phones.
## drm serve
```bash
./bin/drm serve --bind 127.0.0.1:8083 --data .cache/streamd --token SECRET
```
| Flag | Default | Purpose |
|---|---|---|
| `--bind` | `0.0.0.0:8083` | listen address |
| `--data` | `.cache/streamd` | SQLite, work dirs, HLS output, logs |
| `--token` | `$STREAMD_TOKEN` | bearer token for mutating routes |
| `--nre` | `$NRE_PATH`, `bin/`, `PATH` | downloader binary |
| `--ffmpeg` | `$FFMPEG_PATH`, `bin/`, `PATH` | ffmpeg |
| `--mp4decrypt` | `$MP4DECRYPT_PATH`, `bin/`, `PATH` | decrypter |
Open <http://127.0.0.1:8083>. GET routes are unauthenticated; anything mutating
needs `Authorization: Bearer <token>`.
### API
| Route | Purpose |
|---|---|
| `GET /api/health` | liveness + uptime |
| `GET /api/apps` | compiled-in modules, their channels, registered rewriters |
| `GET /api/streams` | all streams with runtime state |
| `POST /api/streams` | create |
| `PATCH /api/streams/:id` | update |
| `DELETE /api/streams/:id` | remove |
| `POST /api/streams/:id/start\|stop\|restart` | desired state |
| `POST /api/streams/:id/credentials` | push a fresh manifest + keys |
| `POST /api/streams/:id/claim` | agent claims the stream |
The dashboard's app and channel pickers are filled from `/api/apps`, so they reflect
whatever modules the binary was built with. Nothing about a provider is hardcoded in
the UI or the schema.
### Creating a stream
```bash
./bin/drm catalog --app tg4 --channel ioi --keys --wvd data/device.wvd --json > s.json
curl -X POST http://127.0.0.1:8083/api/streams \
-H 'Authorization: Bearer SECRET' -H 'Content-Type: application/json' \
--data @s.json
```
A stream created without `app`, `video_select`, `audio_select` or `rewriter` keeps
those empty on purpose. They are resolved when the stream starts:
```text
stored row -> the stream's app module (app.StreamDefaults) -> neutral defaults
```
So a module decides its own downloader settings, an operator override in the row
still wins, and an unknown app gets something harmless rather than another
provider's preferences.
### Refreshing credentials
```bash
curl -X POST http://127.0.0.1:8083/api/streams/1/credentials \
-H 'Authorization: Bearer SECRET' -H 'Content-Type: application/json' \
-d '{"mpd":"https://...","key":"KID:KEY\nKID:KEY","headers_json":"{}"}'
```
This is exactly what the agent does after a successful capture.
### Manifest rewriting
If a stream's `rewriter` names a registered rewriter and the manifest is not HLS,
the supervisor starts a local rewrite server and points the downloader at it:
```text
stream rewriter="rte" -> mpd.Lookup("rte") -> http://127.0.0.1:PORT/manifest.mpd
```
The log line is `stream X: rewritten MPD http://127.0.0.1:…`. Streamd never imports
a provider package to do this — the module registered the rewriter under that name
at init time.
## drm agent run
```bash
export STREAMD_TOKEN=SECRET
./bin/drm agent run --config apps/agent/agent.yaml
```
What it does, in a loop:
1. poll streamd for enabled streams that are down or missing credentials
2. enqueue a job in a durable SQLite queue
3. claim a free ADB device that already has the app installed
4. run a phone capture through the stream's app module
5. `POST /api/streams/:id/credentials`
6. force-stop the app and release the device
It never installs apps and never runs a downloader. A websocket heartbeat keeps its
claims alive, so claims expire automatically if the agent dies.
### Subcommands
```bash
./bin/drm agent devices # phones it can use
./bin/drm agent status # recent jobs
./bin/drm agent enqueue --stream tg4-ioi --reason manual
./bin/drm agent cancel --job 3
```
### Config
`apps/agent/agent.yaml`:
```yaml
streamd_url: http://127.0.0.1:8083
# token: taken from $STREAMD_TOKEN when unset
poll_interval_sec: 30
data_dir: .cache/agent
workers: 1
max_attempts: 5
capture_wait_sec: 180
# devices: [SERIAL1, SERIAL2] # empty = any authorised device
# python: path to python for wvkey.py
# wvd: data/device.wvd
# user_agent: ""
# agent_id: defaults to hostname
```
There is no modules directory to configure — app modules are compiled into the
binary.
`wvd` must be set (here or on the phone capture) or key fetches fail.
## Running both locally
```bash
# terminal 1
./bin/drm serve --bind 127.0.0.1:8083 --data .cache/streamd --token SECRET
# terminal 2
export STREAMD_TOKEN=SECRET
./bin/drm agent run --config apps/agent/agent.yaml
```
Then create a stream, enable it, and watch the agent pick it up:
```bash
./bin/drm agent status
```
## External tools
The media supervisor shells out to three binaries. Put them on `PATH`, in `bin/`,
or name them with flags/env:
| Tool | Flag | Env |
|---|---|---|
| N_m3u8DL-RE | `--nre` | `NRE_PATH`, `N_M3U8DL_RE` |
| ffmpeg | `--ffmpeg` | `FFMPEG_PATH`, `FFMPEG` |
| mp4decrypt | `--mp4decrypt` | `MP4DECRYPT_PATH`, `MP4DECRYPT` |
Streams can be created, claimed and credentialed without any of them; only actually
pulling media needs them. The supervisor is the least exercised part of the system —
treat its health reporting as control-plane bookkeeping.
## Data
```text
.cache/streamd/
streamd.db SQLite: streams, runtime, claims
work/<name>/ downloader scratch + keys.txt
www/<name>/ HLS output
logs/ per-stream downloader + ffmpeg logs
.cache/agent/
queue.db durable job queue
```
Back up `streamd.db` before upgrading — schema defaults change between versions.