fix(config): track the API's raised daily allowance (100000 an account, 5000 an IP)

README.md

@@ -320,20 +320,25 @@ Scheduler, cron and container arrangements.
 
 ### Large libraries
 
-TheIntroDB allows 1000 requests per UTC day with an API key, and 500 without one.
-A library of tens of thousands of items cannot be scanned in a day, so the tool
-is built to be left running rather than to be finished in one pass:
+TheIntroDB allows 100000 requests per UTC day with an API key, and 5000 without
+one. With a key that is more than a full pass over a large library needs, so a
+run is bounded by pacing rather than by the allowance: the client sends at most
+25 requests per 10 seconds, which puts 40,000 items at about four and a half
+hours at that rate. Without a key the allowance is worked through over several
+runs.
+
+What the tool is built around is that each item costs one request, ever:
 
 - **An item is scanned once.** Every lookup is recorded, and a recorded item is
   answered from the ledger afterwards without a request, however long ago it was
   scanned. Expiring that record is what would make a large library never finish,
   so nothing expires it.
-- **A run uses the whole day's allowance.** It scans the items it has never seen
-  until the allowance is spent, then stops cleanly and logs how many are left.
-  That is not an error, and the nightly timer keeps its schedule.
+- **A run spends the allowance on items it has never seen**, then stops cleanly
+  and logs how many are left. That is not an error, and the nightly timer keeps
+  its schedule.
 - **The next run continues.** Because every scan is recorded, it picks up exactly
-  where the last one stopped. A 40,000-item library at 1000 requests a day
-  converges in about 40 days, and each run writes the markers it has data for.
+  where the last one stopped, so an interrupted or allowance-stopped run costs
+  only the items it never reached.
 - **Re-scanning is deliberate.** Nothing is asked about twice on a timer. When a
   submission lands and you want fresh answers, name the items:
 
@@ -371,7 +376,7 @@ Lookup order: `$PLEX_SYNC_CONFIG`, `./plex-sync.toml`,
 | `plex.device_name` | `plex-sync` | what Plex shows for this tool in its device list |
 | `plex.client_id` | stored in `state_dir` | the identity Plex keys this install's device entry on |
 | `theintrodb.api_key` | | optional TheIntroDB API key |
-| `theintrodb.daily_budget` | `1000` | requests per UTC day |
+| `theintrodb.daily_budget` | `100000` | requests per UTC day |
 | `sources.chapters` | `false` | use chapter names as a source |
 | `sources.detection` | `false` | run local fingerprint detection |
 | `segments.intro` | `true` | write intro markers |

docs/scheduling.md

@@ -21,9 +21,15 @@ so `journalctl`, the Docker log or a file all work as a record of what happened.
 
 ## Large libraries, and the daily allowance
 
-TheIntroDB allows 1000 requests per UTC day with an API key and 500 without one,
-so a library of tens of thousands of items is not scanned in one night. That is
-expected, and the schedule is how it is meant to be worked through:
+TheIntroDB allows 100000 requests per UTC day with an API key and 5000 without
+one. With a key that covers a full pass over a large library — 40,000 items need
+40,000 requests, and the client's default pacing limit of 25 requests per 10
+seconds puts that at about four and a half hours at that rate, longer if
+responses are slow or `theintrodb.max_per_window` is set lower — so a nightly
+schedule is enough. Without a key the allowance has to be worked through over
+several nights.
+
+The rules that make either case finish:
 
 - An item that has been looked up is recorded, and a recorded item is answered
   from the ledger afterwards without a request. Nothing expires that record, so
@@ -34,8 +40,10 @@ expected, and the schedule is how it is meant to be worked through:
   that reported an error every night would be a timer nobody reads.
 - The next run continues from the same place, because every scan is recorded.
 
-A schedule therefore converges: a 40,000-item library at 1000 requests a day
-gets through in about 40 days, writing the markers it has data for each night.
+A schedule therefore converges: every item is asked about once, so a run that
+stops — on the clock, on the allowance, or by being interrupted — costs only the
+items it never reached, and the next one carries on from there. With a key, a
+40,000-item library usually finishes in the first night.
 Raising `theintrodb.daily_budget` beyond the allowance does not help — the
 server refuses the request either way; the budget is what stops the client
 before the refusal.

docs/troubleshooting.md

@@ -229,14 +229,15 @@ which puts them back.
 ## "budget" or "usage-limited" in a plan
 
 The day's allowance is spent. The run stops cleanly and resumes tomorrow. Without
-an API key the allowance is 500 requests a day instead of 1000; a key on your
+an API key the allowance is 5000 requests a day instead of 100000; a key on your
 TheIntroDB account raises it.
 
 ## Running out of allowance every day
 
-Expected on a large library: a full pass needs more requests than a day allows.
-Progress accumulates, because every answer including a 404 is cached, so each
-day works on new content.
+Expected without a key, or on a library big enough that one day's allowance does
+not cover it: a full pass needs more requests than a day allows. Progress
+accumulates, because every answer including a 404 is recorded, so each day works
+on new content.
 
 ## Undo
 

internal/config/config.go

@@ -211,7 +211,7 @@ func Default() *Config {
 		TheIntroDB: TheIntroDB{
 			BaseURL:      "https://api.theintrodb.org/v4",
 			TimeoutS:     20,
-			DailyBudget:  1000,
+			DailyBudget:  AccountDailyBudget,
 			MaxPerWindow: 25,
 			WindowS:      10,
 			MissTTLDays:  14,
@@ -272,19 +272,23 @@ func (t TheIntroDB) MinDelay() float64 {
 	return t.WindowS / float64(t.MaxPerWindow)
 }
 
-// AnonymousDailyBudget is the allowance without an API key: 500 per public IP
-// per UTC day, against 1000 for an account.
-const AnonymousDailyBudget = 500
+// AnonymousDailyBudget is the allowance without an API key: 5000 per public IP
+// per UTC day, against 100000 for an account.
+//
+// The numbers are the API's own (UsageLimitMediaPublicIP and
+// UsageLimitMediaAuthenticated, and the X-UsageLimit-Limit header it sends
+// back), and they are higher than they were: an account used to get 1000 a day.
+const AnonymousDailyBudget = 5000
 
 // AccountDailyBudget is the allowance with an API key, and the highest any
 // caller can have.
-const AccountDailyBudget = 1000
+const AccountDailyBudget = 100000
 
 // EffectiveDailyBudget is the allowance that actually applies.
 //
-// Budgeting for 1000 requests when only 500 are allowed means discovering the
-// limit by being rate-limited for the rest of the day, so an anonymous client
-// budgets for the anonymous allowance. A non-positive budget still means
+// Budgeting for 100000 requests when only 5000 are allowed means discovering
+// the limit by being rate-limited for the rest of the day, so an anonymous
+// client budgets for the anonymous allowance. A non-positive budget still means
 // "uncounted".
 func (t TheIntroDB) EffectiveDailyBudget() int {
 	if t.DailyBudget <= 0 {
@@ -674,10 +678,10 @@ func (c *Config) Validate() error {
 	if c.TheIntroDB.DailyBudget <= 0 {
 		problems = append(problems, "theintrodb.daily_budget must be positive")
 	}
-	// An account allowance of 1000 is the highest anyone can have, so a bigger
-	// number is not a bigger budget, it is a number that cannot be true. Left
-	// unchecked it would sit in the file looking like an allowance nobody will
-	// honour.
+	// An account allowance of 100000 is the highest anyone can have, so a
+	// bigger number is not a bigger budget, it is a number that cannot be
+	// true. Left unchecked it would sit in the file looking like an allowance
+	// nobody will honour.
 	if c.TheIntroDB.DailyBudget > AccountDailyBudget {
 		problems = append(problems, fmt.Sprintf(
 			"theintrodb.daily_budget must not exceed the ceiling of %d requests a day",
@@ -871,7 +875,7 @@ client_id = ""
 # The TheIntroDB API key is optional. With a key the daily allowance is higher
 # and your own pending submissions are included in what you get back.
 api_key = ""
-daily_budget = 1000
+daily_budget = 100000
 
 [sources]
 # Alternate marker sources. TheIntroDB always wins a segment type it answered;

internal/config/config_test.go

@@ -65,15 +65,23 @@ func TestPacingStaysUnderTheServerCeiling(t *testing.T) {
 }
 
 func TestAnonymousClientBudgetsForTheAnonymousAllowance(t *testing.T) {
-	anonymous := TheIntroDB{DailyBudget: 1000}
+	// A budget above the anonymous allowance is not a bigger budget without a
+	// key: the client would discover the real limit by being rate-limited.
+	anonymous := TheIntroDB{DailyBudget: AccountDailyBudget}
 	if got := anonymous.EffectiveDailyBudget(); got != AnonymousDailyBudget {
 		t.Errorf("without a key the budget is %d, want %d: budgeting for more means "+
 			"discovering the limit by being rate-limited", got, AnonymousDailyBudget)
 	}
 
-	withKey := TheIntroDB{DailyBudget: 1000, APIKey: "abc"}
-	if got := withKey.EffectiveDailyBudget(); got != 1000 {
-		t.Errorf("with a key the budget is %d, want the configured 1000", got)
+	// A budget under the allowance is what the caller asked for, and stays.
+	modest := TheIntroDB{DailyBudget: 120}
+	if got := modest.EffectiveDailyBudget(); got != 120 {
+		t.Errorf("a budget below the allowance became %d, want 120", got)
+	}
+
+	withKey := TheIntroDB{DailyBudget: AccountDailyBudget, APIKey: "key"}
+	if got := withKey.EffectiveDailyBudget(); got != AccountDailyBudget {
+		t.Errorf("with a key the budget is %d, want the configured %d", got, AccountDailyBudget)
 	}
 
 	// A non-positive budget means "do not count", and must stay that way.
@@ -83,6 +91,40 @@ func TestAnonymousClientBudgetsForTheAnonymousAllowance(t *testing.T) {
 	}
 }
 
+// The allowances are the API's, not invented here, and when the API raises them
+// these are the numbers that move. An account used to get 1000 requests a day;
+// raising it without raising the ceiling below made the real allowance
+// unconfigurable.
+func TestTheAllowancesMatchTheAPI(t *testing.T) {
+	if AccountDailyBudget != 100000 {
+		t.Errorf("account allowance = %d, want the API's 100000 (UsageLimitMediaAuthenticated)",
+			AccountDailyBudget)
+	}
+	if AnonymousDailyBudget != 5000 {
+		t.Errorf("anonymous allowance = %d, want the API's 5000 (UsageLimitMediaPublicIP)",
+			AnonymousDailyBudget)
+	}
+}
+
+// The ceiling has to accept the allowance the API actually grants, or a config
+// naming the real allowance is refused as a number that cannot be true.
+func TestTheBudgetCeilingAcceptsTheAccountAllowance(t *testing.T) {
+	cfg := Default()
+	if got := cfg.TheIntroDB.DailyBudget; got != AccountDailyBudget {
+		t.Errorf("the default budget is %d, want the account allowance %d", got, AccountDailyBudget)
+	}
+
+	cfg.TheIntroDB.DailyBudget = 100000 // the API's usage limit for an account
+	if err := cfg.Validate(); err != nil {
+		t.Errorf("the account allowance must be configurable: %v", err)
+	}
+
+	cfg.TheIntroDB.DailyBudget = 100001
+	if err := cfg.Validate(); err == nil {
+		t.Error("a budget above the account allowance must be refused")
+	}
+}
+
 func TestLoadReadsFileEnvironmentAndDefaults(t *testing.T) {
 	dir := t.TempDir()
 	path := filepath.Join(dir, "config.toml")

internal/ledger/ledger.go

@@ -71,7 +71,7 @@ var schemaStatements = []string{
 	//
 	// The cache answers "do we still trust this body"; this answers "have we
 	// spent a request on this item yet". On a library of tens of thousands of
-	// items against a daily allowance of 500 or 1000, the second question is
+	// items against a daily allowance of 5000 or 100000, the second question is
 	// the one that matters: a run must spend its requests on items it has never
 	// seen, and must be able to stop and pick up where it left off tomorrow.
 	// Expiring these rows is exactly what would make a large library never

internal/tidb/client.go

@@ -9,7 +9,7 @@
 // wrong:
 //
 //   - A 429 is either the rate window (30 requests per 10 seconds) or the daily
-//     allowance (1000 per account, 500 per public IP, reset at UTC midnight).
+//     allowance (100000 per account, 5000 per public IP, reset at UTC midnight).
 //     A rate-limit 429 carries only a Retry-After in seconds. A usage-limit 429
 //     carries X-UsageLimit-Reset in SECONDS UNTIL UTC MIDNIGHT, which can be up
 //     to about 86400, plus a JSON body with a code of usage_limit_exceeded or
@@ -229,7 +229,7 @@ type Client struct {
 // own default, and a non-positive DailyBudget means "uncounted". A nil ledger
 // disables caching and budget counting; a nil httpclient gets a default one.
 func NewClient(cfg config.TheIntroDB, led *ledger.Ledger, hc *httpclient.Client) *Client {
-	// Without a key the public allowance is 500 requests a day, not 1000.
+	// Without a key the public allowance is 5000 requests a day, not 100000.
 	// Budget for the allowance that actually applies rather than discovering it
 	// by being rate-limited for the rest of the day.
 	cfg.DailyBudget = cfg.EffectiveDailyBudget()

internal/tidb/client_test.go

@@ -737,7 +737,7 @@ func TestUsageLimitWaitIsNotClampedToFiveMinutes(t *testing.T) {
 
 	h := newHarness(t, func(call int, w http.ResponseWriter, r *http.Request) {
 		w.Header().Set("X-UsageLimit-Reset", fmt.Sprint(reset))
-		w.Header().Set("X-UsageLimit-Limit", "1000")
+		w.Header().Set("X-UsageLimit-Limit", "100000")
 		w.Header().Set("X-UsageLimit-Remaining", "0")
 		js(w, 429, `{"error":"daily limit reached","retry_after":"2 hours","code":"usage_limit_exceeded"}`)
 	}, nil)

internal/tidb/lookup.go

@@ -63,7 +63,7 @@ func (r LookupResult) OK() bool { return r.Status == 200 || r.Status == 404 }
 // The order is deliberate. First, the item's scan record: if it has been looked
 // up before, the answer stored then is returned and no request is made, however
 // long ago that was. A library of tens of thousands of items against a daily
-// allowance of 500 or 1000 only ever finishes if each item is asked about once,
+// allowance of 5000 or 100000 only ever finishes if each item is asked about once,
 // and an item that was scanned is what "finished" means; the way to ask again
 // is LookupForced, which is what a plan that names the item for a re-scan does.
 //

internal/tui/settings.go

@@ -224,7 +224,7 @@ func settingsRows() []setting {
 		},
 		{
 			section: "TheIntroDB", key: "theintrodb.daily_budget", label: "daily budget", kind: settingInt,
-			help: "Requests per UTC day. The ceiling is 500 without a key and 1000 with one.",
+			help: "Requests per UTC day. The ceiling is 5000 without a key and 100000 with one.",
 			get:  intGet(func(c *config.Config) int { return c.TheIntroDB.DailyBudget }),
 			set:  intSet(func(c *config.Config, n int) { c.TheIntroDB.DailyBudget = n }),
 		},

internal/tui/settings_test.go

@@ -3,6 +3,7 @@ package tui
 import (
 	"os"
 	"path/filepath"
+	"strconv"
 	"strings"
 	"testing"
 
@@ -258,7 +259,11 @@ func TestARejectedValueIsNotApplied(t *testing.T) {
 	for range m.buffer {
 		press(m, "backspace")
 	}
-	typeText(m, "99999") // above the server's ceiling
+	// One above the ceiling the API grants. Taken from the constant so this case
+	// cannot quietly become a valid value when the allowance is raised -- which
+	// is what happened when it was hardcoded to 99999 and the ceiling moved to
+	// 100000.
+	typeText(m, strconv.Itoa(config.AccountDailyBudget+1))
 	press(m, "enter")
 
 	if !m.editing {

internal/tui/view.go

@@ -108,7 +108,7 @@ func (m *Model) statusScreen() string {
 		b.WriteString("  Plex          " + m.ok(m.readiness.PlexOK, m.readiness.PlexVersion, m.readiness.PlexError) + "\n")
 		status := "present"
 		if cfg.TheIntroDB.APIKey == "" {
-			status = "absent, 500 requests a day instead of 1000"
+			status = "absent, 5000 requests a day instead of 100000"
 		}
 		b.WriteString("  TheIntroDB    " + m.ok(m.readiness.TIDBOK, status, m.readiness.TIDBError) + "\n")
 	}