Skip to content
Closed
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 0 additions & 1 deletion specs/001-pathway-pdfs/contracts/cli.md
Original file line number Diff line number Diff line change
Expand Up @@ -128,5 +128,4 @@ The single-event service (contracts/service.md).
| `--figures` | | as for `book`: the release's figures' folder, read for a PDF without an analysis while it passes `book`'s folder checks, ignored otherwise (contracts/service.md) |
| `--render` | `RENDER_SERVICE` | as for `figures`: draws the figures of a PDF with an analysis, and any the folder lacks |
| `--analysis` | `ANALYSIS_SERVICE` | the analysis service's base URL |
| `--retention` | `TOKEN_RETENTION` | how long an analysis token is kept, in days; must match the analysis service's own setting (research R6) |
| `--concurrency` | | PDFs made at once; default 2 |
11 changes: 6 additions & 5 deletions specs/001-pathway-pdfs/contracts/service.md
Original file line number Diff line number Diff line change
Expand Up @@ -15,15 +15,16 @@ and its error envelope are WebsiteAngular's (FR-014). This is the interface behi
| 200 | the PDF is made | `application/pdf`, with the stable identifier in the `Content-Disposition` file name |
| 400 | `depth` is not 0 or 1, or the identifier names something that is not a pathway or reaction | JSON `{ "reason": "not-an-event" \| "bad-depth" }` |
| 404 | the identifier names nothing in the graph | JSON `{ "reason": "no-event" }` |
| 404 | the token is unknown, malformed, or older than the retention period | JSON `{ "reason": "unknown-token" }` |
| 410 | the token's result is gone within the retention period | JSON `{ "reason": "gone-token" }` |
| 404 | the token is malformed, or the analysis service answers 404 for it | JSON `{ "reason": "unknown-token" }` |
| 410 | the analysis service answers 410 for it: no result, though dated within its 7 days or in the future | JSON `{ "reason": "gone-token" }` |
| 503 | the graph, the render service or the analysis service cannot answer, or the service is at its limit | JSON `{ "reason": "unavailable" }` |

- **The order of checks:** the depth, then the identifier, then the token. An unknown or
non-event identifier is answered as such whatever the token.
- **The token** is checked here before the analysis service is asked (research R6): its
form, and its date against the retention period. A malformed or expired token is
`unknown-token`; the service's 404 is `unknown-token` and its 410 `gone-token`.
- **The token's form** is checked here before the analysis service is asked (research
R6): a malformed token is `unknown-token`. Whether it has expired is the service's to
say: its 404 is `unknown-token`, its 410 `gone-token`, and any other failure
`unavailable`.
- **The reasons** are for the content service, which writes WebsiteAngular's envelope.
They never carry a path, a host or a name.
- **Where a figure comes from:** without a token, from the release's figures' folder
Expand Down
12 changes: 7 additions & 5 deletions specs/001-pathway-pdfs/data-model.md
Original file line number Diff line number Diff line change
Expand Up @@ -139,13 +139,15 @@ From the analysis service's API (research R6).
- `token`, `type`, `species`, `resource` (`TOTAL`), `projection`, `interactors`.
- **Requested with FR-006's defaults:** all resources together, no expression column, all
identifiers rather than only importable ones.
- `identifiersFound`, `identifiersTotal`, `pathwaysHit`. Found from the summary if it
carries it, otherwise from the result file (research R6).
- `identifiersFound`, `identifiersTotal`, `pathwaysHit`. Found is the count of distinct
submitted identifiers in the found identifiers' CSV (header skipped), as
`/token/{token}` carries none; the total adds its `identifiersNotFound`; the pathways
hit are its `pathwaysFound`, asked for with `includeDisease=true` (research R6).
- `found(pathwayStId)`: the entities and interactors found. A reaction uses its first
parent pathway's, filtered to the reaction's own identifiers.
- **Errors:** checked here before the service is asked (research R6): a malformed token
or one past the retention period is unknown; the service's 404 is unknown, its 410
gone. The identifier is checked first.
- **Errors:** the token's form is checked here before the service is asked (research
R6): a malformed token is unknown; the service's 404 is unknown, its 410 gone, and
whether a token has expired is the service's to say. The identifier is checked first.

## Run report

Expand Down
3 changes: 2 additions & 1 deletion specs/001-pathway-pdfs/plan.md
Original file line number Diff line number Diff line change
Expand Up @@ -188,7 +188,8 @@ Dockerfile
- a single-event PDF's stages, with the render service's cache off (R7, SC-005);
- the diagrams' widths against FR-016's floor (R4; measured by T008, which replaced the floor);
- what the analysis service answers for an expired and a malformed token, and its
retention period, on a private instance (R6);
retention period (R6; measured by T009 on a non-production instance: expiry is the
service's to judge, so no period is set here);
- the join's memory, and stamping page numbers after it (R4).
2. **Foundations:**
- the package, TypeScript, lint, Prettier and CI;
Expand Down
4 changes: 2 additions & 2 deletions specs/001-pathway-pdfs/quickstart.md
Original file line number Diff line number Diff line change
Expand Up @@ -34,7 +34,7 @@ research R7.
- **Then, first of the new work** (research R7): the render service's throughput at the
book's scale against the 44-minute figures budget, go or no-go (T005: a no-go, and the
figures moved to a step of their own, T006); the analysis service's
answers for an expired and a malformed token on a private instance (R6); the join's
answers for an expired and a malformed token on a non-production instance (R6); the join's
memory and stamping after it (R4).

## 2. CI's own runs
Expand Down Expand Up @@ -72,7 +72,7 @@ Each opens, has the parts of the spec's Background in order and has bookmarks.
pdf.js at the same scale (research R8): no more than 1% of pixels differ.

- **The errors:** an unknown identifier, a non-event, a bad depth, an unknown token, a
malformed token, an expired token and a token gone within the period each give
malformed token, an expired token and a token gone within the service's 7 days each give
contracts/service.md's answer, and a bad identifier with a bad token gives the
identifier's.

Expand Down
116 changes: 96 additions & 20 deletions specs/001-pathway-pdfs/research.md
Original file line number Diff line number Diff line change
Expand Up @@ -365,43 +365,119 @@ synthetic documents, on this host under load.
strings. A TypeScript reader would depend on the analysis library's internal Java
fields, and would break silently when they changed.
- **The analysis service's API reads the same files by the same rules:**
- 404 for an unknown token, and 410 for one gone within its retention, judged by the
date inside the token;
- 404 or 410 for a token with no stored result, judged by the date inside the token
(measured below);
- `/token/{token}` gives the summary, the resource summary, the not-found count and the
pathways hit;
- `/token/{token}/found/entities/{pathway}` and `/found/interactors/{pathway}` give the
found lists, a 404 meaning none;
- `/download/{token}/result.json` gives the count of identifiers found.
- the count of identifiers found is in neither the summary nor `result.json` (measured
below).
- **Today's errors:** a malformed token throws an unhandled Base64 error, and the
content service answers it with an empty response, as it does the other token errors.
- **The render page** reads the same API for its overlays.

**Measured (T009, 3 Nov 2026)**, against a non-production instance of the analysis
service on release 98, never production. Nothing was deleted from its storage: each case
was made from a token the service reads by its own rules, and one small analysis was
submitted (three accessions that map and one that does not; projection to human, without
and then with interactors). The rules were read from the deployed analysis-core
(3.6.0-SNAPSHOT: `TokenUtils`, `Tokenizer`, `AnalysisResult`) and the analysis service's
own source, and the answers agree with them.

- **A token** is Base64 of `{yyyyMMddHHmmss}_{counter}`, URL-encoded (`=` as `%3D`). The
service URL-decodes it until it no longer changes (Java's `URLDecoder`, which reads `+`
as a space), then Base64-decodes it with Java's basic decoder (padding optional).
- **The service reads the stored result first.** Only when there is none does it judge
the token's date: within 7 days, 410; otherwise, or with no date in it, 404. The 7 is
`DAYS_TO_LIVE`, a constant in analysis-core, not a setting, and counted in whole days
(Joda's `daysBetween`, `<= 7` alive) in the service's time zone (UTC on this instance;
the token's date is written and read in the JVM's default zone): a token dated 7 days
23 hours ago is still 410, one dated 8 days ago 404, and one dated in the future 410.
So the 7 days only tell a missing result's 410 from its 404; they delete nothing.
- **How long a result is kept** is the service's own cleanup, hard-coded in its
configuration (`WebConfig`): every 10 s a checker (reactome-utils'
`LruFolderContentChecker`) measures the results folder, and only once it reaches its
maximum less a threshold (2.5 GB less 500 MB) deletes the files last accessed more than
a week ago, least recently used first, until it is under. (The service's API documents
its 410 as a result deleted for a new data release.) So retention depends on the folder's size and on use, not on a token's age: a
result read recently is kept, and a token older than 7 days whose result is still
stored is answered in full. Read from the code; not seen here, as no stored result
older than 7 days was at hand (five recorded tokens from September and October were
all 404).
- **A stored result that cannot be read** (a class changed after an analysis-core
upgrade, say) is the catch-all 500 (read, not run).
- **The answers,** each `application/json;charset=UTF-8`:

| Case | Status | Body |
| -------------------------------------------------------------------------------------------- | ------ | --------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| a good token, `/token/{token}` | 200 | the summary (below) |
| dated within 7 days (or in the future), no result stored | 410 | `{"code":410,"reason":"Gone","messages":[]}` |
| dated 8 days ago or more, no result stored | 404 | `{"code":404,"reason":"Not Found","messages":[]}` |
| Base64 of text with no date (no `_`) | 404 | as above |
| not Base64 (`not-base64!`, or `QQ=` with the wrong padding), or a URL-decoding error (`%25`) | 500 | `{"code":500,"reason":"Internal Server Error","messages":["Please get in touch with our help desk ({help-desk address}) for more details about this problem"]}` |
| a `/` in the token: encoded (`%2F`) | 400 | the servlet container's own HTML page, not the service's JSON |
| a `/` in the token: raw | — | another route: the framework's own JSON 404, or another endpoint (`{x}/resources`) |
| `found/interactors/{pathway}`, none found (with or without interactors in the analysis) | 404 | the 404 body above |

A malformed token is the service's catch-all 500, the same answer as any failure of its
own, so it cannot be told apart from one by its answer.

- **The summary** (`summary` in `/token/{token}`) holds `token`, `projection`,
`interactors`, `type`, `sampleName`, `text` and `includeDisease`. Beside it,
`/token/{token}` gives `identifiersNotFound` (a count), `pathwaysFound`, `pathways` (a
page of them), `resourceSummary`, `speciesSummary`, `expression` and `warnings`. **None
of them is a count of identifiers found,** nor of identifiers submitted.
- **`pathwaysFound` follows `includeDisease`:** the answer is filtered by the stored
analysis's own setting unless the request names one. For the analysis submitted here,
229 with disease pathways and 191 without. Today's book counts the stored result's
pathways unfiltered (229), as `includeDisease=true` gives; `identifiersNotFound` is the
stored not-found set's size, as today's book's.
- **`/download/{token}/result.json`** has no such count either: its keys are `summary`,
`pathways`, `notFound`, `expressionSummary` and `warnings`, and it was 468 KB for this
four-identifier analysis.
- **`/download/{token}/entities/found/TOTAL/{name}.csv`** lists them: a header row,
then a row per submitted identifier and what it maps to (three columns: submitted
identifier, found identifier, resource), 113 bytes here. It is built from the same map
as today's book's count (`getAnalysisIdentifiers()`, the found entities map's keys,
which are equal when their ids are; interactors are a separate map, left out of both),
so the count of distinct submitted identifiers in it, the header skipped, is that
count. Its values are not quoted. Each request writes a temporary file on the service's
host, cleared within about half an hour.

**Decision.**

- **The token is checked here first,** before the service is asked: its form (the
Base64 the service issues), and the date it carries against the retention period.
- **Only the token's form is checked here,** before the service is asked; whether it has
expired is the service's to say. Judging the date here would refuse a result the
service still answers, and the website still shows, whenever its file outlives 7 days.
- The identifier is checked first: an unknown or non-event identifier is answered as
such whatever the token.
- A malformed token, or one older than the retention period: `unknown-token` (404).
- The service's 404: `unknown-token`. Its 410, for a result gone within the period:
`gone-token` (410).
- What the service itself answers for an expired and a malformed token is confirmed
against a private instance in the first tasks. CI's stub has a case for each.
- **The retention period is a setting** of `serve` (contracts/cli.md), and must match
the analysis service's own: if they drift, a token one calls expired the other may
still answer. The first tasks read the service's period from its configuration and
set the default to it.
- A malformed token: `unknown-token` (404), as the service's answers to one (a 500, a
400 page, another route) would read as `unavailable` or worse. Malformed is anything
but the form the service issues: once URL-decoded until it no longer changes (a
decoding error is malformed), letters and digits only, then at most two `=` making
the length a multiple of 4, and an unpadded length that is not 1 modulo 4. The
service's tokens, Base64 of digits and `_`, never hold `+` or `/`, so the rule refuses
none it issues. The token is sent percent-encoded as one path segment.
- The service's 404: `unknown-token`. Its 410, for a result not stored though its date
is within 7 days: `gone-token` (410). Its other answers, a 500 among them, are
`unavailable`.
- **No retention setting:** the period is the service's constant, and it alone applies
it. CI's stub has a case for each row of the table.
- **The requests carry FR-006's defaults:** all resources together (`TOTAL`), no
expression column, all identifiers rather than only importable ones.
- **The found count:** if `/token/{token}`'s summary carries the identifiers found, it
is read there. Otherwise from `/download/{token}/result.json`, which can be large for
a big analysis. Which one is confirmed in the first tasks.
- **The counts,** as today's book's: the identifiers found are the distinct submitted
identifiers in `/download/{token}/entities/found/TOTAL/{name}.csv` (header skipped),
which grows with what was found, not with the pathways hit as `result.json` does; the
total adds `/token/{token}`'s `identifiersNotFound`; the pathways hit are its
`pathwaysFound`, asked for with `includeDisease=true` so that an analysis submitted
without disease pathways is counted as today's book counts it.
- pdf-export reads the analysis through the analysis service's API, at an address that
is a setting. It is the same service, reading the same files by the same rules, and
the render page reads it too, so a PDF's figures and its text agree.
- FR-007's wording ("from where the analysis service writes it") is amended to "through
the analysis service that holds it".
- In tests, a private instance, or a stub of its answers.
- In tests, a non-production instance, or a stub of its answers.

**Alternatives considered.** A Kryo reader in TypeScript: tied to Java's internals.

Expand Down Expand Up @@ -644,8 +720,8 @@ is ignored, not a missing file.

- **To measure in the first tasks:** the render service's throughput (above); the
diagrams' widths, from the release's diagram files, for FR-016's floor (R4); what the
analysis service answers for an expired and a malformed token, and its retention
period (R6); the join's memory and stamping (R4).
analysis service answers for an expired and a malformed token, and how long it keeps
a result (R6; measured by T009); the join's memory and stamping (R4).

- **Parts:** the largest top-level pathway is 3,760 pages today. At about 200 pages a
part (R4), it is split within its chapter, at section boundaries.
Expand Down
Loading
Loading