Skip to content

Spec 002: the report made in the background, behind its address (R4) #104

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
adamjohnwright wants to merge 4 commits into from
Closed

Spec 002: the report made in the background, behind its address (R4) #104

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
4 commits
Select commit Hold shift + click to select a range
db1ecc4
Spec 002: the report made in the background, behind its address (R4)
adamjohnwright Nov 2, 2026
579af8b
Spec 002 R4, from #98's review: the gateway, memory only, one error r…
adamjohnwright Nov 2, 2026
f9076ef
Spec 002 R4, from #100's review: memory bounded, the working page, fa…
adamjohnwright Nov 2, 2026
5fe6f15
Spec 002 R4, from #103's check: the gateway in FR-210, and a client h…
adamjohnwright Nov 2, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 2 additions & 2 deletions specs/002-analysis-report/checklists/requirements.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -37,8 +37,8 @@
the feature's boundaries.
- The two open questions carried from the request are settled by stated defaults rather
than clarification markers: ReactomeGSA results are in scope as far as today's report
covers them (Assumptions), and the 50-pathway time budget is set by the plan from a
measurement of today's report (SC-203).
covers them (Assumptions), and the time budgets are SC-203's: today's time or 2
minutes for 25 pathways, 4 minutes for 50, to a ready report (research R4).
- Requirement numbers start at FR-201 and SC-201, so they cannot be confused with spec
001's when one refers to the other.
- The constitution's Principle I is amended with this spec (1.1.0) to cover the website's
Expand Down
7 changes: 6 additions & 1 deletion specs/002-analysis-report/contracts/analysis-api.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -3,7 +3,12 @@
The analysis service's public API, under `/AnalysisService`, called directly (not
through the gateway). Research R1 says why these and not others.

In this order (research R1):
**The check,** on every request to the report's address, before anything else of the
analysis is read and before a kept report is sent: call 1 below, with a 5 s timeout. Its
404 or 410 is passed on (and drops a kept report); no answer, or any other failure, is 503. It is the only call a request makes when its report is being made or kept; a
joining request makes no other.

In this order, for a report being made (research R1):

| Call | Request | Used for |
| ---- | ------------------------------------------------------------------------------------------------------------------------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------- |
Expand Down
58 changes: 47 additions & 11 deletions specs/002-analysis-report/contracts/service.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -15,20 +15,56 @@ already call. The address and parameters are today's.
`{species}` is a name (any case, `_` as a space), a taxonomy id or a database id.
`{filename}` is ignored.

| Status | When | Body |
| ---------- | ---------------------------------------------------------------------------------------------------------------------------- | ----------------------------------------------------------------------------------------------- |
| 200 | the report is made | `application/pdf`, `Content-Disposition: attachment; filename="report.pdf"`, sent once complete |
| 404 | the species is not in the graph (checked before the token) | JSON `{ "code": 404, "reason": "Not Found", "messages": [ … ] }` |
| 404 | the analysis service does not know the token | as above |
| 410 | the token's result was deleted by a release | JSON, code 410 |
| as today's | a number or flag that is not one | as today's, recorded by the replay (research R5) |
| 500 | making the PDF fails | JSON, code 500 |
| 503 | the analysis service, the graph or the render service cannot answer, or the service is at its limit, or past its time budget | JSON, code 503, with `Retry-After` |
| Status | When | Body |
| ---------- | ---------------------------------------------------------------------------------------------------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| 200 | the report is ready (made within the request's wait, or earlier) | `application/pdf`, `Content-Disposition: attachment; filename="report.pdf"`, sent once complete |
| 202 | the report is being made and was not ready within the request's wait (30 s by default) | `Retry-After: 10`; JSON `{ "code": 202, "reason": "Accepted", "messages": [ … ] }`, or for a request accepting `text/html` a page that asks the same address again after 10 s |
| 404 | the species is not in the graph (checked before the token) | JSON `{ "code": 404, "reason": "Not Found", "messages": [ … ] }` |
| 404 | the analysis service does not know the token | as above |
| 410 | the token's result was deleted by a release | JSON, code 410 |
| as today's | a number or flag that is not one | as today's, recorded by the replay (research R5) |
| 500 | making the PDF fails | JSON, code 500 |
| 503 | the analysis service, the graph or the render service cannot answer, or the service is at its limit, or past its time budget | JSON, code 503, with `Retry-After`; for a browser, the page that asks again |

- **The order of checks:** the species, then the token, then the rest.
- **No message carries a path, a host, a name or a submitted identifier.**
- **Limits** (FR-210): two reports at once by default, a bounded queue, and a time
budget of what the edge's 100 s leaves after the queues (research R4).
- **In the background** (FR-210, research R4, data-model Job): a request starts the
report, or joins the one queued, being made or kept for the same key (the token, the
species' database id, `number` after clamping, `resource` after defaulting,
`importableOnly`), and waits up to 30 s. The same address answers 202 until the
report is ready, then 200 with the PDF.
- **A report that ends in an error** (FR-203's: 404 or 410 when the token goes while it
is made, 500, or 503 including past its budget) answers
that error to every request then waiting and to the next request, then is forgotten,
so the request after starts it again; unasked, it is forgotten after an hour.
- **Kept** (FR-211): a finished report in the service's memory for one hour, at most
512 MB of them by default, the oldest dropped first. Every request asks the analysis
service about the token (contracts/analysis-api.md, the check): a 404 or 410 drops a
kept report and is answered as such; no answer within 5 s is a 503, the kept report
not sent. Every request reads the graph's release with the species: a change drops
every kept report.
- **Limits** (FR-210): two reports made at once by default and eight more waiting (503
beyond that; a joining request is never counted), and at most two of the eight from
one client, as the gateway names it in `X-Reactome-Client`, a header the gateway always
sets to its own `clientOf` (the client's address, IPv6 counted by its /64) and
overwrites whatever a client sent; pdf-export trusts it because only the gateway can
reach it (loopback or the internal network); `X-Forwarded-For` is not used, as nginx
appends to what a client sends; a time budget of 5 minutes from
when a report starts being made. The render requests of all reports together are
capped (two at once by default), so reports never take every one of the render
service's slots from the website's own images.
- **Caching:** 202, its page and every 503 carry `Cache-Control: no-store`; the PDF
`Cache-Control: private, no-store`.
- **A browser** (`Accept: text/html`) is given, for a 202 and for a 503 with
`Retry-After`, a small page that asks the same address again after that time; for
the 200, the PDF downloads (`Content-Disposition: attachment`) and the page stays as
it was, saying the report is ready.
- **One process:** jobs and kept reports live in the service's process. Several
instances do not share them, so the service runs as one instance; a restart loses
them, and the next request starts the report again.
- **The working page** (FR-211): the printer's HTML, holding the submitted identifiers,
in a private folder on a memory-backed filesystem, removed when the print ends; the
folder is emptied when the service starts.
- **A figure that cannot be drawn** leaves its part without it, logged; the report is
still 200.
- **Characters no font draws:** spec 001 FR-017 for PDFs made on request.
Expand Down
21 changes: 19 additions & 2 deletions specs/002-analysis-report/data-model.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -65,5 +65,22 @@ One of the first `number` pathways of the report's species, in the service's ord
- **Log line** (R7): token, species, number of pathways, stage times, size, characters
replaced.

**State:** none kept between requests. The report is made, sent once complete, and
dropped.
**State:** a report is made in the background (FR-210) and, finished, kept in memory
for an hour (FR-211): see Job.

## Job

A report being made or kept, in the service's process (contracts/service.md).

- **Key:** the token, the species' database id, `number` after clamping, `resource`
after defaulting, `importableOnly`.
- **State:** `queued`, `making`, `ready`, `failed`; with the times it was queued and
started, its deadline (5 minutes from starting), and when it is dropped (an hour
after it ended).
- **Result:** the PDF (ready), or the error that ended it (failed).
- **Waiting:** the requests waiting on it; none is needed for it to go on.
- **Dropped:** an hour after it ended; a failed one also once its error has been
answered; a kept one whose token the analysis service no longer knows; all when the
graph's release changes; the oldest first when the kept reports would pass their
total (512 MB by default). Nothing of it is written to disk but the printer's working
page, on memory-backed storage, while it prints (FR-211).
43 changes: 28 additions & 15 deletions specs/002-analysis-report/plan.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -20,17 +20,17 @@ pdf-export makes the website's analysis report on request, in place of
parts are new templates beside spec 001's (R2).
- **The address:** today's, served by pdf-export's service behind WebsiteAngular's
analysis gateway (R6).
- **The time a cold report takes** is likely over what the edge allows (R4): measured
first, and brought to the user with the choices.
- **The time a cold report takes** is over what the edge allows (R4), so the report is
made in the background behind its address and fetched when ready.

## Technical Context

**Language/Version**: TypeScript on Node 24, as spec 001.

**Primary Dependencies**: spec 001's (its plan's table); nothing new.

**Storage**: none. It reads the analysis service's API, the graph and the render
service, and keeps nothing after sending (R7).
**Storage**: no disk. Jobs and finished reports live in the service's memory, a report
for at most an hour (FR-211, R7).

**Testing**: Node's test runner, with a stub analysis service (spec 001's T010, extended)
and the render stub; the fixture graph (R8).
Expand All @@ -39,15 +39,16 @@ and the render stub; the fixture graph (R8).

**Project Type**: a route in spec 001's web service.

**Performance Goals**: SC-203 against today's report on the same private services; the
budget is what the edge's 100 s leaves after the gateway's wait (R4), likely too little
for a cold report; the plan measures it first.
**Performance Goals**: SC-203 against today's report on the same private services. A
report is made in the background behind its address (FR-210, R4): the edge's 100 s
bounds a request, not a report.

**Constraints**: the analysis service's API only (FR-202); no submitted identifier logged
or kept (FR-211); FR-017 for characters; constitution 1.2.0.
**Constraints**: the analysis service's API only (FR-202); no submitted identifier logged,
nor kept beyond the finished report in memory for an hour (FR-211); FR-017 for characters; constitution 1.2.0.

**Scale/Scope**: up to 50 pathways and 51 figures a report; one report at a time behind
the gateway's lane, two in the service by default.
**Scale/Scope**: up to 50 pathways and 51 figures a report; two made at once and eight
waiting, by pdf-export's own limits (FR-210), with a share per client at the gateway,
which no longer queues report requests (step 8).

## Constitution Check

Expand Down Expand Up @@ -115,8 +116,8 @@ else (constitution I).
- a cold report, stage by stage, on a private stack: the analysis calls (R1's call 4
for a large expression dataset), the figures (spec 001's T005, part c), layout and
printing (twice, for the contents);
- **then the time budget's decision with the user** (R4), before the service is
built to one answer.
- the time budget is decided (R4): the report is made in the background; these
measurements set SC-203's baseline and check its 2- and 4-minute targets.
2. **Spec 001's foundations first:** this spec builds on modules spec 001's tasks have
not built yet: the analysis stub (its T010), the analysis client (T020, T021) and the
service (T022), besides the layout and printing already merged. Its own work starts
Expand All @@ -126,12 +127,24 @@ else (constitution I).
found gathered as today's report gathers them.
5. **Layout:** the report's parts, in spec 001's theme; the genome-wide overview.
6. **The service:** the route, its parameters, its errors in the analysis service's
form, its limits, and stopping the work when the client abandons the request (R7).
form; the job (data-model Job): starting and joining, the 30 s wait, 202 with
`Retry-After` and the browser's page, the PDF once ready, an error answered then
forgotten, the queue and the 5-minute budget, the hour's keeping in memory and its
check of the token, the render requests' shared cap; a request abandoned ending only
its wait (R7).
7. **The checks:** the replay (User Story 2), the comparison by FR-208 with each
figure's overlay checked (quickstart §3), SC-203's timing,
SC-205's search of logs and files; each shown failing.
8. **The website's part** (WebsiteAngular's specs): the gateway's report lane sent to
pdf-export, and the render service's `resource` (the overlay wait is step 0).
pdf-export, and the render service's `resource` (the overlay wait is step 0). The
"PDF Result" button opens the report's address in a new tab rather than downloading
it, so the page shown while it is made (202) is seen, not saved as `report.pdf`. The
gateway's report lane sends report requests to pdf-export, a second upstream, without
queueing them one at a time: with pdf-export holding a request up to 30 s, a lane of
one would turn every browser's polling into a wait for the others. It keeps its share
per client (`PER_CLIENT`), names the client in `X-Reactome-Client` (always overwritten
with its `clientOf`), and answers 503 when pdf-export cannot be
reached (R6). The service's image mounts a `tmpfs` for the printer's working folder.

## Complexity Tracking

Expand Down
16 changes: 12 additions & 4 deletions specs/002-analysis-report/quickstart.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -36,10 +36,18 @@ requests, and no ReactomeGSA runs.
- **The overlays:** each figure in a sample of reports carries the analysis overlay for
the result's default resource (FR-204), shown failing on a figure captured without
it.
- **The time** (SC-203): cold and warm, stage by stage, against the baseline and the
budget the user chose (research R4).
- **The time** (SC-203): cold and warm, stage by stage, from the first request to the
PDF, against the baseline and SC-203's 2 and 4 minutes, one report alone.
- **In the background** (FR-210), each shown failing first: a request past 30 s
answered 202 with `Retry-After`, and the page for `Accept: text/html`; two requests
for one key making one report (the stubs count their calls); the PDF from a kept
report, and none after the hour; a failed report's error given to each waiting
request and the next, then a fresh start; a report abandoned at its budget; a kept
report whose token the analysis stub then answers 410 dropped and 410 answered; kept
reports past the total cap dropped oldest first and made again when asked; a third
report waiting from one client refused while others' are taken.
- **The look** (SC-204): spec 001's colour and font check on a sample report.
- **The readers** (SC-207): the report opens, with its bookmarks and links working, in
spec 001's three readers.
- **What is kept** (SC-205): after a sample of reports, the service's logs and working
folder hold none of the submitted identifiers.
- **What is kept** (SC-205): after a sample of reports, some still kept in memory, the
service's logs and working folder hold none of the submitted identifiers.
Loading
Loading