Share one numpy.isclose tolerance check between the symbolic and physical quantity contexts

.github/workflows/staging-deploy.yml

@@ -31,19 +31,24 @@ jobs:
       - name: Install dependencies
         run: |
           python -m venv .venv
-          .venv/bin/pip install pytest
+          .venv/bin/pip install -r requirements-dev.txt
           .venv/bin/pip install -r app/requirements.txt
 
 
       - name: Lint with flake8
         run: |
           source .venv/bin/activate
-          .venv/bin/pip install flake8
           # stop the build if there are Python syntax errors or undefined names
           flake8 ./app --count --select=E9,F63,F7,F82 --show-source --statistics
           # exit-zero treats all errors as warnings. The GitHub editor is 127 chars wide
           flake8 ./app --count --exit-zero --max-complexity=10 --max-line-length=127 --statistics
 
+      - name: Type check with mypy
+        run: |
+          source .venv/bin/activate
+          # configuration (checked files, strictness, untyped dependencies) is in mypy.ini
+          mypy
+
       - name: Run tests
         if: always()
         run: |

.github/workflows/test-lint.yml

@@ -28,18 +28,23 @@ jobs:
       - name: Install dependencies
         run: |
           python -m venv .venv
-          .venv/bin/pip install pytest
+          .venv/bin/pip install -r requirements-dev.txt
           .venv/bin/pip install -r app/requirements.txt
 
       - name: Lint with flake8
         run: |
           source .venv/bin/activate
-          .venv/bin/pip install flake8
           # stop the build if there are Python syntax errors or undefined names
           flake8 ./app --count --select=E9,F63,F7,F82 --show-source --statistics
           # exit-zero treats all errors as warnings. The GitHub editor is 127 chars wide
           flake8 ./app --count --exit-zero --max-complexity=10 --max-line-length=127 --statistics
 
+      - name: Type check with mypy
+        run: |
+          source .venv/bin/activate
+          # configuration (checked files, strictness, untyped dependencies) is in mypy.ini
+          mypy
+
       - name: Run tests
         if: always()
         run: |

.gitignore

@@ -137,3 +137,18 @@ app/benchmark.py
 app/profiling_examples.py
 
 .idea/
+
+# Extracted packages live here as their own separate git repo
+/compareExpressions-packages/
+
+# OS / editor / local tool files
+.DS_Store
+.vscode/
+.junie/
+.mcp.json
+
+# Local virtual environments (e.g. .venv38)
+.venv*/
+
+# Test reports written by CI
+reports/

README.md

@@ -20,22 +20,26 @@ This version is specifically for python, however the ultimate goal is to make si
    - [Production API Gateway Integrations](https://eu-west-2.console.aws.amazon.com/apigateway/main/develop/integrations/attach?api=cttolq2oph&integration=qpbgva8&region=eu-west-2&routes=0xsoy4q)
 
 4. Merge commits into the default branch
-   - This will trigger the `test-and-deploy.yml` workflow, which will build the docker image, push it to a shared ECR repository, then call the backend `grading-function/ensure` route to build the necessary infrastructure to make the function available from the client app.
+   - This will trigger the `staging-deploy.yml` workflow (and `production-deploy.yml` on release), which will build the docker image, push it to a shared ECR repository, then call the backend `grading-function/ensure` route to build the necessary infrastructure to make the function available from the client app.
 
 5. You are now ready to start developing your function:
 
    - Edit the `app/evaluation.py` file, which ultimately gets called when the function is given the `eval` command
-   - Edit the `app/evaluation_tests.py` file to add tests which get run:
-       - Every time you commit to this repo, before the image is built and deployed 
-       - Whenever the `healthcheck` command is supplied to the deployed function
-   - Edit the `app/docs.md` file to reflect your changes. This file is baked into the function's image, and is made available using the `docs` command. This feature is used to display this function's documentation on our [Documentation](https://lambda-feedback.github.io/Documentation/) website once it's been hooked up!
+   - Add pytest tests to `app/tests/`. They get run:
+       - On every pull request and before the image is built and deployed (`pytest` from the repo root)
+       - Whenever the `healthcheck` command is supplied to the deployed function (via `app/evaluation_tests.py`)
+   - Edit `app/docs/user.md` and `app/docs/dev.md` to reflect your changes. These files are baked into the function's image, and is made available using the `docs` command. This feature is used to display this function's documentation on our [Documentation](https://lambda-feedback.github.io/Documentation/) website once it's been hooked up!
 
 ---
 
 ## How it works
 
 The function is built on top of a custom base layer, [BaseEvaluationFunctionLayer](https://github.com/lambda-feedback/BaseEvalutionFunctionLayer), which tools, tests and schema checking relevant to all evaluation functions.
 
+### Packages
+
+Expression parsing, unit handling and the criteria machinery come from the [lambda-feedback-packages](https://github.com/lambda-feedback/lambda-feedback-packages) repository (currently the `v0.3.0-rc.5` release candidate), installed as git dependencies in `app/requirements.txt`. See the "Packages" section of `app/docs/dev.md` for what each package does and how to develop against a local checkout.
+
 ### Docker & Amazon Web Services (AWS)
 
 The grading scripts are hosted AWS Lambda, using containers to run a docker image of the app. Docker is a popular tool in software development that allows programs to be hosted on any machine by bundling all its requirements and dependencies into a single file called an **image**.
@@ -57,7 +61,7 @@ Once the code passes all these tests, it will then be uploaded to AWS and will b
 ## Pre-requisites
 Although all programming can be done through the GitHub interface, it is recommended you do this locally on your machine. To do this, you must have installed:
 
-- Python 3.8 or higher.
+- Python 3.12 (the version used by CI and the production image).
 
 - GitHub Desktop or the `git` CLI.
 

app/.dockerignore

@@ -0,0 +1,12 @@
+# Build files
+Dockerfile
+.dockerignore
+
+# The base image provides the app package's __init__.py and handler
+__init__.py
+
+# Local and generated files
+**/__pycache__
+**/*.py[cod]
+**/.DS_Store
+**/.pytest_cache

app/Dockerfile

@@ -1,6 +1,4 @@
-# Base image that bundles AWS Lambda Python 3.8 image with some middleware functions
-# FROM base-eval-tmp
-# FROM rabidsheep55/python-base-eval-layer
+# Base image that bundles the AWS Lambda Python image with the lambda-feedback middleware (app.handler)
 FROM ghcr.io/lambda-feedback/baseevalutionfunctionlayer:main-3.12
 
 RUN dnf install -y git findutils && dnf clean all
@@ -13,47 +11,11 @@ ENV PYTHONPATH=/app
 COPY requirements.txt .
 RUN pip3 install -r requirements.txt
 
-# Copy main scripts
-COPY evaluation.py ./app/
-COPY evaluation_test.py ./app/
-COPY preview.py ./app/
-COPY preview_test.py ./app/
-
-# Copy contexts
-COPY context/physical_quantity.py ./app/context/
-COPY context/symbolic.py ./app/context/
-
-# Copy feedback messages
-COPY feedback/physical_quantity.py ./app/feedback/
-COPY feedback/symbolic.py ./app/feedback/
-
-# Copy preview implementations
-COPY preview_implementations/physical_quantity_preview.py ./app/preview_implementations/
-COPY preview_implementations/symbolic_preview.py ./app/preview_implementations/
-
-# Copy tests
-COPY tests/evaluation_logging_test.py ./app/tests/
-COPY tests/example_test.py ./app/tests/
-COPY tests/physical_quantity_evaluation_test.py ./app/tests/
-COPY tests/physical_quantity_preview_test.py ./app/tests/
-COPY tests/slr_quantity_test.py ./app/tests/
-COPY tests/symbolic_evaluation_test.py ./app/tests/
-COPY tests/symbolic_preview_test.py ./app/tests/
-
-# Copy utility code
-COPY utility/criteria_graph_utilities.py ./app/utility/
-COPY utility/criteria_parsing.py ./app/utility/
-COPY utility/evaluation_result_utilities.py ./app/utility/
-COPY utility/expression_utilities.py ./app/utility/
-COPY utility/physical_quantity_utilities.py ./app/utility/
-COPY utility/preview_utilities.py ./app/utility/
-COPY utility/slr_parsing_utilities.py ./app/utility/
-COPY utility/syntactical_comparison_utilities.py ./app/utility/
-COPY utility/unit_system_conversions.py ./app/utility/
-
-# Copy Documentation
-COPY docs/dev.md ./app/docs/dev.md
-COPY docs/user.md ./app/docs/user.md
+# The healthcheck command runs the pytest suite (see evaluation_tests.py)
+RUN pip3 install pytest
+
+# Copy the evaluation function source, tests and docs (exclusions in .dockerignore)
+COPY . ./app/
 
 # Set permissions so files and directories can be accessed on AWS
 RUN chmod 644 $(find . -type f)

app/conftest.py

@@ -0,0 +1,5 @@
+import pytest
+
+
+def pytest_configure(config: pytest.Config) -> None:
+    config.addinivalue_line("markers", "slow: excluded from the deployed healthcheck (see evaluation_tests.py)")

app/context/base.py

@@ -0,0 +1,35 @@
+"""The `Context` that specialises the evaluation function for one kind of response (symbolic or physical quantity)."""
+
+from __future__ import annotations
+
+from collections.abc import Callable, Mapping
+from dataclasses import dataclass
+from typing import Any
+
+from lambdafeedback.criteria import CriteriaGraph
+from lambdafeedback.slr_parsing import SLRParser
+
+from app.evaluation_parameters import EvaluationParameters
+from app.preview.common import Result
+from app.result import EvaluationResult
+from app.types import Parameters
+
+
+@dataclass(frozen=True)
+class Context:
+    # Previews the response, as the preview function would
+    expression_preview: Callable[[str, Parameters], Result]
+    # Builds the parser for the criteria from the parsed reserved expressions
+    generate_criteria_parser: Callable[[Mapping[str, Mapping[str, Any]]], SLRParser]
+    # `(name, expression, parameters)`: the expression string, ready to be parsed
+    expression_preprocess: Callable[[str, str, Parameters], str]
+    # `(name, expression, parsing_parameters, evaluation_result)`: the parsed expression
+    expression_parse: Callable[[str, str, Any, EvaluationResult], Any]
+    # Parameters used when they are not given
+    default_parameters: Mapping[str, Any]
+    # Criteria used when none are given
+    default_criteria: tuple[str, ...]
+    # Builds the criteria graph for each parsed criterion
+    feedback_procedure_generator: Callable[[EvaluationParameters], dict[str, CriteriaGraph]]
+    # `(parameters, unsplittable_symbols=...)`: the parsing parameters passed to `expression_parse`
+    parsing_parameters_generator: Callable[..., Any]

app/context/defaults.py

@@ -0,0 +1,9 @@
+"""Default evaluation parameters shared by all contexts; each context adds its own on top of these."""
+
+base_default_parameters = {
+    "complexNumbers": False,
+    "convention": "equal_precedence",
+    "elementary_functions": False,
+    "strict_syntax": False,
+    "multiple_answers_criteria": "all",
+}

app/context/graph_helpers.py

@@ -0,0 +1,46 @@
+"""Helpers for building the `CriteriaGraph`s shared by the symbolic and physical quantity contexts."""
+
+from __future__ import annotations
+
+from collections.abc import Mapping
+from typing import Any
+
+from lambdafeedback.criteria import CriteriaGraph, Evaluate, FeedbackStringGenerator, OutputNode, ReachedCriteria
+
+
+def outcome_tag(label: str, result: bool | None, inputs: Mapping[str, Any] | None = None) -> ReachedCriteria:
+    """`{label+"_TRUE": inputs}` if `result` is True, `label+"_FALSE"` if it is False, otherwise `label+"_UNKNOWN"`."""
+    if result is True:
+        return {label + "_TRUE": inputs}
+    elif result is False:
+        return {label + "_FALSE": inputs}
+    else:
+        return {label + "_UNKNOWN": inputs}
+
+
+def outcome_graph(label: str, summary: str, details: str, evaluate: Evaluate) -> tuple[CriteriaGraph, OutputNode]:
+    """A new graph `label` with an end node and a single evaluation node `label`, ready for its outcomes to be attached."""
+    graph = CriteriaGraph(label)
+    end = graph.add_end_node()
+    graph.add_evaluation_node(label, summary=summary, details=details, evaluate=evaluate)
+    return graph, end
+
+
+def attach_terminal_criterion(
+    graph: CriteriaGraph,
+    evaluation_label: str,
+    label: str,
+    end: OutputNode,
+    summary: str,
+    details: str,
+    feedback_string_generator: FeedbackStringGenerator | None = None,
+) -> None:
+    """Attaches a new criterion `label` to `evaluation_label` and connects it to the `end` node."""
+    graph.attach_new_criterion(
+        evaluation_label,
+        label,
+        summary=summary,
+        details=details,
+        feedback_string_generator=feedback_string_generator
+    )
+    graph.attach(label, end.label)

app/context/operators.py

@@ -0,0 +1,11 @@
+"""The comparison operators that criteria can use, by the symbol used in the criteria string."""
+
+import operator
+
+COMPARISON_OPERATORS = {
+    "=": operator.eq,
+    ">": operator.gt,
+    ">=": operator.ge,
+    "<": operator.lt,
+    "<=": operator.le,
+}