[copybara-service]

Add TCP readiness probes to Pathways head containers and scope custom libtpu to Pathways containers.

Why:
_build_rm_container and _build_proxy_container had no readinessProbe, so Kubernetes marked the pathways-head pod Ready as soon as the container processes spawned. When TPU_LIBRARY_PATH points to a libtpu.so on a cold GCSFuse mount, pathways-rm and pathways-proxy spend ~30-40s inside dlopen before binding their gRPC ports 29001 (PATHWAYS_RM_PORT) and 29000 (PATHWAYS_PROXY_PORT), so anything that acts on the pod's Ready condition (for example kubectl wait --for=condition=Ready followed by kubectl port-forward) hit connection refused.

with_custom_libtpu(..., containers="all") also injected TPU_LIBRARY_PATH into every container and initContainer in the pod spec, including non-Pathways ones (staging init containers, user-workload, colocated-python-sidecar). All three Pathways containers do need the custom libtpu.so: pathways-rm hosts the compilation service and pathways-worker executes the compiled programs, so they must load the same PJRT/TPU runtime ABI; otherwise programs fail with FAILED_PRECONDITION: The base ABI version required by the program (165) is not supported by the current runtime (which has version 164).

What changed:

• Add TCP socket readinessProbes (periodSeconds=1) on PATHWAYS_RM_PORT in _build_rm_container and on PATHWAYS_PROXY_PORT in _build_proxy_container, so pathways-head is Ready only once both gRPC servers are listening.
• Scope containers="all" in PathwaysJobSet.with_custom_libtpu to ("pathways-rm", "pathways-proxy", "pathways-worker").
• Only warn in _apply_container_kwargs when a caller-supplied field differs from the default value.

Behavior change for PathwaysJobSet users:
The generated JobSet uses startupPolicy: InOrder with replicatedJobs: [pathways-head, pathways-worker], and JobSet creates the next ReplicatedJob only once the previous one's Jobs report status.ready (pods with the Ready condition). With these probes, pathways-worker Jobs are created only after pathways-rm and pathways-proxy are listening, which is what InOrder was meant to express; previously the head pod was Ready about a second after start, so InOrder was effectively a no-op. This delays worker Job creation by the head's startup time (~3-5s with a node-local libtpu.so, ~30-40s with a GCSFuse-mounted one). The head pod's DNS name is unaffected (network.publishNotReadyAddresses: true), readiness probes never restart containers, and the probes can be overridden with rm_container_kwargs / proxy_container_kwargs={"readiness_probe": ...}.

[google-cla]

Thanks for your pull request! It looks like this may be your first contribution to a Google open source project. Before we can look at your pull request, you'll need to sign a Contributor License Agreement (CLA).

View this failed invocation of the CLA check for more information.

For the most up to date status, view the checks section at the bottom of the pull request.