[jnunn-aws]

These are fixes for the larger items in my review of #2602, one commit per finding, each with its tests. Merge, cherry-pick or close as you see fit; each commit builds and passes on its own. The smaller items are suggested changes on #2602 instead.

• a6a72dd Exempt only ECR hosts and the configured proxy from the connect guard
• 04185e1 Treat only 404 as absent and report registry denials as authentication errors
• 821b9a9 Add FileManifest.AddDirectory for folder entries
• f88fe01 Write layers as ustar so they are reproducible across processes
• b78ea5a Add LayerPushProgressQueue for polling push progress
• b3c3263 Expose the resolved base image digests

Tests and checks:

• The library suite passes on net10.0, with tests added for each change.
• A real push to a scratch ECR repository on pwsh 7.4 (.NET 8) and pwsh 7.6 (.NET 10) covered:
  • a push;
  • a re-push, where every layer was skipped;
  • an invalid token, rejected in under a second with no base-layer download.
• Layers built in separate processes give the same digests on each runtime. GNU tar and Python's tarfile both read them.
• Proxies: a local proxy, and one set at run time through HttpClient.DefaultProxy, both receive CONNECT. The ECR VPC endpoint case is covered by a test that resolves the ECR host to a private address. It hasn't been run in a real VPC.

Behavior changes worth a look:

• The connect guard no longer checks ECR hosts or the configured proxy. Every other host, including caller-supplied base registries, is still checked.
• Exception types:
  • Denials after the handshake are RegistryAuthenticationException (now with StatusCode), not RegistryRequestException.
  • Transport failures are RegistryException, not HttpRequestException.
• Existence checks: a HEAD redirect now counts as present, and only 404 counts as absent.
• Layer bytes change because entries are ustar now. Builds are reproducible per .NET major version, not across versions, because zlib and the ustar uid/gid formatting differ between .NET 8 and 10.

Internal test seams: RegistryClient.Create takes proxy and resolveHost, there's a RegistryClient constructor overload with anonymous, FromBaseImageAsync has an overload that takes the source client factory, and Layer.CreateEntry/FitsUstar are internal.

[normj]

I don't see how the user can take advantage of the proxy support. Do we need to add an option in EcrPushOptions for user to be able to specify the proxy and timeout?

[normj]

I would prefer AddEmptyDirectory to call out this is a special API to add something that doesn't map to something public.

[normj]

I was nervous about adding this complexity for the reproducible output and figured in our case most build images would change. Looking at your change for this it doesn't look too bad. I'm not fan of using the exception handling for control flow and worry if MSFT changes where the exception is thrown we would break. What do you think of this version:

// ustar header limits: a 100-byte name field, and an 11-digit octal size field, so the
// largest size is 8^11 - 1 (one byte under 8 GiB).
private const int UstarNameFieldLength = 100;
private const long UstarMaxSize = (1L << 33) - 1;

/// <summary>
/// Whether an entry can be written as ustar: its size fits the size field and its UTF-8 name fits
/// the name field. Longer names use PAX. ustar's 155-byte prefix field isn't used, because .NET
/// decides how to split a name between the two fields and doesn't document how.
/// </summary>
internal static bool FitsUstar(string name, long size)
    => size <= UstarMaxSize && Encoding.UTF8.GetByteCount(name) <= UstarNameFieldLength;

We also wouldn't need the HeaderProbeStream type as well with this approach.