[ENH] Make Parquet downloads configurable

[Jayant-kernel]

Metadata

• Reference Issue: Fixes [ENH] Allow change of OPENML_SKIP_PARAQUET_ENV_VAR config #1743
• New Tests Added: Yes
• Documentation Updated: No
• Change Log Entry: Make Parquet downloads configurable

Details

This adds a persistent skip_parquet configuration option for dataset downloads. When enabled, datasets use the ARFF download path even when a Parquet URL is available.

The existing OPENML_SKIP_PARQUET environment variable remains supported. Both dataset download paths now use the same configuration check so the behaviour is consistent.

Tests cover loading skip_parquet from the config file and confirm that the dataset download path skips Parquet and falls back to ARFF when the option is enabled.

[codecov-commenter]

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 53.77%. Comparing base (aee8627) to head (187f1b4).
⚠️ Report is 2 commits behind head on main.

Additional details and impacted files

@@            Coverage Diff             @@
##             main    #1752      +/-   ##
==========================================
- Coverage   53.99%   53.77%   -0.22%     
==========================================
  Files          63       63              
  Lines        5230     5229       -1     
==========================================
- Hits         2824     2812      -12     
- Misses       2406     2417      +11     

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

[L-Agbaria]

There is one failed test that I would like to looked deeper into and verify that it is not caused directly by this, in which case I can directly merge this.

[Jayant-kernel]

@L-Agbaria
Checked test_run_and_upload_maskedarrays — it fails on a malformed XML response from the test service, not the Parquet path. The Parquet code seems innocent here

[Jayant-kernel]

@PGijsbers
Please give it a review, once you are free

[L-Agbaria]

This feature will be added until parquet files become the default, after which this will resolved to always download parquet.

[Jayant-kernel]

Glad this works for the transition period