Skip to content

[SYSTEMDS-3992] Obtain Sparsity Information from In-Memory Matrix - #2637

Closed
ywcb00 wants to merge 3 commits into
apache:mainfrom
ywcb00:feat/recompile/sparsity/optimizer/integrate
Closed

ywcb00 wants to merge 3 commits into
apache:mainfrom
ywcb00:feat/recompile/sparsity/optimizer/integrate

Conversation

@ywcb00

@ywcb00 ywcb00 commented Nov 19, 2026

Copy link
Copy Markdown
Contributor

Hi,
this PR calls the sparsity optimizer from the recompiler instead of the program rewriter. This, in turn, allows leveraging the variable map from the execution context to obtain the sparsity of matrices directly from their in-memory matrix blocks.
All the best,
David

… from the recompiler instead of the program rewriter

feat(main/hops/rewrite/SparsityOptimizer.java): copy the rewrite functionality from the sparsity-based matrix multiplication rewrite into the sparsity optimizer

feat(main/hops/estim/SparsityEstimator.java): create the abstract method for obtaining the sparsity information from an input node by reading it from the matrix block
…ain directly after creating the input MMNode with it
@codecov

codecov Bot commented Nov 19, 2026 •

Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 66.87500% with 53 lines in your changes missing coverage. Please review.
✅ Project coverage is 72.17%. Comparing base (aac5249) to head (f847957).
⚠️ Report is 1 commits behind head on main.

Files with missing lines Patch % Lines
...g/apache/sysds/hops/rewrite/SparsityOptimizer.java 81.66% 8 Missing and 14 partials ⚠️
...g/apache/sysds/hops/estim/EstimatorBasicWorst.java 0.00% 10 Missing ⚠️
...org/apache/sysds/hops/estim/EstimatorBasicAvg.java 50.00% 4 Missing and 1 partial ⚠️
...org/apache/sysds/hops/estim/EstimatorBitsetMM.java 0.00% 3 Missing ⚠️
...g/apache/sysds/hops/estim/EstimatorDensityMap.java 0.00% 3 Missing ⚠️
...che/sysds/hops/estim/EstimatorMatrixHistogram.java 25.00% 3 Missing ⚠️
.../org/apache/sysds/hops/estim/EstimatorRowWise.java 0.00% 2 Missing ⚠️
...va/org/apache/sysds/hops/recompile/Recompiler.java 60.00% 1 Missing and 1 partial ⚠️
...apache/sysds/hops/estim/EstimatorLayeredGraph.java 0.00% 1 Missing ⚠️
...a/org/apache/sysds/hops/estim/EstimatorSample.java 0.00% 1 Missing ⚠️
... and 1 more
Additional details and impacted files
@@              Coverage Diff              @@
##               main    #2637       +/-   ##
=============================================
- Coverage     76.59%   72.17%    -4.42%     
- Complexity        0    51216    +51216     
=============================================
  Files           458     2105     +1647     
  Lines         31204   230158   +198954     
  Branches          0    38694    +38694     
=============================================
+ Hits          23901   166124   +142223     
- Misses         7303    52838    +45535     
- Partials          0    11196    +11196     
Flag Coverage Δ
java 71.48% <66.87%> (?)
python 76.59% <ø> (ø)

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

…mport

chore(main/hops/estim/**): add exception messages
@ywcb00
ywcb00 marked this pull request as ready for review November 19, 2026 13:35
@ywcb00 ywcb00 changed the title [DRAFT][SYSTEMDS-3992] Obtain Sparsity Information from In-Memory Matrix [SYSTEMDS-3992] Obtain Sparsity Information from In-Memory Matrix Nov 19, 2026
@ywcb00 ywcb00 closed this in 1a358e5 Nov 19, 2026
@github-project-automation github-project-automation Bot moved this from In Progress to Done in SystemDS PR Queue Nov 19, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

Status: Done

Development

Successfully merging this pull request may close these issues.

1 participant