Skip to content

Add V2 Importer for Tuxcare advisories - #2104

Open
Samk1710 wants to merge 12 commits into
aboutcode-org:mainfrom
Samk1710:add-tuxcare-importer
Open

Add V2 Importer for Tuxcare advisories#2104
Samk1710 wants to merge 12 commits into
aboutcode-org:mainfrom
Samk1710:add-tuxcare-importer

Conversation

@Samk1710

@Samk1710 Samk1710 commented Jan 4, 2026 •
edited
Loading

Copy link
Copy Markdown
Contributor

Addresses Issue:

Data Source: https://osv-vulnerabilities.storage.googleapis.com/TuxCare/all.zip

Importer Log Excerpt:

python manage.py import tuxcare_importer_v2
Importing data using tuxcare_importer_v2
INFO 2026-10-22 09:47:23.518176 UTC Pipeline [TuxCareImporterPipeline] starting
INFO 2026-10-22 09:47:23.518356 UTC Step [fetch_zip] starting
INFO 2026-10-22 09:47:23.518440 UTC Fetching `https://osv-vulnerabilities.storage.googleapis.com/TuxCare/all.zip`
INFO 2026-10-22 09:47:27.134942 UTC Step [fetch_zip] completed in 4 seconds
INFO 2026-10-22 09:47:27.135528 UTC Step [collect_and_store_advisories] starting
INFO 2026-10-22 09:47:27.284310 UTC Collecting 17,688 advisories
INFO 2026-10-22 09:50:21.282477 UTC Progress: 10% (1769/17688) ETA: 1566 seconds (26.1 minutes)
INFO 2026-10-22 09:53:23.097456 UTC Progress: 20% (3538/17688) ETA: 1423 seconds (23.7 minutes)
INFO 2026-10-22 09:55:57.992284 UTC Progress: 30% (5307/17688) ETA: 1192 seconds (19.9 minutes)
INFO 2026-10-22 09:58:54.480809 UTC Progress: 40% (7076/17688) ETA: 1031 seconds (17.2 minutes)
INFO 2026-10-22 10:02:57.503693 UTC Progress: 50% (8844/17688) ETA: 930 seconds (15.5 minutes)
INFO 2026-10-22 10:05:37.188014 UTC Progress: 60% (10613/17688) ETA: 727 seconds (12.1 minutes)
INFO 2026-10-22 10:06:26.311517 UTC Progress: 70% (12382/17688) ETA: 488 seconds (8.1 minutes)
INFO 2026-10-22 10:07:17.769579 UTC Progress: 80% (14151/17688) ETA: 298 seconds (5.0 minutes)
INFO 2026-10-22 10:08:05.944168 UTC Progress: 90% (15920/17688) ETA: 138 seconds (2.3 minutes)
INFO 2026-10-22 10:09:13.021113 UTC Progress: 100% (17688/17688)
INFO 2026-10-22 10:09:13.051267 UTC Successfully collected 17,688 advisories
INFO 2026-10-22 10:09:13.051511 UTC Step [collect_and_store_advisories] completed in 1306 seconds (21.8 minutes)
INFO 2026-10-22 10:09:13.051590 UTC Pipeline completed in 1310 seconds (21.8 minutes)

@ziadhany
ziadhany self-requested a review January 5, 2026 10:22
ziadhany
ziadhany requested changes Jan 7, 2026
View reviewed changes

@ziadhany ziadhany left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@Samk1710 Thanks , see feedback and suggestions below

Comment thread vulnerabilities/pipelines/v2_importers/tuxcare_importer.py Outdated
Comment thread vulnerabilities/pipelines/v2_importers/tuxcare_importer.py Outdated
Comment thread vulnerabilities/pipelines/v2_importers/tuxcare_importer.py Outdated
Comment thread vulnerabilities/pipelines/v2_importers/tuxcare_importer.py Outdated
Comment thread vulnerabilities/pipelines/v2_importers/tuxcare_importer.py Outdated
Comment thread vulnerabilities/tests/pipelines/v2_importers/test_tuxcare_importer_v2.py Outdated
Comment thread vulnerabilities/pipelines/v2_importers/tuxcare_importer.py Outdated
Comment thread vulnerabilities/pipelines/v2_importers/tuxcare_importer.py Outdated
@Samk1710

Samk1710 commented Jan 7, 2026

Copy link
Copy Markdown
Contributor Author

@ziadhany Thanks for your review.
I have updated the code as per your suggestion and feedback. Requesting a re-review. Thanks again!

@Samk1710
Samk1710 requested a review from ziadhany January 7, 2026 21:18
ziadhany
ziadhany reviewed Jan 8, 2026
View reviewed changes
Comment thread vulnerabilities/pipelines/v2_importers/tuxcare_importer.py Outdated
@ziadhany

ziadhany commented Jan 8, 2026 •
edited
Loading

Copy link
Copy Markdown
Member

@Samk1710, could you please also fix the CI ?

@Samk1710

Samk1710 commented Jan 10, 2026

Copy link
Copy Markdown
Contributor Author

Hey @ziadhany
I have updated the implementation as per your review and suggestion of os_name qualifier. Do let me know if it aligns with what you had in mind. Thanks !

@Samk1710
Samk1710 requested a review from ziadhany January 10, 2026 00:03
ziadhany
ziadhany requested changes Jan 12, 2026
View reviewed changes
Comment thread vulnerabilities/pipelines/v2_importers/tuxcare_importer.py Outdated
@Samk1710

Samk1710 commented Jan 12, 2026

Copy link
Copy Markdown
Contributor Author

Hey @ziadhany
I have refactored the purl as per your suggestion and pushed. Thanks for the guidance :)

@Samk1710
Samk1710 requested a review from ziadhany January 12, 2026 14:23
@Samk1710

Samk1710 commented Jan 13, 2026

Copy link
Copy Markdown
Contributor Author

@Samk1710, could you please also fix the CI ?

Hey @ziadhany could you kindly run the checks. I have fixed the import ordering. Thanks.

ziadhany
ziadhany requested changes Jan 16, 2026
View reviewed changes

@ziadhany ziadhany left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@Samk1710 , The code looks good. I think we just need some refinement of the package URL and the affected and fixed versions.

Comment thread vulnerabilities/pipelines/v2_importers/tuxcare_importer.py Outdated
Comment thread vulnerabilities/pipelines/v2_importers/tuxcare_importer.py Outdated
Comment thread vulnerabilities/pipelines/v2_importers/tuxcare_importer.py Outdated
Comment thread vulnerabilities/pipelines/v2_importers/tuxcare_importer.py Outdated
@Samk1710

Samk1710 commented Jan 21, 2026

Copy link
Copy Markdown
Contributor Author

Hey @ziadhany

I have rectified the PURL. Also added more data to test each OS type with their respective PURLs.
After some digging in I also found the documentation for statuses and also implemented them.
Please see #2104 (comment)

Kindly review the changes when you have time. Thanks.

@Samk1710
Samk1710 requested a review from ziadhany January 22, 2026 18:03
ziadhany
ziadhany reviewed Jan 26, 2026
View reviewed changes
Comment thread vulnerabilities/pipelines/v2_importers/tuxcare_importer.py Outdated
@Samk1710

Samk1710 commented Jan 26, 2026

Copy link
Copy Markdown
Contributor Author

Hey @ziadhany
As per discussions in the weekly meet, I have implemented the Impact Packages and also mapped the version range.
Kindly take a look at them and let me know if anything has to be improved. Please review when time. Thanks.

@Samk1710
Samk1710 requested a review from ziadhany January 26, 2026 19:45
@Samk1710
Samk1710 force-pushed the add-tuxcare-importer branch from 0f27746 to 7d47d46 Compare January 27, 2026 16:24
ziadhany
ziadhany requested changes Feb 2, 2026
View reviewed changes

@ziadhany ziadhany left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The code looks good, just a few nits.

Comment thread vulnerabilities/pipelines/v2_importers/tuxcare_importer.py Outdated
Comment thread vulnerabilities/pipelines/v2_importers/tuxcare_importer.py Outdated
Comment thread vulnerabilities/pipelines/v2_importers/tuxcare_importer.py Outdated
Comment thread vulnerabilities/pipelines/v2_importers/tuxcare_importer.py Outdated
Comment thread vulnerabilities/pipelines/v2_importers/tuxcare_importer.py Outdated
Comment thread vulnerabilities/pipelines/v2_importers/tuxcare_importer.py Outdated
Comment thread vulnerabilities/pipelines/v2_importers/tuxcare_importer.py Outdated
Comment thread vulnerabilities/pipelines/v2_importers/tuxcare_importer.py Outdated
@Samk1710
Samk1710 force-pushed the add-tuxcare-importer branch from e4e1684 to 66be491 Compare February 2, 2026 22:24
@Samk1710

Samk1710 commented Feb 2, 2026

Copy link
Copy Markdown
Contributor Author

The code looks good, just a few nits.

Thanks @ziadhany
I’ve addressed the review comments and pushed the latest changes.
Please let me know if anything else needs adjustment.

@Samk1710
Samk1710 requested a review from ziadhany February 5, 2026 09:15
@Samk1710
Samk1710 force-pushed the add-tuxcare-importer branch from 66be491 to 9597abf Compare February 9, 2026 09:57
@Samk1710

Samk1710 commented Feb 19, 2026

Copy link
Copy Markdown
Contributor Author

Hey @ziadhany. Could you kindly have a look at the changes when time. Thanks.

ziadhany
ziadhany requested changes Feb 21, 2026
View reviewed changes

@ziadhany ziadhany left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@Samk1710, just a few nits to improve code quality and update the importer to use AdvisoryDataV2. Overall, the code looks good.

Comment thread vulnerabilities/pipelines/v2_importers/tuxcare_importer.py Outdated
Comment thread vulnerabilities/pipelines/v2_importers/tuxcare_importer.py Outdated
Comment thread vulnerabilities/pipelines/v2_importers/tuxcare_importer.py Outdated
Comment thread vulnerabilities/pipelines/v2_importers/tuxcare_importer.py Outdated
Comment thread vulnerabilities/pipelines/v2_importers/tuxcare_importer.py Outdated
Comment thread vulnerabilities/pipelines/v2_importers/tuxcare_importer.py Outdated
Comment thread vulnerabilities/pipelines/v2_importers/tuxcare_importer.py Outdated
Comment thread vulnerabilities/pipelines/v2_importers/tuxcare_importer.py Outdated
Comment thread vulnerabilities/pipelines/v2_importers/tuxcare_importer.py Outdated
Comment thread vulnerabilities/pipelines/v2_importers/tuxcare_importer.py Outdated
@Samk1710
Samk1710 force-pushed the add-tuxcare-importer branch 2 times, most recently from e8d2186 to d7ed5f2 Compare February 24, 2026 06:22
This was referenced Feb 24, 2026
Closed
Merged
@Samk1710
Samk1710 force-pushed the add-tuxcare-importer branch 2 times, most recently from 07a5af2 to 37938ee Compare February 24, 2026 18:56
@Samk1710

Samk1710 commented Feb 24, 2026

Copy link
Copy Markdown
Contributor Author

@ziadhany Done with the changes. Please have a look when time

@Samk1710
Samk1710 requested a review from ziadhany February 26, 2026 12:33
@Samk1710
Samk1710 force-pushed the add-tuxcare-importer branch from 37938ee to 6474f3a Compare March 7, 2026 18:47
@Samk1710

Samk1710 commented Oct 2, 2026

Copy link
Copy Markdown
Contributor Author

The new endpoint at https://tuxcare.com/cve-tracker/api/cves/export/json is workable although the PURL resolution might be a bit difficult.

Old advisory

  {
    "cve": "CVE-2023-52922",
    "os_name": "CentOS 8.5 ELS",
    "project_name": "kernel",
    "version": "4.18.0",
    "score": "7.8",
    "severity": "HIGH",
    "status": "Released",
    "last_updated": "2025-05-21 01:43:28.677045"
  }

Sample New Advisory

{
  "cve": "CVE-2026-59797",
  "product": "CentOS 6 ELS",
  "component": "httpd",
  "version": "2.2.15",
  "severity": "Critical",
  "status": "In Research",
  "published_date": "2026-10-01",
  "type": null
}

Currently we were using in https://github.com/aboutcode-org/vulnerablecode/pull/2104/changes#diff-11b0ceee4d3520186d775bd38d8898718ba0b40cea97007fff7dfe0f1fa5737a

        os_mapping = {
            "ubuntu": ("deb", "ubuntu"),
            "debian": ("deb", "debian"),
            "centos": ("rpm", "centos"),
            "almalinux": ("rpm", "almalinux"),
            "rhel": ("rpm", "rhel"),
            "oracle": ("rpm", "oracle"),
            "cloudlinux": ("rpm", "cloudlinux"),
            "alpine": ("apk", "alpine"),
            "unknown": ("generic", "tuxcare"),
            "tuxcare": ("generic", "tuxcare"),
        }

But now Tuxcare has 6814 unique products. This mapping alone will not suffice. See the gist for product frequencies here https://gist.github.com/Samk1710/0229b9dd41a66ba414dbed0464f24428.

Also CVE-2017-20162 alone has 30149 records which is unusual.

These are the top CVEs and their count

CVE-2017-20162: 30149
CVE-2017-16137: 29674
CVE-2017-20165: 29629
CVE-2018-8088: 27253
CVE-2025-67735: 22861
CVE-2025-55163: 22797
CVE-2025-58056: 22536
CVE-2025-58057: 22260

@Samk1710

Samk1710 commented Oct 20, 2026

Copy link
Copy Markdown
Contributor Author

Reply from Tuxcare Team

Hello Sampurna,

Thank you for your time! Addressing your inquiries:
The old endpoint cve.tuxcare.com/els/download-json is indeed permanently discontinued in favor of the export at https://tuxcare.com/cve-tracker/api/cves/export/json. The full file is around 3 GB, regenerated twice a day; with filters, up to 10,000 rows.
Regarding PURLs: the export has no official mapping and carries no PURL or ecosystem fields. For that, the best source is our OSV feed, published on osv.dev under the TuxCare ecosystem. Every record there includes PURLs (for example pkg:rpm/tuxcare/httpd?distro=centos-6) and fixed versions, across rpm, deb, Maven, npm, PyPI and Packagist. The full set can be downloaded from https://osv-vulnerabilities.storage.googleapis.com/TuxCare/all.zip. Keep in mind it only contains released fixes, and a few products in the export, such as Amazon Linux 2 ELS and CloudLinux ELS, aren't in it yet.
On the record counts: the export mixes our ELS distro data with our open-source library monitoring data. In the second set, "product" is an upstream project (Babel, Angular, React) and "component" is one of its dependencies, which is where most of the 6,814 products and the 30,000 rows for CVE-2017-20162 come from. Those rows are distinct, so we'd suggest keying on CVE, product, component and version together rather than deduplicating on CVE alon.

Please let us know if that helps!
Best Regards,
Misael Ramirez | Senior Support Engineer | TuxCare Enterprise Support

Link to conversation
https://support.tuxcare.com/hc/requests/296039

Samk1710 added 11 commits October 22, 2026 15:36
@Samk1710
Add V2 Importer for Tuxcare
704fb9e 
Signed-off-by: Sampurna Pyne <sampurnapyne1710@gmail.com>
@Samk1710
Refactor as per review
66bac4c 
Signed-off-by: Sampurna Pyne <sampurnapyne1710@gmail.com>
@Samk1710
Refactor to PURL qualifier
30c0912 
Signed-off-by: Sampurna Pyne <sampurnapyne1710@gmail.com>
@Samk1710
Codestyle fix
ff4e2fd 
Signed-off-by: Sampurna Pyne <sampurnapyne1710@gmail.com>
@Samk1710
Refactor PURL and fix type-hinting
1a85ee0 
Signed-off-by: Sampurna Pyne <sampurnapyne1710@gmail.com>
@Samk1710
Fix import ordering
8506881 
Signed-off-by: Sampurna Pyne <sampurnapyne1710@gmail.com>
@Samk1710
Refactor PURL and status handling
5848738 
Signed-off-by: Sampurna Pyne <sampurnapyne1710@gmail.com>
@Samk1710
Implement impact packages and specific version with univers
3d2fbe4 
Signed-off-by: Sampurna Pyne <sampurnapyne1710@gmail.com>
@Samk1710
Add docstrings and use pipeline logger
48f97ae 
Signed-off-by: Sampurna Pyne <sampurnapyne1710@gmail.com>
@Samk1710
Add tests for purl creation and use AdvisoryV2
de55b48 
Signed-off-by: Sampurna Pyne <sampurnapyne1710@gmail.com>
@Samk1710
Add precedence to tuxcare importer
d10f7b9 
Signed-off-by: Sampurna Pyne <sampurnapyne1710@gmail.com>
@Samk1710
Samk1710 force-pushed the add-tuxcare-importer branch from 6474f3a to 90fad0a Compare October 22, 2026 10:10
@Samk1710
Update tuxcare Importer to new OSV data source along with tests
cfb7279 
Signed-off-by: Sampurna Pyne <sampurnapyne1710@gmail.com>
@Samk1710
Samk1710 force-pushed the add-tuxcare-importer branch from 90fad0a to cfb7279 Compare October 22, 2026 10:19
@Samk1710

Samk1710 commented Oct 22, 2026

Copy link
Copy Markdown
Contributor Author

@ziadhany I have updated the importer to use the OSV Data Source: https://osv-vulnerabilities.storage.googleapis.com/TuxCare/all.zip
Awaiting a follow up regarding the confirmation of the license from the Support Team.

python manage.py import tuxcare_importer_v2
Importing data using tuxcare_importer_v2
INFO 2026-10-22 09:47:23.518176 UTC Pipeline [TuxCareImporterPipeline] starting
INFO 2026-10-22 09:47:23.518356 UTC Step [fetch_zip] starting
INFO 2026-10-22 09:47:23.518440 UTC Fetching `https://osv-vulnerabilities.storage.googleapis.com/TuxCare/all.zip`
INFO 2026-10-22 09:47:27.134942 UTC Step [fetch_zip] completed in 4 seconds
INFO 2026-10-22 09:47:27.135528 UTC Step [collect_and_store_advisories] starting
INFO 2026-10-22 09:47:27.284310 UTC Collecting 17,688 advisories
INFO 2026-10-22 09:50:21.282477 UTC Progress: 10% (1769/17688) ETA: 1566 seconds (26.1 minutes)
INFO 2026-10-22 09:53:23.097456 UTC Progress: 20% (3538/17688) ETA: 1423 seconds (23.7 minutes)
INFO 2026-10-22 09:55:57.992284 UTC Progress: 30% (5307/17688) ETA: 1192 seconds (19.9 minutes)
INFO 2026-10-22 09:58:54.480809 UTC Progress: 40% (7076/17688) ETA: 1031 seconds (17.2 minutes)
INFO 2026-10-22 10:02:57.503693 UTC Progress: 50% (8844/17688) ETA: 930 seconds (15.5 minutes)
INFO 2026-10-22 10:05:37.188014 UTC Progress: 60% (10613/17688) ETA: 727 seconds (12.1 minutes)
INFO 2026-10-22 10:06:26.311517 UTC Progress: 70% (12382/17688) ETA: 488 seconds (8.1 minutes)
INFO 2026-10-22 10:07:17.769579 UTC Progress: 80% (14151/17688) ETA: 298 seconds (5.0 minutes)
INFO 2026-10-22 10:08:05.944168 UTC Progress: 90% (15920/17688) ETA: 138 seconds (2.3 minutes)
INFO 2026-10-22 10:09:13.021113 UTC Progress: 100% (17688/17688)
INFO 2026-10-22 10:09:13.051267 UTC Successfully collected 17,688 advisories
INFO 2026-10-22 10:09:13.051511 UTC Step [collect_and_store_advisories] completed in 1306 seconds (21.8 minutes)
INFO 2026-10-22 10:09:13.051590 UTC Pipeline completed in 1310 seconds (21.8 minutes)

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@ziadhany ziadhany Awaiting requested review from ziadhany

Requested changes must be addressed to merge this pull request.

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

2 participants

@Samk1710 @ziadhany