Check reference links for rot, not just repositories

The audit only ever asked GitHub about repositories, leaving 60,112
reference URLs unchecked since the corpus was inherited. Sampling put hard
404s at 4%, and one host accounts for a fifth of that on its own: CIS
retired the OVAL Repository, so all 942 links to it are permanently gone
and are removed in the next commit.

The rest is diffuse, so it needs sweeping rather than a rule. Check an
eighth of the URLs each week alongside the repositories, ordered
round-robin by host so a slice does not arrive at one server as a burst.
Only a 404 or 410 removes a link: a timeout, a refused HEAD or a 403 keeps
it, and update_cves already applies the same test to references it
discovers, so this holds inherited links to the standard new ones meet.
This commit is contained in:
0xMarcio committed 2026-08-31 00:27:34 +02:00
1 parent 6822fdbfbc
commit 66a4f41227
2 files changed
+146 -16

No files matched your search

+6 -4
View File
@@ -24,12 +24,13 @@ jobs:
with:
python-version: '3.13'
# A quarter of the corpus a week keeps each run inside the token's
# hourly budget and sweeps everything within the month.
- name: Drop links to deleted repositories
# A quarter of the repositories and an eighth of the reference links each
# week keeps a run inside the token's hourly budget while sweeping
# everything within about two months.
- name: Drop dead links and refresh repository metadata
env:
GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
run: python3 scripts/audit_poc_links.py --limit 12000
run: python3 scripts/audit_poc_links.py --limit 12000 --references 8000
- name: Check for changes and commit
run: |
@@ -41,6 +42,7 @@ jobs:
github.txt \
repo_meta.json \
kev.json \
references.txt \
':(top,glob)[12][0-9][0-9][0-9]/CVE-*.md'
if git diff --cached --quiet; then