mirror of
https://github.com/0xMarcio/cve.git
synced 2026-10-27 16:51:03 +00:00
Check reference links for rot, not just repositories
The audit only ever asked GitHub about repositories, leaving 60,112 reference URLs unchecked since the corpus was inherited. Sampling put hard 404s at 4%, and one host accounts for a fifth of that on its own: CIS retired the OVAL Repository, so all 942 links to it are permanently gone and are removed in the next commit. The rest is diffuse, so it needs sweeping rather than a rule. Check an eighth of the URLs each week alongside the repositories, ordered round-robin by host so a slice does not arrive at one server as a burst. Only a 404 or 410 removes a link: a timeout, a refused HEAD or a 403 keeps it, and update_cves already applies the same test to references it discovers, so this holds inherited links to the standard new ones meet.
This commit is contained in:
1 parent
6822fdbfbc
commit
66a4f41227
2 files changed
+146
-16
No files matched your search
@@ -24,12 +24,13 @@ jobs:
|
||||
with:
|
||||
python-version: '3.13'
|
||||
|
||||
# A quarter of the corpus a week keeps each run inside the token's
|
||||
# hourly budget and sweeps everything within the month.
|
||||
- name: Drop links to deleted repositories
|
||||
# A quarter of the repositories and an eighth of the reference links each
|
||||
# week keeps a run inside the token's hourly budget while sweeping
|
||||
# everything within about two months.
|
||||
- name: Drop dead links and refresh repository metadata
|
||||
env:
|
||||
GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
|
||||
run: python3 scripts/audit_poc_links.py --limit 12000
|
||||
run: python3 scripts/audit_poc_links.py --limit 12000 --references 8000
|
||||
|
||||
- name: Check for changes and commit
|
||||
run: |
|
||||
@@ -41,6 +42,7 @@ jobs:
|
||||
github.txt \
|
||||
repo_meta.json \
|
||||
kev.json \
|
||||
references.txt \
|
||||
':(top,glob)[12][0-9][0-9][0-9]/CVE-*.md'
|
||||
|
||||
if git diff --cached --quiet; then
|
||||
|
||||
Reference in new issue
Block a user