A timeout kept the link either way, but the checker still followed a failed
HEAD with a GET to the same dead host, so every unreachable URL cost two
full timeouts. Return on the first network failure and drop the timeout to
eight seconds: a measured 1,500 references went from roughly three URLs a
second to fifteen.
That pass removed 98 links across 86 inventory lines, and the weekly budget
rises to 12,000 references, which sweeps the corpus every five weeks.
The audit only ever asked GitHub about repositories, leaving 60,112
reference URLs unchecked since the corpus was inherited. Sampling put hard
404s at 4%, and one host accounts for a fifth of that on its own: CIS
retired the OVAL Repository, so all 942 links to it are permanently gone
and are removed in the next commit.
The rest is diffuse, so it needs sweeping rather than a rule. Check an
eighth of the URLs each week alongside the repositories, ordered
round-robin by host so a slice does not arrive at one server as a burst.
Only a 404 or 410 removes a link: a timeout, a refused HEAD or a 403 keeps
it, and update_cves already applies the same test to references it
discovers, so this holds inherited links to the standard new ones meet.
CISA publishes a catalogue of vulnerabilities observed being exploited in
the wild, and 1,111 of its 1,685 entries are CVEs this index already holds
PoCs for. That is the one fact a reader most wants before opening an
exploit, and it was nowhere on the page.
The weekly audit stores the catalogue, the build trims it to the CVEs the
index carries, and a result row shows a KEV chip -- with the ransomware
flag where CISA records one, which covers 352 of them. 37 KB served.
A failed fetch keeps the stored catalogue rather than dropping the badge
off every entry. The chip reuses the header pill's treatment: the palette
still spends green on things you can click and amber on star counts only.
The audit already asks GitHub about every linked repository, so the same
query can carry back a star count and a last-push date for free. Results go
to repo_meta.json, which lets a search result say which of two hundred
linked PoCs is worth opening.
2.1 MB for 45,008 repositories, refreshed weekly.
The trending table sorted and displayed updated_at, which also moves when a
repository merely gains a star, so a four-month-old exploit was claiming it
had been updated three hours ago. Record pushed_at and created_at and let
the page decide: last commit for the age it shows, repository age for the
trending rank.
The job now writes trending.json alongside the README table, so the site
reads the data instead of regex-parsing markdown the same job generated.
Sampling the corpus put 17% of the repositories it links to behind a 404,
so readers were following links to repositories that no longer exist. The
previous audit script was 630 lines, never wired to anything, and kept its
state under a gitignored directory.
Replace it with a check that batches repository lookups through GraphQL
and removes a link only when GitHub answers NOT_FOUND for it, so a
throttle or a network error can never drop one. A quarter of the corpus
runs each week, which stays inside the token's hourly budget and sweeps
everything within the month.
The two scheduled jobs write disjoint paths but pin their checkout when
they are queued, so a commit landing in between throws away a completed
run. Rebase onto the current main and retry instead, deepening the clone
only on the rare occasion it is needed.
The search API was called anonymously on a 10 requests/minute limit while
the workflow already had a token; send it as an Authorization header,
never in the URL, and never print it. Dropping the token from the origin
remote also stops it being written into .git/config on the runner, since
actions/checkout already provides push credentials.
Rewrite on urllib so the job installs nothing, retry rate limits and 5xx,
and fail loudly instead of silently publishing a README with a year
missing. Repository text is flattened so a pipe cannot break the table.
Pages builds from the workflow artifact, so the committed CVE_list.json
was never served while adding a fresh 53 MB blob to history every day.
Generate it during the Pages build instead and drop the duplicate
generator; scripts/build_site.py is now the only thing that writes it.
The jinja2 templates re-rendered a trending table that logic.js replaces
on load, so the page shell goes back to being a plain static file and the
Pages build no longer installs anything. Shallow-clone the sync job too:
it never reads git history, and the three-day lookback already recovers
a skipped day.
0xMarcio