The daily sweep found new repositories within 2 to 9 hours where their
name carried the CVE, and up to 30 hours where only the README did. The
hourly run now searches repositories pushed in the last day for this
year's and last year's CVEs, reads their READMEs and applies the same
classifier, so a PoC published this morning is on the site within the
hour. The daily run keeps covering every year and the CVE record
references; the hourly run skips those, and skipping leaves the delta
checkpoint alone.
A sample of two hundred indexed repositories held six with nothing in
them: an empty README or none, a placeholder, a link to a paywall. A
repository with no directory, no code, no file named for the CVE or as a
PoC, and a README with no code block, command, request or real write-up
is bare, and neither discovery nor the daily audit keeps it. The audit
now reads each repository's README and root to judge that, and sweeps
4,000 repositories a day instead of 1,750, since one link in eight
pointed at a deleted repository under the fortnightly cycle.
The upstream PoC-in-GitHub collection is judged daily rather than
weekly: fourteen of the last thirty-nine additions came from it, which
means it finds repositories our own search does not.
Every writer now has a timeout, so a hung download cannot hold the
repository-updates lock for six hours and cancel the runs queued behind
it. The link audit moves from one 12,000-and-12,000 batch a week, which
measured an hour under that lock, to a seventh of it every day.
The social card PNG is drawn at deploy time and no longer committed: it
changed about four times a day at 68 KB a copy. nuclei.json stops
mirroring EPSS scores; every indexed CVE already has one in epss.json,
the site never fell back to the copy, and the daily recomputation
rewrote 3,400 lines of it each night.
The metadata sync retries GitHub advisory calls on 403, 429 and 5xx
instead of failing on the first one, and refuses to publish a year
shard that an NVD feed would shrink by a tenth or delete outright.
GitHub's cron scheduler created the hourly run about five times a day,
four to five hours late. A systemd timer on the VPS now dispatches it every
hour. The cron stays as fallback and watchdog: a gate job skips it while
the trigger is alive, lets it run when the trending table is over 55
minutes old, and fails the run once it is over three hours old so the
failure email names the cause.
Also stage docs/kev.svg in every writer that redraws it, verify the index
in the link audit before committing like the other writers do, and refresh
the apt index before installing librsvg.
Commits pushed with the workflow token raise no push event, so site.yml
only ran on its 06:20 schedule and the site trailed the index by up to a
day. workflow_dispatch is exempt from that rule, so each writer now
triggers it after a successful push.
The push retry rebased onto the fetched main and, on a conflict, left HEAD
there; the next push then succeeded as a no-op and the run went green with
its changes discarded. Abort the rebase and fail instead.
The site called itself three different things: CVE PoC Hub in the title, CVE
Radar in the h1 and the social metadata, and cve on GitHub. Google had no
consistent site name to settle on, so none of the mentions compounded. CVE
Radar was taken anyway, by SOCRadar's free tool and by cveradar.com.
Every user-facing name and URL now comes from scripts/brand.py, so the next
move is one edit rather than a sweep through the tree.
Also repairs the parts of the site a crawler could not use:
- robots.txt and sitemap.xml were both 404. scripts/build_seo.py writes them,
wired into the Pages job. trending_poc.json, kev.json and stats.json stay
crawlable because the first paint needs them; the five bulk payloads are
disallowed, which is 13.2 MB per crawl spent on data no crawler queries.
- Search state lives in ?q= now. Results were unlinkable, so nobody could cite
one and there was no route for a SearchAction to declare. replaceState
rather than pushState, so typing does not bury the back button.
- A WebSite JSON-LD block carries the name Google prints as the site name.
- The footer carries the MITRE trademark notice.
Twenty-eight year folders sat above the README, so anyone opening the
repository scrolled past 1999 through 2026 before reaching a word about what
it is. The entries move to cves/ and the files the pipeline reads and writes
move to index/, which leaves the root at README, LICENSE and five
directories.
Nothing about the data changes. Every script takes its two roots from the
same place, and all five syncs report zero writes against the moved tree,
which is what a correct repoint looks like.
A timeout kept the link either way, but the checker still followed a failed
HEAD with a GET to the same dead host, so every unreachable URL cost two
full timeouts. Return on the first network failure and drop the timeout to
eight seconds: a measured 1,500 references went from roughly three URLs a
second to fifteen.
That pass removed 98 links across 86 inventory lines, and the weekly budget
rises to 12,000 references, which sweeps the corpus every five weeks.
The audit only ever asked GitHub about repositories, leaving 60,112
reference URLs unchecked since the corpus was inherited. Sampling put hard
404s at 4%, and one host accounts for a fifth of that on its own: CIS
retired the OVAL Repository, so all 942 links to it are permanently gone
and are removed in the next commit.
The rest is diffuse, so it needs sweeping rather than a rule. Check an
eighth of the URLs each week alongside the repositories, ordered
round-robin by host so a slice does not arrive at one server as a burst.
Only a 404 or 410 removes a link: a timeout, a refused HEAD or a 403 keeps
it, and update_cves already applies the same test to references it
discovers, so this holds inherited links to the standard new ones meet.
CISA publishes a catalogue of vulnerabilities observed being exploited in
the wild, and 1,111 of its 1,685 entries are CVEs this index already holds
PoCs for. That is the one fact a reader most wants before opening an
exploit, and it was nowhere on the page.
The weekly audit stores the catalogue, the build trims it to the CVEs the
index carries, and a result row shows a KEV chip -- with the ransomware
flag where CISA records one, which covers 352 of them. 37 KB served.
A failed fetch keeps the stored catalogue rather than dropping the badge
off every entry. The chip reuses the header pill's treatment: the palette
still spends green on things you can click and amber on star counts only.
The audit already asks GitHub about every linked repository, so the same
query can carry back a star count and a last-push date for free. Results go
to repo_meta.json, which lets a search result say which of two hundred
linked PoCs is worth opening.
2.1 MB for 45,008 repositories, refreshed weekly.
Sampling the corpus put 17% of the repositories it links to behind a 404,
so readers were following links to repositories that no longer exist. The
previous audit script was 630 lines, never wired to anything, and kept its
state under a gitignored directory.
Replace it with a check that batches repository lookups through GraphQL
and removes a link only when GitHub answers NOT_FOUND for it, so a
throttle or a network error can never drop one. A quarter of the corpus
runs each week, which stays inside the token's hourly budget and sweeps
everything within the month.
0xMarcio