The daily sweep found new repositories within 2 to 9 hours where their
name carried the CVE, and up to 30 hours where only the README did. The
hourly run now searches repositories pushed in the last day for this
year's and last year's CVEs, reads their READMEs and applies the same
classifier, so a PoC published this morning is on the site within the
hour. The daily run keeps covering every year and the CVE record
references; the hourly run skips those, and skipping leaves the delta
checkpoint alone.
A sample of two hundred indexed repositories held six with nothing in
them: an empty README or none, a placeholder, a link to a paywall. A
repository with no directory, no code, no file named for the CVE or as a
PoC, and a README with no code block, command, request or real write-up
is bare, and neither discovery nor the daily audit keeps it. The audit
now reads each repository's README and root to judge that, and sweeps
4,000 repositories a day instead of 1,750, since one link in eight
pointed at a deleted repository under the fortnightly cycle.
The upstream PoC-in-GitHub collection is judged daily rather than
weekly: fourteen of the last thirty-nine additions came from it, which
means it finds repositories our own search does not.
Every writer now has a timeout, so a hung download cannot hold the
repository-updates lock for six hours and cancel the runs queued behind
it. The link audit moves from one 12,000-and-12,000 batch a week, which
measured an hour under that lock, to a seventh of it every day.
The social card PNG is drawn at deploy time and no longer committed: it
changed about four times a day at 68 KB a copy. nuclei.json stops
mirroring EPSS scores; every indexed CVE already has one in epss.json,
the site never fell back to the copy, and the daily recomputation
rewrote 3,400 lines of it each night.
The metadata sync retries GitHub advisory calls on 403, 429 and 5xx
instead of failing on the first one, and refuses to publish a year
shard that an NVD feed would shrink by a tenth or delete outright.
GitHub's cron scheduler created the hourly run about five times a day,
four to five hours late. A systemd timer on the VPS now dispatches it every
hour. The cron stays as fallback and watchdog: a gate job skips it while
the trigger is alive, lets it run when the trending table is over 55
minutes old, and fails the run once it is over three hours old so the
failure email names the cause.
Also stage docs/kev.svg in every writer that redraws it, verify the index
in the link audit before committing like the other writers do, and refresh
the apt index before installing librsvg.
Commits pushed with the workflow token raise no push event, so site.yml
only ran on its 06:20 schedule and the site trailed the index by up to a
day. workflow_dispatch is exempt from that rule, so each writer now
triggers it after a successful push.
The push retry rebased onto the fetched main and, on a conflict, left HEAD
there; the next push then succeeded as a no-op and the run went green with
its changes discarded. Abort the rebase and fail instead.
The site called itself three different things: CVE PoC Hub in the title, CVE
Radar in the h1 and the social metadata, and cve on GitHub. Google had no
consistent site name to settle on, so none of the mentions compounded. CVE
Radar was taken anyway, by SOCRadar's free tool and by cveradar.com.
Every user-facing name and URL now comes from scripts/brand.py, so the next
move is one edit rather than a sweep through the tree.
Also repairs the parts of the site a crawler could not use:
- robots.txt and sitemap.xml were both 404. scripts/build_seo.py writes them,
wired into the Pages job. trending_poc.json, kev.json and stats.json stay
crawlable because the first paint needs them; the five bulk payloads are
disallowed, which is 13.2 MB per crawl spent on data no crawler queries.
- Search state lives in ?q= now. Results were unlinkable, so nobody could cite
one and there was no route for a SearchAction to declare. replaceState
rather than pushState, so typing does not bury the back button.
- A WebSite JSON-LD block carries the name Google prints as the site name.
- The footer carries the MITRE trademark notice.
Discovery asks GitHub for repositories pushed in the last three days, so a
proof of concept written for a 2019 CVE in 2021 is unreachable however long
the job runs. nomi-sec/PoC-in-GitHub has collected them daily since 1999.
It collects without filtering, which is how this index inherited its junk
the first time, so nothing arrives trusted: every candidate goes through
qualifying_repo_cves, the same gate a repository found by search must pass.
Of 6,945 candidates the gate rejected 4,110 and accepted 2,835, adding
links to 1,528 entries that had none or had fewer.
Vulhub joins the curated archives beside ExploitDB and Metasploit. It is
241 entries, and each one is a docker-compose file and reproduction steps
rather than a description of a flaw.
Candidates are judged again every week rather than remembered as rejected:
an empty shell in March is often a working exploit by June.
0xMarcio