Moving the data files into index/ left three references behind in .github,
and the hourly job has been failing to stage its own output since. The
README has been advertising 104,016 CVEs with PoCs while the site served
89,125, and every KEV marker had dropped out of its table because the
catalogue was being read from a path that no longer exists.
Figures come from the same count build_site publishes, so the badges, the
hero card and the site cannot disagree again.
Discovery asks GitHub for repositories pushed in the last three days, so a
proof of concept written for a 2019 CVE in 2021 is unreachable however long
the job runs. nomi-sec/PoC-in-GitHub has collected them daily since 1999.
It collects without filtering, which is how this index inherited its junk
the first time, so nothing arrives trusted: every candidate goes through
qualifying_repo_cves, the same gate a repository found by search must pass.
Of 6,945 candidates the gate rejected 4,110 and accepted 2,835, adding
links to 1,528 entries that had none or had fewer.
Vulhub joins the curated archives beside ExploitDB and Metasploit. It is
241 entries, and each one is a docker-compose file and reproduction steps
rather than a description of a flaw.
Candidates are judged again every week rather than remembered as rejected:
an empty shell in March is often a working exploit by June.
Discovery only looked at the current year and the one before it, so a PoC
published this week for a 2023 CVE was never found. Exploits for older
vulnerabilities keep appearing, and the daily run has budget to spare: the
window is still three days, so the cost is five sets of search terms instead
of two against the same short window.
Twenty-eight year folders sat above the README, so anyone opening the
repository scrolled past 1999 through 2026 before reaching a word about what
it is. The entries move to cves/ and the files the pipeline reads and writes
move to index/, which leaves the root at README, LICENSE and five
directories.
Nothing about the data changes. Every script takes its two roots from the
same place, and all five syncs report zero writes against the moved tree,
which is what a correct repoint looks like.
The design asked for published and modified in the result column and the
frontend has been rendering them since the reskin, against a field nothing
supplied. Seeded from the CVE Program's daily release: 154,160 of 154,166
CVEs, so coverage on the indexed set is total.
The 590 MB archive is a one-off. Every record the delta log hands the daily
sync afterwards already carries its own dates, so update_cves keeps the file
current from work it was doing anyway.
Costs 0.86 MB gzipped on the index.
Four gaps, measured against the index before any of this was written.
ExploitDB and Metasploit are folded in the way nuclei templates were. ExploitDB
covers 25,066 CVEs and adds 30,327 links across 17,237 entries; Metasploit
covers 3,178 and adds 3,520 across 2,550. Together with the templates the index
goes from 101,809 CVEs carrying a PoC to 103,945. Neither archive skews recent,
which is why nuclei went first, but both are the difference between an old CVE
having nothing and having something that runs. Each keeps its own section and
its own field, and each leads the list ahead of the repositories: a module, an
entry and a template all run as they stand, without reading somebody's code.
EPSS now covers the index rather than the 4,274 CVEs a template happened to
rate. FIRST's daily feed scores 103,783 of 103,945, so a result can say how
likely a flaw is to be exploited in the next thirty days, not only that code
exists for it. It is fetched during the site build and never tracked here: a
daily rescore of 366,000 rows would be nothing but churn.
The trending tables also stop hiding what they exist to show. The star floor
excluded 1,059 of the 1,312 2026 PoC repositories pushed this quarter, and a
repository published this morning has no stars, so the newest row on the front
page was six days old. A separate lane drops the floor entirely and pays for it
with a ten day window, a cap of ten rows, and the same paperwork check the rest
of the table uses. The newest row is now hours old.
git would not expand [12][0-9][0-9][0-9] as a pathspec, so the job did its
work and then died on the commit. Everything it touches is already tracked
apart from nuclei.json, so stage tracked changes and name that one file.
A nuclei template is a runnable check for one CVE, which makes it a proof of
concept whether or not anyone published a repository for it. Measured against
the index: 3,952 CVEs gain a template link and 858 of those had no PoC at all,
584 of the additions falling in 2024 or later. That last figure is the point.
ExploitDB carries more CVEs overall and Metasploit is better weaponised, but
both skew historical; nuclei is where coverage of a recent CVE actually shows
up first.
Templates also carry the ratings the entries never had. Severity, CVSS with
its vector, EPSS with its percentile and CWE now sit in nuclei.json for the
site to read: 4,193 of 4,418 templates state a CVSS score and 4,274 an EPSS
score. A result can finally say how bad a flaw is and how likely it is to be
used, rather than only that somebody wrote code for it.
Templates are kept in their own section and their own field. They are not
somebody's repository and should not be ranked, starred or credited as one,
but they do lead the PoC list, being the one link that runs as it stands.
The index files are also revalidated rather than reused from cache. They are
rebuilt hourly and a returning visitor was holding whatever it downloaded
first, which cost a stale index here until the copy was thrown away by hand.
The page and the README had drifted into looking like two projects. The site
now opens on the same card the README banner draws, built as markup rather
than a second copy of the image, so it fills itself from the index totals the
page already loads and cannot go stale.
Contrast was the real problem. --neutral sat at 4.12:1 against the canvas,
under the 4.5 floor, and it carried the search placeholder, the star glyph and
every separator; descriptions were rendered in the muted token at 6.15:1 while
doing the work of body copy. Descriptions move to a subtle token at 12.26:1,
muted and neutral both come up, and links go to the brighter accent at 7.49:1.
The column headers were 9.5px mono under a tenth of an em of tracking, which
no contrast ratio was going to rescue.
Known-exploited CVEs are marked in the trending table too, not only inside a
result, since which row to read first is the one thing the table could not
say. The README swaps the fire emoji for the same KEV pill, served from the
repository so every flagged row points at one cached file.
The search field no longer repeats the totals now that the hero carries them,
and keeps its slot for the match timing.
The README opened on a bare heading and five year tables, said nothing about
what the project is, and never linked the site. Anyone landing on the repo had
to guess. It now leads with a generated banner carrying the live index totals,
a badge row, and tables trimmed to what fits a table cell.
hero.py draws the banner and the social card as self-contained SVG, since
GitHub proxies README images and drops anything the file has to fetch for
itself. It counts the corpus through build_site's own parser rather than a
second guess at what counts as a PoC, so the banner cannot drift from the site
header, and leaves the figures in docs/stats.json for the README build to
quote. If the count fails it falls back to the deployed site and, failing
that, writes nothing at all: a banner reading zero is worse than yesterday's.
Badges describing the index are baked in at build time so they always match
the banner and cannot degrade to "resource not found"; the two that describe
live GitHub state carry a stamp that defeats the image proxy cache.
The sweep now runs hourly instead of every six hours. It takes under two
minutes and the repository shows up in "recently updated" far more often.
Also adds the OpenGraph tags the site never had, so a shared link renders a
card instead of a bare URL.
A timeout kept the link either way, but the checker still followed a failed
HEAD with a GET to the same dead host, so every unreachable URL cost two
full timeouts. Return on the first network failure and drop the timeout to
eight seconds: a measured 1,500 references went from roughly three URLs a
second to fifteen.
That pass removed 98 links across 86 inventory lines, and the weekly budget
rises to 12,000 references, which sweeps the corpus every five weeks.
The audit only ever asked GitHub about repositories, leaving 60,112
reference URLs unchecked since the corpus was inherited. Sampling put hard
404s at 4%, and one host accounts for a fifth of that on its own: CIS
retired the OVAL Repository, so all 942 links to it are permanently gone
and are removed in the next commit.
The rest is diffuse, so it needs sweeping rather than a rule. Check an
eighth of the URLs each week alongside the repositories, ordered
round-robin by host so a slice does not arrive at one server as a burst.
Only a 404 or 410 removes a link: a timeout, a refused HEAD or a 403 keeps
it, and update_cves already applies the same test to references it
discovers, so this holds inherited links to the standard new ones meet.
CISA publishes a catalogue of vulnerabilities observed being exploited in
the wild, and 1,111 of its 1,685 entries are CVEs this index already holds
PoCs for. That is the one fact a reader most wants before opening an
exploit, and it was nowhere on the page.
The weekly audit stores the catalogue, the build trims it to the CVEs the
index carries, and a result row shows a KEV chip -- with the ransomware
flag where CISA records one, which covers 352 of them. 37 KB served.
A failed fetch keeps the stored catalogue rather than dropping the badge
off every entry. The chip reuses the header pill's treatment: the palette
still spends green on things you can click and amber on star counts only.
The audit already asks GitHub about every linked repository, so the same
query can carry back a star count and a last-push date for free. Results go
to repo_meta.json, which lets a search result say which of two hundred
linked PoCs is worth opening.
2.1 MB for 45,008 repositories, refreshed weekly.
The trending table sorted and displayed updated_at, which also moves when a
repository merely gains a star, so a four-month-old exploit was claiming it
had been updated three hours ago. Record pushed_at and created_at and let
the page decide: last commit for the age it shows, repository age for the
trending rank.
The job now writes trending.json alongside the README table, so the site
reads the data instead of regex-parsing markdown the same job generated.
Sampling the corpus put 17% of the repositories it links to behind a 404,
so readers were following links to repositories that no longer exist. The
previous audit script was 630 lines, never wired to anything, and kept its
state under a gitignored directory.
Replace it with a check that batches repository lookups through GraphQL
and removes a link only when GitHub answers NOT_FOUND for it, so a
throttle or a network error can never drop one. A quarter of the corpus
runs each week, which stays inside the token's hourly budget and sweeps
everything within the month.
The two scheduled jobs write disjoint paths but pin their checkout when
they are queued, so a commit landing in between throws away a completed
run. Rebase onto the current main and retry instead, deepening the clone
only on the rare occasion it is needed.
The search API was called anonymously on a 10 requests/minute limit while
the workflow already had a token; send it as an Authorization header,
never in the URL, and never print it. Dropping the token from the origin
remote also stops it being written into .git/config on the runner, since
actions/checkout already provides push credentials.
Rewrite on urllib so the job installs nothing, retry rate limits and 5xx,
and fail loudly instead of silently publishing a README with a year
missing. Repository text is flattened so a pipe cannot break the table.
Pages builds from the workflow artifact, so the committed CVE_list.json
was never served while adding a fresh 53 MB blob to history every day.
Generate it during the Pages build instead and drop the duplicate
generator; scripts/build_site.py is now the only thing that writes it.
The jinja2 templates re-rendered a trending table that logic.js replaces
on load, so the page shell goes back to being a plain static file and the
Pages build no longer installs anything. Shallow-clone the sync job too:
it never reads git history, and the three-day lookback already recovers
a skipped day.
0xMarcio