Find this year's PoCs within the hour and stop indexing bare shells

The daily sweep found new repositories within 2 to 9 hours where their
name carried the CVE, and up to 30 hours where only the README did. The
hourly run now searches repositories pushed in the last day for this
year's and last year's CVEs, reads their READMEs and applies the same
classifier, so a PoC published this morning is on the site within the
hour. The daily run keeps covering every year and the CVE record
references; the hourly run skips those, and skipping leaves the delta
checkpoint alone.

A sample of two hundred indexed repositories held six with nothing in
them: an empty README or none, a placeholder, a link to a paywall. A
repository with no directory, no code, no file named for the CVE or as a
PoC, and a README with no code block, command, request or real write-up
is bare, and neither discovery nor the daily audit keeps it. The audit
now reads each repository's README and root to judge that, and sweeps
4,000 repositories a day instead of 1,750, since one link in eight
pointed at a deleted repository under the fortnightly cycle.

The upstream PoC-in-GitHub collection is judged daily rather than
weekly: fourteen of the last thirty-nine additions came from it, which
means it finds repositories our own search does not.
This commit is contained in:
0xMarcio committed 2026-09-04 03:46:23 +02:00
1 parent 90355ee5c7
commit c8f213af17
5 files changed
+81 -15

No files matched your search

+5 -6
View File
@@ -16,7 +16,7 @@ concurrency:
jobs:
audit:
runs-on: ubuntu-latest
timeout-minutes: 30
timeout-minutes: 50
steps:
- name: Checkout repository
uses: actions/checkout@v7
@@ -26,14 +26,13 @@ jobs:
with:
python-version: '3.13'
# A seventh of the old weekly batch every day: the 12,000-and-12,000
# weekly run measured an hour, and it holds the repository-updates lock
# the whole time, which stalls the hourly refresh and can cancel a daily
# sync queued behind it. Same sweep period, about nine minutes a day.
# 4,000 repositories and 3,000 reference links a day sweeps the whole
# index about weekly, in around half an hour. A sample found one link in
# eight pointing at a deleted repository under the old fortnightly sweep.
- name: Drop dead links and refresh repository metadata
env:
GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
run: python3 scripts/audit_poc_links.py --limit 1750 --references 1750
run: python3 scripts/audit_poc_links.py --limit 4000 --references 3000
- name: Verify the generated index
run: python3 scripts/build_site.py
+11
View File
@@ -26,6 +26,15 @@ jobs:
with:
python-version: '3.13'
# Every repository pushed in the last day that names a CVE of this year
# or last, with its README read and the classifier applied, so a PoC
# published this morning is on the site within the hour. The daily run
# keeps covering every year and the CVE record references.
- name: Discover this year's PoCs pushed in the last day
env:
GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
run: python3 scripts/update_cves.py --lookback-days 1 --years 2 --skip-cvelist
- name: Ingest the previous hour's candidates
env:
GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
@@ -52,6 +61,8 @@ jobs:
README.md \
index/trending.json \
index/github.txt \
index/references.txt \
index/reference_pocs.txt \
index/cve_dates.json \
index/cve_record_state.json \
docs/hero.svg \
+4 -3
View File
@@ -2,7 +2,7 @@ name: Sync PoC-in-GitHub
on:
schedule:
- cron: '10 5 * * 3'
- cron: '10 5 * * *'
workflow_dispatch:
permissions:
@@ -26,10 +26,11 @@ jobs:
with:
python-version: '3.13'
# Every candidate is judged again each week rather than remembered as
# Every candidate is judged again each day rather than remembered as
# rejected: a repository that was an empty shell in March is often a
# working exploit by June, and re-reading 6,800 of them costs five
# minutes.
# minutes. Daily because the collection keeps finding repositories our
# own search missed.
- name: Judge upstream PoC candidates
env:
GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}