mirror of
https://github.com/0xMarcio/cve.git
synced 2026-10-27 16:51:03 +00:00
Find this year's PoCs within the hour and stop indexing bare shells
The daily sweep found new repositories within 2 to 9 hours where their name carried the CVE, and up to 30 hours where only the README did. The hourly run now searches repositories pushed in the last day for this year's and last year's CVEs, reads their READMEs and applies the same classifier, so a PoC published this morning is on the site within the hour. The daily run keeps covering every year and the CVE record references; the hourly run skips those, and skipping leaves the delta checkpoint alone. A sample of two hundred indexed repositories held six with nothing in them: an empty README or none, a placeholder, a link to a paywall. A repository with no directory, no code, no file named for the CVE or as a PoC, and a README with no code block, command, request or real write-up is bare, and neither discovery nor the daily audit keeps it. The audit now reads each repository's README and root to judge that, and sweeps 4,000 repositories a day instead of 1,750, since one link in eight pointed at a deleted repository under the fortnightly cycle. The upstream PoC-in-GitHub collection is judged daily rather than weekly: fourteen of the last thirty-nine additions came from it, which means it finds repositories our own search does not.
This commit is contained in:
1 parent
90355ee5c7
commit
c8f213af17
5 files changed
+81
-15
No files matched your search
@@ -16,7 +16,7 @@ concurrency:
|
||||
jobs:
|
||||
audit:
|
||||
runs-on: ubuntu-latest
|
||||
timeout-minutes: 30
|
||||
timeout-minutes: 50
|
||||
steps:
|
||||
- name: Checkout repository
|
||||
uses: actions/checkout@v7
|
||||
@@ -26,14 +26,13 @@ jobs:
|
||||
with:
|
||||
python-version: '3.13'
|
||||
|
||||
# A seventh of the old weekly batch every day: the 12,000-and-12,000
|
||||
# weekly run measured an hour, and it holds the repository-updates lock
|
||||
# the whole time, which stalls the hourly refresh and can cancel a daily
|
||||
# sync queued behind it. Same sweep period, about nine minutes a day.
|
||||
# 4,000 repositories and 3,000 reference links a day sweeps the whole
|
||||
# index about weekly, in around half an hour. A sample found one link in
|
||||
# eight pointing at a deleted repository under the old fortnightly sweep.
|
||||
- name: Drop dead links and refresh repository metadata
|
||||
env:
|
||||
GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
|
||||
run: python3 scripts/audit_poc_links.py --limit 1750 --references 1750
|
||||
run: python3 scripts/audit_poc_links.py --limit 4000 --references 3000
|
||||
|
||||
- name: Verify the generated index
|
||||
run: python3 scripts/build_site.py
|
||||
|
||||
@@ -26,6 +26,15 @@ jobs:
|
||||
with:
|
||||
python-version: '3.13'
|
||||
|
||||
# Every repository pushed in the last day that names a CVE of this year
|
||||
# or last, with its README read and the classifier applied, so a PoC
|
||||
# published this morning is on the site within the hour. The daily run
|
||||
# keeps covering every year and the CVE record references.
|
||||
- name: Discover this year's PoCs pushed in the last day
|
||||
env:
|
||||
GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
|
||||
run: python3 scripts/update_cves.py --lookback-days 1 --years 2 --skip-cvelist
|
||||
|
||||
- name: Ingest the previous hour's candidates
|
||||
env:
|
||||
GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
|
||||
@@ -52,6 +61,8 @@ jobs:
|
||||
README.md \
|
||||
index/trending.json \
|
||||
index/github.txt \
|
||||
index/references.txt \
|
||||
index/reference_pocs.txt \
|
||||
index/cve_dates.json \
|
||||
index/cve_record_state.json \
|
||||
docs/hero.svg \
|
||||
|
||||
@@ -2,7 +2,7 @@ name: Sync PoC-in-GitHub
|
||||
|
||||
on:
|
||||
schedule:
|
||||
- cron: '10 5 * * 3'
|
||||
- cron: '10 5 * * *'
|
||||
workflow_dispatch:
|
||||
|
||||
permissions:
|
||||
@@ -26,10 +26,11 @@ jobs:
|
||||
with:
|
||||
python-version: '3.13'
|
||||
|
||||
# Every candidate is judged again each week rather than remembered as
|
||||
# Every candidate is judged again each day rather than remembered as
|
||||
# rejected: a repository that was an empty shell in March is often a
|
||||
# working exploit by June, and re-reading 6,800 of them costs five
|
||||
# minutes.
|
||||
# minutes. Daily because the collection keeps finding repositories our
|
||||
# own search missed.
|
||||
- name: Judge upstream PoC candidates
|
||||
env:
|
||||
GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
|
||||
|
||||
Reference in new issue
Block a user