admin/toolkit
1
0
Fork 0
mirror of https://github.com/indetectables-net/toolkit.git synced 2025-06-21 18:30:51 +00:00
Code Issues Packages Projects Releases Wiki Activity

Sync last updater version

Browse Source
This commit is contained in:
DSR! 2024-12-03 11:15:01 -03:00
parent e61676e8cb
commit a79e755eca
11 changed files with 175 additions and 40 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

26
bin/updater/README.ES.md
View File

@ -31,9 +31,10 @@ re_download = die_win64_portable_(?:\S+).zip
[Portmon]
folder = Monitor\Portmon
url = https://docs.microsoft.com/en-us/sysinternals/downloads/portmon
url = https://raw.githubusercontent.com/MicrosoftDocs/sysinternals/main/sysinternals/downloads/portmon.md
update_url = https://download.sysinternals.com/files/PortMon.zip
re_version = <h1 [^>]*>Portmon for Windows v(.*?)</h1>
from = web
re_version = # Portmon v(\d+\.\d+)
```
Los valores utilizados para la configuración son:
@ -65,6 +66,27 @@ Combinando el uso de `update_url` y `re_download` se consiguen las siguientes es
Esto es útil para arreglar los links de descarga de GitHub o Sourceforge.
4. También se dispone de un método de detección de nuevas versiones que en lugar de regex usa las cabeceras http con las que responde el servidor.
## Parámetros de Línea de Comandos
El actualizador ofrece un conjunto flexible de parámetros para controlar su comportamiento:
| Parámetro | Descripción |
|--------------------------------------------------------------------|---------------------------------------------------------------------------------------------------|
| `-h, --help` | Muestra este mensaje de ayuda y finaliza. |
| `-v, --version` | Muestra el número de versión del programa y finaliza. |
| `-u [UPDATE ...], --update [UPDATE ...]` | Especifica una lista de herramientas a actualizar. Si no se proporciona, se actualizarán todas. |
| `-dsu, --disable-self-update` | Desactiva la auto-actualización automática del script. |
| `-dfc, --disable-folder-clean` | Evita limpiar la carpeta de herramientas durante las actualizaciones. |
| `-dr, --disable-repack` | Impide empaquetar nuevamente las herramientas después del proceso de actualización. |
| `-dic, --disable-install-check` | Omite la verificación de si las herramientas están instaladas correctamente. |
| `-dpb, --disable-progress-bar` | Desactiva la barra de progreso durante las descargas. |
| `-sft {full,version,name}, --save-format-type {full,version,name}` | Especifica el tipo de formato para guardar las actualizaciones comprimidas: `full`, `version` o `name`. |
| `-f, --force` | Fuerza la descarga de actualizaciones, incluso si ya están actualizadas. |
| `-uga USE_GITHUB_API, --use-github-api USE_GITHUB_API` | Usa la API de GitHub para actualizaciones, especificando el token para autenticarse. |
| `-udp, --update-default-params` | Actualiza los parámetros predeterminados almacenados en la configuración. |
| `-dmc, --disable-mutex-check` | Permite ejecutar múltiples instancias del script desactivando la verificación de mutex. |
| `-d, --debug` | Activa la salida detallada de depuración para resolver problemas. |
## Ejemplos
La herramienta soporta varios comandos y combinaciones. Estos son los mas usados.

26
bin/updater/README.md
View File

@ -31,9 +31,10 @@ re_download = die_win64_portable_(?:\S+).zip
[Portmon]
folder = Monitor\Portmon
url = https://docs.microsoft.com/en-us/sysinternals/downloads/portmon
url = https://raw.githubusercontent.com/MicrosoftDocs/sysinternals/main/sysinternals/downloads/portmon.md
update_url = https://download.sysinternals.com/files/PortMon.zip
re_version = <h1 [^>]*>Portmon for Windows v(.*?)</h1>
from = web
re_version = # Portmon v(\d+\.\d+)
```
The values used for configuration are:
@ -63,6 +64,27 @@ Combining the use of `update_url` and `re_download` the following download strat
This is useful for fixing GitHub or Sourceforge download links.
4. A new version detection method is also available that instead of regex uses the http headers with which the server responds.
## Command-line Parameters
The updater provides a flexible set of parameters to control its behavior:
| Parameter | Description |
|--------------------------------------------------------------------|---------------------------------------------------------------------------------------------------|
| `-h, --help` | Show this help message and exit. |
| `-v, --version` | Display the program's version number and exit. |
| `-u [UPDATE ...], --update [UPDATE ...]` | Specify a list of tools to update. Defaults to updating all tools if not provided. |
| `-dsu, --disable-self-update` | Disable automatic self-update of this script. |
| `-dfc, --disable-folder-clean` | Skip cleaning the tool's folder during updates. |
| `-dr, --disable-repack` | Prevent repacking of tools after the update process. |
| `-dic, --disable-install-check` | Skip checking if the tools are properly installed. |
| `-dpb, --disable-progress-bar` | Disable the download progress bar for updates. |
| `-sft {full,version,name}, --save-format-type {full,version,name}` | Specify the save format type for compressed updates: `full`, `version`, or `name`. |
| `-f, --force` | Force the download of updates, even if they appear up to date. |
| `-uga USE_GITHUB_API, --use-github-api USE_GITHUB_API` | Use the GitHub API for updates, specifying the token to authenticate. |
| `-udp, --update-default-params` | Update the default parameters stored in the configuration. |
| `-dmc, --disable-mutex-check` | Allow multiple instances of the script to run simultaneously by disabling the mutex check. |
| `-d, --debug` | Enable detailed debug output for troubleshooting. |
## Examples
The tool supports various commands and combinations. These are the most used.

3
bin/updater/scripts/Toolkit-Updater.bat
View File

@ -4,6 +4,9 @@ echo Update all Universal Update stuff and restart...
:: Kill updater process to be able to update all the files
taskkill /IM updater.exe /F
:: Clean updater mutex
del mutex.lock
:: Backup user tools.ini
move tools.ini tools.ini.old

102
bin/updater/src/UpdateManager.py
View File

@ -4,6 +4,7 @@ import sys
import os
import colorama
import logging
import psutil
from universal_updater.Updater import Updater
from universal_updater.ConfigManager import ConfigManager
@ -19,7 +20,8 @@ class UpdateManager:
"""
Initialize the UpdateManager with a ConfigManager instance and command-line arguments.
"""
self.version = '2.2.0'
self.version = '2.3.1'
self.process_mutex = 'mutex.lock'
self.config_file_name = 'tools.ini'
self.config_section_defaults = 'UpdaterConfig'
self.config_section_self_update = 'UpdaterAutoUpdater'
@ -43,16 +45,49 @@ class UpdateManager:
Version: {self.version}
""")
def exit_handler(self, signal, frame):
def exit_handler(self, signum, frame):
"""
Handles signals like SIGINT for graceful exit.
Handles signals like SIGINT or SIGTERM for graceful exit.
:param signal: Signal type
:param signum: Signal type (e.g., SIGINT, SIGTERM)
:param frame: Current stack frame
"""
print(colorama.Fore.YELLOW + 'SIGINT or CTRL-C detected. Exiting gracefully')
signal_name = 'SIGINT' if signum == signal.SIGINT else 'SIGTERM'
print(colorama.Fore.YELLOW + f'{signal_name} detected. Exiting gracefully')
self.cleanup_mutex()
sys.exit(0)
def check_single_instance(self):
"""
Ensures only a single instance of the script is running by creating a lock file with the current PID.
If the lock file exists and contains a different PID, the script will exit.
"""
if self.arguments.disable_mutex_check:
print('Mutex check is disabled. Multiple instances can run concurrently.')
return
if os.path.exists(self.process_mutex):
with open(self.process_mutex, 'r') as lock:
existing_pid = int(lock.read().strip())
# Verify if the process with the PID exists
if existing_pid and psutil.pid_exists(existing_pid):
print(f"Another instance of the script is already running (PID: {existing_pid}). Exiting.")
sys.exit(1)
else:
print(f"Stale mutex detected (PID: {existing_pid}). Regenerating with current PID.")
# Create a new lock file with the current PID
with open(self.process_mutex, 'w') as lock:
lock.write(str(os.getpid()))
def cleanup_mutex(self):
"""
Removes the mutex file if mutex check is enabled and the file exists.
"""
if not self.arguments.disable_mutex_check and os.path.exists(self.process_mutex):
os.remove(self.process_mutex)
def get_argparse_default(self, option, default, is_bool=True):
"""
Retrieves the default value for a given argparse option from the configuration.
@ -83,54 +118,54 @@ class UpdateManager:
'-u',
'--update',
dest='update',
help='list of tools (default: all)',
help='Specify a list of tools to update. Defaults to updating all tools if not provided.',
nargs='*'
)
parser.add_argument(
'-dsu',
'--disable-self-update',
dest='disable_self_update',
help='disable self update of this script',
action=argparse.BooleanOptionalAction,
help='Disable automatic self-update of this script.',
action='store_true',
default=False
)
parser.add_argument(
'-dfc',
'--disable-folder-clean',
dest='disable_clean',
help='disable tool folder clean',
action=argparse.BooleanOptionalAction,
help='Skip cleaning the tool\'s folder during updates.',
action='store_true',
default=self.get_argparse_default('disable_clean', True)
)
parser.add_argument(
'-dr',
'--disable-repack',
dest='disable_repack',
help='disable tool repack',
action=argparse.BooleanOptionalAction,
help='Prevent repacking of tools after the update process.',
action='store_true',
default=self.get_argparse_default('disable_repack', True)
)
parser.add_argument(
'-dic',
'--disable-install-check',
dest='disable_install_check',
help='disable tool install check',
action=argparse.BooleanOptionalAction,
help='Skip checking if the tools are properly installed.',
action='store_true',
default=self.get_argparse_default('disable_install_check', False)
)
parser.add_argument(
'-dpb',
'--disable-progress-bar',
dest='disable_progress',
help='disable download progress bar',
action=argparse.BooleanOptionalAction,
help='Disable the download progress bar for updates.',
action='store_true',
default=self.get_argparse_default('disable_progress', False)
)
parser.add_argument(
'-sft',
'--save-format-type',
dest='save_format_type',
help='compress save format name',
help='Specify the save format type for compressed updates: "full", "version", or "name".',
choices=['full', 'version', 'name'],
default=self.get_argparse_default('save_format_type', 'full', False)
)
@ -138,31 +173,39 @@ class UpdateManager:
'-f',
'--force',
dest='force_download',
help='force download',
action=argparse.BooleanOptionalAction,
help='Force the download of updates, even if they appear up to date.',
action='store_true',
default=False
)
parser.add_argument(
'-uga',
'--use-github-api',
dest='use_github_api',
help='use github api with this token',
help='Use the GitHub API for updates, specifying the token to authenticate.',
default=self.get_argparse_default('use_github_api', '', False)
)
parser.add_argument(
'-udp',
'--update-default-params',
dest='update_default_params',
help='update default params',
action=argparse.BooleanOptionalAction,
help='Update the default parameters stored in the configuration.',
action='store_true',
default=False
)
parser.add_argument(
'-dmc',
'--disable-mutex-check',
dest='disable_mutex_check',
help='Allow multiple instances of the script to run simultaneously by disabling the mutex check.',
action='store_true',
default=False
)
parser.add_argument(
'-d',
'--debug',
dest='debug',
help='enable debug output',
action=argparse.BooleanOptionalAction,
help='Enable detailed debug output for troubleshooting.',
action='store_true',
default=False
)
@ -249,7 +292,7 @@ class UpdateManager:
try:
updater.update(self.config_section_self_update)
except Exception as exception:
logging.info(exception)
logging.error(exception)
# add missing new line separator
logging.info("\n")
@ -269,7 +312,8 @@ class UpdateManager:
try:
updater.update(tool)
except Exception as exception:
logging.info(exception)
failed_updates += 1
logging.error(exception)
logging.info(colorama.Fore.YELLOW + f"\n[*] Update process completed: {total_updates - failed_updates} succeeded, {failed_updates} failed out of {total_updates} total updates.")
@ -293,13 +337,19 @@ class UpdateManager:
"""
Main entry point for the UpdateManager.
"""
# setup signals
signal.signal(signal.SIGINT, self.exit_handler)
signal.signal(signal.SIGTERM, self.exit_handler)
# setup script
self.change_current_directory()
self.print_banner()
self.parse_arguments()
self.set_logging_level()
self.check_single_instance()
self.update_default_params()
self.handle_updates()
self.cleanup_mutex()
# Entry point for the script

1
bin/updater/src/requirements.txt
View File

@ -3,3 +3,4 @@ colorama>=0.4.4
tqdm>=4.62.3
py7zr>=0.16.1
rarfile>=4.0
psutil>=6.1.0

1
bin/updater/src/universal_updater/Packer.py
View File

@ -62,6 +62,7 @@ class Packer:
"""
# download first "UnRAR for Windows" from https://www.rarlab.com/rar_add.htm
# direct link: https://www.rarlab.com/rar/unrarw32.exe
# new link: https://www.win-rar.com/predownload.html?&Version=32bit&L=0.
with rarfile.RarFile(file_path, 'r') as compressed:
compressed.extractall(unpack_path, pwd=file_pass)

47
bin/updater/src/universal_updater/Scraper.py
View File

@ -65,7 +65,11 @@ class Scraper:
"""
Scrape web for version and download URL based on tool_config.
:return: Dictionary containing 'download_version' and 'download_url'
:return: dict|bool: A dictionary containing:
- 'download_version' (str): Extracted version using regex.
- 'download_url' (str): Extracted or generated download URL.
Returns False if the version cannot be extracted.
:raises Exception: If required configuration fields are missing or HTTP requests fail.
"""
update_url = self.tool_config.get('update_url', None)
re_version = self.tool_config.get('re_version', None)
@ -78,6 +82,9 @@ class Scraper:
# regex shit
download_version = self.check_version_from_web(html_response.text, re_version)
if download_version is None:
return False
download_url = self.get_download_url_from_web(html_response.text, url, update_url, re_download)
logging.debug(f'{self.tool_name}: Regex matching done.')
@ -90,7 +97,11 @@ class Scraper:
"""
Scrape GitHub for version and download URL based on tool_config.
:return: Dictionary containing 'download_version' and 'download_url'
:return: dict|bool: A dictionary containing:
- 'download_version' (str): Extracted version from GitHub.
- 'download_url' (str): The determined or generated download URL.
Returns False if the version cannot be extracted.
:raises Exception: If required configuration fields are missing or HTTP requests fail.
"""
if self.use_github_api:
return self.scrape_github_api()
@ -103,6 +114,9 @@ class Scraper:
logging.debug(f'{self.tool_name}: Version HTML fetched, starting regex matching for version.')
download_version = self.check_version_from_web(version_html_response.text, self.re_github_version)
if download_version is None:
return False
logging.debug(f'{self.tool_name}: Regex matching for version done.')
# load second html
@ -124,7 +138,11 @@ class Scraper:
"""
Scrape GitHub API for version and download URL based on tool_config.
:return: Dictionary containing 'download_version' and 'download_url'
:return: dict|bool: A dictionary containing:
- 'download_version' (str): Extracted version from the API response.
- 'download_url' (str): The determined or generated download URL.
Returns False if the version cannot be extracted.
:raises Exception: If the API request fails or the response is invalid.
"""
logging.debug(f'{self.tool_name}: Consuming GitHub via Api')
github_repo = self.tool_config.get('url', None)
@ -142,6 +160,9 @@ class Scraper:
update_url = self.get_download_url_from_github(json_response)
download_version = self.check_version_from_github(json_response)
if download_version is None:
return False
logging.debug(f'{self.tool_name}: Version and download URL extracted.')
# regex shit
@ -154,7 +175,11 @@ class Scraper:
"""
Scrape HTTP headers for version based on tool_config.
:return: Dictionary containing 'download_version' and 'download_url'
:return: dict|bool: A dictionary containing:
- 'download_version' (str): Extracted version from the headers.
- 'download_url' (str): The update URL.
Returns False if the version cannot be extracted.
:raises Exception: If 'update_url' is missing or an HTTP error occurs.
"""
# get http response
update_url = self.tool_config.get('update_url', None)
@ -171,6 +196,9 @@ class Scraper:
raise Exception(colorama.Fore.RED + f'{self.tool_name}: Error {exception}')
download_version = self.check_version_from_http(http_response.headers)
if download_version is None:
return False
logging.debug(f'{self.tool_name}: Version extracted.')
return {
@ -196,7 +224,8 @@ class Scraper:
raise Exception(colorama.Fore.RED + f'{self.tool_name}: re_version regex not match ({re_version})')
if not self.force_download and local_version == html_regex_version[0]:
raise Exception(f'{self.tool_name}: {local_version} is the latest version')
logging.info(f'{self.tool_name}: {local_version} is the latest version')
return None
logging.info(f'{self.tool_name}: updated from {local_version} --> {html_regex_version[0]}')
@ -225,7 +254,8 @@ class Scraper:
f'{self.tool_name}: no header is found with which to determine if there is an update')
if not self.force_download and local_version == remote_version:
raise Exception(f'{self.tool_name}: {local_version} is the latest version')
logging.info(f'{self.tool_name}: {local_version} is the latest version')
return None
logging.info(f'{self.tool_name}: updated from {local_version} --> {remote_version}')
@ -241,14 +271,15 @@ class Scraper:
local_version = self.tool_config.get('local_version', '0')
if not self.force_download and local_version == json['tag_name']:
raise Exception(f'{self.tool_name}: {local_version} is the latest version')
logging.info(f'{self.tool_name}: {local_version} is the latest version')
return None
logging.info(f'{self.tool_name}: updated from {local_version} --> {json["tag_name"]}')
return json['tag_name']
#################
# Check methods
# Download url methods
#################
def get_download_url_from_web(self, html, html_url, update_url, re_download):
"""

5
bin/updater/src/universal_updater/Updater.py
View File

@ -141,6 +141,8 @@ class Updater:
Perform the update process for a given tool.
:param tool_name: Name of the tool to update
:return: bool: True if the update completes successfully, False if no update is needed.
:raises Exception: If any step in the update process fails.
"""
# tool data setup
self.tool_name = tool_name
@ -155,6 +157,8 @@ class Updater:
# generate version and download data
scrape_data = self.scraper.scrape_step()
if scrape_data is False:
return False
# download and process file
update_file_path = self.download_step(scrape_data['download_url'])
@ -165,3 +169,4 @@ class Updater:
self.post_update(processing_info)
logging.info(f'{self.tool_name}: update complete')
return True

4
bin/updater/tools.ini
View File

@ -39,7 +39,7 @@ folder = ..\..\toolkit\Analysis\PE-Bear
url = hasherezade/pe-bear
from = github
local_version = v0.6.7.3
re_download = PE-bear_(?:\S+)_x64_win_vs13.zip
re_download = PE-bear_(?:\S+)_qt5_x86_win_vs19.zip
[PEStudio]
folder = ..\..\toolkit\Analysis\PEStudio
@ -521,7 +521,7 @@ folder = ..\..\bin\updater
url = indetectables-net/toolkit-updater
update_url = https://github.com/indetectables-net/toolkit-updater/archive/refs/heads/main.zip
from = github
local_version = 2024.7.1
local_version = 2024.7.3
post_unpack = scripts\Toolkit-Updater.bat
disable_repack = True

BIN
bin/updater/unrar.exe
View File

Binary file not shown.

BIN
bin/updater/updater.exe
View File

Binary file not shown.