91a38b269b
Pipeline: parse HTML presets, compare modlists, download from Caddy file server, create junctions/symlinks to Arma 3 Server directory. Includes update/sync flows, missing-mod reporting, OS compat layer, shared config, dep checker, comprehensive test suite (71 tests). Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
515 lines
14 KiB
Markdown
515 lines
14 KiB
Markdown
# arma-modlist-tools
|
|
|
|
Python toolchain for managing Arma 3 mod presets: parse launcher exports, compare presets, download mods from a Caddy file server, and create junction/symlink links to an Arma 3 Server installation.
|
|
|
|
---
|
|
|
|
## Table of Contents
|
|
|
|
1. [Prerequisites](#prerequisites)
|
|
2. [Installation](#installation)
|
|
3. [Configuration](#configuration)
|
|
4. [Quick Start — Full Pipeline](#quick-start--full-pipeline)
|
|
5. [Individual Scripts](#individual-scripts)
|
|
- [check_deps.py](#check_depspy)
|
|
- [parse_modlist.py](#parse_modlistpy)
|
|
- [compare_modlists.py](#compare_modlistspy)
|
|
- [fetch_mods.py](#fetch_modspy)
|
|
- [link_mods.py](#link_modspy)
|
|
- [report_missing.py](#report_missingpy)
|
|
- [sync_missing.py](#sync_missingpy)
|
|
- [update_mods.py](#update_modspy)
|
|
- [run.py](#runpy)
|
|
6. [Folder Structure](#folder-structure)
|
|
7. [Moving to a New Device](#moving-to-a-new-device)
|
|
8. [Running Tests](#running-tests)
|
|
|
|
---
|
|
|
|
## Prerequisites
|
|
|
|
| Requirement | Version | Notes |
|
|
|-------------|---------|-------|
|
|
| Python | >= 3.11 | `python --version` |
|
|
| requests | any | `pip install requests` |
|
|
| tqdm | any | `pip install tqdm` |
|
|
| Windows or Linux | — | Windows uses junctions, Linux uses symlinks |
|
|
|
|
Run the dep checker to confirm everything is ready:
|
|
|
|
```
|
|
python check_deps.py
|
|
```
|
|
|
|
---
|
|
|
|
## Installation
|
|
|
|
```bash
|
|
# Clone the repo
|
|
git clone https://git.revoluxiant.io.vn/revernomad17/arma-modlist-tools.git
|
|
cd arma-modlist-tools
|
|
|
|
# Install dependencies
|
|
pip install -r requirements.txt
|
|
|
|
# Copy config template and fill in your credentials/paths
|
|
cp config.template.json config.json
|
|
```
|
|
|
|
Edit `config.json` — see [Configuration](#configuration) below.
|
|
|
|
---
|
|
|
|
## Configuration
|
|
|
|
All scripts read a single `config.json` in the project root. Copy the template and fill in your values:
|
|
|
|
```json
|
|
{
|
|
"server": {
|
|
"base_url": "https://your-caddy-server/arma3mods/",
|
|
"username": "your_username",
|
|
"password": "your_password"
|
|
},
|
|
"paths": {
|
|
"arma_dir": "C:\\Path\\To\\Arma 3 Server",
|
|
"downloads": "downloads",
|
|
"modlist_html": "modlist_html",
|
|
"modlist_json": "modlist_json"
|
|
}
|
|
}
|
|
```
|
|
|
|
| Key | Description |
|
|
|-----|-------------|
|
|
| `server.base_url` | Root URL of your Caddy file server. Must have a trailing slash. |
|
|
| `server.username` | HTTP Basic Auth username. |
|
|
| `server.password` | HTTP Basic Auth password. |
|
|
| `paths.arma_dir` | Absolute path to the Arma 3 Server directory where links will be created. |
|
|
| `paths.downloads` | Where mod files are downloaded locally. Relative to project root. |
|
|
| `paths.modlist_html` | Folder containing exported Arma 3 Launcher preset `.html` files. |
|
|
| `paths.modlist_json` | Folder where parsed/compared JSON files are saved. |
|
|
|
|
> **Note:** `config.json` is in `.gitignore` because it contains credentials. Never commit it.
|
|
|
|
---
|
|
|
|
## Quick Start — Full Pipeline
|
|
|
|
Place your Arma 3 Launcher preset exports (`.html`) into the `modlist_html/` folder, then run:
|
|
|
|
```bash
|
|
python run.py
|
|
```
|
|
|
|
This runs all four steps in sequence:
|
|
|
|
```
|
|
Step 1/4: Parse presets — modlist_html/*.html -> modlist_json/*.json
|
|
Step 2/4: Compare presets — produces modlist_json/comparison.json
|
|
Step 3/4: Fetch mods — downloads from server -> downloads/
|
|
Step 4/4: Link mods — creates junctions/symlinks in Arma 3 Server dir
|
|
```
|
|
|
|
### Skip flags
|
|
|
|
```bash
|
|
python run.py --skip-fetch --skip-link # parse + compare only
|
|
python run.py --skip-parse --skip-compare --skip-fetch # link only
|
|
python run.py --skip-parse --skip-compare --skip-fetch --group shared # link one group only
|
|
```
|
|
|
|
| Flag | Skips |
|
|
|------|-------|
|
|
| `--skip-parse` | Step 1 (HTML parsing) |
|
|
| `--skip-compare` | Step 2 (preset comparison) |
|
|
| `--skip-fetch` | Step 3 (downloading) |
|
|
| `--skip-link` | Step 4 (linking) |
|
|
| `--group GROUP` | Link step: only link this one group (e.g. `shared`) |
|
|
|
|
---
|
|
|
|
## Individual Scripts
|
|
|
|
### check_deps.py
|
|
|
|
Verify Python version and required packages before running anything else.
|
|
|
|
```bash
|
|
python check_deps.py
|
|
```
|
|
|
|
```
|
|
Python 3.11.9 OK
|
|
OS Windows
|
|
|
|
requests 2.33.0 OK
|
|
tqdm 4.67.3 OK
|
|
|
|
All checks passed. Ready to run.
|
|
```
|
|
|
|
---
|
|
|
|
### parse_modlist.py
|
|
|
|
Parse all `.html` preset exports in `modlist_html/` and save each as JSON in `modlist_json/`.
|
|
|
|
**How to get the HTML files:** In Arma 3 Launcher → Mods → Preset → right-click → Export to HTML. Place the exported files in `modlist_html/`.
|
|
|
|
```bash
|
|
python parse_modlist.py
|
|
```
|
|
|
|
```
|
|
150th_MW_2026_v1.0.html -> modlist_json/150th_MW_2026_v1.0.json (52 mods)
|
|
150th_WW2_2026_V1.0.html -> modlist_json/150th_WW2_2026_V1.0.json (48 mods)
|
|
```
|
|
|
|
Output JSON per preset:
|
|
```json
|
|
{
|
|
"preset_name": "150th_MW_2026_v1.0",
|
|
"source_file": "150th_MW_2026_v1.0.html",
|
|
"mod_count": 52,
|
|
"mods": [
|
|
{
|
|
"name": "CBA_A3",
|
|
"source": "steam",
|
|
"url": "https://steamcommunity.com/sharedfiles/filedetails/?id=450814997",
|
|
"steam_id": "450814997"
|
|
}
|
|
]
|
|
}
|
|
```
|
|
|
|
---
|
|
|
|
### compare_modlists.py
|
|
|
|
Compare all presets in `modlist_html/` and produce `modlist_json/comparison.json` with a shared/unique breakdown.
|
|
|
|
```bash
|
|
python compare_modlists.py
|
|
```
|
|
|
|
```
|
|
Compared: 150th_MW_2026_v1.0, 150th_WW2_2026_V1.0
|
|
Shared mods : 28
|
|
Unique to 150th_MW... : 24
|
|
Unique to 150th_WW2...: 20
|
|
-> modlist_json/comparison.json
|
|
```
|
|
|
|
Output `comparison.json` structure:
|
|
```json
|
|
{
|
|
"compared_presets": ["Preset_A", "Preset_B"],
|
|
"shared": {
|
|
"mod_count": 28,
|
|
"mods": [...]
|
|
},
|
|
"unique": {
|
|
"Preset_A": { "mod_count": 24, "mods": [...] },
|
|
"Preset_B": { "mod_count": 20, "mods": [...] }
|
|
}
|
|
}
|
|
```
|
|
|
|
**Requires:** at least 2 `.html` files in `modlist_html/`.
|
|
|
|
---
|
|
|
|
### fetch_mods.py
|
|
|
|
Download all mods from the Caddy file server into `downloads/`, organized by group.
|
|
|
|
```bash
|
|
python fetch_mods.py
|
|
```
|
|
|
|
```
|
|
Loaded comparison: 150th_MW_2026_v1.0, 150th_WW2_2026_V1.0
|
|
Total mods to consider: 72
|
|
|
|
Building server index... 87 mods indexed
|
|
|
|
[1/70] @ace -> downloads/shared/@ace/ (group: shared)
|
|
addons/ace_common/... 5.2 MB [████████████] 100%
|
|
...
|
|
Done 42 downloaded 8.2 MB
|
|
|
|
Overwrite existing? [s]kip / [o]verwrite: s
|
|
```
|
|
|
|
- Mods shared across all presets go into `downloads/shared/`
|
|
- Preset-unique mods go into `downloads/<preset_name>/`
|
|
- Saves `modlist_json/missing_report.json` listing any mods not found on server
|
|
- Prompts once if destination folders already exist
|
|
|
|
**Requires:** `modlist_json/comparison.json` (run `compare_modlists.py` first).
|
|
|
|
---
|
|
|
|
### link_mods.py
|
|
|
|
Manage junction/symlink links between `downloads/` and the Arma 3 Server directory.
|
|
|
|
#### Check status
|
|
|
|
```bash
|
|
python link_mods.py status --group shared
|
|
```
|
|
|
|
```
|
|
Group : shared
|
|
Path : downloads/shared
|
|
Arma : C:\...\Arma 3 Server
|
|
|
|
Mod Status
|
|
----------------------------------------------------------
|
|
@ace [LINKED]
|
|
@cba_a3 [------]
|
|
|
|
1 / 2 linked
|
|
```
|
|
|
|
#### Create links
|
|
|
|
```bash
|
|
python link_mods.py link --group shared
|
|
```
|
|
|
|
```
|
|
Linking group: shared -> C:\...\Arma 3 Server
|
|
|
|
[+] @ace linked
|
|
[=] @cba_a3 already linked
|
|
```
|
|
|
|
#### Remove links
|
|
|
|
```bash
|
|
python link_mods.py unlink --group shared
|
|
```
|
|
|
|
Prompts for confirmation before removing links. Removing a link does **not** delete the mod files in `downloads/`.
|
|
|
|
#### List available groups
|
|
|
|
Omit `--group` to see all available groups:
|
|
|
|
```bash
|
|
python link_mods.py status
|
|
```
|
|
|
|
> **Windows note:** Creates NTFS directory junctions (`mklink /J`). No administrator rights required.
|
|
> **Linux note:** Creates standard symlinks (`os.symlink`).
|
|
|
|
---
|
|
|
|
### report_missing.py
|
|
|
|
Check which mods from `comparison.json` are missing from the file server. Saves `modlist_json/missing_report.json`.
|
|
|
|
```bash
|
|
python report_missing.py
|
|
```
|
|
|
|
```
|
|
Checking server index... 87 mods indexed
|
|
Cross-referencing 72 required mods...
|
|
|
|
Missing from server (2):
|
|
|
|
steam_id Group Name
|
|
--------------- ----------------------------- ----------------------------------------
|
|
2648308937 150th_WW2_2026_V1.0 IFA3 AIO
|
|
463939057 shared ACE3
|
|
|
|
70 / 72 found on server
|
|
Report saved: modlist_json/missing_report.json
|
|
```
|
|
|
|
**Requires:** `modlist_json/comparison.json`.
|
|
|
|
---
|
|
|
|
### sync_missing.py
|
|
|
|
Re-check the server for mods that were previously missing and download any that have since been added.
|
|
|
|
```bash
|
|
python sync_missing.py
|
|
```
|
|
|
|
```
|
|
Loading missing report: 2 mods previously missing
|
|
Re-checking server index... 89 mods indexed
|
|
|
|
Newly available: 1 mods
|
|
Still missing : 1 mods
|
|
|
|
[+] @ace -> downloads/shared/
|
|
Done 8.2 MB
|
|
|
|
Missing report updated: 1 still missing
|
|
|
|
Linking newly added mods...
|
|
shared 1 new linked, 27 already linked
|
|
```
|
|
|
|
**Flow:**
|
|
1. Loads `modlist_json/missing_report.json`
|
|
2. Re-checks server index for newly available mods
|
|
3. Downloads newly available mods to the correct group folder
|
|
4. Updates `missing_report.json` (removes mods now downloaded)
|
|
5. Runs linker for affected groups — existing links are safely skipped
|
|
|
|
**Requires:** `modlist_json/missing_report.json` (run `report_missing.py` or `fetch_mods.py` first).
|
|
|
|
---
|
|
|
|
### update_mods.py
|
|
|
|
Re-download mod files that have changed on the server without changing the modlist structure (same mods, updated file contents/versions).
|
|
|
|
Detection uses **file size comparison**: a file is considered stale if it is missing locally or its local size differs from the server-reported size.
|
|
|
|
```bash
|
|
python update_mods.py # check all groups and mods
|
|
python update_mods.py --group shared # check one group only
|
|
python update_mods.py --mod @ace # check one specific mod
|
|
python update_mods.py --force # re-download all files regardless of size
|
|
python update_mods.py --force --group shared
|
|
```
|
|
|
|
```
|
|
Building server index... 87 mods indexed
|
|
|
|
Mode: size-check
|
|
Checking 28 mod folder(s)...
|
|
|
|
[=] @cba_a3 shared 4 files up-to-date
|
|
[+] @ace shared 42 files 3 updated (8.2 MB)
|
|
[=] @rhsusaf 150th_MW 88 files up-to-date
|
|
|
|
Total: 134 files checked, 3 updated, 8.2 MB downloaded
|
|
```
|
|
|
|
> **No re-linking needed.** Junctions/symlinks already point at the `downloads/` folders, so updated files are immediately visible to the Arma 3 Server.
|
|
|
|
---
|
|
|
|
### run.py
|
|
|
|
Orchestrator that chains all four pipeline steps. Described in [Quick Start](#quick-start--full-pipeline) above.
|
|
|
|
---
|
|
|
|
## Folder Structure
|
|
|
|
```
|
|
arma-modlist-tools/
|
|
|
|
|
|- arma_modlist_tools/ # Python package (library code)
|
|
| |- __init__.py # Public exports
|
|
| |- parser.py # HTML preset parser
|
|
| |- compare.py # Preset comparison
|
|
| |- fetcher.py # Caddy server downloader
|
|
| |- linker.py # Junction/symlink manager
|
|
| |- reporter.py # Missing-mod report builder
|
|
| |- config.py # config.json loader
|
|
| |- compat.py # OS detection + encoding fix
|
|
|
|
|
|- modlist_html/ # INPUT: put your .html preset exports here
|
|
| |- MyPreset_A.html
|
|
| |- MyPreset_B.html
|
|
|
|
|
|- modlist_json/ # OUTPUT: generated JSON files (gitignored)
|
|
| |- MyPreset_A.json
|
|
| |- MyPreset_B.json
|
|
| |- comparison.json
|
|
| |- missing_report.json
|
|
|
|
|
|- downloads/ # OUTPUT: downloaded mod files (gitignored)
|
|
| |- shared/
|
|
| | |- @ace/
|
|
| | |- @cba_a3/
|
|
| |- MyPreset_A/
|
|
| |- @rhsusaf/
|
|
|
|
|
|- config.json # YOUR config (gitignored — contains credentials)
|
|
|- config.template.json # Template to copy from
|
|
|- requirements.txt
|
|
|
|
|
|- run.py # Orchestrator (parse + compare + fetch + link)
|
|
|- parse_modlist.py # Step 1 standalone
|
|
|- compare_modlists.py # Step 2 standalone
|
|
|- fetch_mods.py # Step 3 standalone
|
|
|- link_mods.py # Link management (status/link/unlink)
|
|
|- report_missing.py # Missing mod report
|
|
|- sync_missing.py # Sync newly available missing mods
|
|
|- update_mods.py # Re-download updated mod files
|
|
|- check_deps.py # Dependency checker
|
|
|- test_suite.py # Test suite
|
|
```
|
|
|
|
---
|
|
|
|
## Moving to a New Device
|
|
|
|
1. **Clone the repo** on the new device:
|
|
```bash
|
|
git clone https://git.revoluxiant.io.vn/revernomad17/arma-modlist-tools.git
|
|
cd arma-modlist-tools
|
|
```
|
|
|
|
2. **Install dependencies:**
|
|
```bash
|
|
pip install -r requirements.txt
|
|
```
|
|
|
|
3. **Create your config:**
|
|
```bash
|
|
cp config.template.json config.json
|
|
# Edit config.json with correct arma_dir and server credentials
|
|
```
|
|
|
|
4. **Add your preset exports:**
|
|
Copy your `.html` files from the Arma 3 Launcher into `modlist_html/`.
|
|
|
|
5. **Run the full pipeline:**
|
|
```bash
|
|
python check_deps.py # verify everything is ready
|
|
python run.py # parse → compare → fetch → link
|
|
```
|
|
|
|
> The `downloads/` folder can be large (several GB of mod files). On a second device you can either let `run.py` re-download everything, or copy the `downloads/` folder manually and run `python run.py --skip-fetch` to skip the download step and just create links.
|
|
|
|
---
|
|
|
|
## Running Tests
|
|
|
|
The test suite covers all modules with 71 tests. No network connection required.
|
|
|
|
```bash
|
|
python test_suite.py
|
|
```
|
|
|
|
```
|
|
------------------------------------------------------------
|
|
compat 6 tests
|
|
config 5 tests
|
|
parser 9 tests
|
|
compare 8 tests
|
|
fetcher 19 tests (pure functions, no network)
|
|
reporter 8 tests
|
|
linker 12 tests (uses temp dirs)
|
|
__init__ 2 tests
|
|
integration 2 tests
|
|
------------------------------------------------------------
|
|
Results: 71 passed, 0 failed, 0 skipped (71 total)
|
|
```
|