arma-modlist-tools/README.md

# arma-modlist-tools

Python toolchain for managing Arma 3 mod presets: parse launcher exports, compare presets, download mods from a Caddy file server, and create junction/symlink links to an Arma 3 Server installation.

---

## Table of Contents

1. [Prerequisites](#prerequisites)
2. [Installation](#installation)
3. [Configuration](#configuration)
4. [Quick Start — Full Pipeline](#quick-start--full-pipeline)
5. [Individual Scripts](#individual-scripts)
   - [check_deps.py](#check_depspy)
   - [parse_modlist.py](#parse_modlistpy)
   - [compare_modlists.py](#compare_modlistspy)
   - [fetch_mods.py](#fetch_modspy)
   - [link_mods.py](#link_modspy)
   - [report_missing.py](#report_missingpy)
   - [sync_missing.py](#sync_missingpy)
   - [update_mods.py](#update_modspy)
   - [check_names.py](#check_namespy)
   - [run.py](#runpy)
6. [Migrating Existing Mods](#migrating-existing-mods)
7. [Folder Structure](#folder-structure)
8. [Moving to a New Device](#moving-to-a-new-device)
9. [Running Tests](#running-tests)

---

## Prerequisites

| Requirement | Version | Notes |
|-------------|---------|-------|
| Python | >= 3.9 | `python --version` |
| requests | any | `pip install requests` |
| tqdm | any | `pip install tqdm` |
| Windows or Linux | — | Windows uses junctions, Linux uses symlinks |

Run the dep checker to confirm everything is ready:

```
python check_deps.py
```

---

## Installation

```bash
# Clone the repo
git clone https://git.revoluxiant.io.vn/revernomad17/arma-modlist-tools.git
cd arma-modlist-tools

# Install dependencies
pip install -r requirements.txt

# Copy config template and fill in your credentials/paths
cp config.template.json config.json
```

Edit `config.json` — see [Configuration](#configuration) below.

---

## Configuration

All scripts read a single `config.json` in the project root. Copy the template and fill in your values:

```json
{
  "server": {
    "base_url": "https://your-caddy-server/arma3mods/",
    "username": "your_username",
    "password": "your_password"
  },
  "paths": {
    "arma_dir":    "C:\\Path\\To\\Arma 3 Server",
    "downloads":   "downloads",
    "modlist_html": "modlist_html",
    "modlist_json": "modlist_json"
  }
}
```

| Key | Description |
|-----|-------------|
| `server.base_url` | Root URL of your Caddy file server. Must have a trailing slash. |
| `server.username` | HTTP Basic Auth username. |
| `server.password` | HTTP Basic Auth password. |
| `paths.arma_dir` | Absolute path to the Arma 3 Server directory where links will be created. |
| `paths.downloads` | Where mod files are downloaded locally. Relative to project root. |
| `paths.modlist_html` | Folder containing exported Arma 3 Launcher preset `.html` files. |
| `paths.modlist_json` | Folder where parsed/compared JSON files are saved. |

> **Note:** `config.json` is in `.gitignore` because it contains credentials. Never commit it.

---

## Quick Start — Full Pipeline

Place your Arma 3 Launcher preset exports (`.html`) into the `modlist_html/` folder, then run:

```bash
python run.py
```

This runs all four steps in sequence:

```
Step 1/4: Parse presets       — modlist_html/*.html  ->  modlist_json/*.json
Step 2/4: Compare presets     — produces modlist_json/comparison.json
Step 3/4: Fetch mods          — downloads from server  ->  downloads/
Step 4/4: Link mods           — creates junctions/symlinks in Arma 3 Server dir
```

### Skip flags

```bash
python run.py --skip-fetch --skip-link          # parse + compare only
python run.py --skip-parse --skip-compare --skip-fetch   # link only
python run.py --skip-parse --skip-compare --skip-fetch --group shared
```

| Flag | Skips |
|------|-------|
| `--skip-parse` | Step 1 (HTML parsing) |
| `--skip-compare` | Step 2 (preset comparison) |
| `--skip-fetch` | Step 3 (downloading) |
| `--skip-link` | Step 4 (linking) |
| `--group GROUP` | Link step: only link this one group (e.g. `shared`) |

> **Safe to re-run.** Every step is idempotent — existing files are skipped, already-linked mods are skipped.

---

## Individual Scripts

### check_deps.py

Verify Python version and required packages before running anything else.

```bash
python check_deps.py
```

```
  Python         3.9.2         OK
  OS             Windows Server

  requests       2.33.0        OK
  tqdm           4.67.3        OK

  All checks passed. Ready to run.
```

---

### parse_modlist.py

Parse all `.html` preset exports in `modlist_html/` and save each as JSON in `modlist_json/`.

**How to get the HTML files:** In Arma 3 Launcher → Mods → Preset → right-click → Export to HTML. Place the exported files in `modlist_html/`.

```bash
python parse_modlist.py
```

```
150th_MW_2026_v1.0.html  ->  modlist_json/150th_MW_2026_v1.0.json  (52 mods)
150th_WW2_2026_V1.0.html ->  modlist_json/150th_WW2_2026_V1.0.json (48 mods)
```

Output JSON per preset:
```json
{
  "preset_name": "150th_MW_2026_v1.0",
  "source_file": "150th_MW_2026_v1.0.html",
  "mod_count": 52,
  "mods": [
    {
      "name": "CBA_A3",
      "source": "steam",
      "url": "https://steamcommunity.com/sharedfiles/filedetails/?id=450814997",
      "steam_id": "450814997"
    }
  ]
}
```

---

### compare_modlists.py

Compare all presets in `modlist_html/` and produce `modlist_json/comparison.json` with a shared/unique breakdown.

```bash
python compare_modlists.py
```

```
Compared: 150th_MW_2026_v1.0, 150th_WW2_2026_V1.0
  Shared mods           : 28
  Unique to 150th_MW... : 24
  Unique to 150th_WW2...: 20
  -> modlist_json/comparison.json
```

Output `comparison.json` structure:
```json
{
  "compared_presets": ["Preset_A", "Preset_B"],
  "shared": {
    "mod_count": 28,
    "mods": [...]
  },
  "unique": {
    "Preset_A": { "mod_count": 24, "mods": [...] },
    "Preset_B": { "mod_count": 20, "mods": [...] }
  }
}
```

**Requires:** at least 2 `.html` files in `modlist_html/`.

---

### fetch_mods.py

Download all mods from the Caddy file server into `downloads/`, organized by group.

```bash
python fetch_mods.py
```

```
Loaded comparison: 150th_MW_2026_v1.0, 150th_WW2_2026_V1.0
Total mods to consider: 72

Building server index...  87 mods indexed

[1/70] @ace  ->  downloads/shared/@ace/  (group: shared)
  addons/ace_common/...    5.2 MB  [################] 100%
  ...
  Done  42 downloaded   8.2 MB

Overwrite existing? [s]kip / [o]verwrite: s
```

- Mods shared across all presets go into `downloads/shared/`
- Preset-unique mods go into `downloads/<preset_name>/`
- Saves `modlist_json/missing_report.json` listing any mods not found on server
- Prompts once if destination folders already exist

**Requires:** `modlist_json/comparison.json` (run `compare_modlists.py` first).

---

### link_mods.py

Manage junction/symlink links between `downloads/` and the Arma 3 Server directory.

#### Check status

```bash
python link_mods.py status --group shared
```

```
  Group  : shared
  Path   : downloads/shared
  Arma   : C:\...\Arma 3 Server

  Mod                                                  Status
  ----------------------------------------------------------
  @ace                                                 [LINKED]
  @cba_a3                                              [------]

  1 / 2 linked
```

#### Create links

```bash
python link_mods.py link --group shared
```

#### Remove links

```bash
python link_mods.py unlink --group shared
```

Prompts for confirmation before removing links. Removing a link does **not** delete the mod files in `downloads/`.

#### List available groups

Omit `--group` to see what groups exist:

```bash
python link_mods.py status
```

> **Windows:** Creates NTFS directory junctions (`mklink /J`). No administrator rights required.
> **Linux:** Creates standard symlinks (`os.symlink`).

---

### report_missing.py

Check which mods from `comparison.json` are missing from the file server. Saves `modlist_json/missing_report.json`.

```bash
python report_missing.py
```

```
Checking server index...  87 mods indexed
Cross-referencing 72 required mods...

  Missing from server (2):

  steam_id        Group                         Name
  --------------- ----------------------------- ----------------------------------------
  2648308937      150th_WW2_2026_V1.0           IFA3 AIO
  463939057       shared                        ACE3

  70 / 72 found on server
  Report saved: modlist_json/missing_report.json
```

**Requires:** `modlist_json/comparison.json`.

---

### sync_missing.py

Re-check the server for mods that were previously missing and download any that have since been added to the server.

```bash
python sync_missing.py
```

```
Loading missing report: 2 mods previously missing
Re-checking server index...  89 mods indexed

  Newly available: 1 mods
  Still missing  : 1 mods

  [+] @ace  ->  downloads/shared/
     Done  8.2 MB

  Missing report updated: 1 still missing

  Linking newly added mods...
    shared    1 new linked, 27 already linked
```

**Flow:**
1. Loads `modlist_json/missing_report.json`
2. Re-checks server index for newly available mods
3. Downloads newly available mods to the correct group folder
4. Updates `missing_report.json` (removes mods now downloaded)
5. Runs linker for affected groups — existing links are safely skipped

**Requires:** `modlist_json/missing_report.json` (run `report_missing.py` or `fetch_mods.py` first).

---

### update_mods.py

Re-download mod files that have changed on the server without changing the modlist structure (same mods, updated file versions).

Detection uses **file size comparison**: a file is re-downloaded if it is missing locally or its local size differs from the server-reported size. Use `--force` to re-download everything unconditionally.

```bash
python update_mods.py                       # check all groups and mods
python update_mods.py --group shared        # one group only
python update_mods.py --mod @ace            # one specific mod
python update_mods.py --force               # re-download all files
python update_mods.py --force --group shared
```

```
Building server index...  87 mods indexed

  Mode: size-check
  Checking 28 mod folder(s)...

  [=] @cba_a3     shared     4 files   up-to-date
  [+] @ace        shared     42 files  3 updated  (8.2 MB)
  [=] @rhsusaf    150th_MW   88 files  up-to-date

  Total: 134 files checked, 3 updated, 8.2 MB downloaded
```

> **No re-linking needed.** Junctions already point at the `downloads/` folders, so updated files are immediately visible to the Arma 3 Server.

---

### check_names.py

Diagnostic tool that compares mod folder names in `downloads/` against the server's canonical names, and optionally fixes mismatches.

**Status codes:**

| Status | Meaning |
|--------|---------|
| `OK` | Disk folder name matches server name exactly |
| `MISMATCH` | Disk name differs from server canonical name (rename needed) |
| `ID_COLLISION` | Local `meta.cpp` has a wrong `publishedid` that belongs to a different mod |
| `NOT_ON_SERVER` | No matching folder found on server |

#### Report only

```bash
python check_names.py
python check_names.py --group shared    # limit to one group
```

```
  Disk name                                     Group                     Status / Server name
  --------------------------------------------  ------------------------  ---------------------------------------------------
  @ace                                          shared                    OK  (@ace)
  @CBA_A3                                       shared                    MISMATCH  ->  @cba_a3
  @NIArms All in One- ACE Compatibility         150th_MW_2026_v1.0        ID_COLLISION  @Realistic Ragdoll Physics (local id: 1234567)
  @150th Languard Zeus Tools                    150th_WW2_2026_V1.0       NOT_ON_SERVER

  80 OK,  1 mismatch,  1 id_collision,  1 not on server

  Run with --fix to rename mismatched folders and --fix-ids to correct wrong steam IDs in meta.cpp.
```

#### Fix mismatched folder names

Renames `MISMATCH` folders on disk to the server's canonical name and updates the `arma_dir` junction to match.

```bash
python check_names.py --fix
```

```
  [+] @CBA_A3  ->  @cba_a3
```

#### Fix wrong steam IDs

Corrects the `publishedid` in local `meta.cpp` files for `ID_COLLISION` entries. Uses `comparison.json` as the authoritative source of steam IDs (which came from the Steam Workshop URLs in your HTML presets).

```bash
python check_names.py --fix-ids
```

```
  [+] @NIArms All in One- ACE Compatibility   meta.cpp: 1234567 -> 9876543
```

**Both fixes at once:**

```bash
python check_names.py --fix --fix-ids
```

**Requires:** `modlist_json/comparison.json` for `--fix-ids` (run `run.py --skip-fetch --skip-link` first).

---

### run.py

Orchestrator that chains all four pipeline steps. Described in [Quick Start](#quick-start--full-pipeline) above.

---

## Migrating Existing Mods

If the Arma 3 Server already has mods installed and you want to bring them under this toolchain without re-downloading:

**Step 1 — Generate comparison.json:**
```bash
python run.py --skip-fetch --skip-link
```

**Step 2 — Check for name mismatches before linking:**
```bash
python check_names.py
```

**Step 3 — Fix any issues:**
```bash
python check_names.py --fix          # rename mismatched folders
python check_names.py --fix-ids      # fix wrong steam IDs in meta.cpp
```

**Step 4 — Create links:**
```bash
python run.py --skip-parse --skip-compare --skip-fetch
```

> Instead of downloading, you can create junctions from `downloads/{group}/@ModName` pointing to wherever the mods already live on disk:
> ```cmd
> mklink /J downloads\shared\@ace "C:\existing\path\@ace"
> ```
> Then run the link step normally.

---

## Folder Structure

```
arma-modlist-tools/
|
|- arma_modlist_tools/        # Python package (library code)
|   |- __init__.py            # Public exports
|   |- parser.py              # HTML preset parser
|   |- compare.py             # Preset comparison
|   |- fetcher.py             # Caddy server downloader
|   |- linker.py              # Junction/symlink manager
|   |- reporter.py            # Missing-mod report builder
|   |- config.py              # config.json loader
|   |- compat.py              # OS detection + encoding fix
|
|- modlist_html/              # INPUT: put your .html preset exports here
|   |- MyPreset_A.html
|   |- MyPreset_B.html
|
|- modlist_json/              # OUTPUT: generated JSON files (gitignored)
|   |- MyPreset_A.json
|   |- MyPreset_B.json
|   |- comparison.json
|   |- missing_report.json
|
|- downloads/                 # OUTPUT: downloaded mod files (gitignored)
|   |- shared/
|   |   |- @ace/
|   |   |- @cba_a3/
|   |- MyPreset_A/
|       |- @rhsusaf/
|
|- config.json                # YOUR config (gitignored — contains credentials)
|- config.template.json       # Template to copy from
|- requirements.txt
|
|- run.py                     # Orchestrator (parse + compare + fetch + link)
|- parse_modlist.py           # Step 1 standalone
|- compare_modlists.py        # Step 2 standalone
|- fetch_mods.py              # Step 3 standalone
|- link_mods.py               # Link management (status/link/unlink)
|- report_missing.py          # Missing mod report
|- sync_missing.py            # Sync newly available missing mods
|- update_mods.py             # Re-download updated mod files
|- check_names.py             # Diagnose and fix folder name / steam_id issues
|- check_deps.py              # Dependency checker
|- test_suite.py              # Test suite
```

---

## Moving to a New Device

1. **Clone the repo:**
   ```bash
   git clone https://git.revoluxiant.io.vn/revernomad17/arma-modlist-tools.git
   cd arma-modlist-tools
   ```

2. **Install dependencies:**
   ```bash
   pip install -r requirements.txt
   ```

3. **Create your config:**
   ```bash
   cp config.template.json config.json
   # Edit config.json with correct arma_dir and server credentials
   ```

4. **Add your preset exports:**
   Copy your `.html` files from the Arma 3 Launcher into `modlist_html/`.

5. **Verify and run:**
   ```bash
   python check_deps.py
   python run.py
   ```

> The `downloads/` folder can be several GB. On a second device you can either let `run.py` re-download everything, or copy the `downloads/` folder manually and run `python run.py --skip-fetch` to skip downloading and just create links.

---

## Running Tests

The test suite covers all modules with 85 tests. No network connection required.

```bash
python test_suite.py
```

```
------------------------------------------------------------
  compat            6 tests
  config            5 tests
  parser            9 tests
  compare           8 tests
  fetcher          19 tests   (pure functions, no network)
  reporter          8 tests
  linker           12 tests   (uses temp dirs)
  __init__          2 tests
  check_names      16 tests
  integration       2 tests
------------------------------------------------------------
  Results: 85 passed, 0 failed, 0 skipped  (85 total)
```