Compare commits

...

8 Commits

Author SHA1 Message Date
Tran G. (Revernomad) Khoa
b24828ac68 docs: update CLAUDE.md, README, and Vietnamese guide for migration step
- CLAUDE.md: document migrator.py algorithm and junction-removal rationale
- README.md: pipeline now 5 steps, add --skip-migrate flag, add migrator.py
  to folder structure, update test count 142 -> 158
- docs/huong-dan-su-dung.md: 5-step pipeline table, new glossary entry,
  updated footer version note
2026-04-14 15:11:39 +07:00
Tran G. (Revernomad) Khoa
48637ffe90 feat: add migrate step to move mod folders between groups on preset change
Before step_fetch, scan all downloads/ subdirs and move any mod that
comparison.json now assigns to a different group. Matching uses steam_id
(via meta.cpp publishedid) first, normalized name as fallback.

Stale junctions in arma_dir are removed before the folder move so
step_link can re-create them pointing to the new location.

- New arma_modlist_tools/migrator.py: migrate_mod_groups()
- run.py: step_migrate(), --skip-migrate flag, wired into dispatch loop
- gui/app.py: step_migrate inserted as Step 3/5 between compare and fetch
- gui/locales.py: add step3/4/5 names (en + vi), renumber old 3->4, 4->5
- test_suite.py: 7 new migrator tests (158 total, 0 failed)
2026-04-14 15:08:40 +07:00
Tran G. (Revernomad) Khoa
45cb023513 fix: align mods view column headers with data rows
- Use padx=(8,4) consistently for the name column in both header and rows,
  removing the leading-space text hack in row labels
- Add a 16px spacer at the right end of the header to compensate for
  CTkScrollableFrame's internal scrollbar width
- Centre-align Downloaded and Linked columns (header + tick/cross labels)
- Unify update button width to 80px to match header col width
2026-04-14 14:42:29 +07:00
Tran G. (Revernomad) Khoa
ecfa5fa636 fix: match mod folder by steam_id when folder name diverges from modlist name
_find_folder in mods.py now has a fourth fallback: reads publishedid from
meta.cpp inside each candidate folder and matches against mod["steam_id"].
Fixes mods appearing as "not downloaded" when the folder name on disk differs
from the name in the modlist but the mod content (meta.cpp) is correct.

Also adds 8 tests covering all four match strategies and edge cases.
2026-04-14 14:37:32 +07:00
Tran G. (Revernomad) Khoa
fd513b3688 docs: document run_tool UTF-8 encoding fix and update_mods orphan removal 2026-04-11 09:28:39 +07:00
Tran G. (Revernomad) Khoa
50990cca4e feat: remove orphan local files that no longer exist on server in update_mods 2026-04-11 09:21:45 +07:00
Tran G. (Revernomad) Khoa
4fde566cf4 Merge branch 'main' of https://git.revoluxiant.io.vn/revernomad17/arma-modlist-tools 2026-04-10 22:38:20 +07:00
Tran G. (Revernomad) Khoa
68fcaaf6d9 fix: decode run_tool subprocess output as UTF-8 to handle tqdm Unicode chars on Windows 2026-04-10 22:37:35 +07:00
11 changed files with 593 additions and 73 deletions

View File

@@ -101,14 +101,15 @@ Pass 2 builds `ok_disk_names` — the set of disk names that already match the s
- `logs.py` — real-time log viewer fed from the stdout/stderr queue
- `settings.py` — in-app editor for `config.json` (server URL, paths, credentials)
**`_find_folder` (mods.py) — three-level name matching:** The mods view resolves a mod's local folder by mod name from `comparison.json`, which may differ from the server-canonical folder name used by the fetcher. Lookup order:
**`_find_folder` (mods.py) — four-level name matching:** The mods view resolves a mod's local folder by mod name from `comparison.json`, which may differ from the server-canonical folder name used by the fetcher. Lookup order:
1. Exact: `@{mod_name}`
2. Case-insensitive: `@CBA_A3` matches `CBA_A3`
3. Normalized (`_normalize_name`): strips all non-alphanumeric — handles punctuation/spacing differences, e.g. `@US GEAr- Units (IFA3)` matches `US GEAr: Units (IFA3)` (both → `usgearunitsifa3`)
4. Steam ID via `meta.cpp`: reads `publishedid` from each folder's `meta.cpp` and matches against `mod["steam_id"]` — handles the case where the folder name bears no resemblance to the modlist name but the mod content is correct
**`selection.json`** — GUI selection state file, tracked in git. Persists which mods/groups are selected between GUI sessions. Written by the GUI; safe to delete (GUI recreates it on next save).
**`run_tool` subprocess streaming:** Tool scripts are launched via `subprocess.Popen` (not `subprocess.run`) with `stdout=PIPE, stderr=STDOUT`, read line-by-line via `iter(proc.stdout.readline, "")`, and posted to the log queue immediately. Python's own output buffering is disabled with the `-u` flag and `PYTHONUNBUFFERED=1` in the environment — without these, output would batch inside the pipe and only appear when the script exits.
**`run_tool` subprocess streaming:** Tool scripts are launched via `subprocess.Popen` (not `subprocess.run`) with `stdout=PIPE, stderr=STDOUT`, read line-by-line via `iter(proc.stdout.readline, "")`, and posted to the log queue immediately. Python's own output buffering is disabled with the `-u` flag and `PYTHONUNBUFFERED=1` in the environment — without these, output would batch inside the pipe and only appear when the script exits. The `Popen` call uses `encoding="utf-8", errors="replace"` and sets `PYTHONUTF8=1` in the child environment so that tqdm's Unicode block characters (e.g. `▉`) don't crash the pipe reader on Windows, where the default `charmap` codec cannot decode them.
**GUI threading model:** Every network or long-running operation runs in a `threading.Thread(daemon=True)` so the Tkinter event loop is never blocked. The only safe way to update widgets from a background thread is `self.after(0, callback)` — never touch widgets directly from a worker thread. `_poll_log` drains the entire log queue in one `after(80, ...)` tick and does a single batched `CTkTextbox.insert()` call rather than one per log entry, keeping the UI smooth even when `tqdm` emits many rapid updates during downloads. The wizard's "Test Connection" button follows the same pattern: `requests.get` runs in a daemon thread; the result is posted back via `self.after(0, ...)` with widget references captured *before* the thread starts, so stale references cannot update the wrong widgets if the user navigates away mid-request.
@@ -116,6 +117,24 @@ Pass 2 builds `ok_disk_names` — the set of disk names that already match the s
**`build_server_index` progress callback:** Accepts an optional `progress_fn(current, total, name)` callback. `step_fetch` in `run.py` uses this to print `Indexing N/M: @FolderName` every 25 folders so the log never goes silent during the server scan phase. The library itself never calls `print` — the caller owns the I/O.
### `migrator.py` — mod group migration
Before `step_fetch` runs, `step_migrate` moves locally-downloaded mod folders to match the group assignments in the new `comparison.json`. This avoids re-downloading mods that already exist on disk under a different group when presets are switched (e.g. `A``A_v1`).
**Algorithm:**
1. `_build_local_index(downloads)` — scans every `downloads/{group}/@Folder`, reads `meta.cpp` to extract `publishedid`, builds `{steam_id → (group, path)}` and `{norm_name → (group, path)}` maps.
2. `_build_target_list(comparison)` — flattens `comparison.json` into `[(new_group, steam_id, mod_name)]`.
3. For each target: locate mod on disk (steam_id first, normalised name fallback); skip if already in correct group or destination exists; remove stale junction from `arma_dir` if present; move folder with `shutil.move`.
**Junction removal is critical:** a stale junction (target moved away) still has the reparse point attribute, so `_is_junction()` returns `True` and `link_group` would skip it as `already_linked` without recreating it at the new path. Removing the junction before the move lets `step_link` recreate it correctly.
**CLI:** `python run.py --skip-migrate` bypasses the step if needed.
### `update_mods.py` — orphan file removal
After downloading updated files, `update_mods.py` compares every file in the local mod folder against the server's file list and **deletes any local files that no longer exist on the server**. This prevents stale `.pbo` or `.bisign` files from accumulating when a mod's content changes upstream. Each removed file is logged as `[-] orphan removed: <rel_path>` and the final summary line includes an orphan count. The orphan check runs even when no files need downloading (e.g. timestamps match but the local folder has extras).
### GUI localization (`gui/locales.py`)
All user-facing strings are centralised in `gui/locales.py`. Two languages are supported: English (`"en"`) and Vietnamese (`"vi"`).

View File

@@ -12,7 +12,7 @@ cp config.template.json config.json # fill in server URL + credentials + arma_
python check_deps.py # verify dependencies
# Day-to-day: full pipeline
python run.py # parse → compare → download → link
python run.py # parse → compare → migrate → download → link
# GUI (recommended)
python gui.py
@@ -24,7 +24,7 @@ python sync_missing.py # retry mods that were absent from server
python check_names.py --fix # fix folder name mismatches
# Testing
python test_suite.py # 142 tests (network tests auto-skip if offline)
python test_suite.py # 158 tests (network tests auto-skip if offline)
```
---
@@ -133,29 +133,34 @@ Place your Arma 3 Launcher preset exports (`.html`) into the `modlist_html/` fol
python run.py
```
This runs all four steps in sequence:
This runs all five steps in sequence:
```
Step 1/4: Parse presets — modlist_html/*.html -> modlist_json/*.json
Step 2/4: Compare presets — produces modlist_json/comparison.json
Step 3/4: Fetch mods — downloads from server -> downloads/
Step 4/4: Link mods creates junctions/symlinks in Arma 3 Server dir
Step 1/5: Parse presets — modlist_html/*.html -> modlist_json/*.json
Step 2/5: Compare presets — produces modlist_json/comparison.json
Step 3/5: Migrate mod groups — moves existing folders to match new group assignments
Step 4/5: Fetch mods — downloads from server -> downloads/
Step 5/5: Link mods — creates junctions/symlinks in Arma 3 Server dir
```
The **migrate step** avoids re-downloading mods that already exist on disk when you switch preset versions (e.g. `A``A_v1`). It matches mods by steam ID (via `meta.cpp`) and moves the folder to the correct group, removing any stale junction first so the link step can re-create it at the new path.
### Skip flags
```bash
python run.py --skip-fetch --skip-link # parse + compare only
python run.py --skip-fetch --skip-link # parse + compare + migrate only
python run.py --skip-parse --skip-compare --skip-fetch # link only
python run.py --skip-parse --skip-compare --skip-fetch --group shared
python run.py --skip-migrate # skip auto-migration
```
| Flag | Skips |
|------|-------|
| `--skip-parse` | Step 1 (HTML parsing) |
| `--skip-compare` | Step 2 (preset comparison) |
| `--skip-fetch` | Step 3 (downloading) |
| `--skip-link` | Step 4 (linking) |
| `--skip-migrate` | Step 3 (mod group migration) |
| `--skip-fetch` | Step 4 (downloading) |
| `--skip-link` | Step 5 (linking) |
| `--group GROUP` | Link step: only link this one group (e.g. `shared`) |
> **Safe to re-run.** Every step is idempotent — existing files are skipped, already-linked mods are skipped.
@@ -526,7 +531,7 @@ The same functionality is available as the **Clean Orphans** tab in the GUI.
### run.py
Orchestrator that chains all four pipeline steps. Described in [Quick Start](#quick-start--full-pipeline) above.
Orchestrator that chains all five pipeline steps. Described in [Quick Start](#quick-start--full-pipeline) above.
---
@@ -602,6 +607,7 @@ arma-modlist-tools/
| |- linker.py # Junction/symlink manager
| |- reporter.py # Missing-mod report builder
| |- cleaner.py # Orphan folder detection
| |- migrator.py # Mod group migration (move folders to match comparison.json)
| |- config.py # config.json loader
| |- compat.py # OS detection + encoding fix
|
@@ -652,7 +658,7 @@ arma-modlist-tools/
|- check_names.py # Diagnose and fix folder name / steam_id issues
|- clean_orphans.py # Find and delete orphaned mod folders
|- check_deps.py # Dependency checker
|- test_suite.py # Test suite (142 tests)
|- test_suite.py # Test suite (158 tests)
```
---
@@ -691,7 +697,7 @@ arma-modlist-tools/
## Running Tests
The test suite covers all modules with 142 tests. Network tests (section 15) auto-skip when the server is unreachable.
The test suite covers all modules with 158 tests. Network tests auto-skip when the server is unreachable.
```bash
python test_suite.py
@@ -713,7 +719,9 @@ python test_suite.py
cleaner 8 tests
e2e — clean_orphans 6 tests (subprocess CLI)
coverage gaps 23 tests (mocked platform branches)
gui.views.mods 8 tests (_find_folder matching)
migrator 7 tests (group migration logic)
live server 9 tests (skipped if server unreachable)
------------------------------------------------------------
Results: 142 passed, 0 failed, 0 skipped (142 total)
Results: 158 passed, 0 failed, 0 skipped (158 total)
```

View File

@@ -12,6 +12,7 @@ from .config import load_config, Config
from .compat import is_windows, is_linux, get_os_label, fix_console_encoding
from .reporter import build_missing_report, save_missing_report
from .cleaner import find_orphan_folders, folder_size
from .migrator import migrate_mod_groups
__all__ = [
# parser
@@ -32,4 +33,6 @@ __all__ = [
"build_missing_report", "save_missing_report",
# cleaner
"find_orphan_folders", "folder_size",
# migrator
"migrate_mod_groups",
]

View File

@@ -0,0 +1,186 @@
"""
arma_modlist_tools.migrator
~~~~~~~~~~~~~~~~~~~~~~~~~~~
Move locally-downloaded mod folders to match the group assignments in
comparison.json, avoiding redundant re-downloads when presets are renamed
or reorganised.
A mod "needs migration" when it exists on disk under
``downloads/{old_group}/@FolderName`` but comparison.json now assigns it
to ``downloads/{new_group}/@FolderName``.
Algorithm
---------
1. Build a *local index*: scan every ``downloads/{group}/@ModDir`` and read
its ``meta.cpp`` to find its steam_id. Also record its normalised folder
name. Result: ``{steam_id: (group, path)}`` and
``{norm_name: (group, path)}``.
2. Build a *target list* from comparison.json: for every mod in every group,
record which group comparison now assigns it to.
3. For each target entry:
- Find the mod in the local index (steam_id first, normalised name fallback).
- If not found on disk: skip (step_fetch will download it).
- If already in the correct group: skip.
- If the destination already exists: skip (step_fetch handles file-level sync).
- Otherwise: remove any stale junction first, then move old → new.
"""
from __future__ import annotations
import shutil
from pathlib import Path
from .fetcher import _normalize_name, _parse_meta_cpp
from .linker import _is_junction, remove_junction
def _build_local_index(downloads: Path) -> dict:
"""Scan ``downloads/`` and return a two-key index of existing mod folders.
:returns: ``{"by_steam_id": {sid: (group, path)},
"by_norm_name": {norm: (group, path)}}``
First match wins for both keys (sorted directory order is deterministic).
"""
by_steam_id: dict[str, tuple[str, Path]] = {}
by_norm_name: dict[str, tuple[str, Path]] = {}
if not downloads.is_dir():
return {"by_steam_id": by_steam_id, "by_norm_name": by_norm_name}
for group_dir in sorted(downloads.iterdir()):
if not group_dir.is_dir():
continue
group_name = group_dir.name
for mod_dir in sorted(group_dir.iterdir()):
if not mod_dir.is_dir() or not mod_dir.name.startswith("@"):
continue
norm = _normalize_name(mod_dir.name)
if norm not in by_norm_name:
by_norm_name[norm] = (group_name, mod_dir)
meta = mod_dir / "meta.cpp"
if meta.exists():
try:
sid = _parse_meta_cpp(
meta.read_text(encoding="utf-8", errors="replace")
)
if sid and sid not in by_steam_id:
by_steam_id[sid] = (group_name, mod_dir)
except OSError:
pass
return {"by_steam_id": by_steam_id, "by_norm_name": by_norm_name}
def _build_target_list(comparison: dict) -> list[tuple[str, str | None, str]]:
"""Flatten comparison.json into ``[(new_group, steam_id_or_None, mod_name)]``.
Shared mods → group ``"shared"``; unique mods → group = preset name.
"""
entries: list[tuple[str, str | None, str]] = []
for mod in comparison.get("shared", {}).get("mods", []):
entries.append(("shared", mod.get("steam_id") or None, mod.get("name", "")))
for preset, pdata in comparison.get("unique", {}).items():
for mod in pdata.get("mods", []):
entries.append((preset, mod.get("steam_id") or None, mod.get("name", "")))
return entries
def migrate_mod_groups(
downloads: Path,
arma_dir: Path | None,
comparison: dict,
) -> dict:
"""Move locally-downloaded mod folders to match *comparison* group assignments.
:param downloads: Path to the ``downloads/`` directory.
:param arma_dir: Path to the Arma 3 server directory used for junction
cleanup. Pass ``None`` to skip junction removal.
:param comparison: Parsed ``comparison.json`` dict.
:returns: Result dict::
{
"moved": int,
"junction_removed": int,
"skipped_correct": int,
"skipped_dest_exists": int,
"skipped_not_found": int,
"errors": {mod_name: error_message},
}
"""
result: dict = {
"moved": 0,
"junction_removed": 0,
"skipped_correct": 0,
"skipped_dest_exists": 0,
"skipped_not_found": 0,
"errors": {},
}
local = _build_local_index(downloads)
for new_group, steam_id, mod_name in _build_target_list(comparison):
# 1. Locate on disk — steam_id first, normalised name fallback
entry: tuple[str, Path] | None = None
if steam_id:
entry = local["by_steam_id"].get(steam_id)
if entry is None:
entry = local["by_norm_name"].get(_normalize_name(mod_name))
if entry is None:
result["skipped_not_found"] += 1
continue
old_group, old_path = entry
# 2. Already in the correct group?
if old_group == new_group:
result["skipped_correct"] += 1
continue
# 3. Destination already present — let step_fetch handle file-level sync
new_path = downloads / new_group / old_path.name
if new_path.exists():
result["skipped_dest_exists"] += 1
continue
# 4. Remove stale junction so step_link can re-create it at the new path
if arma_dir is not None and arma_dir.is_dir():
link = arma_dir / old_path.name
if _is_junction(link):
ok, err = remove_junction(link)
if ok:
result["junction_removed"] += 1
else:
print(f" MIGRATE WARNING: cannot remove junction "
f"{link.name}: {err}")
# 5. Move folder (shutil.move handles cross-device gracefully)
try:
new_path.parent.mkdir(parents=True, exist_ok=True)
shutil.move(str(old_path), str(new_path))
print(f" MIGRATE moved: {old_path.name} {old_group} -> {new_group}")
result["moved"] += 1
# Update index so later targets don't re-match the now-moved path
if steam_id:
local["by_steam_id"][steam_id] = (new_group, new_path)
local["by_norm_name"][_normalize_name(old_path.name)] = (
new_group, new_path
)
except OSError as exc:
print(f" MIGRATE ERROR: {old_path.name}: {exc}")
result["errors"][mod_name] = str(exc)
print(
f" Moved: {result['moved']} "
f"Junctions removed: {result['junction_removed']} "
f"Already correct: {result['skipped_correct']} "
f"Dest exists: {result['skipped_dest_exists']} "
f"Not on disk: {result['skipped_not_found']}"
)
return result

View File

@@ -37,7 +37,8 @@ Tài liệu này dành cho người dùng **chưa biết gì** về dự án. B
Ứng dụng sẽ tự động:
- Đọc danh sách mod từ các preset
- So sánh, tìm mod dùng chung và mod riêng giữa các preset
- Tải mod từ máy chủ về máy tính của bạn
- **Di chuyển** các thư mục mod đã tải về sang đúng nhóm mới (tránh tải lại khi đổi phiên bản preset)
- Tải mod từ máy chủ về máy tính của bạn (chỉ những mod thực sự chưa có)
- Tạo liên kết (junction/symlink) để Arma 3 Server nhận ra các mod
---
@@ -153,13 +154,14 @@ Giao diện gồm thanh điều hướng bên trái và khu vực nội dung bê
### 5.3 Trạng thái Pipeline
Cột bên phải hiển thị 4 bước của quy trình:
Cột bên phải hiển thị 5 bước của quy trình:
| Bước | Mô tả | Dấu hiệu hoàn thành |
|------|-------|-------------------|
| Phân tích preset | Đọc danh sách mod từ tệp HTML | Có ≥ 2 preset được chọn |
| So sánh preset | Tìm mod chung và riêng | Tệp `comparison.json` tồn tại |
| Tải mod | Tải tệp mod từ máy chủ | Có thư mục mod trong `downloads/` |
| Di chuyển nhóm mod | Chuyển thư mục mod sẵn có sang đúng nhóm mới | Tự động (không cần tải lại) |
| Tải mod | Tải tệp mod thực sự còn thiếu từ máy chủ | Có thư mục mod trong `downloads/` |
| Liên kết với Arma | Tạo junction tới thư mục Arma 3 | Thư mục Arma 3 Server tồn tại |
Biểu tượng `✓` (xanh) = đã xong, `○` (xám) = chưa xong.
@@ -391,7 +393,7 @@ Hiển thị các đường dẫn đang dùng: URL máy chủ, thư mục Arma,
| Thuật ngữ | Giải thích |
|-----------|------------|
| **Preset** | Tệp HTML xuất từ Arma 3 Launcher chứa danh sách mod |
| **Pipeline** | Chuỗi 4 bước tự động: phân tích → so sánh → tải → liên kết |
| **Pipeline** | Chuỗi 5 bước tự động: phân tích → so sánh → di chuyển nhóm → tải → liên kết |
| **Junction / Symlink** | Liên kết thư mục ảo — Arma 3 thấy mod trong thư mục của mình nhưng tệp thực sự nằm ở `downloads/` |
| **Shared mods** | Mod xuất hiện trong tất cả preset đã chọn |
| **Unique mods** | Mod chỉ có trong một preset cụ thể |
@@ -401,9 +403,10 @@ Hiển thị các đường dẫn đang dùng: URL máy chủ, thư mục Arma,
| **comparison.json** | Tệp kết quả so sánh preset, lưu danh sách mod theo nhóm |
| **missing_report.json** | Báo cáo mod có trong preset nhưng chưa có trên máy chủ |
| **downloads/** | Thư mục chứa tệp mod đã tải về |
| **Di chuyển nhóm mod** | Bước tự động chuyển thư mục mod từ nhóm cũ sang nhóm mới theo `comparison.json` — tránh tải lại khi đổi phiên bản preset |
| **Mod thừa (Orphan)** | Thư mục mod còn trong `downloads/` nhưng không còn trong preset nào đang dùng |
| **config.json** | Tệp cấu hình lưu thông tin máy chủ và đường dẫn |
---
*Phiên bản tài liệu: 2026-04 (cập nhật: thêm Clean Orphans). Nếu có vấn đề, liên hệ người quản trị máy chủ.*
*Phiên bản tài liệu: 2026-04 (cập nhật: thêm bước Di chuyển nhóm mod, pipeline 5 bước). Nếu có vấn đề, liên hệ người quản trị máy chủ.*

View File

@@ -151,7 +151,7 @@ class ArmaModManagerApp(ctk.CTk):
# run.py calls fix_console_encoding() at import time, which needs
# the real sys.stdout.buffer. Import it before we redirect stdout.
try:
from run import step_fetch, step_link
from run import step_migrate, step_fetch, step_link
except Exception as _import_err:
self.after(0, lambda: self.post_log(
f"\n✗ Failed to load pipeline: {_import_err}\n"
@@ -165,7 +165,7 @@ class ArmaModManagerApp(ctk.CTk):
from arma_modlist_tools.compare import compare_presets
# Step 1 — Parse selected presets
_hdr("Step 1 / 4", t("pipeline.step1_name"))
_hdr("Step 1 / 5", t("pipeline.step1_name"))
cfg.modlist_json.mkdir(exist_ok=True)
presets = []
for fp in sorted(cfg.modlist_html.glob("*.html")):
@@ -182,7 +182,7 @@ class ArmaModManagerApp(ctk.CTk):
presets.append(preset)
# Step 2 — Compare
_hdr("Step 2 / 4", t("pipeline.step2_name"))
_hdr("Step 2 / 5", t("pipeline.step2_name"))
result = compare_presets(*presets)
cfg.comparison.write_text(
json.dumps(result, indent=2, ensure_ascii=False),
@@ -193,12 +193,16 @@ class ArmaModManagerApp(ctk.CTk):
print(f" Shared: {result['shared']['mod_count']} | "
f"Unique: {total_unique}")
# Step 3 — Fetch
_hdr("Step 3 / 4", t("pipeline.step3_name"))
# Step 3 — Migrate
_hdr("Step 3 / 5", t("pipeline.step3_name"))
step_migrate(cfg)
# Step 4 — Fetch
_hdr("Step 4 / 5", t("pipeline.step4_name"))
step_fetch(cfg)
# Step 4 — Link
_hdr("Step 4 / 4", t("pipeline.step4_name"))
# Step 5 — Link
_hdr("Step 5 / 5", t("pipeline.step5_name"))
groups = (
sorted(p.name for p in cfg.downloads.iterdir() if p.is_dir())
if cfg.downloads.is_dir() else []
@@ -227,11 +231,14 @@ class ArmaModManagerApp(ctk.CTk):
try:
env = os.environ.copy()
env["PYTHONUNBUFFERED"] = "1"
env["PYTHONUTF8"] = "1"
proc = subprocess.Popen(
[sys.executable, "-u", str(PROJECT_ROOT / script)] + extra,
stdout=subprocess.PIPE,
stderr=subprocess.STDOUT,
text=True,
encoding="utf-8",
errors="replace",
cwd=str(PROJECT_ROOT),
env=env,
)

View File

@@ -29,8 +29,9 @@ _EN: dict[str, str] = {
"pipeline.starting": "Pipeline started",
"pipeline.step1_name": "Parse presets",
"pipeline.step2_name": "Compare presets",
"pipeline.step3_name": "Download mods",
"pipeline.step4_name": "Link mods",
"pipeline.step3_name": "Migrate mod groups",
"pipeline.step4_name": "Download mods",
"pipeline.step5_name": "Link mods",
# ── app.py dialogs ────────────────────────────────────────────────────────
"app.dlg_presets_title": "Not enough presets selected",
@@ -270,8 +271,9 @@ _VI: dict[str, str] = {
"pipeline.starting": "Pipeline đã bắt đầu",
"pipeline.step1_name": "Phân tích preset",
"pipeline.step2_name": "So sánh preset",
"pipeline.step3_name": "Tải mod",
"pipeline.step4_name": "Liên kết mod",
"pipeline.step3_name": "Di chuyển nhóm mod",
"pipeline.step4_name": "Tải mod",
"pipeline.step5_name": "Liên kết mod",
# ── app.py dialogs ────────────────────────────────────────────────────────
"app.dlg_presets_title": "Chưa chọn đủ preset",

View File

@@ -7,7 +7,7 @@ from typing import TYPE_CHECKING, Optional
import customtkinter as ctk
from arma_modlist_tools.fetcher import _normalize_name
from arma_modlist_tools.fetcher import _normalize_name, _parse_meta_cpp
from gui._constants import COLOR_OK, COLOR_ERROR, COLOR_WARN, COLOR_RUNNING
from gui.locales import t
from gui.views.base import BaseView
@@ -16,13 +16,14 @@ if TYPE_CHECKING:
from gui.app import ArmaModManagerApp
def _find_folder(group_dir: Path, mod_name: str) -> Optional[Path]:
def _find_folder(group_dir: Path, mod_name: str, steam_id: str | None = None) -> Optional[Path]:
"""Return the local mod folder path, or None if not downloaded.
Matches in priority order:
1. Exact folder name ``@{mod_name}``
2. Case-insensitive name (handles ``@CBA_A3`` vs ``CBA_A3``)
3. Normalized name — strips non-alphanumeric (handles ``@cba_a3`` vs ``CBA A3``)
4. Steam ID match via ``meta.cpp`` ``publishedid`` field (folder name differs from modlist name)
"""
if not group_dir.is_dir():
return None
@@ -38,6 +39,18 @@ def _find_folder(group_dir: Path, mod_name: str) -> Optional[Path]:
return p
if _normalize_name(p.name) == target_norm:
return p
# Fallback: match by steam_id via meta.cpp when folder name diverges from modlist name
if steam_id:
for p in group_dir.iterdir():
if not p.is_dir():
continue
meta = p / "meta.cpp"
try:
sid = _parse_meta_cpp(meta.read_text(encoding="utf-8", errors="replace"))
if sid == steam_id:
return p
except OSError:
pass
return None
@@ -171,19 +184,22 @@ class ModsView(BaseView):
fg_color=("gray82", "gray22"), corner_radius=6)
col_hdr.grid(row=0, column=0, sticky="ew", padx=4, pady=(6, 2))
col_hdr.columnconfigure(0, weight=1)
for col, (w, lbl_key) in enumerate([
(0, "mods.col_name"),
(80, "mods.col_downloaded"),
(80, "mods.col_linked"),
(160, "mods.col_server"),
(80, ""),
for col, (w, lbl_key, anc) in enumerate([
(0, "mods.col_name", "w"),
(80, "mods.col_downloaded", "center"),
(80, "mods.col_linked", "center"),
(160, "mods.col_server", "w"),
(80, "", "center"),
]):
ctk.CTkLabel(col_hdr, text=t(lbl_key) if lbl_key else "",
font=ctk.CTkFont(weight="bold"),
anchor="w", width=w or 1).grid(
anchor=anc, width=w or 1).grid(
row=0, column=col,
padx=(10 if col == 0 else 4, 4), pady=5,
padx=(8 if col == 0 else 4, 4), pady=5,
sticky="ew" if col == 0 else "")
# Spacer compensates for CTkScrollableFrame's internal scrollbar width
# so the header columns line up with the data rows below.
ctk.CTkLabel(col_hdr, text="", width=16).grid(row=0, column=5, padx=0)
# Scrollable rows
scroll = ctk.CTkScrollableFrame(tab_frame)
@@ -215,7 +231,7 @@ class ModsView(BaseView):
link_map: dict[str, bool],
) -> None:
for i, mod in enumerate(sorted(mods, key=lambda m: m["name"].lower())):
folder_path = _find_folder(cfg.downloads / group, mod["name"])
folder_path = _find_folder(cfg.downloads / group, mod["name"], mod.get("steam_id"))
downloaded = folder_path is not None
linked = (link_map.get(folder_path.name.lower(), False)
if folder_path else False)
@@ -226,23 +242,23 @@ class ModsView(BaseView):
row.columnconfigure(0, weight=1)
# Mod name
name_lbl = ctk.CTkLabel(row, text=f" {mod['name']}", anchor="w")
name_lbl.grid(row=0, column=0, sticky="ew", padx=4, pady=3)
name_lbl = ctk.CTkLabel(row, text=mod["name"], anchor="w")
name_lbl.grid(row=0, column=0, sticky="ew", padx=(8, 4), pady=3)
# Downloaded
ctk.CTkLabel(
row,
text=" " if downloaded else " ",
text="" if downloaded else "",
text_color=COLOR_OK if downloaded else COLOR_ERROR,
width=80, anchor="w",
width=80, anchor="center",
).grid(row=0, column=1, padx=4)
# Linked
ctk.CTkLabel(
row,
text=" " if linked else (" " if not downloaded else " "),
text="" if linked else ("" if not downloaded else ""),
text_color=COLOR_OK if linked else "gray",
width=80, anchor="w",
width=80, anchor="center",
).grid(row=0, column=2, padx=4)
# Server status
@@ -254,7 +270,7 @@ class ModsView(BaseView):
# Update button (hidden until stale detected)
folder_name = folder_path.name if folder_path else None
update_btn = ctk.CTkButton(
row, text=t("mods.update_btn"), width=70,
row, text=t("mods.update_btn"), width=80,
command=(lambda g=group, fn=folder_name:
self._update_mod(g, fn)) if folder_name else None,
state="normal" if folder_name else "disabled",

24
run.py
View File

@@ -78,6 +78,22 @@ def step_compare(cfg) -> None:
print(f" -> {cfg.comparison}")
def step_migrate(cfg) -> None:
if not cfg.comparison.exists():
print(f" NOTE: {cfg.comparison} not found — skipping migration.")
return
if not cfg.downloads.is_dir():
print(f" NOTE: downloads dir missing ({cfg.downloads}) — nothing to migrate.")
return
from arma_modlist_tools.migrator import migrate_mod_groups
comparison = json.loads(cfg.comparison.read_text(encoding="utf-8"))
arma_dir = cfg.arma_dir if cfg.arma_dir.is_dir() else None
result = migrate_mod_groups(cfg.downloads, arma_dir, comparison)
if result["errors"]:
for name, err in result["errors"].items():
print(f" ERROR {name}: {err}")
def step_fetch(cfg) -> None:
if not cfg.comparison.exists():
print(f" ERROR: {cfg.comparison} not found. Run parse + compare first.")
@@ -178,6 +194,7 @@ def main() -> None:
parser = argparse.ArgumentParser(description="Run the full mod management pipeline.")
parser.add_argument("--skip-parse", action="store_true")
parser.add_argument("--skip-compare", action="store_true")
parser.add_argument("--skip-migrate", action="store_true")
parser.add_argument("--skip-fetch", action="store_true")
parser.add_argument("--skip-link", action="store_true")
parser.add_argument("--group", "-g", metavar="GROUP",
@@ -187,13 +204,15 @@ def main() -> None:
steps = [
(not args.skip_parse, "Parse presets"),
(not args.skip_compare, "Compare presets"),
(not args.skip_migrate, "Migrate mods"),
(not args.skip_fetch, "Fetch mods"),
(not args.skip_link, "Link mods"),
]
active_count = sum(1 for run, _ in steps if run)
step_num = 0
if args.skip_parse and args.skip_compare and args.skip_fetch and args.skip_link:
if (args.skip_parse and args.skip_compare and args.skip_migrate
and args.skip_fetch and args.skip_link):
print("All steps skipped — nothing to do.")
sys.exit(0)
@@ -201,6 +220,7 @@ def main() -> None:
(cfg,),
(cfg,),
(cfg,),
(cfg,),
None, # handled separately
]):
if not run:
@@ -221,6 +241,8 @@ def main() -> None:
step_parse(cfg)
elif name == "Compare presets":
step_compare(cfg)
elif name == "Migrate mods":
step_migrate(cfg)
elif name == "Fetch mods":
step_fetch(cfg)

View File

@@ -2229,6 +2229,240 @@ test("live: list_mod_files entries are (rel_path, url, size) tuples", _test_liv
test("live: find_mod_folder name fallback works (no steam_id)", _test_live_find_mod_by_name_fallback)
# ---------------------------------------------------------------------------
# gui.views.mods — _find_folder
# ---------------------------------------------------------------------------
group("gui.views.mods — _find_folder")
import importlib.util as _mods_ilu
_mods_spec = _mods_ilu.spec_from_file_location(
"gui.views.mods", Path(__file__).parent / "gui" / "views" / "mods.py"
)
_mods_mod = _mods_ilu.module_from_spec(_mods_spec)
# Stub out customtkinter and gui imports so the module loads without a display
import types as _types
import sys as _sys
for _stub in ("customtkinter", "gui", "gui._constants", "gui.locales",
"gui.views", "gui.views.base"):
if _stub not in _sys.modules:
_sys.modules[_stub] = _types.ModuleType(_stub)
# Provide the colour constants the module references at import time
_sys.modules["gui._constants"].COLOR_OK = "#4CAF50"
_sys.modules["gui._constants"].COLOR_ERROR = "#F44336"
_sys.modules["gui._constants"].COLOR_WARN = "#FF9800"
_sys.modules["gui._constants"].COLOR_RUNNING = "#2196F3"
# Stub BaseView so the class body does not fail
_base_stub = _sys.modules["gui.views.base"] = _types.ModuleType("gui.views.base")
_base_stub.BaseView = object
# Stub locales.t so string calls don't crash
_sys.modules["gui.locales"].t = lambda key, **kw: key
_mods_spec.loader.exec_module(_mods_mod)
_find_folder_fn = _mods_mod._find_folder
def _test_ff_returns_none_for_missing_group(tmp_path):
assert _find_folder_fn(tmp_path / "nonexistent", "MyMod") is None
def _test_ff_exact_match(tmp_path):
(tmp_path / "@MyMod").mkdir()
result = _find_folder_fn(tmp_path, "MyMod")
assert result == tmp_path / "@MyMod"
def _test_ff_case_insensitive_match(tmp_path):
(tmp_path / "@mymod").mkdir()
result = _find_folder_fn(tmp_path, "MyMod")
assert result == tmp_path / "@mymod"
def _test_ff_normalized_match(tmp_path):
# Folder on disk uses underscores; modlist name uses spaces — both normalize to same string
(tmp_path / "@My_Mod_Edition").mkdir()
result = _find_folder_fn(tmp_path, "My Mod Edition")
assert result == tmp_path / "@My_Mod_Edition"
def _test_ff_steam_id_fallback(tmp_path):
"""Folder name bears no resemblance to mod name but meta.cpp has correct ID."""
folder = tmp_path / "@ServerCanonicalName"
folder.mkdir()
(folder / "meta.cpp").write_text('publishedid = 123456789;\nname = "Some Mod";\n')
result = _find_folder_fn(tmp_path, "Completely Different Name", steam_id="123456789")
assert result == folder
def _test_ff_steam_id_no_false_positive(tmp_path):
"""Wrong steam_id in meta.cpp must not match."""
folder = tmp_path / "@WrongMod"
folder.mkdir()
(folder / "meta.cpp").write_text('publishedid = 999999999;\n')
result = _find_folder_fn(tmp_path, "My Mod", steam_id="123456789")
assert result is None
def _test_ff_steam_id_skipped_when_none(tmp_path):
"""No steam_id supplied → meta.cpp is never read (no false positives)."""
folder = tmp_path / "@SomeFolder"
folder.mkdir()
(folder / "meta.cpp").write_text('publishedid = 123456789;\n')
result = _find_folder_fn(tmp_path, "My Mod", steam_id=None)
assert result is None
def _test_ff_missing_meta_cpp_skipped(tmp_path):
"""Folders without meta.cpp are silently skipped in the steam_id pass."""
folder = tmp_path / "@NoMeta"
folder.mkdir()
result = _find_folder_fn(tmp_path, "My Mod", steam_id="123456789")
assert result is None
# Wrap tmp_path calls in lambdas that supply a temp dir
def _with_tmp(fn):
def wrapper():
with tempfile.TemporaryDirectory() as d:
fn(Path(d))
return wrapper
test("_find_folder: None when group dir missing",
_with_tmp(_test_ff_returns_none_for_missing_group))
test("_find_folder: exact @ModName match",
_with_tmp(_test_ff_exact_match))
test("_find_folder: case-insensitive name match",
_with_tmp(_test_ff_case_insensitive_match))
test("_find_folder: normalized name match (punctuation differs)",
_with_tmp(_test_ff_normalized_match))
test("_find_folder: steam_id fallback via meta.cpp",
_with_tmp(_test_ff_steam_id_fallback))
test("_find_folder: wrong steam_id in meta.cpp is not a match",
_with_tmp(_test_ff_steam_id_no_false_positive))
test("_find_folder: no steam_id supplied → meta.cpp not checked",
_with_tmp(_test_ff_steam_id_skipped_when_none))
test("_find_folder: missing meta.cpp silently skipped",
_with_tmp(_test_ff_missing_meta_cpp_skipped))
# ---------------------------------------------------------------------------
# migrator — migrate_mod_groups
# ---------------------------------------------------------------------------
group("migrator — migrate_mod_groups")
from arma_modlist_tools.migrator import migrate_mod_groups as _migrate
def _make_mod(dl: Path, grp: str, folder: str, steam_id: str | None = None) -> Path:
"""Create a minimal mod folder under downloads/group/folder."""
d = dl / grp / folder
d.mkdir(parents=True)
if steam_id:
(d / "meta.cpp").write_text(f"publishedid = {steam_id};\n", encoding="utf-8")
(d / "dummy.pbo").write_bytes(b"\x00" * 8)
return d
def _test_migrate_already_correct():
comp = {"shared": {"mods": [{"name": "CBA_A3", "steam_id": "450814997"}]}, "unique": {}}
with tempfile.TemporaryDirectory() as d:
dl = Path(d) / "downloads"
_make_mod(dl, "shared", "@CBA_A3", "450814997")
result = _migrate(dl, None, comp)
assert_eq(result["moved"], 0)
assert_eq(result["skipped_correct"], 1)
def _test_migrate_moves_by_steam_id():
comp = {"shared": {"mods": []}, "unique": {
"A_v1": {"mods": [{"name": "ACE3", "steam_id": "463939057"}]}
}}
with tempfile.TemporaryDirectory() as d:
dl = Path(d) / "downloads"
old = _make_mod(dl, "A", "@ACE3", "463939057")
result = _migrate(dl, None, comp)
assert_eq(result["moved"], 1)
assert not old.exists(), "old folder must be gone"
assert (dl / "A_v1" / "@ACE3").exists(), "new folder must exist"
assert (dl / "A_v1" / "@ACE3" / "dummy.pbo").exists(), "files preserved"
def _test_migrate_moves_by_normalized_name():
"""No meta.cpp — matching falls back to normalised folder name."""
comp = {"shared": {"mods": [{"name": "CBA_A3", "steam_id": None}]}, "unique": {
"A": {"mods": []}
}}
with tempfile.TemporaryDirectory() as d:
dl = Path(d) / "downloads"
_make_mod(dl, "A", "@CBA_A3", steam_id=None)
result = _migrate(dl, None, comp)
assert_eq(result["moved"], 1)
assert (dl / "shared" / "@CBA_A3").exists()
def _test_migrate_skips_dest_exists():
comp = {"shared": {"mods": [{"name": "CBA_A3", "steam_id": "450814997"}]}, "unique": {}}
with tempfile.TemporaryDirectory() as d:
dl = Path(d) / "downloads"
_make_mod(dl, "A", "@CBA_A3", "450814997")
_make_mod(dl, "shared", "@CBA_A3", "450814997")
result = _migrate(dl, None, comp)
assert_eq(result["moved"], 0)
assert_eq(result["skipped_dest_exists"], 1)
def _test_migrate_skips_not_on_disk():
comp = {"shared": {"mods": [{"name": "CBA_A3", "steam_id": "450814997"}]}, "unique": {}}
with tempfile.TemporaryDirectory() as d:
dl = Path(d) / "downloads"
dl.mkdir()
result = _migrate(dl, None, comp)
assert_eq(result["skipped_not_found"], 1)
assert_eq(result["moved"], 0)
def _test_migrate_removes_stale_junction():
from arma_modlist_tools.linker import create_junction, _is_junction
comp = {"shared": {"mods": [{"name": "ACE3", "steam_id": "463939057"}]}, "unique": {}}
with tempfile.TemporaryDirectory() as dl_d, \
tempfile.TemporaryDirectory() as arma_d:
dl = Path(dl_d) / "downloads"
arma = Path(arma_d)
old = _make_mod(dl, "A", "@ACE3", "463939057")
link = arma / "@ACE3"
create_junction(link, old)
assert _is_junction(link), "precondition: junction must exist"
result = _migrate(dl, arma, comp)
assert_eq(result["moved"], 1)
assert_eq(result["junction_removed"], 1)
assert not _is_junction(link), "stale junction must be removed"
assert (dl / "shared" / "@ACE3").exists()
def _test_migrate_missing_downloads_dir():
comp = {"shared": {"mods": [{"name": "X", "steam_id": "1"}]}, "unique": {}}
with tempfile.TemporaryDirectory() as d:
result = _migrate(Path(d) / "nonexistent", None, comp)
assert_eq(result["moved"], 0)
assert_eq(result["skipped_not_found"], 1)
test("migrator: already in correct group → no move", _test_migrate_already_correct)
test("migrator: moves mod by steam_id to new group", _test_migrate_moves_by_steam_id)
test("migrator: moves mod by normalized name (no meta.cpp)", _test_migrate_moves_by_normalized_name)
test("migrator: skips when destination already exists", _test_migrate_skips_dest_exists)
test("migrator: skips mod not on disk", _test_migrate_skips_not_on_disk)
test("migrator: removes stale junction before moving", _test_migrate_removes_stale_junction)
test("migrator: no-op when downloads dir missing", _test_migrate_missing_downloads_dir)
# ---------------------------------------------------------------------------
# Summary
# ---------------------------------------------------------------------------

View File

@@ -102,6 +102,7 @@ def main() -> None:
COL_GROUP = 24
total_checked = total_updated = total_bytes = 0
total_removed = 0
not_on_server = []
for group, folder_name, mod_dir in targets:
@@ -123,13 +124,21 @@ def main() -> None:
all_files = list_mod_files(folder_url, session) if not args.force else stale
checked = len(all_files) if not args.force else len(stale)
if not stale:
# Find local files that no longer exist on the server (orphans)
server_rel = {rel for rel, _, _ in all_files}
orphans = [
f for f in mod_dir.rglob("*") if f.is_file()
and str(f.relative_to(mod_dir)).replace("\\", "/") not in server_rel
]
if not stale and not orphans:
print(f" [=] {folder_name:<{COL_MOD}} {group:<{COL_GROUP}} {checked} files up-to-date")
total_checked += checked
continue
# Download stale files
mod_bytes = 0
if stale:
with tqdm(
total=len(stale), unit="file",
desc=f" {folder_name[-COL_MOD:]:<{COL_MOD}}",
@@ -148,21 +157,32 @@ def main() -> None:
mod_bytes += n
file_bar.update(1)
# Remove orphan files
for orphan in orphans:
tqdm.write(f" [-] orphan removed: {orphan.relative_to(mod_dir)}")
orphan.unlink()
total_checked += checked
total_updated += len(stale)
total_bytes += mod_bytes
print(f" [+] {folder_name:<{COL_MOD}} {group:<{COL_GROUP}} "
f"{checked} files {len(stale)} updated ({_fmt_bytes(mod_bytes)})")
total_removed += len(orphans)
parts = [f"{checked} files"]
if stale:
parts.append(f"{len(stale)} updated ({_fmt_bytes(mod_bytes)})")
if orphans:
parts.append(f"{len(orphans)} orphan(s) removed")
print(f" [+] {folder_name:<{COL_MOD}} {group:<{COL_GROUP}} {' '.join(parts)}")
print(f"\n{'='*56}")
print(f" Total: {total_checked} files checked, "
f"{total_updated} updated, "
f"{total_removed} orphan(s) removed, "
f"{_fmt_bytes(total_bytes)} downloaded")
if not_on_server:
print(f" Not found on server ({len(not_on_server)}): {', '.join(not_on_server)}")
print(f"{'='*56}\n")
if total_updated == 0 and not not_on_server:
if total_updated == 0 and total_removed == 0 and not not_on_server:
print(" All mods are up-to-date.\n")