Compare commits

..

2 Commits

Author SHA1 Message Date
Tran G. (Revernomad) Khoa
ecfa5fa636 fix: match mod folder by steam_id when folder name diverges from modlist name
_find_folder in mods.py now has a fourth fallback: reads publishedid from
meta.cpp inside each candidate folder and matches against mod["steam_id"].
Fixes mods appearing as "not downloaded" when the folder name on disk differs
from the name in the modlist but the mod content (meta.cpp) is correct.

Also adds 8 tests covering all four match strategies and edge cases.
2026-04-14 14:37:32 +07:00
Tran G. (Revernomad) Khoa
fd513b3688 docs: document run_tool UTF-8 encoding fix and update_mods orphan removal 2026-04-11 09:28:39 +07:00
3 changed files with 145 additions and 5 deletions

View File

@@ -101,14 +101,15 @@ Pass 2 builds `ok_disk_names` — the set of disk names that already match the s
- `logs.py` — real-time log viewer fed from the stdout/stderr queue - `logs.py` — real-time log viewer fed from the stdout/stderr queue
- `settings.py` — in-app editor for `config.json` (server URL, paths, credentials) - `settings.py` — in-app editor for `config.json` (server URL, paths, credentials)
**`_find_folder` (mods.py) — three-level name matching:** The mods view resolves a mod's local folder by mod name from `comparison.json`, which may differ from the server-canonical folder name used by the fetcher. Lookup order: **`_find_folder` (mods.py) — four-level name matching:** The mods view resolves a mod's local folder by mod name from `comparison.json`, which may differ from the server-canonical folder name used by the fetcher. Lookup order:
1. Exact: `@{mod_name}` 1. Exact: `@{mod_name}`
2. Case-insensitive: `@CBA_A3` matches `CBA_A3` 2. Case-insensitive: `@CBA_A3` matches `CBA_A3`
3. Normalized (`_normalize_name`): strips all non-alphanumeric — handles punctuation/spacing differences, e.g. `@US GEAr- Units (IFA3)` matches `US GEAr: Units (IFA3)` (both → `usgearunitsifa3`) 3. Normalized (`_normalize_name`): strips all non-alphanumeric — handles punctuation/spacing differences, e.g. `@US GEAr- Units (IFA3)` matches `US GEAr: Units (IFA3)` (both → `usgearunitsifa3`)
4. Steam ID via `meta.cpp`: reads `publishedid` from each folder's `meta.cpp` and matches against `mod["steam_id"]` — handles the case where the folder name bears no resemblance to the modlist name but the mod content is correct
**`selection.json`** — GUI selection state file, tracked in git. Persists which mods/groups are selected between GUI sessions. Written by the GUI; safe to delete (GUI recreates it on next save). **`selection.json`** — GUI selection state file, tracked in git. Persists which mods/groups are selected between GUI sessions. Written by the GUI; safe to delete (GUI recreates it on next save).
**`run_tool` subprocess streaming:** Tool scripts are launched via `subprocess.Popen` (not `subprocess.run`) with `stdout=PIPE, stderr=STDOUT`, read line-by-line via `iter(proc.stdout.readline, "")`, and posted to the log queue immediately. Python's own output buffering is disabled with the `-u` flag and `PYTHONUNBUFFERED=1` in the environment — without these, output would batch inside the pipe and only appear when the script exits. **`run_tool` subprocess streaming:** Tool scripts are launched via `subprocess.Popen` (not `subprocess.run`) with `stdout=PIPE, stderr=STDOUT`, read line-by-line via `iter(proc.stdout.readline, "")`, and posted to the log queue immediately. Python's own output buffering is disabled with the `-u` flag and `PYTHONUNBUFFERED=1` in the environment — without these, output would batch inside the pipe and only appear when the script exits. The `Popen` call uses `encoding="utf-8", errors="replace"` and sets `PYTHONUTF8=1` in the child environment so that tqdm's Unicode block characters (e.g. `▉`) don't crash the pipe reader on Windows, where the default `charmap` codec cannot decode them.
**GUI threading model:** Every network or long-running operation runs in a `threading.Thread(daemon=True)` so the Tkinter event loop is never blocked. The only safe way to update widgets from a background thread is `self.after(0, callback)` — never touch widgets directly from a worker thread. `_poll_log` drains the entire log queue in one `after(80, ...)` tick and does a single batched `CTkTextbox.insert()` call rather than one per log entry, keeping the UI smooth even when `tqdm` emits many rapid updates during downloads. The wizard's "Test Connection" button follows the same pattern: `requests.get` runs in a daemon thread; the result is posted back via `self.after(0, ...)` with widget references captured *before* the thread starts, so stale references cannot update the wrong widgets if the user navigates away mid-request. **GUI threading model:** Every network or long-running operation runs in a `threading.Thread(daemon=True)` so the Tkinter event loop is never blocked. The only safe way to update widgets from a background thread is `self.after(0, callback)` — never touch widgets directly from a worker thread. `_poll_log` drains the entire log queue in one `after(80, ...)` tick and does a single batched `CTkTextbox.insert()` call rather than one per log entry, keeping the UI smooth even when `tqdm` emits many rapid updates during downloads. The wizard's "Test Connection" button follows the same pattern: `requests.get` runs in a daemon thread; the result is posted back via `self.after(0, ...)` with widget references captured *before* the thread starts, so stale references cannot update the wrong widgets if the user navigates away mid-request.
@@ -116,6 +117,10 @@ Pass 2 builds `ok_disk_names` — the set of disk names that already match the s
**`build_server_index` progress callback:** Accepts an optional `progress_fn(current, total, name)` callback. `step_fetch` in `run.py` uses this to print `Indexing N/M: @FolderName` every 25 folders so the log never goes silent during the server scan phase. The library itself never calls `print` — the caller owns the I/O. **`build_server_index` progress callback:** Accepts an optional `progress_fn(current, total, name)` callback. `step_fetch` in `run.py` uses this to print `Indexing N/M: @FolderName` every 25 folders so the log never goes silent during the server scan phase. The library itself never calls `print` — the caller owns the I/O.
### `update_mods.py` — orphan file removal
After downloading updated files, `update_mods.py` compares every file in the local mod folder against the server's file list and **deletes any local files that no longer exist on the server**. This prevents stale `.pbo` or `.bisign` files from accumulating when a mod's content changes upstream. Each removed file is logged as `[-] orphan removed: <rel_path>` and the final summary line includes an orphan count. The orphan check runs even when no files need downloading (e.g. timestamps match but the local folder has extras).
### GUI localization (`gui/locales.py`) ### GUI localization (`gui/locales.py`)
All user-facing strings are centralised in `gui/locales.py`. Two languages are supported: English (`"en"`) and Vietnamese (`"vi"`). All user-facing strings are centralised in `gui/locales.py`. Two languages are supported: English (`"en"`) and Vietnamese (`"vi"`).

View File

@@ -7,7 +7,7 @@ from typing import TYPE_CHECKING, Optional
import customtkinter as ctk import customtkinter as ctk
from arma_modlist_tools.fetcher import _normalize_name from arma_modlist_tools.fetcher import _normalize_name, _parse_meta_cpp
from gui._constants import COLOR_OK, COLOR_ERROR, COLOR_WARN, COLOR_RUNNING from gui._constants import COLOR_OK, COLOR_ERROR, COLOR_WARN, COLOR_RUNNING
from gui.locales import t from gui.locales import t
from gui.views.base import BaseView from gui.views.base import BaseView
@@ -16,13 +16,14 @@ if TYPE_CHECKING:
from gui.app import ArmaModManagerApp from gui.app import ArmaModManagerApp
def _find_folder(group_dir: Path, mod_name: str) -> Optional[Path]: def _find_folder(group_dir: Path, mod_name: str, steam_id: str | None = None) -> Optional[Path]:
"""Return the local mod folder path, or None if not downloaded. """Return the local mod folder path, or None if not downloaded.
Matches in priority order: Matches in priority order:
1. Exact folder name ``@{mod_name}`` 1. Exact folder name ``@{mod_name}``
2. Case-insensitive name (handles ``@CBA_A3`` vs ``CBA_A3``) 2. Case-insensitive name (handles ``@CBA_A3`` vs ``CBA_A3``)
3. Normalized name — strips non-alphanumeric (handles ``@cba_a3`` vs ``CBA A3``) 3. Normalized name — strips non-alphanumeric (handles ``@cba_a3`` vs ``CBA A3``)
4. Steam ID match via ``meta.cpp`` ``publishedid`` field (folder name differs from modlist name)
""" """
if not group_dir.is_dir(): if not group_dir.is_dir():
return None return None
@@ -38,6 +39,18 @@ def _find_folder(group_dir: Path, mod_name: str) -> Optional[Path]:
return p return p
if _normalize_name(p.name) == target_norm: if _normalize_name(p.name) == target_norm:
return p return p
# Fallback: match by steam_id via meta.cpp when folder name diverges from modlist name
if steam_id:
for p in group_dir.iterdir():
if not p.is_dir():
continue
meta = p / "meta.cpp"
try:
sid = _parse_meta_cpp(meta.read_text(encoding="utf-8", errors="replace"))
if sid == steam_id:
return p
except OSError:
pass
return None return None
@@ -215,7 +228,7 @@ class ModsView(BaseView):
link_map: dict[str, bool], link_map: dict[str, bool],
) -> None: ) -> None:
for i, mod in enumerate(sorted(mods, key=lambda m: m["name"].lower())): for i, mod in enumerate(sorted(mods, key=lambda m: m["name"].lower())):
folder_path = _find_folder(cfg.downloads / group, mod["name"]) folder_path = _find_folder(cfg.downloads / group, mod["name"], mod.get("steam_id"))
downloaded = folder_path is not None downloaded = folder_path is not None
linked = (link_map.get(folder_path.name.lower(), False) linked = (link_map.get(folder_path.name.lower(), False)
if folder_path else False) if folder_path else False)

View File

@@ -2229,6 +2229,128 @@ test("live: list_mod_files entries are (rel_path, url, size) tuples", _test_liv
test("live: find_mod_folder name fallback works (no steam_id)", _test_live_find_mod_by_name_fallback) test("live: find_mod_folder name fallback works (no steam_id)", _test_live_find_mod_by_name_fallback)
# ---------------------------------------------------------------------------
# gui.views.mods — _find_folder
# ---------------------------------------------------------------------------
group("gui.views.mods — _find_folder")
import importlib.util as _mods_ilu
_mods_spec = _mods_ilu.spec_from_file_location(
"gui.views.mods", Path(__file__).parent / "gui" / "views" / "mods.py"
)
_mods_mod = _mods_ilu.module_from_spec(_mods_spec)
# Stub out customtkinter and gui imports so the module loads without a display
import types as _types
import sys as _sys
for _stub in ("customtkinter", "gui", "gui._constants", "gui.locales",
"gui.views", "gui.views.base"):
if _stub not in _sys.modules:
_sys.modules[_stub] = _types.ModuleType(_stub)
# Provide the colour constants the module references at import time
_sys.modules["gui._constants"].COLOR_OK = "#4CAF50"
_sys.modules["gui._constants"].COLOR_ERROR = "#F44336"
_sys.modules["gui._constants"].COLOR_WARN = "#FF9800"
_sys.modules["gui._constants"].COLOR_RUNNING = "#2196F3"
# Stub BaseView so the class body does not fail
_base_stub = _sys.modules["gui.views.base"] = _types.ModuleType("gui.views.base")
_base_stub.BaseView = object
# Stub locales.t so string calls don't crash
_sys.modules["gui.locales"].t = lambda key, **kw: key
_mods_spec.loader.exec_module(_mods_mod)
_find_folder_fn = _mods_mod._find_folder
def _test_ff_returns_none_for_missing_group(tmp_path):
assert _find_folder_fn(tmp_path / "nonexistent", "MyMod") is None
def _test_ff_exact_match(tmp_path):
(tmp_path / "@MyMod").mkdir()
result = _find_folder_fn(tmp_path, "MyMod")
assert result == tmp_path / "@MyMod"
def _test_ff_case_insensitive_match(tmp_path):
(tmp_path / "@mymod").mkdir()
result = _find_folder_fn(tmp_path, "MyMod")
assert result == tmp_path / "@mymod"
def _test_ff_normalized_match(tmp_path):
# Folder on disk uses underscores; modlist name uses spaces — both normalize to same string
(tmp_path / "@My_Mod_Edition").mkdir()
result = _find_folder_fn(tmp_path, "My Mod Edition")
assert result == tmp_path / "@My_Mod_Edition"
def _test_ff_steam_id_fallback(tmp_path):
"""Folder name bears no resemblance to mod name but meta.cpp has correct ID."""
folder = tmp_path / "@ServerCanonicalName"
folder.mkdir()
(folder / "meta.cpp").write_text('publishedid = 123456789;\nname = "Some Mod";\n')
result = _find_folder_fn(tmp_path, "Completely Different Name", steam_id="123456789")
assert result == folder
def _test_ff_steam_id_no_false_positive(tmp_path):
"""Wrong steam_id in meta.cpp must not match."""
folder = tmp_path / "@WrongMod"
folder.mkdir()
(folder / "meta.cpp").write_text('publishedid = 999999999;\n')
result = _find_folder_fn(tmp_path, "My Mod", steam_id="123456789")
assert result is None
def _test_ff_steam_id_skipped_when_none(tmp_path):
"""No steam_id supplied → meta.cpp is never read (no false positives)."""
folder = tmp_path / "@SomeFolder"
folder.mkdir()
(folder / "meta.cpp").write_text('publishedid = 123456789;\n')
result = _find_folder_fn(tmp_path, "My Mod", steam_id=None)
assert result is None
def _test_ff_missing_meta_cpp_skipped(tmp_path):
"""Folders without meta.cpp are silently skipped in the steam_id pass."""
folder = tmp_path / "@NoMeta"
folder.mkdir()
result = _find_folder_fn(tmp_path, "My Mod", steam_id="123456789")
assert result is None
# Wrap tmp_path calls in lambdas that supply a temp dir
def _with_tmp(fn):
def wrapper():
with tempfile.TemporaryDirectory() as d:
fn(Path(d))
return wrapper
test("_find_folder: None when group dir missing",
_with_tmp(_test_ff_returns_none_for_missing_group))
test("_find_folder: exact @ModName match",
_with_tmp(_test_ff_exact_match))
test("_find_folder: case-insensitive name match",
_with_tmp(_test_ff_case_insensitive_match))
test("_find_folder: normalized name match (punctuation differs)",
_with_tmp(_test_ff_normalized_match))
test("_find_folder: steam_id fallback via meta.cpp",
_with_tmp(_test_ff_steam_id_fallback))
test("_find_folder: wrong steam_id in meta.cpp is not a match",
_with_tmp(_test_ff_steam_id_no_false_positive))
test("_find_folder: no steam_id supplied → meta.cpp not checked",
_with_tmp(_test_ff_steam_id_skipped_when_none))
test("_find_folder: missing meta.cpp silently skipped",
_with_tmp(_test_ff_missing_meta_cpp_skipped))
# --------------------------------------------------------------------------- # ---------------------------------------------------------------------------
# Summary # Summary
# --------------------------------------------------------------------------- # ---------------------------------------------------------------------------