Enhance P10Y repository handling with pagination and qualified searches

agents/IMPLEMENTATION.md

@@ -30,6 +30,8 @@
 
 ## Recently Completed (June 2026)
 
+- **Quickstart P10Y repository lookup pagination (Jul 8)**: Fixed Quickstart initialization for P10Y organizations with more than one repository page. `create_generation_session_repos.py` now searches generated repos with the qualified GitHub owner/prefix (for example `org/specflow-workspace`) and uses a shared paginated P10Y repository listing helper for ID resolution, re-fetch connection discovery, status checks, and status polling. Tests: `uv run pytest test/scripts/test_create_generation_session_repos.py -q` from `backend` (25 passed).
+
 - **TUI workspace notification dedupe (Jul 1)**: Fixed workspace drill-in causing repeated desktop "Workspace phase progressed" notifications every poll during KB init. `MilestoneTracker` now emits workspace progress only when `last_completed_phase` increases, so label-only differences between session-list and `/status` payloads stay silent. `WorkspaceMessagesScreen` feeds its `/status` poll into the shared per-run tracker, and the app-wide active-session watcher excludes the workspace screen's generation to avoid mixed payload sources. Tests: `test_tui_poller.py::TestMilestoneTracker::test_kb_init_phase_label_change_without_progress_is_silent`, `test_tui_app.py::TestWorkspaceDrillIn::test_workspace_screen_excludes_app_level_session_watcher`; broader TUI check: `uv run pytest tests/test_tui_app.py tests/test_tui_poller.py -v` from `mcp_server` (78 passed).
 
 - **MCP client setup screen in the TUI (Jun 30)**: New `ClientSetupScreen` (`mcp_server/tui/app.py`) + pure registry `mcp_server/tui/mcp_clients.py` that connect SpecFlow's local MCP server to the user's AI tool, replacing the post-onboarding "No active generation sessions." dead-end (shown first-run via `_startup_gate`, re-openable with `c` on Sessions/Dashboard). **Connect UX**: auto-detects installed clients (`shutil.which` + `~/.cursor`), one-key connect, honest per-client status (green = connected/verified, amber = added-can't-verify, red = failed, grey = not-connected / copy-only). Claude Code → `claude mcp remove`+`add-json -s user` then verify `claude mcp get`; Gemini → `gemini mcp add … -s user` (silent overwrite, trust-folder caveat); Cursor → merge `~/.cursor/mcp.json` (refuse-on-malformed, `.bak` only after confirm) + best-effort deeplink; universal "copy config" fallback. Cursor/Gemini have no read-back so they cap at "added — confirm in your client"; revisiting opens a Yes/No inspection that promotes to connected or demotes to failed. **Real per-client status persisted** in the global **`~/.specflow/config.json`** (`clients` section, values = TUI statuses) so an unverified add is never assumed connected across sessions — global because connecting a client is a machine-wide act (`-s user` / `~/.cursor`) and the TUI must reach it from any project. **`~/.specflow/config.json` is the SSOT for future global SpecFlow/MCP settings** — add new sections as sibling top-level keys via `mcp_clients._read_config`/`_write_config` (they preserve unknown keys); do not put global settings in the project-local `.specflow-local/`. **UI follow-ups (Jul):** modals use the app `Footer` ("ESC to close") instead of an inline hint; the works/doesn't-work inspection is an arrow-navigable `ListView` (not buttons); and on screen open a background probe runs `claude mcp get specflow` for verifiable clients — an already-registered server shows **verified without re-adding** (helps users who already have it), and a stale saved "connected" is cleared (`mcp_clients.forget_status`) if it's gone. One registry feeds both the screen and the `init` CLI hint (replaced the hardcoded `_IDE_REGISTRATION_HINT`). **Hang-safe `local_env.run_command`** (timeout + `proc.kill()` + reap) added because verify/probe commands like `claude mcp get` can block on a network socket with no output — the existing `_stream_subprocess` does an unbounded `await proc.wait()` and would freeze the exclusive Textual worker with no escape. Tests: `test_tui_mcp_clients.py` (pure: json forms, Gemini flag translation, deeplink encoding+no-raw-`+`, merge preserves siblings, status/marker), `test_tui_app.py::TestClientSetupScreen` (render/preselect, connect→verified/added-unverified, malformed-refuse-untouched, inspect confirm/reject/later, status persistence), `test_local_env.py::TestRunCommand` (timeout kills hung process, no zombie). 542 unit tests pass.

agents/MEMORY.md

@@ -10,3 +10,4 @@ Concise, generalized lessons (not a changelog — that is `agents/IMPLEMENTATION
 - Firestore emulator imports require the `*overall_export_metadata` file path; export destinations are directories, but `--import-data` must not point at the snapshot directory itself.
 - Global SpecFlow/TUI settings live in `~/.specflow/config.json` (SSOT) — read/write via `mcp_server/tui/mcp_clients.py` `_read_config`/`_write_config`, which preserve unknown top-level keys; add each new setting as its own top-level section. Do NOT put global settings in the project-local `.specflow-local/` (that dir is per-project runtime: `mcp-config.json`, `workspaces.json`, `init.log`). MCP-client connection status is stored globally because connecting a client is a machine-wide act (`claude/gemini mcp add -s user`, Cursor `~/.cursor/mcp.json`).
 - Run MCP server pytest commands from `mcp_server/` (`uv run pytest tests/...`); root-level `uv run pytest ...` may not expose a pytest executable because the repo has per-package `pyproject.toml` environments.
+- P10Y repository list reads in Quickstart must use the narrowest available search (`github_org/prefix` for generated repos) and paginate; first-page org scans miss repos in large Compass organizations.

backend/app/services/p10y/p10y_api_client.py

@@ -19,6 +19,22 @@
 logging.getLogger("httpx").setLevel(logging.INFO)
 logging.getLogger("httpcore").setLevel(logging.INFO)
 
+_REPOSITORY_LIST_PAGE_SIZE = 1000
+_REPOSITORY_LIST_MAX_PAGES = 1000
+
+
+def _response_total_pages(response: Dict[str, Any]) -> Optional[int]:
+    """Extract a positive page count from a paginated P10Y response, if present."""
+    total_pages = response.get("totalPages") or response.get("total_pages")
+    if isinstance(total_pages, bool) or total_pages is None:
+        return None
+    try:
+        parsed = int(total_pages)
+    except (TypeError, ValueError):
+        return None
+    return parsed if parsed > 0 else None
+
+
 class P10YInternalAPIClient:
     """
     Client for interacting with the P10Y Internal API.
@@ -220,6 +236,40 @@ async def list_repositories(
         )
         return response.json()
 
+    async def list_repositories_paginated(
+        self,
+        organisation_id: int,
+        search: Optional[str] = None,
+        page_size: int = _REPOSITORY_LIST_PAGE_SIZE,
+        max_pages: int = _REPOSITORY_LIST_MAX_PAGES,
+    ) -> List[Dict[str, Any]]:
+        """Read all repository pages within an organization for the given search scope."""
+        repositories: List[Dict[str, Any]] = []
+        page = 1
+
+        while True:
+            repos_response = await self.list_repositories(
+                organisation_id=organisation_id,
+                search=search,
+                page=page,
+                page_size=page_size,
+            )
+            page_data = repos_response.get("data", [])
+            repositories.extend(page_data)
+
+            total_pages = _response_total_pages(repos_response)
+            if total_pages is not None:
+                if page >= total_pages:
+                    break
+            elif len(page_data) < page_size:
+                break
+
+            page += 1
+            if page > max_pages:
+                raise RuntimeError(f"P10Y repository listing exceeded {max_pages} pages")
+
+        return repositories
+
     async def update_repository(
         self,
         organisation_id: int,

backend/scripts/create_generation_session_repos.py

@@ -331,11 +331,20 @@ def _normalize_git_url(git_url: str) -> str:
     return s
 
 
+def _p10y_repository_search(prefix: str, github_org: Optional[str]) -> str:
+    """Build the narrowest P10Y repository search value available."""
+    clean_prefix = (prefix or "").strip()
+    clean_org = (github_org or "").strip().strip("/")
+    if clean_prefix and clean_org:
+        return f"{clean_org}/{clean_prefix}"
+    return clean_prefix
+
+
 async def get_repository_ids(
     p10y_client: P10YInternalAPIClient,
     org_id: int,
     repo_names: List[str],
-    prefix: str,
+    search: Optional[str] = None,
     github_org: Optional[str] = None,
 ) -> Dict[str, int]:
     """
@@ -345,20 +354,16 @@ async def get_repository_ids(
         p10y_client: Initialized P10y API client
         org_id: P10y organization ID
         repo_names: List of repository names to find
-        prefix: Search prefix for list_repositories
+        search: Search filter for list_repositories (e.g. the qualified
+            ``<github_org>/<prefix>`` string built by ``_p10y_repository_search``)
         github_org: GitHub org owning the repos. When set, matching is done on
             ``git_url`` (``<org>/<name>``) rather than the bare ``repository_name``.
     Returns:
         Dictionary mapping repository names to their P10y IDs
     """
     print("\n🔍 Looking up P10y repository IDs")
 
-    # Fetch all repositories
-    repos_response = await p10y_client.list_repositories(
-        organisation_id=org_id,
-        search=prefix,
-        page_size=1000,  # Should be enough for our case,
-    )
+    repos = await p10y_client.list_repositories_paginated(org_id, search=search)
 
     # P10Y `repository_name` is the BARE repo name and is NOT unique within a Compass
     # organisation — the same bare name can exist under several GitHub orgs, distinguished
@@ -373,7 +378,7 @@ async def get_repository_ids(
     repo_name_set = set(repo_names)
 
     repo_id_map: Dict[str, int] = {}
-    for repo_data in repos_response.get("data", []):
+    for repo_data in repos:
         if expected_by_git_url:
             matched_name = expected_by_git_url.get(
                 _normalize_git_url(repo_data.get("git_url", ""))
@@ -402,16 +407,24 @@ async def trigger_repository_refetch(
     p10y_client: P10YInternalAPIClient,
     org_id: int,
     github_org: Optional[str],
-    repo_names: List[str],
 ) -> None:
-    """Trigger Compass's 'Re-fetch' on the connection(s) owning the workspace repos; omitting connection_id may return 400 but the sync still completes."""
-    repos = (await p10y_client.list_repositories(organisation_id=org_id, page_size=1000)).get("data", [])
-    expected = (
-        {_normalize_git_url(f"{github_org}/{name}") for name in repo_names} if github_org else None
-    )
+    """Trigger Compass's re-fetch on the connection(s) owning ``github_org``.
+
+    A Compass connection is per GitHub org/account, not per repo, so any repo
+    already ingested under ``github_org`` reveals the right connection — the
+    brand-new repos being provisioned are never yet visible in P10Y (that's
+    why this is being called), so matching only their exact names would
+    always miss and force a broadcast re-fetch across every active GitHub
+    connection instead of just the one that actually owns them.
+    """
+    search = github_org.strip().strip("/") if github_org else None
+    repos = await p10y_client.list_repositories_paginated(org_id, search=search)
+    org_prefix = f"{_normalize_git_url(github_org)}/" if github_org else None
+
     conn_ids: set[int] = set()
     for repo_data in repos:
-        if expected is not None and _normalize_git_url(repo_data.get("git_url", "")) not in expected:
+        git_url = _normalize_git_url(repo_data.get("git_url", ""))
+        if org_prefix is not None and not git_url.startswith(org_prefix):
             continue
         cid = (repo_data.get("_embedded", {}).get("connection") or {}).get("id_connection")
         if cid:
@@ -450,19 +463,17 @@ async def get_repository_statuses(
     p10y_client: P10YInternalAPIClient,
     org_id: int,
     repo_ids: List[int],
+    search: Optional[str] = None,
 ) -> Dict[int, Dict[str, Any]]:
     """Fetch current P10Y statuses for the target repository IDs."""
     if not repo_ids:
         return {}
 
-    repos_response = await p10y_client.list_repositories(
-        organisation_id=org_id,
-        page_size=1000,
-    )
+    repos = await p10y_client.list_repositories_paginated(org_id, search=search)
 
     target_ids = set(repo_ids)
     statuses: Dict[int, Dict[str, Any]] = {}
-    for repo_data in repos_response.get("data", []):
+    for repo_data in repos:
         repo_id = _p10y_repository_id(repo_data)
         if repo_id in target_ids:
             statuses[repo_id] = {
@@ -521,7 +532,8 @@ async def poll_repository_status(
     org_id: int,
     repo_ids: List[int],
     timeout_minutes: int = 5,
-    poll_interval: int = 15
+    poll_interval: int = 15,
+    search: Optional[str] = None,
 ) -> Dict[int, Dict[str, Any]]:
     """
     Poll P10y to check when repositories become live with metrics.
@@ -555,13 +567,10 @@ async def poll_repository_status(
 
         try:
             # Fetch repository details
-            repos_response = await p10y_client.list_repositories(
-                organisation_id=org_id,
-                page_size=1000
-            )
+            repos = await p10y_client.list_repositories_paginated(org_id, search=search)
 
             # Update status for our repos
-            for repo_data in repos_response.get("data", []):
+            for repo_data in repos:
                 repo_id = _p10y_repository_id(repo_data)
                 if repo_id in repo_ids:
                     internal_status = repo_data.get("internal_status")
@@ -591,7 +600,11 @@ async def poll_repository_status(
 
             # Wait before next poll
             await asyncio.sleep(poll_interval)
-
+
+        except RuntimeError:
+            # Structural failure (e.g. pagination cap exceeded) — not a transient
+            # polling error, so fail fast instead of retrying until timeout.
+            raise
         except Exception as e:
             print(f"⚠️  Error polling status: {e}")
             await asyncio.sleep(poll_interval)
@@ -953,7 +966,10 @@ async def main():
     firestore_target_from_cli = bool(gcp_cli and fsdb_cli)
 
     # Validate arguments
-    if args.start > args.end:
+    if args.repos is None and (args.start is None or args.end is None):
+        parser.error("--start and --end are required unless --repos is provided")
+
+    if args.start is not None and args.end is not None and args.start > args.end:
         print("❌ Error: start number must be less than or equal to end number")
         sys.exit(1)
 
@@ -1038,9 +1054,6 @@ async def main():
     elif not git_username:
         git_username = "(not resolved; set GIT_USER_NAME_DEFAULT or GITHUB_TOKEN_DEFAULT)"
 
-    if args.repos is None and (args.start is None or args.end is None):
-        parser.error("--start and --end are required unless --repos is provided")
-
     print("=" * 80)
     title = "🚀 Generation Workspace Repository Setup"
     if args.dry_run:
@@ -1113,8 +1126,12 @@ async def main():
             repo_names = [f"{args.prefix}{num}" for num in range(args.start, args.end + 1)]
 
         # Step 2: Get P10y repository IDs
-        # For --repos, search with empty prefix to match arbitrary names across the full org.
-        p10y_search_prefix = "" if own_repo_list is not None else args.prefix
+        # For --repos, search with an empty string to match arbitrary names across the full org.
+        # Computed once and reused verbatim (never re-qualified) by every lookup below —
+        # get_repository_ids no longer builds its own search string from a raw prefix.
+        p10y_search_prefix = (
+            "" if own_repo_list is not None else _p10y_repository_search(args.prefix, github_org)
+        )
         repo_id_map = await get_repository_ids(
             p10y_client, p10y_org_id, repo_names, p10y_search_prefix, github_org
         )
@@ -1126,7 +1143,7 @@ async def main():
         missing = [r for r in repo_names if r not in repo_id_map]
         if missing:
             print(f"\n🔄 {len(missing)} repo(s) not in P10Y yet: {', '.join(missing)} — triggering re-fetch ...")
-            await trigger_repository_refetch(p10y_client, p10y_org_id, github_org, repo_names)
+            await trigger_repository_refetch(p10y_client, p10y_org_id, github_org)
 
             deadline = time.time() + P10Y_REFETCH_TIMEOUT_SECONDS
             while missing and time.time() < deadline:
@@ -1157,7 +1174,12 @@ async def main():
         # Step 4: Start metrics calculation only for repos that are not already Live.
         # The --repos path skips metrics: those repos already have history in Compass.
         if not args.skip_metrics and own_repo_list is None:
-            current_statuses = await get_repository_statuses(p10y_client, p10y_org_id, repo_ids)
+            current_statuses = await get_repository_statuses(
+                p10y_client,
+                p10y_org_id,
+                repo_ids,
+                search=p10y_search_prefix,
+            )
             metrics_repo_ids = repository_ids_requiring_metrics(repo_ids, current_statuses)
 
             if metrics_repo_ids:
@@ -1172,7 +1194,8 @@ async def main():
                     p10y_org_id,
                     repo_ids,
                     args.poll_timeout,
-                    args.poll_interval
+                    args.poll_interval,
+                    search=p10y_search_prefix,
                 )
             else:
                 final_statuses = current_statuses