diff --git a/CHANGELOG.md b/CHANGELOG.md index a60278e..bbc490b 100755 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -5,6 +5,41 @@ All notable changes to this project will be documented in this file. The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/), and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html). +## [1.4.0] - 2026-01-24 + +### Added +- **Intelligent Source Filtering** - LLM-based relevance scoring before each query + - Uses Gemini or Claude to score each source (1-10) based on semantic relevance + - Automatically deselects irrelevant sources in browser for faster responses + - Fallback chain: Gemini CLI → Claude CLI → keyword matching + - Configurable threshold (`--threshold N`) for filtering sensitivity + - `--keyword-filter` flag to use keyword matching instead of LLM + - `--no-filter` flag to disable filtering entirely + +- **Source Summary Caching** - Automated extraction and caching of source metadata + - New `source_extractor.py` - Click-based extraction of Source Guide content + - New `source_filter.py` - LLM scoring and browser checkbox automation + - Auto-extracts source summaries when adding notebooks + - Cached in `data/library-source-summary/{notebook-id}.md` + - Incremental updates - only fetches NEW sources not in cache + +- **Source Management Commands** + - `update-sources --id ID` - Update source summary for a notebook + - `update-sources --id ID --force` - Re-extract all sources + - `update-sources --all` - Update all notebooks + - `show-sources --id ID` - Display cached source summaries + +### Changed +- **Python 3 Enforcement** - All scripts now use `python3` instead of `python` +- **Browser Switch** - Changed from Chrome to Chromium for better compatibility +- **ask_question.py** - Now integrates source filtering before each query + +### Fixed +- **Checkbox Automation** - Uses Playwright native clicks for reliable Angular interaction + - JavaScript `.click()` didn't trigger Angular change detection + - Now uses `checkbox.click(force=True)` for proper event handling + - Each deselection is verified after clicking + ## [1.3.0] - 2025-11-21 ### Added diff --git a/README.md b/README.md index 0a46e9c..a4170cc 100755 --- a/README.md +++ b/README.md @@ -161,21 +161,30 @@ This is a **Claude Code Skill** - a local folder containing instructions and scr ``` ~/.claude/skills/notebooklm/ -├── SKILL.md # Instructions for Claude -├── scripts/ # Python automation scripts -│ ├── ask_question.py # Query NotebookLM -│ ├── notebook_manager.py # Library management -│ └── auth_manager.py # Google authentication -├── .venv/ # Isolated Python environment (auto-created) -└── data/ # Local notebook library +├── SKILL.md # Instructions for Claude +├── scripts/ # Python automation scripts +│ ├── ask_question.py # Query NotebookLM with source filtering +│ ├── notebook_manager.py # Library + source summary management +│ ├── auth_manager.py # Google authentication +│ ├── source_filter.py # LLM-based relevance scoring +│ ├── source_extractor.py # Extract source summaries from notebooks +│ ├── add_source.py # Add sources (files, URLs, text) +│ ├── list_sources.py # List sources via UI scraping +│ ├── remove_source.py # Remove sources (with safety confirmation) +│ ├── audio_generator.py # Generate Audio Overviews +│ └── video_generator.py # Generate Video Overviews +├── .venv/ # Isolated Python environment (auto-created) +└── data/ # Local notebook library + source summaries ``` When you mention NotebookLM or send a notebook URL, Claude: 1. Loads the skill instructions 2. Runs the appropriate Python script -3. Opens a browser, asks your question -4. Returns the answer directly to you -5. Uses that knowledge to help with your task +3. **Scores source relevance** using Gemini/Claude LLM +4. **Deselects irrelevant sources** in the browser +5. Opens a browser, asks your question +6. Returns the answer directly to you +7. Uses that knowledge to help with your task --- @@ -190,6 +199,39 @@ No copy-paste between browser and editor. Claude asks and receives answers progr ### **Smart Library Management** Save NotebookLM links with tags and descriptions. Claude auto-selects the right notebook for your task. +### **Intelligent Source Filtering** +Before each query, the skill analyzes which sources are relevant: +- **LLM-based scoring**: Uses Gemini or Claude to semantically score each source (1-10) +- **Automatic deselection**: Unchecks irrelevant sources in the browser +- **Faster responses**: NotebookLM only searches relevant documents +- **Fallback support**: Gemini → Claude → keyword matching + +### **Source Summary Caching** +When you add a notebook, the skill automatically: +- Extracts summaries for each source document +- Caches them locally for instant relevance scoring +- Updates incrementally (only fetches new sources) + +### **Source Management** +Add, list, and remove sources directly from Claude Code: +- **Add sources**: Upload PDFs, paste URLs (websites/YouTube), or add text directly +- **List sources**: View all sources in a notebook via reliable UI scraping +- **Remove sources**: Permanently delete sources with safety confirmation + +### **Audio Overview Generation** +Generate podcast-style audio summaries of your notebook content: +- **Formats**: Deep Dive, Brief, Critique, Debate +- **Lengths**: Short, Default, Long +- **Multi-language support**: English, Spanish, Japanese, and more +- **Custom instructions**: Guide what the AI hosts focus on + +### **Video Overview Generation** +Generate video summaries with customizable visual styles: +- **Formats**: Explainer, Brief +- **Visual styles**: Auto, Classic, Whiteboard, Kawaii, Anime +- **Multi-language support** +- **Custom instructions**: Target specific use cases or audiences + ### **Automatic Authentication** One-time Google login, then authentication persists across sessions. @@ -206,10 +248,18 @@ Uses realistic typing speeds and interaction patterns to avoid detection. | What you say | What happens | |--------------|--------------| | *"Set up NotebookLM authentication"* | Opens Chrome for Google login | -| *"Add [link] to my NotebookLM library"* | Saves notebook with metadata | +| *"Add [link] to my NotebookLM library"* | Saves notebook with metadata + extracts source summaries | | *"Show my NotebookLM notebooks"* | Lists all saved notebooks | -| *"Ask my API docs about [topic]"* | Queries the relevant notebook | +| *"Ask my API docs about [topic]"* | Queries with intelligent source filtering | | *"Use the React notebook"* | Sets active notebook | +| *"Update sources for my notebook"* | Re-fetches source summaries | +| *"Show sources in my notebook"* | Displays cached source summaries | +| *"Add this PDF to my notebook"* | Uploads file as a new source | +| *"Add this URL to my notebook"* | Adds website or YouTube video as source | +| *"List sources in my notebook"* | Shows all sources via UI scraping | +| *"Remove [source] from my notebook"* | Permanently deletes a source (with confirmation) | +| *"Generate an audio overview"* | Creates podcast-style audio summary | +| *"Generate a video overview"* | Creates video summary with visuals | | *"Clear NotebookLM data"* | Fresh start (keeps library) | --- @@ -264,11 +314,16 @@ All data is stored locally within the skill directory: ``` ~/.claude/skills/notebooklm/data/ -├── library.json - Your notebook library with metadata -├── auth_info.json - Authentication status info -└── browser_state/ - Browser cookies and session data +├── library.json - Your notebook library with metadata +├── auth_info.json - Authentication status info +├── browser_state/ - Browser cookies and session data +└── library-source-summary/ - Cached source summaries + └── {notebook-id}.md - Source titles and summaries for each notebook ``` +**Source Summary Files:** +Each notebook gets a markdown file with summaries of all sources. These are used for intelligent source filtering - the LLM scores each source's relevance to your question before querying. + **Important Security Note:** - The `data/` directory contains sensitive authentication data and personal notebooks - It's automatically excluded from git via `.gitignore` diff --git a/SKILL.md b/SKILL.md index 2be7e16..92c374a 100755 --- a/SKILL.md +++ b/SKILL.md @@ -23,10 +23,10 @@ When user wants to add a notebook without providing details: **SMART ADD (Recommended)**: Query the notebook first to discover its content: ```bash # Step 1: Query the notebook about its content -python scripts/run.py ask_question.py --question "What is the content of this notebook? What topics are covered? Provide a complete overview briefly and concisely" --notebook-url "[URL]" +python3 scripts/run.py ask_question.py --question "What is the content of this notebook? What topics are covered? Provide a complete overview briefly and concisely" --notebook-url "[URL]" # Step 2: Use the discovered information to add it -python scripts/run.py notebook_manager.py add --url "[URL]" --name "[Based on content]" --description "[Based on content]" --topics "[Based on content]" +python3 scripts/run.py notebook_manager.py add --url "[URL]" --name "[Based on content]" --description "[Based on content]" --topics "[Based on content]" ``` **MANUAL ADD**: If user provides all details: @@ -39,13 +39,15 @@ NEVER guess or use generic descriptions! If details missing, use Smart Add to di ## Critical: Always Use run.py Wrapper -**NEVER call scripts directly. ALWAYS use `python scripts/run.py [script]`:** +**NEVER call scripts directly. ALWAYS use `python3 scripts/run.py [script]`:** + +> **Note:** Use `python3` if available, otherwise use `python`. Check with `which python3 || which python`. ```bash # ✅ CORRECT - Always use run.py: -python scripts/run.py auth_manager.py status -python scripts/run.py notebook_manager.py list -python scripts/run.py ask_question.py --question "..." +python3 scripts/run.py auth_manager.py status +python3 scripts/run.py notebook_manager.py list +python3 scripts/run.py ask_question.py --question "..." # ❌ WRONG - Never call directly: python scripts/auth_manager.py status # Fails without venv! @@ -61,7 +63,7 @@ The `run.py` wrapper automatically: ### Step 1: Check Authentication Status ```bash -python scripts/run.py auth_manager.py status +python3 scripts/run.py auth_manager.py status ``` If not authenticated, proceed to setup. @@ -69,7 +71,7 @@ If not authenticated, proceed to setup. ### Step 2: Authenticate (One-Time Setup) ```bash # Browser MUST be visible for manual Google login -python scripts/run.py auth_manager.py setup +python3 scripts/run.py auth_manager.py setup ``` **Important:** @@ -82,49 +84,227 @@ python scripts/run.py auth_manager.py setup ```bash # List all notebooks -python scripts/run.py notebook_manager.py list +python3 scripts/run.py notebook_manager.py list # BEFORE ADDING: Ask user for metadata if unknown! # "What does this notebook contain?" # "What topics should I tag it with?" # Add notebook to library (ALL parameters are REQUIRED!) -python scripts/run.py notebook_manager.py add \ +python3 scripts/run.py notebook_manager.py add \ --url "https://notebooklm.google.com/notebook/..." \ --name "Descriptive Name" \ --description "What this notebook contains" \ # REQUIRED - ASK USER IF UNKNOWN! --topics "topic1,topic2,topic3" # REQUIRED - ASK USER IF UNKNOWN! # Search notebooks by topic -python scripts/run.py notebook_manager.py search --query "keyword" +python3 scripts/run.py notebook_manager.py search --query "keyword" # Set active notebook -python scripts/run.py notebook_manager.py activate --id notebook-id +python3 scripts/run.py notebook_manager.py activate --id notebook-id # Remove notebook -python scripts/run.py notebook_manager.py remove --id notebook-id +python3 scripts/run.py notebook_manager.py remove --id notebook-id ``` ### Quick Workflow -1. Check library: `python scripts/run.py notebook_manager.py list` -2. Ask question: `python scripts/run.py ask_question.py --question "..." --notebook-id ID` +1. Check library: `python3 scripts/run.py notebook_manager.py list` +2. Ask question: `python3 scripts/run.py ask_question.py --question "..." --notebook-id ID` ### Step 4: Ask Questions ```bash # Basic query (uses active notebook if set) -python scripts/run.py ask_question.py --question "Your question here" +python3 scripts/run.py ask_question.py --question "Your question here" # Query specific notebook -python scripts/run.py ask_question.py --question "..." --notebook-id notebook-id +python3 scripts/run.py ask_question.py --question "..." --notebook-id notebook-id # Query with notebook URL directly -python scripts/run.py ask_question.py --question "..." --notebook-url "https://..." +python3 scripts/run.py ask_question.py --question "..." --notebook-url "https://..." + +# Show browser for debugging +python3 scripts/run.py ask_question.py --question "..." --show-browser +``` + +### Step 5: Add Sources to Notebooks + +Add new sources directly to your NotebookLM notebooks: + +```bash +# Upload a file (PDF, txt, md, docx, etc.) - uses native file upload +python3 scripts/run.py add_source.py --file /path/to/document.pdf + +# Add a website URL +python3 scripts/run.py add_source.py --url "https://example.com/article" + +# Add a YouTube video (transcript will be imported) +python3 scripts/run.py add_source.py --url "https://www.youtube.com/watch?v=VIDEO_ID" + +# Add copied text directly +python3 scripts/run.py add_source.py --text "Your text content here" + +# Add text from a file (reads file and pastes as copied text) +python3 scripts/run.py add_source.py --text-file /path/to/content.txt + +# Specify notebook (uses active notebook if not specified) +python3 scripts/run.py add_source.py --notebook-id ID --file /path/to/file.pdf + +# Show browser for debugging +python3 scripts/run.py add_source.py --file /path/to/file.pdf --show-browser +``` + +**Supported File Types (via --file):** +NotebookLM natively supports: PDF, TXT, Markdown, Google Docs, Google Slides, and more. +Files are uploaded directly and processed by NotebookLM for best results. + +**URL Sources (via --url):** +- Websites: Only visible text is imported; paid articles not supported +- YouTube: Only public videos with transcripts supported + +**Text Sources (via --text or --text-file):** +- Direct text paste for content not in a file +- Use --text-file to read from a local text file + +### Step 6: Generate Audio Overview + +Generate podcast-style audio summaries of your notebook content with customizable prompts: + +```bash +# Basic audio generation (uses default settings) +python3 scripts/run.py audio_generator.py --notebook-url "https://notebooklm.google.com/notebook/..." + +# With custom prompt/instructions +python3 scripts/run.py audio_generator.py --notebook-url URL --instructions "Focus on the main arguments and explain like I'm a beginner" + +# Choose format and length +python3 scripts/run.py audio_generator.py --notebook-id ID --format brief --length short + +# Choose a different language +python3 scripts/run.py audio_generator.py --notebook-id ID --language "Spanish" + +# Full customization with output filename +python3 scripts/run.py audio_generator.py --notebook-id ID \ + --format deep_dive \ + --length long \ + --language "Japanese" \ + --instructions "Discuss the technical implementation details" \ + --output "my_podcast.wav" + +# Show browser for debugging +python3 scripts/run.py audio_generator.py --notebook-url URL --show-browser +``` + +**Audio Format Options:** +| Format | Description | +|--------|-------------| +| `deep_dive` | A lively conversation between two hosts, unpacking and connecting topics (default) | +| `brief` | A bite-sized overview to grasp core ideas quickly | +| `critique` | An expert review with constructive feedback | +| `debate` | A thoughtful debate illuminating different perspectives | + +**Audio Length Options:** +- `short` - Quick summary +- `default` - Standard length +- `long` - Extended discussion + +**Language (--language):** +Specify a language for the audio output (e.g., "English", "Spanish", "Japanese", "French", "German", etc.). Defaults to English if not specified. + +**Custom Instructions (--instructions):** +Guide what the AI hosts focus on: +- Focus on a specific source: "only cover the article about Italy" +- Focus on a specific topic: "just discuss the novel's main character" +- Target an audience: "explain to someone new to biology" +- Request a style: "make it conversational and fun" + +**Note:** Audio generation can take 5-10 minutes. The script will show progress updates while waiting. + +### Step 9: Generate Video Overview + +Generate video summaries of your notebook content with customizable format, visual style, and instructions: + +```bash +# Basic video generation (uses default settings) +python3 scripts/run.py video_generator.py --notebook-url "https://notebooklm.google.com/notebook/..." + +# With custom prompt/instructions +python3 scripts/run.py video_generator.py --notebook-url URL --instructions "Focus on the key concepts and explain visually" + +# Choose format and visual style +python3 scripts/run.py video_generator.py --notebook-id ID --format brief --style whiteboard + +# Choose a different language +python3 scripts/run.py video_generator.py --notebook-id ID --language "French" + +# Full customization with output filename +python3 scripts/run.py video_generator.py --notebook-id ID \ + --format explainer \ + --style kawaii \ + --language "Japanese" \ + --instructions "Present this to a book club" \ + --output "my_video.mp4" + +# Show browser for debugging +python3 scripts/run.py video_generator.py --notebook-url URL --show-browser +``` + +**Video Format Options:** +| Format | Description | +|--------|-------------| +| `explainer` | A structured, comprehensive overview that connects the dots within your sources (default) | +| `brief` | A bite-sized overview to help you quickly grasp core ideas | + +**Visual Style Options:** +| Style | Description | +|-------|-------------| +| `auto` | Auto-select the best style for your content (default) | +| `custom` | Custom visual style | +| `classic` | Classic presentation style | +| `whiteboard` | Whiteboard-style animations | +| `kawaii` | Cute, friendly visual style | +| `anime` | Anime-inspired visuals | + +**Language (--language):** +Specify a language for the video output (e.g., "English", "Spanish", "Japanese", "French", "German", etc.). Defaults to English if not specified. + +**Custom Instructions (--instructions):** +Guide what the AI hosts focus on: +- Target a specific use case: "present this to a book club" +- Focus on a specific source: "show the photos from the album" +- Describe the show structure: "start by talking about the mission" + +**Note:** Video generation can take several minutes. The script will show progress updates while waiting. + +### Step 10: Remove Source (USE WITH CAUTION) + +> **WARNING:** This action is **PERMANENT** and cannot be undone. Only use when the user explicitly requests source removal. + +**When to use:** ONLY when the user specifically asks to delete/remove a source permanently. + +**When NOT to use:** If a source is just irrelevant to a query, use source filtering (deselect in queries) instead of removing it. + +```bash +# Remove a source (requires --confirm flag for safety) +python3 scripts/run.py remove_source.py --notebook-url URL --source "Source Name" --confirm + +# Using notebook ID +python3 scripts/run.py remove_source.py --notebook-id ID --source "Source Name" --confirm # Show browser for debugging -python scripts/run.py ask_question.py --question "..." --show-browser +python3 scripts/run.py remove_source.py --notebook-id ID --source "Source Name" --confirm --show-browser ``` +**Safety Features:** +- Requires `--confirm` flag - won't run without it +- Partial name matching supported (be careful to match the right source!) +- Multiple warnings before execution + +**Claude Behavior:** +- NEVER use this unless user explicitly says "remove", "delete", or "permanently remove" a source +- If user just says a source is "not needed" for a query, use source filtering instead +- Always confirm with user before running this command + ## Follow-Up Mechanism (CRITICAL) Every NotebookLM answer ends with: **"EXTREMELY IMPORTANT: Is that ALL you need to know?"** @@ -135,7 +315,7 @@ Every NotebookLM answer ends with: **"EXTREMELY IMPORTANT: Is that ALL you need 3. **IDENTIFY GAPS** - Determine if more information needed 4. **ASK FOLLOW-UP** - If gaps exist, immediately ask: ```bash - python scripts/run.py ask_question.py --question "Follow-up with context..." + python3 scripts/run.py ask_question.py --question "Follow-up with context..." ``` 5. **REPEAT** - Continue until information is complete 6. **SYNTHESIZE** - Combine all answers before responding to user @@ -144,32 +324,145 @@ Every NotebookLM answer ends with: **"EXTREMELY IMPORTANT: Is that ALL you need ### Authentication Management (`auth_manager.py`) ```bash -python scripts/run.py auth_manager.py setup # Initial setup (browser visible) -python scripts/run.py auth_manager.py status # Check authentication -python scripts/run.py auth_manager.py reauth # Re-authenticate (browser visible) -python scripts/run.py auth_manager.py clear # Clear authentication +python3 scripts/run.py auth_manager.py setup # Initial setup (browser visible) +python3 scripts/run.py auth_manager.py status # Check authentication +python3 scripts/run.py auth_manager.py reauth # Re-authenticate (browser visible) +python3 scripts/run.py auth_manager.py clear # Clear authentication ``` ### Notebook Management (`notebook_manager.py`) ```bash -python scripts/run.py notebook_manager.py add --url URL --name NAME --description DESC --topics TOPICS -python scripts/run.py notebook_manager.py list -python scripts/run.py notebook_manager.py search --query QUERY -python scripts/run.py notebook_manager.py activate --id ID -python scripts/run.py notebook_manager.py remove --id ID -python scripts/run.py notebook_manager.py stats +python3 scripts/run.py notebook_manager.py add --url URL --name NAME --description DESC --topics TOPICS +python3 scripts/run.py notebook_manager.py list +python3 scripts/run.py notebook_manager.py search --query QUERY +python3 scripts/run.py notebook_manager.py activate --id ID +python3 scripts/run.py notebook_manager.py remove --id ID +python3 scripts/run.py notebook_manager.py stats + +# Source Summary Management +python3 scripts/run.py notebook_manager.py update-sources --id ID # Update source summary (incremental) +python3 scripts/run.py notebook_manager.py update-sources --id ID --force # Re-extract all sources +python3 scripts/run.py notebook_manager.py update-sources --all # Update all notebooks +python3 scripts/run.py notebook_manager.py show-sources --id ID # Show cached source summaries ``` +**Source Summary System:** +- When adding a notebook, source summaries are automatically extracted +- Summaries are cached in `data/library-source-summary/{notebook-id}.md` +- Incremental updates only fetch NEW sources not already in the cache +- Use `--force` to re-extract all sources from scratch + ### Question Interface (`ask_question.py`) ```bash -python scripts/run.py ask_question.py --question "..." [--notebook-id ID] [--notebook-url URL] [--show-browser] +# Basic query (uses active notebook if set) +python3 scripts/run.py ask_question.py --question "..." + +# Query specific notebook +python3 scripts/run.py ask_question.py --question "..." --notebook-id ID + +# Query with notebook URL directly +python3 scripts/run.py ask_question.py --question "..." --notebook-url URL + +# Show browser for debugging +python3 scripts/run.py ask_question.py --question "..." --show-browser + +# Source filtering options (default: LLM scoring enabled) +python3 scripts/run.py ask_question.py --question "..." --keyword-filter # Use keyword matching instead of LLM +python3 scripts/run.py ask_question.py --question "..." --threshold 7 # Higher threshold = fewer sources (1-10) +python3 scripts/run.py ask_question.py --question "..." --no-filter # Disable source filtering entirely ``` +### Source Addition (`add_source.py`) +```bash +# Upload file (native upload - best for PDFs) +python3 scripts/run.py add_source.py --file /path/to/file.pdf + +# Add URL (website or YouTube) +python3 scripts/run.py add_source.py --url URL + +# Add copied text +python3 scripts/run.py add_source.py --text "Content" +python3 scripts/run.py add_source.py --text-file /path/to/text.txt + +# Options +--notebook-url URL # Target notebook URL +--notebook-id ID # Target notebook ID from library +--show-browser # Show browser for debugging +``` + +### Source Listing (`list_sources.py`) +```bash +# List sources in active notebook (reads UI directly - more reliable than asking) +python3 scripts/run.py list_sources.py + +# List sources in specific notebook +python3 scripts/run.py list_sources.py --notebook-id ID +python3 scripts/run.py list_sources.py --notebook-url URL + +# Output as JSON +python3 scripts/run.py list_sources.py --json + +# Show browser for debugging +python3 scripts/run.py list_sources.py --show-browser +``` +**Note:** This reads source names directly from the UI, which is more reliable than asking NotebookLM to list sources via a question. Use this to verify sources before/after removal. + +**Source Filtering Behavior:** +Before each query, the skill automatically: +1. Loads cached source summaries from `data/library-source-summary/{notebook-id}.md` +2. Uses LLM (Gemini → Claude → keywords fallback) to score each source's relevance (1-10) +3. Deselects sources below threshold in the browser +4. Queries only relevant sources for faster, focused responses + +### Audio Generation (`audio_generator.py`) +```bash +# Generate with defaults +python3 scripts/run.py audio_generator.py --notebook-url URL + +# With custom prompt +python3 scripts/run.py audio_generator.py --notebook-id ID --instructions "Your prompt here" + +# Full options +python3 scripts/run.py audio_generator.py --notebook-url URL \ + --format deep_dive|brief|critique|debate \ + --length short|default|long \ + --language "English|Spanish|Japanese|..." \ + --instructions "Custom prompt" \ + --output filename.wav \ + --show-browser +``` + +### Video Generation (`video_generator.py`) +```bash +# Generate with defaults +python3 scripts/run.py video_generator.py --notebook-url URL + +# With custom prompt and style +python3 scripts/run.py video_generator.py --notebook-id ID --instructions "Your prompt here" --style whiteboard + +# Full options +python3 scripts/run.py video_generator.py --notebook-url URL \ + --format explainer|brief \ + --style auto|custom|classic|whiteboard|kawaii|anime \ + --language "English|Spanish|Japanese|..." \ + --instructions "Custom prompt" \ + --output filename.mp4 \ + --show-browser +``` + +### Source Removal (`remove_source.py`) - PERMANENT ACTION +```bash +# Remove a source (--confirm REQUIRED) +python3 scripts/run.py remove_source.py --notebook-url URL --source "Source Name" --confirm +python3 scripts/run.py remove_source.py --notebook-id ID --source "Source Name" --confirm --show-browser +``` +**WARNING:** Only use when user explicitly requests permanent removal. Use source filtering for temporary exclusion. + ### Data Cleanup (`cleanup_manager.py`) ```bash -python scripts/run.py cleanup_manager.py # Preview cleanup -python scripts/run.py cleanup_manager.py --confirm # Execute cleanup -python scripts/run.py cleanup_manager.py --preserve-library # Keep notebooks +python3 scripts/run.py cleanup_manager.py # Preview cleanup +python3 scripts/run.py cleanup_manager.py --confirm # Execute cleanup +python3 scripts/run.py cleanup_manager.py --preserve-library # Keep notebooks ``` ## Environment Management @@ -190,10 +483,12 @@ python -m patchright install chromium ## Data Storage -All data stored in `~/.claude/skills/notebooklm/data/`: +All data stored in `data/` directory within the skill folder: - `library.json` - Notebook metadata - `auth_info.json` - Authentication status - `browser_state/` - Browser cookies and session +- `library-source-summary/` - Source summaries for each notebook (auto-generated) + - `{notebook-id}.md` - Markdown file with table of sources and summaries **Security:** Protected by `.gitignore`, never commit to git. @@ -209,22 +504,61 @@ TYPING_WPM_MAX=240 DEFAULT_NOTEBOOK_ID= # Default notebook ``` +## Intelligent Source Filtering + +Before each query, the skill automatically filters sources: + +``` +Question received + ↓ +Load source summaries from cache + ↓ +LLM scores each source (1-10 relevance) + ↓ +Deselect sources below threshold + ↓ +Query only relevant sources + ↓ +Faster, more focused response +``` + +**LLM Fallback Chain:** +1. **Gemini CLI** (default) - Fast, semantic understanding +2. **Claude CLI** - Fallback if Gemini not installed +3. **Keywords** - Final fallback if no LLM available + +**Threshold Guide:** +- `--threshold 3`: Include most sources (broad search) +- `--threshold 5`: Default, balanced filtering +- `--threshold 7`: Only highly relevant sources (focused search) +- `--threshold 9`: Only direct matches + ## Decision Flow ``` User mentions NotebookLM ↓ -Check auth → python scripts/run.py auth_manager.py status +Check auth → python3 scripts/run.py auth_manager.py status + ↓ +If not authenticated → python3 scripts/run.py auth_manager.py setup + ↓ +Check/Add notebook → python3 scripts/run.py notebook_manager.py list/add (with --description) ↓ -If not authenticated → python scripts/run.py auth_manager.py setup +Source summaries auto-extracted and cached ↓ -Check/Add notebook → python scripts/run.py notebook_manager.py list/add (with --description) +Activate notebook → python3 scripts/run.py notebook_manager.py activate --id ID ↓ -Activate notebook → python scripts/run.py notebook_manager.py activate --id ID +┌─────────────────────────────────────────────────────────────┐ +│ Choose action based on user request: │ +├─────────────────────────────────────────────────────────────┤ +│ Ask question → python3 scripts/run.py ask_question.py ... │ +│ Generate audio → python3 scripts/run.py audio_generator.py │ +│ Generate video → python3 scripts/run.py video_generator.py │ +└─────────────────────────────────────────────────────────────┘ ↓ -Ask question → python scripts/run.py ask_question.py --question "..." +(For questions) See "Is that ALL you need?" → Ask follow-ups until complete ↓ -See "Is that ALL you need?" → Ask follow-ups until complete +(For audio/video) Wait for generation → Download media file ↓ Synthesize and respond to user ``` @@ -236,8 +570,10 @@ Synthesize and respond to user | ModuleNotFoundError | Use `run.py` wrapper | | Authentication fails | Browser must be visible for setup! --show-browser | | Rate limit (50/day) | Wait or switch Google account | -| Browser crashes | `python scripts/run.py cleanup_manager.py --preserve-library` | +| Browser crashes | `python3 scripts/run.py cleanup_manager.py --preserve-library` | | Notebook not found | Check with `notebook_manager.py list` | +| Source not relevant | Use `--no-filter` or adjust `--threshold` - DON'T remove it | +| Need to delete source | Only use `remove_source.py` if user explicitly requests permanent deletion | ## Best Practices @@ -247,6 +583,7 @@ Synthesize and respond to user 4. **Browser visible for auth** - Required for manual login 5. **Include context** - Each question is independent 6. **Synthesize answers** - Combine multiple responses +7. **Filter, don't delete** - Use source filtering for irrelevant sources; only use `remove_source.py` when user explicitly asks for permanent deletion ## Limitations @@ -254,6 +591,7 @@ Synthesize and respond to user - Rate limits on free Google accounts (50 queries/day) - Manual upload required (user must add docs to NotebookLM) - Browser overhead (few seconds per question) +- Audio/Video generation takes several minutes (NotebookLM limitation) ## Resources (Skill Structure) diff --git a/docs/plans/2026-01-25-add-source-feature.md b/docs/plans/2026-01-25-add-source-feature.md new file mode 100644 index 0000000..4c2274f --- /dev/null +++ b/docs/plans/2026-01-25-add-source-feature.md @@ -0,0 +1,760 @@ +# Add Source Feature Implementation Plan + +> **For Claude:** REQUIRED SUB-SKILL: Use superpowers:executing-plans to implement this plan task-by-task. + +**Goal:** Add the ability to add sources (local files via native upload, website/YouTube URLs, copied text) to NotebookLM notebooks via browser automation. + +**Architecture:** Create a new `add_source.py` script that uses Playwright browser automation to interact with NotebookLM's "Add sources" dialog. Three source types supported: +1. **File upload** - Uses native OS file picker via Playwright's `expect_file_chooser()` +2. **URLs** - Pastes website/YouTube links into the "Websites" textarea +3. **Text** - Pastes content into the "Copied text" textarea + +**Tech Stack:** Python, Patchright (Playwright fork with stealth features) + +--- + +## UI Reference (from screenshots) + +**Main dialog buttons:** +- "Upload files" → Opens native OS file picker +- "Websites" → Shows textarea with "Paste any links" placeholder +- "Drive" → Google Drive (not implementing - requires OAuth) +- "Copied text" → Shows textarea for pasting text + +**After clicking "Websites":** +- Textarea: "Paste any links" +- Supports multiple URLs separated by space/newline +- "Insert" button to submit + +--- + +### Task 1: Add UI Selectors to config.py + +**Files:** +- Modify: `scripts/config.py` + +**Step 1: Add new selector constants after RESPONSE_SELECTORS (line 40)** + +```python +# Add Source Dialog Selectors (based on NotebookLM UI Jan 2026) +ADD_SOURCE_BUTTON_SELECTORS = [ + 'button:has-text("Add source")', + 'button:has-text("Add sources")', + 'button[aria-label*="Add source"]', + 'button[aria-label*="Add sources"]', + '[data-test-id="add-source-button"]', +] + +# Main dialog option buttons +UPLOAD_FILES_BUTTON_SELECTORS = [ + 'button:has-text("Upload files")', + '[aria-label*="Upload files"]', + 'button:has-text("Upload")', +] + +WEBSITES_BUTTON_SELECTORS = [ + 'button:has-text("Websites")', + '[aria-label*="Websites"]', + 'button:has-text("Website")', +] + +COPIED_TEXT_BUTTON_SELECTORS = [ + 'button:has-text("Copied text")', + '[aria-label*="Copied text"]', + 'button:has-text("Paste")', +] + +# Input fields +URL_TEXTAREA_SELECTORS = [ + 'textarea[placeholder*="Paste any links"]', + 'textarea[placeholder*="link"]', + 'textarea[placeholder*="URL"]', + 'textarea', +] + +TEXT_TEXTAREA_SELECTORS = [ + 'textarea[placeholder*="Paste"]', + 'textarea[placeholder*="paste"]', + 'textarea', +] + +# Action buttons +INSERT_BUTTON_SELECTORS = [ + 'button:has-text("Insert")', + 'button:has-text("Add")', + 'button[type="submit"]', +] + +CLOSE_DIALOG_SELECTORS = [ + 'button[aria-label="Close"]', + 'button:has-text("Close")', + '[aria-label="close"]', +] +``` + +**Step 2: Verify config.py syntax** + +Run: `cd /Users/ellengu/Documents/ObsidianFolder/TheVault/.claude/skills/notebooklm-skill && python3 -c "from scripts.config import *; print('Selectors loaded:', len(ADD_SOURCE_BUTTON_SELECTORS))"` +Expected: `Selectors loaded: 5` + +**Step 3: Commit** + +```bash +git add scripts/config.py +git commit -m "$(cat <<'EOF' +feat(config): add UI selectors for add source dialog + +Add selector constants for NotebookLM's add source interface: +- ADD_SOURCE_BUTTON_SELECTORS - main "Add sources" button +- UPLOAD_FILES_BUTTON_SELECTORS - native file upload +- WEBSITES_BUTTON_SELECTORS - URL input +- COPIED_TEXT_BUTTON_SELECTORS - text paste +- URL_TEXTAREA_SELECTORS, TEXT_TEXTAREA_SELECTORS - input fields +- INSERT_BUTTON_SELECTORS - submit button + +Co-Authored-By: Claude Opus 4.5 +EOF +)" +``` + +--- + +### Task 2: Create add_source.py Script + +**Files:** +- Create: `scripts/add_source.py` + +**Step 1: Create the main script** + +```python +#!/usr/bin/env python3 +""" +Add sources to NotebookLM notebooks. +Supports: File upload (PDF, txt, md, docx), Website/YouTube URLs, Copied text. + +Each source addition opens a fresh browser session, adds the source, and closes. +""" + +import argparse +import sys +import time +import re +from pathlib import Path + +from patchright.sync_api import sync_playwright + +sys.path.insert(0, str(Path(__file__).parent)) + +from auth_manager import AuthManager +from notebook_manager import NotebookLibrary +from config import ( + DATA_DIR, + ADD_SOURCE_BUTTON_SELECTORS, + UPLOAD_FILES_BUTTON_SELECTORS, + WEBSITES_BUTTON_SELECTORS, + COPIED_TEXT_BUTTON_SELECTORS, + URL_TEXTAREA_SELECTORS, + TEXT_TEXTAREA_SELECTORS, + INSERT_BUTTON_SELECTORS, +) +from browser_utils import BrowserFactory, StealthUtils + + +def find_and_click(page, selectors: list, description: str, timeout: int = 5000) -> bool: + """Find element using multiple selectors and click it.""" + for selector in selectors: + try: + element = page.wait_for_selector(selector, timeout=timeout, state="visible") + if element: + print(f" Found {description}: {selector}") + element.click() + return True + except: + continue + print(f" Could not find {description}") + return False + + +def find_element(page, selectors: list, timeout: int = 3000): + """Find element using multiple selectors.""" + for selector in selectors: + try: + element = page.wait_for_selector(selector, timeout=timeout, state="visible") + if element: + return element + except: + continue + return None + + +def navigate_to_add_source_dialog(page, notebook_url: str) -> bool: + """Navigate to notebook and open the Add Sources dialog.""" + print(" Opening notebook...") + page.goto(notebook_url, wait_until="domcontentloaded") + page.wait_for_url(re.compile(r"^https://notebooklm\.google\.com/"), timeout=15000) + time.sleep(3) + + if "/notebook/" not in page.url: + print(f" Redirected away from notebook: {page.url}") + return False + + # Click Sources tab if visible + try: + sources_tab = page.query_selector('button:has-text("Sources"), [role="tab"]:has-text("Sources")') + if sources_tab and sources_tab.is_visible(): + sources_tab.click() + time.sleep(1) + except: + pass + + # Click "Add sources" button + print(" Opening Add Sources dialog...") + if not find_and_click(page, ADD_SOURCE_BUTTON_SELECTORS, "Add sources button"): + debug_path = DATA_DIR / "add_source_debug.png" + page.screenshot(path=str(debug_path)) + print(f" Debug screenshot: {debug_path}") + return False + + time.sleep(1.5) + return True + + +def add_file_source(notebook_url: str, file_path: str, headless: bool = True) -> str: + """ + Upload a file as a source using the native file picker. + + Args: + notebook_url: NotebookLM notebook URL + file_path: Path to file (PDF, txt, md, docx, etc.) + headless: Run browser in headless mode + + Returns: + Success message or None on failure + """ + auth = AuthManager() + if not auth.is_authenticated(): + print(" Not authenticated. Run: python scripts/run.py auth_manager.py setup") + return None + + file_path = Path(file_path).resolve() + if not file_path.exists(): + print(f" File not found: {file_path}") + return None + + print(f" Uploading file: {file_path.name}") + print(f" Size: {file_path.stat().st_size / 1024:.1f} KB") + print(f" Notebook: {notebook_url}") + + playwright = None + context = None + + try: + playwright = sync_playwright().start() + context = BrowserFactory.launch_persistent_context(playwright, headless=headless) + page = context.new_page() + + if not navigate_to_add_source_dialog(page, notebook_url): + return None + + # Click "Upload files" and handle file chooser + print(" Selecting file to upload...") + + with page.expect_file_chooser(timeout=10000) as fc_info: + if not find_and_click(page, UPLOAD_FILES_BUTTON_SELECTORS, "Upload files button"): + return None + + file_chooser = fc_info.value + file_chooser.set_files(str(file_path)) + print(f" File selected: {file_path.name}") + + # Wait for upload to process + print(" Waiting for upload to complete...") + time.sleep(5) + + # Check for success (source should appear in list) + # NotebookLM processes the file and adds it to sources + print(f" File source uploaded: {file_path.name}") + return f"Successfully uploaded file: {file_path.name}" + + except Exception as e: + print(f" Error: {e}") + import traceback + traceback.print_exc() + return None + + finally: + if context: + try: + context.close() + except: + pass + if playwright: + try: + playwright.stop() + except: + pass + + +def add_url_source(notebook_url: str, source_url: str, headless: bool = True) -> str: + """ + Add a website or YouTube URL as a source. + + Args: + notebook_url: NotebookLM notebook URL + source_url: Website or YouTube URL to add + headless: Run browser in headless mode + + Returns: + Success message or None on failure + """ + auth = AuthManager() + if not auth.is_authenticated(): + print(" Not authenticated. Run: python scripts/run.py auth_manager.py setup") + return None + + if not source_url or not source_url.strip(): + print(" URL cannot be empty") + return None + + is_youtube = "youtube.com" in source_url or "youtu.be" in source_url + source_type = "YouTube" if is_youtube else "Website" + + print(f" Adding {source_type} source: {source_url}") + print(f" Notebook: {notebook_url}") + + playwright = None + context = None + + try: + playwright = sync_playwright().start() + context = BrowserFactory.launch_persistent_context(playwright, headless=headless) + page = context.new_page() + + if not navigate_to_add_source_dialog(page, notebook_url): + return None + + # Click "Websites" button + print(" Selecting Websites option...") + if not find_and_click(page, WEBSITES_BUTTON_SELECTORS, "Websites button"): + return None + + time.sleep(1) + + # Find textarea and enter URL + print(" Entering URL...") + textarea = find_element(page, URL_TEXTAREA_SELECTORS) + if not textarea: + print(" Could not find URL textarea") + return None + + textarea.click() + StealthUtils.random_delay(200, 400) + textarea.fill(source_url) + time.sleep(0.5) + + # Click Insert button + print(" Submitting...") + if not find_and_click(page, INSERT_BUTTON_SELECTORS, "Insert button", timeout=3000): + page.keyboard.press("Enter") + + time.sleep(4) + + print(f" {source_type} source added!") + return f"Successfully added {source_type} source: {source_url}" + + except Exception as e: + print(f" Error: {e}") + import traceback + traceback.print_exc() + return None + + finally: + if context: + try: + context.close() + except: + pass + if playwright: + try: + playwright.stop() + except: + pass + + +def add_text_source(notebook_url: str, content: str, headless: bool = True) -> str: + """ + Add copied text as a source. + + Args: + notebook_url: NotebookLM notebook URL + content: Text content to add + headless: Run browser in headless mode + + Returns: + Success message or None on failure + """ + auth = AuthManager() + if not auth.is_authenticated(): + print(" Not authenticated. Run: python scripts/run.py auth_manager.py setup") + return None + + if not content or not content.strip(): + print(" Content cannot be empty") + return None + + preview = content[:100] + "..." if len(content) > 100 else content + print(f" Adding text source ({len(content)} chars)") + print(f" Preview: {preview}") + print(f" Notebook: {notebook_url}") + + playwright = None + context = None + + try: + playwright = sync_playwright().start() + context = BrowserFactory.launch_persistent_context(playwright, headless=headless) + page = context.new_page() + + if not navigate_to_add_source_dialog(page, notebook_url): + return None + + # Click "Copied text" button + print(" Selecting Copied text option...") + if not find_and_click(page, COPIED_TEXT_BUTTON_SELECTORS, "Copied text button"): + return None + + time.sleep(1) + + # Find textarea and enter content + print(" Entering text content...") + textarea = find_element(page, TEXT_TEXTAREA_SELECTORS) + if not textarea: + print(" Could not find text textarea") + return None + + textarea.click() + StealthUtils.random_delay(200, 400) + + # Use fill() for large content + if len(content) > 500: + textarea.fill(content) + else: + StealthUtils.human_type(page, TEXT_TEXTAREA_SELECTORS[0], content) + + time.sleep(0.5) + + # Click Insert button + print(" Submitting...") + time.sleep(1) # Wait for button to enable + + if not find_and_click(page, INSERT_BUTTON_SELECTORS, "Insert button", timeout=3000): + page.keyboard.press("Enter") + + time.sleep(3) + + print(" Text source added!") + return f"Successfully added text source ({len(content)} characters)" + + except Exception as e: + print(f" Error: {e}") + import traceback + traceback.print_exc() + return None + + finally: + if context: + try: + context.close() + except: + pass + if playwright: + try: + playwright.stop() + except: + pass + + +def main(): + parser = argparse.ArgumentParser( + description='Add sources to NotebookLM notebooks', + formatter_class=argparse.RawDescriptionHelpFormatter, + epilog=""" +Examples: + # Upload a PDF file + python3 scripts/run.py add_source.py --file /path/to/document.pdf + + # Add a website URL + python3 scripts/run.py add_source.py --url "https://example.com/article" + + # Add a YouTube video + python3 scripts/run.py add_source.py --url "https://youtube.com/watch?v=..." + + # Add copied text + python3 scripts/run.py add_source.py --text "Your text content here" + + # Read text from a file and paste it (for very large text) + python3 scripts/run.py add_source.py --text-file /path/to/content.txt + """ + ) + + parser.add_argument('--notebook-url', help='NotebookLM notebook URL') + parser.add_argument('--notebook-id', help='Notebook ID from library') + + # Source types (mutually exclusive) + source_group = parser.add_mutually_exclusive_group(required=True) + source_group.add_argument('--file', help='Path to file to upload (PDF, txt, md, docx, etc.)') + source_group.add_argument('--url', help='Website or YouTube URL to add') + source_group.add_argument('--text', help='Text content to paste as source') + source_group.add_argument('--text-file', help='Path to text file to read and paste as source') + + parser.add_argument('--show-browser', action='store_true', help='Show browser window') + + args = parser.parse_args() + + # Resolve notebook URL + notebook_url = args.notebook_url + + if not notebook_url and args.notebook_id: + library = NotebookLibrary() + notebook = library.get_notebook(args.notebook_id) + if notebook: + notebook_url = notebook['url'] + print(f" Using notebook: {notebook['name']}") + else: + print(f" Notebook '{args.notebook_id}' not found") + return 1 + + if not notebook_url: + library = NotebookLibrary() + active = library.get_active_notebook() + if active: + notebook_url = active['url'] + print(f" Using active notebook: {active['name']}") + else: + print(" No notebook specified. Use --notebook-url or --notebook-id") + return 1 + + # Execute based on source type + result = None + + if args.file: + result = add_file_source( + notebook_url=notebook_url, + file_path=args.file, + headless=not args.show_browser + ) + + elif args.url: + result = add_url_source( + notebook_url=notebook_url, + source_url=args.url, + headless=not args.show_browser + ) + + elif args.text: + result = add_text_source( + notebook_url=notebook_url, + content=args.text, + headless=not args.show_browser + ) + + elif args.text_file: + text_path = Path(args.text_file) + if not text_path.exists(): + print(f" File not found: {args.text_file}") + return 1 + + print(f" Reading text from: {args.text_file}") + content = text_path.read_text(encoding='utf-8') + print(f" Content size: {len(content)} characters") + + result = add_text_source( + notebook_url=notebook_url, + content=content, + headless=not args.show_browser + ) + + # Print result + print("\n" + "=" * 50) + if result: + print(f" {result}") + return 0 + else: + print(" Failed to add source") + return 1 + + +if __name__ == "__main__": + sys.exit(main()) +``` + +**Step 2: Make script executable** + +Run: `chmod +x /Users/ellengu/Documents/ObsidianFolder/TheVault/.claude/skills/notebooklm-skill/scripts/add_source.py` + +**Step 3: Verify syntax** + +Run: `cd /Users/ellengu/Documents/ObsidianFolder/TheVault/.claude/skills/notebooklm-skill && python3 -m py_compile scripts/add_source.py && echo "Syntax OK"` +Expected: `Syntax OK` + +**Step 4: Commit** + +```bash +git add scripts/add_source.py +git commit -m "$(cat <<'EOF' +feat(add_source): add script to add sources to NotebookLM + +Three source types supported: +- --file: Native file upload (PDF, txt, md, docx, etc.) via OS file picker +- --url: Website or YouTube URLs via paste textarea +- --text / --text-file: Copied text via paste textarea + +Uses Playwright's expect_file_chooser() for native file upload. + +Co-Authored-By: Claude Opus 4.5 +EOF +)" +``` + +--- + +### Task 3: Update SKILL.md Documentation + +**Files:** +- Modify: `SKILL.md` + +**Step 1: Add new section after "### Step 4: Ask Questions" (around line 127)** + +```markdown +### Step 5: Add Sources to Notebooks + +Add new sources directly to your NotebookLM notebooks: + +```bash +# Upload a file (PDF, txt, md, docx, etc.) - uses native file upload +python3 scripts/run.py add_source.py --file /path/to/document.pdf + +# Add a website URL +python3 scripts/run.py add_source.py --url "https://example.com/article" + +# Add a YouTube video (transcript will be imported) +python3 scripts/run.py add_source.py --url "https://www.youtube.com/watch?v=VIDEO_ID" + +# Add copied text directly +python3 scripts/run.py add_source.py --text "Your text content here" + +# Add text from a file (reads file and pastes as copied text) +python3 scripts/run.py add_source.py --text-file /path/to/content.txt + +# Specify notebook (uses active notebook if not specified) +python3 scripts/run.py add_source.py --notebook-id ID --file /path/to/file.pdf + +# Show browser for debugging +python3 scripts/run.py add_source.py --file /path/to/file.pdf --show-browser +``` + +**Supported File Types (via --file):** +NotebookLM natively supports: PDF, TXT, Markdown, Google Docs, Google Slides, and more. +Files are uploaded directly and processed by NotebookLM for best results. + +**URL Sources (via --url):** +- Websites: Only visible text is imported; paid articles not supported +- YouTube: Only public videos with transcripts supported + +**Text Sources (via --text or --text-file):** +- Direct text paste for content not in a file +- Use --text-file to read from a local text file +``` + +**Step 2: Add to Script Reference section (around line 193)** + +```markdown +### Source Addition (`add_source.py`) +```bash +# Upload file (native upload - best for PDFs) +python3 scripts/run.py add_source.py --file /path/to/file.pdf + +# Add URL (website or YouTube) +python3 scripts/run.py add_source.py --url URL + +# Add copied text +python3 scripts/run.py add_source.py --text "Content" +python3 scripts/run.py add_source.py --text-file /path/to/text.txt + +# Options +--notebook-url URL # Target notebook URL +--notebook-id ID # Target notebook ID from library +--show-browser # Show browser for debugging +``` +``` + +**Step 3: Commit** + +```bash +git add SKILL.md +git commit -m "$(cat <<'EOF' +docs(SKILL.md): add documentation for add_source.py + +Document the new source addition feature: +- Native file upload (--file) for PDF, txt, md, docx +- URL sources (--url) for websites and YouTube +- Text sources (--text, --text-file) + +Co-Authored-By: Claude Opus 4.5 +EOF +)" +``` + +--- + +### Task 4: Manual Testing + +**Files:** None (testing only) + +**Step 1: Test file upload** + +Run: `cd /Users/ellengu/Documents/ObsidianFolder/TheVault/.claude/skills/notebooklm-skill && python3 scripts/run.py add_source.py --file /path/to/test.pdf --show-browser` + +Expected: Browser opens, file picker triggers, file uploads successfully + +**Step 2: Test URL source** + +Run: `cd /Users/ellengu/Documents/ObsidianFolder/TheVault/.claude/skills/notebooklm-skill && python3 scripts/run.py add_source.py --url "https://en.wikipedia.org/wiki/Meiji_Restoration" --show-browser` + +Expected: Browser opens, URL pasted, source added + +**Step 3: Test text source** + +Run: `cd /Users/ellengu/Documents/ObsidianFolder/TheVault/.claude/skills/notebooklm-skill && python3 scripts/run.py add_source.py --text "Test content for NotebookLM" --show-browser` + +Expected: Browser opens, text pasted, source added + +--- + +### Task 5: Cleanup exploration script + +**Files:** +- Delete: `scripts/explore_add_source_ui.py` + +**Step 1: Remove temporary exploration script** + +Run: `rm /Users/ellengu/Documents/ObsidianFolder/TheVault/.claude/skills/notebooklm-skill/scripts/explore_add_source_ui.py` + +--- + +## Summary + +| Task | Description | Files | +|------|-------------|-------| +| 1 | Add UI selectors | `scripts/config.py` | +| 2 | Create add_source.py | `scripts/add_source.py` (new) | +| 3 | Update documentation | `SKILL.md` | +| 4 | Manual testing | (browser tests) | +| 5 | Cleanup | Delete temp script | + +**Key Features:** +- **Native file upload** via `--file` - Uses `expect_file_chooser()` for PDFs and all supported formats +- **URL sources** via `--url` - Websites and YouTube (transcript import) +- **Text sources** via `--text` or `--text-file` - Direct paste + +**No PyPDF2 needed!** Native file upload means NotebookLM handles PDF parsing. diff --git a/references/api_reference.md b/references/api_reference.md index a0ce65e..1e2169c 100755 --- a/references/api_reference.md +++ b/references/api_reference.md @@ -8,7 +8,7 @@ Complete API documentation for all NotebookLM skill modules. ```bash # ✅ CORRECT: -python scripts/run.py [script_name].py [arguments] +python3 scripts/run.py [script_name].py [arguments] # ❌ WRONG: python scripts/[script_name].py [arguments] # Will fail without venv! @@ -17,20 +17,25 @@ python scripts/[script_name].py [arguments] # Will fail without venv! ## Core Scripts ### ask_question.py -Query NotebookLM with automated browser interaction. +Query NotebookLM with automated browser interaction and intelligent source filtering. ```bash # Basic usage -python scripts/run.py ask_question.py --question "Your question" +python3 scripts/run.py ask_question.py --question "Your question" # With specific notebook -python scripts/run.py ask_question.py --question "..." --notebook-id notebook-id +python3 scripts/run.py ask_question.py --question "..." --notebook-id notebook-id # With direct URL -python scripts/run.py ask_question.py --question "..." --notebook-url "https://..." +python3 scripts/run.py ask_question.py --question "..." --notebook-url "https://..." # Show browser (debugging) -python scripts/run.py ask_question.py --question "..." --show-browser +python3 scripts/run.py ask_question.py --question "..." --show-browser + +# Source filtering options +python3 scripts/run.py ask_question.py --question "..." --keyword-filter # Use keywords instead of LLM +python3 scripts/run.py ask_question.py --question "..." --threshold 7 # Higher = fewer sources (1-10) +python3 scripts/run.py ask_question.py --question "..." --no-filter # Disable source filtering ``` **Parameters:** @@ -38,68 +43,85 @@ python scripts/run.py ask_question.py --question "..." --show-browser - `--notebook-id`: Use notebook from library - `--notebook-url`: Use URL directly - `--show-browser`: Make browser visible +- `--keyword-filter`: Use keyword matching instead of LLM scoring +- `--threshold N`: Minimum relevance score for LLM filter (1-10, default: 5) +- `--no-filter`: Disable source filtering entirely + +**Source Filtering Flow:** +1. Loads cached source summaries from `data/library-source-summary/` +2. LLM (Gemini → Claude → keywords) scores each source 1-10 +3. Deselects sources below threshold in browser +4. Queries only relevant sources **Returns:** Answer text with follow-up prompt appended ### notebook_manager.py -Manage notebook library with CRUD operations. +Manage notebook library with CRUD operations and source summary management. ```bash # Smart Add (discover content first) -python scripts/run.py ask_question.py --question "What is the content of this notebook? What topics are covered? Provide a complete overview briefly and concisely" --notebook-url "[URL]" +python3 scripts/run.py ask_question.py --question "What is the content of this notebook? What topics are covered? Provide a complete overview briefly and concisely" --notebook-url "[URL]" # Then add with discovered info -python scripts/run.py notebook_manager.py add \ +python3 scripts/run.py notebook_manager.py add \ --url "https://notebooklm.google.com/notebook/..." \ --name "Name" \ --description "Description" \ --topics "topic1,topic2" # Direct add (when you know the content) -python scripts/run.py notebook_manager.py add \ +python3 scripts/run.py notebook_manager.py add \ --url "https://notebooklm.google.com/notebook/..." \ --name "Name" \ --description "What it contains" \ --topics "topic1,topic2" # List notebooks -python scripts/run.py notebook_manager.py list +python3 scripts/run.py notebook_manager.py list # Search notebooks -python scripts/run.py notebook_manager.py search --query "keyword" +python3 scripts/run.py notebook_manager.py search --query "keyword" # Activate notebook -python scripts/run.py notebook_manager.py activate --id notebook-id +python3 scripts/run.py notebook_manager.py activate --id notebook-id # Remove notebook -python scripts/run.py notebook_manager.py remove --id notebook-id +python3 scripts/run.py notebook_manager.py remove --id notebook-id # Show statistics -python scripts/run.py notebook_manager.py stats +python3 scripts/run.py notebook_manager.py stats + +# Source Summary Management +python3 scripts/run.py notebook_manager.py update-sources --id ID # Update (incremental) +python3 scripts/run.py notebook_manager.py update-sources --id ID --force # Re-extract all +python3 scripts/run.py notebook_manager.py update-sources --all # Update all notebooks +python3 scripts/run.py notebook_manager.py show-sources --id ID # Display cached summaries ``` **Commands:** -- `add`: Add notebook (requires --url, --name, --topics) +- `add`: Add notebook (requires --url, --name, --topics) - auto-extracts source summaries - `list`: Show all notebooks - `search`: Find notebooks by keyword - `activate`: Set default notebook - `remove`: Delete from library - `stats`: Display library statistics +- `update-sources`: Update source summaries (--force to re-extract all) +- `show-sources`: Display cached source summaries ### auth_manager.py Handle Google authentication and browser state. ```bash # Setup (browser visible for login) -python scripts/run.py auth_manager.py setup +python3 scripts/run.py auth_manager.py setup # Check status -python scripts/run.py auth_manager.py status +python3 scripts/run.py auth_manager.py status # Re-authenticate -python scripts/run.py auth_manager.py reauth +python3 scripts/run.py auth_manager.py reauth # Clear authentication -python scripts/run.py auth_manager.py clear +python3 scripts/run.py auth_manager.py clear ``` **Commands:** @@ -113,16 +135,16 @@ Clean skill data with preservation options. ```bash # Preview cleanup -python scripts/run.py cleanup_manager.py +python3 scripts/run.py cleanup_manager.py # Execute cleanup -python scripts/run.py cleanup_manager.py --confirm +python3 scripts/run.py cleanup_manager.py --confirm # Keep library -python scripts/run.py cleanup_manager.py --confirm --preserve-library +python3 scripts/run.py cleanup_manager.py --confirm --preserve-library # Force without prompt -python scripts/run.py cleanup_manager.py --confirm --force +python3 scripts/run.py cleanup_manager.py --confirm --force ``` **Options:** @@ -135,11 +157,11 @@ Script wrapper that handles environment setup. ```bash # Usage -python scripts/run.py [script_name].py [arguments] +python3 scripts/run.py [script_name].py [arguments] # Examples -python scripts/run.py auth_manager.py status -python scripts/run.py ask_question.py --question "..." +python3 scripts/run.py auth_manager.py status +python3 scripts/run.py ask_question.py --question "..." ``` **Automatic actions:** @@ -186,10 +208,33 @@ Location: `~/.claude/skills/notebooklm/data/` ``` data/ -├── library.json # Notebook metadata -├── auth_info.json # Auth status -└── browser_state/ # Browser cookies - └── state.json +├── library.json # Notebook metadata +├── auth_info.json # Auth status +├── browser_state/ # Browser cookies +│ └── state.json +└── library-source-summary/ # Cached source summaries + └── {notebook-id}.md # Source titles and summaries per notebook +``` + +**Source Summary Format:** +```markdown +# Source Summary: Notebook Name + +**Notebook ID:** `notebook-id` +**Generated:** 2026-01-24 12:00:00 +**Total Sources:** 18 + +--- + +### Source Title 1.pdf + +Summary of the source content extracted from NotebookLM's Source Guide... + +--- + +### Source Title 2.md + +Another source summary... ``` **Security:** Protected by `.gitignore`, never commit. @@ -287,6 +332,9 @@ def batch_research(questions, notebook_id): - `get_notebook(notebook_id)` - `activate_notebook(notebook_id)` - `remove_notebook(notebook_id)` +- `fetch_source_summary(notebook_id)` - Extract source summaries via browser +- `update_source_summary(notebook_id, force=False)` - Update cached summaries +- `get_source_summary(notebook_id)` - Load cached summaries ### AuthManager - `is_authenticated()` @@ -295,6 +343,19 @@ def batch_research(questions, notebook_id): - `clear_auth()` - `validate_auth()` +### SourceFilter +- `get_relevant_sources(question, use_llm=True, threshold=5)` - Score and filter sources +- `_get_relevant_sources_llm(question, threshold)` - LLM-based scoring +- `_get_relevant_sources_keywords(question)` - Keyword-based scoring +- `_call_gemini(prompt)` - Call Gemini CLI +- `_call_claude(prompt)` - Call Claude CLI (fallback) +- `get_all_source_titles()` - Get all source titles + +### SourceExtractor +- `extract_all_sources(notebook_url, headless=True)` - Extract all source summaries +- `get_existing_sources(notebook_id)` - Load existing summaries +- Used internally by NotebookLibrary + ### BrowserSession (internal) - Handles browser automation - Manages stealth behavior diff --git a/references/usage_patterns.md b/references/usage_patterns.md index ad517e9..b73ee9d 100755 --- a/references/usage_patterns.md +++ b/references/usage_patterns.md @@ -7,8 +7,8 @@ Advanced patterns for using the NotebookLM skill effectively. **Every command must use the run.py wrapper:** ```bash # ✅ CORRECT: -python scripts/run.py auth_manager.py status -python scripts/run.py ask_question.py --question "..." +python3 scripts/run.py auth_manager.py status +python3 scripts/run.py ask_question.py --question "..." # ❌ WRONG: python scripts/auth_manager.py status # Will fail! @@ -18,16 +18,16 @@ python scripts/auth_manager.py status # Will fail! ```bash # 1. Check authentication (using run.py!) -python scripts/run.py auth_manager.py status +python3 scripts/run.py auth_manager.py status # 2. If not authenticated, setup (Browser MUST be visible!) -python scripts/run.py auth_manager.py setup +python3 scripts/run.py auth_manager.py setup # Tell user: "Please log in to Google in the browser window" # 3. Add first notebook - ASK USER FOR DETAILS FIRST! # Ask: "What does this notebook contain?" # Ask: "What topics should I tag it with?" -python scripts/run.py notebook_manager.py add \ +python3 scripts/run.py notebook_manager.py add \ --url "https://notebooklm.google.com/notebook/..." \ --name "User provided name" \ --description "User provided description" \ # NEVER GUESS! @@ -46,12 +46,12 @@ python scripts/run.py notebook_manager.py add \ **OPTION A: Smart Discovery (Recommended)** ```bash # 1. Query the notebook to discover its content -python scripts/run.py ask_question.py \ +python3 scripts/run.py ask_question.py \ --question "What is the content of this notebook? What topics are covered? Provide a complete overview briefly and concisely" \ --notebook-url "[URL]" # 2. Use discovered info to add it -python scripts/run.py notebook_manager.py add \ +python3 scripts/run.py notebook_manager.py add \ --url "[URL]" \ --name "[Based on content]" \ --description "[From discovery]" \ @@ -65,7 +65,7 @@ python scripts/run.py notebook_manager.py add \ "What topics does it cover?" # Then add with user-provided info: -python scripts/run.py notebook_manager.py add \ +python3 scripts/run.py notebook_manager.py add \ --url "[URL]" \ --name "[User's answer]" \ --description "[User's description]" \ @@ -81,18 +81,54 @@ python scripts/run.py notebook_manager.py add \ ```bash # Check library -python scripts/run.py notebook_manager.py list +python3 scripts/run.py notebook_manager.py list -# Research with comprehensive questions -python scripts/run.py ask_question.py \ +# Research with comprehensive questions (auto source filtering) +python3 scripts/run.py ask_question.py \ --question "Detailed question with all context" \ --notebook-id notebook-id # Follow-up when you see "Is that ALL you need to know?" -python scripts/run.py ask_question.py \ +python3 scripts/run.py ask_question.py \ --question "Follow-up question with previous context" ``` +## Pattern 3.5: Source Filtering Options + +```bash +# Default: LLM-based semantic scoring (Gemini → Claude → keywords) +python3 scripts/run.py ask_question.py \ + --question "Specific topic question" \ + --notebook-id notebook-id +# → LLM scores each source 1-10, deselects irrelevant ones + +# Higher threshold for focused search +python3 scripts/run.py ask_question.py \ + --question "Very specific question" \ + --threshold 7 # Only sources scoring 7+ included + +# Lower threshold for broad search +python3 scripts/run.py ask_question.py \ + --question "General overview question" \ + --threshold 3 # Include more sources + +# Keyword matching (faster, less accurate) +python3 scripts/run.py ask_question.py \ + --question "Question with specific keywords" \ + --keyword-filter + +# Disable filtering entirely +python3 scripts/run.py ask_question.py \ + --question "Use all sources" \ + --no-filter + +# Update source summaries (after adding new documents) +python3 scripts/run.py notebook_manager.py update-sources --id notebook-id + +# Force re-extract all summaries +python3 scripts/run.py notebook_manager.py update-sources --id notebook-id --force +``` + ## Pattern 4: Follow-Up Questions (CRITICAL!) When NotebookLM responds with "EXTREMELY IMPORTANT: Is that ALL you need to know?": @@ -101,7 +137,7 @@ When NotebookLM responds with "EXTREMELY IMPORTANT: Is that ALL you need to know # 1. STOP - Don't respond to user yet # 2. ANALYZE - Is answer complete? # 3. If gaps exist, ask follow-up: -python scripts/run.py ask_question.py \ +python3 scripts/run.py ask_question.py \ --question "Specific follow-up with context from previous answer" # 4. Repeat until complete @@ -112,11 +148,11 @@ python scripts/run.py ask_question.py \ ```python # Query different notebooks for comparison -python scripts/run.py notebook_manager.py activate --id notebook-1 -python scripts/run.py ask_question.py --question "Question" +python3 scripts/run.py notebook_manager.py activate --id notebook-1 +python3 scripts/run.py ask_question.py --question "Question" -python scripts/run.py notebook_manager.py activate --id notebook-2 -python scripts/run.py ask_question.py --question "Same question" +python3 scripts/run.py notebook_manager.py activate --id notebook-2 +python3 scripts/run.py ask_question.py --question "Same question" # Compare and synthesize answers ``` @@ -125,16 +161,16 @@ python scripts/run.py ask_question.py --question "Same question" ```bash # If authentication fails -python scripts/run.py auth_manager.py status -python scripts/run.py auth_manager.py reauth # Browser visible! +python3 scripts/run.py auth_manager.py status +python3 scripts/run.py auth_manager.py reauth # Browser visible! # If browser crashes -python scripts/run.py cleanup_manager.py --preserve-library -python scripts/run.py auth_manager.py setup # Browser visible! +python3 scripts/run.py cleanup_manager.py --preserve-library +python3 scripts/run.py auth_manager.py setup # Browser visible! # If rate limited # Wait or switch accounts -python scripts/run.py auth_manager.py reauth # Login with different account +python3 scripts/run.py auth_manager.py reauth # Login with different account ``` ## Pattern 7: Batch Processing @@ -150,7 +186,7 @@ QUESTIONS=( for question in "${QUESTIONS[@]}"; do echo "Asking: $question" - python scripts/run.py ask_question.py \ + python3 scripts/run.py ask_question.py \ --question "$question" \ --notebook-id "$NOTEBOOK_ID" sleep 2 # Avoid rate limits @@ -198,8 +234,8 @@ add_notebook("React Docs", "React framework documentation", "react,frontend") add_notebook("CSS Framework", "Styling documentation", "css,styling,frontend") # Search by domain -python scripts/run.py notebook_manager.py search --query "backend" -python scripts/run.py notebook_manager.py search --query "frontend" +python3 scripts/run.py notebook_manager.py search --query "backend" +python3 scripts/run.py notebook_manager.py search --query "frontend" ``` ## Pattern 10: Integration with Development @@ -324,7 +360,7 @@ for q in questions: ```bash # Always use run.py! -python scripts/run.py [script].py [args] +python3 scripts/run.py [script].py [args] # Common operations run.py auth_manager.py status # Check auth diff --git a/scripts/add_source.py b/scripts/add_source.py new file mode 100755 index 0000000..4a18d49 --- /dev/null +++ b/scripts/add_source.py @@ -0,0 +1,476 @@ +#!/usr/bin/env python3 +""" +Add sources to NotebookLM notebooks. +Supports: File upload (PDF, txt, md, docx), Website/YouTube URLs, Copied text. + +Each source addition opens a fresh browser session, adds the source, and closes. +""" + +import argparse +import sys +import time +import re +import traceback +from pathlib import Path + +from patchright.sync_api import sync_playwright + +sys.path.insert(0, str(Path(__file__).parent)) + +from auth_manager import AuthManager +from notebook_manager import NotebookLibrary +from config import ( + DATA_DIR, + ADD_SOURCE_BUTTON_SELECTORS, + UPLOAD_FILES_BUTTON_SELECTORS, + WEBSITES_BUTTON_SELECTORS, + COPIED_TEXT_BUTTON_SELECTORS, + URL_TEXTAREA_SELECTORS, + TEXT_TEXTAREA_SELECTORS, + INSERT_BUTTON_SELECTORS, +) +from browser_utils import BrowserFactory, StealthUtils + + +def find_and_click(page, selectors: list, description: str, timeout: int = 5000) -> bool: + """Find element using multiple selectors and click it.""" + for selector in selectors: + try: + element = page.wait_for_selector(selector, timeout=timeout, state="visible") + if element: + element.click() + print(f" Clicked {description}") + return True + except Exception: + continue + print(f" Could not find {description}") + return False + + +def find_element(page, selectors: list, timeout: int = 3000): + """Find element using multiple selectors.""" + for selector in selectors: + try: + element = page.wait_for_selector(selector, timeout=timeout, state="visible") + if element: + return element + except Exception: + continue + return None + + +def navigate_to_add_source_dialog(page, notebook_url: str) -> bool: + """Navigate to notebook and open the Add Sources dialog.""" + print(" Opening notebook...") + page.goto(notebook_url, wait_until="domcontentloaded") + page.wait_for_url(re.compile(r"^https://notebooklm\.google\.com/"), timeout=15000) + time.sleep(3) + + if "/notebook/" not in page.url: + print(f" Redirected away from notebook: {page.url}") + return False + + # Close any existing dialog that might be open (press Escape) + try: + page.keyboard.press("Escape") + time.sleep(0.5) + except Exception: + pass + + # Click Sources tab if visible + try: + sources_tab = page.query_selector('button:has-text("Sources"), [role="tab"]:has-text("Sources")') + if sources_tab and sources_tab.is_visible(): + sources_tab.click() + time.sleep(1) + except Exception: + pass + + # Click "Add sources" button + print(" Opening Add Sources dialog...") + find_and_click(page, ADD_SOURCE_BUTTON_SELECTORS, "Add sources button") + + # Wait and verify dialog opened by checking for dialog buttons + time.sleep(1.5) + + # Check if dialog opened successfully + for verify_selector in UPLOAD_FILES_BUTTON_SELECTORS + WEBSITES_BUTTON_SELECTORS: + try: + dialog_btn = page.wait_for_selector(verify_selector, timeout=3000, state="visible") + if dialog_btn: + print(" Dialog opened successfully") + return True + except Exception: + continue + + # Dialog didn't open + debug_path = DATA_DIR / "add_source_debug.png" + page.screenshot(path=str(debug_path)) + print(f" Dialog did not open. Debug screenshot: {debug_path}") + return False + + +def add_file_source(notebook_url: str, file_path: str, headless: bool = True) -> str: + """ + Upload a file as a source using the native file picker. + + Args: + notebook_url: NotebookLM notebook URL + file_path: Path to file (PDF, txt, md, docx, etc.) + headless: Run browser in headless mode + + Returns: + Success message or None on failure + """ + auth = AuthManager() + if not auth.is_authenticated(): + print(" Not authenticated. Run: python scripts/run.py auth_manager.py setup") + return None + + file_path = Path(file_path).resolve() + if not file_path.exists(): + print(f" File not found: {file_path}") + return None + + print(f" Uploading file: {file_path.name}") + print(f" Size: {file_path.stat().st_size / 1024:.1f} KB") + print(f" Notebook: {notebook_url}") + + playwright = None + context = None + + try: + playwright = sync_playwright().start() + context = BrowserFactory.launch_persistent_context(playwright, headless=headless) + page = context.new_page() + + if not navigate_to_add_source_dialog(page, notebook_url): + return None + + # Click "Upload files" and handle file chooser + print(" Selecting file to upload...") + + with page.expect_file_chooser(timeout=10000) as fc_info: + if not find_and_click(page, UPLOAD_FILES_BUTTON_SELECTORS, "Upload files button"): + return None + + file_chooser = fc_info.value + file_chooser.set_files(str(file_path)) + print(f" File selected: {file_path.name}") + + # Wait for upload to process + print(" Waiting for upload to complete...") + time.sleep(5) + + # Check for success (source should appear in list) + # NotebookLM processes the file and adds it to sources + print(f" File source uploaded: {file_path.name}") + return f"Successfully uploaded file: {file_path.name}" + + except Exception as e: + print(f" Error: {e}") + traceback.print_exc() + return None + + finally: + if context: + try: + context.close() + except Exception: + pass + if playwright: + try: + playwright.stop() + except Exception: + pass + + +def add_url_source(notebook_url: str, source_url: str, headless: bool = True) -> str: + """ + Add a website or YouTube URL as a source. + + Args: + notebook_url: NotebookLM notebook URL + source_url: Website or YouTube URL to add + headless: Run browser in headless mode + + Returns: + Success message or None on failure + """ + auth = AuthManager() + if not auth.is_authenticated(): + print(" Not authenticated. Run: python scripts/run.py auth_manager.py setup") + return None + + if not source_url or not source_url.strip(): + print(" URL cannot be empty") + return None + + is_youtube = "youtube.com" in source_url or "youtu.be" in source_url + source_type = "YouTube" if is_youtube else "Website" + + print(f" Adding {source_type} source: {source_url}") + print(f" Notebook: {notebook_url}") + + playwright = None + context = None + + try: + playwright = sync_playwright().start() + context = BrowserFactory.launch_persistent_context(playwright, headless=headless) + page = context.new_page() + + if not navigate_to_add_source_dialog(page, notebook_url): + return None + + # Click "Websites" button + print(" Selecting Websites option...") + if not find_and_click(page, WEBSITES_BUTTON_SELECTORS, "Websites button"): + return None + + time.sleep(1) + + # Find textarea and enter URL + print(" Entering URL...") + textarea = find_element(page, URL_TEXTAREA_SELECTORS) + if not textarea: + print(" Could not find URL textarea") + return None + + textarea.click() + StealthUtils.random_delay(200, 400) + textarea.fill(source_url) + time.sleep(0.5) + + # Click Insert button + print(" Submitting...") + if not find_and_click(page, INSERT_BUTTON_SELECTORS, "Insert button", timeout=3000): + page.keyboard.press("Enter") + + time.sleep(4) + + print(f" {source_type} source added!") + return f"Successfully added {source_type} source: {source_url}" + + except Exception as e: + print(f" Error: {e}") + traceback.print_exc() + return None + + finally: + if context: + try: + context.close() + except Exception: + pass + if playwright: + try: + playwright.stop() + except Exception: + pass + + +def add_text_source(notebook_url: str, content: str, headless: bool = True) -> str: + """ + Add copied text as a source. + + Args: + notebook_url: NotebookLM notebook URL + content: Text content to add + headless: Run browser in headless mode + + Returns: + Success message or None on failure + """ + auth = AuthManager() + if not auth.is_authenticated(): + print(" Not authenticated. Run: python scripts/run.py auth_manager.py setup") + return None + + if not content or not content.strip(): + print(" Content cannot be empty") + return None + + preview = content[:100] + "..." if len(content) > 100 else content + print(f" Adding text source ({len(content)} chars)") + print(f" Preview: {preview}") + print(f" Notebook: {notebook_url}") + + playwright = None + context = None + + try: + playwright = sync_playwright().start() + context = BrowserFactory.launch_persistent_context(playwright, headless=headless) + page = context.new_page() + + if not navigate_to_add_source_dialog(page, notebook_url): + return None + + # Click "Copied text" button + print(" Selecting Copied text option...") + if not find_and_click(page, COPIED_TEXT_BUTTON_SELECTORS, "Copied text button"): + return None + + time.sleep(1) + + # Find textarea and enter content + print(" Entering text content...") + textarea = find_element(page, TEXT_TEXTAREA_SELECTORS) + if not textarea: + print(" Could not find text textarea") + return None + + textarea.click() + StealthUtils.random_delay(200, 400) + + # Use fill() for large content + if len(content) > 500: + textarea.fill(content) + else: + StealthUtils.human_type(page, TEXT_TEXTAREA_SELECTORS[0], content) + + time.sleep(0.5) + + # Click Insert button + print(" Submitting...") + time.sleep(1) # Wait for button to enable + + if not find_and_click(page, INSERT_BUTTON_SELECTORS, "Insert button", timeout=3000): + page.keyboard.press("Enter") + + time.sleep(3) + + print(" Text source added!") + return f"Successfully added text source ({len(content)} characters)" + + except Exception as e: + print(f" Error: {e}") + traceback.print_exc() + return None + + finally: + if context: + try: + context.close() + except Exception: + pass + if playwright: + try: + playwright.stop() + except Exception: + pass + + +def main(): + parser = argparse.ArgumentParser( + description='Add sources to NotebookLM notebooks', + formatter_class=argparse.RawDescriptionHelpFormatter, + epilog=""" +Examples: + # Upload a PDF file + python3 scripts/run.py add_source.py --file /path/to/document.pdf + + # Add a website URL + python3 scripts/run.py add_source.py --url "https://example.com/article" + + # Add a YouTube video + python3 scripts/run.py add_source.py --url "https://youtube.com/watch?v=..." + + # Add copied text + python3 scripts/run.py add_source.py --text "Your text content here" + + # Read text from a file and paste it (for very large text) + python3 scripts/run.py add_source.py --text-file /path/to/content.txt + """ + ) + + parser.add_argument('--notebook-url', help='NotebookLM notebook URL') + parser.add_argument('--notebook-id', help='Notebook ID from library') + + # Source types (mutually exclusive) + source_group = parser.add_mutually_exclusive_group(required=True) + source_group.add_argument('--file', help='Path to file to upload (PDF, txt, md, docx, etc.)') + source_group.add_argument('--url', help='Website or YouTube URL to add') + source_group.add_argument('--text', help='Text content to paste as source') + source_group.add_argument('--text-file', help='Path to text file to read and paste as source') + + parser.add_argument('--show-browser', action='store_true', help='Show browser window') + + args = parser.parse_args() + + # Resolve notebook URL + notebook_url = args.notebook_url + + if not notebook_url and args.notebook_id: + library = NotebookLibrary() + notebook = library.get_notebook(args.notebook_id) + if notebook: + notebook_url = notebook['url'] + print(f" Using notebook: {notebook['name']}") + else: + print(f" Notebook '{args.notebook_id}' not found") + return 1 + + if not notebook_url: + library = NotebookLibrary() + active = library.get_active_notebook() + if active: + notebook_url = active['url'] + print(f" Using active notebook: {active['name']}") + else: + print(" No notebook specified. Use --notebook-url or --notebook-id") + return 1 + + # Execute based on source type + result = None + + if args.file: + result = add_file_source( + notebook_url=notebook_url, + file_path=args.file, + headless=not args.show_browser + ) + + elif args.url: + result = add_url_source( + notebook_url=notebook_url, + source_url=args.url, + headless=not args.show_browser + ) + + elif args.text: + result = add_text_source( + notebook_url=notebook_url, + content=args.text, + headless=not args.show_browser + ) + + elif args.text_file: + text_path = Path(args.text_file) + if not text_path.exists(): + print(f" File not found: {args.text_file}") + return 1 + + print(f" Reading text from: {args.text_file}") + content = text_path.read_text(encoding='utf-8') + print(f" Content size: {len(content)} characters") + + result = add_text_source( + notebook_url=notebook_url, + content=content, + headless=not args.show_browser + ) + + # Print result + print("\n" + "=" * 50) + if result: + print(f" {result}") + return 0 + else: + print(" Failed to add source") + return 1 + + +if __name__ == "__main__": + sys.exit(main()) diff --git a/scripts/ask_question.py b/scripts/ask_question.py index aa47e4b..5a0c7ba 100755 --- a/scripts/ask_question.py +++ b/scripts/ask_question.py @@ -13,6 +13,7 @@ import sys import time import re +import traceback from pathlib import Path from patchright.sync_api import sync_playwright @@ -24,6 +25,7 @@ from notebook_manager import NotebookLibrary from config import QUERY_INPUT_SELECTORS, RESPONSE_SELECTORS from browser_utils import BrowserFactory, StealthUtils +from source_filter import SourceFilter, select_sources_in_browser # Follow-up reminder (adapted from MCP server for stateless operation) @@ -37,14 +39,20 @@ ) -def ask_notebooklm(question: str, notebook_url: str, headless: bool = True) -> str: +def ask_notebooklm(question: str, notebook_url: str, notebook_id: str = None, + headless: bool = True, filter_sources: bool = True, + use_llm_filter: bool = True, relevance_threshold: int = 5) -> str: """ Ask a question to NotebookLM Args: question: Question to ask notebook_url: NotebookLM notebook URL + notebook_id: Notebook ID (for source filtering) headless: Run browser in headless mode + filter_sources: Whether to filter sources based on question relevance + use_llm_filter: Use Gemini for semantic relevance scoring (default True) + relevance_threshold: Minimum score (1-10) for LLM filter (default 5) Returns: Answer text from NotebookLM @@ -55,6 +63,21 @@ def ask_notebooklm(question: str, notebook_url: str, headless: bool = True) -> s print("⚠️ Not authenticated. Run: python auth_manager.py setup") return None + # Load source filter if enabled and notebook_id provided + source_filter = None + relevant_sources = None + all_sources = None + + if filter_sources and notebook_id: + source_filter = SourceFilter(notebook_id) + if source_filter.sources: + relevant_sources = source_filter.get_relevant_sources( + question, + use_llm=use_llm_filter, + threshold=relevance_threshold + ) + all_sources = source_filter.get_all_source_titles() + print(f"💬 Asking: {question}") print(f"📚 Notebook: {notebook_url}") @@ -93,13 +116,27 @@ def ask_notebooklm(question: str, notebook_url: str, headless: bool = True) -> s if query_element: print(f" ✓ Found input: {selector}") break - except: + except Exception: continue if not query_element: print(" ❌ Could not find query input") return None + # Filter sources if enabled + if relevant_sources and all_sources and len(relevant_sources) < len(all_sources): + print(f" 🔧 Filtering to {len(relevant_sources)}/{len(all_sources)} relevant sources...") + + # Wait for source containers to load (they may load after query input) + try: + page.wait_for_selector('.single-source-container', timeout=5000, state="attached") + StealthUtils.random_delay(500, 800) # Small extra wait for all sources + except Exception: + print(" ⚠️ Source containers not found, skipping filter") + + select_sources_in_browser(page, relevant_sources, all_sources) + StealthUtils.random_delay(500, 1000) + # Type question (human-like, fast) print(" ⏳ Typing question...") @@ -129,7 +166,7 @@ def ask_notebooklm(question: str, notebook_url: str, headless: bool = True) -> s if thinking_element and thinking_element.is_visible(): time.sleep(1) continue - except: + except Exception: pass # Try to find response with MCP selectors @@ -150,7 +187,7 @@ def ask_notebooklm(question: str, notebook_url: str, headless: bool = True) -> s else: stable_count = 0 last_text = text - except: + except Exception: continue if answer: @@ -168,7 +205,6 @@ def ask_notebooklm(question: str, notebook_url: str, headless: bool = True) -> s except Exception as e: print(f" ❌ Error: {e}") - import traceback traceback.print_exc() return None @@ -177,13 +213,13 @@ def ask_notebooklm(question: str, notebook_url: str, headless: bool = True) -> s if context: try: context.close() - except: + except Exception: pass if playwright: try: playwright.stop() - except: + except Exception: pass @@ -194,17 +230,24 @@ def main(): parser.add_argument('--notebook-url', help='NotebookLM notebook URL') parser.add_argument('--notebook-id', help='Notebook ID from library') parser.add_argument('--show-browser', action='store_true', help='Show browser') + parser.add_argument('--no-filter', action='store_true', help='Disable source filtering') + parser.add_argument('--keyword-filter', action='store_true', + help='Use keyword matching instead of LLM for source filtering') + parser.add_argument('--threshold', type=int, default=5, + help='Relevance threshold for LLM filter (1-10, default: 5)') args = parser.parse_args() - # Resolve notebook URL + # Resolve notebook URL and ID notebook_url = args.notebook_url + notebook_id = args.notebook_id if not notebook_url and args.notebook_id: library = NotebookLibrary() notebook = library.get_notebook(args.notebook_id) if notebook: notebook_url = notebook['url'] + notebook_id = notebook['id'] else: print(f"❌ Notebook '{args.notebook_id}' not found") return 1 @@ -215,6 +258,7 @@ def main(): active = library.get_active_notebook() if active: notebook_url = active['url'] + notebook_id = active['id'] print(f"📚 Using active notebook: {active['name']}") else: # Show available notebooks @@ -225,17 +269,21 @@ def main(): mark = " [ACTIVE]" if nb.get('id') == library.active_notebook_id else "" print(f" {nb['id']}: {nb['name']}{mark}") print("\nSpecify with --notebook-id or set active:") - print("python scripts/run.py notebook_manager.py activate --id ID") + print("python3 scripts/run.py notebook_manager.py activate --id ID") else: print("❌ No notebooks in library. Add one first:") - print("python scripts/run.py notebook_manager.py add --url URL --name NAME --description DESC --topics TOPICS") + print("python3 scripts/run.py notebook_manager.py add --url URL --name NAME --description DESC --topics TOPICS") return 1 # Ask the question answer = ask_notebooklm( question=args.question, notebook_url=notebook_url, - headless=not args.show_browser + notebook_id=notebook_id, + headless=not args.show_browser, + filter_sources=not args.no_filter, + use_llm_filter=not args.keyword_filter, + relevance_threshold=args.threshold ) if answer: diff --git a/scripts/audio_generator.py b/scripts/audio_generator.py new file mode 100644 index 0000000..f253f05 --- /dev/null +++ b/scripts/audio_generator.py @@ -0,0 +1,614 @@ +#!/usr/bin/env python3 +""" +Audio Generator for NotebookLM +Generates Audio Overview with custom prompts and downloads the audio file +""" + +import argparse +import sys +import re +import time +import traceback +from pathlib import Path + +from patchright.sync_api import sync_playwright + +# Add parent directory to path +sys.path.insert(0, str(Path(__file__).parent)) + +from auth_manager import AuthManager +from notebook_manager import NotebookLibrary +from config import ( + AUDIO_OVERVIEW_CARD_SELECTORS, + AUDIO_CUSTOMIZE_BUTTON_SELECTORS, + AUDIO_FORMAT_SELECTORS, + AUDIO_LENGTH_SELECTORS, + AUDIO_INSTRUCTIONS_SELECTORS, + AUDIO_GENERATE_BUTTON_SELECTORS, + AUDIO_GENERATING_SELECTORS, + AUDIO_GENERATION_TIMEOUT, + PAGE_LOAD_TIMEOUT, +) +from browser_utils import ( + BrowserFactory, + StealthUtils, + find_element_with_selectors, + select_language, + get_all_sources_from_ui, +) +from source_filter import select_sources_in_browser + + +def generate_audio( + notebook_url: str, + format: str = "deep_dive", + length: str = "default", + language: str = None, + instructions: str = None, + sources: list = None, + output: str = None, + headless: bool = True +) -> str: + """ + Generate Audio Overview for a NotebookLM notebook and download it. + + Args: + notebook_url: The NotebookLM notebook URL + format: Audio format - deep_dive, brief, critique, or debate + length: Audio length - short, default, or long + language: Language for the audio (e.g., "English", "Spanish", "Japanese") + instructions: Custom instructions for AI hosts (the prompt) + sources: List of source names to include (deselects others). If None, uses all sources. + output: Custom output filename + headless: Run browser in headless mode + + Returns: + Path to downloaded audio file, or None if failed + """ + auth = AuthManager() + + if not auth.is_authenticated(): + print(" Not authenticated. Run: python scripts/run.py auth_manager.py setup") + return None + + print(" Generating Audio Overview...") + print(f" Notebook: {notebook_url}") + print(f" Format: {format}") + print(f" Length: {length}") + if language: + print(f" Language: {language}") + if sources: + print(f" Sources: {len(sources)} selected") + for s in sources[:3]: + print(f" - {s[:50]}...") + if len(sources) > 3: + print(f" ... and {len(sources) - 3} more") + if instructions: + print(f" Instructions: {instructions[:50]}...") + + playwright = None + context = None + download_dir = os.getcwd() + + try: + playwright = sync_playwright().start() + + context = BrowserFactory.launch_persistent_context( + playwright, + headless=headless + ) + + page = context.new_page() + print(" Opening notebook...") + page.goto(notebook_url, wait_until="domcontentloaded") + + # Wait for notebook to load + page.wait_for_url(re.compile(r"^https://notebooklm\.google\.com/"), timeout=60000) + StealthUtils.random_delay(2000, 3000) + + # Handle source selection if specified + if sources: + print(" Selecting specific sources...") + StealthUtils.random_delay(1000, 1500) + + # Get all sources from UI + all_sources = get_all_sources_from_ui(page) + print(f" Found {len(all_sources)} sources in notebook") + + if all_sources: + # Find matching sources (partial match supported) + sources_to_keep = [] + for source_name in sources: + for ui_source in all_sources: + if source_name.lower() in ui_source.lower() or ui_source.lower() in source_name.lower(): + sources_to_keep.append(ui_source) + break + + if sources_to_keep: + print(f" Keeping {len(sources_to_keep)} sources, deselecting {len(all_sources) - len(sources_to_keep)}") + select_sources_in_browser(page, sources_to_keep, all_sources) + StealthUtils.random_delay(500, 1000) + else: + print(" Warning: No matching sources found, using all sources") + + # First, check if we need to click on "Studio" tab (responsive layout) + studio_tab = page.query_selector('button:has-text("Studio"), [role="tab"]:has-text("Studio")') + if studio_tab: + print(" Found Studio tab, clicking...") + studio_tab.click() + StealthUtils.random_delay(1000, 1500) + + # Find the Audio Overview card in Studio panel + print(" Looking for Audio Overview in Studio panel...") + + # Click on the pencil/customize icon next to Audio Overview + customize_button, selector = find_element_with_selectors( + page, AUDIO_CUSTOMIZE_BUTTON_SELECTORS, timeout=10000 + ) + + if not customize_button: + # Try clicking on Audio Overview card directly + audio_card, card_selector = find_element_with_selectors( + page, AUDIO_OVERVIEW_CARD_SELECTORS, timeout=5000 + ) + if audio_card: + print(f" Found Audio Overview card: {card_selector}") + # Look for pencil icon within or near the card + pencil = page.query_selector(f'{card_selector} button, {card_selector} + button') + if pencil: + pencil.click() + StealthUtils.random_delay(1000, 1500) + else: + # Click the card itself + audio_card.click() + StealthUtils.random_delay(1000, 1500) + else: + print(" Could not find Audio Overview option") + return None + else: + print(f" Found customize button: {selector}") + customize_button.click() + StealthUtils.random_delay(1000, 1500) + + # Wait for customize dialog to appear + print(" Waiting for Customize Audio Overview dialog...") + StealthUtils.random_delay(1000, 2000) + + # Select format if specified + if format: + print(f" Selecting format: {format}") + format_names = { + 'deep_dive': 'Deep Dive', + 'brief': 'Brief', + 'critique': 'Critique', + 'debate': 'Debate' + } + format_name = format_names.get(format, format) + + # Try using Playwright's text locator first + format_selected = False + try: + # Look for a div that starts with the format name (the card) + format_card = page.locator(f'div:has(div:text-is("{format_name}"))').first + if format_card.is_visible(timeout=2000): + format_card.click() + format_selected = True + except Exception: + pass + + if not format_selected: + # Fallback: use getByText + try: + format_el = page.get_by_text(format_name, exact=True).first + if format_el.is_visible(timeout=2000): + # Click the parent card + format_el.locator('..').locator('..').click() + format_selected = True + except Exception: + pass + + if format_selected: + StealthUtils.random_delay(500, 800) + print(f" Selected: {format_name}") + else: + print(f" Warning: Could not find format card for {format_name}") + # Take a screenshot for debugging + page.screenshot(path="/tmp/notebooklm_format_debug.png") + print(f" Debug screenshot: /tmp/notebooklm_format_debug.png") + + # Select length if specified + if length and length in AUDIO_LENGTH_SELECTORS: + print(f" Selecting length: {length}") + length_selectors = AUDIO_LENGTH_SELECTORS[length] + length_element, len_selector = find_element_with_selectors(page, length_selectors, timeout=3000) + if length_element: + length_element.click() + StealthUtils.random_delay(500, 800) + print(f" Selected: {len_selector}") + + # Select language if specified + if language: + select_language(page, language) + + # Add custom instructions if provided + if instructions: + print(" Adding custom instructions...") + # Re-find the instructions input (DOM may have changed after format/language selection) + try: + # Try using Playwright's locator for textarea + textarea = page.locator('textarea').first + if textarea.is_visible(timeout=3000): + textarea.click() + StealthUtils.random_delay(200, 400) + textarea.fill(instructions) + StealthUtils.random_delay(500, 800) + print(" Instructions added") + else: + raise Exception("Textarea not visible") + except Exception as e: + # Fallback to selector-based approach + instructions_input, instr_selector = find_element_with_selectors( + page, AUDIO_INSTRUCTIONS_SELECTORS, timeout=3000 + ) + if instructions_input: + instructions_input.click() + StealthUtils.random_delay(200, 400) + instructions_input.fill(instructions) + StealthUtils.random_delay(500, 800) + print(" Instructions added") + else: + print(f" Could not find instructions field: {e}") + + # Click Generate button + print(" Looking for Generate button...") + generate_button, gen_selector = find_element_with_selectors( + page, AUDIO_GENERATE_BUTTON_SELECTORS, timeout=5000 + ) + + if not generate_button: + # Try more specific selectors + generate_button = page.query_selector('button.generate-button, button[type="submit"], button:has-text("Generate")') + + if not generate_button: + print(" Could not find Generate button") + page.screenshot(path="/tmp/notebooklm_audio_debug.png") + print(" Debug screenshot saved to /tmp/notebooklm_audio_debug.png") + return None + + print(f" Found Generate button") + + # Wait for button to be enabled (it may be disabled initially) + print(" Waiting for Generate button to be enabled...") + try: + page.wait_for_selector('button:has-text("Generate"):not([disabled])', timeout=10000) + except Exception: + print(" Generate button seems disabled, trying to click anyway...") + + StealthUtils.random_delay(500, 1000) + + # Scroll into view and wait for stability + generate_button.scroll_into_view_if_needed() + StealthUtils.random_delay(300, 500) + + try: + generate_button.click(timeout=10000) + except Exception as click_error: + print(f" Click failed: {click_error}") + # Try JavaScript click as fallback + try: + page.evaluate('(el) => el.click()', generate_button) + print(" Used JavaScript click fallback") + except Exception as js_error: + print(f" JavaScript click also failed: {js_error}") + page.screenshot(path="/tmp/notebooklm_generate_debug.png") + print(" Debug screenshot saved to /tmp/notebooklm_generate_debug.png") + return None + + print(" Started audio generation...") + + # Wait for generation to complete (can take 5-10 minutes) + print(" Waiting for audio generation (this may take several minutes)...") + generation_start = time.time() + generation_timeout = AUDIO_GENERATION_TIMEOUT / 1000 # Convert to seconds + + # First, wait a few seconds for generation indicator to appear + StealthUtils.random_delay(3000, 5000) + + last_print_time = 0 + while time.time() - generation_start < generation_timeout: + # Check if still generating by looking for the generation indicator + generating = False + + for selector in AUDIO_GENERATING_SELECTORS: + try: + gen_indicator = page.query_selector(selector) + if gen_indicator and gen_indicator.is_visible(): + generating = True + break + except Exception: + continue + + if generating: + elapsed = int(time.time() - generation_start) + # Print status every 30 seconds + if elapsed - last_print_time >= 30: + print(f" Still generating... ({elapsed}s)") + last_print_time = elapsed + time.sleep(5) + continue + else: + # Generation indicator disappeared - check if new audio appeared + print(" Generation indicator disappeared, verifying completion...") + StealthUtils.random_delay(2000, 3000) + + # Double-check that generation is really complete + still_generating = False + for selector in AUDIO_GENERATING_SELECTORS: + try: + gen_indicator = page.query_selector(selector) + if gen_indicator and gen_indicator.is_visible(): + still_generating = True + break + except Exception: + continue + + if not still_generating: + print(" Audio generation complete!") + break + else: + # False alarm, continue waiting + time.sleep(5) + continue + + else: + print(" Generation timeout exceeded") + return None + + # Allow UI to settle after generation + print(" Waiting for UI to settle...") + StealthUtils.random_delay(3000, 5000) + + # Find and click the three-dot menu on the most recently generated audio + print(" Looking for audio menu...") + + # The generated audio items are in the Studio panel (right side) + # Each item has: icon, title/metadata, play button (optional), three-dot menu button + # We need to find the most recent one (usually shows "Xm ago" or "just now") + + # Use JavaScript to find the correct menu button + menu_clicked = page.evaluate(""" + () => { + // Find the Studio panel - it's on the right side of the page + const studioPanelSelectors = [ + '[class*="studio"]', + '[aria-label*="Studio"]', + 'div:has(> div:has-text("Audio"))' + ]; + + // Find all audio/video items - they typically have an icon and metadata like "X source · Xm ago" + const items = document.querySelectorAll('div'); + let targetButton = null; + let mostRecent = null; + + for (const item of items) { + const text = item.textContent || ''; + // Look for items that look like generated audio (have source count and time) + if ((text.includes('source') && (text.includes('ago') || text.includes('just now'))) || + (text.includes('Deep Dive') && text.includes('source')) || + (text.includes('Brief') && text.includes('source'))) { + + // Check if this item is in the right part of the page (Studio panel) + const rect = item.getBoundingClientRect(); + if (rect.x < 600) continue; // Studio panel is on the right + + // Find the three-dot menu button within this item + const menuButtons = item.querySelectorAll('button'); + for (const btn of menuButtons) { + const btnRect = btn.getBoundingClientRect(); + // Menu button is typically small and on the right side of the item + if (btnRect.width < 50 && btnRect.height < 50) { + const ariaLabel = btn.getAttribute('aria-label') || ''; + const innerHTML = btn.innerHTML || ''; + if (ariaLabel.includes('option') || ariaLabel.includes('more') || + innerHTML.includes('more_vert') || innerHTML.includes('⋮')) { + targetButton = btn; + break; + } + } + } + + if (targetButton) { + // Check if this is more recent than previous finds + if (text.includes('just now') || text.includes('1m ago') || text.includes('2m ago')) { + mostRecent = targetButton; + break; // Found a very recent one + } + if (!mostRecent) mostRecent = targetButton; + } + } + } + + if (mostRecent) { + mostRecent.click(); + return true; + } + + // Fallback: find any menu button in the Studio area + const allButtons = document.querySelectorAll('button'); + for (const btn of allButtons) { + const rect = btn.getBoundingClientRect(); + if (rect.x < 600) continue; // Must be in right panel + + const ariaLabel = btn.getAttribute('aria-label') || ''; + if (ariaLabel.toLowerCase().includes('option') || ariaLabel.toLowerCase().includes('more')) { + btn.click(); + return true; + } + } + + return false; + } + """) + + if not menu_clicked: + print(" Could not find audio menu button") + page.screenshot(path="/tmp/notebooklm_menu_debug.png") + print(" Debug screenshot saved to /tmp/notebooklm_menu_debug.png") + return None + + print(" Clicked menu button") + StealthUtils.random_delay(1000, 1500) + + # Click Download option from the menu + print(" Looking for Download option...") + + # Wait for download - set up expectation before clicking + print(" Waiting for download...") + + try: + with page.expect_download(timeout=60000) as download_info: + # Click Download in the menu + download_clicked = page.evaluate(""" + () => { + const menuItems = document.querySelectorAll('[role="menuitem"], li, button, span'); + for (const item of menuItems) { + const text = item.textContent?.trim() || ''; + if (text === 'Download') { + const rect = item.getBoundingClientRect(); + if (rect.width > 0 && rect.height > 0) { + item.click(); + return true; + } + } + } + return false; + } + """) + + if not download_clicked: + raise Exception("Could not find Download option") + + download = download_info.value + suggested_name = download.suggested_filename + print(f" Download started: {suggested_name}") + + if output: + download_path = f"{download_dir}/{output}" + else: + download_path = f"{download_dir}/{suggested_name}" + + download.save_as(download_path) + print(f" Saved to: {download_path}") + + except Exception as dl_error: + print(f" Download error: {dl_error}") + page.screenshot(path="/tmp/notebooklm_download_menu_debug.png") + print(" Debug screenshot saved to /tmp/notebooklm_download_menu_debug.png") + download_path = None + + if download_path: + print(f" Audio downloaded successfully: {download_path}") + return download_path + else: + print(" Download failed") + return None + + except Exception as e: + print(f" Error: {e}") + traceback.print_exc() + return None + + finally: + if context: + try: + context.close() + except Exception: + pass + if playwright: + try: + playwright.stop() + except Exception: + pass + + +def main(): + parser = argparse.ArgumentParser(description='Generate NotebookLM Audio Overview') + + # Notebook selection (mutually exclusive) + notebook_group = parser.add_mutually_exclusive_group() + notebook_group.add_argument('--notebook-url', help='NotebookLM notebook URL') + notebook_group.add_argument('--notebook-id', help='Notebook ID from library') + + # Audio customization + parser.add_argument('--format', choices=['deep_dive', 'brief', 'critique', 'debate'], + default='deep_dive', help='Audio format (default: deep_dive)') + parser.add_argument('--length', choices=['short', 'default', 'long'], + default='default', help='Audio length (default: default)') + parser.add_argument('--language', help='Language for the audio (e.g., English, Spanish, Japanese)') + parser.add_argument('--instructions', help='Custom instructions/prompt for AI hosts') + parser.add_argument('--sources', help='Comma-separated list of source names to include (deselects others)') + parser.add_argument('--output', help='Custom output filename') + parser.add_argument('--show-browser', action='store_true', help='Show browser window') + + args = parser.parse_args() + + # Resolve notebook URL + library = NotebookLibrary() + + if args.notebook_id: + notebook = library.get_notebook(args.notebook_id) + if not notebook: + print(f"Notebook not found: {args.notebook_id}") + return 1 + notebook_url = notebook['url'] + notebook_name = notebook['name'] + elif args.notebook_url: + notebook_url = args.notebook_url + notebook_name = "Unknown" + else: + # Use active notebook + active = library.get_active_notebook() + if active: + notebook_url = active['url'] + notebook_name = active['name'] + print(f"Using active notebook: {notebook_name}") + else: + print("No notebook specified and no active notebook set") + return 1 + + # Parse sources if provided + sources_list = None + if args.sources: + sources_list = [s.strip() for s in args.sources.split(',')] + + result = generate_audio( + notebook_url=notebook_url, + format=args.format, + length=args.length, + language=args.language, + instructions=args.instructions, + sources=sources_list, + output=args.output, + headless=not args.show_browser + ) + + if result: + print("\n" + "=" * 60) + print(" Audio Overview Generated") + print("=" * 60) + print(f"File: {result}") + print(f"Format: {args.format}") + print(f"Length: {args.length}") + if args.language: + print(f"Language: {args.language}") + if args.instructions: + print(f"Instructions: {args.instructions[:50]}...") + print("=" * 60) + return 0 + else: + print("\n Failed to generate audio") + return 1 + + +if __name__ == "__main__": + sys.exit(main()) diff --git a/scripts/browser_utils.py b/scripts/browser_utils.py index 60a1210..9c45559 100755 --- a/scripts/browser_utils.py +++ b/scripts/browser_utils.py @@ -6,10 +6,11 @@ import json import time import random -from typing import Optional, List +import traceback +from typing import Optional, List, Tuple, Any from patchright.sync_api import Playwright, BrowserContext, Page -from config import BROWSER_PROFILE_DIR, STATE_FILE, BROWSER_ARGS, USER_AGENT +from config import BROWSER_PROFILE_DIR, STATE_FILE, BROWSER_ARGS, USER_AGENT, get_language_display_name class BrowserFactory: @@ -28,7 +29,7 @@ def launch_persistent_context( # Launch persistent context context = playwright.chromium.launch_persistent_context( user_data_dir=user_data_dir, - channel="chrome", # Use real Chrome + # channel="chrome", # Disabled - using Chromium instead (Chrome not installed) headless=headless, no_viewport=True, ignore_default_args=["--enable-automation"], @@ -72,7 +73,7 @@ def human_type(page: Page, selector: str, text: str, wpm_min: int = 320, wpm_max # Try waiting if not immediately found try: element = page.wait_for_selector(selector, timeout=2000) - except: + except Exception: pass if not element: @@ -105,3 +106,228 @@ def realistic_click(page: Page, selector: str): StealthUtils.random_delay(100, 300) element.click() StealthUtils.random_delay(100, 300) + + +class DownloadHandler: + """Utilities for handling file downloads""" + + @staticmethod + def wait_for_download( + page: Page, + trigger_action, + download_dir: str, + timeout: int = 60000 + ) -> Optional[str]: + """ + Wait for a download to complete after triggering an action. + Returns the path to the downloaded file. + """ + try: + with page.expect_download(timeout=timeout) as download_info: + trigger_action() + + download = download_info.value + suggested_filename = download.suggested_filename + download_path = f"{download_dir}/{suggested_filename}" + + download.save_as(download_path) + return download_path + + except Exception as e: + print(f" Download failed: {e}") + return None + + @staticmethod + def download_with_custom_name( + page: Page, + trigger_action, + download_dir: str, + custom_filename: str, + timeout: int = 60000 + ) -> Optional[str]: + """ + Wait for a download and save with a custom filename. + Returns the path to the downloaded file. + """ + try: + with page.expect_download(timeout=timeout) as download_info: + trigger_action() + + download = download_info.value + + # Preserve extension from suggested filename if custom doesn't have one + suggested = download.suggested_filename + if '.' not in custom_filename and '.' in suggested: + ext = suggested.rsplit('.', 1)[-1] + custom_filename = f"{custom_filename}.{ext}" + + download_path = f"{download_dir}/{custom_filename}" + download.save_as(download_path) + return download_path + + except Exception as e: + print(f" Download failed: {e}") + return None + + +def find_element_with_selectors(page: Page, selectors: list, timeout: int = 5000) -> Tuple[Optional[Any], Optional[str]]: + """ + Try multiple selectors and return first visible match. + + Args: + page: Playwright page object + selectors: List of CSS selectors to try + timeout: Timeout in milliseconds for each selector + + Returns: + Tuple of (element, matched_selector) or (None, None) if not found + """ + for selector in selectors: + try: + element = page.wait_for_selector(selector, timeout=timeout, state="visible") + if element: + return element, selector + except Exception: + continue + return None, None + + +def select_language(page: Page, language: str) -> bool: + """ + Select a language from the NotebookLM dropdown menu. + + Args: + page: Playwright page object + language: Language name in English (e.g., "Japanese") or native script (e.g., "日本語") + + Returns: + True if language was selected, False otherwise + """ + display_name = get_language_display_name(language) + print(f" Selecting language: {language} → {display_name}") + + try: + # Step 1: Find and click the language dropdown + dropdown_opened = False + + # Try clicking on the element that shows current language selection + try: + lang_dropdown = page.locator('mat-select:has-text("English"), div[role="combobox"]:has-text("English"), div[role="listbox"]:has-text("English")').first + if lang_dropdown.is_visible(timeout=2000): + lang_dropdown.click() + dropdown_opened = True + except Exception: + pass + + if not dropdown_opened: + # Try finding by structure - look for the language section + try: + lang_section = page.locator('div:has-text("Choose language")').first + if lang_section.is_visible(timeout=1000): + dropdown = lang_section.locator('mat-select, [role="combobox"], div:has-text("English")').first + if dropdown.is_visible(timeout=1000): + dropdown.click() + dropdown_opened = True + except Exception: + pass + + if not dropdown_opened: + print(" Could not find language dropdown") + page.screenshot(path="/tmp/notebooklm_language_debug.png") + return False + + StealthUtils.random_delay(500, 800) + + # Step 2: Find and click the language option by its display name + try: + lang_option = page.locator(f'div:text-is("{display_name}"), span:text-is("{display_name}")').first + if lang_option.is_visible(timeout=3000): + lang_option.scroll_into_view_if_needed() + lang_option.click() + StealthUtils.random_delay(500, 800) + print(f" Selected: {display_name}") + return True + except Exception: + pass + + # Fallback: try get_by_text with exact match + try: + lang_option = page.get_by_text(display_name, exact=True).first + if lang_option.is_visible(timeout=2000): + lang_option.scroll_into_view_if_needed() + lang_option.click() + StealthUtils.random_delay(500, 800) + print(f" Selected: {display_name}") + return True + except Exception: + pass + + # Another fallback: use role="option" selector + try: + lang_option = page.locator(f'[role="option"]:has-text("{display_name}")').first + if lang_option.is_visible(timeout=2000): + lang_option.scroll_into_view_if_needed() + lang_option.click() + StealthUtils.random_delay(500, 800) + print(f" Selected: {display_name}") + return True + except Exception: + pass + + print(f" Could not find language option: {display_name}") + page.screenshot(path="/tmp/notebooklm_language_options_debug.png") + page.keyboard.press("Escape") + return False + + except Exception as e: + print(f" Language selection error: {e}") + return False + + +def get_all_sources_from_ui(page: Page) -> List[str]: + """ + Get all source titles from the NotebookLM UI. + + Args: + page: Playwright page object + + Returns: + List of source titles + """ + sources = page.evaluate(""" + () => { + const sources = []; + const checkboxes = document.querySelectorAll('input[type="checkbox"], [role="checkbox"], mat-checkbox, .mat-checkbox'); + + for (const checkbox of checkboxes) { + let parent = checkbox.parentElement; + for (let i = 0; i < 5 && parent; i++) { + const text = parent.innerText?.trim(); + if (text && text.length > 5 && text.length < 300) { + const lines = text.split('\\n').filter(l => l.trim()); + for (const line of lines) { + const cleaned = line.trim(); + if (cleaned.length < 5) continue; + if (['check_box', 'more_vert', 'description', 'drive_pdf', + 'markdown', 'web', 'Select all'].some(p => cleaned.includes(p))) continue; + + const hasExtension = /\\.(pdf|md|txt|docx)$/i.test(cleaned); + const isWikipedia = cleaned.includes('Wikipedia'); + const hasCJK = /[\\u4e00-\\u9fff]/.test(cleaned); + const hasPattern = /--/.test(cleaned) || /_/.test(cleaned); + + if ((hasExtension || isWikipedia || hasCJK || hasPattern) && + !sources.includes(cleaned)) { + sources.push(cleaned); + break; + } + } + break; + } + parent = parent.parentElement; + } + } + return sources; + } + """) + return sources diff --git a/scripts/config.py b/scripts/config.py index 4486b55..44618e2 100755 --- a/scripts/config.py +++ b/scripts/config.py @@ -13,6 +13,18 @@ STATE_FILE = BROWSER_STATE_DIR / "state.json" AUTH_INFO_FILE = DATA_DIR / "auth_info.json" LIBRARY_FILE = DATA_DIR / "library.json" +SOURCE_SUMMARY_DIR = DATA_DIR / "library-source-summary" + +# Source Summary Query +SOURCE_SUMMARY_QUERY = """List ALL the sources/documents in this notebook. For each source, provide: +1. The exact source name/title as shown in NotebookLM +2. A brief 1-2 sentence summary of what that source contains + +Format your response as a markdown table with these columns: +| Source Name | Summary | +|-------------|---------| + +Include EVERY source in this notebook, even if there are many.""" # NotebookLM Selectors QUERY_INPUT_SELECTORS = [ @@ -27,6 +39,61 @@ "[data-message-author='assistant']", ] +# Add Source Dialog Selectors (based on NotebookLM UI Jan 2026) +ADD_SOURCE_BUTTON_SELECTORS = [ + 'button:has-text("Add source")', + 'button:has-text("Add sources")', + 'button[aria-label*="Add source"]', + 'button[aria-label*="Add sources"]', + '[data-test-id="add-source-button"]', +] + +# Main dialog option buttons +UPLOAD_FILES_BUTTON_SELECTORS = [ + 'button:has-text("Upload files")', + '[aria-label*="Upload files"]', + 'button:has-text("Upload")', +] + +WEBSITES_BUTTON_SELECTORS = [ + 'button:has-text("Websites")', + '[aria-label*="Websites"]', + 'button:has-text("Website")', +] + +COPIED_TEXT_BUTTON_SELECTORS = [ + 'button:has-text("Copied text")', + '[aria-label*="Copied text"]', + 'button:has-text("Paste")', +] + +# Input fields +URL_TEXTAREA_SELECTORS = [ + 'textarea[placeholder*="Paste any links"]', + 'textarea[placeholder*="link"]', + 'textarea[placeholder*="URL"]', + 'textarea', +] + +TEXT_TEXTAREA_SELECTORS = [ + 'textarea[placeholder*="Paste"]', + 'textarea[placeholder*="paste"]', + 'textarea', +] + +# Action buttons +INSERT_BUTTON_SELECTORS = [ + 'button:has-text("Insert")', + 'button:has-text("Add")', + 'button[type="submit"]', +] + +CLOSE_DIALOG_SELECTORS = [ + 'button[aria-label="Close"]', + 'button:has-text("Close")', + '[aria-label="close"]', +] + # Browser Configuration BROWSER_ARGS = [ '--disable-blink-features=AutomationControlled', # Patches navigator.webdriver @@ -42,3 +109,385 @@ LOGIN_TIMEOUT_MINUTES = 10 QUERY_TIMEOUT_SECONDS = 120 PAGE_LOAD_TIMEOUT = 30000 +AUDIO_GENERATION_TIMEOUT = 900000 # 15 minutes for audio generation + +# Audio Overview Selectors +AUDIO_OVERVIEW_CARD_SELECTORS = [ + 'div:has-text("Audio Overview")', + '[aria-label="Audio Overview"]', +] + +AUDIO_CUSTOMIZE_BUTTON_SELECTORS = [ + 'button[aria-label="Customize Audio Overview"]', + 'div:has-text("Audio Overview") button', +] + +AUDIO_FORMAT_SELECTORS = { + 'deep_dive': [ + 'div:has(> div:text-is("Deep Dive"))', + 'div.format-card:has-text("Deep Dive")', + '[data-format="deep_dive"]', + ], + 'brief': [ + 'div:has(> div:text-is("Brief"))', + 'div.format-card:has-text("Brief")', + '[data-format="brief"]', + ], + 'critique': [ + 'div:has(> div:text-is("Critique"))', + 'div.format-card:has-text("Critique")', + '[data-format="critique"]', + ], + 'debate': [ + 'div:has(> div:text-is("Debate"))', + 'div.format-card:has-text("Debate")', + '[data-format="debate"]', + ], +} + +AUDIO_LENGTH_SELECTORS = { + 'short': [ + 'button:has-text("Short")', + '[data-length="short"]', + ], + 'default': [ + 'button:has-text("Default")', + '[data-length="default"]', + ], + 'long': [ + 'button:has-text("Long")', + '[data-length="long"]', + ], +} + +AUDIO_INSTRUCTIONS_SELECTORS = [ + 'textarea[placeholder*="Things to try"]', + 'textarea[placeholder*="focus"]', + '[aria-label*="What should the AI hosts focus on"]', + 'textarea', +] + +AUDIO_LANGUAGE_DROPDOWN_SELECTORS = [ + 'div:has-text("Choose language") + div', + '[aria-label*="language"]', + '[aria-haspopup="listbox"]', + 'mat-select', + '.language-select', +] + +# Language name mapping: English name -> NotebookLM display name (native script) +# Based on NotebookLM's language dropdown options +LANGUAGE_MAP = { + # East Asian + 'japanese': '日本語', + 'chinese': '中文(简体)', + 'simplified chinese': '中文(简体)', + 'traditional chinese': '中文(繁體)', + 'korean': '한국어', + + # European - Latin script + 'english': 'English', + 'spanish': 'español', + 'french': 'français', + 'german': 'Deutsch', + 'italian': 'italiano', + 'portuguese': 'português', + 'dutch': 'Nederlands', + 'polish': 'polski', + 'swedish': 'svenska', + 'danish': 'dansk', + 'norwegian': 'norsk', + 'finnish': 'suomi', + 'czech': 'čeština', + 'romanian': 'română', + 'hungarian': 'magyar', + 'turkish': 'Türkçe', + 'indonesian': 'Indonesia', + 'vietnamese': 'Tiếng Việt', + 'malay': 'Melayu', + 'tagalog': 'Tagalog', + 'estonian': 'eesti', + 'latvian': 'latviešu', + 'lithuanian': 'lietuvių', + 'slovenian': 'slovenščina', + 'croatian': 'hrvatski', + 'slovak': 'slovenčina', + 'catalan': 'català', + 'haitian creole': 'créole haïtien', + + # Cyrillic + 'russian': 'русский', + 'ukrainian': 'українська', + 'bulgarian': 'български', + + # Other scripts + 'arabic': 'العربية', + 'hebrew': 'עברית', + 'hindi': 'हिन्दी', + 'thai': 'ไทย', + 'greek': 'Ελληνικά', + 'bengali': 'বাংলা', + 'tamil': 'தமிழ்', + 'telugu': 'తెలుగు', + 'marathi': 'मराठी', + 'gujarati': 'ગુજરાતી', + 'kannada': 'ಕನ್ನಡ', + 'malayalam': 'മലയാളം', + 'punjabi': 'ਪੰਜਾਬੀ', + 'urdu': 'اردو', + 'persian': 'فارسی', + 'farsi': 'فارسی', + 'swahili': 'Kiswahili', + 'afrikaans': 'Afrikaans', +} + +def get_language_display_name(language: str) -> str: + """ + Get the NotebookLM display name for a language. + + Args: + language: Language name in English (case-insensitive) or native script + + Returns: + The display name used in NotebookLM's dropdown + """ + # Check if it's already a native script name (pass through) + if language in LANGUAGE_MAP.values(): + return language + + # Look up by English name (case-insensitive) + return LANGUAGE_MAP.get(language.lower(), language) + +AUDIO_GENERATE_BUTTON_SELECTORS = [ + 'button:has-text("Generate")', + '[aria-label="Generate"]', + 'button.generate-button', +] + +AUDIO_MENU_BUTTON_SELECTORS = [ + 'button[aria-label="More options"]', + 'button[aria-label="Options"]', + 'button:has([data-icon="more_vert"])', +] + +AUDIO_DOWNLOAD_MENU_SELECTORS = [ + '[role="menuitem"]:has-text("Download")', + 'li:has-text("Download")', + 'button:has-text("Download")', + 'div:has-text("Download")', +] + +AUDIO_GENERATING_SELECTORS = [ + 'div:has-text("Generating Audio Overview")', + ':text("Generating Audio Overview")', + ':text("Come back in a few minutes")', + '[aria-label="Generating"]', +] + +# Source Management Selectors (for removing/renaming sources) +SOURCE_ITEM_SELECTORS = [ + '[data-test-id="source-item"]', + '.source-item', + 'div[role="listitem"]', +] + +SOURCE_MENU_BUTTON_SELECTORS = [ + 'button[aria-label="More options"]', + 'button[aria-label="Source options"]', + 'button:has([data-icon="more_vert"])', + 'button:has-text("⋮")', +] + +SOURCE_REMOVE_MENU_SELECTORS = [ + '[role="menuitem"]:has-text("Remove source")', + 'li:has-text("Remove source")', + 'button:has-text("Remove source")', + 'div:has-text("Remove source")', +] + +SOURCE_CONFIRM_REMOVE_SELECTORS = [ + 'button:has-text("Remove")', + 'button:has-text("Delete")', + 'button:has-text("Confirm")', + '[data-action="confirm-remove"]', +] + +# Video Overview Selectors +VIDEO_GENERATION_TIMEOUT = 900000 # 15 minutes for video generation + +VIDEO_OVERVIEW_CARD_SELECTORS = [ + 'div:has-text("Video Overview")', + '[aria-label="Video Overview"]', +] + +VIDEO_CUSTOMIZE_BUTTON_SELECTORS = [ + 'button[aria-label="Customize Video Overview"]', + 'div:has-text("Video Overview") button', +] + +VIDEO_FORMAT_SELECTORS = { + 'explainer': [ + 'div:has-text("Explainer")', + '[data-format="explainer"]', + ], + 'brief': [ + 'div:has-text("Brief")', + '[data-format="brief"]', + ], +} + +VIDEO_STYLE_SELECTORS = { + 'auto': [ + 'div:has-text("Auto-select")', + '[data-style="auto"]', + ], + 'custom': [ + 'div:has-text("Custom")', + '[data-style="custom"]', + ], + 'classic': [ + 'div:has-text("Classic")', + '[data-style="classic"]', + ], + 'whiteboard': [ + 'div:has-text("Whiteboard")', + '[data-style="whiteboard"]', + ], + 'kawaii': [ + 'div:has-text("Kawaii")', + '[data-style="kawaii"]', + ], + 'anime': [ + 'div:has-text("Anime")', + '[data-style="anime"]', + ], +} + +VIDEO_LANGUAGE_DROPDOWN_SELECTORS = [ + 'div:has-text("Choose language") + div', + '[aria-label*="language"]', + '[aria-haspopup="listbox"]', + 'mat-select', + '.language-select', +] + +VIDEO_INSTRUCTIONS_SELECTORS = [ + 'textarea[placeholder*="Things to try"]', + 'textarea[placeholder*="focus"]', + '[aria-label*="What should the AI hosts focus on"]', + 'textarea', +] + +VIDEO_GENERATE_BUTTON_SELECTORS = [ + 'button:has-text("Generate")', + '[aria-label="Generate"]', + 'button.generate-button', +] + +VIDEO_MENU_BUTTON_SELECTORS = [ + 'button[aria-label="More options"]', + 'button[aria-label="Options"]', + 'button:has([data-icon="more_vert"])', +] + +VIDEO_DOWNLOAD_MENU_SELECTORS = [ + '[role="menuitem"]:has-text("Download")', + 'li:has-text("Download")', + 'button:has-text("Download")', + 'div:has-text("Download")', +] + +VIDEO_GENERATING_SELECTORS = [ + 'div:has-text("Generating Video Overview")', + ':text("Generating Video Overview")', + ':text("Come back in a few minutes")', + '[aria-label="Generating"]', +] + +# Shared JavaScript for source detection from UI +# Used by list_sources.py and source_extractor.py +SOURCE_DETECTION_JS = """ +() => { + const sources = []; + const seenTexts = new Set(); + + // UI text to skip (exact matches) + const skipTexts = new Set(['check_box', 'more_vert', 'description', 'drive_pdf', + 'markdown', 'web', 'select all', 'select all sources', + 'sources', 'add source', 'add sources', 'video_youtube', 'link', 'pdf', + 'more', 'less', 'expand', 'collapse', 'chat', 'studio', + 'save to note', 'try deep research for an in-depth report and new sources!']); + + // Helper to check if text is a valid source name + const isValidSource = (text) => { + if (!text || text.length < 3 || text.length > 150) return false; + if (text.includes('\\n')) return false; + + // Skip UI elements (exact match, case insensitive) + if (skipTexts.has(text.toLowerCase())) return false; + + // Skip if it starts with bullet points or looks like content + if (text.startsWith('•') || text.startsWith('-') || text.startsWith('*')) return false; + if (text.startsWith('How ') || text.startsWith('What ') || text.startsWith('Based on')) return false; + + return true; + }; + + // Primary method: Find checkbox elements and extract source titles from their rows + const checkboxes = document.querySelectorAll('mat-checkbox, [role="checkbox"]'); + + for (const checkbox of checkboxes) { + const rect = checkbox.getBoundingClientRect(); + // Must be in left panel (x < 450) - filters out main content + if (rect.x > 450 || rect.x < 0) continue; + + // Get the row container (parent that contains both title and checkbox) + let row = checkbox.parentElement; + for (let i = 0; i < 4 && row; i++) { + const rowRect = row.getBoundingClientRect(); + // Row should be reasonably sized (not too tall = contains multiple items) + if (rowRect.height > 30 && rowRect.height < 80) { + break; + } + row = row.parentElement; + } + + if (!row) continue; + + // Extract the title from this row + const textElements = row.querySelectorAll('span, div'); + for (const el of textElements) { + const text = el.innerText?.trim(); + if (!text) continue; + + // Skip if too short or contains multiple lines + if (text.length < 3 || text.length > 150) continue; + if (text.includes('\\n')) continue; + + // Skip icon text + if (skipTexts.has(text.toLowerCase())) continue; + + // This is likely the source title + if (!seenTexts.has(text) && isValidSource(text)) { + seenTexts.add(text); + sources.push(text); + break; // Found title for this row + } + } + } + + // Fallback: If we didn't find enough, try looking for .source-title class + if (sources.length < 3) { + const titleElements = document.querySelectorAll('.source-title, .source-name, [class*="source"][class*="title"]'); + for (const el of titleElements) { + const text = el.innerText?.trim(); + if (text && isValidSource(text) && !seenTexts.has(text)) { + seenTexts.add(text); + sources.push(text); + } + } + } + + return sources; +} +""" diff --git a/scripts/list_sources.py b/scripts/list_sources.py new file mode 100644 index 0000000..dd8216a --- /dev/null +++ b/scripts/list_sources.py @@ -0,0 +1,179 @@ +#!/usr/bin/env python3 +""" +List Sources from NotebookLM +Reads source names directly from the UI (more reliable than asking questions) +""" + +import argparse +import sys +import re +import time +import traceback +from pathlib import Path + +from patchright.sync_api import sync_playwright + +# Add parent directory to path +sys.path.insert(0, str(Path(__file__).parent)) + +from auth_manager import AuthManager +from notebook_manager import NotebookLibrary +from browser_utils import BrowserFactory +from config import SOURCE_DETECTION_JS + + +def list_sources_from_ui(notebook_url: str, headless: bool = True) -> list: + """ + List all sources in a NotebookLM notebook by reading the UI. + + This is more reliable than asking NotebookLM to list sources via a question. + + Args: + notebook_url: The NotebookLM notebook URL + headless: Run browser in headless mode + + Returns: + List of source names/titles + """ + auth = AuthManager() + + if not auth.is_authenticated(): + print(" Not authenticated. Run: python scripts/run.py auth_manager.py setup") + return [] + + print(f" Reading sources from: {notebook_url}") + + playwright = None + context = None + sources = [] + + try: + playwright = sync_playwright().start() + + context = BrowserFactory.launch_persistent_context( + playwright, + headless=headless + ) + + page = context.new_page() + print(" Opening notebook...") + page.goto(notebook_url, wait_until="domcontentloaded") + + # Wait for notebook to load + page.wait_for_url(re.compile(r"^https://notebooklm\.google\.com/"), timeout=60000) + + # Wait for sources panel to load + print(" Waiting for sources panel...") + time.sleep(3) + + # Scroll through the sources panel to load all sources + # (some sources may be lazy-loaded or virtualized) + print(" Scrolling through sources panel...") + + # Find and scroll the sources container + page.evaluate(""" + () => { + // Find the scrollable sources container + const containers = document.querySelectorAll('[class*="source"], [class*="scroll"], [role="list"]'); + for (const container of containers) { + const rect = container.getBoundingClientRect(); + // Left panel container + if (rect.x < 350 && rect.height > 200) { + // Scroll down multiple times to load all sources + for (let i = 0; i < 10; i++) { + container.scrollTop += 300; + } + // Scroll back to top + container.scrollTop = 0; + break; + } + } + } + """) + time.sleep(1) + + # Extract source titles using shared JavaScript detection + sources = page.evaluate(SOURCE_DETECTION_JS) + # Sort alphabetically for easier reading + sources.sort() + + print(f" Found {len(sources)} sources") + return sources + + except Exception as e: + print(f" Error: {e}") + traceback.print_exc() + return [] + + finally: + if context: + try: + context.close() + except Exception: + pass + if playwright: + try: + playwright.stop() + except Exception: + pass + + +def main(): + parser = argparse.ArgumentParser(description='List sources in a NotebookLM notebook') + + # Notebook selection (mutually exclusive) + notebook_group = parser.add_mutually_exclusive_group() + notebook_group.add_argument('--notebook-url', help='NotebookLM notebook URL') + notebook_group.add_argument('--notebook-id', help='Notebook ID from library') + + parser.add_argument('--show-browser', action='store_true', help='Show browser window') + parser.add_argument('--json', action='store_true', help='Output as JSON') + + args = parser.parse_args() + + # Resolve notebook URL + library = NotebookLibrary() + + if args.notebook_id: + notebook = library.get_notebook(args.notebook_id) + if not notebook: + print(f"Notebook not found: {args.notebook_id}") + return 1 + notebook_url = notebook['url'] + notebook_name = notebook['name'] + elif args.notebook_url: + notebook_url = args.notebook_url + notebook_name = "Unknown" + else: + # Use active notebook + active = library.get_active_notebook() + if active: + notebook_url = active['url'] + notebook_name = active['name'] + print(f"Using active notebook: {notebook_name}") + else: + print("No notebook specified and no active notebook set") + return 1 + + sources = list_sources_from_ui( + notebook_url=notebook_url, + headless=not args.show_browser + ) + + if args.json: + import json + print(json.dumps(sources, indent=2, ensure_ascii=False)) + else: + print("\n" + "=" * 60) + print(f" Sources in: {notebook_name}") + print("=" * 60) + for i, source in enumerate(sources, 1): + print(f" {i}. {source}") + print("=" * 60) + print(f" Total: {len(sources)} sources") + + return 0 + + +if __name__ == "__main__": + sys.exit(main()) diff --git a/scripts/notebook_manager.py b/scripts/notebook_manager.py index e10e156..16eadca 100755 --- a/scripts/notebook_manager.py +++ b/scripts/notebook_manager.py @@ -7,12 +7,15 @@ import json import argparse -import uuid +import subprocess +import sys import os from pathlib import Path from typing import Dict, List, Optional, Any from datetime import datetime +from config import SOURCE_SUMMARY_DIR, SOURCE_SUMMARY_QUERY + class NotebookLibrary: """Manages a collection of NotebookLM notebooks with metadata""" @@ -118,6 +121,11 @@ def add_notebook( self._save_library() print(f"✅ Added notebook: {name} ({notebook_id})") + + # Auto-fetch source summary for new notebooks + print("📋 Fetching source summary...") + self.fetch_source_summary(notebook_id) + return notebook def remove_notebook(self, notebook_id: str) -> bool: @@ -304,6 +312,111 @@ def get_stats(self) -> Dict[str, Any]: 'library_path': str(self.library_file) } + def get_source_summary_path(self, notebook_id: str) -> Path: + """Get the path to the source summary file for a notebook""" + SOURCE_SUMMARY_DIR.mkdir(parents=True, exist_ok=True) + return SOURCE_SUMMARY_DIR / f"{notebook_id}.md" + + def has_source_summary(self, notebook_id: str) -> bool: + """Check if a source summary file exists for a notebook""" + return self.get_source_summary_path(notebook_id).exists() + + def fetch_source_summary(self, notebook_id: str, force: bool = False) -> Optional[str]: + """ + Fetch source summary from NotebookLM by clicking each source and extracting Source Guide + + Args: + notebook_id: ID of the notebook + force: If True, re-extract all sources; if False, only extract new sources + + Returns: + The source summary content, or None if failed + """ + notebook = self.get_notebook(notebook_id) + if not notebook: + print(f"❌ Notebook not found: {notebook_id}") + return None + + summary_path = self.get_source_summary_path(notebook_id) + + # Always run the extractor - it handles incremental logic internally + print(f"🔍 Checking sources for: {notebook['name']}") + + # Call source_extractor.py for click-based extraction + skill_dir = Path(__file__).parent.parent + extractor_script = skill_dir / "scripts" / "source_extractor.py" + venv_python = skill_dir / ".venv" / "bin" / "python" + + if not venv_python.exists(): + # Fallback to system python3 + venv_python = "python3" + + try: + cmd = [str(venv_python), str(extractor_script), "--notebook-id", notebook_id] + if force: + cmd.append("--force") + + result = subprocess.run( + cmd, + capture_output=True, + text=True, + timeout=600 # 10 minute timeout (needs to click through all sources) + ) + + if result.returncode != 0: + print(f"❌ Failed to fetch source summary: {result.stderr}") + return None + + # The extractor saves directly to the file, so just read it back + if summary_path.exists(): + with open(summary_path, 'r', encoding='utf-8') as f: + return f.read() + else: + print("❌ Source summary file was not created") + return None + + except subprocess.TimeoutExpired: + print("❌ Timeout while fetching source summary") + return None + except Exception as e: + print(f"❌ Error fetching source summary: {e}") + return None + + def _save_source_summary(self, notebook_id: str, notebook_name: str, content: str): + """Save source summary to markdown file""" + summary_path = self.get_source_summary_path(notebook_id) + + # Create header with metadata + header = f"""# Source Summary: {notebook_name} + +**Notebook ID:** `{notebook_id}` +**Generated:** {datetime.now().strftime('%Y-%m-%d %H:%M:%S')} + +--- + +## Sources + +""" + + full_content = header + content + + with open(summary_path, 'w') as f: + f.write(full_content) + + print(f"✅ Saved source summary: {summary_path}") + + def update_source_summary(self, notebook_id: str) -> Optional[str]: + """Force update the source summary for a notebook""" + return self.fetch_source_summary(notebook_id, force=True) + + def get_source_summary(self, notebook_id: str) -> Optional[str]: + """Get the source summary content for a notebook""" + summary_path = self.get_source_summary_path(notebook_id) + if summary_path.exists(): + with open(summary_path, 'r') as f: + return f.read() + return None + def main(): """Command-line interface for notebook management""" @@ -338,6 +451,16 @@ def main(): # Stats command subparsers.add_parser('stats', help='Show library statistics') + # Update sources command + update_sources_parser = subparsers.add_parser('update-sources', help='Update source summary for a notebook') + update_sources_parser.add_argument('--id', help='Notebook ID (uses active if not specified)') + update_sources_parser.add_argument('--all', action='store_true', help='Update all notebooks') + update_sources_parser.add_argument('--force', action='store_true', help='Force re-extraction of all sources (ignore existing)') + + # Show sources command + show_sources_parser = subparsers.add_parser('show-sources', help='Show source summary for a notebook') + show_sources_parser.add_argument('--id', help='Notebook ID (uses active if not specified)') + args = parser.parse_args() # Initialize library @@ -402,6 +525,48 @@ def main(): print(f" Most used: {stats['most_used_notebook']['name']} ({stats['most_used_notebook']['use_count']} uses)") print(f" Library path: {stats['library_path']}") + elif args.command == 'update-sources': + force = getattr(args, 'force', False) + if args.all: + # Update all notebooks + notebooks = library.list_notebooks() + if not notebooks: + print("📚 No notebooks in library") + else: + print(f"🔄 Updating source summaries for {len(notebooks)} notebooks...") + for notebook in notebooks: + library.fetch_source_summary(notebook['id'], force=force) + else: + # Update specific or active notebook + notebook_id = args.id + if not notebook_id: + active = library.get_active_notebook() + if active: + notebook_id = active['id'] + else: + print("❌ No notebook ID specified and no active notebook set") + sys.exit(1) + + library.fetch_source_summary(notebook_id, force=force) + + elif args.command == 'show-sources': + # Show source summary for a notebook + notebook_id = args.id + if not notebook_id: + active = library.get_active_notebook() + if active: + notebook_id = active['id'] + else: + print("❌ No notebook ID specified and no active notebook set") + sys.exit(1) + + summary = library.get_source_summary(notebook_id) + if summary: + print(summary) + else: + print(f"📄 No source summary found for: {notebook_id}") + print(" Run: python3 scripts/run.py notebook_manager.py update-sources --id " + notebook_id) + else: parser.print_help() diff --git a/scripts/remove_source.py b/scripts/remove_source.py new file mode 100644 index 0000000..9fab495 --- /dev/null +++ b/scripts/remove_source.py @@ -0,0 +1,377 @@ +#!/usr/bin/env python3 +""" +Remove Source from NotebookLM +CAUTION: This permanently removes a source from a notebook. Use with care. +""" + +import argparse +import sys +import re +import traceback +from pathlib import Path + +from patchright.sync_api import sync_playwright + +# Add parent directory to path +sys.path.insert(0, str(Path(__file__).parent)) + +from auth_manager import AuthManager +from notebook_manager import NotebookLibrary +from config import ( + SOURCE_MENU_BUTTON_SELECTORS, + SOURCE_REMOVE_MENU_SELECTORS, + SOURCE_CONFIRM_REMOVE_SELECTORS, + PAGE_LOAD_TIMEOUT +) +from browser_utils import BrowserFactory, StealthUtils, find_element_with_selectors + + +def remove_source( + notebook_url: str, + source_name: str, + headless: bool = True, + confirm: bool = False +) -> bool: + """ + Remove a source from a NotebookLM notebook. + + CAUTION: This is a PERMANENT action. The source cannot be recovered. + + Args: + notebook_url: The NotebookLM notebook URL + source_name: Name/title of the source to remove (partial match supported) + headless: Run browser in headless mode + confirm: Must be True to actually remove (safety check) + + Returns: + True if source was removed, False otherwise + """ + if not confirm: + print(" WARNING: Remove source requires --confirm flag") + print(" This action is PERMANENT and cannot be undone.") + print(" Re-run with --confirm to proceed.") + return False + + auth = AuthManager() + + if not auth.is_authenticated(): + print(" Not authenticated. Run: python scripts/run.py auth_manager.py setup") + return False + + print(" Removing source from notebook...") + print(f" Notebook: {notebook_url}") + print(f" Source to remove: {source_name}") + print("") + print(" WARNING: This action is PERMANENT!") + print("") + + playwright = None + context = None + + try: + playwright = sync_playwright().start() + + context = BrowserFactory.launch_persistent_context( + playwright, + headless=headless + ) + + page = context.new_page() + print(" Opening notebook...") + page.goto(notebook_url, wait_until="domcontentloaded") + + # Wait for notebook to load + page.wait_for_url(re.compile(r"^https://notebooklm\.google\.com/"), timeout=60000) + StealthUtils.random_delay(2000, 3000) + + # Look for the source in the sources panel (left sidebar) + print(f" Looking for source: {source_name}") + + # The sources panel is on the left side of the page + # Each source appears as a row with: [menu button (⋮)] [icon] [source name] [checkbox] + # We need to find the specific row containing our source name + + source_row = None + source_text_found = None + + # Strategy: Find all source rows and look for the one with matching text + # Source rows are typically in a scrollable list under "Sources" heading + + # First, try to find the sources container + sources_container = page.query_selector('[aria-label="Sources"], .sources-panel, [data-testid="sources-list"]') + if not sources_container: + # Fallback: look for elements that contain source names + sources_container = page + + # Look for source items - they typically have the source name as text + # and a checkbox to the right, with a menu button to the left + potential_sources = page.query_selector_all('div, span, li') + + for el in potential_sources: + try: + text = el.text_content() or '' + # Check if this element contains our source name (case insensitive, partial match) + if source_name.lower() in text.lower(): + # Verify this is a source item (not some other element with the text) + # Source items are typically compact and don't contain huge amounts of text + if len(text) < 200: # Source names are relatively short + # Check if there's a checkbox or icon nearby (indicators of source row) + has_checkbox = el.query_selector('input[type="checkbox"], [role="checkbox"]') + parent = el.evaluate_handle('el => el.parentElement') + + # Get bounding box to verify it's in the left panel (x < 400 typically) + box = el.bounding_box() + if box and box['x'] < 400: # Left panel check + source_row = el + source_text_found = text[:50] + break + except Exception: + continue + + if not source_row: + print(f" Could not find source: {source_name}") + page.screenshot(path="/tmp/notebooklm_source_debug.png") + print(" Debug screenshot saved to /tmp/notebooklm_source_debug.png") + return False + + print(f" Found source: {source_text_found}...") + + # Now find the menu button for THIS specific source + # The menu button (⋮) is to the LEFT of the source name + print(" Looking for source menu button...") + + menu_button = None + source_box = source_row.bounding_box() + + if source_box: + # Hover over the source row to make the menu button visible + page.mouse.move(source_box['x'] + source_box['width'] / 2, + source_box['y'] + source_box['height'] / 2) + StealthUtils.random_delay(500, 800) + + # Look for menu buttons that are: + # 1. On the same vertical level (same Y coordinate ± some tolerance) + # 2. To the LEFT of the source text (smaller X coordinate) + # 3. Within the sources panel (X < 400) + + all_buttons = page.query_selector_all('button') + for btn in all_buttons: + try: + btn_box = btn.bounding_box() + if not btn_box: + continue + + # Check if button is on same row (Y within 30px) + y_diff = abs(btn_box['y'] - source_box['y']) + if y_diff > 30: + continue + + # Check if button is in left panel + if btn_box['x'] > 350: + continue + + # Check if button is to the left of source (or at start of row) + if btn_box['x'] > source_box['x']: + continue + + # Check aria-label or other indicators that this is a menu button + aria_label = btn.get_attribute('aria-label') or '' + inner_html = btn.inner_html() or '' + + # Menu buttons often have "more", "options", or the ⋮ icon + if ('more' in aria_label.lower() or + 'option' in aria_label.lower() or + 'menu' in aria_label.lower() or + 'more_vert' in inner_html.lower() or + '⋮' in inner_html): + menu_button = btn + print(f" Found menu button at ({btn_box['x']}, {btn_box['y']})") + break + + # If no aria-label, check if it's a small button (likely icon-only menu button) + if btn_box['width'] < 50 and btn_box['height'] < 50: + # This could be the menu button - verify by checking if it's visible + if btn.is_visible(): + menu_button = btn + print(f" Found potential menu button at ({btn_box['x']}, {btn_box['y']})") + break + except Exception: + continue + + if not menu_button: + # Fallback: try clicking directly on the source row to see if menu appears + print(" Menu button not found, trying direct interaction...") + try: + # Right-click might show context menu + source_row.click(button='right') + StealthUtils.random_delay(500, 800) + + # Check if a menu appeared with "Remove" option + remove_check = page.query_selector('[role="menuitem"]:has-text("Remove"), li:has-text("Remove")') + if remove_check: + print(" Context menu appeared!") + # Skip to the remove step + menu_button = "context_menu_used" + except Exception: + pass + + if not menu_button: + print(" Could not find source menu button") + page.screenshot(path="/tmp/notebooklm_menu_debug.png") + print(" Debug screenshot saved to /tmp/notebooklm_menu_debug.png") + return False + + if menu_button != "context_menu_used": + print(" Clicking menu button...") + menu_button.click() + StealthUtils.random_delay(800, 1200) + + # Look for "Remove source" option in the dropdown + print(" Looking for 'Remove source' option...") + + remove_option, remove_selector = find_element_with_selectors( + page, SOURCE_REMOVE_MENU_SELECTORS, timeout=3000 + ) + + if not remove_option: + print(" Could not find 'Remove source' option") + page.screenshot(path="/tmp/notebooklm_remove_debug.png") + print(" Debug screenshot saved to /tmp/notebooklm_remove_debug.png") + return False + + print(f" Found: {remove_selector}") + remove_option.click() + StealthUtils.random_delay(500, 800) + + # Handle confirmation dialog if one appears + print(" Checking for confirmation dialog...") + StealthUtils.random_delay(500, 800) + + # Try to find and click confirm button + confirm_button, confirm_selector = find_element_with_selectors( + page, SOURCE_CONFIRM_REMOVE_SELECTORS, timeout=3000 + ) + + if confirm_button: + print(" Found confirmation dialog, confirming removal...") + confirm_button.click() + StealthUtils.random_delay(1000, 1500) + + # Verify removal + print(" Verifying source was removed...") + StealthUtils.random_delay(1500, 2000) + + # Try to find the source again - it should not exist + still_exists = False + + # Check all elements in the left panel for the source name + all_elements = page.query_selector_all('div, span') + for el in all_elements: + try: + box = el.bounding_box() + if not box or box['x'] > 400: # Only check left panel + continue + text = el.text_content() or '' + if source_name.lower() in text.lower() and len(text) < 200: + if el.is_visible(): + still_exists = True + break + except Exception: + continue + + if still_exists: + print(" Source still appears to exist - removal may have failed") + return False + + print(" Source removed successfully!") + return True + + except Exception as e: + print(f" Error: {e}") + traceback.print_exc() + return False + + finally: + if context: + try: + context.close() + except Exception: + pass + if playwright: + try: + playwright.stop() + except Exception: + pass + + +def main(): + parser = argparse.ArgumentParser( + description='Remove a source from NotebookLM notebook (PERMANENT ACTION)', + epilog='WARNING: This action cannot be undone. Use with caution.' + ) + + # Notebook selection (mutually exclusive) + notebook_group = parser.add_mutually_exclusive_group(required=True) + notebook_group.add_argument('--notebook-url', help='NotebookLM notebook URL') + notebook_group.add_argument('--notebook-id', help='Notebook ID from library') + + # Source to remove + parser.add_argument('--source', required=True, + help='Name of the source to remove (partial match supported)') + + # Safety flag + parser.add_argument('--confirm', action='store_true', + help='REQUIRED: Confirm you want to permanently remove this source') + + parser.add_argument('--show-browser', action='store_true', + help='Show browser window') + + args = parser.parse_args() + + # Resolve notebook URL + if args.notebook_id: + library = NotebookLibrary() + notebook = library.get_notebook(args.notebook_id) + if not notebook: + print(f"Notebook not found: {args.notebook_id}") + return 1 + notebook_url = notebook['url'] + else: + notebook_url = args.notebook_url + + # Extra warning + if args.confirm: + print("") + print("=" * 60) + print(" WARNING: PERMANENT ACTION") + print("=" * 60) + print(f" You are about to PERMANENTLY remove: {args.source}") + print(" This cannot be undone!") + print("=" * 60) + print("") + + result = remove_source( + notebook_url=notebook_url, + source_name=args.source, + headless=not args.show_browser, + confirm=args.confirm + ) + + if result: + print("\n" + "=" * 60) + print(" Source Removed") + print("=" * 60) + print(f" Source: {args.source}") + print(" Status: Permanently removed") + print("=" * 60) + return 0 + else: + if not args.confirm: + print("\n To remove the source, re-run with --confirm flag") + else: + print("\n Failed to remove source") + return 1 + + +if __name__ == "__main__": + sys.exit(main()) diff --git a/scripts/run.py b/scripts/run.py index 7c47a92..e6c2d70 100755 --- a/scripts/run.py +++ b/scripts/run.py @@ -7,9 +7,22 @@ import os import sys import subprocess +import shutil from pathlib import Path +def get_system_python_command(): + """Get the system Python command (python3 or python)""" + # Check python3 first (preferred) + if shutil.which('python3'): + return 'python3' + # Fall back to python + if shutil.which('python'): + return 'python' + # Last resort - use whatever ran this script + return sys.executable + + def get_venv_python(): """Get the virtual environment Python executable""" skill_dir = Path(__file__).parent.parent @@ -48,10 +61,16 @@ def ensure_venv(): def main(): """Main runner""" if len(sys.argv) < 2: - print("Usage: python run.py [args...]") + python_cmd = get_system_python_command() + print(f"Usage: {python_cmd} run.py [args...]") print("\nAvailable scripts:") - print(" ask_question.py - Query NotebookLM") + print(" ask_question.py - Query NotebookLM") print(" notebook_manager.py - Manage notebook library") + print(" audio_generator.py - Generate Audio Overview with custom prompts") + print(" video_generator.py - Generate Video Overview with custom prompts") + print(" add_source.py - Add sources to notebooks") + print(" list_sources.py - List sources by reading UI (reliable)") + print(" remove_source.py - Remove source from notebook (PERMANENT)") print(" session_manager.py - Manage sessions") print(" auth_manager.py - Handle authentication") print(" cleanup_manager.py - Clean up skill data") diff --git a/scripts/source_extractor.py b/scripts/source_extractor.py new file mode 100644 index 0000000..634694f --- /dev/null +++ b/scripts/source_extractor.py @@ -0,0 +1,533 @@ +#!/usr/bin/env python3 +""" +Source Extractor for NotebookLM +Extracts source titles and their Source Guide summaries by clicking each source +""" + +import argparse +import sys +import time +import re +import traceback +from pathlib import Path +from datetime import datetime + +from patchright.sync_api import sync_playwright + +# Add parent directory to path +sys.path.insert(0, str(Path(__file__).parent)) + +from auth_manager import AuthManager +from notebook_manager import NotebookLibrary +from config import SOURCE_SUMMARY_DIR, SOURCE_DETECTION_JS +from browser_utils import BrowserFactory, StealthUtils + + +def get_existing_sources(notebook_id: str) -> set: + """ + Get set of source titles already in the summary file + + Args: + notebook_id: ID of the notebook + + Returns: + Set of source titles that already exist + """ + summary_path = SOURCE_SUMMARY_DIR / f"{notebook_id}.md" + existing_titles = set() + + if summary_path.exists(): + try: + with open(summary_path, 'r', encoding='utf-8') as f: + content = f.read() + # Parse out source titles (lines starting with ###) + for line in content.split('\n'): + if line.startswith('### '): + title = line[4:].strip() + existing_titles.add(title) + except Exception as e: + print(f" ⚠️ Could not read existing summary: {e}") + + return existing_titles + + +def extract_sources(notebook_url: str, headless: bool = True, existing_titles: set = None) -> list: + """ + Extract all sources and their Source Guide content from a NotebookLM notebook + + Args: + notebook_url: NotebookLM notebook URL + headless: Run browser in headless mode + existing_titles: Set of titles to skip (already extracted) + + Returns: + List of dicts with 'title' and 'summary' keys + """ + if existing_titles is None: + existing_titles = set() + + auth = AuthManager() + + if not auth.is_authenticated(): + print("⚠️ Not authenticated. Run: python3 scripts/run.py auth_manager.py setup") + return [] + + print(f"📚 Extracting sources from: {notebook_url}") + + playwright = None + context = None + sources = [] + + try: + # Start playwright + playwright = sync_playwright().start() + + # Launch persistent browser context + context = BrowserFactory.launch_persistent_context( + playwright, + headless=headless + ) + + # Navigate to notebook + page = context.new_page() + print(" 🌐 Opening notebook...") + page.goto(notebook_url, wait_until="domcontentloaded") + + # Wait for NotebookLM to load + page.wait_for_url(re.compile(r"^https://notebooklm\.google\.com/"), timeout=15000) + + # Wait for the sources panel to load + print(" ⏳ Waiting for sources panel...") + time.sleep(5) # Give the page time to fully render + + # Find all source items in the sidebar + # NotebookLM uses a list of source items - we need to find them + # The sources are typically in a scrollable list on the left + + # Try multiple selectors for source items + source_selectors = [ + 'div[data-source-id]', # Sources with data attribute + '.source-item', # Class-based + '.source-list-item', + '[role="listitem"]', # ARIA role + 'mat-list-item', # Material list item + ] + + source_elements = [] + for selector in source_selectors: + try: + elements = page.query_selector_all(selector) + if elements and len(elements) > 0: + print(f" ✓ Found {len(elements)} sources using: {selector}") + source_elements = elements + break + except Exception: + continue + + if not source_elements: + # Try to find sources by looking for clickable items in the source panel + print(" 🔍 Trying alternative source detection...") + + # Look for the Sources header and find items below it + try: + # Wait a bit more for dynamic content + time.sleep(4) + + # Try finding source titles directly - look for common patterns + alt_selectors = [ + '.source-title', + '.source-name', + '[data-testid*="source"]', + 'button[class*="source"]', + 'div[class*="source-list"] button', + 'div[class*="source-list"] [role="button"]', + ] + + for selector in alt_selectors: + source_elements = page.query_selector_all(selector) + if source_elements and len(source_elements) > 0: + print(f" ✓ Found {len(source_elements)} sources using: {selector}") + break + except Exception: + pass + + if not source_elements: + print(" ⚠️ Could not find source elements automatically.") + print(" 📸 Taking screenshot for debugging...") + + # Take a screenshot to help debug + screenshot_path = SOURCE_SUMMARY_DIR / "debug_screenshot.png" + SOURCE_SUMMARY_DIR.mkdir(parents=True, exist_ok=True) + page.screenshot(path=str(screenshot_path)) + print(f" 📸 Screenshot saved: {screenshot_path}") + + # Try to get page HTML for debugging + print(" 🔍 Attempting to extract sources from page structure...") + + # Use JavaScript to find source elements + source_data = page.evaluate(""" + () => { + const results = []; + + // Look for source items in various ways + // Method 1: Find elements with source-related classes + const possibleSources = document.querySelectorAll('[class*="source"], [class*="Source"]'); + + // Method 2: Look for list items that might be sources + const listItems = document.querySelectorAll('mat-list-item, .mat-mdc-list-item, [role="listitem"]'); + + // Log what we find + console.log('Found possible sources:', possibleSources.length); + console.log('Found list items:', listItems.length); + + // Return info about the page structure + return { + possibleSourcesCount: possibleSources.length, + listItemsCount: listItems.length, + bodyClasses: document.body.className, + mainContent: document.querySelector('main')?.innerHTML?.substring(0, 500) || 'No main element' + }; + } + """) + print(f" 📊 Page analysis: {source_data}") + + # If we have existing sources and can't find new ones, that's okay + if existing_titles: + print(f" ℹ️ Could not check for new sources, but {len(existing_titles)} sources already exist") + return [] + + return [] + + # First, get all source titles via shared JavaScript detection + print(" 📝 Collecting all source titles...") + source_titles = page.evaluate(SOURCE_DETECTION_JS) + + if not source_titles: + print(" ⚠️ Could not get source titles") + return [] + + total_sources = len(source_titles) + + # Filter out already extracted sources + titles_to_extract = [t for t in source_titles if t not in existing_titles] + skipped_count = total_sources - len(titles_to_extract) + + if skipped_count > 0: + print(f" 📋 Found {total_sources} sources, {skipped_count} already extracted, {len(titles_to_extract)} to fetch") + else: + print(f" 📋 Found {total_sources} source titles to extract") + + if not titles_to_extract: + print(" ✅ All sources already extracted!") + return [] + + for i, title in enumerate(titles_to_extract): + try: + print(f" [{i+1}/{len(titles_to_extract)}] {title[:60]}...") + + # Navigate back to the notebook main page to ensure source list is visible + if i > 0: + page.goto(notebook_url, wait_until="domcontentloaded") + time.sleep(4) # Wait for sources panel to reload + + # Use Playwright locator to find and click the source by text + try: + # Try to find the source by its text + source_locator = page.locator(f'text="{title}"').first + source_locator.click(timeout=5000) + except Exception: + # Fallback: use JavaScript + clicked = page.evaluate(""" + (title) => { + const elements = document.querySelectorAll('button, [role="button"], [role="listitem"], .source-title, .source-name, span, div'); + for (const el of elements) { + const text = el.innerText?.trim(); + if (text === title || (text && text.includes(title.substring(0, 20)))) { + el.click(); + return true; + } + } + return false; + } + """, title) + + if not clicked: + print(f" ⚠️ Could not click source") + continue + + StealthUtils.random_delay(4000, 6000) # Wait longer for Source Guide to load + + # Extract the Source Guide content + source_guide_text = page.evaluate(""" + () => { + // Look for the Source guide container specifically + // The Source Guide is typically in a collapsible section with specific structure + + // Method 1: Find elements that contain "Source guide" header + const allElements = document.querySelectorAll('*'); + let bestMatch = ''; + let bestScore = 0; + + for (const el of allElements) { + const text = el.innerText || ''; + + // Skip elements that are too short or too long + if (text.length < 100 || text.length > 3000) continue; + + // Must contain "Source guide" + if (!text.includes('Source guide')) continue; + + // Score based on how focused the content is + const hasSourceGuideHeader = text.startsWith('Source guide') || + text.match(/^[\\s\\S]{0,50}Source guide/); + + // Extract content after "Source guide" header + const guideIdx = text.indexOf('Source guide'); + let content = text.substring(guideIdx + 12).trim(); + + // Remove topic chips at the end (short lines that are keywords) + const lines = content.split('\\n'); + const cleanLines = []; + let foundContent = false; + + for (const line of lines) { + const trimmed = line.trim(); + if (trimmed.length === 0) continue; + + // If this is a substantial line (part of the summary), keep it + if (trimmed.length > 50) { + foundContent = true; + cleanLines.push(trimmed); + } else if (foundContent && trimmed.length < 40) { + // Short line after content = likely a topic chip, stop + break; + } + } + + const result = cleanLines.join(' ').trim(); + + // Score this result - prefer medium-length results with content + if (result.length > 100 && result.length < 2000) { + const score = result.length; + if (score > bestScore) { + bestScore = score; + bestMatch = result; + } + } + } + + if (bestMatch) { + return bestMatch; + } + + // Fallback: Look for any paragraph that seems like a summary + const paragraphs = document.querySelectorAll('p, div'); + for (const p of paragraphs) { + const text = p.innerText?.trim(); + if (text && text.length > 200 && text.length < 2000) { + // Check if it looks like a summary (full sentences) + if (text.includes('.') && !text.includes('Source guide')) { + return text; + } + } + } + + return ''; + } + """) + + if source_guide_text and len(source_guide_text) > 30: + sources.append({ + 'title': title, + 'summary': source_guide_text + }) + print(f" ✓ Got summary ({len(source_guide_text)} chars)") + else: + print(f" ⚠️ No Source Guide found") + sources.append({ + 'title': title, + 'summary': '(No Source Guide available)' + }) + + # No need to navigate back - we'll reload the page at the start of next iteration + + except Exception as e: + print(f" ❌ Error: {e}") + continue + + print(f"\n ✅ Extracted {len(sources)} sources!") + return sources + + except Exception as e: + print(f" ❌ Error: {e}") + traceback.print_exc() + return [] + + finally: + # Clean up + if context: + try: + context.close() + except Exception: + pass + + if playwright: + try: + playwright.stop() + except Exception: + pass + + +def save_sources_markdown(notebook_id: str, notebook_name: str, sources: list, merge: bool = True) -> Path: + """ + Save extracted sources to a markdown file, optionally merging with existing + + Args: + notebook_id: ID of the notebook + notebook_name: Display name of the notebook + sources: List of dicts with 'title' and 'summary' + merge: If True, merge with existing file; if False, overwrite + + Returns: + Path to the saved file + """ + SOURCE_SUMMARY_DIR.mkdir(parents=True, exist_ok=True) + output_path = SOURCE_SUMMARY_DIR / f"{notebook_id}.md" + + existing_sources = [] + + # If merging, load existing sources + if merge and output_path.exists(): + try: + with open(output_path, 'r', encoding='utf-8') as f: + content = f.read() + + # Parse existing sources + current_title = None + current_summary_lines = [] + + for line in content.split('\n'): + if line.startswith('### '): + # Save previous source if exists + if current_title: + existing_sources.append({ + 'title': current_title, + 'summary': '\n'.join(current_summary_lines).strip() + }) + current_title = line[4:].strip() + current_summary_lines = [] + elif line.startswith('---'): + continue + elif line.startswith('# Source Summary:') or line.startswith('**'): + continue + elif current_title: + current_summary_lines.append(line) + + # Don't forget the last one + if current_title: + existing_sources.append({ + 'title': current_title, + 'summary': '\n'.join(current_summary_lines).strip() + }) + + print(f" 📄 Loaded {len(existing_sources)} existing sources") + except Exception as e: + print(f" ⚠️ Could not parse existing file: {e}") + + # Merge: add new sources to existing + all_sources = existing_sources + sources + total_count = len(all_sources) + + # Build markdown content + content = f"""# Source Summary: {notebook_name} + +**Notebook ID:** `{notebook_id}` +**Generated:** {datetime.now().strftime('%Y-%m-%d %H:%M:%S')} +**Total Sources:** {total_count} + +--- + +""" + + for source in all_sources: + content += f"### {source['title']}\n\n" + content += f"{source['summary']}\n\n" + content += "---\n\n" + + with open(output_path, 'w', encoding='utf-8') as f: + f.write(content) + + if sources: + print(f"✅ Added {len(sources)} new sources (total: {total_count})") + print(f"✅ Saved to: {output_path}") + return output_path + + +def main(): + parser = argparse.ArgumentParser(description='Extract sources from NotebookLM') + + parser.add_argument('--notebook-url', help='NotebookLM notebook URL') + parser.add_argument('--notebook-id', help='Notebook ID from library') + parser.add_argument('--show-browser', action='store_true', help='Show browser window') + parser.add_argument('--force', action='store_true', help='Force re-extraction of all sources') + + args = parser.parse_args() + + # Resolve notebook + notebook_url = args.notebook_url + notebook_id = args.notebook_id + notebook_name = "Unknown Notebook" + + library = NotebookLibrary() + + if not notebook_url and args.notebook_id: + notebook = library.get_notebook(args.notebook_id) + if notebook: + notebook_url = notebook['url'] + notebook_name = notebook['name'] + notebook_id = notebook['id'] + else: + print(f"❌ Notebook '{args.notebook_id}' not found") + return 1 + + if not notebook_url: + # Use active notebook + active = library.get_active_notebook() + if active: + notebook_url = active['url'] + notebook_name = active['name'] + notebook_id = active['id'] + print(f"📚 Using active notebook: {notebook_name}") + else: + print("❌ No notebook specified and no active notebook set") + return 1 + + # Get existing sources (unless force mode) + existing_titles = set() + if not args.force: + existing_titles = get_existing_sources(notebook_id) + if existing_titles: + print(f"📄 Found {len(existing_titles)} existing sources in summary file") + + # Extract sources (only new ones) + sources = extract_sources( + notebook_url=notebook_url, + headless=not args.show_browser, + existing_titles=existing_titles + ) + + if sources: + # Save to markdown (merge with existing) + save_sources_markdown(notebook_id, notebook_name, sources, merge=True) + return 0 + elif existing_titles: + # No new sources, but we have existing ones - that's okay + print("✅ No new sources to extract") + return 0 + else: + print("❌ No sources extracted") + return 1 + + +if __name__ == "__main__": + sys.exit(main()) diff --git a/scripts/source_filter.py b/scripts/source_filter.py new file mode 100644 index 0000000..199e780 --- /dev/null +++ b/scripts/source_filter.py @@ -0,0 +1,435 @@ +#!/usr/bin/env python3 +""" +Source Filter for NotebookLM +Determines which sources are relevant to a query and handles source selection in browser + +Supports two modes: +1. Keyword matching (fast, no API calls) +2. LLM scoring via Gemini (smarter, understands semantics) +""" + +import re +import sys +import json +import subprocess +import traceback +from pathlib import Path +from typing import List, Dict, Set, Optional, Tuple + +# Add parent directory to path +sys.path.insert(0, str(Path(__file__).parent)) + +from config import SOURCE_SUMMARY_DIR + + +class SourceFilter: + """Handles source relevance detection and selection""" + + def __init__(self, notebook_id: str): + """ + Initialize source filter for a notebook + + Args: + notebook_id: ID of the notebook + """ + self.notebook_id = notebook_id + self.sources = self._load_sources() + + def _load_sources(self) -> List[Dict[str, str]]: + """Load sources from the summary file""" + summary_path = SOURCE_SUMMARY_DIR / f"{self.notebook_id}.md" + sources = [] + + if not summary_path.exists(): + print(f" ⚠️ No source summary file found for: {self.notebook_id}") + return sources + + try: + with open(summary_path, 'r', encoding='utf-8') as f: + content = f.read() + + # Parse sources from markdown + current_title = None + current_summary_lines = [] + + for line in content.split('\n'): + if line.startswith('### '): + # Save previous source + if current_title: + sources.append({ + 'title': current_title, + 'summary': '\n'.join(current_summary_lines).strip() + }) + current_title = line[4:].strip() + current_summary_lines = [] + elif line.startswith('---') or line.startswith('# ') or line.startswith('**'): + continue + elif current_title: + current_summary_lines.append(line) + + # Don't forget the last one + if current_title: + sources.append({ + 'title': current_title, + 'summary': '\n'.join(current_summary_lines).strip() + }) + + print(f" 📚 Loaded {len(sources)} sources for filtering") + return sources + + except Exception as e: + print(f" ⚠️ Error loading sources: {e}") + return [] + + def get_relevant_sources(self, question: str, use_llm: bool = True, threshold: int = 5) -> List[str]: + """ + Determine which sources are relevant to the question + + Args: + question: The question to ask + use_llm: If True, use Gemini for semantic scoring. If False, use keyword matching. + threshold: Minimum relevance score (1-10) for LLM mode. Default 5. + + Returns: + List of source titles that are relevant + """ + if not self.sources: + return [] # No filtering if no sources loaded + + if use_llm: + return self._get_relevant_sources_llm(question, threshold) + else: + return self._get_relevant_sources_keywords(question) + + def _get_relevant_sources_llm(self, question: str, threshold: int = 5) -> List[str]: + """Use LLM to score source relevance semantically. + + Fallback chain: Gemini CLI → Claude CLI → Keyword matching + """ + print(f" 🤖 Using LLM to score source relevance (threshold: {threshold}/10)...") + + # Build prompt with source list + source_list = "\n".join([ + f"{i+1}. **{s['title']}**: {s['summary'][:200]}..." + for i, s in enumerate(self.sources) + ]) + + prompt = f"""Given this question: "{question}" + +Rate how relevant each source is on a scale of 1-10: +- 10: Directly answers the question +- 7-9: Highly relevant, contains key information +- 4-6: Somewhat relevant, provides context +- 1-3: Not relevant to this question + +Sources: +{source_list} + +IMPORTANT: Respond ONLY with a JSON array of objects, each with "index" (1-based) and "score" (1-10). +Example: [{{"index": 1, "score": 8}}, {{"index": 2, "score": 3}}] + +Be generous with scores - if a source MIGHT contain relevant information, give it at least 5. +Consider semantic relationships, not just keyword matches.""" + + # Try Gemini first + result = self._call_gemini(prompt) + + # If Gemini fails, try Claude + if result is None: + print(f" → Trying Claude CLI...") + result = self._call_claude(prompt) + + # If both fail, fall back to keywords + if result is None: + print(f" → Falling back to keyword matching") + return self._get_relevant_sources_keywords(question) + + # Parse and return results + return self._parse_llm_scores(result, threshold) + + def _call_gemini(self, prompt: str) -> Optional[str]: + """Call Gemini CLI and return response, or None on failure""" + try: + result = subprocess.run( + ['gemini', '-m', 'gemini-2.0-flash', '-p', prompt], + capture_output=True, + text=True, + timeout=30 + ) + if result.returncode == 0 and result.stdout.strip(): + return result.stdout.strip() + return None + except FileNotFoundError: + print(f" ⚠️ Gemini CLI not installed") + return None + except subprocess.TimeoutExpired: + print(f" ⚠️ Gemini timeout") + return None + except Exception as e: + print(f" ⚠️ Gemini error: {e}") + return None + + def _call_claude(self, prompt: str) -> Optional[str]: + """Call Claude CLI and return response, or None on failure""" + try: + # Use claude CLI with -p for print mode (non-interactive) + result = subprocess.run( + ['claude', '-p', prompt, '--model', 'claude-sonnet-4-20250514'], + capture_output=True, + text=True, + timeout=60 + ) + if result.returncode == 0 and result.stdout.strip(): + return result.stdout.strip() + return None + except FileNotFoundError: + print(f" ⚠️ Claude CLI not installed") + return None + except subprocess.TimeoutExpired: + print(f" ⚠️ Claude timeout") + return None + except Exception as e: + print(f" ⚠️ Claude error: {e}") + return None + + def _parse_llm_scores(self, output: str, threshold: int) -> List[str]: + """Parse LLM response and return relevant source titles""" + try: + # Extract JSON from response (might have markdown code blocks) + json_match = re.search(r'\[.*\]', output, re.DOTALL) + if not json_match: + print(f" ⚠️ Could not find JSON in LLM response") + return self._get_relevant_sources_keywords("") + + scores = json.loads(json_match.group()) + + # Build results + relevant = [] + for item in scores: + idx = item.get('index', 0) - 1 # Convert to 0-based + score = item.get('score', 0) + + if 0 <= idx < len(self.sources) and score >= threshold: + relevant.append({ + 'title': self.sources[idx]['title'], + 'score': score + }) + + # Sort by score + relevant.sort(key=lambda x: x['score'], reverse=True) + + if relevant: + print(f" ✓ LLM selected {len(relevant)} relevant sources (out of {len(self.sources)})") + for s in relevant[:5]: + print(f" - {s['title'][:50]}... (score: {s['score']}/10)") + if len(relevant) > 5: + print(f" ... and {len(relevant) - 5} more") + return [s['title'] for s in relevant] + else: + print(f" ℹ️ No sources scored >= {threshold}, using all sources") + return [s['title'] for s in self.sources] + + except json.JSONDecodeError as e: + print(f" ⚠️ JSON parse error: {e}") + return [s['title'] for s in self.sources] + except Exception as e: + print(f" ⚠️ Parse error: {e}") + return [s['title'] for s in self.sources] + + def _get_relevant_sources_keywords(self, question: str) -> List[str]: + """Use keyword matching for source relevance (fallback method)""" + question_lower = question.lower() + + # Extract keywords from the question + stop_words = { + 'the', 'a', 'an', 'is', 'are', 'was', 'were', 'what', 'who', 'when', + 'where', 'why', 'how', 'did', 'do', 'does', 'had', 'has', 'have', + 'this', 'that', 'these', 'those', 'it', 'its', 'in', 'on', 'at', + 'to', 'for', 'of', 'with', 'by', 'from', 'about', 'into', 'and', + 'or', 'but', 'if', 'then', 'so', 'as', 'be', 'been', 'being', + 'can', 'could', 'would', 'should', 'may', 'might', 'must', 'will', + 'me', 'my', 'your', 'his', 'her', 'their', 'our', 'please', 'tell', + 'explain', 'describe', 'give', 'list', 'show', 'find', 'answer' + } + + words = re.findall(r'\b[a-zA-Z\u4e00-\u9fff\u3040-\u309f\u30a0-\u30ff]+\b', question_lower) + keywords = [w for w in words if w not in stop_words and len(w) > 2] + + names = re.findall(r'\b[A-Z][a-zA-Z]+\b', question) + keywords.extend([n.lower() for n in names]) + + cjk_terms = re.findall(r'[\u4e00-\u9fff\u3040-\u309f\u30a0-\u30ff]+', question) + keywords.extend(cjk_terms) + + keywords = list(set(keywords)) + + print(f" 🔍 Keywords extracted: {keywords[:10]}{'...' if len(keywords) > 10 else ''}") + + relevant_sources = [] + + for source in self.sources: + title_lower = source['title'].lower() + summary_lower = source['summary'].lower() + combined_text = title_lower + ' ' + summary_lower + + score = 0 + matched_keywords = [] + + for keyword in keywords: + if keyword in combined_text: + if keyword in title_lower: + score += 3 + else: + score += 1 + matched_keywords.append(keyword) + + if score > 0: + relevant_sources.append({ + 'title': source['title'], + 'score': score, + 'matched': matched_keywords + }) + + relevant_sources.sort(key=lambda x: x['score'], reverse=True) + + if relevant_sources: + threshold = 1 + filtered = [s for s in relevant_sources if s['score'] >= threshold] + + print(f" ✓ Found {len(filtered)} relevant sources (out of {len(self.sources)})") + for s in filtered[:5]: + print(f" - {s['title'][:50]}... (score: {s['score']})") + if len(filtered) > 5: + print(f" ... and {len(filtered) - 5} more") + + return [s['title'] for s in filtered] + else: + print(f" ℹ️ No specific matches found, using all {len(self.sources)} sources") + return [s['title'] for s in self.sources] + + def get_all_source_titles(self) -> List[str]: + """Get all source titles""" + return [s['title'] for s in self.sources] + + +def select_sources_in_browser(page, relevant_titles: List[str], all_titles: List[str]) -> bool: + """ + Select only relevant sources in the NotebookLM browser UI + + Uses Playwright's native click for reliable checkbox interaction. + + Args: + page: Playwright page object + relevant_titles: List of source titles to SELECT (keep checked) + all_titles: List of ALL source titles (to know what to deselect) + + Returns: + True if selection was successful + """ + import time + from browser_utils import StealthUtils + + if not relevant_titles or not all_titles: + print(" ℹ️ No source filtering needed") + return True + + # If all sources are relevant, no need to filter + if set(relevant_titles) == set(all_titles): + print(" ℹ️ All sources are relevant, no filtering needed") + return True + + titles_to_keep = set(relevant_titles) + titles_to_deselect = set(all_titles) - titles_to_keep + + print(f" 🔧 Filtering sources: keeping {len(titles_to_keep)}, removing {len(titles_to_deselect)}") + + # Helper function to check if a title should be deselected + def should_deselect(element_text: str) -> bool: + text = element_text.lower().strip() + for title in titles_to_deselect: + t = title.lower() + if text.find(t[:25]) >= 0 or t.find(text[:25]) >= 0: + return True + return False + + try: + # Use Playwright's native selectors and clicks for reliable interaction + # Find all source containers + containers = page.query_selector_all('.single-source-container') + print(f" Found {len(containers)} source containers") + + deselected = 0 + errors = [] + + for container in containers: + try: + # Get the text content of this source + text = container.inner_text() or "" + short_text = text[:40].replace('\n', ' ') + + # Check if this source should be deselected + if not should_deselect(text): + continue + + # Find the checkbox + checkbox = container.query_selector('mat-checkbox.select-checkbox') + if not checkbox: + errors.append(f"No checkbox: {short_text}") + continue + + # Check if currently checked + checkbox_class = checkbox.get_attribute('class') or "" + if 'mat-mdc-checkbox-checked' not in checkbox_class: + print(f" Already unchecked: {short_text}") + continue + + # Use Playwright's native click on the checkbox + # This properly triggers Angular's change detection + checkbox.click(force=True) + StealthUtils.random_delay(100, 200) + + # Verify + new_class = checkbox.get_attribute('class') or "" + if 'mat-mdc-checkbox-checked' not in new_class: + print(f" ✓ Unchecked: {short_text}") + deselected += 1 + else: + print(f" ⚠ Click didn't work: {short_text}") + + except Exception as e: + errors.append(f"Error: {str(e)[:50]}") + continue + + if deselected > 0: + print(f" ✓ Successfully deselected {deselected} sources") + StealthUtils.random_delay(300, 600) + elif errors: + for err in errors[:3]: + print(f" ⚠️ {err}") + else: + print(f" ℹ️ No sources needed deselecting") + + return True + + except Exception as e: + print(f" ⚠️ Error filtering sources: {e}") + traceback.print_exc() + return False + + +if __name__ == "__main__": + # Test the filter + import argparse + + parser = argparse.ArgumentParser(description='Test source filtering') + parser.add_argument('--notebook-id', required=True, help='Notebook ID') + parser.add_argument('--question', required=True, help='Question to analyze') + + args = parser.parse_args() + + filter = SourceFilter(args.notebook_id) + relevant = filter.get_relevant_sources(args.question) + + print(f"\nRelevant sources for: {args.question}") + for title in relevant: + print(f" - {title}") diff --git a/scripts/video_generator.py b/scripts/video_generator.py new file mode 100644 index 0000000..05b7fa8 --- /dev/null +++ b/scripts/video_generator.py @@ -0,0 +1,582 @@ +#!/usr/bin/env python3 +""" +Video Generator for NotebookLM +Generates Video Overview with custom format, visual style, and instructions +""" + +import argparse +import sys +import re +import time +import traceback +from pathlib import Path + +from patchright.sync_api import sync_playwright + +# Add parent directory to path +sys.path.insert(0, str(Path(__file__).parent)) + +from auth_manager import AuthManager +from notebook_manager import NotebookLibrary +from config import ( + VIDEO_OVERVIEW_CARD_SELECTORS, + VIDEO_CUSTOMIZE_BUTTON_SELECTORS, + VIDEO_FORMAT_SELECTORS, + VIDEO_STYLE_SELECTORS, + VIDEO_INSTRUCTIONS_SELECTORS, + VIDEO_GENERATE_BUTTON_SELECTORS, + VIDEO_GENERATING_SELECTORS, + VIDEO_GENERATION_TIMEOUT, + PAGE_LOAD_TIMEOUT, +) +from browser_utils import ( + BrowserFactory, + StealthUtils, + find_element_with_selectors, + select_language, + get_all_sources_from_ui, +) +from source_filter import select_sources_in_browser + + +def generate_video( + notebook_url: str, + format: str = "explainer", + style: str = "auto", + language: str = None, + instructions: str = None, + sources: list = None, + output: str = None, + headless: bool = True +) -> str: + """ + Generate Video Overview for a NotebookLM notebook and download it. + + Args: + notebook_url: The NotebookLM notebook URL + format: Video format - explainer or brief + style: Visual style - auto, custom, classic, whiteboard, kawaii, anime + language: Language for the video (e.g., "English", "Spanish", "Japanese") + instructions: Custom instructions for AI hosts (the prompt) + sources: List of source names to include (deselects others). If None, uses all sources. + output: Custom output filename + headless: Run browser in headless mode + + Returns: + Path to downloaded video file, or None if failed + """ + auth = AuthManager() + + if not auth.is_authenticated(): + print(" Not authenticated. Run: python scripts/run.py auth_manager.py setup") + return None + + print(" Generating Video Overview...") + print(f" Notebook: {notebook_url}") + print(f" Format: {format}") + print(f" Style: {style}") + if language: + print(f" Language: {language}") + if sources: + print(f" Sources: {len(sources)} selected") + for s in sources[:3]: + print(f" - {s[:50]}...") + if len(sources) > 3: + print(f" ... and {len(sources) - 3} more") + if instructions: + print(f" Instructions: {instructions[:50]}...") + + playwright = None + context = None + download_dir = os.getcwd() + + try: + playwright = sync_playwright().start() + + context = BrowserFactory.launch_persistent_context( + playwright, + headless=headless + ) + + page = context.new_page() + print(" Opening notebook...") + page.goto(notebook_url, wait_until="domcontentloaded") + + # Wait for notebook to load + page.wait_for_url(re.compile(r"^https://notebooklm\.google\.com/"), timeout=60000) + StealthUtils.random_delay(2000, 3000) + + # Handle source selection if specified + if sources: + print(" Selecting specific sources...") + StealthUtils.random_delay(1000, 1500) + + # Get all sources from UI + all_sources = get_all_sources_from_ui(page) + print(f" Found {len(all_sources)} sources in notebook") + + if all_sources: + # Find matching sources (partial match supported) + sources_to_keep = [] + for source_name in sources: + for ui_source in all_sources: + if source_name.lower() in ui_source.lower() or ui_source.lower() in source_name.lower(): + sources_to_keep.append(ui_source) + break + + if sources_to_keep: + print(f" Keeping {len(sources_to_keep)} sources, deselecting {len(all_sources) - len(sources_to_keep)}") + select_sources_in_browser(page, sources_to_keep, all_sources) + StealthUtils.random_delay(500, 1000) + else: + print(" Warning: No matching sources found, using all sources") + + # First, check if we need to click on "Studio" tab (responsive layout) + studio_tab = page.query_selector('button:has-text("Studio"), [role="tab"]:has-text("Studio")') + if studio_tab: + print(" Found Studio tab, clicking...") + studio_tab.click() + StealthUtils.random_delay(1000, 1500) + + # Find the Video Overview card in Studio panel + print(" Looking for Video Overview in Studio panel...") + + # Click on the pencil/customize icon next to Video Overview + customize_button, selector = find_element_with_selectors( + page, VIDEO_CUSTOMIZE_BUTTON_SELECTORS, timeout=10000 + ) + + if not customize_button: + # Try clicking on Video Overview card directly + video_card, card_selector = find_element_with_selectors( + page, VIDEO_OVERVIEW_CARD_SELECTORS, timeout=5000 + ) + if video_card: + print(f" Found Video Overview card: {card_selector}") + # Look for pencil icon within or near the card + pencil = page.query_selector(f'{card_selector} button, {card_selector} + button') + if pencil: + pencil.click() + StealthUtils.random_delay(1000, 1500) + else: + # Click the card itself + video_card.click() + StealthUtils.random_delay(1000, 1500) + else: + print(" Could not find Video Overview option") + return None + else: + print(f" Found customize button: {selector}") + customize_button.click() + StealthUtils.random_delay(1000, 1500) + + # Wait for customize dialog to appear + print(" Waiting for Customize Video Overview dialog...") + StealthUtils.random_delay(1000, 2000) + + # Select format if specified + if format: + print(f" Selecting format: {format}") + format_names = { + 'explainer': 'Explainer', + 'brief': 'Brief' + } + format_name = format_names.get(format, format) + + # Use Playwright locators to find and click format card + format_selected = False + try: + # Try finding a div that contains exactly the format name as a child div + format_card = page.locator(f'div:has(div:text-is("{format_name}"))').first + if format_card.is_visible(timeout=2000): + format_card.click() + format_selected = True + except Exception: + pass + + if not format_selected: + try: + # Fallback: find the text and click its parent card + format_el = page.get_by_text(format_name, exact=True).first + if format_el.is_visible(timeout=2000): + format_el.locator('..').locator('..').click() + format_selected = True + except Exception: + pass + + if format_selected: + StealthUtils.random_delay(500, 800) + print(f" Selected: {format_name}") + else: + print(f" Warning: Could not find format card for {format_name}") + + # Select visual style if specified + if style and style in VIDEO_STYLE_SELECTORS: + print(f" Selecting visual style: {style}") + style_selectors = VIDEO_STYLE_SELECTORS[style] + style_element, sty_selector = find_element_with_selectors(page, style_selectors, timeout=3000) + if style_element: + style_element.click() + StealthUtils.random_delay(500, 800) + print(f" Selected: {sty_selector}") + + # Select language if specified + if language: + select_language(page, language) + + # Add custom instructions if provided + if instructions: + print(" Adding custom instructions...") + instructions_input, instr_selector = find_element_with_selectors( + page, VIDEO_INSTRUCTIONS_SELECTORS, timeout=3000 + ) + if instructions_input: + instructions_input.click() + StealthUtils.random_delay(200, 400) + # Clear existing content and type new instructions + instructions_input.fill("") + StealthUtils.random_delay(100, 200) + instructions_input.fill(instructions) + StealthUtils.random_delay(500, 800) + print(" Instructions added") + else: + print(" Could not find instructions field") + + # Click Generate button + print(" Looking for Generate button...") + generate_button, gen_selector = find_element_with_selectors( + page, VIDEO_GENERATE_BUTTON_SELECTORS, timeout=5000 + ) + + if not generate_button: + # Try more specific selectors + generate_button = page.query_selector('button.generate-button, button[type="submit"], button:has-text("Generate")') + + if not generate_button: + print(" Could not find Generate button") + page.screenshot(path="/tmp/notebooklm_video_debug.png") + print(" Debug screenshot saved to /tmp/notebooklm_video_debug.png") + return None + + print(f" Found Generate button") + + # Wait for button to be enabled (it may be disabled initially) + print(" Waiting for Generate button to be enabled...") + try: + page.wait_for_selector('button:has-text("Generate"):not([disabled])', timeout=10000) + except Exception: + print(" Generate button seems disabled, trying to click anyway...") + + StealthUtils.random_delay(500, 1000) + + # Scroll into view and wait for stability + generate_button.scroll_into_view_if_needed() + StealthUtils.random_delay(300, 500) + + try: + generate_button.click(timeout=10000) + except Exception as click_error: + print(f" Click failed: {click_error}") + # Try JavaScript click as fallback + try: + page.evaluate('(el) => el.click()', generate_button) + print(" Used JavaScript click fallback") + except Exception as js_error: + print(f" JavaScript click also failed: {js_error}") + page.screenshot(path="/tmp/notebooklm_generate_debug.png") + print(" Debug screenshot saved to /tmp/notebooklm_generate_debug.png") + return None + + print(" Started video generation...") + + # Wait for generation to complete (can take several minutes) + print(" Waiting for video generation (this may take several minutes)...") + generation_start = time.time() + generation_timeout = VIDEO_GENERATION_TIMEOUT / 1000 # Convert to seconds + + # First, wait a few seconds for generation indicator to appear + StealthUtils.random_delay(3000, 5000) + + last_print_time = 0 + while time.time() - generation_start < generation_timeout: + # Check if still generating by looking for the generation indicator + generating = False + + for selector in VIDEO_GENERATING_SELECTORS: + try: + gen_indicator = page.query_selector(selector) + if gen_indicator and gen_indicator.is_visible(): + generating = True + break + except Exception: + continue + + if generating: + elapsed = int(time.time() - generation_start) + # Print status every 30 seconds + if elapsed - last_print_time >= 30: + print(f" Still generating... ({elapsed}s)") + last_print_time = elapsed + time.sleep(5) + continue + else: + # Generation indicator disappeared - check if new video appeared + print(" Generation indicator disappeared, verifying completion...") + StealthUtils.random_delay(2000, 3000) + + # Double-check that generation is really complete + still_generating = False + for selector in VIDEO_GENERATING_SELECTORS: + try: + gen_indicator = page.query_selector(selector) + if gen_indicator and gen_indicator.is_visible(): + still_generating = True + break + except Exception: + continue + + if not still_generating: + print(" Video generation complete!") + break + else: + # False alarm, continue waiting + time.sleep(5) + continue + + else: + print(" Generation timeout exceeded") + return None + + # Allow UI to settle after generation + print(" Waiting for UI to settle...") + StealthUtils.random_delay(3000, 5000) + + # Find and click the three-dot menu on the most recently generated video + print(" Looking for video menu...") + + # Use JavaScript to find the correct menu button in the Studio panel + menu_clicked = page.evaluate(""" + () => { + // Find all video items - they have metadata like "X source · Xm ago" + const items = document.querySelectorAll('div'); + let targetButton = null; + let mostRecent = null; + + for (const item of items) { + const text = item.textContent || ''; + // Look for items that look like generated video + if ((text.includes('source') && (text.includes('ago') || text.includes('just now'))) || + (text.includes('Explainer') && text.includes('source')) || + (text.includes('Brief') && text.includes('source'))) { + + // Check if this item is in the right part of the page (Studio panel) + const rect = item.getBoundingClientRect(); + if (rect.x < 600) continue; // Studio panel is on the right + + // Find the three-dot menu button within this item + const menuButtons = item.querySelectorAll('button'); + for (const btn of menuButtons) { + const btnRect = btn.getBoundingClientRect(); + if (btnRect.width < 50 && btnRect.height < 50) { + const ariaLabel = btn.getAttribute('aria-label') || ''; + const innerHTML = btn.innerHTML || ''; + if (ariaLabel.includes('option') || ariaLabel.includes('more') || + innerHTML.includes('more_vert') || innerHTML.includes('⋮')) { + targetButton = btn; + break; + } + } + } + + if (targetButton) { + if (text.includes('just now') || text.includes('1m ago') || text.includes('2m ago')) { + mostRecent = targetButton; + break; + } + if (!mostRecent) mostRecent = targetButton; + } + } + } + + if (mostRecent) { + mostRecent.click(); + return true; + } + + // Fallback + const allButtons = document.querySelectorAll('button'); + for (const btn of allButtons) { + const rect = btn.getBoundingClientRect(); + if (rect.x < 600) continue; + + const ariaLabel = btn.getAttribute('aria-label') || ''; + if (ariaLabel.toLowerCase().includes('option') || ariaLabel.toLowerCase().includes('more')) { + btn.click(); + return true; + } + } + + return false; + } + """) + + if not menu_clicked: + print(" Could not find video menu button") + page.screenshot(path="/tmp/notebooklm_menu_debug.png") + print(" Debug screenshot saved to /tmp/notebooklm_menu_debug.png") + return None + + print(" Clicked menu button") + StealthUtils.random_delay(1000, 1500) + + # Click Download option from the menu + print(" Looking for Download option...") + print(" Waiting for download...") + + try: + with page.expect_download(timeout=60000) as download_info: + # Click Download in the menu + download_clicked = page.evaluate(""" + () => { + const menuItems = document.querySelectorAll('[role="menuitem"], li, button, span'); + for (const item of menuItems) { + const text = item.textContent?.trim() || ''; + if (text === 'Download') { + const rect = item.getBoundingClientRect(); + if (rect.width > 0 && rect.height > 0) { + item.click(); + return true; + } + } + } + return false; + } + """) + + if not download_clicked: + raise Exception("Could not find Download option") + + download = download_info.value + suggested_name = download.suggested_filename + print(f" Download started: {suggested_name}") + + if output: + download_path = f"{download_dir}/{output}" + else: + download_path = f"{download_dir}/{suggested_name}" + + download.save_as(download_path) + print(f" Saved to: {download_path}") + + except Exception as dl_error: + print(f" Download error: {dl_error}") + page.screenshot(path="/tmp/notebooklm_download_menu_debug.png") + print(" Debug screenshot saved to /tmp/notebooklm_download_menu_debug.png") + download_path = None + + if download_path: + print(f" Video downloaded successfully: {download_path}") + return download_path + else: + print(" Download failed") + return None + + except Exception as e: + print(f" Error: {e}") + traceback.print_exc() + return None + + finally: + if context: + try: + context.close() + except Exception: + pass + if playwright: + try: + playwright.stop() + except Exception: + pass + + +def main(): + parser = argparse.ArgumentParser(description='Generate NotebookLM Video Overview') + + # Notebook selection (mutually exclusive) + notebook_group = parser.add_mutually_exclusive_group() + notebook_group.add_argument('--notebook-url', help='NotebookLM notebook URL') + notebook_group.add_argument('--notebook-id', help='Notebook ID from library') + + # Video customization + parser.add_argument('--format', choices=['explainer', 'brief'], + default='explainer', help='Video format (default: explainer)') + parser.add_argument('--style', choices=['auto', 'custom', 'classic', 'whiteboard', 'kawaii', 'anime'], + default='auto', help='Visual style (default: auto)') + parser.add_argument('--language', help='Language for the video (e.g., English, Spanish, Japanese)') + parser.add_argument('--instructions', help='Custom instructions/prompt for AI hosts') + parser.add_argument('--sources', help='Comma-separated list of source names to include (deselects others)') + parser.add_argument('--output', help='Custom output filename') + parser.add_argument('--show-browser', action='store_true', help='Show browser window') + + args = parser.parse_args() + + # Resolve notebook URL + library = NotebookLibrary() + + if args.notebook_id: + notebook = library.get_notebook(args.notebook_id) + if not notebook: + print(f"Notebook not found: {args.notebook_id}") + return 1 + notebook_url = notebook['url'] + notebook_name = notebook['name'] + elif args.notebook_url: + notebook_url = args.notebook_url + notebook_name = "Unknown" + else: + # Use active notebook + active = library.get_active_notebook() + if active: + notebook_url = active['url'] + notebook_name = active['name'] + print(f"Using active notebook: {notebook_name}") + else: + print("No notebook specified and no active notebook set") + return 1 + + # Parse sources if provided + sources_list = None + if args.sources: + sources_list = [s.strip() for s in args.sources.split(',')] + + result = generate_video( + notebook_url=notebook_url, + format=args.format, + style=args.style, + language=args.language, + instructions=args.instructions, + sources=sources_list, + output=args.output, + headless=not args.show_browser + ) + + if result: + print("\n" + "=" * 60) + print(" Video Overview Generated") + print("=" * 60) + print(f"File: {result}") + print(f"Format: {args.format}") + print(f"Style: {args.style}") + if args.language: + print(f"Language: {args.language}") + if args.instructions: + print(f"Instructions: {args.instructions[:50]}...") + print("=" * 60) + return 0 + else: + print("\n Failed to generate video") + return 1 + + +if __name__ == "__main__": + sys.exit(main())