Skip to content

⚡ Bolt: Optimize skill categorization in LinkedInSync #326

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
anchapin wants to merge 1 commit into
base: main
Choose a base branch
from
Open

⚡ Bolt: Optimize skill categorization in LinkedInSync #326

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 4 additions & 0 deletions .jules/bolt.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -13,3 +13,7 @@
## 2025-02-18 - Regex Pre-compilation in Hot Paths
**Learning:** Re-compiling regexes inside a frequently called function (like `latex_escape` which runs for every string) creates significant overhead. Pre-compiling them at module level yielded a ~3.2x speedup.
**Action:** Always look for regex compilations inside loops or frequently called functions and move them to module level constants.

## 2024-05-29 - Regex Alternate Pattern Pre-compilation
**Learning:** Re-compiling simple keyword search loops into single alternate regex patterns (`re.compile(r"kw1|kw2")`) and moving them to the module level turns O(skills * categories * keywords) evaluations into O(skills * categories) pre-compiled searches, dramatically improving categorization routines in places like `LinkedInSync._categorize_skills` by > 30x.
**Action:** When categorizing elements against static word lists inside a loop, pre-compile the lists as a single alternated regex pattern at the module level.
215 changes: 123 additions & 92 deletions cli/integrations/linkedin.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -7,6 +7,127 @@
from pathlib import Path
from typing import Any, Dict, List, Optional

# Pre-compile regex patterns for skill categorization
_LANGUAGE_PATTERN = re.compile(
r"\b(?:"
+ "|".join(
[
"python",
"javascript",
"java",
"go",
"rust",
r"c\+\+",
"c#",
"ruby",
"php",
"swift",
"kotlin",
"scala",
"haskell",
"typescript",
"sql",
]
)
+ r")\b"
)

_FRAMEWORK_PATTERN = re.compile(
r"\b(?:"
+ "|".join(
[
"django",
"flask",
"fastapi",
"spring",
"react",
"angular",
"vue",
"express",
"rails",
"laravel",
r"next\.js",
"nuxt",
"tensorflow",
"pytorch",
"keras",
"pandas",
"numpy",
"scikit",
"langchain",
]
)
+ r")\b"
)

_CLOUD_PATTERN = re.compile(
r"\b(?:"
+ "|".join(
[
"aws",
"azure",
"gcp",
"google cloud",
"amazon web services",
"heroku",
"vercel",
"netlify",
"digitalocean",
"linode",
]
)
+ r")\b"
)

_DATABASE_PATTERN = re.compile(
r"\b(?:"
+ "|".join(
[
"postgres",
"postgresql",
"mysql",
"mongodb",
"redis",
"sqlite",
"oracle",
"sql server",
"cassandra",
"elasticsearch",
"dynamodb",
]
)
+ r")\b"
)

_TOOL_PATTERN = re.compile(
r"\b(?:"
+ "|".join(
[
"docker",
"kubernetes",
"git",
"github",
"gitlab",
"jenkins",
"circleci",
"terraform",
"ansible",
"nagios",
"grafana",
"prometheus",
]
)
+ r")\b"
)

_SKILL_PATTERNS = [
(_LANGUAGE_PATTERN, "languages"),
(_FRAMEWORK_PATTERN, "frameworks"),
(_CLOUD_PATTERN, "cloud_platforms"),
(_DATABASE_PATTERN, "databases"),
(_TOOL_PATTERN, "tools"),
]


class LinkedInSync:
"""Sync LinkedIn profile data to/from resume.yaml."""
Expand Down Expand Up @@ -442,103 +563,13 @@ def _categorize_skills(self, skills: List[str]) -> Dict[str, List[str]]:
"other": [],
}

language_keywords = [
"python",
"javascript",
"java",
"go",
"rust",
"c\\+\\+",
"c#",
"ruby",
"php",
"swift",
"kotlin",
"scala",
"haskell",
"typescript",
"sql",
]

framework_keywords = [
"django",
"flask",
"fastapi",
"spring",
"react",
"angular",
"vue",
"express",
"rails",
"laravel",
"next\\.js",
"nuxt",
"tensorflow",
"pytorch",
"keras",
"pandas",
"numpy",
"scikit",
"langchain",
]

cloud_keywords = [
"aws",
"azure",
"gcp",
"google cloud",
"amazon web services",
"heroku",
"vercel",
"netlify",
"digitalocean",
"linode",
]

database_keywords = [
"postgres",
"postgresql",
"mysql",
"mongodb",
"redis",
"sqlite",
"oracle",
"sql server",
"cassandra",
"elasticsearch",
"dynamodb",
]

tool_keywords = [
"docker",
"kubernetes",
"git",
"github",
"gitlab",
"jenkins",
"circleci",
"terraform",
"ansible",
"nagios",
"grafana",
"prometheus",
]

for skill in skills:
skill_lower = skill.lower()

# Check each category (use first match)
matched = False
patterns = [
(language_keywords, "languages"),
(framework_keywords, "frameworks"),
(cloud_keywords, "cloud_platforms"),
(database_keywords, "databases"),
(tool_keywords, "tools"),
]

for keywords, category in patterns:
if any(re.search(rf"\b{kw}\b", skill_lower) for kw in keywords):
for pattern, category in _SKILL_PATTERNS:
if pattern.search(skill_lower):
categories[category].append(skill)
matched = True
break
Expand Down
Loading