Skip to content
vnk8071Public

About

Recent job postings in Vietnam and Global Hub

Resources

Contributing

Stars

15 stars

Watchers

0 watching

Forks

Latest commit

ย 

History

230 Commits

Folders and files

Repository files navigation

๐ŸŽฏ Good Jobs

Docker FastAPI Python TypeScript Redis Playwright License PRs welcome

One search box โ†’ every job board. Good Jobs is a self-hosted job search aggregator that scrapes recently posted jobs from 21 sources and streams results to your browser in real time over SSE โ€” with AI-powered intent detection, skill extraction, vector-based related-job supplements, and a permanent Redis cache warmed every 2 hours.

๐Ÿ‡ป๐Ÿ‡ณ Vietnam is the core market โ€” LinkedIn, ITViec, TopCV, VietnamWorks, CareerViet, TopDev, JobsGo, CareerLink, Glints, ViecOi, Indeed, TechJobs, Xรณm Jobs. ๐ŸŒ Opt-in global mode covers the US, UK, and Singapore โ€” LinkedIn, Indeed, RemoteOK, We Work Remotely, Glassdoor, USAJOBS, Dice. Just type the city; the country is inferred automatically ("nyc" โ†’ New York โ†’ US sources).

๐Ÿ—“๏ธ Release Timeline

Date Feature
2026-11-17 LinkedIn auto-posting โ€” each published news article is posted automatically to the Good Jobs company page via the LinkedIn REST API (Community Management API, w_organization_social)
2026-11-07 News desk, homepage filters & All Jobs browser โ€” goodjobs.io.vn/news publishes AI-generated Vietnamese articles twice a day (Workers AI + D1, RSS, news sitemap); the homepage feed merges the live Redis cache with the durable Cloudflare D1 archive so every job ever scraped stays browsable, with category/location/source filters and a jobs-per-page selector
2026-10-03 MCP server & career-ops submodule โ€” mcp/goodjobs_mcp.py exposes search, recent jobs, semantic search, keyword classification, city normalization, and cache status as read-only tools for MCP clients such as Claude Desktop and Claude Code; career-ops (branch feat/goodjobs-provider) vendored at thirdparty/career-ops with the goodjobs provider tests
2026-08-30 career-ops integration โ€” career-ops added Good Jobs as a providers/goodjobs.mjs source; ongoing collaboration on integration depth (/recent-jobs, /search-semantic) and a self-hosting guide in career-ops-hq/career-ops#3521
2026-08-22 Global search โ€” opt-in coverage for the US, UK, and Singapore (LinkedIn, Indeed, RemoteOK, We Work Remotely, Glassdoor, USAJOBS, Dice) alongside the Vietnam market
2026-04-11 CV / skill matching โ€” paste a CV or skill list and AI intent detection maps it to a canonical job keyword, with results scored against it
2026-03-25 Summary description โ€” background LLM summarization of long job descriptions, surfaced inline in the UI

๐Ÿ”Ž The Job Boards

21 sources, one query. Every board below is scraped live on each search and merged into a single deduplicated feed.

๐Ÿ‡ป๐Ÿ‡ณ Vietnam

Board Best known for Scraping
LinkedIn International companies & senior roles Headless Chromium, full description enrichment
ITViec IT jobs, transparent salary ranges Static HTML, full enrichment
TopCV Largest VN general job board Static HTML, full enrichment
VietnamWorks Established enterprises, HR-first Static HTML, full enrichment
TopDev Developer-focused roles Static HTML, full enrichment
JobsGo Startups & SMEs Static HTML, full enrichment
CareerLink Japanese-affiliated companies Static HTML, full enrichment
CareerViet Broad coverage across industries Static HTML, full enrichment
Glints Southeast Asia startups Headless Chromium, full enrichment
ViecOi Foreign-invested firms in VN Static HTML, full enrichment
Indeed VN Aggregated listings at scale Static HTML, inline enrichment
TechJobs IT job aggregator, company career pages Public website, server-rendered data
Xรณm Jobs Data & AI roles Public website, static HTML
Google Careers Google's own openings in Vietnam Headless Chromium

๐ŸŒ Global (US ยท UK ยท SG)

Type any supported city โ€” new york, london, singapore โ€” and the right boards are chosen automatically.

Board Best known for Scraping
LinkedIn All three markets, one scraper Headless Chromium
Indeed Per-country domains, huge volume Static HTML
RemoteOK Fully remote dev jobs Public JSON API
We Work Remotely Remote-first companies Public RSS feed
Glassdoor Salary data + company reviews Headless Chromium, list-page only
USAJOBS US federal government jobs Official API
Dice US tech contracting & full-time Static HTML

Every posting keeps its original link โ€” Good Jobs never republishes, it points you home.

โœจ Highlights

โšก Real-time streaming Results arrive job-by-job via SSE as each scraper finishes โ€” no waiting for the slowest board
๐Ÿค– AI intent detection Paste a CV, a skill list, or a vague phrase โ€” an LLM maps it to a canonical job keyword (with Vietnamese โ†” English translation)
๐Ÿง  Vector supplement Related jobs from other cached keywords are retrieved via embeddings and appended on cache misses
๐Ÿท๏ธ Skill extraction Tech skills auto-tagged per job from title + description, rendered as pills in the UI
๐Ÿ” Self-warming cache ~30 keywordร—location pairs re-scraped every 2 hours; new jobs merged by link dedup; jobs pruned after 14 days
๐ŸŒ Free-text locations No dropdowns. "hcmc" โ†’ Ho Chi Minh City, typos corrected, country inferred from the city
๐Ÿšฆ Fair concurrency A semaphore caps concurrent scrapes; user requests jump ahead of background warmups
๐Ÿงฎ All Jobs filters Homepage feed merges the Redis cache with the D1 archive; filter by category, location, or source, with a jobs-per-page selector
๐Ÿ“ฐ News desk /news โ€” AI-generated Vietnamese labor-market articles, auto-published twice a day and cross-posted to LinkedIn

๐Ÿš€ Quick Start

docker compose up
Service URL
Frontend http://localhost
Backend API http://localhost:8000
API docs (Swagger) http://localhost:8000/docs

That's it โ€” type a job title (or paste your CV) and a city, and watch results stream in.

๐Ÿ“ก API

Endpoint Method Description
/scrape-stream POST Main SSE endpoint โ€” streams cached, data, vector-results, and enrichment events
/scrape POST Blocking scrape (returns full job list)
/classify-input POST LLM intent detection: free text โ†’ canonical keyword
/normalize-city POST Typo correction / abbreviation expansion for cities
/suggest-query POST Query suggestions
/search-semantic GET Vector search across cached jobs
/recent-jobs, /stats GET VN-facing views over the cache
/cache/status, /cache/overview GET Cache keys with fetched_ago and job_count
/cache/scrape GET Trigger a scrape for a specific key
/warmup/keywords GET / POST Inspect / extend warmup keyword pairs
/health GET Health probe

๐Ÿ—๏ธ Architecture

User request
     โ”‚
     โ–ผ
 Cache hit? โ”€โ”€yesโ”€โ”€โ–ถ Stream cached jobs instantly (event: cached)
     โ”‚                       โ”‚
     no                      โ–ผ
     โ”‚               Vector supplement from related
     โ–ผ               cached keywords (event: vector-results)
 Semaphore
  queue
     โ”‚
     โ–ผ
Phase 1 โ”€โ”€ All scrapers for the resolved country run concurrently
     โ”‚      Results streamed as each finishes (event: data)
     โ”‚
     โ–ผ
Phase 2 โ”€โ”€ Per-site description enrichment โ€” VN sources (LinkedIn, TopCV,
           ITViec, TopDev, JobsGo, CareerLink, Glints, ViecOi,
           VietnamWorks, CareerViet) or global sources (LinkedIn only;
           RemoteOK/We Work Remotely return full descriptions in Phase 1,
           Glassdoor is list-page-only, Indeed enriches inline in Phase 1)
           โ€” streamed job-by-job in background

Caching & warmup

  • Results are cached permanently in Redis by (keyword, location); individual jobs are pruned after 14 days (RECENT_DAYS). Global entries are namespaced (jobs:{country}:โ€ฆ) and excluded from VN-facing views and warmup.
  • ๐Ÿ“ Direct submissions: Headhunters can submit a job via POST /submissions; after admin review (/admin/submissions/*, gated by ADMIN_SECRET) it's merged into search results as source "Direct". Stored in Redis under submissions:{pending,approved,rejected}, never written into the keyword/location cache.
  • Background warmup: ~30 VN keywordร—location pairs every 2 hours, checked every 10 minutes. LinkedIn uses f_TPR=r7200 for incremental 2-hour windows; other scrapers refetch the full window. New jobs merge into existing caches, deduplicated by link.

Project structure

goodjobs/
โ”œโ”€โ”€ backend/
โ”‚   โ”œโ”€โ”€ main.py                    # FastAPI app, routes, semaphore queue, warmup scheduler
โ”‚   โ””โ”€โ”€ src/
โ”‚       โ”œโ”€โ”€ constants.py           # HEADERS, CHROMIUM_ARGS, RECENT_DAYS, skill patterns
โ”‚       โ”œโ”€โ”€ models.py              # Pydantic models (Job, ScrapeRequest)
โ”‚       โ”œโ”€โ”€ matching.py            # Title matching, skill extraction, level tagging
โ”‚       โ”œโ”€โ”€ cache.py               # Redis helpers (VN/global namespacing, merge, prune)
โ”‚       โ”œโ”€โ”€ vector.py              # Embeddings, vector index, supplement search
โ”‚       โ”œโ”€โ”€ intent.py              # LLM intent detection, city normalization
โ”‚       โ”œโ”€โ”€ warmup.py              # Warmup scheduler, per-site enrich pipeline
โ”‚       โ”œโ”€โ”€ summarizer.py          # On-demand description summariser
โ”‚       โ”œโ”€โ”€ ratelimit.py           # Per-IP rate limiting middleware
โ”‚       โ”œโ”€โ”€ submissions.py         # Headhunter job-submission store (pending/approved/rejected)
โ”‚       โ”œโ”€โ”€ graphql_schema.py      # GraphQL schema (jobs query)
โ”‚       โ””โ”€โ”€ scrapers/              # One module per job board (see tables above)
โ”œโ”€โ”€ frontend/
โ”‚   โ”œโ”€โ”€ index.html                 # UI layout and styles
โ”‚   โ”œโ”€โ”€ privacy/, terms/, contact/, admin/, post-job/
โ”‚   โ””โ”€โ”€ src/
โ”‚       โ”œโ”€โ”€ main.ts                # SSE client, fetch orchestration
โ”‚       โ”œโ”€โ”€ api.ts                 # Stream parser, AI classify/normalize calls
โ”‚       โ”œโ”€โ”€ ui.ts                  # Table rendering, modal, skill pills, badges
โ”‚       โ””โ”€โ”€ types.ts               # Shared TypeScript interfaces
โ”œโ”€โ”€ thirdparty/
โ”‚   โ””โ”€โ”€ career-ops/                # Git submodule: career-ops (consumes this API via providers/goodjobs.mjs)
โ”œโ”€โ”€ docker-compose.yml             # Local development
โ”œโ”€โ”€ docker-compose.server.yml      # Production (+ cloudflared tunnel)
โ””โ”€โ”€ .claude/skills/                # Agent skills for common repo tasks

๐Ÿ› ๏ธ Extending

Common tasks (also available to coding agents as skills):

  • Add a scraper: create backend/src/scrapers/<site>.py, export from backend/src/scrapers/__init__.py, register in backend/main.py โ†’ see add-scraper
  • Add a skill keyword: update _SKILL_PATTERNS in backend/src/constants.py
  • Add a warmup keyword: POST /warmup/keywords or edit _WARMUP_KEYWORDS in backend/src/warmup.py
  • Check cache state: GET /cache/status

โ˜๏ธ Deployment

Pushing to main triggers GitHub Actions:

  1. Backend โ€” Docker image built โ†’ pushed to Docker Hub โ†’ deployed over SSH (backend.yml)
  2. Frontend โ€” Built and published to GitHub Pages (frontend.yml)

Required repository secrets:

Secret Purpose
DOCKERHUB_USERNAME / DOCKERHUB_TOKEN Docker Hub login
SERVER_HOST / SERVER_USER / SERVER_SSH_KEY / SERVER_PASSWORD / SERVER_PORT SSH deploy target
VITE_API_URL Backend API URL baked into the frontend build
CLOUDFLARE_TUNNEL_TOKEN Cloudflare tunnel auth token

A separately-operated instance runs outside this CI/CD flow, with its own cron watchdog (restarts on DNS breakage, a crashed container, a wedged event loop, or a stalled scrape scheduler via GET /warmup/heartbeat). That script is versioned at ops/watchdog.sh for review; deploying a change to it is a manual copy to that host, not part of this repo's automation.

๐Ÿค– Agent Skills

This repo ships agent skills so coding agents (Claude Code, opencode, โ€ฆ) can work on it effectively:

  • add-scraper โ€” end-to-end recipe for adding a new job board
  • debug-scrapers โ€” diagnosing broken selectors, anti-bot blocks, and empty results
  • cache-ops โ€” inspecting, flushing, and reasoning about the Redis cache
  • run-and-verify โ€” start the stack and verify changes end-to-end

๐Ÿงฉ Third-party: career-ops

career-ops is vendored as a git submodule at thirdparty/career-ops (branch feat/goodjobs-provider). It consumes Good Jobs as a source via providers/goodjobs.mjs, a thin client that calls the public POST /scrape endpoint and normalizes the results.

# Clone with the submodule
git clone --recurse-submodules https://github.com/vnk8071/goodjobs.git

# Or, in an existing clone
git submodule update --init --recursive

Test the provider:

cd thirdparty/career-ops
node --test tests/providers/goodjobs.test.mjs

The unit tests cover URL resolution, request building, normalization, and retry behavior. They do not hit the network. The provider's default backend is https://api.goodjobs.io.vn. Because career-ops' SSRF guard blocks loopback addresses, a local docker compose up instance cannot be used as a source; point api: at a publicly reachable HTTPS deployment instead.

To update the submodule to the latest branch commit:

git submodule update --remote thirdparty/career-ops

๐Ÿ”Œ MCP Server

mcp/goodjobs_mcp.py exposes Good Jobs to MCP clients (Claude Code, Claude Desktop, โ€ฆ) as read-only tools over stdio. It calls the public backend by default, so no local stack is needed.

Tool Wraps Purpose
search_jobs POST /scrape Live search by keyword and location; newest first
recent_jobs GET /recent-jobs Latest postings across cached searches, optionally one board
semantic_search GET /search-semantic Match by meaning, e.g. a pasted CV summary
classify_keyword POST /classify-input Turn free text or a CV into a canonical job title
normalize_city POST /normalize-city Fix city typos and abbreviations ("hcmc" โ†’ Ho Chi Minh City)
cache_status GET /cache/status Cache freshness and job counts per keyword and location

Admin and submission routes are deliberately not exposed.

cd mcp
python3 -m venv .venv && . .venv/bin/activate
pip install -r requirements.txt
python -m pytest -q test_goodjobs_mcp.py   # unit tests, no network

Register it with Claude Code:

claude mcp add goodjobs -- /absolute/path/to/goodjobs/mcp/.venv/bin/python /absolute/path/to/goodjobs/mcp/goodjobs_mcp.py

Set GOODJOBS_API_URL to point at another deployment. Use http://localhost:8000 for a local docker compose up. The first search_jobs call on an uncached keyword can take a minute, because it runs a full multi-board scrape.

๐Ÿค Contributing

PRs are welcome! The most valuable contribution is a new job board for your country โ€” see CONTRIBUTING.md and the add-scraper guide.

๐Ÿ“„ License

MIT โ€” each job posting links back to its original source. See also our privacy and terms pages.

About

Recent job postings in Vietnam and Global Hub

Resources

Contributing

Stars

15 stars

Watchers

0 watching

Forks

Contributors

Languages