Spaces:
Running
Running
Commit History
-
Stop the search chain treating an empty answer as an answer
Claude +1: Claude Opus 5Unverified signature`/serp/search` returned zero results and the explanation "All backends are rate-limited." - a claim with nothing behind it. Seen once against the live deployment, then not reproducible, which is itself the problem: the message described a diagnosis rather than what happened. Two causes, both fixed here. A backend that returned an empty list without raising ended the fallback chain and was recorded as a success. Brave does that whenever its block page lacks the word "suspicious", and Bing whenever no result item yields a title and href - so one quiet backend could stop the chain before a working one was ever tried. This is the same bug class fixed for DuckDuckGo in d341a7a, which raises on empty; the other two never got the same treatment. Fixing it in the chain rather than in each scraper covers every backend, including any added later, and leaves the single-backend endpoints free to report an honest empty result. An empty answer still counts neither for nor against the backend, as …73c1aa6 -
Classify a blocked Scholar page before waiting, not after
Claude +1: Claude Opus 5Unverified signature#8 made the classification correct but left it arriving 30s late. Measured against the deployment: the error now says "Google Scholar blocked the request", but it still takes 30.7s, because every signal was consulted only in the selector-timeout handler. All of them are available the moment navigation completes, so they are checked there instead. A page carrying no results at all is decidable immediately: Scholar is server-rendered, so a genuine results page already has div.gs_ri in the DOM. That "no results" guard is also what makes the wording check safe this early - and it caught a real false positive during this change. Without it, a legitimate search for "unusual traffic" returns papers whose snippets quote the interstitial and was misreported as a block. The test written for that risk failed against the first version of this commit, which is the whole reason it exists. The block exception now carries the evidence that classified it - redirect URL, challenge element, or wording…8a62982 -
Make a failed Scholar scrape say what it actually got
Claude +1: Claude Opus 5Unverified signatureThe block detection added in #7 did not work. It waited for challenge elements I had guessed at without being able to observe the real page - Scholar is unreachable from the development sandbox - and matched none of them, so a blocked scrape still cost the full 30s timeout and still reported a bare "Timeout 30000ms exceeded". Guessing again would repeat the mistake, so this changes the shape of the failure instead. An unclassifiable page now raises GoogleScholarUnavailableError carrying the final URL, the page title and a text excerpt. Whatever Google is actually serving, the next live call identifies it - no more deploy-and-hope rounds. The error path is best-effort throughout: failing to read the page must not replace the original problem with a new one. Two real defects in the first attempt, both now covered: - Challenge elements were matched by visibility. reCAPTCHA renders in an iframe whose container is commonly zero-height until it loads, so the element can be attached bu…addbf67 -
Detect Google Scholar's block page, and unwrap Bing's redirect links
Claude +1: Claude Opus 5Unverified signatureBoth found by running the live smoke test against the deployed MCP server, not by the suite - the division of labour CONTRIBUTING.md already describes. Google Scholar serves an anti-bot interstitial to the Space's IP instead of results. That page has no `div.gs_ri`, so the scraper sat in `wait_for_selector` for the full 30 seconds and then reported a bare "Timeout 30000ms exceeded" - slow for every caller, and worded like a broken selector rather than a blocked deployment. It now waits for the results selector and Google's challenge elements together, so a block is recognised as soon as the page loads and raises GoogleScholarBlockedException with a message that says what actually happened. A challenge page carrying none of those elements falls back to matching Google's wording, but only after the selector wait has failed: sniffing the text up front would be faster and wrong, since a legitimate Scholar search for "unusual traffic" returns papers whose snippets contain that phrase. A p…f493ed5 -
Move orchestration out of the route handlers into a service layer
Claude +1: Claude Opus 5Unverified signatureThe backend policy - which backend to try, in what order, when to fall back, what counts as evidence a backend is unhealthy, when a patent is absent rather than merely unreachable - was written inline in FastAPI route handlers. It is not HTTP-specific: it would be equally true behind a CLI or a queue consumer, and putting it there meant exercising any of it cost an ASGI round-trip plus monkeypatching module globals. It now lives in services.py as SearchService and PatentService, which take their stateful collaborators (HTTP client, browser provider, circuit breaker, OPS credentials) as constructor arguments. app.py keeps the module-level instances and wires them in through `Depends`, so `app.dependency_overrides` replaces them in a test. The services raise domain errors - PatentNotFound, UpstreamUnavailable, OPSUnconfigured - rather than HTTPException. Mapping those onto status codes is the HTTP layer's job and is now three exception handlers instead of the same branching repeated ac…e44fdef -
Housekeeping: lint in CI, client lifecycle, and a truer breaker signal
Claude +1: Claude Opus 5Unverified signatureIndependent cleanups from the review, grouped because each is small. A3 - utils.log_gathered_exceptions imported SerpQuery from serp.py purely to read one attribute, so the lowest layer in the codebase depended on the domain layer. It now takes any iterable of query strings, which also makes it usable from the patent and OPS paths. A4 - the shared httpx client was created at import and never closed; the lifespan tore down the browser but left its connection pool open. It is now closed on shutdown, including when Playwright failed to start. Two lifespan tests cover this, and the existing one no longer closes the real client out from under the rest of the session. A5 - the circuit breaker could not tell "this query had no hits" from "this backend is refusing us". query_ddg_search raises on an empty result because a soft block can parse to zero results, but _search_one counted every exception against the backend - and the breaker is a process-wide singleton, so three unusual-but-legiti…9841346 -
Pin the MCP tool surface and give every tool an output schema
Claude +1: Claude Opus 5Unverified signaturemcp_server.py had no tests, and it is the file least able to afford that: `_tool_names` walks the route tree through three version-dependent shapes (DefaultPlaceholder unwrapping, nested `.routes`, FastAPI >= 0.141's `original_router`), each a reasonable guess about a library's internals. When one stops resolving, tools quietly revert to generated operationIds like `search_arxiv_serp_search_arxiv_post`, tool selection degrades, and nothing fails. tests/test_mcp_server.py pins the surface an agent actually sees: the curated six-tool set, handler names rather than operationIds, the /ops/* and single-backend exclusions holding, and anchored exclusion patterns. It also caught a live defect. FastMCP derives tool schemas from the app's OpenAPI document, and three handlers had no return annotation, so FastAPI emitted an empty 200 schema and those tools advertised no output schema at all - `search`, `search_arxiv` and `search_google_scholar`. `search` is the tool MCP_INSTRUCTIONS pushes agen…060118a -
Test the search URLs, and delete the harness that hid them
Claude +1: Claude Opus 5Unverified signatureThe Playwright tests drove a real browser through a wrapper that replaced `page.goto` with a stand-in accepting the URL and discarding it. So every scraper's navigation target was untested, and invisibly so: mutation testing showed that replacing the Bing URL with a constant, dropping `&num=` from the Scholar URL, and making the Google Patents URL ignore the query all left the full suite green. A scraper searching for the wrong thing entirely passed. Finish the fetch/parse split that scrap.py and _extract_bing_results had already started: - Each scraper's URL construction becomes a named pure builder (bing_search_url and friends), asserted directly in tests/test_serp_urls.py - query present, query encoded, result count honoured, expected host. 22 tests, no browser, 0.11s. All four URL mutants now fail the suite. - The extraction tests take an already-loaded page via a `page_factory` fixture using `set_content`, exactly as the bing test already did. This removes the need for…232eb80 -
Stop reporting upstream failures as "patent not found"
Claude +1: Claude Opus 5Unverified signature`scrap_patent` caught HTTPStatusError and bare Exception identically and fell through to a single 404, so every failure mode collapsed into "not found": a 503 from Google Patents, a connect timeout, and an unparseable interstitial all told the caller the patent does not exist. That is worse than an inaccurate status code here, because the MCP instructions this server ships tell the model in as many words: "A patent that returns 'not found' is genuinely absent from every backend; move on to the next id rather than retrying." Correct advice given a truthful 404, actively harmful given this one - a transient blip taught an agent, with the server's explicit encouragement, that a real patent does not exist. Now 404 is reserved for a backend positively saying the document is absent, and is only returned when *both* backends said so (or when the only backend asked did). Upstream failures map to 502, timeouts to 504. The same mapping is applied to the direct /ops endpoint. tests/test_patent…2e4089d -
Bound the list inputs and the outbound fan-out they cause
Claude +1: Claude Opus 5Unverified signature`n_results` was defended by a validator and a seven-case parametrized test while the list fields alongside it were undefended entirely. Two consequences, both reachable from an unauthenticated request: An empty query list crashed every search endpoint. `_shape_serp_results` reads `results[-1]` to build its error message, so `{"queries": []}` raised IndexError - an unhandled 500 on all six endpoints, from a two-character body. The bulk endpoints fanned out without any limit. `patent_ids` had no maximum and both bulk helpers gathered the whole list at once, so 3000 ids opened 3000 concurrent scrapes; the OPS variant is three times worse again, since each id costs a biblio, a claims and a description request. The Playwright scrapers have had a concurrency semaphore from the start - this brings the HTTP paths in line with them. - min_length/max_length on SerpQuery.queries and ScrapPatentsRequest.patent_ids - semaphores in scrap_patent_bulk_async and ops_scrap_patent_bulk - _shape_serp_r…d4440e7 -
Run the blocking DuckDuckGo call in a worker thread
Claude +1: Claude Opus 5Unverified signature`DDGS.text()` is a synchronous method doing blocking HTTP I/O, and it was being called directly from an `async def`. That froze the entire event loop for the duration of every DuckDuckGo request: no other request progressed, no in-flight Playwright scrape progressed, nothing else on the loop got a turn. DuckDuckGo is the first backend tried for every query in `/serp/search`, which is the flagship MCP tool, and both the endpoint docstrings and MCP_INSTRUCTIONS tell clients to batch their query variations into a single call - so the advertised concurrency was not just absent on the primary path, it was inverted into N sequential stalls. Measured before the fix, with a 0.25s blocking stand-in: four gathered queries took 1.00s (fully serialized) and the event loop got 0 turns. After: the same four take about as long as one, and the loop keeps ticking throughout. tests/test_serp_ddg_concurrency.py asserts both properties directly - the gathered queries don't serialize, and a heartbeat ta…8bbdd72 -
Test the API-key middleware, and stop it 500ing on a non-ASCII key
Claude +1: Claude Opus 5Unverified signatureThe optional API-key guard was the only security control in the codebase and had no tests at all. Mutation testing showed what that cost: replacing `secrets.compare_digest` with `==` (a timing-oracle regression) or disabling the guard's condition outright both left the whole suite green. Add tests/test_auth.py covering the behaviour that was already there - open when unset, 401 on a missing/wrong/prefix key, both header forms accepted, bearer scheme matched case-insensitively, and the docs paths staying public - all confirmed green against the unchanged middleware first. Three of the four mutants above now fail the suite. The fourth cannot be caught behaviourally: `==` and `compare_digest` return the same answer for every input and differ only in timing. Since a timing oracle on the API key is worth pinning anyway, that one test asserts structurally that the comparison goes through compare_digest. Then fix a real defect those tests exposed. `compare_digest` raises TypeError on str a…d03af21 -
Add a circuit breaker to the multi-backend search fallback (item from review)
ClaudeUnverified signature/serp/search (and the MCP `search` tool) tries DuckDuckGo, then Brave, then Bing per query with no memory between calls - if a backend is rate-limiting or has blocked this deployment's IP, every single query still pays its full timeout hammering that backend before falling through, which only makes the block worse. - circuit_breaker.py: a small per-key CircuitBreaker (new module, no third- party dependency) - opens after `failure_threshold` consecutive failures for a key, short-circuits further calls during `cooldown_seconds`, then allows one half-open trial; success closes it, another failure reopens it for a fresh cooldown. Unit-tested with a fake clock for determinism. - app.py: wired a module-level `_backend_circuit_breaker` (3 failures, 60s cooldown) into `_search_one`'s backend loop. _search_one/`/serp/search` had zero test coverage before this - added characterization tests for its current fallback behavior first (first- success, falls-through-on-failure, all-fail er…ba2278a -
Run the container as a non-root user (item from review: Dockerfile hardening)
ClaudeUnverified signatureThe image ran as root with no USER directive - if a scraping dependency or a malicious response ever led to arbitrary code execution, that code ran as root inside the container. Also dropped a redundant `pip install playwright` (already pulled in by requirements.txt). - Dockerfile: added `useradd -m -u 1000 user` + `USER user`, following Hugging Face Spaces' own documented convention for custom Docker Spaces (this app deploys to a HF Space). - app.py: api_lifespan now launches Chromium with `args=["--no-sandbox"]`. Chromium's own internal sandbox needs privileges a non-root container user doesn't have and fails to launch at all without this - the container itself remains the isolation boundary. Added a test (written first, confirmed failing, then made to pass) asserting the launch call carries this flag. Validated end-to-end rather than by inspection alone, since this is the one change here with real risk of breaking the live deployment: started dockerd in this sandbox,…285bc99 -
Validate patent_id shape before hitting Google Patents/OPS
ClaudeUnverified signaturepatent_id (both the /scrap and /ops path params, and scrap_patents_bulk's list) was an unvalidated string interpolated straight into a Google Patents/OPS URL. Not a host-controllable SSRF (the host is a fixed literal), but garbage input still reached the upstream API and came back as a confusing 404/502 instead of a clear client error. - serp.py: split PATENT_ID_REGEX into an unanchored PATENT_ID_CORE (used to find an id inside scraped text) and the existing bounded regex built from it, so there's one definition of "what a patent id looks like" instead of two that could drift. - app.py: added a PatentId = Annotated[str, StringConstraints(pattern=...)] built from that same core pattern, applied to both path params and ScrapPatentsRequest.patent_ids. Malformed input now gets a 422 with a clear message instead of reaching the upstream API. Wrote the new tests first (expecting 422 on malformed input), confirmed they failed against the unvalidated endpoints, then added the val…55c0bf1 -
Deduplicate app.py's search endpoints (item 4 from review)
ClaudeUnverified signatureSix of the seven /serp and /ops search endpoints repeated the identical gather -> log -> filter -> flatten -> shape-a-SerpResults pattern verbatim, differing only in which query_* function they called. Extracted into two helpers: - _shape_serp_results(results): flatten a gather's per-query results (or Exceptions), surfacing the last error only when every query failed. - _run_serp_queries(fn, params, context): run fn concurrently across every query, log failures, and shape via _shape_serp_results. search_patents keeps its distinct OPS-fallback loop (genuinely different logic, not boilerplate) but now reuses _shape_serp_results for its tail. app.py had zero test coverage before this - added tests/test_app.py first (characterizing the endpoints' current behavior via ASGITransport, with the query_*/ops_search functions monkeypatched rather than hitting real search engines or needing a real Playwright browser) and confirmed those pass against the pre-refactor code, then did the extra…3352a6e -
Split fetch from parse in scrap.py and serp.py (item 2 from review)
ClaudeUnverified signatureBoth scrap_patent_async and the four Playwright scrapers previously fused navigation/fetching with DOM parsing in one function, so there was no way to unit-test "given this page, what does the current selector contract extract" without also driving a real network request or browser navigation through the exact same call. - scrap.py: extracted parse_patent_html(html, patent_url) as a plain synchronous function; scrap_patent_async is now just fetch + delegate. - serp.py: extracted _extract_google_scholar_results, _extract_google_patents_results, _extract_brave_results and _extract_bing_results, each taking an already-loaded Page. The public query_* functions are now just navigate + delegate. Also hoisted the identical `_block_resources` closure (copy-pasted four times) to one module-level _block_stylesheet_and_image_resources, and PATENT_ID_REGEX out of query_google_patents's body. No behavior change - all 67 existing characterization tests pass unmodified, which is the p…4e043b3 -
Fix bare except and misattributed logger (item 3 from review)
ClaudeUnverified signature- serp.py: query_google_patents's per-item extraction caught bare `except:`, which also swallows asyncio.CancelledError (BaseException in 3.8+), silently absorbing task cancellation instead of letting it propagate. Narrowed to `except Exception:`. - utils.py: `from asyncio.log import logger` reused asyncio's internal logger, so every warning here was misattributed to the 'asyncio' logging namespace instead of the app's own. Now uses logging.getLogger(__name__). - Updated the characterization test that had pinned the old logger name to assert the fixed one instead.21042c7
-
Fix hanging Playwright tests: make the browser fixture function-scoped
ClaudeUnverified signatureThe session-scoped `browser` fixture hung indefinitely on the first test that used it. Root cause: pytest-asyncio runs each test on its own event loop by default, but the fixture was set up once against whichever loop was active on first use. Playwright's connection keeps a background reader task bound to that original loop, so a later test's awaits (on a different loop) don't error - they just wait forever for a reply the old loop's task can never deliver. Confirmed by reproducing the hang, then ruling out the scraping/fixture logic itself by running the same call in a plain script outside pytest (single event loop throughout), where it completed instantly. Switching to a function-scoped fixture (fresh browser per test, ~1s extra each) sidesteps the whole cross-loop class of bug instead of fighting pytest-asyncio's loop-scope configuration. All 67 tests now pass in ~6s.30d9456 -
Add pytest harness with characterization tests for scrap/ops/serp/utils
ClaudeUnverified signatureAdds a test safety net ahead of any refactoring, per the TDD/architecture review of this codebase (no test infrastructure existed before this): - pytest + pytest-asyncio + respx as dev-only deps (requirements-dev.txt) - Pure-function tests for SerpQuery's n_results clamping, utils.py's exception logging, and every OPS JSON-walking helper (ops.py handles OPS's XML-to-JSON quirk where a repeated element is a list but a single one stays a bare dict - tests cover both shapes for each helper) - HTTP-mocked (respx) tests for scrap_patent_async's HTML parsing (abstract/description/claims/classification-code extraction, plus the documented ValueError path for an unexpected page layout), the OPS token manager's caching/refresh/retry-on-stale-token behavior, and query_arxiv's Atom feed parsing - A DDGS-mocked test pinning the DuckDuckGo empty-results-means-blocked fix from d341a7a, so a regression there fails a test instead of shipping silently - Real-headless-Chromium tests f…5796881