Spaces:
Running
Running
Commit History
-
Stop the search chain treating an empty answer as an answer (#10) Running
Unverified signature/serp/search returned zero results and the explanation "All backends are rate-limited." - a claim with nothing behind it. Seen once against the live deployment, then not reproducible, which is itself the problem: the message described a diagnosis rather than what happened. A backend returning an empty list without raising ended the fallback chain and was recorded as a success. Brave does that whenever its block page lacks the word "suspicious", and Bing whenever no result item yields a title and href, so one quiet backend could stop the chain before a working one was tried. Same bug class as the DuckDuckGo fix in d341a7a, which the other two never got. Fixed in the chain rather than each scraper, so it covers every backend including later ones and leaves the single-backend endpoints free to report an honest empty result. An empty answer still counts neither for nor against the backend, as an exception-raised empty already did: recording it as a success would reset a backend part-way …6fd0195 -
Stop the search chain treating an empty answer as an answer
Claude +1: Claude Opus 5Unverified signature`/serp/search` returned zero results and the explanation "All backends are rate-limited." - a claim with nothing behind it. Seen once against the live deployment, then not reproducible, which is itself the problem: the message described a diagnosis rather than what happened. Two causes, both fixed here. A backend that returned an empty list without raising ended the fallback chain and was recorded as a success. Brave does that whenever its block page lacks the word "suspicious", and Bing whenever no result item yields a title and href - so one quiet backend could stop the chain before a working one was ever tried. This is the same bug class fixed for DuckDuckGo in d341a7a, which raises on empty; the other two never got the same treatment. Fixing it in the chain rather than in each scraper covers every backend, including any added later, and leaves the single-backend endpoints free to report an honest empty result. An empty answer still counts neither for nor against the backend, as …73c1aa6 -
Classify a blocked Scholar page before waiting, not after (#9)
Unverified signature#8 made the classification correct but left it arriving 30s late. Measured against the deployment: the error said "Google Scholar blocked the request", but still took 30.7s, because every signal was consulted only in the selector-timeout handler. All of them are available the moment navigation completes, so they are checked there instead - and deliberately all of them, not just the one suspected of firing in production, since a fix contingent on that guess would repeat the mistake from #7. A page carrying no results at all is decidable immediately: Scholar is server-rendered, so a genuine results page already has div.gs_ri in the DOM. That "no results" guard is what makes the wording check safe this early, and it caught a real false positive during this change: without it, a legitimate search for "unusual traffic" returns papers whose snippets quote the interstitial and was misreported as a block. The block exception now carries the evidence that classified it. Diagnostics had only …fabb9dc -
Classify a blocked Scholar page before waiting, not after
Claude +1: Claude Opus 5Unverified signature#8 made the classification correct but left it arriving 30s late. Measured against the deployment: the error now says "Google Scholar blocked the request", but it still takes 30.7s, because every signal was consulted only in the selector-timeout handler. All of them are available the moment navigation completes, so they are checked there instead. A page carrying no results at all is decidable immediately: Scholar is server-rendered, so a genuine results page already has div.gs_ri in the DOM. That "no results" guard is also what makes the wording check safe this early - and it caught a real false positive during this change. Without it, a legitimate search for "unusual traffic" returns papers whose snippets quote the interstitial and was misreported as a block. The test written for that risk failed against the first version of this commit, which is the whole reason it exists. The block exception now carries the evidence that classified it - redirect URL, challenge element, or wording…8a62982 -
Make a failed Scholar scrape say what it actually got (#8)
Unverified signatureThe block detection added in #7 did not work. It waited for challenge elements guessed at without being able to observe the real page - Scholar is unreachable from the development sandbox - and matched none of them, so a blocked scrape still cost the full 30s timeout and still reported a bare "Timeout exceeded". Guessing again would repeat the mistake, so this changes the shape of the failure instead. An unclassifiable page now raises GoogleScholarUnavailableError carrying the final URL, the page title and a text excerpt, so the next live call identifies whatever Google is actually serving. Two real defects in the first attempt, both now covered: challenge elements were matched by visibility, but reCAPTCHA's container is commonly zero-height until it loads, so detection sat through the entire timeout on exactly the pages it was written to catch; and the evidence was purely markup-based, where a /sorry redirect survives Google restyling the page. The hidden-element test originally as…57c466b -
Make a failed Scholar scrape say what it actually got
Claude +1: Claude Opus 5Unverified signatureThe block detection added in #7 did not work. It waited for challenge elements I had guessed at without being able to observe the real page - Scholar is unreachable from the development sandbox - and matched none of them, so a blocked scrape still cost the full 30s timeout and still reported a bare "Timeout 30000ms exceeded". Guessing again would repeat the mistake, so this changes the shape of the failure instead. An unclassifiable page now raises GoogleScholarUnavailableError carrying the final URL, the page title and a text excerpt. Whatever Google is actually serving, the next live call identifies it - no more deploy-and-hope rounds. The error path is best-effort throughout: failing to read the page must not replace the original problem with a new one. Two real defects in the first attempt, both now covered: - Challenge elements were matched by visibility. reCAPTCHA renders in an iframe whose container is commonly zero-height until it loads, so the element can be attached bu…addbf67 -
Detect Google Scholar's block page, and unwrap Bing's redirect links (#7)
Unverified signatureBoth found by running the live smoke test against the deployed MCP server after #6 shipped, not by the suite. Google Scholar serves an anti-bot interstitial to the Space's IP. That page has no div.gs_ri, so the scraper waited out the full 30s selector timeout and then reported a bare "Timeout exceeded" - slow, and worded like a broken selector rather than a blocked deployment. It now watches for the results selector and Google's challenge elements together, failing fast with a message that says what happened. The text-matching fallback runs only after the selector wait fails, because a legitimate search for "unusual traffic" would otherwise be misreported as a block; a page with neither results nor a challenge still raises the original timeout, so a real selector regression cannot hide behind the excuse. Bing does not put the destination in a result's href - every link is wrapped in its /ck/a redirector - so callers were handed tracking links instead of pages. decode_bing_redirect un…a8e3022 -
Detect Google Scholar's block page, and unwrap Bing's redirect links
Claude +1: Claude Opus 5Unverified signatureBoth found by running the live smoke test against the deployed MCP server, not by the suite - the division of labour CONTRIBUTING.md already describes. Google Scholar serves an anti-bot interstitial to the Space's IP instead of results. That page has no `div.gs_ri`, so the scraper sat in `wait_for_selector` for the full 30 seconds and then reported a bare "Timeout 30000ms exceeded" - slow for every caller, and worded like a broken selector rather than a blocked deployment. It now waits for the results selector and Google's challenge elements together, so a block is recognised as soon as the page loads and raises GoogleScholarBlockedException with a message that says what actually happened. A challenge page carrying none of those elements falls back to matching Google's wording, but only after the selector wait has failed: sniffing the text up front would be faster and wrong, since a legitimate Scholar search for "unusual traffic" returns papers whose snippets contain that phrase. A p…f493ed5 -
Fix the defects a mutation probe found, and the architecture that hid them (#6)
Unverified signatureA mutation probe of the existing suite killed 5 of 6 mutants aimed at pure functions and 1 of 7 aimed at the I/O boundary - deleting the API-key middleware entirely left it green, as did replacing the Bing search URL with a constant. Line coverage read 93% and could not see the difference. Fixes every defect that hid behind that gap: the blocking DuckDuckGo call that froze the event loop on the primary search path, an empty query list 500ing all six search endpoints, a non-ASCII API key crashing the auth check, upstream failures reported as "patent not found" (which MCP_INSTRUCTIONS tells agents to believe and not retry), unbounded bulk fan-out, and three MCP tools advertising no output schema. Then the cause: orchestration policy moved out of the route handlers into services.py, with its stateful collaborators injected via Depends. Also adds ruff to CI, closes the httpx client in the lifespan, and stops the circuit breaker treating "zero results" as evidence a backend is down. 97 -…632b2d9 -
Stop tracking .coverage, and ignore local tool output
Claude +1: Claude Opus 5Unverified signatureA `git add -A` in the previous commit swept up .coverage, the binary SQLite file `pytest --cov` writes. It is machine-specific build output and has no business in the tree. Plain `pytest` - the command CONTRIBUTING.md documents - doesn't produce it, which is why the gap went unnoticed: pytest and ruff both write self-ignoring cache directories, and coverage.py does not. Ignore .coverage, htmlcov and .ruff_cache so running any of the documented commands leaves the tree clean. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01MSNSYnceqvdVz7Csis4K9e72a7212 -
Move orchestration out of the route handlers into a service layer
Claude +1: Claude Opus 5Unverified signatureThe backend policy - which backend to try, in what order, when to fall back, what counts as evidence a backend is unhealthy, when a patent is absent rather than merely unreachable - was written inline in FastAPI route handlers. It is not HTTP-specific: it would be equally true behind a CLI or a queue consumer, and putting it there meant exercising any of it cost an ASGI round-trip plus monkeypatching module globals. It now lives in services.py as SearchService and PatentService, which take their stateful collaborators (HTTP client, browser provider, circuit breaker, OPS credentials) as constructor arguments. app.py keeps the module-level instances and wires them in through `Depends`, so `app.dependency_overrides` replaces them in a test. The services raise domain errors - PatentNotFound, UpstreamUnavailable, OPSUnconfigured - rather than HTTPException. Mapping those onto status codes is the HTTP layer's job and is now three exception handlers instead of the same branching repeated ac…e44fdef -
Housekeeping: lint in CI, client lifecycle, and a truer breaker signal
Claude +1: Claude Opus 5Unverified signatureIndependent cleanups from the review, grouped because each is small. A3 - utils.log_gathered_exceptions imported SerpQuery from serp.py purely to read one attribute, so the lowest layer in the codebase depended on the domain layer. It now takes any iterable of query strings, which also makes it usable from the patent and OPS paths. A4 - the shared httpx client was created at import and never closed; the lifespan tore down the browser but left its connection pool open. It is now closed on shutdown, including when Playwright failed to start. Two lifespan tests cover this, and the existing one no longer closes the real client out from under the rest of the session. A5 - the circuit breaker could not tell "this query had no hits" from "this backend is refusing us". query_ddg_search raises on an empty result because a soft block can parse to zero results, but _search_one counted every exception against the backend - and the breaker is a process-wide singleton, so three unusual-but-legiti…9841346 -
Pin the MCP tool surface and give every tool an output schema
Claude +1: Claude Opus 5Unverified signaturemcp_server.py had no tests, and it is the file least able to afford that: `_tool_names` walks the route tree through three version-dependent shapes (DefaultPlaceholder unwrapping, nested `.routes`, FastAPI >= 0.141's `original_router`), each a reasonable guess about a library's internals. When one stops resolving, tools quietly revert to generated operationIds like `search_arxiv_serp_search_arxiv_post`, tool selection degrades, and nothing fails. tests/test_mcp_server.py pins the surface an agent actually sees: the curated six-tool set, handler names rather than operationIds, the /ops/* and single-backend exclusions holding, and anchored exclusion patterns. It also caught a live defect. FastMCP derives tool schemas from the app's OpenAPI document, and three handlers had no return annotation, so FastAPI emitted an empty 200 schema and those tools advertised no output schema at all - `search`, `search_arxiv` and `search_google_scholar`. `search` is the tool MCP_INSTRUCTIONS pushes agen…060118a -
Test the search URLs, and delete the harness that hid them
Claude +1: Claude Opus 5Unverified signatureThe Playwright tests drove a real browser through a wrapper that replaced `page.goto` with a stand-in accepting the URL and discarding it. So every scraper's navigation target was untested, and invisibly so: mutation testing showed that replacing the Bing URL with a constant, dropping `&num=` from the Scholar URL, and making the Google Patents URL ignore the query all left the full suite green. A scraper searching for the wrong thing entirely passed. Finish the fetch/parse split that scrap.py and _extract_bing_results had already started: - Each scraper's URL construction becomes a named pure builder (bing_search_url and friends), asserted directly in tests/test_serp_urls.py - query present, query encoded, result count honoured, expected host. 22 tests, no browser, 0.11s. All four URL mutants now fail the suite. - The extraction tests take an already-loaded page via a `page_factory` fixture using `set_content`, exactly as the bing test already did. This removes the need for…232eb80 -
Stop reporting upstream failures as "patent not found"
Claude +1: Claude Opus 5Unverified signature`scrap_patent` caught HTTPStatusError and bare Exception identically and fell through to a single 404, so every failure mode collapsed into "not found": a 503 from Google Patents, a connect timeout, and an unparseable interstitial all told the caller the patent does not exist. That is worse than an inaccurate status code here, because the MCP instructions this server ships tell the model in as many words: "A patent that returns 'not found' is genuinely absent from every backend; move on to the next id rather than retrying." Correct advice given a truthful 404, actively harmful given this one - a transient blip taught an agent, with the server's explicit encouragement, that a real patent does not exist. Now 404 is reserved for a backend positively saying the document is absent, and is only returned when *both* backends said so (or when the only backend asked did). Upstream failures map to 502, timeouts to 504. The same mapping is applied to the direct /ops endpoint. tests/test_patent…2e4089d -
Bound the list inputs and the outbound fan-out they cause
Claude +1: Claude Opus 5Unverified signature`n_results` was defended by a validator and a seven-case parametrized test while the list fields alongside it were undefended entirely. Two consequences, both reachable from an unauthenticated request: An empty query list crashed every search endpoint. `_shape_serp_results` reads `results[-1]` to build its error message, so `{"queries": []}` raised IndexError - an unhandled 500 on all six endpoints, from a two-character body. The bulk endpoints fanned out without any limit. `patent_ids` had no maximum and both bulk helpers gathered the whole list at once, so 3000 ids opened 3000 concurrent scrapes; the OPS variant is three times worse again, since each id costs a biblio, a claims and a description request. The Playwright scrapers have had a concurrency semaphore from the start - this brings the HTTP paths in line with them. - min_length/max_length on SerpQuery.queries and ScrapPatentsRequest.patent_ids - semaphores in scrap_patent_bulk_async and ops_scrap_patent_bulk - _shape_serp_r…d4440e7 -
Run the blocking DuckDuckGo call in a worker thread
Claude +1: Claude Opus 5Unverified signature`DDGS.text()` is a synchronous method doing blocking HTTP I/O, and it was being called directly from an `async def`. That froze the entire event loop for the duration of every DuckDuckGo request: no other request progressed, no in-flight Playwright scrape progressed, nothing else on the loop got a turn. DuckDuckGo is the first backend tried for every query in `/serp/search`, which is the flagship MCP tool, and both the endpoint docstrings and MCP_INSTRUCTIONS tell clients to batch their query variations into a single call - so the advertised concurrency was not just absent on the primary path, it was inverted into N sequential stalls. Measured before the fix, with a 0.25s blocking stand-in: four gathered queries took 1.00s (fully serialized) and the event loop got 0 turns. After: the same four take about as long as one, and the loop keeps ticking throughout. tests/test_serp_ddg_concurrency.py asserts both properties directly - the gathered queries don't serialize, and a heartbeat ta…8bbdd72 -
Test the API-key middleware, and stop it 500ing on a non-ASCII key
Claude +1: Claude Opus 5Unverified signatureThe optional API-key guard was the only security control in the codebase and had no tests at all. Mutation testing showed what that cost: replacing `secrets.compare_digest` with `==` (a timing-oracle regression) or disabling the guard's condition outright both left the whole suite green. Add tests/test_auth.py covering the behaviour that was already there - open when unset, 401 on a missing/wrong/prefix key, both header forms accepted, bearer scheme matched case-insensitively, and the docs paths staying public - all confirmed green against the unchanged middleware first. Three of the four mutants above now fail the suite. The fourth cannot be caught behaviourally: `==` and `compare_digest` return the same answer for every input and differ only in timing. Since a timing oracle on the API key is worth pinning anyway, that one test asserts structurally that the comparison goes through compare_digest. Then fix a real defect those tests exposed. `compare_digest` raises TypeError on str a…d03af21 -
Safer concurrent deploys, pinned dependencies, and a CONTRIBUTING guide (#5)
Unverified signatureSafer concurrent deploys, pinned dependencies, and a CONTRIBUTING guidea604513 -
Document the development methodology (CONTRIBUTING.md)
ClaudeUnverified signatureWrites down the TDD workflow this project has followed throughout - test/characterize first, run the full suite (a shared helper's bug should surface as multiple failures), distrust a test that never fails (break the code on purpose and confirm it catches it), watch for shared mutable state leaking across tests, and validate genuinely risky changes for real rather than by review alone - plus the CI/deploy pipeline, the dependency pinning policy, and which secrets to check first when something breaks for no obvious code reason. This isn't GitHub-only documentation: deploy-to-hf.yml pushes the whole repo tree to the Hugging Face Space on every deploy, not a curated subset, so this file (like the tests, the pinned requirements, and the CI config) ships there too once this merges and deploys. Linked from README.md.5a8a721 -
Pin dependencies and add CI on pull requests
ClaudeUnverified signatureTwo related gaps found while writing the "coming back after a month" runbook: - requirements.txt/requirements-dev.txt had no version pins at all. A fresh `pip install` any time in the future would silently grab whatever's newest on PyPI, with no way to tell "my change broke this" from "an unrelated dependency update broke this." Pinned every package to the version this codebase is actually tested against, with a range up to its next breaking boundary (next major for a stable package, next minor for a 0.x one). playwright and duckduckgo_search are pinned tighter (patch-only): playwright ships in lockstep with a specific browser revision - a mismatch is a real failure mode we already hit and worked around in tests/conftest.py - and duckduckgo_search has a history of breaking changes between versions. pytest-asyncio (dev-only) is pinned the same way, for the same reason: its event-loop-scope semantics bit us once already this project (see the `browser` fixture's c…2d3711f -
Make the HF Space deploy fail loudly on a concurrent-write collision
ClaudeUnverified signaturegit push --force to the Space was a blind overwrite: it never checked what was currently there, so if anyone else has independent push access to the same Space (direct git push, HF's web editor, their own CI) and deploys around the same time, whichever push landed second would win silently. The loser gets no error - git push --force always succeeds regardless of what it's overwriting - so neither side would know a collision happened without manually diffing the Space's history after the fact. Switched to `git push --force-with-lease`, which refuses to push if the remote's current value doesn't match what we last fetched - i.e. someone else changed it since we looked. Needs an explicit `git fetch space main` right before the push: --force-with-lease compares against the local remote-tracking ref, and this job adds the `space` remote fresh each run with nothing to compare against otherwise (it would either fail every single deploy, or on some git versions silently fall back to acting li…56b0d37 -
Address remaining hardening items from the TDD/architecture review (#4)
Unverified signatureAddress remaining hardening items from the TDD/architecture review33657f8 -
Add a circuit breaker to the multi-backend search fallback (item from review)
ClaudeUnverified signature/serp/search (and the MCP `search` tool) tries DuckDuckGo, then Brave, then Bing per query with no memory between calls - if a backend is rate-limiting or has blocked this deployment's IP, every single query still pays its full timeout hammering that backend before falling through, which only makes the block worse. - circuit_breaker.py: a small per-key CircuitBreaker (new module, no third- party dependency) - opens after `failure_threshold` consecutive failures for a key, short-circuits further calls during `cooldown_seconds`, then allows one half-open trial; success closes it, another failure reopens it for a fresh cooldown. Unit-tested with a fake clock for determinism. - app.py: wired a module-level `_backend_circuit_breaker` (3 failures, 60s cooldown) into `_search_one`'s backend loop. _search_one/`/serp/search` had zero test coverage before this - added characterization tests for its current fallback behavior first (first- success, falls-through-on-failure, all-fail er…ba2278a -
Run the container as a non-root user (item from review: Dockerfile hardening)
ClaudeUnverified signatureThe image ran as root with no USER directive - if a scraping dependency or a malicious response ever led to arbitrary code execution, that code ran as root inside the container. Also dropped a redundant `pip install playwright` (already pulled in by requirements.txt). - Dockerfile: added `useradd -m -u 1000 user` + `USER user`, following Hugging Face Spaces' own documented convention for custom Docker Spaces (this app deploys to a HF Space). - app.py: api_lifespan now launches Chromium with `args=["--no-sandbox"]`. Chromium's own internal sandbox needs privileges a non-root container user doesn't have and fails to launch at all without this - the container itself remains the isolation boundary. Added a test (written first, confirmed failing, then made to pass) asserting the launch call carries this flag. Validated end-to-end rather than by inspection alone, since this is the one change here with real risk of breaking the live deployment: started dockerd in this sandbox,…285bc99 -
Validate patent_id shape before hitting Google Patents/OPS
ClaudeUnverified signaturepatent_id (both the /scrap and /ops path params, and scrap_patents_bulk's list) was an unvalidated string interpolated straight into a Google Patents/OPS URL. Not a host-controllable SSRF (the host is a fixed literal), but garbage input still reached the upstream API and came back as a confusing 404/502 instead of a clear client error. - serp.py: split PATENT_ID_REGEX into an unanchored PATENT_ID_CORE (used to find an id inside scraped text) and the existing bounded regex built from it, so there's one definition of "what a patent id looks like" instead of two that could drift. - app.py: added a PatentId = Annotated[str, StringConstraints(pattern=...)] built from that same core pattern, applied to both path params and ScrapPatentsRequest.patent_ids. Malformed input now gets a 422 with a clear message instead of reaching the upstream API. Wrote the new tests first (expecting 422 on malformed input), confirmed they failed against the unvalidated endpoints, then added the val…55c0bf1 -
Add test harness and fix TDD/architecture review findings (#3)
Unverified signatureAdd test harness and fix TDD/architecture review findings5e51b32 -
Gate the HF Space deploy on tests passing (item 5 from review)
ClaudeUnverified signaturedeploy-to-hf.yml previously went straight from checkout to `git push --force` on every push to main - no lint, no type-check, no test run in between. Added a `test` job (install deps, install Playwright's Chromium, run pytest) and made `deploy` depend on it via `needs: test`, so a commit that breaks the test suite no longer reaches the production HF Space.caf91fb -
Deduplicate app.py's search endpoints (item 4 from review)
ClaudeUnverified signatureSix of the seven /serp and /ops search endpoints repeated the identical gather -> log -> filter -> flatten -> shape-a-SerpResults pattern verbatim, differing only in which query_* function they called. Extracted into two helpers: - _shape_serp_results(results): flatten a gather's per-query results (or Exceptions), surfacing the last error only when every query failed. - _run_serp_queries(fn, params, context): run fn concurrently across every query, log failures, and shape via _shape_serp_results. search_patents keeps its distinct OPS-fallback loop (genuinely different logic, not boilerplate) but now reuses _shape_serp_results for its tail. app.py had zero test coverage before this - added tests/test_app.py first (characterizing the endpoints' current behavior via ASGITransport, with the query_*/ops_search functions monkeypatched rather than hitting real search engines or needing a real Playwright browser) and confirmed those pass against the pre-refactor code, then did the extra…3352a6e -
Split fetch from parse in scrap.py and serp.py (item 2 from review)
ClaudeUnverified signatureBoth scrap_patent_async and the four Playwright scrapers previously fused navigation/fetching with DOM parsing in one function, so there was no way to unit-test "given this page, what does the current selector contract extract" without also driving a real network request or browser navigation through the exact same call. - scrap.py: extracted parse_patent_html(html, patent_url) as a plain synchronous function; scrap_patent_async is now just fetch + delegate. - serp.py: extracted _extract_google_scholar_results, _extract_google_patents_results, _extract_brave_results and _extract_bing_results, each taking an already-loaded Page. The public query_* functions are now just navigate + delegate. Also hoisted the identical `_block_resources` closure (copy-pasted four times) to one module-level _block_stylesheet_and_image_resources, and PATENT_ID_REGEX out of query_google_patents's body. No behavior change - all 67 existing characterization tests pass unmodified, which is the p…4e043b3 -
Fix bare except and misattributed logger (item 3 from review)
ClaudeUnverified signature- serp.py: query_google_patents's per-item extraction caught bare `except:`, which also swallows asyncio.CancelledError (BaseException in 3.8+), silently absorbing task cancellation instead of letting it propagate. Narrowed to `except Exception:`. - utils.py: `from asyncio.log import logger` reused asyncio's internal logger, so every warning here was misattributed to the 'asyncio' logging namespace instead of the app's own. Now uses logging.getLogger(__name__). - Updated the characterization test that had pinned the old logger name to assert the fixed one instead.21042c7
-
Document how to run the new test suite
ClaudeUnverified signature18225f3 -
Fix hanging Playwright tests: make the browser fixture function-scoped
ClaudeUnverified signatureThe session-scoped `browser` fixture hung indefinitely on the first test that used it. Root cause: pytest-asyncio runs each test on its own event loop by default, but the fixture was set up once against whichever loop was active on first use. Playwright's connection keeps a background reader task bound to that original loop, so a later test's awaits (on a different loop) don't error - they just wait forever for a reply the old loop's task can never deliver. Confirmed by reproducing the hang, then ruling out the scraping/fixture logic itself by running the same call in a plain script outside pytest (single event loop throughout), where it completed instantly. Switching to a function-scoped fixture (fresh browser per test, ~1s extra each) sidesteps the whole cross-loop class of bug instead of fighting pytest-asyncio's loop-scope configuration. All 67 tests now pass in ~6s.30d9456 -
Add pytest harness with characterization tests for scrap/ops/serp/utils
ClaudeUnverified signatureAdds a test safety net ahead of any refactoring, per the TDD/architecture review of this codebase (no test infrastructure existed before this): - pytest + pytest-asyncio + respx as dev-only deps (requirements-dev.txt) - Pure-function tests for SerpQuery's n_results clamping, utils.py's exception logging, and every OPS JSON-walking helper (ops.py handles OPS's XML-to-JSON quirk where a repeated element is a list but a single one stays a bare dict - tests cover both shapes for each helper) - HTTP-mocked (respx) tests for scrap_patent_async's HTML parsing (abstract/description/claims/classification-code extraction, plus the documented ValueError path for an unexpected page layout), the OPS token manager's caching/refresh/retry-on-stale-token behavior, and query_arxiv's Atom feed parsing - A DDGS-mocked test pinning the DuckDuckGo empty-results-means-blocked fix from d341a7a, so a regression there fails a test instead of shipping silently - Real-headless-Chromium tests f…5796881 -
Fix DuckDuckGo backend silently returning empty results (#2)
Unverified signaturequery_ddg_search() didn't raise when the duckduckgo_search library got blocked/rate-limited and parsed zero results instead of throwing. This made /serp/search_duck report {"error": null, "results": []} on failure, and worse, made the /serp/search fallback chain (_search_one) treat DDG's empty return as a success and never try Brave or Bing — the reason /serp/search was returning "All backends are rate-limited." even when Bing worked fine on its own. Now an empty DDG result raises, so both endpoints behave correctly. Claude-Session: https://claude.ai/code/session_012sXQyRk3EZHbYHas8fQsiS Co-authored-by: Claude <noreply@anthropic.com>d341a7a -
Add GitHub Actions workflow to deploy to HF Space on push to main (#1)
Unverified signaturePushes the repo to huggingface.co/spaces/OrganizedProgrammers/SERPent using the HF_TOKEN repo secret. Claude-Session: https://claude.ai/code/session_012sXQyRk3EZHbYHas8fQsiS Co-authored-by: Claude <noreply@anthropic.com>cd869cb -
Mirror SERPent from Hugging Face Space with bug fixes
ClaudeUnverified signatureMirrors OrganizedProgrammers/SERPent (https://huggingface.co/spaces/OrganizedProgrammers/SERPent) with the following fixes applied on top: - app.py: make /serp/search try all queries concurrently instead of sequentially, and surface per-query backend failures in the response's error field instead of silently dropping them when at least one other query in the batch succeeded. - app.py: add opt-in SERPENT_API_KEY protection (X-API-Key or Authorization: Bearer) gating REST and MCP alike via HTTP middleware, closing the unauthenticated open-scraping-proxy surface. Unset by default, so behavior is unchanged until explicitly enabled. - app.py: load docs/docs.md safely (context-managed read, fallback description) instead of an unguarded open().read() at import time. - serp.py: raise a clear BrowserUnavailableError instead of an AttributeError when Playwright failed to start and pw_browser is None. - serp.py: clamp n_results to [1, 100] in SerpQuery so a negative or huge val…cb1e234