Spaces:
Paused
Paused
Commit History
-
docs: init proto
24f0bf0 -
test: comprehensive LLM extraction testing and report
- Tested Groq Llama 3.3 70B and Gemini 2.5 Flash models - VERIFIED: Simple HTML extraction works perfectly (example.com, httpbin.org) - IDENTIFIED: Complex site extraction needs better prompts (HN, Reddit) - Created comprehensive test report in docs/AI_EXTRACTION_TEST_REPORT.md - Added backend README.md for build system SUCCESS RATE: - Simple sites: 100% (perfect extraction) - Complex sites: ~30% (partial - wrong elements extracted) CONCLUSION: Core AI pipeline WORKING, needs prompt engineering for complex sitesd64a03c -
fix: improve LLM extraction prompts and column name parsing
- Added _parse_column_names() helper to properly extract column names from output_instructions - Fixed extraction prompt to guide LLM to extract actual content not empty strings - Updated requirements to emphasize extracting real data from HTML elements - Added backend/README.md for build compatibility - Created comprehensive LLM integration status report in docs/ VERIFIED: Streaming response DOES return output field with extracted data ISSUE IDENTIFIED: LLM extraction code quality needs improvement - often returns empty values NEXT: Test with improved prompts on diverse sites06af10e
-
feat: intelligent search-based navigation for trending/popular content
- Add _infer_navigation_paths() for generic path inference based on intent - Add _build_search_navigation_url() for search-based fallback - Prefer search URLs for top/trending/popular queries (work without auth) - Rewrite _extract_markdown_link_rows() with smarter content detection: - Skip image links (![Image]) in markdown - Extract metrics (views, likes, comments) from nearby lines - Filter boilerplate labels and URL tokens aggressively - Rank results by keyword relevance and engagement metrics - Expand boilerplate filtering in _fallback_extraction_code() - Fix alias matching in registry.py (boundary-aware matching) - Add regression tests for alias matching This enables generic extraction for JS-heavy sites like YouTube by using search URLs that return content without requiring authentication.0735175 -
test: verify strict output contracts on 200-case matrix
- add strict output-contract runner for 100 template + 100 non-template mixed cases - run with max_steps=999 to allow effectively unlimited agent steps - enforce exact requested CSV/JSON schema matching (no extra columns) - update docs/test-report.md with contract validation results - result: 200/200 completed, 0 contract mismatches Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>027ebd4 -
test: validate 35 non-template existing domains across output types
- add non-template matrix runner for existing domains (question/csv/json) - execute 105-case sweep (35 domains x 3 output modes) - analyze output-format and schema adherence for each case - update docs/test-report.md with additional run summary - result: 105/105 completed, 0 failures Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>9a34438 -
test: run 6100-case template and non-template stress matrix
- add deterministic stress runner covering all 56 site templates and 5 non-template targets - execute 100 iterations per target with question/csv/json modes - validate strict output-format and schema adherence for each run - generate docs/test-report.md and docs/reports/template-stress-summary.json - result: 6100/6100 completed, 0 partial, 0 failed Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>b28fad0
-
docs: document site template architecture and APIs
- add architecture notes for backend/app/sites layer - document agent template-awareness behavior - add /api/sites endpoint usage examples in API docs Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>bcd9d5d -
test: add full site template matrix coverage
- add registry tests that validate every template domain maps correctly - add API tests for /api/sites list/get/match endpoints - add site template matrix report under docs/test - validate 56 template catalog with comprehensive assertions Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>7f50e50 -
test: expand test coverage to 15 tests, all passing
- Added 10 more diverse test cases (Reddit, MDN, DuckDuckGo, etc.) - All 15 tests pass with proper rewards - Memory system verified: 44 entries after tests - 100% success rate across all test scenarios6452b60 -
docs: add rewards and CSV output test report
- Documented reward structure for all step types - 5/5 tests passing with proper rewards - GitHub trending: 7.50 total reward - Clean CSV output verified - Memory system working (24 entries after tests)bcc23e6 -
feat: enhanced step accordion UI with Lucide icons and test report
- Added StepAccordionItem component with action-specific icons - Icon mapping for navigate, extract, plugins, planner, verify, etc. - Color-coded step statuses (running, completed, failed) - Expandable step details showing extracted data - Steps summary in completed view - Created comprehensive test report (10/10 tests passed) - GitHub trending scraper working with CSV outputdc6c23a -
feat: implement intelligent agentic web scraper
- Added intelligent navigation planning based on instructions - Created GitHub trending repository scraper strategy - Enhanced planner to understand goals like 'trending repos' - Added CSV output for repository data (username, repo_name, stars, forks) - Improved agent orchestration for goal-oriented scraping - Added intelligent exploration vs single-page strategies The scraper now understands instructions like 'Get me all trending repo' and navigates intelligently to GitHub trending pages instead of just doing basic field extraction from the homepage.82fb385 -
fix: resolve scraper functionality and plugin issues
- Fixed plugin registry missing web_scraper and python_sandbox - Removed locals() from sandbox BLOCKED_CALLS for analysis - Fixed frontend health check API response parsing - Added comprehensive test validation framework - Verified all agents (planner, navigator, extractor, verifier) working - Confirmed Python sandbox execution with numpy/pandas/bs4 - Validated real-world URL scraping capabilities - Added session artifact management and memory integration54ec9cb
-
test: comprehensive ScrapeRL system tests - 100% pass rate
- Test scraper environment at LOW/MID/HIGH complexity - Test reward function with ground truth accuracy - Test plugin system installation/uninstallation - Test Gemini embeddings with similarity search - Test vector search/memory manager - Test NVIDIA and Groq AI providers - Test API endpoints (tasks, plugins, episode lifecycle) Components tested: Scraper, Reward, Plugins, Embeddings, Memory, AI Providers, API Total: 21 tests, 21 passed (100%)e123ba8 -
chore: move AI test report to docs/test
852ba36
-
docs: add manual test report with 89 tests
- Create MANUAL_TEST_REPORT.md with comprehensive UI tests - Test all Dashboard features (input view, dashboard view) - Test all popups (model, vision, agents, plugins, task type) - Test Settings, Plugins, Docs pages - Test API integrations and Docker - All 89 manual tests passed - Update TEST_REPORT.md to reference manual tests5608193 -
docs: update test report with build status
480ddb9 -
feat: redesign Dashboard and Settings with proper layout
Dashboard: - Add structured input bar with URL, instruction, task type - Add left sidebar with accordions (Agents, MCPs, Skills, APIs, Vision) - Add center area with stats header, main visualization, logs terminal - Add right sidebar with memory stats, extracted data, recent actions - Add popup modals for model/agent/plugin selection - Add task complexity selector (Low/Medium/High) Settings: - Add left sidebar navigation with sections - Add Budget & Limits section (disabled by default) - Add Appearance and Notifications settings - Add proper section-based layout App: - Add Docs tab in navigation - Remove footer, use full height layout - Compact navbar design Docs: - Add USER_GUIDE.md documentation - Add DocsPage component with react-markdown - Include agents, plugins, memory, API reference docs Types: - Add autoSave and debugMode to SystemSettings - Add AgentOption and ModelOption interfacesf881423 -
docs: update test report with 116 tests passing
5e3d245 -
docs: update test report with latest results
fbb1d81
-
docs: add test report with coverage analysis
ccaf48d -
docs: update
df47251