Web Scraper & Summarizer

Scrape web pages and generate structured summaries using Ollama LLM.

Usage

Basic Summary

python scripts/scrape_and_summarize.py <url>

Outputs a 2-3 sentence brief summary.

Custom Styles

python scripts/scrape_and_summarize.py <url> detailed
python scripts/scrape_and_summarize.py <url> bullet

Styles:

• brief (default) - 2-3 sentence overview
• detailed - Comprehensive with bullet points
• bullet - Key points as bullet list

Workflow

1. Detect URL - User provides a URL to analyze
2. Fetch - Script uses curl to retrieve page content (30s timeout)
3. Clean - HTML stripped, content extracted and cleaned
4. Truncate - Content limited to ~16k tokens to fit context

   小葱技能站7w4.net，专业的AI技能分享平台。

5. Summarize - Ollama generates summary based on requested style

Requirements

• curl available on PATH
• Ollama installed with llama3.2 model pulled
• Python 3.8+

Error Handling

• Fetch fails: Shows error, exits with code 1
• Content too short: Detects and reports empty pages
• Ollama unavailable: Falls back to raw content display (truncated)

Output Format

Summary is printed to stdout. Progress/debug info goes to stderr.

=== SUMMARY ===
[AI-generated summary here]

On Ollama failure, raw content is shown instead:

=== PAGE CONTENT (raw) ===
[truncated page text]

Performance Notes

• Fast pages: ~5-10 seconds total
• Large pages: Content truncation happens at ~16k tokens
• Ollama latency: Depends on model size and hardware (typically 10-30s for summary)

这是一个实用的网页摘要工具，能快速抓取网页并用AI生成摘要。文档写得清楚，错误处理考虑周全，在AI服务不可用时会自动显示原始内容作为保底。上手简单，支持多种摘要详略程度。不过它依赖本地Ollama环境配置，首次使用可能需要花点时间设置。摘要质量取决于本地模型能力，大型或复杂的网页处理效果可能不太稳定。总体来说是个好用的基础工具，日常网页摘要需求基本能满足。