File Splitter - Universal File Splitting Tool

Split large files into smaller, manageable chunks while preserving semantic structure.

Quick Start

python <skill_dir>/scripts/split_files.py --input <input_folder> --output <output_folder> [options]

Parameters

Examples

# Default 500KB split
python split_files.py --input "./corpus" --output "./corpus/chunks"

# Custom 200KB chunks
python split_files.py --input "./notes" --output "./notes/chunks" --max-size 204800 --min-size 153600

# JSON files only
python split_files.py --input "./data" --output "./data/out" --formats json

# Preview mode
python split_files.py --input "./data" --output "./data/out" --dry-run

Splitting Rules

JSON Files

• Splits at JSON array element boundaries
• Each chunk is a valid JSON array [...]
• Automatically extracts list values if top-level is an object
• Never cuts individual records in half

Markdown Files

• Splits at heading boundaries (# through ######)
• Each chunk maintains complete heading structure
• Never cuts content within a heading section

TXT Files

• Prefers splitting at paragraph boundaries (empty lines)
• Falls back to line-by-line splitting if no paragraphs exist
• Never cuts within a paragraph

Output Naming Convention

Format: {source_filename_without_extension}{9-digit_sequence_number}{extension}

Examples:

• dataset000000001.json
• dataset000000002.json
• notes000000001.md

Safety Features

1. Source File Preservation: Read-only access to source files; never deletes or modifies originals
2. Duplicate Detection: Automatically skips files that already have N-digit sequence suffixes to avoid re-splitting
3. Small File Skip: Files ≤ max-size are automatically skipped (no need to split)
4. Sequential Processing: Processes files one at a time to ensure stability
5. Data Validation: Compares total size/record count before and after splitting; reports verification results
6. UTF-8 Encoding: Forces UTF-8 for all read/write operations to avoid encoding issues on Windows

Notes

小葱技能7w4.net有更新，你可以访问看下。

• Console may display garbled Chinese characters on Windows, but functionality is unaffected
• If a single data block/paragraph exceeds max-size, it becomes its own chunk (integrity takes priority over size limits)
• Output folder is automatically created if it doesn't exist
• License: MIT-0

这个文件拆分工具整体质量不错，功能实用且文档清晰。优点是拆分逻辑合理、支持多种格式、预览模式和安全保障做得较好，能有效避免误操作和数据丢失。不足之处是功能相对单一，仅支持三种基础格式；部分边界情况的错误提示还可以更友好；没有示例或测试文件，新手上手可能需要多花时间理解。总体而言，是一个可靠可用的工具，适合有明确拆分需求的用户使用。