Skip to content

A high-performance command-line utility for finding and managing duplicate and similar files. Built with Rust, DedCore uses multi-algorithm hashing (Blake3, xxHash) and parallel processing for maximum speed.

License

Notifications You must be signed in to change notification settings

keirsalterego/dedcore

Repository files navigation

      ██████╗ ███████╗██████╗  ██████╗ ██████╗ ██████╗ ███████╗
      ██╔══██╗██╔════╝██╔══██╗██╔════╝██╔═══██╗██╔══██╗██╔════╝
      ██║  ██║█████╗  ██║  ██║██║     ██║   ██║██████╔╝█████╗  
      ██║  ██║██╔══╝  ██║  ██║██║     ██║   ██║██╔══██╗██╔══╝  
               ██████╔╝███████╗██████╔╝╚██████╗╚██████╔╝██║  ██║███████╗ v 0.1.0
              DEDCORE

DedCore: Because Your Storage Space Deserves Better

"I'm not saying you have a hoarding problem, but your hard drive called..."

License: MIT Rust Version Platforms Crates.ioTwitter Follow

🚀 Features That Won't Make You Cry

Rust Performance Security Easy to Use

Core Superpowers

  • 🦸 Multi-Algorithm Hashing: SHA-256, Blake3, and xxHash walk into a bar... your files don't stand a chance
  • Parallel Processing: Because watching paint dry is only fun the first 10,000 times
  • 🎯 Smart Filtering: Size, type, age, regex - because not all files are created equal (but some are exactly equal)
  • 🛡️ Safe Mode: Quarantine before delete - like a witness protection program for your files
  • 📊 Fancy Reports: JSON/HTML reports so detailed they make your cat photos jealous

Advanced Magic

  • 🔍 Text Similarity: Finds files that are suspiciously similar (great for finding that essay you rewrote 14 times)
  • 🖼️ Image Dedupe: Because you don't need 37 copies of the same cat picture (or do you?)
  • ⏱️ Incremental Scans: Only checks what's changed - like a detective who's good at their job
  • ⏮️ Undo Button: Because sometimes you DO need that 14th copy of the cat picture
  • 💾 Space Analysis: Shows you exactly how much space you'll save (spoiler: it's never enough)

🛠️ Installation: Choose Your Own Adventure

Cargo Install AUR Homebrew

The "I Trust Random Internet Code" Method (Recommended)

# Install via cargo (requires Rust toolchain)
cargo install dedcore

# Add to PATH (because typing the full path is so 1990s)
echo 'export PATH="$HOME/.cargo/bin:$PATH"' >> ~/.bashrc  # or ~/.zshrc
source ~/.bashrc  # or restart your terminal

The "I Like to Live Dangerously" Method

# Build from source (for the truly brave)
git clone https://github.com/manishyoudumb/dedcore.git
cd dedcore
cargo build --release
# The binary is now in ./target/release/dedcore

The "I Use Arch BTW" Method

yay -S dedcore  # Coming soon to an AUR near you!

🎯 Usage: It's Like Magic, But With More Buttons

Getting Started

Just run dedcore in your terminal and let the beautiful TUI guide you through the process:

dedcore

What to Expect

  1. Main Menu: Choose between scanning for duplicates, managing quarantined files, or getting help
  2. Interactive Scanning:
    • Select directories to scan
    • Choose file types to include or exclude
    • Set similarity thresholds with easy sliders
    • Preview results before taking any action
  3. Quarantine Management:
    • Review and manage quarantined files
    • Restore files if you change your mind
    • Permanently delete when you're ready

Pro Tips

  • Use arrow keys to navigate
  • Press Tab to switch between elements
  • Enter selects the highlighted option
  • Esc or q to go back/exit

🚧 Upcoming Features (AKA The "We're Working On It" Section)

Coming Soon™

  • 🤖 AI-Powered Dedupe: Because sometimes only AI can understand why you have 37 versions of the same document
  • 🌐 Cloud Integration: Because the cloud is just someone else's computer
  • 📱 Mobile App: For when you need to dedupe on the go (we don't judge)
  • 🎨 Better UI: Because terminal colors are nice, but have you seen gradients?

Future Dreams (AKA The "Maybe One Day" List)

  • 🔮 Predictive Dedupe: Knows what you're going to duplicate before you do
  • 🧠 Blockchain Integration: Because why not make deduplication 1000x slower?
  • 🤖 Robot Butler: Optional feature that also makes you coffee (batteries not included)

🤝 Contributing

Found a bug? Have a feature request? Want to make fun of our code? We'd love to hear from you!

  1. Fork it (please be gentle)
  2. Create your feature branch (git checkout -b feature/amazing)
  3. Commit your changes (git commit -am 'Add some amazing feature')
  4. Push to the branch (git push origin feature/amazing)
  5. Open a Pull Request (and pray to the merge gods)

📜 License

This project is licensed under the MIT License - see the LICENSE file for details.

Disclaimer: We're not responsible for any existential crises caused by realizing how many duplicate files you actually have.


💖 Support

If you like DedCore, consider giving it a ⭐ on GitHub (or a new hard drive):


Made with ❤️, 🦀, and way too many duplicate files.

Text Similarity Options

  • Minimum similarity threshold (0.0 to 1.0) for grouping text files

    • 1.0 = files must be identical
    • 0.8 = files must be at least 80% similar
    • Lower values will group more files together
  • Supported text file extensions: .txt, .md, .rs, .py, .js, .ts, .java, .c, .cpp, .h, .hpp, .html, .css, .json, .toml, .yaml, .yml, .xml, .csv, .log

Image Similarity

Finding Similar Images

DedCore's TUI makes it easy to find similar images:

  1. Select "Scan for Duplicates" from the main menu
  2. Choose the directory containing your images
  3. Adjust the similarity threshold using the intuitive slider
  4. Let DedCore do its magic!

How It Works

Image similarity is calculated using perceptual hashing algorithms that generate fingerprints of images. These fingerprints can be compared to determine how similar two images are, even if they have different sizes, formats, or minor modifications.


📚 License

MIT License

About

A high-performance command-line utility for finding and managing duplicate and similar files. Built with Rust, DedCore uses multi-algorithm hashing (Blake3, xxHash) and parallel processing for maximum speed.

Resources

License

Stars

Watchers

Forks

Packages

No packages published

Languages