remove duplicates

my single list become bigger and bigger, but i don't have a very good tool to remove duplicats, i try to use Once Is Enough, but it always say error

is there any tool i can use to remove duplicates, my single world list is more than 50MB, or even 100mb

thank you guys

[slysnake]

that list might be big for raptorIII, might try ultra edit

i have download it
but i can't find how to delete duplicates
could u help me to explain it in detail, thank you

[daemon.azazel]

right click on the file. open with raptor / wordlist tool
click on remove duplicates tab on the left side.
check up the checkbox "keep original" on the right side.
click the "remove duplicates" button (that one with small flash unde the checkboxes)

that's all ;)

[sPlico]

I find onceisenough4 better for deduping then raptor. It chokes if you try to load too big wlist in memory, it does ok when you dedupe "from file" as it says there, but it's still easier and faster for me to do it in OIE4.

[RiceDude]

Yup I use OIE4 for all dupes removal now (even proxies). Raptor simply doesn't work as well :(

Whenever I find myself beat I resort to the old MS Excel... Lots of functionality; sort, add, delete... and the possibility to construct any macros you might want...

[High|ander]

OIE4 can merge big files?

[RiceDude]

It sure can Highlander :) I have merged over 200mb in under a minute