Update README.txt

README.txt

@@ -1,22 +1,14 @@
 rarbg
 Backup of magnets from RARBG
 
-Currently:
+everything.7z holds everything that i compiled (3,468,029)
-clean.py is my Python script for cleaning up magnets post-extraction. I think it might have some finnicky thing going on with the way it fixes two magnets in one line, but it works.
+xrmb.7z holds all of xrmb's rarbg magnets (2,843,291)
-moviesrarbg.txt holds my original post, cleaned up a lot. (117,392)
+sorted holds nicely categorized and useful magnets for shows and movies
-showsother.txt holds my original post, cleaned up a little. (137,671)
-showsrarbg.txt holds my original post, cleaned up a lot. (11,699)
-everything.7z holds everything that i've compiled (3,468,029)
 
---- everything.7z is complete ---
+Some of the stuff in everything.7z did not come from RARBG, but I just left it all. Xrmb's database is all from RARBG.
 
-(and it would take a large new backup find to convince me otherwise)
+This repository was mentioned on TorrentFreak - 
-(and convince me to update it)
+https://torrentfreak.com/rarbg-over-267000-movie-tv-show-magnet-links-appear-online-230601/
 
-Some of the stuff in everything.7z did not come from RARBG, and that is my next step. I'm accepting suggestions for how to filter those. My current best idea is to write something to look for commonly formatted titles, like TITLE.YEAR.RESOLUTION.SOURCE.ENCODING-GROUP, but I'll need input on what porn/music/games titles usually looked like on RARBG, I'm not familiar with them.
+More info about this project can be found on Reddit - 
-I'll filter everything.7z and split it into its relevant categories. For example, the one I'm most excited for is a .txt file dedicated to solely 1080p BluRay x265 -RARBG movies.
+https://www.reddit.com/r/Piracy/comments/13wn554/my_rarbg_magnet_backup_268k/
-Thanks guys.
-
-This repository was mentioned on TorrentFreak - https://torrentfreak.com/rarbg-over-267000-movie-tv-show-magnet-links-appear-online-230601/
-
-Much more info about this project can be found on Reddit - https://www.reddit.com/r/Piracy/comments/13wn554/my_rarbg_magnet_backup_268k/

clean.py

@@ -1,70 +0,0 @@
-def fix(line, data):
-    try:
-        hash = line[20:[pos for pos, char in enumerate(line) if char == '&'][0]].lower()#hash is end of prefix to first '&', lowercased
-    except:#if no '&dn='
-        hash = line[20:]
-        line = line + '&dn='
-    try:
-        int(hash, 16)#check if hash is hexadecimal
-    except:
-        return
-    if line.count('&') > 1:#look for trackers
-        location = 0
-        tocheck = []
-        while location < len(line):#find all occurences of '&'
-            location = line.find('&', location)
-            if location == -1:
-                break
-            tocheck.append(location)
-            location += 1
-        for index in tocheck:#iterate through occurences of '&'
-            try:
-                if (line[index + 1] == 't') and (line[index + 2] == 'r') and (line[index + 3] == '='):#if occurence is part of a tracker then ignore
-                    pass
-                else:#if not, it's part of the title so replace it
-                    line = line[:index] + line[index + 1:]
-            except IndexError:
-                line = line[:index] + line[index + 1:]
-        if line.count('&') > 1:#if it actually has only trackers now
-            title = line[[pos for pos, char in enumerate(line) if char == '='][1] + 1:[pos for pos, char in enumerate(line) if char == '&'][1]]#title is second '=' to second '&'
-        else:
-            title = line[[pos for pos, char in enumerate(line) if char == '='][1] + 1:]#title is second '=' to end if no trackers
-    else:
-        title = line[[pos for pos, char in enumerate(line) if char == '='][1] + 1:]#title is second '=' to end if no trackers
-    title = ''.join(char for char in title if ord(char) < 128)#strip non-ascii characters
-    linesplit = ['magnet:?xt=urn:btih:', hash, '&dn=', title]
-    data.append(linesplit)
-    return data
-
-data = []#lists within list
-with open('everything.txt', encoding='utf-8') as file:#open file
-    for line in file:
-        line = line.strip()
-        if line.startswith('magnet:?xt=urn:btih:'):#check for validity
-            if 'magnet:?xt=urn:btih:' in line[20:]:#check for paste errors on my part
-                secondline = line[line.find('magnet:?xt=urn:btih:', 20):]#the second magnet link in this line
-                line = line[:line.find('magnet:?xt=urn:btih:', 20)]#the first magnet link in this line
-                data = fix(secondline, data)#go ahead and add the second to data
-            if 'magnetxturnbtih' in line[20:]:#paste errors that got symbols removed (and 'd' after the first '&', for some reason)
-                hash = line[line.find('magnetxturnbtih', 20) + 15:line.find('n', line.find('magnetxturnbtih', 20) + 15)]#pull just the hash of the second magnet, which stretches from the end of the magnet prefix to the first occurrence of 'n' past the prefix
-                title = line[line.find('n', line.find('magnetxturnbtih', 20) + 15) + 1:]#title stretches from that 'n' to the end (any trackers will be stripped out later)
-                secondline = 'magnet:?xt=urn:btih:' + hash + '&dn=' + title#put it back together
-                line = line[:line.find('magnetxturnbtih', 20)]
-                data = fix(secondline, data)
-            data = fix(line, data)#add split line to data
-
-for magnet in data:
-    for character in ['`', '~', '!', '@', '#', '$', '%', '^', '&', '*', '(', ')', '_', '+', '=', '[', '{', ']', '}', '\\', '|', ';', ':', '\'', '\"', ',', '<', '>', '?', '/']:
-        magnet[3] = magnet[3].replace(character, '')#get rid of symbols except '.' and '-'
-    magnet[3] = magnet[3].replace(' ', '.')#replace spaces
-
-dic = {}#dictionary to eliminate duplicate hashes
-for i in sorted(data, key=lambda x: x[3]):#sorted data because it lets me replace null titles because the last duplicate keeps the title and nulls are listed first in sort
-    dic[i[0] + i[1]] = i[2] + i[3]
-results = []
-for value in sorted(dic, key=dic.get):#sort dictionary
-    results.append('{}{}'.format(value, dic[value]))
-
-with open('output.txt', 'a', encoding='utf-8') as output:
-    for i in results:
-        output.write(i + '\n')