Update CI to create an automatic beta release

Co-authored-by: Satsuoni <satsuoni@hotmail.com>

.github/workflows/main.yml

@@ -9,8 +9,10 @@ jobs:
     steps:
       - name: Checkout
         uses: actions/checkout@v2
+
       - name: Package
         run: python3 make_release.py
+
       - name: Upload
         uses: actions/upload-artifact@v2
         with: 
@@ -18,3 +20,45 @@ jobs:
           path: |
             DeDRM_tools_*.zip
             DeDRM_tools.zip
+
+      - name: Delete old release
+        uses: cb80/delrel@latest
+        with:
+          tag: autorelease
+          token: ${{ github.token }}
+
+      - name: Delete old tag
+        uses: dev-drprasad/delete-tag-and-release@v1.0
+        with:
+          tag_name: autorelease
+          github_token: ${{ github.token }}
+          delete_release: true
+      
+      - name: Prepare release
+        run: cp DeDRM_tools.zip DeDRM_alpha_${{ github.sha }}.zip
+
+      - name: Auto-release
+        id: autorelease
+        uses: softprops/action-gh-release@v1
+        with: 
+          tag_name: autorelease
+          token: ${{ github.token }}
+          name: Automatic alpha release with latest changes
+          body: |
+            This release is automatically generated by Github for each commit. 
+
+            This means, every time a change is made to this repo, this release will be updated to contain an untested copy of the plugin at that stage. This will contain the most up-to-date code, but it's not tested at all and may be broken.
+
+            Last update based on Git commit ${{ github.sha }}.
+          prerelease: true
+          draft: true
+          files: DeDRM_alpha_${{ github.sha }}.zip
+
+      - name: Make release public
+        uses: irongut/EditRelease@v1.2.0
+        with:
+          token: ${{ github.token }}
+          id: ${{ steps.autorelease.outputs.id }}
+          draft: false
+          prerelease: true
+

CHANGELOG.md

@@ -73,7 +73,22 @@ List of changes since the fork of Apprentice Harper's repository:
 - Fix some more Calibre-6 bugs in the Obok plugin (should fix #114).
 - Fix a bug where invalid Adobe keys could cause the plugin to stop trying subsequent keys (partially fixes #109).
 - Fix DRM removal sometimes resetting the ZIP's internal "external_attr" value on Calibre 5 and newer.
-- Fix PDF decryption issues on Calibre 4 (hopefully fixes #104).
+- Fix tons of PDF decryption issues (hopefully fixes #104 and other PDF-related issues).
 - Small Python 2 / Calibre 4 bugfix for Obok.
 - Removing ancient AlfCrypto machine code libraries, moving all encryption / decryption to Python code.
 - General cleanup and removal of dead code.
+- Fix a bug where ADE account keys weren't automatically imported from the DeACSM plugin when importing a PDF file.
+- Re-enable Xrefs in exported PDF files since the file corruption bug is hopefully fixed. Please open bug reports if you encounter new issues with PDF files.
+- Fix a bug that would sometimes cause corrupted keys to be added when adding them through the config dialog (fixes #145, #134, #119, #116, #115, #109).
+- Update the README (fixes #136) to indicate that Apprentice Harper's version is no longer being updated.
+- Fix a bug where PDFs with empty arrays (`<>`) in a PDF object failed to decrypt, fixes #183.
+- Automatically strip whitespace from entered Amazon Kindle serial numbers, should fix #158.
+- Obok: Add new setting option "Add new entry" for duplicate books to always add them to the Calibre database as a new book. Fixes #148.
+- Obok: Fix where changing the Calibre UI language to some languages would cause the "duplicate book" setting to reset.
+- Fix Python3 bug in stylexml2css.php script, fixes #232.
+- PDF: Ignore invalid PDF objids unless the script is running in strict mode. Fixes some PDFs, apparently. Fixes #233. 
+- Bugfix: EPUBs with remaining content in the encryption.xml after decryption weren't written correctly. 
+- Support for Adobe's 'aes128-cbc-uncompressed' encryption method (fixes #242).
+- Two bugfixes for Amazon DeDRM from Satuoni ( https://github.com/noDRM/DeDRM_tools/issues/315#issuecomment-1508305428 ) and andrewc12 ( https://github.com/andrewc12/DeDRM_tools/commit/d9233d61f00d4484235863969919059f4d0b2057 ) that might make the plugin work with newer versions.
+- Fix font decryption not working with some books (fixes #347), thanks for the patch @bydioeds. 
+- Fix a couple unicode errors for Python2 in Kindle and Nook code.

DeDRM_plugin/__init__.py

@@ -134,7 +134,7 @@ except:
     config_dir = ""
 
 
-from utilities import SafeUnbuffered
+import utilities
 
 
 PLUGIN_NAME = __version.PLUGIN_NAME
@@ -733,10 +733,10 @@ class DeDRM(FileTypePlugin):
             if newkey is not None: 
                 if codecs.encode(newkey, 'hex').decode('ascii') not in dedrmprefs['adeptkeys'].values():
                     print("{0} v{1}: Found new key '{2}' in DeACSM plugin".format(PLUGIN_NAME, PLUGIN_VERSION, newname))
-                    newkeys.append(keyvalue)
+                    newkeys.append(newkey)
                     newnames.append(newname)
         except:
-            pass
+            traceback.print_exc()
 
         if len(newkeys) > 0:
             try:
@@ -770,7 +770,7 @@ class DeDRM(FileTypePlugin):
 
                     print("{0} v{1}: Failed to decrypt with new default key after {2:.1f} seconds".format(PLUGIN_NAME, PLUGIN_VERSION,time.time()-self.starttime))
             except Exception as e:
-                pass
+                traceback.print_exc()
 
 
         # Unable to decrypt the PDF with any of the existing keys. Is it a B&N PDF?
@@ -998,8 +998,8 @@ class DeDRM(FileTypePlugin):
     def run(self, path_to_ebook):
 
         # make sure any unicode output gets converted safely with 'replace'
-        sys.stdout=SafeUnbuffered(sys.stdout)
+        sys.stdout=utilities.SafeUnbuffered(sys.stdout)
-        sys.stderr=SafeUnbuffered(sys.stderr)
+        sys.stderr=utilities.SafeUnbuffered(sys.stderr)
 
         print("{0} v{1}: Trying to decrypt {2}".format(PLUGIN_NAME, PLUGIN_VERSION, os.path.basename(path_to_ebook)))
         self.starttime = time.time()

DeDRM_plugin/androidkindlekey.py

@@ -396,7 +396,7 @@ def gui_main():
                 return
             self.status['text'] = "Select backup.ab file"
 
-    argv=unicode_argv()
+    argv=sys.argv()
     progpath, progname = os.path.split(argv[0])
     root = tkinter.Tk()
     root.title("Kindle for Android Key Extraction v.{0}".format(__version__))

DeDRM_plugin/config.py

@@ -1152,7 +1152,8 @@ class AddAdeptDialog():
                 zip_function = zip
 
             for key, name in zip_function(defaultkeys, defaultnames):
-                if codecs.encode(key,'hex').decode("latin-1") in self.parent.plugin_keys.values():
+                key = codecs.encode(key,'hex').decode("latin-1")
+                if key in self.parent.plugin_keys.values():
                     print("Found key '{0}' in ADE - already present, skipping.".format(name))
                 else:
                     self.new_keys.append(key)
@@ -1167,8 +1168,8 @@ class AddAdeptDialog():
             key, name = checkForDeACSMkeys()
 
             if key is not None: 
-
+                key = codecs.encode(key,'hex').decode("latin-1")
-                if codecs.encode(key,'hex').decode("latin-1") in self.parent.plugin_keys.values():
+                if key in self.parent.plugin_keys.values():
                     print("Found key '{0}' in DeACSM - already present, skipping.".format(name))
                 else: 
                     # Found new key, add that.
@@ -1202,7 +1203,7 @@ class AddAdeptDialog():
 
     @property
     def key_value(self):
-        return codecs.encode(self.new_keys[0],'hex').decode("utf-8")
+        return codecs.encode(self.new_keys[0],'hex').decode("latin-1")
 
     
     @property
@@ -1325,7 +1326,7 @@ class AddSerialDialog(QDialog):
 
     @property
     def key_value(self):
-        return str(self.key_ledit.text()).replace(' ', '')
+        return str(self.key_ledit.text()).replace(' ', '').replace('\r', '').replace('\n', '').replace('\t', '')
 
     def accept(self):
         if len(self.key_name) == 0 or self.key_name.isspace():

DeDRM_plugin/epubfontdecrypt.py

@@ -2,7 +2,7 @@
 # -*- coding: utf-8 -*-
 
 # epubfontdecrypt.py
-# Copyright © 2021 by noDRM
+# Copyright © 2021-2023 by noDRM
 
 # Released under the terms of the GNU General Public Licence, version 3
 # <http://www.gnu.org/licenses/>
@@ -10,6 +10,7 @@
 
 # Revision history:
 #   1 - Initial release
+#   2 - Bugfix for multiple book IDs, reported at #347
 
 """
 Decrypts / deobfuscates font files in EPUB files
@@ -18,7 +19,7 @@ Decrypts / deobfuscates font files in EPUB files
 from __future__ import print_function
 
 __license__ = 'GPL v3'
-__version__ = "1"
+__version__ = "2"
 
 import os
 import traceback
@@ -193,9 +194,10 @@ def decryptFontsBook(inpath, outpath):
                 pass
 
             try: 
-                identify_element = container.find(packageNS("metadata")).find(metadataDCNS("identifier"))
+                identify_elements = container.find(packageNS("metadata")).findall(metadataDCNS("identifier"))
-                if (secret_key_name is None or secret_key_name == identify_element.get("id")):
+                for element in identify_elements:
-                    font_master_key = identify_element.text
+                    if (secret_key_name is None or secret_key_name == element.get("id")):
+                        font_master_key = element.text
             except: 
                 pass
 

DeDRM_plugin/ignoblekeyNookStudy.py

@@ -54,15 +54,26 @@ def getNookLogFiles():
         paths = set()
         if 'LOCALAPPDATA' in os.environ.keys():
             # Python 2.x does not return unicode env. Use Python 3.x
-            path = winreg.ExpandEnvironmentStrings("%LOCALAPPDATA%")
+            if sys.version_info[0] == 2:
+                path = winreg.ExpandEnvironmentStrings(u"%LOCALAPPDATA%")
+            else:
+                path = winreg.ExpandEnvironmentStrings("%LOCALAPPDATA%")
             if os.path.isdir(path):
                 paths.add(path)
         if 'USERPROFILE' in os.environ.keys():
             # Python 2.x does not return unicode env. Use Python 3.x
-            path = winreg.ExpandEnvironmentStrings("%USERPROFILE%")+"\\AppData\\Local"
+            if sys.version_info[0] == 2:
+                path = winreg.ExpandEnvironmentStrings(u"%USERPROFILE%")+u"\\AppData\\Local"
+            else:
+                path = winreg.ExpandEnvironmentStrings("%USERPROFILE%")+"\\AppData\\Local"
+
             if os.path.isdir(path):
                 paths.add(path)
-            path = winreg.ExpandEnvironmentStrings("%USERPROFILE%")+"\\AppData\\Roaming"
+
+            if sys.version_info[0] == 2:  
+                path = winreg.ExpandEnvironmentStrings(u"%USERPROFILE%")+u"\\AppData\\Roaming"
+            else:
+                path = winreg.ExpandEnvironmentStrings("%USERPROFILE%")+"\\AppData\\Roaming"
             if os.path.isdir(path):
                 paths.add(path)
         # User Shell Folders show take precedent over Shell Folders if present

DeDRM_plugin/ineptepub.py

@@ -89,15 +89,16 @@ class Decryptor(object):
     def __init__(self, bookkey, encryption):
         enc = lambda tag: '{%s}%s' % (NSMAP['enc'], tag)
         self._aes = AES.new(bookkey, AES.MODE_CBC, b'\x00'*16)
-        encryption = etree.fromstring(encryption)
+        self._encryption = etree.fromstring(encryption)
         self._encrypted = encrypted = set()
+        self._encryptedForceNoDecomp = encryptedForceNoDecomp = set()
         self._otherData = otherData = set()
 
         self._json_elements_to_remove = json_elements_to_remove = set()
         self._has_remaining_xml = False
         expr = './%s/%s/%s' % (enc('EncryptedData'), enc('CipherData'),
                                enc('CipherReference'))
-        for elem in encryption.findall(expr):
+        for elem in self._encryption.findall(expr):
             path = elem.get('URI', None)
             encryption_type_url = (elem.getparent().getparent().find("./%s" % (enc('EncryptionMethod'))).get('Algorithm', None))
             if path is not None:
@@ -106,6 +107,11 @@ class Decryptor(object):
                     path = path.encode('utf-8')
                     encrypted.add(path)
                     json_elements_to_remove.add(elem.getparent().getparent())
+                elif (encryption_type_url == "http://ns.adobe.com/adept/xmlenc#aes128-cbc-uncompressed"):
+                    # Adobe uncompressed, for stuff like video files
+                    path = path.encode('utf-8')
+                    encryptedForceNoDecomp.add(path)
+                    json_elements_to_remove.add(elem.getparent().getparent())
                 else:
                     path = path.encode('utf-8')
                     otherData.add(path)
@@ -134,14 +140,15 @@ class Decryptor(object):
         return decompressed_bytes
 
     def decrypt(self, path, data):
-        if path.encode('utf-8') in self._encrypted:
+        if path.encode('utf-8') in self._encrypted or path.encode('utf-8') in self._encryptedForceNoDecomp:
             data = self._aes.decrypt(data)[16:]
             if type(data[-1]) != int:
                 place = ord(data[-1])
             else:
                 place = data[-1]
             data = data[:-place]
-            data = self.decompress(data)
+            if not path.encode('utf-8') in self._encryptedForceNoDecomp:
+                data = self.decompress(data)
         return data
 
 # check file to make check whether it's probably an Adobe Adept encrypted ePub

DeDRM_plugin/ineptpdf.py

@@ -117,7 +117,7 @@ def SHA256(message):
 # 1 = only if present in input
 # 2 = always
 
-GEN_XREF_STM = 0
+GEN_XREF_STM = 1
 
 # This is the value for the current document
 gen_xref_stm = False # will be set in PDFSerializer
@@ -270,6 +270,11 @@ END_STRING = re.compile(br'[()\\]')
 OCT_STRING = re.compile(br'[0-7]')
 ESC_STRING = { b'b':8, b't':9, b'n':10, b'f':12, b'r':13, b'(':40, b')':41, b'\\':92 }
 
+class EmptyArrayValue(object):
+    def __str__(self):
+        return "<>"
+
+
 class PSBaseParser(object):
 
     '''
@@ -519,6 +524,13 @@ class PSBaseParser(object):
         if c == b'<':
             self.add_token(KEYWORD_DICT_BEGIN)
             i += 1
+        if c == b'>':
+            # Empty array without any contents. Why though?
+            # We need to add some dummy python object that will serialize to 
+            # nothing, otherwise the code removes the whole array.
+            self.add_token(EmptyArrayValue())
+            i += 1
+
         return (self.parse_main, i)
 
     def parse_wclose(self, s, i):
@@ -544,7 +556,6 @@ class PSBaseParser(object):
         else: 
             token = HEX_PAIR.sub(lambda m: bytes([int(m.group(0), 16)]),
                                                  SPC.sub(b'', self.token))
-
         self.add_token(token)
         return (self.parse_main, j)
 
@@ -565,7 +576,11 @@ class PSBaseParser(object):
         while 1:
             self.fillbuf()
             if eol:
-                c = bytes([self.buf[self.charpos]])
+                if sys.version_info[0] == 2: 
+                    c = self.buf[self.charpos]
+                else: 
+                    c = bytes([self.buf[self.charpos]])
+
                 # handle '\r\n'
                 if c == b'\n':
                     linebuf += c
@@ -575,10 +590,17 @@ class PSBaseParser(object):
             if m:
                 linebuf += self.buf[self.charpos:m.end(0)]
                 self.charpos = m.end(0)
-                if bytes([linebuf[-1]]) == b'\r':
+                if sys.version_info[0] == 2:
-                    eol = True
+                    if linebuf[-1] == b'\r':
+                        eol = True
+                    else:
+                        break
                 else: 
-                    break
+                    if bytes([linebuf[-1]]) == b'\r':
+                        eol = True
+                    else:
+                        break
+                    
             else:
                 linebuf += self.buf[self.charpos:]
                 self.charpos = len(self.buf)
@@ -954,9 +976,14 @@ class PDFStream(PDFObject):
                     for i in range(0, len(data), columns+1):
                         pred = data[i]
                         ent1 = data[i+1:i+1+columns]
-                        if pred == 2:
+                        if sys.version_info[0] == 2:
-                            ent1 = b''.join(bytes([(a+b) & 255]) \
+                            if pred == '\x02':
-                                           for (a,b) in zip(ent0,ent1))
+                                ent1 = ''.join(chr((ord(a)+ord(b)) & 255) \
+                                               for (a,b) in zip(ent0,ent1))
+                        else: 
+                            if pred == 2:
+                                ent1 = b''.join(bytes([(a+b) & 255]) \
+                                            for (a,b) in zip(ent0,ent1))
                         buf += ent1
                         ent0 = ent1
                     data = buf
@@ -1404,7 +1431,10 @@ class PDFDocument(object):
         x = ARC4.new(hash).decrypt(Odata) # 4
         if R >= 3:
             for i in range(1,19+1):
-                k = b''.join(bytes([c ^ i]) for c in hash )
+                if sys.version_info[0] == 2:
+                    k = b''.join(chr(ord(c) ^ i) for c in hash )
+                else: 
+                    k = b''.join(bytes([c ^ i]) for c in hash )
                 x = ARC4.new(k).decrypt(x)
 
 
@@ -1462,7 +1492,10 @@ class PDFDocument(object):
             hash.update(docid[0]) # 3
             x = ARC4.new(key).decrypt(hash.digest()[:16]) # 4
             for i in range(1,19+1):
-                k = b''.join(bytes([c ^ i]) for c in key )
+                if sys.version_info[0] == 2:
+                    k = b''.join(chr(ord(c) ^ i) for c in key )
+                else: 
+                    k = b''.join(bytes([c ^ i]) for c in key )
                 x = ARC4.new(k).decrypt(x)
             u1 = x+x # 32bytes total
         if R == 2:
@@ -1490,8 +1523,8 @@ class PDFDocument(object):
 
         # check owner pass:
         retval = self.check_owner_password(password, docid, param)
-        if retval is True or retval is not None:
+        if retval is True or (retval is not False and retval is not None):
-            #print("Owner pass is valid - " + str(retval))
+            #print("Owner pass is valid")
             if retval is True:
                 self.decrypt_key = self.recover_encryption_key_with_password(password, docid, param)
             else:
@@ -1500,7 +1533,7 @@ class PDFDocument(object):
         if self.decrypt_key is None or self.decrypt_key is True or self.decrypt_key is False:
             # That's not the owner password. Check if it's the user password.
             retval = self.check_user_password(password, docid, param)
-            if retval is True or retval is not None:
+            if retval is True or (retval is not False and retval is not None):
                 #print("User pass is valid")
                 if retval is True:
                     self.decrypt_key = self.recover_encryption_key_with_password(password, docid, param)
@@ -1569,7 +1602,13 @@ class PDFDocument(object):
 
     def initialize_ebx_ignoble(self, keyb64, docid, param):
         self.is_printable = self.is_modifiable = self.is_extractable = True
-        key = keyb64.decode('base64')[:16]
+
+        try: 
+            key = keyb64.decode('base64')[:16]
+            # This will probably always error, but I'm not 100% sure, so lets leave the old code in.
+        except AttributeError: 
+            key = codecs.decode(keyb64.encode("ascii"), 'base64')[:16]
+
 
         length = int_value(param.get('Length', 0)) / 8
         rights = codecs.decode(str_value(param.get('ADEPT_LICENSE')), "base64")
@@ -1597,13 +1636,15 @@ class PDFDocument(object):
             else:
                 print("ebx_V is %d  and ebx_type is %d" % (ebx_V, ebx_type))
                 print("length is %d and len(bookkey) is %d" % (length, len(bookkey)))
-                print("bookkey[0] is %d" % bookkey[0])
+                if len(bookkey) > 0:
+                    print("bookkey[0] is %d" % bookkey[0])
                 raise ADEPTError('error decrypting book session key - mismatched length')
         else:
             # proper length unknown try with whatever you have
             print("ebx_V is %d  and ebx_type is %d" % (ebx_V, ebx_type))
             print("length is %d and len(bookkey) is %d" % (length, len(bookkey)))
-            print("bookkey[0] is %d" % ord(bookkey[0]))
+            if len(bookkey) > 0:
+                print("bookkey[0] is %d" % ord(bookkey[0]))
             if ebx_V == 3:
                 V = 3
             else:
@@ -1669,13 +1710,15 @@ class PDFDocument(object):
             else:
                 print("ebx_V is %d  and ebx_type is %d" % (ebx_V, ebx_type))
                 print("length is %d and len(bookkey) is %d" % (length, len(bookkey)))
-                print("bookkey[0] is %d" % bookkey[0])
+                if len(bookkey) > 0:
+                    print("bookkey[0] is %d" % bookkey[0])
                 raise ADEPTError('error decrypting book session key - mismatched length')
         else:
             # proper length unknown try with whatever you have
             print("ebx_V is %d  and ebx_type is %d" % (ebx_V, ebx_type))
             print("length is %d and len(bookkey) is %d" % (length, len(bookkey)))
-            print("bookkey[0] is %d" % bookkey[0])
+            if len(bookkey) > 0:
+                print("bookkey[0] is %d" % bookkey[0])
             if ebx_V == 3:
                 V = 3
             else:
@@ -1723,7 +1766,11 @@ class PDFDocument(object):
         data = data[16:]
         plaintext = AES.new(key,AES.MODE_CBC,ivector).decrypt(data)
         # remove pkcs#5 aes padding
-        cutter = -1 * plaintext[-1]
+        if sys.version_info[0] == 2: 
+            cutter = -1 * ord(plaintext[-1])
+        else: 
+            cutter = -1 * plaintext[-1]
+
         plaintext = plaintext[:cutter]
         return plaintext
 
@@ -1784,7 +1831,19 @@ class PDFDocument(object):
                 try:
                     obj = objs[i]
                 except IndexError:
-                    raise PDFSyntaxError('Invalid object number: objid=%r' % (objid))
+                    # This IndexError used to just raise an exception.
+                    # Unfortunately that seems to break some PDFs, see this issue:
+                    # https://github.com/noDRM/DeDRM_tools/issues/233
+                    # I'm not sure why this is the case, but lets try only raising that exception
+                    # when in STRICT mode, and make it a warning otherwise.
+                    if STRICT:
+                        raise PDFSyntaxError('Invalid object number: objid=%r' % (objid))
+
+                    print('Invalid object number: objid=%r' % (objid))
+                    print("Continuing anyways?")
+                    print("If the resulting PDF is corrupted, please open a bug report.")
+                    return None
+
                 if isinstance(obj, PDFStream):
                     obj.set_objid(objid, 0)
             else:
@@ -2199,7 +2258,7 @@ class PDFSerializer(object):
         elif isinstance(obj, bytearray):
             self.write(b'(%s)' % self.escape_string(obj))
         elif isinstance(obj, bytes):
-            self.write(b'(%s)' % self.escape_string(obj))
+            self.write(b'<%s>' % binascii.hexlify(obj).upper())
         elif isinstance(obj, str):
             self.write(b'(%s)' % self.escape_string(obj.encode('utf-8')))
         elif isinstance(obj, bool):
@@ -2226,6 +2285,20 @@ class PDFSerializer(object):
                 self.write(b'(deleted)')
             else:
                 data = obj.get_decdata()
+
+                # Fix length:
+                # We've decompressed and then recompressed the PDF stream.
+                # Depending on the algorithm, the implementation, and the compression level, 
+                # the resulting recompressed stream is unlikely to have the same length as the original.
+                # So we need to update the PDF object to contain the new proper length.
+
+                # Without this change, all PDFs exported by this plugin are slightly corrupted - 
+                # even though most if not all PDF readers can correct that on-the-fly.
+
+                if 'Length' in obj.dic: 
+                    obj.dic['Length'] = len(data)
+
+
                 self.serialize_object(obj.dic)
                 self.write(b'stream\n')
                 self.write(data)

DeDRM_plugin/ion.py

@@ -57,6 +57,7 @@ except ImportError:
                 # Windows-friendly choice: pylzma wheels
                 import pylzma as lzma
 
+from kfxtables import *
 
 TID_NULL = 0
 TID_BOOLEAN = 1
@@ -769,6 +770,7 @@ def pkcs7unpad(msg, blocklen):
 
 
 # every VoucherEnvelope version has a corresponding "word" and magic number, used in obfuscating the shared secret
+# 4-digit versions use their own obfuscation/scramble. It does not seem to depend on the "word" and number
 OBFUSCATION_TABLE = {
     "V1":    (0x00, None),
     "V2":    (0x05, b'Antidisestablishmentarianism'),
@@ -779,26 +781,26 @@ OBFUSCATION_TABLE = {
     "V7":    (0x05, b'\x10\x1bJ\x18\nh!\x10"\x03>Z\'\r\x01]W\x06\x1c\x1e?\x0f\x13'),
     "V8":    (0x09, b"K\x0c6\x1d\x1a\x17pO}Rk\x1d'w1^\x1f$\x1c{C\x02Q\x06\x1d`"),
     "V9":    (0x05, b'X.\x0eW\x1c*K\x12\x12\t\n\n\x17Wx\x01\x02Yf\x0f\x18\x1bVXPi\x01'),
-    "V10":   (0x07, b'z3\n\x039\x12\x13`\x06=v,\x02MTK\x1e%}L\x1c\x1f\x15\x0c\x11\x02\x0c\n8\x17p'),
+    "V10":   (0x07, b'z3\n\x039\x12\x13`\x06=v;\x02MTK\x1e%}L\x1c\x1f\x15\x0c\x11\x02\x0c\n8\x17p'),
     "V11":   (0x05, b'L=\nhVm\x07go\n6\x14\x06\x16L\r\x02\x0b\x0c\x1b\x04#p\t'),
-    "V12":   (0x06, b',n\x1d\rl\x13\x1c\x13\x16p\x14\x07U\x0c\x1f\x19w\x16\x16\x1d5T'),
+    "V12":   (0x06, b';n\x1d\rl\x13\x1c\x13\x16p\x14\x07U\x0c\x1f\x19w\x16\x16\x1d5T'),
     "V13":   (0x07, b'I\x05\t\x08\x03r)\x01$N\x0fr3n\x0b062D\x0f\x13'),
     "V14":   (0x05, b"\x03\x02\x1c9\x19\x15\x15q\x1057\x08\x16\x0cF\x1b.Fw\x01\x12\x03\x13\x02\x17S'hk6"),
     "V15":   (0x0A, b'&,4B\x1dcI\x0bU\x03I\x07\x04\x1c\t\x05c\x07%ws\x0cj\t\x1a\x08\x0f'),
-    "V16":   (0x0A, b'\x06\x18`h,b><\x06PqR\x02Zc\x034\n\x16\x1e\x18\x06#e'),
+    "V16":   (0x0A, b'\x06\x18`h;b><\x06PqR\x02Zc\x034\n\x16\x1e\x18\x06#e'),
     "V17":   (0x07, b'y\r\x12\x08fw.[\x02\t\n\x13\x11\x0c\x11b\x1e8L\x10(\x13<Jx6c\x0f'),
-    "V18":   (0x07, b'I\x0b\x0e,\x19\x1aIa\x10s\x19g\\\x1b\x11!\x18yf\x0f\t\x1d7[bSp\x03'),
+    "V18":   (0x07, b'I\x0b\x0e;\x19\x1aIa\x10s\x19g\\\x1b\x11!\x18yf\x0f\t\x1d7[bSp\x03'),
     "V19":   (0x05, b'\n6>)N\x02\x188\x016s\x13\x14\x1b\x16jeN\n\x146\x04\x18\x1c\x0c\x19\x1f,\x02]'),
-    "V20":   (0x08, b'_\r\x01\x12]\\\x14*\x17i\x14\r\t!\x1e,~hZ\x12jK\x17\x1e*1'),
+    "V20":   (0x08, b'_\r\x01\x12]\\\x14*\x17i\x14\r\t!\x1e;~hZ\x12jK\x17\x1e*1'),
     "V21":   (0x07, b'e\x1d\x19|\ty\x1di|N\x13\x0e\x04\x1bj<h\x13\x15k\x12\x08=\x1f\x16~\x13l'),
     "V22":   (0x08, b'?\x17yi$k7Pc\tEo\x0c\x07\x07\t\x1f,*i\x12\x0cI0\x10I\x1a?2\x04'),
     "V23":   (0x08, b'\x16+db\x13\x04\x18\rc%\x14\x17\x0f\x13F\x0c[\t9\x1ay\x01\x1eH'),
-    "V24":   (0x06, b'|6\\\x1a\r\x10\nP\x07\x0fu\x1f\t,\rr`uv\\~55\x11]N'),
+    "V24":   (0x06, b'|6\\\x1a\r\x10\nP\x07\x0fu\x1f\t;\rr`uv\\~55\x11]N'),
-    "V25":   (0x09, b'\x07\x14w\x1e,^y\x01:\x08\x07\x1fr\tU#j\x16\x12\x1eB\x04\x16=\x06fZ\x07\x02\x06'),
+    "V25":   (0x09, b'\x07\x14w\x1e;^y\x01:\x08\x07\x1fr\tU#j\x16\x12\x1eB\x04\x16=\x06fZ\x07\x02\x06'),
     "V26":   (0x06, b'\x03IL\x1e"K\x1f\x0f\x1fp0\x01`X\x02z0`\x03\x0eN\x07'),
     "V27":   (0x07, b'Xk\x10y\x02\x18\x10\x17\x1d,\x0e\x05e\x10\x15"e\x0fh(\x06s\x1c\x08I\x0c\x1b\x0e'),
     "V28":   (0x0A, b'6P\x1bs\x0f\x06V.\x1cM\x14\x02\n\x1b\x07{P0:\x18zaU\x05'),
-    "V9708": (0x05, b'\x1diIm\x08a\x17\x1e!am\x1d\x1aQ.\x16!\x06*\}x04\x11\t\x06\x04?'),
+    "V9708": (0x05, b'\x1diIm\x08a\x17\x1e!am\x1d\x1aQ.\x16!\x06*\x04\x11\t\x06\x04?'),
     "V1031": (0x08, b'Antidisestablishmentarianism'),
     "V2069": (0x07, b'Floccinaucinihilipilification'),
     "V9041": (0x06, b'>\x14\x0c\x12\x10-\x13&\x18U\x1d\x05Rlt\x03!\x19\x1b\x13\x04]Y\x19,\t\x1b'),
@@ -807,10 +809,367 @@ OBFUSCATION_TABLE = {
     "V9479": (0x09, b'\x10\x1bJ\x18\nh!\x10"\x03>Z\'\r\x01]W\x06\x1c\x1e?\x0f\x13'),
     "V9888": (0x05, b"K\x0c6\x1d\x1a\x17pO}Rk\x1d'w1^\x1f$\x1c{C\x02Q\x06\x1d`"),
     "V4648": (0x07, b'X.\x0eW\x1c*K\x12\x12\t\n\n\x17Wx\x01\x02Yf\x0f\x18\x1bVXPi\x01'),
-    "V5683": (0x05, b'z3\n\x039\x12\x13`\x06=v,\x02MTK\x1e%}L\x1c\x1f\x15\x0c\x11\x02\x0c\n8\x17p'),
+    "V5683": (0x05, b'z3\n\x039\x12\x13`\x06=v;\x02MTK\x1e%}L\x1c\x1f\x15\x0c\x11\x02\x0c\n8\x17p'),
 }
 
 
+#common str:  "PIDv3AESAES/CBC/PKCS5PaddingHmacSHA256"
+class workspace(object):
+  def __init__(self,initial_list):
+    self.work=initial_list
+  def shuffle(self,shuflist):
+    ll=len(shuflist)
+    rt=[]
+    for i in range(ll):
+      rt.append(self.work[shuflist[i]])
+    self.work=rt
+  def sbox(self,table,matrix,skplist=[]): #table is list of 4-byte integers
+    offset=0
+    nwork=list(self.work)
+    wo=0
+    toff=0
+    while offset<0x6000:
+      uv5=table[toff+nwork[wo+0]]
+      uv1=table[toff+nwork[wo+1]+0x100]
+      uv2=table[toff+nwork[wo+2]+0x200]
+      uv3=table[toff+nwork[wo+3]+0x300]
+      moff=0
+      if 0 in skplist:
+        moff+=0x400
+      else:
+        nib1=matrix[moff+offset+(uv1>>0x1c)|( (uv5>>0x18)&0xf0)]
+        moff+=0x100
+        nib2=matrix[moff+offset+(uv3>>0x1c)|( (uv2>>0x18)&0xf0)]
+        moff+=0x100
+        nib3=matrix[moff+offset+((uv1>>0x18)&0xf) |( (uv5>>0x14)&0xf0)]
+        moff+=0x100
+        nib4=matrix[moff+offset+((uv3>>0x18)&0xf) |( (uv2>>0x14)&0xf0)]
+        moff+=0x100
+      rnib1=matrix[moff+offset+nib1*0x10+nib2]
+      moff+=0x100
+      rnib2=matrix[moff+offset+nib3*0x10+nib4]
+      moff+=0x100
+      nwork[wo+0]=rnib1*0x10+rnib2
+      if 1 in skplist:
+        moff+=0x400
+      else:
+        nib1=matrix[moff+offset+((uv1>>0x14)&0xf)|( (uv5>>0x10)&0xf0)]
+        moff+=0x100
+        nib2=matrix[moff+offset+((uv3>>0x14)&0xf)|( (uv2>>0x10)&0xf0)]
+        moff+=0x100
+        nib3=matrix[moff+offset+((uv1>>0x10)&0xf) |( (uv5>>0xc)&0xf0)]
+        moff+=0x100
+        nib4=matrix[moff+offset+((uv3>>0x10)&0xf) |( (uv2>>0xc)&0xf0)]
+        moff+=0x100
+
+      rnib1=matrix[moff+offset+nib1*0x10+nib2]
+      moff+=0x100
+      rnib2=matrix[moff+offset+nib3*0x10+nib4]
+      moff+=0x100
+      nwork[wo+1]=rnib1*0x10+rnib2
+      if 2 in skplist:
+        moff+=0x400
+      else:
+        nib1=matrix[moff+offset+((uv1>>0xc)&0xf)|( (uv5>>0x8)&0xf0)]
+        moff+=0x100
+        nib2=matrix[moff+offset+((uv3>>0xc)&0xf)|( (uv2>>0x8)&0xf0)]
+        moff+=0x100
+        nib3=matrix[moff+offset+((uv1>>0x8)&0xf) |( (uv5>>0x4)&0xf0)]
+        moff+=0x100
+        nib4=matrix[moff+offset+((uv3>>0x8)&0xf) |( (uv2>>0x4)&0xf0)]
+        moff+=0x100
+      rnib1=matrix[moff+offset+nib1*0x10+nib2]
+      moff+=0x100
+      rnib2=matrix[moff+offset+nib3*0x10+nib4]
+      moff+=0x100
+      nwork[wo+2]=rnib1*0x10+rnib2
+      if 3 in skplist:
+        moff+=0x400
+      else:
+        nib1=matrix[moff+offset+((uv1>>0x4)&0xf)|( (uv5)&0xf0)]
+        moff+=0x100
+        nib2=matrix[moff+offset+((uv3>>0x4)&0xf)|( (uv2)&0xf0)]
+        moff+=0x100
+        nib3=matrix[moff+offset+((uv1)&0xf)|( (uv5<<4)&0xf0) ]
+        moff+=0x100
+        nib4=matrix[moff+offset+((uv3)&0xf)|( (uv2<<4)&0xf0) ]
+        moff+=0x100
+      ##############
+      rnib1=matrix[moff+offset+nib1*0x10+nib2]
+      moff+=0x100
+      rnib2=matrix[moff+offset+nib3*0x10+nib4]
+      moff+=0x100
+      nwork[wo+3]=rnib1*0x10+rnib2
+      offset = offset + 0x1800
+      wo+=4
+      toff+=0x400
+    self.work=nwork
+  def lookup(self,ltable):
+    for a in range(len(self.work)):
+      self.work[a]=ltable[a]
+  def exlookup(self,ltable):
+    lookoffs=0
+    for a in range(len(self.work)):
+      self.work[a]=ltable[self.work[a]+lookoffs]
+      lookoffs+=0x100
+  def mask(self, chunk):
+    out=[]
+    for a in range(len(chunk)):
+      self.work[a]=self.work[a]^chunk[a]
+      out.append(self.work[a])
+    return out
+
+def process_V9708(st):
+  #e9c457a7dae6aa24365e7ef219b934b17ed58ee7d5329343fc3aea7860ed51f9a73de14351c9
+  ws=workspace([0x11]*16)
+  repl=[0,5,10,15,4,9,14,3,8,13,2,7,12,1,6,11]
+  remln=len(st)
+  sto=0
+  out=[]
+  while(remln>0):
+    ws.shuffle(repl)
+    ws.sbox(d0x6a06ea70,d0x6a0dab50)
+    ws.sbox(d0x6a073a70,d0x6a0dab50)
+    ws.shuffle(repl)
+    ws.exlookup(d0x6a072a70)
+    dat=ws.mask(st[sto:sto+16])
+    out+=dat
+    sto+=16
+    remln-=16;
+  return bytes(out)
+
+def process_V1031(st):
+  #d53efea7fdd0fda3e1e0ebbae87cad0e8f5ef413c471c3ae81f39222a9ec8b8ed582e045918c
+  ws=workspace([0x06,0x18,0x60,0x68,0x3b,0x62,0x3e,0x3c,0x06,0x50,0x71,0x52,0x02,0x5a,0x63,0x03])
+  repl=[0,5,10,15,4,9,14,3,8,13,2,7,12,1,6,11]
+  remln=len(st)
+  sto=0
+  out=[]
+  while(remln>0):
+    ws.shuffle(repl)
+    ws.sbox(d0x6a0797c0,d0x6a0dab50,[3])
+    ws.sbox(d0x6a07e7c0,d0x6a0dab50,[3])
+    ws.shuffle(repl)
+    ws.sbox(d0x6a0797c0,d0x6a0dab50,[3])
+    ws.sbox(d0x6a07e7c0,d0x6a0dab50,[3])
+    ws.exlookup(d0x6a07d7c0)
+    dat=ws.mask(st[sto:sto+16])
+    out+=dat
+    sto+=16
+    remln-=16
+    #break
+  return bytes(out)
+
+def process_V2069(st):
+  #8e6196d754a304c9354e91b5d79f07b048026d31c7373a8691e513f2c802c706742731caa858
+  ws=workspace([0x79,0x0d,0x12,0x08,0x66,0x77,0x2e,0x5b,0x02,0x09,0x0a,0x13,0x11,0x0c,0x11,0x62])
+  repl=[0,5,10,15,4,9,14,3,8,13,2,7,12,1,6,11]
+  remln=len(st)
+  sto=0
+  out=[]
+  while(remln>0):
+    ws.sbox(d0x6a084498,d0x6a0dab50,[2])
+    ws.shuffle(repl)
+    ws.sbox(d0x6a089498,d0x6a0dab50,[2])
+    ws.sbox(d0x6a089498,d0x6a0dab50,[2])
+    ws.sbox(d0x6a084498,d0x6a0dab50,[2])
+    ws.shuffle(repl)
+    ws.exlookup(d0x6a088498)
+    dat=ws.mask(st[sto:sto+16])
+    out+=dat
+    sto+=16
+    remln-=16
+  return bytes(out)
+
+
+def process_V9041(st):
+  #11f7db074b24e560dfa6fae3252b383c3b936e51f6ded570dc936cb1da9f4fc4a97ec686e7d8
+  ws=workspace([0x49,0x0b,0x0e,0x3b,0x19,0x1a,0x49,0x61,0x10,0x73,0x19,0x67,0x5c,0x1b,0x11,0x21])
+  repl=[0,5,10,15,4,9,14,3,8,13,2,7,12,1,6,11]
+  remln=len(st)
+  sto=0
+  out=[]
+  while(remln>0):
+    ws.sbox(d0x6a094170,d0x6a0dab50,[1])
+    ws.shuffle(repl)
+    ws.shuffle(repl)
+    ws.sbox(d0x6a08f170,d0x6a0dab50,[1])
+    ws.sbox(d0x6a08f170,d0x6a0dab50,[1])
+    ws.sbox(d0x6a094170,d0x6a0dab50,[1])
+
+    ws.exlookup(d0x6a093170)
+    dat=ws.mask(st[sto:sto+16])
+    out+=dat
+    sto+=16
+    remln-=16
+    #break
+  return bytes(out)
+
+def process_V3646(st):
+  #d468aa362b44479282291983243b38197c4b4aa24c2c58e62c76ec4b81e08556ca0c54301664
+  ws=workspace([0x0a,0x36,0x3e,0x29,0x4e,0x02,0x18,0x38,0x01,0x36,0x73,0x13,0x14,0x1b,0x16,0x6a])
+  repl=[0,5,10,15,4,9,14,3,8,13,2,7,12,1,6,11]
+  remln=len(st)
+  sto=0
+  out=[]
+  while(remln>0):
+    ws.shuffle(repl)
+    ws.sbox(d0x6a099e48,d0x6a0dab50,[2,3])
+    ws.sbox(d0x6a09ee48,d0x6a0dab50,[2,3])
+    ws.sbox(d0x6a09ee48,d0x6a0dab50,[2,3])
+    ws.shuffle(repl)
+    ws.sbox(d0x6a099e48,d0x6a0dab50,[2,3])
+    ws.sbox(d0x6a099e48,d0x6a0dab50,[2,3])
+    ws.shuffle(repl)
+    ws.sbox(d0x6a09ee48,d0x6a0dab50,[2,3])
+    ws.exlookup(d0x6a09de48)
+    dat=ws.mask(st[sto:sto+16])
+    out+=dat
+    sto+=16
+    remln-=16
+  return bytes(out)
+
+
+def process_V6052(st):
+  #d683c8c4e4f46ae45812196f37e218eabce0fae08994f25fabb01d3e569b8bf3866b99d36f57
+  ws=workspace([0x5f,0x0d,0x01,0x12,0x5d,0x5c,0x14,0x2a,0x17,0x69,0x14,0x0d,0x09,0x21,0x1e,0x3b])
+  repl=[0,5,10,15,4,9,14,3,8,13,2,7,12,1,6,11]
+  remln=len(st)
+  sto=0
+  out=[]
+  while(remln>0):
+    ws.shuffle(repl)
+    ws.sbox(d0x6a0a4b20,d0x6a0dab50,[1,3])
+    ws.shuffle(repl)
+    ws.sbox(d0x6a0a4b20,d0x6a0dab50,[1,3])
+    ws.sbox(d0x6a0a9b20,d0x6a0dab50,[1,3])
+    ws.shuffle(repl)
+    ws.sbox(d0x6a0a9b20,d0x6a0dab50,[1,3])
+    ws.sbox(d0x6a0a9b20,d0x6a0dab50,[1,3])
+    ws.sbox(d0x6a0a4b20,d0x6a0dab50,[1,3])
+
+    ws.exlookup(d0x6a0a8b20)
+    dat=ws.mask(st[sto:sto+16])
+    out+=dat
+    sto+=16
+    remln-=16
+  return bytes(out)
+
+def process_V9479(st):
+  #925635db434bccd3f4791eb87b89d2dfc7c93be06e794744eb9de58e6d721e696980680ab551
+  ws=workspace([0x65,0x1d,0x19,0x7c,0x09,0x79,0x1d,0x69,0x7c,0x4e,0x13,0x0e,0x04,0x1b,0x6a,0x3c ])
+  repl=[0,5,10,15,4,9,14,3,8,13,2,7,12,1,6,11]
+  remln=len(st)
+  sto=0
+  out=[]
+  while(remln>0):
+    ws.sbox(d0x6a0af7f8,d0x6a0dab50,[1,2,3])
+    ws.sbox(d0x6a0af7f8,d0x6a0dab50,[1,2,3])
+    ws.sbox(d0x6a0b47f8,d0x6a0dab50,[1,2,3])
+    ws.sbox(d0x6a0af7f8,d0x6a0dab50,[1,2,3])
+    ws.shuffle(repl)
+    ws.sbox(d0x6a0b47f8,d0x6a0dab50,[1,2,3])
+    ws.shuffle(repl)
+    ws.shuffle(repl)
+    ws.sbox(d0x6a0b47f8,d0x6a0dab50,[1,2,3])
+    ws.exlookup(d0x6a0b37f8)
+
+    dat=ws.mask(st[sto:sto+16])
+    out+=dat
+    sto+=16
+    remln-=16
+  return bytes(out)
+
+def process_V9888(st):
+  #54c470723f8c105ba0186b6319050869de673ce31a5ec15d4439921d4cd05c5e860cb2a41fea
+  ws=workspace([0x3f,0x17,0x79,0x69,0x24,0x6b,0x37,0x50,0x63,0x09,0x45,0x6f,0x0c,0x07,0x07,0x09])
+  repl=[0,5,10,15,4,9,14,3,8,13,2,7,12,1,6,11]
+  remln=len(st)
+  sto=0
+  out=[]
+  while(remln>0):
+    ws.sbox(d0x6a0ba4d0,d0x6a0dab50,[1,2])
+    ws.sbox(d0x6a0bf4d0,d0x6a0dab50,[1,2])
+    ws.sbox(d0x6a0bf4d0,d0x6a0dab50,[1,2])
+    ws.sbox(d0x6a0ba4d0,d0x6a0dab50,[1,2])
+    ws.shuffle(repl)
+    ws.shuffle(repl)    
+    ws.shuffle(repl)
+    ws.sbox(d0x6a0bf4d0,d0x6a0dab50,[1,2])
+    ws.sbox(d0x6a0ba4d0,d0x6a0dab50,[1,2])
+    ws.exlookup(d0x6a0be4d0)    
+    dat=ws.mask(st[sto:sto+16])
+    out+=dat
+    sto+=16
+    remln-=16
+  return bytes(out)
+
+def process_V4648(st):
+  #705bd4cd8b61d4596ef4ca40774d68e71f1f846c6e94bd23fd26e5c127e0beaa650a50171f1b
+  ws=workspace([0x16,0x2b,0x64,0x62,0x13,0x04,0x18,0x0d,0x63,0x25,0x14,0x17,0x0f,0x13,0x46,0x0c])
+  repl=[0,5,10,15,4,9,14,3,8,13,2,7,12,1,6,11]
+  remln=len(st)
+  sto=0
+  out=[]
+  while(remln>0):
+    ws.sbox(d0x6a0ca1a8,d0x6a0dab50,[1,3])
+    ws.shuffle(repl)
+    ws.sbox(d0x6a0ca1a8,d0x6a0dab50,[1,3])
+    ws.sbox(d0x6a0c51a8,d0x6a0dab50,[1,3])
+    ws.sbox(d0x6a0ca1a8,d0x6a0dab50,[1,3])
+    ws.sbox(d0x6a0c51a8,d0x6a0dab50,[1,3])
+    ws.sbox(d0x6a0c51a8,d0x6a0dab50,[1,3])
+    ws.shuffle(repl)
+    ws.shuffle(repl)
+    ws.exlookup(d0x6a0c91a8)    
+    dat=ws.mask(st[sto:sto+16])
+    out+=dat
+    sto+=16
+    remln-=16
+  return bytes(out)
+
+def process_V5683(st):
+  #1f5af733423e5104afb9d5594e682ecf839a776257f33747c9beee671c57ab3f84943f69d8fd
+  ws=workspace([0x7c,0x36,0x5c,0x1a,0x0d,0x10,0x0a,0x50,0x07,0x0f,0x75,0x1f,0x09,0x3b,0x0d,0x72])
+  repl=[0,5,10,15,4,9,14,3,8,13,2,7,12,1,6,11]
+  remln=len(st)
+  sto=0
+  out=[]
+  while(remln>0):
+    ws.sbox(d0x6a0d4e80,d0x6a0dab50,[])
+    ws.shuffle(repl)
+    ws.sbox(d0x6a0cfe80,d0x6a0dab50,[])
+    ws.sbox(d0x6a0d4e80,d0x6a0dab50,[])
+    ws.sbox(d0x6a0cfe80,d0x6a0dab50,[])
+    ws.sbox(d0x6a0d4e80,d0x6a0dab50,[])
+    ws.shuffle(repl)
+    ws.sbox(d0x6a0cfe80,d0x6a0dab50,[])
+    ws.shuffle(repl)
+    ws.exlookup(d0x6a0d3e80)    
+    dat=ws.mask(st[sto:sto+16])
+    out+=dat
+    sto+=16
+    remln-=16
+  return bytes(out)
+
+
+# def a2hex(arr):
+#   ax=[]
+#   ha="0123456789abcdef"
+#   for a in arr:
+#     if a<0: a=256+a
+#     ax.append(ha[(a>>4)]+ha[a%16])
+#   return "".join(ax)
+# 
+# def memhex(adr,sz):
+#   emu=EmulatorHelper(currentProgram)
+#   arr=emu.readMemory(getAddress(adr),sz)
+#   return a2hex(arr)
+# 
+
+
+
+
 # obfuscate shared secret according to the VoucherEnvelope version
 def obfuscate(secret, version):
     if version == 1:  # v1 does not use obfuscation
@@ -835,6 +1194,107 @@ def obfuscate(secret, version):
     return obfuscated
 
 
+
+# scramble() and obfuscate2() from https://github.com/andrewc12/DeDRM_tools/commit/d9233d61f00d4484235863969919059f4d0b2057
+
+def scramble(st,magic):
+    ret=bytearray(len(st))
+    padlen=len(st)
+    for counter in range(len(st)):
+        ivar2=(padlen//2)-2*(counter%magic)+magic+counter-1
+        ret[ivar2%padlen]=st[counter]
+    return ret
+
+
+def obfuscate2(secret, version):
+    if version == 1:  # v1 does not use obfuscation
+        return secret
+    magic, word = OBFUSCATION_TABLE["V%d" % version]
+    # extend secret so that its length is divisible by the magic number
+    if len(secret) % magic != 0:
+        secret = secret + b'\x00' * (magic - len(secret) % magic)
+    obfuscated = bytearray(len(secret))
+    wordhash = bytearray(hashlib.sha256(word).digest()[16:])
+    #print(wordhash.hex())
+    shuffled = bytearray(scramble(secret,magic))
+    for i in range(0, len(secret)):
+        obfuscated[i] = shuffled[i] ^ wordhash[i % 16]
+    return obfuscated
+
+# scramble3() and obfuscate3() from https://github.com/Satsuoni/DeDRM_tools/commit/da6b6a0c911b6d45fe1b13042b690daebc1cc22f
+
+def scramble3(st,magic):
+  ret=bytearray(len(st))
+  padlen=len(st)
+  divs = padlen // magic
+  cntr = 0
+  iVar6 = 0
+  offset = 0
+  if (0 < ((magic - 1) + divs)):
+    while True:
+      if (offset & 1) == 0 :
+        uVar4 = divs - 1
+        if offset < divs:
+          iVar3 = 0
+          uVar4 = offset
+        else:
+          iVar3 = (offset - divs) + 1
+        if uVar4>=0:
+          iVar5 = uVar4 * magic
+          index =  ((padlen - 1) - cntr)
+          while True:
+            if (magic <= iVar3): break
+            ret[index] = st[iVar3 + iVar5]
+            iVar3 = iVar3 + 1
+            cntr = cntr + 1
+            uVar4 = uVar4 - 1
+            iVar5 = iVar5 - magic
+            index -= 1
+            if uVar4<=-1: break
+      else: 
+        if (offset < magic):
+          iVar3 = 0
+        else :
+          iVar3 = (offset - magic) + 1
+        if (iVar3 < divs):
+          uVar4 = offset
+          if (magic <= offset):
+            uVar4 = magic - 1
+
+          index = ((padlen - 1) - cntr)
+          iVar5 = iVar3 * magic
+          while True:
+            if (uVar4 < 0) : break
+            iVar3 += 1
+            ret[index] = st[uVar4 + iVar5]
+            uVar4 -= 1
+            index=index-1
+            iVar5 = iVar5 + magic;
+            cntr += 1;
+            if iVar3>=divs: break 
+      offset = offset + 1
+      if offset >= ((magic - 1) + divs) :break 
+  return ret
+
+#not sure if the third variant is used anywhere, but it is in Kindle, so I tried to add it
+def obfuscate3(secret, version):
+    if version == 1:  # v1 does not use obfuscation
+        return secret
+    magic, word = OBFUSCATION_TABLE["V%d" % version]
+    # extend secret so that its length is divisible by the magic number
+    if len(secret) % magic != 0:
+        secret = secret + b'\x00' * (magic - len(secret) % magic)
+    #secret = bytearray(secret)
+    obfuscated = bytearray(len(secret))
+    wordhash = bytearray(hashlib.sha256(word).digest())
+    #print(wordhash.hex())
+    shuffled=bytearray(scramble3(secret,magic))
+    #print(shuffled)
+    # shuffle secret and xor it with the first half of the word hash
+    for i in range(0, len(secret)):
+        obfuscated[i] = shuffled[i] ^ wordhash[i % 16]
+    return obfuscated
+
 class DrmIonVoucher(object):
     envelope = None
     version = None
@@ -878,18 +1338,34 @@ class DrmIonVoucher(object):
             else:
                 _assert(False, "Unknown lock parameter: %s" % param)
 
-        sharedsecret = obfuscate(shared, self.version)
 
-        key = hmac.new(sharedsecret, b"PIDv3", digestmod=hashlib.sha256).digest()
+        # i know that version maps to scramble pretty much 1 to 1, but there was precendent where they changed it, so... 
-        aes = AES.new(key[:32], AES.MODE_CBC, self.cipheriv[:16])
+        sharedsecrets = [obfuscate(shared, self.version),obfuscate2(shared, self.version),obfuscate3(shared, self.version),
-        b = aes.decrypt(self.ciphertext)
+                         process_V9708(shared), process_V1031(shared), process_V2069(shared), process_V9041(shared),
-        b = pkcs7unpad(b, 16)
+                         process_V3646(shared), process_V6052(shared), process_V9479(shared), process_V9888(shared),
+                         process_V4648(shared), process_V5683(shared)]
         
-        self.drmkey = BinaryIonParser(BytesIO(b))
+        decrypted=False
-        addprottable(self.drmkey)
+        ex=None
+        for sharedsecret in sharedsecrets:
+            key = hmac.new(sharedsecret, b"PIDv3", digestmod=hashlib.sha256).digest()
+            aes = AES.new(key[:32], AES.MODE_CBC, self.cipheriv[:16])
+            try:
+                b = aes.decrypt(self.ciphertext)
+                b = pkcs7unpad(b, 16)
+                self.drmkey = BinaryIonParser(BytesIO(b))
+                addprottable(self.drmkey)
 
-        _assert(self.drmkey.hasnext() and self.drmkey.next() == TID_LIST and self.drmkey.gettypename() == "com.amazon.drm.KeySet@1.0",
+                _assert(self.drmkey.hasnext() and self.drmkey.next() == TID_LIST and self.drmkey.gettypename() == "com.amazon.drm.KeySet@1.0",
-                "Expected KeySet, got %s" % self.drmkey.gettypename())
+                    "Expected KeySet, got %s" % self.drmkey.gettypename())
+                decrypted=True 
+
+                print("Decryption succeeded")
+                break
+            except Exception as ex:
+                print("Decryption failed, trying next fallback ")
+        if not decrypted:
+            raise ex
 
         self.drmkey.stepin()
         while self.drmkey.hasnext():

DeDRM_plugin/kindlekey.py

@@ -279,7 +279,10 @@ if iswindows:
         path = ""
         if 'LOCALAPPDATA' in os.environ.keys():
             # Python 2.x does not return unicode env. Use Python 3.x
-            path = winreg.ExpandEnvironmentStrings("%LOCALAPPDATA%")
+            if sys.version_info[0] == 2:
+                path = winreg.ExpandEnvironmentStrings(u"%LOCALAPPDATA%")
+            else:
+                path = winreg.ExpandEnvironmentStrings("%LOCALAPPDATA%")
             # this is just another alternative.
             # path = getEnvironmentVariable('LOCALAPPDATA')
             if not os.path.isdir(path):

DeDRM_plugin/stylexml2css.py

@@ -181,7 +181,7 @@ class DocParser(object):
                                 print("Scale not defined!")
                                 scale = 1.0
 
-                            if val == "":
+                            if not val:
                                 val = 0
 
                             if not ((attr == b'hang') and (int(val) == 0)):

DeDRM_plugin/topazextract.py

@@ -1,6 +1,8 @@
 #!/usr/bin/env python3
 # -*- coding: utf-8 -*-
 
+from __future__ import print_function
+
 # topazextract.py
 # Mostly written by some_updates based on code from many others
 

DeDRM_plugin/utilities.py

@@ -3,7 +3,6 @@
 
 #@@CALIBRE_COMPAT_CODE@@
 
-from ignoblekeyGenPassHash import generate_key
 import sys
 
 __license__ = 'GPL v3'
@@ -22,17 +21,6 @@ def uStrCmp (s1, s2, caseless=False):
     else:
         return ud.normalize('NFC', str1) == ud.normalize('NFC', str2)
 
-def parseCustString(keystuff):
-    userkeys = []
-    ar = keystuff.split(':')
-    for i in ar:
-        try:
-            name, ccn = i.split(',')
-            # Generate Barnes & Noble EPUB user key from name and credit card number.
-            userkeys.append(generate_key(name, ccn))
-        except:
-            pass
-    return userkeys
 
 
 # Wrap a stream so that output gets flushed immediately

FAQs.md

@@ -31,7 +31,7 @@ Verify the one of the following cryptographic hash values, using software of you
 * SHA-1: 7AB9A86B954CB23D622BD79E3257F8E2182D791C
 * SHA-256: 28DC21246A9C7CDEDD2D6F0F4082E6BF7EF9DB9CE9D485548E8A9E1D19EAE2AC
 
-You will need to go to the preferences and uncheck the auto update checkbox. Then download and install 1.17 over the top of the newer installation. You'll also need to delete the KFX folders from your My Kindle Content folder. You may also need to take further action to prevent an auto update. The simplest wayis to find the 'updates' folder and replace it with a file. See [this thread] (http://www.mobileread.com/forums/showthread.php?t=283371) at MobileRead for a Script to do this on a PC. On a Mac you can find the folder at ~/Library/Application Support/Kindle/. Make the 'updates' folder read-only, or delete it and save a blank text file called 'updates' in its place.
+You will need to go to the preferences and uncheck the auto update checkbox. Then download and install 1.17 over the top of the newer installation. You'll also need to delete the KFX folders from your My Kindle Content folder. You may also need to take further action to prevent an auto update. The simplest way is to find the 'updates' folder and replace it with a file. See [this thread] (http://www.mobileread.com/forums/showthread.php?t=283371) at MobileRead for a Script to do this on a PC. On a Mac you can find the folder at ~/Library/Application Support/Kindle/. Make the 'updates' folder read-only, or delete it and save a blank text file called 'updates' in its place.
 
 Another possible solution is to use 1.19 or later, but disable KFX by renaming or disabling a necessary component of the application. This may or may not work on versions after 1.25. In a command window, enter the following commands when Kindle for PC/Mac is not running:
 
@@ -183,7 +183,7 @@ Amazon turned off backup for Kindle for Android, so the tools can no longer find
 Apple regularly change the details of their DRM and so the tools in the main tools archive will not work with these ebooks. Apple’s Fairplay DRM scheme can be removed using Requiem if the appropriate version of iTunes can still be installed and used. See the post Apple and ebooks: iBookstore DRM and how to remove it at Apprentice Alf's blog for more details.
 
 ## Why don't the tools work with LCP-encrypted ebooks? / Error message about a "DMCA takedown"
-Support for LCP DRM removal was included in the past, but Readium (the company who developed that particular DRM) has decided to [open a DMCA takedown request](https://github.com/github/dmca/blob/master/2022/01/2022-01-04-readium.md) in January 2022. This means that for legal reasons, this GitHun repository no longer contains the code needed to remove DRM from LCP-encrypted books. For more information please read [this bug report](https://github.com/noDRM/DeDRM_tools/issues/18). 
+Support for LCP DRM removal was included in the past, but Readium (the company who developed that particular DRM) has decided to [open a DMCA takedown request](https://github.com/github/dmca/blob/master/2022/01/2022-01-04-readium.md) in January 2022. This means that for legal reasons, this GitHub repository no longer contains the code needed to remove DRM from LCP-encrypted books. For more information please read [this bug report](https://github.com/noDRM/DeDRM_tools/issues/18). 
 
 ## I’ve got the tools archive and I’ve read all the FAQs but I still can’t install the tools and/or the DRM removal doesn’t work
 * Read the `ReadMe_Overview.txt` file in the top level of the tools archive

Obok_plugin/action.py

@@ -237,7 +237,10 @@ class InterfacePluginAction(InterfaceAction):
 
         :param books_to_add: List of calibre bookmaps (created in get_decrypted_kobo_books)
         '''
-        added = self.db.add_books(books_to_add, add_duplicates=False, run_hooks=False)
+
+        cfg_add_duplicates = (cfg['finding_homes_for_formats'] == 'Add new entry')
+
+        added = self.db.add_books(books_to_add, add_duplicates=cfg_add_duplicates, run_hooks=False)
         if len(added[0]):
             # Record the id(s) that got added
             for id in added[0]:

Obok_plugin/config.py

@@ -39,8 +39,13 @@ class ConfigWidget(QWidget):
         self.find_homes = QComboBox()
         self.find_homes.setToolTip(_('<p>Default behavior when duplicates are detected. None of the choices will cause calibre ebooks to be overwritten'))
         layout.addWidget(self.find_homes)
-        self.find_homes.addItems([_('Ask'), _('Always'), _('Never')])
+
+        self.find_homes.addItems([_('Ask'), _('Always'), _('Never'), _('Add new entry')])
+
         index = self.find_homes.findText(plugin_prefs['finding_homes_for_formats'])
+        if index == -1:
+            index = self.find_homes.findText(_(plugin_prefs['finding_homes_for_formats']))
+
         self.find_homes.setCurrentIndex(index)
 
         self.serials_button = QtGui.QPushButton(self)
@@ -69,7 +74,24 @@ class ConfigWidget(QWidget):
 
 
     def save_settings(self):
-        plugin_prefs['finding_homes_for_formats'] = self.find_homes.currentText()
+
+
+        # Make sure the config file string is *always* english.
+        find_homes = None
+        if self.find_homes.currentText() == _('Ask'):
+            find_homes = 'Ask'
+        elif self.find_homes.currentText() == _('Always'):
+            find_homes = 'Always'
+        elif self.find_homes.currentText() == _('Never'):
+            find_homes = 'Never'
+        elif self.find_homes.currentText() == _('Add new entry'):
+            find_homes = 'Add new entry'
+        
+        if find_homes is None:
+            # Fallback
+            find_homes = self.find_homes.currentText()
+
+        plugin_prefs['finding_homes_for_formats'] = find_homes
         plugin_prefs['kobo_serials'] = self.tmpserials
         plugin_prefs['kobo_directory'] = self.kobodirectory
 

Obok_plugin/obok/obok.py

@@ -312,11 +312,17 @@ class KoboLibrary(object):
                     if sys.getwindowsversion().major > 5:
                         if 'LOCALAPPDATA' in os.environ.keys():
                             # Python 2.x does not return unicode env. Use Python 3.x
-                            self.kobodir = winreg.ExpandEnvironmentStrings("%LOCALAPPDATA%")
+                            if sys.version_info[0] == 2:
+                                self.kobodir = winreg.ExpandEnvironmentStrings(u"%LOCALAPPDATA%")
+                            else: 
+                                self.kobodir = winreg.ExpandEnvironmentStrings("%LOCALAPPDATA%")
                     if (self.kobodir == u""):
                         if 'USERPROFILE' in os.environ.keys():
                             # Python 2.x does not return unicode env. Use Python 3.x
-                            self.kobodir = os.path.join(winreg.ExpandEnvironmentStrings("%USERPROFILE%"), "Local Settings", "Application Data")
+                            if sys.version_info[0] == 2:
+                                self.kobodir = os.path.join(winreg.ExpandEnvironmentStrings(u"%USERPROFILE%"), "Local Settings", "Application Data")
+                            else: 
+                                self.kobodir = os.path.join(winreg.ExpandEnvironmentStrings("%USERPROFILE%"), "Local Settings", "Application Data")
                     self.kobodir = os.path.join(self.kobodir, "Kobo", "Kobo Desktop Edition")
                 elif sys.platform.startswith('darwin'):
                     self.kobodir = os.path.join(os.environ['HOME'], "Library", "Application Support", "Kobo", "Kobo Desktop Edition")

README.md

@@ -1,9 +1,11 @@
 # DeDRM_tools
 DeDRM tools for ebooks
 
-This is a fork of Apprentice Harper's version of the DeDRM tools. I've added some of the PRs that still haven't been merged, as well as added some more features / bugfixes myself.
+This is a fork of Apprentice Harper's version of the DeDRM tools. Apprentice Harper said that the original version of the plugin [is no longer maintained](https://github.com/apprenticeharper/DeDRM_tools#no-longer-maintained), so I've taken over, merged a bunch of open PRs, and added a ton more features and bugfixes. 
 
-Take a look at [the CHANGELOG](https://github.com/noDRM/DeDRM_tools/blob/master/CHANGELOG.md) to see a list of changes since the last version by Apprentice Harper (v7.2.1). This plugin will start with version v10.0.0 so there won't be conflicting / duplicate version numbers when Apprentice Harper's version is updated again.
+The latest stable (released) version is v10.0.3 which [can be downloaded here](https://github.com/noDRM/DeDRM_tools/releases/tag/v10.0.3). The latest `master` build (will be automatically updated with every code change, may be unstable) [can be found here](https://github.com/noDRM/DeDRM_tools/releases/tag/autorelease). 
+
+Take a look at [the CHANGELOG](https://github.com/noDRM/DeDRM_tools/blob/master/CHANGELOG.md) to see a list of changes since the last version by Apprentice Harper (v7.2.1). This plugin will start with version v10.0.0.
 
 The v10.0.0 versions of this plugin should both work with Calibre 5.x (Python 3) as well as Calibre 4.x and lower (Python 2). If you encounter issues with this plugin in Calibre 4.x or lower, please open a bug report.