Merge ad33aea18d into e82d2b5c9c

The Kobo Desktop program will (when running under Wine on Linux) put all
of its data in the current working directory. This means that there
may be more than one Kobo.sqlite floating around by the user, which
leads to Obok showing an outdated list of books and the user being
confused by Obok cannot find the full list of books.

Solving this completely appears to be a bit too complicated, so this
patch is a best-effort improvement for the worst cases which can be
caused by this:

 1. If the user deletes the files but Obok has already cached the path,
    previously Obok would just error out rather than trying to search
    for a new Kobo.sqlite path and updating the cache.

 2. We search $HOME before searching /, which speeds up initial usage of
    Obok in the common case (usually Kobo Desktop will be installed in
    ~/.wine/drive_c) and also ensures that we correctly preference the
    current user's Kobo Desktop installation.

Signed-off-by: Aleksa Sarai <cyphar@cyphar.com>

.github/ISSUE_TEMPLATE/QUESTION.yml

@@ -10,16 +10,16 @@ body:
     id: calibre-version
     attributes:
       label: Which version of Calibre are you running?
-      description: "Example: 5.32"
-      placeholder: "5.32"
+      description: "Example: 6.23"
+      placeholder: "6.23"
     validations: 
       required: true
   - type: input
     id: plugin-version
     attributes:
       label: Which version of the DeDRM plugin are you running?
-      description: "Example: v10.0.0"
-      placeholder: "v10.0.0"
+      description: "Example: v10.0.2"
+      placeholder: "v10.0.2"
     validations: 
       required: true
   - type: input

.github/workflows/main.yml

@@ -21,44 +21,45 @@ jobs:
             DeDRM_tools_*.zip
             DeDRM_tools.zip
 
-      - name: Delete old release
-        uses: cb80/delrel@latest
-        with:
-          tag: autorelease
-          token: ${{ github.token }}
-
-      - name: Delete old tag
-        uses: dev-drprasad/delete-tag-and-release@v1.0
-        with:
-          tag_name: autorelease
-          github_token: ${{ github.token }}
-          delete_release: true
-      
-      - name: Prepare release
-        run: cp DeDRM_tools.zip DeDRM_alpha_${{ github.sha }}.zip
-
-      - name: Auto-release
-        id: autorelease
-        uses: softprops/action-gh-release@v1
-        with: 
-          tag_name: autorelease
-          token: ${{ github.token }}
-          name: Automatic alpha release with latest changes
-          body: |
-            This release is automatically generated by Github for each commit. 
-
-            This means, every time a change is made to this repo, this release will be updated to contain an untested copy of the plugin at that stage. This will contain the most up-to-date code, but it's not tested at all and may be broken.
-
-            Last update based on Git commit ${{ github.sha }}.
-          prerelease: true
-          draft: true
-          files: DeDRM_alpha_${{ github.sha }}.zip
-
-      - name: Make release public
-        uses: irongut/EditRelease@v1.2.0
-        with:
-          token: ${{ github.token }}
-          id: ${{ steps.autorelease.outputs.id }}
-          draft: false
-          prerelease: true
-
+#      - name: Delete old release
+#        uses: cb80/delrel@latest
+#        with:
+#          tag: autorelease
+#          token: ${{ github.token }}
+#
+#      - name: Delete old tag
+#        uses: dev-drprasad/delete-tag-and-release@v1.0
+#        with:
+#          tag_name: autorelease
+#          github_token: ${{ github.token }}
+#          delete_release: true
+#      
+#      - name: Prepare release
+#        run: cp DeDRM_tools.zip DeDRM_alpha_${{ github.sha }}.zip
+#
+#      - name: Auto-release
+#        id: autorelease
+#        uses: softprops/action-gh-release@v1
+#        with: 
+#          tag_name: autorelease
+#          token: ${{ github.token }}
+#          name: Automatic alpha release with latest changes
+#          body: |
+#            This release is automatically generated by Github for each commit. 
+#
+#            This means, every time a change is made to this repo, this release will be updated to contain an untested copy of the plugin at that stage. This will contain the most up-to-date code, but it's not tested at all and may be broken.
+#
+#            Last update based on Git commit ${{ github.sha }}.
+#          prerelease: true
+#          draft: true
+#          files: DeDRM_alpha_${{ github.sha }}.zip
+#
+#      - name: Make release public
+#        uses: irongut/EditRelease@v1.2.0
+#        with:
+#          token: ${{ github.token }}
+#          id: ${{ steps.autorelease.outputs.id }}
+#          draft: false
+#          prerelease: true
+#
+#

CHANGELOG.md

@@ -67,7 +67,11 @@ List of changes since the fork of Apprentice Harper's repository:
 - Fix Nook Study key retrieval code (partially fixes #50).
 - Make the plugin work on Calibre 6 (Qt 6). (fixes #54 and #98) If you're running Calibre 6 and you notice any issues, please open a bug report.
 
-## Fixes on master (not yet released):
+## Fixes in v10.0.9 (RC for v10.1.0, 2023-08-02):
+
+Note that versions v10.0.4(s), v10.0.5(s) and v10.0.6(s) were released by other people in various forks, so I have decided to make a larger version jump so there are no conflicting version numbers / different builds with the same version number. 
+
+This is v10.0.9, a release candidate for v10.1.0. I don't expect there to be major issues / bugs, but since a lot of code has changed in the last year I wanted to get some "extended testing" before this becomes v10.1.0. 
 
 - Fix a bug introduced with #48 that breaks DeDRM'ing on Calibre 4 (fixes #101).
 - Fix some more Calibre-6 bugs in the Obok plugin (should fix #114).
@@ -92,3 +96,8 @@ List of changes since the fork of Apprentice Harper's repository:
 - Two bugfixes for Amazon DeDRM from Satuoni ( https://github.com/noDRM/DeDRM_tools/issues/315#issuecomment-1508305428 ) and andrewc12 ( https://github.com/andrewc12/DeDRM_tools/commit/d9233d61f00d4484235863969919059f4d0b2057 ) that might make the plugin work with newer versions.
 - Fix font decryption not working with some books (fixes #347), thanks for the patch @bydioeds. 
 - Fix a couple unicode errors for Python2 in Kindle and Nook code.
+
+## Fixes on master (not yet released):
+
+- Fix a bug where decrypting a 40-bit RC4 pdf with R=2 didn't work.
+- Fix a bug where decrypting a 256-bit AES pdf with V=5 didn't work.

DeDRM_plugin/DeDRM_Help.htm

@@ -17,7 +17,7 @@ p {margin-top: 0}
 
 <body>
 
-<h1>DeDRM Plugin <span class="version">(v10.0.3)</span></h1>
+<h1>DeDRM Plugin <span class="version">(v10.0.9 / v10.1.0 RC1)</span></h1>
 
 <p>This plugin removes DRM from ebooks when they are imported into calibre. If you already have DRMed ebooks in your calibre library, you will need to remove them and import them again.</p>
 
@@ -26,6 +26,8 @@ p {margin-top: 0}
 <h3>Installation</h3>
 <p>You have obviously managed to install the plugin, as otherwise you wouldn’t be reading this help file. However, you should also delete any older DRM removal plugins, as this DeDRM plugin replaces the five older plugins: Kindle and Mobipocket DeDRM (K4MobiDeDRM), Ignoble Epub DeDRM (ignobleepub), Inept Epub DeDRM (ineptepub), Inept PDF DeDRM (ineptepub) and eReader PDB 2 PML (eReaderPDB2PML).</p>
 
+<p>This plugin (in versions v10.0.0 and above) will automatically replace the older 7.X and below versions from Apprentice Alf and Apprentice Harper.</p>
+
 <h3>Configuration</h3>
 <p>On Windows and Mac, the keys for ebooks downloaded for Kindle for Mac/PC and Adobe Digital Editions are automatically generated. If all your DRMed ebooks can be opened and read in Kindle for Mac/PC and/or Adobe Digital Editions on the same computer on which you are running calibre, you do not need to do any configuration of this plugin. On Linux, keys for Kindle for PC and Adobe Digital Editions need to be generated separately (see the Linux section below).</p>
 
@@ -60,7 +62,7 @@ p {margin-top: 0}
 <li>And probably many more.</li>
 </ul>
 
-<h3>For additional help read the <a href="https://github.com/noDRM/DeDRM_tools/blob/master/FAQs.md">FAQs</a> at <a href="https://github.com/noDRM/DeDRM_tools">NoDRM's GitHub repository</a> (or the corresponding <a href="https://github.com/apprenticeharper/DeDRM_tools/blob/master/FAQs.md">FAQs</a> at <a href="https://github.com/apprenticeharper/DeDRM_tools/">Apprentice Harpers’s GitHub repository</a>). You can <a href="https://github.com/noDRM/DeDRM_tools/issues">open issue reports</a>related to this fork at NoDRM's GitHub repository.</h3>
+<h4>For additional help read the <a href="https://github.com/noDRM/DeDRM_tools/blob/master/FAQs.md">FAQs</a> at <a href="https://github.com/noDRM/DeDRM_tools">NoDRM's GitHub repository</a> (or the corresponding <a href="https://github.com/apprenticeharper/DeDRM_tools/blob/master/FAQs.md">FAQs</a> at <a href="https://github.com/apprenticeharper/DeDRM_tools/">Apprentice Harpers’s GitHub repository</a>). You can <a href="https://github.com/noDRM/DeDRM_tools/issues">open issue reports</a> related to this fork at NoDRM's GitHub repository.</h4>
 
 
 <h2>Linux Systems Only</h2>

DeDRM_plugin/__init__.py

@@ -5,7 +5,7 @@ from __future__ import print_function
 
 # __init__.py for DeDRM_plugin
 # Copyright © 2008-2020 Apprentice Harper et al.
-# Copyright © 2021 NoDRM
+# Copyright © 2021-2023 NoDRM
 
 __license__   = 'GPL v3'
 __docformat__ = 'restructuredtext en'
@@ -82,6 +82,7 @@ __docformat__ = 'restructuredtext en'
 #  10.0.0 - First forked version by NoDRM. See CHANGELOG.md for details.
 #  10.0.1 - Fixes a bug in the watermark code.
 #  10.0.2 - Fix Kindle for Mac & update Adobe key retrieval
+#  For changes made in 10.0.3 and above, see the CHANGELOG.md file
 
 """
 Decrypt DRMed ebooks.

DeDRM_plugin/__version.py

@@ -4,7 +4,7 @@
 #@@CALIBRE_COMPAT_CODE@@
 
 PLUGIN_NAME = "DeDRM"
-__version__ = '10.0.3'
+__version__ = '10.0.9'
 
 PLUGIN_VERSION_TUPLE = tuple([int(x) for x in __version__.split(".")])
 PLUGIN_VERSION = ".".join([str(x)for x in PLUGIN_VERSION_TUPLE])

DeDRM_plugin/ineptpdf.py

@@ -1366,14 +1366,14 @@ class PDFDocument(object):
 
     def process_with_aes(self, key, encrypt, data, repetitions = 1, iv = None):
         if iv is None:
-            keylen = len(key)
-            iv = bytes([0x00]*keylen)
+            iv = bytes(bytearray(16))
+
+        aes = AES.new(key, AES.MODE_CBC, iv)
 
         if not encrypt:
-            plaintext = AES.new(key,AES.MODE_CBC,iv, True).decrypt(data)
+            plaintext = aes.decrypt(data)
             return plaintext
         else:
-            aes = AES.new(key, AES.MODE_CBC, iv, False)
             new_data = bytes(data * repetitions)
             crypt = aes.encrypt(new_data)
             return crypt
@@ -1394,10 +1394,18 @@ class PDFDocument(object):
                     raise Exception("K1 < 32 ...")
                 #def process_with_aes(self, key: bytes, encrypt: bool, data: bytes, repetitions: int = 1, iv: bytes = None):
                 E = self.process_with_aes(K[:16], True, K1, 64, K[16:32])
-                K = (hashlib.sha256, hashlib.sha384, hashlib.sha512)[sum(E) % 3](E).digest()
+                E = bytearray(E)
+
+                E_mod_3 = 0
+                for i in range(16):
+                    E_mod_3 += E[i]
+
+                E_mod_3 %= 3
+
+                K = (hashlib.sha256, hashlib.sha384, hashlib.sha512)[E_mod_3](E).digest()
 
                 if round_number >= 64:
-                    ch = int.from_bytes(E[-1:], "big", signed=False)
+                    ch = E[-1:][0]  # get last byte
                     if ch <= round_number - 32:
                         done = True
 
@@ -1478,14 +1486,23 @@ class PDFDocument(object):
             EncMetadata = b'True'
         if (EncMetadata == ('False' or 'false') or V < 4) and R >= 4:
             hash.update(codecs.decode(b'ffffffff','hex'))
+
+        # Finish hash:
+        hash = hash.digest()
+        
         if R >= 3:
             # 8
             for _ in range(50):
-                hash = hashlib.md5(hash.digest()[:length//8])
-        key = hash.digest()[:length//8]
+                hash = hashlib.md5(hash[:length//8]).digest()
+        if R == 2:
+            # R=2 only uses first five bytes.
+            key = hash[:5]
+        else:
+            key = hash[:length//8]
+
         if R == 2:
             # Algorithm 3.4
-            u1 = ARC4.new(key).decrypt(password)
+            u1 = ARC4.new(key).decrypt(self.PASSWORD_PADDING)
         elif R >= 3:
             # Algorithm 3.5
             hash = hashlib.md5(self.PASSWORD_PADDING) # 2
@@ -1498,6 +1515,7 @@ class PDFDocument(object):
                     k = b''.join(bytes([c ^ i]) for c in key )
                 x = ARC4.new(k).decrypt(x)
             u1 = x+x # 32bytes total
+        
         if R == 2:
             is_authenticated = (u1 == U)
         else:

DeDRM_plugin/standalone/__init__.py

@@ -7,6 +7,18 @@ from __future__ import absolute_import, print_function
 
 # Copyright © 2021 NoDRM
 
+"""
+
+NOTE: This code is not functional (yet). I started working on it a while ago
+to make a standalone version of the plugins that could work without Calibre, 
+too, but for now there's only a rough code structure and no working code yet.
+
+Currently, to use these plugins, you will need to use Calibre. Hopwfully that'll
+change in the future. 
+
+"""
+
+
 OPT_SHORT_TO_LONG = [
     ["c", "config"],
     ["e", "extract"],

DeDRM_plugin/standalone/jsonconfig.py

@@ -8,6 +8,17 @@ from __future__ import absolute_import, print_function
 
 # Taken from Calibre code - Copyright © 2008, Kovid Goyal kovid@kovidgoyal.net, GPLv3
 
+"""
+
+NOTE: This code is not functional (yet). I started working on it a while ago
+to make a standalone version of the plugins that could work without Calibre, 
+too, but for now there's only a rough code structure and no working code yet.
+
+Currently, to use these plugins, you will need to use Calibre. Hopwfully that'll
+change in the future. 
+
+"""
+
 #@@CALIBRE_COMPAT_CODE@@
 
 import sys, os, codecs, json

DeDRM_plugin/standalone/passhash.py

@@ -8,6 +8,17 @@ from __future__ import absolute_import, print_function
 
 # Copyright © 2021 NoDRM
 
+"""
+
+NOTE: This code is not functional (yet). I started working on it a while ago
+to make a standalone version of the plugins that could work without Calibre, 
+too, but for now there's only a rough code structure and no working code yet.
+
+Currently, to use these plugins, you will need to use Calibre. Hopwfully that'll
+change in the future. 
+
+"""
+
 #@@CALIBRE_COMPAT_CODE@@
 
 import os, sys

DeDRM_plugin/standalone/remove_drm.py

@@ -8,6 +8,17 @@ from __future__ import absolute_import, print_function
 
 # Copyright © 2021 NoDRM
 
+"""
+
+NOTE: This code is not functional (yet). I started working on it a while ago
+to make a standalone version of the plugins that could work without Calibre, 
+too, but for now there's only a rough code structure and no working code yet.
+
+Currently, to use these plugins, you will need to use Calibre. Hopwfully that'll
+change in the future. 
+
+"""
+
 #@@CALIBRE_COMPAT_CODE@@
 
 import os, sys

Obok_plugin/__init__.py

@@ -3,7 +3,7 @@ from __future__ import (unicode_literals, division, absolute_import,
                         print_function)
 
 __license__   = 'GPL v3'
-__version__ = '10.0.3'
+__version__ = '10.0.9'
 __docformat__ = 'restructuredtext en'
 
 #####################################################################
@@ -20,7 +20,7 @@ except NameError:
 PLUGIN_NAME = 'Obok DeDRM'
 PLUGIN_SAFE_NAME = PLUGIN_NAME.strip().lower().replace(' ', '_')
 PLUGIN_DESCRIPTION = _('Removes DRM from Kobo kepubs and adds them to the library.')
-PLUGIN_VERSION_TUPLE = (10, 0, 3)
+PLUGIN_VERSION_TUPLE = (10, 0, 9)
 PLUGIN_VERSION = '.'.join([str(x) for x in PLUGIN_VERSION_TUPLE])
 HELPFILE_NAME = PLUGIN_SAFE_NAME + '_Help.htm'
 PLUGIN_AUTHORS = 'Anon'

Obok_plugin/obok/obok.py

@@ -168,8 +168,8 @@
 """Manage all Kobo books, either encrypted or DRM-free."""
 from __future__ import print_function
 
-__version__ = '10.0.1'
-__about__ =  "Obok v{0}\nCopyright © 2012-2022 Physisticated et al.".format(__version__)
+__version__ = '10.0.9'
+__about__ =  "Obok v{0}\nCopyright © 2012-2023 Physisticated et al.".format(__version__)
 
 import sys
 import os
@@ -327,36 +327,50 @@ class KoboLibrary(object):
                 elif sys.platform.startswith('darwin'):
                     self.kobodir = os.path.join(os.environ['HOME'], "Library", "Application Support", "Kobo", "Kobo Desktop Edition")
                 elif sys.platform.startswith('linux'):
+                    # Since on Linux, you have to run Kobo Desktop within Wine,
+                    # there is no guarantee where Kobo.sqlite (and the rest of
+                    # the kobo directory) might be.
+                    #
+                    # It should also be noted that Kobo Desktop will store all
+                    # of it files in the current directory where you run it,
+                    # meaning that a user might accidentally create several
+                    # Kobo.sqlite files which are all separate and then be
+                    # confused why Obok can't find any of the new books. Sadly
+                    # there isn't a trivial way to deal with this.
 
-                    #sets ~/.config/calibre as the location to store the kobodir location info file and creates this directory if necessary
-                    kobodir_cache_dir = os.path.join(os.environ['HOME'], ".config", "calibre")
+                    # We cache the kobodir we find in ~/.config/calibre.
+                    kobodir_cache_dir = os.path.join(os.environ['HOME'], ".config", "calibre", "plugins", "obok")
                     if not os.path.isdir(kobodir_cache_dir):
                         os.mkdir(kobodir_cache_dir)
-                    
-                    #appends the name of the file we're storing the kobodir location info to the above path
-                    kobodir_cache_file = str(kobodir_cache_dir) + "/" + "kobo location"
-                    
-                    """if the above file does not exist, recursively searches from the root
-                    of the filesystem until kobodir is found and stores the location of kobodir
-                    in that file so this loop can be skipped in the future"""
-                    original_stdout = sys.stdout
-                    if not os.path.isfile(kobodir_cache_file):
-                        for root, dirs, files in os.walk('/'):
-                            for file in files:
-                                if file == 'Kobo.sqlite':
-                                    kobo_linux_path = str(root)
-                                    with open(kobodir_cache_file, 'w') as f:
-                                        sys.stdout = f
-                                        print(kobo_linux_path, end='')
-                                        sys.stdout = original_stdout
+                    kobodir_cache_file = os.path.join(kobodir_cache_dir, "kobo-location")
 
-                    f = open(kobodir_cache_file, 'r' )
-                    self.kobodir = f.read()
+                    try:
+                        # If the cached version exists and the path does really
+                        # contain Kobo.sqlite, use that.
+                        with open(kobodir_cache_file, "r") as f:
+                            cached_kobodir = f.read().strip()
+                            assert os.path.isfile(os.path.join(cached_kobodir, "Kobo.sqlite"))
+                            self.kobodir = cached_kobodir
+                    except (AssertionError, FileNotFoundError):
+                        # If there was no cached version, search the entire
+                        # filesystem tree for a directory containing
+                        # "Kobo.sqlite".
+                        #
+                        # We first search $HOME to avoid picking another user's
+                        # Kobo.sqlite file, but then fallback to / if there was
+                        # nothing in $HOME.
+                        for candidate_root in (os.environ["HOME"], "/"):
+                            for root, _, files in os.walk(candidate_root):
+                                if "Kobo.sqlite" in files:
+                                    with open(kobodir_cache_file, "w") as f:
+                                        f.write("%s/\n" % (root,))
+                                    self.kobodir = root
+                                    break
 
-            # desktop versions use Kobo.sqlite
+            # Desktop versions use Kobo.sqlite.
             kobodb = os.path.join(self.kobodir, "Kobo.sqlite")
             # check for existence of file
-            if (not(os.path.isfile(kobodb))):
+            if not os.path.isfile(kobodb):
                 # give up here, we haven't found anything useful
                 self.kobodir = u""
                 kobodb  = u""
@@ -430,7 +444,7 @@ class KoboLibrary(object):
         macaddrs = []
         if sys.platform.startswith('win'):
             c = re.compile('\s?(' + '[0-9a-f]{2}[:\-]' * 5 + '[0-9a-f]{2})(\s|$)', re.IGNORECASE)
-            try: 
+            try:
                 output = subprocess.Popen('ipconfig /all', shell=True, stdout=subprocess.PIPE, text=True).stdout
                 for line in output:
                     m = c.search(line)

Obok_plugin/obok_dedrm_Help.htm

@@ -8,7 +8,7 @@
 <body>
 
 <h1>Obok DeDRM Plugin</h1>
-<h3>(version 10.0.2)</h3>
+<h3>(version 10.0.9 / 10.1.0 RC1)</h3>
 
 <h3>Installation:</h3>
 

README.md

@@ -3,7 +3,7 @@ DeDRM tools for ebooks
 
 This is a fork of Apprentice Harper's version of the DeDRM tools. Apprentice Harper said that the original version of the plugin [is no longer maintained](https://github.com/apprenticeharper/DeDRM_tools#no-longer-maintained), so I've taken over, merged a bunch of open PRs, and added a ton more features and bugfixes. 
 
-The latest stable (released) version is v10.0.3 which [can be downloaded here](https://github.com/noDRM/DeDRM_tools/releases/tag/v10.0.3). The latest `master` build (will be automatically updated with every code change, may be unstable) [can be found here](https://github.com/noDRM/DeDRM_tools/releases/tag/autorelease). 
+The latest stable (released) version is v10.0.3 which [can be downloaded here](https://github.com/noDRM/DeDRM_tools/releases/tag/v10.0.3).
 
 Take a look at [the CHANGELOG](https://github.com/noDRM/DeDRM_tools/blob/master/CHANGELOG.md) to see a list of changes since the last version by Apprentice Harper (v7.2.1). This plugin will start with version v10.0.0.