v0.5.0

.gitignore

@@ -11,14 +11,12 @@ dist/
 sfx/
 .venv/
 
-# sublime
+# ide
 *.sublime-workspace
 
 # winmerge
 *.bak
 
-# other licenses
-contrib/
-
-# deps
-copyparty/web/deps
+# derived
+copyparty/web/deps/
+srv/

.vscode/launch.json

@@ -9,8 +9,6 @@
             "console": "integratedTerminal",
             "cwd": "${workspaceFolder}",
             "args": [
-                "-j",
-                "0",
                 //"-nw",
                 "-a",
                 "ed:wark",

README.md

@@ -19,6 +19,8 @@ turn your phone or raspi into a portable file server with resumable uploads/down
 * Android-Chrome: set max "parallel uploads" for 200% upload speed (android bug)
 * Android-Firefox: takes a while to select files (in order to avoid the above android-chrome issue)
 * Desktop-Firefox: may use gigabytes of RAM if your connection is great and your files are massive
+* paper-printing is affected by dark/light-mode! use lightmode for color, darkmode for grayscale
+  * because no browsers currently implement the media-query to do this properly orz
 
 
 ## status
@@ -36,10 +38,20 @@ turn your phone or raspi into a portable file server with resumable uploads/down
 * [x] accounts
 * [x] markdown viewer
 * [x] markdown editor
+* [x] FUSE client
 
 summary: it works! you can use it! (but technically not even close to beta)
 
 
+# client examples
+
+* javascript: dump some state into a file (two separate examples)
+  * `await fetch('https://127.0.0.1:3923/', {method:"PUT", body: JSON.stringify(foo)});`
+  * `var xhr = new XMLHttpRequest(); xhr.open('POST', 'https://127.0.0.1:3923/msgs?raw'); xhr.send('foo');`
+
+* FUSE: mount a copyparty server as a local filesystem (see [./bin/](bin/))
+
+
 # dependencies
 
 * `jinja2`
@@ -55,17 +67,23 @@ currently there are two self-contained binaries:
 * `copyparty-sfx.sh` for unix (linux and osx) -- smaller, more robust
 * `copyparty-sfx.py` for windows (unix too) -- crossplatform, beta
 
-launch either of them and it'll unpack and run copyparty, assuming you have python installed of course
+launch either of them (**use sfx.py on systemd**) and it'll unpack and run copyparty, assuming you have python installed of course
 
 pls note that `copyparty-sfx.sh` will fail if you rename `copyparty-sfx.py` to `copyparty.py` and keep it in the same folder because `sys.path` is funky
 
-if you don't need all the features you can repack the sfx and save a bunch of space, tho currently the only removable feature is the opus/vorbis javascript decoder which is needed by apple devices to play foss audio files
 
-steps to reduce the sfx size from `720 kB` to `250 kB` roughly:
-* run one of the sfx'es once to unpack it
-* `./scripts/make-sfx.sh re no-ogv` creates a new pair of sfx
+## sfx repack
 
-no internet connection needed, just download an sfx and the repo zip (also if you're on windows use msys2)
+if you don't need all the features you can repack the sfx and save a bunch of space; all you need is an sfx and a copy of this repo (nothing else to download or build, except for either msys2 or WSL if you're on windows)
+* `724K` original size as of v0.4.0
+* `256K` after `./scripts/make-sfx.sh re no-ogv`
+* `164K` after `./scripts/make-sfx.sh re no-ogv no-cm`
+
+the features you can opt to drop are
+* `ogv`.js, the opus/vorbis decoder which is needed by apple devices to play foss audio files
+* `cm`/easymde, the "fancy" markdown editor
+
+for the `re`pack to work, first run one of the sfx'es once to unpack it
 
 
 # install on android
@@ -104,10 +122,8 @@ in the `scripts` folder:
 
 roughly sorted by priority
 
-* sortable browser columns
 * up2k handle filename too long
 * up2k fails on empty files? alert then stuck
-* unexpected filepath on dupe up2k
 * drop onto folders
 * look into android thumbnail cache file format
 * support pillow-simd

bin/README.md

@@ -0,0 +1,34 @@
+# copyparty-fuse.py
+* mount a copyparty server as a local filesystem (read-only)
+* **supports Windows!** -- expect `194 MiB/s` sequential read
+* **supports Linux** -- expect `117 MiB/s` sequential read
+* **supports macos** -- expect `85 MiB/s` sequential read
+
+filecache is default-on for windows and macos;
+* macos readsize is 64kB, so speed ~32 MiB/s without the cache
+* windows readsize varies by software; explorer=1M, pv=32k
+
+note that copyparty should run with `-ed` to enable dotfiles (hidden otherwise)
+
+
+## to run this on windows:
+* install [winfsp](https://github.com/billziss-gh/winfsp/releases/latest) and [python 3](https://www.python.org/downloads/)
+  * [x] add python 3.x to PATH (it asks during install)
+* `python -m pip install --user fusepy`
+* `python ./copyparty-fuse.py n: http://192.168.1.69:3923/`
+
+10% faster in [msys2](https://www.msys2.org/), 700% faster if debug prints are enabled:
+* `pacman -S mingw64/mingw-w64-x86_64-python{,-pip}`
+* `/mingw64/bin/python3 -m pip install --user fusepy`
+* `/mingw64/bin/python3 ./copyparty-fuse.py [...]`
+
+you could replace winfsp with [dokan](https://github.com/dokan-dev/dokany/releases/latest), let me know if you [figure out how](https://github.com/dokan-dev/dokany/wiki/FUSE)  
+(winfsp's sshfs leaks, doesn't look like winfsp itself does, should be fine)
+
+
+
+# copyparty-fuse🅱️.py
+* mount a copyparty server as a local filesystem (read-only)
+* does the same thing except more correct, `samba` approves
+* **supports Linux** -- expect `18 MiB/s` (wait what)
+* **supports Macos** -- probably

bin/copyparty-fuse.py

@@ -7,47 +7,80 @@ __copyright__ = 2019
 __license__ = "MIT"
 __url__ = "https://github.com/9001/copyparty/"
 
-import re
-import sys
-import time
-import stat
-import errno
-import struct
-import threading
-import http.client  # py2: httplib
-import urllib.parse
-from datetime import datetime
-from urllib.parse import quote_from_bytes as quote
-
-try:
-    from fuse import FUSE, FuseOSError, Operations
-except:
-    print("\n    could not import fuse;\n    pip install fusepy\n")
-    raise
-
 
 """
 mount a copyparty server (local or remote) as a filesystem
 
 usage:
-  python copyparty-fuse.py ./music http://192.168.1.69:1234/
+  python copyparty-fuse.py ./music http://192.168.1.69:3923/
 
 dependencies:
-  sudo apk add fuse-dev
-  python3 -m venv ~/pe/ve.fusepy
-  . ~/pe/ve.fusepy/bin/activate
-  pip install fusepy
-
-
-MB/s
- 28   cache NOthread
- 24   cache   thread
- 29   cache NOthread NOmutex
- 67 NOcache NOthread NOmutex  (　´・ω・) nyoro~n
- 10 NOcache   thread NOmutex
+  python3 -m pip install --user fusepy
+  + on Linux: sudo apk add fuse
+  + on Macos: https://osxfuse.github.io/
+  + on Windows: https://github.com/billziss-gh/winfsp/releases/latest
 """
 
 
+import re
+import os
+import sys
+import time
+import stat
+import errno
+import struct
+import builtins
+import platform
+import threading
+import traceback
+import http.client  # py2: httplib
+import urllib.parse
+from datetime import datetime
+from urllib.parse import quote_from_bytes as quote
+
+
+DEBUG = False  # ctrl-f this to configure logging
+
+
+WINDOWS = sys.platform == "win32"
+MACOS = platform.system() == "Darwin"
+
+
+try:
+    from fuse import FUSE, FuseOSError, Operations
+except:
+    if WINDOWS:
+        libfuse = "install https://github.com/billziss-gh/winfsp/releases/latest"
+    elif MACOS:
+        libfuse = "install https://osxfuse.github.io/"
+    else:
+        libfuse = "apt install libfuse\n    modprobe fuse"
+
+    print(
+        "\n  could not import fuse; these may help:"
+        + "\n    python3 -m pip install --user fusepy\n    "
+        + libfuse
+        + "\n"
+    )
+    raise
+
+
+def print(*args, **kwargs):
+    try:
+        builtins.print(*list(args), **kwargs)
+    except:
+        builtins.print(termsafe(" ".join(str(x) for x in args)), **kwargs)
+
+
+def termsafe(txt):
+    try:
+        return txt.encode(sys.stdout.encoding, "backslashreplace").decode(
+            sys.stdout.encoding
+        )
+    except:
+        return txt.encode(sys.stdout.encoding, "replace").decode(sys.stdout.encoding)
+
+
 def threadless_log(msg):
     print(msg + "\n", end="")
 
@@ -60,27 +93,100 @@ def boring_log(msg):
 def rice_tid():
     tid = threading.current_thread().ident
     c = struct.unpack(b"B" * 5, struct.pack(b">Q", tid)[-5:])
-    return "".join("\033[1;37;48;5;{}m{:02x}".format(x, x) for x in c)
+    return "".join("\033[1;37;48;5;{}m{:02x}".format(x, x) for x in c) + "\033[0m"
 
 
 def fancy_log(msg):
-    print("{}\033[0m {}\n".format(rice_tid(), msg), end="")
+    print("{} {}\n".format(rice_tid(), msg), end="")
 
 
 def null_log(msg):
     pass
 
 
-log = boring_log
-log = fancy_log
-log = threadless_log
-dbg = null_log
+class RecentLog(object):
+    def __init__(self):
+        self.mtx = threading.Lock()
+        self.f = None  # open("copyparty-fuse.log", "wb")
+        self.q = []
+
+        thr = threading.Thread(target=self.printer)
+        thr.daemon = True
+        thr.start()
+
+    def put(self, msg):
+        msg = "{} {}\n".format(rice_tid(), msg)
+        if self.f:
+            fmsg = " ".join([datetime.utcnow().strftime("%H%M%S.%f"), str(msg)])
+            self.f.write(fmsg.encode("utf-8"))
+
+        with self.mtx:
+            self.q.append(msg)
+            if len(self.q) > 200:
+                self.q = self.q[-50:]
+
+    def printer(self):
+        while True:
+            time.sleep(0.05)
+            with self.mtx:
+                q = self.q
+                if not q:
+                    continue
+
+                self.q = []
+
+            print("".join(q), end="")
+
+
+if DEBUG:
+    # debug=on,
+    #   windows terminals are slow (cmd.exe, mintty)
+    #   otoh fancy_log beats RecentLog on linux
+    logger = RecentLog().put if WINDOWS else fancy_log
+
+    info = logger
+    log = logger
+    dbg = logger
+else:
+    # debug=off, speed is dontcare
+    info = fancy_log
+    log = null_log
+    dbg = null_log
+
+
+# [windows/cmd/cpy3]  python dev\copyparty\bin\copyparty-fuse.py q: http://192.168.1.159:1234/
+# [windows/cmd/msys2] C:\msys64\mingw64\bin\python3 dev\copyparty\bin\copyparty-fuse.py q: http://192.168.1.159:1234/
+# [windows/mty/msys2] /mingw64/bin/python3 /c/Users/ed/dev/copyparty/bin/copyparty-fuse.py q: http://192.168.1.159:1234/
+#
+# [windows] find /q/music/albums/Phant*24bit -printf '%s %p\n' | sort -n | tail -n 8 | sed -r 's/^[0-9]+ //' | while IFS= read -r x; do dd if="$x" of=/dev/null bs=4k count=8192 & done
+# [alpine]  ll t; for x in t/2020_0724_16{2,3}*; do dd if="$x" of=/dev/null bs=4k count=10240 & done
+#
+#  72.4983 windows mintty msys2 fancy_log
+# 219.5781 windows cmd msys2 fancy_log
+# nope.avi windows cmd cpy3 fancy_log
+#   9.8817 windows mintty msys2 RecentLog 200 50 0.1
+#  10.2241 windows cmd cpy3 RecentLog 200 50 0.1
+#   9.8494 windows cmd msys2 RecentLog 200 50 0.1
+#   7.8061 windows mintty msys2 fancy_log <info-only>
+#   7.9961 windows mintty msys2 RecentLog <info-only>
+#   4.2603 alpine xfce4 cpy3 RecentLog
+#   4.1538 alpine xfce4 cpy3 fancy_log
+#   3.1742 alpine urxvt cpy3 fancy_log
 
 
 def get_tid():
     return threading.current_thread().ident
 
 
+def html_dec(txt):
+    return (
+        txt.replace("&lt;", "<")
+        .replace("&gt;", ">")
+        .replace("&quot;", '"')
+        .replace("&amp;", "&")
+    )
+
+
 class CacheNode(object):
     def __init__(self, tag, data):
         self.tag = tag
@@ -118,7 +224,7 @@ class Gateway(object):
         try:
             return self.conns[tid]
         except:
-            log("new conn [{}] [{}]".format(self.web_host, self.web_port))
+            info("new conn [{}] [{}]".format(self.web_host, self.web_port))
 
             conn = http.client.HTTPConnection(self.web_host, self.web_port, timeout=260)
 
@@ -146,29 +252,36 @@ class Gateway(object):
             return c.getresponse()
 
     def listdir(self, path):
-        web_path = "/" + "/".join([self.web_root, path])
-
-        r = self.sendreq("GET", self.quotep(web_path))
+        web_path = self.quotep("/" + "/".join([self.web_root, path])) + "?dots"
+        r = self.sendreq("GET", web_path)
         if r.status != 200:
             self.closeconn()
             raise Exception(
-                "http error {} reading dir {} in {:x}".format(
+                "http error {} reading dir {} in {}".format(
                     r.status, web_path, rice_tid()
                 )
             )
 
-        return self.parse_html(r)
+        try:
+            return self.parse_html(r)
+        except:
+            traceback.print_exc()
+            raise
 
     def download_file_range(self, path, ofs1, ofs2):
-        web_path = "/" + "/".join([self.web_root, path])
-        hdr_range = "bytes={}-{}".format(ofs1, ofs2)
-        log("downloading {}".format(hdr_range))
+        web_path = self.quotep("/" + "/".join([self.web_root, path])) + "?raw"
+        hdr_range = "bytes={}-{}".format(ofs1, ofs2 - 1)
+        info(
+            "DL {:4.0f}K\033[36m{:>9}-{:<9}\033[0m{}".format(
+                (ofs2 - ofs1) / 1024.0, ofs1, ofs2 - 1, path
+            )
+        )
 
-        r = self.sendreq("GET", self.quotep(web_path), headers={"Range": hdr_range})
+        r = self.sendreq("GET", web_path, headers={"Range": hdr_range})
         if r.status != http.client.PARTIAL_CONTENT:
             self.closeconn()
             raise Exception(
-                "http error {} reading file {} range {} in {:x}".format(
+                "http error {} reading file {} range {} in {}".format(
                     r.status, web_path, hdr_range, rice_tid()
                 )
             )
@@ -202,8 +315,16 @@ class Gateway(object):
                     continue
 
                 ftype, fname, fsize, fdate = m.groups()
-                ts = datetime.strptime(fdate, "%Y-%m-%d %H:%M:%S").timestamp()
-                sz = int(fsize)
+                fname = html_dec(fname)
+                sz = 1
+                ts = 60 * 60 * 24 * 2
+                try:
+                    sz = int(fsize)
+                    ts = datetime.strptime(fdate, "%Y-%m-%d %H:%M:%S").timestamp()
+                except:
+                    info("bad HTML or OS [{}] [{}]".format(fdate, fsize))
+                    # python cannot strptime(1959-01-01) on windows
+
                 if ftype == "-":
                     ret.append([fname, self.stat_file(ts, sz), 0])
                 else:
@@ -213,7 +334,7 @@ class Gateway(object):
 
     def stat_dir(self, ts, sz=4096):
         return {
-            "st_mode": 0o555 | stat.S_IFDIR,
+            "st_mode": stat.S_IFDIR | 0o555,
             "st_uid": 1000,
             "st_gid": 1000,
             "st_size": sz,
@@ -225,7 +346,7 @@ class Gateway(object):
 
     def stat_file(self, ts, sz):
         return {
-            "st_mode": 0o444 | stat.S_IFREG,
+            "st_mode": stat.S_IFREG | 0o444,
             "st_uid": 1000,
             "st_gid": 1000,
             "st_size": sz,
@@ -237,8 +358,11 @@ class Gateway(object):
 
 
 class CPPF(Operations):
-    def __init__(self, base_url):
+    def __init__(self, base_url, dircache, filecache):
         self.gw = Gateway(base_url)
+        self.junk_fh_ctr = 3
+        self.n_dircache = dircache
+        self.n_filecache = filecache
 
         self.dircache = []
         self.dircache_mtx = threading.Lock()
@@ -246,14 +370,25 @@ class CPPF(Operations):
         self.filecache = []
         self.filecache_mtx = threading.Lock()
 
-        log("up")
+        info("up")
+
+    def _describe(self):
+        msg = ""
+        with self.filecache_mtx:
+            for n, cn in enumerate(self.filecache):
+                cache_path, cache1 = cn.tag
+                cache2 = cache1 + len(cn.data)
+                msg += "\n{:<2} {:>7} {:>10}:{:<9} {}".format(
+                    n, len(cn.data), cache1, cache2, cache_path
+                )
+        return msg
 
     def clean_dircache(self):
         """not threadsafe"""
         now = time.time()
         cutoff = 0
         for cn in self.dircache:
-            if cn.ts - now > 1:
+            if now - cn.ts > self.n_dircache:
                 cutoff += 1
             else:
                 break
@@ -262,8 +397,7 @@ class CPPF(Operations):
             self.dircache = self.dircache[cutoff:]
 
     def get_cached_dir(self, dirpath):
-        # with self.dircache_mtx:
-        if True:
+        with self.dircache_mtx:
             self.clean_dircache()
             for cn in self.dircache:
                 if cn.tag == dirpath:
@@ -300,9 +434,8 @@ class CPPF(Operations):
         car = None
         cdr = None
         ncn = -1
-        # with self.filecache_mtx:
-        if True:
-            dbg("cache request from {} to {}, size {}".format(get1, get2, file_sz))
+        dbg("cache request {}:{} |{}|".format(get1, get2, file_sz) + self._describe())
+        with self.filecache_mtx:
             for cn in self.filecache:
                 ncn += 1
 
@@ -312,6 +445,12 @@ class CPPF(Operations):
 
                 cache2 = cache1 + len(cn.data)
                 if get2 <= cache1 or get1 >= cache2:
+                    # request does not overlap with cached area at all
+                    continue
+
+                if get1 < cache1 and get2 > cache2:
+                    # cached area does overlap, but must specifically contain
+                    # either the first or last byte in the requested range
                     continue
 
                 if get1 >= cache1 and get2 <= cache2:
@@ -322,7 +461,7 @@ class CPPF(Operations):
                     buf_ofs = get1 - cache1
                     buf_end = buf_ofs + (get2 - get1)
                     dbg(
-                        "found all ({}, {} to {}, len {}) [{}:{}] = {}".format(
+                        "found all (#{} {}:{} |{}|) [{}:{}] = {}".format(
                             ncn,
                             cache1,
                             cache2,
@@ -334,11 +473,11 @@ class CPPF(Operations):
                     )
                     return cn.data[buf_ofs:buf_end]
 
-                if get2 < cache2:
+                if get2 <= cache2:
                     x = cn.data[: get2 - cache1]
                     if not cdr or len(cdr) < len(x):
                         dbg(
-                            "found car ({}, {} to {}, len {}) [:{}-{}] = [:{}] = {}".format(
+                            "found cdr (#{} {}:{} |{}|) [:{}-{}] = [:{}] = {}".format(
                                 ncn,
                                 cache1,
                                 cache2,
@@ -353,11 +492,11 @@ class CPPF(Operations):
 
                     continue
 
-                if get1 > cache1:
-                    x = cn.data[-(cache2 - get1) :]
+                if get1 >= cache1:
+                    x = cn.data[-(max(0, cache2 - get1)) :]
                     if not car or len(car) < len(x):
                         dbg(
-                            "found cdr ({}, {} to {}, len {}) [-({}-{}):] = [-{}:] = {}".format(
+                            "found car (#{} {}:{} |{}|) [-({}-{}):] = [-{}:] = {}".format(
                                 ncn,
                                 cache1,
                                 cache2,
@@ -372,38 +511,52 @@ class CPPF(Operations):
 
                     continue
 
-                raise Exception("what")
+                msg = "cache fallthrough\n{} {} {}\n{} {} {}\n{} {} --\n".format(
+                    get1,
+                    get2,
+                    get2 - get1,
+                    cache1,
+                    cache2,
+                    cache2 - cache1,
+                    get1 - cache1,
+                    get2 - cache2,
+                )
+                msg += self._describe()
+                raise Exception(msg)
 
-        if car and cdr:
+        if car and cdr and len(car) + len(cdr) == get2 - get1:
             dbg("<cache> have both")
+            return car + cdr
 
-            ret = car + cdr
-            if len(ret) == get2 - get1:
-                return ret
-
-            raise Exception("{} + {} != {} - {}".format(len(car), len(cdr), get2, get1))
-
-        elif cdr:
+        elif cdr and (not car or len(car) < len(cdr)):
             h_end = get1 + (get2 - get1) - len(cdr)
-            h_ofs = h_end - 512 * 1024
+            h_ofs = min(get1, h_end - 512 * 1024)
 
             if h_ofs < 0:
                 h_ofs = 0
 
-            buf_ofs = (get2 - get1) - len(cdr)
+            buf_ofs = get1 - h_ofs
 
             dbg(
-                "<cache> cdr {}, car {}-{}={} [-{}:]".format(
+                "<cache> cdr {}, car {}:{} |{}| [{}:]".format(
                     len(cdr), h_ofs, h_end, h_end - h_ofs, buf_ofs
                 )
             )
 
-            buf = self.gw.download_file_range(path, h_ofs, h_end - 1)
-            ret = buf[-buf_ofs:] + cdr
+            buf = self.gw.download_file_range(path, h_ofs, h_end)
+            if len(buf) == h_end - h_ofs:
+                ret = buf[buf_ofs:] + cdr
+            else:
+                ret = buf[get1 - h_ofs :]
+                info(
+                    "remote truncated {}:{} to |{}|, will return |{}|".format(
+                        h_ofs, h_end, len(buf), len(ret)
+                    )
+                )
 
         elif car:
             h_ofs = get1 + len(car)
-            h_end = h_ofs + 1024 * 1024
+            h_end = max(get2, h_ofs + 1024 * 1024)
 
             if h_end > file_sz:
                 h_end = file_sz
@@ -411,17 +564,22 @@ class CPPF(Operations):
             buf_ofs = (get2 - get1) - len(car)
 
             dbg(
-                "<cache> car {}, cdr {}-{}={} [:{}]".format(
+                "<cache> car {}, cdr {}:{} |{}| [:{}]".format(
                     len(car), h_ofs, h_end, h_end - h_ofs, buf_ofs
                 )
             )
 
-            buf = self.gw.download_file_range(path, h_ofs, h_end - 1)
+            buf = self.gw.download_file_range(path, h_ofs, h_end)
             ret = car + buf[:buf_ofs]
 
         else:
-            h_ofs = get1 - 256 * 1024
-            h_end = get2 + 1024 * 1024
+            if get2 - get1 <= 1024 * 1024:
+                h_ofs = get1 - 256 * 1024
+                h_end = get2 + 1024 * 1024
+            else:
+                # big enough, doesn't need pads
+                h_ofs = get1
+                h_end = get2
 
             if h_ofs < 0:
                 h_ofs = 0
@@ -433,54 +591,93 @@ class CPPF(Operations):
             buf_end = buf_ofs + get2 - get1
 
             dbg(
-                "<cache> {}-{}={} [{}:{}]".format(
+                "<cache> {}:{} |{}| [{}:{}]".format(
                     h_ofs, h_end, h_end - h_ofs, buf_ofs, buf_end
                 )
             )
 
-            buf = self.gw.download_file_range(path, h_ofs, h_end - 1)
+            buf = self.gw.download_file_range(path, h_ofs, h_end)
             ret = buf[buf_ofs:buf_end]
 
         cn = CacheNode([path, h_ofs], buf)
-        # with self.filecache_mtx:
-        if True:
-            if len(self.filecache) > 6:
+        with self.filecache_mtx:
+            if len(self.filecache) >= self.n_filecache:
                 self.filecache = self.filecache[1:] + [cn]
             else:
                 self.filecache.append(cn)
 
         return ret
 
-    def readdir(self, path, fh=None):
+    def _readdir(self, path, fh=None):
         path = path.strip("/")
-        log("readdir {}".format(path))
+        log("readdir [{}] [{}]".format(path, fh))
 
         ret = self.gw.listdir(path)
+        if not self.n_dircache:
+            return ret
 
-        # with self.dircache_mtx:
-        if True:
+        with self.dircache_mtx:
             cn = CacheNode(path, ret)
             self.dircache.append(cn)
             self.clean_dircache()
 
         return ret
 
+    def readdir(self, path, fh=None):
+        return [".", ".."] + self._readdir(path, fh)
+
     def read(self, path, length, offset, fh=None):
+        req_max = 1024 * 1024 * 8
+        cache_max = 1024 * 1024 * 2
+        if length > req_max:
+            # windows actually doing 240 MiB read calls, sausage
+            info("truncate |{}| to {}MiB".format(length, req_max >> 20))
+            length = req_max
+
         path = path.strip("/")
-
         ofs2 = offset + length
-        log("read {} @ {} len {} end {}".format(path, offset, length, ofs2))
-
         file_sz = self.getattr(path)["st_size"]
-        if ofs2 >= file_sz:
-            ofs2 = file_sz - 1
-            log("truncate to len {} end {}".format((ofs2 - offset) + 1, ofs2))
+        log("read {} |{}| {}:{} max {}".format(path, length, offset, ofs2, file_sz))
+        if ofs2 > file_sz:
+            ofs2 = file_sz
+            log("truncate to |{}| :{}".format(ofs2 - offset, ofs2))
 
-        # toggle cache here i suppose
-        # return self.get_cached_file(path, offset, ofs2, file_sz)
-        return self.gw.download_file_range(path, offset, ofs2 - 1)
+        if file_sz == 0 or offset >= ofs2:
+            return b""
+
+        if self.n_filecache and length <= cache_max:
+            ret = self.get_cached_file(path, offset, ofs2, file_sz)
+        else:
+            ret = self.gw.download_file_range(path, offset, ofs2)
+
+        return ret
+
+        fn = "cppf-{}-{}-{}".format(time.time(), offset, length)
+        if False:
+            with open(fn, "wb", len(ret)) as f:
+                f.write(ret)
+        elif self.n_filecache:
+            ret2 = self.gw.download_file_range(path, offset, ofs2)
+            if ret != ret2:
+                info(fn)
+                for v in [ret, ret2]:
+                    try:
+                        info(len(v))
+                    except:
+                        info("uhh " + repr(v))
+
+                with open(fn + ".bad", "wb") as f:
+                    f.write(ret)
+                with open(fn + ".good", "wb") as f:
+                    f.write(ret2)
+
+                raise Exception("cache bork")
+
+        return ret
 
     def getattr(self, path, fh=None):
+        log("getattr [{}]".format(path))
+
         path = path.strip("/")
         try:
             dirpath, fname = path.rsplit("/", 1)
@@ -488,23 +685,25 @@ class CPPF(Operations):
             dirpath = ""
             fname = path
 
-        log("getattr {}".format(path))
-
         if not path:
-            return self.gw.stat_dir(time.time())
+            ret = self.gw.stat_dir(time.time())
+            # dbg("=" + repr(ret))
+            return ret
 
         cn = self.get_cached_dir(dirpath)
         if cn:
-            # log('cache ok')
+            log("cache ok")
             dents = cn.data
         else:
-            log("cache miss")
-            dents = self.readdir(dirpath)
+            dbg("cache miss")
+            dents = self._readdir(dirpath)
 
         for cache_name, cache_stat, _ in dents:
             if cache_name == fname:
+                # dbg("=" + repr(cache_stat))
                 return cache_stat
 
+        info("=ENOENT ({})".format(path))
         raise FuseOSError(errno.ENOENT)
 
     access = None
@@ -517,17 +716,136 @@ class CPPF(Operations):
     releasedir = None
     statfs = None
 
+    if False:
+        # incorrect semantics but good for debugging stuff like samba and msys2
+        def access(self, path, mode):
+            log("@@ access [{}] [{}]".format(path, mode))
+            return 1 if self.getattr(path) else 0
+
+        def flush(self, path, fh):
+            log("@@ flush [{}] [{}]".format(path, fh))
+            return True
+
+        def getxattr(self, *args):
+            log("@@ getxattr [{}]".format("] [".join(str(x) for x in args)))
+            return False
+
+        def listxattr(self, *args):
+            log("@@ listxattr [{}]".format("] [".join(str(x) for x in args)))
+            return False
+
+        def open(self, path, flags):
+            log("@@ open [{}] [{}]".format(path, flags))
+            return 42
+
+        def opendir(self, fh):
+            log("@@ opendir [{}]".format(fh))
+            return 69
+
+        def release(self, ino, fi):
+            log("@@ release [{}] [{}]".format(ino, fi))
+            return True
+
+        def releasedir(self, ino, fi):
+            log("@@ releasedir [{}] [{}]".format(ino, fi))
+            return True
+
+        def statfs(self, path):
+            log("@@ statfs [{}]".format(path))
+            return {}
+
+    if sys.platform == "win32":
+        # quick compat for /mingw64/bin/python3 (msys2)
+        def _open(self, path):
+            try:
+                x = self.getattr(path)
+                if x["st_mode"] <= 0:
+                    raise Exception()
+
+                self.junk_fh_ctr += 1
+                if self.junk_fh_ctr > 32000:  # TODO untested
+                    self.junk_fh_ctr = 4
+
+                return self.junk_fh_ctr
+
+            except Exception as ex:
+                log("open ERR {}".format(repr(ex)))
+                raise FuseOSError(errno.ENOENT)
+
+        def open(self, path, flags):
+            dbg("open [{}] [{}]".format(path, flags))
+            return self._open(path)
+
+        def opendir(self, path):
+            dbg("opendir [{}]".format(path))
+            return self._open(path)
+
+        def flush(self, path, fh):
+            dbg("flush [{}] [{}]".format(path, fh))
+
+        def release(self, ino, fi):
+            dbg("release [{}] [{}]".format(ino, fi))
+
+        def releasedir(self, ino, fi):
+            dbg("releasedir [{}] [{}]".format(ino, fi))
+
+        def access(self, path, mode):
+            dbg("access [{}] [{}]".format(path, mode))
+            try:
+                x = self.getattr(path)
+                if x["st_mode"] <= 0:
+                    raise Exception()
+            except:
+                raise FuseOSError(errno.ENOENT)
+
 
 def main():
+    # filecache helps for reads that are ~64k or smaller;
+    #   linux generally does 128k so the cache is a slowdown,
+    #   windows likes to use 4k and 64k so cache is required,
+    #   value is numChunks (1~3M each) to keep in the cache
+    nf = 24 if WINDOWS or MACOS else 0
+
+    # dircache is always a boost,
+    #   only want to disable it for tests etc,
+    #   value is numSec until an entry goes stale
+    nd = 1
+
     try:
-        local, remote = sys.argv[1:]
+        local, remote = sys.argv[1:3]
+        filecache = nf if len(sys.argv) <= 3 else int(sys.argv[3])
+        dircache = nd if len(sys.argv) <= 4 else float(sys.argv[4])
     except:
-        print("need arg 1: local directory")
+        where = "local directory"
+        if WINDOWS:
+            where += " or DRIVE:"
+
+        print("need arg 1: " + where)
         print("need arg 2: root url")
+        print("optional 3: num files in filecache ({})".format(nf))
+        print("optional 4: num seconds / dircache ({})".format(nd))
+        print()
+        print("example:")
+        print("  copyparty-fuse.py ./music http://192.168.1.69:3923/music/")
+        if WINDOWS:
+            print("  copyparty-fuse.py M: http://192.168.1.69:3923/music/")
+
         return
 
-    FUSE(CPPF(remote), local, foreground=True, nothreads=True)
-    # if nothreads=False also uncomment the `with *_mtx` things
+    if WINDOWS:
+        os.system("")
+
+    try:
+        with open("/etc/fuse.conf", "rb") as f:
+            allow_other = b"\nuser_allow_other" in f.read()
+    except:
+        allow_other = WINDOWS or MACOS
+
+    args = {"foreground": True, "nothreads": True, "allow_other": allow_other}
+    if not MACOS:
+        args["nonempty"] = True
+
+    FUSE(CPPF(remote, dircache, filecache), local, **args)
 
 
 if __name__ == "__main__":

bin/copyparty-fuseb.py

@@ -0,0 +1,590 @@
+#!/usr/bin/env python3
+from __future__ import print_function, unicode_literals
+
+"""copyparty-fuseb: remote copyparty as a local filesystem"""
+__author__ = "ed <copyparty@ocv.me>"
+__copyright__ = 2020
+__license__ = "MIT"
+__url__ = "https://github.com/9001/copyparty/"
+
+import re
+import os
+import sys
+import time
+import stat
+import errno
+import struct
+import threading
+import http.client  # py2: httplib
+import urllib.parse
+from datetime import datetime
+from urllib.parse import quote_from_bytes as quote
+
+try:
+    import fuse
+    from fuse import Fuse
+
+    fuse.fuse_python_api = (0, 2)
+    if not hasattr(fuse, "__version__"):
+        raise Exception("your fuse-python is way old")
+except:
+    print(
+        "\n  could not import fuse; these may help:\n    python3 -m pip install --user fuse-python\n    apt install libfuse\n    modprobe fuse\n"
+    )
+    raise
+
+
+"""
+mount a copyparty server (local or remote) as a filesystem
+
+usage:
+  python ./copyparty-fuseb.py -f -o allow_other,auto_unmount,nonempty,url=http://192.168.1.69:3923 /mnt/nas
+
+dependencies:
+  sudo apk add fuse-dev python3-dev
+  python3 -m pip install --user fuse-python
+
+fork of copyparty-fuse.py based on fuse-python which
+  appears to be more compliant than fusepy? since this works with samba
+    (probably just my garbage code tbh)
+"""
+
+
+def threadless_log(msg):
+    print(msg + "\n", end="")
+
+
+def boring_log(msg):
+    msg = "\033[36m{:012x}\033[0m {}\n".format(threading.current_thread().ident, msg)
+    print(msg[4:], end="")
+
+
+def rice_tid():
+    tid = threading.current_thread().ident
+    c = struct.unpack(b"B" * 5, struct.pack(b">Q", tid)[-5:])
+    return "".join("\033[1;37;48;5;{}m{:02x}".format(x, x) for x in c) + "\033[0m"
+
+
+def fancy_log(msg):
+    print("{} {}\n".format(rice_tid(), msg), end="")
+
+
+def null_log(msg):
+    pass
+
+
+info = fancy_log
+log = fancy_log
+dbg = fancy_log
+log = null_log
+dbg = null_log
+
+
+def get_tid():
+    return threading.current_thread().ident
+
+
+def html_dec(txt):
+    return (
+        txt.replace("&lt;", "<")
+        .replace("&gt;", ">")
+        .replace("&quot;", '"')
+        .replace("&amp;", "&")
+    )
+
+
+class CacheNode(object):
+    def __init__(self, tag, data):
+        self.tag = tag
+        self.data = data
+        self.ts = time.time()
+
+
+class Stat(fuse.Stat):
+    def __init__(self):
+        self.st_mode = 0
+        self.st_ino = 0
+        self.st_dev = 0
+        self.st_nlink = 1
+        self.st_uid = 1000
+        self.st_gid = 1000
+        self.st_size = 0
+        self.st_atime = 0
+        self.st_mtime = 0
+        self.st_ctime = 0
+
+
+class Gateway(object):
+    def __init__(self, base_url):
+        self.base_url = base_url
+
+        ui = urllib.parse.urlparse(base_url)
+        self.web_root = ui.path.strip("/")
+        try:
+            self.web_host, self.web_port = ui.netloc.split(":")
+            self.web_port = int(self.web_port)
+        except:
+            self.web_host = ui.netloc
+            if ui.scheme == "http":
+                self.web_port = 80
+            elif ui.scheme == "https":
+                raise Exception("todo")
+            else:
+                raise Exception("bad url?")
+
+        self.conns = {}
+
+    def quotep(self, path):
+        # TODO: mojibake support
+        path = path.encode("utf-8", "ignore")
+        return quote(path, safe="/")
+
+    def getconn(self, tid=None):
+        tid = tid or get_tid()
+        try:
+            return self.conns[tid]
+        except:
+            info("new conn [{}] [{}]".format(self.web_host, self.web_port))
+
+            conn = http.client.HTTPConnection(self.web_host, self.web_port, timeout=260)
+
+            self.conns[tid] = conn
+            return conn
+
+    def closeconn(self, tid=None):
+        tid = tid or get_tid()
+        try:
+            self.conns[tid].close()
+            del self.conns[tid]
+        except:
+            pass
+
+    def sendreq(self, *args, **kwargs):
+        tid = get_tid()
+        try:
+            c = self.getconn(tid)
+            c.request(*list(args), **kwargs)
+            return c.getresponse()
+        except:
+            self.closeconn(tid)
+            c = self.getconn(tid)
+            c.request(*list(args), **kwargs)
+            return c.getresponse()
+
+    def listdir(self, path):
+        web_path = self.quotep("/" + "/".join([self.web_root, path])) + "?dots"
+        r = self.sendreq("GET", web_path)
+        if r.status != 200:
+            self.closeconn()
+            raise Exception(
+                "http error {} reading dir {} in {}".format(
+                    r.status, web_path, rice_tid()
+                )
+            )
+
+        return self.parse_html(r)
+
+    def download_file_range(self, path, ofs1, ofs2):
+        web_path = self.quotep("/" + "/".join([self.web_root, path])) + "?raw"
+        hdr_range = "bytes={}-{}".format(ofs1, ofs2 - 1)
+        log("downloading {}".format(hdr_range))
+
+        r = self.sendreq("GET", web_path, headers={"Range": hdr_range})
+        if r.status != http.client.PARTIAL_CONTENT:
+            self.closeconn()
+            raise Exception(
+                "http error {} reading file {} range {} in {}".format(
+                    r.status, web_path, hdr_range, rice_tid()
+                )
+            )
+
+        return r.read()
+
+    def parse_html(self, datasrc):
+        ret = []
+        remainder = b""
+        ptn = re.compile(
+            r"^<tr><td>(-|DIR)</td><td><a [^>]+>([^<]+)</a></td><td>([^<]+)</td><td>([^<]+)</td></tr>$"
+        )
+
+        while True:
+            buf = remainder + datasrc.read(4096)
+            # print('[{}]'.format(buf.decode('utf-8')))
+            if not buf:
+                break
+
+            remainder = b""
+            endpos = buf.rfind(b"\n")
+            if endpos >= 0:
+                remainder = buf[endpos + 1 :]
+                buf = buf[:endpos]
+
+            lines = buf.decode("utf-8").split("\n")
+            for line in lines:
+                m = ptn.match(line)
+                if not m:
+                    # print(line)
+                    continue
+
+                ftype, fname, fsize, fdate = m.groups()
+                fname = html_dec(fname)
+                ts = datetime.strptime(fdate, "%Y-%m-%d %H:%M:%S").timestamp()
+                sz = int(fsize)
+                if ftype == "-":
+                    ret.append([fname, self.stat_file(ts, sz), 0])
+                else:
+                    ret.append([fname, self.stat_dir(ts, sz), 0])
+
+        return ret
+
+    def stat_dir(self, ts, sz=4096):
+        ret = Stat()
+        ret.st_mode = stat.S_IFDIR | 0o555
+        ret.st_nlink = 2
+        ret.st_size = sz
+        ret.st_atime = ts
+        ret.st_mtime = ts
+        ret.st_ctime = ts
+        return ret
+
+    def stat_file(self, ts, sz):
+        ret = Stat()
+        ret.st_mode = stat.S_IFREG | 0o444
+        ret.st_size = sz
+        ret.st_atime = ts
+        ret.st_mtime = ts
+        ret.st_ctime = ts
+        return ret
+
+
+class CPPF(Fuse):
+    def __init__(self, *args, **kwargs):
+        Fuse.__init__(self, *args, **kwargs)
+
+        self.url = None
+
+        self.dircache = []
+        self.dircache_mtx = threading.Lock()
+
+        self.filecache = []
+        self.filecache_mtx = threading.Lock()
+
+    def init2(self):
+        # TODO figure out how python-fuse wanted this to go
+        self.gw = Gateway(self.url)  # .decode('utf-8'))
+        info("up")
+
+    def clean_dircache(self):
+        """not threadsafe"""
+        now = time.time()
+        cutoff = 0
+        for cn in self.dircache:
+            if now - cn.ts > 1:
+                cutoff += 1
+            else:
+                break
+
+        if cutoff > 0:
+            self.dircache = self.dircache[cutoff:]
+
+    def get_cached_dir(self, dirpath):
+        # with self.dircache_mtx:
+        if True:
+            self.clean_dircache()
+            for cn in self.dircache:
+                if cn.tag == dirpath:
+                    return cn
+
+        return None
+
+    """
+            ,-------------------------------,  g1>=c1, g2<=c2
+            |cache1                   cache2|  buf[g1-c1:(g1-c1)+(g2-g1)]
+            `-------------------------------'
+                    ,---------------,
+                    |get1       get2|
+                    `---------------'
+    __________________________________________________________________________
+
+            ,-------------------------------,  g2<=c2, (g2>=c1)
+            |cache1                   cache2|  cdr=buf[:g2-c1]
+            `-------------------------------'  dl car; g1-512K:c1
+    ,---------------,
+    |get1       get2|
+    `---------------'
+    __________________________________________________________________________
+
+            ,-------------------------------,  g1>=c1, (g1<=c2)
+            |cache1                   cache2|  car=buf[c2-g1:]
+            `-------------------------------'  dl cdr; c2:c2+1M
+                                    ,---------------,
+                                    |get1       get2|
+                                    `---------------'
+    """
+
+    def get_cached_file(self, path, get1, get2, file_sz):
+        car = None
+        cdr = None
+        ncn = -1
+        # with self.filecache_mtx:
+        if True:
+            dbg("cache request from {} to {}, size {}".format(get1, get2, file_sz))
+            for cn in self.filecache:
+                ncn += 1
+
+                cache_path, cache1 = cn.tag
+                if cache_path != path:
+                    continue
+
+                cache2 = cache1 + len(cn.data)
+                if get2 <= cache1 or get1 >= cache2:
+                    continue
+
+                if get1 >= cache1 and get2 <= cache2:
+                    # keep cache entry alive by moving it to the end
+                    self.filecache = (
+                        self.filecache[:ncn] + self.filecache[ncn + 1 :] + [cn]
+                    )
+                    buf_ofs = get1 - cache1
+                    buf_end = buf_ofs + (get2 - get1)
+                    dbg(
+                        "found all ({}, {} to {}, len {}) [{}:{}] = {}".format(
+                            ncn,
+                            cache1,
+                            cache2,
+                            len(cn.data),
+                            buf_ofs,
+                            buf_end,
+                            buf_end - buf_ofs,
+                        )
+                    )
+                    return cn.data[buf_ofs:buf_end]
+
+                if get2 < cache2:
+                    x = cn.data[: get2 - cache1]
+                    if not cdr or len(cdr) < len(x):
+                        dbg(
+                            "found car ({}, {} to {}, len {}) [:{}-{}] = [:{}] = {}".format(
+                                ncn,
+                                cache1,
+                                cache2,
+                                len(cn.data),
+                                get2,
+                                cache1,
+                                get2 - cache1,
+                                len(x),
+                            )
+                        )
+                        cdr = x
+
+                    continue
+
+                if get1 > cache1:
+                    x = cn.data[-(cache2 - get1) :]
+                    if not car or len(car) < len(x):
+                        dbg(
+                            "found cdr ({}, {} to {}, len {}) [-({}-{}):] = [-{}:] = {}".format(
+                                ncn,
+                                cache1,
+                                cache2,
+                                len(cn.data),
+                                cache2,
+                                get1,
+                                cache2 - get1,
+                                len(x),
+                            )
+                        )
+                        car = x
+
+                    continue
+
+                raise Exception("what")
+
+        if car and cdr:
+            dbg("<cache> have both")
+
+            ret = car + cdr
+            if len(ret) == get2 - get1:
+                return ret
+
+            raise Exception("{} + {} != {} - {}".format(len(car), len(cdr), get2, get1))
+
+        elif cdr:
+            h_end = get1 + (get2 - get1) - len(cdr)
+            h_ofs = h_end - 512 * 1024
+
+            if h_ofs < 0:
+                h_ofs = 0
+
+            buf_ofs = (get2 - get1) - len(cdr)
+
+            dbg(
+                "<cache> cdr {}, car {}-{}={} [-{}:]".format(
+                    len(cdr), h_ofs, h_end, h_end - h_ofs, buf_ofs
+                )
+            )
+
+            buf = self.gw.download_file_range(path, h_ofs, h_end)
+            ret = buf[-buf_ofs:] + cdr
+
+        elif car:
+            h_ofs = get1 + len(car)
+            h_end = h_ofs + 1024 * 1024
+
+            if h_end > file_sz:
+                h_end = file_sz
+
+            buf_ofs = (get2 - get1) - len(car)
+
+            dbg(
+                "<cache> car {}, cdr {}-{}={} [:{}]".format(
+                    len(car), h_ofs, h_end, h_end - h_ofs, buf_ofs
+                )
+            )
+
+            buf = self.gw.download_file_range(path, h_ofs, h_end)
+            ret = car + buf[:buf_ofs]
+
+        else:
+            h_ofs = get1 - 256 * 1024
+            h_end = get2 + 1024 * 1024
+
+            if h_ofs < 0:
+                h_ofs = 0
+
+            if h_end > file_sz:
+                h_end = file_sz
+
+            buf_ofs = get1 - h_ofs
+            buf_end = buf_ofs + get2 - get1
+
+            dbg(
+                "<cache> {}-{}={} [{}:{}]".format(
+                    h_ofs, h_end, h_end - h_ofs, buf_ofs, buf_end
+                )
+            )
+
+            buf = self.gw.download_file_range(path, h_ofs, h_end)
+            ret = buf[buf_ofs:buf_end]
+
+        cn = CacheNode([path, h_ofs], buf)
+        # with self.filecache_mtx:
+        if True:
+            if len(self.filecache) > 6:
+                self.filecache = self.filecache[1:] + [cn]
+            else:
+                self.filecache.append(cn)
+
+        return ret
+
+    def _readdir(self, path):
+        path = path.strip("/")
+        log("readdir {}".format(path))
+
+        ret = self.gw.listdir(path)
+
+        # with self.dircache_mtx:
+        if True:
+            cn = CacheNode(path, ret)
+            self.dircache.append(cn)
+            self.clean_dircache()
+
+        return ret
+
+    def readdir(self, path, offset):
+        for e in self._readdir(path)[offset:]:
+            # log("yield [{}]".format(e[0]))
+            yield fuse.Direntry(e[0])
+
+    def open(self, path, flags):
+        if (flags & (os.O_RDONLY | os.O_WRONLY | os.O_RDWR)) != os.O_RDONLY:
+            return -errno.EACCES
+
+        st = self.getattr(path)
+        try:
+            if st.st_nlink > 0:
+                return st
+        except:
+            return st  # -int(os.errcode)
+
+    def read(self, path, length, offset, fh=None, *args):
+        if args:
+            log("unexpected args [" + "] [".join(repr(x) for x in args) + "]")
+            raise Exception()
+
+        path = path.strip("/")
+
+        ofs2 = offset + length
+        log("read {} @ {} len {} end {}".format(path, offset, length, ofs2))
+
+        st = self.getattr(path)
+        try:
+            file_sz = st.st_size
+        except:
+            return st  # -int(os.errcode)
+
+        if ofs2 > file_sz:
+            ofs2 = file_sz
+            log("truncate to len {} end {}".format(ofs2 - offset, ofs2))
+
+        if file_sz == 0 or offset >= ofs2:
+            return b""
+
+        # toggle cache here i suppose
+        # return self.get_cached_file(path, offset, ofs2, file_sz)
+        return self.gw.download_file_range(path, offset, ofs2)
+
+    def getattr(self, path):
+        log("getattr [{}]".format(path))
+
+        path = path.strip("/")
+        try:
+            dirpath, fname = path.rsplit("/", 1)
+        except:
+            dirpath = ""
+            fname = path
+
+        if not path:
+            ret = self.gw.stat_dir(time.time())
+            dbg("=root")
+            return ret
+
+        cn = self.get_cached_dir(dirpath)
+        if cn:
+            log("cache ok")
+            dents = cn.data
+        else:
+            log("cache miss")
+            dents = self._readdir(dirpath)
+
+        for cache_name, cache_stat, _ in dents:
+            if cache_name == fname:
+                dbg("=file")
+                return cache_stat
+
+        log("=404")
+        return -errno.ENOENT
+
+
+def main():
+    server = CPPF()
+    server.parser.add_option(mountopt="url", metavar="BASE_URL", default=None)
+    server.parse(values=server, errex=1)
+    if not server.url or not str(server.url).startswith("http"):
+        print("\nerror:")
+        print("  need argument: -o url=<...>")
+        print("  need argument: mount-path")
+        print("example:")
+        print(
+            "  ./copyparty-fuseb.py -f -o allow_other,auto_unmount,nonempty,url=http://192.168.1.69:3923 /mnt/nas"
+        )
+        sys.exit(1)
+
+    server.init2()
+    threading.Thread(target=server.main, daemon=True).start()
+    while True:
+        time.sleep(9001)
+
+
+if __name__ == "__main__":
+    main()

bin/up2k.sh

@@ -118,7 +118,7 @@ printf ']}' >> /dev/shm/$salt.hs
 
 printf '\033[36m'
 
-#curl "http://$target:1234$posturl/handshake.php" -H "Content-Type: text/plain;charset=UTF-8" -H "Cookie: cppwd=$passwd" --data "$(cat "/dev/shm/$salt.hs")" | tee /dev/shm/$salt.res
+#curl "http://$target:3923$posturl/handshake.php" -H "Content-Type: text/plain;charset=UTF-8" -H "Cookie: cppwd=$passwd" --data "$(cat "/dev/shm/$salt.hs")" | tee /dev/shm/$salt.res
 
 {
     {
@@ -135,7 +135,7 @@ EOF
     cat /dev/shm/$salt.hs
 } |
 tee /dev/shm/$salt.hsb |
-ncat $target 1234 |
+ncat $target 3923 |
 tee /dev/shm/$salt.hs1r
 
 wark="$(cat /dev/shm/$salt.hs1r | getwark)"
@@ -190,7 +190,7 @@ EOF
     nchunk=$((nchunk+1))
 
 done |
-ncat $target 1234 |
+ncat $target 3923 |
 tee /dev/shm/$salt.pr
 
 t=$(date +%s.%N)
@@ -201,7 +201,7 @@ t=$(date +%s.%N)
 
 printf '\033[36m'
 
-ncat $target 1234 < /dev/shm/$salt.hsb |
+ncat $target 3923 < /dev/shm/$salt.hsb |
 tee /dev/shm/$salt.hs2r |
 grep -E '"hash": ?\[ *\]'
 

copyparty/__init__.py

@@ -16,6 +16,8 @@ if platform.system() == "Windows":
 VT100 = not WINDOWS or WINDOWS >= [10, 0, 14393]
 # introduced in anniversary update
 
+MACOS = platform.system() == "Darwin"
+
 
 class EnvParams(object):
     def __init__(self):

copyparty/__main__.py

@@ -127,13 +127,16 @@ def main():
         "-c", metavar="PATH", type=str, action="append", help="add config file"
     )
     ap.add_argument("-i", metavar="IP", type=str, default="0.0.0.0", help="ip to bind")
-    ap.add_argument("-p", metavar="PORT", type=int, default=1234, help="port to bind")
+    ap.add_argument("-p", metavar="PORT", type=int, default=3923, help="port to bind")
     ap.add_argument("-nc", metavar="NUM", type=int, default=16, help="max num clients")
-    ap.add_argument("-j", metavar="CORES", type=int, help="max num cpu cores")
+    ap.add_argument(
+        "-j", metavar="CORES", type=int, default=1, help="max num cpu cores"
+    )
     ap.add_argument("-a", metavar="ACCT", type=str, action="append", help="add account")
     ap.add_argument("-v", metavar="VOL", type=str, action="append", help="add volume")
     ap.add_argument("-q", action="store_true", help="quiet")
-    ap.add_argument("-nw", action="store_true", help="benchmark: disable writing")
+    ap.add_argument("-ed", action="store_true", help="enable ?dots")
+    ap.add_argument("-nw", action="store_true", help="disable writes (benchmark)")
     al = ap.parse_args()
 
     SvcHub(al).run()

copyparty/__version__.py

@@ -1,8 +1,8 @@
 # coding: utf-8
 
-VERSION = (0, 3, 0)
-CODENAME = "docuparty"
-BUILD_DT = (2020, 5, 6)
+VERSION = (0, 5, 0)
+CODENAME = "fuse jelly"
+BUILD_DT = (2020, 8, 17)
 
 S_VERSION = ".".join(map(str, VERSION))
 S_BUILD_DT = "{0:04d}-{1:02d}-{2:02d}".format(*BUILD_DT)

copyparty/authsrv.py

@@ -135,9 +135,9 @@ class AuthSrv(object):
         self.warn_anonwrite = True
 
         if WINDOWS:
-            self.re_vol = re.compile(r"^([a-zA-Z]:[\\/][^:]*|[^:]*):([^:]*):(.*)")
+            self.re_vol = re.compile(r"^([a-zA-Z]:[\\/][^:]*|[^:]*):([^:]*):(.*)$")
         else:
-            self.re_vol = re.compile(r"^([^:]*):([^:]*):(.*)")
+            self.re_vol = re.compile(r"^([^:]*):([^:]*):(.*)$")
 
         self.mutex = threading.Lock()
         self.reload()
@@ -220,12 +220,13 @@ class AuthSrv(object):
         if self.args.v:
             # list of src:dst:permset:permset:...
             # permset is [rwa]username
-            for vol_match in [self.re_vol.match(x) for x in self.args.v]:
-                try:
-                    src, dst, perms = vol_match.groups()
-                except:
-                    raise Exception("invalid -v argument")
+            for v_str in self.args.v:
+                m = self.re_vol.match(v_str)
+                if not m:
+                    raise Exception("invalid -v argument: [{}]".format(v_str))
 
+                src, dst, perms = m.groups()
+                # print("\n".join([src, dst, perms]))
                 src = fsdec(os.path.abspath(fsenc(src)))
                 dst = dst.strip("/")
                 mount[dst] = src

copyparty/broker_mp.py

@@ -29,7 +29,7 @@ class BrokerMp(object):
         self.mutex = threading.Lock()
 
         cores = self.args.j
-        if cores is None:
+        if not cores:
             cores = mp.cpu_count()
 
         self.log("broker", "booting {} subprocesses".format(cores))

copyparty/broker_util.py

@@ -4,7 +4,6 @@ from __future__ import print_function, unicode_literals
 
 import traceback
 
-from .__init__ import PY2
 from .util import Pebkac, Queue
 
 

copyparty/httpcli.py

@@ -36,13 +36,13 @@ class HttpCli(object):
 
         self.bufsz = 1024 * 32
         self.absolute_urls = False
-        self.out_headers = {}
+        self.out_headers = {"Access-Control-Allow-Origin": "*"}
 
     def log(self, msg):
         self.log_func(self.log_src, msg)
 
     def _check_nonfatal(self, ex):
-        return ex.code in [403, 404]
+        return ex.code in [404]
 
     def _assert_safe_rem(self, rem):
         # sanity check to prevent any disasters
@@ -128,6 +128,10 @@ class HttpCli(object):
                 return self.handle_get() and self.keepalive
             elif self.mode == "POST":
                 return self.handle_post() and self.keepalive
+            elif self.mode == "PUT":
+                return self.handle_put() and self.keepalive
+            elif self.mode == "OPTIONS":
+                return self.handle_options() and self.keepalive
             else:
                 raise Pebkac(400, 'invalid HTTP mode "{0}"'.format(self.mode))
 
@@ -143,9 +147,7 @@ class HttpCli(object):
     def send_headers(self, length, status=200, mime=None, headers={}):
         response = ["HTTP/1.1 {} {}".format(status, HTTPCODE[status])]
 
-        if length is None:
-            self.keepalive = False
-        else:
+        if length is not None:
             response.append("Content-Length: " + str(length))
 
         # close if unknown length, otherwise take client's preference
@@ -230,6 +232,30 @@ class HttpCli(object):
 
         return self.tx_browser()
 
+    def handle_options(self):
+        self.log("OPTIONS " + self.req)
+        self.send_headers(
+            None,
+            204,
+            headers={
+                "Access-Control-Allow-Origin": "*",
+                "Access-Control-Allow-Methods": "*",
+                "Access-Control-Allow-Headers": "*",
+            },
+        )
+        return True
+
+    def handle_put(self):
+        self.log("PUT " + self.req)
+
+        if self.headers.get("expect", "").lower() == "100-continue":
+            try:
+                self.s.sendall(b"HTTP/1.1 100 Continue\r\n\r\n")
+            except:
+                raise Pebkac(400, "client d/c before 100 continue")
+
+        return self.handle_stash()
+
     def handle_post(self):
         self.log("POST " + self.req)
 
@@ -243,6 +269,9 @@ class HttpCli(object):
         if not ctype:
             raise Pebkac(400, "you can't post without a content-type header")
 
+        if "raw" in self.uparam:
+            return self.handle_stash()
+
         if "multipart/form-data" in ctype:
             return self.handle_post_multipart()
 
@@ -255,6 +284,28 @@ class HttpCli(object):
 
         raise Pebkac(405, "don't know how to handle {} POST".format(ctype))
 
+    def handle_stash(self):
+        remains = int(self.headers.get("content-length", None))
+        if remains is None:
+            reader = read_socket_unbounded(self.sr)
+            self.keepalive = False
+        else:
+            reader = read_socket(self.sr, remains)
+
+        vfs, rem = self.conn.auth.vfs.get(self.vpath, self.uname, False, True)
+        fdir = os.path.join(vfs.realpath, rem)
+
+        addr = self.conn.addr[0].replace(":", ".")
+        fn = "put-{:.6f}-{}.bin".format(time.time(), addr)
+        path = os.path.join(fdir, fn)
+
+        with open(path, "wb", 512 * 1024) as f:
+            post_sz, _, sha_b64 = hashcopy(self.conn, reader, f)
+
+        self.log("wrote {}/{} bytes to {}".format(post_sz, remains, path))
+        self.reply("{}\n{}\n".format(post_sz, sha_b64).encode("utf-8"))
+        return True
+
     def handle_post_multipart(self):
         self.parser = MultipartParser(self.log, self.sr, self.headers)
         self.parser.parse()
@@ -420,9 +471,11 @@ class HttpCli(object):
         vfs, rem = self.conn.auth.vfs.get(self.vpath, self.uname, False, True)
         self._assert_safe_rem(rem)
 
+        sanitized = sanitize_fn(new_dir)
+
         if not nullwrite:
             fdir = os.path.join(vfs.realpath, rem)
-            fn = os.path.join(fdir, sanitize_fn(new_dir))
+            fn = os.path.join(fdir, sanitized)
 
             if not os.path.isdir(fsenc(fdir)):
                 raise Pebkac(500, "parent folder does not exist")
@@ -435,7 +488,7 @@ class HttpCli(object):
             except:
                 raise Pebkac(500, "mkdir failed, check the logs")
 
-        vpath = "{}/{}".format(self.vpath, new_dir).lstrip("/")
+        vpath = "{}/{}".format(self.vpath, sanitized).lstrip("/")
         html = self.conn.tpl_msg.render(
             h2='<a href="/{}">go to /{}</a>'.format(
                 quotep(vpath), html_escape(vpath, quote=False)
@@ -457,9 +510,11 @@ class HttpCli(object):
         if not new_file.endswith(".md"):
             new_file += ".md"
 
+        sanitized = sanitize_fn(new_file)
+
         if not nullwrite:
             fdir = os.path.join(vfs.realpath, rem)
-            fn = os.path.join(fdir, sanitize_fn(new_file))
+            fn = os.path.join(fdir, sanitized)
 
             if os.path.exists(fsenc(fn)):
                 raise Pebkac(500, "that file exists already")
@@ -467,7 +522,7 @@ class HttpCli(object):
             with open(fsenc(fn), "wb") as f:
                 f.write(b"`GRUNNUR`\n")
 
-        vpath = "{}/{}".format(self.vpath, new_file).lstrip("/")
+        vpath = "{}/{}".format(self.vpath, sanitized).lstrip("/")
         html = self.conn.tpl_msg.render(
             h2='<a href="/{}?edit">go to /{}?edit</a>'.format(
                 quotep(vpath), html_escape(vpath, quote=False)
@@ -600,10 +655,10 @@ class HttpCli(object):
             self.reply(response.encode("utf-8"))
             return True
 
-        fn = os.path.join(vfs.realpath, rem)
+        fp = os.path.join(vfs.realpath, rem)
         srv_lastmod = -1
         try:
-            st = os.stat(fsenc(fn))
+            st = os.stat(fsenc(fp))
             srv_lastmod = st.st_mtime
             srv_lastmod3 = int(srv_lastmod * 1000)
         except OSError as ex:
@@ -612,7 +667,16 @@ class HttpCli(object):
 
         # if file exists, chekc that timestamp matches the client's
         if srv_lastmod >= 0:
-            if cli_lastmod3 not in [-1, srv_lastmod3]:
+            same_lastmod = cli_lastmod3 in [-1, srv_lastmod3]
+            if not same_lastmod:
+                # some filesystems/transports limit precision to 1sec, hopefully floored
+                same_lastmod = (
+                    srv_lastmod == int(srv_lastmod)
+                    and cli_lastmod3 > srv_lastmod3
+                    and cli_lastmod3 - srv_lastmod3 < 1000
+                )
+
+            if not same_lastmod:
                 response = json.dumps(
                     {
                         "ok": False,
@@ -631,16 +695,22 @@ class HttpCli(object):
                 return True
 
             # TODO another hack re: pending permissions rework
-            os.rename(fn, "{}.bak-{:.3f}.md".format(fn[:-3], srv_lastmod))
+            mdir, mfile = os.path.split(fp)
+            mfile2 = "{}.{:.3f}.md".format(mfile[:-3], srv_lastmod)
+            try:
+                os.mkdir(os.path.join(mdir, ".hist"))
+            except:
+                pass
+            os.rename(fp, os.path.join(mdir, ".hist", mfile2))
 
         p_field, _, p_data = next(self.parser.gen)
         if p_field != "body":
             raise Pebkac(400, "expected body, got {}".format(p_field))
 
-        with open(fn, "wb") as f:
+        with open(fp, "wb") as f:
             sz, sha512, _ = hashcopy(self.conn, p_data, f)
 
-        new_lastmod = os.stat(fsenc(fn)).st_mtime
+        new_lastmod = os.stat(fsenc(fp)).st_mtime
         new_lastmod3 = int(new_lastmod * 1000)
         sha512 = sha512[:56]
 
@@ -759,11 +829,20 @@ class HttpCli(object):
                 else:
                     upper = file_sz
 
-                if lower < 0 or lower >= file_sz or upper < 0 or upper > file_sz:
+                if upper > file_sz:
+                    upper = file_sz
+
+                if lower < 0 or lower >= upper:
                     raise Exception()
 
             except:
-                raise Pebkac(400, "invalid range requested: " + hrange)
+                err = "invalid range ({}), size={}".format(hrange, file_sz)
+                self.loud_reply(
+                    err,
+                    status=416,
+                    headers={"Content-Range": "bytes */{}".format(file_sz)},
+                )
+                return True
 
             status = 206
             self.out_headers["Content-Range"] = "bytes {}-{}/{}".format(
@@ -824,7 +903,7 @@ class HttpCli(object):
 
     def tx_md(self, fs_path):
         logmsg = "{:4} {} ".format("", self.req)
-        if "edit" in self.uparam:
+        if "edit2" in self.uparam:
             html_path = "web/mde.html"
             template = self.conn.tpl_mde
         else:
@@ -834,18 +913,26 @@ class HttpCli(object):
         html_path = os.path.join(E.mod, html_path)
 
         st = os.stat(fsenc(fs_path))
-        sz_md = st.st_size
+        # sz_md = st.st_size
         ts_md = st.st_mtime
 
         st = os.stat(fsenc(html_path))
         ts_html = st.st_mtime
 
+        # TODO dont load into memory ;_;
+        #   (trivial fix, count the &'s)
+        with open(fsenc(fs_path), "rb") as f:
+            md = f.read().replace(b"&", b"&amp;")
+            sz_md = len(md)
+
         file_ts = max(ts_md, ts_html)
         file_lastmod, do_send = self._chk_lastmod(file_ts)
         self.out_headers["Last-Modified"] = file_lastmod
+        self.out_headers["Cache-Control"] = "no-cache"
         status = 200 if do_send else 304
 
         targs = {
+            "edit": "edit" in self.uparam,
             "title": html_escape(self.vpath, quote=False),
             "lastmod": int(ts_md * 1000),
             "md": "",
@@ -858,9 +945,7 @@ class HttpCli(object):
             self.log(logmsg)
             return True
 
-        with open(fsenc(fs_path), "rb") as f:
-            md = f.read()
-
+        # TODO jinja2 can stream this right?
         targs["md"] = md.decode("utf-8", "replace")
         html = template.render(**targs).encode("utf-8")
         try:
@@ -909,12 +994,34 @@ class HttpCli(object):
         fsroot, vfs_ls, vfs_virt = vn.ls(rem, self.uname)
         vfs_ls.extend(vfs_virt.keys())
 
+        # check for old versions of files,
+        hist = {}  # [num-backups, most-recent, hist-path]
+        histdir = os.path.join(fsroot, ".hist")
+        ptn = re.compile(r"(.*)\.([0-9]+\.[0-9]{3})(\.[^\.]+)$")
+        try:
+            for hfn in os.listdir(histdir):
+                m = ptn.match(hfn)
+                if not m:
+                    continue
+
+                fn = m.group(1) + m.group(3)
+                n, ts, _ = hist.get(fn, [0, 0, ""])
+                hist[fn] = [n + 1, max(ts, float(m.group(2))), hfn]
+        except:
+            pass
+
+        # show dotfiles if permitted and requested
+        if not self.args.ed or "dots" not in self.uparam:
+            vfs_ls = exclude_dotfiles(vfs_ls)
+
         dirs = []
         files = []
-        for fn in exclude_dotfiles(vfs_ls):
+        for fn in vfs_ls:
+            base = ""
             href = fn
             if self.absolute_urls and vpath:
-                href = "/" + vpath + "/" + fn
+                base = "/" + vpath + "/"
+                href = base + fn
 
             if fn in vfs_virt:
                 fspath = vfs_virt[fn].realpath
@@ -931,6 +1038,10 @@ class HttpCli(object):
             if is_dir:
                 margin = "DIR"
                 href += "/"
+            elif fn in hist:
+                margin = '<a href="{}.hist/{}">#{}</a>'.format(
+                    base, html_escape(hist[fn][2], quote=True), hist[fn][0]
+                )
             else:
                 margin = "-"
 
@@ -951,6 +1062,17 @@ class HttpCli(object):
                 with open(fsenc(fn), "rb") as f:
                     logues[n] = f.read().decode("utf-8")
 
+        if False:
+            # this is a mistake
+            md = None
+            for fn in [x[2] for x in files]:
+                if fn.lower() == "readme.md":
+                    fn = os.path.join(abspath, fn)
+                    with open(fn, "rb") as f:
+                        md = f.read().decode("utf-8")
+
+                    break
+
         ts = ""
         # ts = "?{}".format(time.time())
 

copyparty/httpconn.py

@@ -86,7 +86,7 @@ class HttpConn(object):
                 self.s.send(b"HTTP/1.1 400 Bad Request\r\n\r\n" + err.encode("utf-8"))
                 return
 
-        if method not in [None, b"GET ", b"HEAD", b"POST"]:
+        if method not in [None, b"GET ", b"HEAD", b"POST", b"PUT ", b"OPTI"]:
             if self.sr:
                 self.log("\033[1;31mTODO: cannot do https in jython\033[0m")
                 return

copyparty/httpsrv.py

@@ -6,7 +6,7 @@ import time
 import socket
 import threading
 
-from .__init__ import E
+from .__init__ import E, MACOS
 from .httpconn import HttpConn
 from .authsrv import AuthSrv
 
@@ -75,11 +75,14 @@ class HttpSrv(object):
                 sck.shutdown(socket.SHUT_RDWR)
                 sck.close()
             except (OSError, socket.error) as ex:
-                self.log(
-                    "%s %s" % addr, "shut_rdwr err:\n  {}\n  {}".format(repr(sck), ex),
-                )
-                if ex.errno not in [10038, 107, 57, 9]:
+                if not MACOS:
+                    self.log(
+                        "%s %s" % addr,
+                        "shut_rdwr err:\n  {}\n  {}".format(repr(sck), ex),
+                    )
+                if ex.errno not in [10038, 10054, 107, 57, 9]:
                     # 10038 No longer considered a socket
+                    # 10054 Foribly closed by remote
                     #   107 Transport endpoint not connected
                     #    57 Socket is not connected
                     #     9 Bad file descriptor

copyparty/svchub.py

@@ -8,7 +8,7 @@ import threading
 from datetime import datetime, timedelta
 import calendar
 
-from .__init__ import PY2, WINDOWS, VT100
+from .__init__ import PY2, WINDOWS, MACOS, VT100
 from .tcpsrv import TcpSrv
 from .up2k import Up2k
 from .util import mp
@@ -111,6 +111,8 @@ class SvcHub(object):
                 return msg
             elif vmin < 3:
                 return msg
+        elif MACOS:
+            return "multiprocessing is wonky on mac osx;"
         else:
             msg = "need python 2.7 or 3.3+ for multiprocessing;"
             if not PY2 and vmin < 3:
@@ -127,13 +129,13 @@ class SvcHub(object):
         return None
 
     def check_mp_enable(self):
-        if self.args.j == 0:
-            self.log("root", "multiprocessing disabled by argument -j 0;")
+        if self.args.j == 1:
+            self.log("root", "multiprocessing disabled by argument -j 1;")
             return False
 
         if mp.cpu_count() <= 1:
             return False
-        
+
         try:
             # support vscode debugger (bonus: same behavior as on windows)
             mp.set_start_method("spawn", True)

copyparty/tcpsrv.py

@@ -24,7 +24,7 @@ class TcpSrv(object):
         ip = "127.0.0.1"
         eps = {ip: "local only"}
         if self.args.i != ip:
-            eps = self.detect_interfaces(self.args.i) or eps
+            eps = self.detect_interfaces(self.args.i) or {self.args.i: "external"}
 
         for ip, desc in sorted(eps.items(), key=lambda x: x[1]):
             self.log(

copyparty/up2k.py

@@ -13,7 +13,7 @@ import threading
 from copy import deepcopy
 
 from .__init__ import WINDOWS
-from .util import Pebkac, Queue, fsenc
+from .util import Pebkac, Queue, fsenc, sanitize_fn
 
 
 class Up2k(object):
@@ -48,6 +48,7 @@ class Up2k(object):
         self.r_hash = re.compile("^[0-9a-zA-Z_-]{43}$")
 
     def handle_json(self, cj):
+        cj["name"] = sanitize_fn(cj["name"])
         wark = self._get_wark(cj)
         now = time.time()
         with self.mutex:

copyparty/util.py

@@ -42,6 +42,7 @@ if WINDOWS and PY2:
 
 HTTPCODE = {
     200: "OK",
+    204: "No Content",
     206: "Partial Content",
     304: "Not Modified",
     400: "Bad Request",
@@ -49,6 +50,7 @@ HTTPCODE = {
     404: "Not Found",
     405: "Method Not Allowed",
     413: "Payload Too Large",
+    416: "Requested Range Not Satisfiable",
     422: "Unprocessable Entity",
     500: "Internal Server Error",
     501: "Not Implemented",
@@ -309,18 +311,7 @@ def get_boundary(headers):
 def read_header(sr):
     ret = b""
     while True:
-        if ret.endswith(b"\r\n\r\n"):
-            break
-        elif ret.endswith(b"\r\n\r"):
-            n = 1
-        elif ret.endswith(b"\r\n"):
-            n = 2
-        elif ret.endswith(b"\r"):
-            n = 3
-        else:
-            n = 4
-
-        buf = sr.recv(n)
+        buf = sr.recv(1024)
         if not buf:
             if not ret:
                 return None
@@ -332,11 +323,15 @@ def read_header(sr):
             )
 
         ret += buf
+        ofs = ret.find(b"\r\n\r\n")
+        if ofs < 0:
+            if len(ret) > 1024 * 64:
+                raise Pebkac(400, "header 2big")
+            else:
+                continue
 
-        if len(ret) > 1024 * 64:
-            raise Pebkac(400, "header 2big")
-
-    return ret[:-4].decode("utf-8", "surrogateescape").split("\r\n")
+        sr.unrecv(ret[ofs + 4 :])
+        return ret[:ofs].decode("utf-8", "surrogateescape").split("\r\n")
 
 
 def undot(path):
@@ -356,7 +351,30 @@ def undot(path):
 
 
 def sanitize_fn(fn):
-    return fn.replace("\\", "/").split("/")[-1].strip()
+    fn = fn.replace("\\", "/").split("/")[-1]
+
+    if WINDOWS:
+        for bad, good in [
+            ["<", "＜"],
+            [">", "＞"],
+            [":", "："],
+            ['"', "＂"],
+            ["/", "／"],
+            ["\\", "＼"],
+            ["|", "｜"],
+            ["?", "？"],
+            ["*", "＊"],
+        ]:
+            fn = fn.replace(bad, good)
+
+        bad = ["con", "prn", "aux", "nul"]
+        for n in range(1, 10):
+            bad += "com{0} lpt{0}".format(n).split(" ")
+
+        if fn.lower() in bad:
+            fn = "_" + fn
+
+    return fn.strip()
 
 
 def exclude_dotfiles(filepaths):
@@ -428,6 +446,15 @@ def read_socket(sr, total_size):
         yield buf
 
 
+def read_socket_unbounded(sr):
+    while True:
+        buf = sr.recv(32 * 1024)
+        if not buf:
+            return
+
+        yield buf
+
+
 def hashcopy(actor, fin, fout):
     u32_lim = int((2 ** 31) * 0.9)
     hashobj = hashlib.sha512()

copyparty/web/browser.css

@@ -68,7 +68,7 @@ a {
 }
 #files thead th:last-child {
 	background: #444;
-	border-radius: .7em 0 0 0;
+	border-radius: .7em .7em 0 0;
 }
 #files thead th:first-child {
 	background: #222;

copyparty/web/browser.html

@@ -33,7 +33,7 @@
             <tr>
                 <th></th>
                 <th>File Name</th>
-                <th>File Size</th>
+                <th sort="int">File Size</th>
                 <th>Date</th>
             </tr>
         </thead>

copyparty/web/browser.js

@@ -34,6 +34,7 @@ window.onerror = function (msg, url, lineNo, columnNo, error) {
 					esc(String(error[find[a]])).replace(/\n/g, '<br />\n'));
 	}
 	document.body.style.fontSize = '0.8em';
+	document.body.style.padding = '0 1em 1em 1em';
 	hcroak(html.join('\n'));
 };
 
@@ -78,6 +79,41 @@ function ev(e) {
 }
 
 
+function sortTable(table, col) {
+	var tb = table.tBodies[0], // use `<tbody>` to ignore `<thead>` and `<tfoot>` rows
+		th = table.tHead.rows[0].cells,
+		tr = Array.prototype.slice.call(tb.rows, 0),
+		i, reverse = th[col].className == 'sort1' ? -1 : 1;
+	for (var a = 0, thl = th.length; a < thl; a++)
+		th[a].className = '';
+	th[col].className = 'sort' + reverse;
+	var stype = th[col].getAttribute('sort');
+	tr = tr.sort(function (a, b) {
+		var v1 = a.cells[col].textContent.trim();
+		var v2 = b.cells[col].textContent.trim();
+		if (stype == 'int') {
+			v1 = parseInt(v1.replace(/,/g, ''));
+			v2 = parseInt(v2.replace(/,/g, ''));
+			return reverse * (v1 - v2);
+		}
+		return reverse * (v1.localeCompare(v2));
+	});
+	for (i = 0; i < tr.length; ++i) tb.appendChild(tr[i]);
+}
+function makeSortable(table) {
+	var th = table.tHead, i;
+	th && (th = th.rows[0]) && (th = th.cells);
+	if (th) i = th.length;
+	else return; // if no `<thead>` then do nothing
+	while (--i >= 0) (function (i) {
+		th[i].onclick = function () {
+			sortTable(table, i);
+		};
+	}(i));
+}
+makeSortable(o('files'));
+
+
 // extract songs + add play column
 var mp = (function () {
 	var tracks = [];
@@ -89,7 +125,6 @@ var mp = (function () {
 		'cover_url': ''
 	};
 	var re_audio = new RegExp('\.(opus|ogg|m4a|aac|mp3|wav|flac)$', 'i');
-	var re_cover = new RegExp('^(cover|folder|cd|front|back)\.(jpe?g|png|gif)$', 'i');
 
 	var trs = document.getElementById('files').getElementsByTagName('tbody')[0].getElementsByTagName('tr');
 	for (var a = 0, aa = trs.length; a < aa; a++) {
@@ -380,15 +415,6 @@ var vbar = (function () {
 		var x = e.clientX - rect.left;
 		var mul = x * 1.0 / rect.width;
 
-		/*
-		dbg(//Math.round(rect.width) + 'x' + Math.round(rect.height) + '+' +
-			//Math.round(rect.left) + '+' + Math.round(rect.top) + ', ' +
-			//Math.round(e.clientX) + 'x' + Math.round(e.clientY) + ', ' +
-			Math.round(mp.au.currentTime * 10) / 10 + ', ' +
-			Math.round(mp.au.duration * 10) / 10 + '*' +
-			Math.round(mul * 1000) / 1000);
-		*/
-
 		mp.au.currentTime = mp.au.duration * mul;
 
 		if (mp.au === mp.au_native)
@@ -449,8 +475,14 @@ function setclass(id, clas) {
 }
 
 
-var iOS = !!navigator.platform &&
-	/iPad|iPhone|iPod/.test(navigator.platform);
+var need_ogv = true;
+try {
+	need_ogv = new Audio().canPlayType('audio/ogg; codecs=opus') !== 'probably';
+
+	if (/ Edge\//.exec(navigator.userAgent + ''))
+		need_ogv = true;
+}
+catch (ex) { }
 
 
 // plays the tid'th audio file on the page
@@ -473,7 +505,7 @@ function play(tid, call_depth) {
 	var hack_attempt_play = true;
 
 	var url = mp.tracks[tid];
-	if (iOS && /\.(ogg|opus)$/i.test(url)) {
+	if (need_ogv && /\.(ogg|opus)$/i.test(url)) {
 		if (mp.au_ogvjs) {
 			mp.au = mp.au_ogvjs;
 		}
@@ -560,7 +592,6 @@ function evau_error(e) {
 	err += '\n\nFile: «' + decodeURIComponent(eplaya.src.split('/').slice(-1)[0]) + '»';
 
 	alert(err);
-	play(eplaya.tid + 1);
 }
 
 
@@ -579,7 +610,7 @@ function show_modal(html) {
 function unblocked() {
 	var dom = o('blocked');
 	if (dom)
-		dom.remove();
+		dom.parentNode.removeChild(dom);
 }
 
 

copyparty/web/md.css

@@ -1,13 +1,19 @@
+@font-face {
+	font-family: 'scp';
+	src: local('Source Code Pro Regular'), local('SourceCodePro-Regular'), url(/.cpr/deps/scp.woff2) format('woff2');
+}
 html, body {
 	color: #333;
 	background: #eee;
 	font-family: sans-serif;
 	line-height: 1.5em;
 }
+#mtw {
+	display: none;
+}
 #mw {
-	width: 48.5em;
 	margin: 0 auto;
-	margin-bottom: 6em;
+	padding: 0 1.5em;
 }
 pre, code, a {
 	color: #480;
@@ -21,7 +27,7 @@ code {
 	font-size: .96em;
 }
 pre, code {
-	font-family: monospace, monospace;
+	font-family: 'scp', monospace, monospace;
 	white-space: pre-wrap;
 	word-break: break-all;
 }
@@ -41,7 +47,7 @@ pre code {
 pre code:last-child {
 	border-bottom: none;
 }
-pre code:before {
+pre code::before {
 	content: counter(precode);
 	-webkit-user-select: none;
 	display: inline-block;
@@ -76,27 +82,31 @@ h2 {
 	padding-left: .4em;
 	margin-top: 3em;
 }
+h3 {
+	border-bottom: .1em solid #999;
+}
 h1 a, h3 a, h5 a,
 h2 a, h4 a, h6 a {
 	color: inherit;
+	display: block;
 	background: none;
 	border: none;
 	padding: 0;
 	margin: 0;
 }
-#m ul,
-#m ol {
+#mp ul,
+#mp ol {
 	border-left: .3em solid #ddd;
 }
 #m>ul,
 #m>ol {
 	border-color: #bbb;
 }
-#m ul>li {
+#mp ul>li {
 	list-style-type: disc;
 }
-#m ul>li,
-#m ol>li {
+#mp ul>li,
+#mp ol>li {
 	margin: .7em 0;
 }
 p>em,
@@ -116,8 +126,9 @@ small {
 	opacity: .8;
 }
 #toc {
-	width: 48.5em;
-	margin: 0 auto;
+	margin: 0 1em;
+	-ms-scroll-chaining: none;
+	overscroll-behavior-y: none;
 }
 #toc ul {
 	padding-left: 1em;
@@ -162,14 +173,12 @@ small {
 }
 table {
 	border-collapse: collapse;
+	margin: 1em 0;
 }
-td {
+th, td {
 	padding: .2em .5em;
 	border: .12em solid #aaa;
 }
-th {
-	border: .12em solid #aaa;
-}
 blink {
 	animation: blinker .7s cubic-bezier(.9, 0, .1, 1) infinite;
 }
@@ -181,10 +190,26 @@ blink {
 		opacity: 1;
 	}
 }
+
 @media screen {
 	html, body {
 		margin: 0;
 		padding: 0;
+		outline: 0;
+		border: none;
+		width: 100%;
+		height: 100%;
+	}
+	#mw {
+		margin: 0 auto;
+		right: 0;
+	}
+	#mp {
+		max-width: 52em;
+		margin-bottom: 6em;
+		word-break: break-word;
+		overflow-wrap: break-word;
+		word-wrap: break-word; /*ie*/
 	}
 	a {
 		color: #fff;
@@ -212,15 +237,17 @@ blink {
 		padding: .5em 0;
 	}
 	#mn {
-		font-weight: normal;
 		padding: 1.3em 0 .7em 1em;
-		font-size: 1.4em;
+		border-bottom: 1px solid #ccc;
+		background: #eee;
+		z-index: 10;
+		width: calc(100% - 1em);
 	}
 	#mn a {
 		color: #444;
 		background: none;
 		margin: 0 0 0 -.2em;
-		padding: 0 0 0 .4em;
+		padding: .3em 0 .3em .4em;
 		text-decoration: none;
 		border: none;
 		/* ie: */
@@ -233,14 +260,14 @@ blink {
 	#mn a:last-child {
 		padding-right: .5em;
 	}
-	#mn a:not(:last-child):after {
+	#mn a:not(:last-child)::after {
 		content: '';
 		width: 1.05em;
 		height: 1.05em;
 		margin: -.2em .3em -.2em -.4em;
 		display: inline-block;
-		border: 1px solid rgba(0,0,0,0.3);
-		border-width: .05em .05em 0 0;
+		border: 1px solid rgba(0,0,0,0.2);
+		border-width: .2em .2em 0 0;
 		transform: rotate(45deg);
 	}
 	#mn a:hover {
@@ -248,7 +275,19 @@ blink {
 		text-decoration: underline;
 	}
 	#mh {
-		margin: 0 0 1.5em 0;
+		padding: .4em 1em;
+		position: relative;
+		width: 100%;
+		width: calc(100% - 3em);
+		background: #eee;
+		z-index: 9;
+		top: 0;
+	}
+	#mh a {
+		color: #444;
+		background: none;
+		text-decoration: underline;
+		border: none;
 	}
 
 
@@ -270,13 +309,12 @@ blink {
 	html.dark #toc li {
 		border-width: 0;
 	}
-	html.dark #m a,
-	html.dark #mh a {
+	html.dark #mp a {
 		background: #057;
 	}
-	html.dark #m h1 a, html.dark #m h4 a,
-	html.dark #m h2 a, html.dark #m h5 a,
-	html.dark #m h3 a, html.dark #m h6 a {
+	html.dark #mp h1 a, html.dark #mp h4 a,
+	html.dark #mp h2 a, html.dark #mp h5 a,
+	html.dark #mp h3 a, html.dark #mp h6 a {
 		color: inherit;
 		background: none;
 	}
@@ -286,8 +324,8 @@ blink {
 		background: #1a1a1a;
 		border: .07em solid #333;
 	}
-	html.dark #m ul,
-	html.dark #m ol {
+	html.dark #mp ul,
+	html.dark #mp ol {
 		border-color: #444;
 	}
 	html.dark #m>ul,
@@ -316,32 +354,50 @@ blink {
 		background: #282828;
 		border: .07em dashed #444;
 	}
-	html.dark #mn a:not(:last-child):after {
+	html.dark #mn a:not(:last-child)::after {
 		border-color: rgba(255,255,255,0.3);
 	}
 	html.dark #mn a {
 		color: #ccc;
 	}
+	html.dark #mn {
+		border-bottom: 1px solid #333;
+	}
+	html.dark #mn,
+	html.dark #mh {
+		background: #222;
+	}
+	html.dark #mh a {
+		color: #ccc;
+		background: none;
+	}
 }
-@media screen and (min-width: 64em) {
+
+@media screen and (min-width: 66em) {
 	#mw {
-		margin-left: 14em;
-		margin-left: calc(100% - 50em);
+		position: fixed;
+		overflow-y: auto;
+		left: 14em;
+		left: calc(100% - 55em);
+		max-width: none;
+		bottom: 0;
+		scrollbar-color: #eb0 #f7f7f7;
 	}
 	#toc {
 		width: 13em;
-		width: calc(100% - 52.3em);
+		width: calc(100% - 55.3em);
+		max-width: 30em;
 		background: #eee;
 		position: fixed;
+		overflow-y: auto;
 		top: 0;
 		left: 0;
-		height: 100%;
-		overflow-y: auto;
+		bottom: 0;
 		padding: 0;
 		margin: 0;
-		box-shadow: 0 0 1em #ccc;
 		scrollbar-color: #eb0 #f7f7f7;
-		xscrollbar-width: thin;
+		box-shadow: 0 0 1em rgba(0,0,0,0.1);
+		border-top: 1px solid #d7d7d7;
 	}
 	#toc li {
 		border-left: .3em solid #ccc;
@@ -361,30 +417,133 @@ blink {
 	
 	html.dark #toc {
 		background: #282828;
+		border-top: 1px solid #2c2c2c;
 		box-shadow: 0 0 1em #181818;
+	}
+	html.dark #toc,
+	html.dark #mw {
 		scrollbar-color: #b80 #282828;
 	}
+	html.dark #toc::-webkit-scrollbar-track {
+		background: #282828;
+	}
+	html.dark #toc::-webkit-scrollbar {
+		background: #282828;
+		width: .8em;
+	}
+	html.dark #toc::-webkit-scrollbar-thumb {
+		background: #b80;
+	}
 }
-@media screen and (min-width: 84em) {
+@media screen and (min-width: 85.5em) {
 	#toc { width: 30em }
-	#mw { margin-left: 32em }
+	#mw { left: 30.5em }
 }
 @media print {
+	@page {
+		size: A4;
+		padding: 0;
+		margin: .5in .6in;
+		mso-header-margin: .6in;
+		mso-footer-margin: .6in;
+		mso-paper-source: 0;
+	}
 	a {
 		color: #079;
 		text-decoration: none;
 		border-bottom: .07em solid #4ac;
 		padding: 0 .3em;
 	}
-	#toc {
-		margin: 0 !important;
-	}
 	#toc>ul {
 		border-left: .1em solid #84c4dd;
 	}
 	#mn, #mh {
 		display: none;
 	}
+	html, body, #toc, #mw {
+		margin: 0 !important;
+		word-break: break-word;
+		width: 52em;
+	}
+	#toc {
+		margin-left: 1em !important;
+	}
+	#toc a {
+		color: #000 !important;
+	}
+	#toc a::after {
+		/* hopefully supported by browsers eventually */
+		content: leader('.') target-counter(attr(href), page);
+	}
+	a[ctr]::before {
+		content: attr(ctr) '. ';
+	}
+	h1 {
+		margin: 2em 0;
+	}
+	h2 {
+		margin: 2em 0 0 0;
+	}
+	h1, h2, h3 {
+		page-break-inside: avoid;
+	}
+	h1::after,
+	h2::after,
+	h3::after {
+		content: 'orz';
+		color: transparent;
+		display: block;
+		line-height: 1em;
+		padding: 4em 0 0 0;
+		margin: 0 0 -5em 0;
+	}
+	p {
+		page-break-inside: avoid;
+	}
+	table {
+		page-break-inside: auto;
+	}
+	tr {
+		page-break-inside: avoid;
+		page-break-after: auto;
+	}
+	thead {
+		display: table-header-group;
+	}
+	tfoot {
+		display: table-footer-group;
+	}
+	#mp a.vis::after {
+		content: ' (' attr(href) ')';
+		border-bottom: 1px solid #bbb;
+		color: #444;
+	}
+	blockquote {
+		border-color: #555;
+	}
+	code {
+		border-color: #bbb;
+	}
+	pre, pre code {
+		border-color: #999;
+	}
+	pre code::before {
+		color: #058;
+	}
+
+
+	
+	html.dark a {
+		color: #000;
+	}
+	html.dark pre,
+	html.dark code {
+		color: #240;
+	}
+	html.dark p>em,
+	html.dark li>em {
+		color: #940;
+	}
 }
 
 /*

copyparty/web/md.html

@@ -4,32 +4,122 @@
 	<meta http-equiv="X-UA-Compatible" content="IE=edge">
 	<meta name="viewport" content="width=device-width, initial-scale=0.7">
 	<link href="/.cpr/md.css" rel="stylesheet">
+	{%- if edit %}
+	<link href="/.cpr/md2.css" rel="stylesheet">
+	{%- endif %}
 </head>
 <body>
-	<div id="mn"></div>
+	<div id="mn">navbar</div>
+	<div id="mh">
+		<a id="lightswitch" href="#">go dark</a>
+		<a id="navtoggle" href="#">hide nav</a>
+		{%- if edit %}
+			<a id="save" href="?edit">save</a>
+			<a id="sbs" href="#">sbs</a>
+			<a id="nsbs" href="#">editor</a>
+			<a id="help" href="#">help</a>
+		{%- else %}
+			<a href="?edit">edit (basic)</a>
+			<a href="?edit2">edit (fancy)</a>
+			<a href="?raw">view raw</a>
+		{%- endif %}
+	</div>
 	<div id="toc"></div>
+	<div id="mtw">
+		<textarea id="mt" autocomplete="off">{{ md }}</textarea>
+	</div>
 	<div id="mw">
-		<div id="mh">
-			<a id="lightswitch" href="#">go dark</a> //
-			<a id="edit" href="?edit">edit this</a>
-		</div>
 		<div id="ml">
 			<div style="text-align:center;margin:5em 0">
 				<div style="font-size:2em;margin:1em 0">Loading</div>
 				if you're still reading this, check that javascript is allowed
 			</div>
 		</div>
-		<div id="m">
-			<textarea id="mt" style="display:none">{{ md }}</textarea>
-		</div>
+		<div id="mp"></div>
 	</div>
+	
+	{%- if edit %}
+	<div id="helpbox">
+		<textarea autocomplete="off">
+
+write markdown (most html is 🙆 too)
+
+## hotkey list
+* `Ctrl-S` to save
+* `Ctrl-H` / `Ctrl-Shift-H` to create a header
+* `TAB` / `Shift-TAB` to indent/dedent a selection
+
+## toolbar
+1. toggle dark mode
+2. show/hide navigation bar
+3. save changes on server
+4. side-by-side editing
+5. toggle editor/preview
+6. this thing :^)
+
+## markdown
+|||
+|--|--|
+|`**bold**`|**bold**|
+|`_italic_`|_italic_|
+|`~~strike~~`|~~strike~~|
+|`` `code` ``|`code`|
+|`[](#hotkey-list)`|[](#hotkey-list)|
+|`[](/foo/bar.md#header)`|[](/foo/bar.md#header)|
+|`<blink>💯</blink>`|<blink>💯</blink>|
+
+## tables
+    |left-aligned|centered|right-aligned
+    | ---------- | :----: | ----------:
+    |one         |two     |three
+
+|left-aligned|centered|right-aligned
+| ---------- | :----: | ----------:
+|one         |two     |three
+
+## lists
+	* one
+	* two
+	1. one
+	1. two
+* one
+* two
+1. one
+1. two
+
+## headers
+	# level 1
+	## level 2
+	### level 3
+
+## quote
+	> hello
+> hello
+
+## codeblock
+		four spaces (no tab pls)
+
+## code in lists
+	* foo
+	  bar
+          six spaces total
+* foo
+  bar
+      six spaces total
+.
+		</textarea>
+	</div>
+	{%- endif %}
+	
 	<script>
 
 var link_md_as_html = false;  // TODO (does nothing)
+var last_modified = {{ lastmod }};
 
 (function () {
     var btn = document.getElementById("lightswitch");
-    var toggle = function () {
+    var toggle = function (e) {
+		if (e) e.preventDefault();
         var dark = !document.documentElement.getAttribute("class");
         document.documentElement.setAttribute("class", dark ? "dark" : "");
         btn.innerHTML = "go " + (dark ? "light" : "dark");
@@ -41,7 +131,17 @@ var link_md_as_html = false;  // TODO (does nothing)
 		toggle();
 })();
 
+if (!String.startsWith) {
+	String.prototype.startsWith = function(s, i) {
+		i = i>0 ? i|0 : 0;
+		return this.substring(i, i + s.length) === s;
+	};
+}
+
 	</script>
 	<script src="/.cpr/deps/marked.full.js"></script>
 	<script src="/.cpr/md.js"></script>
+	{%- if edit %}
+		<script src="/.cpr/md2.js"></script>
+	{%- endif %}
 </body></html>

copyparty/web/md.js

@@ -1,17 +1,54 @@
-/*var conv = new showdown.Converter();
-conv.setFlavor('github');
-conv.setOption('tasklists', 0);
-var mhtml = conv.makeHtml(dom_md.value);
-*/
-
 var dom_toc = document.getElementById('toc');
 var dom_wrap = document.getElementById('mw');
-var dom_head = document.getElementById('mh');
+var dom_hbar = document.getElementById('mh');
 var dom_nav = document.getElementById('mn');
-var dom_doc = document.getElementById('m');
-var dom_md = document.getElementById('mt');
+var dom_pre = document.getElementById('mp');
+var dom_src = document.getElementById('mt');
+var dom_navtgl = document.getElementById('navtoggle');
 
-// add toolbar buttons
+
+// chrome 49 needs this
+var chromedbg = function () { console.log(arguments); }
+
+// null-logger
+var dbg = function () { };
+
+// replace dbg with the real deal here or in the console:
+// dbg = chromedbg
+// dbg = console.log
+
+
+function hesc(txt) {
+    return txt.replace(/&/g, "&amp;").replace(/</g, "&lt;").replace(/>/g, "&gt;");
+}
+
+
+function cls(dom, name, add) {
+    var re = new RegExp('(^| )' + name + '( |$)');
+    var lst = (dom.getAttribute('class') + '').replace(re, "$1$2").replace(/  /, "");
+    dom.setAttribute('class', lst + (add ? ' ' + name : ''));
+}
+
+
+function static(obj) {
+    return JSON.parse(JSON.stringify(obj));
+}
+
+
+// dodge browser issues
+(function () {
+    var ua = navigator.userAgent;
+    if (ua.indexOf(') Gecko/') !== -1 && /Linux| Mac /.exec(ua)) {
+        // necessary on ff-68.7 at least
+        var s = document.createElement('style');
+        s.innerHTML = '@page { margin: .5in .6in .8in .6in; }';
+        console.log(s.innerHTML);
+        document.head.appendChild(s);
+    }
+})();
+
+
+// add navbar
 (function () {
     var n = document.location + '';
     n = n.substr(n.indexOf('//') + 2).split('?')[0].split('/');
@@ -22,25 +59,123 @@ var dom_md = document.getElementById('mt');
         if (a > 0)
             loc.push(n[a]);
 
-        nav.push('<a href="/' + loc.join('/') + '">' + n[a] + '</a>');
+        var dec = hesc(decodeURIComponent(n[a]));
+
+        nav.push('<a href="/' + loc.join('/') + '">' + dec + '</a>');
     }
     dom_nav.innerHTML = nav.join('');
 })();
 
-function convert_markdown(md_text) {
+
+// faster than replacing the entire html (chrome 1.8x, firefox 1.6x)
+function copydom(src, dst, lv) {
+    var sc = src.childNodes,
+        dc = dst.childNodes;
+
+    if (sc.length !== dc.length) {
+        dbg("replace L%d (%d/%d) |%d|",
+            lv, sc.length, dc.length, src.innerHTML.length);
+
+        dst.innerHTML = src.innerHTML;
+        return;
+    }
+
+    var rpl = [];
+    for (var a = sc.length - 1; a >= 0; a--) {
+        var st = sc[a].tagName,
+            dt = dc[a].tagName;
+
+        if (st !== dt) {
+            dbg("replace L%d (%d/%d) type %s/%s", lv, a, sc.length, st, dt);
+            rpl.push(a);
+            continue;
+        }
+
+        var sa = sc[a].attributes || [],
+            da = dc[a].attributes || [];
+
+        if (sa.length !== da.length) {
+            dbg("replace L%d (%d/%d) attr# %d/%d",
+                lv, a, sc.length, sa.length, da.length);
+
+            rpl.push(a);
+            continue;
+        }
+
+        var dirty = false;
+        for (var b = sa.length - 1; b >= 0; b--) {
+            var name = sa[b].name,
+                sv = sa[b].value,
+                dv = dc[a].getAttribute(name);
+
+            if (name == "data-ln" && sv !== dv) {
+                dc[a].setAttribute(name, sv);
+                continue;
+            }
+
+            if (sv !== dv) {
+                dbg("replace L%d (%d/%d) attr %s [%s] [%s]",
+                    lv, a, sc.length, name, sv, dv);
+
+                dirty = true;
+                break;
+            }
+        }
+        if (dirty)
+            rpl.push(a);
+    }
+
+    // TODO pure guessing
+    if (rpl.length > sc.length / 3) {
+        dbg("replace L%d fully, %s (%d/%d) |%d|",
+            lv, rpl.length, sc.length, src.innerHTML.length);
+
+        dst.innerHTML = src.innerHTML;
+        return;
+    }
+
+    // repl is reversed; build top-down
+    var nbytes = 0;
+    for (var a = rpl.length - 1; a >= 0; a--) {
+        var html = sc[rpl[a]].outerHTML;
+        dc[rpl[a]].outerHTML = html;
+        nbytes += html.length;
+    }
+    if (nbytes > 0)
+        dbg("replaced %d bytes L%d", nbytes, lv);
+
+    for (var a = 0; a < sc.length; a++)
+        copydom(sc[a], dc[a], lv + 1);
+
+    if (src.innerHTML !== dst.innerHTML) {
+        dbg("setting %d bytes L%d", src.innerHTML.length, lv);
+        dst.innerHTML = src.innerHTML;
+    }
+}
+
+
+function convert_markdown(md_text, dest_dom) {
     marked.setOptions({
         //headerPrefix: 'h-',
         breaks: true,
         gfm: true
     });
-    var html = marked(md_text);
-    dom_doc.innerHTML = html;
+    var md_html = marked(md_text);
+    var md_dom = new DOMParser().parseFromString(md_html, "text/html").body;
 
-    var loader = document.getElementById('ml');
-    loader.parentNode.removeChild(loader);
+    var nodes = md_dom.getElementsByTagName('a');
+    for (var a = nodes.length - 1; a >= 0; a--) {
+        var href = nodes[a].getAttribute('href');
+        var txt = nodes[a].textContent;
+
+        if (!txt)
+            nodes[a].textContent = href;
+        else if (href !== txt)
+            nodes[a].setAttribute('class', 'vis');
+    }
 
     // todo-lists (should probably be a marked extension)
-    var nodes = dom_doc.getElementsByTagName('input');
+    nodes = md_dom.getElementsByTagName('input');
     for (var a = nodes.length - 1; a >= 0; a--) {
         var dom_box = nodes[a];
         if (dom_box.getAttribute('type') !== 'checkbox')
@@ -59,34 +194,77 @@ function convert_markdown(md_text) {
             '<span class="todo_' + clas + '">' + char + '</span>' +
             html.substr(html.indexOf('>') + 1);
     }
+
+    // separate <code> for each line in <pre>
+    var nodes = md_dom.getElementsByTagName('pre');
+    for (var a = nodes.length - 1; a >= 0; a--) {
+        var el = nodes[a];
+
+        var is_precode =
+            el.tagName == 'PRE' &&
+            el.childNodes.length === 1 &&
+            el.childNodes[0].tagName == 'CODE';
+
+        if (!is_precode)
+            continue;
+
+        var nline = parseInt(el.getAttribute('data-ln')) + 1;
+        var lines = el.innerHTML.replace(/\r?\n<\/code>$/i, '</code>').split(/\r?\n/g);
+        for (var b = 0; b < lines.length - 1; b++)
+            lines[b] += '</code>\n<code data-ln="' + (nline + b) + '">';
+
+        el.innerHTML = lines.join('');
+    }
+
+    // self-link headers
+    var id_seen = {},
+        dyn = md_dom.getElementsByTagName('*');
+
+    nodes = [];
+    for (var a = 0, aa = dyn.length; a < aa; a++)
+        if (/^[Hh]([1-6])/.exec(dyn[a].tagName) !== null)
+            nodes.push(dyn[a]);
+
+    for (var a = 0; a < nodes.length; a++) {
+        el = nodes[a];
+        var id = el.getAttribute('id'),
+            orig_id = id;
+
+        if (id_seen[id]) {
+            for (var n = 1; n < 4096; n++) {
+                id = orig_id + '-' + n;
+                if (!id_seen[id])
+                    break;
+            }
+            el.setAttribute('id', id);
+        }
+        id_seen[id] = 1;
+        el.innerHTML = '<a href="#' + id + '">' + el.innerHTML + '</a>';
+    }
+
+    copydom(md_dom, dest_dom, 0);
 }
 
+
 function init_toc() {
+    var loader = document.getElementById('ml');
+    loader.parentNode.removeChild(loader);
+
     var anchors = [];  // list of toc entries, complex objects
     var anchor = null; // current toc node
-    var id_seen = {};  // taken IDs
     var html = [];     // generated toc html
     var lv = 0;        // current indentation level in the toc html
-    var re = new RegExp('^[Hh]([1-3])');
+    var ctr = [0, 0, 0, 0, 0, 0];
 
-    var manip_nodes_dyn = dom_doc.getElementsByTagName('*');
+    var manip_nodes_dyn = dom_pre.getElementsByTagName('*');
     var manip_nodes = [];
     for (var a = 0, aa = manip_nodes_dyn.length; a < aa; a++)
         manip_nodes.push(manip_nodes_dyn[a]);
 
     for (var a = 0, aa = manip_nodes.length; a < aa; a++) {
         var elm = manip_nodes[a];
-        var m = re.exec(elm.tagName);
-
-        var is_header =
-            m !== null;
-
-        var is_precode =
-            !is_header &&
-            elm.tagName == 'PRE' &&
-            elm.childNodes.length === 1 &&
-            elm.childNodes[0].tagName == 'CODE';
-
+        var m = /^[Hh]([1-6])/.exec(elm.tagName);
+        var is_header = m !== null;
         if (is_header) {
             var nlv = m[1];
             while (lv < nlv) {
@@ -97,24 +275,13 @@ function init_toc() {
                 html.push('</ul>');
                 lv--;
             }
+            ctr[lv - 1]++;
+            for (var b = lv; b < 6; b++)
+                ctr[b] = 0;
 
-            var orig_id = elm.getAttribute('id');
-            var id = orig_id;
-            if (id_seen[id]) {
-                for (var n = 1; n < 4096; n++) {
-                    id = orig_id + '-' + n;
-                    if (!id_seen[id])
-                        break;
-                }
-                elm.setAttribute('id', id);
-            }
-            id_seen[id] = 1;
+            elm.childNodes[0].setAttribute('ctr', ctr.slice(0, lv).join('.'));
 
-            var ahref = '<a href="#' + id + '">' +
-                elm.innerHTML + '</a>';
-
-            html.push('<li>' + ahref + '</li>');
-            elm.innerHTML = ahref;
+            html.push('<li>' + elm.innerHTML + '</li>');
 
             if (anchor != null)
                 anchors.push(anchor);
@@ -125,17 +292,6 @@ function init_toc() {
                 y: null
             };
         }
-        else if (is_precode) {
-            // not actually toc-related (sorry),
-            // split <pre><code /></pre> into one <code> per line
-            var nline = parseInt(elm.getAttribute('data-ln')) + 1;
-            var lines = elm.innerHTML.replace(/\r?\n<\/code>$/i, '</code>').split(/\r?\n/g);
-            for (var b = 0; b < lines.length - 1; b++)
-                lines[b] += '</code>\n<code data-ln="' + (nline + b) + '">';
-
-            elm.innerHTML = lines.join('');
-        }
-
         if (!is_header && anchor)
             anchor.kids.push(elm);
     }
@@ -207,41 +363,47 @@ function init_toc() {
 
 
 // "main" :p
-convert_markdown(dom_md.value);
+convert_markdown(dom_src.value, dom_pre);
 var toc = init_toc();
 
 
 // scroll handler
-(function () {
-    var timer_active = false;
-    var final = null;
+var redraw = (function () {
+    var sbs = false;
+    function onresize() {
+        sbs = window.matchMedia('(min-width: 64em)').matches;
+        var y = (dom_hbar.offsetTop + dom_hbar.offsetHeight) + 'px';
+        if (sbs) {
+            dom_toc.style.top = y;
+            dom_wrap.style.top = y;
+            dom_toc.style.marginTop = '0';
+        }
+        onscroll();
+    }
 
     function onscroll() {
-        clearTimeout(final);
-        timer_active = false;
         toc.refresh();
-
-        var y = 0;
-        if (window.matchMedia('(min-width: 64em)').matches)
-            y = parseInt(dom_nav.offsetHeight) - window.scrollY;
-
-        dom_toc.style.marginTop = y < 0 ? 0 : y + "px";
     }
-    onscroll();
 
-    function ev_onscroll() {
-        // long timeout: scroll ended
-        clearTimeout(final);
-        final = setTimeout(onscroll, 100);
+    window.onresize = onresize;
+    window.onscroll = onscroll;
+    dom_wrap.onscroll = onscroll;
 
-        // short timeout: continuous updates
-        if (timer_active)
-            return;
-
-        timer_active = true;
-        setTimeout(onscroll, 10);
-    };
-
-    window.onscroll = ev_onscroll;
-    window.onresize = ev_onscroll;
+    onresize();
+    return onresize;
 })();
+
+
+dom_navtgl.onclick = function () {
+    var hidden = dom_navtgl.innerHTML == 'hide nav';
+    dom_navtgl.innerHTML = hidden ? 'show nav' : 'hide nav';
+    dom_nav.style.display = hidden ? 'none' : 'block';
+
+    if (window.localStorage)
+        localStorage.setItem('hidenav', hidden ? 1 : 0);
+
+    redraw();
+};
+
+if (window.localStorage && localStorage.getItem('hidenav') == 1)
+    dom_navtgl.onclick();

copyparty/web/md2.css

@@ -0,0 +1,108 @@
+#toc {
+    display: none;
+}
+#mtw {
+    display: block;
+    position: fixed;
+    left: .5em;
+    bottom: 0;
+    width: calc(100% - 56em);
+}
+#mw {
+    left: calc(100% - 55em);
+    overflow-y: auto;
+    position: fixed;
+    bottom: 0;
+}
+
+
+/* single-screen */
+#mtw.preview,
+#mw.editor {
+    opacity: 0;
+    z-index: 1;
+}
+#mw.preview,
+#mtw.editor {
+    z-index: 5;
+}
+#mtw.single,
+#mw.single {
+    margin: 0;
+    left: 1em;
+    left: max(1em, calc((100% - 56em) / 2));
+}
+#mtw.single {
+    width: 55em;
+    width: min(55em, calc(100% - 2em));
+}
+
+
+#mp {
+    position: relative;
+}
+#mt, #mtr {
+    width: 100%;
+    height: calc(100% - 1px);
+    color: #444;
+    background: #f7f7f7;
+    border: 1px solid #999;
+    outline: none;
+    padding: 0;
+    margin: 0;
+    font-family: 'consolas', monospace, monospace;
+    white-space: pre-wrap;
+    word-break: break-word;
+    overflow-wrap: break-word;
+    word-wrap: break-word; /*ie*/
+    overflow-y: scroll;
+    line-height: 1.3em;
+    font-size: .9em;
+    position: relative;
+    scrollbar-color: #eb0 #f7f7f7;
+}
+html.dark #mt {
+    color: #eee;
+    background: #222;
+    border: 1px solid #777;
+    scrollbar-color: #b80 #282828;
+}
+#mtr {
+    position: absolute;
+    top: 0;
+    left: 0;
+}
+#save.force-save {
+    color: #400;
+    background: #f97;
+    border-radius: .15em;
+}
+#save.disabled {
+    opacity: .4;
+}
+#helpbox {
+    display: none;
+    position: fixed;
+    background: #f7f7f7;
+    box-shadow: 0 .5em 2em #777;
+    border-radius: .4em;
+    padding: 2em;
+    top: 4em;
+    overflow-y: auto;
+    height: calc(100% - 12em);
+    left: calc(50% - 15em);
+    right: 0;
+    width: 30em;
+    z-index: 9001;
+}
+#helpclose {
+    display: block;
+}
+html.dark #helpbox {
+    background: #222;
+    box-shadow: 0 .5em 2em #444;
+    border: 1px solid #079;
+    border-width: 1px 0;
+}
+
+# mt {opacity: .5;top:1px}

copyparty/web/md2.js

@@ -0,0 +1,756 @@
+// server state
+var server_md = dom_src.value;
+
+
+// dom nodes
+var dom_swrap = document.getElementById('mtw');
+var dom_sbs = document.getElementById('sbs');
+var dom_nsbs = document.getElementById('nsbs');
+var dom_ref = (function () {
+    var d = document.createElement('div');
+    d.setAttribute('id', 'mtr');
+    dom_swrap.appendChild(d);
+    d = document.getElementById('mtr');
+    // hide behind the textarea (offsetTop is not computed if display:none)
+    dom_src.style.zIndex = '4';
+    d.style.zIndex = '3';
+    return d;
+})();
+
+
+// line->scrollpos maps
+function genmapq(dom, query) {
+    var ret = [];
+    var last_y = -1;
+    var parent_y = 0;
+    var parent_n = null;
+    var nodes = dom.querySelectorAll(query);
+    for (var a = 0; a < nodes.length; a++) {
+        var n = nodes[a];
+        var ln = parseInt(n.getAttribute('data-ln'));
+        if (ln in ret)
+            continue;
+
+        var y = 0;
+        var par = n.offsetParent;
+        if (par && par != parent_n) {
+            while (par && par != dom) {
+                y += par.offsetTop;
+                par = par.offsetParent;
+            }
+            if (par != dom)
+                continue;
+
+            parent_y = y;
+            parent_n = n.offsetParent;
+        }
+        while (ln > ret.length)
+            ret.push(null);
+
+        y = parent_y + n.offsetTop;
+        if (y <= last_y)
+            //console.log('awawa');
+            continue;
+
+        //console.log('%d  %d  (%d+%d)', a, y, parent_y, n.offsetTop);
+        ret.push(y);
+        last_y = y;
+    }
+    return ret;
+}
+var map_src = [];
+var map_pre = [];
+function genmap(dom, oldmap) {
+    var find = nlines;
+    while (oldmap && find --> 0) {
+        var tmap = genmapq(dom, '*[data-ln="' + find + '"]');
+        if (!tmap || !tmap.length)
+            continue;
+
+        var cy = tmap[find];
+        var oy = parseInt(oldmap[find]);
+        if (cy + 24 > oy && cy - 24 < oy)
+            return oldmap;
+
+        console.log('map regen', dom.getAttribute('id'), find, oy, cy, oy - cy);
+        break;
+    }
+    return genmapq(dom, '*[data-ln]');
+}
+
+
+// input handler
+var action_stack = null;
+var nlines = 0;
+var draw_md = (function () {
+    var delay = 1;
+    function draw_md() {
+        var t0 = new Date().getTime();
+        var src = dom_src.value;
+        convert_markdown(src, dom_pre);
+
+        var lines = hesc(src).replace(/\r/g, "").split('\n');
+        nlines = lines.length;
+        var html = [];
+        for (var a = 0; a < lines.length; a++)
+            html.push('<span data-ln="' + (a + 1) + '">' + lines[a] + "</span>");
+
+        dom_ref.innerHTML = html.join('\n');
+        map_src = genmap(dom_ref, map_src);
+        map_pre = genmap(dom_pre, map_pre);
+
+        cls(document.getElementById('save'), 'disabled', src == server_md);
+
+        var t1 = new Date().getTime();
+        delay = t1 - t0 > 100 ? 25 : 1;
+    }
+
+    var timeout = null;
+    dom_src.oninput = function (e) {
+        clearTimeout(timeout);
+        timeout = setTimeout(draw_md, delay);
+        if (action_stack)
+            action_stack.push();
+    };
+
+    draw_md();
+    return draw_md;
+})();
+
+
+// resize handler
+redraw = (function () {
+    function onresize() {
+        var y = (dom_hbar.offsetTop + dom_hbar.offsetHeight) + 'px';
+        dom_wrap.style.top = y;
+        dom_swrap.style.top = y;
+        dom_ref.style.width = getComputedStyle(dom_src).offsetWidth + 'px';
+        map_src = genmap(dom_ref, map_src);
+        map_pre = genmap(dom_pre, map_pre);
+        dbg(document.body.clientWidth + 'x' + document.body.clientHeight);
+    }
+    function setsbs() {
+        dom_wrap.setAttribute('class', '');
+        dom_swrap.setAttribute('class', '');
+        onresize();
+    }
+    function modetoggle() {
+        mode = dom_nsbs.innerHTML;
+        dom_nsbs.innerHTML = mode == 'editor' ? 'preview' : 'editor';
+        mode += ' single';
+        dom_wrap.setAttribute('class', mode);
+        dom_swrap.setAttribute('class', mode);
+        onresize();
+    }
+
+    window.onresize = onresize;
+    window.onscroll = null;
+    dom_wrap.onscroll = null;
+    dom_sbs.onclick = setsbs;
+    dom_nsbs.onclick = modetoggle;
+
+    onresize();
+    return onresize;
+})();
+
+
+// scroll handlers
+(function () {
+    var skip_src = false, skip_pre = false;
+
+    function scroll(src, srcmap, dst, dstmap) {
+        var y = src.scrollTop;
+        if (y < 8) {
+            dst.scrollTop = 0;
+            return;
+        }
+        if (y + 8 + src.clientHeight > src.scrollHeight) {
+            dst.scrollTop = dst.scrollHeight - dst.clientHeight;
+            return;
+        }
+        y += src.clientHeight / 2;
+        var sy1 = -1, sy2 = -1, dy1 = -1, dy2 = -1;
+        for (var a = 1; a < nlines + 1; a++) {
+            if (srcmap[a] === null || dstmap[a] === null)
+                continue;
+
+            if (srcmap[a] > y) {
+                sy2 = srcmap[a];
+                dy2 = dstmap[a];
+                break;
+            }
+            sy1 = srcmap[a];
+            dy1 = dstmap[a];
+        }
+        if (sy1 == -1)
+            return;
+
+        var dy = dy1;
+        if (sy2 != -1 && dy2 != -1) {
+            var mul = (y - sy1) / (sy2 - sy1);
+            dy = dy1 + (dy2 - dy1) * mul;
+        }
+        dst.scrollTop = dy - dst.clientHeight / 2;
+    }
+
+    dom_src.onscroll = function () {
+        //dbg: dom_ref.scrollTop = dom_src.scrollTop;
+        if (skip_src) {
+            skip_src = false;
+            return;
+        }
+        skip_pre = true;
+        scroll(dom_src, map_src, dom_wrap, map_pre);
+    };
+
+    dom_wrap.onscroll = function () {
+        if (skip_pre) {
+            skip_pre = false;
+            return;
+        }
+        skip_src = true;
+        scroll(dom_wrap, map_pre, dom_src, map_src);
+    };
+})();
+
+
+// save handler
+function save(e) {
+    if (e) e.preventDefault();
+    var save_btn = document.getElementById("save"),
+        save_cls = save_btn.getAttribute('class') + '';
+
+    if (save_cls.indexOf('disabled') >= 0) {
+        toast('font-size:2em;color:#fc6;width:9em;', 'no changes');
+        return;
+    }
+
+    var force = (save_cls.indexOf('force-save') >= 0);
+    if (force && !confirm('confirm that you wish to lose the changes made on the server since you opened this document')) {
+        alert('ok, aborted');
+        return;
+    }
+
+    var txt = dom_src.value;
+
+    var fd = new FormData();
+    fd.append("act", "tput");
+    fd.append("lastmod", (force ? -1 : last_modified));
+    fd.append("body", txt);
+
+    var url = (document.location + '').split('?')[0];
+    var xhr = new XMLHttpRequest();
+    xhr.open('POST', url, true);
+    xhr.responseType = 'text';
+    xhr.onreadystatechange = save_cb;
+    xhr.btn = save_btn;
+    xhr.txt = txt;
+    xhr.send(fd);
+}
+
+function save_cb() {
+    if (this.readyState != XMLHttpRequest.DONE)
+        return;
+
+    if (this.status !== 200) {
+        alert('Error!  The file was NOT saved.\n\n' + this.status + ": " + (this.responseText + '').replace(/^<pre>/, ""));
+        return;
+    }
+
+    var r;
+    try {
+        r = JSON.parse(this.responseText);
+    }
+    catch (ex) {
+        alert('Failed to parse reply from server:\n\n' + this.responseText);
+        return;
+    }
+
+    if (!r.ok) {
+        if (!this.btn.classList.contains('force-save')) {
+            this.btn.classList.add('force-save');
+            var msg = [
+                'This file has been modified since you started editing it!\n',
+                'if you really want to overwrite, press save again.\n',
+                'modified ' + ((r.now - r.lastmod) / 1000) + ' seconds ago,',
+                ((r.lastmod - last_modified) / 1000) + ' sec after you opened it\n',
+                last_modified + ' lastmod when you opened it,',
+                r.lastmod + ' lastmod on the server now,',
+                r.now + ' server time now,\n',
+            ];
+            alert(msg.join('\n'));
+        }
+        else {
+            alert('Error! Save failed.  Maybe this JSON explains why:\n\n' + this.responseText);
+        }
+        return;
+    }
+
+    this.btn.classList.remove('force-save');
+    //alert('save OK -- wrote ' + r.size + ' bytes.\n\nsha512: ' + r.sha512);
+
+    // download the saved doc from the server and compare
+    var url = (document.location + '').split('?')[0] + '?raw';
+    var xhr = new XMLHttpRequest();
+    xhr.open('GET', url, true);
+    xhr.responseType = 'text';
+    xhr.onreadystatechange = save_chk;
+    xhr.btn = this.save_btn;
+    xhr.txt = this.txt;
+    xhr.lastmod = r.lastmod;
+    xhr.send();
+}
+
+function save_chk() {
+    if (this.readyState != XMLHttpRequest.DONE)
+        return;
+
+    if (this.status !== 200) {
+        alert('Error!  The file was NOT saved.\n\n' + this.status + ": " + (this.responseText + '').replace(/^<pre>/, ""));
+        return;
+    }
+
+    var doc1 = this.txt.replace(/\r\n/g, "\n");
+    var doc2 = this.responseText.replace(/\r\n/g, "\n");
+    if (doc1 != doc2) {
+        alert(
+            'Error! The document on the server does not appear to have saved correctly (your editor contents and the server copy is not identical). Place the document on your clipboard for now and check the server logs for hints\n\n' +
+            'Length: yours=' + doc1.length + ', server=' + doc2.length
+        );
+        alert('yours, ' + doc1.length + ' byte:\n[' + doc1 + ']');
+        alert('server, ' + doc2.length + ' byte:\n[' + doc2 + ']');
+        return;
+    }
+
+    last_modified = this.lastmod;
+    server_md = this.txt;
+    draw_md();
+    toast('font-size:6em;font-family:serif;color:#cf6;width:4em;', 'OK✔️');
+}
+
+function toast(style, msg) {
+    var ok = document.createElement('div');
+    style += 'font-weight:bold;background:#444;border-radius:.3em;padding:.6em 0;position:fixed;top:30%;left:calc(50% - 2em);text-align:center;z-index:9001;transition:opacity 0.2s ease-in-out;opacity:1';
+    ok.setAttribute('style', style);
+    ok.innerHTML = msg;
+    var parent = document.getElementById('m');
+    document.documentElement.appendChild(ok);
+    setTimeout(function () {
+        ok.style.opacity = 0;
+    }, 500);
+    setTimeout(function () {
+        ok.parentNode.removeChild(ok);
+    }, 750);
+}
+
+
+// firefox bug: initial selection offset isn't cleared properly through js
+var ff_clearsel = (function () {
+    if (navigator.userAgent.indexOf(') Gecko/') === -1)
+        return function () { }
+
+    return function () {
+        var txt = dom_src.value;
+        var y = dom_src.scrollTop;
+        dom_src.value = '';
+        dom_src.value = txt;
+        dom_src.scrollTop = y;
+    };
+})();
+
+
+// returns car/cdr (selection bounds) and n1/n2 (grown to full lines)
+function linebounds(just_car, greedy_growth) {
+    var car = dom_src.selectionStart,
+        cdr = dom_src.selectionEnd;
+
+    if (just_car)
+        cdr = car;
+
+    var md = dom_src.value,
+        n1 = Math.max(car, 0),
+        n2 = Math.min(cdr, md.length - 1);
+
+    if (greedy_growth !== true) {
+        if (n1 < n2 && md[n1] == '\n')
+            n1++;
+
+        if (n1 < n2 && md[n2 - 1] == '\n')
+            n2 -= 2;
+    }
+
+    n1 = md.lastIndexOf('\n', n1 - 1) + 1;
+    n2 = md.indexOf('\n', n2);
+    if (n2 < n1)
+        n2 = md.length;
+
+    return {
+        "car": car,
+        "cdr": cdr,
+        "n1": n1,
+        "n2": n2,
+        "md": md
+    }
+}
+
+
+// linebounds + the three textranges
+function getsel() {
+    var s = linebounds(false);
+    s.pre = s.md.substring(0, s.n1);
+    s.sel = s.md.substring(s.n1, s.n2);
+    s.post = s.md.substring(s.n2);
+    return s;
+}
+
+
+// place modified getsel into markdown
+function setsel(s) {
+    if (s.car != s.cdr) {
+        s.car = s.pre.length;
+        s.cdr = s.pre.length + s.sel.length;
+    }
+    dom_src.value = [s.pre, s.sel, s.post].join('');
+    dom_src.setSelectionRange(s.car, s.cdr, dom_src.selectionDirection);
+    dom_src.oninput();
+}
+
+
+// indent/dedent
+function md_indent(dedent) {
+    var s = getsel(),
+        sel0 = s.sel;
+
+    if (dedent)
+        s.sel = s.sel.replace(/^  /, "").replace(/\n  /g, "\n");
+    else
+        s.sel = '  ' + s.sel.replace(/\n/g, '\n  ');
+
+    if (s.car == s.cdr)
+        s.car = s.cdr += s.sel.length - sel0.length;
+
+    setsel(s);
+}
+
+
+// header
+function md_header(dedent) {
+    var s = getsel(),
+        sel0 = s.sel;
+
+    if (dedent)
+        s.sel = s.sel.replace(/^#/, "").replace(/^ +/, "");
+    else
+        s.sel = s.sel.replace(/^(#*) ?/, "#$1 ");
+
+    if (s.car == s.cdr)
+        s.car = s.cdr += s.sel.length - sel0.length;
+
+    setsel(s);
+}
+
+
+// smart-home
+function md_home(shift) {
+    var s = linebounds(false, true),
+        ln = s.md.substring(s.n1, s.n2),
+        dir = dom_src.selectionDirection,
+        rev = dir === 'backward',
+        p1 = rev ? s.car : s.cdr,
+        p2 = rev ? s.cdr : s.car,
+        home = 0,
+        lf = ln.lastIndexOf('\n') + 1,
+        re = /^[ \t#>+-]*(\* )?([0-9]+\. +)?/;
+
+    if (rev)
+        home = s.n1 + re.exec(ln)[0].length;
+    else
+        home = s.n1 + lf + re.exec(ln.substring(lf))[0].length;
+
+    p1 = (p1 !== home) ? home : (rev ? s.n1 : s.n1 + lf);
+    if (!shift)
+        p2 = p1;
+
+    if (rev !== p1 < p2)
+        dir = rev ? 'forward' : 'backward';
+
+    if (!shift)
+        ff_clearsel();
+
+    dom_src.setSelectionRange(Math.min(p1, p2), Math.max(p1, p2), dir);
+}
+
+
+// autoindent
+function md_newline() {
+    var s = linebounds(true),
+        ln = s.md.substring(s.n1, s.n2),
+        m1 = /^( *)([0-9]+)(\. +)/.exec(ln),
+        m2 = /^[ \t>+-]*(\* )?/.exec(ln);
+
+    var pre = m2[0];
+    if (m1 !== null)
+        pre = m1[1] + (parseInt(m1[2]) + 1) + m1[3];
+
+    if (pre.length > s.car - s.n1)
+        // in gutter, do nothing
+        return true;
+
+    s.pre = s.md.substring(0, s.car) + '\n' + pre;
+    s.sel = '';
+    s.post = s.md.substring(s.car);
+    s.car = s.cdr = s.pre.length;
+    setsel(s);
+    return false;
+}
+
+
+// backspace
+function md_backspace() {
+    var s = linebounds(true),
+        ln = s.md.substring(s.n1, s.n2),
+        m = /^[ \t>+-]*(\* )?([0-9]+\. +)?/.exec(ln);
+
+    var v = m[0].replace(/[^ ]/g, " ");
+    if (v === m[0] || v.length !== ln.length)
+        return true;
+
+    s.pre = s.md.substring(0, s.n1) + v;
+    s.sel = '';
+    s.post = s.md.substring(s.car);
+    s.car = s.cdr = s.pre.length;
+    setsel(s);
+    return false;
+}
+
+
+// paragraph jump
+function md_p_jump(down) {
+    var ofs = dom_src.selectionStart;
+    var txt = dom_src.value;
+
+    if (down) {
+        while (txt[ofs] == '\n' && --ofs > 0);
+        ofs = txt.indexOf("\n\n", ofs);
+        if (ofs < 0)
+            ofs = txt.length - 1;
+
+        while (txt[ofs] == '\n' && ++ofs < txt.length - 1);
+    }
+    else {
+        txt += '\n\n';
+        while (ofs > 1 && txt[ofs - 1] == '\n') ofs--;
+        ofs = Math.max(0, txt.lastIndexOf("\n\n", ofs - 1));
+        while (txt[ofs] == '\n' && ++ofs < txt.length - 1);
+    }
+
+    dom_src.setSelectionRange(ofs, ofs, "none");
+}
+
+
+// hotkeys / toolbar
+(function () {
+    function keydown(ev) {
+        ev = ev || window.event;
+        var kc = ev.keyCode || ev.which;
+        var ctrl = ev.ctrlKey || ev.metaKey;
+        //console.log(ev.code, kc);
+        if (ctrl && (ev.code == "KeyS" || kc == 83)) {
+            save();
+            return false;
+        }
+        if (ev.code == "Escape" || kc == 27) {
+            var d = document.getElementById('helpclose');
+            if (d)
+                d.click();
+        }
+        if (document.activeElement == dom_src) {
+            if (ev.code == "Tab" || kc == 9) {
+                md_indent(ev.shiftKey);
+                return false;
+            }
+            if (ctrl && (ev.code == "KeyH" || kc == 72)) {
+                md_header(ev.shiftKey);
+                return false;
+            }
+            if (!ctrl && (ev.code == "Home" || kc == 36)) {
+                md_home(ev.shiftKey);
+                return false;
+            }
+            if (!ctrl && !ev.shiftKey && (ev.code == "Enter" || kc == 13)) {
+                return md_newline();
+            }
+            if (ctrl && (ev.code == "KeyZ" || kc == 90)) {
+                if (ev.shiftKey)
+                    action_stack.redo();
+                else
+                    action_stack.undo();
+
+                return false;
+            }
+            if (ctrl && (ev.code == "KeyY" || kc == 89)) {
+                action_stack.redo();
+                return false;
+            }
+            if (!ctrl && !ev.shiftKey && kc == 8) {
+                return md_backspace();
+            }
+            var up = ev.code == "ArrowUp" || kc == 38;
+            var dn = ev.code == "ArrowDown" || kc == 40;
+            if (ctrl && (up || dn)) {
+                md_p_jump(dn);
+                return false;
+            }
+        }
+    }
+    document.onkeydown = keydown;
+    document.getElementById('save').onclick = save;
+})();
+
+
+document.getElementById('help').onclick = function (e) {
+    if (e) e.preventDefault();
+    var dom = document.getElementById('helpbox');
+    var dtxt = dom.getElementsByTagName('textarea');
+    if (dtxt.length > 0) {
+        convert_markdown(dtxt[0].value, dom);
+        dom.innerHTML = '<a href="#" id="helpclose">close</a>' + dom.innerHTML;
+    }
+
+    dom.style.display = 'block';
+    document.getElementById('helpclose').onclick = function () {
+        dom.style.display = 'none';
+    };
+};
+
+
+// blame steen
+action_stack = (function () {
+    var hist = {
+        un: [],
+        re: []
+    };
+    var sched_cpos = 0;
+    var sched_timer = null;
+    var ignore = false;
+    var ref = dom_src.value;
+
+    function diff(from, to, cpos) {
+        if (from === to)
+            return null;
+
+        var car = 0,
+            max = Math.max(from.length, to.length);
+
+        for (; car < max; car++)
+            if (from[car] != to[car])
+                break;
+
+        var p1 = from.length,
+            p2 = to.length;
+
+        while (p1 --> 0 && p2 --> 0)
+            if (from[p1] != to[p2])
+                break;
+
+        if (car > ++p1) {
+            car = p1;
+        }
+
+        var txt = from.substring(car, p1)
+        return {
+            car: car,
+            cdr: ++p2,
+            txt: txt,
+            cpos: cpos
+        };
+    }
+
+    function undiff(from, change) {
+        return {
+            txt: from.substring(0, change.car) + change.txt + from.substring(change.cdr),
+            cpos: change.cpos
+        };
+    }
+
+    function apply(src, dst) {
+        dbg('undos(%d) redos(%d)', hist.un.length, hist.re.length);
+
+        if (src.length === 0)
+            return false;
+
+        var patch = src.pop(),
+            applied = undiff(ref, patch),
+            cpos = patch.cpos - (patch.cdr - patch.car) + patch.txt.length,
+            reverse = diff(ref, applied.txt, cpos);
+
+        if (reverse === null)
+            return false;
+
+        dst.push(reverse);
+        ref = applied.txt;
+        ignore = true; // just some browsers
+        dom_src.value = ref;
+        dom_src.setSelectionRange(cpos, cpos);
+        ignore = true; // all browsers
+        dom_src.oninput();
+        return true;
+    }
+
+    function schedule_push() {
+        if (ignore) {
+            ignore = false;
+            return;
+        }
+        hist.re = [];
+        clearTimeout(sched_timer);
+        sched_cpos = dom_src.selectionEnd;
+        sched_timer = setTimeout(push, 500);
+    }
+
+    function undo() {
+        if (hist.re.length == 0) {
+            clearTimeout(sched_timer);
+            push();
+        }
+        return apply(hist.un, hist.re);
+    }
+
+    function redo() {
+        return apply(hist.re, hist.un);
+    }
+
+    function push() {
+        var newtxt = dom_src.value;
+        var change = diff(ref, newtxt, sched_cpos);
+        if (change !== null)
+            hist.un.push(change);
+
+        ref = newtxt;
+        dbg('undos(%d) redos(%d)', hist.un.length, hist.re.length);
+        if (hist.un.length > 0)
+            dbg(static(hist.un.slice(-1)[0]));
+        if (hist.re.length > 0)
+            dbg(static(hist.re.slice(-1)[0]));
+    }
+
+    return {
+        push: push,
+        undo: undo,
+        redo: redo,
+        push: schedule_push,
+        _hist: hist,
+        _ref: ref
+    }
+})();
+
+/*
+document.getElementById('help').onclick = function () {
+    var c1 = getComputedStyle(dom_src).cssText.split(';');
+    var c2 = getComputedStyle(dom_ref).cssText.split(';');
+    var max = Math.min(c1.length, c2.length);
+    for (var a = 0; a < max; a++)
+        if (c1[a] !== c2[a])
+            console.log(c1[a] + '\n' + c2[a]);
+}
+*/

copyparty/web/mde.css

@@ -21,7 +21,6 @@ html, body {
 #mn {
     font-weight: normal;
     margin: 1.3em 0 .7em 1em;
-    font-size: 1.4em;
 }
 #mn a {
     color: #444;
@@ -44,8 +43,8 @@ html, body {
     height: 1.05em;
     margin: -.2em .3em -.2em -.4em;
     display: inline-block;
-    border: 1px solid rgba(0,0,0,0.3);
-    border-width: .05em .05em 0 0;
+    border: 1px solid rgba(0,0,0,0.2);
+    border-width: .2em .2em 0 0;
     transform: rotate(45deg);
 }
 #mn a:hover {

copyparty/web/mde.html

@@ -17,7 +17,7 @@
 			</div>
 		</div>
 		<div id="m">
-			<textarea id="mt" style="display:none">{{ md }}</textarea>
+			<textarea id="mt" style="display:none" autocomplete="off">{{ md }}</textarea>
 		</div>
 	</div>
 	<script>
@@ -39,6 +39,6 @@ var lightswitch = (function () {
 })();
 
 	</script>
-	<script src="/.cpr/deps/easymde.full.js"></script>
+	<script src="/.cpr/deps/easymde.js"></script>
 	<script src="/.cpr/mde.js"></script>
 </body></html>

copyparty/web/mde.js

@@ -13,7 +13,9 @@ var dom_md = document.getElementById('mt');
         if (a > 0)
             loc.push(n[a]);
 
-        nav.push('<a href="/' + loc.join('/') + '">' + n[a] + '</a>');
+        var dec = decodeURIComponent(n[a]).replace(/&/g, "&amp;").replace(/</g, "&lt;").replace(/>/g, "&gt;");
+
+        nav.push('<a href="/' + loc.join('/') + '">' + dec + '</a>');
     }
     dom_nav.innerHTML = nav.join('');
 })();
@@ -51,7 +53,8 @@ var mde = (function () {
             "save": "Ctrl-S"
         },
         insertTexts: ["[](", ")"],
-        tabSize: 4,
+        indentWithTabs: false,
+        tabSize: 2,
         toolbar: tbar,
         previewClass: 'mdo',
         onToggleFullScreen: set_jumpto,

copyparty/web/splash.css

@@ -13,6 +13,7 @@ h1 {
 	border-bottom: 1px solid #ccc;
 	margin: 2em 0 .4em 0;
 	padding: 0 0 .2em 0;
+	font-weight: normal;
 }
 li {
 	margin: 1em 0;
@@ -24,4 +25,29 @@ a {
 	border-bottom: 1px solid #aaa;
 	border-radius: .2em;
 	padding: .2em .8em;
+}
+
+
+html.dark,
+html.dark body,
+html.dark #wrap {
+	background: #222;
+	color: #ccc;
+}
+html.dark h1 {
+	border-color: #777;
+}
+html.dark a {
+	color: #fff;
+	background: #057;
+	border-color: #37a;
+}
+html.dark input {
+	color: #fff;
+	background: #624;
+	border: 1px solid #c27;
+	border-width: 1px 0 0 0;
+	border-radius: .5em;
+	padding: .5em .7em;
+	margin: 0 .5em 0 0;
 }

copyparty/web/splash.html

@@ -36,7 +36,11 @@
             </form>
         </ul>
     </div>
-    <!-- script src="/.cpr/splash.js"></script -->
-</body>
+    <script>
 
+if (window.localStorage && localStorage.getItem('darkmode') == 1)
+    document.documentElement.setAttribute("class", "dark");
+
+</script>
+</body>
 </html>

copyparty/web/up2k.js

@@ -34,6 +34,7 @@ window.onerror = function (msg, url, lineNo, columnNo, error) {
                     esc(String(error[find[a]])).replace(/\n/g, '<br />\n'));
     }
     document.body.style.fontSize = '0.8em';
+    document.body.style.padding = '0 1em 1em 1em';
     hcroak(html.join('\n'));
 };
 
@@ -211,7 +212,8 @@ function up2k_init(have_crypto) {
     // handle user intent to use the basic uploader instead
     o('u2nope').onclick = function (e) {
         e.preventDefault();
-        un2k();
+        setmsg('');
+        goto('bup');
     };
 
     if (!String.prototype.format) {

copyparty/web/upload.html

@@ -66,5 +66,5 @@
             </table>
 
             <p id="u2foot"></p>
-            <p>( if you don't need lastmod timestamps, resumable uploads or progress bars just use the <a href="#" id="u2nope" onclick="javascript:goto('bup');">basic uploader</a>)</p>
+            <p>( if you don't need lastmod timestamps, resumable uploads or progress bars just use the <a href="#" id="u2nope">basic uploader</a>)</p>
     </div>

docs/notes.sh

@@ -13,7 +13,7 @@ head -c $((2*1024*1024*1024)) /dev/zero | openssl enc -aes-256-ctr -pass pass:hu
 ## testing multiple parallel uploads
 ## usage:  para | tee log
 
-para() { for s in 1 2 3 4 5 6 7 8 12 16 24 32 48 64; do echo $s; for r in {1..4}; do for ((n=0;n<s;n++)); do curl -sF "act=bput" -F "f=@garbage.file" http://127.0.0.1:1234/ 2>&1 & done; wait; echo; done; done; }
+para() { for s in 1 2 3 4 5 6 7 8 12 16 24 32 48 64; do echo $s; for r in {1..4}; do for ((n=0;n<s;n++)); do curl -sF "act=bput" -F "f=@garbage.file" http://127.0.0.1:3923/ 2>&1 & done; wait; echo; done; done; }
 
 
 ##
@@ -36,13 +36,13 @@ for dir in "${dirs[@]}"; do for fn in ふが "$(printf \\xed\\x93)" 'qwe,rty;asd
 
 fn=$(printf '\xba\xdc\xab.cab')
 echo asdf > "$fn"
-curl --cookie cppwd=wark -sF "act=bput" -F "f=@$fn" http://127.0.0.1:1234/moji/%ED%91/
+curl --cookie cppwd=wark -sF "act=bput" -F "f=@$fn" http://127.0.0.1:3923/moji/%ED%91/
 
 
 ##
 ## test compression
 
-wget -S --header='Accept-Encoding: gzip' -U 'MSIE 6.0; SV1' http://127.0.0.1:1234/.cpr/deps/ogv.js -O- | md5sum; p=~ed/dev/copyparty/copyparty/web/deps/ogv.js.gz; md5sum $p; gzip -d < $p | md5sum
+wget -S --header='Accept-Encoding: gzip' -U 'MSIE 6.0; SV1' http://127.0.0.1:3923/.cpr/deps/ogv.js -O- | md5sum; p=~ed/dev/copyparty/copyparty/web/deps/ogv.js.gz; md5sum $p; gzip -d < $p | md5sum
 
 
 ##
@@ -80,3 +80,45 @@ for d in /usr /var; do find $d -type f -size +30M 2>/dev/null; done | while IFS=
 # py2 on osx
 brew install python@2
 pip install virtualenv
+
+
+##
+## http 206
+
+# az = abcdefghijklmnopqrstuvwxyz
+
+printf '%s\r\n' 'GET /az HTTP/1.1' 'Host: ocv.me' 'Range: bytes=5-10' '' | ncat ocv.me 80 
+# Content-Range: bytes 5-10/26
+# Content-Length: 6
+# fghijk
+
+Range: bytes=0-1    "ab" Content-Range: bytes 0-1/26
+Range: bytes=24-24  "y"  Content-Range: bytes 24-24/26
+Range: bytes=24-25  "yz" Content-Range: bytes 24-25/26
+Range: bytes=24-    "yz" Content-Range: bytes 24-25/26
+Range: bytes=25-29  "z"  Content-Range: bytes 25-25/26
+Range: bytes=26-         Content-Range: bytes */26
+  HTTP/1.1 416 Requested Range Not Satisfiable
+
+
+##
+## md perf
+
+var tsh = [];
+function convert_markdown(md_text, dest_dom) {
+    tsh.push(new Date().getTime());
+    while (tsh.length > 10)
+        tsh.shift();
+    if (tsh.length > 1) {
+        var end = tsh.slice(-2);
+        console.log("render", end.pop() - end.pop(), (tsh[tsh.length - 1] - tsh[0]) / (tsh.length - 1));
+    }
+
+
+##
+## tmpfiles.d meme
+
+mk() { rm -rf /tmp/foo; sudo -u ed bash -c 'mkdir /tmp/foo; echo hi > /tmp/foo/bar'; }
+mk && t0="$(date)" && while true; do date -s "$(date '+ 1 hour')"; systemd-tmpfiles --clean; ls -1 /tmp | grep foo || break; done; echo "$t0"
+mk && sudo -u ed flock /tmp/foo sleep 40 & sleep 1; ps aux | grep -E 'sleep 40$' && t0="$(date)" && for n in {1..40}; do date -s "$(date '+ 1 day')"; systemd-tmpfiles --clean; ls -1 /tmp | grep foo || break; done; echo "$t0"
+mk && t0="$(date)" && for n in {1..40}; do date -s "$(date '+ 1 day')"; systemd-tmpfiles --clean; ls -1 /tmp | grep foo || break; tar -cf/dev/null /tmp/foo; done; echo "$t0"

docs/unirange.py

@@ -0,0 +1,10 @@
+v = "U+0000-00FF, U+0131, U+0152-0153, U+02BB-02BC, U+02C6, U+02DA, U+02DC, U+2000-206F, U+2074, U+20AC, U+2122, U+2191, U+2193, U+2212, U+2215, U+FEFF, U+FFFD"
+for v in v.split(","):
+    if "+" in v:
+        v = v.split("+")[1]
+    if "-" in v:
+        lo, hi = v.split("-")
+    else:
+        lo = hi = v
+    for v in range(int(lo, 16), int(hi, 16) + 1):
+        print("{:4x} [{}]".format(v, chr(v)))

scripts/deps-docker/Dockerfile

@@ -3,7 +3,7 @@ WORKDIR /z
 ENV     ver_asmcrypto=2821dd1dedd1196c378f5854037dda5c869313f3 \
         ver_markdownit=10.0.0 \
         ver_showdown=1.9.1 \
-        ver_marked=1.0.0 \
+        ver_marked=1.1.0 \
         ver_ogvjs=1.6.1 \
         ver_mde=2.10.1 \
         ver_codemirror=5.53.2 \
@@ -11,8 +11,11 @@ ENV     ver_asmcrypto=2821dd1dedd1196c378f5854037dda5c869313f3 \
         ver_zopfli=1.0.3
 
 
-# download
-RUN     apk add cmake make g++ git bash npm patch wget tar pigz brotli gzip unzip python3 python3-dev \
+# download;
+# the scp url is latin from https://fonts.googleapis.com/css2?family=Source+Code+Pro&display=swap
+RUN     mkdir -p /z/dist/no-pk \
+        && wget https://fonts.gstatic.com/s/sourcecodepro/v11/HI_SiYsKILxRpg3hIP6sJ7fM7PqlPevW.woff2 -O scp.woff2 \
+        && apk add cmake make g++ git bash npm patch wget tar pigz brotli gzip unzip python3 python3-dev brotli py3-brotli \
         && wget https://github.com/brion/ogv.js/releases/download/$ver_ogvjs/ogvjs-$ver_ogvjs.zip -O ogvjs.zip \
         && wget https://github.com/asmcrypto/asmcrypto.js/archive/$ver_asmcrypto.tar.gz -O asmcrypto.tgz \
         && wget https://github.com/markedjs/marked/archive/v$ver_marked.tar.gz -O marked.tgz \
@@ -36,23 +39,7 @@ RUN     apk add cmake make g++ git bash npm patch wget tar pigz brotli gzip unzi
             && npm install \
             && npm i gulp-cli -g ) \
         && unzip fontawesome.zip \
-        && tar -xf zopfli.tgz \
-        && mkdir -p /z/dist/no-pk
-
-
-# uncomment if you wanna test the abandoned markdown converters
-#ENV     build_abandoned=1
-
-
-RUN     [ $build_abandoned ] || exit 0; \
-        git clone --depth 1 --branch $ver_showdown https://github.com/showdownjs/showdown/ \
-        && wget https://github.com/markdown-it/markdown-it/archive/$ver_markdownit.tar.gz -O markdownit.tgz \
-        && (cd showdown \
-            && npm install \
-            && npm i grunt -g ) \
-        && (tar -xf markdownit.tgz \
-            && cd markdown-it-$ver_markdownit \
-            && npm install )
+        && tar -xf zopfli.tgz
 
 
 # build fonttools (which needs zopfli)
@@ -80,31 +67,27 @@ RUN     cd ogvjs-$ver_ogvjs \
         && cp -pv \
             ogv.js \
             ogv-worker-audio.js \
-            ogv-demuxer-ogg.js \
             ogv-demuxer-ogg-wasm.js \
             ogv-demuxer-ogg-wasm.wasm \
-            ogv-demuxer-webm.js \
             ogv-demuxer-webm-wasm.js \
             ogv-demuxer-webm-wasm.wasm \
-            ogv-decoder-audio-opus.js \
             ogv-decoder-audio-opus-wasm.js \
             ogv-decoder-audio-opus-wasm.wasm \
-            ogv-decoder-audio-vorbis.js \
             ogv-decoder-audio-vorbis-wasm.js \
             ogv-decoder-audio-vorbis-wasm.wasm \
-            dynamicaudio.swf \
             /z/dist
 
+#            ogv-demuxer-ogg.js \
+#            ogv-demuxer-webm.js \
+#            ogv-decoder-audio-opus.js \
+#            ogv-decoder-audio-vorbis.js \
+#            dynamicaudio.swf \
+
 
 # build marked
-RUN     wget https://github.com/markedjs/marked/commit/5c166d4164791f643693478e4ac094d63d6e0c9a.patch -O marked-git-1.patch \
-        && wget https://patch-diff.githubusercontent.com/raw/markedjs/marked/pull/1652.patch -O marked-git-2.patch
-
 COPY    marked.patch /z/
 COPY    marked-ln.patch /z/
 RUN     cd marked-$ver_marked \
-        && patch -p1 < /z/marked-git-1.patch \
-        && patch -p1 < /z/marked-git-2.patch \
         && patch -p1 < /z/marked-ln.patch \
         && patch -p1 < /z/marked.patch \
         && npm run build \
@@ -138,57 +121,10 @@ RUN     cd easy-markdown-editor-$ver_mde \
         && patch -p1 < /z/easymde-ln.patch \
         && gulp \
         && cp -pv dist/easymde.min.css /z/dist/easymde.css \
-        && cp -pv dist/easymde.min.js /z/dist/easymde.js \
-        && sed -ri '/pipe.terser/d; /cleanCSS/d' gulpfile.js \
-        && gulp \
-        && cp -pv dist/easymde.min.css /z/dist/easymde.full.css \
-        && cp -pv dist/easymde.min.js /z/dist/easymde.full.js
+        && cp -pv dist/easymde.min.js /z/dist/easymde.js
 
 
-# build showdown (abandoned; disabled by default)
-COPY    showdown.patch /z/
-RUN     [ $build_abandoned ] || exit 0; \
-        cd showdown \
-        && rm -rf bin dist \
-#       # remove ellipsis plugin \
-        && rm \
-            src/subParsers/ellipsis.js \
-            test/cases/ellipsis* \
-#       # remove html-to-md converter \
-        && rm \
-            test/node/testsuite.makemd.js \
-            test/node/showdown.Converter.makeMarkdown.js \
-#       # remove emojis \
-        && rm src/subParsers/emoji.js \
-        && awk '/^showdown.helper.emojis/ {o=1} !o; /^\}/ {o=0}' \
-            >f <src/helpers.js \
-        && mv f src/helpers.js \
-        && rm -rf test/features/emojis \
-#       # remove ghmentions \
-        && rm test/features/ghMentions.* \
-#       # remove option descriptions \
-        && sed -ri '/descri(ption|be): /d' src/options.js \
-        && patch -p1 < /z/showdown.patch
-
-RUN     [ $build_abandoned ] || exit 0; \
-        cd showdown \
-        && grunt build \
-        && sed -ri '/sourceMappingURL=showdown.min.js.map/d' dist/showdown.min.js \
-        && mv dist/showdown.min.js /z/dist/showdown.js \
-        && ls -al /z/dist/showdown.js
-
-
-# build markdownit (abandoned; disabled by default)
-COPY    markdown-it.patch /z/
-RUN     [ $build_abandoned ] || exit 0; \
-        cd markdown-it-$ver_markdownit \
-        && patch -p1 < /z/markdown-it.patch \
-        && make browserify \
-        && cp -pv dist/markdown-it.min.js /z/dist/markdown-it.js \
-        && cp -pv dist/markdown-it.js /z/dist/markdown-it-full.js
-
-
-# build fontawesome
+# build fontawesome and scp
 COPY    mini-fa.sh /z
 COPY    mini-fa.css /z
 RUN     /bin/ash /z/mini-fa.sh
@@ -203,38 +139,6 @@ RUN     cd /z/dist \
         && rmdir no-pk
 
 
-# showdown: abandoned due to code-blocks in lists failing
-# 22770 orig
-# 12154 no-emojis
-# 12134 no-srcmap
-# 11189 no-descriptions
-# 11152 no-ellipsis
-# 10617 no-this.makeMd
-#  9569 no-extensions
-#  9537 no-extensions
-#  9410 no-mentions
-
-
-# markdown-it: abandoned because no header anchors (and too big)
-#       32322 107754 orig (wowee)
-# 19619 21392  71540 less entities
-
-
-# marked:
-# 9253 29773 orig
-# 9159 29633 no copyright (reverted)
-# 9040 29057 no sanitize
-# 8870 28631 no email-mangle
-# so really not worth it, just drop the patch when that stops working
-
-
-# easymde:
-# 91836 orig
-# 88635 no spellcheck
-# 88392 no urlRE
-# 85651 less bidi
-# 82855 less mode meta
-
-
-# d=/home/ed/dev/copyparty/scripts/deps-docker/; tar -cf ../x . && ssh root@$bip "cd $d && tar -xv >&2 && make >&2 && tar -cC ../../copyparty/web deps" <../x | (cd ../../copyparty/web/; cat > the.tgz; tar -xvf the.tgz)
+# git diff -U2 --no-index marked-1.1.0-orig/ marked-1.1.0-edit/ -U2 | sed -r '/^index /d;s`^(diff --git a/)[^/]+/(.* b/)[^/]+/`\1\2`; s`^(---|\+\+\+) ([ab]/)[^/]+/`\1 \2`' > ../dev/copyparty/scripts/deps-docker/marked-ln.patch
+# d=/home/ed/dev/copyparty/scripts/deps-docker/; tar -cf ../x . && ssh root@$bip "cd $d && tar -xv >&2 && make >&2 && tar -cC ../../copyparty/web deps" <../x | (cd ../../copyparty/web/; cat > the.tgz; tar -xvf the.tgz; rm the.tgz)
 # gzip -dkf ../dev/copyparty/copyparty/web/deps/deps/marked.full.js.gz && diff -NarU2 ../dev/copyparty/copyparty/web/deps/{,deps/}marked.full.js

scripts/deps-docker/marked-ln.patch

@@ -35,7 +35,7 @@ add data-ln="%d" to most tags, %d is the source markdown line
 +      // this.ln will be bumped by recursive calls into this func;
 +      // reset the count and rely on the outermost token's raw only
 +      ln = this.ln;
-+      
++
        // newline
        if (token = this.tokenizer.space(src)) {
          src = src.substring(token.raw.length);
@@ -180,7 +180,7 @@ diff --git a/src/Parser.js b/src/Parser.js
 +            // similar to tables, writing contents before the <ul> tag
 +            // so update the tag attribute as we go
 +            // (assuming all list entries got tagged with a source-line, probably safe w)
-+            body += this.renderer.tag_ln(item.tokens[0].ln).listitem(itemBody, task, checked);
++            body += this.renderer.tag_ln((item.tokens[0] || token).ln).listitem(itemBody, task, checked);
            }
  
 -          out += this.renderer.list(body, ordered, start);
@@ -234,7 +234,7 @@ diff --git a/src/Renderer.js b/src/Renderer.js
 -      return '<pre><code>'
 +      return '<pre' + this.ln + '><code>'
          + (escaped ? code : escape(code, true))
-         + '</code></pre>';
+         + '</code></pre>\n';
      }
  
 -    return '<pre><code class="'

scripts/deps-docker/marked.patch

@@ -1,7 +1,141 @@
-diff -NarU1 marked-1.0.0-orig/src/defaults.js marked-1.0.0-edit/src/defaults.js
---- marked-1.0.0-orig/src/defaults.js	2020-04-21 01:03:48.000000000 +0000
-+++ marked-1.0.0-edit/src/defaults.js	2020-04-25 19:16:56.124621393 +0000
-@@ -9,10 +9,6 @@
+diff --git a/src/Lexer.js b/src/Lexer.js
+--- a/src/Lexer.js
++++ b/src/Lexer.js
+@@ -5,5 +5,5 @@ const { block, inline } = require('./rules.js');
+ /**
+  * smartypants text replacement
+- */
++ *
+ function smartypants(text) {
+   return text
+@@ -26,5 +26,5 @@ function smartypants(text) {
+ /**
+  * mangle email addresses
+- */
++ *
+ function mangle(text) {
+   let out = '',
+@@ -439,5 +439,5 @@ module.exports = class Lexer {
+ 
+       // autolink
+-      if (token = this.tokenizer.autolink(src, mangle)) {
++      if (token = this.tokenizer.autolink(src)) {
+         src = src.substring(token.raw.length);
+         tokens.push(token);
+@@ -446,5 +446,5 @@ module.exports = class Lexer {
+ 
+       // url (gfm)
+-      if (!inLink && (token = this.tokenizer.url(src, mangle))) {
++      if (!inLink && (token = this.tokenizer.url(src))) {
+         src = src.substring(token.raw.length);
+         tokens.push(token);
+@@ -453,5 +453,5 @@ module.exports = class Lexer {
+ 
+       // text
+-      if (token = this.tokenizer.inlineText(src, inRawBlock, smartypants)) {
++      if (token = this.tokenizer.inlineText(src, inRawBlock)) {
+         src = src.substring(token.raw.length);
+         tokens.push(token);
+diff --git a/src/Renderer.js b/src/Renderer.js
+--- a/src/Renderer.js
++++ b/src/Renderer.js
+@@ -140,5 +140,5 @@ module.exports = class Renderer {
+ 
+   link(href, title, text) {
+-    href = cleanUrl(this.options.sanitize, this.options.baseUrl, href);
++    href = cleanUrl(this.options.baseUrl, href);
+     if (href === null) {
+       return text;
+@@ -153,5 +153,5 @@ module.exports = class Renderer {
+ 
+   image(href, title, text) {
+-    href = cleanUrl(this.options.sanitize, this.options.baseUrl, href);
++    href = cleanUrl(this.options.baseUrl, href);
+     if (href === null) {
+       return text;
+diff --git a/src/Tokenizer.js b/src/Tokenizer.js
+--- a/src/Tokenizer.js
++++ b/src/Tokenizer.js
+@@ -287,11 +287,8 @@ module.exports = class Tokenizer {
+     if (cap) {
+       return {
+-        type: this.options.sanitize
+-          ? 'paragraph'
+-          : 'html',
++        type: 'html',
+         raw: cap[0],
+-        pre: !this.options.sanitizer
+-          && (cap[1] === 'pre' || cap[1] === 'script' || cap[1] === 'style'),
+-        text: this.options.sanitize ? (this.options.sanitizer ? this.options.sanitizer(cap[0]) : escape(cap[0])) : cap[0]
++        pre: cap[1] === 'pre' || cap[1] === 'script' || cap[1] === 'style',
++        text: cap[0]
+       };
+     }
+@@ -421,15 +418,9 @@ module.exports = class Tokenizer {
+ 
+       return {
+-        type: this.options.sanitize
+-          ? 'text'
+-          : 'html',
++        type: 'html',
+         raw: cap[0],
+         inLink,
+         inRawBlock,
+-        text: this.options.sanitize
+-          ? (this.options.sanitizer
+-            ? this.options.sanitizer(cap[0])
+-            : escape(cap[0]))
+-          : cap[0]
++        text: cap[0]
+       };
+     }
+@@ -550,10 +541,10 @@ module.exports = class Tokenizer {
+   }
+ 
+-  autolink(src, mangle) {
++  autolink(src) {
+     const cap = this.rules.inline.autolink.exec(src);
+     if (cap) {
+       let text, href;
+       if (cap[2] === '@') {
+-        text = escape(this.options.mangle ? mangle(cap[1]) : cap[1]);
++        text = escape(cap[1]);
+         href = 'mailto:' + text;
+       } else {
+@@ -578,10 +569,10 @@ module.exports = class Tokenizer {
+   }
+ 
+-  url(src, mangle) {
++  url(src) {
+     let cap;
+     if (cap = this.rules.inline.url.exec(src)) {
+       let text, href;
+       if (cap[2] === '@') {
+-        text = escape(this.options.mangle ? mangle(cap[0]) : cap[0]);
++        text = escape(cap[0]);
+         href = 'mailto:' + text;
+       } else {
+@@ -615,12 +606,12 @@ module.exports = class Tokenizer {
+   }
+ 
+-  inlineText(src, inRawBlock, smartypants) {
++  inlineText(src, inRawBlock) {
+     const cap = this.rules.inline.text.exec(src);
+     if (cap) {
+       let text;
+       if (inRawBlock) {
+-        text = this.options.sanitize ? (this.options.sanitizer ? this.options.sanitizer(cap[0]) : escape(cap[0])) : cap[0];
++        text = cap[0];
+       } else {
+-        text = escape(this.options.smartypants ? smartypants(cap[0]) : cap[0]);
++        text = escape(cap[0]);
+       }
+       return {
+diff --git a/src/defaults.js b/src/defaults.js
+--- a/src/defaults.js
++++ b/src/defaults.js
+@@ -8,12 +8,8 @@ function getDefaults() {
+     highlight: null,
      langPrefix: 'language-',
 -    mangle: true,
      pedantic: false,
@@ -12,10 +146,12 @@ diff -NarU1 marked-1.0.0-orig/src/defaults.js marked-1.0.0-edit/src/defaults.js
      smartLists: false,
 -    smartypants: false,
      tokenizer: null,
-diff -NarU1 marked-1.0.0-orig/src/helpers.js marked-1.0.0-edit/src/helpers.js
---- marked-1.0.0-orig/src/helpers.js	2020-04-21 01:03:48.000000000 +0000
-+++ marked-1.0.0-edit/src/helpers.js	2020-04-25 18:58:43.001320210 +0000
-@@ -65,16 +65,3 @@
+     walkTokens: null,
+diff --git a/src/helpers.js b/src/helpers.js
+--- a/src/helpers.js
++++ b/src/helpers.js
+@@ -64,18 +64,5 @@ function edit(regex, opt) {
+ const nonWordAndColonTest = /[^\w:]/g;
  const originIndependentUrl = /^$|^[a-z][a-z0-9+.-]*:|^[?#]/i;
 -function cleanUrl(sanitize, base, href) {
 -  if (sanitize) {
@@ -33,7 +169,9 @@ diff -NarU1 marked-1.0.0-orig/src/helpers.js marked-1.0.0-edit/src/helpers.js
 -  }
 +function cleanUrl(base, href) {
    if (base && !originIndependentUrl.test(href)) {
-@@ -224,8 +211,2 @@
+     href = resolveUrl(base, href);
+@@ -223,10 +210,4 @@ function findClosingBracket(str, b) {
+ }
  
 -function checkSanitizeDeprecation(opt) {
 -  if (opt && opt.sanitize && !opt.silent) {
@@ -42,228 +180,161 @@ diff -NarU1 marked-1.0.0-orig/src/helpers.js marked-1.0.0-edit/src/helpers.js
 -}
 -
  module.exports = {
-@@ -240,4 +221,3 @@
+   escape,
+@@ -239,5 +220,4 @@ module.exports = {
+   splitCells,
    rtrim,
 -  findClosingBracket,
 -  checkSanitizeDeprecation
 +  findClosingBracket
  };
-diff -NarU1 marked-1.0.0-orig/src/Lexer.js marked-1.0.0-edit/src/Lexer.js
---- marked-1.0.0-orig/src/Lexer.js	2020-04-21 01:03:48.000000000 +0000
-+++ marked-1.0.0-edit/src/Lexer.js	2020-04-25 22:46:54.107584066 +0000
-@@ -6,3 +6,3 @@
-  * smartypants text replacement
-- */
-+ *
- function smartypants(text) {
-@@ -27,3 +27,3 @@
-  * mangle email addresses
-- */
-+ *
- function mangle(text) {
-@@ -388,3 +388,3 @@
-       // autolink
--      if (token = this.tokenizer.autolink(src, mangle)) {
-+      if (token = this.tokenizer.autolink(src)) {
-         src = src.substring(token.raw.length);
-@@ -395,3 +395,3 @@
-       // url (gfm)
--      if (!inLink && (token = this.tokenizer.url(src, mangle))) {
-+      if (!inLink && (token = this.tokenizer.url(src))) {
-         src = src.substring(token.raw.length);
-@@ -402,3 +402,3 @@
-       // text
--      if (token = this.tokenizer.inlineText(src, inRawBlock, smartypants)) {
-+      if (token = this.tokenizer.inlineText(src, inRawBlock)) {
-         src = src.substring(token.raw.length);
-diff -NarU1 marked-1.0.0-orig/src/marked.js marked-1.0.0-edit/src/marked.js
---- marked-1.0.0-orig/src/marked.js	2020-04-21 01:03:48.000000000 +0000
-+++ marked-1.0.0-edit/src/marked.js	2020-04-25 22:42:55.140924439 +0000
-@@ -8,3 +8,2 @@
+diff --git a/src/marked.js b/src/marked.js
+--- a/src/marked.js
++++ b/src/marked.js
+@@ -7,5 +7,4 @@ const Slugger = require('./Slugger.js');
+ const {
    merge,
 -  checkSanitizeDeprecation,
    escape
-@@ -37,3 +36,2 @@
-     opt = merge({}, marked.defaults, opt || {});
--    checkSanitizeDeprecation(opt);
-     const highlight = opt.highlight;
-@@ -101,6 +99,5 @@
-     opt = merge({}, marked.defaults, opt || {});
--    checkSanitizeDeprecation(opt);
-     return Parser.parse(Lexer.lex(src, opt), opt);
+ } = require('./helpers.js');
+@@ -35,5 +34,4 @@ function marked(src, opt, callback) {
+ 
+   opt = merge({}, marked.defaults, opt || {});
+-  checkSanitizeDeprecation(opt);
+ 
+   if (callback) {
+@@ -108,5 +106,5 @@ function marked(src, opt, callback) {
+     return Parser.parse(tokens, opt);
    } catch (e) {
 -    e.message += '\nPlease report this to https://github.com/markedjs/marked.';
 +    e.message += '\nmake issue @ https://github.com/9001/copyparty';
-     if ((opt || marked.defaults).silent) {
-diff -NarU1 marked-1.0.0-orig/src/Renderer.js marked-1.0.0-edit/src/Renderer.js
---- marked-1.0.0-orig/src/Renderer.js	2020-04-21 01:03:48.000000000 +0000
-+++ marked-1.0.0-edit/src/Renderer.js	2020-04-25 18:59:15.091319265 +0000
-@@ -134,3 +134,3 @@
-   link(href, title, text) {
--    href = cleanUrl(this.options.sanitize, this.options.baseUrl, href);
-+    href = cleanUrl(this.options.baseUrl, href);
-     if (href === null) {
-@@ -147,3 +147,3 @@
-   image(href, title, text) {
--    href = cleanUrl(this.options.sanitize, this.options.baseUrl, href);
-+    href = cleanUrl(this.options.baseUrl, href);
-     if (href === null) {
-diff -NarU1 marked-1.0.0-orig/src/Tokenizer.js marked-1.0.0-edit/src/Tokenizer.js
---- marked-1.0.0-orig/src/Tokenizer.js	2020-04-21 01:03:48.000000000 +0000
-+++ marked-1.0.0-edit/src/Tokenizer.js	2020-04-25 22:47:07.610917004 +0000
-@@ -256,9 +256,6 @@
-       return {
--        type: this.options.sanitize
--          ? 'paragraph'
--          : 'html',
--        raw: cap[0],
--        pre: !this.options.sanitizer
--          && (cap[1] === 'pre' || cap[1] === 'script' || cap[1] === 'style'),
--        text: this.options.sanitize ? (this.options.sanitizer ? this.options.sanitizer(cap[0]) : escape(cap[0])) : cap[0]
-+        type: 'html',
-+        raw: cap[0],
-+        pre: cap[1] === 'pre' || cap[1] === 'script' || cap[1] === 'style',
-+        text: cap[0]
-       };
-@@ -382,5 +379,3 @@
-       return {
--        type: this.options.sanitize
--          ? 'text'
--          : 'html',
-+        type: 'html',
-         raw: cap[0],
-@@ -388,7 +383,3 @@
-         inRawBlock,
--        text: this.options.sanitize
--          ? (this.options.sanitizer
--            ? this.options.sanitizer(cap[0])
--            : escape(cap[0]))
--          : cap[0]
-+        text: cap[0]
-       };
-@@ -504,3 +495,3 @@
- 
--  autolink(src, mangle) {
-+  autolink(src) {
-     const cap = this.rules.inline.autolink.exec(src);
-@@ -509,3 +500,3 @@
-       if (cap[2] === '@') {
--        text = escape(this.options.mangle ? mangle(cap[1]) : cap[1]);
-+        text = escape(cap[1]);
-         href = 'mailto:' + text;
-@@ -532,3 +523,3 @@
- 
--  url(src, mangle) {
-+  url(src) {
-     let cap;
-@@ -537,3 +528,3 @@
-       if (cap[2] === '@') {
--        text = escape(this.options.mangle ? mangle(cap[0]) : cap[0]);
-+        text = escape(cap[0]);
-         href = 'mailto:' + text;
-@@ -569,3 +560,3 @@
- 
--  inlineText(src, inRawBlock, smartypants) {
-+  inlineText(src, inRawBlock) {
-     const cap = this.rules.inline.text.exec(src);
-@@ -574,5 +565,5 @@
-       if (inRawBlock) {
--        text = this.options.sanitize ? (this.options.sanitizer ? this.options.sanitizer(cap[0]) : escape(cap[0])) : cap[0];
-+        text = cap[0];
-       } else {
--        text = escape(this.options.smartypants ? smartypants(cap[0]) : cap[0]);
-+        text = escape(cap[0]);
-       }
-diff -NarU1 marked-1.0.0-orig/test/bench.js marked-1.0.0-edit/test/bench.js
---- marked-1.0.0-orig/test/bench.js	2020-04-21 01:03:48.000000000 +0000
-+++ marked-1.0.0-edit/test/bench.js	2020-04-25 19:02:27.227980287 +0000
-@@ -34,3 +34,2 @@
+     if (opt.silent) {
+       return '<p>An error occurred:</p><pre>'
+diff --git a/test/bench.js b/test/bench.js
+--- a/test/bench.js
++++ b/test/bench.js
+@@ -33,5 +33,4 @@ async function runBench(options) {
+     breaks: false,
      pedantic: false,
 -    sanitize: false,
      smartLists: false
-@@ -46,3 +45,2 @@
+   });
+@@ -45,5 +44,4 @@ async function runBench(options) {
+     breaks: false,
      pedantic: false,
 -    sanitize: false,
      smartLists: false
-@@ -59,3 +57,2 @@
+   });
+@@ -58,5 +56,4 @@ async function runBench(options) {
+     breaks: false,
      pedantic: false,
 -    sanitize: false,
      smartLists: false
-@@ -71,3 +68,2 @@
+   });
+@@ -70,5 +67,4 @@ async function runBench(options) {
+     breaks: false,
      pedantic: false,
 -    sanitize: false,
      smartLists: false
-@@ -84,3 +80,2 @@
+   });
+@@ -83,5 +79,4 @@ async function runBench(options) {
+     breaks: false,
      pedantic: true,
 -    sanitize: false,
      smartLists: false
-@@ -96,3 +91,2 @@
+   });
+@@ -95,5 +90,4 @@ async function runBench(options) {
+     breaks: false,
      pedantic: true,
 -    sanitize: false,
      smartLists: false
-diff -NarU1 marked-1.0.0-orig/test/specs/run-spec.js marked-1.0.0-edit/test/specs/run-spec.js
---- marked-1.0.0-orig/test/specs/run-spec.js	2020-04-21 01:03:48.000000000 +0000
-+++ marked-1.0.0-edit/test/specs/run-spec.js	2020-04-25 19:05:24.321308408 +0000
-@@ -21,6 +21,2 @@
+   });
+diff --git a/test/specs/run-spec.js b/test/specs/run-spec.js
+--- a/test/specs/run-spec.js
++++ b/test/specs/run-spec.js
+@@ -22,8 +22,4 @@ function runSpecs(title, dir, showCompletionTable, options) {
            }
+ 
 -          if (spec.options.sanitizer) {
 -            // eslint-disable-next-line no-eval
 -            spec.options.sanitizer = eval(spec.options.sanitizer);
 -          }
+ 
            (spec.only ? fit : (spec.skip ? xit : it))('should ' + passFail + example, async() => {
-@@ -49,2 +45 @@
+@@ -53,3 +49,2 @@ runSpecs('Original', './original', false, { gfm: false, pedantic: true });
+ runSpecs('New', './new');
  runSpecs('ReDOS', './redos');
 -runSpecs('Security', './security', false, { silent: true }); // silent - do not show deprecation warning
-diff -NarU1 marked-1.0.0-orig/test/unit/Lexer-spec.js marked-1.0.0-edit/test/unit/Lexer-spec.js
---- marked-1.0.0-orig/test/unit/Lexer-spec.js	2020-04-21 01:03:48.000000000 +0000
-+++ marked-1.0.0-edit/test/unit/Lexer-spec.js	2020-04-25 22:47:27.170916427 +0000
-@@ -464,3 +464,3 @@
+diff --git a/test/unit/Lexer-spec.js b/test/unit/Lexer-spec.js
+--- a/test/unit/Lexer-spec.js
++++ b/test/unit/Lexer-spec.js
+@@ -465,5 +465,5 @@ a | b
+     });
  
 -    it('sanitize', () => {
 +    /*it('sanitize', () => {
        expectTokens({
-@@ -482,3 +482,3 @@
+         md: '<div>html</div>',
+@@ -483,5 +483,5 @@ a | b
+         ]
        });
 -    });
 +    });*/
    });
-@@ -586,3 +586,3 @@
+ 
+@@ -587,5 +587,5 @@ a | b
+       });
  
 -      it('html sanitize', () => {
 +      /*it('html sanitize', () => {
          expectInlineTokens({
-@@ -596,3 +596,3 @@
+           md: '<div>html</div>',
+@@ -597,5 +597,5 @@ a | b
+           ]
          });
 -      });
 +      });*/
  
-@@ -825,3 +825,3 @@
+       it('link', () => {
+@@ -909,5 +909,5 @@ a | b
+         });
  
 -        it('autolink mangle email', () => {
 +        /*it('autolink mangle email', () => {
            expectInlineTokens({
-@@ -845,3 +845,3 @@
+             md: '<test@example.com>',
+@@ -929,5 +929,5 @@ a | b
+             ]
            });
 -        });
 +        });*/
  
-@@ -882,3 +882,3 @@
+         it('url', () => {
+@@ -966,5 +966,5 @@ a | b
+         });
  
 -        it('url mangle email', () => {
 +        /*it('url mangle email', () => {
            expectInlineTokens({
-@@ -902,3 +902,3 @@
+             md: 'test@example.com',
+@@ -986,5 +986,5 @@ a | b
+             ]
            });
 -        });
 +        });*/
        });
-@@ -918,3 +918,3 @@
+ 
+@@ -1002,5 +1002,5 @@ a | b
+       });
  
 -      describe('smartypants', () => {
 +      /*describe('smartypants', () => {
          it('single quotes', () => {
-@@ -988,3 +988,3 @@
+           expectInlineTokens({
+@@ -1072,5 +1072,5 @@ a | b
+           });
          });
 -      });
 +      });*/
      });
+   });

scripts/deps-docker/mini-fa.sh

@@ -26,3 +26,6 @@ awk '/:before .content:"\\/ {sub(/[^"]+"./,""); sub(/".*/,""); print}' </z/dist/
 
 # and finally create a woff with just our icons
 pyftsubset "$orig_woff" --unicodes-file=/z/icon.list --no-ignore-missing-unicodes --flavor=woff --with-zopfli --output-file=/z/dist/no-pk/mini-fa.woff --verbose
+
+# scp is easier, just want basic latin
+pyftsubset /z/scp.woff2 --unicodes="20-7e,ab,b7,bb,2022" --no-ignore-missing-unicodes --flavor=woff2 --output-file=/z/dist/no-pk/scp.woff2 --verbose

scripts/fusefuzz.py

@@ -0,0 +1,100 @@
+#!/usr/bin/env python3
+
+import os
+import time
+
+"""
+mkdir -p /dev/shm/fusefuzz/{r,v}
+PYTHONPATH=.. python3 -m copyparty -v /dev/shm/fusefuzz/r::r -i 127.0.0.1
+../bin/copyparty-fuse.py /dev/shm/fusefuzz/v http://127.0.0.1:3923/ 2 0
+(d="$PWD"; cd /dev/shm/fusefuzz && "$d"/fusefuzz.py)
+"""
+
+
+def chk(fsz, rsz, ofs0, shift, ofs, rf, vf):
+    if ofs != rf.tell():
+        rf.seek(ofs)
+        vf.seek(ofs)
+
+    rb = rf.read(rsz)
+    vb = vf.read(rsz)
+
+    print(f"fsz {fsz} rsz {rsz} ofs {ofs0} shift {shift} ofs {ofs} = {len(rb)}")
+
+    if rb != vb:
+        for n, buf in enumerate([rb, vb]):
+            with open("buf." + str(n), "wb") as f:
+                f.write(buf)
+
+        raise Exception(f"{len(rb)} != {len(vb)}")
+
+    return rb, vb
+
+
+def main():
+    v = "v"
+    for n in range(5):
+        with open(f"r/{n}", "wb") as f:
+            f.write(b"h" * n)
+
+    rand = os.urandom(7919)  # prime
+    for fsz in range(1024 * 1024 * 2 - 3, 1024 * 1024 * 2 + 3):
+        with open("r/f", "wb", fsz) as f:
+            f.write((rand * int(fsz / len(rand) + 1))[:fsz])
+
+        for rsz in range(64 * 1024 - 2, 64 * 1024 + 2):
+            ofslist = [0, 1, 2]
+            for n in range(3):
+                ofslist.append(fsz - n)
+                ofslist.append(fsz - (rsz * 1 + n))
+                ofslist.append(fsz - (rsz * 2 + n))
+
+            for ofs0 in ofslist:
+                for shift in range(-3, 3):
+                    print(f"fsz {fsz} rsz {rsz} ofs {ofs0} shift {shift}")
+                    ofs = ofs0
+                    if ofs < 0 or ofs >= fsz:
+                        continue
+
+                    for n in range(1, 3):
+                        with open(f"{v}/{n}", "rb") as f:
+                            f.read()
+
+                    prev_ofs = -99
+                    with open("r/f", "rb", rsz) as rf:
+                        with open(f"{v}/f", "rb", rsz) as vf:
+                            while True:
+                                ofs += shift
+                                if ofs < 0 or ofs > fsz or ofs == prev_ofs:
+                                    break
+
+                                prev_ofs = ofs
+
+                                rb, vb = chk(fsz, rsz, ofs0, shift, ofs, rf, vf)
+
+                                if not rb:
+                                    break
+
+                                ofs += len(rb)
+
+                    for n in range(1, 3):
+                        with open(f"{v}/{n}", "rb") as f:
+                            f.read()
+
+                    with open("r/f", "rb", rsz) as rf:
+                        with open(f"{v}/f", "rb", rsz) as vf:
+                            for n in range(2):
+                                ofs += shift
+                                if ofs < 0 or ofs > fsz:
+                                    break
+
+                                rb, vb = chk(fsz, rsz, ofs0, shift, ofs, rf, vf)
+
+                                ofs -= rsz
+
+        # bumping fsz, sleep away the dentry cache in cppf
+        time.sleep(1)
+
+
+if __name__ == "__main__":
+    main()

scripts/make-sfx.sh

@@ -13,6 +13,9 @@ echo
 #
 # `no-ogv` saves ~500k by removing the opus/vorbis audio codecs
 #   (only affects apple devices; everything else has native support)
+#
+# `no-cm` saves ~90k by removing easymde/codemirror
+#   (the fancy markdown editor)
 
 
 command -v gtar  >/dev/null &&
@@ -21,6 +24,7 @@ command -v gfind >/dev/null && {
 	sed()  { gsed  "$@"; }
 	find() { gfind "$@"; }
 	sort() { gsort "$@"; }
+	unexpand() { gunexpand "$@"; }
 }
 
 [ -e copyparty/__main__.py ] || cd ..
@@ -35,9 +39,15 @@ while [ ! -z "$1" ]; do
 	[ "$1" = clean  ] && clean=1  && shift && continue
 	[ "$1" = re     ] && repack=1 && shift && continue
 	[ "$1" = no-ogv ] && no_ogv=1 && shift && continue
+	[ "$1" = no-cm  ] && no_cm=1  && shift && continue
 	break
 done
 
+tmv() {
+	touch -r "$1" t
+	mv t "$1"
+}
+
 rm -rf sfx/*
 mkdir -p sfx build
 cd sfx
@@ -62,7 +72,15 @@ cd sfx
 
 	tar -zxf $f
 	mv Jinja2-*/jinja2 .
-	rm -rf Jinja2-* jinja2/testsuite
+	rm -rf Jinja2-* jinja2/testsuite jinja2/_markupsafe/tests.py jinja2/_stringdefs.py
+	
+	f=jinja2/lexer.py
+	sed -r '/.*föö.*/    raise SyntaxError/' <$f >t
+	tmv $f
+	
+	f=jinja2/_markupsafe/_constants.py
+	awk '!/: [0-9]+,?$/ || /(amp|gt|lt|quot|apos|nbsp).:/' <$f >t
+	tmv $f
 
 	# msys2 tar is bad, make the best of it
 	echo collecting source
@@ -98,10 +116,27 @@ rm -f copyparty/web/deps/*.full.*
 
 # it's fine dw
 grep -lE '\.full\.(js|css)' copyparty/web/* |
-while IFS= read -r x; do sed -ri 's/\.full\.(js|css)/.\1/g' "$x"; done
+while IFS= read -r x; do
+	sed -r 's/\.full\.(js|css)/.\1/g' <"$x" >t
+	tmv "$x"
+done
 
 [ $no_ogv ] &&
-	rm -rf copyparty/web/deps/{dynamicaudio,ogv}* copyparty/web/browser.js
+	rm -rf copyparty/web/deps/{dynamicaudio,ogv}*
+
+[ $no_cm ] && {
+	rm -rf copyparty/web/mde.* copyparty/web/deps/easymde*
+	echo h > copyparty/web/mde.html
+	f=copyparty/web/md.html
+	sed -r '/edit2">edit \(fancy/d' <$f >t && tmv "$f"
+}
+
+# up2k goes from 28k to 22k laff
+echo entabbening
+find | grep -E '\.(js|css|html|py)$' | while IFS= read -r f; do
+	unexpand -t 4 --first-only <"$f" >t
+	tmv "$f"
+done
 
 echo creating tar
 args=(--owner=1000 --group=1000)
@@ -132,19 +167,5 @@ printf "done:\n"
 printf "  %s\n" "$(realpath $sfx_out)."{sh,py}
 # rm -rf *
 
-# -rw-r--r--  1 ed ed 811271 May  5 14:35 tar.bz2
-# -rw-r--r--  1 ed ed 732016 May  5 14:35 tar.xz
-
-# -rwxr-xr-x  1 ed ed 830425 May  5 14:35 copyparty-sfx.py*
-# -rwxr-xr-x  1 ed ed 734088 May  5 14:35 copyparty-sfx.sh*
-
-# -rwxr-xr-x  1 ed ed 799690 May  5 14:45 copyparty-sfx.py*
-# -rwxr-xr-x  1 ed ed 735004 May  5 14:45 copyparty-sfx.sh*
-
-# time pigz -11 -J 34 -I 5730 < tar > tar.gz.5730
-# real	8m50.622s
-# user	33m9.821s
-# -rw-r--r--  1 ed ed 1136640 May  5 14:50 tar
-# -rw-r--r--  1 ed ed  296334 May  5 14:50 tar.bz2
-# -rw-r--r--  1 ed ed  324705 May  5 15:01 tar.gz.5730
-# -rw-r--r--  1 ed ed  257208 May  5 14:50 tar.xz
+# tar -tvf ../sfx/tar | sed -r 's/(.* ....-..-.. ..:.. )(.*)/\2 `` \1/' | sort | sed -r 's/(.*) `` (.*)/\2 \1/'| less
+# for n in {1..9}; do tar -tf tar | grep -vE '/$' | sed -r 's/(.*)\.(.*)/\2.\1/' | sort | sed -r 's/([^\.]+)\.(.*)/\2.\1/' | tar -cT- | bzip2 -c$n | wc -c; done 

scripts/sfx.py

@@ -260,7 +260,7 @@ def read_py(binp):
 
 def get_pys():
     ver, chk = read_py(sys.executable)
-    if chk:
+    if chk or PY2:
         return [[chk, ver, sys.executable]]
 
     hits = {sys.executable.lower(): sys.executable}
@@ -426,6 +426,16 @@ def run(tmp, py):
     msg("will use:", py)
     msg("bound to:", tmp)
 
+    # "systemd-tmpfiles-clean.timer"?? HOW do you even come up with this shit
+    try:
+        import fcntl
+
+        fd = os.open(tmp, os.O_RDONLY)
+        fcntl.flock(fd, fcntl.LOCK_EX | fcntl.LOCK_NB)
+        tmp = os.readlink(tmp)  # can't flock a symlink, even with O_NOFOLLOW
+    except:
+        pass
+
     fp_py = os.path.join(tmp, "py")
     with open(fp_py, "wb") as f:
         f.write(py.encode("utf-8") + b"\n")

srv/ceditable.html

@@ -0,0 +1,84 @@
+<!DOCTYPE html><html><head>
+<meta charset="utf-8">
+<meta http-equiv="X-UA-Compatible" content="IE=edge">
+<meta name="viewport" content="width=device-width, initial-scale=0.8">
+<style>
+
+* {
+    margin: 0;
+    padding: 0;
+    outline: 0;
+    border: none;
+    font-size: 1em;
+    line-height: 1em;
+    font-family: monospace, monospace;
+    color: #333;
+}
+html, body {
+    width: 100%;
+    height: 100%;
+    background: #ddd;
+}
+html {
+    font-size: 1.3em;
+}
+li, #edit {
+    list-style-type: none;
+    white-space: pre-wrap;
+    word-break: break-all;
+    overflow-wrap: break-word;
+    word-wrap: break-word; /*ie*/
+}
+li:nth-child(even) {
+    background: #ddd;
+}
+#edit, #html, #txt1, #txt2 {
+    background: #eee;
+    position: fixed;
+    width: calc(50% - .8em);
+    height: calc(50% - .8em);
+}
+#txt1 { top: .5em; left: .5em }
+#edit { top: .5em; right: .5em }
+#html { bottom: .5em; left: .5em }
+#txt2 { bottom: .5em; right: .5em }
+
+</style></head><body>
+<pre id="edit" contenteditable="true"></pre>
+<textarea id="html"></textarea>
+<ul id="txt1"></ul>
+<ul id="txt2"></ul>
+<script>
+
+var edit = document.getElementById('edit'),
+    html = document.getElementById('html'),
+    txt1 = document.getElementById('txt1'),
+    txt2 = document.getElementById('txt2');
+
+var oh = null;
+function fun() {
+    var h = edit.innerHTML;
+    if (oh != h) {
+        oh = h;
+        html.value = h;
+        var t = edit.innerText;
+        if (h.indexOf('<div><br></div>') >= 0)
+            t = t.replace(/\n\n/g, "\n");
+        
+        t = '<li>' + t.
+            replace(/&/g, "&amp;").
+            replace(/</g, "&lt;").
+            replace(/>/g, "&gt;").
+            split('\n').join('</li>\n<li>') + '</li>';
+        
+        t = t.replace(/<li><\/li>/g, '<li> </li>');
+        txt1.innerHTML = t;
+        txt2.innerHTML = t;
+    }
+    setTimeout(fun, 100);
+}
+fun();
+
+</script>
+</body>
+</html>

srv/test.md

@@ -1,3 +1,41 @@
+### hello world
+
+```
+[72....................................................................]
+[80............................................................................]
+```
+
+* foo
+  ```
+  [72....................................................................]
+  [80............................................................................]
+  ```
+
+  * bar
+    ```
+    [72....................................................................]
+    [80............................................................................]
+    ```
+
+```
+l[i]=1I;(){}o0O</> var foo = "$(`bar`)"; a's'd
+```
+
+[](#s1)
+[s1](#s1)
+[#s1](#s1)
+
+a123456789b123456789c123456789d123456789e123456789f123456789g123456789h123456789i123456789j123456789k123456789l123456789m123456789n123456789o123456789p123456789q123456789r123456789s123456789t123456789u123456789v123456789w123456789x123456789y123456789z123456789
+
+<foo> &nbsp; bar &amp; <span>baz</span>
+<a href="?foo=bar&baz=qwe&amp;rty">?foo=bar&baz=qwe&amp;rty</a>
+<!-- hidden -->
+```
+<foo> &nbsp; bar &amp; <span>baz</span>
+<a href="?foo=bar&baz=qwe&amp;rty">?foo=bar&baz=qwe&amp;rty</a>
+<!-- visible -->
+```
+
 *fails marked/showdown/tui/simplemde (just italics), **OK: markdown-it/simplemde:***  
 testing just google.com and underscored _google.com_ also with _google.com,_ trailing comma and _google.com_, comma after