v0.6.0

* show per-connection and per-transfer speeds
* support multiple cookies in parser
* set SameSite=Lax
* restore macos support in sfx.sh
* md-editor: add mojibake/unicode hunter
* md-editor: add table formatter
* md-editor: make bold bolder
* md-editor: more hotkeys
* md-editor: fix saving in fancy
* md-editor: fix eof-scrolling in chrome
* md-editor: fix text erasure with newline
* md-editor: fix backspace behavior in gutter

.eslintrc.json

@@ -0,0 +1,12 @@
+{
+    "env": {
+        "browser": true,
+        "es2021": true
+    },
+    "extends": "eslint:recommended",
+    "parserOptions": {
+        "ecmaVersion": 12
+    },
+    "rules": {
+    }
+}

.vscode/launch.json

@@ -9,9 +9,9 @@
             "console": "integratedTerminal",
             "cwd": "${workspaceFolder}",
             "args": [
-                "-j",
-                "0",
                 //"-nw",
+                "-ed",
+                "-emp",
                 "-a",
                 "ed:wark",
                 "-v",

.vscode/settings.json

@@ -37,7 +37,7 @@
     "python.linting.banditEnabled": true,
     "python.linting.flake8Args": [
         "--max-line-length=120",
-        "--ignore=E722,F405,E203,W503,W293",
+        "--ignore=E722,F405,E203,W503,W293,E402",
     ],
     "python.linting.banditArgs": [
         "--ignore=B104"
@@ -55,6 +55,6 @@
     //
     //  things you may wanna edit:
     //
-    "python.pythonPath": ".venv/bin/python",
+    "python.pythonPath": "/usr/bin/python3",
     //"python.linting.enabled": true,
 }

README.md

@@ -19,6 +19,8 @@ turn your phone or raspi into a portable file server with resumable uploads/down
 * Android-Chrome: set max "parallel uploads" for 200% upload speed (android bug)
 * Android-Firefox: takes a while to select files (in order to avoid the above android-chrome issue)
 * Desktop-Firefox: may use gigabytes of RAM if your connection is great and your files are massive
+* paper-printing is affected by dark/light-mode! use lightmode for color, darkmode for grayscale
+  * because no browsers currently implement the media-query to do this properly orz
 
 
 ## status
@@ -36,10 +38,22 @@ turn your phone or raspi into a portable file server with resumable uploads/down
 * [x] accounts
 * [x] markdown viewer
 * [x] markdown editor
+* [x] FUSE client (read-only)
 
 summary: it works! you can use it! (but technically not even close to beta)
 
 
+# client examples
+
+* javascript: dump some state into a file (two separate examples)
+  * `await fetch('https://127.0.0.1:3923/', {method:"PUT", body: JSON.stringify(foo)});`
+  * `var xhr = new XMLHttpRequest(); xhr.open('POST', 'https://127.0.0.1:3923/msgs?raw'); xhr.send('foo');`
+
+* FUSE: mount a copyparty server as a local filesystem
+  * cross-platform python client available in [./bin/](bin/)
+  * [rclone](https://rclone.org/) as client can give ~5x performance, see [./docs/rclone.md](docs/rclone.md)
+
+
 # dependencies
 
 * `jinja2`
@@ -55,10 +69,13 @@ currently there are two self-contained binaries:
 * `copyparty-sfx.sh` for unix (linux and osx) -- smaller, more robust
 * `copyparty-sfx.py` for windows (unix too) -- crossplatform, beta
 
-launch either of them and it'll unpack and run copyparty, assuming you have python installed of course
+launch either of them (**use sfx.py on systemd**) and it'll unpack and run copyparty, assuming you have python installed of course
 
 pls note that `copyparty-sfx.sh` will fail if you rename `copyparty-sfx.py` to `copyparty.py` and keep it in the same folder because `sys.path` is funky
 
+
+## sfx repack
+
 if you don't need all the features you can repack the sfx and save a bunch of space; all you need is an sfx and a copy of this repo (nothing else to download or build, except for either msys2 or WSL if you're on windows)
 * `724K` original size as of v0.4.0
 * `256K` after `./scripts/make-sfx.sh re no-ogv`
@@ -70,16 +87,18 @@ the features you can opt to drop are
 
 for the `re`pack to work, first run one of the sfx'es once to unpack it
 
+**note:** you can also just download and run [scripts/copyparty-repack.sh](scripts/copyparty-repack.sh) -- this will grab the latest copyparty release from github and do a `no-ogv no-cm` repack; works on linux/macos (and windows with msys2 or WSL)
+
 
 # install on android
 
 install [Termux](https://termux.com/) (see [ocv.me/termux](https://ocv.me/termux/)) and then copy-paste this into Termux (long-tap) all at once:
 ```sh
-apt update && apt -y full-upgrade && termux-setup-storage && apt -y install curl && cd && curl -L https://github.com/9001/copyparty/raw/master/scripts/copyparty-android.sh > copyparty-android.sh && chmod 755 copyparty-android.sh && ./copyparty-android.sh -h
+apt update && apt -y full-upgrade && termux-setup-storage && apt -y install python && python -m ensurepip && python -m pip install -U copyparty
 echo $?
 ```
 
-after the initial setup (and restarting bash), you can launch copyparty at any time by running "copyparty" in Termux
+after the initial setup, you can launch copyparty at any time by running `copyparty` anywhere in Termux
 
 
 # dev env setup

bin/README.md

@@ -0,0 +1,36 @@
+# copyparty-fuse.py
+* mount a copyparty server as a local filesystem (read-only)
+* **supports Windows!** -- expect `194 MiB/s` sequential read
+* **supports Linux** -- expect `117 MiB/s` sequential read
+* **supports macos** -- expect `85 MiB/s` sequential read
+
+filecache is default-on for windows and macos;
+* macos readsize is 64kB, so speed ~32 MiB/s without the cache
+* windows readsize varies by software; explorer=1M, pv=32k
+
+note that copyparty should run with `-ed` to enable dotfiles (hidden otherwise)
+
+also consider using [../docs/rclone.md](../docs/rclone.md) instead for 5x performance
+
+
+## to run this on windows:
+* install [winfsp](https://github.com/billziss-gh/winfsp/releases/latest) and [python 3](https://www.python.org/downloads/)
+  * [x] add python 3.x to PATH (it asks during install)
+* `python -m pip install --user fusepy`
+* `python ./copyparty-fuse.py n: http://192.168.1.69:3923/`
+
+10% faster in [msys2](https://www.msys2.org/), 700% faster if debug prints are enabled:
+* `pacman -S mingw64/mingw-w64-x86_64-python{,-pip}`
+* `/mingw64/bin/python3 -m pip install --user fusepy`
+* `/mingw64/bin/python3 ./copyparty-fuse.py [...]`
+
+you could replace winfsp with [dokan](https://github.com/dokan-dev/dokany/releases/latest), let me know if you [figure out how](https://github.com/dokan-dev/dokany/wiki/FUSE)  
+(winfsp's sshfs leaks, doesn't look like winfsp itself does, should be fine)
+
+
+
+# copyparty-fuse🅱️.py
+* mount a copyparty server as a local filesystem (read-only)
+* does the same thing except more correct, `samba` approves
+* **supports Linux** -- expect `18 MiB/s` (wait what)
+* **supports Macos** -- probably

bin/copyparty-fuse.py

@@ -7,47 +7,83 @@ __copyright__ = 2019
 __license__ = "MIT"
 __url__ = "https://github.com/9001/copyparty/"
 
-import re
-import sys
-import time
-import stat
-import errno
-import struct
-import threading
-import http.client  # py2: httplib
-import urllib.parse
-from datetime import datetime
-from urllib.parse import quote_from_bytes as quote
-
-try:
-    from fuse import FUSE, FuseOSError, Operations
-except:
-    print("\n    could not import fuse;\n    pip install fusepy\n")
-    raise
-
 
 """
 mount a copyparty server (local or remote) as a filesystem
 
 usage:
-  python copyparty-fuse.py ./music http://192.168.1.69:1234/
+  python copyparty-fuse.py ./music http://192.168.1.69:3923/
 
 dependencies:
-  sudo apk add fuse-dev
-  python3 -m venv ~/pe/ve.fusepy
-  . ~/pe/ve.fusepy/bin/activate
-  pip install fusepy
+  python3 -m pip install --user fusepy
+  + on Linux: sudo apk add fuse
+  + on Macos: https://osxfuse.github.io/
+  + on Windows: https://github.com/billziss-gh/winfsp/releases/latest
 
-
-MB/s
- 28   cache NOthread
- 24   cache   thread
- 29   cache NOthread NOmutex
- 67 NOcache NOthread NOmutex  (　´・ω・) nyoro~n
- 10 NOcache   thread NOmutex
+get server cert:
+  awk '/-BEGIN CERTIFICATE-/ {a=1} a; /-END CERTIFICATE-/{exit}' <(openssl s_client -connect 127.0.0.1:3923 </dev/null 2>/dev/null) >cert.pem
 """
 
 
+import re
+import os
+import sys
+import time
+import stat
+import errno
+import struct
+import codecs
+import builtins
+import platform
+import argparse
+import threading
+import traceback
+import http.client  # py2: httplib
+import urllib.parse
+from datetime import datetime
+from urllib.parse import quote_from_bytes as quote
+from urllib.parse import unquote_to_bytes as unquote
+
+WINDOWS = sys.platform == "win32"
+MACOS = platform.system() == "Darwin"
+info = log = dbg = None
+
+
+try:
+    from fuse import FUSE, FuseOSError, Operations
+except:
+    if WINDOWS:
+        libfuse = "install https://github.com/billziss-gh/winfsp/releases/latest"
+    elif MACOS:
+        libfuse = "install https://osxfuse.github.io/"
+    else:
+        libfuse = "apt install libfuse\n    modprobe fuse"
+
+    print(
+        "\n  could not import fuse; these may help:"
+        + "\n    python3 -m pip install --user fusepy\n    "
+        + libfuse
+        + "\n"
+    )
+    raise
+
+
+def print(*args, **kwargs):
+    try:
+        builtins.print(*list(args), **kwargs)
+    except:
+        builtins.print(termsafe(" ".join(str(x) for x in args)), **kwargs)
+
+
+def termsafe(txt):
+    try:
+        return txt.encode(sys.stdout.encoding, "backslashreplace").decode(
+            sys.stdout.encoding
+        )
+    except:
+        return txt.encode(sys.stdout.encoding, "replace").decode(sys.stdout.encoding)
+
+
 def threadless_log(msg):
     print(msg + "\n", end="")
 
@@ -60,27 +96,127 @@ def boring_log(msg):
 def rice_tid():
     tid = threading.current_thread().ident
     c = struct.unpack(b"B" * 5, struct.pack(b">Q", tid)[-5:])
-    return "".join("\033[1;37;48;5;{}m{:02x}".format(x, x) for x in c)
+    return "".join("\033[1;37;48;5;{}m{:02x}".format(x, x) for x in c) + "\033[0m"
 
 
 def fancy_log(msg):
-    print("{}\033[0m {}\n".format(rice_tid(), msg), end="")
+    print("{} {}\n".format(rice_tid(), msg), end="")
 
 
 def null_log(msg):
     pass
 
 
-log = boring_log
-log = fancy_log
-log = threadless_log
-dbg = null_log
+def hexler(binary):
+    return binary.replace("\r", "\\r").replace("\n", "\\n")
+    return " ".join(["{}\033[36m{:02x}\033[0m".format(b, ord(b)) for b in binary])
+    return " ".join(map(lambda b: format(ord(b), "02x"), binary))
+
+
+def register_wtf8():
+    def wtf8_enc(text):
+        return str(text).encode("utf-8", "surrogateescape"), len(text)
+
+    def wtf8_dec(binary):
+        return bytes(binary).decode("utf-8", "surrogateescape"), len(binary)
+
+    def wtf8_search(encoding_name):
+        return codecs.CodecInfo(wtf8_enc, wtf8_dec, name="wtf-8")
+
+    codecs.register(wtf8_search)
+
+
+bad_good = {}
+good_bad = {}
+
+
+def enwin(txt):
+    return "".join([bad_good.get(x, x) for x in txt])
+
+    for bad, good in bad_good.items():
+        txt = txt.replace(bad, good)
+
+    return txt
+
+
+def dewin(txt):
+    return "".join([good_bad.get(x, x) for x in txt])
+
+    for bad, good in bad_good.items():
+        txt = txt.replace(good, bad)
+
+    return txt
+
+
+class RecentLog(object):
+    def __init__(self):
+        self.mtx = threading.Lock()
+        self.f = None  # open("copyparty-fuse.log", "wb")
+        self.q = []
+
+        thr = threading.Thread(target=self.printer)
+        thr.daemon = True
+        thr.start()
+
+    def put(self, msg):
+        msg = "{} {}\n".format(rice_tid(), msg)
+        if self.f:
+            fmsg = " ".join([datetime.utcnow().strftime("%H%M%S.%f"), str(msg)])
+            self.f.write(fmsg.encode("utf-8"))
+
+        with self.mtx:
+            self.q.append(msg)
+            if len(self.q) > 200:
+                self.q = self.q[-50:]
+
+    def printer(self):
+        while True:
+            time.sleep(0.05)
+            with self.mtx:
+                q = self.q
+                if not q:
+                    continue
+
+                self.q = []
+
+            print("".join(q), end="")
+
+
+# [windows/cmd/cpy3]  python dev\copyparty\bin\copyparty-fuse.py q: http://192.168.1.159:1234/
+# [windows/cmd/msys2] C:\msys64\mingw64\bin\python3 dev\copyparty\bin\copyparty-fuse.py q: http://192.168.1.159:1234/
+# [windows/mty/msys2] /mingw64/bin/python3 /c/Users/ed/dev/copyparty/bin/copyparty-fuse.py q: http://192.168.1.159:1234/
+#
+# [windows] find /q/music/albums/Phant*24bit -printf '%s %p\n' | sort -n | tail -n 8 | sed -r 's/^[0-9]+ //' | while IFS= read -r x; do dd if="$x" of=/dev/null bs=4k count=8192 & done
+# [alpine]  ll t; for x in t/2020_0724_16{2,3}*; do dd if="$x" of=/dev/null bs=4k count=10240 & done
+#
+#  72.4983 windows mintty msys2 fancy_log
+# 219.5781 windows cmd msys2 fancy_log
+# nope.avi windows cmd cpy3 fancy_log
+#   9.8817 windows mintty msys2 RecentLog 200 50 0.1
+#  10.2241 windows cmd cpy3 RecentLog 200 50 0.1
+#   9.8494 windows cmd msys2 RecentLog 200 50 0.1
+#   7.8061 windows mintty msys2 fancy_log <info-only>
+#   7.9961 windows mintty msys2 RecentLog <info-only>
+#   4.2603 alpine xfce4 cpy3 RecentLog
+#   4.1538 alpine xfce4 cpy3 fancy_log
+#   3.1742 alpine urxvt cpy3 fancy_log
 
 
 def get_tid():
     return threading.current_thread().ident
 
 
+def html_dec(txt):
+    return (
+        txt.replace("&lt;", "<")
+        .replace("&gt;", ">")
+        .replace("&quot;", '"')
+        .replace("&#13;", "\r")
+        .replace("&#10;", "\n")
+        .replace("&amp;", "&")
+    )
+
+
 class CacheNode(object):
     def __init__(self, tag, data):
         self.tag = tag
@@ -89,10 +225,11 @@ class CacheNode(object):
 
 
 class Gateway(object):
-    def __init__(self, base_url):
-        self.base_url = base_url
+    def __init__(self, ar):
+        self.base_url = ar.base_url
+        self.password = ar.a
 
-        ui = urllib.parse.urlparse(base_url)
+        ui = urllib.parse.urlparse(self.base_url)
         self.web_root = ui.path.strip("/")
         try:
             self.web_host, self.web_port = ui.netloc.split(":")
@@ -102,15 +239,25 @@ class Gateway(object):
             if ui.scheme == "http":
                 self.web_port = 80
             elif ui.scheme == "https":
-                raise Exception("todo")
+                self.web_port = 443
             else:
                 raise Exception("bad url?")
 
+        self.ssl_context = None
+        self.use_tls = ui.scheme.lower() == "https"
+        if self.use_tls:
+            import ssl
+
+            if ar.td:
+                self.ssl_context = ssl._create_unverified_context()
+            elif ar.te:
+                self.ssl_context = ssl.SSLContext(ssl.PROTOCOL_TLS)
+                self.ssl_context.load_verify_locations(ar.te)
+
         self.conns = {}
 
     def quotep(self, path):
-        # TODO: mojibake support
-        path = path.encode("utf-8", "ignore")
+        path = path.encode("wtf-8")
         return quote(path, safe="/")
 
     def getconn(self, tid=None):
@@ -118,9 +265,17 @@ class Gateway(object):
         try:
             return self.conns[tid]
         except:
-            log("new conn [{}] [{}]".format(self.web_host, self.web_port))
+            info("new conn [{}] [{}]".format(self.web_host, self.web_port))
 
-            conn = http.client.HTTPConnection(self.web_host, self.web_port, timeout=260)
+            args = {}
+            if not self.use_tls:
+                C = http.client.HTTPConnection
+            else:
+                C = http.client.HTTPSConnection
+                if self.ssl_context:
+                    args = {"context": self.ssl_context}
+
+            conn = C(self.web_host, self.web_port, timeout=260, **args)
 
             self.conns[tid] = conn
             return conn
@@ -133,42 +288,75 @@ class Gateway(object):
         except:
             pass
 
-    def sendreq(self, *args, **kwargs):
+    def sendreq(self, *args, headers={}, **kwargs):
         tid = get_tid()
+        if self.password:
+            headers["Cookie"] = "=".join(["cppwd", self.password])
+
         try:
             c = self.getconn(tid)
-            c.request(*list(args), **kwargs)
+            c.request(*list(args), headers=headers, **kwargs)
             return c.getresponse()
         except:
-            self.closeconn(tid)
+            dbg("bad conn")
+
+        self.closeconn(tid)
+        try:
             c = self.getconn(tid)
-            c.request(*list(args), **kwargs)
+            c.request(*list(args), headers=headers, **kwargs)
             return c.getresponse()
+        except:
+            info("http connection failed:\n" + traceback.format_exc())
+            if self.use_tls and not self.ssl_context:
+                import ssl
+
+                cert = ssl.get_server_certificate((self.web_host, self.web_port))
+                info("server certificate probably not trusted:\n" + cert)
+
+            raise
 
     def listdir(self, path):
-        web_path = "/" + "/".join([self.web_root, path])
+        if bad_good:
+            path = dewin(path)
 
-        r = self.sendreq("GET", self.quotep(web_path))
+        web_path = self.quotep("/" + "/".join([self.web_root, path])) + "?dots"
+        r = self.sendreq("GET", web_path)
         if r.status != 200:
             self.closeconn()
-            raise Exception(
-                "http error {} reading dir {} in {:x}".format(
+            log(
+                "http error {} reading dir {} in {}".format(
                     r.status, web_path, rice_tid()
                 )
             )
+            raise FuseOSError(errno.ENOENT)
 
-        return self.parse_html(r)
+        if not r.getheader("Content-Type", "").startswith("text/html"):
+            log("listdir on file: {}".format(path))
+            raise FuseOSError(errno.ENOENT)
+
+        try:
+            return self.parse_html(r)
+        except:
+            info(repr(path) + "\n" + traceback.format_exc())
+            raise
 
     def download_file_range(self, path, ofs1, ofs2):
-        web_path = "/" + "/".join([self.web_root, path])
-        hdr_range = "bytes={}-{}".format(ofs1, ofs2)
-        log("downloading {}".format(hdr_range))
+        if bad_good:
+            path = dewin(path)
 
-        r = self.sendreq("GET", self.quotep(web_path), headers={"Range": hdr_range})
+        web_path = self.quotep("/" + "/".join([self.web_root, path])) + "?raw"
+        hdr_range = "bytes={}-{}".format(ofs1, ofs2 - 1)
+        info(
+            "DL {:4.0f}K\033[36m{:>9}-{:<9}\033[0m{}".format(
+                (ofs2 - ofs1) / 1024.0, ofs1, ofs2 - 1, hexler(path)
+            )
+        )
+
+        r = self.sendreq("GET", web_path, headers={"Range": hdr_range})
         if r.status != http.client.PARTIAL_CONTENT:
             self.closeconn()
             raise Exception(
-                "http error {} reading file {} range {} in {:x}".format(
+                "http error {} reading file {} range {} in {}".format(
                     r.status, web_path, hdr_range, rice_tid()
                 )
             )
@@ -179,7 +367,7 @@ class Gateway(object):
         ret = []
         remainder = b""
         ptn = re.compile(
-            r"^<tr><td>(-|DIR)</td><td><a [^>]+>([^<]+)</a></td><td>([^<]+)</td><td>([^<]+)</td></tr>$"
+            r'^<tr><td>(-|DIR)</td><td><a[^>]* href="([^"]+)"[^>]*>([^<]+)</a></td><td>([^<]+)</td><td>([^<]+)</td></tr>$'
         )
 
         while True:
@@ -201,9 +389,22 @@ class Gateway(object):
                     # print(line)
                     continue
 
-                ftype, fname, fsize, fdate = m.groups()
-                ts = datetime.strptime(fdate, "%Y-%m-%d %H:%M:%S").timestamp()
-                sz = int(fsize)
+                ftype, furl, fname, fsize, fdate = m.groups()
+                fname = furl.rstrip("/").split("/")[-1]
+                fname = unquote(fname)
+                fname = fname.decode("wtf-8")
+                if bad_good:
+                    fname = enwin(fname)
+
+                sz = 1
+                ts = 60 * 60 * 24 * 2
+                try:
+                    sz = int(fsize)
+                    ts = datetime.strptime(fdate, "%Y-%m-%d %H:%M:%S").timestamp()
+                except:
+                    info("bad HTML or OS [{}] [{}]".format(fdate, fsize))
+                    # python cannot strptime(1959-01-01) on windows
+
                 if ftype == "-":
                     ret.append([fname, self.stat_file(ts, sz), 0])
                 else:
@@ -213,7 +414,7 @@ class Gateway(object):
 
     def stat_dir(self, ts, sz=4096):
         return {
-            "st_mode": 0o555 | stat.S_IFDIR,
+            "st_mode": stat.S_IFDIR | 0o555,
             "st_uid": 1000,
             "st_gid": 1000,
             "st_size": sz,
@@ -225,7 +426,7 @@ class Gateway(object):
 
     def stat_file(self, ts, sz):
         return {
-            "st_mode": 0o444 | stat.S_IFREG,
+            "st_mode": stat.S_IFREG | 0o444,
             "st_uid": 1000,
             "st_gid": 1000,
             "st_size": sz,
@@ -237,8 +438,11 @@ class Gateway(object):
 
 
 class CPPF(Operations):
-    def __init__(self, base_url):
-        self.gw = Gateway(base_url)
+    def __init__(self, ar):
+        self.gw = Gateway(ar)
+        self.junk_fh_ctr = 3
+        self.n_dircache = ar.cd
+        self.n_filecache = ar.cf
 
         self.dircache = []
         self.dircache_mtx = threading.Lock()
@@ -246,14 +450,29 @@ class CPPF(Operations):
         self.filecache = []
         self.filecache_mtx = threading.Lock()
 
-        log("up")
+        info("up")
+
+    def _describe(self):
+        msg = ""
+        with self.filecache_mtx:
+            for n, cn in enumerate(self.filecache):
+                cache_path, cache1 = cn.tag
+                cache2 = cache1 + len(cn.data)
+                msg += "\n{:<2} {:>7} {:>10}:{:<9} {}".format(
+                    n,
+                    len(cn.data),
+                    cache1,
+                    cache2,
+                    cache_path.replace("\r", "\\r").replace("\n", "\\n"),
+                )
+        return msg
 
     def clean_dircache(self):
         """not threadsafe"""
         now = time.time()
         cutoff = 0
         for cn in self.dircache:
-            if cn.ts - now > 1:
+            if now - cn.ts > self.n_dircache:
                 cutoff += 1
             else:
                 break
@@ -262,8 +481,7 @@ class CPPF(Operations):
             self.dircache = self.dircache[cutoff:]
 
     def get_cached_dir(self, dirpath):
-        # with self.dircache_mtx:
-        if True:
+        with self.dircache_mtx:
             self.clean_dircache()
             for cn in self.dircache:
                 if cn.tag == dirpath:
@@ -300,9 +518,8 @@ class CPPF(Operations):
         car = None
         cdr = None
         ncn = -1
-        # with self.filecache_mtx:
-        if True:
-            dbg("cache request from {} to {}, size {}".format(get1, get2, file_sz))
+        dbg("cache request {}:{} |{}|".format(get1, get2, file_sz) + self._describe())
+        with self.filecache_mtx:
             for cn in self.filecache:
                 ncn += 1
 
@@ -312,6 +529,12 @@ class CPPF(Operations):
 
                 cache2 = cache1 + len(cn.data)
                 if get2 <= cache1 or get1 >= cache2:
+                    # request does not overlap with cached area at all
+                    continue
+
+                if get1 < cache1 and get2 > cache2:
+                    # cached area does overlap, but must specifically contain
+                    # either the first or last byte in the requested range
                     continue
 
                 if get1 >= cache1 and get2 <= cache2:
@@ -322,7 +545,7 @@ class CPPF(Operations):
                     buf_ofs = get1 - cache1
                     buf_end = buf_ofs + (get2 - get1)
                     dbg(
-                        "found all ({}, {} to {}, len {}) [{}:{}] = {}".format(
+                        "found all (#{} {}:{} |{}|) [{}:{}] = {}".format(
                             ncn,
                             cache1,
                             cache2,
@@ -334,11 +557,11 @@ class CPPF(Operations):
                     )
                     return cn.data[buf_ofs:buf_end]
 
-                if get2 < cache2:
+                if get2 <= cache2:
                     x = cn.data[: get2 - cache1]
                     if not cdr or len(cdr) < len(x):
                         dbg(
-                            "found car ({}, {} to {}, len {}) [:{}-{}] = [:{}] = {}".format(
+                            "found cdr (#{} {}:{} |{}|) [:{}-{}] = [:{}] = {}".format(
                                 ncn,
                                 cache1,
                                 cache2,
@@ -353,11 +576,11 @@ class CPPF(Operations):
 
                     continue
 
-                if get1 > cache1:
-                    x = cn.data[-(cache2 - get1) :]
+                if get1 >= cache1:
+                    x = cn.data[-(max(0, cache2 - get1)) :]
                     if not car or len(car) < len(x):
                         dbg(
-                            "found cdr ({}, {} to {}, len {}) [-({}-{}):] = [-{}:] = {}".format(
+                            "found car (#{} {}:{} |{}|) [-({}-{}):] = [-{}:] = {}".format(
                                 ncn,
                                 cache1,
                                 cache2,
@@ -372,38 +595,52 @@ class CPPF(Operations):
 
                     continue
 
-                raise Exception("what")
+                msg = "cache fallthrough\n{} {} {}\n{} {} {}\n{} {} --\n".format(
+                    get1,
+                    get2,
+                    get2 - get1,
+                    cache1,
+                    cache2,
+                    cache2 - cache1,
+                    get1 - cache1,
+                    get2 - cache2,
+                )
+                msg += self._describe()
+                raise Exception(msg)
 
-        if car and cdr:
+        if car and cdr and len(car) + len(cdr) == get2 - get1:
             dbg("<cache> have both")
+            return car + cdr
 
-            ret = car + cdr
-            if len(ret) == get2 - get1:
-                return ret
-
-            raise Exception("{} + {} != {} - {}".format(len(car), len(cdr), get2, get1))
-
-        elif cdr:
+        elif cdr and (not car or len(car) < len(cdr)):
             h_end = get1 + (get2 - get1) - len(cdr)
-            h_ofs = h_end - 512 * 1024
+            h_ofs = min(get1, h_end - 512 * 1024)
 
             if h_ofs < 0:
                 h_ofs = 0
 
-            buf_ofs = (get2 - get1) - len(cdr)
+            buf_ofs = get1 - h_ofs
 
             dbg(
-                "<cache> cdr {}, car {}-{}={} [-{}:]".format(
+                "<cache> cdr {}, car {}:{} |{}| [{}:]".format(
                     len(cdr), h_ofs, h_end, h_end - h_ofs, buf_ofs
                 )
             )
 
-            buf = self.gw.download_file_range(path, h_ofs, h_end - 1)
-            ret = buf[-buf_ofs:] + cdr
+            buf = self.gw.download_file_range(path, h_ofs, h_end)
+            if len(buf) == h_end - h_ofs:
+                ret = buf[buf_ofs:] + cdr
+            else:
+                ret = buf[get1 - h_ofs :]
+                info(
+                    "remote truncated {}:{} to |{}|, will return |{}|".format(
+                        h_ofs, h_end, len(buf), len(ret)
+                    )
+                )
 
         elif car:
             h_ofs = get1 + len(car)
-            h_end = h_ofs + 1024 * 1024
+            h_end = max(get2, h_ofs + 1024 * 1024)
 
             if h_end > file_sz:
                 h_end = file_sz
@@ -411,17 +648,22 @@ class CPPF(Operations):
             buf_ofs = (get2 - get1) - len(car)
 
             dbg(
-                "<cache> car {}, cdr {}-{}={} [:{}]".format(
+                "<cache> car {}, cdr {}:{} |{}| [:{}]".format(
                     len(car), h_ofs, h_end, h_end - h_ofs, buf_ofs
                 )
             )
 
-            buf = self.gw.download_file_range(path, h_ofs, h_end - 1)
+            buf = self.gw.download_file_range(path, h_ofs, h_end)
             ret = car + buf[:buf_ofs]
 
         else:
-            h_ofs = get1 - 256 * 1024
-            h_end = get2 + 1024 * 1024
+            if get2 - get1 <= 1024 * 1024:
+                h_ofs = get1 - 256 * 1024
+                h_end = get2 + 1024 * 1024
+            else:
+                # big enough, doesn't need pads
+                h_ofs = get1
+                h_end = get2
 
             if h_ofs < 0:
                 h_ofs = 0
@@ -433,54 +675,99 @@ class CPPF(Operations):
             buf_end = buf_ofs + get2 - get1
 
             dbg(
-                "<cache> {}-{}={} [{}:{}]".format(
+                "<cache> {}:{} |{}| [{}:{}]".format(
                     h_ofs, h_end, h_end - h_ofs, buf_ofs, buf_end
                 )
             )
 
-            buf = self.gw.download_file_range(path, h_ofs, h_end - 1)
+            buf = self.gw.download_file_range(path, h_ofs, h_end)
             ret = buf[buf_ofs:buf_end]
 
         cn = CacheNode([path, h_ofs], buf)
-        # with self.filecache_mtx:
-        if True:
-            if len(self.filecache) > 6:
+        with self.filecache_mtx:
+            if len(self.filecache) >= self.n_filecache:
                 self.filecache = self.filecache[1:] + [cn]
             else:
                 self.filecache.append(cn)
 
         return ret
 
-    def readdir(self, path, fh=None):
+    def _readdir(self, path, fh=None):
         path = path.strip("/")
-        log("readdir {}".format(path))
+        log("readdir [{}] [{}]".format(hexler(path), fh))
 
         ret = self.gw.listdir(path)
+        if not self.n_dircache:
+            return ret
 
-        # with self.dircache_mtx:
-        if True:
+        with self.dircache_mtx:
             cn = CacheNode(path, ret)
             self.dircache.append(cn)
             self.clean_dircache()
 
         return ret
 
+    def readdir(self, path, fh=None):
+        return [".", ".."] + self._readdir(path, fh)
+
     def read(self, path, length, offset, fh=None):
+        req_max = 1024 * 1024 * 8
+        cache_max = 1024 * 1024 * 2
+        if length > req_max:
+            # windows actually doing 240 MiB read calls, sausage
+            info("truncate |{}| to {}MiB".format(length, req_max >> 20))
+            length = req_max
+
         path = path.strip("/")
-
         ofs2 = offset + length
-        log("read {} @ {} len {} end {}".format(path, offset, length, ofs2))
-
         file_sz = self.getattr(path)["st_size"]
-        if ofs2 >= file_sz:
-            ofs2 = file_sz - 1
-            log("truncate to len {} end {}".format((ofs2 - offset) + 1, ofs2))
+        log(
+            "read {} |{}| {}:{} max {}".format(
+                hexler(path), length, offset, ofs2, file_sz
+            )
+        )
+        if ofs2 > file_sz:
+            ofs2 = file_sz
+            log("truncate to |{}| :{}".format(ofs2 - offset, ofs2))
 
-        # toggle cache here i suppose
-        # return self.get_cached_file(path, offset, ofs2, file_sz)
-        return self.gw.download_file_range(path, offset, ofs2 - 1)
+        if file_sz == 0 or offset >= ofs2:
+            return b""
+
+        if self.n_filecache and length <= cache_max:
+            ret = self.get_cached_file(path, offset, ofs2, file_sz)
+        else:
+            ret = self.gw.download_file_range(path, offset, ofs2)
+
+        return ret
+
+        fn = "cppf-{}-{}-{}".format(time.time(), offset, length)
+        if False:
+            with open(fn, "wb", len(ret)) as f:
+                f.write(ret)
+        elif self.n_filecache:
+            ret2 = self.gw.download_file_range(path, offset, ofs2)
+            if ret != ret2:
+                info(fn)
+                for v in [ret, ret2]:
+                    try:
+                        info(len(v))
+                    except:
+                        info("uhh " + repr(v))
+
+                with open(fn + ".bad", "wb") as f:
+                    f.write(ret)
+                with open(fn + ".good", "wb") as f:
+                    f.write(ret2)
+
+                raise Exception("cache bork")
+
+        return ret
 
     def getattr(self, path, fh=None):
+        log("getattr [{}]".format(hexler(path)))
+        if WINDOWS:
+            path = enwin(path)  # windows occasionally decodes f0xx to xx
+
         path = path.strip("/")
         try:
             dirpath, fname = path.rsplit("/", 1)
@@ -488,23 +775,34 @@ class CPPF(Operations):
             dirpath = ""
             fname = path
 
-        log("getattr {}".format(path))
-
         if not path:
-            return self.gw.stat_dir(time.time())
+            ret = self.gw.stat_dir(time.time())
+            # dbg("=" + repr(ret))
+            return ret
 
         cn = self.get_cached_dir(dirpath)
         if cn:
-            # log('cache ok')
+            log("cache ok")
             dents = cn.data
         else:
-            log("cache miss")
-            dents = self.readdir(dirpath)
+            dbg("cache miss")
+            dents = self._readdir(dirpath)
 
         for cache_name, cache_stat, _ in dents:
+            # if "qw" in cache_name and "qw" in fname:
+            #     info(
+            #         "cmp\n  [{}]\n  [{}]\n\n{}\n".format(
+            #             hexler(cache_name),
+            #             hexler(fname),
+            #             "\n".join(traceback.format_stack()[:-1]),
+            #         )
+            #     )
+
             if cache_name == fname:
+                # dbg("=" + repr(cache_stat))
                 return cache_stat
 
+        info("=ENOENT ({})".format(hexler(path)))
         raise FuseOSError(errno.ENOENT)
 
     access = None
@@ -517,17 +815,178 @@ class CPPF(Operations):
     releasedir = None
     statfs = None
 
+    if False:
+        # incorrect semantics but good for debugging stuff like samba and msys2
+        def access(self, path, mode):
+            log("@@ access [{}] [{}]".format(path, mode))
+            return 1 if self.getattr(path) else 0
+
+        def flush(self, path, fh):
+            log("@@ flush [{}] [{}]".format(path, fh))
+            return True
+
+        def getxattr(self, *args):
+            log("@@ getxattr [{}]".format("] [".join(str(x) for x in args)))
+            return False
+
+        def listxattr(self, *args):
+            log("@@ listxattr [{}]".format("] [".join(str(x) for x in args)))
+            return False
+
+        def open(self, path, flags):
+            log("@@ open [{}] [{}]".format(path, flags))
+            return 42
+
+        def opendir(self, fh):
+            log("@@ opendir [{}]".format(fh))
+            return 69
+
+        def release(self, ino, fi):
+            log("@@ release [{}] [{}]".format(ino, fi))
+            return True
+
+        def releasedir(self, ino, fi):
+            log("@@ releasedir [{}] [{}]".format(ino, fi))
+            return True
+
+        def statfs(self, path):
+            log("@@ statfs [{}]".format(path))
+            return {}
+
+    if sys.platform == "win32":
+        # quick compat for /mingw64/bin/python3 (msys2)
+        def _open(self, path):
+            try:
+                x = self.getattr(path)
+                if x["st_mode"] <= 0:
+                    raise Exception()
+
+                self.junk_fh_ctr += 1
+                if self.junk_fh_ctr > 32000:  # TODO untested
+                    self.junk_fh_ctr = 4
+
+                return self.junk_fh_ctr
+
+            except Exception as ex:
+                log("open ERR {}".format(repr(ex)))
+                raise FuseOSError(errno.ENOENT)
+
+        def open(self, path, flags):
+            dbg("open [{}] [{}]".format(hexler(path), flags))
+            return self._open(path)
+
+        def opendir(self, path):
+            dbg("opendir [{}]".format(hexler(path)))
+            return self._open(path)
+
+        def flush(self, path, fh):
+            dbg("flush [{}] [{}]".format(hexler(path), fh))
+
+        def release(self, ino, fi):
+            dbg("release [{}] [{}]".format(hexler(ino), fi))
+
+        def releasedir(self, ino, fi):
+            dbg("releasedir [{}] [{}]".format(hexler(ino), fi))
+
+        def access(self, path, mode):
+            dbg("access [{}] [{}]".format(hexler(path), mode))
+            try:
+                x = self.getattr(path)
+                if x["st_mode"] <= 0:
+                    raise Exception()
+            except:
+                raise FuseOSError(errno.ENOENT)
+
+
+class TheArgparseFormatter(
+    argparse.RawTextHelpFormatter, argparse.ArgumentDefaultsHelpFormatter
+):
+    pass
+
 
 def main():
-    try:
-        local, remote = sys.argv[1:]
-    except:
-        print("need arg 1: local directory")
-        print("need arg 2: root url")
-        return
+    global info, log, dbg
 
-    FUSE(CPPF(remote), local, foreground=True, nothreads=True)
-    # if nothreads=False also uncomment the `with *_mtx` things
+    # filecache helps for reads that are ~64k or smaller;
+    #   linux generally does 128k so the cache is a slowdown,
+    #   windows likes to use 4k and 64k so cache is required,
+    #   value is numChunks (1~3M each) to keep in the cache
+    nf = 24
+
+    # dircache is always a boost,
+    #   only want to disable it for tests etc,
+    #   value is numSec until an entry goes stale
+    nd = 1
+
+    where = "local directory"
+    if WINDOWS:
+        where += " or DRIVE:"
+
+    ex_pre = "\n  " + os.path.basename(__file__) + "  "
+    examples = ["http://192.168.1.69:3923/music/  ./music"]
+    if WINDOWS:
+        examples.append("http://192.168.1.69:3923/music/  M:")
+
+    ap = argparse.ArgumentParser(
+        formatter_class=TheArgparseFormatter,
+        epilog="example:" + ex_pre + ex_pre.join(examples),
+    )
+    ap.add_argument(
+        "-cd", metavar="NUM_SECONDS", type=float, default=nd, help="directory cache"
+    )
+    ap.add_argument(
+        "-cf", metavar="NUM_BLOCKS", type=int, default=nf, help="file cache"
+    )
+    ap.add_argument("-a", metavar="PASSWORD", help="password")
+    ap.add_argument("-d", action="store_true", help="enable debug")
+    ap.add_argument("-te", metavar="PEM_FILE", help="certificate to expect/verify")
+    ap.add_argument("-td", action="store_true", help="disable certificate check")
+    ap.add_argument("base_url", type=str, help="remote copyparty URL to mount")
+    ap.add_argument("local_path", type=str, help=where + " to mount it on")
+    ar = ap.parse_args()
+
+    if ar.d:
+        # windows terminals are slow (cmd.exe, mintty)
+        # otoh fancy_log beats RecentLog on linux
+        logger = RecentLog().put if WINDOWS else fancy_log
+
+        info = logger
+        log = logger
+        dbg = logger
+    else:
+        # debug=off, speed is dontcare
+        info = fancy_log
+        log = null_log
+        dbg = null_log
+
+    if WINDOWS:
+        os.system("")
+
+        for ch in '<>:"\\|?*':
+            # microsoft maps illegal characters to f0xx
+            # (e000 to f8ff is basic-plane private-use)
+            bad_good[ch] = chr(ord(ch) + 0xF000)
+
+        for n in range(0, 0x100):
+            # map surrogateescape to another private-use area
+            bad_good[chr(n + 0xDC00)] = chr(n + 0xF100)
+
+        for k, v in bad_good.items():
+            good_bad[v] = k
+
+    register_wtf8()
+
+    try:
+        with open("/etc/fuse.conf", "rb") as f:
+            allow_other = b"\nuser_allow_other" in f.read()
+    except:
+        allow_other = WINDOWS or MACOS
+
+    args = {"foreground": True, "nothreads": True, "allow_other": allow_other}
+    if not MACOS:
+        args["nonempty"] = True
+
+    FUSE(CPPF(ar), ar.local_path, encoding="wtf-8", **args)
 
 
 if __name__ == "__main__":

bin/copyparty-fuseb.py

@@ -0,0 +1,590 @@
+#!/usr/bin/env python3
+from __future__ import print_function, unicode_literals
+
+"""copyparty-fuseb: remote copyparty as a local filesystem"""
+__author__ = "ed <copyparty@ocv.me>"
+__copyright__ = 2020
+__license__ = "MIT"
+__url__ = "https://github.com/9001/copyparty/"
+
+import re
+import os
+import sys
+import time
+import stat
+import errno
+import struct
+import threading
+import http.client  # py2: httplib
+import urllib.parse
+from datetime import datetime
+from urllib.parse import quote_from_bytes as quote
+
+try:
+    import fuse
+    from fuse import Fuse
+
+    fuse.fuse_python_api = (0, 2)
+    if not hasattr(fuse, "__version__"):
+        raise Exception("your fuse-python is way old")
+except:
+    print(
+        "\n  could not import fuse; these may help:\n    python3 -m pip install --user fuse-python\n    apt install libfuse\n    modprobe fuse\n"
+    )
+    raise
+
+
+"""
+mount a copyparty server (local or remote) as a filesystem
+
+usage:
+  python ./copyparty-fuseb.py -f -o allow_other,auto_unmount,nonempty,url=http://192.168.1.69:3923 /mnt/nas
+
+dependencies:
+  sudo apk add fuse-dev python3-dev
+  python3 -m pip install --user fuse-python
+
+fork of copyparty-fuse.py based on fuse-python which
+  appears to be more compliant than fusepy? since this works with samba
+    (probably just my garbage code tbh)
+"""
+
+
+def threadless_log(msg):
+    print(msg + "\n", end="")
+
+
+def boring_log(msg):
+    msg = "\033[36m{:012x}\033[0m {}\n".format(threading.current_thread().ident, msg)
+    print(msg[4:], end="")
+
+
+def rice_tid():
+    tid = threading.current_thread().ident
+    c = struct.unpack(b"B" * 5, struct.pack(b">Q", tid)[-5:])
+    return "".join("\033[1;37;48;5;{}m{:02x}".format(x, x) for x in c) + "\033[0m"
+
+
+def fancy_log(msg):
+    print("{} {}\n".format(rice_tid(), msg), end="")
+
+
+def null_log(msg):
+    pass
+
+
+info = fancy_log
+log = fancy_log
+dbg = fancy_log
+log = null_log
+dbg = null_log
+
+
+def get_tid():
+    return threading.current_thread().ident
+
+
+def html_dec(txt):
+    return (
+        txt.replace("&lt;", "<")
+        .replace("&gt;", ">")
+        .replace("&quot;", '"')
+        .replace("&amp;", "&")
+    )
+
+
+class CacheNode(object):
+    def __init__(self, tag, data):
+        self.tag = tag
+        self.data = data
+        self.ts = time.time()
+
+
+class Stat(fuse.Stat):
+    def __init__(self):
+        self.st_mode = 0
+        self.st_ino = 0
+        self.st_dev = 0
+        self.st_nlink = 1
+        self.st_uid = 1000
+        self.st_gid = 1000
+        self.st_size = 0
+        self.st_atime = 0
+        self.st_mtime = 0
+        self.st_ctime = 0
+
+
+class Gateway(object):
+    def __init__(self, base_url):
+        self.base_url = base_url
+
+        ui = urllib.parse.urlparse(base_url)
+        self.web_root = ui.path.strip("/")
+        try:
+            self.web_host, self.web_port = ui.netloc.split(":")
+            self.web_port = int(self.web_port)
+        except:
+            self.web_host = ui.netloc
+            if ui.scheme == "http":
+                self.web_port = 80
+            elif ui.scheme == "https":
+                raise Exception("todo")
+            else:
+                raise Exception("bad url?")
+
+        self.conns = {}
+
+    def quotep(self, path):
+        # TODO: mojibake support
+        path = path.encode("utf-8", "ignore")
+        return quote(path, safe="/")
+
+    def getconn(self, tid=None):
+        tid = tid or get_tid()
+        try:
+            return self.conns[tid]
+        except:
+            info("new conn [{}] [{}]".format(self.web_host, self.web_port))
+
+            conn = http.client.HTTPConnection(self.web_host, self.web_port, timeout=260)
+
+            self.conns[tid] = conn
+            return conn
+
+    def closeconn(self, tid=None):
+        tid = tid or get_tid()
+        try:
+            self.conns[tid].close()
+            del self.conns[tid]
+        except:
+            pass
+
+    def sendreq(self, *args, **kwargs):
+        tid = get_tid()
+        try:
+            c = self.getconn(tid)
+            c.request(*list(args), **kwargs)
+            return c.getresponse()
+        except:
+            self.closeconn(tid)
+            c = self.getconn(tid)
+            c.request(*list(args), **kwargs)
+            return c.getresponse()
+
+    def listdir(self, path):
+        web_path = self.quotep("/" + "/".join([self.web_root, path])) + "?dots"
+        r = self.sendreq("GET", web_path)
+        if r.status != 200:
+            self.closeconn()
+            raise Exception(
+                "http error {} reading dir {} in {}".format(
+                    r.status, web_path, rice_tid()
+                )
+            )
+
+        return self.parse_html(r)
+
+    def download_file_range(self, path, ofs1, ofs2):
+        web_path = self.quotep("/" + "/".join([self.web_root, path])) + "?raw"
+        hdr_range = "bytes={}-{}".format(ofs1, ofs2 - 1)
+        log("downloading {}".format(hdr_range))
+
+        r = self.sendreq("GET", web_path, headers={"Range": hdr_range})
+        if r.status != http.client.PARTIAL_CONTENT:
+            self.closeconn()
+            raise Exception(
+                "http error {} reading file {} range {} in {}".format(
+                    r.status, web_path, hdr_range, rice_tid()
+                )
+            )
+
+        return r.read()
+
+    def parse_html(self, datasrc):
+        ret = []
+        remainder = b""
+        ptn = re.compile(
+            r"^<tr><td>(-|DIR)</td><td><a [^>]+>([^<]+)</a></td><td>([^<]+)</td><td>([^<]+)</td></tr>$"
+        )
+
+        while True:
+            buf = remainder + datasrc.read(4096)
+            # print('[{}]'.format(buf.decode('utf-8')))
+            if not buf:
+                break
+
+            remainder = b""
+            endpos = buf.rfind(b"\n")
+            if endpos >= 0:
+                remainder = buf[endpos + 1 :]
+                buf = buf[:endpos]
+
+            lines = buf.decode("utf-8").split("\n")
+            for line in lines:
+                m = ptn.match(line)
+                if not m:
+                    # print(line)
+                    continue
+
+                ftype, fname, fsize, fdate = m.groups()
+                fname = html_dec(fname)
+                ts = datetime.strptime(fdate, "%Y-%m-%d %H:%M:%S").timestamp()
+                sz = int(fsize)
+                if ftype == "-":
+                    ret.append([fname, self.stat_file(ts, sz), 0])
+                else:
+                    ret.append([fname, self.stat_dir(ts, sz), 0])
+
+        return ret
+
+    def stat_dir(self, ts, sz=4096):
+        ret = Stat()
+        ret.st_mode = stat.S_IFDIR | 0o555
+        ret.st_nlink = 2
+        ret.st_size = sz
+        ret.st_atime = ts
+        ret.st_mtime = ts
+        ret.st_ctime = ts
+        return ret
+
+    def stat_file(self, ts, sz):
+        ret = Stat()
+        ret.st_mode = stat.S_IFREG | 0o444
+        ret.st_size = sz
+        ret.st_atime = ts
+        ret.st_mtime = ts
+        ret.st_ctime = ts
+        return ret
+
+
+class CPPF(Fuse):
+    def __init__(self, *args, **kwargs):
+        Fuse.__init__(self, *args, **kwargs)
+
+        self.url = None
+
+        self.dircache = []
+        self.dircache_mtx = threading.Lock()
+
+        self.filecache = []
+        self.filecache_mtx = threading.Lock()
+
+    def init2(self):
+        # TODO figure out how python-fuse wanted this to go
+        self.gw = Gateway(self.url)  # .decode('utf-8'))
+        info("up")
+
+    def clean_dircache(self):
+        """not threadsafe"""
+        now = time.time()
+        cutoff = 0
+        for cn in self.dircache:
+            if now - cn.ts > 1:
+                cutoff += 1
+            else:
+                break
+
+        if cutoff > 0:
+            self.dircache = self.dircache[cutoff:]
+
+    def get_cached_dir(self, dirpath):
+        # with self.dircache_mtx:
+        if True:
+            self.clean_dircache()
+            for cn in self.dircache:
+                if cn.tag == dirpath:
+                    return cn
+
+        return None
+
+    """
+            ,-------------------------------,  g1>=c1, g2<=c2
+            |cache1                   cache2|  buf[g1-c1:(g1-c1)+(g2-g1)]
+            `-------------------------------'
+                    ,---------------,
+                    |get1       get2|
+                    `---------------'
+    __________________________________________________________________________
+
+            ,-------------------------------,  g2<=c2, (g2>=c1)
+            |cache1                   cache2|  cdr=buf[:g2-c1]
+            `-------------------------------'  dl car; g1-512K:c1
+    ,---------------,
+    |get1       get2|
+    `---------------'
+    __________________________________________________________________________
+
+            ,-------------------------------,  g1>=c1, (g1<=c2)
+            |cache1                   cache2|  car=buf[c2-g1:]
+            `-------------------------------'  dl cdr; c2:c2+1M
+                                    ,---------------,
+                                    |get1       get2|
+                                    `---------------'
+    """
+
+    def get_cached_file(self, path, get1, get2, file_sz):
+        car = None
+        cdr = None
+        ncn = -1
+        # with self.filecache_mtx:
+        if True:
+            dbg("cache request from {} to {}, size {}".format(get1, get2, file_sz))
+            for cn in self.filecache:
+                ncn += 1
+
+                cache_path, cache1 = cn.tag
+                if cache_path != path:
+                    continue
+
+                cache2 = cache1 + len(cn.data)
+                if get2 <= cache1 or get1 >= cache2:
+                    continue
+
+                if get1 >= cache1 and get2 <= cache2:
+                    # keep cache entry alive by moving it to the end
+                    self.filecache = (
+                        self.filecache[:ncn] + self.filecache[ncn + 1 :] + [cn]
+                    )
+                    buf_ofs = get1 - cache1
+                    buf_end = buf_ofs + (get2 - get1)
+                    dbg(
+                        "found all ({}, {} to {}, len {}) [{}:{}] = {}".format(
+                            ncn,
+                            cache1,
+                            cache2,
+                            len(cn.data),
+                            buf_ofs,
+                            buf_end,
+                            buf_end - buf_ofs,
+                        )
+                    )
+                    return cn.data[buf_ofs:buf_end]
+
+                if get2 < cache2:
+                    x = cn.data[: get2 - cache1]
+                    if not cdr or len(cdr) < len(x):
+                        dbg(
+                            "found car ({}, {} to {}, len {}) [:{}-{}] = [:{}] = {}".format(
+                                ncn,
+                                cache1,
+                                cache2,
+                                len(cn.data),
+                                get2,
+                                cache1,
+                                get2 - cache1,
+                                len(x),
+                            )
+                        )
+                        cdr = x
+
+                    continue
+
+                if get1 > cache1:
+                    x = cn.data[-(cache2 - get1) :]
+                    if not car or len(car) < len(x):
+                        dbg(
+                            "found cdr ({}, {} to {}, len {}) [-({}-{}):] = [-{}:] = {}".format(
+                                ncn,
+                                cache1,
+                                cache2,
+                                len(cn.data),
+                                cache2,
+                                get1,
+                                cache2 - get1,
+                                len(x),
+                            )
+                        )
+                        car = x
+
+                    continue
+
+                raise Exception("what")
+
+        if car and cdr:
+            dbg("<cache> have both")
+
+            ret = car + cdr
+            if len(ret) == get2 - get1:
+                return ret
+
+            raise Exception("{} + {} != {} - {}".format(len(car), len(cdr), get2, get1))
+
+        elif cdr:
+            h_end = get1 + (get2 - get1) - len(cdr)
+            h_ofs = h_end - 512 * 1024
+
+            if h_ofs < 0:
+                h_ofs = 0
+
+            buf_ofs = (get2 - get1) - len(cdr)
+
+            dbg(
+                "<cache> cdr {}, car {}-{}={} [-{}:]".format(
+                    len(cdr), h_ofs, h_end, h_end - h_ofs, buf_ofs
+                )
+            )
+
+            buf = self.gw.download_file_range(path, h_ofs, h_end)
+            ret = buf[-buf_ofs:] + cdr
+
+        elif car:
+            h_ofs = get1 + len(car)
+            h_end = h_ofs + 1024 * 1024
+
+            if h_end > file_sz:
+                h_end = file_sz
+
+            buf_ofs = (get2 - get1) - len(car)
+
+            dbg(
+                "<cache> car {}, cdr {}-{}={} [:{}]".format(
+                    len(car), h_ofs, h_end, h_end - h_ofs, buf_ofs
+                )
+            )
+
+            buf = self.gw.download_file_range(path, h_ofs, h_end)
+            ret = car + buf[:buf_ofs]
+
+        else:
+            h_ofs = get1 - 256 * 1024
+            h_end = get2 + 1024 * 1024
+
+            if h_ofs < 0:
+                h_ofs = 0
+
+            if h_end > file_sz:
+                h_end = file_sz
+
+            buf_ofs = get1 - h_ofs
+            buf_end = buf_ofs + get2 - get1
+
+            dbg(
+                "<cache> {}-{}={} [{}:{}]".format(
+                    h_ofs, h_end, h_end - h_ofs, buf_ofs, buf_end
+                )
+            )
+
+            buf = self.gw.download_file_range(path, h_ofs, h_end)
+            ret = buf[buf_ofs:buf_end]
+
+        cn = CacheNode([path, h_ofs], buf)
+        # with self.filecache_mtx:
+        if True:
+            if len(self.filecache) > 6:
+                self.filecache = self.filecache[1:] + [cn]
+            else:
+                self.filecache.append(cn)
+
+        return ret
+
+    def _readdir(self, path):
+        path = path.strip("/")
+        log("readdir {}".format(path))
+
+        ret = self.gw.listdir(path)
+
+        # with self.dircache_mtx:
+        if True:
+            cn = CacheNode(path, ret)
+            self.dircache.append(cn)
+            self.clean_dircache()
+
+        return ret
+
+    def readdir(self, path, offset):
+        for e in self._readdir(path)[offset:]:
+            # log("yield [{}]".format(e[0]))
+            yield fuse.Direntry(e[0])
+
+    def open(self, path, flags):
+        if (flags & (os.O_RDONLY | os.O_WRONLY | os.O_RDWR)) != os.O_RDONLY:
+            return -errno.EACCES
+
+        st = self.getattr(path)
+        try:
+            if st.st_nlink > 0:
+                return st
+        except:
+            return st  # -int(os.errcode)
+
+    def read(self, path, length, offset, fh=None, *args):
+        if args:
+            log("unexpected args [" + "] [".join(repr(x) for x in args) + "]")
+            raise Exception()
+
+        path = path.strip("/")
+
+        ofs2 = offset + length
+        log("read {} @ {} len {} end {}".format(path, offset, length, ofs2))
+
+        st = self.getattr(path)
+        try:
+            file_sz = st.st_size
+        except:
+            return st  # -int(os.errcode)
+
+        if ofs2 > file_sz:
+            ofs2 = file_sz
+            log("truncate to len {} end {}".format(ofs2 - offset, ofs2))
+
+        if file_sz == 0 or offset >= ofs2:
+            return b""
+
+        # toggle cache here i suppose
+        # return self.get_cached_file(path, offset, ofs2, file_sz)
+        return self.gw.download_file_range(path, offset, ofs2)
+
+    def getattr(self, path):
+        log("getattr [{}]".format(path))
+
+        path = path.strip("/")
+        try:
+            dirpath, fname = path.rsplit("/", 1)
+        except:
+            dirpath = ""
+            fname = path
+
+        if not path:
+            ret = self.gw.stat_dir(time.time())
+            dbg("=root")
+            return ret
+
+        cn = self.get_cached_dir(dirpath)
+        if cn:
+            log("cache ok")
+            dents = cn.data
+        else:
+            log("cache miss")
+            dents = self._readdir(dirpath)
+
+        for cache_name, cache_stat, _ in dents:
+            if cache_name == fname:
+                dbg("=file")
+                return cache_stat
+
+        log("=404")
+        return -errno.ENOENT
+
+
+def main():
+    server = CPPF()
+    server.parser.add_option(mountopt="url", metavar="BASE_URL", default=None)
+    server.parse(values=server, errex=1)
+    if not server.url or not str(server.url).startswith("http"):
+        print("\nerror:")
+        print("  need argument: -o url=<...>")
+        print("  need argument: mount-path")
+        print("example:")
+        print(
+            "  ./copyparty-fuseb.py -f -o allow_other,auto_unmount,nonempty,url=http://192.168.1.69:3923 /mnt/nas"
+        )
+        sys.exit(1)
+
+    server.init2()
+    threading.Thread(target=server.main, daemon=True).start()
+    while True:
+        time.sleep(9001)
+
+
+if __name__ == "__main__":
+    main()

bin/up2k.sh

@@ -118,7 +118,7 @@ printf ']}' >> /dev/shm/$salt.hs
 
 printf '\033[36m'
 
-#curl "http://$target:1234$posturl/handshake.php" -H "Content-Type: text/plain;charset=UTF-8" -H "Cookie: cppwd=$passwd" --data "$(cat "/dev/shm/$salt.hs")" | tee /dev/shm/$salt.res
+#curl "http://$target:3923$posturl/handshake.php" -H "Content-Type: text/plain;charset=UTF-8" -H "Cookie: cppwd=$passwd" --data "$(cat "/dev/shm/$salt.hs")" | tee /dev/shm/$salt.res
 
 {
     {
@@ -135,7 +135,7 @@ EOF
     cat /dev/shm/$salt.hs
 } |
 tee /dev/shm/$salt.hsb |
-ncat $target 1234 |
+ncat $target 3923 |
 tee /dev/shm/$salt.hs1r
 
 wark="$(cat /dev/shm/$salt.hs1r | getwark)"
@@ -190,7 +190,7 @@ EOF
     nchunk=$((nchunk+1))
 
 done |
-ncat $target 1234 |
+ncat $target 3923 |
 tee /dev/shm/$salt.pr
 
 t=$(date +%s.%N)
@@ -201,7 +201,7 @@ t=$(date +%s.%N)
 
 printf '\033[36m'
 
-ncat $target 1234 < /dev/shm/$salt.hsb |
+ncat $target 3923 < /dev/shm/$salt.hsb |
 tee /dev/shm/$salt.hs2r |
 grep -E '"hash": ?\[ *\]'
 

contrib/README.md

@@ -0,0 +1,19 @@
+### [`copyparty.bat`](copyparty.bat)
+* launches copyparty with no arguments (anon read+write within same folder)
+* intended for windows machines with no python.exe in PATH
+* works on windows, linux and macos
+* assumes `copyparty-sfx.py` was renamed to `copyparty.py` in the same folder as `copyparty.bat`
+
+### [`index.html`](index.html)
+* drop-in redirect from an httpd to copyparty
+* assumes the webserver and copyparty is running on the same server/IP
+* modify `10.13.1.1` as necessary if you wish to support browsers without javascript
+
+# OS integration
+init-scripts to start copyparty as a service
+* [`systemd/copyparty.service`](systemd/copyparty.service)
+* [`openrc/copyparty`](openrc/copyparty)
+
+# Reverse-proxy
+copyparty has basic support for running behind another webserver
+* [`nginx/copyparty.conf`](nginx/copyparty.conf)

contrib/copyparty.bat

@@ -0,0 +1,33 @@
+exec python "$(dirname "$0")"/copyparty.py
+
+@rem on linux, the above will execute and the script will terminate
+@rem on windows, the rest of this script will run
+
+@echo off
+cls
+
+set py=
+for /f %%i in ('where python 2^>nul') do (
+    set "py=%%i"
+    goto c1
+)
+:c1
+
+if [%py%] == [] (
+    for /f %%i in ('where /r "%localappdata%\programs\python" python 2^>nul') do (
+        set "py=%%i"
+        goto c2
+    )
+)
+:c2
+
+if [%py%] == [] set "py=c:\python27\python.exe"
+
+if not exist "%py%" (
+    echo could not find python
+    echo(
+    pause
+    exit /b
+)
+
+start cmd /c %py% "%~dp0\copyparty.py"

contrib/index.html

@@ -0,0 +1,43 @@
+<!DOCTYPE html>
+<html lang="en">
+
+<head>
+	<meta charset="utf-8">
+	<title>⇆🎉 redirect</title>
+	<meta http-equiv="X-UA-Compatible" content="IE=edge">
+	<style>
+
+html, body {
+	font-family: sans-serif;
+}
+body {
+	padding: 1em 2em;
+	font-size: 1.5em;
+}
+a {
+	font-size: 1.2em;
+	padding: .1em;
+}
+
+</style>
+</head>
+<body>
+	<span id="desc">you probably want</span> <a id="redir" href="//10.13.1.1:3923/">copyparty</a>
+	<script>
+
+var a = document.getElementById('redir'),
+	proto = window.location.protocol.indexOf('https') === 0 ? 'https' : 'http',
+	loc = window.location.hostname || '127.0.0.1',
+	port = a.getAttribute('href').split(':').pop().split('/')[0],
+	url = proto + '://' + loc + ':' + port + '/';
+
+a.setAttribute('href', url);
+document.getElementById('desc').innerHTML = 'redirecting to';
+
+setTimeout(function() {
+	window.location.href = url;
+}, 500);
+
+</script>
+</body>
+</html>

contrib/nginx/copyparty.conf

@@ -0,0 +1,26 @@
+upstream cpp {
+	server 127.0.0.1:3923;
+	keepalive 120;
+}
+server {
+	listen 443 ssl;
+	listen [::]:443 ssl;
+
+	server_name fs.example.com;
+	
+	location / {
+		proxy_pass http://cpp;
+		proxy_redirect off;
+		# disable buffering (next 4 lines)
+		proxy_http_version 1.1;
+		client_max_body_size 0;
+		proxy_buffering off;
+		proxy_request_buffering off;
+
+		proxy_set_header   Host              $host;
+		proxy_set_header   X-Real-IP         $remote_addr;
+		proxy_set_header   X-Forwarded-For   $proxy_add_x_forwarded_for;
+		proxy_set_header   X-Forwarded-Proto $scheme;
+		proxy_set_header   Connection        "Keep-Alive";
+	}
+}

contrib/openrc/copyparty

@@ -0,0 +1,18 @@
+#!/sbin/openrc-run
+
+# this will start `/usr/local/bin/copyparty-sfx.py`
+# and share '/mnt' with anonymous read+write
+#
+# installation:
+#   cp -pv copyparty /etc/init.d && rc-update add copyparty
+#
+# you may want to:
+#   change '/usr/bin/python' to another interpreter
+#   change '/mnt::a' to another location or permission-set
+
+name="$SVCNAME"
+command_background=true
+pidfile="/var/run/$SVCNAME.pid"
+
+command="/usr/bin/python /usr/local/bin/copyparty-sfx.py"
+command_args="-q -v /mnt::a"

contrib/systemd/copyparty.service

@@ -0,0 +1,19 @@
+# this will start `/usr/local/bin/copyparty-sfx.py`
+# and share '/mnt' with anonymous read+write
+#
+# installation:
+#   cp -pv copyparty.service /etc/systemd/system && systemctl enable --now copyparty
+#
+# you may want to:
+#   change '/usr/bin/python' to another interpreter
+#   change '/mnt::a' to another location or permission-set
+
+[Unit]
+Description=copyparty file server
+
+[Service]
+ExecStart=/usr/bin/python /usr/local/bin/copyparty-sfx.py -q -v /mnt::a
+ExecStartPre=/bin/bash -c 'mkdir -p /run/tmpfiles.d/ && echo "x /tmp/pe-copyparty*" > /run/tmpfiles.d/copyparty.conf'
+
+[Install]
+WantedBy=multi-user.target

copyparty/__main__.py

@@ -123,17 +123,20 @@ def main():
             """
         ),
     )
-    ap.add_argument(
-        "-c", metavar="PATH", type=str, action="append", help="add config file"
-    )
+    ap.add_argument("-c", metavar="PATH", type=str, action="append", help="add config file")
     ap.add_argument("-i", metavar="IP", type=str, default="0.0.0.0", help="ip to bind")
-    ap.add_argument("-p", metavar="PORT", type=int, default=1234, help="port to bind")
-    ap.add_argument("-nc", metavar="NUM", type=int, default=16, help="max num clients")
-    ap.add_argument("-j", metavar="CORES", type=int, help="max num cpu cores")
+    ap.add_argument("-p", metavar="PORT", type=int, default=3923, help="port to bind")
+    ap.add_argument("-nc", metavar="NUM", type=int, default=64, help="max num clients")
+    ap.add_argument("-j", metavar="CORES", type=int, default=1, help="max num cpu cores")
     ap.add_argument("-a", metavar="ACCT", type=str, action="append", help="add account")
     ap.add_argument("-v", metavar="VOL", type=str, action="append", help="add volume")
     ap.add_argument("-q", action="store_true", help="quiet")
-    ap.add_argument("-nw", action="store_true", help="benchmark: disable writing")
+    ap.add_argument("-ed", action="store_true", help="enable ?dots")
+    ap.add_argument("-emp", action="store_true", help="enable markdown plugins")
+    ap.add_argument("-mcr", metavar="SEC", type=int, default=60, help="md-editor mod-chk rate")
+    ap.add_argument("-nw", action="store_true", help="disable writes (benchmark)")
+    ap.add_argument("-nih", action="store_true", help="no info hostname")
+    ap.add_argument("-nid", action="store_true", help="no info disk-usage")
     al = ap.parse_args()
 
     SvcHub(al).run()

copyparty/__version__.py

@@ -1,8 +1,8 @@
 # coding: utf-8
 
-VERSION = (0, 4, 1)
-CODENAME = "NIH"
-BUILD_DT = (2020, 5, 14)
+VERSION = (0, 6, 0)
+CODENAME = "CHRISTMAAAAAS"
+BUILD_DT = (2020, 12, 1)
 
 S_VERSION = ".".join(map(str, VERSION))
 S_BUILD_DT = "{0:04d}-{1:02d}-{2:02d}".format(*BUILD_DT)

copyparty/authsrv.py

@@ -135,9 +135,9 @@ class AuthSrv(object):
         self.warn_anonwrite = True
 
         if WINDOWS:
-            self.re_vol = re.compile(r"^([a-zA-Z]:[\\/][^:]*|[^:]*):([^:]*):(.*)")
+            self.re_vol = re.compile(r"^([a-zA-Z]:[\\/][^:]*|[^:]*):([^:]*):(.*)$")
         else:
-            self.re_vol = re.compile(r"^([^:]*):([^:]*):(.*)")
+            self.re_vol = re.compile(r"^([^:]*):([^:]*):(.*)$")
 
         self.mutex = threading.Lock()
         self.reload()
@@ -220,12 +220,13 @@ class AuthSrv(object):
         if self.args.v:
             # list of src:dst:permset:permset:...
             # permset is [rwa]username
-            for vol_match in [self.re_vol.match(x) for x in self.args.v]:
-                try:
-                    src, dst, perms = vol_match.groups()
-                except:
-                    raise Exception("invalid -v argument")
+            for v_str in self.args.v:
+                m = self.re_vol.match(v_str)
+                if not m:
+                    raise Exception("invalid -v argument: [{}]".format(v_str))
 
+                src, dst, perms = m.groups()
+                # print("\n".join([src, dst, perms]))
                 src = fsdec(os.path.abspath(fsenc(src)))
                 dst = dst.strip("/")
                 mount[dst] = src

copyparty/broker_mp.py

@@ -29,7 +29,7 @@ class BrokerMp(object):
         self.mutex = threading.Lock()
 
         cores = self.args.j
-        if cores is None:
+        if not cores:
             cores = mp.cpu_count()
 
         self.log("broker", "booting {} subprocesses".format(cores))

copyparty/httpcli.py

@@ -6,6 +6,8 @@ import stat
 import gzip
 import time
 import json
+import socket
+import ctypes
 from datetime import datetime
 import calendar
 
@@ -14,9 +16,6 @@ from .util import *  # noqa  # pylint: disable=unused-wildcard-import
 
 if not PY2:
     unicode = str
-    from html import escape as html_escape
-else:
-    from cgi import escape as html_escape  # pylint: disable=no-name-in-module
 
 
 class HttpCli(object):
@@ -25,6 +24,7 @@ class HttpCli(object):
     """
 
     def __init__(self, conn):
+        self.t0 = time.time()
         self.conn = conn
         self.s = conn.s
         self.sr = conn.sr
@@ -36,13 +36,13 @@ class HttpCli(object):
 
         self.bufsz = 1024 * 32
         self.absolute_urls = False
-        self.out_headers = {}
+        self.out_headers = {"Access-Control-Allow-Origin": "*"}
 
     def log(self, msg):
         self.log_func(self.log_src, msg)
 
     def _check_nonfatal(self, ex):
-        return ex.code in [403, 404]
+        return ex.code in [404]
 
     def _assert_safe_rem(self, rem):
         # sanity check to prevent any disasters
@@ -83,11 +83,15 @@ class HttpCli(object):
         v = self.headers.get("connection", "").lower()
         self.keepalive = not v.startswith("close")
 
+        v = self.headers.get("x-forwarded-for", None)
+        if v is not None and self.conn.addr[0] in ["127.0.0.1", "::1"]:
+            self.log_src = self.conn.set_rproxy(v.split(",")[0])
+
         self.uname = "*"
         if "cookie" in self.headers:
             cookies = self.headers["cookie"].split(";")
             for k, v in [x.split("=", 1) for x in cookies]:
-                if k != "cppwd":
+                if k.strip() != "cppwd":
                     continue
 
                 v = unescape_cookie(v)
@@ -123,11 +127,20 @@ class HttpCli(object):
         self.uparam = uparam
         self.vpath = unquotep(vpath)
 
+        ua = self.headers.get("user-agent", "")
+        if ua.startswith("rclone/"):
+            uparam["raw"] = True
+            uparam["dots"] = True
+
         try:
             if self.mode in ["GET", "HEAD"]:
                 return self.handle_get() and self.keepalive
             elif self.mode == "POST":
                 return self.handle_post() and self.keepalive
+            elif self.mode == "PUT":
+                return self.handle_put() and self.keepalive
+            elif self.mode == "OPTIONS":
+                return self.handle_options() and self.keepalive
             else:
                 raise Pebkac(400, 'invalid HTTP mode "{0}"'.format(self.mode))
 
@@ -135,7 +148,7 @@ class HttpCli(object):
             try:
                 # self.log("pebkac at httpcli.run #2: " + repr(ex))
                 self.keepalive = self._check_nonfatal(ex)
-                self.loud_reply(str(ex), status=ex.code)
+                self.loud_reply("{}: {}".format(str(ex), self.vpath), status=ex.code)
                 return self.keepalive
             except Pebkac:
                 return False
@@ -143,9 +156,7 @@ class HttpCli(object):
     def send_headers(self, length, status=200, mime=None, headers={}):
         response = ["HTTP/1.1 {} {}".format(status, HTTPCODE[status])]
 
-        if length is None:
-            self.keepalive = False
-        else:
+        if length is not None:
             response.append("Content-Length: " + str(length))
 
         # close if unknown length, otherwise take client's preference
@@ -176,7 +187,8 @@ class HttpCli(object):
         self.send_headers(len(body), status, mime, headers)
 
         try:
-            self.s.sendall(body)
+            if self.mode != "HEAD":
+                self.s.sendall(body)
         except:
             raise Pebkac(400, "client d/c while replying body")
 
@@ -184,7 +196,7 @@ class HttpCli(object):
 
     def loud_reply(self, body, *args, **kwargs):
         self.log(body.rstrip())
-        self.reply(b"<pre>" + body.encode("utf-8"), *list(args), **kwargs)
+        self.reply(b"<pre>" + body.encode("utf-8") + b"\r\n", *list(args), **kwargs)
 
     def handle_get(self):
         logmsg = "{:4} {}".format(self.mode, self.req)
@@ -230,6 +242,30 @@ class HttpCli(object):
 
         return self.tx_browser()
 
+    def handle_options(self):
+        self.log("OPTIONS " + self.req)
+        self.send_headers(
+            None,
+            204,
+            headers={
+                "Access-Control-Allow-Origin": "*",
+                "Access-Control-Allow-Methods": "*",
+                "Access-Control-Allow-Headers": "*",
+            },
+        )
+        return True
+
+    def handle_put(self):
+        self.log("PUT " + self.req)
+
+        if self.headers.get("expect", "").lower() == "100-continue":
+            try:
+                self.s.sendall(b"HTTP/1.1 100 Continue\r\n\r\n")
+            except:
+                raise Pebkac(400, "client d/c before 100 continue")
+
+        return self.handle_stash()
+
     def handle_post(self):
         self.log("POST " + self.req)
 
@@ -243,6 +279,9 @@ class HttpCli(object):
         if not ctype:
             raise Pebkac(400, "you can't post without a content-type header")
 
+        if "raw" in self.uparam:
+            return self.handle_stash()
+
         if "multipart/form-data" in ctype:
             return self.handle_post_multipart()
 
@@ -255,6 +294,37 @@ class HttpCli(object):
 
         raise Pebkac(405, "don't know how to handle {} POST".format(ctype))
 
+    def handle_stash(self):
+        remains = int(self.headers.get("content-length", None))
+        if remains is None:
+            reader = read_socket_unbounded(self.sr)
+            self.keepalive = False
+        else:
+            reader = read_socket(self.sr, remains)
+
+        vfs, rem = self.conn.auth.vfs.get(self.vpath, self.uname, False, True)
+        fdir = os.path.join(vfs.realpath, rem)
+
+        addr = self.conn.addr[0].replace(":", ".")
+        fn = "put-{:.6f}-{}.bin".format(time.time(), addr)
+        path = os.path.join(fdir, fn)
+
+        with open(path, "wb", 512 * 1024) as f:
+            post_sz, _, sha_b64 = hashcopy(self.conn, reader, f)
+
+        spd = self._spd(post_sz)
+        self.log("{} wrote {}/{} bytes to {}".format(spd, post_sz, remains, path))
+        self.reply("{}\n{}\n".format(post_sz, sha_b64).encode("utf-8"))
+        return True
+
+    def _spd(self, nbytes, add=True):
+        if add:
+            self.conn.nbyte += nbytes
+
+        spd1 = get_spd(nbytes, self.t0)
+        spd2 = get_spd(self.conn.nbyte, self.conn.t0)
+        return spd1 + " " + spd2
+
     def handle_post_multipart(self):
         self.parser = MultipartParser(self.log, self.sr, self.headers)
         self.parser.parse()
@@ -394,7 +464,9 @@ class HttpCli(object):
             except:
                 self.log("failed to utime ({}, {})".format(path, times))
 
-        self.loud_reply("thank")
+        spd = self._spd(post_sz)
+        self.log("{} thank".format(spd))
+        self.reply(b"thank")
         return True
 
     def handle_login(self):
@@ -407,7 +479,7 @@ class HttpCli(object):
             msg = "naw dude"
             pwd = "x"  # nosec
 
-        h = {"Set-Cookie": "cppwd={}; Path=/".format(pwd)}
+        h = {"Set-Cookie": "cppwd={}; Path=/; SameSite=Lax".format(pwd)}
         html = self.conn.tpl_msg.render(h1=msg, h2='<a href="/">ack</a>', redir="/")
         self.reply(html.encode("utf-8"), headers=h)
         return True
@@ -440,7 +512,7 @@ class HttpCli(object):
         vpath = "{}/{}".format(self.vpath, sanitized).lstrip("/")
         html = self.conn.tpl_msg.render(
             h2='<a href="/{}">go to /{}</a>'.format(
-                quotep(vpath), html_escape(vpath, quote=False)
+                quotep(vpath), html_escape(vpath)
             ),
             pre="aight",
             click=True,
@@ -474,7 +546,7 @@ class HttpCli(object):
         vpath = "{}/{}".format(self.vpath, sanitized).lstrip("/")
         html = self.conn.tpl_msg.render(
             h2='<a href="/{}?edit">go to /{}?edit</a>'.format(
-                quotep(vpath), html_escape(vpath, quote=False)
+                quotep(vpath), html_escape(vpath)
             ),
             pre="aight",
             click=True,
@@ -519,6 +591,7 @@ class HttpCli(object):
                             raise Pebkac(400, "empty files in post")
 
                         files.append([sz, sha512_hex])
+                        self.conn.nbyte += sz
 
                 except Pebkac:
                     if fn != os.devnull:
@@ -546,7 +619,9 @@ class HttpCli(object):
             # truncated SHA-512 prevents length extension attacks;
             # using SHA-512/224, optionally SHA-512/256 = :64
 
-        self.log(msg)
+        vspd = self._spd(sz_total, False)
+        self.log("{} {}".format(vspd, msg))
+
         if not nullwrite:
             # TODO this is bad
             log_fn = "up.{:.6f}.txt".format(t0)
@@ -568,7 +643,7 @@ class HttpCli(object):
 
         html = self.conn.tpl_msg.render(
             h2='<a href="/{}">return to /{}</a>'.format(
-                quotep(self.vpath), html_escape(self.vpath, quote=False)
+                quotep(self.vpath), html_escape(self.vpath)
             ),
             pre=msg,
         )
@@ -616,7 +691,16 @@ class HttpCli(object):
 
         # if file exists, chekc that timestamp matches the client's
         if srv_lastmod >= 0:
-            if cli_lastmod3 not in [-1, srv_lastmod3]:
+            same_lastmod = cli_lastmod3 in [-1, srv_lastmod3]
+            if not same_lastmod:
+                # some filesystems/transports limit precision to 1sec, hopefully floored
+                same_lastmod = (
+                    srv_lastmod == int(srv_lastmod)
+                    and cli_lastmod3 > srv_lastmod3
+                    and cli_lastmod3 - srv_lastmod3 < 1000
+                )
+
+            if not same_lastmod:
                 response = json.dumps(
                     {
                         "ok": False,
@@ -672,9 +756,12 @@ class HttpCli(object):
                 cli_dt = time.strptime(cli_lastmod, "%a, %d %b %Y %H:%M:%S GMT")
                 cli_ts = calendar.timegm(cli_dt)
                 return file_lastmod, int(file_ts) > int(cli_ts)
-            except:
-                self.log("bad lastmod format: {}".format(cli_lastmod))
-                self.log("   expected format: {}".format(file_lastmod))
+            except Exception as ex:
+                self.log(
+                    "lastmod {}\nremote: [{}]\n local: [{}]".format(
+                        repr(ex), cli_lastmod, file_lastmod
+                    )
+                )
                 return file_lastmod, file_lastmod != cli_lastmod
 
         return file_lastmod, True
@@ -769,11 +856,20 @@ class HttpCli(object):
                 else:
                     upper = file_sz
 
-                if lower < 0 or lower >= file_sz or upper < 0 or upper > file_sz:
+                if upper > file_sz:
+                    upper = file_sz
+
+                if lower < 0 or lower >= upper:
                     raise Exception()
 
             except:
-                raise Pebkac(400, "invalid range requested: " + hrange)
+                err = "invalid range ({}), size={}".format(hrange, file_sz)
+                self.loud_reply(
+                    err,
+                    status=416,
+                    headers={"Content-Range": "bytes */{}".format(file_sz)},
+                )
+                return True
 
             status = 206
             self.out_headers["Content-Range"] = "bytes {}-{}/{}".format(
@@ -795,6 +891,9 @@ class HttpCli(object):
         #
         # send reply
 
+        if not is_compressed:
+            self.out_headers["Cache-Control"] = "no-cache"
+
         self.out_headers["Accept-Ranges"] = "bytes"
         self.send_headers(
             length=upper - lower,
@@ -808,6 +907,7 @@ class HttpCli(object):
             self.log(logmsg)
             return True
 
+        ret = True
         with open_func(*open_args) as f:
             remains = upper - lower
             f.seek(lower)
@@ -820,17 +920,17 @@ class HttpCli(object):
                 if remains < len(buf):
                     buf = buf[:remains]
 
-                remains -= len(buf)
-
                 try:
                     self.s.sendall(buf)
+                    remains -= len(buf)
                 except:
                     logmsg += " \033[31m" + str(upper - remains) + "\033[0m"
-                    self.log(logmsg)
-                    return False
+                    ret = False
+                    break
 
-        self.log(logmsg)
-        return True
+        spd = self._spd((upper - lower) - remains)
+        self.log("{},  {}".format(logmsg, spd))
+        return ret
 
     def tx_md(self, fs_path):
         logmsg = "{:4} {} ".format("", self.req)
@@ -864,8 +964,10 @@ class HttpCli(object):
 
         targs = {
             "edit": "edit" in self.uparam,
-            "title": html_escape(self.vpath, quote=False),
+            "title": html_escape(self.vpath),
             "lastmod": int(ts_md * 1000),
+            "md_plug": "true" if self.args.emp else "false",
+            "md_chk_rate": self.args.mcr,
             "md": "",
         }
         sz_html = len(template.render(**targs).encode("utf-8"))
@@ -905,7 +1007,7 @@ class HttpCli(object):
                 else:
                     vpath += "/" + node
 
-                vpnodes.append([quotep(vpath) + "/", html_escape(node, quote=False)])
+                vpnodes.append([quotep(vpath) + "/", html_escape(node)])
 
         vn, rem = self.auth.vfs.get(
             self.vpath, self.uname, self.readable, self.writable
@@ -941,9 +1043,13 @@ class HttpCli(object):
         except:
             pass
 
+        # show dotfiles if permitted and requested
+        if not self.args.ed or "dots" not in self.uparam:
+            vfs_ls = exclude_dotfiles(vfs_ls)
+
         dirs = []
         files = []
-        for fn in exclude_dotfiles(vfs_ls):
+        for fn in vfs_ls:
             base = ""
             href = fn
             if self.absolute_urls and vpath:
@@ -976,7 +1082,12 @@ class HttpCli(object):
             dt = datetime.utcfromtimestamp(inf.st_mtime)
             dt = dt.strftime("%Y-%m-%d %H:%M:%S")
 
-            item = [margin, quotep(href), html_escape(fn, quote=False), sz, dt]
+            try:
+                ext = "---" if is_dir else fn.rsplit(".", 1)[1]
+            except:
+                ext = "%"
+
+            item = [margin, quotep(href), html_escape(fn), sz, ext, dt]
             if is_dir:
                 dirs.append(item)
             else:
@@ -989,6 +1100,45 @@ class HttpCli(object):
                 with open(fsenc(fn), "rb") as f:
                     logues[n] = f.read().decode("utf-8")
 
+        if False:
+            # this is a mistake
+            md = None
+            for fn in [x[2] for x in files]:
+                if fn.lower() == "readme.md":
+                    fn = os.path.join(abspath, fn)
+                    with open(fn, "rb") as f:
+                        md = f.read().decode("utf-8")
+
+                    break
+
+        srv_info = []
+
+        try:
+            if not self.args.nih:
+                srv_info.append(str(socket.gethostname()).split(".")[0])
+        except:
+            self.log("#wow #whoa")
+            pass
+
+        try:
+            # some fuses misbehave
+            if not self.args.nid:
+                if WINDOWS:
+                    bfree = ctypes.c_ulonglong(0)
+                    ctypes.windll.kernel32.GetDiskFreeSpaceExW(
+                        ctypes.c_wchar_p(abspath), None, None, ctypes.pointer(bfree)
+                    )
+                    srv_info.append(humansize(bfree.value) + " free")
+                else:
+                    sv = os.statvfs(abspath)
+                    free = humansize(sv.f_frsize * sv.f_bfree, True)
+                    total = humansize(sv.f_frsize * sv.f_blocks, True)
+
+                    srv_info.append(free + " free")
+                    srv_info.append(total)
+        except:
+            pass
+
         ts = ""
         # ts = "?{}".format(time.time())
 
@@ -1002,7 +1152,8 @@ class HttpCli(object):
             ts=ts,
             prologue=logues[0],
             epilogue=logues[1],
-            title=html_escape(self.vpath, quote=False),
+            title=html_escape(self.vpath),
+            srv_info="</span> /// <span>".join(srv_info),
         )
         self.reply(html.encode("utf-8", "replace"))
         return True

copyparty/httpconn.py

@@ -4,6 +4,7 @@ from __future__ import print_function, unicode_literals
 import os
 import sys
 import ssl
+import time
 import socket
 
 try:
@@ -41,9 +42,11 @@ class HttpConn(object):
         self.auth = hsrv.auth
         self.cert_path = hsrv.cert_path
 
+        self.t0 = time.time()
+        self.nbyte = 0
         self.workload = 0
         self.log_func = hsrv.log
-        self.log_src = "{} \033[36m{}".format(addr[0], addr[1]).ljust(26)
+        self.set_rproxy()
 
         env = jinja2.Environment()
         env.loader = jinja2.FileSystemLoader(os.path.join(E.mod, "web"))
@@ -53,6 +56,18 @@ class HttpConn(object):
         self.tpl_md = env.get_template("md.html")
         self.tpl_mde = env.get_template("mde.html")
 
+    def set_rproxy(self, ip=None):
+        if ip is None:
+            color = 36
+            ip = self.addr[0]
+            self.rproxy = None
+        else:
+            color = 34
+            self.rproxy = ip
+
+        self.log_src = "{} \033[{}m{}".format(ip, color, self.addr[1]).ljust(26)
+        return self.log_src
+
     def respath(self, res_name):
         return os.path.join(E.mod, "web", res_name)
 
@@ -86,7 +101,7 @@ class HttpConn(object):
                 self.s.send(b"HTTP/1.1 400 Bad Request\r\n\r\n" + err.encode("utf-8"))
                 return
 
-        if method not in [None, b"GET ", b"HEAD", b"POST"]:
+        if method not in [None, b"GET ", b"HEAD", b"POST", b"PUT ", b"OPTI"]:
             if self.sr:
                 self.log("\033[1;31mTODO: cannot do https in jython\033[0m")
                 return

copyparty/httpsrv.py

@@ -80,8 +80,9 @@ class HttpSrv(object):
                         "%s %s" % addr,
                         "shut_rdwr err:\n  {}\n  {}".format(repr(sck), ex),
                     )
-                if ex.errno not in [10038, 107, 57, 9]:
+                if ex.errno not in [10038, 10054, 107, 57, 9]:
                     # 10038 No longer considered a socket
+                    # 10054 Foribly closed by remote
                     #   107 Transport endpoint not connected
                     #    57 Socket is not connected
                     #     9 Bad file descriptor

copyparty/svchub.py

@@ -129,8 +129,8 @@ class SvcHub(object):
         return None
 
     def check_mp_enable(self):
-        if self.args.j == 0:
-            self.log("root", "multiprocessing disabled by argument -j 0;")
+        if self.args.j == 1:
+            self.log("root", "multiprocessing disabled by argument -j 1;")
             return False
 
         if mp.cpu_count() <= 1:

copyparty/util.py

@@ -3,6 +3,7 @@ from __future__ import print_function, unicode_literals
 
 import re
 import sys
+import time
 import base64
 import struct
 import hashlib
@@ -42,6 +43,7 @@ if WINDOWS and PY2:
 
 HTTPCODE = {
     200: "OK",
+    204: "No Content",
     206: "Partial Content",
     304: "Not Modified",
     400: "Bad Request",
@@ -49,6 +51,7 @@ HTTPCODE = {
     404: "Not Found",
     405: "Method Not Allowed",
     413: "Payload Too Large",
+    416: "Requested Range Not Satisfiable",
     422: "Unprocessable Entity",
     500: "Internal Server Error",
     501: "Not Implemented",
@@ -309,18 +312,7 @@ def get_boundary(headers):
 def read_header(sr):
     ret = b""
     while True:
-        if ret.endswith(b"\r\n\r\n"):
-            break
-        elif ret.endswith(b"\r\n\r"):
-            n = 1
-        elif ret.endswith(b"\r\n"):
-            n = 2
-        elif ret.endswith(b"\r"):
-            n = 3
-        else:
-            n = 4
-
-        buf = sr.recv(n)
+        buf = sr.recv(1024)
         if not buf:
             if not ret:
                 return None
@@ -332,11 +324,40 @@ def read_header(sr):
             )
 
         ret += buf
+        ofs = ret.find(b"\r\n\r\n")
+        if ofs < 0:
+            if len(ret) > 1024 * 64:
+                raise Pebkac(400, "header 2big")
+            else:
+                continue
 
-        if len(ret) > 1024 * 64:
-            raise Pebkac(400, "header 2big")
+        sr.unrecv(ret[ofs + 4 :])
+        return ret[:ofs].decode("utf-8", "surrogateescape").split("\r\n")
 
-    return ret[:-4].decode("utf-8", "surrogateescape").split("\r\n")
+
+def humansize(sz, terse=False):
+    for unit in ["B", "KiB", "MiB", "GiB", "TiB"]:
+        if sz < 1024:
+            break
+
+        sz /= 1024.0
+
+    ret = " ".join([str(sz)[:4].rstrip("."), unit])
+
+    if not terse:
+        return ret
+
+    return ret.replace("iB", "").replace(" ", "")
+
+
+def get_spd(nbyte, t0, t=None):
+    if t is None:
+        t = time.time()
+
+    bps = nbyte / ((t - t0) + 0.001)
+    s1 = humansize(nbyte).replace(" ", "\033[33m").replace("iB", "")
+    s2 = humansize(bps).replace(" ", "\033[35m").replace("iB", "")
+    return "{} \033[0m{}/s\033[0m".format(s1, s2)
 
 
 def undot(path):
@@ -388,6 +409,21 @@ def exclude_dotfiles(filepaths):
             yield fpath
 
 
+def html_escape(s, quote=False):
+    """html.escape but also newlines"""
+    s = (
+        s.replace("&", "&amp;")
+        .replace("<", "&lt;")
+        .replace(">", "&gt;")
+        .replace("\r", "&#13;")
+        .replace("\n", "&#10;")
+    )
+    if quote:
+        s = s.replace('"', "&quot;").replace("'", "&#x27;")
+
+    return s
+
+
 def quotep(txt):
     """url quoter which deals with bytes correctly"""
     btxt = w8enc(txt)
@@ -402,8 +438,8 @@ def quotep(txt):
 def unquotep(txt):
     """url unquoter which deals with bytes correctly"""
     btxt = w8enc(txt)
-    unq1 = btxt.replace(b"+", b" ")
-    unq2 = unquote(unq1)
+    # btxt = btxt.replace(b"+", b" ")
+    unq2 = unquote(btxt)
     return w8dec(unq2)
 
 
@@ -451,6 +487,15 @@ def read_socket(sr, total_size):
         yield buf
 
 
+def read_socket_unbounded(sr):
+    while True:
+        buf = sr.recv(32 * 1024)
+        if not buf:
+            return
+
+        yield buf
+
+
 def hashcopy(actor, fin, fout):
     u32_lim = int((2 ** 31) * 0.9)
     hashobj = hashlib.sha512()

copyparty/web/Makefile

@@ -0,0 +1,12 @@
+# run me to zopfli all the static files
+# which should help on really slow connections
+# but then why are you using copyparty in the first place
+
+pk: $(addsuffix .gz, $(wildcard *.js *.css))
+un: $(addsuffix .un, $(wildcard *.gz))
+
+%.gz: %
+	pigz -11 -J 34 -I 5730 $<
+
+%.un: %
+	pigz -d $<

copyparty/web/browser.css

@@ -131,6 +131,17 @@ a {
 .logue {
 	padding: .2em 1.5em;
 }
+#srv_info {
+	opacity: .5;
+	font-size: .8em;
+	color: #fc5;
+	position: absolute;
+	top: .5em;
+	left: 2em;
+}
+#srv_info span {
+	color: #fff;
+}
 a.play {
 	color: #e70;
 }

copyparty/web/browser.html

@@ -33,14 +33,15 @@
             <tr>
                 <th></th>
                 <th>File Name</th>
-                <th>File Size</th>
+                <th sort="int">File Size</th>
+                <th>T</th>
                 <th>Date</th>
             </tr>
         </thead>
         <tbody>
 
 {%- for f in files %}
-<tr><td>{{ f[0] }}</td><td><a href="{{ f[1] }}">{{ f[2] }}</a></td><td>{{ f[3] }}</td><td>{{ f[4] }}</td></tr>
+<tr><td>{{ f[0] }}</td><td><a href="{{ f[1] }}">{{ f[2] }}</a></td><td>{{ f[3] }}</td><td>{{ f[4] }}</td><td>{{ f[5] }}</td></tr>
 {%- endfor %}
 
         </tbody>
@@ -53,6 +54,10 @@
 
     <h2><a href="?h">control-panel</a></h2>
 
+    {%- if srv_info %}
+    <div id="srv_info"><span>{{ srv_info }}</span></div>
+    {%- endif %}
+
     <div id="widget">
         <div id="wtoggle">♫</div>
         <div id="widgeti">
@@ -63,6 +68,8 @@
         </div>
     </div>
     
+    <script src="/.cpr/util.js{{ ts }}"></script>
+
     {%- if can_read %}
     <script src="/.cpr/browser.js{{ ts }}"></script>
     {%- endif %}

copyparty/web/browser.js

@@ -1,75 +1,9 @@
 "use strict";
 
-// error handler for mobile devices
-function hcroak(msg) {
-	document.body.innerHTML = msg;
-	window.onerror = undefined;
-	throw 'fatal_err';
-}
-function croak(msg) {
-	document.body.textContent = msg;
-	window.onerror = undefined;
-	throw msg;
-}
-function esc(txt) {
-	return txt.replace(/[&"<>]/g, function (c) {
-		return {
-			'&': '&amp;',
-			'"': '&quot;',
-			'<': '&lt;',
-			'>': '&gt;'
-		}[c];
-	});
-}
-window.onerror = function (msg, url, lineNo, columnNo, error) {
-	window.onerror = undefined;
-	var html = ['<h1>you hit a bug!</h1><p>please screenshot this error and send me a copy arigathanks gozaimuch (ed/irc.rizon.net or ed#2644)</p><p>',
-		esc(String(msg)), '</p><p>', esc(url + ' @' + lineNo + ':' + columnNo), '</p>'];
-
-	if (error) {
-		var find = ['desc', 'stack', 'trace'];
-		for (var a = 0; a < find.length; a++)
-			if (String(error[find[a]]) !== 'undefined')
-				html.push('<h2>' + find[a] + '</h2>' +
-					esc(String(error[find[a]])).replace(/\n/g, '<br />\n'));
-	}
-	document.body.style.fontSize = '0.8em';
-	document.body.style.padding = '0 1em 1em 1em';
-	hcroak(html.join('\n'));
-};
-
-
-// https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Global_Objects/String/endsWith
-if (!String.prototype.endsWith) {
-	String.prototype.endsWith = function (search, this_len) {
-		if (this_len === undefined || this_len > this.length) {
-			this_len = this.length;
-		}
-		return this.substring(this_len - search.length, this_len) === search;
-	};
-}
-
-
-// https://stackoverflow.com/a/950146
-function import_js(url, cb) {
-	var head = document.head || document.getElementsByTagName('head')[0];
-	var script = document.createElement('script');
-	script.type = 'text/javascript';
-	script.src = url;
-
-	script.onreadystatechange = cb;
-	script.onload = cb;
-
-	head.appendChild(script);
-}
-
-
-function o(id) {
-	return document.getElementById(id);
-}
+window.onerror = vis_exh;
 
 function dbg(msg) {
-	o('path').innerHTML = msg;
+	ebi('path').innerHTML = msg;
 }
 
 function ev(e) {
@@ -78,38 +12,7 @@ function ev(e) {
 	return e;
 }
 
-
-function sortTable(table, col) {
-	var tb = table.tBodies[0], // use `<tbody>` to ignore `<thead>` and `<tfoot>` rows
-		th = table.tHead.rows[0].cells,
-		tr = Array.prototype.slice.call(tb.rows, 0),
-		i, reverse = th[col].className == 'sort1' ? -1 : 1;
-	for (var a = 0, thl = th.length; a < thl; a++)
-		th[a].className = '';
-	th[col].className = 'sort' + reverse;
-	var stype = th[col].getAttribute('sort');
-	tr = tr.sort(function (a, b) {
-		var v1 = a.cells[col].textContent.trim();
-		var v2 = b.cells[col].textContent.trim();
-		if (stype == 'int') {
-			v1 = parseInt(v1.replace(/,/g, ''));
-			v2 = parseInt(v2.replace(/,/g, ''));
-			return reverse * (v1 - v2);
-		}
-		return reverse * (v1.localeCompare(v2));
-	});
-	for (i = 0; i < tr.length; ++i) tb.appendChild(tr[i]);
-}
-function makeSortable(table) {
-	var th = table.tHead, i;
-	th && (th = th.rows[0]) && (th = th.cells);
-	if (th) i = th.length;
-	else return; // if no `<thead>` then do nothing
-	while (--i >= 0) (function (i) {
-		th[i].addEventListener('click', function () { sortTable(table, i) });
-	}(i));
-}
-makeSortable(o('files'));
+makeSortable(ebi('files'));
 
 
 // extract songs + add play column
@@ -122,10 +25,9 @@ var mp = (function () {
 		'tracks': tracks,
 		'cover_url': ''
 	};
-	var re_audio = new RegExp('\.(opus|ogg|m4a|aac|mp3|wav|flac)$', 'i');
-	var re_cover = new RegExp('^(cover|folder|cd|front|back)\.(jpe?g|png|gif)$', 'i');
+	var re_audio = /\.(opus|ogg|m4a|aac|mp3|wav|flac)$/i;
 
-	var trs = document.getElementById('files').getElementsByTagName('tbody')[0].getElementsByTagName('tr');
+	var trs = ebi('files').getElementsByTagName('tbody')[0].getElementsByTagName('tr');
 	for (var a = 0, aa = trs.length; a < aa; a++) {
 		var tds = trs[a].getElementsByTagName('td');
 		var link = tds[1].getElementsByTagName('a')[0];
@@ -141,7 +43,7 @@ var mp = (function () {
 	}
 
 	for (var a = 0, aa = tracks.length; a < aa; a++)
-		o('trk' + a).onclick = ev_play;
+		ebi('trk' + a).onclick = ev_play;
 
 	ret.vol = localStorage.getItem('vol');
 	if (ret.vol !== null)
@@ -168,8 +70,8 @@ var mp = (function () {
 // toggle player widget
 var widget = (function () {
 	var ret = {};
-	var widget = document.getElementById('widget');
-	var wtoggle = document.getElementById('wtoggle');
+	var widget = ebi('widget');
+	var wtoggle = ebi('wtoggle');
 	var touchmode = false;
 	var side_open = false;
 	var was_paused = true;
@@ -198,7 +100,7 @@ var widget = (function () {
 	ret.paused = function (paused) {
 		if (was_paused != paused) {
 			was_paused = paused;
-			o('bplay').innerHTML = paused ? '▶' : '⏸';
+			ebi('bplay').innerHTML = paused ? '▶' : '⏸';
 		}
 	};
 	var click_handler = function (e) {
@@ -222,8 +124,8 @@ var widget = (function () {
 // buffer/position bar
 var pbar = (function () {
 	var r = {};
-	r.bcan = o('barbuf');
-	r.pcan = o('barpos');
+	r.bcan = ebi('barbuf');
+	r.pcan = ebi('barpos');
 	r.bctx = r.bcan.getContext('2d');
 	r.pctx = r.pcan.getContext('2d');
 
@@ -288,7 +190,7 @@ var pbar = (function () {
 // volume bar
 var vbar = (function () {
 	var r = {};
-	r.can = o('pvol');
+	r.can = ebi('pvol');
 	r.ctx = r.can.getContext('2d');
 
 	var bctx = r.ctx;
@@ -385,7 +287,7 @@ var vbar = (function () {
 		else
 			play(0);
 	};
-	o('bplay').onclick = function (e) {
+	ebi('bplay').onclick = function (e) {
 		ev(e);
 		if (mp.au) {
 			if (mp.au.paused)
@@ -396,15 +298,15 @@ var vbar = (function () {
 		else
 			play(0);
 	};
-	o('bprev').onclick = function (e) {
+	ebi('bprev').onclick = function (e) {
 		ev(e);
 		bskip(-1);
 	};
-	o('bnext').onclick = function (e) {
+	ebi('bnext').onclick = function (e) {
 		ev(e);
 		bskip(1);
 	};
-	o('barpos').onclick = function (e) {
+	ebi('barpos').onclick = function (e) {
 		if (!mp.au) {
 			//dbg((new Date()).getTime());
 			return play(0);
@@ -414,15 +316,6 @@ var vbar = (function () {
 		var x = e.clientX - rect.left;
 		var mul = x * 1.0 / rect.width;
 
-		/*
-		dbg(//Math.round(rect.width) + 'x' + Math.round(rect.height) + '+' +
-			//Math.round(rect.left) + '+' + Math.round(rect.top) + ', ' +
-			//Math.round(e.clientX) + 'x' + Math.round(e.clientY) + ', ' +
-			Math.round(mp.au.currentTime * 10) / 10 + ', ' +
-			Math.round(mp.au.duration * 10) / 10 + '*' +
-			Math.round(mul * 1000) / 1000);
-		*/
-
 		mp.au.currentTime = mp.au.duration * mul;
 
 		if (mp.au === mp.au_native)
@@ -479,12 +372,18 @@ function ev_play(e) {
 
 
 function setclass(id, clas) {
-	o(id).setAttribute('class', clas);
+	ebi(id).setAttribute('class', clas);
 }
 
 
-var iOS = !!navigator.platform &&
-	/iPad|iPhone|iPod/.test(navigator.platform);
+var need_ogv = true;
+try {
+	need_ogv = new Audio().canPlayType('audio/ogg; codecs=opus') !== 'probably';
+
+	if (/ Edge\//.exec(navigator.userAgent + ''))
+		need_ogv = true;
+}
+catch (ex) { }
 
 
 // plays the tid'th audio file on the page
@@ -507,7 +406,7 @@ function play(tid, call_depth) {
 	var hack_attempt_play = true;
 
 	var url = mp.tracks[tid];
-	if (iOS && /\.(ogg|opus)$/i.test(url)) {
+	if (need_ogv && /\.(ogg|opus)$/i.test(url)) {
 		if (mp.au_ogvjs) {
 			mp.au = mp.au_ogvjs;
 		}
@@ -569,7 +468,6 @@ function play(tid, call_depth) {
 function evau_error(e) {
 	var err = '';
 	var eplaya = (e && e.target) || (window.event && window.event.srcElement);
-	var url = eplaya.src;
 
 	switch (eplaya.error.code) {
 		case eplaya.error.MEDIA_ERR_ABORTED:
@@ -594,7 +492,6 @@ function evau_error(e) {
 	err += '\n\nFile: «' + decodeURIComponent(eplaya.src.split('/').slice(-1)[0]) + '»';
 
 	alert(err);
-	play(eplaya.tid + 1);
 }
 
 
@@ -611,26 +508,27 @@ function show_modal(html) {
 
 // hide fullscreen message
 function unblocked() {
-	var dom = o('blocked');
+	var dom = ebi('blocked');
 	if (dom)
-		dom.remove();
+		dom.parentNode.removeChild(dom);
 }
 
 
 // show ui to manually start playback of a linked song
-function autoplay_blocked(tid) {
+function autoplay_blocked() {
 	show_modal(
-		'<div id="blk_play"><a id="blk_go"></a></div>' +
-		'<div id="blk_abrt"><a id="blk_na">Cancel<br />(show file list)</a></div>');
+		'<div id="blk_play"><a href="#" id="blk_go"></a></div>' +
+		'<div id="blk_abrt"><a href="#" id="blk_na">Cancel<br />(show file list)</a></div>');
 
-	var go = o('blk_go');
-	var na = o('blk_na');
+	var go = ebi('blk_go');
+	var na = ebi('blk_na');
 
 	var fn = mp.tracks[mp.au.tid].split(/\//).pop();
 	fn = decodeURIComponent(fn.replace(/\+/g, ' '));
 
 	go.textContent = 'Play "' + fn + '"';
-	go.onclick = function () {
+	go.onclick = function (e) {
+		if (e) e.preventDefault();
 		unblocked();
 		mp.au.play();
 	};

copyparty/web/md.css

@@ -1,3 +1,7 @@
+@font-face {
+	font-family: 'scp';
+	src: local('Source Code Pro Regular'), local('SourceCodePro-Regular'), url(/.cpr/deps/scp.woff2) format('woff2');
+}
 html, body {
 	color: #333;
 	background: #eee;
@@ -9,6 +13,7 @@ html, body {
 }
 #mw {
 	margin: 0 auto;
+	padding: 0 1.5em;
 }
 pre, code, a {
 	color: #480;
@@ -22,7 +27,7 @@ code {
 	font-size: .96em;
 }
 pre, code {
-	font-family: monospace, monospace;
+	font-family: 'scp', monospace, monospace;
 	white-space: pre-wrap;
 	word-break: break-all;
 }
@@ -42,7 +47,7 @@ pre code {
 pre code:last-child {
 	border-bottom: none;
 }
-pre code:before {
+pre code::before {
 	content: counter(precode);
 	-webkit-user-select: none;
 	display: inline-block;
@@ -104,8 +109,12 @@ h2 a, h4 a, h6 a {
 #mp ol>li {
 	margin: .7em 0;
 }
+strong {
+	color: #000;
+}
 p>em,
-li>em {
+li>em,
+td>em {
 	color: #c50;
 	padding: .1em;
 	border-bottom: .1em solid #bbb;
@@ -168,14 +177,12 @@ small {
 }
 table {
 	border-collapse: collapse;
+	margin: 1em 0;
 }
-td {
+th, td {
 	padding: .2em .5em;
 	border: .12em solid #aaa;
 }
-th {
-	border: .12em solid #aaa;
-}
 blink {
 	animation: blinker .7s cubic-bezier(.9, 0, .1, 1) infinite;
 }
@@ -198,13 +205,15 @@ blink {
 		height: 100%;
 	}
 	#mw {
-		padding: 0 1em;
 		margin: 0 auto;
 		right: 0;
 	}
 	#mp {
-		max-width: 54em;
+		max-width: 52em;
 		margin-bottom: 6em;
+		word-break: break-word;
+		overflow-wrap: break-word;
+		word-wrap: break-word; /*ie*/
 	}
 	a {
 		color: #fff;
@@ -238,12 +247,6 @@ blink {
 		z-index: 10;
 		width: calc(100% - 1em);
 	}
-	#mn.undocked {
-        position: fixed;
-		padding: 1.7em 0 1.5em 1em;
-		box-shadow: 0 0 .5em rgba(0, 0, 0, 0.3);
-		background: #f7f7f7;
-	}
 	#mn a {
 		color: #444;
 		background: none;
@@ -261,7 +264,7 @@ blink {
 	#mn a:last-child {
 		padding-right: .5em;
 	}
-	#mn a:not(:last-child):after {
+	#mn a:not(:last-child)::after {
 		content: '';
 		width: 1.05em;
 		height: 1.05em;
@@ -290,6 +293,32 @@ blink {
 		text-decoration: underline;
 		border: none;
 	}
+	#mh a:hover {
+		color: #000;
+		background: #ddd;
+	}
+	#toolsbox {
+		overflow: hidden;
+		display: inline-block;
+		background: #eee;
+		height: 1.5em;
+		padding: 0 .2em;
+		margin: 0 .2em;
+		position: absolute;
+	}
+	#toolsbox.open {
+		height: auto;
+		overflow: visible;
+		background: #eee;
+		box-shadow: 0 .2em .2em #ccc;
+		padding-bottom: .2em;
+	}
+	#toolsbox a {
+		display: block;
+	}
+	#toolsbox a+a {
+		text-decoration: none;
+	}
 
 
 
@@ -333,8 +362,12 @@ blink {
 	html.dark #m>ol {
 		border-color: #555;
 	}
+	html.dark strong {
+		color: #fff;
+	}
 	html.dark p>em,
-	html.dark li>em {
+	html.dark li>em,
+	html.dark td>em {
 		color: #f94;
 		border-color: #666;
 	}
@@ -355,7 +388,7 @@ blink {
 		background: #282828;
 		border: .07em dashed #444;
 	}
-	html.dark #mn a:not(:last-child):after {
+	html.dark #mn a:not(:last-child)::after {
 		border-color: rgba(255,255,255,0.3);
 	}
 	html.dark #mn a {
@@ -372,21 +405,32 @@ blink {
 		color: #ccc;
 		background: none;
 	}
+	html.dark #mh a:hover {
+		background: #333;
+		color: #fff;
+	}
+	html.dark #toolsbox {
+		background: #222;
+	}
+	html.dark #toolsbox.open {
+		box-shadow: 0 .2em .2em #069;
+		border-radius: 0 0 .4em .4em;
+	}
 }
 
-@media screen and (min-width: 70em) {
+@media screen and (min-width: 66em) {
 	#mw {
 		position: fixed;
 		overflow-y: auto;
 		left: 14em;
-		left: calc(100% - 57em);
+		left: calc(100% - 55em);
 		max-width: none;
 		bottom: 0;
 		scrollbar-color: #eb0 #f7f7f7;
 	}
 	#toc {
 		width: 13em;
-		width: calc(100% - 57.3em);
+		width: calc(100% - 55.3em);
 		max-width: 30em;
 		background: #eee;
 		position: fixed;
@@ -433,34 +477,119 @@ blink {
 		width: .8em;
 	}
 	html.dark #toc::-webkit-scrollbar-thumb {
-		background: #eb0;
-	}
-	html.dark #mn.undocked {
-		box-shadow: 0 0 .5em #555;
-		border: none;
-		background: #0a0a0a;
+		background: #b80;
 	}
 }
-@media screen and (min-width: 87.5em) {
+@media screen and (min-width: 85.5em) {
 	#toc { width: 30em }
 	#mw { left: 30.5em }
 }
 @media print {
+	@page {
+		size: A4;
+		padding: 0;
+		margin: .5in .6in;
+		mso-header-margin: .6in;
+		mso-footer-margin: .6in;
+		mso-paper-source: 0;
+	}
 	a {
 		color: #079;
 		text-decoration: none;
 		border-bottom: .07em solid #4ac;
 		padding: 0 .3em;
 	}
-	#toc {
-		margin: 0 !important;
-	}
 	#toc>ul {
 		border-left: .1em solid #84c4dd;
 	}
 	#mn, #mh {
 		display: none;
 	}
+	html, body, #toc, #mw {
+		margin: 0 !important;
+		word-break: break-word;
+		width: 52em;
+	}
+	#toc {
+		margin-left: 1em !important;
+	}
+	#toc a {
+		color: #000 !important;
+	}
+	#toc a::after {
+		/* hopefully supported by browsers eventually */
+		content: leader('.') target-counter(attr(href), page);
+	}
+	a[ctr]::before {
+		content: attr(ctr) '. ';
+	}
+	h1 {
+		margin: 2em 0;
+	}
+	h2 {
+		margin: 2em 0 0 0;
+	}
+	h1, h2, h3 {
+		page-break-inside: avoid;
+	}
+	h1::after,
+	h2::after,
+	h3::after {
+		content: 'orz';
+		color: transparent;
+		display: block;
+		line-height: 1em;
+		padding: 4em 0 0 0;
+		margin: 0 0 -5em 0;
+	}
+	p {
+		page-break-inside: avoid;
+	}
+	table {
+		page-break-inside: auto;
+	}
+	tr {
+		page-break-inside: avoid;
+		page-break-after: auto;
+	}
+	thead {
+		display: table-header-group;
+	}
+	tfoot {
+		display: table-footer-group;
+	}
+	#mp a.vis::after {
+		content: ' (' attr(href) ')';
+		border-bottom: 1px solid #bbb;
+		color: #444;
+	}
+	blockquote {
+		border-color: #555;
+	}
+	code {
+		border-color: #bbb;
+	}
+	pre, pre code {
+		border-color: #999;
+	}
+	pre code::before {
+		color: #058;
+	}
+
+
+	
+	html.dark a {
+		color: #000;
+	}
+	html.dark pre,
+	html.dark code {
+		color: #240;
+	}
+	html.dark p>em,
+	html.dark li>em,
+	html.dark td>em {
+		color: #940;
+	}
 }
 
 /*

copyparty/web/md.html

@@ -17,15 +17,23 @@
 			<a id="save" href="?edit">save</a>
 			<a id="sbs" href="#">sbs</a>
 			<a id="nsbs" href="#">editor</a>
-			<a id="help" href="#">help</a>
+			<div id="toolsbox">
+				<a id="tools" href="#">tools</a>
+				<a id="fmt_table" href="#">prettify table (ctrl-k)</a>
+				<a id="iter_uni" href="#">non-ascii: iterate (ctrl-u)</a>
+				<a id="mark_uni" href="#">non-ascii: markup</a>
+				<a id="cfg_uni" href="#">non-ascii: whitelist</a>
+				<a id="help" href="#">help</a>
+			</div>
 		{%- else %}
 			<a href="?edit">edit (basic)</a>
 			<a href="?edit2">edit (fancy)</a>
+			<a href="?raw">view raw</a>
 		{%- endif %}
 	</div>
 	<div id="toc"></div>
 	<div id="mtw">
-		<textarea id="mt">{{ md }}</textarea>
+		<textarea id="mt" autocomplete="off">{{ md }}</textarea>
 	</div>
 	<div id="mw">
 		<div id="ml">
@@ -39,16 +47,19 @@
 	
 	{%- if edit %}
 	<div id="helpbox">
-		<textarea>
+		<textarea autocomplete="off">
 
-write markdown (html is permitted)
+write markdown (most html is 🙆 too)
 
-### hotkey list
+## hotkey list
 * `Ctrl-S` to save
+* `Ctrl-E` to toggle mode
+* `Ctrl-K` to prettyprint a table
+* `Ctrl-U` to iterate non-ascii chars
 * `Ctrl-H` / `Ctrl-Shift-H` to create a header
 * `TAB` / `Shift-TAB` to indent/dedent a selection
 
-### toolbar
+## toolbar
 1. toggle dark mode
 2. show/hide navigation bar
 3. save changes on server
@@ -56,16 +67,68 @@ write markdown (html is permitted)
 5. toggle editor/preview
 6. this thing :^)
 
-.
+## markdown
+|||
+|--|--|
+|`**bold**`|**bold**|
+|`_italic_`|_italic_|
+|`~~strike~~`|~~strike~~|
+|`` `code` ``|`code`|
+|`[](#hotkey-list)`|[](#hotkey-list)|
+|`[](/foo/bar.md#header)`|[](/foo/bar.md#header)|
+|`<blink>💯</blink>`|<blink>💯</blink>|
 
+## tables
+    |left-aligned|centered|right-aligned
+    | ---------- | :----: | ----------:
+    |one         |two     |three
+
+|left-aligned|centered|right-aligned
+| ---------- | :----: | ----------:
+|one         |two     |three
+
+## lists
+	* one
+	* two
+	1. one
+	1. two
+* one
+* two
+1. one
+1. two
+
+## headers
+	# level 1
+	## level 2
+	### level 3
+
+## quote
+	> hello
+> hello
+
+## codeblock
+		four spaces (no tab pls)
+
+## code in lists
+	* foo
+	  bar
+          six spaces total
+* foo
+  bar
+      six spaces total
+.
 		</textarea>
 	</div>
 	{%- endif %}
 	
 	<script>
 
-var link_md_as_html = false;  // TODO (does nothing)
 var last_modified = {{ lastmod }};
+var md_opt = {
+	link_md_as_html: false,
+	allow_plugins: {{ md_plug }},
+	modpoll_freq: {{ md_chk_rate }}
+};
 
 (function () {
     var btn = document.getElementById("lightswitch");
@@ -82,17 +145,11 @@ var last_modified = {{ lastmod }};
 		toggle();
 })();
 
-if (!String.startsWith) {
-	String.prototype.startsWith = function(s, i) {
-		i = i>0 ? i|0 : 0;
-		return this.substring(i, i + s.length) === s;
-	};
-}
-
 	</script>
+    <script src="/.cpr/util.js"></script>
 	<script src="/.cpr/deps/marked.full.js"></script>
 	<script src="/.cpr/md.js"></script>
 	{%- if edit %}
-		<script src="/.cpr/md2.js"></script>
+	<script src="/.cpr/md2.js"></script>
 	{%- endif %}
 </body></html>

copyparty/web/md.js

@@ -1,10 +1,12 @@
-var dom_toc = document.getElementById('toc');
-var dom_wrap = document.getElementById('mw');
-var dom_hbar = document.getElementById('mh');
-var dom_nav = document.getElementById('mn');
-var dom_pre = document.getElementById('mp');
-var dom_src = document.getElementById('mt');
-var dom_navtgl = document.getElementById('navtoggle');
+"use strict";
+
+var dom_toc = ebi('toc');
+var dom_wrap = ebi('mw');
+var dom_hbar = ebi('mh');
+var dom_nav = ebi('mn');
+var dom_pre = ebi('mp');
+var dom_src = ebi('mt');
+var dom_navtgl = ebi('navtoggle');
 
 
 // chrome 49 needs this
@@ -18,6 +20,10 @@ var dbg = function () { };
 // dbg = console.log
 
 
+// plugins
+var md_plug = {};
+
+
 function hesc(txt) {
     return txt.replace(/&/g, "&amp;").replace(/</g, "&lt;").replace(/>/g, "&gt;");
 }
@@ -30,6 +36,24 @@ function cls(dom, name, add) {
 }
 
 
+function statify(obj) {
+    return JSON.parse(JSON.stringify(obj));
+}
+
+
+// dodge browser issues
+(function () {
+    var ua = navigator.userAgent;
+    if (ua.indexOf(') Gecko/') !== -1 && /Linux| Mac /.exec(ua)) {
+        // necessary on ff-68.7 at least
+        var s = document.createElement('style');
+        s.innerHTML = '@page { margin: .5in .6in .8in .6in; }';
+        console.log(s.innerHTML);
+        document.head.appendChild(s);
+    }
+})();
+
+
 // add navbar
 (function () {
     var n = document.location + '';
@@ -136,17 +160,125 @@ function copydom(src, dst, lv) {
 }
 
 
-function convert_markdown(md_text) {
-    marked.setOptions({
+function md_plug_err(ex, js) {
+    var errbox = ebi('md_errbox');
+    if (errbox)
+        errbox.parentNode.removeChild(errbox);
+
+    if (!ex)
+        return;
+
+    var msg = (ex + '').split('\n')[0];
+    var ln = ex.lineNumber;
+    var o = null;
+    if (ln) {
+        msg = "Line " + ln + ", " + msg;
+        var lns = js.split('\n');
+        if (ln < lns.length) {
+            o = document.createElement('span');
+            o.style.cssText = 'color:#ac2;font-size:.9em;font-family:scp;display:block';
+            o.textContent = lns[ln - 1];
+        }
+    }
+    errbox = document.createElement('div');
+    errbox.setAttribute('id', 'md_errbox');
+    errbox.style.cssText = 'position:absolute;top:0;left:0;padding:1em .5em;background:#2b2b2b;color:#fc5'
+    errbox.textContent = msg;
+    errbox.onclick = function () {
+        alert('' + ex.stack);
+    };
+    if (o) {
+        errbox.appendChild(o);
+        errbox.style.padding = '.25em .5em';
+    }
+    dom_nav.appendChild(errbox);
+
+    try {
+        console.trace();
+    }
+    catch (ex2) { }
+}
+
+
+function load_plug(md_text, plug_type) {
+    if (!md_opt.allow_plugins)
+        return md_text;
+
+    var find = '\n```copyparty_' + plug_type + '\n';
+    var ofs = md_text.indexOf(find);
+    if (ofs === -1)
+        return md_text;
+
+    var ofs2 = md_text.indexOf('\n```', ofs + 1);
+    if (ofs2 == -1)
+        return md_text;
+
+    var js = md_text.slice(ofs + find.length, ofs2 + 1);
+    var md = md_text.slice(0, ofs + 1) + md_text.slice(ofs2 + 4);
+
+    var old_plug = md_plug[plug_type];
+    if (!old_plug || old_plug[1] != js) {
+        js = 'const x = { ' + js + ' }; x;';
+        try {
+            var x = eval(js);
+        }
+        catch (ex) {
+            md_plug[plug_type] = null;
+            md_plug_err(ex, js);
+            return md;
+        }
+        if (x['ctor']) {
+            x['ctor']();
+            delete x['ctor'];
+        }
+        md_plug[plug_type] = [x, js];
+    }
+
+    return md;
+}
+
+
+function convert_markdown(md_text, dest_dom) {
+    md_text = md_text.replace(/\r/g, '');
+
+    md_plug_err(null);
+    md_text = load_plug(md_text, 'pre');
+    md_text = load_plug(md_text, 'post');
+
+    var marked_opts = {
         //headerPrefix: 'h-',
         breaks: true,
         gfm: true
-    });
-    var md_html = marked(md_text);
+    };
+
+    var ext = md_plug['pre'];
+    if (ext)
+        Object.assign(marked_opts, ext[0]);
+
+    try {
+        var md_html = marked(md_text, marked_opts);
+    }
+    catch (ex) {
+        if (ext)
+            md_plug_err(ex, ext[1]);
+
+        throw ex;
+    }
     var md_dom = new DOMParser().parseFromString(md_html, "text/html").body;
 
+    var nodes = md_dom.getElementsByTagName('a');
+    for (var a = nodes.length - 1; a >= 0; a--) {
+        var href = nodes[a].getAttribute('href');
+        var txt = nodes[a].textContent;
+
+        if (!txt)
+            nodes[a].textContent = href;
+        else if (href !== txt)
+            nodes[a].setAttribute('class', 'vis');
+    }
+
     // todo-lists (should probably be a marked extension)
-    var nodes = md_dom.getElementsByTagName('input');
+    nodes = md_dom.getElementsByTagName('input');
     for (var a = nodes.length - 1; a >= 0; a--) {
         var dom_box = nodes[a];
         if (dom_box.getAttribute('type') !== 'checkbox')
@@ -167,7 +299,7 @@ function convert_markdown(md_text) {
     }
 
     // separate <code> for each line in <pre>
-    var nodes = md_dom.getElementsByTagName('pre');
+    nodes = md_dom.getElementsByTagName('pre');
     for (var a = nodes.length - 1; a >= 0; a--) {
         var el = nodes[a];
 
@@ -180,7 +312,7 @@ function convert_markdown(md_text) {
             continue;
 
         var nline = parseInt(el.getAttribute('data-ln')) + 1;
-        var lines = el.innerHTML.replace(/\r?\n<\/code>$/i, '</code>').split(/\r?\n/g);
+        var lines = el.innerHTML.replace(/\n<\/code>$/i, '</code>').split(/\n/g);
         for (var b = 0; b < lines.length - 1; b++)
             lines[b] += '</code>\n<code data-ln="' + (nline + b) + '">';
 
@@ -213,18 +345,36 @@ function convert_markdown(md_text) {
         el.innerHTML = '<a href="#' + id + '">' + el.innerHTML + '</a>';
     }
 
-    copydom(md_dom, dom_pre, 0);
+    ext = md_plug['post'];
+    if (ext && ext[0].render)
+        try {
+            ext[0].render(md_dom);
+        }
+        catch (ex) {
+            md_plug_err(ex, ext[1]);
+        }
+
+    copydom(md_dom, dest_dom, 0);
+
+    if (ext && ext[0].render2)
+        try {
+            ext[0].render2(dest_dom);
+        }
+        catch (ex) {
+            md_plug_err(ex, ext[1]);
+        }
 }
 
 
 function init_toc() {
-    var loader = document.getElementById('ml');
+    var loader = ebi('ml');
     loader.parentNode.removeChild(loader);
 
     var anchors = [];  // list of toc entries, complex objects
     var anchor = null; // current toc node
     var html = [];     // generated toc html
     var lv = 0;        // current indentation level in the toc html
+    var ctr = [0, 0, 0, 0, 0, 0];
 
     var manip_nodes_dyn = dom_pre.getElementsByTagName('*');
     var manip_nodes = [];
@@ -245,8 +395,18 @@ function init_toc() {
                 html.push('</ul>');
                 lv--;
             }
+            ctr[lv - 1]++;
+            for (var b = lv; b < 6; b++)
+                ctr[b] = 0;
 
-            html.push('<li>' + elm.innerHTML + '</li>');
+            elm.childNodes[0].setAttribute('ctr', ctr.slice(0, lv).join('.'));
+
+            var elm2 = elm.cloneNode(true);
+            elm2.childNodes[0].textContent = elm.textContent;
+            while (elm2.childNodes.length > 1)
+                elm2.removeChild(elm2.childNodes[1]);
+
+            html.push('<li>' + elm2.innerHTML + '</li>');
 
             if (anchor != null)
                 anchors.push(anchor);
@@ -328,7 +488,7 @@ function init_toc() {
 
 
 // "main" :p
-convert_markdown(dom_src.value);
+convert_markdown(dom_src.value, dom_pre);
 var toc = init_toc();
 
 
@@ -360,40 +520,10 @@ var redraw = (function () {
 
 
 dom_navtgl.onclick = function () {
-    var timeout = null;
-    function show_nav(e) {
-        if (e && e.target == dom_hbar && e.pageX && e.pageX < dom_hbar.offsetWidth / 2)
-            return;
-
-        clearTimeout(timeout);
-        dom_nav.style.display = 'block';
-    }
-    function hide_nav() {
-        clearTimeout(timeout);
-        timeout = setTimeout(function () {
-            dom_nav.style.display = 'none';
-        }, 30);
-    }
     var hidden = dom_navtgl.innerHTML == 'hide nav';
     dom_navtgl.innerHTML = hidden ? 'show nav' : 'hide nav';
-    if (hidden) {
-        dom_nav.setAttribute('class', 'undocked');
-        dom_nav.style.display = 'none';
-        dom_nav.style.top = dom_hbar.offsetHeight + 'px';
-        dom_nav.onmouseenter = show_nav;
-        dom_nav.onmouseleave = hide_nav;
-        dom_hbar.onmouseenter = show_nav;
-        dom_hbar.onmouseleave = hide_nav;
-    }
-    else {
-        dom_nav.setAttribute('class', '');
-        dom_nav.style.display = 'block';
-        dom_nav.style.top = '0';
-        dom_nav.onmouseenter = null;
-        dom_nav.onmouseleave = null;
-        dom_hbar.onmouseenter = null;
-        dom_hbar.onmouseleave = null;
-    }
+    dom_nav.style.display = hidden ? 'none' : 'block';
+
     if (window.localStorage)
         localStorage.setItem('hidenav', hidden ? 1 : 0);
 

copyparty/web/md2.css

@@ -4,12 +4,12 @@
 #mtw {
     display: block;
     position: fixed;
-    left: 0;
+    left: .5em;
     bottom: 0;
-    width: calc(100% - 58em);
+    width: calc(100% - 56em);
 }
 #mw {
-    left: calc(100% - 57em);
+    left: calc(100% - 55em);
     overflow-y: auto;
     position: fixed;
     bottom: 0;
@@ -30,11 +30,11 @@
 #mw.single {
     margin: 0;
     left: 1em;
-    left: max(1em, calc((100% - 58em) / 2));
+    left: max(1em, calc((100% - 56em) / 2));
 }
 #mtw.single {
-    width: 57em;
-    width: min(57em, calc(100% - 2em));
+    width: 55em;
+    width: min(55em, calc(100% - 2em));
 }
 
 
@@ -43,7 +43,7 @@
 }
 #mt, #mtr {
     width: 100%;
-    height: calc(100% - 5px);
+    height: calc(100% - 1px);
     color: #444;
     background: #f7f7f7;
     border: 1px solid #999;
@@ -77,32 +77,52 @@ html.dark #mt {
     background: #f97;
     border-radius: .15em;
 }
+html.dark #save.force-save {
+    color: #fca;
+    background: #720;
+}
 #save.disabled {
     opacity: .4;
 }
+#helpbox,
+#toast {
+    background: #f7f7f7;
+    border-radius: .4em;
+    z-index: 9001;
+}
 #helpbox {
     display: none;
     position: fixed;
-    background: #f7f7f7;
-    box-shadow: 0 .5em 2em #777;
-    border-radius: .4em;
     padding: 2em;
     top: 4em;
     overflow-y: auto;
+    box-shadow: 0 .5em 2em #777;
     height: calc(100% - 12em);
     left: calc(50% - 15em);
     right: 0;
     width: 30em;
-    z-index: 9001;
 }
 #helpclose {
     display: block;
 }
 html.dark #helpbox {
-    background: #222;
     box-shadow: 0 .5em 2em #444;
+}
+html.dark #helpbox,
+html.dark #toast {
+    background: #222;
     border: 1px solid #079;
     border-width: 1px 0;
 }
+#toast {
+    font-weight: bold;
+    text-align: center;
+    padding: .6em 0;
+    position: fixed;
+    z-index: 9001;
+    top: 30%;
+    transition: opacity 0.2s ease-in-out;
+    opacity: 1;
+}
 
 # mt {opacity: .5;top:1px}

copyparty/web/md2.js

@@ -1,16 +1,25 @@
+"use strict";
+
+
 // server state
 var server_md = dom_src.value;
 
 
+// the non-ascii whitelist
+var esc_uni_whitelist = '\\n\\t\\x20-\\x7eÆØÅæøå';
+var js_uni_whitelist = eval('\'' + esc_uni_whitelist + '\'');
+
+
 // dom nodes
-var dom_swrap = document.getElementById('mtw');
-var dom_sbs = document.getElementById('sbs');
-var dom_nsbs = document.getElementById('nsbs');
+var dom_swrap = ebi('mtw');
+var dom_sbs = ebi('sbs');
+var dom_nsbs = ebi('nsbs');
+var dom_tbox = ebi('toolsbox');
 var dom_ref = (function () {
     var d = document.createElement('div');
     d.setAttribute('id', 'mtr');
     dom_swrap.appendChild(d);
-    d = document.getElementById('mtr');
+    d = ebi('mtr');
     // hide behind the textarea (offsetTop is not computed if display:none)
     dom_src.style.zIndex = '4';
     d.style.zIndex = '3';
@@ -19,14 +28,12 @@ var dom_ref = (function () {
 
 
 // line->scrollpos maps
-var map_src = [];
-var map_pre = [];
-function genmap(dom) {
+function genmapq(dom, query) {
     var ret = [];
     var last_y = -1;
     var parent_y = 0;
     var parent_n = null;
-    var nodes = dom.querySelectorAll('*[data-ln]');
+    var nodes = dom.querySelectorAll(query);
     for (var a = 0; a < nodes.length; a++) {
         var n = nodes[a];
         var ln = parseInt(n.getAttribute('data-ln'));
@@ -35,7 +42,7 @@ function genmap(dom) {
 
         var y = 0;
         var par = n.offsetParent;
-        if (par != parent_n) {
+        if (par && par != parent_n) {
             while (par && par != dom) {
                 y += par.offsetTop;
                 par = par.offsetParent;
@@ -49,7 +56,7 @@ function genmap(dom) {
         while (ln > ret.length)
             ret.push(null);
 
-        var y = parent_y + n.offsetTop;
+        y = parent_y + n.offsetTop;
         if (y <= last_y)
             //console.log('awawa');
             continue;
@@ -60,6 +67,25 @@ function genmap(dom) {
     }
     return ret;
 }
+var map_src = [];
+var map_pre = [];
+function genmap(dom, oldmap) {
+    var find = nlines;
+    while (oldmap && find --> 0) {
+        var tmap = genmapq(dom, '*[data-ln="' + find + '"]');
+        if (!tmap || !tmap.length)
+            continue;
+
+        var cy = tmap[find];
+        var oy = parseInt(oldmap[find]);
+        if (cy + 24 > oy && cy - 24 < oy)
+            return oldmap;
+
+        console.log('map regen', dom.getAttribute('id'), find, oy, cy, oy - cy);
+        break;
+    }
+    return genmapq(dom, '*[data-ln]');
+}
 
 
 // input handler
@@ -70,7 +96,7 @@ var draw_md = (function () {
     function draw_md() {
         var t0 = new Date().getTime();
         var src = dom_src.value;
-        convert_markdown(src);
+        convert_markdown(src, dom_pre);
 
         var lines = hesc(src).replace(/\r/g, "").split('\n');
         nlines = lines.length;
@@ -79,13 +105,13 @@ var draw_md = (function () {
             html.push('<span data-ln="' + (a + 1) + '">' + lines[a] + "</span>");
 
         dom_ref.innerHTML = html.join('\n');
-        map_src = genmap(dom_ref);
-        map_pre = genmap(dom_pre);
+        map_src = genmap(dom_ref, map_src);
+        map_pre = genmap(dom_pre, map_pre);
 
-        cls(document.getElementById('save'), 'disabled', src == server_md);
+        cls(ebi('save'), 'disabled', src == server_md);
 
         var t1 = new Date().getTime();
-        delay = t1 - t0 > 150 ? 25 : 1;
+        delay = t1 - t0 > 100 ? 25 : 1;
     }
 
     var timeout = null;
@@ -108,8 +134,8 @@ redraw = (function () {
         dom_wrap.style.top = y;
         dom_swrap.style.top = y;
         dom_ref.style.width = getComputedStyle(dom_src).offsetWidth + 'px';
-        map_src = genmap(dom_ref);
-        map_pre = genmap(dom_pre);
+        map_src = genmap(dom_ref, map_src);
+        map_pre = genmap(dom_pre, map_pre);
         dbg(document.body.clientWidth + 'x' + document.body.clientHeight);
     }
     function setsbs() {
@@ -118,7 +144,7 @@ redraw = (function () {
         onresize();
     }
     function modetoggle() {
-        mode = dom_nsbs.innerHTML;
+        var mode = dom_nsbs.innerHTML;
         dom_nsbs.innerHTML = mode == 'editor' ? 'preview' : 'editor';
         mode += ' single';
         dom_wrap.setAttribute('class', mode);
@@ -147,14 +173,14 @@ redraw = (function () {
             dst.scrollTop = 0;
             return;
         }
-        if (y + 8 + src.clientHeight > src.scrollHeight) {
+        if (y + 48 + src.clientHeight > src.scrollHeight) {
             dst.scrollTop = dst.scrollHeight - dst.clientHeight;
             return;
         }
         y += src.clientHeight / 2;
         var sy1 = -1, sy2 = -1, dy1 = -1, dy2 = -1;
         for (var a = 1; a < nlines + 1; a++) {
-            if (srcmap[a] === null || dstmap[a] === null)
+            if (srcmap[a] == null || dstmap[a] == null)
                 continue;
 
             if (srcmap[a] > y) {
@@ -197,14 +223,108 @@ redraw = (function () {
 })();
 
 
+// modification checker
+function Modpoll() {
+    this.skip_one = true;
+    this.disabled = false;
+
+    this.periodic = function () {
+        var that = this;
+        setTimeout(function () {
+            that.periodic();
+        }, 1000 * md_opt.modpoll_freq);
+
+        var skip = null;
+
+        if (ebi('toast'))
+            skip = 'toast';
+
+        else if (this.skip_one)
+            skip = 'saved';
+
+        else if (this.disabled)
+            skip = 'disabled';
+
+        if (skip) {
+            console.log('modpoll skip, ' + skip);
+            this.skip_one = false;
+            return;
+        }
+
+        console.log('modpoll...');
+        var url = (document.location + '').split('?')[0] + '?raw&_=' + new Date().getTime();
+        var xhr = new XMLHttpRequest();
+        xhr.modpoll = this;
+        xhr.open('GET', url, true);
+        xhr.responseType = 'text';
+        xhr.onreadystatechange = this.cb;
+        xhr.send();
+    }
+
+    this.cb = function () {
+        if (this.modpoll.disabled || this.modpoll.skip_one) {
+            console.log('modpoll abort');
+            return;
+        }
+
+        if (this.readyState != XMLHttpRequest.DONE)
+            return;
+
+        if (this.status !== 200) {
+            console.log('modpoll err ' + this.status + ": " + this.responseText);
+            return;
+        }
+
+        if (!this.responseText)
+            return;
+
+        var server_ref = server_md.replace(/\r/g, '');
+        var server_now = this.responseText.replace(/\r/g, '');
+
+        if (server_ref != server_now) {
+            console.log("modpoll diff |" + server_ref.length + "|, |" + server_now.length + "|");
+            this.modpoll.disabled = true;
+            var msg = [
+                "The document has changed on the server.<br />" +
+                "The changes will NOT be loaded into your editor automatically.",
+
+                "Press F5 or CTRL-R to refresh the page,<br />" +
+                "replacing your document with the server copy.",
+
+                "You can click this message to ignore and contnue."
+            ];
+            return toast(false, "box-shadow:0 1em 2em rgba(64,64,64,0.8);font-weight:normal",
+                36, "<p>" + msg.join('</p>\n<p>') + '</p>');
+        }
+
+        console.log('modpoll eq');
+    }
+
+    if (md_opt.modpoll_freq > 0)
+        this.periodic();
+
+    return this;
+}
+var modpoll = new Modpoll();
+
+
+window.onbeforeunload = function (e) {
+    if ((ebi("save").getAttribute('class') + '').indexOf('disabled') >= 0)
+        return; //nice (todo)
+
+    e.preventDefault(); //ff
+    e.returnValue = ''; //chrome
+};
+
+
 // save handler
 function save(e) {
     if (e) e.preventDefault();
-    var save_btn = document.getElementById("save"),
+    var save_btn = ebi("save"),
         save_cls = save_btn.getAttribute('class') + '';
 
     if (save_cls.indexOf('disabled') >= 0) {
-        alert('there is nothing to save');
+        toast(true, ";font-size:2em;color:#c90", 9, "no changes");
         return;
     }
 
@@ -221,13 +341,15 @@ function save(e) {
     fd.append("lastmod", (force ? -1 : last_modified));
     fd.append("body", txt);
 
-    var url = (document.location + '').split('?')[0] + '?raw';
+    var url = (document.location + '').split('?')[0];
     var xhr = new XMLHttpRequest();
     xhr.open('POST', url, true);
     xhr.responseType = 'text';
     xhr.onreadystatechange = save_cb;
     xhr.btn = save_btn;
     xhr.txt = txt;
+
+    modpoll.skip_one = true;  // skip one iteration while we save
     xhr.send(fd);
 }
 
@@ -272,19 +394,24 @@ function save_cb() {
     this.btn.classList.remove('force-save');
     //alert('save OK -- wrote ' + r.size + ' bytes.\n\nsha512: ' + r.sha512);
 
+    run_savechk(r.lastmod, this.txt, this.btn, 0);
+}
+
+function run_savechk(lastmod, txt, btn, ntry) {
     // download the saved doc from the server and compare
-    var url = (document.location + '').split('?')[0] + '?raw';
+    var url = (document.location + '').split('?')[0] + '?raw&_=' + new Date().getTime();
     var xhr = new XMLHttpRequest();
     xhr.open('GET', url, true);
     xhr.responseType = 'text';
-    xhr.onreadystatechange = save_chk;
-    xhr.btn = this.save_btn;
-    xhr.txt = this.txt;
-    xhr.lastmod = r.lastmod;
+    xhr.onreadystatechange = savechk_cb;
+    xhr.lastmod = lastmod;
+    xhr.txt = txt;
+    xhr.btn = btn;
+    xhr.ntry = ntry;
     xhr.send();
 }
 
-function save_chk() {
+function savechk_cb() {
     if (this.readyState != XMLHttpRequest.DONE)
         return;
 
@@ -296,6 +423,14 @@ function save_chk() {
     var doc1 = this.txt.replace(/\r\n/g, "\n");
     var doc2 = this.responseText.replace(/\r\n/g, "\n");
     if (doc1 != doc2) {
+        var that = this;
+        if (that.ntry < 10) {
+            // qnap funny, try a few more times
+            setTimeout(function () {
+                run_savechk(that.lastmod, that.txt, that.btn, that.ntry + 1)
+            }, 100);
+            return;
+        }
         alert(
             'Error! The document on the server does not appear to have saved correctly (your editor contents and the server copy is not identical). Place the document on your clipboard for now and check the server logs for hints\n\n' +
             'Length: yours=' + doc1.length + ', server=' + doc2.length
@@ -308,28 +443,67 @@ function save_chk() {
     last_modified = this.lastmod;
     server_md = this.txt;
     draw_md();
+    toast(true, ";font-size:6em;font-family:serif;color:#9b4", 4,
+        'OK✔️<span style="font-size:.2em;color:#999;position:absolute">' + this.ntry + '</span>');
 
-    var ok = document.createElement('div');
-    ok.setAttribute('style', 'font-size:6em;font-family:serif;font-weight:bold;color:#cf6;background:#444;border-radius:.3em;padding:.6em 0;position:fixed;top:30%;left:calc(50% - 2em);width:4em;text-align:center;z-index:9001;transition:opacity 0.2s ease-in-out;opacity:1');
-    ok.innerHTML = 'OK✔️';
-    var parent = document.getElementById('m');
-    document.documentElement.appendChild(ok);
-    setTimeout(function () {
-        ok.style.opacity = 0;
-    }, 500);
-    setTimeout(function () {
+    modpoll.disabled = false;
+}
+
+function toast(autoclose, style, width, msg) {
+    var ok = ebi("toast");
+    if (ok)
         ok.parentNode.removeChild(ok);
-    }, 750);
+
+    style = "width:" + width + "em;left:calc(50% - " + (width / 2) + "em);" + style;
+    ok = document.createElement('div');
+    ok.setAttribute('id', 'toast');
+    ok.setAttribute('style', style);
+    ok.innerHTML = msg;
+    var parent = ebi('m');
+    document.documentElement.appendChild(ok);
+
+    var hide = function (delay) {
+        delay = delay || 0;
+
+        setTimeout(function () {
+            ok.style.opacity = 0;
+        }, delay);
+
+        setTimeout(function () {
+            if (ok.parentNode)
+                ok.parentNode.removeChild(ok);
+        }, delay + 250);
+    }
+
+    ok.onclick = function () {
+        hide(0);
+    };
+
+    if (autoclose)
+        hide(500);
 }
 
 
+// firefox bug: initial selection offset isn't cleared properly through js
+var ff_clearsel = (function () {
+    if (navigator.userAgent.indexOf(') Gecko/') === -1)
+        return function () { }
+
+    return function () {
+        var txt = dom_src.value;
+        var y = dom_src.scrollTop;
+        dom_src.value = '';
+        dom_src.value = txt;
+        dom_src.scrollTop = y;
+    };
+})();
+
+
 // returns car/cdr (selection bounds) and n1/n2 (grown to full lines)
-function linebounds(just_car) {
+function linebounds(just_car, greedy_growth) {
     var car = dom_src.selectionStart,
         cdr = dom_src.selectionEnd;
 
-    dbg(car, cdr);
-
     if (just_car)
         cdr = car;
 
@@ -337,11 +511,13 @@ function linebounds(just_car) {
         n1 = Math.max(car, 0),
         n2 = Math.min(cdr, md.length - 1);
 
-    if (n1 < n2 && md[n1] == '\n')
-        n1++;
+    if (greedy_growth !== true) {
+        if (n1 < n2 && md[n1] == '\n')
+            n1++;
 
-    if (n1 < n2 && md[n2 - 1] == '\n')
-        n2 -= 2;
+        if (n1 < n2 && md[n2 - 1] == '\n')
+            n2 -= 2;
+    }
 
     n1 = md.lastIndexOf('\n', n1 - 1) + 1;
     n2 = md.indexOf('\n', n2);
@@ -375,8 +551,11 @@ function setsel(s) {
         s.cdr = s.pre.length + s.sel.length;
     }
     dom_src.value = [s.pre, s.sel, s.post].join('');
-    dom_src.setSelectionRange(s.car, s.cdr);
+    dom_src.setSelectionRange(s.car, s.cdr, dom_src.selectionDirection);
     dom_src.oninput();
+    // support chrome:
+    dom_src.blur();
+    dom_src.focus();
 }
 
 
@@ -416,17 +595,32 @@ function md_header(dedent) {
 
 // smart-home
 function md_home(shift) {
-    var s = linebounds(!shift),
+    var s = linebounds(false, true),
         ln = s.md.substring(s.n1, s.n2),
-        m = /^[ \t#>+-]*(\* )?([0-9]+\. +)?/.exec(ln),
-        home = s.n1 + m[0].length,
-        car = (s.car == home) ? s.n1 : home,
-        cdr = shift ? s.cdr : car;
+        dir = dom_src.selectionDirection,
+        rev = dir === 'backward',
+        p1 = rev ? s.car : s.cdr,
+        p2 = rev ? s.cdr : s.car,
+        home = 0,
+        lf = ln.lastIndexOf('\n') + 1,
+        re = /^[ \t#>+-]*(\* )?([0-9]+\. +)?/;
 
-    if (car > cdr)
-        car = [cdr, cdr = car][0];
+    if (rev)
+        home = s.n1 + re.exec(ln)[0].length;
+    else
+        home = s.n1 + lf + re.exec(ln.substring(lf))[0].length;
 
-    dom_src.setSelectionRange(car, cdr);
+    p1 = (p1 !== home) ? home : (rev ? s.n1 : s.n1 + lf);
+    if (!shift)
+        p2 = p1;
+
+    if (rev !== p1 < p2)
+        dir = rev ? 'forward' : 'backward';
+
+    if (!shift)
+        ff_clearsel();
+
+    dom_src.setSelectionRange(Math.min(p1, p2), Math.max(p1, p2), dir);
 }
 
 
@@ -434,13 +628,291 @@ function md_home(shift) {
 function md_newline() {
     var s = linebounds(true),
         ln = s.md.substring(s.n1, s.n2),
-        m = /^[ \t>+-]*(\* )?([0-9]+\. +)?/.exec(ln);
+        m1 = /^( *)([0-9]+)(\. +)/.exec(ln),
+        m2 = /^[ \t>+-]*(\* )?/.exec(ln),
+        drop = dom_src.selectionEnd - dom_src.selectionStart;
 
-    s.pre = s.md.substring(0, s.car) + '\n' + m[0];
+    var pre = m2[0];
+    if (m1 !== null)
+        pre = m1[1] + (parseInt(m1[2]) + 1) + m1[3];
+
+    if (pre.length > s.car - s.n1)
+        // in gutter, do nothing
+        return true;
+
+    s.pre = s.md.substring(0, s.car) + '\n' + pre;
+    s.sel = '';
+    s.post = s.md.substring(s.car + drop);
+    s.car = s.cdr = s.pre.length;
+    setsel(s);
+    return false;
+}
+
+
+// backspace
+function md_backspace() {
+    var s = linebounds(true),
+        o0 = dom_src.selectionStart,
+        left = s.md.slice(s.n1, o0),
+        m = /^[ \t>+-]*(\* )?([0-9]+\. +)?/.exec(left);
+
+    // if car is in whitespace area, do nothing
+    if (/^\s*$/.test(left))
+        return true;
+
+    // same if line is all-whitespace or non-markup
+    var v = m[0].replace(/[^ ]/g, " ");
+    if (v === m[0] || v.length !== left.length)
+        return true;
+
+    s.pre = s.md.substring(0, s.n1) + v;
     s.sel = '';
     s.post = s.md.substring(s.car);
     s.car = s.cdr = s.pre.length;
     setsel(s);
+    return false;
+}
+
+
+// paragraph jump
+function md_p_jump(down) {
+    var txt = dom_src.value,
+        ofs = dom_src.selectionStart;
+
+    if (down) {
+        while (txt[ofs] == '\n' && --ofs > 0);
+        ofs = txt.indexOf("\n\n", ofs);
+        if (ofs < 0)
+            ofs = txt.length - 1;
+
+        while (txt[ofs] == '\n' && ++ofs < txt.length - 1);
+    }
+    else {
+        txt += '\n\n';
+        while (ofs > 1 && txt[ofs - 1] == '\n') ofs--;
+        ofs = Math.max(0, txt.lastIndexOf("\n\n", ofs - 1));
+        while (txt[ofs] == '\n' && ++ofs < txt.length - 1);
+    }
+
+    dom_src.setSelectionRange(ofs, ofs, "none");
+}
+
+
+function reLastIndexOf(txt, ptn, end) {
+    var ofs = (typeof end !== 'undefined') ? end : txt.length;
+    end = ofs;
+    while (ofs >= 0) {
+        var sub = txt.slice(ofs, end);
+        if (ptn.test(sub))
+            return ofs;
+
+        ofs--;
+    }
+    return -1;
+}
+
+
+// table formatter
+function fmt_table(e) {
+    if (e) e.preventDefault();
+    //dom_tbox.setAttribute('class', '');
+
+    var txt = dom_src.value,
+        ofs = dom_src.selectionStart,
+        //o0 = txt.lastIndexOf('\n\n', ofs),
+        //o1 = txt.indexOf('\n\n', ofs);
+        o0 = reLastIndexOf(txt, /\n\s*\n/m, ofs),
+        o1 = txt.slice(ofs).search(/\n\s*\n|\n\s*$/m);
+    // note \s contains \n but its fine
+
+    if (o0 < 0)
+        o0 = 0;
+    else {
+        // seek past the hit
+        var m = /\n\s*\n/m.exec(txt.slice(o0));
+        o0 += m[0].length;
+    }
+
+    o1 = o1 < 0 ? txt.length : o1 + ofs;
+
+    var err = 'cannot format table due to ',
+        tab = txt.slice(o0, o1).split(/\s*\n/),
+        re_ind = /^\s*/,
+        ind = tab[1].match(re_ind)[0],
+        r0_ind = tab[0].slice(0, ind.length),
+        lpipe = tab[1].indexOf('|') < tab[1].indexOf('-'),
+        rpipe = tab[1].lastIndexOf('|') > tab[1].lastIndexOf('-'),
+        re_lpipe = lpipe ? /^\s*\|\s*/ : /^\s*/,
+        re_rpipe = rpipe ? /\s*\|\s*$/ : /\s*$/,
+        ncols;
+
+    // the second row defines the table,
+    // need to process that first
+    var tmp = tab[0];
+    tab[0] = tab[1];
+    tab[1] = tmp;
+
+    for (var a = 0; a < tab.length; a++) {
+        var row_name = (a == 1) ? 'header' : 'row#' + (a + 1);
+
+        var ind2 = tab[a].match(re_ind)[0];
+        if (ind != ind2 && a != 1)  // the table can be a list entry or something, ignore [0]
+            return alert(err + 'indentation mismatch on row#2 and ' + row_name + ',\n' + tab[a]);
+
+        var t = tab[a].slice(ind.length);
+        t = t.replace(re_lpipe, "");
+        t = t.replace(re_rpipe, "");
+        tab[a] = t.split(/\s*\|\s*/g);
+
+        if (a == 0)
+            ncols = tab[a].length;
+        else if (ncols < tab[a].length)
+            return alert(err + 'num.columns(' + row_name + ') exceeding row#2;  ' + ncols + ' < ' + tab[a].length);
+
+        // if row has less columns than row2, fill them in
+        while (tab[a].length < ncols)
+            tab[a].push('');
+    }
+
+    // aight now swap em back
+    tmp = tab[0];
+    tab[0] = tab[1];
+    tab[1] = tmp;
+
+    var re_align = /^ *(:?)-+(:?) *$/;
+    var align = [];
+    for (var col = 0; col < tab[1].length; col++) {
+        var m = tab[1][col].match(re_align);
+        if (!m)
+            return alert(err + 'invalid column specification, row#2, col ' + (col + 1) + ', [' + tab[1][col] + ']');
+
+        if (m[2]) {
+            if (m[1])
+                align.push('c');
+            else
+                align.push('r');
+        }
+        else
+            align.push('l');
+    }
+
+    var pad = [];
+    var tmax = 0;
+    for (var col = 0; col < ncols; col++) {
+        var max = 0;
+        for (var row = 0; row < tab.length; row++)
+            if (row != 1)
+                max = Math.max(max, tab[row][col].length);
+
+        var s = '';
+        for (var n = 0; n < max; n++)
+            s += ' ';
+
+        pad.push(s);
+        tmax = Math.max(max, tmax);
+    }
+
+    var dashes = '';
+    for (var a = 0; a < tmax; a++)
+        dashes += '-';
+
+    var ret = [];
+    for (var row = 0; row < tab.length; row++) {
+        var ln = [];
+        for (var col = 0; col < tab[row].length; col++) {
+            var p = pad[col];
+            var s = tab[row][col];
+
+            if (align[col] == 'l') {
+                s = (s + p).slice(0, p.length);
+            }
+            else if (align[col] == 'r') {
+                s = (p + s).slice(-p.length);
+            }
+            else {
+                var pt = p.length - s.length;
+                var pl = p.slice(0, Math.floor(pt / 2));
+                var pr = p.slice(0, pt - pl.length);
+                s = pl + s + pr;
+            }
+
+            if (row == 1) {
+                if (align[col] == 'l')
+                    s = dashes.slice(0, p.length);
+                else if (align[col] == 'r')
+                    s = dashes.slice(0, p.length - 1) + ':';
+                else
+                    s = ':' + dashes.slice(0, p.length - 2) + ':';
+            }
+            ln.push(s);
+        }
+        ret.push(ind + '| ' + ln.join(' | ') + ' |');
+    }
+
+    // restore any markup in the row0 gutter
+    ret[0] = r0_ind + ret[0].slice(ind.length);
+
+    ret = {
+        "pre": txt.slice(0, o0),
+        "sel": ret.join('\n'),
+        "post": txt.slice(o1),
+        "car": o0,
+        "cdr": o0
+    };
+    setsel(ret);
+}
+
+
+// show unicode
+function mark_uni(e) {
+    if (e) e.preventDefault();
+    dom_tbox.setAttribute('class', '');
+
+    var txt = dom_src.value,
+        ptn = new RegExp('([^' + js_uni_whitelist + ']+)', 'g'),
+        mod = txt.replace(/\r/g, "").replace(ptn, "\u2588\u2770$1\u2771");
+
+    if (txt == mod) {
+        alert('no results;  no modifications were made');
+        return;
+    }
+    dom_src.value = mod;
+}
+
+
+// iterate unicode
+function iter_uni(e) {
+    if (e) e.preventDefault();
+
+    var txt = dom_src.value,
+        ofs = dom_src.selectionDirection == "forward" ? dom_src.selectionEnd : dom_src.selectionStart,
+        re = new RegExp('([^' + js_uni_whitelist + ']+)'),
+        m = re.exec(txt.slice(ofs));
+
+    if (!m) {
+        alert('no more hits from cursor onwards');
+        return;
+    }
+    ofs += m.index;
+
+    dom_src.setSelectionRange(ofs, ofs + m[0].length, "forward");
+    dom_src.oninput();
+    // support chrome:
+    dom_src.blur();
+    dom_src.focus();
+}
+
+
+// configure whitelist
+function cfg_uni(e) {
+    if (e) e.preventDefault();
+
+    var reply = prompt("unicode whitelist", esc_uni_whitelist);
+    if (reply === null)
+        return;
+
+    esc_uni_whitelist = reply;
+    js_uni_whitelist = eval('\'' + esc_uni_whitelist + '\'');
 }
 
 
@@ -455,6 +927,11 @@ function md_newline() {
             save();
             return false;
         }
+        if (ev.code == "Escape" || kc == 27) {
+            var d = ebi('helpclose');
+            if (d)
+                d.click();
+        }
         if (document.activeElement == dom_src) {
             if (ev.code == "Tab" || kc == 9) {
                 md_indent(ev.shiftKey);
@@ -469,8 +946,7 @@ function md_newline() {
                 return false;
             }
             if (!ctrl && !ev.shiftKey && (ev.code == "Enter" || kc == 13)) {
-                md_newline();
-                return false;
+                return md_newline();
             }
             if (ctrl && (ev.code == "KeyZ" || kc == 90)) {
                 if (ev.shiftKey)
@@ -484,31 +960,72 @@ function md_newline() {
                 action_stack.redo();
                 return false;
             }
+            if (!ctrl && !ev.shiftKey && kc == 8) {
+                return md_backspace();
+            }
+            if (ctrl && (ev.code == "KeyK")) {
+                fmt_table();
+                return false;
+            }
+            if (ctrl && (ev.code == "KeyU")) {
+                iter_uni();
+                return false;
+            }
+            if (ctrl && (ev.code == "KeyE")) {
+                dom_nsbs.click();
+                //fmt_table();
+                return false;
+            }
+            var up = ev.code == "ArrowUp" || kc == 38;
+            var dn = ev.code == "ArrowDown" || kc == 40;
+            if (ctrl && (up || dn)) {
+                md_p_jump(dn);
+                return false;
+            }
         }
     }
     document.onkeydown = keydown;
-    document.getElementById('save').onclick = save;
+    ebi('save').onclick = save;
 })();
 
 
-document.getElementById('help').onclick = function (e) {
+ebi('tools').onclick = function (e) {
     if (e) e.preventDefault();
-    var dom = document.getElementById('helpbox');
+    var is_open = dom_tbox.getAttribute('class') != 'open';
+    dom_tbox.setAttribute('class', is_open ? 'open' : '');
+};
+
+
+ebi('help').onclick = function (e) {
+    if (e) e.preventDefault();
+    dom_tbox.setAttribute('class', '');
+
+    var dom = ebi('helpbox');
     var dtxt = dom.getElementsByTagName('textarea');
-    if (dtxt.length > 0)
-        dom.innerHTML = '<a href="#" id="helpclose">close</a>' + marked(dtxt[0].value);
+    if (dtxt.length > 0) {
+        convert_markdown(dtxt[0].value, dom);
+        dom.innerHTML = '<a href="#" id="helpclose">close</a>' + dom.innerHTML;
+    }
 
     dom.style.display = 'block';
-    document.getElementById('helpclose').onclick = function () {
+    ebi('helpclose').onclick = function () {
         dom.style.display = 'none';
     };
 };
 
 
+ebi('fmt_table').onclick = fmt_table;
+ebi('mark_uni').onclick = mark_uni;
+ebi('iter_uni').onclick = iter_uni;
+ebi('cfg_uni').onclick = cfg_uni;
+
+
 // blame steen
 action_stack = (function () {
-    var undos = [];
-    var redos = [];
+    var hist = {
+        un: [],
+        re: []
+    };
     var sched_cpos = 0;
     var sched_timer = null;
     var ignore = false;
@@ -553,7 +1070,7 @@ action_stack = (function () {
     }
 
     function apply(src, dst) {
-        dbg('undos(%d) redos(%d)', undos.length, redos.length);
+        dbg('undos(%d) redos(%d)', hist.un.length, hist.re.length);
 
         if (src.length === 0)
             return false;
@@ -572,7 +1089,7 @@ action_stack = (function () {
         dom_src.value = ref;
         dom_src.setSelectionRange(cpos, cpos);
         ignore = true; // all browsers
-        draw_md();
+        dom_src.oninput();
         return true;
     }
 
@@ -581,51 +1098,49 @@ action_stack = (function () {
             ignore = false;
             return;
         }
-        redos = [];
+        hist.re = [];
         clearTimeout(sched_timer);
         sched_cpos = dom_src.selectionEnd;
         sched_timer = setTimeout(push, 500);
     }
 
     function undo() {
-        if (redos.length == 0) {
+        if (hist.re.length == 0) {
             clearTimeout(sched_timer);
             push();
         }
-        return apply(undos, redos);
+        return apply(hist.un, hist.re);
     }
 
     function redo() {
-        return apply(redos, undos);
+        return apply(hist.re, hist.un);
     }
 
     function push() {
         var newtxt = dom_src.value;
         var change = diff(ref, newtxt, sched_cpos);
         if (change !== null)
-            undos.push(change);
+            hist.un.push(change);
 
         ref = newtxt;
-        dbg('undos(%d) redos(%d)', undos.length, redos.length);
-        if (undos.length > 0)
-            dbg(undos.slice(-1)[0]);
-        if (redos.length > 0)
-            dbg(redos.slice(-1)[0]);
+        dbg('undos(%d) redos(%d)', hist.un.length, hist.re.length);
+        if (hist.un.length > 0)
+            dbg(statify(hist.un.slice(-1)[0]));
+        if (hist.re.length > 0)
+            dbg(statify(hist.re.slice(-1)[0]));
     }
 
     return {
-        push: push,
         undo: undo,
         redo: redo,
         push: schedule_push,
-        _undos: undos,
-        _redos: redos,
+        _hist: hist,
         _ref: ref
     }
 })();
 
 /*
-document.getElementById('help').onclick = function () {
+ebi('help').onclick = function () {
     var c1 = getComputedStyle(dom_src).cssText.split(';');
     var c2 = getComputedStyle(dom_ref).cssText.split(';');
     var max = Math.min(c1.length, c2.length);
@@ -633,4 +1148,4 @@ document.getElementById('help').onclick = function () {
         if (c1[a] !== c2[a])
             console.log(c1[a] + '\n' + c2[a]);
 }
-*/
+*/

copyparty/web/mde.css

@@ -160,8 +160,12 @@ h2 {
 .mdo ol>li {
 	margin: .7em 0;
 }
+strong {
+	color: #000;
+}
 p>em,
-li>em {
+li>em,
+td>em {
 	color: #c50;
 	padding: .1em;
 	border-bottom: .1em solid #bbb;
@@ -253,8 +257,12 @@ html.dark .mdo>ul,
 html.dark .mdo>ol {
     border-color: #555;
 }
+html.dark strong {
+    color: #fff;
+}
 html.dark p>em,
-html.dark li>em {
+html.dark li>em,
+html.dark td>em {
     color: #f94;
     border-color: #666;
 }

copyparty/web/mde.html

@@ -17,13 +17,17 @@
 			</div>
 		</div>
 		<div id="m">
-			<textarea id="mt" style="display:none">{{ md }}</textarea>
+			<textarea id="mt" style="display:none" autocomplete="off">{{ md }}</textarea>
 		</div>
 	</div>
 	<script>
 
-var link_md_as_html = false;  // TODO (does nothing)
 var last_modified = {{ lastmod }};
+var md_opt = {
+	link_md_as_html: false,
+	allow_plugins: {{ md_plug }},
+	modpoll_freq: {{ md_chk_rate }}
+};
 
 var lightswitch = (function () {
 	var fun = function () {
@@ -39,6 +43,7 @@ var lightswitch = (function () {
 })();
 
 	</script>
-	<script src="/.cpr/deps/easymde.full.js"></script>
+    <script src="/.cpr/util.js"></script>
+	<script src="/.cpr/deps/easymde.js"></script>
 	<script src="/.cpr/mde.js"></script>
 </body></html>

copyparty/web/mde.js

@@ -1,7 +1,9 @@
-var dom_wrap = document.getElementById('mw');
-var dom_nav = document.getElementById('mn');
-var dom_doc = document.getElementById('m');
-var dom_md = document.getElementById('mt');
+"use strict";
+
+var dom_wrap = ebi('mw');
+var dom_nav = ebi('mn');
+var dom_doc = ebi('m');
+var dom_md = ebi('mt');
 
 (function () {
     var n = document.location + '';
@@ -63,7 +65,7 @@ var mde = (function () {
     mde.codemirror.on("change", function () {
         md_changed(mde);
     });
-    var loader = document.getElementById('ml');
+    var loader = ebi('ml');
     loader.parentNode.removeChild(loader);
     return mde;
 })();
@@ -121,7 +123,7 @@ function save(mde) {
     fd.append("lastmod", (force ? -1 : last_modified));
     fd.append("body", txt);
 
-    var url = (document.location + '').split('?')[0] + '?raw';
+    var url = (document.location + '').split('?')[0];
     var xhr = new XMLHttpRequest();
     xhr.open('POST', url, true);
     xhr.responseType = 'text';
@@ -213,7 +215,7 @@ function save_chk() {
     var ok = document.createElement('div');
     ok.setAttribute('style', 'font-size:6em;font-family:serif;font-weight:bold;color:#cf6;background:#444;border-radius:.3em;padding:.6em 0;position:fixed;top:30%;left:calc(50% - 2em);width:4em;text-align:center;z-index:9001;transition:opacity 0.2s ease-in-out;opacity:1');
     ok.innerHTML = 'OK✔️';
-    var parent = document.getElementById('m');
+    var parent = ebi('m');
     document.documentElement.appendChild(ok);
     setTimeout(function () {
         ok.style.opacity = 0;

copyparty/web/up2k.js

@@ -1,61 +1,6 @@
 "use strict";
 
-// error handler for mobile devices
-function hcroak(msg) {
-    document.body.innerHTML = msg;
-    window.onerror = undefined;
-    throw 'fatal_err';
-}
-function croak(msg) {
-    document.body.textContent = msg;
-    window.onerror = undefined;
-    throw msg;
-}
-function esc(txt) {
-    return txt.replace(/[&"<>]/g, function (c) {
-        return {
-            '&': '&amp;',
-            '"': '&quot;',
-            '<': '&lt;',
-            '>': '&gt;'
-        }[c];
-    });
-}
-window.onerror = function (msg, url, lineNo, columnNo, error) {
-    window.onerror = undefined;
-    var html = ['<h1>you hit a bug!</h1><p>please screenshot this error and send me a copy arigathanks gozaimuch (ed/irc.rizon.net or ed#2644)</p><p>',
-        esc(String(msg)), '</p><p>', esc(url + ' @' + lineNo + ':' + columnNo), '</p>'];
-
-    if (error) {
-        var find = ['desc', 'stack', 'trace'];
-        for (var a = 0; a < find.length; a++)
-            if (String(error[find[a]]) !== 'undefined')
-                html.push('<h2>' + find[a] + '</h2>' +
-                    esc(String(error[find[a]])).replace(/\n/g, '<br />\n'));
-    }
-    document.body.style.fontSize = '0.8em';
-    document.body.style.padding = '0 1em 1em 1em';
-    hcroak(html.join('\n'));
-};
-
-
-// https://stackoverflow.com/a/950146
-function import_js(url, cb) {
-    var head = document.head || document.getElementsByTagName('head')[0];
-    var script = document.createElement('script');
-    script.type = 'text/javascript';
-    script.src = url;
-
-    script.onreadystatechange = cb;
-    script.onload = cb;
-
-    head.appendChild(script);
-}
-
-
-function o(id) {
-    return document.getElementById(id);
-}
+window.onerror = vis_exh;
 
 
 (function () {
@@ -88,12 +33,12 @@ function goto(dest) {
     for (var a = obj.length - 1; a >= 0; a--)
         obj[a].classList.remove('act');
 
-    var obj = document.querySelectorAll('#ops>a');
+    obj = document.querySelectorAll('#ops>a');
     for (var a = obj.length - 1; a >= 0; a--)
         obj[a].classList.remove('act');
 
     if (dest) {
-        document.getElementById('op_' + dest).classList.add('act');
+        ebi('op_' + dest).classList.add('act');
         document.querySelector('#ops>a[data-dest=' + dest + ']').classList.add('act');
 
         var fn = window['goto_' + dest];
@@ -121,7 +66,7 @@ function goto_up2k() {
         if (op !== null && op !== '.')
             goto(op);
     }
-    document.getElementById('ops').style.display = 'block';
+    ebi('ops').style.display = 'block';
 })();
 
 
@@ -150,21 +95,21 @@ function up2k_init(have_crypto) {
 
     // show modal message
     function showmodal(msg) {
-        o('u2notbtn').innerHTML = msg;
-        o('u2btn').style.display = 'none';
-        o('u2notbtn').style.display = 'block';
-        o('u2conf').style.opacity = '0.5';
+        ebi('u2notbtn').innerHTML = msg;
+        ebi('u2btn').style.display = 'none';
+        ebi('u2notbtn').style.display = 'block';
+        ebi('u2conf').style.opacity = '0.5';
     }
 
     // hide modal message
     function unmodal() {
-        o('u2notbtn').style.display = 'none';
-        o('u2btn').style.display = 'block';
-        o('u2conf').style.opacity = '1';
-        o('u2notbtn').innerHTML = '';
+        ebi('u2notbtn').style.display = 'none';
+        ebi('u2btn').style.display = 'block';
+        ebi('u2conf').style.opacity = '1';
+        ebi('u2notbtn').innerHTML = '';
     }
 
-    var post_url = o('op_bup').getElementsByTagName('form')[0].getAttribute('action');
+    var post_url = ebi('op_bup').getElementsByTagName('form')[0].getAttribute('action');
     if (post_url && post_url.charAt(post_url.length - 1) !== '/')
         post_url += '/';
 
@@ -181,25 +126,25 @@ function up2k_init(have_crypto) {
             import_js('/.cpr/deps/sha512.js', unmodal);
 
             if (is_https)
-                o('u2foot').innerHTML = shame + ' so <em>this</em> uploader will do like 500kB/s at best';
+                ebi('u2foot').innerHTML = shame + ' so <em>this</em> uploader will do like 500kB/s at best';
             else
-                o('u2foot').innerHTML = 'seems like ' + shame + ' so do that if you want more performance';
+                ebi('u2foot').innerHTML = 'seems like ' + shame + ' so do that if you want more performance';
         }
-    };
+    }
 
     // show uploader if the user only has write-access
-    if (!o('files'))
+    if (!ebi('files'))
         goto('up2k');
 
     // shows or clears an error message in the basic uploader ui
     function setmsg(msg) {
         if (msg !== undefined) {
-            o('u2err').setAttribute('class', 'err');
-            o('u2err').innerHTML = msg;
+            ebi('u2err').setAttribute('class', 'err');
+            ebi('u2err').innerHTML = msg;
         }
         else {
-            o('u2err').setAttribute('class', '');
-            o('u2err').innerHTML = '';
+            ebi('u2err').setAttribute('class', '');
+            ebi('u2err').innerHTML = '';
         }
     }
 
@@ -210,7 +155,7 @@ function up2k_init(have_crypto) {
     }
 
     // handle user intent to use the basic uploader instead
-    o('u2nope').onclick = function (e) {
+    ebi('u2nope').onclick = function (e) {
         e.preventDefault();
         setmsg('');
         goto('bup');
@@ -229,9 +174,9 @@ function up2k_init(have_crypto) {
     function cfg_get(name) {
         var val = localStorage.getItem(name);
         if (val === null)
-            return parseInt(o(name).value);
+            return parseInt(ebi(name).value);
 
-        o(name).value = val;
+        ebi(name).value = val;
         return val;
     }
 
@@ -242,7 +187,7 @@ function up2k_init(have_crypto) {
         else
             val = (val == '1');
 
-        o(name).checked = val;
+        ebi(name).checked = val;
         return val;
     }
 
@@ -250,7 +195,7 @@ function up2k_init(have_crypto) {
         localStorage.setItem(
             name, val ? '1' : '0');
 
-        o(name).checked = val;
+        ebi(name).checked = val;
         return val;
     }
 
@@ -284,9 +229,9 @@ function up2k_init(have_crypto) {
         return un2k("this is the basic uploader; up2k needs at least<br />chrome 21 // firefox 13 // edge 12 // opera 12 // safari 5.1");
 
     function nav() {
-        o('file' + fdom_ctr).click();
+        ebi('file' + fdom_ctr).click();
     }
-    o('u2btn').addEventListener('click', nav, false);
+    ebi('u2btn').addEventListener('click', nav, false);
 
     function ondrag(ev) {
         ev.stopPropagation();
@@ -294,8 +239,8 @@ function up2k_init(have_crypto) {
         ev.dataTransfer.dropEffect = 'copy';
         ev.dataTransfer.effectAllowed = 'copy';
     }
-    o('u2btn').addEventListener('dragover', ondrag, false);
-    o('u2btn').addEventListener('dragenter', ondrag, false);
+    ebi('u2btn').addEventListener('dragover', ondrag, false);
+    ebi('u2btn').addEventListener('dragenter', ondrag, false);
 
     function gotfile(ev) {
         ev.stopPropagation();
@@ -357,7 +302,7 @@ function up2k_init(have_crypto) {
             var tr = document.createElement('tr');
             tr.innerHTML = '<td id="f{0}n"></td><td id="f{0}t">hashing</td><td id="f{0}p" class="prog"></td>'.format(st.files.length);
             tr.getElementsByTagName('td')[0].textContent = entry.name;
-            o('u2tab').appendChild(tr);
+            ebi('u2tab').appendChild(tr);
 
             st.files.push(entry);
             st.todo.hash.push(entry);
@@ -374,14 +319,14 @@ function up2k_init(have_crypto) {
             alert(msg);
         }
     }
-    o('u2btn').addEventListener('drop', gotfile, false);
+    ebi('u2btn').addEventListener('drop', gotfile, false);
 
     function more_one_file() {
         fdom_ctr++;
         var elm = document.createElement('div')
         elm.innerHTML = '<input id="file{0}" type="file" name="file{0}[]" multiple="multiple" />'.format(fdom_ctr);
-        o('u2form').appendChild(elm);
-        o('file' + fdom_ctr).addEventListener('change', gotfile, false);
+        ebi('u2form').appendChild(elm);
+        ebi('file' + fdom_ctr).addEventListener('change', gotfile, false);
     }
     more_one_file();
 
@@ -451,17 +396,6 @@ function up2k_init(have_crypto) {
     ///   hashing
     //
 
-    // https://gist.github.com/jonleighton/958841
-    function buf2b64_maybe_fucky(buffer) {
-        var ret = '';
-        var view = new DataView(buffer);
-        for (var i = 0; i < view.byteLength; i++) {
-            ret += String.fromCharCode(view.getUint8(i));
-        }
-        return window.btoa(ret).replace(
-            /\+/g, '-').replace(/\//g, '_').replace(/=+$/, '');
-    }
-
     // https://gist.github.com/jonleighton/958841
     function buf2b64(arrayBuffer) {
         var base64 = '';
@@ -502,20 +436,6 @@ function up2k_init(have_crypto) {
         return base64;
     }
 
-    // https://developer.mozilla.org/en-US/docs/Web/API/SubtleCrypto/digest
-    function buf2hex(buffer) {
-        var hexCodes = [];
-        var view = new DataView(buffer);
-        for (var i = 0; i < view.byteLength; i += 4) {
-            var value = view.getUint32(i) // 4 bytes per iter
-            var stringValue = value.toString(16) // doesn't pad
-            var padding = '00000000'
-            var paddedValue = (padding + stringValue).slice(-padding.length)
-            hexCodes.push(paddedValue);
-        }
-        return hexCodes.join("");
-    }
-
     function get_chunksize(filesize) {
         var chunksize = 1024 * 1024;
         var stepsize = 512 * 1024;
@@ -602,7 +522,7 @@ function up2k_init(have_crypto) {
             pb_html += '<div id="f{0}p{1}" style="width:{2}%"><div></div></div>'.format(
                 t.n, a, pb_perc);
 
-        o('f{0}p'.format(t.n)).innerHTML = pb_html;
+        ebi('f{0}p'.format(t.n)).innerHTML = pb_html;
 
         var reader = new FileReader();
 
@@ -677,7 +597,7 @@ function up2k_init(have_crypto) {
                 alert('{0} ms, {1} MB/s\n'.format(t.t2 - t.t1, spd.toFixed(3)) + t.hash.join('\n'));
             }
 
-            o('f{0}t'.format(t.n)).innerHTML = 'connecting';
+            ebi('f{0}t'.format(t.n)).innerHTML = 'connecting';
             st.busy.hash.splice(st.busy.hash.indexOf(t), 1);
             st.todo.handshake.push(t);
         };
@@ -706,7 +626,7 @@ function up2k_init(have_crypto) {
                 if (response.name !== t.name) {
                     // file exists; server renamed us
                     t.name = response.name;
-                    o('f{0}n'.format(t.n)).textContent = t.name;
+                    ebi('f{0}n'.format(t.n)).textContent = t.name;
                 }
 
                 t.postlist = [];
@@ -736,13 +656,13 @@ function up2k_init(have_crypto) {
                     msg = 'uploading';
                     done = false;
                 }
-                o('f{0}t'.format(t.n)).innerHTML = msg;
+                ebi('f{0}t'.format(t.n)).innerHTML = msg;
                 st.busy.handshake.splice(st.busy.handshake.indexOf(t), 1);
 
                 if (done) {
                     var spd1 = (t.size / ((t.t2 - t.t1) / 1000.)) / (1024 * 1024.);
                     var spd2 = (t.size / ((t.t3 - t.t2) / 1000.)) / (1024 * 1024.);
-                    o('f{0}p'.format(t.n)).innerHTML = 'hash {0}, up {1} MB/s'.format(
+                    ebi('f{0}p'.format(t.n)).innerHTML = 'hash {0}, up {1} MB/s'.format(
                         spd1.toFixed(2), spd2.toFixed(2));
                 }
                 tasker();
@@ -803,7 +723,7 @@ function up2k_init(have_crypto) {
                     t.postlist.splice(t.postlist.indexOf(npart), 1);
                     if (t.postlist.length == 0) {
                         t.t3 = new Date().getTime();
-                        o('f{0}t'.format(t.n)).innerHTML = 'verifying';
+                        ebi('f{0}t'.format(t.n)).innerHTML = 'verifying';
                         st.todo.handshake.push(t);
                     }
                     tasker();
@@ -834,7 +754,7 @@ function up2k_init(have_crypto) {
     //
 
     function prog(nfile, nchunk, color, percent) {
-        var n1 = o('f{0}p{1}'.format(nfile, nchunk));
+        var n1 = ebi('f{0}p{1}'.format(nfile, nchunk));
         var n2 = n1.getElementsByTagName('div')[0];
         if (percent === undefined) {
             n1.style.background = color;
@@ -857,7 +777,7 @@ function up2k_init(have_crypto) {
             dir.preventDefault();
         } catch (ex) { }
 
-        var obj = o('nthread');
+        var obj = ebi('nthread');
         if (dir.target) {
             obj.style.background = '#922';
             var v = Math.floor(parseInt(obj.value));
@@ -892,19 +812,19 @@ function up2k_init(have_crypto) {
         this.click();
     }
 
-    o('nthread_add').onclick = function (ev) {
+    ebi('nthread_add').onclick = function (ev) {
         ev.preventDefault();
         bumpthread(1);
     };
-    o('nthread_sub').onclick = function (ev) {
+    ebi('nthread_sub').onclick = function (ev) {
         ev.preventDefault();
         bumpthread(-1);
     };
 
-    o('nthread').addEventListener('input', bumpthread, false);
-    o('multitask').addEventListener('click', tgl_multitask, false);
+    ebi('nthread').addEventListener('input', bumpthread, false);
+    ebi('multitask').addEventListener('click', tgl_multitask, false);
 
-    var nodes = o('u2conf').getElementsByTagName('a');
+    var nodes = ebi('u2conf').getElementsByTagName('a');
     for (var a = nodes.length - 1; a >= 0; a--)
         nodes[a].addEventListener('touchend', nop, false);
 

copyparty/web/util.js

@@ -0,0 +1,109 @@
+"use strict";
+
+// error handler for mobile devices
+function hcroak(msg) {
+    document.body.innerHTML = msg;
+    window.onerror = undefined;
+    throw 'fatal_err';
+}
+function croak(msg) {
+    document.body.textContent = msg;
+    window.onerror = undefined;
+    throw msg;
+}
+function esc(txt) {
+    return txt.replace(/[&"<>]/g, function (c) {
+        return {
+            '&': '&amp;',
+            '"': '&quot;',
+            '<': '&lt;',
+            '>': '&gt;'
+        }[c];
+    });
+}
+function vis_exh(msg, url, lineNo, columnNo, error) {
+    window.onerror = undefined;
+    var html = ['<h1>you hit a bug!</h1><p>please screenshot this error and send me a copy arigathanks gozaimuch (ed/irc.rizon.net or ed#2644)</p><p>',
+        esc(String(msg)), '</p><p>', esc(url + ' @' + lineNo + ':' + columnNo), '</p>'];
+
+    if (error) {
+        var find = ['desc', 'stack', 'trace'];
+        for (var a = 0; a < find.length; a++)
+            if (String(error[find[a]]) !== 'undefined')
+                html.push('<h2>' + find[a] + '</h2>' +
+                    esc(String(error[find[a]])).replace(/\n/g, '<br />\n'));
+    }
+    document.body.style.fontSize = '0.8em';
+    document.body.style.padding = '0 1em 1em 1em';
+    hcroak(html.join('\n'));
+}
+
+
+function ebi(id) {
+    return document.getElementById(id);
+}
+
+
+// https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Global_Objects/String/endsWith
+if (!String.prototype.endsWith) {
+    String.prototype.endsWith = function (search, this_len) {
+        if (this_len === undefined || this_len > this.length) {
+            this_len = this.length;
+        }
+        return this.substring(this_len - search.length, this_len) === search;
+    };
+}
+if (!String.startsWith) {
+    String.prototype.startsWith = function (s, i) {
+        i = i > 0 ? i | 0 : 0;
+        return this.substring(i, i + s.length) === s;
+    };
+}
+
+
+// https://stackoverflow.com/a/950146
+function import_js(url, cb) {
+    var head = document.head || document.getElementsByTagName('head')[0];
+    var script = document.createElement('script');
+    script.type = 'text/javascript';
+    script.src = url;
+
+    script.onreadystatechange = cb;
+    script.onload = cb;
+
+    head.appendChild(script);
+}
+
+
+function sortTable(table, col) {
+    var tb = table.tBodies[0], // use `<tbody>` to ignore `<thead>` and `<tfoot>` rows
+        th = table.tHead.rows[0].cells,
+        tr = Array.prototype.slice.call(tb.rows, 0),
+        i, reverse = th[col].className == 'sort1' ? -1 : 1;
+    for (var a = 0, thl = th.length; a < thl; a++)
+        th[a].className = '';
+    th[col].className = 'sort' + reverse;
+    var stype = th[col].getAttribute('sort');
+    tr = tr.sort(function (a, b) {
+        var v1 = a.cells[col].textContent.trim();
+        var v2 = b.cells[col].textContent.trim();
+        if (stype == 'int') {
+            v1 = parseInt(v1.replace(/,/g, ''));
+            v2 = parseInt(v2.replace(/,/g, ''));
+            return reverse * (v1 - v2);
+        }
+        return reverse * (v1.localeCompare(v2));
+    });
+    for (i = 0; i < tr.length; ++i) tb.appendChild(tr[i]);
+}
+function makeSortable(table) {
+    var th = table.tHead, i;
+    th && (th = th.rows[0]) && (th = th.cells);
+    if (th) i = th.length;
+    else return; // if no `<thead>` then do nothing
+    while (--i >= 0) (function (i) {
+        th[i].onclick = function () {
+            sortTable(table, i);
+        };
+    }(i));
+}

docs/notes.sh

@@ -13,7 +13,7 @@ head -c $((2*1024*1024*1024)) /dev/zero | openssl enc -aes-256-ctr -pass pass:hu
 ## testing multiple parallel uploads
 ## usage:  para | tee log
 
-para() { for s in 1 2 3 4 5 6 7 8 12 16 24 32 48 64; do echo $s; for r in {1..4}; do for ((n=0;n<s;n++)); do curl -sF "act=bput" -F "f=@garbage.file" http://127.0.0.1:1234/ 2>&1 & done; wait; echo; done; done; }
+para() { for s in 1 2 3 4 5 6 7 8 12 16 24 32 48 64; do echo $s; for r in {1..4}; do for ((n=0;n<s;n++)); do curl -sF "act=bput" -F "f=@garbage.file" http://127.0.0.1:3923/ 2>&1 & done; wait; echo; done; done; }
 
 
 ##
@@ -36,13 +36,13 @@ for dir in "${dirs[@]}"; do for fn in ふが "$(printf \\xed\\x93)" 'qwe,rty;asd
 
 fn=$(printf '\xba\xdc\xab.cab')
 echo asdf > "$fn"
-curl --cookie cppwd=wark -sF "act=bput" -F "f=@$fn" http://127.0.0.1:1234/moji/%ED%91/
+curl --cookie cppwd=wark -sF "act=bput" -F "f=@$fn" http://127.0.0.1:3923/moji/%ED%91/
 
 
 ##
 ## test compression
 
-wget -S --header='Accept-Encoding: gzip' -U 'MSIE 6.0; SV1' http://127.0.0.1:1234/.cpr/deps/ogv.js -O- | md5sum; p=~ed/dev/copyparty/copyparty/web/deps/ogv.js.gz; md5sum $p; gzip -d < $p | md5sum
+wget -S --header='Accept-Encoding: gzip' -U 'MSIE 6.0; SV1' http://127.0.0.1:3923/.cpr/deps/ogv.js -O- | md5sum; p=~ed/dev/copyparty/copyparty/web/deps/ogv.js.gz; md5sum $p; gzip -d < $p | md5sum
 
 
 ##
@@ -80,3 +80,45 @@ for d in /usr /var; do find $d -type f -size +30M 2>/dev/null; done | while IFS=
 # py2 on osx
 brew install python@2
 pip install virtualenv
+
+
+##
+## http 206
+
+# az = abcdefghijklmnopqrstuvwxyz
+
+printf '%s\r\n' 'GET /az HTTP/1.1' 'Host: ocv.me' 'Range: bytes=5-10' '' | ncat ocv.me 80 
+# Content-Range: bytes 5-10/26
+# Content-Length: 6
+# fghijk
+
+Range: bytes=0-1    "ab" Content-Range: bytes 0-1/26
+Range: bytes=24-24  "y"  Content-Range: bytes 24-24/26
+Range: bytes=24-25  "yz" Content-Range: bytes 24-25/26
+Range: bytes=24-    "yz" Content-Range: bytes 24-25/26
+Range: bytes=25-29  "z"  Content-Range: bytes 25-25/26
+Range: bytes=26-         Content-Range: bytes */26
+  HTTP/1.1 416 Requested Range Not Satisfiable
+
+
+##
+## md perf
+
+var tsh = [];
+function convert_markdown(md_text, dest_dom) {
+    tsh.push(new Date().getTime());
+    while (tsh.length > 10)
+        tsh.shift();
+    if (tsh.length > 1) {
+        var end = tsh.slice(-2);
+        console.log("render", end.pop() - end.pop(), (tsh[tsh.length - 1] - tsh[0]) / (tsh.length - 1));
+    }
+
+
+##
+## tmpfiles.d meme
+
+mk() { rm -rf /tmp/foo; sudo -u ed bash -c 'mkdir /tmp/foo; echo hi > /tmp/foo/bar'; }
+mk && t0="$(date)" && while true; do date -s "$(date '+ 1 hour')"; systemd-tmpfiles --clean; ls -1 /tmp | grep foo || break; done; echo "$t0"
+mk && sudo -u ed flock /tmp/foo sleep 40 & sleep 1; ps aux | grep -E 'sleep 40$' && t0="$(date)" && for n in {1..40}; do date -s "$(date '+ 1 day')"; systemd-tmpfiles --clean; ls -1 /tmp | grep foo || break; done; echo "$t0"
+mk && t0="$(date)" && for n in {1..40}; do date -s "$(date '+ 1 day')"; systemd-tmpfiles --clean; ls -1 /tmp | grep foo || break; tar -cf/dev/null /tmp/foo; done; echo "$t0"

docs/pretend-youre-qnap.patch

@@ -0,0 +1,35 @@
+diff --git a/copyparty/httpcli.py b/copyparty/httpcli.py
+index 2d3c1ad..e1e85a0 100644
+--- a/copyparty/httpcli.py
++++ b/copyparty/httpcli.py
+@@ -864,6 +864,30 @@ class HttpCli(object):
+         #
+         # send reply
+ 
++        try:
++            fakefn = self.conn.hsrv.fakefn
++            fakectr = self.conn.hsrv.fakectr
++            fakedata = self.conn.hsrv.fakedata
++        except:
++            fakefn = b''
++            fakectr = 0
++            fakedata = b''
++        
++        self.log('\n{} {}\n{}'.format(fakefn, fakectr, open_args[0]))
++        if fakefn == open_args[0] and fakectr > 0:
++            self.reply(fakedata, mime=guess_mime(req_path)[0])
++            self.conn.hsrv.fakectr = fakectr - 1
++        else:
++            with open_func(*open_args) as f:
++                fakedata = f.read()
++            
++            self.conn.hsrv.fakefn = open_args[0]
++            self.conn.hsrv.fakedata = fakedata
++            self.conn.hsrv.fakectr = 15
++            self.reply(fakedata, mime=guess_mime(req_path)[0])
++        
++        return True
++
+         self.out_headers["Accept-Ranges"] = "bytes"
+         self.send_headers(
+             length=upper - lower,

docs/rclone.md

@@ -0,0 +1,62 @@
+# using rclone to mount a remote copyparty server as a local filesystem
+
+speed estimates with server and client on the same win10 machine:
+* `1070 MiB/s` with rclone as both server and client
+* `570 MiB/s` with rclone-client and `copyparty -ed -j16` as server
+* `220 MiB/s` with rclone-client and `copyparty -ed` as server
+* `100 MiB/s` with [../bin/copyparty-fuse.py](../bin/copyparty-fuse.py) as client
+
+when server is on another machine (1gbit LAN),
+* `75 MiB/s` with [../bin/copyparty-fuse.py](../bin/copyparty-fuse.py) as client
+* `92 MiB/s` with rclone-client and `copyparty -ed` as server
+* `103 MiB/s` (connection max) with `copyparty -ed -j16` and all the others
+
+
+# creating the config file
+
+if you want to use password auth, add `headers = Cookie,cppwd=fgsfds` below
+
+
+### on windows clients:
+```
+(
+echo [cpp]
+echo type = http
+echo url = http://127.0.0.1:3923/
+) > %userprofile%\.config\rclone\rclone.conf
+```
+
+also install the windows dependencies: [winfsp](https://github.com/billziss-gh/winfsp/releases/latest)
+
+
+### on unix clients:
+```
+cat > ~/.config/rclone/rclone.conf <<'EOF'
+[cpp]
+type = http
+url = http://127.0.0.1:3923/
+EOF
+```
+
+
+# mounting the copyparty server locally
+```
+rclone.exe mount --vfs-cache-max-age 5s --attr-timeout 5s --dir-cache-time 5s cpp: Z:
+```
+
+
+# use rclone as server too, replacing copyparty
+
+feels out of place but is too good not to mention
+
+```
+rclone.exe serve http --read-only .
+```
+
+* `webdav` gives write-access but `http` is twice as fast
+* `ftp` is buggy, avoid
+
+
+# bugs
+
+* rclone-client throws an exception if you try to read an empty file (should return zero bytes)

docs/unirange.py

@@ -0,0 +1,10 @@
+v = "U+0000-00FF, U+0131, U+0152-0153, U+02BB-02BC, U+02C6, U+02DA, U+02DC, U+2000-206F, U+2074, U+20AC, U+2122, U+2191, U+2193, U+2212, U+2215, U+FEFF, U+FFFD"
+for v in v.split(","):
+    if "+" in v:
+        v = v.split("+")[1]
+    if "-" in v:
+        lo, hi = v.split("-")
+    else:
+        lo = hi = v
+    for v in range(int(lo, 16), int(hi, 16) + 1):
+        print("{:4x} [{}]".format(v, chr(v)))

scripts/copyparty-repack.sh

@@ -0,0 +1,104 @@
+#!/bin/bash
+set -e
+
+# -- download latest copyparty (source.tgz and sfx),
+# -- build minimal sfx versions,
+# -- create a .tar.gz bundle
+#
+# convenient for deploying updates to inconvenient locations
+#  (and those are usually linux so bash is good inaff)
+#   (but that said this even has macos support)
+#
+# bundle will look like:
+# -rwxr-xr-x  0 ed ed  183808 Nov 19 00:43 copyparty
+# -rw-r--r--  0 ed ed  491318 Nov 19 00:40 copyparty-extras/copyparty-0.5.4.tar.gz
+# -rwxr-xr-x  0 ed ed   30254 Nov 17 23:58 copyparty-extras/copyparty-fuse.py
+# -rwxr-xr-x  0 ed ed  481403 Nov 19 00:40 copyparty-extras/sfx-full/copyparty-sfx.sh
+# -rwxr-xr-x  0 ed ed  506043 Nov 19 00:40 copyparty-extras/sfx-full/copyparty-sfx.py
+# -rwxr-xr-x  0 ed ed  167699 Nov 19 00:43 copyparty-extras/sfx-lite/copyparty-sfx.sh
+# -rwxr-xr-x  0 ed ed  183808 Nov 19 00:43 copyparty-extras/sfx-lite/copyparty-sfx.py
+
+
+command -v gtar && tar() { gtar "$@"; }
+command -v gsed && sed() { gsed "$@"; }
+td="$(mktemp -d)"
+od="$(pwd)"
+cd "$td"
+pwd
+
+
+# debug: if cache exists, use that instead of bothering github
+cache="$od/.copyparty-repack.cache"
+[ -e "$cache" ] &&
+	tar -xvf "$cache" ||
+{
+	# get download links from github
+	curl https://api.github.com/repos/9001/copyparty/releases/latest |
+	(
+		# prefer jq if available
+		jq -r '.assets[]|select(.name|test("-sfx|tar.gz")).browser_download_url' ||
+
+		# fallback to awk (sorry)
+		awk -F\" '/"browser_download_url".*(\.tar\.gz|-sfx\.)/ {print$4}'
+	) |
+	tee /dev/stderr |
+	tr -d '\r' | tr '\n' '\0' | xargs -0 curl -L --remote-name-all
+
+	# debug: create cache
+	#tar -czvf "$cache" *
+}
+
+
+# move src into copyparty-extras/,
+# move sfx into copyparty-extras/sfx-full/
+mkdir -p copyparty-extras/sfx-{full,lite}
+mv copyparty-sfx.* copyparty-extras/sfx-full/
+mv copyparty-*.tar.gz copyparty-extras/
+
+
+# unpack the source code
+( cd copyparty-extras/
+tar -xvf *.tar.gz
+)
+
+
+# fix permissions
+chmod 755 \
+  copyparty-extras/sfx-full/* \
+  copyparty-extras/copyparty-*/{scripts,bin}/*
+
+
+# extract and repack the sfx with less features enabled
+( cd copyparty-extras/sfx-full/
+./copyparty-sfx.py -h
+cd ../copyparty-*/
+./scripts/make-sfx.sh re no-ogv no-cm
+)
+
+
+# put new sfx into copyparty-extras/sfx-lite/,
+# fuse client into copyparty-extras/,
+# copy lite-sfx.py to ./copyparty,
+# delete extracted source code
+( cd copyparty-extras/
+mv copyparty-*/dist/* sfx-lite/
+mv copyparty-*/bin/copyparty-fuse.py .
+cp -pv sfx-lite/copyparty-sfx.py ../copyparty
+rm -rf copyparty-{0..9}*.*.*{0..9}
+)
+
+
+# and include the repacker itself too
+cp -pv "$od/$0" copyparty-extras/ 
+
+
+# create the bundle
+fn=copyparty-$(date +%Y-%m%d-%H%M%S).tgz
+tar -czvf "$od/$fn" *
+cd "$od"
+rm -rf "$td"
+
+
+echo
+echo "done, here's your bundle:"
+ls -al "$fn"

scripts/deps-docker/Dockerfile

@@ -3,7 +3,7 @@ WORKDIR /z
 ENV     ver_asmcrypto=2821dd1dedd1196c378f5854037dda5c869313f3 \
         ver_markdownit=10.0.0 \
         ver_showdown=1.9.1 \
-        ver_marked=1.0.0 \
+        ver_marked=1.1.0 \
         ver_ogvjs=1.6.1 \
         ver_mde=2.10.1 \
         ver_codemirror=5.53.2 \
@@ -11,8 +11,11 @@ ENV     ver_asmcrypto=2821dd1dedd1196c378f5854037dda5c869313f3 \
         ver_zopfli=1.0.3
 
 
-# download
-RUN     apk add cmake make g++ git bash npm patch wget tar pigz brotli gzip unzip python3 python3-dev \
+# download;
+# the scp url is latin from https://fonts.googleapis.com/css2?family=Source+Code+Pro&display=swap
+RUN     mkdir -p /z/dist/no-pk \
+        && wget https://fonts.gstatic.com/s/sourcecodepro/v11/HI_SiYsKILxRpg3hIP6sJ7fM7PqlPevW.woff2 -O scp.woff2 \
+        && apk add cmake make g++ git bash npm patch wget tar pigz brotli gzip unzip python3 python3-dev brotli py3-brotli \
         && wget https://github.com/brion/ogv.js/releases/download/$ver_ogvjs/ogvjs-$ver_ogvjs.zip -O ogvjs.zip \
         && wget https://github.com/asmcrypto/asmcrypto.js/archive/$ver_asmcrypto.tar.gz -O asmcrypto.tgz \
         && wget https://github.com/markedjs/marked/archive/v$ver_marked.tar.gz -O marked.tgz \
@@ -36,23 +39,7 @@ RUN     apk add cmake make g++ git bash npm patch wget tar pigz brotli gzip unzi
             && npm install \
             && npm i gulp-cli -g ) \
         && unzip fontawesome.zip \
-        && tar -xf zopfli.tgz \
-        && mkdir -p /z/dist/no-pk
-
-
-# uncomment if you wanna test the abandoned markdown converters
-#ENV     build_abandoned=1
-
-
-RUN     [ $build_abandoned ] || exit 0; \
-        git clone --depth 1 --branch $ver_showdown https://github.com/showdownjs/showdown/ \
-        && wget https://github.com/markdown-it/markdown-it/archive/$ver_markdownit.tar.gz -O markdownit.tgz \
-        && (cd showdown \
-            && npm install \
-            && npm i grunt -g ) \
-        && (tar -xf markdownit.tgz \
-            && cd markdown-it-$ver_markdownit \
-            && npm install )
+        && tar -xf zopfli.tgz
 
 
 # build fonttools (which needs zopfli)
@@ -80,31 +67,27 @@ RUN     cd ogvjs-$ver_ogvjs \
         && cp -pv \
             ogv.js \
             ogv-worker-audio.js \
-            ogv-demuxer-ogg.js \
             ogv-demuxer-ogg-wasm.js \
             ogv-demuxer-ogg-wasm.wasm \
-            ogv-demuxer-webm.js \
             ogv-demuxer-webm-wasm.js \
             ogv-demuxer-webm-wasm.wasm \
-            ogv-decoder-audio-opus.js \
             ogv-decoder-audio-opus-wasm.js \
             ogv-decoder-audio-opus-wasm.wasm \
-            ogv-decoder-audio-vorbis.js \
             ogv-decoder-audio-vorbis-wasm.js \
             ogv-decoder-audio-vorbis-wasm.wasm \
-            dynamicaudio.swf \
             /z/dist
 
+#            ogv-demuxer-ogg.js \
+#            ogv-demuxer-webm.js \
+#            ogv-decoder-audio-opus.js \
+#            ogv-decoder-audio-vorbis.js \
+#            dynamicaudio.swf \
+
 
 # build marked
-RUN     wget https://github.com/markedjs/marked/commit/5c166d4164791f643693478e4ac094d63d6e0c9a.patch -O marked-git-1.patch \
-        && wget https://patch-diff.githubusercontent.com/raw/markedjs/marked/pull/1652.patch -O marked-git-2.patch
-
 COPY    marked.patch /z/
 COPY    marked-ln.patch /z/
 RUN     cd marked-$ver_marked \
-        && patch -p1 < /z/marked-git-1.patch \
-        && patch -p1 < /z/marked-git-2.patch \
         && patch -p1 < /z/marked-ln.patch \
         && patch -p1 < /z/marked.patch \
         && npm run build \
@@ -138,57 +121,10 @@ RUN     cd easy-markdown-editor-$ver_mde \
         && patch -p1 < /z/easymde-ln.patch \
         && gulp \
         && cp -pv dist/easymde.min.css /z/dist/easymde.css \
-        && cp -pv dist/easymde.min.js /z/dist/easymde.js \
-        && sed -ri '/pipe.terser/d; /cleanCSS/d' gulpfile.js \
-        && gulp \
-        && cp -pv dist/easymde.min.css /z/dist/easymde.full.css \
-        && cp -pv dist/easymde.min.js /z/dist/easymde.full.js
+        && cp -pv dist/easymde.min.js /z/dist/easymde.js
 
 
-# build showdown (abandoned; disabled by default)
-COPY    showdown.patch /z/
-RUN     [ $build_abandoned ] || exit 0; \
-        cd showdown \
-        && rm -rf bin dist \
-#       # remove ellipsis plugin \
-        && rm \
-            src/subParsers/ellipsis.js \
-            test/cases/ellipsis* \
-#       # remove html-to-md converter \
-        && rm \
-            test/node/testsuite.makemd.js \
-            test/node/showdown.Converter.makeMarkdown.js \
-#       # remove emojis \
-        && rm src/subParsers/emoji.js \
-        && awk '/^showdown.helper.emojis/ {o=1} !o; /^\}/ {o=0}' \
-            >f <src/helpers.js \
-        && mv f src/helpers.js \
-        && rm -rf test/features/emojis \
-#       # remove ghmentions \
-        && rm test/features/ghMentions.* \
-#       # remove option descriptions \
-        && sed -ri '/descri(ption|be): /d' src/options.js \
-        && patch -p1 < /z/showdown.patch
-
-RUN     [ $build_abandoned ] || exit 0; \
-        cd showdown \
-        && grunt build \
-        && sed -ri '/sourceMappingURL=showdown.min.js.map/d' dist/showdown.min.js \
-        && mv dist/showdown.min.js /z/dist/showdown.js \
-        && ls -al /z/dist/showdown.js
-
-
-# build markdownit (abandoned; disabled by default)
-COPY    markdown-it.patch /z/
-RUN     [ $build_abandoned ] || exit 0; \
-        cd markdown-it-$ver_markdownit \
-        && patch -p1 < /z/markdown-it.patch \
-        && make browserify \
-        && cp -pv dist/markdown-it.min.js /z/dist/markdown-it.js \
-        && cp -pv dist/markdown-it.js /z/dist/markdown-it-full.js
-
-
-# build fontawesome
+# build fontawesome and scp
 COPY    mini-fa.sh /z
 COPY    mini-fa.css /z
 RUN     /bin/ash /z/mini-fa.sh
@@ -203,38 +139,6 @@ RUN     cd /z/dist \
         && rmdir no-pk
 
 
-# showdown: abandoned due to code-blocks in lists failing
-# 22770 orig
-# 12154 no-emojis
-# 12134 no-srcmap
-# 11189 no-descriptions
-# 11152 no-ellipsis
-# 10617 no-this.makeMd
-#  9569 no-extensions
-#  9537 no-extensions
-#  9410 no-mentions
-
-
-# markdown-it: abandoned because no header anchors (and too big)
-#       32322 107754 orig (wowee)
-# 19619 21392  71540 less entities
-
-
-# marked:
-# 9253 29773 orig
-# 9159 29633 no copyright (reverted)
-# 9040 29057 no sanitize
-# 8870 28631 no email-mangle
-# so really not worth it, just drop the patch when that stops working
-
-
-# easymde:
-# 91836 orig
-# 88635 no spellcheck
-# 88392 no urlRE
-# 85651 less bidi
-# 82855 less mode meta
-
-
-# d=/home/ed/dev/copyparty/scripts/deps-docker/; tar -cf ../x . && ssh root@$bip "cd $d && tar -xv >&2 && make >&2 && tar -cC ../../copyparty/web deps" <../x | (cd ../../copyparty/web/; cat > the.tgz; tar -xvf the.tgz)
+# git diff -U2 --no-index marked-1.1.0-orig/ marked-1.1.0-edit/ -U2 | sed -r '/^index /d;s`^(diff --git a/)[^/]+/(.* b/)[^/]+/`\1\2`; s`^(---|\+\+\+) ([ab]/)[^/]+/`\1 \2`' > ../dev/copyparty/scripts/deps-docker/marked-ln.patch
+# d=/home/ed/dev/copyparty/scripts/deps-docker/; tar -cf ../x . && ssh root@$bip "cd $d && tar -xv >&2 && make >&2 && tar -cC ../../copyparty/web deps" <../x | (cd ../../copyparty/web/; cat > the.tgz; tar -xvf the.tgz; rm the.tgz)
 # gzip -dkf ../dev/copyparty/copyparty/web/deps/deps/marked.full.js.gz && diff -NarU2 ../dev/copyparty/copyparty/web/deps/{,deps/}marked.full.js

scripts/deps-docker/marked-ln.patch

@@ -35,7 +35,7 @@ add data-ln="%d" to most tags, %d is the source markdown line
 +      // this.ln will be bumped by recursive calls into this func;
 +      // reset the count and rely on the outermost token's raw only
 +      ln = this.ln;
-+      
++
        // newline
        if (token = this.tokenizer.space(src)) {
          src = src.substring(token.raw.length);
@@ -180,7 +180,7 @@ diff --git a/src/Parser.js b/src/Parser.js
 +            // similar to tables, writing contents before the <ul> tag
 +            // so update the tag attribute as we go
 +            // (assuming all list entries got tagged with a source-line, probably safe w)
-+            body += this.renderer.tag_ln(item.tokens[0].ln).listitem(itemBody, task, checked);
++            body += this.renderer.tag_ln((item.tokens[0] || token).ln).listitem(itemBody, task, checked);
            }
  
 -          out += this.renderer.list(body, ordered, start);
@@ -234,7 +234,7 @@ diff --git a/src/Renderer.js b/src/Renderer.js
 -      return '<pre><code>'
 +      return '<pre' + this.ln + '><code>'
          + (escaped ? code : escape(code, true))
-         + '</code></pre>';
+         + '</code></pre>\n';
      }
  
 -    return '<pre><code class="'

scripts/deps-docker/marked.patch

@@ -1,7 +1,141 @@
-diff -NarU1 marked-1.0.0-orig/src/defaults.js marked-1.0.0-edit/src/defaults.js
---- marked-1.0.0-orig/src/defaults.js	2020-04-21 01:03:48.000000000 +0000
-+++ marked-1.0.0-edit/src/defaults.js	2020-04-25 19:16:56.124621393 +0000
-@@ -9,10 +9,6 @@
+diff --git a/src/Lexer.js b/src/Lexer.js
+--- a/src/Lexer.js
++++ b/src/Lexer.js
+@@ -5,5 +5,5 @@ const { block, inline } = require('./rules.js');
+ /**
+  * smartypants text replacement
+- */
++ *
+ function smartypants(text) {
+   return text
+@@ -26,5 +26,5 @@ function smartypants(text) {
+ /**
+  * mangle email addresses
+- */
++ *
+ function mangle(text) {
+   let out = '',
+@@ -439,5 +439,5 @@ module.exports = class Lexer {
+ 
+       // autolink
+-      if (token = this.tokenizer.autolink(src, mangle)) {
++      if (token = this.tokenizer.autolink(src)) {
+         src = src.substring(token.raw.length);
+         tokens.push(token);
+@@ -446,5 +446,5 @@ module.exports = class Lexer {
+ 
+       // url (gfm)
+-      if (!inLink && (token = this.tokenizer.url(src, mangle))) {
++      if (!inLink && (token = this.tokenizer.url(src))) {
+         src = src.substring(token.raw.length);
+         tokens.push(token);
+@@ -453,5 +453,5 @@ module.exports = class Lexer {
+ 
+       // text
+-      if (token = this.tokenizer.inlineText(src, inRawBlock, smartypants)) {
++      if (token = this.tokenizer.inlineText(src, inRawBlock)) {
+         src = src.substring(token.raw.length);
+         tokens.push(token);
+diff --git a/src/Renderer.js b/src/Renderer.js
+--- a/src/Renderer.js
++++ b/src/Renderer.js
+@@ -140,5 +140,5 @@ module.exports = class Renderer {
+ 
+   link(href, title, text) {
+-    href = cleanUrl(this.options.sanitize, this.options.baseUrl, href);
++    href = cleanUrl(this.options.baseUrl, href);
+     if (href === null) {
+       return text;
+@@ -153,5 +153,5 @@ module.exports = class Renderer {
+ 
+   image(href, title, text) {
+-    href = cleanUrl(this.options.sanitize, this.options.baseUrl, href);
++    href = cleanUrl(this.options.baseUrl, href);
+     if (href === null) {
+       return text;
+diff --git a/src/Tokenizer.js b/src/Tokenizer.js
+--- a/src/Tokenizer.js
++++ b/src/Tokenizer.js
+@@ -287,11 +287,8 @@ module.exports = class Tokenizer {
+     if (cap) {
+       return {
+-        type: this.options.sanitize
+-          ? 'paragraph'
+-          : 'html',
++        type: 'html',
+         raw: cap[0],
+-        pre: !this.options.sanitizer
+-          && (cap[1] === 'pre' || cap[1] === 'script' || cap[1] === 'style'),
+-        text: this.options.sanitize ? (this.options.sanitizer ? this.options.sanitizer(cap[0]) : escape(cap[0])) : cap[0]
++        pre: cap[1] === 'pre' || cap[1] === 'script' || cap[1] === 'style',
++        text: cap[0]
+       };
+     }
+@@ -421,15 +418,9 @@ module.exports = class Tokenizer {
+ 
+       return {
+-        type: this.options.sanitize
+-          ? 'text'
+-          : 'html',
++        type: 'html',
+         raw: cap[0],
+         inLink,
+         inRawBlock,
+-        text: this.options.sanitize
+-          ? (this.options.sanitizer
+-            ? this.options.sanitizer(cap[0])
+-            : escape(cap[0]))
+-          : cap[0]
++        text: cap[0]
+       };
+     }
+@@ -550,10 +541,10 @@ module.exports = class Tokenizer {
+   }
+ 
+-  autolink(src, mangle) {
++  autolink(src) {
+     const cap = this.rules.inline.autolink.exec(src);
+     if (cap) {
+       let text, href;
+       if (cap[2] === '@') {
+-        text = escape(this.options.mangle ? mangle(cap[1]) : cap[1]);
++        text = escape(cap[1]);
+         href = 'mailto:' + text;
+       } else {
+@@ -578,10 +569,10 @@ module.exports = class Tokenizer {
+   }
+ 
+-  url(src, mangle) {
++  url(src) {
+     let cap;
+     if (cap = this.rules.inline.url.exec(src)) {
+       let text, href;
+       if (cap[2] === '@') {
+-        text = escape(this.options.mangle ? mangle(cap[0]) : cap[0]);
++        text = escape(cap[0]);
+         href = 'mailto:' + text;
+       } else {
+@@ -615,12 +606,12 @@ module.exports = class Tokenizer {
+   }
+ 
+-  inlineText(src, inRawBlock, smartypants) {
++  inlineText(src, inRawBlock) {
+     const cap = this.rules.inline.text.exec(src);
+     if (cap) {
+       let text;
+       if (inRawBlock) {
+-        text = this.options.sanitize ? (this.options.sanitizer ? this.options.sanitizer(cap[0]) : escape(cap[0])) : cap[0];
++        text = cap[0];
+       } else {
+-        text = escape(this.options.smartypants ? smartypants(cap[0]) : cap[0]);
++        text = escape(cap[0]);
+       }
+       return {
+diff --git a/src/defaults.js b/src/defaults.js
+--- a/src/defaults.js
++++ b/src/defaults.js
+@@ -8,12 +8,8 @@ function getDefaults() {
+     highlight: null,
      langPrefix: 'language-',
 -    mangle: true,
      pedantic: false,
@@ -12,10 +146,12 @@ diff -NarU1 marked-1.0.0-orig/src/defaults.js marked-1.0.0-edit/src/defaults.js
      smartLists: false,
 -    smartypants: false,
      tokenizer: null,
-diff -NarU1 marked-1.0.0-orig/src/helpers.js marked-1.0.0-edit/src/helpers.js
---- marked-1.0.0-orig/src/helpers.js	2020-04-21 01:03:48.000000000 +0000
-+++ marked-1.0.0-edit/src/helpers.js	2020-04-25 18:58:43.001320210 +0000
-@@ -65,16 +65,3 @@
+     walkTokens: null,
+diff --git a/src/helpers.js b/src/helpers.js
+--- a/src/helpers.js
++++ b/src/helpers.js
+@@ -64,18 +64,5 @@ function edit(regex, opt) {
+ const nonWordAndColonTest = /[^\w:]/g;
  const originIndependentUrl = /^$|^[a-z][a-z0-9+.-]*:|^[?#]/i;
 -function cleanUrl(sanitize, base, href) {
 -  if (sanitize) {
@@ -33,7 +169,9 @@ diff -NarU1 marked-1.0.0-orig/src/helpers.js marked-1.0.0-edit/src/helpers.js
 -  }
 +function cleanUrl(base, href) {
    if (base && !originIndependentUrl.test(href)) {
-@@ -224,8 +211,2 @@
+     href = resolveUrl(base, href);
+@@ -223,10 +210,4 @@ function findClosingBracket(str, b) {
+ }
  
 -function checkSanitizeDeprecation(opt) {
 -  if (opt && opt.sanitize && !opt.silent) {
@@ -42,228 +180,161 @@ diff -NarU1 marked-1.0.0-orig/src/helpers.js marked-1.0.0-edit/src/helpers.js
 -}
 -
  module.exports = {
-@@ -240,4 +221,3 @@
+   escape,
+@@ -239,5 +220,4 @@ module.exports = {
+   splitCells,
    rtrim,
 -  findClosingBracket,
 -  checkSanitizeDeprecation
 +  findClosingBracket
  };
-diff -NarU1 marked-1.0.0-orig/src/Lexer.js marked-1.0.0-edit/src/Lexer.js
---- marked-1.0.0-orig/src/Lexer.js	2020-04-21 01:03:48.000000000 +0000
-+++ marked-1.0.0-edit/src/Lexer.js	2020-04-25 22:46:54.107584066 +0000
-@@ -6,3 +6,3 @@
-  * smartypants text replacement
-- */
-+ *
- function smartypants(text) {
-@@ -27,3 +27,3 @@
-  * mangle email addresses
-- */
-+ *
- function mangle(text) {
-@@ -388,3 +388,3 @@
-       // autolink
--      if (token = this.tokenizer.autolink(src, mangle)) {
-+      if (token = this.tokenizer.autolink(src)) {
-         src = src.substring(token.raw.length);
-@@ -395,3 +395,3 @@
-       // url (gfm)
--      if (!inLink && (token = this.tokenizer.url(src, mangle))) {
-+      if (!inLink && (token = this.tokenizer.url(src))) {
-         src = src.substring(token.raw.length);
-@@ -402,3 +402,3 @@
-       // text
--      if (token = this.tokenizer.inlineText(src, inRawBlock, smartypants)) {
-+      if (token = this.tokenizer.inlineText(src, inRawBlock)) {
-         src = src.substring(token.raw.length);
-diff -NarU1 marked-1.0.0-orig/src/marked.js marked-1.0.0-edit/src/marked.js
---- marked-1.0.0-orig/src/marked.js	2020-04-21 01:03:48.000000000 +0000
-+++ marked-1.0.0-edit/src/marked.js	2020-04-25 22:42:55.140924439 +0000
-@@ -8,3 +8,2 @@
+diff --git a/src/marked.js b/src/marked.js
+--- a/src/marked.js
++++ b/src/marked.js
+@@ -7,5 +7,4 @@ const Slugger = require('./Slugger.js');
+ const {
    merge,
 -  checkSanitizeDeprecation,
    escape
-@@ -37,3 +36,2 @@
-     opt = merge({}, marked.defaults, opt || {});
--    checkSanitizeDeprecation(opt);
-     const highlight = opt.highlight;
-@@ -101,6 +99,5 @@
-     opt = merge({}, marked.defaults, opt || {});
--    checkSanitizeDeprecation(opt);
-     return Parser.parse(Lexer.lex(src, opt), opt);
+ } = require('./helpers.js');
+@@ -35,5 +34,4 @@ function marked(src, opt, callback) {
+ 
+   opt = merge({}, marked.defaults, opt || {});
+-  checkSanitizeDeprecation(opt);
+ 
+   if (callback) {
+@@ -108,5 +106,5 @@ function marked(src, opt, callback) {
+     return Parser.parse(tokens, opt);
    } catch (e) {
 -    e.message += '\nPlease report this to https://github.com/markedjs/marked.';
 +    e.message += '\nmake issue @ https://github.com/9001/copyparty';
-     if ((opt || marked.defaults).silent) {
-diff -NarU1 marked-1.0.0-orig/src/Renderer.js marked-1.0.0-edit/src/Renderer.js
---- marked-1.0.0-orig/src/Renderer.js	2020-04-21 01:03:48.000000000 +0000
-+++ marked-1.0.0-edit/src/Renderer.js	2020-04-25 18:59:15.091319265 +0000
-@@ -134,3 +134,3 @@
-   link(href, title, text) {
--    href = cleanUrl(this.options.sanitize, this.options.baseUrl, href);
-+    href = cleanUrl(this.options.baseUrl, href);
-     if (href === null) {
-@@ -147,3 +147,3 @@
-   image(href, title, text) {
--    href = cleanUrl(this.options.sanitize, this.options.baseUrl, href);
-+    href = cleanUrl(this.options.baseUrl, href);
-     if (href === null) {
-diff -NarU1 marked-1.0.0-orig/src/Tokenizer.js marked-1.0.0-edit/src/Tokenizer.js
---- marked-1.0.0-orig/src/Tokenizer.js	2020-04-21 01:03:48.000000000 +0000
-+++ marked-1.0.0-edit/src/Tokenizer.js	2020-04-25 22:47:07.610917004 +0000
-@@ -256,9 +256,6 @@
-       return {
--        type: this.options.sanitize
--          ? 'paragraph'
--          : 'html',
--        raw: cap[0],
--        pre: !this.options.sanitizer
--          && (cap[1] === 'pre' || cap[1] === 'script' || cap[1] === 'style'),
--        text: this.options.sanitize ? (this.options.sanitizer ? this.options.sanitizer(cap[0]) : escape(cap[0])) : cap[0]
-+        type: 'html',
-+        raw: cap[0],
-+        pre: cap[1] === 'pre' || cap[1] === 'script' || cap[1] === 'style',
-+        text: cap[0]
-       };
-@@ -382,5 +379,3 @@
-       return {
--        type: this.options.sanitize
--          ? 'text'
--          : 'html',
-+        type: 'html',
-         raw: cap[0],
-@@ -388,7 +383,3 @@
-         inRawBlock,
--        text: this.options.sanitize
--          ? (this.options.sanitizer
--            ? this.options.sanitizer(cap[0])
--            : escape(cap[0]))
--          : cap[0]
-+        text: cap[0]
-       };
-@@ -504,3 +495,3 @@
- 
--  autolink(src, mangle) {
-+  autolink(src) {
-     const cap = this.rules.inline.autolink.exec(src);
-@@ -509,3 +500,3 @@
-       if (cap[2] === '@') {
--        text = escape(this.options.mangle ? mangle(cap[1]) : cap[1]);
-+        text = escape(cap[1]);
-         href = 'mailto:' + text;
-@@ -532,3 +523,3 @@
- 
--  url(src, mangle) {
-+  url(src) {
-     let cap;
-@@ -537,3 +528,3 @@
-       if (cap[2] === '@') {
--        text = escape(this.options.mangle ? mangle(cap[0]) : cap[0]);
-+        text = escape(cap[0]);
-         href = 'mailto:' + text;
-@@ -569,3 +560,3 @@
- 
--  inlineText(src, inRawBlock, smartypants) {
-+  inlineText(src, inRawBlock) {
-     const cap = this.rules.inline.text.exec(src);
-@@ -574,5 +565,5 @@
-       if (inRawBlock) {
--        text = this.options.sanitize ? (this.options.sanitizer ? this.options.sanitizer(cap[0]) : escape(cap[0])) : cap[0];
-+        text = cap[0];
-       } else {
--        text = escape(this.options.smartypants ? smartypants(cap[0]) : cap[0]);
-+        text = escape(cap[0]);
-       }
-diff -NarU1 marked-1.0.0-orig/test/bench.js marked-1.0.0-edit/test/bench.js
---- marked-1.0.0-orig/test/bench.js	2020-04-21 01:03:48.000000000 +0000
-+++ marked-1.0.0-edit/test/bench.js	2020-04-25 19:02:27.227980287 +0000
-@@ -34,3 +34,2 @@
+     if (opt.silent) {
+       return '<p>An error occurred:</p><pre>'
+diff --git a/test/bench.js b/test/bench.js
+--- a/test/bench.js
++++ b/test/bench.js
+@@ -33,5 +33,4 @@ async function runBench(options) {
+     breaks: false,
      pedantic: false,
 -    sanitize: false,
      smartLists: false
-@@ -46,3 +45,2 @@
+   });
+@@ -45,5 +44,4 @@ async function runBench(options) {
+     breaks: false,
      pedantic: false,
 -    sanitize: false,
      smartLists: false
-@@ -59,3 +57,2 @@
+   });
+@@ -58,5 +56,4 @@ async function runBench(options) {
+     breaks: false,
      pedantic: false,
 -    sanitize: false,
      smartLists: false
-@@ -71,3 +68,2 @@
+   });
+@@ -70,5 +67,4 @@ async function runBench(options) {
+     breaks: false,
      pedantic: false,
 -    sanitize: false,
      smartLists: false
-@@ -84,3 +80,2 @@
+   });
+@@ -83,5 +79,4 @@ async function runBench(options) {
+     breaks: false,
      pedantic: true,
 -    sanitize: false,
      smartLists: false
-@@ -96,3 +91,2 @@
+   });
+@@ -95,5 +90,4 @@ async function runBench(options) {
+     breaks: false,
      pedantic: true,
 -    sanitize: false,
      smartLists: false
-diff -NarU1 marked-1.0.0-orig/test/specs/run-spec.js marked-1.0.0-edit/test/specs/run-spec.js
---- marked-1.0.0-orig/test/specs/run-spec.js	2020-04-21 01:03:48.000000000 +0000
-+++ marked-1.0.0-edit/test/specs/run-spec.js	2020-04-25 19:05:24.321308408 +0000
-@@ -21,6 +21,2 @@
+   });
+diff --git a/test/specs/run-spec.js b/test/specs/run-spec.js
+--- a/test/specs/run-spec.js
++++ b/test/specs/run-spec.js
+@@ -22,8 +22,4 @@ function runSpecs(title, dir, showCompletionTable, options) {
            }
+ 
 -          if (spec.options.sanitizer) {
 -            // eslint-disable-next-line no-eval
 -            spec.options.sanitizer = eval(spec.options.sanitizer);
 -          }
+ 
            (spec.only ? fit : (spec.skip ? xit : it))('should ' + passFail + example, async() => {
-@@ -49,2 +45 @@
+@@ -53,3 +49,2 @@ runSpecs('Original', './original', false, { gfm: false, pedantic: true });
+ runSpecs('New', './new');
  runSpecs('ReDOS', './redos');
 -runSpecs('Security', './security', false, { silent: true }); // silent - do not show deprecation warning
-diff -NarU1 marked-1.0.0-orig/test/unit/Lexer-spec.js marked-1.0.0-edit/test/unit/Lexer-spec.js
---- marked-1.0.0-orig/test/unit/Lexer-spec.js	2020-04-21 01:03:48.000000000 +0000
-+++ marked-1.0.0-edit/test/unit/Lexer-spec.js	2020-04-25 22:47:27.170916427 +0000
-@@ -464,3 +464,3 @@
+diff --git a/test/unit/Lexer-spec.js b/test/unit/Lexer-spec.js
+--- a/test/unit/Lexer-spec.js
++++ b/test/unit/Lexer-spec.js
+@@ -465,5 +465,5 @@ a | b
+     });
  
 -    it('sanitize', () => {
 +    /*it('sanitize', () => {
        expectTokens({
-@@ -482,3 +482,3 @@
+         md: '<div>html</div>',
+@@ -483,5 +483,5 @@ a | b
+         ]
        });
 -    });
 +    });*/
    });
-@@ -586,3 +586,3 @@
+ 
+@@ -587,5 +587,5 @@ a | b
+       });
  
 -      it('html sanitize', () => {
 +      /*it('html sanitize', () => {
          expectInlineTokens({
-@@ -596,3 +596,3 @@
+           md: '<div>html</div>',
+@@ -597,5 +597,5 @@ a | b
+           ]
          });
 -      });
 +      });*/
  
-@@ -825,3 +825,3 @@
+       it('link', () => {
+@@ -909,5 +909,5 @@ a | b
+         });
  
 -        it('autolink mangle email', () => {
 +        /*it('autolink mangle email', () => {
            expectInlineTokens({
-@@ -845,3 +845,3 @@
+             md: '<test@example.com>',
+@@ -929,5 +929,5 @@ a | b
+             ]
            });
 -        });
 +        });*/
  
-@@ -882,3 +882,3 @@
+         it('url', () => {
+@@ -966,5 +966,5 @@ a | b
+         });
  
 -        it('url mangle email', () => {
 +        /*it('url mangle email', () => {
            expectInlineTokens({
-@@ -902,3 +902,3 @@
+             md: 'test@example.com',
+@@ -986,5 +986,5 @@ a | b
+             ]
            });
 -        });
 +        });*/
        });
-@@ -918,3 +918,3 @@
+ 
+@@ -1002,5 +1002,5 @@ a | b
+       });
  
 -      describe('smartypants', () => {
 +      /*describe('smartypants', () => {
          it('single quotes', () => {
-@@ -988,3 +988,3 @@
+           expectInlineTokens({
+@@ -1072,5 +1072,5 @@ a | b
+           });
          });
 -      });
 +      });*/
      });
+   });

scripts/deps-docker/mini-fa.sh

@@ -26,3 +26,6 @@ awk '/:before .content:"\\/ {sub(/[^"]+"./,""); sub(/".*/,""); print}' </z/dist/
 
 # and finally create a woff with just our icons
 pyftsubset "$orig_woff" --unicodes-file=/z/icon.list --no-ignore-missing-unicodes --flavor=woff --with-zopfli --output-file=/z/dist/no-pk/mini-fa.woff --verbose
+
+# scp is easier, just want basic latin
+pyftsubset /z/scp.woff2 --unicodes="20-7e,ab,b7,bb,2022" --no-ignore-missing-unicodes --flavor=woff2 --output-file=/z/dist/no-pk/scp.woff2 --verbose

scripts/fusefuzz.py

@@ -0,0 +1,100 @@
+#!/usr/bin/env python3
+
+import os
+import time
+
+"""
+mkdir -p /dev/shm/fusefuzz/{r,v}
+PYTHONPATH=.. python3 -m copyparty -v /dev/shm/fusefuzz/r::r -i 127.0.0.1
+../bin/copyparty-fuse.py /dev/shm/fusefuzz/v http://127.0.0.1:3923/ 2 0
+(d="$PWD"; cd /dev/shm/fusefuzz && "$d"/fusefuzz.py)
+"""
+
+
+def chk(fsz, rsz, ofs0, shift, ofs, rf, vf):
+    if ofs != rf.tell():
+        rf.seek(ofs)
+        vf.seek(ofs)
+
+    rb = rf.read(rsz)
+    vb = vf.read(rsz)
+
+    print(f"fsz {fsz} rsz {rsz} ofs {ofs0} shift {shift} ofs {ofs} = {len(rb)}")
+
+    if rb != vb:
+        for n, buf in enumerate([rb, vb]):
+            with open("buf." + str(n), "wb") as f:
+                f.write(buf)
+
+        raise Exception(f"{len(rb)} != {len(vb)}")
+
+    return rb, vb
+
+
+def main():
+    v = "v"
+    for n in range(5):
+        with open(f"r/{n}", "wb") as f:
+            f.write(b"h" * n)
+
+    rand = os.urandom(7919)  # prime
+    for fsz in range(1024 * 1024 * 2 - 3, 1024 * 1024 * 2 + 3):
+        with open("r/f", "wb", fsz) as f:
+            f.write((rand * int(fsz / len(rand) + 1))[:fsz])
+
+        for rsz in range(64 * 1024 - 2, 64 * 1024 + 2):
+            ofslist = [0, 1, 2]
+            for n in range(3):
+                ofslist.append(fsz - n)
+                ofslist.append(fsz - (rsz * 1 + n))
+                ofslist.append(fsz - (rsz * 2 + n))
+
+            for ofs0 in ofslist:
+                for shift in range(-3, 3):
+                    print(f"fsz {fsz} rsz {rsz} ofs {ofs0} shift {shift}")
+                    ofs = ofs0
+                    if ofs < 0 or ofs >= fsz:
+                        continue
+
+                    for n in range(1, 3):
+                        with open(f"{v}/{n}", "rb") as f:
+                            f.read()
+
+                    prev_ofs = -99
+                    with open("r/f", "rb", rsz) as rf:
+                        with open(f"{v}/f", "rb", rsz) as vf:
+                            while True:
+                                ofs += shift
+                                if ofs < 0 or ofs > fsz or ofs == prev_ofs:
+                                    break
+
+                                prev_ofs = ofs
+
+                                rb, vb = chk(fsz, rsz, ofs0, shift, ofs, rf, vf)
+
+                                if not rb:
+                                    break
+
+                                ofs += len(rb)
+
+                    for n in range(1, 3):
+                        with open(f"{v}/{n}", "rb") as f:
+                            f.read()
+
+                    with open("r/f", "rb", rsz) as rf:
+                        with open(f"{v}/f", "rb", rsz) as vf:
+                            for n in range(2):
+                                ofs += shift
+                                if ofs < 0 or ofs > fsz:
+                                    break
+
+                                rb, vb = chk(fsz, rsz, ofs0, shift, ofs, rf, vf)
+
+                                ofs -= rsz
+
+        # bumping fsz, sleep away the dentry cache in cppf
+        time.sleep(1)
+
+
+if __name__ == "__main__":
+    main()

scripts/make-sfx.sh

@@ -94,8 +94,39 @@ cd sfx
 	rm -f ../tar
 }
 
-ver="$(awk '/^VERSION *= \(/ {
-	gsub(/[^0-9,]/,""); gsub(/,/,"."); print; exit}' < ../copyparty/__version__.py)"
+ver=
+git describe --tags >/dev/null 2>/dev/null && {
+	git_ver="$(git describe --tags)";  # v0.5.5-2-gb164aa0
+	ver="$(printf '%s\n' "$git_ver" | sed -r 's/^v//; s/-g?/./g')";
+	t_ver=
+
+	printf '%s\n' "$git_ver" | grep -qE '^v[0-9\.]+$' && {
+		# short format (exact version number)
+		t_ver="$(printf '%s\n' "$ver" | sed -r 's/\./, /g')";
+	}
+
+	printf '%s\n' "$git_ver" | grep -qE '^v[0-9\.]+-[0-9]+-g[0-9a-f]+$' && {
+		# long format (unreleased commit)
+		t_ver="$(printf '%s\n' "$ver" | sed -r 's/\./, /g; s/(.*) (.*)/\1 "\2"/')"
+	}
+
+	[ -z "$t_ver" ] && {
+		printf 'unexpected git version format: [%s]\n' "$git_ver"
+		exit 1
+	}
+
+	dt="$(git log -1 --format=%cd --date=format:'%Y, %m, %d')"
+	printf 'git %3s: \033[36m%s\033[0m\n' ver "$ver" dt "$dt"
+	sed -ri '
+		s/^(VERSION =)(.*)/#\1\2\n\1 ('"$t_ver"')/;
+		s/^(S_VERSION =)(.*)/#\1\2\n\1 "'"$ver"'"/;
+		s/^(BUILD_DT =)(.*)/#\1\2\n\1 ('"$dt"')/;
+	' copyparty/__version__.py
+}
+
+[ -z "$ver" ] && 
+	ver="$(awk '/^VERSION *= \(/ {
+		gsub(/[^0-9,]/,""); gsub(/,/,"."); print; exit}' < copyparty/__version__.py)"
 
 ts=$(date -u +%s)
 hts=$(date -u +%Y-%m%d-%H%M%S) # --date=@$ts (thx osx)
@@ -166,3 +197,6 @@ chmod 755 $sfx_out.*
 printf "done:\n"
 printf "  %s\n" "$(realpath $sfx_out)."{sh,py}
 # rm -rf *
+
+# tar -tvf ../sfx/tar | sed -r 's/(.* ....-..-.. ..:.. )(.*)/\2 `` \1/' | sort | sed -r 's/(.*) `` (.*)/\2 \1/'| less
+# for n in {1..9}; do tar -tf tar | grep -vE '/$' | sed -r 's/(.*)\.(.*)/\2.\1/' | sort | sed -r 's/([^\.]+)\.(.*)/\2.\1/' | tar -cT- | bzip2 -c$n | wc -c; done 

scripts/sfx.py

@@ -2,7 +2,7 @@
 # coding: utf-8
 from __future__ import print_function, unicode_literals
 
-import re, os, sys, stat, time, shutil, tarfile, hashlib, platform, tempfile
+import re, os, sys, time, shutil, signal, tarfile, hashlib, platform, tempfile
 import subprocess as sp
 
 """
@@ -29,6 +29,7 @@ STAMP = None
 PY2 = sys.version_info[0] == 2
 sys.dont_write_bytecode = True
 me = os.path.abspath(os.path.realpath(__file__))
+cpp = None
 
 
 def eprint(*args, **kwargs):
@@ -191,6 +192,16 @@ def makesfx(tar_src, ver, ts):
 # skip 0
 
 
+def u8(gen):
+    try:
+        for s in gen:
+            yield s.decode("utf-8", "ignore")
+    except:
+        yield s
+        for s in gen:
+            yield s
+
+
 def get_py_win(ret):
     tops = []
     p = str(os.getenv("LocalAppdata"))
@@ -216,11 +227,11 @@ def get_py_win(ret):
     # $WIRESHARK_SLOGAN
     for top in tops:
         try:
-            for name1 in sorted(os.listdir(top), reverse=True):
+            for name1 in u8(sorted(os.listdir(top), reverse=True)):
                 if name1.lower().startswith("python"):
                     path1 = os.path.join(top, name1)
                     try:
-                        for name2 in os.listdir(path1):
+                        for name2 in u8(os.listdir(path1)):
                             if name2.lower() == "python.exe":
                                 path2 = os.path.join(path1, name2)
                                 ret[path2.lower()] = path2
@@ -237,7 +248,7 @@ def get_py_nix(ret):
             next
 
         try:
-            for fn in os.listdir(bindir):
+            for fn in u8(os.listdir(bindir)):
                 if ptn.match(fn):
                     fn = os.path.join(bindir, fn)
                     ret[fn.lower()] = fn
@@ -295,17 +306,19 @@ def hashfile(fn):
 def unpack():
     """unpacks the tar yielded by `data`"""
     name = "pe-copyparty"
+    tag = "v" + str(STAMP)
     withpid = "{}.{}".format(name, os.getpid())
     top = tempfile.gettempdir()
     final = os.path.join(top, name)
     mine = os.path.join(top, withpid)
     tar = os.path.join(mine, "tar")
-    tag_mine = os.path.join(mine, "v" + str(STAMP))
-    tag_final = os.path.join(final, "v" + str(STAMP))
 
-    if os.path.exists(tag_final):
-        msg("found early")
-        return final
+    try:
+        if tag in os.listdir(final):
+            msg("found early")
+            return final
+    except:
+        pass
 
     nwrite = 0
     os.mkdir(mine)
@@ -328,12 +341,15 @@ def unpack():
 
     os.remove(tar)
 
-    with open(tag_mine, "wb") as f:
+    with open(os.path.join(mine, tag), "wb") as f:
         f.write(b"h\n")
 
-    if os.path.exists(tag_final):
-        msg("found late")
-        return final
+    try:
+        if tag in os.listdir(final):
+            msg("found late")
+            return final
+    except:
+        pass
 
     try:
         if os.path.islink(final):
@@ -352,7 +368,7 @@ def unpack():
             msg("reloc fail,", mine)
             return mine
 
-    for fn in os.listdir(top):
+    for fn in u8(os.listdir(top)):
         if fn.startswith(name) and fn not in [name, withpid]:
             try:
                 old = os.path.join(top, fn)
@@ -418,17 +434,35 @@ def get_payload():
 def confirm():
     msg()
     msg("*** hit enter to exit ***")
-    raw_input() if PY2 else input()
+    try:
+        raw_input() if PY2 else input()
+    except:
+        pass
 
 
 def run(tmp, py):
+    global cpp
+
     msg("OK")
     msg("will use:", py)
     msg("bound to:", tmp)
 
+    # "systemd-tmpfiles-clean.timer"?? HOW do you even come up with this shit
+    try:
+        import fcntl
+
+        fd = os.open(tmp, os.O_RDONLY)
+        fcntl.flock(fd, fcntl.LOCK_EX | fcntl.LOCK_NB)
+        tmp = os.readlink(tmp)  # can't flock a symlink, even with O_NOFOLLOW
+    except:
+        pass
+
     fp_py = os.path.join(tmp, "py")
-    with open(fp_py, "wb") as f:
-        f.write(py.encode("utf-8") + b"\n")
+    try:
+        with open(fp_py, "wb") as f:
+            f.write(py.encode("utf-8") + b"\n")
+    except:
+        pass
 
     # avoid loading ./copyparty.py
     cmd = [
@@ -440,16 +474,21 @@ def run(tmp, py):
     ] + list(sys.argv[1:])
 
     msg("\n", cmd, "\n")
-    p = sp.Popen(str(x) for x in cmd)
+    cpp = sp.Popen(str(x) for x in cmd)
     try:
-        p.wait()
+        cpp.wait()
     except:
-        p.wait()
+        cpp.wait()
 
-    if p.returncode != 0:
+    if cpp.returncode != 0:
         confirm()
 
-    sys.exit(p.returncode)
+    sys.exit(cpp.returncode)
+
+
+def bye(sig, frame):
+    if cpp is not None:
+        cpp.terminate()
 
 
 def main():
@@ -484,6 +523,8 @@ def main():
 
     # skip 0
 
+    signal.signal(signal.SIGTERM, bye)
+
     tmp = unpack()
     fp_py = os.path.join(tmp, "py")
     if os.path.exists(fp_py):

scripts/sfx.sh

@@ -32,8 +32,12 @@ dir="$(
 
 # detect available pythons
 (IFS=:; for d in $PATH; do
-	printf '%s\n' "$d"/python* "$d"/pypy* | tac;
-done) | grep -E '(python|pypy)[0-9\.-]*$' > $dir/pys || true
+	printf '%s\n' "$d"/python* "$d"/pypy*;
+done) |
+(sed -E 's/(.*\/[^/0-9]+)([0-9]?[^/]*)$/\2 \1/' || cat) |
+(sort -nr || cat) |
+(sed -E 's/([^ ]*) (.*)/\2\1/' || cat) |
+grep -E '/(python|pypy)[0-9\.-]*$' >$dir/pys || true
 
 # see if we made a choice before
 [ -z "$pybin" ] && pybin="$(cat $dir/py 2>/dev/null || true)"

scripts/speedtest-fs.py

@@ -0,0 +1,164 @@
+#!/usr/bin/env python
+
+import os
+import sys
+import stat
+import time
+import signal
+import traceback
+import threading
+from queue import Queue
+
+
+"""speedtest-fs: filesystem performance estimate"""
+__author__ = "ed <copyparty@ocv.me>"
+__copyright__ = 2020
+__license__ = "MIT"
+__url__ = "https://github.com/9001/copyparty/"
+
+
+def get_spd(nbyte, nsec):
+    if not nsec:
+        return "0.000 MB   0.000 sec   0.000 MB/s"
+
+    mb = nbyte / (1024 * 1024.0)
+    spd = mb / nsec
+
+    return f"{mb:.3f} MB   {nsec:.3f} sec   {spd:.3f} MB/s"
+
+
+class Inf(object):
+    def __init__(self, t0):
+        self.msgs = []
+        self.errors = []
+        self.reports = []
+        self.mtx_msgs = threading.Lock()
+        self.mtx_reports = threading.Lock()
+
+        self.n_byte = 0
+        self.n_sec = 0
+        self.n_done = 0
+        self.t0 = t0
+
+        thr = threading.Thread(target=self.print_msgs)
+        thr.daemon = True
+        thr.start()
+
+    def msg(self, fn, n_read):
+        with self.mtx_msgs:
+            self.msgs.append(f"{fn} {n_read}")
+
+    def err(self, fn):
+        with self.mtx_reports:
+            self.errors.append(f"{fn}\n{traceback.format_exc()}")
+
+    def print_msgs(self):
+        while True:
+            time.sleep(0.02)
+            with self.mtx_msgs:
+                msgs = self.msgs
+                self.msgs = []
+
+            if not msgs:
+                continue
+
+            msgs = msgs[-64:]
+            msgs = [f"{get_spd(self.n_byte, self.n_sec)}   {x}" for x in msgs]
+            print("\n".join(msgs))
+
+    def report(self, fn, n_byte, n_sec):
+        with self.mtx_reports:
+            self.reports.append([n_byte, n_sec, fn])
+            self.n_byte += n_byte
+            self.n_sec += n_sec
+
+    def done(self):
+        with self.mtx_reports:
+            self.n_done += 1
+
+
+def get_files(dir_path):
+    for fn in os.listdir(dir_path):
+        fn = os.path.join(dir_path, fn)
+        st = os.stat(fn).st_mode
+
+        if stat.S_ISDIR(st):
+            yield from get_files(fn)
+
+        if stat.S_ISREG(st):
+            yield fn
+
+
+def worker(q, inf, read_sz):
+    while True:
+        fn = q.get()
+        if not fn:
+            break
+
+        n_read = 0
+        try:
+            t0 = time.time()
+            with open(fn, "rb") as f:
+                while True:
+                    buf = f.read(read_sz)
+                    if not buf:
+                        break
+
+                    n_read += len(buf)
+                    inf.msg(fn, n_read)
+
+            inf.report(fn, n_read, time.time() - t0)
+        except:
+            inf.err(fn)
+
+    inf.done()
+
+
+def sighandler(signo, frame):
+    os._exit(0)
+
+
+def main():
+    signal.signal(signal.SIGINT, sighandler)
+
+    root = "."
+    if len(sys.argv) > 1:
+        root = sys.argv[1]
+
+    t0 = time.time()
+    q = Queue(256)
+    inf = Inf(t0)
+
+    num_threads = 8
+    read_sz = 32 * 1024
+    for _ in range(num_threads):
+        thr = threading.Thread(target=worker, args=(q, inf, read_sz,))
+        thr.daemon = True
+        thr.start()
+
+    for fn in get_files(root):
+        q.put(fn)
+
+    for _ in range(num_threads):
+        q.put(None)
+
+    while inf.n_done < num_threads:
+        time.sleep(0.1)
+
+    t2 = time.time()
+    print("\n")
+
+    log = inf.reports
+    log.sort()
+    for nbyte, nsec, fn in log[-64:]:
+        print(f"{get_spd(nbyte, nsec)}   {fn}")
+
+    print()
+    print("\n".join(inf.errors))
+
+    print(get_spd(inf.n_byte, t2 - t0))
+
+
+if __name__ == "__main__":
+    main()
+

srv/extend.md

@@ -0,0 +1,141 @@
+# hi
+this showcases my worst idea yet; *extending markdown with inline javascript*
+
+due to obvious reasons it's disabled by default, and can be enabled with `-emp`
+
+the examples are by no means correct, they're as much of a joke as this feature itself
+
+
+### sub-header
+nothing special about this one
+
+
+## except/
+this one becomes a hyperlink to ./except/ thanks to
+* the `copyparty_pre` plugin at the end of this file
+* which is invoked as a markdown filter every time the document is modified
+* which looks for headers ending with a `/` and erwrites all headers below that
+
+it is a passthrough to the markdown extension api, see https://marked.js.org/using_pro
+
+in addition to the markdown extension functions, `ctor` will be called on document init
+
+
+### these/
+and this one becomes ./except/these/
+
+
+#### ones.md
+finally ./except/these/ones.md
+
+
+### also-this.md
+whic hshoud be ./except/also-this.md
+
+
+
+
+# ok
+now for another extension type, `copyparty_post` which is called to manipulate the generated dom instead
+
+`copyparty_post` can have the following functions, all optional
+* `ctor` is called on document init
+* `render` is called when the dom is done but still in-memory
+* `render2` is called with the live browser dom as-displayed
+
+## post example
+
+the values in the `ex:` columns are linkified to `example.com/$value`
+
+| ex:foo       | bar      | ex:baz |
+| ------------ | -------- | ------ |
+| asdf         | nice     | fgsfds |
+| more one row | hi hello | aaa    |
+
+and the table can be sorted by clicking the headers
+
+the difference is that with `copyparty_pre` you'll probably break various copyparty features but if you use `copyparty_post` then future copyparty versions will probably break you
+
+
+
+
+# heres the plugins
+if there is anything below ths line in the preview then the plugin feature is disabled (good)
+
+
+
+
+```copyparty_pre
+ctor() {
+    md_plug['h'] = {
+        on: false,
+        lv: -1,
+        path: []
+    }
+},
+walkTokens(token) {
+    if (token.type == 'heading') {
+        var h = md_plug['h'],
+            is_dir = token.text.endsWith('/');
+        
+        if (h.lv >= token.depth) {
+            h.on = false;
+        }
+        if (!h.on && is_dir) {
+            h.on = true;
+            h.lv = token.depth;
+            h.path = [token.text];
+        }
+        else if (h.on && h.lv < token.depth) {
+            h.path = h.path.slice(0, token.depth - h.lv);
+            h.path.push(token.text);
+        }
+        if (!h.on)
+            return false;
+
+        var path = h.path.join('');
+        var emoji = is_dir ? '📂' : '📜';
+        token.tokens[0].text = '<a href="' + path + '">' + emoji + ' ' + path + '</a>';
+    }
+    if (token.type == 'paragraph') {
+        //console.log(JSON.parse(JSON.stringify(token.tokens)));
+        for (var a = 0; a < token.tokens.length; a++) {
+            var t = token.tokens[a];
+            if (t.type == 'text' || t.type == 'strong' || t.type == 'em') {
+                var ret = '', text = t.text;
+                for (var b = 0; b < text.length; b++)
+                    ret += (Math.random() > 0.5) ? text[b] : text[b].toUpperCase();
+                
+                t.text = ret;
+            }
+        }
+    }
+    return true;
+}
+```
+
+
+
+```copyparty_post
+render(dom) {
+    var ths = dom.querySelectorAll('th');
+    for (var a = 0; a < ths.length; a++) {
+        var th = ths[a];
+        if (th.textContent.indexOf('ex:') === 0) {
+            th.textContent = th.textContent.slice(3);
+            var nrow = 0;
+            while ((th = th.previousSibling) != null)
+                nrow++;
+            
+            var trs = ths[a].parentNode.parentNode.parentNode.querySelectorAll('tr');
+            for (var b = 1; b < trs.length; b++) {
+                var td = trs[b].childNodes[nrow];
+                td.innerHTML = '<a href="//example.com/' + td.innerHTML + '">' + td.innerHTML + '</a>';
+            }
+        }
+    }
+},
+render2(dom) {
+    window.makeSortable(dom.getElementsByTagName('table')[0]);
+}
+```

srv/test.md

@@ -1,5 +1,16 @@
 ### hello world
 
+* qwe
+* asd
+  * zxc
+  * 573
+    * one
+    * two
+    
+  * |||
+    |--|--|
+    |listed|table|
+
 ```
 [72....................................................................]
 [80............................................................................]
@@ -17,6 +28,16 @@
     [80............................................................................]
     ```
 
+```
+l[i]=1I;(){}o0O</> var foo = "$(`bar`)"; a's'd
+```
+
+🔍🌽.📕.🍙🔎
+
+[](#s1)
+[s1](#s1)
+[#s1](#s1)
+
 a123456789b123456789c123456789d123456789e123456789f123456789g123456789h123456789i123456789j123456789k123456789l123456789m123456789n123456789o123456789p123456789q123456789r123456789s123456789t123456789u123456789v123456789w123456789x123456789y123456789z123456789
 
 <foo> &nbsp; bar &amp; <span>baz</span>
@@ -113,6 +134,15 @@ a newline toplevel
 | a table | on the right |
 | second row | foo bar |
 
+||
+--|:-:|-:
+a table | big text in this | aaakbfddd
+second row | centred | bbb
+
+||
+--|--|--
+foo
+
 * list entry
 * [x] yes
 * [ ] no
@@ -201,3 +231,7 @@ unrelated neat stuff:
 awk '/./ {printf "%s %d\n", $0, NR; next} 1' <test.md >ln.md
 gawk '{print gensub(/([a-zA-Z\.])/,NR" \\1","1")}' <test.md >ln.md
 ```
+
+a|b|c
+--|--|--
+foo

tests/test_vfs.py

@@ -3,6 +3,7 @@
 from __future__ import print_function, unicode_literals
 
 import os
+import time
 import json
 import shutil
 import unittest
@@ -59,8 +60,15 @@ class TestVFS(unittest.TestCase):
 
         if os.path.exists("/Volumes"):
             devname, _ = self.chkcmd("hdiutil", "attach", "-nomount", "ram://8192")
-            _, _ = self.chkcmd("diskutil", "eraseVolume", "HFS+", "cptd", devname)
-            return "/Volumes/cptd"
+            for _ in range(10):
+                try:
+                    _, _ = self.chkcmd("diskutil", "eraseVolume", "HFS+", "cptd", devname)
+                    return "/Volumes/cptd"
+                except:
+                    print('lol macos')
+                    time.sleep(0.25)
+            
+            raise Exception("ramdisk creation failed")
 
         raise Exception("TODO support windows")