v0.11.38

fix ogvjs paused-seek

README.md

@@ -20,8 +20,10 @@ turn your phone or raspi into a portable file server with resumable uploads/down
 
 * top
     * [quickstart](#quickstart)
+        * [on debian](#on-debian)
     * [notes](#notes)
     * [status](#status)
+    * [testimonials](#testimonials)
 * [bugs](#bugs)
     * [general bugs](#general-bugs)
     * [not my bugs](#not-my-bugs)
@@ -44,6 +46,7 @@ turn your phone or raspi into a portable file server with resumable uploads/down
 * [browser support](#browser-support)
 * [client examples](#client-examples)
 * [up2k](#up2k)
+* [performance](#performance)
 * [dependencies](#dependencies)
     * [optional dependencies](#optional-dependencies)
     * [install recommended deps](#install-recommended-deps)
@@ -51,8 +54,10 @@ turn your phone or raspi into a portable file server with resumable uploads/down
 * [sfx](#sfx)
     * [sfx repack](#sfx-repack)
 * [install on android](#install-on-android)
-* [dev env setup](#dev-env-setup)
+* [building](#building)
-* [how to release](#how-to-release)
+    * [dev env setup](#dev-env-setup)
+    * [just the sfx](#just-the-sfx)
+    * [complete release](#complete-release)
 * [todo](#todo)
 
 
@@ -62,11 +67,33 @@ download [copyparty-sfx.py](https://github.com/9001/copyparty/releases/latest/do
 
 running the sfx without arguments (for example doubleclicking it on Windows) will give everyone full access to the current folder; see `-h` for help if you want accounts and volumes etc
 
+some recommended options:
+* `-e2dsa` enables general file indexing, see [search configuration](#search-configuration)
+* `-e2ts` enables audio metadata indexing (needs either FFprobe or mutagen), see [optional dependencies](#optional-dependencies)
+* `-v /mnt/music:/music:r:afoo -a foo:bar` shares `/mnt/music` as `/music`, `r`eadable by anyone, with user `foo` as `a`dmin (read/write), password `bar`
+  * the syntax is `-v src:dst:perm:perm:...` so local-path, url-path, and one or more permissions to set
+  * replace `:r:afoo` with `:rfoo` to only make the folder readable by `foo` and nobody else
+  * in addition to `r`ead and `a`dmin, `w`rite makes a folder write-only, so cannot list/access files in it
+* `--ls '**,*,ln,p,r'` to crash on startup if any of the volumes contain a symlink which point outside the volume, as that could give users unintended access
+
 you may also want these, especially on servers:
 * [contrib/systemd/copyparty.service](contrib/systemd/copyparty.service) to run copyparty as a systemd service
 * [contrib/nginx/copyparty.conf](contrib/nginx/copyparty.conf) to reverse-proxy behind nginx (for better https)
 
 
+### on debian
+
+recommended steps to enable audio metadata and thumbnails (from images and videos):
+
+* as root, run the following:  
+  `apt install python3 python3-pip python3-dev ffmpeg`
+
+* then, as the user which will be running copyparty (so hopefully not root), run this:  
+  `python3 -m pip install --user -U Pillow pillow-avif-plugin`
+
+(skipped `pyheif-pillow-opener` because apparently debian is too old to build it)
+
+
 ## notes
 
 general:
@@ -101,11 +128,12 @@ summary: all planned features work! now please enjoy the bloatening
   * ☑ FUSE client (read-only)
 * browser
   * ☑ tree-view
-  * ☑ media player
+  * ☑ audio player (with OS media controls)
   * ☑ thumbnails
     * ☑ images using Pillow
     * ☑ videos using FFmpeg
     * ☑ cache eviction (max-age; maybe max-size eventually)
+  * ☑ image gallery
   * ☑ SPA (browse while uploading)
     * if you use the file-tree on the left only, not folders in the file list
 * server indexing
@@ -117,6 +145,13 @@ summary: all planned features work! now please enjoy the bloatening
   * ☑ editor (sure why not)
 
 
+## testimonials
+
+small collection of user feedback
+
+`good enough`, `surprisingly correct`, `certified good software`, `just works`, `why`
+
+
 # bugs
 
 * Windows: python 3.7 and older cannot read tags with ffprobe, so use mutagen or upgrade
@@ -128,10 +163,15 @@ summary: all planned features work! now please enjoy the bloatening
 
 * all volumes must exist / be available on startup; up2k (mtp especially) gets funky otherwise
 * cannot mount something at `/d1/d2/d3` unless `d2` exists inside `d1`
+* dupe files will not have metadata (audio tags etc) displayed in the file listing
+  * because they don't get `up` entries in the db (probably best fix) and `tx_browser` does not `lstat`
 * probably more, pls let me know
 
 ## not my bugs
 
+* Windows: folders cannot be accessed if the name ends with `.`
+  * python or windows bug
+
 * Windows: msys2-python 3.8.6 occasionally throws "RuntimeError: release unlocked lock" when leaving a scoped mutex in up2k
   * this is an msys2 bug, the regular windows edition of python is fine
 
@@ -154,25 +194,34 @@ summary: all planned features work! now please enjoy the bloatening
 ## hotkeys
 
 the browser has the following hotkeys
+* `B` toggle breadcrumbs / directory tree
 * `I/K` prev/next folder
-* `P` parent folder
+* `M` parent folder
 * `G` toggle list / grid view
 * `T` toggle thumbnails / icons
 * when playing audio:
-  * `0..9` jump to 10%..90%
-  * `U/O` skip 10sec back/forward
   * `J/L` prev/next song
-    * `J` also starts playing the folder
+  * `U/O` skip 10sec back/forward
+  * `0..9` jump to 10%..90%
+  * `P` play/pause (also starts playing the folder)
+* when viewing images / playing videos:
+  * `J/L, Left/Right` prev/next file
+  * `Home/End` first/last file
+  * `U/O` skip 10sec back/forward
+  * `P/K/Space` play/pause video
+  * `Esc` close viewer
+* when tree-sidebar is open:
+  * `A/D` adjust tree width
 * in the grid view:
   * `S` toggle multiselect
-  * `A/D` zoom
+  * shift+`A/D` zoom
 
 
 ## tree-mode
 
-by default there's a breadcrumbs path; you can replace this with a tree-browser sidebar thing by clicking the 🌲
+by default there's a breadcrumbs path; you can replace this with a tree-browser sidebar thing by clicking the `🌲` or pressing the `B` hotkey
 
-click `[-]` and `[+]` to adjust the size, and the `[a]` toggles if the tree should widen dynamically as you go deeper or stay fixed-size
+click `[-]` and `[+]` (or hotkeys `A`/`D`) to adjust the size, and the `[a]` toggles if the tree should widen dynamically as you go deeper or stay fixed-size
 
 
 ## thumbnails
@@ -183,6 +232,8 @@ it does static images with Pillow and uses FFmpeg for video files, so you may wa
 
 images named `folder.jpg` and `folder.png` become the thumbnail of the folder they're in
 
+in the grid/thumbnail view, if the audio player panel is open, songs will start playing when clicked
+
 
 ## zip downloads
 
@@ -266,6 +317,8 @@ up2k has saved a few uploads from becoming corrupted in-transfer already; caught
 
 * you can link a particular timestamp in an audio file by adding it to the URL, such as `&20` / `&20s` / `&1m20` / `&t=1:20` after the `.../#af-c8960dab`
 
+* if you are using media hotkeys to switch songs and are getting tired of seeing the OSD popup which Windows doesn't let you disable, consider https://ocv.me/dev/?media-osd-bgone.ps1
+
 
 # searching
 
@@ -300,7 +353,7 @@ the same arguments can be set as volume flags, in addition to `d2d` and `d2t` fo
 * `-v ~/music::r:cd2t` disables all `-e2t*` (tags), does not affect `-e2d*`
 
 note:
-* `e2tsr` is probably always overkill, since `e2ds`/`e2dsa` would pick up any file modifications and cause `e2ts` to reindex those
+* `e2tsr` is probably always overkill, since `e2ds`/`e2dsa` would pick up any file modifications and `e2ts` would then reindex those
 * the rescan button in the admin panel has no effect unless the volume has `-e2ds` or higher
 
 you can choose to only index filename/path/size/last-modified (and not the hash of the file contents) by setting `--no-hash` or the volume-flag `cdhash`, this has the following consequences:
@@ -428,7 +481,7 @@ quick summary of more eccentric web-browsers trying to view a directory index:
 
 copyparty returns a truncated sha512sum of your PUT/POST as base64; you can generate the same checksum locally to verify uplaods:
 
-    b512(){ printf "$((sha512sum||shasum -a512)|sed -E 's/ .*//;s/(..)/\\x\1/g')"|base64|head -c43;}
+    b512(){ printf "$((sha512sum||shasum -a512)|sed -E 's/ .*//;s/(..)/\\x\1/g')"|base64|tr '+/' '-_'|head -c44;}
     b512 <movie.mkv
 
 
@@ -448,6 +501,23 @@ quick outline of the up2k protocol, see [uploading](#uploading) for the web-clie
 * client does another handshake with the hashlist; server replies with OK or a list of chunks to reupload
 
 
+# performance
+
+defaults are good for most cases, don't mind the `cannot efficiently use multiple CPU cores` message, it's very unlikely to be a problem
+
+below are some tweaks roughly ordered by usefulness:
+
+* `-q` disables logging and can help a bunch, even when combined with `-lo` to redirect logs to file
+* `--http-only` or `--https-only` (unless you want to support both protocols) will reduce the delay before a new connection is established
+* `--hist` pointing to a fast location (ssd) will make directory listings and searches faster when `-e2d` or `-e2t` is set
+* `--no-hash` when indexing a networked disk if you don't care about the actual filehashes and only want the names/tags searchable
+* `-j` enables multiprocessing (actual multithreading) and can make copyparty perform better in cpu-intensive workloads, for example:
+  * huge amount of short-lived connections
+  * really heavy traffic (downloads/uploads)
+  
+  ...however it adds an overhead to internal communication so it might be a net loss, see if it works 4 u
+
+
 # dependencies
 
 * `jinja2` (is built into the SFX)
@@ -523,18 +593,45 @@ echo $?
 after the initial setup, you can launch copyparty at any time by running `copyparty` anywhere in Termux
 
 
-# dev env setup
+# building
+
+## dev env setup
+
+mostly optional; if you need a working env for vscode or similar
 
 ```sh
 python3 -m venv .venv
 . .venv/bin/activate
-pip install jinja2  # mandatory deps
+pip install jinja2  # mandatory
-pip install Pillow  # thumbnail deps
+pip install mutagen  # audio metadata
+pip install Pillow pyheif-pillow-opener pillow-avif-plugin  # thumbnails
 pip install black bandit pylint flake8  # vscode tooling
 ```
 
 
-# how to release
+## just the sfx
+
+unless you need to modify something in the web-dependencies, it's faster to grab those from a previous release:
+
+```sh
+rm -rf copyparty/web/deps
+curl -L https://github.com/9001/copyparty/releases/latest/download/copyparty-sfx.py >x.py
+python3 x.py -h
+rm x.py
+mv /tmp/pe-copyparty/copyparty/web/deps/ copyparty/web/
+```
+
+then build the sfx using any of the following examples:
+
+```sh
+./scripts/make-sfx.sh  # both python and sh editions
+./scripts/make-sfx.sh no-sh gz  # just python with gzip
+```
+
+
+## complete release
+
+also builds the sfx so disregard the sfx section above
 
 in the `scripts` folder:
 
@@ -550,13 +647,15 @@ in the `scripts` folder:
 roughly sorted by priority
 
 * readme.md as epilogue
-* single sha512 across all up2k chunks? maybe
 * reduce up2k roundtrips
   * start from a chunk index and just go
   * terminate client on bad data
+* logging to file
 
 discarded ideas
 
+* single sha512 across all up2k chunks?
+  * crypto.subtle cannot into streaming, would have to use hashwasm, expensive
 * separate sqlite table per tag
   * performance fixed by skipping some indexes (`+mt.k`)
 * audio fingerprinting
@@ -571,3 +670,6 @@ discarded ideas
   * nah
 * look into android thumbnail cache file format
   * absolutely not
+* indexedDB for hashes, cfg enable/clear/sz, 2gb avail, ~9k for 1g, ~4k for 100m, 500k items before autoeviction
+  * blank hashlist when up-ok to skip handshake
+    * too many confusing side-effects

bin/README.md

@@ -48,15 +48,16 @@ you could replace winfsp with [dokan](https://github.com/dokan-dev/dokany/releas
 
 
 # [`dbtool.py`](dbtool.py)
-upgrade utility which can show db info and help transfer data between databases, for example when a new version of copyparty recommends to wipe the DB and reindex because it now collects additional metadata during analysis, but you have some really expensive `-mtp` parsers and want to copy over the tags from the old db
+upgrade utility which can show db info and help transfer data between databases, for example when a new version of copyparty is incompatible with the old DB and automatically rebuilds the DB from scratch, but you have some really expensive `-mtp` parsers and want to copy over the tags from the old db
 
-for that example (upgrading to v0.11.0), first move the old db aside, launch copyparty, let it rebuild the db until the point where it starts running mtp (colored messages as it adds the mtp tags), then CTRL-C and patch in the old mtp tags from the old db instead
+for that example (upgrading to v0.11.20), first launch the new version of copyparty like usual, let it make a backup of the old db and rebuild the new db until the point where it starts running mtp (colored messages as it adds the mtp tags), that's when you hit CTRL-C and patch in the old mtp tags from the old db instead
 
 so assuming you have `-mtp` parsers to provide the tags `key` and `.bpm`:
 
 ```
-~/bin/dbtool.py -ls up2k.db
+cd /mnt/nas/music/.hist
-~/bin/dbtool.py -src up2k.db.v0.10.22 up2k.db -cmp
+~/src/copyparty/bin/dbtool.py -ls up2k.db
-~/bin/dbtool.py -src up2k.db.v0.10.22 up2k.db -rm-mtp-flag -copy key
+~/src/copyparty/bin/dbtool.py -src up2k.*.v3 up2k.db -cmp
-~/bin/dbtool.py -src up2k.db.v0.10.22 up2k.db -rm-mtp-flag -copy .bpm -vac
+~/src/copyparty/bin/dbtool.py -src up2k.*.v3 up2k.db -rm-mtp-flag -copy key
+~/src/copyparty/bin/dbtool.py -src up2k.*.v3 up2k.db -rm-mtp-flag -copy .bpm -vac
 ```

bin/copyparty-fuse-streaming.py

@@ -345,7 +345,7 @@ class Gateway(object):
         except:
             pass
 
-    def sendreq(self, *args, headers={}, **kwargs):
+    def sendreq(self, meth, path, headers, **kwargs):
         if self.password:
             headers["Cookie"] = "=".join(["cppwd", self.password])
 
@@ -354,21 +354,21 @@ class Gateway(object):
             if c.rx_path:
                 raise Exception()
 
-            c.request(*list(args), headers=headers, **kwargs)
+            c.request(meth, path, headers=headers, **kwargs)
             c.rx = c.getresponse()
             return c
         except:
             tid = threading.current_thread().ident
             dbg(
-                "\033[1;37;44mbad conn {:x}\n  {}\n  {}\033[0m".format(
+                "\033[1;37;44mbad conn {:x}\n  {} {}\n  {}\033[0m".format(
-                    tid, " ".join(str(x) for x in args), c.rx_path if c else "(null)"
+                    tid, meth, path, c.rx_path if c else "(null)"
                 )
             )
 
         self.closeconn(c)
         c = self.getconn()
         try:
-            c.request(*list(args), headers=headers, **kwargs)
+            c.request(meth, path, headers=headers, **kwargs)
             c.rx = c.getresponse()
             return c
         except:
@@ -386,7 +386,7 @@ class Gateway(object):
             path = dewin(path)
 
         web_path = self.quotep("/" + "/".join([self.web_root, path])) + "?dots"
-        c = self.sendreq("GET", web_path)
+        c = self.sendreq("GET", web_path, {})
         if c.rx.status != 200:
             self.closeconn(c)
             log(
@@ -440,7 +440,7 @@ class Gateway(object):
             )
         )
 
-        c = self.sendreq("GET", web_path, headers={"Range": hdr_range})
+        c = self.sendreq("GET", web_path, {"Range": hdr_range})
         if c.rx.status != http.client.PARTIAL_CONTENT:
             self.closeconn(c)
             raise Exception(

bin/copyparty-fuse.py

@@ -54,10 +54,13 @@ MACOS = platform.system() == "Darwin"
 info = log = dbg = None
 
 
-print("{} v{} @ {}".format(
+print(
-    platform.python_implementation(),
+    "{} v{} @ {}".format(
-    ".".join([str(x) for x in sys.version_info]),
+        platform.python_implementation(),
-    sys.executable))
+        ".".join([str(x) for x in sys.version_info]),
+        sys.executable,
+    )
+)
 
 
 try:
@@ -299,14 +302,14 @@ class Gateway(object):
         except:
             pass
 
-    def sendreq(self, *args, headers={}, **kwargs):
+    def sendreq(self, meth, path, headers, **kwargs):
         tid = get_tid()
         if self.password:
             headers["Cookie"] = "=".join(["cppwd", self.password])
 
         try:
             c = self.getconn(tid)
-            c.request(*list(args), headers=headers, **kwargs)
+            c.request(meth, path, headers=headers, **kwargs)
             return c.getresponse()
         except:
             dbg("bad conn")
@@ -314,7 +317,7 @@ class Gateway(object):
         self.closeconn(tid)
         try:
             c = self.getconn(tid)
-            c.request(*list(args), headers=headers, **kwargs)
+            c.request(meth, path, headers=headers, **kwargs)
             return c.getresponse()
         except:
             info("http connection failed:\n" + traceback.format_exc())
@@ -331,7 +334,7 @@ class Gateway(object):
             path = dewin(path)
 
         web_path = self.quotep("/" + "/".join([self.web_root, path])) + "?dots&ls"
-        r = self.sendreq("GET", web_path)
+        r = self.sendreq("GET", web_path, {})
         if r.status != 200:
             self.closeconn()
             log(
@@ -368,7 +371,7 @@ class Gateway(object):
             )
         )
 
-        r = self.sendreq("GET", web_path, headers={"Range": hdr_range})
+        r = self.sendreq("GET", web_path, {"Range": hdr_range})
         if r.status != http.client.PARTIAL_CONTENT:
             self.closeconn()
             raise Exception(

bin/dbtool.py

@@ -2,10 +2,13 @@
 
 import os
 import sys
+import time
+import shutil
 import sqlite3
 import argparse
 
-DB_VER = 3
+DB_VER1 = 3
+DB_VER2 = 4
 
 
 def die(msg):
@@ -45,18 +48,21 @@ def compare(n1, d1, n2, d2, verbose):
     nt = next(d1.execute("select count(w) from up"))[0]
     n = 0
     miss = 0
-    for w, rd, fn in d1.execute("select w, rd, fn from up"):
+    for w1, rd, fn in d1.execute("select w, rd, fn from up"):
         n += 1
         if n % 25_000 == 0:
             m = f"\033[36mchecked {n:,} of {nt:,} files in {n1} against {n2}\033[0m"
             print(m)
 
-        q = "select w from up where substr(w,1,16) = ?"
+        if rd.split("/", 1)[0] == ".hist":
-        hit = d2.execute(q, (w[:16],)).fetchone()
+            continue
+
+        q = "select w from up where rd = ? and fn = ?"
+        hit = d2.execute(q, (rd, fn)).fetchone()
         if not hit:
             miss += 1
             if verbose:
-                print(f"file in {n1} missing in {n2}: [{w}] {rd}/{fn}")
+                print(f"file in {n1} missing in {n2}: [{w1}] {rd}/{fn}")
 
     print(f" {miss} files in {n1} missing in {n2}\n")
 
@@ -64,15 +70,30 @@ def compare(n1, d1, n2, d2, verbose):
     n = 0
     miss = {}
     nmiss = 0
-    for w, k, v in d1.execute("select * from mt"):
+    for w1, k, v in d1.execute("select * from mt"):
+
         n += 1
         if n % 100_000 == 0:
             m = f"\033[36mchecked {n:,} of {nt:,} tags in {n1} against {n2}, so far {nmiss} missing tags\033[0m"
             print(m)
 
-        v2 = d2.execute("select v from mt where w = ? and +k = ?", (w, k)).fetchone()
+        q = "select rd, fn from up where substr(w,1,16) = ?"
-        if v2:
+        rd, fn = d1.execute(q, (w1,)).fetchone()
-            v2 = v2[0]
+        if rd.split("/", 1)[0] == ".hist":
+            continue
+
+        q = "select substr(w,1,16) from up where rd = ? and fn = ?"
+        w2 = d2.execute(q, (rd, fn)).fetchone()
+        if w2:
+            w2 = w2[0]
+
+        v2 = None
+        if w2:
+            v2 = d2.execute(
+                "select v from mt where w = ? and +k = ?", (w2, k)
+            ).fetchone()
+            if v2:
+                v2 = v2[0]
 
         # if v != v2 and v2 and k in [".bpm", "key"] and n2 == "src":
         #    print(f"{w} [{rd}/{fn}] {k} = [{v}] / [{v2}]")
@@ -99,9 +120,7 @@ def compare(n1, d1, n2, d2, verbose):
                 miss[k] = 1
 
             if verbose:
-                q = "select rd, fn from up where substr(w,1,16) = ?"
+                print(f"missing in {n2}: [{w1}] [{rd}/{fn}] {k} = {v}")
-                rd, fn = d1.execute(q, (w,)).fetchone()
-                print(f"missing in {n2}: [{w}] [{rd}/{fn}] {k} = {v}")
 
     for k, v in sorted(miss.items()):
         if v:
@@ -114,24 +133,35 @@ def copy_mtp(d1, d2, tag, rm):
     nt = next(d1.execute("select count(w) from mt where k = ?", (tag,)))[0]
     n = 0
     ndone = 0
-    for w, k, v in d1.execute("select * from mt where k = ?", (tag,)):
+    for w1, k, v in d1.execute("select * from mt where k = ?", (tag,)):
         n += 1
         if n % 25_000 == 0:
             m = f"\033[36m{n:,} of {nt:,} tags checked, so far {ndone} copied\033[0m"
             print(m)
 
-        hit = d2.execute("select v from mt where w = ? and +k = ?", (w, k)).fetchone()
+        q = "select rd, fn from up where substr(w,1,16) = ?"
+        rd, fn = d1.execute(q, (w1,)).fetchone()
+        if rd.split("/", 1)[0] == ".hist":
+            continue
+
+        q = "select substr(w,1,16) from up where rd = ? and fn = ?"
+        w2 = d2.execute(q, (rd, fn)).fetchone()
+        if not w2:
+            continue
+
+        w2 = w2[0]
+        hit = d2.execute("select v from mt where w = ? and +k = ?", (w2, k)).fetchone()
         if hit:
             hit = hit[0]
 
         if hit != v:
             ndone += 1
             if hit is not None:
-                d2.execute("delete from mt where w = ? and +k = ?", (w, k))
+                d2.execute("delete from mt where w = ? and +k = ?", (w2, k))
 
-            d2.execute("insert into mt values (?,?,?)", (w, k, v))
+            d2.execute("insert into mt values (?,?,?)", (w2, k, v))
             if rm:
-                d2.execute("delete from mt where w = ? and +k = 't:mtp'", (w,))
+                d2.execute("delete from mt where w = ? and +k = 't:mtp'", (w2,))
 
     d2.commit()
     print(f"copied {ndone} {tag} tags over")
@@ -140,7 +170,7 @@ def copy_mtp(d1, d2, tag, rm):
 def main():
     os.system("")
     print()
-    
+
     ap = argparse.ArgumentParser()
     ap.add_argument("db", help="database to work on")
     ap.add_argument("-src", metavar="DB", type=str, help="database to copy from")
@@ -168,6 +198,23 @@ def main():
     db = sqlite3.connect(ar.db)
     ds = sqlite3.connect(ar.src) if ar.src else None
 
+    # revert journals
+    for d, p in [[db, ar.db], [ds, ar.src]]:
+        if not d:
+            continue
+
+        pj = "{}-journal".format(p)
+        if not os.path.exists(pj):
+            continue
+
+        d.execute("create table foo (bar int)")
+        d.execute("drop table foo")
+
+    if ar.copy:
+        db.close()
+        shutil.copy2(ar.db, "{}.bak.dbtool.{:x}".format(ar.db, int(time.time())))
+        db = sqlite3.connect(ar.db)
+
     for d, n in [[ds, "src"], [db, "dst"]]:
         if not d:
             continue
@@ -176,8 +223,8 @@ def main():
         if ver == "corrupt":
             die("{} database appears to be corrupt, sorry")
 
-        if ver != DB_VER:
+        if ver < DB_VER1 or ver > DB_VER2:
-            m = f"{n} db is version {ver}, this tool only supports version {DB_VER}, please upgrade it with copyparty first"
+            m = f"{n} db is version {ver}, this tool only supports versions between {DB_VER1} and {DB_VER2}, please upgrade it with copyparty first"
             die(m)
 
     if ar.ls:

bin/mtag/audio-bpm.py

@@ -60,7 +60,7 @@ def main():
     try:
         det(tf)
     except:
-        pass
+        pass  # mute
     finally:
         os.unlink(tf)
 

bin/mtag/audio-key-slicing.py

@@ -0,0 +1,123 @@
+#!/usr/bin/env python
+
+import re
+import os
+import sys
+import tempfile
+import subprocess as sp
+
+import keyfinder
+
+from copyparty.util import fsenc
+
+"""
+dep: github/mixxxdj/libkeyfinder
+dep: pypi/keyfinder
+dep: ffmpeg
+
+note: this is a janky edition of the regular audio-key.py,
+  slicing the files at 20sec intervals and keeping 5sec from each,
+  surprisingly accurate but still garbage (446 ok, 69 bad, 13% miss)
+
+  it is fast tho
+"""
+
+
+def get_duration():
+    # TODO provide ffprobe tags to mtp as json
+
+    # fmt: off
+    dur = sp.check_output([
+        "ffprobe",
+        "-hide_banner",
+        "-v", "fatal",
+        "-show_streams",
+        "-show_format",
+        fsenc(sys.argv[1])
+    ])
+    # fmt: on
+
+    dur = dur.decode("ascii", "replace").split("\n")
+    dur = [x.split("=")[1] for x in dur if x.startswith("duration=")]
+    dur = [float(x) for x in dur if re.match(r"^[0-9\.,]+$", x)]
+    return list(sorted(dur))[-1] if dur else None
+
+
+def get_segs(dur):
+    # keep first 5s of each 20s,
+    # keep entire last segment
+    ofs = 0
+    segs = []
+    while True:
+        seg = [ofs, 5]
+        segs.append(seg)
+        if dur - ofs < 20:
+            seg[-1] = int(dur - seg[0])
+            break
+
+        ofs += 20
+
+    return segs
+
+
+def slice(tf):
+    dur = get_duration()
+    dur = min(dur, 600)  # max 10min
+    segs = get_segs(dur)
+
+    # fmt: off
+    cmd = [
+        "ffmpeg",
+        "-nostdin",
+        "-hide_banner",
+        "-v", "fatal",
+        "-y"
+    ]
+
+    for seg in segs:
+        cmd.extend([
+            "-ss", str(seg[0]),
+            "-i", fsenc(sys.argv[1])
+        ])
+    
+    filt = ""
+    for n, seg in enumerate(segs):
+        filt += "[{}:a:0]atrim=duration={}[a{}]; ".format(n, seg[1], n)
+    
+    prev = "a0"
+    for n in range(1, len(segs)):
+        nxt = "b{}".format(n)
+        filt += "[{}][a{}]acrossfade=d=0.5[{}]; ".format(prev, n, nxt)
+        prev = nxt
+
+    cmd.extend([
+        "-filter_complex", filt[:-2],
+        "-map", "[{}]".format(nxt),
+        "-sample_fmt", "s16",
+        tf
+    ])
+    # fmt: on
+
+    # print(cmd)
+    sp.check_call(cmd)
+
+
+def det(tf):
+    slice(tf)
+    print(keyfinder.key(tf).camelot())
+
+
+def main():
+    with tempfile.NamedTemporaryFile(suffix=".flac", delete=False) as f:
+        f.write(b"h")
+        tf = f.name
+
+    try:
+        det(tf)
+    finally:
+        os.unlink(tf)
+        pass
+
+
+if __name__ == "__main__":
+    main()

bin/mtag/audio-key.py

@@ -1,18 +1,54 @@
 #!/usr/bin/env python
 
+import os
 import sys
+import tempfile
+import subprocess as sp
 import keyfinder
 
+from copyparty.util import fsenc
+
 """
 dep: github/mixxxdj/libkeyfinder
 dep: pypi/keyfinder
 dep: ffmpeg
-
-note: cannot fsenc
 """
 
 
-try:
+# tried trimming the first/last 5th, bad idea,
-    print(keyfinder.key(sys.argv[1]).camelot())
+# misdetects 9a law field (Sphere Caliber) as 10b,
-except:
+# obvious when mixing 9a ghostly parapara ship
-    pass
+
+
+def det(tf):
+    # fmt: off
+    sp.check_call([
+        "ffmpeg",
+        "-nostdin",
+        "-hide_banner",
+        "-v", "fatal",
+        "-y", "-i", fsenc(sys.argv[1]),
+        "-t", "300",
+        "-sample_fmt", "s16",
+        tf
+    ])
+    # fmt: on
+
+    print(keyfinder.key(tf).camelot())
+
+
+def main():
+    with tempfile.NamedTemporaryFile(suffix=".flac", delete=False) as f:
+        f.write(b"h")
+        tf = f.name
+
+    try:
+        det(tf)
+    except:
+        pass  # mute
+    finally:
+        os.unlink(tf)
+
+
+if __name__ == "__main__":
+    main()

contrib/nginx/copyparty.conf

@@ -1,3 +1,16 @@
+# when running copyparty behind a reverse proxy,
+# the following arguments are recommended:
+#
+#   -nc 512         important, see next paragraph
+#   --http-only     lower latency on initial connection
+#   -i 127.0.0.1    only accept connections from nginx
+#
+# -nc must match or exceed the webserver's max number of concurrent clients;
+# nginx default is 512  (worker_processes 1, worker_connections 512)
+#
+# you may also consider adding -j0 for CPU-intensive configurations
+# (not that i can really think of any good examples)
+
 upstream cpp {
 	server 127.0.0.1:3923;
 	keepalive 120;

copyparty/__init__.py

@@ -9,6 +9,9 @@ import os
 PY2 = sys.version_info[0] == 2
 if PY2:
     sys.dont_write_bytecode = True
+    unicode = unicode
+else:
+    unicode = str
 
 WINDOWS = False
 if platform.system() == "Windows":

copyparty/__main__.py

@@ -20,7 +20,7 @@ import threading
 import traceback
 from textwrap import dedent
 
-from .__init__ import E, WINDOWS, VT100, PY2
+from .__init__ import E, WINDOWS, VT100, PY2, unicode
 from .__version__ import S_VERSION, S_BUILD_DT, CODENAME
 from .svchub import SvcHub
 from .util import py_desc, align_tab, IMPLICATIONS
@@ -31,6 +31,8 @@ try:
 except:
     HAVE_SSL = False
 
+printed = ""
+
 
 class RiceFormatter(argparse.HelpFormatter):
     def _get_help_string(self, action):
@@ -61,8 +63,15 @@ class Dodge11874(RiceFormatter):
         super(Dodge11874, self).__init__(*args, **kwargs)
 
 
+def lprint(*a, **ka):
+    global printed
+
+    printed += " ".join(unicode(x) for x in a) + ka.get("end", "\n")
+    print(*a, **ka)
+
+
 def warn(msg):
-    print("\033[1mwarning:\033[0;33m {}\033[0m\n".format(msg))
+    lprint("\033[1mwarning:\033[0;33m {}\033[0m\n".format(msg))
 
 
 def ensure_locale():
@@ -73,7 +82,7 @@ def ensure_locale():
     ]:
         try:
             locale.setlocale(locale.LC_ALL, x)
-            print("Locale:", x)
+            lprint("Locale:", x)
             break
         except:
             continue
@@ -94,7 +103,7 @@ def ensure_cert():
 
     try:
         if filecmp.cmp(cert_cfg, cert_insec):
-            print(
+            lprint(
                 "\033[33m  using default TLS certificate; https will be insecure."
                 + "\033[36m\n  certificate location: {}\033[0m\n".format(cert_cfg)
             )
@@ -123,7 +132,7 @@ def configure_ssl_ver(al):
     if "help" in sslver:
         avail = [terse_sslver(x[6:]) for x in flags]
         avail = " ".join(sorted(avail) + ["all"])
-        print("\navailable ssl/tls versions:\n  " + avail)
+        lprint("\navailable ssl/tls versions:\n  " + avail)
         sys.exit(0)
 
     al.ssl_flags_en = 0
@@ -143,7 +152,7 @@ def configure_ssl_ver(al):
 
     for k in ["ssl_flags_en", "ssl_flags_de"]:
         num = getattr(al, k)
-        print("{}: {:8x} ({})".format(k, num, num))
+        lprint("{}: {:8x} ({})".format(k, num, num))
 
     # think i need that beer now
 
@@ -160,13 +169,13 @@ def configure_ssl_ciphers(al):
         try:
             ctx.set_ciphers(al.ciphers)
         except:
-            print("\n\033[1;31mfailed to set ciphers\033[0m\n")
+            lprint("\n\033[1;31mfailed to set ciphers\033[0m\n")
 
     if not hasattr(ctx, "get_ciphers"):
-        print("cannot read cipher list: openssl or python too old")
+        lprint("cannot read cipher list: openssl or python too old")
     else:
         ciphers = [x["description"] for x in ctx.get_ciphers()]
-        print("\n  ".join(["\nenabled ciphers:"] + align_tab(ciphers) + [""]))
+        lprint("\n  ".join(["\nenabled ciphers:"] + align_tab(ciphers) + [""]))
 
     if is_help:
         sys.exit(0)
@@ -222,10 +231,6 @@ def run_argparse(argv, formatter):
               "print,get" prints the data in the log and returns GET
               (leave out the ",get" to return an error instead)
 
-            --ciphers help = available ssl/tls ciphers,
-            --ssl-ver help = available ssl/tls versions,
-              default is what python considers safe, usually >= TLS1
-            
             values for --ls:
               "USR" is a user to browse as; * is anonymous, ** is all users
               "VOL" is a single volume to scan, default is * (all vols)
@@ -243,28 +248,50 @@ def run_argparse(argv, formatter):
         ),
     )
     # fmt: off
-    ap.add_argument("-c", metavar="PATH", type=str, action="append", help="add config file")
+    u = unicode
-    ap.add_argument("-i", metavar="IP", type=str, default="0.0.0.0", help="ip to bind (comma-sep.)")
+    ap2 = ap.add_argument_group('general options')
-    ap.add_argument("-p", metavar="PORT", type=str, default="3923", help="ports to bind (comma/range)")
+    ap2.add_argument("-c", metavar="PATH", type=u, action="append", help="add config file")
-    ap.add_argument("-nc", metavar="NUM", type=int, default=64, help="max num clients")
+    ap2.add_argument("-nc", metavar="NUM", type=int, default=64, help="max num clients")
-    ap.add_argument("-j", metavar="CORES", type=int, default=1, help="max num cpu cores")
+    ap2.add_argument("-j", metavar="CORES", type=int, default=1, help="max num cpu cores")
-    ap.add_argument("-a", metavar="ACCT", type=str, action="append", help="add account")
+    ap2.add_argument("-a", metavar="ACCT", type=u, action="append", help="add account, USER:PASS; example [ed:wark")
-    ap.add_argument("-v", metavar="VOL", type=str, action="append", help="add volume")
+    ap2.add_argument("-v", metavar="VOL", type=u, action="append", help="add volume, SRC:DST:FLAG; example [.::r], [/mnt/nas/music:/music:r:aed")
-    ap.add_argument("-q", action="store_true", help="quiet")
+    ap2.add_argument("-ed", action="store_true", help="enable ?dots")
-    ap.add_argument("-ed", action="store_true", help="enable ?dots")
+    ap2.add_argument("-emp", action="store_true", help="enable markdown plugins")
-    ap.add_argument("-emp", action="store_true", help="enable markdown plugins")
+    ap2.add_argument("-mcr", metavar="SEC", type=int, default=60, help="md-editor mod-chk rate")
-    ap.add_argument("-mcr", metavar="SEC", type=int, default=60, help="md-editor mod-chk rate")
+    ap2.add_argument("--dotpart", action="store_true", help="dotfile incomplete uploads")
-    ap.add_argument("-nw", action="store_true", help="disable writes (benchmark)")
+    ap2.add_argument("--sparse", metavar="MiB", type=int, default=4, help="up2k min.size threshold (mswin-only)")
-    ap.add_argument("-nih", action="store_true", help="no info hostname")
+    ap2.add_argument("--urlform", metavar="MODE", type=u, default="print,get", help="how to handle url-forms; examples: [stash], [save,get]")
-    ap.add_argument("-nid", action="store_true", help="no info disk-usage")
-    ap.add_argument("--dotpart", action="store_true", help="dotfile incomplete uploads")
-    ap.add_argument("--no-zip", action="store_true", help="disable download as zip/tar")
-    ap.add_argument("--sparse", metavar="MiB", type=int, default=4, help="up2k min.size threshold (mswin-only)")
-    ap.add_argument("--urlform", metavar="MODE", type=str, default="print,get", help="how to handle url-forms")
-    ap.add_argument("--salt", type=str, default="hunter2", help="up2k file-hash salt")
 
-    ap2 = ap.add_argument_group('appearance options')
+    ap2 = ap.add_argument_group('network options')
-    ap2.add_argument("--css-browser", metavar="L", help="URL to additional CSS to include")
+    ap2.add_argument("-i", metavar="IP", type=u, default="0.0.0.0", help="ip to bind (comma-sep.)")
+    ap2.add_argument("-p", metavar="PORT", type=u, default="3923", help="ports to bind (comma/range)")
+    ap2.add_argument("--rproxy", metavar="DEPTH", type=int, default=1, help="which ip to keep; 0 = tcp, 1 = origin (first x-fwd), 2 = cloudflare, 3 = nginx, -1 = closest proxy")
+    
+    ap2 = ap.add_argument_group('SSL/TLS options')
+    ap2.add_argument("--http-only", action="store_true", help="disable ssl/tls")
+    ap2.add_argument("--https-only", action="store_true", help="disable plaintext")
+    ap2.add_argument("--ssl-ver", metavar="LIST", type=u, help="set allowed ssl/tls versions; [help] shows available versions; default is what your python version considers safe")
+    ap2.add_argument("--ciphers", metavar="LIST", type=u, help="set allowed ssl/tls ciphers; [help] shows available ciphers")
+    ap2.add_argument("--ssl-dbg", action="store_true", help="dump some tls info")
+    ap2.add_argument("--ssl-log", metavar="PATH", type=u, help="log master secrets")
+
+    ap2 = ap.add_argument_group('opt-outs')
+    ap2.add_argument("-nw", action="store_true", help="disable writes (benchmark)")
+    ap2.add_argument("-nih", action="store_true", help="no info hostname")
+    ap2.add_argument("-nid", action="store_true", help="no info disk-usage")
+    ap2.add_argument("--no-zip", action="store_true", help="disable download as zip/tar")
+
+    ap2 = ap.add_argument_group('safety options')
+    ap2.add_argument("--ls", metavar="U[,V[,F]]", type=u, help="scan all volumes; arguments USER,VOL,FLAGS; example [**,*,ln,p,r]")
+    ap2.add_argument("--salt", type=u, default="hunter2", help="up2k file-hash salt")
+
+    ap2 = ap.add_argument_group('logging options')
+    ap2.add_argument("-q", action="store_true", help="quiet")
+    ap2.add_argument("-lo", metavar="PATH", type=u, help="logfile, example: cpp-%%Y-%%m%%d-%%H%%M%%S.txt.xz")
+    ap2.add_argument("--log-conn", action="store_true", help="print tcp-server msgs")
+    ap2.add_argument("--log-htp", action="store_true", help="print http-server threadpool scaling")
+    ap2.add_argument("--ihead", metavar="HEADER", type=u, action='append', help="dump incoming header")
+    ap2.add_argument("--lf-url", metavar="RE", type=u, default=r"^/\.cpr/|\?th=[wj]$", help="dont log URLs matching")
 
     ap2 = ap.add_argument_group('admin panel options')
     ap2.add_argument("--no-rescan", action="store_true", help="disable ?scan (volume reindexing)")
@@ -279,8 +306,9 @@ def run_argparse(argv, formatter):
     ap2.add_argument("--th-no-webp", action="store_true", help="disable webp output")
     ap2.add_argument("--th-ff-jpg", action="store_true", help="force jpg for video thumbs")
     ap2.add_argument("--th-poke", metavar="SEC", type=int, default=300, help="activity labeling cooldown")
-    ap2.add_argument("--th-clean", metavar="SEC", type=int, default=43200, help="cleanup interval")
+    ap2.add_argument("--th-clean", metavar="SEC", type=int, default=43200, help="cleanup interval; 0=disabled")
     ap2.add_argument("--th-maxage", metavar="SEC", type=int, default=604800, help="max folder age")
+    ap2.add_argument("--th-covers", metavar="N,N", type=u, default="folder.png,folder.jpg,cover.png,cover.jpg", help="folder thumbnails to stat for")
 
     ap2 = ap.add_argument_group('database options')
     ap2.add_argument("-e2d", action="store_true", help="enable up2k database")
@@ -289,32 +317,26 @@ def run_argparse(argv, formatter):
     ap2.add_argument("-e2t", action="store_true", help="enable metadata indexing")
     ap2.add_argument("-e2ts", action="store_true", help="enable metadata scanner, sets -e2t")
     ap2.add_argument("-e2tsr", action="store_true", help="rescan all metadata, sets -e2ts")
-    ap2.add_argument("--hist", metavar="PATH", type=str, help="where to store volume state")
+    ap2.add_argument("--hist", metavar="PATH", type=u, help="where to store volume state")
     ap2.add_argument("--no-hash", action="store_true", help="disable hashing during e2ds folder scans")
     ap2.add_argument("--no-mutagen", action="store_true", help="use ffprobe for tags instead")
     ap2.add_argument("--no-mtag-mt", action="store_true", help="disable tag-read parallelism")
-    ap2.add_argument("-mtm", metavar="M=t,t,t", action="append", type=str, help="add/replace metadata mapping")
+    ap2.add_argument("-mtm", metavar="M=t,t,t", type=u, action="append", help="add/replace metadata mapping")
-    ap2.add_argument("-mte", metavar="M,M,M", type=str, help="tags to index/display (comma-sep.)",
+    ap2.add_argument("-mte", metavar="M,M,M", type=u, help="tags to index/display (comma-sep.)",
         default="circle,album,.tn,artist,title,.bpm,key,.dur,.q,.vq,.aq,ac,vc,res,.fps")
-    ap2.add_argument("-mtp", metavar="M=[f,]bin", action="append", type=str, help="read tag M using bin")
+    ap2.add_argument("-mtp", metavar="M=[f,]bin", type=u, action="append", help="read tag M using bin")
     ap2.add_argument("--srch-time", metavar="SEC", type=int, default=30, help="search deadline")
 
-    ap2 = ap.add_argument_group('SSL/TLS options')
+    ap2 = ap.add_argument_group('appearance options')
-    ap2.add_argument("--http-only", action="store_true", help="disable ssl/tls")
+    ap2.add_argument("--css-browser", metavar="L", type=u, help="URL to additional CSS to include")
-    ap2.add_argument("--https-only", action="store_true", help="disable plaintext")
-    ap2.add_argument("--ssl-ver", metavar="LIST", type=str, help="ssl/tls versions to allow")
-    ap2.add_argument("--ciphers", metavar="LIST", help="set allowed ciphers")
-    ap2.add_argument("--ssl-dbg", action="store_true", help="dump some tls info")
-    ap2.add_argument("--ssl-log", metavar="PATH", help="log master secrets")
 
     ap2 = ap.add_argument_group('debug options')
-    ap2.add_argument("--ls", metavar="U[,V[,F]]", help="scan all volumes")
-    ap2.add_argument("--log-conn", action="store_true", help="print tcp-server msgs")
     ap2.add_argument("--no-sendfile", action="store_true", help="disable sendfile")
     ap2.add_argument("--no-scandir", action="store_true", help="disable scandir")
     ap2.add_argument("--no-fastboot", action="store_true", help="wait for up2k indexing")
-    ap2.add_argument("--ihead", metavar="HEADER", action='append', help="dump incoming header")
+    ap2.add_argument("--no-htp", action="store_true", help="disable httpserver threadpool, create threads as-needed instead")
-    ap2.add_argument("--lf-url", metavar="RE", type=str, default=r"^/\.cpr/|\?th=[wj]$", help="dont log URLs matching")
+    ap2.add_argument("--stackmon", metavar="P,S", type=u, help="write stacktrace to Path every S second")
+    ap2.add_argument("--log-thrs", metavar="SEC", type=float, help="list active threads every SEC")
     
     return ap.parse_args(args=argv[1:])
     # fmt: on
@@ -331,7 +353,7 @@ def main(argv=None):
     desc = py_desc().replace("[", "\033[1;30m[")
 
     f = '\033[36mcopyparty v{} "\033[35m{}\033[36m" ({})\n{}\033[0m\n'
-    print(f.format(S_VERSION, CODENAME, S_BUILD_DT, desc))
+    lprint(f.format(S_VERSION, CODENAME, S_BUILD_DT, desc))
 
     ensure_locale()
     if HAVE_SSL:
@@ -345,7 +367,7 @@ def main(argv=None):
             continue
 
         msg = "\033[1;31mWARNING:\033[0;1m\n  {} \033[0;33mwas replaced with\033[0;1m {} \033[0;33mand will be removed\n\033[0m"
-        print(msg.format(dk, nk))
+        lprint(msg.format(dk, nk))
         argv[idx] = nk
         time.sleep(2)
 
@@ -384,12 +406,12 @@ def main(argv=None):
             + "  (if you crash with codec errors then that is why)"
         )
 
-    if WINDOWS and sys.version_info < (3, 6):
+    if sys.version_info < (3, 6):
         al.no_scandir = True
 
     # signal.signal(signal.SIGINT, sighandler)
 
-    SvcHub(al).run()
+    SvcHub(al, argv, printed).run()
 
 
 if __name__ == "__main__":

copyparty/__version__.py

@@ -1,8 +1,8 @@
 # coding: utf-8
 
-VERSION = (0, 11, 12)
+VERSION = (0, 11, 38)
 CODENAME = "the grid"
-BUILD_DT = (2021, 6, 12)
+BUILD_DT = (2021, 7, 13)
 
 S_VERSION = ".".join(map(str, VERSION))
 S_BUILD_DT = "{0:04d}-{1:02d}-{2:02d}".format(*BUILD_DT)

copyparty/authsrv.py

@@ -10,13 +10,14 @@ import hashlib
 import threading
 
 from .__init__ import WINDOWS
-from .util import IMPLICATIONS, undot, Pebkac, fsdec, fsenc, statdir, nuprint
+from .util import IMPLICATIONS, uncyg, undot, Pebkac, fsdec, fsenc, statdir
 
 
 class VFS(object):
     """single level in the virtual fs"""
 
-    def __init__(self, realpath, vpath, uread=[], uwrite=[], uadm=[], flags={}):
+    def __init__(self, log, realpath, vpath, uread, uwrite, uadm, flags):
+        self.log = log
         self.realpath = realpath  # absolute path on host filesystem
         self.vpath = vpath  # absolute path in the virtual filesystem
         self.uread = uread  # users who can read this
@@ -62,6 +63,7 @@ class VFS(object):
                 return self.nodes[name].add(src, dst)
 
             vn = VFS(
+                self.log,
                 os.path.join(self.realpath, name) if self.realpath else None,
                 "{}/{}".format(self.vpath, name).lstrip("/"),
                 self.uread,
@@ -79,7 +81,7 @@ class VFS(object):
 
         # leaf does not exist; create and keep permissions blank
         vp = "{}/{}".format(self.vpath, dst).lstrip("/")
-        vn = VFS(src, vp)
+        vn = VFS(self.log, src, vp, [], [], [], {})
         vn.dbv = self.dbv or self
         self.nodes[dst] = vn
         return vn
@@ -181,7 +183,7 @@ class VFS(object):
         """return user-readable [fsdir,real,virt] items at vpath"""
         virt_vis = {}  # nodes readable by user
         abspath = self.canonical(rem)
-        real = list(statdir(nuprint, scandir, lstat, abspath))
+        real = list(statdir(self.log, scandir, lstat, abspath))
         real.sort()
         if not rem:
             for name, vn2 in sorted(self.nodes.items()):
@@ -208,8 +210,13 @@ class VFS(object):
             rem, uname, scandir, incl_wo=False, lstat=lstat
         )
 
-        if seen and not fsroot.startswith(seen[-1]) and fsroot in seen:
+        if (
-            print("bailing from symlink loop,\n  {}\n  {}".format(seen[-1], fsroot))
+            seen
+            and (not fsroot.startswith(seen[-1]) or fsroot == seen[-1])
+            and fsroot in seen
+        ):
+            m = "bailing from symlink loop,\n  prev: {}\n  curr: {}\n  from: {}/{}"
+            self.log("vfs.walk", m.format(seen[-1], fsroot, self.vpath, rem), 3)
             return
 
         seen = seen[:] + [fsroot]
@@ -242,6 +249,10 @@ class VFS(object):
         if flt:
             flt = {k: True for k in flt}
 
+        f1 = "{0}.hist{0}up2k.".format(os.sep)
+        f2a = os.sep + "dir.txt"
+        f2b = "{0}.hist{0}".format(os.sep)
+
         for vpath, apath, files, rd, vd in self.walk(
             "", vrem, [], uname, dots, scandir, False
         ):
@@ -275,7 +286,11 @@ class VFS(object):
                     del vd[x]
 
             # up2k filetring based on actual abspath
-            files = [x for x in files if "{0}.hist{0}up2k.".format(os.sep) not in x[1]]
+            files = [
+                x
+                for x in files
+                if f1 not in x[1] and (not x[1].endswith(f2a) or f2b not in x[1])
+            ]
 
             for f in [{"vp": v, "ap": a, "st": n[1]} for v, a, n in files]:
                 yield f
@@ -439,8 +454,8 @@ class AuthSrv(object):
                     raise Exception("invalid -v argument: [{}]".format(v_str))
 
                 src, dst, perms = m.groups()
-                if WINDOWS and src.startswith("/"):
+                if WINDOWS:
-                    src = "{}:\\{}".format(src[1], src[3:])
+                    src = uncyg(src)
 
                 # print("\n".join([src, dst, perms]))
                 src = fsdec(os.path.abspath(fsenc(src)))
@@ -466,15 +481,26 @@ class AuthSrv(object):
                         )
                     except:
                         m = "\n\033[1;31m\nerror in config file {} on line {}:\n\033[0m"
-                        print(m.format(cfg_fn, self.line_ctr))
+                        self.log(m.format(cfg_fn, self.line_ctr), 1)
                         raise
 
+        # case-insensitive; normalize
+        if WINDOWS:
+            cased = {}
+            for k, v in mount.items():
+                try:
+                    cased[k] = fsdec(os.path.realpath(fsenc(v)))
+                except:
+                    cased[k] = v
+
+            mount = cased
+
         if not mount:
             # -h says our defaults are CWD at root and read/write for everyone
-            vfs = VFS(os.path.abspath("."), "", ["*"], ["*"])
+            vfs = VFS(self.log_func, os.path.abspath("."), "", ["*"], ["*"], ["*"], {})
         elif "" not in mount:
             # there's volumes but no root; make root inaccessible
-            vfs = VFS(None, "")
+            vfs = VFS(self.log_func, None, "", [], [], [], {})
             vfs.flags["d2d"] = True
 
         maxdepth = 0
@@ -486,7 +512,13 @@ class AuthSrv(object):
             if dst == "":
                 # rootfs was mapped; fully replaces the default CWD vfs
                 vfs = VFS(
-                    mount[dst], dst, mread[dst], mwrite[dst], madm[dst], mflags[dst]
+                    self.log_func,
+                    mount[dst],
+                    dst,
+                    mread[dst],
+                    mwrite[dst],
+                    madm[dst],
+                    mflags[dst],
                 )
                 continue
 
@@ -524,9 +556,7 @@ class AuthSrv(object):
             if vflag == "-":
                 pass
             elif vflag:
-                if WINDOWS and vflag.startswith("/"):
+                vol.histpath = uncyg(vflag) if WINDOWS else vflag
-                    vflag = "{}:\\{}".format(vflag[1], vflag[3:])
-                vol.histpath = vflag
             elif self.args.hist:
                 for nch in range(len(hid)):
                     hpath = os.path.join(self.args.hist, hid[: nch + 1])
@@ -684,6 +714,11 @@ class AuthSrv(object):
             self.user = user
             self.iuser = {v: k for k, v in user.items()}
 
+            self.re_pwd = None
+            pwds = [re.escape(x) for x in self.iuser.keys()]
+            if pwds:
+                self.re_pwd = re.compile("=(" + "|".join(pwds) + ")([]&; ]|$)")
+
         # import pprint
         # pprint.pprint({"usr": user, "rd": mread, "wr": mwrite, "mnt": mount})
 
@@ -766,7 +801,7 @@ class AuthSrv(object):
                         msg = [x[1] for x in files]
 
                     if msg:
-                        nuprint("\n".join(msg))
+                        self.log("\n" + "\n".join(msg))
 
                 if n_bads and flag_p:
                     raise Exception("found symlink leaving volume, and strict is set")

copyparty/broker_mp.py

@@ -4,17 +4,11 @@ from __future__ import print_function, unicode_literals
 import time
 import threading
 
-from .__init__ import PY2, WINDOWS, VT100
 from .broker_util import try_exec
 from .broker_mpw import MpWorker
 from .util import mp
 
 
-if PY2 and not WINDOWS:
-    from multiprocessing.reduction import ForkingPickler
-    from StringIO import StringIO as MemesIO  # pylint: disable=import-error
-
-
 class BrokerMp(object):
     """external api; manages MpWorkers"""
 
@@ -33,19 +27,17 @@ class BrokerMp(object):
             cores = mp.cpu_count()
 
         self.log("broker", "booting {} subprocesses".format(cores))
-        for n in range(cores):
+        for n in range(1, cores + 1):
             q_pend = mp.Queue(1)
             q_yield = mp.Queue(64)
 
             proc = mp.Process(target=MpWorker, args=(q_pend, q_yield, self.args, n))
             proc.q_pend = q_pend
             proc.q_yield = q_yield
-            proc.nid = n
             proc.clients = {}
-            proc.workload = 0
 
             thr = threading.Thread(
-                target=self.collector, args=(proc,), name="mp-collector"
+                target=self.collector, args=(proc,), name="mp-sink-{}".format(n)
             )
             thr.daemon = True
             thr.start()
@@ -53,13 +45,6 @@ class BrokerMp(object):
             self.procs.append(proc)
             proc.start()
 
-        if not self.args.q:
-            thr = threading.Thread(
-                target=self.debug_load_balancer, name="mp-dbg-loadbalancer"
-            )
-            thr.daemon = True
-            thr.start()
-
     def shutdown(self):
         self.log("broker", "shutting down")
         for n, proc in enumerate(self.procs):
@@ -89,20 +74,6 @@ class BrokerMp(object):
             if dest == "log":
                 self.log(*args)
 
-            elif dest == "workload":
-                with self.mutex:
-                    proc.workload = args[0]
-
-            elif dest == "httpdrop":
-                addr = args[0]
-
-                with self.mutex:
-                    del proc.clients[addr]
-                    if not proc.clients:
-                        proc.workload = 0
-
-                self.hub.tcpsrv.num_clients.add(-1)
-
             elif dest == "retq":
                 # response from previous ipc call
                 with self.retpend_mutex:
@@ -128,38 +99,9 @@ class BrokerMp(object):
         returns a Queue object which eventually contains the response if want_retval
         (not-impl here since nothing uses it yet)
         """
-        if dest == "httpconn":
+        if dest == "listen":
-            sck, addr = args
+            for p in self.procs:
-            sck2 = sck
+                p.q_pend.put([0, dest, [args[0], len(self.procs)]])
-            if PY2:
-                buf = MemesIO()
-                ForkingPickler(buf).dump(sck)
-                sck2 = buf.getvalue()
-
-            proc = sorted(self.procs, key=lambda x: x.workload)[0]
-            proc.q_pend.put([0, dest, [sck2, addr]])
-
-            with self.mutex:
-                proc.clients[addr] = 50
-                proc.workload += 50
 
         else:
             raise Exception("what is " + str(dest))
-
-    def debug_load_balancer(self):
-        fmt = "\033[1m{}\033[0;36m{:4}\033[0m "
-        if not VT100:
-            fmt = "({}{:4})"
-
-        last = ""
-        while self.procs:
-            msg = ""
-            for proc in self.procs:
-                msg += fmt.format(len(proc.clients), proc.workload)
-
-            if msg != last:
-                last = msg
-                with self.hub.log_mutex:
-                    print(msg)
-
-            time.sleep(0.1)

copyparty/broker_mpw.py

@@ -3,18 +3,13 @@ from __future__ import print_function, unicode_literals
 from copyparty.authsrv import AuthSrv
 
 import sys
-import time
 import signal
 import threading
 
-from .__init__ import PY2, WINDOWS
 from .broker_util import ExceptionalQueue
 from .httpsrv import HttpSrv
 from .util import FAKE_MP
 
-if PY2 and not WINDOWS:
-    import pickle  # nosec
-
 
 class MpWorker(object):
     """one single mp instance"""
@@ -25,10 +20,11 @@ class MpWorker(object):
         self.args = args
         self.n = n
 
+        self.log = self._log_disabled if args.q and not args.lo else self._log_enabled
+
         self.retpend = {}
         self.retpend_mutex = threading.Lock()
         self.mutex = threading.Lock()
-        self.workload_thr_alive = False
 
         # we inherited signal_handler from parent,
         # replace it with something harmless
@@ -39,8 +35,7 @@ class MpWorker(object):
         self.asrv = AuthSrv(args, None, False)
 
         # instantiate all services here (TODO: inheritance?)
-        self.httpsrv = HttpSrv(self, True)
+        self.httpsrv = HttpSrv(self, n)
-        self.httpsrv.disconnect_func = self.httpdrop
 
         # on winxp and some other platforms,
         # use thr.join() to block all signals
@@ -53,43 +48,28 @@ class MpWorker(object):
         # print('k')
         pass
 
-    def log(self, src, msg, c=0):
+    def _log_enabled(self, src, msg, c=0):
         self.q_yield.put([0, "log", [src, msg, c]])
 
+    def _log_disabled(self, src, msg, c=0):
+        pass
+
     def logw(self, msg, c=0):
         self.log("mp{}".format(self.n), msg, c)
 
-    def httpdrop(self, addr):
-        self.q_yield.put([0, "httpdrop", [addr]])
-
     def main(self):
         while True:
             retq_id, dest, args = self.q_pend.get()
 
             # self.logw("work: [{}]".format(d[0]))
             if dest == "shutdown":
+                self.httpsrv.shutdown()
                 self.logw("ok bye")
                 sys.exit(0)
                 return
 
-            elif dest == "httpconn":
+            elif dest == "listen":
-                sck, addr = args
+                self.httpsrv.listen(args[0], args[1])
-                if PY2:
-                    sck = pickle.loads(sck)  # nosec
-
-                if self.args.log_conn:
-                    self.log("%s %s" % addr, "|%sC-qpop" % ("-" * 4,), c="1;30")
-
-                self.httpsrv.accept(sck, addr)
-
-                with self.mutex:
-                    if not self.workload_thr_alive:
-                        self.workload_thr_alive = True
-                        thr = threading.Thread(
-                            target=self.thr_workload, name="mpw-workload"
-                        )
-                        thr.daemon = True
-                        thr.start()
 
             elif dest == "retq":
                 # response from previous ipc call
@@ -113,16 +93,3 @@ class MpWorker(object):
 
         self.q_yield.put([retq_id, dest, args])
         return retq
-
-    def thr_workload(self):
-        """announce workloads to MpSrv (the mp controller / loadbalancer)"""
-        # avoid locking in extract_filedata by tracking difference here
-        while True:
-            time.sleep(0.2)
-            with self.mutex:
-                if self.httpsrv.num_clients() == 0:
-                    # no clients rn, termiante thread
-                    self.workload_thr_alive = False
-                    return
-
-            self.q_yield.put([0, "workload", [self.httpsrv.workload]])

copyparty/broker_thr.py

@@ -3,7 +3,6 @@ from __future__ import print_function, unicode_literals
 
 import threading
 
-from .authsrv import AuthSrv
 from .httpsrv import HttpSrv
 from .broker_util import ExceptionalQueue, try_exec
 
@@ -20,20 +19,16 @@ class BrokerThr(object):
         self.mutex = threading.Lock()
 
         # instantiate all services here (TODO: inheritance?)
-        self.httpsrv = HttpSrv(self)
+        self.httpsrv = HttpSrv(self, None)
-        self.httpsrv.disconnect_func = self.httpdrop
 
     def shutdown(self):
         # self.log("broker", "shutting down")
+        self.httpsrv.shutdown()
         pass
 
     def put(self, want_retval, dest, *args):
-        if dest == "httpconn":
+        if dest == "listen":
-            sck, addr = args
+            self.httpsrv.listen(args[0], 1)
-            if self.args.log_conn:
-                self.log("%s %s" % addr, "|%sC-qpop" % ("-" * 4,), c="1;30")
-
-            self.httpsrv.accept(sck, addr)
 
         else:
             # new ipc invoking managed service in hub
@@ -50,6 +45,3 @@ class BrokerThr(object):
             retq = ExceptionalQueue(1)
             retq.put(rv)
             return retq
-
-    def httpdrop(self, addr):
-        self.hub.tcpsrv.num_clients.add(-1)

copyparty/httpcli.py

@@ -10,19 +10,15 @@ import json
 import string
 import socket
 import ctypes
-import traceback
 from datetime import datetime
 import calendar
 
-from .__init__ import E, PY2, WINDOWS, ANYWIN
+from .__init__ import E, PY2, WINDOWS, ANYWIN, unicode
 from .util import *  # noqa  # pylint: disable=unused-wildcard-import
 from .authsrv import AuthSrv
 from .szip import StreamZip
 from .star import StreamTar
 
-if not PY2:
-    unicode = str
-
 
 NO_CACHE = {"Cache-Control": "no-cache"}
 NO_STORE = {"Cache-Control": "no-store; max-age=0"}
@@ -41,7 +37,6 @@ class HttpCli(object):
         self.ip = conn.addr[0]
         self.addr = conn.addr  # type: tuple[str, int]
         self.args = conn.args
-        self.is_mp = conn.is_mp
         self.asrv = conn.asrv  # type: AuthSrv
         self.ico = conn.ico
         self.thumbcli = conn.thumbcli
@@ -50,12 +45,21 @@ class HttpCli(object):
         self.tls = hasattr(self.s, "cipher")
 
         self.bufsz = 1024 * 32
+        self.hint = None
         self.absolute_urls = False
         self.out_headers = {"Access-Control-Allow-Origin": "*"}
 
     def log(self, msg, c=0):
+        ptn = self.asrv.re_pwd
+        if ptn and ptn.search(msg):
+            msg = ptn.sub(self.unpwd, msg)
+
         self.log_func(self.log_src, msg, c)
 
+    def unpwd(self, m):
+        a, b = m.groups()
+        return "=\033[7m {} \033[27m{}".format(self.asrv.iuser[a], b)
+
     def _check_nonfatal(self, ex):
         return ex.code < 400 or ex.code in [404, 429]
 
@@ -64,14 +68,19 @@ class HttpCli(object):
         if rem.startswith("/") or rem.startswith("../") or "/../" in rem:
             raise Exception("that was close")
 
-    def j2(self, name, **kwargs):
+    def j2(self, name, **ka):
         tpl = self.conn.hsrv.j2[name]
-        return tpl.render(**kwargs) if kwargs else tpl
+        if ka:
+            ka["ts"] = self.conn.hsrv.cachebuster()
+            return tpl.render(**ka)
+
+        return tpl
 
     def run(self):
         """returns true if connection can be reused"""
         self.keepalive = False
         self.headers = {}
+        self.hint = None
         try:
             headerlines = read_header(self.sr)
             if not headerlines:
@@ -85,9 +94,13 @@ class HttpCli(object):
             try:
                 self.mode, self.req, self.http_ver = headerlines[0].split(" ")
             except:
-                raise Pebkac(400, "bad headers:\n" + "\n".join(headerlines))
+                msg = " ]\n#[ ".join(headerlines)
+                raise Pebkac(400, "bad headers:\n#[ " + msg + " ]")
 
         except Pebkac as ex:
+            self.mode = "GET"
+            self.req = "[junk]"
+            self.http_ver = "HTTP/1.1"
             # self.log("pebkac at httpcli.run #1: " + repr(ex))
             self.keepalive = self._check_nonfatal(ex)
             self.loud_reply(unicode(ex), status=ex.code)
@@ -104,10 +117,21 @@ class HttpCli(object):
         v = self.headers.get("connection", "").lower()
         self.keepalive = not v.startswith("close") and self.http_ver != "HTTP/1.0"
 
-        v = self.headers.get("x-forwarded-for", None)
+        n = self.args.rproxy
-        if v is not None and self.conn.addr[0] in ["127.0.0.1", "::1"]:
+        if n:
-            self.ip = v.split(",")[0]
+            v = self.headers.get("x-forwarded-for")
-            self.log_src = self.conn.set_rproxy(self.ip)
+            if v and self.conn.addr[0] in ["127.0.0.1", "::1"]:
+                if n > 0:
+                    n -= 1
+
+                vs = v.split(",")
+                try:
+                    self.ip = vs[n].strip()
+                except:
+                    self.ip = vs[0].strip()
+                    self.log("rproxy={} oob x-fwd {}".format(self.args.rproxy, v), c=3)
+
+                self.log_src = self.conn.set_rproxy(self.ip)
 
         if self.args.ihead:
             keys = self.args.ihead
@@ -119,6 +143,9 @@ class HttpCli(object):
                 if v is not None:
                     self.log("[H] {}: \033[33m[{}]".format(k, v), 6)
 
+        if "&" in self.req and "?" not in self.req:
+            self.hint = "did you mean '?' instead of '&'"
+
         # split req into vpath + uparam
         uparam = {}
         if "?" not in self.req:
@@ -158,6 +185,9 @@ class HttpCli(object):
         self.rvol, self.wvol, self.avol = [[], [], []]
         self.asrv.vfs.user_tree(self.uname, self.rvol, self.wvol, self.avol)
 
+        if pwd and "pw" in self.ouparam and pwd != cookies.get("cppwd"):
+            self.out_headers["Set-Cookie"] = self.get_pwd_cookie(pwd)[0]
+
         ua = self.headers.get("user-agent", "")
         self.is_rclone = ua.startswith("rclone/")
         if self.is_rclone:
@@ -188,12 +218,15 @@ class HttpCli(object):
 
                 self.log("{}\033[0m, {}".format(str(ex), self.vpath), 3)
                 msg = "<pre>{}\r\nURL: {}\r\n".format(str(ex), self.vpath)
+                if self.hint:
+                    msg += "hint: {}\r\n".format(self.hint)
+
                 self.reply(msg.encode("utf-8", "replace"), status=ex.code)
                 return self.keepalive
             except Pebkac:
                 return False
 
-    def send_headers(self, length, status=200, mime=None, headers={}):
+    def send_headers(self, length, status=200, mime=None, headers=None):
         response = ["{} {} {}".format(self.http_ver, status, HTTPCODE[status])]
 
         if length is not None:
@@ -203,7 +236,8 @@ class HttpCli(object):
         response.append("Connection: " + ("Keep-Alive" if self.keepalive else "Close"))
 
         # headers{} overrides anything set previously
-        self.out_headers.update(headers)
+        if headers:
+            self.out_headers.update(headers)
 
         # default to utf8 html if no content-type is set
         if not mime:
@@ -220,7 +254,7 @@ class HttpCli(object):
         except:
             raise Pebkac(400, "client d/c while replying headers")
 
-    def reply(self, body, status=200, mime=None, headers={}):
+    def reply(self, body, status=200, mime=None, headers=None):
         # TODO something to reply with user-supplied values safely
         self.send_headers(len(body), status, mime, headers)
 
@@ -236,7 +270,7 @@ class HttpCli(object):
         self.log(body.rstrip())
         self.reply(b"<pre>" + body.encode("utf-8") + b"\r\n", *list(args), **kwargs)
 
-    def urlq(self, add={}, rm=[]):
+    def urlq(self, add, rm):
         """
         generates url query based on uparam (b, pw, all others)
         removing anything in rm, adding pairs in add
@@ -245,10 +279,11 @@ class HttpCli(object):
         if self.is_rclone:
             return ""
 
+        cmap = {"pw": "cppwd"}
         kv = {
             k: v
             for k, v in self.uparam.items()
-            if k not in rm and self.cookies.get(k) != v
+            if k not in rm and self.cookies.get(cmap.get(k, k)) != v
         }
         kv.update(add)
         if not kv:
@@ -307,6 +342,9 @@ class HttpCli(object):
         if "tree" in self.uparam:
             return self.tx_tree()
 
+        if "stack" in self.uparam:
+            return self.tx_stack()
+
         # conditional redirect to single volumes
         if self.vpath == "" and not self.ouparam:
             nread = len(self.rvol)
@@ -336,9 +374,6 @@ class HttpCli(object):
         if "scan" in self.uparam:
             return self.scanvol()
 
-        if "stack" in self.uparam:
-            return self.tx_stack()
-
         return self.tx_browser()
 
     def handle_options(self):
@@ -444,15 +479,17 @@ class HttpCli(object):
         addr = self.ip.replace(":", ".")
         fn = "put-{:.6f}-{}.bin".format(time.time(), addr)
         path = os.path.join(fdir, fn)
+        if self.args.nw:
+            path = os.devnull
 
         with open(fsenc(path), "wb", 512 * 1024) as f:
-            post_sz, _, sha_b64 = hashcopy(self.conn, reader, f)
+            post_sz, _, sha_b64 = hashcopy(reader, f)
 
-        vfs, vrem = vfs.get_dbv(rem)
+        if not self.args.nw:
-
+            vfs, vrem = vfs.get_dbv(rem)
-        self.conn.hsrv.broker.put(
+            self.conn.hsrv.broker.put(
-            False, "up2k.hash_file", vfs.realpath, vfs.flags, vrem, fn
+                False, "up2k.hash_file", vfs.realpath, vfs.flags, vrem, fn
-        )
+            )
 
         return post_sz, sha_b64, remains, path
 
@@ -469,7 +506,7 @@ class HttpCli(object):
 
         spd1 = get_spd(nbytes, self.t0)
         spd2 = get_spd(self.conn.nbyte, self.conn.t0)
-        return spd1 + " " + spd2
+        return "{} {} n{}".format(spd1, spd2, self.conn.nreq)
 
     def handle_post_multipart(self):
         self.parser = MultipartParser(self.log, self.sr, self.headers)
@@ -569,10 +606,20 @@ class HttpCli(object):
         if sub:
             try:
                 dst = os.path.join(vfs.realpath, rem)
-                os.makedirs(fsenc(dst))
-            except:
                 if not os.path.isdir(fsenc(dst)):
-                    raise Pebkac(400, "some file got your folder name")
+                    os.makedirs(fsenc(dst))
+            except OSError as ex:
+                self.log("makedirs failed [{}]".format(dst))
+                if not os.path.isdir(fsenc(dst)):
+                    if ex.errno == 13:
+                        raise Pebkac(500, "the server OS denied write-access")
+
+                    if ex.errno == 17:
+                        raise Pebkac(400, "some file got your folder name")
+
+                    raise Pebkac(500, min_ex())
+            except:
+                raise Pebkac(500, min_ex())
 
         x = self.conn.hsrv.broker.put(True, "up2k.handle_json", body)
         ret = x.get()
@@ -602,7 +649,7 @@ class HttpCli(object):
             penalty = 0.7
             t_idle = t0 - idx.p_end
             if idx.p_dur > 0.7 and t_idle < penalty:
-                m = "rate-limit ({:.1f} sec), cost {:.2f}, idle {:.2f}"
+                m = "rate-limit {:.1f} sec, cost {:.2f}, idle {:.2f}"
                 raise Pebkac(429, m.format(penalty, idx.p_dur, t_idle))
 
         if "srch" in body:
@@ -668,7 +715,7 @@ class HttpCli(object):
 
         with open(fsenc(path), "rb+", 512 * 1024) as f:
             f.seek(cstart[0])
-            post_sz, _, sha_b64 = hashcopy(self.conn, reader, f)
+            post_sz, _, sha_b64 = hashcopy(reader, f)
 
             if sha_b64 != chash:
                 raise Pebkac(
@@ -722,6 +769,12 @@ class HttpCli(object):
         pwd = self.parser.require("cppwd", 64)
         self.parser.drop()
 
+        ck, msg = self.get_pwd_cookie(pwd)
+        html = self.j2("msg", h1=msg, h2='<a href="/">ack</a>', redir="/")
+        self.reply(html.encode("utf-8"), headers={"Set-Cookie": ck})
+        return True
+
+    def get_pwd_cookie(self, pwd):
         if pwd in self.asrv.iuser:
             msg = "login ok"
             dt = datetime.utcfromtimestamp(time.time() + 60 * 60 * 24 * 365)
@@ -732,9 +785,7 @@ class HttpCli(object):
             exp = "Fri, 15 Aug 1997 01:00:00 GMT"
 
         ck = "cppwd={}; Path=/; Expires={}; SameSite=Lax".format(pwd, exp)
-        html = self.j2("msg", h1=msg, h2='<a href="/">ack</a>', redir="/")
+        return [ck, msg]
-        self.reply(html.encode("utf-8"), headers={"Set-Cookie": ck})
-        return True
 
     def handle_mkdir(self):
         new_dir = self.parser.require("name", 512)
@@ -744,7 +795,7 @@ class HttpCli(object):
         vfs, rem = self.asrv.vfs.get(self.vpath, self.uname, False, True)
         self._assert_safe_rem(rem)
 
-        sanitized = sanitize_fn(new_dir)
+        sanitized = sanitize_fn(new_dir, "", [])
 
         if not nullwrite:
             fdir = os.path.join(vfs.realpath, rem)
@@ -758,8 +809,13 @@ class HttpCli(object):
 
             try:
                 os.mkdir(fsenc(fn))
+            except OSError as ex:
+                if ex.errno == 13:
+                    raise Pebkac(500, "the server OS denied write-access")
+
+                raise Pebkac(500, "mkdir failed:\n" + min_ex())
             except:
-                raise Pebkac(500, "mkdir failed, check the logs")
+                raise Pebkac(500, min_ex())
 
         vpath = "{}/{}".format(self.vpath, sanitized).lstrip("/")
         self.redirect(vpath)
@@ -776,7 +832,7 @@ class HttpCli(object):
         if not new_file.endswith(".md"):
             new_file += ".md"
 
-        sanitized = sanitize_fn(new_file)
+        sanitized = sanitize_fn(new_file, "", [])
 
         if not nullwrite:
             fdir = os.path.join(vfs.realpath, rem)
@@ -809,7 +865,7 @@ class HttpCli(object):
                 if p_file and not nullwrite:
                     fdir = os.path.join(vfs.realpath, rem)
                     fname = sanitize_fn(
-                        p_file, bad=[".prologue.html", ".epilogue.html"]
+                        p_file, "", [".prologue.html", ".epilogue.html"]
                     )
 
                     if not os.path.isdir(fsenc(fdir)):
@@ -826,7 +882,7 @@ class HttpCli(object):
                     with ren_open(fname, "wb", 512 * 1024, **open_args) as f:
                         f, fname = f["orz"]
                         self.log("writing to {}/{}".format(fdir, fname))
-                        sz, sha512_hex, _ = hashcopy(self.conn, p_data, f)
+                        sz, sha512_hex, _ = hashcopy(p_data, f)
                         if sz == 0:
                             raise Pebkac(400, "empty files in post")
 
@@ -1009,7 +1065,7 @@ class HttpCli(object):
             raise Pebkac(400, "expected body, got {}".format(p_field))
 
         with open(fsenc(fp), "wb", 512 * 1024) as f:
-            sz, sha512, _ = hashcopy(self.conn, p_data, f)
+            sz, sha512, _ = hashcopy(p_data, f)
 
         new_lastmod = os.stat(fsenc(fp)).st_mtime
         new_lastmod3 = int(new_lastmod * 1000)
@@ -1176,7 +1232,7 @@ class HttpCli(object):
         #
         # send reply
 
-        if not is_compressed:
+        if not is_compressed and "cache" not in self.uparam:
             self.out_headers.update(NO_CACHE)
 
         self.out_headers["Accept-Ranges"] = "bytes"
@@ -1199,8 +1255,7 @@ class HttpCli(object):
             if use_sendfile:
                 remains = sendfile_kern(lower, upper, f, self.s)
             else:
-                actor = self.conn if self.is_mp else None
+                remains = sendfile_py(lower, upper, f, self.s)
-                remains = sendfile_py(lower, upper, f, self.s, actor)
 
         if remains > 0:
             logmsg += " \033[31m" + unicode(upper - remains) + "\033[0m"
@@ -1257,7 +1312,7 @@ class HttpCli(object):
 
         fgen = vn.zipgen(rem, items, self.uname, dots, not self.args.no_scandir)
         # for f in fgen: print(repr({k: f[k] for k in ["vp", "ap"]}))
-        bgen = packer(fgen, utf8="utf" in uarg, pre_crc="crc" in uarg)
+        bgen = packer(self.log, fgen, utf8="utf" in uarg, pre_crc="crc" in uarg)
         bsent = 0
         for buf in bgen.gen():
             if not buf:
@@ -1279,7 +1334,7 @@ class HttpCli(object):
             ext = "folder"
             exact = True
 
-        bad = re.compile(r"[](){}/[]|^[0-9_-]*$")
+        bad = re.compile(r"[](){}/ []|^[0-9_-]*$")
         n = ext.split(".")[::-1]
         if not exact:
             n = n[:-1]
@@ -1335,6 +1390,7 @@ class HttpCli(object):
             "md_plug": "true" if self.args.emp else "false",
             "md_chk_rate": self.args.mcr,
             "md": boundary,
+            "ts": self.conn.hsrv.cachebuster(),
         }
         html = template.render(**targs).encode("utf-8", "replace")
         html = html.split(boundary.encode("utf-8"))
@@ -1367,7 +1423,7 @@ class HttpCli(object):
         return True
 
     def tx_mounts(self):
-        suf = self.urlq(rm=["h"])
+        suf = self.urlq({}, ["h"])
         rvol, wvol, avol = [
             [("/" + x).rstrip("/") + "/" for x in y]
             for y in [self.rvol, self.wvol, self.avol]
@@ -1417,39 +1473,14 @@ class HttpCli(object):
         raise Pebkac(500, x)
 
     def tx_stack(self):
-        if not self.readable or not self.writable:
+        if not self.avol:
             raise Pebkac(403, "not admin")
 
         if self.args.no_stack:
             raise Pebkac(403, "disabled by argv")
 
-        threads = {}
+        ret = "<pre>{}\n{}".format(time.time(), alltrace())
-        names = dict([(t.ident, t.name) for t in threading.enumerate()])
+        self.reply(ret.encode("utf-8"))
-        for tid, stack in sys._current_frames().items():
-            name = "{} ({:x})".format(names.get(tid), tid)
-            threads[name] = stack
-
-        rret = []
-        bret = []
-        for name, stack in sorted(threads.items()):
-            ret = ["\n\n# {}".format(name)]
-            pad = None
-            for fn, lno, name, line in traceback.extract_stack(stack):
-                fn = os.sep.join(fn.split(os.sep)[-3:])
-                ret.append('File: "{}", line {}, in {}'.format(fn, lno, name))
-                if line:
-                    ret.append("  " + str(line.strip()))
-                    if "self.not_empty.wait()" in line:
-                        pad = " " * 4
-
-            if pad:
-                bret += [ret[0]] + [pad + x for x in ret[1:]]
-            else:
-                rret += ret
-
-        ret = rret + bret
-        ret = ("<pre>" + "\n".join(ret)).encode("utf-8")
-        self.reply(ret)
 
     def tx_tree(self):
         top = self.uparam["tree"] or ""
@@ -1532,14 +1563,16 @@ class HttpCli(object):
             raise Pebkac(404)
 
         if self.readable:
-            if rem.startswith(".hist/up2k."):
+            if rem.startswith(".hist/up2k.") or (
+                rem.endswith("/dir.txt") and rem.startswith(".hist/th/")
+            ):
                 raise Pebkac(403)
 
             is_dir = stat.S_ISDIR(st.st_mode)
             th_fmt = self.uparam.get("th")
             if th_fmt is not None:
                 if is_dir:
-                    for fn in ["folder.png", "folder.jpg"]:
+                    for fn in self.args.th_covers.split(","):
                         fp = os.path.join(abspath, fn)
                         if os.path.exists(fp):
                             vrem = "{}/{}".format(vrem.rstrip("/"), fn)
@@ -1601,9 +1634,8 @@ class HttpCli(object):
         if self.writable:
             perms.append("write")
 
-        url_suf = self.urlq()
+        url_suf = self.urlq({}, [])
         is_ls = "ls" in self.uparam
-        ts = ""  # "?{}".format(time.time())
 
         tpl = "browser"
         if "b" in self.uparam:
@@ -1628,7 +1660,6 @@ class HttpCli(object):
             "vdir": quotep(self.vpath),
             "vpnodes": vpnodes,
             "files": [],
-            "ts": ts,
             "perms": json.dumps(perms),
             "taglist": [],
             "tag_order": [],
@@ -1764,28 +1795,44 @@ class HttpCli(object):
             fn = f["name"]
             rd = f["rd"]
             del f["rd"]
-            if icur:
+            if not icur:
-                if vn != dbv:
+                break
-                    _, rd = vn.get_dbv(rd)
+
+            if vn != dbv:
+                _, rd = vn.get_dbv(rd)
+
+            q = "select w from up where rd = ? and fn = ?"
+            r = None
+            try:
+                r = icur.execute(q, (rd, fn)).fetchone()
+            except Exception as ex:
+                if "database is locked" in str(ex):
+                    break
 
-                q = "select w from up where rd = ? and fn = ?"
                 try:
-                    r = icur.execute(q, (rd, fn)).fetchone()
-                except:
                     args = s3enc(idx.mem_cur, rd, fn)
                     r = icur.execute(q, args).fetchone()
+                except:
+                    m = "tag list error, {}/{}\n{}"
+                    self.log(m.format(rd, fn, min_ex()))
+                    break
 
-                tags = {}
+            tags = {}
-                f["tags"] = tags
+            f["tags"] = tags
 
-                if not r:
+            if not r:
-                    continue
+                continue
 
-                w = r[0][:16]
+            w = r[0][:16]
-                q = "select k, v from mt where w = ? and k != 'x'"
+            q = "select k, v from mt where w = ? and k != 'x'"
+            try:
                 for k, v in icur.execute(q, (w,)):
                     taglist[k] = True
                     tags[k] = v
+            except:
+                m = "tag read error, {}/{} [{}]:\n{}"
+                self.log(m.format(rd, fn, w, min_ex()))
+                break
 
         if icur:
             taglist = [k for k in vn.flags.get("mte", "").split(",") if k in taglist]

copyparty/httpconn.py

@@ -3,7 +3,6 @@ from __future__ import print_function, unicode_literals
 
 import re
 import os
-import sys
 import time
 import socket
 
@@ -35,7 +34,6 @@ class HttpConn(object):
 
         self.args = hsrv.args
         self.asrv = hsrv.asrv
-        self.is_mp = hsrv.is_mp
         self.cert_path = hsrv.cert_path
 
         enth = HAVE_PIL and not self.args.no_thumb
@@ -43,13 +41,22 @@ class HttpConn(object):
         self.ico = Ico(self.args)
 
         self.t0 = time.time()
+        self.stopping = False
+        self.nreq = 0
         self.nbyte = 0
-        self.workload = 0
         self.u2idx = None
         self.log_func = hsrv.log
         self.lf_url = re.compile(self.args.lf_url) if self.args.lf_url else None
         self.set_rproxy()
 
+    def shutdown(self):
+        self.stopping = True
+        try:
+            self.s.shutdown(socket.SHUT_RDWR)
+            self.s.close()
+        except:
+            pass
+
     def set_rproxy(self, ip=None):
         if ip is None:
             color = 36
@@ -163,7 +170,7 @@ class HttpConn(object):
                     self.log("client rejected our certificate (nice)")
 
                 elif "ALERT_CERTIFICATE_UNKNOWN" in em:
-                    # chrome-android keeps doing this
+                    # android-chrome keeps doing this
                     pass
 
                 else:
@@ -174,12 +181,8 @@ class HttpConn(object):
         if not self.sr:
             self.sr = Unrecv(self.s)
 
-        while True:
+        while not self.stopping:
-            if self.is_mp:
+            self.nreq += 1
-                self.workload += 50
-                if self.workload >= 2 ** 31:
-                    self.workload = 100
-
             cli = HttpCli(self)
             if not cli.run():
                 return

copyparty/httpsrv.py

@@ -4,6 +4,8 @@ from __future__ import print_function, unicode_literals
 import os
 import sys
 import time
+import math
+import base64
 import socket
 import threading
 
@@ -24,10 +26,15 @@ except ImportError:
     )
     sys.exit(1)
 
-from .__init__ import E, MACOS
+from .__init__ import E, PY2, MACOS
-from .authsrv import AuthSrv
+from .util import spack, min_ex, start_stackmon, start_log_thrs
 from .httpconn import HttpConn
 
+if PY2:
+    import Queue as queue
+else:
+    import queue
+
 
 class HttpSrv(object):
     """
@@ -35,19 +42,28 @@ class HttpSrv(object):
     relying on MpSrv for performance (HttpSrv is just plain threads)
     """
 
-    def __init__(self, broker, is_mp=False):
+    def __init__(self, broker, nid):
         self.broker = broker
-        self.is_mp = is_mp
+        self.nid = nid
         self.args = broker.args
         self.log = broker.log
         self.asrv = broker.asrv
 
-        self.disconnect_func = None
+        self.name = "httpsrv" + ("-n{}-i{:x}".format(nid, os.getpid()) if nid else "")
         self.mutex = threading.Lock()
+        self.stopping = False
 
-        self.clients = {}
+        self.tp_nthr = 0  # actual
-        self.workload = 0
+        self.tp_ncli = 0  # fading
-        self.workload_thr_alive = False
+        self.tp_time = None  # latest worker collect
+        self.tp_q = None if self.args.no_htp else queue.LifoQueue()
+
+        self.srvs = []
+        self.ncli = 0  # exact
+        self.clients = {}  # laggy
+        self.nclimax = 0
+        self.cb_ts = 0
+        self.cb_v = 0
 
         env = jinja2.Environment()
         env.loader = jinja2.FileSystemLoader(os.path.join(E.mod, "web"))
@@ -62,25 +78,170 @@ class HttpSrv(object):
         else:
             self.cert_path = None
 
+        if self.tp_q:
+            self.start_threads(4)
+
+            name = "httpsrv-scaler" + ("-{}".format(nid) if nid else "")
+            t = threading.Thread(target=self.thr_scaler, name=name)
+            t.daemon = True
+            t.start()
+
+        if nid:
+            if self.args.stackmon:
+                start_stackmon(self.args.stackmon, nid)
+
+            if self.args.log_thrs:
+                start_log_thrs(self.log, self.args.log_thrs, nid)
+
+    def start_threads(self, n):
+        self.tp_nthr += n
+        if self.args.log_htp:
+            self.log(self.name, "workers += {} = {}".format(n, self.tp_nthr), 6)
+
+        for _ in range(n):
+            thr = threading.Thread(
+                target=self.thr_poolw,
+                name=self.name + "-poolw",
+            )
+            thr.daemon = True
+            thr.start()
+
+    def stop_threads(self, n):
+        self.tp_nthr -= n
+        if self.args.log_htp:
+            self.log(self.name, "workers -= {} = {}".format(n, self.tp_nthr), 6)
+
+        for _ in range(n):
+            self.tp_q.put(None)
+
+    def thr_scaler(self):
+        while True:
+            time.sleep(2 if self.tp_ncli else 30)
+            with self.mutex:
+                self.tp_ncli = max(self.ncli, self.tp_ncli - 2)
+                if self.tp_nthr > self.tp_ncli + 8:
+                    self.stop_threads(4)
+
+    def listen(self, sck, nlisteners):
+        ip, port = sck.getsockname()
+        self.srvs.append(sck)
+        self.nclimax = math.ceil(self.args.nc * 1.0 / nlisteners)
+        t = threading.Thread(
+            target=self.thr_listen,
+            args=(sck,),
+            name="httpsrv-n{}-listen-{}-{}".format(self.nid or "0", ip, port),
+        )
+        t.daemon = True
+        t.start()
+
+    def thr_listen(self, srv_sck):
+        """listens on a shared tcp server"""
+        ip, port = srv_sck.getsockname()
+        fno = srv_sck.fileno()
+        msg = "subscribed @ {}:{}  f{}".format(ip, port, fno)
+        self.log(self.name, msg)
+        while not self.stopping:
+            if self.args.log_conn:
+                self.log(self.name, "|%sC-ncli" % ("-" * 1,), c="1;30")
+
+            if self.ncli >= self.nclimax:
+                self.log(self.name, "at connection limit; waiting", 3)
+                while self.ncli >= self.nclimax:
+                    time.sleep(0.1)
+
+            if self.args.log_conn:
+                self.log(self.name, "|%sC-acc1" % ("-" * 2,), c="1;30")
+
+            try:
+                sck, addr = srv_sck.accept()
+            except (OSError, socket.error) as ex:
+                self.log(self.name, "accept({}): {}".format(fno, ex), c=6)
+                time.sleep(0.02)
+                continue
+
+            if self.args.log_conn:
+                m = "|{}C-acc2 \033[0;36m{} \033[3{}m{}".format(
+                    "-" * 3, ip, port % 8, port
+                )
+                self.log("%s %s" % addr, m, c="1;30")
+
+            self.accept(sck, addr)
+
     def accept(self, sck, addr):
         """takes an incoming tcp connection and creates a thread to handle it"""
-        if self.args.log_conn:
+        now = time.time()
-            self.log("%s %s" % addr, "|%sC-cthr" % ("-" * 5,), c="1;30")
+
+        if self.tp_time and now - self.tp_time > 300:
+            self.tp_q = None
+
+        if self.tp_q:
+            self.tp_q.put((sck, addr))
+            with self.mutex:
+                self.ncli += 1
+                self.tp_time = self.tp_time or now
+                self.tp_ncli = max(self.tp_ncli, self.ncli + 1)
+                if self.tp_nthr < self.ncli + 4:
+                    self.start_threads(8)
+            return
+
+        if not self.args.no_htp:
+            m = "looks like the httpserver threadpool died; please make an issue on github and tell me the story of how you pulled that off, thanks and dog bless\n"
+            self.log(self.name, m, 1)
+
+        with self.mutex:
+            self.ncli += 1
 
         thr = threading.Thread(
             target=self.thr_client,
             args=(sck, addr),
-            name="httpsrv-{}-{}".format(addr[0].split(".", 2)[-1][-6:], addr[1]),
+            name="httpconn-{}-{}".format(addr[0].split(".", 2)[-1][-6:], addr[1]),
         )
         thr.daemon = True
         thr.start()
 
-    def num_clients(self):
+    def thr_poolw(self):
-        with self.mutex:
+        while True:
-            return len(self.clients)
+            task = self.tp_q.get()
+            if not task:
+                break
+
+            with self.mutex:
+                self.tp_time = None
+
+            try:
+                sck, addr = task
+                me = threading.current_thread()
+                me.name = "httpconn-{}-{}".format(
+                    addr[0].split(".", 2)[-1][-6:], addr[1]
+                )
+                self.thr_client(sck, addr)
+                me.name = self.name + "-poolw"
+            except:
+                self.log(self.name, "thr_client: " + min_ex(), 3)
 
     def shutdown(self):
-        self.log("ok bye")
+        self.stopping = True
+        for srv in self.srvs:
+            try:
+                srv.close()
+            except:
+                pass
+
+        clients = list(self.clients.keys())
+        for cli in clients:
+            try:
+                cli.shutdown()
+            except:
+                pass
+
+        if self.tp_q:
+            self.stop_threads(self.tp_nthr)
+            for _ in range(10):
+                time.sleep(0.05)
+                if self.tp_q.empty():
+                    break
+
+        self.log(self.name, "ok bye")
 
     def thr_client(self, sck, addr):
         """thread managing one tcp client"""
@@ -90,73 +251,68 @@ class HttpSrv(object):
         with self.mutex:
             self.clients[cli] = 0
 
-            if self.is_mp:
+        fno = sck.fileno()
-                self.workload += 50
-                if not self.workload_thr_alive:
-                    self.workload_thr_alive = True
-                    thr = threading.Thread(
-                        target=self.thr_workload, name="httpsrv-workload"
-                    )
-                    thr.daemon = True
-                    thr.start()
-
         try:
             if self.args.log_conn:
-                self.log("%s %s" % addr, "|%sC-crun" % ("-" * 6,), c="1;30")
+                self.log("%s %s" % addr, "|%sC-crun" % ("-" * 4,), c="1;30")
 
             cli.run()
 
+        except (OSError, socket.error) as ex:
+            if ex.errno not in [10038, 10054, 107, 57, 49, 9]:
+                self.log(
+                    "%s %s" % addr,
+                    "run({}): {}".format(fno, ex),
+                    c=6,
+                )
+
         finally:
             sck = cli.s
             if self.args.log_conn:
-                self.log("%s %s" % addr, "|%sC-cdone" % ("-" * 7,), c="1;30")
+                self.log("%s %s" % addr, "|%sC-cdone" % ("-" * 5,), c="1;30")
 
             try:
+                fno = sck.fileno()
                 sck.shutdown(socket.SHUT_RDWR)
                 sck.close()
             except (OSError, socket.error) as ex:
                 if not MACOS:
                     self.log(
                         "%s %s" % addr,
-                        "shut({}): {}".format(sck.fileno(), ex),
+                        "shut({}): {}".format(fno, ex),
                         c="1;30",
                     )
-                if ex.errno not in [10038, 10054, 107, 57, 9]:
+                if ex.errno not in [10038, 10054, 107, 57, 49, 9]:
                     # 10038 No longer considered a socket
                     # 10054 Foribly closed by remote
                     #   107 Transport endpoint not connected
                     #    57 Socket is not connected
+                    #    49 Can't assign requested address (wifi down)
                     #     9 Bad file descriptor
                     raise
             finally:
                 with self.mutex:
                     del self.clients[cli]
+                    self.ncli -= 1
 
-                if self.disconnect_func:
+    def cachebuster(self):
-                    self.disconnect_func(addr)  # pylint: disable=not-callable
+        if time.time() - self.cb_ts < 1:
+            return self.cb_v
 
-    def thr_workload(self):
+        with self.mutex:
-        """indicates the python interpreter workload caused by this HttpSrv"""
+            if time.time() - self.cb_ts < 1:
-        # avoid locking in extract_filedata by tracking difference here
+                return self.cb_v
-        while True:
-            time.sleep(0.2)
-            with self.mutex:
-                if not self.clients:
-                    # no clients rn, termiante thread
-                    self.workload_thr_alive = False
-                    self.workload = 0
-                    return
 
-            total = 0
+            v = E.t0
-            with self.mutex:
+            try:
-                for cli in self.clients.keys():
+                with os.scandir(os.path.join(E.mod, "web")) as dh:
-                    now = cli.workload
+                    for fh in dh:
-                    delta = now - self.clients[cli]
+                        inf = fh.stat(follow_symlinks=False)
-                    if delta < 0:
+                        v = max(v, inf.st_mtime)
-                        # was reset in HttpCli to prevent overflow
+            except:
-                        delta = now
+                pass
 
-                    total += delta
+            v = base64.urlsafe_b64encode(spack(b">xxL", int(v)))
-                    self.clients[cli] = now
+            self.cb_v = v.decode("ascii")[-4:]
-
+            self.cb_ts = time.time()
-            self.workload = total
+            return self.cb_v

copyparty/mtag.py

@@ -7,15 +7,13 @@ import json
 import shutil
 import subprocess as sp
 
-from .__init__ import PY2, WINDOWS
+from .__init__ import PY2, WINDOWS, unicode
-from .util import fsenc, fsdec, REKOBO_LKEY
+from .util import fsenc, fsdec, uncyg, REKOBO_LKEY
-
-if not PY2:
-    unicode = str
 
 
 def have_ff(cmd):
     if PY2:
+        print("# checking {}".format(cmd))
         cmd = (cmd + " -version").encode("ascii").split(b" ")
         try:
             sp.Popen(cmd, stdout=sp.PIPE, stderr=sp.PIPE).communicate()
@@ -43,6 +41,9 @@ class MParser(object):
         while True:
             try:
                 bp = os.path.expanduser(args)
+                if WINDOWS:
+                    bp = uncyg(bp)
+
                 if os.path.exists(bp):
                     self.bin = bp
                     return
@@ -111,6 +112,19 @@ def parse_ffprobe(txt):
     ret = {}  # processed
     md = {}  # raw tags
 
+    is_audio = fmt.get("format_name") in ["mp3", "ogg", "flac", "wav"]
+    if fmt.get("filename", "").split(".")[-1].lower() in ["m4a", "aac"]:
+        is_audio = True
+
+    # if audio file, ensure audio stream appears first
+    if (
+        is_audio
+        and len(streams) > 2
+        and streams[1].get("codec_type") != "audio"
+        and streams[2].get("codec_type") == "audio"
+    ):
+        streams = [fmt, streams[2], streams[1]] + streams[3:]
+
     have = {}
     for strm in streams:
         typ = strm.get("codec_type")
@@ -130,9 +144,7 @@ def parse_ffprobe(txt):
             ]
 
         if typ == "video":
-            if strm.get("DISPOSITION:attached_pic") == "1" or fmt.get(
+            if strm.get("DISPOSITION:attached_pic") == "1" or is_audio:
-                "format_name"
-            ) in ["mp3", "ogg", "flac"]:
                 continue
 
             kvm = [
@@ -176,7 +188,7 @@ def parse_ffprobe(txt):
 
             k = k[4:].strip()
             v = v.strip()
-            if k and v:
+            if k and v and k not in md:
                 md[k] = [v]
 
     for k in [".q", ".vq", ".aq"]:

copyparty/star.py

@@ -33,10 +33,11 @@ class QFile(object):
 class StreamTar(object):
     """construct in-memory tar file from the given path"""
 
-    def __init__(self, fgen, **kwargs):
+    def __init__(self, log, fgen, **kwargs):
         self.ci = 0
         self.co = 0
         self.qfile = QFile()
+        self.log = log
         self.fgen = fgen
         self.errf = None
 
@@ -91,7 +92,8 @@ class StreamTar(object):
                 errors.append([f["vp"], repr(ex)])
 
         if errors:
-            self.errf = errdesc(errors)
+            self.errf, txt = errdesc(errors)
+            self.log("\n".join(([repr(self.errf)] + txt[1:])))
             self.ser(self.errf)
 
         self.tar.close()

copyparty/sutil.py

@@ -25,4 +25,4 @@ def errdesc(errors):
         "vp": "archive-errors-{}.txt".format(dt),
         "ap": tf_path,
         "st": os.stat(tf_path),
-    }
+    }, report

copyparty/svchub.py

@@ -5,12 +5,13 @@ import re
 import os
 import sys
 import time
+import shlex
 import threading
 from datetime import datetime, timedelta
 import calendar
 
-from .__init__ import PY2, WINDOWS, MACOS, VT100
+from .__init__ import E, PY2, WINDOWS, MACOS, VT100
-from .util import mp
+from .util import mp, start_log_thrs, start_stackmon
 from .authsrv import AuthSrv
 from .tcpsrv import TcpSrv
 from .up2k import Up2k
@@ -28,14 +29,24 @@ class SvcHub(object):
     put() can return a queue (if want_reply=True) which has a blocking get() with the response.
     """
 
-    def __init__(self, args):
+    def __init__(self, args, argv, printed):
         self.args = args
+        self.argv = argv
+        self.logf = None
 
         self.ansi_re = re.compile("\033\\[[^m]*m")
         self.log_mutex = threading.Lock()
         self.next_day = 0
 
         self.log = self._log_disabled if args.q else self._log_enabled
+        if args.lo:
+            self._setup_logfile(printed)
+
+        if args.stackmon:
+            start_stackmon(args.stackmon, 0)
+
+        if args.log_thrs:
+            start_log_thrs(self.log, args.log_thrs, 0)
 
         # initiate all services to manage
         self.asrv = AuthSrv(self.args, self.log, False)
@@ -69,6 +80,52 @@ class SvcHub(object):
 
         self.broker = Broker(self)
 
+    def _logname(self):
+        dt = datetime.utcfromtimestamp(time.time())
+        fn = self.args.lo
+        for fs in "YmdHMS":
+            fs = "%" + fs
+            if fs in fn:
+                fn = fn.replace(fs, dt.strftime(fs))
+
+        return fn
+
+    def _setup_logfile(self, printed):
+        base_fn = fn = sel_fn = self._logname()
+        if fn != self.args.lo:
+            ctr = 0
+            # yup this is a race; if started sufficiently concurrently, two
+            # copyparties can grab the same logfile (considered and ignored)
+            while os.path.exists(sel_fn):
+                ctr += 1
+                sel_fn = "{}.{}".format(fn, ctr)
+
+        fn = sel_fn
+
+        try:
+            import lzma
+
+            lh = lzma.open(fn, "wt", encoding="utf-8", errors="replace", preset=0)
+
+        except:
+            import codecs
+
+            lh = codecs.open(fn, "w", encoding="utf-8", errors="replace")
+
+        lh.base_fn = base_fn
+
+        argv = [sys.executable] + self.argv
+        if hasattr(shlex, "quote"):
+            argv = [shlex.quote(x) for x in argv]
+        else:
+            argv = ['"{}"'.format(x) for x in argv]
+
+        msg = "[+] opened logfile [{}]\n".format(fn)
+        printed += msg
+        lh.write("t0: {:.3f}\nargv: {}\n\n{}".format(E.t0, " ".join(argv), printed))
+        self.logf = lh
+        print(msg, end="")
+
     def run(self):
         thr = threading.Thread(target=self.tcpsrv.run, name="svchub-main")
         thr.daemon = True
@@ -99,9 +156,36 @@ class SvcHub(object):
             print("nailed it", end="")
         finally:
             print("\033[0m")
+            if self.logf:
+                self.logf.close()
 
     def _log_disabled(self, src, msg, c=0):
-        pass
+        if not self.logf:
+            return
+
+        with self.log_mutex:
+            ts = datetime.utcfromtimestamp(time.time())
+            ts = ts.strftime("%Y-%m%d-%H%M%S.%f")[:-3]
+            self.logf.write("@{} [{}] {}\n".format(ts, src, msg))
+
+            now = time.time()
+            if now >= self.next_day:
+                self._set_next_day()
+
+    def _set_next_day(self):
+        if self.next_day and self.logf and self.logf.base_fn != self._logname():
+            self.logf.close()
+            self._setup_logfile("")
+
+        dt = datetime.utcfromtimestamp(time.time())
+
+        # unix timestamp of next 00:00:00 (leap-seconds safe)
+        day_now = dt.day
+        while dt.day == day_now:
+            dt += timedelta(hours=12)
+
+        dt = dt.replace(hour=0, minute=0, second=0)
+        self.next_day = calendar.timegm(dt.utctimetuple())
 
     def _log_enabled(self, src, msg, c=0):
         """handles logging from all components"""
@@ -110,14 +194,7 @@ class SvcHub(object):
             if now >= self.next_day:
                 dt = datetime.utcfromtimestamp(now)
                 print("\033[36m{}\033[0m\n".format(dt.strftime("%Y-%m-%d")), end="")
-
+                self._set_next_day()
-                # unix timestamp of next 00:00:00 (leap-seconds safe)
-                day_now = dt.day
-                while dt.day == day_now:
-                    dt += timedelta(hours=12)
-
-                dt = dt.replace(hour=0, minute=0, second=0)
-                self.next_day = calendar.timegm(dt.utctimetuple())
 
             fmt = "\033[36m{} \033[33m{:21} \033[0m{}\n"
             if not VT100:
@@ -144,20 +221,20 @@ class SvcHub(object):
                 except:
                     print(msg.encode("ascii", "replace").decode(), end="")
 
+            if self.logf:
+                self.logf.write(msg)
+
     def check_mp_support(self):
         vmin = sys.version_info[1]
         if WINDOWS:
             msg = "need python 3.3 or newer for multiprocessing;"
-            if PY2:
+            if PY2 or vmin < 3:
-                # py2 pickler doesn't support winsock
-                return msg
-            elif vmin < 3:
                 return msg
         elif MACOS:
             return "multiprocessing is wonky on mac osx;"
         else:
-            msg = "need python 2.7 or 3.3+ for multiprocessing;"
+            msg = "need python 3.3+ for multiprocessing;"
-            if not PY2 and vmin < 3:
+            if PY2 or vmin < 3:
                 return msg
 
         try:
@@ -189,5 +266,5 @@ class SvcHub(object):
         if not err:
             return True
         else:
-            self.log("root", err)
+            self.log("svchub", err)
             return False

copyparty/szip.py

@@ -4,15 +4,14 @@ from __future__ import print_function, unicode_literals
 import os
 import time
 import zlib
-import struct
 from datetime import datetime
 
 from .sutil import errdesc
-from .util import yieldfile, sanitize_fn
+from .util import yieldfile, sanitize_fn, spack, sunpack
 
 
 def dostime2unix(buf):
-    t, d = struct.unpack("<HH", buf)
+    t, d = sunpack(b"<HH", buf)
 
     ts = (t & 0x1F) * 2
     tm = (t >> 5) & 0x3F
@@ -36,13 +35,13 @@ def unixtime2dos(ts):
 
     bd = ((dy - 1980) << 9) + (dm << 5) + dd
     bt = (th << 11) + (tm << 5) + ts // 2
-    return struct.pack("<HH", bt, bd)
+    return spack(b"<HH", bt, bd)
 
 
 def gen_fdesc(sz, crc32, z64):
     ret = b"\x50\x4b\x07\x08"
-    fmt = "<LQQ" if z64 else "<LLL"
+    fmt = b"<LQQ" if z64 else b"<LLL"
-    ret += struct.pack(fmt, crc32, sz, sz)
+    ret += spack(fmt, crc32, sz, sz)
     return ret
 
 
@@ -66,7 +65,7 @@ def gen_hdr(h_pos, fn, sz, lastmod, utf8, crc32, pre_crc):
     req_ver = b"\x2d\x00" if z64 else b"\x0a\x00"
 
     if crc32:
-        crc32 = struct.pack("<L", crc32)
+        crc32 = spack(b"<L", crc32)
     else:
         crc32 = b"\x00" * 4
 
@@ -87,14 +86,14 @@ def gen_hdr(h_pos, fn, sz, lastmod, utf8, crc32, pre_crc):
     # however infozip does actual sz and it even works on winxp
     # (same reasning for z64 extradata later)
     vsz = 0xFFFFFFFF if z64 else sz
-    ret += struct.pack("<LL", vsz, vsz)
+    ret += spack(b"<LL", vsz, vsz)
 
     # windows support (the "?" replace below too)
-    fn = sanitize_fn(fn, ok="/")
+    fn = sanitize_fn(fn, "/", [])
     bfn = fn.encode("utf-8" if utf8 else "cp437", "replace").replace(b"?", b"_")
 
     z64_len = len(z64v) * 8 + 4 if z64v else 0
-    ret += struct.pack("<HH", len(bfn), z64_len)
+    ret += spack(b"<HH", len(bfn), z64_len)
 
     if h_pos is not None:
         # 2b comment, 2b diskno
@@ -106,12 +105,12 @@ def gen_hdr(h_pos, fn, sz, lastmod, utf8, crc32, pre_crc):
         ret += b"\x01\x00\x00\x00\xa4\x81"
 
         # 4b local-header-ofs
-        ret += struct.pack("<L", min(h_pos, 0xFFFFFFFF))
+        ret += spack(b"<L", min(h_pos, 0xFFFFFFFF))
 
     ret += bfn
 
     if z64v:
-        ret += struct.pack("<HH" + "Q" * len(z64v), 1, len(z64v) * 8, *z64v)
+        ret += spack(b"<HH" + b"Q" * len(z64v), 1, len(z64v) * 8, *z64v)
 
     return ret
 
@@ -136,7 +135,7 @@ def gen_ecdr(items, cdir_pos, cdir_end):
     need_64 = nitems == 0xFFFF or 0xFFFFFFFF in [csz, cpos]
 
     # 2b tnfiles, 2b dnfiles, 4b dir sz, 4b dir pos
-    ret += struct.pack("<HHLL", nitems, nitems, csz, cpos)
+    ret += spack(b"<HHLL", nitems, nitems, csz, cpos)
 
     # 2b comment length
     ret += b"\x00\x00"
@@ -163,7 +162,7 @@ def gen_ecdr64(items, cdir_pos, cdir_end):
 
     # 8b tnfiles, 8b dnfiles, 8b dir sz, 8b dir pos
     cdir_sz = cdir_end - cdir_pos
-    ret += struct.pack("<QQQQ", len(items), len(items), cdir_sz, cdir_pos)
+    ret += spack(b"<QQQQ", len(items), len(items), cdir_sz, cdir_pos)
 
     return ret
 
@@ -178,13 +177,14 @@ def gen_ecdr64_loc(ecdr64_pos):
     ret = b"\x50\x4b\x06\x07"
 
     # 4b cdisk, 8b start of ecdr64, 4b ndisks
-    ret += struct.pack("<LQL", 0, ecdr64_pos, 1)
+    ret += spack(b"<LQL", 0, ecdr64_pos, 1)
 
     return ret
 
 
 class StreamZip(object):
-    def __init__(self, fgen, utf8=False, pre_crc=False):
+    def __init__(self, log, fgen, utf8=False, pre_crc=False):
+        self.log = log
         self.fgen = fgen
         self.utf8 = utf8
         self.pre_crc = pre_crc
@@ -247,8 +247,8 @@ class StreamZip(object):
                 errors.append([f["vp"], repr(ex)])
 
         if errors:
-            errf = errdesc(errors)
+            errf, txt = errdesc(errors)
-            print(repr(errf))
+            self.log("\n".join(([repr(errf)] + txt[1:])))
             for x in self.ser(errf):
                 yield x
 

copyparty/tcpsrv.py

@@ -2,11 +2,9 @@
 from __future__ import print_function, unicode_literals
 
 import re
-import time
 import socket
-import select
 
-from .util import chkcmd, Counter
+from .util import chkcmd
 
 
 class TcpSrv(object):
@@ -20,7 +18,7 @@ class TcpSrv(object):
         self.args = hub.args
         self.log = hub.log
 
-        self.num_clients = Counter()
+        self.stopping = False
 
         ip = "127.0.0.1"
         eps = {ip: "local only"}
@@ -65,36 +63,22 @@ class TcpSrv(object):
         for srv in self.srv:
             srv.listen(self.args.nc)
             ip, port = srv.getsockname()
-            self.log("tcpsrv", "listening @ {0}:{1}".format(ip, port))
+            fno = srv.fileno()
+            msg = "listening @ {}:{}  f{}".format(ip, port, fno)
+            self.log("tcpsrv", msg)
+            if self.args.q:
+                print(msg)
 
-        while True:
+            self.hub.broker.put(False, "listen", srv)
-            if self.args.log_conn:
-                self.log("tcpsrv", "|%sC-ncli" % ("-" * 1,), c="1;30")
-
-            if self.num_clients.v >= self.args.nc:
-                time.sleep(0.1)
-                continue
-
-            if self.args.log_conn:
-                self.log("tcpsrv", "|%sC-acc1" % ("-" * 2,), c="1;30")
-
-            ready, _, _ = select.select(self.srv, [], [])
-            for srv in ready:
-                sck, addr = srv.accept()
-                sip, sport = srv.getsockname()
-                if self.args.log_conn:
-                    self.log(
-                        "%s %s" % addr,
-                        "|{}C-acc2 \033[0;36m{} \033[3{}m{}".format(
-                            "-" * 3, sip, sport % 8, sport
-                        ),
-                        c="1;30",
-                    )
-
-                self.num_clients.add()
-                self.hub.broker.put(False, "httpconn", sck, addr)
 
     def shutdown(self):
+        self.stopping = True
+        try:
+            for srv in self.srv:
+                srv.close()
+        except:
+            pass
+
         self.log("tcpsrv", "ok bye")
 
     def detect_interfaces(self, listen_ips):

copyparty/th_srv.py

@@ -9,15 +9,11 @@ import hashlib
 import threading
 import subprocess as sp
 
-from .__init__ import PY2
+from .__init__ import PY2, unicode
 from .util import fsenc, runcmd, Queue, Cooldown, BytesIO, min_ex
 from .mtag import HAVE_FFMPEG, HAVE_FFPROBE, ffprobe
 
 
-if not PY2:
-    unicode = str
-
-
 HAVE_PIL = False
 HAVE_HEIF = False
 HAVE_AVIF = False
@@ -53,7 +49,7 @@ except:
 # https://pillow.readthedocs.io/en/stable/handbook/image-file-formats.html
 # ffmpeg -formats
 FMT_PIL = "bmp dib gif icns ico jpg jpeg jp2 jpx pcx png pbm pgm ppm pnm sgi tga tif tiff webp xbm dds xpm"
-FMT_FF = "av1 asf avi flv m4v mkv mjpeg mjpg mpg mpeg mpg2 mpeg2 h264 avc h265 hevc mov 3gp mp4 ts mpegts nut ogv ogm rm vob webm wmv"
+FMT_FF = "av1 asf avi flv m4v mkv mjpeg mjpg mpg mpeg mpg2 mpeg2 h264 avc mts h265 hevc mov 3gp mp4 ts mpegts nut ogv ogm rm vob webm wmv"
 
 if HAVE_HEIF:
     FMT_PIL += " heif heifs heic heics"
@@ -84,14 +80,14 @@ def thumb_path(histpath, rem, mtime, fmt):
         fn = rem
 
     if rd:
-        h = hashlib.sha512(fsenc(rd)).digest()[:24]
+        h = hashlib.sha512(fsenc(rd)).digest()
         b64 = base64.urlsafe_b64encode(h).decode("ascii")[:24]
         rd = "{}/{}/".format(b64[:2], b64[2:4]).lower() + b64
     else:
         rd = "top"
 
     # could keep original filenames but this is safer re pathlen
-    h = hashlib.sha512(fsenc(fn)).digest()[:24]
+    h = hashlib.sha512(fsenc(fn)).digest()
     fn = base64.urlsafe_b64encode(h).decode("ascii")[:24]
 
     return "{}/th/{}/{}.{:x}.{}".format(
@@ -134,9 +130,10 @@ class ThumbSrv(object):
             msg += ", ".join(missing)
             self.log(msg, c=3)
 
-        t = threading.Thread(target=self.cleaner, name="thumb-cleaner")
+        if self.args.th_clean:
-        t.daemon = True
+            t = threading.Thread(target=self.cleaner, name="thumb-cleaner")
-        t.start()
+            t.daemon = True
+            t.start()
 
     def log(self, msg, c=0):
         self.log_func("thumb", msg, c)
@@ -154,7 +151,8 @@ class ThumbSrv(object):
         histpath = self.asrv.vfs.histtab[ptop]
         tpath = thumb_path(histpath, rem, mtime, fmt)
         abspath = os.path.join(ptop, rem)
-        cond = threading.Condition()
+        cond = threading.Condition(self.mutex)
+        do_conv = False
         with self.mutex:
             try:
                 self.busy[tpath].append(cond)
@@ -172,8 +170,11 @@ class ThumbSrv(object):
                         f.write(fsenc(os.path.dirname(abspath)))
 
                 self.busy[tpath] = [cond]
-                self.q.put([abspath, tpath])
+                do_conv = True
-                self.log("conv {} \033[0m{}".format(tpath, abspath), c=6)
+
+        if do_conv:
+            self.q.put([abspath, tpath])
+            self.log("conv {} \033[0m{}".format(tpath, abspath), c=6)
 
         while not self.stopping:
             with self.mutex:
@@ -181,7 +182,7 @@ class ThumbSrv(object):
                     break
 
             with cond:
-                cond.wait()
+                cond.wait(3)
 
         try:
             st = os.stat(tpath)
@@ -259,7 +260,7 @@ class ThumbSrv(object):
                 pass  # default q = 75
 
             if im.mode not in fmts:
-                print("conv {}".format(im.mode))
+                # print("conv {}".format(im.mode))
                 im = im.convert("RGB")
 
             im.save(tpath, quality=40, method=6)
@@ -335,29 +336,32 @@ class ThumbSrv(object):
         interval = self.args.th_clean
         while True:
             time.sleep(interval)
+            ndirs = 0
             for vol, histpath in self.asrv.vfs.histtab.items():
                 if histpath.startswith(vol):
                     self.log("\033[Jcln {}/\033[A".format(histpath))
                 else:
                     self.log("\033[Jcln {} ({})/\033[A".format(histpath, vol))
 
-                self.clean(histpath)
+                ndirs += self.clean(histpath)
 
-            self.log("\033[Jcln ok")
+            self.log("\033[Jcln ok; rm {} dirs".format(ndirs))
 
     def clean(self, histpath):
-        # self.log("cln {}".format(histpath))
+        thumbpath = os.path.join(histpath, "th")
+        # self.log("cln {}".format(thumbpath))
         maxage = self.args.th_maxage
         now = time.time()
         prev_b64 = None
         prev_fp = None
         try:
-            ents = os.listdir(histpath)
+            ents = os.listdir(thumbpath)
         except:
-            return
+            return 0
 
+        ndirs = 0
         for f in sorted(ents):
-            fp = os.path.join(histpath, f)
+            fp = os.path.join(thumbpath, f)
             cmp = fp.lower().replace("\\", "/")
 
             # "top" or b64 prefix/full (a folder)
@@ -372,10 +376,11 @@ class ThumbSrv(object):
                                 break
 
                         if safe:
+                            ndirs += 1
                             self.log("rm -rf [{}]".format(fp))
                             shutil.rmtree(fp, ignore_errors=True)
                 else:
-                    self.clean(fp)
+                    ndirs += self.clean(fp)
                 continue
 
             # thumb file
@@ -397,3 +402,5 @@ class ThumbSrv(object):
 
             prev_b64 = b64
             prev_fp = fp
+
+        return ndirs

copyparty/u2idx.py

@@ -26,7 +26,7 @@ class U2idx(object):
         self.timeout = self.args.srch_time
 
         if not HAVE_SQLITE3:
-            self.log("could not load sqlite3; searchign wqill be disabled")
+            self.log("your python does not have sqlite3; searching will be disabled")
             return
 
         self.cur = {}
@@ -57,6 +57,9 @@ class U2idx(object):
             raise Pebkac(500, min_ex())
 
     def get_cur(self, ptop):
+        if not HAVE_SQLITE3:
+            return None
+
         cur = self.cur.get(ptop)
         if cur:
             return cur
@@ -66,7 +69,7 @@ class U2idx(object):
         if not os.path.exists(db_path):
             return None
 
-        cur = sqlite3.connect(db_path).cursor()
+        cur = sqlite3.connect(db_path, 2).cursor()
         self.cur[ptop] = cur
         return cur
 

copyparty/up2k.py

@@ -16,7 +16,7 @@ import traceback
 import subprocess as sp
 from copy import deepcopy
 
-from .__init__ import WINDOWS, ANYWIN
+from .__init__ import WINDOWS, ANYWIN, PY2
 from .util import (
     Pebkac,
     Queue,
@@ -30,6 +30,7 @@ from .util import (
     s3dec,
     statdir,
     s2hms,
+    min_ex,
 )
 from .mtag import MTag, MParser
 
@@ -39,6 +40,8 @@ try:
 except:
     HAVE_SQLITE3 = False
 
+DB_VER = 4
+
 
 class Up2k(object):
     """
@@ -91,7 +94,7 @@ class Up2k(object):
             thr.start()
 
         # static
-        self.r_hash = re.compile("^[0-9a-zA-Z_-]{43}$")
+        self.r_hash = re.compile("^[0-9a-zA-Z_-]{44}$")
 
         if not HAVE_SQLITE3:
             self.log("could not initialize sqlite3, will use in-memory registry only")
@@ -100,13 +103,15 @@ class Up2k(object):
             self.deferred_init()
         else:
             t = threading.Thread(
-                target=self.deferred_init,
+                target=self.deferred_init, name="up2k-deferred-init", args=(0.5,)
-                name="up2k-deferred-init",
             )
             t.daemon = True
             t.start()
 
-    def deferred_init(self):
+    def deferred_init(self, wait=0):
+        if wait:
+            time.sleep(wait)
+
         all_vols = self.asrv.vfs.all_vols
         have_e2d = self.init_indexes(all_vols)
 
@@ -134,7 +139,7 @@ class Up2k(object):
     def get_state(self):
         mtpq = 0
         q = "select count(w) from mt where k = 't:mtp'"
-        got_lock = self.mutex.acquire(timeout=0.5)
+        got_lock = False if PY2 else self.mutex.acquire(timeout=0.5)
         if got_lock:
             for cur in self.cur.values():
                 try:
@@ -190,7 +195,7 @@ class Up2k(object):
 
         return True, ret
 
-    def init_indexes(self, all_vols, scan_vols=[]):
+    def init_indexes(self, all_vols, scan_vols=None):
         self.pp = ProgressPrinter()
         vols = all_vols.values()
         t0 = time.time()
@@ -339,7 +344,15 @@ class Up2k(object):
             for k, v in flags.items()
         ]
         if a:
-            self.log(" ".join(sorted(a)) + "\033[0m")
+            vpath = "?"
+            for k, v in self.asrv.vfs.all_vols.items():
+                if v.realpath == ptop:
+                    vpath = k
+
+            if vpath:
+                vpath += "/"
+
+            self.log("/{} {}".format(vpath, " ".join(sorted(a))), "35")
 
         reg = {}
         path = os.path.join(histpath, "up2k.snap")
@@ -398,7 +411,7 @@ class Up2k(object):
             if WINDOWS:
                 excl = [x.replace("/", "\\") for x in excl]
 
-            n_add = self._build_dir(dbw, top, set(excl), top, nohash)
+            n_add = self._build_dir(dbw, top, set(excl), top, nohash, [])
             n_rm = self._drop_lost(dbw[0], top)
             if dbw[1]:
                 self.log("commit {} new files".format(dbw[1]))
@@ -406,11 +419,25 @@ class Up2k(object):
 
             return True, n_add or n_rm or do_vac
 
-    def _build_dir(self, dbw, top, excl, cdir, nohash):
+    def _build_dir(self, dbw, top, excl, cdir, nohash, seen):
+        rcdir = cdir
+        if not ANYWIN:
+            try:
+                # a bit expensive but worth
+                rcdir = os.path.realpath(cdir)
+            except:
+                pass
+
+        if rcdir in seen:
+            m = "bailing from symlink loop,\n  prev: {}\n  curr: {}\n  from: {}"
+            self.log(m.format(seen[-1], rcdir, cdir), 3)
+            return 0
+
+        seen = seen + [cdir]
         self.pp.msg = "a{} {}".format(self.pp.n, cdir)
         histpath = self.asrv.vfs.histtab[top]
         ret = 0
-        g = statdir(self.log, not self.args.no_scandir, False, cdir)
+        g = statdir(self.log_func, not self.args.no_scandir, False, cdir)
         for iname, inf in sorted(g):
             abspath = os.path.join(cdir, iname)
             lmod = int(inf.st_mtime)
@@ -419,10 +446,13 @@ class Up2k(object):
                 if abspath in excl or abspath == histpath:
                     continue
                 # self.log(" dir: {}".format(abspath))
-                ret += self._build_dir(dbw, top, excl, abspath, nohash)
+                ret += self._build_dir(dbw, top, excl, abspath, nohash, seen)
             else:
                 # self.log("file: {}".format(abspath))
-                rp = abspath[len(top) :].replace("\\", "/").strip("/")
+                rp = abspath[len(top) + 1 :]
+                if WINDOWS:
+                    rp = rp.replace("\\", "/").strip("/")
+
                 rd, fn = rp.rsplit("/", 1) if "/" in rp else ["", rp]
                 sql = "select w, mt, sz from up where rd = ? and fn = ?"
                 try:
@@ -647,7 +677,7 @@ class Up2k(object):
             try:
                 parser = MParser(parser)
             except:
-                self.log("invalid argument: " + parser, 1)
+                self.log("invalid argument (could not find program): " + parser, 1)
                 return
 
             for tag in entags:
@@ -887,59 +917,31 @@ class Up2k(object):
         if not existed and ver is None:
             return self._create_db(db_path, cur)
 
-        orig_ver = ver
+        if ver == DB_VER:
-        if not ver or ver < 3:
-            bak = "{}.bak.{:x}.v{}".format(db_path, int(time.time()), ver)
-            db = cur.connection
-            cur.close()
-            db.close()
-            msg = "creating new DB (old is bad); backup: {}"
-            if ver:
-                msg = "creating backup before upgrade: {}"
-
-            self.log(msg.format(bak))
-            shutil.copy2(db_path, bak)
-            cur = self._orz(db_path)
-
-        if ver == 1:
-            cur = self._upgrade_v1(cur, db_path)
-            if cur:
-                ver = 2
-
-        if ver == 2:
-            cur = self._create_v3(cur)
-            ver = self._read_ver(cur) if cur else None
-
-        if ver == 3:
-            if orig_ver != ver:
-                cur.connection.commit()
-                cur.execute("vacuum")
-                cur.connection.commit()
-
             try:
                 nfiles = next(cur.execute("select count(w) from up"))[0]
                 self.log("OK: {} |{}|".format(db_path, nfiles))
                 return cur
-            except Exception as ex:
+            except:
-                self.log("WARN: could not list files, DB corrupt?\n  " + repr(ex))
+                self.log("WARN: could not list files; DB corrupt?\n" + min_ex())
 
-        if cur:
+        if (ver or 0) > DB_VER:
-            db = cur.connection
+            m = "database is version {}, this copyparty only supports versions <= {}"
-            cur.close()
+            raise Exception(m.format(ver, DB_VER))
-            db.close()
+
+        bak = "{}.bak.{:x}.v{}".format(db_path, int(time.time()), ver)
+        db = cur.connection
+        cur.close()
+        db.close()
+        msg = "creating new DB (old is bad); backup: {}"
+        if ver:
+            msg = "creating new DB (too old to upgrade); backup: {}"
+
+        self.log(msg.format(bak))
+        os.rename(fsenc(db_path), fsenc(bak))
 
         return self._create_db(db_path, None)
 
-    def _create_db(self, db_path, cur):
-        if not cur:
-            cur = self._orz(db_path)
-
-        self._create_v2(cur)
-        self._create_v3(cur)
-        cur.connection.commit()
-        self.log("created DB at {}".format(db_path))
-        return cur
-
     def _read_ver(self, cur):
         for tab in ["ki", "kv"]:
             try:
@@ -951,78 +953,51 @@ class Up2k(object):
             if rows:
                 return int(rows[0][0])
 
-    def _create_v2(self, cur):
+    def _create_db(self, db_path, cur):
-        for cmd in [
-            r"create table up (w text, mt int, sz int, rd text, fn text)",
-            r"create index up_rd on up(rd)",
-            r"create index up_fn on up(fn)",
-        ]:
-            cur.execute(cmd)
-        return cur
-
-    def _create_v3(self, cur):
         """
         collision in 2^(n/2) files where n = bits (6 bits/ch)
           10*6/2 = 2^30 =       1'073'741'824, 24.1mb idx  1<<(3*10)
           12*6/2 = 2^36 =      68'719'476'736, 24.8mb idx
           16*6/2 = 2^48 = 281'474'976'710'656, 26.1mb idx
         """
-        for c, ks in [["drop table k", "isv"], ["drop index up_", "w"]]:
+        if not cur:
-            for k in ks:
+            cur = self._orz(db_path)
-                try:
-                    cur.execute(c + k)
-                except:
-                    pass
 
         idx = r"create index up_w on up(substr(w,1,16))"
         if self.no_expr_idx:
             idx = r"create index up_w on up(w)"
 
         for cmd in [
+            r"create table up (w text, mt int, sz int, rd text, fn text)",
+            r"create index up_rd on up(rd)",
+            r"create index up_fn on up(fn)",
             idx,
             r"create table mt (w text, k text, v int)",
             r"create index mt_w on mt(w)",
             r"create index mt_k on mt(k)",
             r"create index mt_v on mt(v)",
             r"create table kv (k text, v int)",
-            r"insert into kv values ('sver', 3)",
+            r"insert into kv values ('sver', {})".format(DB_VER),
         ]:
             cur.execute(cmd)
+
+        cur.connection.commit()
+        self.log("created DB at {}".format(db_path))
         return cur
 
-    def _upgrade_v1(self, odb, db_path):
-        npath = db_path + ".next"
-        if os.path.exists(npath):
-            os.unlink(npath)
-
-        ndb = self._orz(npath)
-        self._create_v2(ndb)
-
-        c = odb.execute("select * from up")
-        for wark, ts, sz, rp in c:
-            rd, fn = rp.rsplit("/", 1) if "/" in rp else ["", rp]
-            v = (wark, ts, sz, rd, fn)
-            ndb.execute("insert into up values (?,?,?,?,?)", v)
-
-        ndb.connection.commit()
-        ndb.connection.close()
-        odb.connection.close()
-        atomic_move(npath, db_path)
-        return self._orz(db_path)
-
     def handle_json(self, cj):
         with self.mutex:
             if not self.register_vpath(cj["ptop"], cj["vcfg"]):
                 if cj["ptop"] not in self.registry:
                     raise Pebkac(410, "location unavailable")
 
-        cj["name"] = sanitize_fn(cj["name"], bad=[".prologue.html", ".epilogue.html"])
+        cj["name"] = sanitize_fn(cj["name"], "", [".prologue.html", ".epilogue.html"])
         cj["poke"] = time.time()
         wark = self._get_wark(cj)
         now = time.time()
         job = None
         with self.mutex:
-            cur = self.cur.get(cj["ptop"], None)
+            cur = self.cur.get(cj["ptop"])
             reg = self.registry[cj["ptop"]]
             if cur:
                 if self.no_expr_idx:
@@ -1068,7 +1043,8 @@ class Up2k(object):
                                 break
                         except:
                             # missing; restart
-                            job = None
+                            if not self.args.nw:
+                                job = None
                             break
                 else:
                     # file contents match, but not the path
@@ -1095,8 +1071,9 @@ class Up2k(object):
                         pdir = os.path.join(cj["ptop"], cj["prel"])
                         job["name"] = self._untaken(pdir, cj["name"], now, cj["addr"])
                         dst = os.path.join(job["ptop"], job["prel"], job["name"])
-                        os.unlink(fsenc(dst))  # TODO ed pls
+                        if not self.args.nw:
-                        self._symlink(src, dst)
+                            os.unlink(fsenc(dst))  # TODO ed pls
+                            self._symlink(src, dst)
 
             if not job:
                 job = {
@@ -1138,6 +1115,9 @@ class Up2k(object):
             }
 
     def _untaken(self, fdir, fname, ts, ip):
+        if self.args.nw:
+            return fname
+
         # TODO broker which avoid this race and
         # provides a new filename if taken (same as bup)
         suffix = ".{:.6f}-{}".format(ts, ip)
@@ -1147,6 +1127,9 @@ class Up2k(object):
     def _symlink(self, src, dst):
         # TODO store this in linktab so we never delete src if there are links to it
         self.log("linking dupe:\n  {0}\n  {1}".format(src, dst))
+        if self.args.nw:
+            return
+
         try:
             lsrc = src
             ldst = dst
@@ -1180,7 +1163,7 @@ class Up2k(object):
 
     def handle_chunk(self, ptop, wark, chash):
         with self.mutex:
-            job = self.registry[ptop].get(wark, None)
+            job = self.registry[ptop].get(wark)
             if not job:
                 known = " ".join([x for x in self.registry[ptop].keys()])
                 self.log("unknown wark [{}], known: {}".format(wark, known))
@@ -1224,6 +1207,10 @@ class Up2k(object):
             if ret > 0:
                 return ret, src
 
+            if self.args.nw:
+                # del self.registry[ptop][wark]
+                return ret, dst
+
             atomic_move(src, dst)
 
             if ANYWIN:
@@ -1245,7 +1232,7 @@ class Up2k(object):
         return ret, dst
 
     def idx_wark(self, ptop, wark, rd, fn, lmod, sz):
-        cur = self.cur.get(ptop, None)
+        cur = self.cur.get(ptop)
         if not cur:
             return False
 
@@ -1316,9 +1303,9 @@ class Up2k(object):
                     hashobj.update(buf)
                     rem -= len(buf)
 
-                digest = hashobj.digest()[:32]
+                digest = hashobj.digest()[:33]
                 digest = base64.urlsafe_b64encode(digest)
-                ret.append(digest.decode("utf-8").rstrip("="))
+                ret.append(digest.decode("utf-8"))
 
         return ret
 
@@ -1333,6 +1320,10 @@ class Up2k(object):
         if self.args.dotpart:
             tnam = "." + tnam
 
+        if self.args.nw:
+            job["tnam"] = tnam
+            return
+
         suffix = ".{:.6f}-{}".format(job["t0"], job["addr"])
         with ren_open(tnam, "wb", fdir=pdir, suffix=suffix) as f:
             f, job["tnam"] = f["orz"]
@@ -1414,7 +1405,7 @@ class Up2k(object):
 
         newest = max(x["poke"] for _, x in reg.items()) if reg else 0
         etag = [len(reg), newest]
-        if etag == prev.get(ptop, None):
+        if etag == prev.get(ptop):
             return
 
         try:
@@ -1518,12 +1509,12 @@ def up2k_wark_from_hashlist(salt, filesize, hashes):
     ident.extend(hashes)
     ident = "\n".join(ident)
 
-    wark = hashlib.sha512(ident.encode("utf-8")).digest()
+    wark = hashlib.sha512(ident.encode("utf-8")).digest()[:33]
     wark = base64.urlsafe_b64encode(wark)
-    return wark.decode("ascii")[:43]
+    return wark.decode("ascii")
 
 
 def up2k_wark_from_metadata(salt, sz, lastmod, rd, fn):
     ret = fsenc("{}\n{}\n{}\n{}\n{}".format(salt, lastmod, sz, rd, fn))
     ret = base64.urlsafe_b64encode(hashlib.sha512(ret).digest())
-    return "#{}".format(ret[:42].decode("ascii"))
+    return "#{}".format(ret.decode("ascii"))[:44]

copyparty/util.py

@@ -16,6 +16,7 @@ import mimetypes
 import contextlib
 import subprocess as sp  # nosec
 from datetime import datetime
+from collections import Counter
 
 from .__init__ import PY2, WINDOWS, ANYWIN
 from .stolen import surrogateescape
@@ -42,6 +43,20 @@ else:
     from Queue import Queue  # pylint: disable=import-error,no-name-in-module
     from StringIO import StringIO as BytesIO
 
+
+try:
+    struct.unpack(b">i", b"idgi")
+    spack = struct.pack
+    sunpack = struct.unpack
+except:
+
+    def spack(f, *a, **ka):
+        return struct.pack(f.decode("ascii"), *a, **ka)
+
+    def sunpack(f, *a, **ka):
+        return struct.unpack(f.decode("ascii"), *a, **ka)
+
+
 surrogateescape.register_surrogateescape()
 FS_ENCODING = sys.getfilesystemencoding()
 if WINDOWS and PY2:
@@ -123,20 +138,6 @@ REKOBO_KEY = {
 REKOBO_LKEY = {k.lower(): v for k, v in REKOBO_KEY.items()}
 
 
-class Counter(object):
-    def __init__(self, v=0):
-        self.v = v
-        self.mutex = threading.Lock()
-
-    def add(self, delta=1):
-        with self.mutex:
-            self.v += delta
-
-    def set(self, absval):
-        with self.mutex:
-            self.v = absval
-
-
 class Cooldown(object):
     def __init__(self, maxage):
         self.maxage = maxage
@@ -231,7 +232,7 @@ def nuprint(msg):
 
 def rice_tid():
     tid = threading.current_thread().ident
-    c = struct.unpack(b"B" * 5, struct.pack(b">Q", tid)[-5:])
+    c = sunpack(b"B" * 5, spack(b">Q", tid)[-5:])
     return "".join("\033[1;37;48;5;{}m{:02x}".format(x, x) for x in c) + "\033[0m"
 
 
@@ -254,15 +255,97 @@ def trace(*args, **kwargs):
     nuprint(msg)
 
 
+def alltrace():
+    threads = {}
+    names = dict([(t.ident, t.name) for t in threading.enumerate()])
+    for tid, stack in sys._current_frames().items():
+        name = "{} ({:x})".format(names.get(tid), tid)
+        threads[name] = stack
+
+    rret = []
+    bret = []
+    for name, stack in sorted(threads.items()):
+        ret = ["\n\n# {}".format(name)]
+        pad = None
+        for fn, lno, name, line in traceback.extract_stack(stack):
+            fn = os.sep.join(fn.split(os.sep)[-3:])
+            ret.append('File: "{}", line {}, in {}'.format(fn, lno, name))
+            if line:
+                ret.append("  " + str(line.strip()))
+                if "self.not_empty.wait()" in line:
+                    pad = " " * 4
+
+        if pad:
+            bret += [ret[0]] + [pad + x for x in ret[1:]]
+        else:
+            rret += ret
+
+    return "\n".join(rret + bret)
+
+
+def start_stackmon(arg_str, nid):
+    suffix = "-{}".format(nid) if nid else ""
+    fp, f = arg_str.rsplit(",", 1)
+    f = int(f)
+    t = threading.Thread(
+        target=stackmon,
+        args=(fp, f, suffix),
+        name="stackmon" + suffix,
+    )
+    t.daemon = True
+    t.start()
+
+
+def stackmon(fp, ival, suffix):
+    ctr = 0
+    while True:
+        ctr += 1
+        time.sleep(ival)
+        st = "{}, {}\n{}".format(ctr, time.time(), alltrace())
+        with open(fp + suffix, "wb") as f:
+            f.write(st.encode("utf-8", "replace"))
+
+
+def start_log_thrs(logger, ival, nid):
+    ival = int(ival)
+    tname = lname = "log-thrs"
+    if nid:
+        tname = "logthr-n{}-i{:x}".format(nid, os.getpid())
+        lname = tname[3:]
+
+    t = threading.Thread(
+        target=log_thrs,
+        args=(logger, ival, lname),
+        name=tname,
+    )
+    t.daemon = True
+    t.start()
+
+
+def log_thrs(log, ival, name):
+    while True:
+        time.sleep(ival)
+        tv = [x.name for x in threading.enumerate()]
+        tv = [
+            x.split("-")[0]
+            if x.startswith("httpconn-") or x.startswith("thumb-")
+            else "listen"
+            if "-listen-" in x
+            else x
+            for x in tv
+            if not x.startswith("pydevd.")
+        ]
+        tv = ["{}\033[36m{}".format(v, k) for k, v in sorted(Counter(tv).items())]
+        log(name, "\033[0m \033[33m".join(tv), 3)
+
+
 def min_ex():
     et, ev, tb = sys.exc_info()
-    tb = traceback.extract_tb(tb, 2)
+    tb = traceback.extract_tb(tb)
-    ex = [
+    fmt = "{} @ {} <{}>: {}"
-        "{} @ {} <{}>: {}".format(fp.split(os.sep)[-1], ln, fun, txt)
+    ex = [fmt.format(fp.split(os.sep)[-1], ln, fun, txt) for fp, ln, fun, txt in tb]
-        for fp, ln, fun, txt in tb
+    ex.append("[{}] {}".format(et.__name__, ev))
-    ]
+    return "\n".join(ex[-8:])
-    ex.append("{}: {}".format(et.__name__, ev))
-    return "\n".join(ex)
 
 
 @contextlib.contextmanager
@@ -323,7 +406,7 @@ def ren_open(fname, *args, **kwargs):
         if not b64:
             b64 = (bname + ext).encode("utf-8", "replace")
             b64 = hashlib.sha512(b64).digest()[:12]
-            b64 = base64.urlsafe_b64encode(b64).decode("utf-8").rstrip("=")
+            b64 = base64.urlsafe_b64encode(b64).decode("utf-8")
 
         badlen = len(fname)
         while len(fname) >= badlen:
@@ -620,6 +703,16 @@ def s2hms(s, optional_h=False):
     return "{}:{:02}:{:02}".format(h, m, s)
 
 
+def uncyg(path):
+    if len(path) < 2 or not path.startswith("/"):
+        return path
+
+    if len(path) > 2 and path[2] != "/":
+        return path
+
+    return "{}:\\{}".format(path[1], path[3:])
+
+
 def undot(path):
     ret = []
     for node in path.split("/"):
@@ -636,7 +729,7 @@ def undot(path):
     return "/".join(ret)
 
 
-def sanitize_fn(fn, ok="", bad=[]):
+def sanitize_fn(fn, ok, bad):
     if "/" not in ok:
         fn = fn.replace("\\", "/").split("/")[-1]
 
@@ -866,35 +959,24 @@ def yieldfile(fn):
             yield buf
 
 
-def hashcopy(actor, fin, fout):
+def hashcopy(fin, fout):
-    is_mp = actor.is_mp
     hashobj = hashlib.sha512()
     tlen = 0
     for buf in fin:
-        if is_mp:
-            actor.workload += 1
-            if actor.workload > 2 ** 31:
-                actor.workload = 100
-
         tlen += len(buf)
         hashobj.update(buf)
         fout.write(buf)
 
-    digest32 = hashobj.digest()[:32]
+    digest = hashobj.digest()[:33]
-    digest_b64 = base64.urlsafe_b64encode(digest32).decode("utf-8").rstrip("=")
+    digest_b64 = base64.urlsafe_b64encode(digest).decode("utf-8")
 
     return tlen, hashobj.hexdigest(), digest_b64
 
 
-def sendfile_py(lower, upper, f, s, actor=None):
+def sendfile_py(lower, upper, f, s):
     remains = upper - lower
     f.seek(lower)
     while remains > 0:
-        if actor:
-            actor.workload += 1
-            if actor.workload > 2 ** 31:
-                actor.workload = 100
-
         # time.sleep(0.01)
         buf = f.read(min(1024 * 32, remains))
         if not buf:
@@ -941,8 +1023,7 @@ def statdir(logger, scandir, lstat, top):
                     try:
                         yield [fsdec(fh.name), fh.stat(follow_symlinks=not lstat)]
                     except Exception as ex:
-                        msg = "scan-stat: \033[36m{} @ {}"
+                        logger(src, "[s] {} @ {}".format(repr(ex), fsdec(fh.path)), 6)
-                        logger(msg.format(repr(ex), fsdec(fh.path)))
         else:
             src = "listdir"
             fun = os.lstat if lstat else os.stat
@@ -951,11 +1032,10 @@ def statdir(logger, scandir, lstat, top):
                 try:
                     yield [fsdec(name), fun(abspath)]
                 except Exception as ex:
-                    msg = "list-stat: \033[36m{} @ {}"
+                    logger(src, "[s] {} @ {}".format(repr(ex), fsdec(abspath)), 6)
-                    logger(msg.format(repr(ex), fsdec(abspath)))
 
     except Exception as ex:
-        logger("{}: \033[31m{} @ {}".format(src, repr(ex), top))
+        logger(src, "{} @ {}".format(repr(ex), top), 1)
 
 
 def unescape_cookie(orig):
@@ -992,7 +1072,13 @@ def guess_mime(url, fallback="application/octet-stream"):
     except:
         return fallback
 
-    return MIMES.get(ext) or mimetypes.guess_type(url)[0] or fallback
+    ret = MIMES.get(ext) or mimetypes.guess_type(url)[0] or fallback
+
+    if ";" not in ret:
+        if ret.startswith("text/") or ret.endswith("/javascript"):
+            ret += "; charset=UTF-8"
+
+    return ret
 
 
 def runcmd(*argv):
@@ -1026,10 +1112,7 @@ def gzip_orig_sz(fn):
     with open(fsenc(fn), "rb") as f:
         f.seek(-4, 2)
         rv = f.read(4)
-        try:
+        return sunpack(b"I", rv)[0]
-            return struct.unpack(b"I", rv)[0]
-        except:
-            return struct.unpack("I", rv)[0]
 
 
 def py_desc():

copyparty/web/baguettebox.js

@@ -0,0 +1,642 @@
+/*!
+ * baguetteBox.js
+ * @author  feimosi
+ * @version 1.11.1-mod
+ * @url https://github.com/feimosi/baguetteBox.js
+ */
+
+window.baguetteBox = (function () {
+    'use strict';
+
+    var options = {},
+        defaults = {
+            captions: true,
+            buttons: 'auto',
+            noScrollbars: false,
+            bodyClass: 'baguetteBox-open',
+            titleTag: false,
+            async: false,
+            preload: 2,
+            animation: 'slideIn',
+            afterShow: null,
+            afterHide: null,
+            onChange: null,
+        },
+        overlay, slider, previousButton, nextButton, closeButton,
+        currentGallery = [],
+        currentIndex = 0,
+        isOverlayVisible = false,
+        touch = {},  // start-pos
+        touchFlag = false,  // busy
+        re_i = /.+\.(gif|jpe?g|png|webp)(\?|$)/i,
+        re_v = /.+\.(webm|mp4)(\?|$)/i,
+        data = {},  // all galleries
+        imagesElements = [],
+        documentLastFocus = null,
+        isFullscreen = false;
+
+    var onFSC = function (e) {
+        isFullscreen = !!document.fullscreenElement;
+    };
+
+    var overlayClickHandler = function (event) {
+        if (event.target.id.indexOf('baguette-img') !== -1) {
+            hideOverlay();
+        }
+    };
+
+    var touchstartHandler = function (event) {
+        touch.count++;
+        if (touch.count > 1) {
+            touch.multitouch = true;
+        }
+        touch.startX = event.changedTouches[0].pageX;
+        touch.startY = event.changedTouches[0].pageY;
+    };
+    var touchmoveHandler = function (event) {
+        if (touchFlag || touch.multitouch) {
+            return;
+        }
+        event.preventDefault ? event.preventDefault() : event.returnValue = false;
+        var touchEvent = event.touches[0] || event.changedTouches[0];
+        if (touchEvent.pageX - touch.startX > 40) {
+            touchFlag = true;
+            showPreviousImage();
+        } else if (touchEvent.pageX - touch.startX < -40) {
+            touchFlag = true;
+            showNextImage();
+        } else if (touch.startY - touchEvent.pageY > 100) {
+            hideOverlay();
+        }
+    };
+    var touchendHandler = function () {
+        touch.count--;
+        if (touch.count <= 0) {
+            touch.multitouch = false;
+        }
+        touchFlag = false;
+    };
+    var contextmenuHandler = function () {
+        touchendHandler();
+    };
+
+    var trapFocusInsideOverlay = function (event) {
+        if (overlay.style.display === 'block' && (overlay.contains && !overlay.contains(event.target))) {
+            event.stopPropagation();
+            initFocus();
+        }
+    };
+
+    function run(selector, userOptions) {
+        buildOverlay();
+        removeFromCache(selector);
+        return bindImageClickListeners(selector, userOptions);
+    }
+
+    function bindImageClickListeners(selector, userOptions) {
+        var galleryNodeList = document.querySelectorAll(selector);
+        var selectorData = {
+            galleries: [],
+            nodeList: galleryNodeList
+        };
+        data[selector] = selectorData;
+
+        [].forEach.call(galleryNodeList, function (galleryElement) {
+            var tagsNodeList = [];
+            if (galleryElement.tagName === 'A') {
+                tagsNodeList = [galleryElement];
+            } else {
+                tagsNodeList = galleryElement.getElementsByTagName('a');
+            }
+
+            tagsNodeList = [].filter.call(tagsNodeList, function (element) {
+                if (element.className.indexOf(userOptions && userOptions.ignoreClass) === -1) {
+                    return re_i.test(element.href) || re_v.test(element.href);
+                }
+            });
+            if (tagsNodeList.length === 0) {
+                return;
+            }
+
+            var gallery = [];
+            [].forEach.call(tagsNodeList, function (imageElement, imageIndex) {
+                var imageElementClickHandler = function (event) {
+                    if (event && (event.ctrlKey || event.metaKey))
+                        return true;
+
+                    event.preventDefault ? event.preventDefault() : event.returnValue = false;
+                    prepareOverlay(gallery, userOptions);
+                    showOverlay(imageIndex);
+                };
+                var imageItem = {
+                    eventHandler: imageElementClickHandler,
+                    imageElement: imageElement
+                };
+                bind(imageElement, 'click', imageElementClickHandler);
+                gallery.push(imageItem);
+            });
+            selectorData.galleries.push(gallery);
+        });
+
+        return selectorData.galleries;
+    }
+
+    function clearCachedData() {
+        for (var selector in data) {
+            if (data.hasOwnProperty(selector)) {
+                removeFromCache(selector);
+            }
+        }
+    }
+
+    function removeFromCache(selector) {
+        if (!data.hasOwnProperty(selector)) {
+            return;
+        }
+        var galleries = data[selector].galleries;
+        [].forEach.call(galleries, function (gallery) {
+            [].forEach.call(gallery, function (imageItem) {
+                unbind(imageItem.imageElement, 'click', imageItem.eventHandler);
+            });
+
+            if (currentGallery === gallery) {
+                currentGallery = [];
+            }
+        });
+
+        delete data[selector];
+    }
+
+    function buildOverlay() {
+        overlay = ebi('baguetteBox-overlay');
+        if (overlay) {
+            slider = ebi('baguetteBox-slider');
+            previousButton = ebi('previous-button');
+            nextButton = ebi('next-button');
+            closeButton = ebi('close-button');
+            return;
+        }
+        overlay = mknod('div');
+        overlay.setAttribute('role', 'dialog');
+        overlay.id = 'baguetteBox-overlay';
+        document.getElementsByTagName('body')[0].appendChild(overlay);
+
+        slider = mknod('div');
+        slider.id = 'baguetteBox-slider';
+        overlay.appendChild(slider);
+
+        previousButton = mknod('button');
+        previousButton.setAttribute('type', 'button');
+        previousButton.id = 'previous-button';
+        previousButton.setAttribute('aria-label', 'Previous');
+        previousButton.innerHTML = '&lt;';
+        overlay.appendChild(previousButton);
+
+        nextButton = mknod('button');
+        nextButton.setAttribute('type', 'button');
+        nextButton.id = 'next-button';
+        nextButton.setAttribute('aria-label', 'Next');
+        nextButton.innerHTML = '&gt;';
+        overlay.appendChild(nextButton);
+
+        closeButton = mknod('button');
+        closeButton.setAttribute('type', 'button');
+        closeButton.id = 'close-button';
+        closeButton.setAttribute('aria-label', 'Close');
+        closeButton.innerHTML = '&times;';
+        overlay.appendChild(closeButton);
+
+        previousButton.className = nextButton.className = closeButton.className = 'baguetteBox-button';
+
+        bindEvents();
+    }
+
+    function keyDownHandler(e) {
+        if (e.ctrlKey || e.altKey || e.metaKey || e.isComposing)
+            return;
+
+        var k = e.code + '';
+
+        if (k == "ArrowLeft" || k == "KeyJ")
+            showPreviousImage();
+        else if (k == "ArrowRight" || k == "KeyL")
+            showNextImage();
+        else if (k == "Escape")
+            hideOverlay();
+        else if (k == "Home")
+            showFirstImage(e);
+        else if (k == "End")
+            showLastImage(e);
+        else if (k == "Space" || k == "KeyP" || k == "KeyK")
+            playpause();
+        else if (k == "KeyU" || k == "KeyO")
+            relseek(k == "KeyU" ? -10 : 10);
+        else if (k == "KeyM" && vid())
+            vid().muted = !vid().muted;
+        else if (k == "KeyF")
+            try {
+                if (isFullscreen)
+                    document.exitFullscreen();
+                else
+                    vid().requestFullscreen();
+            }
+            catch (ex) { }
+    }
+
+    function keyUpHandler(e) {
+        if (e.ctrlKey || e.altKey || e.metaKey || e.isComposing)
+            return;
+
+        var k = e.code + '';
+
+        if (k == "Space")
+            ev(e);
+    }
+
+    var passiveSupp = false;
+    try {
+        var opts = {
+            get passive() {
+                passiveSupp = true;
+                return false;
+            }
+        };
+        window.addEventListener('test', null, opts);
+        window.removeEventListener('test', null, opts);
+    }
+    catch (ex) {
+        passiveSupp = false;
+    }
+    var passiveEvent = passiveSupp ? { passive: false } : null;
+    var nonPassiveEvent = passiveSupp ? { passive: true } : null;
+
+    function bindEvents() {
+        bind(overlay, 'click', overlayClickHandler);
+        bind(previousButton, 'click', showPreviousImage);
+        bind(nextButton, 'click', showNextImage);
+        bind(closeButton, 'click', hideOverlay);
+        bind(slider, 'contextmenu', contextmenuHandler);
+        bind(overlay, 'touchstart', touchstartHandler, nonPassiveEvent);
+        bind(overlay, 'touchmove', touchmoveHandler, passiveEvent);
+        bind(overlay, 'touchend', touchendHandler);
+        bind(document, 'focus', trapFocusInsideOverlay, true);
+    }
+
+    function unbindEvents() {
+        unbind(overlay, 'click', overlayClickHandler);
+        unbind(previousButton, 'click', showPreviousImage);
+        unbind(nextButton, 'click', showNextImage);
+        unbind(closeButton, 'click', hideOverlay);
+        unbind(slider, 'contextmenu', contextmenuHandler);
+        unbind(overlay, 'touchstart', touchstartHandler, nonPassiveEvent);
+        unbind(overlay, 'touchmove', touchmoveHandler, passiveEvent);
+        unbind(overlay, 'touchend', touchendHandler);
+        unbind(document, 'focus', trapFocusInsideOverlay, true);
+    }
+
+    function prepareOverlay(gallery, userOptions) {
+        if (currentGallery === gallery) {
+            return;
+        }
+        currentGallery = gallery;
+        setOptions(userOptions);
+        slider.innerHTML = '';
+        imagesElements.length = 0;
+
+        var imagesFiguresIds = [];
+        var imagesCaptionsIds = [];
+        for (var i = 0, fullImage; i < gallery.length; i++) {
+            fullImage = mknod('div');
+            fullImage.className = 'full-image';
+            fullImage.id = 'baguette-img-' + i;
+            imagesElements.push(fullImage);
+
+            imagesFiguresIds.push('baguetteBox-figure-' + i);
+            imagesCaptionsIds.push('baguetteBox-figcaption-' + i);
+            slider.appendChild(imagesElements[i]);
+        }
+        overlay.setAttribute('aria-labelledby', imagesFiguresIds.join(' '));
+        overlay.setAttribute('aria-describedby', imagesCaptionsIds.join(' '));
+    }
+
+    function setOptions(newOptions) {
+        if (!newOptions) {
+            newOptions = {};
+        }
+        for (var item in defaults) {
+            options[item] = defaults[item];
+            if (typeof newOptions[item] !== 'undefined') {
+                options[item] = newOptions[item];
+            }
+        }
+        slider.style.transition = (options.animation === 'fadeIn' ? 'opacity .4s ease' :
+            options.animation === 'slideIn' ? '' : 'none');
+
+        if (options.buttons === 'auto' && ('ontouchstart' in window || currentGallery.length === 1)) {
+            options.buttons = false;
+        }
+
+        previousButton.style.display = nextButton.style.display = (options.buttons ? '' : 'none');
+    }
+
+    function showOverlay(chosenImageIndex) {
+        if (options.noScrollbars) {
+            document.documentElement.style.overflowY = 'hidden';
+            document.body.style.overflowY = 'scroll';
+        }
+        if (overlay.style.display === 'block') {
+            return;
+        }
+
+        bind(document, 'keydown', keyDownHandler);
+        bind(document, 'keyup', keyUpHandler);
+        bind(document, 'fullscreenchange', onFSC);
+        currentIndex = chosenImageIndex;
+        touch = {
+            count: 0,
+            startX: null,
+            startY: null
+        };
+        loadImage(currentIndex, function () {
+            preloadNext(currentIndex);
+            preloadPrev(currentIndex);
+        });
+
+        updateOffset();
+        overlay.style.display = 'block';
+        // Fade in overlay
+        setTimeout(function () {
+            overlay.className = 'visible';
+            if (options.bodyClass && document.body.classList) {
+                document.body.classList.add(options.bodyClass);
+            }
+            if (options.afterShow) {
+                options.afterShow();
+            }
+        }, 50);
+        if (options.onChange) {
+            options.onChange(currentIndex, imagesElements.length);
+        }
+        documentLastFocus = document.activeElement;
+        initFocus();
+        isOverlayVisible = true;
+    }
+
+    function initFocus() {
+        if (options.buttons) {
+            previousButton.focus();
+        } else {
+            closeButton.focus();
+        }
+    }
+
+    function hideOverlay(e) {
+        ev(e);
+        playvid(false);
+        if (options.noScrollbars) {
+            document.documentElement.style.overflowY = 'auto';
+            document.body.style.overflowY = 'auto';
+        }
+        if (overlay.style.display === 'none') {
+            return;
+        }
+
+        unbind(document, 'keydown', keyDownHandler);
+        unbind(document, 'keyup', keyUpHandler);
+        unbind(document, 'fullscreenchange', onFSC);
+        // Fade out and hide the overlay
+        overlay.className = '';
+        setTimeout(function () {
+            overlay.style.display = 'none';
+            if (options.bodyClass && document.body.classList) {
+                document.body.classList.remove(options.bodyClass);
+            }
+            if (options.afterHide) {
+                options.afterHide();
+            }
+            documentLastFocus && documentLastFocus.focus();
+            isOverlayVisible = false;
+        }, 500);
+    }
+
+    function loadImage(index, callback) {
+        var imageContainer = imagesElements[index];
+        var galleryItem = currentGallery[index];
+
+        if (typeof imageContainer === 'undefined' || typeof galleryItem === 'undefined') {
+            return;  // out-of-bounds or gallery dirty
+        }
+
+        if (imageContainer.querySelector('img, video')) {
+            // was loaded, cb and bail
+            if (callback) {
+                callback();
+            }
+            return;
+        }
+
+        var imageElement = galleryItem.imageElement,
+            imageSrc = imageElement.href,
+            thumbnailElement = imageElement.querySelector('img, video'),
+            imageCaption = typeof options.captions === 'function' ?
+                options.captions.call(currentGallery, imageElement) :
+                imageElement.getAttribute('data-caption') || imageElement.title;
+
+        var figure = mknod('figure');
+        figure.id = 'baguetteBox-figure-' + index;
+        figure.innerHTML = '<div class="baguetteBox-spinner">' +
+            '<div class="baguetteBox-double-bounce1"></div>' +
+            '<div class="baguetteBox-double-bounce2"></div>' +
+            '</div>';
+
+        if (options.captions && imageCaption) {
+            var figcaption = mknod('figcaption');
+            figcaption.id = 'baguetteBox-figcaption-' + index;
+            figcaption.innerHTML = imageCaption;
+            figure.appendChild(figcaption);
+        }
+        imageContainer.appendChild(figure);
+
+        var is_vid = re_v.test(imageSrc),
+            image = mknod(is_vid ? 'video' : 'img');
+
+        clmod(imageContainer, 'vid', is_vid);
+
+        image.addEventListener(is_vid ? 'loadedmetadata' : 'load', function () {
+            // Remove loader element
+            var spinner = document.querySelector('#baguette-img-' + index + ' .baguetteBox-spinner');
+            figure.removeChild(spinner);
+            if (!options.async && callback)
+                callback();
+        });
+        image.setAttribute('src', imageSrc);
+        image.setAttribute('controls', 'controls');
+        image.alt = thumbnailElement ? thumbnailElement.alt || '' : '';
+        if (options.titleTag && imageCaption) {
+            image.title = imageCaption;
+        }
+        figure.appendChild(image);
+
+        if (options.async && callback) {
+            callback();
+        }
+    }
+
+    function showNextImage(e) {
+        ev(e);
+        return show(currentIndex + 1);
+    }
+
+    function showPreviousImage(e) {
+        ev(e);
+        return show(currentIndex - 1);
+    }
+
+    function showFirstImage(event) {
+        if (event) {
+            event.preventDefault();
+        }
+        return show(0);
+    }
+
+    function showLastImage(event) {
+        if (event) {
+            event.preventDefault();
+        }
+        return show(currentGallery.length - 1);
+    }
+
+    /**
+     * Move the gallery to a specific index
+     * @param `index` {number} - the position of the image
+     * @param `gallery` {array} - gallery which should be opened, if omitted assumes the currently opened one
+     * @return {boolean} - true on success or false if the index is invalid
+     */
+    function show(index, gallery) {
+        if (!isOverlayVisible && index >= 0 && index < gallery.length) {
+            prepareOverlay(gallery, options);
+            showOverlay(index);
+            return true;
+        }
+        if (index < 0) {
+            if (options.animation) {
+                bounceAnimation('left');
+            }
+            return false;
+        }
+        if (index >= imagesElements.length) {
+            if (options.animation) {
+                bounceAnimation('right');
+            }
+            return false;
+        }
+
+        playvid(false);
+        currentIndex = index;
+        loadImage(currentIndex, function () {
+            preloadNext(currentIndex);
+            preloadPrev(currentIndex);
+        });
+        updateOffset();
+
+        if (options.onChange) {
+            options.onChange(currentIndex, imagesElements.length);
+        }
+
+        return true;
+    }
+
+    function vid() {
+        return imagesElements[currentIndex].querySelector('video');
+    }
+
+    function playvid(play) {
+        if (vid())
+            vid()[play ? 'play' : 'pause']();
+    }
+
+    function playpause() {
+        var v = vid();
+        if (v)
+            v[v.paused ? "play" : "pause"]();
+    }
+
+    function relseek(sec) {
+        if (vid())
+            vid().currentTime += sec;
+    }
+
+    /**
+     * Triggers the bounce animation
+     * @param {('left'|'right')} direction - Direction of the movement
+     */
+    function bounceAnimation(direction) {
+        slider.className = 'bounce-from-' + direction;
+        setTimeout(function () {
+            slider.className = '';
+        }, 400);
+    }
+
+    function updateOffset() {
+        var offset = -currentIndex * 100 + '%';
+        if (options.animation === 'fadeIn') {
+            slider.style.opacity = 0;
+            setTimeout(function () {
+                slider.style.transform = 'translate3d(' + offset + ',0,0)';
+                slider.style.opacity = 1;
+            }, 400);
+        } else {
+            slider.style.transform = 'translate3d(' + offset + ',0,0)';
+        }
+        playvid(false);
+        playvid(true);
+    }
+
+    function preloadNext(index) {
+        if (index - currentIndex >= options.preload) {
+            return;
+        }
+        loadImage(index + 1, function () {
+            preloadNext(index + 1);
+        });
+    }
+
+    function preloadPrev(index) {
+        if (currentIndex - index >= options.preload) {
+            return;
+        }
+        loadImage(index - 1, function () {
+            preloadPrev(index - 1);
+        });
+    }
+
+    function bind(element, event, callback, options) {
+        element.addEventListener(event, callback, options);
+    }
+
+    function unbind(element, event, callback, options) {
+        element.removeEventListener(event, callback, options);
+    }
+
+    function destroyPlugin() {
+        unbindEvents();
+        clearCachedData();
+        unbind(document, 'keydown', keyDownHandler);
+        unbind(document, 'keyup', keyUpHandler);
+        document.getElementsByTagName('body')[0].removeChild(ebi('baguetteBox-overlay'));
+        data = {};
+        currentGallery = [];
+        currentIndex = 0;
+    }
+
+    return {
+        run: run,
+        show: show,
+        showNext: showNextImage,
+        showPrevious: showPreviousImage,
+        relseek: relseek,
+        playpause: playpause,
+        hide: hideOverlay,
+        destroy: destroyPlugin
+    };
+})();

copyparty/web/browser.css

@@ -25,6 +25,47 @@ html, body {
 body {
 	padding-bottom: 5em;
 }
+#tt {
+	position: fixed;
+	max-width: 34em;
+	background: #222;
+	border: 0 solid #777;
+	overflow: hidden;
+	margin-top: 1em;
+	padding: 0 1.3em;
+	height: 0;
+	opacity: .1;
+	transition: opacity 0.14s, height 0.14s, padding 0.14s;
+	box-shadow: 0 .2em .5em #222;
+	border-radius: .4em;
+	z-index: 9001;
+}
+#tt.b {
+	padding: 0 2em;
+	border-radius: .5em;
+	box-shadow: 0 .2em 1em #000;
+}
+#tt.show {
+	padding: 1em 1.3em;
+	border-width: .4em 0;
+	height: auto;
+	opacity: 1;
+}
+#tt.show.b {
+	padding: 1.5em 2em;
+	border-width: .5em 0;
+}
+#tt code {
+	background: #3c3c3c;
+	padding: .1em .3em;
+	border-top: 1px solid #777;
+	border-radius: .3em;
+	font-family: monospace, monospace;
+	line-height: 1.7em;
+}
+#tt em {
+	color: #f6a;
+}
 #path,
 #path * {
 	font-size: 1em;
@@ -53,6 +94,7 @@ body {
 #files tbody a {
 	display: block;
 	padding: .3em 0;
+	scroll-margin-top: 45vh;
 }
 #files tbody div a {
 	color: #f5a;
@@ -68,7 +110,6 @@ a, #files tbody div a:last-child {
 	text-decoration: underline;
 }
 #files thead {
-	background: #333;
 	position: sticky;
 	top: 0;
 }
@@ -76,29 +117,30 @@ a, #files tbody div a:last-child {
 	color: #999;
 	font-weight: normal;
 }
-#files tr:hover {
+#files tr:hover td {
 	background: #1c1c1c;
 }
 #files thead th {
-	padding: .5em 1.3em .3em 1.3em;
+	padding: .5em .3em .3em .3em;
+	border-right: 2px solid #3c3c3c;
+	border-bottom: 2px solid #444;
+	background: #333;
 	cursor: pointer;
 }
+#files thead th+th {
+	border-left: 2px solid #2a2a2a;
+}
 #files thead th:last-child {
-	background: #444;
+	border-right: none;
-	border-radius: .7em .7em 0 0;
 }
-#files thead th:first-child {
+#files tbody {
 	background: #222;
 }
-#files tbody,
-#files thead th:nth-child(2) {
-	background: #222;
-	border-radius: 0 .7em 0 0;
-}
 #files td {
 	margin: 0;
 	padding: 0 .5em;
 	border-bottom: 1px solid #111;
+	border-left: 1px solid #2c2c2c;
 }
 #files td+td+td {
 	max-width: 30em;
@@ -185,9 +227,17 @@ a, #files tbody div a:last-child {
 	margin: -.2em;
 }
 #files tbody a.play.act {
-	color: #840;
+	color: #720;
 	text-shadow: 0 0 .3em #b80;
 }
+#ggrid a.play,
+html.light #ggrid a.play {
+	color: #fff;
+	background: #750;
+	border-color: #c90;
+	border-top: 1px solid #da4;
+	box-shadow: 0 .1em 1.2em #b83;
+}
 #files tbody tr.sel td,
 #ggrid a.sel,
 html.light #ggrid a.sel {
@@ -209,11 +259,17 @@ html.light #ggrid a.sel {
 	box-shadow: 0 .1em 1.2em #b36;
 	transition: all 0.2s cubic-bezier(.2, 2.2, .5, 1); /* https://cubic-bezier.com/#.4,2,.7,1 */
 }
-#ggrid a.sel img {
+#ggrid a.sel img,
+#ggrid a.play img {
 	opacity: .7;
-	box-shadow: 0 0 1em #b36;
 	filter: contrast(130%) brightness(107%);
 }
+#ggrid a.sel img {
+	box-shadow: 0 0 1em #b36;
+}
+#ggrid a.play img {
+	box-shadow: 0 0 1em #b83;
+}
 #files tr.sel a {
 	color: #fff;
 }
@@ -267,6 +323,7 @@ html.light #ggrid a.sel {
 	height: 6em;
 	width: 100%;
 	z-index: 3;
+	touch-action: none;
 	transition: bottom 0.15s;
 }
 #widget.open {
@@ -483,20 +540,56 @@ html.light #ggrid a.sel {
 	margin: .5em;
 }
 .opview input[type=text] {
-	color: #fff;
 	background: #383838;
+	color: #fff;
 	border: none;
 	box-shadow: 0 0 .3em #222;
 	border-bottom: 1px solid #fc5;
 	border-radius: .2em;
 	padding: .2em .3em;
 }
+.opview input.err,
+html.light .opview input[type="text"].err {
+	color: #fff;
+	background: #a20;
+	border-color: #f00;
+	box-shadow: 0 0 .7em #f00;
+	text-shadow: 1px 1px 0 #500;
+	outline: none;
+}
 input[type="checkbox"]+label {
 	color: #f5a;
 }
 input[type="checkbox"]:checked+label {
 	color: #fc5;
 }
+input[type="radio"]:checked+label {
+	color: #fc0;
+}
+html.light input[type="radio"]:checked+label {
+	color: #07c;
+}
+input.eq_gain {
+	width: 3em;
+	text-align: center;
+	margin: 0 .6em;
+}
+#audio_eq table {
+	border-collapse: collapse;
+}
+#audio_eq td {
+	text-align: center;
+}
+#audio_eq a.eq_step {
+	font-size: 1.5em;
+	display: block;
+	padding: 0;
+}
+#au_eq {
+	display: block;
+	margin-top: .5em;
+	padding: 1.3em .3em;
+}
 
 
 
@@ -526,7 +619,7 @@ input[type="checkbox"]:checked+label {
 #srch_q {
 	white-space: pre;
 	color: #f80;
-	height: 1em;
+	min-height: 1em;
 	margin: .2em 0 -1em 1.6em;
 }
 #tq_raw {
@@ -563,6 +656,7 @@ input[type="checkbox"]:checked+label {
 }
 #wrap {
 	margin-top: 2em;
+	min-height: 90vh;
 }
 #tree {
 	display: none;
@@ -575,8 +669,15 @@ input[type="checkbox"]:checked+label {
 	overscroll-behavior-y: none;
 	scrollbar-color: #eb0 #333;
 }
+#treeh {
+	background: #333;
+	position: sticky;
+	z-index: 1;
+	top: 0;
+}
 #thx_ff {
 	padding: 5em 0;
+	/* widget */
 }
 #tree::-webkit-scrollbar-track,
 #tree::-webkit-scrollbar {
@@ -600,6 +701,7 @@ input[type="checkbox"]:checked+label {
 	box-shadow: 0 .1em .2em #222 inset;
 	border-radius: .3em;
 	margin: .2em;
+	white-space: pre;
 	position: relative;
 	top: -.2em;
 }
@@ -636,10 +738,10 @@ input[type="checkbox"]:checked+label {
 #treeul a.hl {
 	color: #400;
 	background: #fc4;
-	border-radius: .3em;
 	text-shadow: none;
 }
 #treeul a {
+	border-radius: .3em;
 	display: inline-block;
 }
 #treeul a+a {
@@ -667,34 +769,20 @@ input[type="checkbox"]:checked+label {
 	font-size: 2em;
 	white-space: nowrap;
 }
-#files th:hover .cfg,
+#files th:hover .cfg {
-#files th.min .cfg {
 	display: block;
 	width: 1em;
 	border-radius: .2em;
 	margin: -1.3em auto 0 auto;
 	background: #444;
 }
-#files th.min .cfg {
+#files>thead>tr>th.min,
-	margin: -.6em;
+#files td.min {
-}
+		display: none;
-#files>thead>tr>th.min span {
-	position: absolute;
-	transform: rotate(270deg);
-	background: linear-gradient(90deg, rgba(68,68,68,0), rgba(68,68,68,0.5) 70%, #444);
-	margin-left: -4.6em;
-	padding: .4em;
-	top: 5.4em;
-	width: 8em;
-	text-align: right;
-	letter-spacing: .04em;
 }
 #files td:nth-child(2n) {
 	color: #f5a;
 }
-#files td.min a {
-	display: none;
-}
 #files tr.play td,
 #files tr.play div a {
 	background: #fc4;
@@ -709,50 +797,42 @@ input[type="checkbox"]:checked+label {
 	color: #300;
 	background: #fea;
 }
-#op_cfg {
+.opwide {
 	max-width: none;
 	margin-right: 1.5em;
 }
-#op_cfg>div>a {
+.opwide>div {
+	display: inline-block;
+	vertical-align: top;
+	border-left: .2em solid #4c4c4c;
+	margin-left: .5em;
+	padding-left: .5em;
+}
+.opwide>div.fill {
+	display: block;
+}
+.opwide>div>div>a {
 	line-height: 2em;
 }
-#op_cfg>div>span {
+#op_cfg>div>div>span {
 	display: inline-block;
 	padding: .2em .4em;
 }
-#op_cfg h3 {
+.opbox h3 {
 	margin: .8em 0 0 .6em;
 	padding: 0;
 	border-bottom: 1px solid #555;
 }
-#opdesc {
+#thumbs,
-	display: none;
+#au_osd_cv,
-}
+#u2tdate {
-#ops:hover #opdesc {
-	display: block;
-	background: linear-gradient(0deg,#555, #4c4c4c 80%, #444);
-	box-shadow: 0 .3em 1em #222;
-	padding: 1em;
-	border-radius: .3em;
-	position: absolute;
-	z-index: 3;
-	top: 6em;
-	right: 1.5em;
-}
-#ops:hover #opdesc.off {
-	display: none;
-}
-#opdesc code {
-	background: #3c3c3c;
-	padding: .2em .3em;
-	border-top: 1px solid #777;
-	border-radius: .3em;
-	font-family: monospace, monospace;
-	line-height: 2em;
-}
-#griden.on+#thumbs {
 	opacity: .3;
 }
+#griden.on+#thumbs,
+#au_os_ctl.on+#au_osd_cv,
+#u2turbo.on+#u2tdate {
+	opacity: 1;
+}
 #ghead {
 	background: #3c3c3c;
 	border: 1px solid #444;
@@ -801,8 +881,7 @@ html.light #ghead {
 	content: '📂';
 	line-height: 0;
 	font-size: 2em;
-	display: inline-block;
+	margin: -.7em .1em -.5em -.3em;
-	margin: -.7em 0 -.5em -.3em;
 }
 #ggrid a:hover {
 	background: #444;
@@ -854,6 +933,18 @@ html.light {
 	background: #eee;
 	text-shadow: none;
 }
+html.light #tt {
+	background: #fff;
+	border-color: #888 #000 #777 #000;
+	box-shadow: 0 .3em 1em rgba(0,0,0,0.4);
+}
+html.light #tt code {
+	background: #060;
+	color: #fff;
+}
+html.light #tt em {
+	color: #d38;
+}
 html.light #ops,
 html.light .opbox,
 html.light #srch_form {
@@ -897,8 +988,14 @@ html.light #treeul a.hl {
 	background: #07a;
 	color: #fff;
 }
+html.light #treeul a.hl:hover {
+	background: #059;
+}
 html.light #tree li {
-	border-color: #ddd #fff #f7f7f7 #fff;
+	border-color: #f7f7f7 #fff #ddd #fff;
+}
+html.light #tree a:hover {
+	background: #fff;
 }
 html.light #tree ul {
 	border-color: #ccc;
@@ -916,13 +1013,14 @@ html.light #files {
 }
 html.light #files thead th {
 	background: #eee;
-	border-radius: 0;
+	border: 1px solid #ccc;
+	border-top: none;
 }
-html.light #files tr td {
+html.light #files thead th+th {
-	border-top: 1px solid #ddd;
+	border-left: 1px solid #f7f7f7;
 }
 html.light #files td {
-	border-bottom: 1px solid #f7f7f7;
+	border-color: #fff #fff #ddd #ddd;
 }
 html.light #files tbody tr:last-child td {
 	border-bottom: .2em solid #ccc;
@@ -930,25 +1028,28 @@ html.light #files tbody tr:last-child td {
 html.light #files td:nth-child(2n) {
 	color: #d38;
 }
-html.light #files tr:hover td {
+html.light #files tr.play td:nth-child(2n) {
-	background: #fff;
+	color: #c16;
 }
 html.light #files tbody a.play {
 	color: #c0f;
 }
-html.light tr.play td {
+html.light #files tbody a.play.act {
+	color: #90c;
+}
+html.light #files tr.play td {
 	background: #fc5;
+	border-color: #eb1;
+}
+html.light #files tr:hover td {
+	background: #fff;
 }
 html.light tr.play a {
 	color: #406;
 }
-html.light #files th:hover .cfg,
+html.light #files th:hover .cfg {
-html.light #files th.min .cfg {
 	background: #ccc;
 }
-html.light #files > thead > tr > th.min span {
-	background: linear-gradient(90deg, rgba(204,204,204,0), rgba(204,204,204,0.5) 70%, #ccc);
-}
 html.light #blocked {
 	background: #eee;
 }
@@ -958,7 +1059,24 @@ html.light #blk_abrt a {
 	box-shadow: 0 .2em .4em #ddd;
 }
 html.light #widget a {
-	color: #fc5;
+	color: #06a;
+}
+html.light #wtoggle,
+html.light #widgeti {
+	background: #eee;
+}
+html.light #wtoggle {
+	box-shadow: 0 0 .5em #bbb;
+}
+html.light #widget.open {
+	border-top: .2em solid #f7f7f7;
+}
+html.light #wzip,
+html.light #wnp {
+	border-color: #ccc;
+}
+html.light #barbuf {
+	background: none;
 }
 html.light #files tr.sel:hover td {
 	background: #c37;
@@ -975,20 +1093,15 @@ html.light #files tr.sel a.play.act {
 html.light input[type="checkbox"] + label {
 	color: #333;
 }
+html.light .opwide>div {
+	border-color: #ccc;
+}
 html.light .opview input[type="text"] {
 	background: #fff;
 	color: #333;
 	box-shadow: 0 0 2px #888;
 	border-color: #38d;
 }
-html.light #ops:hover #opdesc {
-	background: #fff;
-	box-shadow: 0 .3em 1em #ccc;
-}
-html.light #opdesc code {
-	background: #060;
-	color: #fff;
-}
 html.light #u2tab a>span,
 html.light #files td div span {
 	color: #000;
@@ -998,9 +1111,6 @@ html.light #path {
 	text-shadow: none;
 	box-shadow: 0 0 .3em #bbb;
 }
-html.light #path a {
-	color: #333;
-}
 html.light #path a:not(:last-child)::after {
 	border-color: #ccc;
 	background: none;
@@ -1009,7 +1119,7 @@ html.light #path a:not(:last-child)::after {
 }
 html.light #path a:hover {
 	background: none;
-	color: #60a;
+	color: #90d;
 }
 html.light #files tbody div a {
 	color: #d38;
@@ -1019,6 +1129,9 @@ html.light #files tr.sel a:hover {
 	color: #000;
 	background: #fff;
 }
+html.light #treeh {
+	background: #eee;
+}
 html.light #tree {
 	scrollbar-color: #a70 #ddd;
 }
@@ -1028,4 +1141,167 @@ html.light #tree::-webkit-scrollbar {
 }
 #tree::-webkit-scrollbar-thumb {
 	background: #da0;
-}
+}
+
+
+
+#baguetteBox-overlay {
+	display: none;
+	opacity: 0;
+	position: fixed;
+	overflow: hidden;
+	touch-action: none;
+	top: 0;
+	left: 0;
+	width: 100%;
+	height: 100%;
+	z-index: 1000000;
+	background: rgba(0, 0, 0, 0.8);
+	transition: opacity .3s ease;
+}
+#baguetteBox-overlay.visible {
+	opacity: 1;
+}
+#baguetteBox-overlay .full-image {
+    display: inline-block;
+    position: relative;
+    width: 100%;
+    height: 100%;
+    text-align: center;
+}
+#baguetteBox-overlay .full-image figure {
+	display: inline;
+	margin: 0;
+	height: 100%;
+}
+#baguetteBox-overlay .full-image img,
+#baguetteBox-overlay .full-image video {
+	display: inline-block;
+	width: auto;
+	height: auto;
+	max-width: 100%;
+	max-height: 100%;
+	max-height: calc(100% - 1.4em);
+	margin-bottom: 1.4em;
+	vertical-align: middle;
+	box-shadow: 0 0 8px rgba(0, 0, 0, 0.6);
+}
+#baguetteBox-overlay .full-image video {
+	background: #333;
+}
+#baguetteBox-overlay .full-image figcaption {
+	display: block;
+	position: fixed;
+	bottom: .1em;
+	width: 100%;
+	text-align: center;
+	white-space: normal;
+	color: #ccc;
+}
+#baguetteBox-overlay figcaption a {
+	background: rgba(0, 0, 0, 0.6);
+	border-radius: .4em;
+	padding: .3em .6em;
+}
+#baguetteBox-overlay .full-image:before {
+	content: "";
+	display: inline-block;
+	height: 50%;
+	width: 1px;
+	margin-right: -1px;
+}
+#baguetteBox-slider {
+	position: absolute;
+	left: 0;
+	top: 0;
+	height: 100%;
+	width: 100%;
+	white-space: nowrap;
+	transition: left .2s ease, transform .2s ease;
+}
+#baguetteBox-slider.bounce-from-right {
+	animation: bounceFromRight .4s ease-out;
+}
+#baguetteBox-slider.bounce-from-left {
+	animation: bounceFromLeft .4s ease-out;
+}
+@keyframes bounceFromRight {
+	0% {margin-left: 0}
+	50% {margin-left: -30px}
+	100% {margin-left: 0}
+}
+@keyframes bounceFromLeft {
+	0% {margin-left: 0}
+	50% {margin-left: 30px}
+	100% {margin-left: 0}
+}
+.baguetteBox-button#next-button,
+.baguetteBox-button#previous-button {
+	top: 50%;
+	top: calc(50% - 30px);
+	width: 44px;
+	height: 60px;
+} 
+.baguetteBox-button {
+	position: absolute;
+	cursor: pointer;
+	outline: none;
+	padding: 0;
+	margin: 0;
+	border: 0;
+	border-radius: 15%;
+	background: rgba(50, 50, 50, 0.5);
+	color: #ddd;
+	font: 1.6em sans-serif;
+	transition: background-color .3s ease;
+}
+.baguetteBox-button:focus,
+.baguetteBox-button:hover {
+	background: rgba(50, 50, 50, 0.9);
+}
+#next-button {
+	right: 2%;
+}
+#previous-button {
+	left: 2%;
+}
+#close-button {
+	top: 20px;
+	right: 2%;
+	width: 30px;
+	height: 30px;
+}
+.baguetteBox-button svg {
+	position: absolute;
+	left: 0;
+	top: 0;
+}
+.baguetteBox-spinner {
+	width: 40px;
+	height: 40px;
+	display: inline-block;
+	position: absolute;
+	top: 50%;
+	left: 50%;
+	margin-top: -20px;
+	margin-left: -20px;
+}
+.baguetteBox-double-bounce1,
+.baguetteBox-double-bounce2 {
+	width: 100%;
+	height: 100%;
+	border-radius: 50%;
+	background-color: #fff;
+	opacity: .6;
+	position: absolute;
+	top: 0;
+	left: 0;
+	animation: bounce 2s infinite ease-in-out;
+}
+.baguetteBox-double-bounce2 {
+	animation-delay: -1s;
+}
+@keyframes bounce {
+	0%, 100% {transform: scale(0)}
+	50% {transform: scale(1)}
+}

copyparty/web/browser.html

@@ -2,134 +2,134 @@
 <html lang="en">
 
 <head>
-    <meta charset="utf-8">
+	<meta charset="utf-8">
-    <title>⇆🎉 {{ title }}</title>
+	<title>⇆🎉 {{ title }}</title>
-    <meta http-equiv="X-UA-Compatible" content="IE=edge">
+	<meta http-equiv="X-UA-Compatible" content="IE=edge">
-    <meta name="viewport" content="width=device-width, initial-scale=0.8">
+	<meta name="viewport" content="width=device-width, initial-scale=0.8">
-    <link rel="stylesheet" type="text/css" media="screen" href="/.cpr/browser.css{{ ts }}">
+	<link rel="stylesheet" type="text/css" media="screen" href="/.cpr/browser.css?_={{ ts }}">
-    <link rel="stylesheet" type="text/css" media="screen" href="/.cpr/upload.css{{ ts }}">
+	<link rel="stylesheet" type="text/css" media="screen" href="/.cpr/upload.css?_={{ ts }}">
-    {%- if css %}
+	{%- if css %}
-    <link rel="stylesheet" type="text/css" media="screen" href="{{ css }}{{ ts }}">
+	<link rel="stylesheet" type="text/css" media="screen" href="{{ css }}?_={{ ts }}">
-    {%- endif %}
+	{%- endif %}
 </head>
 
 <body>
-    <div id="ops">
+	<div id="ops"></div>
-        <a href="#" data-dest="" data-desc="close submenu">---</a>
-        {%- if have_up2k_idx %}
-        <a href="#" data-perm="read" data-dest="search" data-desc="search for files by attributes, path/name, music tags, or any combination of those.&lt;br /&gt;&lt;br /&gt;&lt;code&gt;foo bar&lt;/code&gt; = must contain both foo and bar,&lt;br /&gt;&lt;code&gt;foo -bar&lt;/code&gt; = must contain foo but not bar,&lt;br /&gt;&lt;code&gt;^yana .opus$&lt;/code&gt; = must start with yana and have the opus extension">🔎</a>
-        <a href="#" data-dest="up2k" data-desc="up2k: upload files (if you have write-access) or toggle into the search-mode and drag files onto the search button to see if they exist somewhere on the server">🚀</a>
-        {%- else %}
-        <a href="#" data-perm="write" data-dest="up2k" data-desc="up2k: upload files with resume support (close your browser and drop the same files in later)">🚀</a>
-        {%- endif %}
-        <a href="#" data-perm="write" data-dest="bup" data-desc="bup: basic uploader, even supports netscape 4.0">🎈</a>
-        <a href="#" data-perm="write" data-dest="mkdir" data-desc="mkdir: create a new directory">📂</a>
-        <a href="#" data-perm="read write" data-dest="new_md" data-desc="new-md: create a new markdown document">📝</a>
-        <a href="#" data-perm="write" data-dest="msg" data-desc="msg: send a message to the server log">📟</a>
-        <a href="#" data-dest="cfg" data-desc="configuration options">⚙️</a>
-        <div id="opdesc"></div>
-    </div>
 
-    <div id="op_search" class="opview">
+	<div id="op_search" class="opview">
-        {%- if have_tags_idx %}
+		{%- if have_tags_idx %}
-        <div id="srch_form" class="tags"></div>
+		<div id="srch_form" class="tags"></div>
-        {%- else %}
+		{%- else %}
-        <div id="srch_form"></div>
+		<div id="srch_form"></div>
-        {%- endif %}
+		{%- endif %}
-        <div id="srch_q"></div>
+		<div id="srch_q"></div>
-    </div>
+	</div>
 
-    {%- include 'upload.html' %}
+	<div id="op_player" class="opview opbox opwide"></div>
 
-    <div id="op_cfg" class="opview opbox">
+	<div id="op_bup" class="opview opbox act">
-        <h3>switches</h3>
+		<div id="u2err"></div>
-        <div>
+		<form method="post" enctype="multipart/form-data" accept-charset="utf-8" action="{{ url_suf }}">
-            <a id="tooltips" class="tgl btn" href="#">tooltips</a>
+			<input type="hidden" name="act" value="bput" />
-            <a id="lightmode" class="tgl btn" href="#">lightmode</a>
+			<input type="file" name="f" multiple><br />
-            <a id="griden" class="tgl btn" href="#">the grid</a>
+			<input type="submit" value="start upload">
-            <a id="thumbs" class="tgl btn" href="#">thumbs</a>
+		</form>
-        </div>
+	</div>
-        {%- if have_zip %}
+
-        <h3>folder download</h3>
+	<div id="op_mkdir" class="opview opbox act">
-        <div id="arc_fmt"></div>
+		<form method="post" enctype="multipart/form-data" accept-charset="utf-8" action="{{ url_suf }}">
-        {%- endif %}
+			<input type="hidden" name="act" value="mkdir" />
-        <h3>key notation</h3>
+			<input type="text" name="name" size="30">
-        <div id="key_notation"></div>
+			<input type="submit" value="mkdir">
-    </div>
+		</form>
-    
+	</div>
-    <h1 id="path">
+
-        <a href="#" id="entree">🌲</a>
+	<div id="op_new_md" class="opview opbox">
-        {%- for n in vpnodes %}
+		<form method="post" enctype="multipart/form-data" accept-charset="utf-8" action="{{ url_suf }}">
-        <a href="/{{ n[0] }}">{{ n[1] }}</a>
+			<input type="hidden" name="act" value="new_md" />
-        {%- endfor %}
+			<input type="text" name="name" size="30">
-    </h1>
+			<input type="submit" value="create doc">
-    
+		</form>
-    <div id="tree">
+	</div>
-        <a href="#" id="detree">🍞...</a>
+
-        <a href="#" class="btn" step="2" id="twobytwo">+</a>
+	<div id="op_msg" class="opview opbox act">
-        <a href="#" class="btn" step="-2" id="twig">&ndash;</a>
+		<form method="post" enctype="application/x-www-form-urlencoded" accept-charset="utf-8" action="{{ url_suf }}">
-        <a href="#" class="tgl btn" id="dyntree">a</a>
+			<input type="text" name="msg" size="30">
-        <ul id="treeul"></ul>
+			<input type="submit" value="send msg">
-        <div id="thx_ff">&nbsp;</div>
+		</form>
-    </div>
+	</div>
+
+	<div id="op_up2k" class="opview"></div>
+
+	<div id="op_cfg" class="opview opbox opwide"></div>
+	
+	<h1 id="path">
+		<a href="#" id="entree" tt="show directory tree$NHotkey: B">🌲</a>
+		{%- for n in vpnodes %}
+		<a href="/{{ n[0] }}">{{ n[1] }}</a>
+		{%- endfor %}
+	</h1>
+	
+	<div id="tree"></div>
 
 <div id="wrap">
 
-    <div id="pro" class="logue">{{ logues[0] }}</div>
+	<div id="pro" class="logue">{{ logues[0] }}</div>
 
-    <table id="files">
+	<table id="files">
-        <thead>
+		<thead>
-            <tr>
+			<tr>
-                <th name="lead"><span>c</span></th>
+				<th name="lead"><span>c</span></th>
-                <th name="href"><span>File Name</span></th>
+				<th name="href"><span>File Name</span></th>
-                <th name="sz" sort="int"><span>Size</span></th>
+				<th name="sz" sort="int"><span>Size</span></th>
-                {%- for k in taglist %}
+				{%- for k in taglist %}
-                    {%- if k.startswith('.') %}
+					{%- if k.startswith('.') %}
-                        <th name="tags/{{ k }}" sort="int"><span>{{ k[1:] }}</span></th>
+				<th name="tags/{{ k }}" sort="int"><span>{{ k[1:] }}</span></th>
-                    {%- else %}
+					{%- else %}
-                        <th name="tags/{{ k }}"><span>{{ k[0]|upper }}{{ k[1:] }}</span></th>
+				<th name="tags/{{ k }}"><span>{{ k[0]|upper }}{{ k[1:] }}</span></th>
-                    {%- endif %}
+					{%- endif %}
-                {%- endfor %}
+				{%- endfor %}
-                <th name="ext"><span>T</span></th>
+				<th name="ext"><span>T</span></th>
-                <th name="ts"><span>Date</span></th>
+				<th name="ts"><span>Date</span></th>
-            </tr>
+			</tr>
-        </thead>
+		</thead>
-        <tbody>
+<tbody>
 
 {%- for f in files %}
-    <tr><td>{{ f.lead }}</td><td><a href="{{ f.href }}">{{ f.name|e }}</a></td><td>{{ f.sz }}</td>
+<tr><td>{{ f.lead }}</td><td><a href="{{ f.href }}">{{ f.name|e }}</a></td><td>{{ f.sz }}</td>
-    {%- if f.tags is defined %}
+	{%- if f.tags is defined %}
-        {%- for k in taglist %}
+		{%- for k in taglist %}
-            <td>{{ f.tags[k] }}</td>
+<td>{{ f.tags[k] }}</td>
-        {%- endfor %}
+		{%- endfor %}
-    {%- endif %}
+	{%- endif %}
-    <td>{{ f.ext }}</td><td>{{ f.dt }}</td></tr>
+<td>{{ f.ext }}</td><td>{{ f.dt }}</td></tr>
 {%- endfor %}
 
-        </tbody>
+		</tbody>
-    </table>
+	</table>
-    
+	
-    <div id="epi" class="logue">{{ logues[1] }}</div>
+	<div id="epi" class="logue">{{ logues[1] }}</div>
 
-    <h2><a href="?h">control-panel</a></h2>
+	<h2><a href="/?h">control-panel</a></h2>
 
 </div>
 
-    {%- if srv_info %}
+	{%- if srv_info %}
-    <div id="srv_info"><span>{{ srv_info }}</span></div>
+	<div id="srv_info"><span>{{ srv_info }}</span></div>
-    {%- endif %}
+	{%- endif %}
 
-    <div id="widget"></div>
+	<div id="widget"></div>
 
-    <script>
+	<script>
-        var tag_order_cfg = {{ tag_order }};
+		var perms = {{ perms }},
-    </script>
+			tag_order_cfg = {{ tag_order }},
-    <script src="/.cpr/util.js{{ ts }}"></script>
+			have_up2k_idx = {{ have_up2k_idx|tojson }},
-    <script src="/.cpr/browser.js{{ ts }}"></script>
+			have_tags_idx = {{ have_tags_idx|tojson }},
-    <script src="/.cpr/up2k.js{{ ts }}"></script>
+			have_zip = {{ have_zip|tojson }};
-    <script>
+	</script>
-        apply_perms({{ perms }});
+	<script src="/.cpr/util.js?_={{ ts }}"></script>
-    </script>
+	<script src="/.cpr/browser.js?_={{ ts }}"></script>
+	<script src="/.cpr/up2k.js?_={{ ts }}"></script>
 </body>
 
 </html>

copyparty/web/browser2.html

@@ -2,59 +2,59 @@
 <html lang="en">
 
 <head>
-    <meta charset="utf-8">
+	<meta charset="utf-8">
-    <title>{{ title }}</title>
+	<title>{{ title }}</title>
-    <meta http-equiv="X-UA-Compatible" content="IE=edge">
+	<meta http-equiv="X-UA-Compatible" content="IE=edge">
-    <meta name="viewport" content="width=device-width, initial-scale=0.8">
+	<meta name="viewport" content="width=device-width, initial-scale=0.8">
-    <style>
+	<style>
-        html{font-family:sans-serif}
+		html{font-family:sans-serif}
-        td{border:1px solid #999;border-width:1px 1px 0 0;padding:0 5px}
+		td{border:1px solid #999;border-width:1px 1px 0 0;padding:0 5px}
-        a{display:block}
+		a{display:block}
-    </style>
+	</style>
 </head>
 
 <body>
-    {%- if srv_info %}
+	{%- if srv_info %}
-    <p><span>{{ srv_info }}</span></p>
+	<p><span>{{ srv_info }}</span></p>
-    {%- endif %}
+	{%- endif %}
 
-    {%- if have_b_u %}
+	{%- if have_b_u %}
-    <form method="post" enctype="multipart/form-data" accept-charset="utf-8" action="{{ url_suf }}">
+	<form method="post" enctype="multipart/form-data" accept-charset="utf-8" action="{{ url_suf }}">
-        <input type="hidden" name="act" value="bput" />
+		<input type="hidden" name="act" value="bput" />
-        <input type="file" name="f" multiple /><br />
+		<input type="file" name="f" multiple /><br />
-        <input type="submit" value="start upload" />
+		<input type="submit" value="start upload" />
-    </form>
+	</form>
-    <br />
+	<br />
-    {%- endif %}
+	{%- endif %}
 
-    {%- if logues[0] %}
+	{%- if logues[0] %}
-    <div>{{ logues[0] }}</div><br />
+	<div>{{ logues[0] }}</div><br />
-    {%- endif %}
+	{%- endif %}
 
-    <table id="files">
+	<table id="files">
-        <thead>
+		<thead>
-            <tr>
+			<tr>
-                <th name="lead"><span>c</span></th>
+				<th name="lead"><span>c</span></th>
-                <th name="href"><span>File Name</span></th>
+				<th name="href"><span>File Name</span></th>
-                <th name="sz" sort="int"><span>Size</span></th>
+				<th name="sz" sort="int"><span>Size</span></th>
-                <th name="ts"><span>Date</span></th>
+				<th name="ts"><span>Date</span></th>
-            </tr>
+			</tr>
-        </thead>
+		</thead>
-        <tbody>
+		<tbody>
-            <tr><td></td><td><a href="../{{ url_suf }}">parent folder</a></td><td>-</td><td>-</td></tr>
+<tr><td></td><td><a href="../{{ url_suf }}">parent folder</a></td><td>-</td><td>-</td></tr>
 
 {%- for f in files %}
-    <tr><td>{{ f.lead }}</td><td><a href="{{ f.href }}{{ url_suf }}">{{ f.name|e }}</a></td><td>{{ f.sz }}</td><td>{{ f.dt }}</td></tr>
+<tr><td>{{ f.lead }}</td><td><a href="{{ f.href }}{{ url_suf }}">{{ f.name|e }}</a></td><td>{{ f.sz }}</td><td>{{ f.dt }}</td></tr>
 {%- endfor %}
 
-        </tbody>
+		</tbody>
-    </table>
+	</table>
-    
+	
-    {%- if logues[1] %}
+	{%- if logues[1] %}
-    <div>{{ logues[1] }}</div><br />
+	<div>{{ logues[1] }}</div><br />
-    {%- endif %}
+	{%- endif %}
-    
+	
-    <h2><a href="{{ url_suf }}{{ url_suf and '&amp;' or '?' }}h">control-panel</a></h2>
+	<h2><a href="/{{ url_suf }}{{ url_suf and '&amp;' or '?' }}h">control-panel</a></h2>
 
 </body>
 </html>

copyparty/web/dbg-audio.js

@@ -0,0 +1,61 @@
+var ofun = audio_eq.apply.bind(audio_eq);
+audio_eq.apply = function () {
+    var ac1 = mp.ac;
+    ofun();
+    var ac = mp.ac,
+        w = 2048,
+        h = 256;
+
+    if (!audio_eq.filters.length) {
+        audio_eq.ana = null;
+        return;
+    }
+
+    var can = ebi('fft_can');
+    if (!can) {
+        can = mknod('canvas');
+        can.setAttribute('id', 'fft_can');
+        can.style.cssText = 'position:absolute;left:0;bottom:5em;width:' + w + 'px;height:' + h + 'px;z-index:9001';
+        document.body.appendChild(can);
+        can.width = w;
+        can.height = h;
+    }
+    var cc = can.getContext('2d');
+    if (!ac)
+        return;
+
+    var ana = ac.createAnalyser();
+    ana.smoothingTimeConstant = 0;
+    ana.fftSize = 8192;
+
+    audio_eq.filters[0].connect(ana);
+    audio_eq.ana = ana;
+
+    var buf = new Uint8Array(ana.frequencyBinCount),
+        colw = can.width / buf.length;
+
+    cc.fillStyle = '#fc0';
+    function draw() {
+        if (ana == audio_eq.ana)
+            requestAnimationFrame(draw);
+
+        ana.getByteFrequencyData(buf);
+
+        cc.clearRect(0, 0, can.width, can.height);
+
+        /*var x = 0, w = 1;
+        for (var a = 0; a < buf.length; a++) {
+            cc.fillRect(x, h - buf[a], w, h);
+            x += w;
+        }*/
+        var mul = Math.pow(w, 4) / buf.length;
+        for (var x = 0; x < w; x++) {
+            var a = Math.floor(Math.pow(x, 4) / mul),
+                v = buf[a];
+
+            cc.fillRect(x, h - v, 1, v);
+        }
+    }
+    draw();
+};
+audio_eq.apply();

copyparty/web/md.html

@@ -3,9 +3,9 @@
 	<title>📝🎉 {{ title }}</title> <!-- 📜 -->
 	<meta http-equiv="X-UA-Compatible" content="IE=edge">
 	<meta name="viewport" content="width=device-width, initial-scale=0.7">
-	<link href="/.cpr/md.css" rel="stylesheet">
+	<link href="/.cpr/md.css?_={{ ts }}" rel="stylesheet">
 	{%- if edit %}
-	<link href="/.cpr/md2.css" rel="stylesheet">
+	<link href="/.cpr/md2.css?_={{ ts }}" rel="stylesheet">
 	{%- endif %}
 </head>
 <body>
@@ -146,10 +146,10 @@ var md_opt = {
 })();
 
 	</script>
-    <script src="/.cpr/util.js"></script>
+    <script src="/.cpr/util.js?_={{ ts }}"></script>
-	<script src="/.cpr/deps/marked.js"></script>
+	<script src="/.cpr/deps/marked.js?_={{ ts }}"></script>
-	<script src="/.cpr/md.js"></script>
+	<script src="/.cpr/md.js?_={{ ts }}"></script>
 	{%- if edit %}
-	<script src="/.cpr/md2.js"></script>
+	<script src="/.cpr/md2.js?_={{ ts }}"></script>
 	{%- endif %}
 </body></html>

copyparty/web/mde.html

@@ -3,9 +3,9 @@
 	<title>📝🎉 {{ title }}</title>
 	<meta http-equiv="X-UA-Compatible" content="IE=edge">
 	<meta name="viewport" content="width=device-width, initial-scale=0.7">
-	<link href="/.cpr/mde.css" rel="stylesheet">
+	<link href="/.cpr/mde.css?_={{ ts }}" rel="stylesheet">
-	<link href="/.cpr/deps/mini-fa.css" rel="stylesheet">
+	<link href="/.cpr/deps/mini-fa.css?_={{ ts }}" rel="stylesheet">
-	<link href="/.cpr/deps/easymde.css" rel="stylesheet">
+	<link href="/.cpr/deps/easymde.css?_={{ ts }}" rel="stylesheet">
 </head>
 <body>
 	<div id="mw">
@@ -43,7 +43,7 @@ var lightswitch = (function () {
 })();
 
 	</script>
-    <script src="/.cpr/util.js"></script>
+    <script src="/.cpr/util.js?_={{ ts }}"></script>
-	<script src="/.cpr/deps/easymde.js"></script>
+	<script src="/.cpr/deps/easymde.js?_={{ ts }}"></script>
-	<script src="/.cpr/mde.js"></script>
+	<script src="/.cpr/mde.js?_={{ ts }}"></script>
 </body></html>

copyparty/web/msg.html

@@ -6,7 +6,7 @@
     <title>copyparty</title>
     <meta http-equiv="X-UA-Compatible" content="IE=edge">
     <meta name="viewport" content="width=device-width, initial-scale=0.8">
-    <link rel="stylesheet" type="text/css" media="screen" href="/.cpr/msg.css">
+    <link rel="stylesheet" type="text/css" media="screen" href="/.cpr/msg.css?_={{ ts }}">
 </head>
 
 <body>

copyparty/web/splash.html

@@ -6,7 +6,7 @@
     <title>copyparty</title>
     <meta http-equiv="X-UA-Compatible" content="IE=edge">
     <meta name="viewport" content="width=device-width, initial-scale=0.8">
-    <link rel="stylesheet" type="text/css" media="screen" href="/.cpr/splash.css">
+    <link rel="stylesheet" type="text/css" media="screen" href="/.cpr/splash.css?_={{ ts }}">
 </head>
 
 <body>
@@ -35,7 +35,7 @@
             </table>
         </td></tr></table>
         <div class="btns">
-            <a href="{{ avol[0] }}?stack">dump stack</a>
+            <a href="/?stack">dump stack</a>
         </div>
         {%- endif %}
 
@@ -59,7 +59,7 @@
 
         <h1>login for more:</h1>
         <ul>
-            <form method="post" enctype="multipart/form-data" action="/{{ url_suf }}">
+            <form method="post" enctype="multipart/form-data" action="/">
                 <input type="hidden" name="act" value="login" />
                 <input type="password" name="cppwd" />
                 <input type="submit" value="Login" />

copyparty/web/upload.css

@@ -211,36 +211,35 @@
 	box-shadow: none;
 	opacity: .2;
 }
-#u2cdesc {
-	position: absolute;
-	width: 34em;
-	left: calc(50% - 15em);
-	background: #222;
-	border: 0 solid #555;
-	text-align: center;
-	overflow: hidden;
-	margin: 0 -2em;
-	padding: 0 1em;
-	height: 0;
-	opacity: .1;
-	transition: all 0.14s ease-in-out;
-	box-shadow: 0 .2em .5em #222;
-	border-radius: .4em;
-	z-index: 1;
-}
-#u2cdesc.show {
-	padding: 1em;
-	height: auto;
-	border-width: .2em 0;
-	opacity: 1;
-}
 #u2foot {
 	color: #fff;
 	font-style: italic;
 }
+#u2foot .warn {
+	font-size: 1.3em;
+	padding: .5em .8em;
+	margin: 1em -.6em;
+	color: #f74;
+	background: #322;
+	border: 1px solid #633;
+	border-width: .1em 0;
+	text-align: center;
+}
+#u2foot .warn span {
+	color: #f86;
+}
+html.light #u2foot .warn {
+	color: #b00;
+	background: #fca;
+	border-color: #f70;
+}
+html.light #u2foot .warn span {
+	color: #930;
+}
 #u2foot span {
 	color: #999;
 	font-size: .9em;
+	font-weight: normal;
 }
 #u2footfoot {
 	margin-bottom: -1em;
@@ -286,10 +285,6 @@ html.light #u2conf .txtbox.err {
 	background: #f96;
 	color: #300;
 }
-html.light #u2cdesc {
-	background: #fff;
-	border: none;
-}
 html.light #op_up2k.srch #u2btn {
 	border-color: #a80;
 }

copyparty/web/upload.html

@@ -1,103 +0,0 @@
-
-    <div id="op_bup" class="opview opbox act">
-        <div id="u2err"></div>
-        <form method="post" enctype="multipart/form-data" accept-charset="utf-8" action="{{ url_suf }}">
-            <input type="hidden" name="act" value="bput" />
-            <input type="file" name="f" multiple><br />
-            <input type="submit" value="start upload">
-        </form>
-    </div>
-
-    <div id="op_mkdir" class="opview opbox act">
-        <form method="post" enctype="multipart/form-data" accept-charset="utf-8" action="{{ url_suf }}">
-            <input type="hidden" name="act" value="mkdir" />
-            <input type="text" name="name" size="30">
-            <input type="submit" value="mkdir">
-        </form>
-    </div>
-
-    <div id="op_new_md" class="opview opbox">
-        <form method="post" enctype="multipart/form-data" accept-charset="utf-8" action="{{ url_suf }}">
-            <input type="hidden" name="act" value="new_md" />
-            <input type="text" name="name" size="30">
-            <input type="submit" value="create doc">
-        </form>
-    </div>
-
-    <div id="op_msg" class="opview opbox act">
-        <form method="post" enctype="application/x-www-form-urlencoded" accept-charset="utf-8" action="{{ url_suf }}">
-            <input type="text" name="msg" size="30">
-            <input type="submit" value="send msg">
-        </form>
-    </div>
-
-    <div id="op_up2k" class="opview">
-        <form id="u2form" method="post" enctype="multipart/form-data" onsubmit="return false;"></form>
-
-            <table id="u2conf">
-                <tr>
-                    <td><br />parallel uploads:</td>
-                    <td rowspan="2">
-                        <input type="checkbox" id="multitask" />
-                        <label for="multitask" alt="continue hashing other files while uploading">🏃</label>
-                    </td>
-                    <td rowspan="2">
-                        <input type="checkbox" id="ask_up" />
-                        <label for="ask_up" alt="ask for confirmation befofre upload starts">💭</label>
-                    </td>
-                    <td rowspan="2">
-                        <input type="checkbox" id="flag_en" />
-                        <label for="flag_en" alt="ensure only one tab is uploading at a time $N (other tabs must have this enabled too)">💤</label>
-                    </td>
-                {%- if have_up2k_idx %}
-                    <td data-perm="read" rowspan="2">
-                        <input type="checkbox" id="fsearch" />
-                        <label for="fsearch" alt="don't actually upload, instead check if the files already $N exist on the server (will scan all folders you can read)">🔎</label>
-                    </td>
-                {%- endif %}
-                    <td data-perm="read" rowspan="2" id="u2btn_cw"></td>
-                </tr>
-                <tr>
-                    <td>
-                        <a href="#" id="nthread_sub">&ndash;</a><input
-                            class="txtbox" id="nthread" value="2"/><a
-                            href="#" id="nthread_add">+</a><br />&nbsp;
-                    </td>
-                </tr>
-            </table>
-
-            <div id="u2cdesc"></div>
-
-            <div id="u2notbtn"></div>
-
-            <div id="u2btn_ct">
-                <div id="u2btn">
-                    <span id="u2bm"></span><br />
-                    drag/drop files<br />
-                    and folders here<br />
-                    (or click me)
-                </div>
-            </div>
-
-            <div id="u2cards">
-                <a href="#" act="ok">ok <span>0</span></a><a
-                href="#" act="ng">ng <span>0</span></a><a
-                href="#" act="done">done <span>0</span></a><a
-                href="#" act="bz" class="act">busy <span>0</span></a><a
-                href="#" act="q">que <span>0</span></a>
-            </div>
-
-            <table id="u2tab">
-                <thead>
-                    <tr>
-                        <td>filename</td>
-                        <td>status</td>
-                        <td>progress<a href="#" id="u2cleanup">cleanup</a></td>
-                    </tr>
-                </thead>
-                <tbody></tbody>
-            </table>
-
-            <p id="u2foot"></p>
-            <p id="u2footfoot" data-perm="write">( you can use the <a href="#" id="u2nope">basic uploader</a> if you don't need lastmod timestamps, resumable uploads, or progress bars )</p>
-    </div>

copyparty/web/util.js

@@ -6,21 +6,11 @@ if (!window['console'])
     };
 
 
-var clickev = window.Touch ? 'touchstart' : 'click',
+var is_touch = 'ontouchstart' in window,
     ANDROID = /(android)/i.test(navigator.userAgent);
 
 
 // error handler for mobile devices
-function hcroak(msg) {
-    document.body.innerHTML = msg;
-    window.onerror = undefined;
-    throw 'fatal_err';
-}
-function croak(msg) {
-    document.body.textContent = msg;
-    window.onerror = undefined;
-    throw msg;
-}
 function esc(txt) {
     return txt.replace(/[&"<>]/g, function (c) {
         return {
@@ -32,9 +22,12 @@ function esc(txt) {
     });
 }
 function vis_exh(msg, url, lineNo, columnNo, error) {
+    if (!window.onerror)
+        return;
+
     window.onerror = undefined;
     window['vis_exh'] = null;
-    var html = ['<h1>you hit a bug!</h1><p>please screenshot this error and send me a copy arigathanks gozaimuch (ed/irc.rizon.net or ed#2644)</p><p>',
+    var html = ['<h1>you hit a bug!</h1><p>please send me a screenshot arigathanks gozaimuch: <code>ed/irc.rizon.net</code> or <code>ed#2644</code><br />&nbsp; (and if you can, press F12 and include the "Console" tab in the screenshot too)</p><p>',
         esc(String(msg)), '</p><p>', esc(url + ' @' + lineNo + ':' + columnNo), '</p>'];
 
     if (error) {
@@ -44,9 +37,14 @@ function vis_exh(msg, url, lineNo, columnNo, error) {
                 html.push('<h2>' + find[a] + '</h2>' +
                     esc(String(error[find[a]])).replace(/\n/g, '<br />\n'));
     }
-    document.body.style.fontSize = '0.8em';
+    html.push('<p style="border-top:1px solid #999;margin-top:1.5em;font-size:1.4em">if you are stuck here, try to <a href="#" onclick="localStorage.clear();location.reload();">reset copyparty settings</a></p>');
-    document.body.style.padding = '0 1em 1em 1em';
+    document.body.innerHTML = html.join('\n');
-    hcroak(html.join('\n'));
+
+    var s = mknod('style');
+    s.innerHTML = 'body{background:#333;color:#ddd;font-family:sans-serif;font-size:0.8em;padding:0 1em 1em 1em} code{color:#bf7;background:#222;padding:.1em;margin:.2em;font-size:1.1em;font-family:monospace,monospace} *{line-height:1.5em}';
+    document.head.appendChild(s);
+
+    throw 'fatal_err';
 }
 
 
@@ -67,6 +65,9 @@ function ev(e) {
     if (e.stopPropagation)
         e.stopPropagation();
 
+    if (e.stopImmediatePropagation)
+        e.stopImmediatePropagation();
+
     e.returnValue = false;
     return e;
 }
@@ -285,63 +286,6 @@ function makeSortable(table, cb) {
 }
 
 
-(function () {
-    var ops = QSA('#ops>a');
-    for (var a = 0; a < ops.length; a++) {
-        ops[a].onclick = opclick;
-    }
-})();
-
-
-function opclick(e) {
-    ev(e);
-
-    var dest = this.getAttribute('data-dest');
-    goto(dest);
-
-    swrite('opmode', dest || null);
-
-    var input = QS('.opview.act input:not([type="hidden"])')
-    if (input)
-        input.focus();
-}
-
-
-function goto(dest) {
-    var obj = QSA('.opview.act');
-    for (var a = obj.length - 1; a >= 0; a--)
-        clmod(obj[a], 'act');
-
-    obj = QSA('#ops>a');
-    for (var a = obj.length - 1; a >= 0; a--)
-        clmod(obj[a], 'act');
-
-    if (dest) {
-        var ui = ebi('op_' + dest);
-        clmod(ui, 'act', true);
-        QS('#ops>a[data-dest=' + dest + ']').className += " act";
-
-        var fn = window['goto_' + dest];
-        if (fn)
-            fn();
-    }
-
-    if (window['treectl'])
-        treectl.onscroll();
-}
-
-
-(function () {
-    goto();
-    var op = sread('opmode');
-    if (op !== null && op !== '.')
-        try {
-            goto(op);
-        }
-        catch (ex) { }
-})();
-
-
 function linksplit(rp) {
     var ret = [];
     var apath = '/';
@@ -416,6 +360,15 @@ function get_vpath() {
 }
 
 
+function get_pwd() {
+    var pwd = ('; ' + document.cookie).split('; cppwd=');
+    if (pwd.length < 2)
+        return null;
+
+    return pwd[1].split(';')[0];
+}
+
+
 function unix2iso(ts) {
     return new Date(ts * 1000).toISOString().replace("T", " ").slice(0, -5);
 }
@@ -437,6 +390,18 @@ function has(haystack, needle) {
 }
 
 
+function apop(arr, v) {
+    var ofs = arr.indexOf(v);
+    if (ofs !== -1)
+        arr.splice(ofs, 1);
+}
+
+
+function jcp(obj) {
+    return JSON.parse(JSON.stringify(obj));
+}
+
+
 function sread(key) {
     if (window.localStorage)
         return localStorage.getItem(key);
@@ -528,3 +493,69 @@ function hist_replace(url) {
     console.log("h-repl " + url);
     history.replaceState(url, url, url);
 }
+
+
+var tt = (function () {
+    var r = {
+        "tt": mknod("div"),
+        "en": true
+    };
+
+    r.tt.setAttribute('id', 'tt');
+    document.body.appendChild(r.tt);
+
+    function show() {
+        var cfg = sread('tooltips');
+        if (cfg !== null && cfg != '1')
+            return;
+
+        var msg = this.getAttribute('tt');
+        if (!msg)
+            return;
+
+        var pos = this.getBoundingClientRect(),
+            left = pos.left < window.innerWidth / 2,
+            top = pos.top < window.innerHeight / 2,
+            big = this.className.indexOf(' ttb') !== -1;
+
+        clmod(r.tt, 'b', big);
+        r.tt.style.top = top ? pos.bottom + 'px' : 'auto';
+        r.tt.style.bottom = top ? 'auto' : (window.innerHeight - pos.top) + 'px';
+        r.tt.style.left = left ? pos.left + 'px' : 'auto';
+        r.tt.style.right = left ? 'auto' : (window.innerWidth - pos.right) + 'px';
+
+        r.tt.innerHTML = msg.replace(/\$N/g, "<br />");
+        clmod(r.tt, 'show', 1);
+    }
+
+    function hide() {
+        clmod(r.tt, 'show');
+    }
+
+    r.init = function () {
+        var ttb = ebi('tooltips');
+        if (ttb) {
+            ttb.onclick = function (e) {
+                ev(e);
+                r.en = !r.en;
+                bcfg_set('tooltips', r.en);
+                r.init();
+            };
+            r.en = bcfg_get('tooltips', true)
+        }
+
+        var _show = r.en ? show : null,
+            _hide = r.en ? hide : null;
+
+        var o = QSA('*[tt]');
+        for (var a = o.length - 1; a >= 0; a--) {
+            o[a].onfocus = _show;
+            o[a].onblur = _hide;
+            o[a].onmouseenter = _show;
+            o[a].onmouseleave = _hide;
+        }
+        hide();
+    };
+
+    return r;
+})();

docs/README.md

@@ -1,8 +1,19 @@
-## [`minimal-up2k.html`](minimal-up2k.html)
+# example `.epilogue.html`
-* save as `.epilogue.html` inside a folder to [simplify the ui](https://user-images.githubusercontent.com/241032/118311195-dd6ca380-b4ef-11eb-86f3-75a3ff2e1332.png)
+save one of these as `.epilogue.html` inside a folder to customize it:
 
-## [`browser.css`](browser.css)
+* [`minimal-up2k.html`](minimal-up2k.html) will [simplify the upload ui](https://user-images.githubusercontent.com/241032/118311195-dd6ca380-b4ef-11eb-86f3-75a3ff2e1332.png)
-* example for `--css-browser`
+
+
+
+# example browser-css
+point `--css-browser` to one of these by URL:
+
+* [`browser.css`](browser.css) changes the background
+* [`browser-icons.css`](browser-icons.css) adds filetype icons
+
+
+
+# other stuff
 
 ## [`rclone.md`](rclone.md)
 * notes on using rclone as a fuse client/server

docs/biquad.html

@@ -0,0 +1,95 @@
+<!DOCTYPE html><html><head></head><body><script>
+
+setTimeout(location.reload.bind(location), 700);
+document.documentElement.scrollLeft = 0;
+
+var can = document.createElement('canvas'),
+    cc = can.getContext('2d'),
+    w = 2048,
+    h = 1024;
+
+w = 2048;
+
+can.width = w;
+can.height = h;
+document.body.appendChild(can);
+can.style.cssText = 'width:' + w + 'px;height:' + h + 'px';
+
+cc.fillStyle = '#000';
+cc.fillRect(0, 0, w, h);
+
+var cfg = [ // hz, q, g
+    [31.25 * 0.88, 0, 1.4],  // shelf
+    [31.25 * 1.04, 0.7, 0.96],  // peak
+    [62.5, 0.7, 1],
+    [125, 0.8, 1],
+    [250, 0.9, 1.03],
+    [500, 0.9, 1.1],
+    [1000, 0.9, 1.1],
+    [2000, 0.9, 1.105],
+    [4000, 0.88, 1.05],
+    [8000 * 1.006, 0.73, 1.24],
+    //[16000 * 1.00, 0.5, 1.75],  // peak.v1
+    //[16000 * 1.19, 0, 1.8]  // shelf.v1
+    [16000 * 0.89, 0.7, 1.26],  // peak
+    [16000 * 1.13, 0.82, 1.09],  // peak
+    [16000 * 1.205, 0, 1.9]  // shelf
+];
+
+var freqs = new Float32Array(22000),
+    sum = new Float32Array(freqs.length),
+    ac = new AudioContext(),
+    step = w / freqs.length,
+    colors = [
+        'rgba(255, 0, 0, 0.7)',
+        'rgba(0, 224, 0, 0.7)',
+        'rgba(0, 64, 255, 0.7)'
+    ];
+
+var order = [];
+
+for (var a = 0; a < cfg.length; a += 2)
+    order.push(a);
+
+for (var a = 1; a < cfg.length; a += 2)
+    order.push(a);
+
+for (var ia = 0; ia < order.length; ia++) {
+    var a = order[ia],
+        fi = ac.createBiquadFilter(),
+        mag = new Float32Array(freqs.length),
+        phase = new Float32Array(freqs.length);
+
+    for (var b = 0; b < freqs.length; b++)
+        freqs[b] = b;
+
+    fi.type = a == 0 ? 'lowshelf' : a == cfg.length - 1 ? 'highshelf' : 'peaking';
+    fi.frequency.value = cfg[a][0];
+    fi.Q.value = cfg[a][1];
+    fi.gain.value = 1;
+
+    fi.getFrequencyResponse(freqs, mag, phase);
+    cc.fillStyle = colors[a % colors.length];
+    for (var b = 0; b < sum.length; b++) {
+        mag[b] -= 1;
+        sum[b] += mag[b] * cfg[a][2];
+        var y = h - (mag[b] * h * 3);
+        cc.fillRect(b * step, y, step, h - y);
+        cc.fillRect(b * step - 1, y - 1, 3, 3);
+    }
+}
+
+var min = 999999, max = 0;
+for (var a = 0; a < sum.length; a++) {
+    min = Math.min(min, sum[a]);
+    max = Math.max(max, sum[a]);
+}
+cc.fillStyle = 'rgba(255,255,255,1)';
+for (var a = 0; a < sum.length; a++) {
+    var v = (sum[a] - min) / (max - min);
+    cc.fillRect(a * step, 0, step, v * h / 2);
+}
+
+cc.fillRect(0, 460, w, 1);
+
+</script></body></html>

docs/browser-icons.css

@@ -0,0 +1,66 @@
+/* put filetype icons inline with text
+#ggrid>a>span:before,
+#ggrid>a>span.dir:before {
+	display: inline;
+	line-height: 0;
+	font-size: 1.7em;
+	margin: -.7em .1em -.5em -.6em;
+}
+*/
+
+
+/* move folder icons top-left */
+#ggrid>a>span.dir:before {
+	content: initial;
+}
+#ggrid>a[href$="/"]:before {
+	content: '📂';
+}
+
+
+/* put filetype icons top-left */
+#ggrid>a:before {
+    display: block;
+    position: absolute;
+	padding: .3em 0;
+    margin: -.4em;
+    text-shadow: 0 0 .1em #000;
+	background: linear-gradient(135deg,rgba(255,255,255,0) 50%,rgba(255,255,255,0.2));
+	border-radius: .3em;
+    font-size: 2em;
+}
+
+
+/* video */
+#ggrid>a:is(
+[href$=".mkv"i],
+[href$=".mp4"i],
+[href$=".webm"i],
+):before {
+    content: '📺';
+}
+
+
+/* audio */
+#ggrid>a:is(
+[href$=".mp3"i],
+[href$=".ogg"i],
+[href$=".opus"i],
+[href$=".flac"i],
+[href$=".m4a"i],
+[href$=".aac"i],
+):before {
+    content: '🎵';
+}
+
+
+/* image */
+#ggrid>a:is(
+[href$=".jpg"i],
+[href$=".jpeg"i],
+[href$=".png"i],
+[href$=".gif"i],
+[href$=".webp"i],
+):before {
+    content: '🎨';
+}

docs/browser.css

@@ -1,5 +1,5 @@
 html {
-    background: url('/wp/wallhaven-mdjrqy.jpg') center / cover no-repeat fixed;
+    background: #333 url('/wp/wallhaven-mdjrqy.jpg') center / cover no-repeat fixed;
 }
 #files th {
     background: rgba(32, 32, 32, 0.9) !important;
@@ -12,7 +12,7 @@ html {
 
 
 html.light {
-    background: url('/wp/wallhaven-dpxl6l.png') center / cover no-repeat fixed;
+    background: #eee url('/wp/wallhaven-dpxl6l.png') center / cover no-repeat fixed;
 }
 html.light #files th {
     background: rgba(255, 255, 255, 0.9) !important;

docs/hls.html

@@ -0,0 +1,51 @@
+<!DOCTYPE html><html lang="en"><head>
+	<meta charset="utf-8">
+	<title>hls-test</title>
+	<meta http-equiv="X-UA-Compatible" content="IE=edge">
+</head><body>
+
+<video id="vid" controls></video>
+<script src="hls.light.js"></script>
+<script>
+
+var video = document.getElementById('vid');
+var hls = new Hls({
+	debug: true,
+	autoStartLoad: false
+});
+hls.loadSource('live/v.m3u8');
+hls.attachMedia(video);
+hls.on(Hls.Events.MANIFEST_PARSED, function() {
+	hls.startLoad(0);
+});
+hls.on(Hls.Events.MEDIA_ATTACHED, function() {
+	video.muted = true;
+	video.play();
+});
+
+/*
+general good news:
+- doesn't need fixed-length segments; ok to let x264 pick optimal keyframes and slice on those
+- hls.js polls the m3u8 for new segments, scales the duration accordingly, seeking works great
+- the sfx will grow by 66 KiB since that's how small hls.js can get, wait thats not good
+
+# vod, creates m3u8 at the end, fixed keyframes, v bad
+ffmpeg -hide_banner -threads 0 -flags -global_header -i ..\CowboyBebopMovie-OP1.webm -vf scale=1280:-4,format=yuv420p -ac 2 -c:a libopus -b:a 128k -c:v libx264 -preset slow -crf 24 -maxrate:v 5M -bufsize:v 10M -g 120 -keyint_min 120 -sc_threshold 0 -hls_time 4 -hls_playlist_type vod -hls_segment_filename v%05d.ts v.m3u8
+
+# live, updates m3u8 as it goes, dynamic keyframes, streamable with hls.js
+ffmpeg -hide_banner -threads 0 -flags -global_header -i ..\..\CowboyBebopMovie-OP1.webm -vf scale=1280:-4,format=yuv420p -ac 2 -c:a libopus -b:a 128k -c:v libx264 -preset slow -crf 24 -maxrate:v 5M -bufsize:v 10M -f segment -segment_list v.m3u8 -segment_format mpegts -segment_list_flags live v%05d.ts
+
+# fmp4 (fragmented mp4), doesn't work with hls.js, gets duratoin 149:07:51 (536871s), probably the tkhd/mdhd 0xffffffff (timebase 8000? ok)
+ffmpeg -re -hide_banner -threads 0 -flags +cgop -i ..\..\CowboyBebopMovie-OP1.webm -vf scale=1280:-4,format=yuv420p -ac 2 -c:a libopus -b:a 128k -c:v libx264 -preset slow -crf 24 -maxrate:v 5M -bufsize:v 10M -f segment -segment_list v.m3u8 -segment_format fmp4 -segment_list_flags live v%05d.mp4
+
+# try 2, works, uses tempfiles for m3u8 updates, good, 6% smaller
+ffmpeg -re -hide_banner -threads 0 -flags +cgop -i ..\..\CowboyBebopMovie-OP1.webm -vf scale=1280:-4,format=yuv420p -ac 2 -c:a libopus -b:a 128k -c:v libx264 -preset slow -crf 24 -maxrate:v 5M -bufsize:v 10M -f hls -hls_segment_type fmp4 -hls_list_size 0 -hls_segment_filename v%05d.mp4 v.m3u8
+
+more notes
+- adding -hls_flags single_file makes duration wack during playback (for both fmp4 and ts), ok once finalized and refreshed, gives no size reduction anyways
+- bebop op has good keyframe spacing for testing hls.js, in particular it hops one seg back and immediately resumes if it hits eof with the explicit hls.startLoad(0); otherwise it jumps into the middle of a seg and becomes art
+- can probably -c:v copy most of the time, is there a way to check for cgop? todo
+
+*/
+</script>
+</body></html>

docs/notes.sh

@@ -86,6 +86,9 @@ var t=[]; var b=document.location.href.split('#')[0].slice(0, -1); document.quer
 # get the size and video-id of all youtube vids in folder, assuming filename ends with -id.ext, and create a copyparty search query
 find -maxdepth 1 -printf '%s %p\n' | sort -n | awk '!/-([0-9a-zA-Z_-]{11})\.(mkv|mp4|webm)$/{next} {sub(/\.[^\.]+$/,"");n=length($0);v=substr($0,n-10);print $1, v}' | tee /dev/stderr | awk 'BEGIN {p="("} {printf("%s name like *-%s.* ",p,$2);p="or"} END {print ")\n"}' | cat >&2
 
+# unique stacks in a stackdump
+f=a; rm -rf stacks; mkdir stacks; grep -E '^#' $f | while IFS= read -r n; do awk -v n="$n" '!$0{o=0} o; $0==n{o=1}' <$f >stacks/f; h=$(sha1sum <stacks/f | cut -c-16); mv stacks/f stacks/$h-"$n"; done ; find stacks/ | sort | uniq -cw24
+
 
 ##
 ## sqlite3 stuff
@@ -100,6 +103,15 @@ cat warks | while IFS= read -r x; do sqlite3 up2k.db "delete from mt where w = '
 # dump all dbs
 find -iname up2k.db | while IFS= read -r x; do sqlite3 "$x" 'select substr(w,1,12), rd, fn from up' | sed -r 's/\|/ \| /g' | while IFS= read -r y; do printf '%s | %s\n' "$x" "$y"; done; done
 
+# unschedule mtp scan for all files somewhere under "enc/"
+sqlite3 -readonly up2k.db 'select substr(up.w,1,16) from up inner join mt on mt.w = substr(up.w,1,16) where rd like "enc/%" and +mt.k = "t:mtp"' > keys; awk '{printf "delete from mt where w = \"%s\" and +k = \"t:mtp\";\n", $0}' <keys | tee /dev/stderr | sqlite3 up2k.db
+
+# compare metadata key "key" between two databases
+sqlite3 -readonly up2k.db.key-full 'select w, v from mt where k = "key" order by w' > k1; sqlite3 -readonly up2k.db 'select w, v from mt where k = "key" order by w' > k2; ok=0; ng=0; while IFS='|' read w k2; do k1="$(grep -E "^$w" k1 | sed -r 's/.*\|//')"; [ "$k1" = "$k2" ] && ok=$((ok+1)) || { ng=$((ng+1)); printf '%3s %3s  %s\n' "$k1" "$k2" "$(sqlite3 -readonly up2k.db.key-full "select * from up where substr(w,1,16) = '$w'" | sed -r 's/\|/ | /g')"; }; done < <(cat k2); echo "match $ok   diff $ng"
+
+# actually this is much better
+sqlite3 -readonly up2k.db.key-full 'select w, v from mt where k = "key" order by w' > k1; sqlite3 -readonly up2k.db 'select mt.w, mt.v, up.rd, up.fn from mt inner join up on mt.w = substr(up.w,1,16) where mt.k = "key" order by up.rd, up.fn' > k2; ok=0; ng=0; while IFS='|' read w k2 path; do k1="$(grep -E "^$w" k1 | sed -r 's/.*\|//')"; [ "$k1" = "$k2" ] && ok=$((ok+1)) || { ng=$((ng+1)); printf '%3s %3s  %s\n' "$k1" "$k2" "$path"; }; done < <(cat k2); echo "match $ok   diff $ng"
+
 
 ##
 ## media
@@ -154,7 +166,7 @@ dbg.asyncStore.pendingBreakpoints = {}
 about:config >> devtools.debugger.prefs-schema-version = -1
 
 # determine server version
-git reset --hard origin/HEAD && git log --format=format:"%H %ai %d" --decorate=full > /dev/shm/revs && cat /dev/shm/revs | while read -r rev extra; do (git reset --hard $rev >/dev/null 2>/dev/null && dsz=$(cat copyparty/web/{util,browser,up2k}.js 2>/dev/null | diff -wNarU0 - <(cat /mnt/Users/ed/Downloads/ref/{util,browser,up2k}.js) | wc -c) && printf '%s %6s %s\n' "$rev" $dsz "$extra") </dev/null; done                
+git pull; git reset --hard origin/HEAD && git log --format=format:"%H %ai %d" --decorate=full > ../revs && cat ../{util,browser}.js >../vr && cat ../revs | while read -r rev extra; do (git reset --hard $rev >/dev/null 2>/dev/null && dsz=$(cat copyparty/web/{util,browser}.js >../vg 2>/dev/null && diff -wNarU0 ../{vg,vr} | wc -c) && printf '%s %6s %s\n' "$rev" $dsz "$extra") </dev/null; done                
 
 
 ##
@@ -197,3 +209,4 @@ mk() { rm -rf /tmp/foo; sudo -u ed bash -c 'mkdir /tmp/foo; echo hi > /tmp/foo/b
 mk && t0="$(date)" && while true; do date -s "$(date '+ 1 hour')"; systemd-tmpfiles --clean; ls -1 /tmp | grep foo || break; done; echo "$t0"
 mk && sudo -u ed flock /tmp/foo sleep 40 & sleep 1; ps aux | grep -E 'sleep 40$' && t0="$(date)" && for n in {1..40}; do date -s "$(date '+ 1 day')"; systemd-tmpfiles --clean; ls -1 /tmp | grep foo || break; done; echo "$t0"
 mk && t0="$(date)" && for n in {1..40}; do date -s "$(date '+ 1 day')"; systemd-tmpfiles --clean; ls -1 /tmp | grep foo || break; tar -cf/dev/null /tmp/foo; done; echo "$t0"
+

docs/tcp-debug.sh

@@ -0,0 +1,32 @@
+(cd ~/dev/copyparty && strace -Tttyyvfs 256 -o strace.strace python3 -um copyparty -i 127.0.0.1 --http-only --stackmon /dev/shm/cpps,10 ) 2>&1 | tee /dev/stderr > ~/log-copyparty-$(date +%Y-%m%d-%H%M%S).txt
+
+14/Jun/2021:16:34:02 1623688447.212405 death
+14/Jun/2021:16:35:02 1623688502.420860 back
+
+tcpdump -nni lo -w /home/ed/lo.pcap
+
+# 16:35:25.324662 IP 127.0.0.1.48632 > 127.0.0.1.3920: Flags [F.], seq 849, ack 544, win 359, options [nop,nop,TS val 809396796 ecr 809396796], length 0
+
+tcpdump -nnr /home/ed/lo.pcap | awk '/ > 127.0.0.1.3920: /{sub(/ > .*/,"");sub(/.*\./,"");print}' | sort -n | uniq | while IFS= read -r port; do echo; tcpdump -nnr /home/ed/lo.pcap 2>/dev/null | grep -E "\.$port( > |: F)" | sed -r 's/ > .*, /, /'; done | grep -E '^16:35:0.*length [^0]' -C50
+
+16:34:02.441732 IP 127.0.0.1.48638, length 0
+16:34:02.441738 IP 127.0.0.1.3920, length 0
+16:34:02.441744 IP 127.0.0.1.48638, length 0
+16:34:02.441756 IP 127.0.0.1.48638, length 791
+16:34:02.441759 IP 127.0.0.1.3920, length 0
+16:35:02.445529 IP 127.0.0.1.48638, length 0
+16:35:02.489194 IP 127.0.0.1.3920, length 0
+16:35:02.515595 IP 127.0.0.1.3920, length 216
+16:35:02.515600 IP 127.0.0.1.48638, length 0
+
+grep 48638 "$(find ~ -maxdepth 1 -name log-copyparty-\*.txt | sort | tail -n 1)"
+
+1623688502.510380 48638 rh
+1623688502.511291 48638 Unrecv direct ...
+1623688502.511827 48638 rh = 791
+16:35:02.518 127.0.0.1 48638       shut(8): [Errno 107] Socket not connected
+Exception in thread httpsrv-0.1-48638:
+
+grep 48638 ~/dev/copyparty/strace.strace
+14561 16:35:02.506310 <... accept4 resumed> {sa_family=AF_INET, sin_port=htons(48638), sin_addr=inet_addr("127.0.0.1")}, [16], SOCK_CLOEXEC) = 8<TCP:[127.0.0.1:3920->127.0.0.1:48638]> <0.000012>
+15230 16:35:02.510725 write(1<pipe:[256639555]>, "1623688502.510380 48638 rh\n", 27 <unfinished ...>

scripts/copyparty-repack.sh

@@ -92,20 +92,34 @@ chmod 755 \
   copyparty-extras/copyparty-*/{scripts,bin}/*
 
 
-# extract and repack the sfx with less features enabled
+# extract the sfx
 ( cd copyparty-extras/sfx-full/
 ./copyparty-sfx.py -h
-cd ../copyparty-*/
-./scripts/make-sfx.sh re no-ogv no-cm
 )
 
 
-# put new sfx into copyparty-extras/sfx-lite/,
+repack() {
-# fuse client into copyparty-extras/,
+
+	# do the repack
+	(cd copyparty-extras/copyparty-*/
+	./scripts/make-sfx.sh $2
+	)
+
+	# put new sfx into copyparty-extras/$name/,
+	( cd copyparty-extras/
+	mv copyparty-*/dist/* $1/
+	)
+}
+
+repack sfx-full "re gz no-sh"
+repack sfx-lite "re no-ogv no-cm"
+repack sfx-lite "re no-ogv no-cm gz no-sh"
+
+
+# move fuse client into copyparty-extras/,
 # copy lite-sfx.py to ./copyparty,
 # delete extracted source code
 ( cd copyparty-extras/
-mv copyparty-*/dist/* sfx-lite/
 mv copyparty-*/bin/copyparty-fuse.py .
 cp -pv sfx-lite/copyparty-sfx.py ../copyparty
 rm -rf copyparty-{0..9}*.*.*{0..9}
@@ -119,6 +133,7 @@ true
 
 
 # create the bundle
+printf '\n\n'
 fn=copyparty-$(date +%Y-%m%d-%H%M%S).tgz
 tar -czvf "$od/$fn" *
 cd "$od"

scripts/deps-docker/Dockerfile

@@ -1,6 +1,7 @@
 FROM    alpine:3.13
 WORKDIR /z
 ENV     ver_asmcrypto=5b994303a9d3e27e0915f72a10b6c2c51535a4dc \
+        ver_hashwasm=4.7.0 \
         ver_marked=1.1.0 \
         ver_ogvjs=1.8.0 \
         ver_mde=2.14.0 \
@@ -9,12 +10,6 @@ ENV     ver_asmcrypto=5b994303a9d3e27e0915f72a10b6c2c51535a4dc \
         ver_zopfli=1.0.3
 
 
-# TODO
-# sha512.hw.js https://github.com/Daninet/hash-wasm
-# sha512.kc.js https://github.com/chm-diederichs/sha3-wasm
-# awk '/HMAC state/{o=1}  /var HEAP/{o=0}  /function hmac_reset/{o=1}  /return \{/{o=0}  /var __extends =/{o=1}  /var Hash =/{o=0}  /hmac_|pbkdf2_/{next}  o{next}  {gsub(/IllegalStateError/,"Exception")}  {sub(/^ +/,"");sub(/^\/\/ .*/,"");sub(/;$/," ;")}  1' <sha512.ac.js.orig >sha512.ac.js; for fn in sha512.ac.js.orig sha512.ac.js; do wc -c <$fn; wc -c <$fn.gz ; for n in {1..9}; do printf '%8d %d bz\n' $(bzip2 -c$n <$fn | wc -c) $n; done; done
-
-
 # download;
 # the scp url is latin from https://fonts.googleapis.com/css2?family=Source+Code+Pro&display=swap
 RUN     mkdir -p /z/dist/no-pk \
@@ -27,7 +22,11 @@ RUN     mkdir -p /z/dist/no-pk \
         && wget https://github.com/codemirror/CodeMirror/archive/$ver_codemirror.tar.gz -O codemirror.tgz \
         && wget https://github.com/FortAwesome/Font-Awesome/releases/download/$ver_fontawesome/fontawesome-free-$ver_fontawesome-web.zip -O fontawesome.zip \
         && wget https://github.com/google/zopfli/archive/zopfli-$ver_zopfli.tar.gz -O zopfli.tgz \
+        && wget https://github.com/Daninet/hash-wasm/releases/download/v$ver_hashwasm/hash-wasm@$ver_hashwasm.zip -O hash-wasm.zip \
         && unzip ogvjs.zip \
+        && (mkdir hash-wasm \
+            && cd hash-wasm \
+            && unzip ../hash-wasm.zip) \
         && (tar -xf asmcrypto.tgz \
             && cd asmcrypto.js-$ver_asmcrypto \
             && npm install ) \
@@ -64,7 +63,12 @@ RUN     tar -xf zopfli.tgz \
 RUN     cd asmcrypto.js-$ver_asmcrypto \
         && echo "export { Sha512 } from './hash/sha512/sha512';" > src/entry-export_all.ts \
         && node -r esm build.js \
-        && mv asmcrypto.all.es5.js /z/dist/sha512.js
+        && awk '/HMAC state/{o=1}  /var HEAP/{o=0}  /function hmac_reset/{o=1}  /return \{/{o=0}  /var __extends =/{o=1}  /var Hash =/{o=0}  /hmac_|pbkdf2_/{next}  o{next}  {gsub(/IllegalStateError/,"Exception")}  {sub(/^ +/,"");sub(/^\/\/ .*/,"");sub(/;$/," ;")}  1' < asmcrypto.all.es5.js > /z/dist/sha512.ac.js
+
+
+# build hash-wasm
+RUN     cd hash-wasm \
+        && mv sha512.umd.min.js /z/dist/sha512.hw.js
 
 
 # build ogvjs

scripts/make-sfx.sh

@@ -11,6 +11,10 @@ echo
 # `re` does a repack of an sfx which you already executed once
 #   (grabs files from the sfx-created tempdir), overrides `clean`
 #
+# `gz` creates a gzip-compressed python sfx instead of bzip2
+#
+# `no-sh` makes just the python sfx, skips the sh/unix sfx
+#
 # `no-ogv` saves ~500k by removing the opus/vorbis audio codecs
 #   (only affects apple devices; everything else has native support)
 #
@@ -167,7 +171,7 @@ find .. -type f \( -name .DS_Store -or -name ._.DS_Store \) -delete
 find .. -type f -name ._\* | while IFS= read -r f; do cmp <(printf '\x00\x05\x16') <(head -c 3 -- "$f") && rm -f -- "$f"; done
 
 echo use smol web deps
-rm -f copyparty/web/deps/*.full.* copyparty/web/Makefile
+rm -f copyparty/web/deps/*.full.* copyparty/web/dbg-* copyparty/web/Makefile
 
 # it's fine dw
 grep -lE '\.full\.(js|css)' copyparty/web/* |
@@ -208,7 +212,7 @@ find | grep -E '\.css$' | while IFS= read -r f; do
 	}
 	!/\}$/ {printf "%s",$0;next}
 	1
-	' <$f | gsed 's/;\}$/}/' >t
+	' <$f | sed 's/;\}$/}/' >t
 	tmv "$f"
 done
 find | grep -E '\.(js|html)$' | while IFS= read -r f; do
@@ -223,7 +227,7 @@ command -v pigz &&
 	pk='gzip'
 
 echo "$pk"
-find | grep -E '\.(js|css)$' | while IFS= read -r f; do
+find | grep -E '\.(js|css)$' | grep -vF /deps/ | while IFS= read -r f; do
 	echo -n .
 	$pk "$f"
 done

scripts/sfx.py

@@ -6,10 +6,10 @@ import re, os, sys, time, shutil, signal, threading, tarfile, hashlib, platform,
 import subprocess as sp
 
 """
-run me with any version of python, i will unpack and run copyparty
+pls don't edit this file with a text editor,
+  it breaks the compressed stuff at the end
 
-(but please don't edit this file with a text editor
+run me with any version of python, i will unpack and run copyparty
-  since that would probably corrupt the binary stuff at the end)
 
 there's zero binaries! just plaintext python scripts all the way down
   so you can easily unpack the archive and inspect it for shady stuff

scripts/test/race.py

@@ -0,0 +1,105 @@
+#!/usr/bin/env python3
+
+import os
+import sys
+import time
+import json
+import threading
+import http.client
+
+
+class Conn(object):
+    def __init__(self, ip, port):
+        self.s = http.client.HTTPConnection(ip, port, timeout=260)
+        self.st = []
+
+    def get(self, vpath):
+        self.st = [time.time()]
+
+        self.s.request("GET", vpath)
+        self.st.append(time.time())
+
+        ret = self.s.getresponse()
+        self.st.append(time.time())
+
+        if ret.status < 200 or ret.status >= 400:
+            raise Exception(ret.status)
+
+        ret = ret.read()
+        self.st.append(time.time())
+
+        return ret
+
+    def get_json(self, vpath):
+        ret = self.get(vpath)
+        return json.loads(ret)
+
+
+class CState(threading.Thread):
+    def __init__(self, cs):
+        threading.Thread.__init__(self)
+        self.daemon = True
+        self.cs = cs
+        self.start()
+
+    def run(self):
+        colors = [5, 1, 3, 2, 7]
+        remotes = []
+        remotes_ok = False
+        while True:
+            time.sleep(0.001)
+            if not remotes_ok:
+                remotes = []
+                remotes_ok = True
+                for conn in self.cs:
+                    try:
+                        remotes.append(conn.s.sock.getsockname()[1])
+                    except:
+                        remotes.append("?")
+                        remotes_ok = False
+
+            m = []
+            for conn, remote in zip(self.cs, remotes):
+                stage = len(conn.st)
+                m.append(f"\033[3{colors[stage]}m{remote}")
+
+            m = " ".join(m)
+            print(f"{m}\033[0m\n\033[A", end="")
+
+
+def allget(cs, urls):
+    thrs = []
+    for c, url in zip(cs, urls):
+        t = threading.Thread(target=c.get, args=(url,))
+        t.start()
+        thrs.append(t)
+
+    for t in thrs:
+        t.join()
+
+
+def main():
+    os.system("")
+
+    ip, port = sys.argv[1].split(":")
+    port = int(port)
+
+    cs = []
+    for _ in range(64):
+        cs.append(Conn(ip, 3923))
+
+    CState(cs)
+
+    urlbase = "/doujin/c95"
+    j = cs[0].get_json(f"{urlbase}?ls")
+    urls = []
+    for d in j["dirs"]:
+        urls.append(f"{urlbase}/{d['href']}?th=w")
+
+    for n in range(100):
+        print(n)
+        allget(cs, urls)
+
+
+if __name__ == "__main__":
+    main()

tests/test_httpcli.py

@@ -23,12 +23,14 @@ def hdr(query):
 
 
 class Cfg(Namespace):
-    def __init__(self, a=[], v=[], c=None):
+    def __init__(self, a=None, v=None, c=None):
         super(Cfg, self).__init__(
-            a=a,
+            a=a or [],
-            v=v,
+            v=v or [],
             c=c,
+            rproxy=0,
             ed=False,
+            nw=False,
             no_zip=False,
             no_scandir=False,
             no_sendfile=True,
@@ -39,6 +41,7 @@ class Cfg(Namespace):
             mte="a",
             hist=None,
             no_hash=False,
+            css_browser=None,
             **{k: False for k in "e2d e2ds e2dsa e2t e2ts e2tsr".split()}
         )
 

tests/test_vfs.py

@@ -16,11 +16,18 @@ from copyparty import util
 
 
 class Cfg(Namespace):
-    def __init__(self, a=[], v=[], c=None):
+    def __init__(self, a=None, v=None, c=None):
-        ex = {k: False for k in "e2d e2ds e2dsa e2t e2ts e2tsr".split()}
+        ex = {k: False for k in "nw e2d e2ds e2dsa e2t e2ts e2tsr".split()}
-        ex2 = {"mtp": [], "mte": "a", "hist": None, "no_hash": False}
+        ex2 = {
+            "mtp": [],
+            "mte": "a",
+            "hist": None,
+            "no_hash": False,
+            "css_browser": None,
+            "rproxy": 0,
+        }
         ex.update(ex2)
-        super(Cfg, self).__init__(a=a, v=v, c=c, **ex)
+        super(Cfg, self).__init__(a=a or [], v=v or [], c=c, **ex)
 
 
 class TestVFS(unittest.TestCase):

tests/util.py

@@ -108,6 +108,9 @@ class VHttpSrv(object):
         aliases = ["splash", "browser", "browser2", "msg", "md", "mde"]
         self.j2 = {x: J2_FILES for x in aliases}
 
+    def cachebuster(self):
+        return "a"
+
 
 class VHttpConn(object):
     def __init__(self, args, asrv, log, buf):
@@ -121,8 +124,8 @@ class VHttpConn(object):
         self.log_src = "a"
         self.lf_url = None
         self.hsrv = VHttpSrv()
+        self.nreq = 0
         self.nbyte = 0
-        self.workload = 0
         self.ico = None
         self.thumbcli = None
         self.t0 = time.time()