v0.10.17

README.md

@@ -12,6 +12,8 @@ turn your phone or raspi into a portable file server with resumable uploads/down
 * *resumable* uploads need `firefox 12+` / `chrome 6+` / `safari 6+` / `IE 10+`
 * code standard: `black`
 
+📷 screenshots: [browser](#the-browser) // [upload](#uploading) // [md-viewer](#markdown-viewer) // [search](#searching) // [fsearch](#file-search) // [zip-DL](#zip-downloads) // [ie4](#browser-support)
+
 
 ## readme toc
 
@@ -20,8 +22,16 @@ turn your phone or raspi into a portable file server with resumable uploads/down
     * [notes](#notes)
     * [status](#status)
 * [bugs](#bugs)
-* [usage](#usage)
+    * [not my bugs](#not-my-bugs)
+* [the browser](#the-browser)
+    * [tabs](#tabs)
+    * [hotkeys](#hotkeys)
+    * [tree-mode](#tree-mode)
     * [zip downloads](#zip-downloads)
+    * [uploading](#uploading)
+        * [file-search](#file-search)
+    * [markdown viewer](#markdown-viewer)
+    * [other tricks](#other-tricks)
 * [searching](#searching)
     * [search configuration](#search-configuration)
     * [metadata from audio files](#metadata-from-audio-files)
@@ -29,6 +39,7 @@ turn your phone or raspi into a portable file server with resumable uploads/down
     * [complete examples](#complete-examples)
 * [browser support](#browser-support)
 * [client examples](#client-examples)
+* [up2k](#up2k)
 * [dependencies](#dependencies)
     * [optional gpl stuff](#optional-gpl-stuff)
 * [sfx](#sfx)
@@ -53,9 +64,9 @@ you may also want these, especially on servers:
 ## notes
 
 * iPhone/iPad: use Firefox to download files
-* Android-Chrome: set max "parallel uploads" for 200% upload speed (android bug)
-* Android-Firefox: takes a while to select files (in order to avoid the above android-chrome issue)
-* Desktop-Firefox: may use gigabytes of RAM if your connection is great and your files are massive
+* Android-Chrome: increase "parallel uploads" for higher speed (android bug)
+* Android-Firefox: takes a while to select files (their fix for ☝️)
+* Desktop-Firefox: ~~may use gigabytes of RAM if your files are massive~~ *seems to be OK now*
 * paper-printing is affected by dark/light-mode! use lightmode for color, darkmode for grayscale
   * because no browsers currently implement the media-query to do this properly orz
 
@@ -101,15 +112,44 @@ summary: it works! you can use it! (but technically not even close to beta)
 * hiding the contents at url `/d1/d2/d3` using `-v :d1/d2/d3:cd2d` has the side-effect of creating databases (for files/tags) inside folders d1 and d2, and those databases take precedence over the main db at the top of the vfs - this means all files in d2 and below will be reindexed unless you already had a vfs entry at or below d2
 * probably more, pls let me know
 
+## not my bugs
 
-# usage
+* Windows: msys2-python 3.8.6 occasionally throws "RuntimeError: release unlocked lock" when leaving a scoped mutex in up2k
+  * this is an msys2 bug, the regular windows edition of python is fine
+
+
+# the browser
+
+![copyparty-browser-fs8](https://user-images.githubusercontent.com/241032/115978054-65106380-a57d-11eb-98f8-59e3dee73557.png)
+
+
+## tabs
+
+* `[🔎]` search by size, date, path/name, mp3-tags ... see [searching](#searching)
+* `[🚀]` and `[🎈]` are the uploaders, see [uploading](#uploading)
+* `[📂]` mkdir, create directories
+* `[📝]` new-md, create a new markdown document
+* `[📟]` send-msg, either to server-log or into textfiles if `--urlform save`
+* `[⚙️]` client configuration options
+
+
+## hotkeys
 
 the browser has the following hotkeys
-* `0..9` jump to 10%..90%
-* `U/O` skip 10sec back/forward
-* `J/L` prev/next song
 * `I/K` prev/next folder
 * `P` parent folder
+* when playing audio:
+  * `0..9` jump to 10%..90%
+  * `U/O` skip 10sec back/forward
+  * `J/L` prev/next song
+    * `J` also starts playing the folder
+
+
+## tree-mode
+
+by default there's a breadcrumbs path; you can replace this with a tree-browser sidebar thing by clicking the 🌲
+
+click `[-]` and `[+]` to adjust the size, and the `[a]` toggles if the tree should widen dynamically as you go deeper or stay fixed-size
 
 
 ## zip downloads
@@ -128,12 +168,80 @@ the `zip` link next to folders can produce various types of zip/tar files using
 * `zip_crc` will take longer to download since the server has to read each file twice
   * please let me know if you find a program old enough to actually need this
 
+you can also zip a selection of files or folders by clicking them in the browser, that brings up a selection editor and zip button in the bottom right
+
+![copyparty-zipsel-fs8](https://user-images.githubusercontent.com/241032/116008321-372a2e00-a614-11eb-9a4a-4a1fd9074224.png)
+
+## uploading
+
+two upload methods are available in the html client:
+* 🎈 bup, the basic uploader, supports almost every browser since netscape 4.0
+* 🚀 up2k, the fancy one
+
+up2k has several advantages:
+* you can drop folders into the browser (files are added recursively)
+* files are processed in chunks, and each chunk is checksummed
+  * uploads resume if they are interrupted (for example by a reboot)
+  * server detects any corruption; the client reuploads affected chunks
+  * the client doesn't upload anything that already exists on the server
+* the last-modified timestamp of the file is preserved
+
+see [up2k](#up2k) for details on how it works
+
+![copyparty-upload-fs8](https://user-images.githubusercontent.com/241032/115978061-680b5400-a57d-11eb-9ef6-cbb5f60aeccc.png)
+
+the up2k UI is the epitome of polished inutitive experiences:
+* "parallel uploads" specifies how many chunks to upload at the same time
+* `[🏃]` analysis of other files should continue while one is uploading
+* `[💭]` ask for confirmation before files are added to the list
+* `[💤]` sync uploading between other copyparty tabs so only one is active
+* `[🔎]` switch between upload and file-search mode
+
+and then theres the tabs below it,
+* `[ok]` is uploads which completed successfully
+* `[ng]` is the uploads which failed / got rejected (already exists, ...)
+* `[done]` shows a combined list of `[ok]` and `[ng]`, chronological order
+* `[busy]` files which are currently hashing, pending-upload, or uploading
+  * plus up to 3 entries each from `[done]` and `[que]` for context
+* `[que]` is all the files that are still queued
+
+protip: you can avoid scaring away users by hiding some of the UI with hacks like [docs/minimal-up2k.html](docs/minimal-up2k.html)
+
+### file-search
+
+![copyparty-fsearch-fs8](https://user-images.githubusercontent.com/241032/116008320-36919780-a614-11eb-803f-04162326a700.png)
+
+in the 🚀 up2k tab, after toggling the `[🔎]` switch green, any files/folders you drop onto the dropzone will be hashed on the client-side. Each hash is sent to the server which checks if that file exists somewhere already
+
+files go into `[ok]` if they exist (and you get a link to where it is), otherwise they land in `[ng]`
+* the main reason filesearch is combined with the uploader is cause the code was too spaghetti to separate it out somewhere else
+
+adding the same file multiple times is blocked, so if you first search for a file and then decide to upload it, you have to click the `[cleanup]` button to discard `[done]` files
+
+note that since up2k has to read the file twice, 🎈 bup can be up to 2x faster if your internet connection is faster than the read-speed of your HDD
+
+up2k has saved a few uploads from becoming corrupted in-transfer already; caught an android phone on wifi redhanded in wireshark with a bitflip, however bup with https would *probably* have noticed as well thanks to tls also functioning as an integrity check
+
+
+## markdown viewer
+
+![copyparty-md-read-fs8](https://user-images.githubusercontent.com/241032/115978057-66419080-a57d-11eb-8539-d2be843991aa.png)
+
+* the document preview has a max-width which is the same as an A4 paper when printed
+
+
+## other tricks
+
+* you can link a particular timestamp in an audio file by adding it to the URL, such as `&20` / `&20s` / `&1m20` / `&t=1:20` after the `.../#af-c8960dab`
+
 
 # searching
 
+![copyparty-search-fs8](https://user-images.githubusercontent.com/241032/115978060-6772bd80-a57d-11eb-81d3-174e869b72c3.png)
+
 when started with `-e2dsa` copyparty will scan/index all your files. This avoids duplicates on upload, and also makes the volumes searchable through the web-ui:
 * make search queries by `size`/`date`/`directory-path`/`filename`, or...
-* drag/drop a local file to see if the same contents exist somewhere on the server (you get the URL if it does)
+* drag/drop a local file to see if the same contents exist somewhere on the server, see [file-search](#file-search)
 
 path/name queries are space-separated, AND'ed together, and words are negated with a `-` prefix, so for example:
 * path: `shibayan -bossa` finds all files where one of the folders contain `shibayan` but filters out any results where `bossa` exists somewhere in the path
@@ -199,6 +307,8 @@ copyparty can invoke external programs to collect additional metadata for files
 
 # browser support
 
+![copyparty-ie4-fs8](https://user-images.githubusercontent.com/241032/116009043-a1909d80-a617-11eb-9140-037ad6604899.png)
+
 `ie` = internet-explorer, `ff` = firefox, `c` = chrome, `iOS` = iPhone/iPad, `Andr` = Android
 
 | feature         | ie6 | ie9 | ie10 | ie11 | ff 52 | c 49 | iOS | Andr |
@@ -229,7 +339,7 @@ quick summary of more eccentric web-browsers trying to view a directory index:
 * lynx (2.8.9/macports) can browse, login, upload/mkdir/msg
 * w3m (0.5.3/macports) can browse, login, upload at 100kB/s, mkdir/msg
 * netsurf (3.10/arch) is basically ie6 with much better css (javascript has almost no effect)
-* netscape 4.0 and 4.5 can browse (text is yellow on white), upload with `?b=u`
+* ie4 and netscape 4.0 can browse (text is yellow on white), upload with `?b=u`
 * SerenityOS (22d13d8) hits a page fault, works with `?b=u`, file input not-impl, url params are multiplying
 
 # client examples
@@ -256,6 +366,22 @@ copyparty returns a truncated sha512sum of your PUT/POST as base64; you can gene
     b512 <movie.mkv
 
 
+# up2k
+
+quick outline of the up2k protocol, see [uploading](#uploading) for the web-client
+* the up2k client splits a file into an "optimal" number of chunks
+  * 1 MiB each, unless that becomes more than 256 chunks
+  * tries 1.5M, 2M, 3, 4, 6, ... until <= 256# or chunksize >= 32M
+* client posts the list of hashes, filename, size, last-modified
+* server creates the `wark`, an identifier for this upload
+  * `sha512( salt + filesize + chunk_hashes )`
+  * and a sparse file is created for the chunks to drop into
+* client uploads each chunk
+  * header entries for the chunk-hash and wark
+  * server writes chunks into place based on the hash
+* client does another handshake with the hashlist; server replies with OK or a list of chunks to reupload
+
+
 # dependencies
 
 * `jinja2` (is built into the SFX)
@@ -272,12 +398,12 @@ copyparty returns a truncated sha512sum of your PUT/POST as base64; you can gene
 
 some bundled tools have copyleft dependencies, see [./bin/#mtag](bin/#mtag)
 
-these are standalone and will never be imported / evaluated by copyparty
+these are standalone programs and will never be imported / evaluated by copyparty
 
 
 # sfx
 
-currently there are two self-contained binaries:
+currently there are two self-contained "binaries":
 * [copyparty-sfx.py](https://github.com/9001/copyparty/releases/latest/download/copyparty-sfx.py) -- pure python, works everywhere
 * [copyparty-sfx.sh](https://github.com/9001/copyparty/releases/latest/download/copyparty-sfx.sh) -- smaller, but only for linux and macos
 
@@ -339,7 +465,6 @@ in the `scripts` folder:
 
 roughly sorted by priority
 
-* audio link with timestamp
 * separate sqlite table per tag
 * audio fingerprinting
 * readme.md as epilogue

copyparty/__init__.py

@@ -16,12 +16,17 @@ if platform.system() == "Windows":
 VT100 = not WINDOWS or WINDOWS >= [10, 0, 14393]
 # introduced in anniversary update
 
+ANYWIN = WINDOWS or sys.platform in ["msys"]
+
 MACOS = platform.system() == "Darwin"
 
 
 class EnvParams(object):
     def __init__(self):
         self.mod = os.path.dirname(os.path.realpath(__file__))
+        if self.mod.endswith("__init__"):
+            self.mod = os.path.dirname(self.mod)
+
         if sys.platform == "win32":
             self.cfg = os.path.normpath(os.environ["APPDATA"] + "/copyparty")
         elif sys.platform == "darwin":

copyparty/__main__.py

@@ -247,6 +247,7 @@ def run_argparse(argv, formatter):
     ap.add_argument("--no-zip", action="store_true", help="disable download as zip/tar")
     ap.add_argument("--no-sendfile", action="store_true", help="disable sendfile (for debugging)")
     ap.add_argument("--no-scandir", action="store_true", help="disable scandir (for debugging)")
+    ap.add_argument("--sparse", metavar="MiB", type=int, default=4, help="up2k min.size threshold (mswin-only)")
     ap.add_argument("--urlform", metavar="MODE", type=str, default="print,get", help="how to handle url-forms")
     ap.add_argument("--salt", type=str, default="hunter2", help="up2k file-hash salt")
 

copyparty/__version__.py

@@ -1,8 +1,8 @@
 # coding: utf-8
 
-VERSION = (0, 10, 9)
+VERSION = (0, 10, 17)
 CODENAME = "zip it"
-BUILD_DT = (2021, 4, 17)
+BUILD_DT = (2021, 5, 12)
 
 S_VERSION = ".".join(map(str, VERSION))
 S_BUILD_DT = "{0:04d}-{1:02d}-{2:02d}".format(*BUILD_DT)

copyparty/authsrv.py

@@ -141,7 +141,12 @@ class VFS(object):
         real.sort()
         if not rem:
             for name, vn2 in sorted(self.nodes.items()):
-                if uname in vn2.uread or "*" in vn2.uread:
+                if (
+                    uname in vn2.uread
+                    or "*" in vn2.uread
+                    or uname in vn2.uwrite
+                    or "*" in vn2.uwrite
+                ):
                     virt_vis[name] = vn2
 
             # no vfs nodes in the list of real inodes
@@ -241,6 +246,7 @@ class AuthSrv(object):
         self.args = args
         self.log_func = log_func
         self.warn_anonwrite = warn_anonwrite
+        self.line_ctr = 0
 
         if WINDOWS:
             self.re_vol = re.compile(r"^([a-zA-Z]:[\\/][^:]*|[^:]*):([^:]*):(.*)$")
@@ -266,7 +272,9 @@ class AuthSrv(object):
     def _parse_config_file(self, fd, user, mread, mwrite, mflags, mount):
         vol_src = None
         vol_dst = None
+        self.line_ctr = 0
         for ln in [x.decode("utf-8").strip() for x in fd]:
+            self.line_ctr += 1
             if not ln and vol_src is not None:
                 vol_src = None
                 vol_dst = None
@@ -296,7 +304,12 @@ class AuthSrv(object):
                 mflags[vol_dst] = {}
                 continue
 
-            lvl, uname = ln.split(" ")
+            if len(ln) > 1:
+                lvl, uname = ln.split(" ")
+            else:
+                lvl = ln
+                uname = "*"
+
             self._read_vol_str(
                 lvl, uname, mread[vol_dst], mwrite[vol_dst], mflags[vol_dst]
             )
@@ -374,7 +387,12 @@ class AuthSrv(object):
         if self.args.c:
             for cfg_fn in self.args.c:
                 with open(cfg_fn, "rb") as f:
-                    self._parse_config_file(f, user, mread, mwrite, mflags, mount)
+                    try:
+                        self._parse_config_file(f, user, mread, mwrite, mflags, mount)
+                    except:
+                        m = "\n\033[1;31m\nerror in config file {} on line {}:\n\033[0m"
+                        print(m.format(cfg_fn, self.line_ctr))
+                        raise
 
         if not mount:
             # -h says our defaults are CWD at root and read/write for everyone

copyparty/httpcli.py

@@ -13,7 +13,7 @@ import ctypes
 from datetime import datetime
 import calendar
 
-from .__init__ import E, PY2, WINDOWS
+from .__init__ import E, PY2, WINDOWS, ANYWIN
 from .util import *  # noqa  # pylint: disable=unused-wildcard-import
 from .szip import StreamZip
 from .star import StreamTar
@@ -182,10 +182,8 @@ class HttpCli(object):
         self.out_headers.update(headers)
 
         # default to utf8 html if no content-type is set
-        try:
-            mime = mime or self.out_headers["Content-Type"]
-        except KeyError:
-            mime = "text/html; charset=UTF-8"
+        if not mime:
+            mime = self.out_headers.get("Content-Type", "text/html; charset=UTF-8")
 
         self.out_headers["Content-Type"] = mime
 
@@ -261,12 +259,14 @@ class HttpCli(object):
 
                 self.absolute_urls = True
 
-        # go home if verboten
         self.readable, self.writable = self.conn.auth.vfs.can_access(
             self.vpath, self.uname
         )
         if not self.readable and not self.writable:
-            self.log("inaccessible: [{}]".format(self.vpath))
+            if self.vpath:
+                self.log("inaccessible: [{}]".format(self.vpath))
+                raise Pebkac(404)
+
             self.uparam = {"h": False}
 
         if "h" in self.uparam:
@@ -534,7 +534,7 @@ class HttpCli(object):
             self.log("qj: " + repr(vbody))
             hits = idx.fsearch(vols, body)
             msg = repr(hits)
-            taglist = []
+            taglist = {}
         else:
             # search by query params
             self.log("qj: " + repr(body))
@@ -626,7 +626,7 @@ class HttpCli(object):
             self.loud_reply(x, status=500)
             return False
 
-        if not WINDOWS and num_left == 0:
+        if not ANYWIN and num_left == 0:
             times = (int(time.time()), int(lastmod))
             self.log("no more chunks, setting times {}".format(times))
             try:
@@ -680,7 +680,7 @@ class HttpCli(object):
                 raise Pebkac(500, "mkdir failed, check the logs")
 
         vpath = "{}/{}".format(self.vpath, sanitized).lstrip("/")
-        esc_paths = [quotep(vpath), html_escape(vpath)]
+        esc_paths = [quotep(vpath), html_escape(vpath, crlf=True)]
         html = self.j2(
             "msg",
             h2='<a href="/{}">go to /{}</a>'.format(*esc_paths),
@@ -741,7 +741,9 @@ class HttpCli(object):
 
                 if p_file and not nullwrite:
                     fdir = os.path.join(vfs.realpath, rem)
-                    fname = sanitize_fn(p_file)
+                    fname = sanitize_fn(
+                        p_file, bad=[".prologue.html", ".epilogue.html"]
+                    )
 
                     if not os.path.isdir(fsenc(fdir)):
                         raise Pebkac(404, "that folder does not exist")
@@ -1181,17 +1183,16 @@ class HttpCli(object):
         template = self.j2(tpl)
 
         st = os.stat(fsenc(fs_path))
-        # sz_md = st.st_size
         ts_md = st.st_mtime
 
         st = os.stat(fsenc(html_path))
         ts_html = st.st_mtime
 
-        # TODO dont load into memory ;_;
-        #   (trivial fix, count the &'s)
-        with open(fsenc(fs_path), "rb") as f:
-            md = f.read().replace(b"&", b"&amp;")
-            sz_md = len(md)
+        sz_md = 0
+        for buf in yieldfile(fs_path):
+            sz_md += len(buf)
+            for c, v in [[b"&", 4], [b"<", 3], [b">", 3]]:
+                sz_md += (len(buf) - len(buf.replace(c, b""))) * v
 
         file_ts = max(ts_md, ts_html)
         file_lastmod, do_send = self._chk_lastmod(file_ts)
@@ -1199,27 +1200,34 @@ class HttpCli(object):
         self.out_headers["Cache-Control"] = "no-cache"
         status = 200 if do_send else 304
 
+        boundary = "\roll\tide"
         targs = {
             "edit": "edit" in self.uparam,
-            "title": html_escape(self.vpath),
+            "title": html_escape(self.vpath, crlf=True),
             "lastmod": int(ts_md * 1000),
             "md_plug": "true" if self.args.emp else "false",
             "md_chk_rate": self.args.mcr,
-            "md": "",
+            "md": boundary,
         }
-        sz_html = len(template.render(**targs).encode("utf-8"))
-        self.send_headers(sz_html + sz_md, status)
+        html = template.render(**targs).encode("utf-8")
+        html = html.split(boundary.encode("utf-8"))
+        if len(html) != 2:
+            raise Exception("boundary appears in " + html_path)
+
+        self.send_headers(sz_md + len(html[0]) + len(html[1]), status)
 
         logmsg += unicode(status)
         if self.mode == "HEAD" or not do_send:
             self.log(logmsg)
             return True
 
-        # TODO jinja2 can stream this right?
-        targs["md"] = md.decode("utf-8", "replace")
-        html = template.render(**targs).encode("utf-8")
         try:
-            self.s.sendall(html)
+            self.s.sendall(html[0])
+            for buf in yieldfile(fs_path):
+                self.s.sendall(html_bescape(buf))
+
+            self.s.sendall(html[1])
+
         except:
             self.log(logmsg + " \033[31md/c\033[0m")
             return False
@@ -1300,7 +1308,7 @@ class HttpCli(object):
                 else:
                     vpath += "/" + node
 
-                vpnodes.append([quotep(vpath) + "/", html_escape(node)])
+                vpnodes.append([quotep(vpath) + "/", html_escape(node, crlf=True)])
 
         vn, rem = self.auth.vfs.get(
             self.vpath, self.uname, self.readable, self.writable
@@ -1311,6 +1319,94 @@ class HttpCli(object):
             # print(abspath)
             raise Pebkac(404)
 
+        srv_info = []
+
+        try:
+            if not self.args.nih:
+                srv_info.append(unicode(socket.gethostname()).split(".")[0])
+        except:
+            self.log("#wow #whoa")
+
+        try:
+            # some fuses misbehave
+            if not self.args.nid:
+                if WINDOWS:
+                    bfree = ctypes.c_ulonglong(0)
+                    ctypes.windll.kernel32.GetDiskFreeSpaceExW(
+                        ctypes.c_wchar_p(abspath), None, None, ctypes.pointer(bfree)
+                    )
+                    srv_info.append(humansize(bfree.value) + " free")
+                else:
+                    sv = os.statvfs(abspath)
+                    free = humansize(sv.f_frsize * sv.f_bfree, True)
+                    total = humansize(sv.f_frsize * sv.f_blocks, True)
+
+                    srv_info.append(free + " free")
+                    srv_info.append(total)
+        except:
+            pass
+
+        srv_info = "</span> /// <span>".join(srv_info)
+
+        perms = []
+        if self.readable:
+            perms.append("read")
+        if self.writable:
+            perms.append("write")
+
+        url_suf = self.urlq()
+        is_ls = "ls" in self.uparam
+        ts = ""  # "?{}".format(time.time())
+
+        tpl = "browser"
+        if "b" in self.uparam:
+            tpl = "browser2"
+
+        logues = ["", ""]
+        for n, fn in enumerate([".prologue.html", ".epilogue.html"]):
+            fn = os.path.join(abspath, fn)
+            if os.path.exists(fsenc(fn)):
+                with open(fsenc(fn), "rb") as f:
+                    logues[n] = f.read().decode("utf-8")
+
+        ls_ret = {
+            "dirs": [],
+            "files": [],
+            "taglist": [],
+            "srvinf": srv_info,
+            "perms": perms,
+            "logues": logues,
+        }
+        j2a = {
+            "vdir": quotep(self.vpath),
+            "vpnodes": vpnodes,
+            "files": [],
+            "ts": ts,
+            "perms": json.dumps(perms),
+            "taglist": [],
+            "tag_order": [],
+            "have_up2k_idx": ("e2d" in vn.flags),
+            "have_tags_idx": ("e2t" in vn.flags),
+            "have_zip": (not self.args.no_zip),
+            "have_b_u": (self.writable and self.uparam.get("b") == "u"),
+            "url_suf": url_suf,
+            "logues": logues,
+            "title": html_escape(self.vpath, crlf=True),
+            "srv_info": srv_info,
+        }
+        if not self.readable:
+            if is_ls:
+                ret = json.dumps(ls_ret)
+                self.reply(ret.encode("utf-8", "replace"), mime="application/json")
+                return True
+
+            if not os.path.isdir(fsenc(abspath)):
+                raise Pebkac(404)
+
+            html = self.j2(tpl, **j2a)
+            self.reply(html.encode("utf-8", "replace"))
+            return True
+
         if not os.path.isdir(fsenc(abspath)):
             if abspath.endswith(".md") and "raw" not in self.uparam:
                 return self.tx_md(abspath)
@@ -1354,15 +1450,11 @@ class HttpCli(object):
         if rem == ".hist":
             hidden = ["up2k."]
 
-        is_ls = "ls" in self.uparam
-
         icur = None
         if "e2t" in vn.flags:
             idx = self.conn.get_u2idx()
             icur = idx.get_cur(vn.realpath)
 
-        url_suf = self.urlq()
-
         dirs = []
         files = []
         for fn in vfs_ls:
@@ -1394,7 +1486,7 @@ class HttpCli(object):
                     margin = '<a href="{}?zip">zip</a>'.format(quotep(href))
             elif fn in hist:
                 margin = '<a href="{}.hist/{}">#{}</a>'.format(
-                    base, html_escape(hist[fn][2], quote=True), hist[fn][0]
+                    base, html_escape(hist[fn][2], quote=True, crlf=True), hist[fn][0]
                 )
             else:
                 margin = "-"
@@ -1453,91 +1545,21 @@ class HttpCli(object):
             for f in dirs:
                 f["tags"] = {}
 
-        srv_info = []
-
-        try:
-            if not self.args.nih:
-                srv_info.append(unicode(socket.gethostname()).split(".")[0])
-        except:
-            self.log("#wow #whoa")
-            pass
-
-        try:
-            # some fuses misbehave
-            if not self.args.nid:
-                if WINDOWS:
-                    bfree = ctypes.c_ulonglong(0)
-                    ctypes.windll.kernel32.GetDiskFreeSpaceExW(
-                        ctypes.c_wchar_p(abspath), None, None, ctypes.pointer(bfree)
-                    )
-                    srv_info.append(humansize(bfree.value) + " free")
-                else:
-                    sv = os.statvfs(abspath)
-                    free = humansize(sv.f_frsize * sv.f_bfree, True)
-                    total = humansize(sv.f_frsize * sv.f_blocks, True)
-
-                    srv_info.append(free + " free")
-                    srv_info.append(total)
-        except:
-            pass
-
-        srv_info = "</span> /// <span>".join(srv_info)
-
-        perms = []
-        if self.readable:
-            perms.append("read")
-        if self.writable:
-            perms.append("write")
-
-        logues = ["", ""]
-        for n, fn in enumerate([".prologue.html", ".epilogue.html"]):
-            fn = os.path.join(abspath, fn)
-            if os.path.exists(fsenc(fn)):
-                with open(fsenc(fn), "rb") as f:
-                    logues[n] = f.read().decode("utf-8")
-
         if is_ls:
             [x.pop(k) for k in ["name", "dt"] for y in [dirs, files] for x in y]
-            ret = {
-                "dirs": dirs,
-                "files": files,
-                "srvinf": srv_info,
-                "perms": perms,
-                "logues": logues,
-                "taglist": taglist,
-            }
-            ret = json.dumps(ret)
+            ls_ret["dirs"] = dirs
+            ls_ret["files"] = files
+            ls_ret["taglist"] = taglist
+            ret = json.dumps(ls_ret)
             self.reply(ret.encode("utf-8", "replace"), mime="application/json")
             return True
 
-        ts = ""
-        # ts = "?{}".format(time.time())
+        j2a["files"] = dirs + files
+        j2a["logues"] = logues
+        j2a["taglist"] = taglist
+        if "mte" in vn.flags:
+            j2a["tag_order"] = json.dumps(vn.flags["mte"].split(","))
 
-        dirs.extend(files)
-
-        tpl = "browser"
-        if "b" in self.uparam:
-            tpl = "browser2"
-
-        html = self.j2(
-            tpl,
-            vdir=quotep(self.vpath),
-            vpnodes=vpnodes,
-            files=dirs,
-            ts=ts,
-            perms=json.dumps(perms),
-            taglist=taglist,
-            tag_order=json.dumps(
-                vn.flags["mte"].split(",") if "mte" in vn.flags else []
-            ),
-            have_up2k_idx=("e2d" in vn.flags),
-            have_tags_idx=("e2t" in vn.flags),
-            have_zip=(not self.args.no_zip),
-            have_b_u=(self.writable and self.uparam.get("b") == "u"),
-            url_suf=url_suf,
-            logues=logues,
-            title=html_escape(self.vpath),
-            srv_info=srv_info,
-        )
+        html = self.j2(tpl, **j2a)
         self.reply(html.encode("utf-8", "replace"))
         return True

copyparty/szip.py

@@ -87,7 +87,7 @@ def gen_hdr(h_pos, fn, sz, lastmod, utf8, crc32, pre_crc):
     ret += struct.pack("<LL", vsz, vsz)
 
     # windows support (the "?" replace below too)
-    fn = sanitize_fn(fn, "/")
+    fn = sanitize_fn(fn, ok="/")
     bfn = fn.encode("utf-8" if utf8 else "cp437", "replace").replace(b"?", b"_")
 
     z64_len = len(z64v) * 8 + 4 if z64v else 0

copyparty/up2k.py

@@ -16,7 +16,7 @@ import traceback
 import subprocess as sp
 from copy import deepcopy
 
-from .__init__ import WINDOWS
+from .__init__ import WINDOWS, ANYWIN
 from .util import (
     Pebkac,
     Queue,
@@ -79,7 +79,7 @@ class Up2k(object):
             if self.sqlite_ver < (3, 9):
                 self.no_expr_idx = True
 
-        if WINDOWS:
+        if ANYWIN:
             # usually fails to set lastmod too quickly
             self.lastmod_q = Queue()
             thr = threading.Thread(target=self._lastmodder)
@@ -101,17 +101,18 @@ class Up2k(object):
             thr.daemon = True
             thr.start()
 
-            thr = threading.Thread(target=self._tagger)
-            thr.daemon = True
-            thr.start()
-
             thr = threading.Thread(target=self._hasher)
             thr.daemon = True
             thr.start()
 
-            thr = threading.Thread(target=self._run_all_mtp)
-            thr.daemon = True
-            thr.start()
+            if self.mtag:
+                thr = threading.Thread(target=self._tagger)
+                thr.daemon = True
+                thr.start()
+
+                thr = threading.Thread(target=self._run_all_mtp)
+                thr.daemon = True
+                thr.start()
 
     def log(self, msg, c=0):
         self.log_func("up2k", msg + "\033[K", c)
@@ -667,12 +668,6 @@ class Up2k(object):
             cur.close()
 
     def _start_mpool(self):
-        if WINDOWS and False:
-            nah = open(os.devnull, "wb")
-            wmic = "processid={}".format(os.getpid())
-            wmic = ["wmic", "process", "where", wmic, "call", "setpriority"]
-            sp.call(wmic + ["below normal"], stdout=nah, stderr=nah)
-
         # mp.pool.ThreadPool and concurrent.futures.ThreadPoolExecutor
         # both do crazy runahead so lets reinvent another wheel
         nw = os.cpu_count() if hasattr(os, "cpu_count") else 4
@@ -697,12 +692,6 @@ class Up2k(object):
 
         mpool.join()
         done = self._flush_mpool(wcur)
-        if WINDOWS and False:
-            nah = open(os.devnull, "wb")
-            wmic = "processid={}".format(os.getpid())
-            wmic = ["wmic", "process", "where", wmic, "call", "setpriority"]
-            sp.call(wmic + ["below normal"], stdout=nah, stderr=nah)
-
         return done
 
     def _tag_thr(self, q):
@@ -902,7 +891,7 @@ class Up2k(object):
             if cj["ptop"] not in self.registry:
                 raise Pebkac(410, "location unavailable")
 
-        cj["name"] = sanitize_fn(cj["name"])
+        cj["name"] = sanitize_fn(cj["name"], bad=[".prologue.html", ".epilogue.html"])
         cj["poke"] = time.time()
         wark = self._get_wark(cj)
         now = time.time()
@@ -1109,8 +1098,9 @@ class Up2k(object):
 
             atomic_move(src, dst)
 
-            if WINDOWS:
-                self.lastmod_q.put([dst, (int(time.time()), int(job["lmod"]))])
+            if ANYWIN:
+                a = [dst, job["size"], (int(time.time()), int(job["lmod"]))]
+                self.lastmod_q.put(a)
 
             # legit api sware 2 me mum
             if self.idx_wark(
@@ -1211,6 +1201,17 @@ class Up2k(object):
         suffix = ".{:.6f}-{}".format(job["t0"], job["addr"])
         with ren_open(tnam, "wb", fdir=pdir, suffix=suffix) as f:
             f, job["tnam"] = f["orz"]
+            if (
+                ANYWIN
+                and self.args.sparse
+                and self.args.sparse * 1024 * 1024 <= job["size"]
+            ):
+                fp = os.path.join(pdir, job["tnam"])
+                try:
+                    sp.check_call(["fsutil", "sparse", "setflag", fp])
+                except:
+                    self.log("could not sparse [{}]".format(fp), 3)
+
             f.seek(job["size"] - 1)
             f.write(b"e")
 
@@ -1222,13 +1223,19 @@ class Up2k(object):
 
             # self.log("lmod: got {}".format(len(ready)))
             time.sleep(5)
-            for path, times in ready:
+            for path, sz, times in ready:
                 self.log("lmod: setting times {} on {}".format(times, path))
                 try:
                     os.utime(fsenc(path), times)
                 except:
                     self.log("lmod: failed to utime ({}, {})".format(path, times))
 
+                if self.args.sparse and self.args.sparse * 1024 * 1024 <= sz:
+                    try:
+                        sp.check_call(["fsutil", "sparse", "setflag", path, "0"])
+                    except:
+                        self.log("could not unsparse [{}]".format(path), 3)
+
     def _snapshot(self):
         persist_interval = 30  # persist unfinished uploads index every 30 sec
         discard_interval = 21600  # drop unfinished uploads after 6 hours inactivity

copyparty/util.py

@@ -16,7 +16,7 @@ import mimetypes
 import contextlib
 import subprocess as sp  # nosec
 
-from .__init__ import PY2, WINDOWS
+from .__init__ import PY2, WINDOWS, ANYWIN
 from .stolen import surrogateescape
 
 FAKE_MP = False
@@ -576,12 +576,12 @@ def undot(path):
     return "/".join(ret)
 
 
-def sanitize_fn(fn, ok=""):
+def sanitize_fn(fn, ok="", bad=[]):
     if "/" not in ok:
         fn = fn.replace("\\", "/").split("/")[-1]
 
-    if WINDOWS:
-        for bad, good in [x for x in [
+    if ANYWIN:
+        remap = [
             ["<", "＜"],
             [">", "＞"],
             [":", "："],
@@ -591,15 +591,16 @@ def sanitize_fn(fn, ok=""):
             ["|", "｜"],
             ["?", "？"],
             ["*", "＊"],
-        ] if x[0] not in ok]:
+        ]
+        for bad, good in [x for x in remap if x[0] not in ok]:
             fn = fn.replace(bad, good)
 
-        bad = ["con", "prn", "aux", "nul"]
+        bad.extend(["con", "prn", "aux", "nul"])
         for n in range(1, 10):
             bad += "com{0} lpt{0}".format(n).split(" ")
 
-        if fn.lower() in bad:
-            fn = "_" + fn
+    if fn.lower() in bad:
+        fn = "_" + fn
 
     return fn.strip()
 
@@ -615,17 +616,24 @@ def exclude_dotfiles(filepaths):
     return [x for x in filepaths if not x.split("/")[-1].startswith(".")]
 
 
-def html_escape(s, quote=False):
+def html_escape(s, quote=False, crlf=False):
     """html.escape but also newlines"""
-    s = (
-        s.replace("&", "&amp;")
-        .replace("<", "&lt;")
-        .replace(">", "&gt;")
-        .replace("\r", "&#13;")
-        .replace("\n", "&#10;")
-    )
+    s = s.replace("&", "&amp;").replace("<", "&lt;").replace(">", "&gt;")
     if quote:
         s = s.replace('"', "&quot;").replace("'", "&#x27;")
+    if crlf:
+        s = s.replace("\r", "&#13;").replace("\n", "&#10;")
+
+    return s
+
+
+def html_bescape(s, quote=False, crlf=False):
+    """html.escape but bytestrings"""
+    s = s.replace(b"&", b"&amp;").replace(b"<", b"&lt;").replace(b">", b"&gt;")
+    if quote:
+        s = s.replace(b'"', b"&quot;").replace(b"'", b"&#x27;")
+    if crlf:
+        s = s.replace(b"\r", b"&#13;").replace(b"\n", b"&#10;")
 
     return s
 

copyparty/web/browser.css

@@ -6,7 +6,7 @@ html,body,tr,th,td,#files,a {
 	background: none;
 	font-weight: inherit;
 	font-size: inherit;
-	padding: none;
+	padding: 0;
 	border: none;
 }
 html {
@@ -68,7 +68,7 @@ a, #files tbody div a:last-child {
 	color: #999;
 	font-weight: normal;
 }
-#files tr+tr:hover {
+#files tr:hover {
 	background: #1c1c1c;
 }
 #files thead th {
@@ -90,8 +90,6 @@ a, #files tbody div a:last-child {
 #files td {
 	margin: 0;
 	padding: 0 .5em;
-}
-#files td {
 	border-bottom: 1px solid #111;
 }
 #files td+td+td {
@@ -187,6 +185,16 @@ a, #files tbody div a:last-child {
 	background: #925;
 	border-color: #c37;
 }
+#files tr.sel a {
+	color: #fff;
+}
+#files tr.sel a.play {
+	color: #fc5;
+}
+#files tr.sel a.play.act {
+	color: #fff;
+	text-shadow: 0 0 1px #fff;
+}
 #blocked {
 	position: fixed;
 	top: 0;
@@ -284,7 +292,7 @@ a, #files tbody div a:last-child {
 	line-height: 1em;
 }
 #wtoggle.sel {
-	width: 6em;
+	width: 6.4em;
 }
 #wtoggle.sel #wzip {
 	display: inline-block;
@@ -343,10 +351,10 @@ a, #files tbody div a:last-child {
 	width: calc(100% - 10.5em);
 	background: rgba(0,0,0,0.2);
 }
-@media (min-width: 90em) {
+@media (min-width: 80em) {
 	#barpos,
 	#barbuf {
-		width: calc(100% - 24em);
+		width: calc(100% - 21em);
 		left: 9.8em;
 		top: .7em;
 		height: 1.6em;
@@ -356,6 +364,9 @@ a, #files tbody div a:last-child {
 		bottom: -3.2em;
 		height: 3.2em;
 	}
+	#pvol {
+		max-width: 9em;
+	}
 }
 
 
@@ -487,9 +498,6 @@ input[type="checkbox"]:checked+label {
 	border-collapse: collapse;
 	width: 100%;
 }
-#files td div a:last-child {
-	width: 100%;
-}
 #wrap {
 	margin-top: 2em;
 }
@@ -508,9 +516,7 @@ input[type="checkbox"]:checked+label {
 #thx_ff {
 	padding: 5em 0;
 }
-#tree::-webkit-scrollbar-track {
-	background: #333;
-}
+#tree::-webkit-scrollbar-track,
 #tree::-webkit-scrollbar {
 	background: #333;
 }
@@ -685,3 +691,199 @@ input[type="checkbox"]:checked+label {
 	font-family: monospace, monospace;
 	line-height: 2em;
 }
+#pvol,
+#barbuf,
+#barpos,
+#u2conf label {
+	-webkit-user-select: none;
+	-moz-user-select: none;
+	-ms-user-select: none;
+	user-select: none;
+}
+
+
+
+
+
+html.light {
+	color: #333;
+	background: #eee;
+	text-shadow: none;
+}
+html.light #ops,
+html.light .opbox,
+html.light #srch_form {
+	background: #f7f7f7;
+	box-shadow: 0 0 .3em #ddd;
+	border-color: #f7f7f7;
+}
+html.light #ops a.act {
+	box-shadow: 0 .2em .2em #ccc;
+	background: #fff;
+	border-color: #07a;
+	padding-top: .4em;
+}
+html.light #op_cfg h3 {
+	border-color: #ccc;
+}
+html.light .tglbtn,
+html.light #tree > a + a {
+	color: #666;
+	background: #ddd;
+	box-shadow: none;
+}
+html.light .tglbtn:hover,
+html.light #tree > a + a:hover {
+	background: #caf;
+}
+html.light .tglbtn.on,
+html.light #tree > a + a.on {
+	background: #4a0;
+	color: #fff;
+}
+html.light #srv_info {
+	color: #c83;
+	text-shadow: 1px 1px 0 #fff;
+}
+html.light #srv_info span {
+	color: #000;
+}
+html.light #treeul a+a {
+	background: inherit;
+	color: #06a;
+}
+html.light #treeul a.hl {
+	background: #07a;
+	color: #fff;
+}
+html.light #tree li {
+	border-color: #ddd #fff #f7f7f7 #fff;
+}
+html.light #tree ul {
+	border-color: #ccc;
+}
+html.light a,
+html.light #ops a,
+html.light #files tbody div a:last-child {
+	color: #06a;
+}
+html.light #files tbody {
+	background: #f7f7f7;
+}
+html.light #files {
+	box-shadow: 0 0 .3em #ccc;
+}
+html.light #files thead th {
+	background: #eee;
+}
+html.light #files tr td {
+	border-top: 1px solid #ddd;
+}
+html.light #files td {
+	border-bottom: 1px solid #f7f7f7;
+}
+html.light #files tbody tr:last-child td {
+	border-bottom: .2em solid #ccc;
+}
+html.light #files td:nth-child(2n) {
+	color: #d38;
+}
+html.light #files tr:hover td {
+	background: #fff;
+}
+html.light #files tbody a.play {
+	color: #c0f;
+}
+html.light tr.play td {
+	background: #fc5;
+}
+html.light tr.play a {
+	color: #406;
+}
+html.light #files th:hover .cfg,
+html.light #files th.min .cfg {
+	background: #ccc;
+}
+html.light #files > thead > tr > th.min span {
+	background: linear-gradient(90deg, rgba(204,204,204,0), rgba(204,204,204,0.5) 70%, #ccc);
+}
+html.light #blocked {
+	background: #eee;
+}
+html.light #blk_play a,
+html.light #blk_abrt a {
+	background: #fff;
+	box-shadow: 0 .2em .4em #ddd;
+}
+html.light #widget a {
+	color: #fc5;
+}
+html.light #files tr.sel:hover td {
+	background: #c37;
+}
+html.light #files tr.sel td {
+	color: #fff;
+}
+html.light #files tr.sel a {
+	color: #fff;
+}
+html.light #files tr.sel a.play.act {
+	color: #fb0;
+}
+html.light input[type="checkbox"] + label {
+	color: #333;
+}
+html.light .opview input[type="text"] {
+	background: #fff;
+	color: #333;
+	box-shadow: 0 0 2px #888;
+	border-color: #38d;
+}
+html.light #ops:hover #opdesc {
+	background: #fff;
+	box-shadow: 0 .3em 1em #ccc;
+}
+html.light #opdesc code {
+	background: #060;
+	color: #fff;
+}
+html.light #u2tab a>span,
+html.light #files td div span {
+	color: #000;
+}
+html.light #path {
+	background: #f7f7f7;
+	text-shadow: none;
+	box-shadow: 0 0 .3em #bbb;
+}
+html.light #path a {
+	color: #333;
+}
+html.light #path a:not(:last-child)::after {
+	border-color: #ccc;
+	background: none;
+	border-width: .1em .1em 0 0;
+	margin: -.2em .3em -.2em -.3em;
+}
+html.light #path a:hover {
+	background: none;
+	color: #60a;
+}
+html.light #files tbody div a {
+	color: #d38;
+}
+html.light #files a:hover,
+html.light #files tr.sel a:hover {
+	color: #000;
+	background: #fff;
+}
+html.light #tree {
+	scrollbar-color: #a70 #ddd;
+}
+html.light #tree::-webkit-scrollbar-track,
+html.light #tree::-webkit-scrollbar {
+	background: #ddd;
+}
+#tree::-webkit-scrollbar-thumb {
+	background: #da0;
+}

copyparty/web/browser.html

@@ -39,14 +39,17 @@
     {%- include 'upload.html' %}
 
     <div id="op_cfg" class="opview opbox">
-        <h3>key notation</h3>
-        <div id="key_notation"></div>
+        <h3>switches</h3>
+        <div>
+            <a id="tooltips" class="tglbtn" href="#">tooltips</a>
+            <a id="lightmode" class="tglbtn" href="#">lightmode</a>
+        </div>
         {%- if have_zip %}
         <h3>folder download</h3>
         <div id="arc_fmt"></div>
         {%- endif %}
-        <h3>tooltips</h3>
-        <div><a id="tooltips" class="tglbtn" href="#">enable</a></div>
+        <h3>key notation</h3>
+        <div id="key_notation"></div>
     </div>
     
     <h1 id="path">

copyparty/web/md.css

@@ -50,6 +50,9 @@ pre code:last-child {
 pre code::before {
 	content: counter(precode);
 	-webkit-user-select: none;
+	-moz-user-select: none;
+	-ms-user-select: none;
+	user-select: none;
 	display: inline-block;
 	text-align: right;
 	font-size: .75em;
@@ -591,12 +594,3 @@ blink {
 		color: #940;
 	}
 }
-
-/*
-*[data-ln]:before {
-	content: attr(data-ln);
-	font-size: .8em;
-	margin: 0 .4em;
-	color: #f0c;
-}
-*/

copyparty/web/md.html

@@ -138,10 +138,10 @@ var md_opt = {
         document.documentElement.setAttribute("class", dark ? "dark" : "");
         btn.innerHTML = "go " + (dark ? "light" : "dark");
         if (window.localStorage)
-            localStorage.setItem('darkmode', dark ? 1 : 0);
+            localStorage.setItem('lightmode', dark ? 0 : 1);
     };
     btn.onclick = toggle;
-    if (window.localStorage && localStorage.getItem('darkmode') == 1)
+    if (window.localStorage && localStorage.getItem('lightmode') != 1)
 		toggle();
 })();
 

copyparty/web/md.js

@@ -46,7 +46,7 @@ function statify(obj) {
     var ua = navigator.userAgent;
     if (ua.indexOf(') Gecko/') !== -1 && /Linux| Mac /.exec(ua)) {
         // necessary on ff-68.7 at least
-        var s = document.createElement('style');
+        var s = mknod('style');
         s.innerHTML = '@page { margin: .5in .6in .8in .6in; }';
         console.log(s.innerHTML);
         document.head.appendChild(s);
@@ -175,12 +175,12 @@ function md_plug_err(ex, js) {
         msg = "Line " + ln + ", " + msg;
         var lns = js.split('\n');
         if (ln < lns.length) {
-            o = document.createElement('span');
+            o = mknod('span');
             o.style.cssText = 'color:#ac2;font-size:.9em;font-family:scp;display:block';
             o.textContent = lns[ln - 1];
         }
     }
-    errbox = document.createElement('div');
+    errbox = mknod('div');
     errbox.setAttribute('id', 'md_errbox');
     errbox.style.cssText = 'position:absolute;top:0;left:0;padding:1em .5em;background:#2b2b2b;color:#fc5'
     errbox.textContent = msg;

copyparty/web/md2.css

@@ -1,126 +1,125 @@
 #toc {
-    display: none;
+	display: none;
 }
 #mtw {
-    display: block;
-    position: fixed;
-    left: .5em;
-    bottom: 0;
-    width: calc(100% - 56em);
+	display: block;
+	position: fixed;
+	left: .5em;
+	bottom: 0;
+	width: calc(100% - 56em);
 }
 #mw {
-    left: calc(100% - 55em);
-    overflow-y: auto;
-    position: fixed;
-    bottom: 0;
+	left: calc(100% - 55em);
+	overflow-y: auto;
+	position: fixed;
+	bottom: 0;
 }
 
 
 /* single-screen */
 #mtw.preview,
 #mw.editor {
-    opacity: 0;
-    z-index: 1;
+	opacity: 0;
+	z-index: 1;
 }
 #mw.preview,
 #mtw.editor {
-    z-index: 5;
+	z-index: 5;
 }
 #mtw.single,
 #mw.single {
-    margin: 0;
-    left: 1em;
-    left: max(1em, calc((100% - 56em) / 2));
+	margin: 0;
+	left: 1em;
+	left: max(1em, calc((100% - 56em) / 2));
 }
 #mtw.single {
-    width: 55em;
-    width: min(55em, calc(100% - 2em));
+	width: 55em;
+	width: min(55em, calc(100% - 2em));
 }
 
 
 #mp {
-    position: relative;
+	position: relative;
 }
 #mt, #mtr {
-    width: 100%;
-    height: calc(100% - 1px);
-    color: #444;
-    background: #f7f7f7;
-    border: 1px solid #999;
-    outline: none;
-    padding: 0;
-    margin: 0;
-    font-family: 'consolas', monospace, monospace;
-    white-space: pre-wrap;
-    word-break: break-word;
-    overflow-wrap: break-word;
-    word-wrap: break-word; /*ie*/
-    overflow-y: scroll;
-    line-height: 1.3em;
-    font-size: .9em;
-    position: relative;
-    scrollbar-color: #eb0 #f7f7f7;
+	width: 100%;
+	height: calc(100% - 1px);
+	color: #444;
+	background: #f7f7f7;
+	border: 1px solid #999;
+	outline: none;
+	padding: 0;
+	margin: 0;
+	font-family: 'consolas', monospace, monospace;
+	white-space: pre-wrap;
+	word-break: break-word;
+	overflow-wrap: break-word;
+	word-wrap: break-word; /*ie*/
+	overflow-y: scroll;
+	line-height: 1.3em;
+	font-size: .9em;
+	position: relative;
+	scrollbar-color: #eb0 #f7f7f7;
 }
 html.dark #mt {
-    color: #eee;
-    background: #222;
-    border: 1px solid #777;
-    scrollbar-color: #b80 #282828;
+	color: #eee;
+	background: #222;
+	border: 1px solid #777;
+	scrollbar-color: #b80 #282828;
 }
 #mtr {
-    position: absolute;
-    top: 0;
-    left: 0;
+	position: absolute;
+	top: 0;
+	left: 0;
 }
 #save.force-save {
-    color: #400;
-    background: #f97;
-    border-radius: .15em;
+	color: #400;
+	background: #f97;
+	border-radius: .15em;
 }
 html.dark #save.force-save {
-    color: #fca;
-    background: #720;
+	color: #fca;
+	background: #720;
 }
 #save.disabled {
-    opacity: .4;
+	opacity: .4;
 }
 #helpbox,
 #toast {
-    background: #f7f7f7;
-    border-radius: .4em;
-    z-index: 9001;
+	background: #f7f7f7;
+	border-radius: .4em;
+	z-index: 9001;
 }
 #helpbox {
-    display: none;
-    position: fixed;
-    padding: 2em;
-    top: 4em;
-    overflow-y: auto;
-    box-shadow: 0 .5em 2em #777;
-    height: calc(100% - 12em);
-    left: calc(50% - 15em);
-    right: 0;
-    width: 30em;
+	display: none;
+	position: fixed;
+	padding: 2em;
+	top: 4em;
+	overflow-y: auto;
+	box-shadow: 0 .5em 2em #777;
+	height: calc(100% - 12em);
+	left: calc(50% - 15em);
+	right: 0;
+	width: 30em;
 }
 #helpclose {
-    display: block;
+	display: block;
 }
 html.dark #helpbox {
-    box-shadow: 0 .5em 2em #444;
+	box-shadow: 0 .5em 2em #444;
 }
 html.dark #helpbox,
 html.dark #toast {
-    background: #222;
-    border: 1px solid #079;
-    border-width: 1px 0;
+	background: #222;
+	border: 1px solid #079;
+	border-width: 1px 0;
 }
 #toast {
-    font-weight: bold;
-    text-align: center;
-    padding: .6em 0;
-    position: fixed;
-    z-index: 9001;
-    top: 30%;
-    transition: opacity 0.2s ease-in-out;
-    opacity: 1;
+	font-weight: bold;
+	text-align: center;
+	padding: .6em 0;
+	position: fixed;
+	top: 30%;
+	transition: opacity 0.2s ease-in-out;
+	opacity: 1;
 }

copyparty/web/md2.js

@@ -16,7 +16,7 @@ var dom_sbs = ebi('sbs');
 var dom_nsbs = ebi('nsbs');
 var dom_tbox = ebi('toolsbox');
 var dom_ref = (function () {
-    var d = document.createElement('div');
+    var d = mknod('div');
     d.setAttribute('id', 'mtr');
     dom_swrap.appendChild(d);
     d = ebi('mtr');
@@ -71,7 +71,7 @@ var map_src = [];
 var map_pre = [];
 function genmap(dom, oldmap) {
     var find = nlines;
-    while (oldmap && find --> 0) {
+    while (oldmap && find-- > 0) {
         var tmap = genmapq(dom, '*[data-ln="' + find + '"]');
         if (!tmap || !tmap.length)
             continue;
@@ -94,7 +94,7 @@ var nlines = 0;
 var draw_md = (function () {
     var delay = 1;
     function draw_md() {
-        var t0 = new Date().getTime();
+        var t0 = Date.now();
         var src = dom_src.value;
         convert_markdown(src, dom_pre);
 
@@ -110,7 +110,7 @@ var draw_md = (function () {
 
         cls(ebi('save'), 'disabled', src == server_md);
 
-        var t1 = new Date().getTime();
+        var t1 = Date.now();
         delay = t1 - t0 > 100 ? 25 : 1;
     }
 
@@ -252,7 +252,7 @@ function Modpoll() {
         }
 
         console.log('modpoll...');
-        var url = (document.location + '').split('?')[0] + '?raw&_=' + new Date().getTime();
+        var url = (document.location + '').split('?')[0] + '?raw&_=' + Date.now();
         var xhr = new XMLHttpRequest();
         xhr.modpoll = this;
         xhr.open('GET', url, true);
@@ -399,7 +399,7 @@ function save_cb() {
 
 function run_savechk(lastmod, txt, btn, ntry) {
     // download the saved doc from the server and compare
-    var url = (document.location + '').split('?')[0] + '?raw&_=' + new Date().getTime();
+    var url = (document.location + '').split('?')[0] + '?raw&_=' + Date.now();
     var xhr = new XMLHttpRequest();
     xhr.open('GET', url, true);
     xhr.responseType = 'text';
@@ -455,7 +455,7 @@ function toast(autoclose, style, width, msg) {
         ok.parentNode.removeChild(ok);
 
     style = "width:" + width + "em;left:calc(50% - " + (width / 2) + "em);" + style;
-    ok = document.createElement('div');
+    ok = mknod('div');
     ok.setAttribute('id', 'toast');
     ok.setAttribute('style', style);
     ok.innerHTML = msg;
@@ -1049,7 +1049,7 @@ action_stack = (function () {
         var p1 = from.length,
             p2 = to.length;
 
-        while (p1 --> 0 && p2 --> 0)
+        while (p1-- > 0 && p2-- > 0)
             if (from[p1] != to[p2])
                 break;
 
@@ -1142,14 +1142,3 @@ action_stack = (function () {
         _ref: ref
     }
 })();
-
-/*
-ebi('help').onclick = function () {
-    var c1 = getComputedStyle(dom_src).cssText.split(';');
-    var c2 = getComputedStyle(dom_ref).cssText.split(';');
-    var max = Math.min(c1.length, c2.length);
-    for (var a = 0; a < max; a++)
-        if (c1[a] !== c2[a])
-            console.log(c1[a] + '\n' + c2[a]);
-}
-*/

copyparty/web/mde.css

@@ -8,68 +8,58 @@ html .editor-toolbar>i.separator { border-left: 1px solid #ccc; }
 html .editor-toolbar.disabled-for-preview>button:not(.no-disable) { opacity: .35 }
 
 html {
-    line-height: 1.5em;
+	line-height: 1.5em;
 }
 html, body {
-    margin: 0;
-    padding: 0;
-    min-height: 100%;
-    font-family: sans-serif;
-    background: #f7f7f7;
-    color: #333;
+	margin: 0;
+	padding: 0;
+	min-height: 100%;
+	font-family: sans-serif;
+	background: #f7f7f7;
+	color: #333;
 }
 #mn {
-    font-weight: normal;
-    margin: 1.3em 0 .7em 1em;
+	font-weight: normal;
+	margin: 1.3em 0 .7em 1em;
 }
 #mn a {
-    color: #444;
-    margin: 0 0 0 -.2em;
-    padding: 0 0 0 .4em;
-    text-decoration: none;
-    /* ie: */
-    border-bottom: .1em solid #777\9;
-    margin-right: 1em\9;
+	color: #444;
+	margin: 0 0 0 -.2em;
+	padding: 0 0 0 .4em;
+	text-decoration: none;
+	/* ie: */
+	border-bottom: .1em solid #777\9;
+	margin-right: 1em\9;
 }
 #mn a:first-child {
-    padding-left: .5em;
+	padding-left: .5em;
 }
 #mn a:last-child {
-    padding-right: .5em;
+	padding-right: .5em;
 }
 #mn a:not(:last-child):after {
-    content: '';
-    width: 1.05em;
-    height: 1.05em;
-    margin: -.2em .3em -.2em -.4em;
-    display: inline-block;
-    border: 1px solid rgba(0,0,0,0.2);
-    border-width: .2em .2em 0 0;
-    transform: rotate(45deg);
+	content: '';
+	width: 1.05em;
+	height: 1.05em;
+	margin: -.2em .3em -.2em -.4em;
+	display: inline-block;
+	border: 1px solid rgba(0,0,0,0.2);
+	border-width: .2em .2em 0 0;
+	transform: rotate(45deg);
 }
 #mn a:hover {
-    color: #000;
-    text-decoration: underline;
+	color: #000;
+	text-decoration: underline;
 }
 
 html .editor-toolbar>button.disabled {
-    opacity: .35;
-    pointer-events: none;
+	opacity: .35;
+	pointer-events: none;
 }
 html .editor-toolbar>button.save.force-save {
-    background: #f97;
+	background: #f97;
 }
 
-/*
-*[data-ln]:before {
-	content: attr(data-ln);
-	font-size: .8em;
-	margin: 0 .4em;
-	color: #f0c;
-}
-.cm-header { font-size: .4em !important }
-*/
-
 
 
 
@@ -101,29 +91,29 @@ html .editor-toolbar>button.save.force-save {
 	line-height: 1.1em;
 }
 .mdo a {
-    color: #fff;
-    background: #39b;
-    text-decoration: none;
-    padding: 0 .3em;
-    border: none;
-    border-bottom: .07em solid #079;
+	color: #fff;
+	background: #39b;
+	text-decoration: none;
+	padding: 0 .3em;
+	border: none;
+	border-bottom: .07em solid #079;
 }
 .mdo h2 {
-    color: #fff;
-    background: #555;
-    margin-top: 2em;
-    border-bottom: .22em solid #999;
-    border-top: none;
+	color: #fff;
+	background: #555;
+	margin-top: 2em;
+	border-bottom: .22em solid #999;
+	border-top: none;
 }
 .mdo h1 {
-    color: #fff;
-    background: #444;
-    font-weight: normal;
-    border-top: .4em solid #fb0;
-    border-bottom: .4em solid #777;
-    border-radius: 0 1em 0 1em;
-    margin: 3em 0 1em 0;
-    padding: .5em 0;
+	color: #fff;
+	background: #444;
+	font-weight: normal;
+	border-top: .4em solid #fb0;
+	border-bottom: .4em solid #777;
+	border-radius: 0 1em 0 1em;
+	margin: 3em 0 1em 0;
+	padding: .5em 0;
 }
 h1, h2 {
 	line-height: 1.5em;
@@ -197,14 +187,14 @@ th {
 
 /* mde support */
 .mdo {
-    padding: 1em;
-    background: #f7f7f7;
+	padding: 1em;
+	background: #f7f7f7;
 }
 html.dark .mdo {
-    background: #1c1c1c;
+	background: #1c1c1c;
 }
 .CodeMirror {
-    background: #f7f7f7;
+	background: #f7f7f7;
 }
 
 
@@ -214,108 +204,108 @@ html.dark .mdo {
 /* darkmode */
 html.dark .mdo,
 html.dark .CodeMirror {
-    border-color: #222;
+	border-color: #222;
 }
 html.dark,
 html.dark body,
 html.dark .CodeMirror {
-    background: #222;
-    color: #ccc;
+	background: #222;
+	color: #ccc;
 }
 html.dark .CodeMirror-cursor {
-    border-color: #fff;
+	border-color: #fff;
 }
 html.dark .CodeMirror-selected {
-    box-shadow: 0 0 1px #0cf inset;
+	box-shadow: 0 0 1px #0cf inset;
 }
 html.dark .CodeMirror-selected,
 html.dark .CodeMirror-selectedtext {
-    border-radius: .1em;
-    background: #246;
-    color: #fff;
+	border-radius: .1em;
+	background: #246;
+	color: #fff;
 }
 html.dark .mdo a {
-    background: #057;
+	background: #057;
 }
 html.dark .mdo h1 a, html.dark .mdo h4 a,
 html.dark .mdo h2 a, html.dark .mdo h5 a,
 html.dark .mdo h3 a, html.dark .mdo h6 a {
-    color: inherit;
-    background: none;
+	color: inherit;
+	background: none;
 }
 html.dark pre,
 html.dark code {
-    color: #8c0;
-    background: #1a1a1a;
-    border: .07em solid #333;
+	color: #8c0;
+	background: #1a1a1a;
+	border: .07em solid #333;
 }
 html.dark .mdo ul,
 html.dark .mdo ol {
-    border-color: #444;
+	border-color: #444;
 }
 html.dark .mdo>ul,
 html.dark .mdo>ol {
-    border-color: #555;
+	border-color: #555;
 }
 html.dark strong {
-    color: #fff;
+	color: #fff;
 }
 html.dark p>em,
 html.dark li>em,
 html.dark td>em {
-    color: #f94;
-    border-color: #666;
+	color: #f94;
+	border-color: #666;
 }
 html.dark h1 {
-    background: #383838;
-    border-top: .4em solid #b80;
-    border-bottom: .4em solid #4c4c4c;
+	background: #383838;
+	border-top: .4em solid #b80;
+	border-bottom: .4em solid #4c4c4c;
 }
 html.dark h2 {
-    background: #444;
-    border-bottom: .22em solid #555;
+	background: #444;
+	border-bottom: .22em solid #555;
 }
 html.dark td,
 html.dark th {
-    border-color: #444;
+	border-color: #444;
 }
 html.dark blockquote {
-    background: #282828;
-    border: .07em dashed #444;
+	background: #282828;
+	border: .07em dashed #444;
 }
 
 
 
 html.dark #mn a {
-    color: #ccc;
+	color: #ccc;
 }
 html.dark #mn a:not(:last-child):after {
-    border-color: rgba(255,255,255,0.3);
+	border-color: rgba(255,255,255,0.3);
 }
 html.dark .editor-toolbar {
-    border-color: #2c2c2c;
-    background: #1c1c1c;
+	border-color: #2c2c2c;
+	background: #1c1c1c;
 }
 html.dark .editor-toolbar>i.separator {
-    border-left: 1px solid #444;
-    border-right: 1px solid #111;
+	border-left: 1px solid #444;
+	border-right: 1px solid #111;
 }
 html.dark .editor-toolbar>button {
-    margin-left: -1px; border: 1px solid rgba(255,255,255,0.1);
-    color: #aaa;
+	margin-left: -1px; border: 1px solid rgba(255,255,255,0.1);
+	color: #aaa;
 }
 
 
 
 html.dark .editor-toolbar>button:hover {
-    color: #333;
+	color: #333;
 }
 html.dark .editor-toolbar>button.active {
-    color: #333;
-    border-color: #ec1;
-    background: #c90;
+	color: #333;
+	border-color: #ec1;
+	background: #c90;
 }
 html.dark .editor-toolbar::after,
 html.dark .editor-toolbar::before {
-    background: none;
+	background: none;
 }

copyparty/web/mde.html

@@ -31,12 +31,12 @@ var md_opt = {
 
 var lightswitch = (function () {
 	var fun = function () {
-		var dark = !!!document.documentElement.getAttribute("class");
+		var dark = !document.documentElement.getAttribute("class");
 		document.documentElement.setAttribute("class", dark ? "dark" : "");
 		if (window.localStorage)
-			localStorage.setItem('darkmode', dark ? 1 : 0);
+			localStorage.setItem('lightmode', dark ? 0 : 1);
 	};
-	if (window.localStorage && localStorage.getItem('darkmode') == 1)
+	if (window.localStorage && localStorage.getItem('lightmode') != 1)
 		fun();
 	
 	return fun;

copyparty/web/mde.js

@@ -71,7 +71,7 @@ var mde = (function () {
 })();
 
 function set_jumpto() {
-    document.querySelector('.editor-preview-side').onclick = jumpto;
+    QS('.editor-preview-side').onclick = jumpto;
 }
 
 function jumpto(ev) {
@@ -94,7 +94,7 @@ function md_changed(mde, on_srv) {
         window.md_saved = mde.value();
 
     var md_now = mde.value();
-    var save_btn = document.querySelector('.editor-toolbar button.save');
+    var save_btn = QS('.editor-toolbar button.save');
 
     if (md_now == window.md_saved)
         save_btn.classList.add('disabled');
@@ -105,7 +105,7 @@ function md_changed(mde, on_srv) {
 }
 
 function save(mde) {
-    var save_btn = document.querySelector('.editor-toolbar button.save');
+    var save_btn = QS('.editor-toolbar button.save');
     if (save_btn.classList.contains('disabled')) {
         alert('there is nothing to save');
         return;
@@ -212,7 +212,7 @@ function save_chk() {
     last_modified = this.lastmod;
     md_changed(this.mde, true);
 
-    var ok = document.createElement('div');
+    var ok = mknod('div');
     ok.setAttribute('style', 'font-size:6em;font-family:serif;font-weight:bold;color:#cf6;background:#444;border-radius:.3em;padding:.6em 0;position:fixed;top:30%;left:calc(50% - 2em);width:4em;text-align:center;z-index:9001;transition:opacity 0.2s ease-in-out;opacity:1');
     ok.innerHTML = 'OK✔️';
     var parent = ebi('m');

copyparty/web/msg.css

@@ -3,7 +3,7 @@ html,body,tr,th,td,#files,a {
 	background: none;
 	font-weight: inherit;
 	font-size: inherit;
-	padding: none;
+	padding: 0;
 	border: none;
 }
 html {
@@ -20,8 +20,8 @@ body {
 	padding-bottom: 5em;
 }
 #box {
-    padding: .5em 1em;
-    background: #2c2c2c;
+	padding: .5em 1em;
+	background: #2c2c2c;
 }
 pre {
 	font-family: monospace, monospace;

copyparty/web/splash.html

@@ -38,7 +38,7 @@
     </div>
     <script>
 
-if (window.localStorage && localStorage.getItem('darkmode') == 1)
+if (window.localStorage && localStorage.getItem('lightmode') != 1)
     document.documentElement.setAttribute("class", "dark");
 
 </script>

copyparty/web/up2k.js

@@ -55,7 +55,7 @@ function up2k_flagbus() {
             dbg(who, 'hi me (??)');
             return;
         }
-        flag.act = new Date().getTime();
+        flag.act = Date.now();
         if (what == "want") {
             // lowest id wins, don't care if that's us
             if (who < flag.id) {
@@ -141,7 +141,7 @@ function U2pvis(act, btns) {
     this.tail = -1;
     this.wsz = 3;
 
-    this.addfile = function (entry) {
+    this.addfile = function (entry, sz) {
         this.tab.push({
             "hn": entry[0],
             "ht": entry[1],
@@ -149,8 +149,10 @@ function U2pvis(act, btns) {
             "in": 'q',
             "nh": 0, //hashed
             "nd": 0, //done
-            "pa": [], //percents
-            "pb": []  //active-list
+            "cb": [], // bytes done in chunk
+            "bt": sz, // bytes total
+            "bd": 0,  // bytes done
+            "bd0": 0  // upload start
         });
         this.ctr["q"]++;
         this.drawcard("q");
@@ -172,6 +174,9 @@ function U2pvis(act, btns) {
     this.seth = function (nfile, field, html) {
         var fo = this.tab[nfile];
         field = ['hn', 'ht', 'hp'][field];
+        if (fo[field] === html)
+            return;
+
         fo[field] = html;
         if (!this.is_act(fo.in))
             return;
@@ -184,61 +189,62 @@ function U2pvis(act, btns) {
         }
     };
 
-    this.setab = function (nfile, blocks) {
+    this.setab = function (nfile, nblocks) {
         var t = [];
-        for (var a = 0; a < blocks; a++)
+        for (var a = 0; a < nblocks; a++)
             t.push(0);
 
-        this.tab[nfile].pa = t;
+        this.tab[nfile].cb = t;
     };
 
-    this.perc = function (n, t, e, sz, t0) {
-        var p = (n + e) * 100.0 / t,
-            td = new Date().getTime() - t0,
-            pp = (td / 1000) / p,
-            spd = (sz / 100) / pp,
-            eta = pp * (100 - p);
+    this.setat = function (nfile, blocktab) {
+        this.tab[nfile].cb = blocktab;
+
+        var bd = 0;
+        for (var a = 0; a < blocktab.length; a++)
+            bd += blocktab[a];
+
+        this.tab[nfile].bd = bd;
+        this.tab[nfile].bd0 = bd;
+    };
+
+    this.perc = function (bd, bd0, sz, t0) {
+        var td = Date.now() - t0,
+            p = bd * 100.0 / sz,
+            nb = bd - bd0,
+            spd = nb / (td / 1000),
+            eta = (sz - bd) / spd;
 
         return [p, s2ms(eta), spd / (1024 * 1024)];
     };
 
     this.hashed = function (fobj) {
-        var fo = this.tab[fobj.n];
-        fo.nh++;
-        var p = this.perc(fo.nh, fo.pa.length, 0, fobj.size, fobj.t1);
+        var fo = this.tab[fobj.n],
+            nb = fo.bt * (++fo.nh / fo.cb.length),
+            p = this.perc(nb, 0, fobj.size, fobj.t1);
+
         fo.hp = '{0}%, {1}, {2} MB/s'.format(
             p[0].toFixed(2), p[1], p[2].toFixed(2)
         );
         if (!this.is_act(fo.in))
             return;
 
-        var obj = ebi('f{0}p'.format(fobj.n));
+        var obj = ebi('f{0}p'.format(fobj.n)),
+            o1 = p[0] - 2, o2 = p[0] - 0.1, o3 = p[0];
+
         obj.innerHTML = fo.hp;
         obj.style.color = '#fff';
-        var o1 = p[0] - 2, o2 = p[0] - 0.1, o3 = p[0];
-        obj.style.background = 'linear-gradient(90deg, #025, #06a ' + o1 + '%, #08d ' + o2 + '%, #333 ' + o3 + '%)';
+        obj.style.background = 'linear-gradient(90deg, #025, #06a ' + o1 + '%, #09d ' + o2 + '%, #333 ' + o3 + '%, #333 99%, #777)';
     };
 
-    this.prog = function (fobj, nchunk, percent) {
-        var fo = this.tab[fobj.n], pb = fo.pb;
-        var i = pb.indexOf(nchunk);
-        fo.pa[nchunk] = percent;
-        if (percent == 101) {
-            fo.nd++;
-            if (i >= 0)
-                pb.splice(i);
-        }
-        else if (i == -1) {
-            pb.push(nchunk);
-        }
+    this.prog = function (fobj, nchunk, cbd) {
+        var fo = this.tab[fobj.n],
+            delta = cbd - fo.cb[nchunk];
 
-        var extra = 0;
-        for (var a = 0; a < pb.length; a++)
-            extra += fo.pa[a];
+        fo.cb[nchunk] = cbd;
+        fo.bd += delta;
 
-        extra /= fo.pa.length;
-
-        var p = this.perc(fo.nd, fo.pa.length, extra, fobj.size, fobj.t3);
+        var p = this.perc(fo.bd, fo.bd0, fo.bt, fobj.t3);
         fo.hp = '{0}%, {1}, {2} MB/s'.format(
             p[0].toFixed(2), p[1], p[2].toFixed(2)
         );
@@ -246,11 +252,12 @@ function U2pvis(act, btns) {
         if (!this.is_act(fo.in))
             return;
 
-        var obj = ebi('f{0}p'.format(fobj.n));
+        var obj = ebi('f{0}p'.format(fobj.n)),
+            o1 = p[0] - 2, o2 = p[0] - 0.1, o3 = p[0];
+
         obj.innerHTML = fo.hp;
         obj.style.color = '#fff';
-        var o1 = p[0] - 2, o2 = p[0] - 0.1, o3 = p[0];
-        obj.style.background = 'linear-gradient(90deg, #050, #270 ' + o1 + '%, #4b0 ' + o2 + '%, #333 ' + o3 + '%)';
+        obj.style.background = 'linear-gradient(90deg, #050, #270 ' + o1 + '%, #4b0 ' + o2 + '%, #333 ' + o3 + '%, #333 99%, #777)';
     };
 
     this.move = function (nfile, newcat) {
@@ -284,24 +291,14 @@ function U2pvis(act, btns) {
         }
     };
 
-    this.bzw_log = function (first, last) {
-        console.log("first %d   head %d   tail %d   last %d", first, this.head, this.tail, last);
-        var trs = document.querySelectorAll('#u2tab>tbody>tr'), msg = [];
-        for (var a = 0; a < trs.length; a++)
-            msg.push(trs[a].getAttribute('id'));
-
-        console.log(msg.join(' '));
-    }
-
     this.bzw = function () {
-        var first = document.querySelector('#u2tab>tbody>tr:first-child');
+        var first = QS('#u2tab>tbody>tr:first-child');
         if (!first)
             return;
 
-        var last = document.querySelector('#u2tab>tbody>tr:last-child');
+        var last = QS('#u2tab>tbody>tr:last-child');
         first = parseInt(first.getAttribute('id').slice(1));
         last = parseInt(last.getAttribute('id').slice(1));
-        //this.bzw_log(first, last);
 
         while (this.head - first > this.wsz) {
             var obj = ebi('f' + (first++));
@@ -312,12 +309,10 @@ function U2pvis(act, btns) {
             if (!obj)
                 this.addrow(last);
         }
-        //this.bzw_log(first, last);
-        //console.log('--');
     };
 
     this.drawcard = function (cat) {
-        var cards = document.querySelectorAll('#u2cards>a>span');
+        var cards = QSA('#u2cards>a>span');
 
         if (cat == "q") {
             cards[4].innerHTML = this.ctr[cat];
@@ -340,9 +335,9 @@ function U2pvis(act, btns) {
 
     this.changecard = function (card) {
         this.act = card;
-        var html = [];
         this.head = -1;
         this.tail = -1;
+        var html = [];
         for (var a = 0; a < this.tab.length; a++) {
             var rt = this.tab[a].in;
             if (this.is_act(rt)) {
@@ -379,7 +374,7 @@ function U2pvis(act, btns) {
         if (as_html)
             return '<tr id="f' + nfile + '">' + ret + '</tr>';
 
-        var obj = document.createElement('tr');
+        var obj = mknod('tr');
         obj.setAttribute('id', 'f' + nfile);
         obj.innerHTML = ret;
         return obj;
@@ -391,7 +386,7 @@ function U2pvis(act, btns) {
     };
 
     var that = this;
-    btns = document.querySelectorAll(btns + '>a[act]');
+    btns = QSA(btns + '>a[act]');
     for (var a = 0; a < btns.length; a++) {
         btns[a].onclick = function (e) {
             ev(e);
@@ -408,7 +403,6 @@ function U2pvis(act, btns) {
 
 function up2k_init(have_crypto) {
     //have_crypto = false;
-    var need_filereader_cache = undefined;
 
     // show modal message
     function showmodal(msg) {
@@ -426,8 +420,11 @@ function up2k_init(have_crypto) {
         ebi('u2notbtn').innerHTML = '';
     }
 
-    var shame = 'your browser <a href="https://www.chromium.org/blink/webcrypto">disables sha512</a> unless you <a href="' + (window.location + '').replace(':', 's:') + '">use https</a>'
-    var is_https = (window.location + '').indexOf('https:') === 0;
+    var suggest_up2k = 'this is the basic uploader; <a href="#" id="u2yea">up2k</a> is better';
+
+    var shame = 'your browser <a href="https://www.chromium.org/blink/webcrypto">disables sha512</a> unless you <a href="' + (window.location + '').replace(':', 's:') + '">use https</a>',
+        is_https = (window.location + '').indexOf('https:') === 0;
+
     if (is_https)
         // chrome<37 firefox<34 edge<12 ie<11 opera<24 safari<10.1
         shame = 'your browser is impressively ancient';
@@ -446,34 +443,43 @@ function up2k_init(have_crypto) {
     }
 
     // show uploader if the user only has write-access
-    if (!ebi('files'))
+    var perms = (document.body.getAttribute('perms') + '').split(' ');
+    if (!has(perms, 'read'))
         goto('up2k');
 
-    // shows or clears an error message in the basic uploader ui
-    function setmsg(msg) {
+    // shows or clears a message in the basic uploader ui
+    function setmsg(msg, type) {
         if (msg !== undefined) {
-            ebi('u2err').setAttribute('class', 'err');
+            ebi('u2err').setAttribute('class', type);
             ebi('u2err').innerHTML = msg;
         }
         else {
             ebi('u2err').setAttribute('class', '');
             ebi('u2err').innerHTML = '';
         }
+        if (msg == suggest_up2k) {
+            ebi('u2yea').onclick = function (e) {
+                ev(e);
+                goto('up2k');
+            };
+        }
     }
 
     // switches to the basic uploader with msg as error message
     function un2k(msg) {
-        setmsg(msg);
+        setmsg(msg, 'err');
         return false;
     }
 
     // handle user intent to use the basic uploader instead
     ebi('u2nope').onclick = function (e) {
         ev(e);
-        setmsg();
+        setmsg(suggest_up2k, 'msg');
         goto('bup');
     };
 
+    setmsg(suggest_up2k, 'msg');
+
     if (!String.prototype.format) {
         String.prototype.format = function () {
             var args = arguments;
@@ -484,13 +490,14 @@ function up2k_init(have_crypto) {
         };
     }
 
-    var parallel_uploads = icfg_get('nthread');
-    var multitask = bcfg_get('multitask', true);
-    var ask_up = bcfg_get('ask_up', true);
-    var flag_en = bcfg_get('flag_en', false);
-    var fsearch = bcfg_get('fsearch', false);
+    var parallel_uploads = icfg_get('nthread'),
+        multitask = bcfg_get('multitask', true),
+        ask_up = bcfg_get('ask_up', true),
+        flag_en = bcfg_get('flag_en', false),
+        fsearch = bcfg_get('fsearch', false),
+        fdom_ctr = 0,
+        min_filebuf = 0;
 
-    var fdom_ctr = 0;
     var st = {
         "files": [],
         "todo": {
@@ -540,8 +547,9 @@ function up2k_init(have_crypto) {
         e.stopPropagation();
         e.preventDefault();
 
-        var files;
-        var is_itemlist = false;
+        var files,
+            is_itemlist = false;
+
         if (e.dataTransfer) {
             if (e.dataTransfer.items) {
                 files = e.dataTransfer.items; // DataTransferItemList
@@ -555,9 +563,10 @@ function up2k_init(have_crypto) {
             return alert('no files selected??');
 
         more_one_file();
-        var bad_files = [];
-        var good_files = [];
-        var dirs = [];
+        var bad_files = [],
+            good_files = [],
+            dirs = [];
+
         for (var a = 0; a < files.length; a++) {
             var fobj = files[a];
             if (is_itemlist) {
@@ -642,12 +651,13 @@ function up2k_init(have_crypto) {
 
     function gotallfiles(good_files, bad_files) {
         if (bad_files.length > 0) {
-            var ntot = bad_files.length + good_files.length;
-            var msg = 'These {0} files (of {1} total) were skipped because they are empty:\n'.format(bad_files.length, ntot);
+            var ntot = bad_files.length + good_files.length,
+                msg = 'These {0} files (of {1} total) were skipped because they are empty:\n'.format(bad_files.length, ntot);
+
             for (var a = 0, aa = Math.min(20, bad_files.length); a < aa; a++)
                 msg += '-- ' + bad_files[a] + '\n';
 
-            if (good_files.length - bad_files.length <= 1 && /(android)/i.test(navigator.userAgent))
+            if (good_files.length - bad_files.length <= 1 && ANDROID)
                 msg += '\nFirefox-Android has a bug which prevents selecting multiple files. Try selecting one file at a time. For more info, see firefox bug 1456557';
 
             alert(msg);
@@ -661,9 +671,10 @@ function up2k_init(have_crypto) {
             return;
 
         for (var a = 0; a < good_files.length; a++) {
-            var fobj = good_files[a][0];
-            var now = new Date().getTime();
-            var lmod = fobj.lastModified || now;
+            var fobj = good_files[a][0],
+                now = Date.now(),
+                lmod = fobj.lastModified || now;
+
             var entry = {
                 "n": parseInt(st.files.length.toString()),
                 "t0": now,
@@ -690,7 +701,7 @@ function up2k_init(have_crypto) {
                     esc(uricom_dec(entry.purl)[0] + entry.name)).join(' '),
                 '📐 hash',
                 ''
-            ]);
+            ], fobj.size);
             st.files.push(entry);
             st.todo.hash.push(entry);
         }
@@ -699,7 +710,7 @@ function up2k_init(have_crypto) {
 
     function more_one_file() {
         fdom_ctr++;
-        var elm = document.createElement('div')
+        var elm = mknod('div');
         elm.innerHTML = '<input id="file{0}" type="file" name="file{0}[]" multiple="multiple" />'.format(fdom_ctr);
         ebi('u2form').appendChild(elm);
         ebi('file' + fdom_ctr).addEventListener('change', gotfile, false);
@@ -737,7 +748,7 @@ function up2k_init(have_crypto) {
     function hashing_permitted() {
         if (multitask) {
             var ahead = st.bytes.hashed - st.bytes.uploaded;
-            return ahead < 1024 * 1024 * 128 &&
+            return ahead < 1024 * 1024 * 1024 * 4 &&
                 st.todo.handshake.length + st.busy.handshake.length < 16;
         }
         return handshakes_permitted() && 0 ==
@@ -746,26 +757,23 @@ function up2k_init(have_crypto) {
     }
 
     var tasker = (function () {
-        var mutex = false;
-        var was_busy = false;
+        var tto = null,
+            running = false,
+            was_busy = false;
+
+        function defer() {
+            running = false;
+            clearTimeout(tto);
+            tto = setTimeout(taskerd, 100);
+        }
 
         function taskerd() {
-            if (mutex)
+            if (running)
                 return;
 
-            mutex = true;
+            clearTimeout(tto);
+            running = true;
             while (true) {
-                if (false) {
-                    ebi('srv_info').innerHTML =
-                        new Date().getTime() + ", " +
-                        st.todo.hash.length + ", " +
-                        st.todo.handshake.length + ", " +
-                        st.todo.upload.length + ", " +
-                        st.busy.hash.length + ", " +
-                        st.busy.handshake.length + ", " +
-                        st.busy.upload.length;
-                }
-
                 var is_busy = 0 !=
                     st.todo.hash.length +
                     st.todo.handshake.length +
@@ -777,21 +785,16 @@ function up2k_init(have_crypto) {
                 if (was_busy != is_busy) {
                     was_busy = is_busy;
 
-                    if (is_busy)
-                        window.addEventListener("beforeunload", warn_uploader_busy);
-                    else
-                        window.removeEventListener("beforeunload", warn_uploader_busy);
+                    window[(is_busy ? "add" : "remove") +
+                        "EventListener"]("beforeunload", warn_uploader_busy);
                 }
 
                 if (flag) {
                     if (is_busy) {
-                        var now = new Date().getTime();
+                        var now = Date.now();
                         flag.take(now);
-                        if (!flag.ours) {
-                            setTimeout(taskerd, 100);
-                            mutex = false;
-                            return;
-                        }
+                        if (!flag.ours)
+                            return defer();
                     }
                     else if (flag.ours) {
                         flag.give();
@@ -833,11 +836,8 @@ function up2k_init(have_crypto) {
                     mou_ikkai = true;
                 }
 
-                if (!mou_ikkai) {
-                    setTimeout(taskerd, 100);
-                    mutex = false;
-                    return;
-                }
+                if (!mou_ikkai)
+                    return defer();
             }
         }
         taskerd();
@@ -851,47 +851,47 @@ function up2k_init(have_crypto) {
 
     // https://gist.github.com/jonleighton/958841
     function buf2b64(arrayBuffer) {
-        var base64 = '';
-        var encodings = 'ABCDEFGHIJKLMNOPQRSTUVWXYZabcdefghijklmnopqrstuvwxyz0123456789-_';
-        var bytes = new Uint8Array(arrayBuffer);
-        var byteLength = bytes.byteLength;
-        var byteRemainder = byteLength % 3;
-        var mainLength = byteLength - byteRemainder;
-        var a, b, c, d;
-        var chunk;
+        var base64 = '',
+            cset = 'ABCDEFGHIJKLMNOPQRSTUVWXYZabcdefghijklmnopqrstuvwxyz0123456789-_',
+            src = new Uint8Array(arrayBuffer),
+            nbytes = src.byteLength,
+            byteRem = nbytes % 3,
+            mainLen = nbytes - byteRem,
+            a, b, c, d, chunk;
 
-        for (var i = 0; i < mainLength; i = i + 3) {
-            chunk = (bytes[i] << 16) | (bytes[i + 1] << 8) | bytes[i + 2];
+        for (var i = 0; i < mainLen; i = i + 3) {
+            chunk = (src[i] << 16) | (src[i + 1] << 8) | src[i + 2];
             // create 8*3=24bit segment then split into 6bit segments
-            a = (chunk & 16515072) >> 18; // 16515072 = (2^6 - 1) << 18
-            b = (chunk & 258048) >> 12; // 258048   = (2^6 - 1) << 12
-            c = (chunk & 4032) >> 6; // 4032     = (2^6 - 1) << 6
-            d = chunk & 63;               // 63       = 2^6 - 1
+            a = (chunk & 16515072) >> 18; // (2^6 - 1) << 18
+            b = (chunk & 258048) >> 12; // (2^6 - 1) << 12
+            c = (chunk & 4032) >> 6; // (2^6 - 1) << 6
+            d = chunk & 63; // 2^6 - 1
 
             // Convert the raw binary segments to the appropriate ASCII encoding
-            base64 += encodings[a] + encodings[b] + encodings[c] + encodings[d];
+            base64 += cset[a] + cset[b] + cset[c] + cset[d];
         }
 
-        if (byteRemainder == 1) {
-            chunk = bytes[mainLength];
-            a = (chunk & 252) >> 2; // 252 = (2^6 - 1) << 2
-            b = (chunk & 3) << 4; // 3   = 2^2 - 1  (zero 4 LSB)
-            base64 += encodings[a] + encodings[b];//+ '==';
+        if (byteRem == 1) {
+            chunk = src[mainLen];
+            a = (chunk & 252) >> 2; // (2^6 - 1) << 2
+            b = (chunk & 3) << 4; // 2^2 - 1  (zero 4 LSB)
+            base64 += cset[a] + cset[b];//+ '==';
         }
-        else if (byteRemainder == 2) {
-            chunk = (bytes[mainLength] << 8) | bytes[mainLength + 1];
-            a = (chunk & 64512) >> 10; // 64512 = (2^6 - 1) << 10
-            b = (chunk & 1008) >> 4; // 1008  = (2^6 - 1) << 4
-            c = (chunk & 15) << 2; // 15    = 2^4 - 1  (zero 2 LSB)
-            base64 += encodings[a] + encodings[b] + encodings[c];//+ '=';
+        else if (byteRem == 2) {
+            chunk = (src[mainLen] << 8) | src[mainLen + 1];
+            a = (chunk & 64512) >> 10; // (2^6 - 1) << 10
+            b = (chunk & 1008) >> 4; // (2^6 - 1) << 4
+            c = (chunk & 15) << 2; // 2^4 - 1  (zero 2 LSB)
+            base64 += cset[a] + cset[b] + cset[c];//+ '=';
         }
 
         return base64;
     }
 
     function get_chunksize(filesize) {
-        var chunksize = 1024 * 1024;
-        var stepsize = 512 * 1024;
+        var chunksize = 1024 * 1024,
+            stepsize = 512 * 1024;
+
         while (true) {
             for (var mul = 1; mul <= 2; mul++) {
                 var nchunks = Math.ceil(filesize / chunksize);
@@ -904,156 +904,99 @@ function up2k_init(have_crypto) {
         }
     }
 
-    function test_filereader_speed(segm_err) {
-        var f = st.todo.hash[0].fobj,
-            sz = Math.min(2, f.size),
-            reader = new FileReader(),
-            t0, ctr = 0;
-
-        var segm_next = function () {
-            var t = new Date().getTime(),
-                td = t - t0;
-
-            if (++ctr > 2) {
-                need_filereader_cache = td > 50;
-                st.busy.hash.pop();
-                return;
-            }
-            t0 = t;
-            reader.onload = segm_next;
-            reader.onerror = segm_err;
-            reader.readAsArrayBuffer(
-                bobslice.call(f, 0, sz));
-        };
-
-        segm_next();
-    }
-
-    function ensure_rendered(func) {
-        var hidden = false;
-        var keys = ['hidden', 'msHidden', 'webkitHidden'];
-        for (var a = 0; a < keys.length; a++)
-            if (typeof document[keys[a]] !== "undefined")
-                hidden = document[keys[a]];
-
-        if (hidden)
-            return func();
-
-        window.requestAnimationFrame(func);
-    }
-
     function exec_hash() {
-        if (need_filereader_cache === undefined) {
-            st.busy.hash.push(1);
-            return test_filereader_speed(segm_err);
-        }
-
         var t = st.todo.hash.shift();
         st.busy.hash.push(t);
         st.bytes.hashed += t.size;
         t.bytes_uploaded = 0;
-        t.t1 = new Date().getTime();
 
-        var nchunk = 0;
-        var chunksize = get_chunksize(t.size);
-        var nchunks = Math.ceil(t.size / chunksize);
-
-        // android-chrome has 180ms latency on FileReader calls,
-        // detect this and do 32MB at a time
-        var cache_buf = undefined,
-            cache_ofs = 0,
-            subchunks = 2;
-
-        while (subchunks * chunksize <= 32 * 1024 * 1024)
-            subchunks++;
-
-        subchunks--;
-        if (!need_filereader_cache)
-            subchunks = 1;
+        var bpend = 0,
+            nchunk = 0,
+            chunksize = get_chunksize(t.size),
+            nchunks = Math.ceil(t.size / chunksize),
+            hashtab = {};
 
         pvis.setab(t.n, nchunks);
         pvis.move(t.n, 'bz');
 
-        var reader = new FileReader();
-
         var segm_next = function () {
-            if (cache_buf) {
-                return hash_calc();
-            }
-            reader.onload = segm_load;
-            reader.onerror = segm_err;
+            if (nchunk >= nchunks || (bpend > chunksize && bpend >= min_filebuf))
+                return false;
+
+            var reader = new FileReader(),
+                nch = nchunk++,
+                car = nch * chunksize,
+                cdr = car + chunksize,
+                t0 = Date.now();
 
-            var car = nchunk * chunksize;
-            var cdr = car + chunksize * subchunks;
             if (cdr >= t.size)
                 cdr = t.size;
 
+            bpend += cdr - car;
+
+            reader.onload = function (e) {
+                if (!min_filebuf && nch == 1) {
+                    min_filebuf = 1;
+                    var td = Date.now() - t0;
+                    if (td > 50) {
+                        ebi('u2foot').innerHTML += "<p>excessive filereader latency (" + td + " ms), increasing readahead</p>";
+                        min_filebuf = 32 * 1024 * 1024;
+                    }
+                }
+                hash_calc(nch, e.target.result);
+            };
+            reader.onerror = function () {
+                alert('y o u   b r o k e    i t\nerror: ' + reader.error);
+            };
             reader.readAsArrayBuffer(
                 bobslice.call(t.fobj, car, cdr));
+
+            return true;
         };
 
-        var segm_load = function (e) {
-            cache_buf = e.target.result;
-            cache_ofs = 0;
-            hash_calc();
-        };
+        var hash_calc = function (nch, buf) {
+            while (segm_next());
 
-        var hash_calc = function () {
-            var buf = cache_buf;
-            if (chunksize >= buf.byteLength)
-                cache_buf = undefined;
-            else {
-                var ofs = cache_ofs;
-                var ofs2 = ofs + Math.min(chunksize, cache_buf.byteLength - cache_ofs);
-                cache_ofs = ofs2;
-                buf = new Uint8Array(cache_buf).subarray(ofs, ofs2);
-                if (ofs2 >= cache_buf.byteLength)
-                    cache_buf = undefined;
-            }
+            var hash_done = function (hashbuf) {
+                var hslice = new Uint8Array(hashbuf).subarray(0, 32),
+                    b64str = buf2b64(hslice).replace(/=$/, '');
 
-            var func = function () {
-                if (have_crypto)
-                    crypto.subtle.digest('SHA-512', buf).then(hash_done);
-                else {
-                    var hasher = new asmCrypto.Sha512();
-                    hasher.process(new Uint8Array(buf));
-                    hasher.finish();
-                    hash_done(hasher.result);
+                hashtab[nch] = b64str;
+                t.hash.push(nch);
+                pvis.hashed(t);
+
+                bpend -= buf.byteLength;
+                if (t.hash.length < nchunks) {
+                    return segm_next();
                 }
+                t.hash = [];
+                for (var a = 0; a < nchunks; a++) {
+                    t.hash.push(hashtab[a]);
+                }
+
+                t.t2 = Date.now();
+                if (t.n == 0 && window.location.hash == '#dbg') {
+                    var spd = (t.size / ((t.t2 - t.t1) / 1000.)) / (1024 * 1024.);
+                    alert('{0} ms, {1} MB/s\n'.format(t.t2 - t.t1, spd.toFixed(3)) + t.hash.join('\n'));
+                }
+
+                pvis.seth(t.n, 2, 'hashing done');
+                pvis.seth(t.n, 1, '📦 wait');
+                st.busy.hash.splice(st.busy.hash.indexOf(t), 1);
+                st.todo.handshake.push(t);
             };
 
-            if (cache_buf)
-                ensure_rendered(func);
-            else
-                func();
-        };
-
-        var hash_done = function (hashbuf) {
-            var hslice = new Uint8Array(hashbuf).subarray(0, 32);
-            var b64str = buf2b64(hslice).replace(/=$/, '');
-            t.hash.push(b64str);
-
-            pvis.hashed(t);
-            if (++nchunk < nchunks) {
-                return segm_next();
+            if (have_crypto)
+                crypto.subtle.digest('SHA-512', buf).then(hash_done);
+            else {
+                var hasher = new asmCrypto.Sha512();
+                hasher.process(new Uint8Array(buf));
+                hasher.finish();
+                hash_done(hasher.result);
             }
-
-            t.t2 = new Date().getTime();
-            if (t.n == 0 && window.location.hash == '#dbg') {
-                var spd = (t.size / ((t.t2 - t.t1) / 1000.)) / (1024 * 1024.);
-                alert('{0} ms, {1} MB/s\n'.format(t.t2 - t.t1, spd.toFixed(3)) + t.hash.join('\n'));
-            }
-
-            pvis.seth(t.n, 2, 'hashing done');
-            pvis.seth(t.n, 1, '📦 wait');
-            st.busy.hash.splice(st.busy.hash.indexOf(t), 1);
-            st.todo.handshake.push(t);
-        };
-
-        var segm_err = function () {
-            alert('y o u   b r o k e    i t\n\n(was that a folder? just files please)');
         };
 
+        t.t1 = Date.now();
         segm_next();
     }
 
@@ -1072,8 +1015,9 @@ function up2k_init(have_crypto) {
                 var response = JSON.parse(xhr.responseText);
 
                 if (!response.name) {
-                    var msg = '';
-                    var smsg = '';
+                    var msg = '',
+                        smsg = '';
+
                     if (!response || !response.hits || !response.hits.length) {
                         msg = 'not found on server';
                         smsg = '404';
@@ -1106,6 +1050,15 @@ function up2k_init(have_crypto) {
                     pvis.seth(t.n, 0, linksplit(esc(t.purl + t.name)).join(' '));
                 }
 
+                var chunksize = get_chunksize(t.size),
+                    cdr_idx = Math.ceil(t.size / chunksize) - 1,
+                    cdr_sz = (t.size % chunksize) || chunksize,
+                    cbd = [];
+
+                for (var a = 0; a <= cdr_idx; a++) {
+                    cbd.push(a == cdr_idx ? cdr_sz : chunksize);
+                }
+
                 t.postlist = [];
                 t.wark = response.wark;
                 var missing = response.hash;
@@ -1116,10 +1069,15 @@ function up2k_init(have_crypto) {
                             missing[a], JSON.stringify(t)));
 
                     t.postlist.push(idx);
+                    cbd[idx] = 0;
                 }
 
-                var done = true;
-                var msg = '&#x1f3b7;&#x1f41b;';
+                pvis.setat(t.n, cbd);
+                pvis.prog(t, 0, cbd[0]);
+
+                var done = true,
+                    msg = '&#x1f3b7;&#x1f41b;';
+
                 if (t.postlist.length > 0) {
                     for (var a = 0; a < t.postlist.length; a++)
                         st.todo.upload.push({
@@ -1136,10 +1094,12 @@ function up2k_init(have_crypto) {
                 if (done) {
                     t.done = true;
                     st.bytes.uploaded += t.size - t.bytes_uploaded;
-                    var spd1 = (t.size / ((t.t2 - t.t1) / 1000.)) / (1024 * 1024.);
-                    var spd2 = (t.size / ((t.t4 - t.t3) / 1000.)) / (1024 * 1024.);
+                    var spd1 = (t.size / ((t.t2 - t.t1) / 1000.)) / (1024 * 1024.),
+                        spd2 = (t.size / ((t.t4 - t.t3) / 1000.)) / (1024 * 1024.);
+
                     pvis.seth(t.n, 2, 'hash {0}, up {1} MB/s'.format(
                         spd1.toFixed(2), spd2.toFixed(2)));
+
                     pvis.move(t.n, 'ok');
                 }
                 else t.t4 = undefined;
@@ -1206,66 +1166,55 @@ function up2k_init(have_crypto) {
         var upt = st.todo.upload.shift();
         st.busy.upload.push(upt);
 
-        var npart = upt.npart;
-        var t = st.files[upt.nfile];
+        var npart = upt.npart,
+            t = st.files[upt.nfile];
+
+        if (!t.t3)
+            t.t3 = Date.now();
 
         pvis.seth(t.n, 1, "🚀 send");
 
-        var chunksize = get_chunksize(t.size);
-        var car = npart * chunksize;
-        var cdr = car + chunksize;
+        var chunksize = get_chunksize(t.size),
+            car = npart * chunksize,
+            cdr = car + chunksize;
+
         if (cdr >= t.size)
             cdr = t.size;
 
-        var reader = new FileReader();
-
-        reader.onerror = function () {
-            alert('y o u   b r o k e    i t\n\n(was that a folder? just files please)');
+        var xhr = new XMLHttpRequest();
+        xhr.upload.onprogress = function (xev) {
+            pvis.prog(t, npart, xev.loaded);
         };
-
-        reader.onload = function (e) {
-            var xhr = new XMLHttpRequest();
-            xhr.upload.onprogress = function (xev) {
-                var perc = xev.loaded / (cdr - car) * 100;
-                pvis.prog(t, npart, perc, t);
-            };
-            xhr.onload = function (xev) {
-                if (xhr.status == 200) {
-                    pvis.prog(t, npart, 101, t);
-                    st.bytes.uploaded += cdr - car;
-                    t.bytes_uploaded += cdr - car;
-                    st.busy.upload.splice(st.busy.upload.indexOf(upt), 1);
-                    t.postlist.splice(t.postlist.indexOf(npart), 1);
-                    if (t.postlist.length == 0) {
-                        t.t4 = new Date().getTime();
-                        pvis.seth(t.n, 1, 'verifying');
-                        st.todo.handshake.unshift(t);
-                    }
-                    tasker();
+        xhr.onload = function (xev) {
+            if (xhr.status == 200) {
+                pvis.prog(t, npart, cdr - car);
+                st.bytes.uploaded += cdr - car;
+                t.bytes_uploaded += cdr - car;
+                st.busy.upload.splice(st.busy.upload.indexOf(upt), 1);
+                t.postlist.splice(t.postlist.indexOf(npart), 1);
+                if (t.postlist.length == 0) {
+                    t.t4 = Date.now();
+                    pvis.seth(t.n, 1, 'verifying');
+                    st.todo.handshake.unshift(t);
                 }
-                else
-                    alert("server broke; cu-err {0} on file [{1}]:\n".format(
-                        xhr.status, t.name) + (
-                            (xhr.response && xhr.response.err) ||
-                            (xhr.responseText && xhr.responseText) ||
-                            "no further information"));
-            };
-            xhr.open('POST', t.purl + 'chunkpit.php', true);
-            //xhr.setRequestHeader("X-Up2k-Hash", t.hash[npart].substr(1) + "x");
-            xhr.setRequestHeader("X-Up2k-Hash", t.hash[npart]);
-            xhr.setRequestHeader("X-Up2k-Wark", t.wark);
-            xhr.setRequestHeader('Content-Type', 'application/octet-stream');
-            if (xhr.overrideMimeType)
-                xhr.overrideMimeType('Content-Type', 'application/octet-stream');
-
-            xhr.responseType = 'text';
-            xhr.send(e.target.result);
-
-            if (!t.t3)
-                t.t3 = new Date().getTime();
+                tasker();
+            }
+            else
+                alert("server broke; cu-err {0} on file [{1}]:\n".format(
+                    xhr.status, t.name) + (
+                        (xhr.response && xhr.response.err) ||
+                        (xhr.responseText && xhr.responseText) ||
+                        "no further information"));
         };
+        xhr.open('POST', t.purl + 'chunkpit.php', true);
+        xhr.setRequestHeader("X-Up2k-Hash", t.hash[npart]);
+        xhr.setRequestHeader("X-Up2k-Wark", t.wark);
+        xhr.setRequestHeader('Content-Type', 'application/octet-stream');
+        if (xhr.overrideMimeType)
+            xhr.overrideMimeType('Content-Type', 'application/octet-stream');
 
-        reader.readAsArrayBuffer(bobslice.call(t.fobj, car, cdr));
+        xhr.responseType = 'text';
+        xhr.send(bobslice.call(t.fobj, car, cdr));
     }
 
     /////
@@ -1293,20 +1242,20 @@ function up2k_init(have_crypto) {
     onresize();
 
     function desc_show(e) {
-        var msg = this.getAttribute('alt');
-        msg = msg.replace(/\$N/g, "<br />");
-        var cdesc = ebi('u2cdesc');
-        cdesc.innerHTML = msg;
+        var msg = this.getAttribute('alt'),
+            cdesc = ebi('u2cdesc');
+
+        cdesc.innerHTML = msg.replace(/\$N/g, "<br />");
         cdesc.setAttribute('class', 'show');
     }
     function desc_hide(e) {
         ebi('u2cdesc').setAttribute('class', '');
     }
-    var o = document.querySelectorAll('#u2conf *[alt]');
+    var o = QSA('#u2conf *[alt]');
     for (var a = o.length - 1; a >= 0; a--) {
         o[a].parentNode.getElementsByTagName('input')[0].setAttribute('alt', o[a].getAttribute('alt'));
     }
-    var o = document.querySelectorAll('#u2conf *[alt]');
+    var o = QSA('#u2conf *[alt]');
     for (var a = 0; a < o.length; a++) {
         o[a].onfocus = desc_show;
         o[a].onblur = desc_hide;
@@ -1322,14 +1271,14 @@ function up2k_init(have_crypto) {
 
         var obj = ebi('nthread');
         if (dir.target) {
-            obj.style.background = '#922';
+            clmod(obj, 'err', 1);
             var v = Math.floor(parseInt(obj.value));
             if (v < 1 || v > 8 || v !== v)
                 return;
 
             parallel_uploads = v;
             swrite('nthread', v);
-            obj.style.background = '#444';
+            clmod(obj, 'err');
             return;
         }
 
@@ -1360,8 +1309,8 @@ function up2k_init(have_crypto) {
     }
 
     function set_fsearch(new_state) {
-        var perms = document.body.getAttribute('perms');
-        var read_only = false;
+        var perms = document.body.getAttribute('perms'),
+            read_only = false;
 
         if (!ebi('fsearch')) {
             new_state = false;
@@ -1377,16 +1326,16 @@ function up2k_init(have_crypto) {
         }
 
         try {
-            document.querySelector('label[for="fsearch"]').style.opacity = read_only ? '0' : '1';
+            QS('label[for="fsearch"]').style.opacity = read_only ? '0' : '1';
         }
         catch (ex) { }
 
         try {
-            var fun = fsearch ? 'add' : 'remove';
-            ebi('op_up2k').classList[fun]('srch');
+            var fun = fsearch ? 'add' : 'remove',
+                ico = fsearch ? '🔎' : '🚀',
+                desc = fsearch ? 'Search' : 'Upload';
 
-            var ico = fsearch ? '🔎' : '🚀';
-            var desc = fsearch ? 'Search' : 'Upload';
+            ebi('op_up2k').classList[fun]('srch');
             ebi('u2bm').innerHTML = ico + ' <sup>' + desc + '</sup>';
         }
         catch (ex) { }
@@ -1453,5 +1402,5 @@ function warn_uploader_busy(e) {
 }
 
 
-if (document.querySelector('#op_up2k.act'))
+if (QS('#op_up2k.act'))
     goto_up2k();

copyparty/web/upload.css

@@ -19,6 +19,10 @@
 	color: #f87;
 	padding: .5em;
 }
+#u2err.msg {
+	color: #999;
+	padding: .5em;
+}
 #u2btn {
 	color: #eee;
 	background: #555;
@@ -86,12 +90,14 @@
 	font-family: sans-serif;
 	width: auto;
 }
-#u2tab tr+tr:hover td {
+#u2tab tbody tr:hover td {
 	background: #222;
 }
 #u2cards {
-	margin: 2.5em auto -2.5em auto;
+	padding: 1em 0 .3em 1em;
+	margin: 1.5em auto -2.5em auto;
 	text-align: center;
+	overflow: hidden;
 }
 #u2cards.w {
 	width: 45em;
@@ -110,10 +116,15 @@
 	border-radius: 0 .4em 0 0;
 }
 #u2cards a.act {
-	border-width: 1px 1px 0 1px;
+	padding-bottom: .5em;
+	border-width: 1px 1px .1em 1px;
 	border-radius: .3em .3em 0 0;
 	margin-left: -1px;
-	background: transparent;
+	background: linear-gradient(to bottom, #464, #333 80%);
+	box-shadow: 0 -.17em .67em #280;
+	border-color: #7c5 #583 #333 #583;
+	position: relative;
+	color: #fd7;
 }
 #u2cards span {
 	color: #fff;
@@ -134,12 +145,16 @@
 	outline: none;
 }
 #u2conf .txtbox {
-	width: 4em;
+	width: 3em;
 	color: #fff;
 	background: #444;
 	border: 1px solid #777;
 	font-size: 1.2em;
 	padding: .15em 0;
+	height: 1.05em;
+}
+#u2conf .txtbox.err {
+	background: #922;
 }
 #u2conf a {
 	color: #fff;
@@ -148,13 +163,12 @@
 	border-radius: .1em;
 	font-size: 1.5em;
 	padding: .1em 0;
-	margin: 0 -.25em;
+	margin: 0 -1px;
 	width: 1.5em;
 	height: 1em;
 	display: inline-block;
 	position: relative;
-	line-height: 1em;
-	bottom: -.08em;
+	bottom: -0.08em;
 }
 #u2conf input+a {
 	background: #d80;
@@ -165,7 +179,6 @@
 	height: 1em;
 	padding: .4em 0;
 	display: block;
-	user-select: none;
 	border-radius: .25em;
 }
 #u2conf input[type="checkbox"] {
@@ -205,12 +218,13 @@
 	text-align: center;
 	overflow: hidden;
 	margin: 0 -2em;
-	height: 0;
 	padding: 0 1em;
+	height: 0;
 	opacity: .1;
-    transition: all 0.14s ease-in-out;
-	border-radius: .4em;
+	transition: all 0.14s ease-in-out;
 	box-shadow: 0 .2em .5em #222;
+	border-radius: .4em;
+	z-index: 1;
 }
 #u2cdesc.show {
 	padding: 1em;
@@ -238,3 +252,44 @@
 	float: right;
 	margin-bottom: -.3em;
 }
+
+
+
+
+
+html.light #u2btn {
+	box-shadow: .4em .4em 0 #ccc;
+}
+html.light #u2cards span {
+	color: #000;
+}
+html.light #u2cards a {
+	background: linear-gradient(to bottom, #eee, #fff);
+}
+html.light #u2cards a.act {
+	color: #037;
+	background: inherit;
+	box-shadow: 0 -.17em .67em #0ad;
+	border-color: #09c #05a #eee #05a;
+}
+html.light #u2conf .txtbox {
+	background: #fff;
+	color: #444;
+}
+html.light #u2conf .txtbox.err {
+	background: #f96;
+	color: #300;
+}
+html.light #u2cdesc {
+	background: #fff;
+	border: none;
+}
+html.light #op_up2k.srch #u2btn {
+	border-color: #a80;
+}
+html.light #u2foot {
+	color: #000;
+}
+html.light #u2tab tbody tr:hover td {
+	background: #fff;
+}

copyparty/web/upload.html

@@ -36,7 +36,7 @@
 
             <table id="u2conf">
                 <tr>
-                    <td>parallel uploads</td>
+                    <td>parallel uploads:</td>
                     <td rowspan="2">
                         <input type="checkbox" id="multitask" />
                         <label for="multitask" alt="continue hashing other files while uploading">🏃</label>
@@ -59,9 +59,9 @@
                 </tr>
                 <tr>
                     <td>
-                        <a href="#" id="nthread_sub">&ndash;</a>
-                        <input class="txtbox" id="nthread" value="2" />
-                        <a href="#" id="nthread_add">+</a>
+                        <a href="#" id="nthread_sub">&ndash;</a><input
+                            class="txtbox" id="nthread" value="2"/><a
+                            href="#" id="nthread_add">+</a>
                     </td>
                 </tr>
             </table>
@@ -99,5 +99,5 @@
             </table>
 
             <p id="u2foot"></p>
-            <p id="u2footfoot">( if you don't need lastmod timestamps, resumable uploads or progress bars just use the <a href="#" id="u2nope">basic uploader</a>)</p>
+            <p id="u2footfoot">( you can use the <a href="#" id="u2nope">basic uploader</a> if you don't need lastmod timestamps, resumable uploads, or progress bars )</p>
     </div>

copyparty/web/util.js

@@ -6,7 +6,8 @@ if (!window['console'])
     };
 
 
-var clickev = window.Touch ? 'touchstart' : 'click';
+var clickev = window.Touch ? 'touchstart' : 'click',
+    ANDROID = /(android)/i.test(navigator.userAgent);
 
 
 // error handler for mobile devices
@@ -49,9 +50,11 @@ function vis_exh(msg, url, lineNo, columnNo, error) {
 }
 
 
-function ebi(id) {
-    return document.getElementById(id);
-}
+var ebi = document.getElementById.bind(document),
+    QS = document.querySelector.bind(document),
+    QSA = document.querySelectorAll.bind(document),
+    mknod = document.createElement.bind(document);
+
 
 function ev(e) {
     e = e || window.event;
@@ -89,7 +92,7 @@ if (!String.startsWith) {
 // https://stackoverflow.com/a/950146
 function import_js(url, cb) {
     var head = document.head || document.getElementsByTagName('head')[0];
-    var script = document.createElement('script');
+    var script = mknod('script');
     script.type = 'text/javascript';
     script.src = url;
 
@@ -274,7 +277,7 @@ function makeSortable(table, cb) {
 
 
 (function () {
-    var ops = document.querySelectorAll('#ops>a');
+    var ops = QSA('#ops>a');
     for (var a = 0; a < ops.length; a++) {
         ops[a].onclick = opclick;
     }
@@ -289,25 +292,25 @@ function opclick(e) {
 
     swrite('opmode', dest || null);
 
-    var input = document.querySelector('.opview.act input:not([type="hidden"])')
+    var input = QS('.opview.act input:not([type="hidden"])')
     if (input)
         input.focus();
 }
 
 
 function goto(dest) {
-    var obj = document.querySelectorAll('.opview.act');
+    var obj = QSA('.opview.act');
     for (var a = obj.length - 1; a >= 0; a--)
         clmod(obj[a], 'act');
 
-    obj = document.querySelectorAll('#ops>a');
+    obj = QSA('#ops>a');
     for (var a = obj.length - 1; a >= 0; a--)
         clmod(obj[a], 'act');
 
     if (dest) {
         var ui = ebi('op_' + dest);
         clmod(ui, 'act', true);
-        document.querySelector('#ops>a[data-dest=' + dest + ']').className += " act";
+        QS('#ops>a[data-dest=' + dest + ']').className += " act";
 
         var fn = window['goto_' + dest];
         if (fn)

docs/example.conf

@@ -32,9 +32,13 @@ r
 
 # and a folder where anyone can upload
 # but nobody can see the contents
+# and set the e2d flag to enable the uploads database
+# and set the nodupe flag to reject duplicate uploads
 /home/ed/inc
 /dump
 w
+c e2d
+c nodupe
 
 # this entire config file can be replaced with these arguments:
 # -u ed:123 -u k:k -v .::r:aed -v priv:priv:rk:aed -v /home/ed/Music:music:r -v /home/ed/inc:dump:w

docs/minimal-up2k.html

@@ -0,0 +1,29 @@
+<!--
+    save this as .epilogue.html inside a
+    write-only folder to declutter the UI
+-->
+
+<style>
+
+    /* make the up2k ui REALLY minimal by hiding a bunch of stuff: */
+
+    #ops, #tree, #path,  /* main tabs and navigators (tree/breadcrumbs) */
+
+    #u2cleanup, #u2conf tr:first-child>td[rowspan]:not(#u2btn_cw),  /* most of the config options */
+
+    #u2cards  /* and the upload progress tabs */
+
+    {display:none!important}  /* do it! */
+
+
+
+    /* add some margins because now it's weird */
+    .opview {margin-top: 2.5em}
+    #op_up2k {margin-top: 5em}
+
+    /* and embiggen the upload button */
+    #u2conf #u2btn, #u2btn {padding:1.5em 0}
+
+</style>
+
+<a href="#" onclick="this.parentNode.innerHTML='';">show advanced options</a>

docs/notes.sh

@@ -171,7 +171,7 @@ Range: bytes=26-         Content-Range: bytes */26
 
 var tsh = [];
 function convert_markdown(md_text, dest_dom) {
-    tsh.push(new Date().getTime());
+    tsh.push(Date.now());
     while (tsh.length > 10)
         tsh.shift();
     if (tsh.length > 1) {

scripts/make-sfx.sh

@@ -117,7 +117,7 @@ cd sfx
 ver=
 git describe --tags >/dev/null 2>/dev/null && {
 	git_ver="$(git describe --tags)";  # v0.5.5-2-gb164aa0
-	ver="$(printf '%s\n' "$git_ver" | sed -r 's/^v//; s/-g?/./g')";
+	ver="$(printf '%s\n' "$git_ver" | sed -r 's/^v//')";
 	t_ver=
 
 	printf '%s\n' "$git_ver" | grep -qE '^v[0-9\.]+$' && {
@@ -199,12 +199,19 @@ find | grep -E '\.(js|css|html)$' | while IFS= read -r f; do
 	tmv "$f"
 done
 
+echo gen tarlist
+for d in copyparty dep-j2; do find $d -type f; done |
+sed -r 's/(.*)\.(.*)/\2 \1/' | LC_ALL=C sort |
+sed -r 's/([^ ]*) (.*)/\2.\1/' | grep -vE '/list1?$' > list1
+
+(grep -vE 'gz$' list1; grep -E 'gz$' list1) >list
+
 echo creating tar
 args=(--owner=1000 --group=1000)
 [ "$OSTYPE" = msys ] &&
 	args=()
 
-tar -cf tar "${args[@]}" --numeric-owner copyparty dep-j2
+tar -cf tar "${args[@]}" --numeric-owner -T list
 
 pc=bzip2
 pe=bz2

scripts/sfx.py

@@ -2,7 +2,8 @@
 # coding: latin-1
 from __future__ import print_function, unicode_literals
 
-import os, sys, time, shutil, threading, tarfile, hashlib, platform, tempfile, traceback
+import re, os, sys, time, shutil, signal, threading, tarfile, hashlib, platform, tempfile, traceback
+import subprocess as sp
 
 """
 run me with any version of python, i will unpack and run copyparty
@@ -26,22 +27,21 @@ CKSUM = None
 STAMP = None
 
 PY2 = sys.version_info[0] == 2
-WINDOWS = sys.platform == "win32"
+WINDOWS = sys.platform in ["win32", "msys"]
 sys.dont_write_bytecode = True
 me = os.path.abspath(os.path.realpath(__file__))
-cpp = None
 
 
-def eprint(*args, **kwargs):
-    kwargs["file"] = sys.stderr
-    print(*args, **kwargs)
+def eprint(*a, **ka):
+    ka["file"] = sys.stderr
+    print(*a, **ka)
 
 
-def msg(*args, **kwargs):
-    if args:
-        args = ["[SFX]", args[0]] + list(args[1:])
+def msg(*a, **ka):
+    if a:
+        a = ["[SFX]", a[0]] + list(a[1:])
 
-    eprint(*args, **kwargs)
+    eprint(*a, **ka)
 
 
 # skip 1
@@ -156,6 +156,9 @@ def encode(data, size, cksum, ver, ts):
                 skip = True
                 continue
 
+            if ln.strip().startswith("# fmt: "):
+                continue
+
             unpk += ln + "\n"
 
         for k, v in [
@@ -209,11 +212,11 @@ def yieldfile(fn):
 
 
 def hashfile(fn):
-    hasher = hashlib.md5()
+    h = hashlib.md5()
     for block in yieldfile(fn):
-        hasher.update(block)
+        h.update(block)
 
-    return hasher.hexdigest()
+    return h.hexdigest()
 
 
 def unpack():
@@ -222,9 +225,10 @@ def unpack():
     tag = "v" + str(STAMP)
     withpid = "{}.{}".format(name, os.getpid())
     top = tempfile.gettempdir()
-    final = os.path.join(top, name)
-    mine = os.path.join(top, withpid)
-    tar = os.path.join(mine, "tar")
+    opj = os.path.join
+    final = opj(top, name)
+    mine = opj(top, withpid)
+    tar = opj(mine, "tar")
 
     try:
         if tag in os.listdir(final):
@@ -233,28 +237,24 @@ def unpack():
     except:
         pass
 
-    nwrite = 0
+    sz = 0
     os.mkdir(mine)
     with open(tar, "wb") as f:
         for buf in get_payload():
-            nwrite += len(buf)
+            sz += len(buf)
             f.write(buf)
 
-    if nwrite != SIZE:
-        t = "\n\n  bad file:\n    expected {} bytes, got {}\n".format(SIZE, nwrite)
-        raise Exception(t)
-
-    cksum = hashfile(tar)
-    if cksum != CKSUM:
-        t = "\n\n  bad file:\n    {} expected,\n    {} obtained\n".format(CKSUM, cksum)
-        raise Exception(t)
+    ck = hashfile(tar)
+    if ck != CKSUM:
+        t = "\n\nexpected {} ({} byte)\nobtained {} ({} byte)\nsfx corrupt"
+        raise Exception(t.format(CKSUM, SIZE, ck, sz))
 
     with tarfile.open(tar, "r:bz2") as tf:
         tf.extractall(mine)
 
     os.remove(tar)
 
-    with open(os.path.join(mine, tag), "wb") as f:
+    with open(opj(mine, tag), "wb") as f:
         f.write(b"h\n")
 
     try:
@@ -272,25 +272,25 @@ def unpack():
     except:
         pass
 
+    for fn in u8(os.listdir(top)):
+        if fn.startswith(name) and fn != withpid:
+            try:
+                old = opj(top, fn)
+                if time.time() - os.path.getmtime(old) > 86400:
+                    shutil.rmtree(old)
+            except:
+                pass
+
     try:
         os.symlink(mine, final)
     except:
         try:
             os.rename(mine, final)
+            return final
         except:
             msg("reloc fail,", mine)
-            return mine
 
-    for fn in u8(os.listdir(top)):
-        if fn.startswith(name) and fn not in [name, withpid]:
-            try:
-                old = os.path.join(top, fn)
-                if time.time() - os.path.getmtime(old) > 10:
-                    shutil.rmtree(old)
-            except:
-                pass
-
-    return final
+    return mine
 
 
 def get_payload():
@@ -307,37 +307,33 @@ def get_payload():
         if ofs < 0:
             raise Exception("could not find archive marker")
 
-        # start reading from the final b"\n"
+        # start at final b"\n"
         fpos = ofs + len(ptn) - 3
-        # msg("tar found at", fpos)
         f.seek(fpos)
         dpos = 0
-        leftovers = b""
+        rem = b""
         while True:
             rbuf = f.read(1024 * 32)
             if rbuf:
-                buf = leftovers + rbuf
+                buf = rem + rbuf
                 ofs = buf.rfind(b"\n")
                 if len(buf) <= 4:
-                    leftovers = buf
+                    rem = buf
                     continue
 
                 if ofs >= len(buf) - 4:
-                    leftovers = buf[ofs:]
+                    rem = buf[ofs:]
                     buf = buf[:ofs]
                 else:
-                    leftovers = b"\n# "
+                    rem = b"\n# "
             else:
-                buf = leftovers
+                buf = rem
 
             fpos += len(buf) + 1
-            buf = (
-                buf.replace(b"\n# ", b"")
-                .replace(b"\n#r", b"\r")
-                .replace(b"\n#n", b"\n")
-            )
-            dpos += len(buf) - 1
+            for a, b in [[b"\n# ", b""], [b"\n#r", b"\r"], [b"\n#n", b"\n"]]:
+                buf = buf.replace(a, b)
 
+            dpos += len(buf) - 1
             yield buf
 
             if not rbuf:
@@ -361,7 +357,7 @@ def utime(top):
 
 def confirm(rv):
     msg()
-    msg(traceback.format_exc())
+    msg("retcode", rv if rv else traceback.format_exc())
     msg("*** hit enter to exit ***")
     try:
         raw_input() if PY2 else input()
@@ -371,10 +367,8 @@ def confirm(rv):
     sys.exit(rv)
 
 
-def run(tmp, j2ver):
-    global cpp
-
-    msg("jinja2:", j2ver or "bundled")
+def run(tmp, j2):
+    msg("jinja2:", j2 or "bundled")
     msg("sfxdir:", tmp)
     msg()
 
@@ -384,7 +378,6 @@ def run(tmp, j2ver):
 
         fd = os.open(tmp, os.O_RDONLY)
         fcntl.flock(fd, fcntl.LOCK_EX | fcntl.LOCK_NB)
-        tmp = os.readlink(tmp)  # can't flock a symlink, even with O_NOFOLLOW
     except Exception as ex:
         if not WINDOWS:
             msg("\033[31mflock:", repr(ex))
@@ -394,22 +387,39 @@ def run(tmp, j2ver):
     t.start()
 
     ld = [tmp, os.path.join(tmp, "dep-j2")]
-    if j2ver:
+    if j2:
         del ld[-1]
 
+    if any([re.match(r"^-.*j[0-9]", x) for x in sys.argv]):
+        run_s(ld)
+    else:
+        run_i(ld)
+
+
+def run_i(ld):
     for x in ld:
         sys.path.insert(0, x)
 
-    try:
-        from copyparty.__main__ import main as copyparty
+    from copyparty.__main__ import main as p
 
-        copyparty()
+    p()
 
-    except SystemExit as ex:
-        if ex.code:
-            confirm(ex.code)
-    except:
-        confirm(1)
+
+def run_s(ld):
+    # fmt: off
+    c = "import sys,runpy;" + "".join(['sys.path.insert(0,r"' + x + '");' for x in ld]) + 'runpy.run_module("copyparty",run_name="__main__")'
+    c = [str(x) for x in [sys.executable, "-c", c] + list(sys.argv[1:])]
+    # fmt: on
+    msg("\n", c, "\n")
+    p = sp.Popen(c)
+
+    def bye(*a):
+        p.send_signal(signal.SIGINT)
+
+    signal.signal(signal.SIGTERM, bye)
+    p.wait()
+
+    raise SystemExit(p.returncode)
 
 
 def main():
@@ -443,14 +453,23 @@ def main():
 
     # skip 0
 
-    tmp = unpack()
+    tmp = os.path.realpath(unpack())
 
     try:
-        from jinja2 import __version__ as j2ver
+        from jinja2 import __version__ as j2
     except:
-        j2ver = None
+        j2 = None
 
-    run(tmp, j2ver)
+    try:
+        run(tmp, j2)
+    except SystemExit as ex:
+        c = ex.code
+        if c not in [0, -15]:
+            confirm(ex.code)
+    except KeyboardInterrupt:
+        pass
+    except:
+        confirm(0)
 
 
 if __name__ == "__main__":

tests/run.py

@@ -0,0 +1,33 @@
+#!/usr/bin/env python3
+
+import sys
+import runpy
+
+host = sys.argv[1]
+sys.argv = sys.argv[:1] + sys.argv[2:]
+sys.path.insert(0, ".")
+
+
+def rp():
+    runpy.run_module("unittest", run_name="__main__")
+
+
+if host == "vmprof":
+    rp()
+
+elif host == "cprofile":
+    import cProfile
+    import pstats
+
+    log_fn = "cprofile.log"
+    cProfile.run("rp()", log_fn)
+    p = pstats.Stats(log_fn)
+    p.sort_stats(pstats.SortKey.CUMULATIVE).print_stats(64)
+
+
+"""
+python3.9 tests/run.py cprofile -v tests/test_httpcli.py
+
+python3.9 -m pip install --user vmprof
+python3.9 -m vmprof --lines -o vmprof.log tests/run.py vmprof -v tests/test_httpcli.py
+"""

tests/test_httpcli.py

@@ -0,0 +1,202 @@
+#!/usr/bin/env python
+# coding: utf-8
+from __future__ import print_function, unicode_literals
+
+import io
+import os
+import time
+import shutil
+import pprint
+import tarfile
+import unittest
+
+from argparse import Namespace
+from copyparty.authsrv import AuthSrv
+from copyparty.httpcli import HttpCli
+
+from tests import util as tu
+
+
+def hdr(query):
+    h = "GET /{} HTTP/1.1\r\nCookie: cppwd=o\r\nConnection: close\r\n\r\n"
+    return h.format(query).encode("utf-8")
+
+
+class Cfg(Namespace):
+    def __init__(self, a=[], v=[], c=None):
+        super(Cfg, self).__init__(
+            a=a,
+            v=v,
+            c=c,
+            ed=False,
+            no_zip=False,
+            no_scandir=False,
+            no_sendfile=True,
+            nih=True,
+            mtp=[],
+            mte="a",
+            **{k: False for k in "e2d e2ds e2dsa e2t e2ts e2tsr".split()}
+        )
+
+
+class TestHttpCli(unittest.TestCase):
+    def test(self):
+        td = os.path.join(tu.get_ramdisk(), "vfs")
+        try:
+            shutil.rmtree(td)
+        except OSError:
+            pass
+
+        os.mkdir(td)
+        os.chdir(td)
+
+        self.dtypes = ["ra", "ro", "rx", "wa", "wo", "wx", "aa", "ao", "ax"]
+        self.can_read = ["ra", "ro", "aa", "ao"]
+        self.can_write = ["wa", "wo", "aa", "ao"]
+        self.fn = "g{:x}g".format(int(time.time() * 3))
+
+        allfiles = []
+        allvols = []
+        for top in self.dtypes:
+            allvols.append(top)
+            allfiles.append("/".join([top, self.fn]))
+            for s1 in self.dtypes:
+                p = "/".join([top, s1])
+                allvols.append(p)
+                allfiles.append(p + "/" + self.fn)
+                allfiles.append(p + "/n/" + self.fn)
+                for s2 in self.dtypes:
+                    p = "/".join([top, s1, "n", s2])
+                    os.makedirs(p)
+                    allvols.append(p)
+                    allfiles.append(p + "/" + self.fn)
+
+        for fp in allfiles:
+            with open(fp, "w") as f:
+                f.write("ok {}\n".format(fp))
+
+        for top in self.dtypes:
+            vcfg = []
+            for vol in allvols:
+                if not vol.startswith(top):
+                    continue
+
+                mode = vol[-2]
+                usr = vol[-1]
+                if usr == "a":
+                    usr = ""
+
+                if "/" not in vol:
+                    vol += "/"
+
+                top, sub = vol.split("/", 1)
+                vcfg.append("{0}/{1}:{1}:{2}{3}".format(top, sub, mode, usr))
+
+            pprint.pprint(vcfg)
+
+            self.args = Cfg(v=vcfg, a=["o:o", "x:x"])
+            self.auth = AuthSrv(self.args, self.log)
+            vfiles = [x for x in allfiles if x.startswith(top)]
+            for fp in vfiles:
+                rok, wok = self.can_rw(fp)
+                furl = fp.split("/", 1)[1]
+                durl = furl.rsplit("/", 1)[0] if "/" in furl else ""
+
+                # file download
+                h, ret = self.curl(furl)
+                res = "ok " + fp in ret
+                print("[{}] {} {} = {}".format(fp, rok, wok, res))
+                if rok != res:
+                    print("\033[33m{}\n# {}\033[0m".format(ret, furl))
+                    self.fail()
+
+                # file browser: html
+                h, ret = self.curl(durl)
+                res = "'{}'".format(self.fn) in ret
+                print(res)
+                if rok != res:
+                    print("\033[33m{}\n# {}\033[0m".format(ret, durl))
+                    self.fail()
+
+                # file browser: json
+                url = durl + "?ls"
+                h, ret = self.curl(url)
+                res = '"{}"'.format(self.fn) in ret
+                print(res)
+                if rok != res:
+                    print("\033[33m{}\n# {}\033[0m".format(ret, url))
+                    self.fail()
+
+                # tar
+                url = durl + "?tar"
+                h, b = self.curl(url, True)
+                # with open(os.path.join(td, "tar"), "wb") as f:
+                #    f.write(b)
+                try:
+                    tar = tarfile.open(fileobj=io.BytesIO(b)).getnames()
+                except:
+                    tar = []
+                tar = ["/".join([y for y in [top, durl, x] if y]) for x in tar]
+                tar = [[x] + self.can_rw(x) for x in tar]
+                tar_ok = [x[0] for x in tar if x[1]]
+                tar_ng = [x[0] for x in tar if not x[1]]
+                self.assertEqual([], tar_ng)
+
+                if durl.split("/")[-1] in self.can_read:
+                    ref = [x for x in vfiles if self.in_dive(top + "/" + durl, x)]
+                    for f in ref:
+                        print("{}: {}".format("ok" if f in tar_ok else "NG", f))
+                    ref.sort()
+                    tar_ok.sort()
+                    self.assertEqual(ref, tar_ok)
+
+                # stash
+                h, ret = self.put(url)
+                res = h.startswith("HTTP/1.1 200 ")
+                self.assertEqual(res, wok)
+
+    def can_rw(self, fp):
+        # lowest non-neutral folder declares permissions
+        expect = fp.split("/")[:-1]
+        for x in reversed(expect):
+            if x != "n":
+                expect = x
+                break
+
+        return [expect in self.can_read, expect in self.can_write]
+
+    def in_dive(self, top, fp):
+        # archiver bails at first inaccessible subvolume
+        top = top.strip("/").split("/")
+        fp = fp.split("/")
+        for f1, f2 in zip(top, fp):
+            if f1 != f2:
+                return False
+
+        for f in fp[len(top) :]:
+            if f == self.fn:
+                return True
+            if f not in self.can_read and f != "n":
+                return False
+
+        return True
+
+    def put(self, url):
+        buf = "PUT /{0} HTTP/1.1\r\nCookie: cppwd=o\r\nConnection: close\r\nContent-Length: {1}\r\n\r\nok {0}\n"
+        buf = buf.format(url, len(url) + 4).encode("utf-8")
+        conn = tu.VHttpConn(self.args, self.auth, self.log, buf)
+        HttpCli(conn).run()
+        return conn.s._reply.decode("utf-8").split("\r\n\r\n", 1)
+
+    def curl(self, url, binary=False):
+        conn = tu.VHttpConn(self.args, self.auth, self.log, hdr(url))
+        HttpCli(conn).run()
+        if binary:
+            h, b = conn.s._reply.split(b"\r\n\r\n", 1)
+            return [h.decode("utf-8"), b]
+
+        return conn.s._reply.decode("utf-8").split("\r\n\r\n", 1)
+
+    def log(self, src, msg, c=0):
+        # print(repr(msg))
+        pass

tests/test_vfs.py

@@ -3,18 +3,18 @@
 from __future__ import print_function, unicode_literals
 
 import os
-import time
 import json
 import shutil
 import tempfile
 import unittest
-import subprocess as sp  # nosec
 
 from textwrap import dedent
 from argparse import Namespace
 from copyparty.authsrv import AuthSrv
 from copyparty import util
 
+from tests import util as tu
+
 
 class Cfg(Namespace):
     def __init__(self, a=[], v=[], c=None):
@@ -51,52 +51,11 @@ class TestVFS(unittest.TestCase):
         real = [x[0] for x in real]
         return fsdir, real, virt
 
-    def runcmd(self, *argv):
-        p = sp.Popen(argv, stdout=sp.PIPE, stderr=sp.PIPE)
-        stdout, stderr = p.communicate()
-        stdout = stdout.decode("utf-8")
-        stderr = stderr.decode("utf-8")
-        return [p.returncode, stdout, stderr]
-
-    def chkcmd(self, *argv):
-        ok, sout, serr = self.runcmd(*argv)
-        if ok != 0:
-            raise Exception(serr)
-
-        return sout, serr
-
-    def get_ramdisk(self):
-        for vol in ["/dev/shm", "/Volumes/cptd"]:  # nosec (singleton test)
-            if os.path.exists(vol):
-                return vol
-
-        if os.path.exists("/Volumes"):
-            devname, _ = self.chkcmd("hdiutil", "attach", "-nomount", "ram://8192")
-            devname = devname.strip()
-            print("devname: [{}]".format(devname))
-            for _ in range(10):
-                try:
-                    _, _ = self.chkcmd(
-                        "diskutil", "eraseVolume", "HFS+", "cptd", devname
-                    )
-                    return "/Volumes/cptd"
-                except Exception as ex:
-                    print(repr(ex))
-                    time.sleep(0.25)
-
-            raise Exception("ramdisk creation failed")
-
-        ret = os.path.join(tempfile.gettempdir(), "copyparty-test")
-        try:
-            os.mkdir(ret)
-        finally:
-            return ret
-
     def log(self, src, msg, c=0):
         pass
 
     def test(self):
-        td = os.path.join(self.get_ramdisk(), "vfs")
+        td = os.path.join(tu.get_ramdisk(), "vfs")
         try:
             shutil.rmtree(td)
         except OSError:
@@ -268,7 +227,7 @@ class TestVFS(unittest.TestCase):
         self.assertEqual(list(v1), list(v2))
 
         # config file parser
-        cfg_path = os.path.join(self.get_ramdisk(), "test.cfg")
+        cfg_path = os.path.join(tu.get_ramdisk(), "test.cfg")
         with open(cfg_path, "wb") as f:
             f.write(
                 dedent(

tests/util.py

@@ -0,0 +1,97 @@
+import os
+import time
+import jinja2
+import tempfile
+import subprocess as sp
+
+from copyparty.util import Unrecv
+
+
+J2_ENV = jinja2.Environment(loader=jinja2.BaseLoader)
+J2_FILES = J2_ENV.from_string("{{ files|join('\n') }}")
+
+
+def runcmd(*argv):
+    p = sp.Popen(argv, stdout=sp.PIPE, stderr=sp.PIPE)
+    stdout, stderr = p.communicate()
+    stdout = stdout.decode("utf-8")
+    stderr = stderr.decode("utf-8")
+    return [p.returncode, stdout, stderr]
+
+
+def chkcmd(*argv):
+    ok, sout, serr = runcmd(*argv)
+    if ok != 0:
+        raise Exception(serr)
+
+    return sout, serr
+
+
+def get_ramdisk():
+    for vol in ["/dev/shm", "/Volumes/cptd"]:  # nosec (singleton test)
+        if os.path.exists(vol):
+            return vol
+
+    if os.path.exists("/Volumes"):
+        devname, _ = chkcmd("hdiutil", "attach", "-nomount", "ram://32768")
+        devname = devname.strip()
+        print("devname: [{}]".format(devname))
+        for _ in range(10):
+            try:
+                _, _ = chkcmd("diskutil", "eraseVolume", "HFS+", "cptd", devname)
+                return "/Volumes/cptd"
+            except Exception as ex:
+                print(repr(ex))
+                time.sleep(0.25)
+
+        raise Exception("ramdisk creation failed")
+
+    ret = os.path.join(tempfile.gettempdir(), "copyparty-test")
+    try:
+        os.mkdir(ret)
+    finally:
+        return ret
+
+
+class NullBroker(object):
+    def put(*args):
+        pass
+
+
+class VSock(object):
+    def __init__(self, buf):
+        self._query = buf
+        self._reply = b""
+        self.sendall = self.send
+
+    def recv(self, sz):
+        ret = self._query[:sz]
+        self._query = self._query[sz:]
+        return ret
+
+    def send(self, buf):
+        self._reply += buf
+        return len(buf)
+
+
+class VHttpSrv(object):
+    def __init__(self):
+        self.broker = NullBroker()
+
+        aliases = ["splash", "browser", "browser2", "msg", "md", "mde"]
+        self.j2 = {x: J2_FILES for x in aliases}
+
+
+class VHttpConn(object):
+    def __init__(self, args, auth, log, buf):
+        self.s = VSock(buf)
+        self.sr = Unrecv(self.s)
+        self.addr = ("127.0.0.1", "42069")
+        self.args = args
+        self.auth = auth
+        self.log_func = log
+        self.log_src = "a"
+        self.hsrv = VHttpSrv()
+        self.nbyte = 0
+        self.workload = 0
+        self.t0 = time.time()