v0.12.5

README.md

@@ -16,6 +16,11 @@ turn your phone or raspi into a portable file server with resumable uploads/down
 📷 **screenshots:** [browser](#the-browser) // [upload](#uploading) // [thumbnails](#thumbnails) // [md-viewer](#markdown-viewer) // [search](#searching) // [fsearch](#file-search) // [zip-DL](#zip-downloads) // [ie4](#browser-support)
 
 
+## breaking changes \o/
+
+this is the readme for v0.12 which has a different expression for volume permissions (`-v`); see [the v0.11.x readme](https://github.com/9001/copyparty/tree/15b59822112dda56cee576df30f331252fc62628#readme) for stuff regarding the [current stable release](https://github.com/9001/copyparty/releases/tag/v0.11.47)
+
+
 ## readme toc
 
 * top
@@ -30,11 +35,12 @@ turn your phone or raspi into a portable file server with resumable uploads/down
 * [the browser](#the-browser)
     * [tabs](#tabs)
     * [hotkeys](#hotkeys)
-    * [tree-mode](#tree-mode)
+    * [navpane](#navpane)
     * [thumbnails](#thumbnails)
     * [zip downloads](#zip-downloads)
     * [uploading](#uploading)
         * [file-search](#file-search)
+    * [file manager](#file-manager)
     * [markdown viewer](#markdown-viewer)
     * [other tricks](#other-tricks)
 * [searching](#searching)
@@ -66,15 +72,14 @@ turn your phone or raspi into a portable file server with resumable uploads/down
 
 download [copyparty-sfx.py](https://github.com/9001/copyparty/releases/latest/download/copyparty-sfx.py) and you're all set!
 
-running the sfx without arguments (for example doubleclicking it on Windows) will give everyone full access to the current folder; see `-h` for help if you want accounts and volumes etc
+running the sfx without arguments (for example doubleclicking it on Windows) will give everyone full access to the current folder; see `-h` for help if you want [accounts and volumes](#accounts-and-volumes) etc
 
 some recommended options:
 * `-e2dsa` enables general file indexing, see [search configuration](#search-configuration)
 * `-e2ts` enables audio metadata indexing (needs either FFprobe or Mutagen), see [optional dependencies](#optional-dependencies)
-* `-v /mnt/music:/music:r:afoo -a foo:bar` shares `/mnt/music` as `/music`, `r`eadable by anyone, with user `foo` as `a`dmin (read/write), password `bar`
-  * the syntax is `-v src:dst:perm:perm:...` so local-path, url-path, and one or more permissions to set
-  * replace `:r:afoo` with `:rfoo` to only make the folder readable by `foo` and nobody else
-  * in addition to `r`ead and `a`dmin, `w`rite makes a folder write-only, so cannot list/access files in it
+* `-v /mnt/music:/music:r:rw,foo -a foo:bar` shares `/mnt/music` as `/music`, `r`eadable by anyone, and read-write for user `foo`, password `bar`
+  * replace `:r:rw,foo` with `:r,foo` to only make the folder readable by `foo` and nobody else
+  * see [accounts and volumes](#accounts-and-volumes) for the syntax and other access levels (`r`ead, `w`rite, `m`ove, `d`elete)
 * `--ls '**,*,ln,p,r'` to crash on startup if any of the volumes contain a symlink which point outside the volume, as that could give users unintended access
 
 you may also want these, especially on servers:
@@ -117,18 +122,19 @@ summary: all planned features work! now please enjoy the bloatening
   * ☑ sanic multipart parser
   * ☑ multiprocessing (actual multithreading)
   * ☑ volumes (mountpoints)
-  * ☑ accounts
+  * ☑ [accounts](#accounts-and-volumes)
 * upload
   * ☑ basic: plain multipart, ie6 support
   * ☑ up2k: js, resumable, multithreaded
   * ☑ stash: simple PUT filedropper
+  * ☑ unpost: undo/delete accidental uploads
   * ☑ symlink/discard existing files (content-matching)
 * download
   * ☑ single files in browser
   * ☑ folders as zip / tar files
   * ☑ FUSE client (read-only)
 * browser
-  * ☑ tree-view
+  * ☑ navpane (directory tree sidebar)
   * ☑ audio player (with OS media controls)
   * ☑ thumbnails
     * ☑ ...of images using Pillow
@@ -136,7 +142,7 @@ summary: all planned features work! now please enjoy the bloatening
     * ☑ cache eviction (max-age; maybe max-size eventually)
   * ☑ image gallery with webm player
   * ☑ SPA (browse while uploading)
-    * if you use the file-tree on the left only, not folders in the file list
+    * if you use the navpane to navigate, not folders in the file list
 * server indexing
   * ☑ locate files by contents
   * ☑ search by name/path/date/size
@@ -164,8 +170,6 @@ small collection of user feedback
 
 * all volumes must exist / be available on startup; up2k (mtp especially) gets funky otherwise
 * cannot mount something at `/d1/d2/d3` unless `d2` exists inside `d1`
-* dupe files will not have metadata (audio tags etc) displayed in the file listing
-  * because they don't get `up` entries in the db (probably best fix) and `tx_browser` does not `lstat`
 * probably more, pls let me know
 
 ## not my bugs
@@ -180,6 +184,30 @@ small collection of user feedback
   * use `--hist` or the `hist` volflag (`-v [...]:chist=/tmp/foo`) to place the db inside the vm instead
 
 
+# accounts and volumes
+
+* `-a usr:pwd` adds account `usr` with password `pwd`
+* `-v .::r` adds current-folder `.` as the webroot, `r`eadable by anyone
+  * the syntax is `-v src:dst:perm:perm:...` so local-path, url-path, and one or more permissions to set
+  * when granting permissions to an account, the names are comma-separated: `-v .::r,usr1,usr2:rw,usr3,usr4`
+
+permissions:
+* `r` (read): browse folder contents, download files, download as zip/tar
+* `w` (write): upload files, move files *into* folder
+* `m` (move): move files/folders *from* folder
+* `d` (delete): delete files/folders
+
+example:
+* add accounts named u1, u2, u3 with passwords p1, p2, p3: `-a u1:p1 -a u2:p2 -a u3:p3`
+* make folder `/srv` the root of the filesystem, read-only by anyone: `-v /srv::r`
+* make folder `/mnt/music` available at `/music`, read-only for u1 and u2, read-write for u3: `-v /mnt/music:music:r,u1,u2:rw,u3`
+  * unauthorized users accessing the webroot can see that the `music` folder exists, but cannot open it
+* make folder `/mnt/incoming` available at `/inc`, write-only for u1, read-move for u2: `-v /mnt/incoming:inc:w,u1:rm,u2`
+  * unauthorized users accessing the webroot can see that the `inc` folder exists, but cannot open it
+  * `u1` can open the `inc` folder, but cannot see the contents, only upload new files to it
+  * `u2` can browse it and move files *from* `/inc` into any folder where `u2` has write-access
+
+
 # the browser
 
 ![copyparty-browser-fs8](https://user-images.githubusercontent.com/241032/115978054-65106380-a57d-11eb-98f8-59e3dee73557.png)
@@ -188,10 +216,11 @@ small collection of user feedback
 ## tabs
 
 * `[🔎]` search by size, date, path/name, mp3-tags ... see [searching](#searching)
+* `[🧯]` unpost: undo/delete accidental uploads
 * `[🚀]` and `[🎈]` are the uploaders, see [uploading](#uploading)
-* `[📂]` mkdir, create directories
-* `[📝]` new-md, create a new markdown document
-* `[📟]` send-msg, either to server-log or into textfiles if `--urlform save`
+* `[📂]` mkdir: create directories
+* `[📝]` new-md: create a new markdown document
+* `[📟]` send-msg: either to server-log or into textfiles if `--urlform save`
 * `[🎺]` audio-player config options
 * `[⚙️]` general client config options
 
@@ -199,11 +228,20 @@ small collection of user feedback
 ## hotkeys
 
 the browser has the following hotkeys (assumes qwerty, ignores actual layout)
-* `B` toggle breadcrumbs / directory tree
+* `B` toggle breadcrumbs / navpane
 * `I/K` prev/next folder
 * `M` parent folder (or unexpand current)
 * `G` toggle list / grid view
 * `T` toggle thumbnails / icons
+* `ctrl-X` cut selected files/folders
+* `ctrl-V` paste
+* `F2` rename selected file/folder
+* when a file/folder is selected (in not-grid-view):
+  * `Up/Down` move cursor
+  * shift+`Up/Down` select and move cursor
+  * ctrl+`Up/Down` move cursor and scroll viewport
+  * `Space` toggle file selection
+  * `Ctrl-A` toggle select all
 * when playing audio:
   * `J/L` prev/next song
   * `U/O` skip 10sec back/forward
@@ -220,7 +258,7 @@ the browser has the following hotkeys (assumes qwerty, ignores actual layout)
     * `C` continue playing next video
     * `R` loop
     * `M` mute
-* when tree-sidebar is open:
+* when the navpane is open:
   * `A/D` adjust tree width
 * in the grid view:
   * `S` toggle multiselect
@@ -233,9 +271,10 @@ the browser has the following hotkeys (assumes qwerty, ignores actual layout)
   * `^e` toggle editor / preview
   * `^up, ^down` jump paragraphs
 
-## tree-mode
 
-by default there's a breadcrumbs path; you can replace this with a tree-browser sidebar thing by clicking the `🌲` or pressing the `B` hotkey
+## navpane
+
+by default there's a breadcrumbs path; you can replace this with a navpane (tree-browser sidebar thing) by clicking the `🌲` or pressing the `B` hotkey
 
 click `[-]` and `[+]` (or hotkeys `A`/`D`) to adjust the size, and the `[a]` toggles if the tree should widen dynamically as you go deeper or stay fixed-size
 
@@ -275,8 +314,10 @@ you can also zip a selection of files or folders by clicking them in the browser
 ## uploading
 
 two upload methods are available in the html client:
-* `🎈 bup`, the basic uploader, supports almost every browser since netscape 4.0
-* `🚀 up2k`, the fancy one
+* `[🎈] bup`, the basic uploader, supports almost every browser since netscape 4.0
+* `[🚀] up2k`, the fancy one
+
+you can undo/delete uploads using `[🧯] unpost` if the server is running with `-e2d`
 
 up2k has several advantages:
 * you can drop folders into the browser (files are added recursively)
@@ -325,6 +366,13 @@ note that since up2k has to read the file twice, `[🎈 bup]` can be up to 2x fa
 up2k has saved a few uploads from becoming corrupted in-transfer already; caught an android phone on wifi redhanded in wireshark with a bitflip, however bup with https would *probably* have noticed as well (thanks to tls also functioning as an integrity check)
 
 
+## file manager
+
+if you have the required permissions, you can cut/paste, rename, and delete files/folders
+
+you can move files across browser tabs (cut in one tab, paste in another)
+
+
 ## markdown viewer
 
 ![copyparty-md-read-fs8](https://user-images.githubusercontent.com/241032/115978057-66419080-a57d-11eb-8539-d2be843991aa.png)
@@ -451,7 +499,7 @@ copyparty can invoke external programs to collect additional metadata for files
 | send message    | yep | yep | yep  | yep  | yep   | yep  | yep | yep  |
 | set sort order  |  -  | yep | yep  | yep  | yep   | yep  | yep | yep  |
 | zip selection   |  -  | yep | yep  | yep  | yep   | yep  | yep | yep  |
-| directory tree  |  -  |  -  | `*1` | yep  | yep   | yep  | yep | yep  |
+| navpane         |  -  |  -  | `*1` | yep  | yep   | yep  | yep | yep  |
 | up2k            |  -  |  -  | yep  | yep  | yep   | yep  | yep | yep  |
 | markdown editor |  -  |  -  | yep  | yep  | yep   | yep  | yep | yep  |
 | markdown viewer |  -  |  -  | yep  | yep  | yep   | yep  | yep | yep  |

contrib/systemd/copyparty.service

@@ -13,6 +13,10 @@
 #   But note that journalctl will get the timestamps wrong due to
 #   python disabling line-buffering, so messages are out-of-order:
 #   https://user-images.githubusercontent.com/241032/126040249-cb535cc7-c599-4931-a796-a5d9af691bad.png
+#
+# enable line-buffering for realtime logging (slight performance cost):
+#   modify ExecStart and prefix it with `/bin/stdbuf -oL` like so:
+#   ExecStart=/bin/stdbuf -oL /usr/bin/python3 [...]
 
 [Unit]
 Description=copyparty file server

copyparty/__main__.py

@@ -199,24 +199,30 @@ def run_argparse(argv, formatter):
         epilog=dedent(
             """
             -a takes username:password,
-            -v takes src:dst:permset:permset:cflag:cflag:...
-               where "permset" is accesslevel followed by username (no separator)
+            -v takes src:dst:perm1:perm2:permN:cflag1:cflag2:cflagN:...
+               where "perm" is "accesslevels,username1,username2,..."
                and "cflag" is config flags to set on this volume
             
+            list of accesslevels:
+              "r" (read):   list folder contents, download files
+              "w" (write):  upload files; need "r" to see the uploads
+              "m" (move):   move files and folders; need "w" at destination
+              "d" (delete): permanently delete files and folders
+
             list of cflags:
-              "cnodupe" rejects existing files (instead of symlinking them)
-              "ce2d" sets -e2d (all -e2* args can be set using ce2* cflags)
-              "cd2t" disables metadata collection, overrides -e2t*
-              "cd2d" disables all database stuff, overrides -e2*
+              "c,nodupe" rejects existing files (instead of symlinking them)
+              "c,e2d" sets -e2d (all -e2* args can be set using ce2* cflags)
+              "c,d2t" disables metadata collection, overrides -e2t*
+              "c,d2d" disables all database stuff, overrides -e2*
 
             example:\033[35m
-              -a ed:hunter2 -v .::r:aed -v ../inc:dump:w:aed:cnodupe  \033[36m
+              -a ed:hunter2 -v .::r:rw,ed -v ../inc:dump:w:rw,ed:c,nodupe  \033[36m
               mount current directory at "/" with
                * r (read-only) for everyone
-               * a (read+write) for ed
+               * rw (read+write) for ed
               mount ../inc at "/dump" with
                * w (write-only) for everyone
-               * a (read+write) for ed
+               * rw (read+write) for ed
                * reject duplicate files  \033[0m
             
             if no accounts or volumes are configured,
@@ -258,9 +264,12 @@ def run_argparse(argv, formatter):
     ap2.add_argument("-ed", action="store_true", help="enable ?dots")
     ap2.add_argument("-emp", action="store_true", help="enable markdown plugins")
     ap2.add_argument("-mcr", metavar="SEC", type=int, default=60, help="md-editor mod-chk rate")
+    ap2.add_argument("--urlform", metavar="MODE", type=u, default="print,get", help="how to handle url-forms; examples: [stash], [save,get]")
+
+    ap2 = ap.add_argument_group('upload options')
     ap2.add_argument("--dotpart", action="store_true", help="dotfile incomplete uploads")
     ap2.add_argument("--sparse", metavar="MiB", type=int, default=4, help="up2k min.size threshold (mswin-only)")
-    ap2.add_argument("--urlform", metavar="MODE", type=u, default="print,get", help="how to handle url-forms; examples: [stash], [save,get]")
+    ap2.add_argument("--unpost", metavar="SEC", type=int, default=3600*12, help="grace period where uploads can be deleted by the uploader, even without delete permissions; 0=disabled")
 
     ap2 = ap.add_argument_group('network options')
     ap2.add_argument("-i", metavar="IP", type=u, default="0.0.0.0", help="ip to bind (comma-sep.)")
@@ -277,6 +286,8 @@ def run_argparse(argv, formatter):
 
     ap2 = ap.add_argument_group('opt-outs')
     ap2.add_argument("-nw", action="store_true", help="disable writes (benchmark)")
+    ap2.add_argument("--no-del", action="store_true", help="disable delete operations")
+    ap2.add_argument("--no-mv", action="store_true", help="disable move/rename operations")
     ap2.add_argument("-nih", action="store_true", help="no info hostname")
     ap2.add_argument("-nid", action="store_true", help="no info disk-usage")
     ap2.add_argument("--no-zip", action="store_true", help="disable download as zip/tar")
@@ -288,6 +299,7 @@ def run_argparse(argv, formatter):
     ap2 = ap.add_argument_group('logging options')
     ap2.add_argument("-q", action="store_true", help="quiet")
     ap2.add_argument("-lo", metavar="PATH", type=u, help="logfile, example: cpp-%%Y-%%m%%d-%%H%%M%%S.txt.xz")
+    ap2.add_argument("--no-voldump", action="store_true", help="do not list volumes and permissions on startup")
     ap2.add_argument("--log-conn", action="store_true", help="print tcp-server msgs")
     ap2.add_argument("--log-htp", action="store_true", help="print http-server threadpool scaling")
     ap2.add_argument("--ihead", metavar="HEADER", type=u, action='append', help="dump incoming header")
@@ -310,15 +322,20 @@ def run_argparse(argv, formatter):
     ap2.add_argument("--th-maxage", metavar="SEC", type=int, default=604800, help="max folder age")
     ap2.add_argument("--th-covers", metavar="N,N", type=u, default="folder.png,folder.jpg,cover.png,cover.jpg", help="folder thumbnails to stat for")
 
-    ap2 = ap.add_argument_group('database options')
+    ap2 = ap.add_argument_group('general db options')
     ap2.add_argument("-e2d", action="store_true", help="enable up2k database")
     ap2.add_argument("-e2ds", action="store_true", help="enable up2k db-scanner, sets -e2d")
     ap2.add_argument("-e2dsa", action="store_true", help="scan all folders (for search), sets -e2ds")
+    ap2.add_argument("--hist", metavar="PATH", type=u, help="where to store volume state")
+    ap2.add_argument("--no-hash", action="store_true", help="disable hashing during e2ds folder scans")
+    ap2.add_argument("--re-int", metavar="SEC", type=int, default=30, help="disk rescan check interval")
+    ap2.add_argument("--re-maxage", metavar="SEC", type=int, default=0, help="disk rescan volume interval (0=off)")
+    ap2.add_argument("--srch-time", metavar="SEC", type=int, default=30, help="search deadline")
+    
+    ap2 = ap.add_argument_group('metadata db options')
     ap2.add_argument("-e2t", action="store_true", help="enable metadata indexing")
     ap2.add_argument("-e2ts", action="store_true", help="enable metadata scanner, sets -e2t")
     ap2.add_argument("-e2tsr", action="store_true", help="rescan all metadata, sets -e2ts")
-    ap2.add_argument("--hist", metavar="PATH", type=u, help="where to store volume state")
-    ap2.add_argument("--no-hash", action="store_true", help="disable hashing during e2ds folder scans")
     ap2.add_argument("--no-mutagen", action="store_true", help="use FFprobe for tags instead")
     ap2.add_argument("--no-mtag-mt", action="store_true", help="disable tag-read parallelism")
     ap2.add_argument("--no-mtag-ff", action="store_true", help="never use FFprobe as tag reader")
@@ -326,7 +343,6 @@ def run_argparse(argv, formatter):
     ap2.add_argument("-mte", metavar="M,M,M", type=u, help="tags to index/display (comma-sep.)",
         default="circle,album,.tn,artist,title,.bpm,key,.dur,.q,.vq,.aq,ac,vc,res,.fps")
     ap2.add_argument("-mtp", metavar="M=[f,]bin", type=u, action="append", help="read tag M using bin")
-    ap2.add_argument("--srch-time", metavar="SEC", type=int, default=30, help="search deadline")
 
     ap2 = ap.add_argument_group('appearance options')
     ap2.add_argument("--css-browser", metavar="L", type=u, help="URL to additional CSS to include")
@@ -377,6 +393,36 @@ def main(argv=None):
     except AssertionError:
         al = run_argparse(argv, Dodge11874)
 
+    nstrs = []
+    anymod = False
+    for ostr in al.v or []:
+        mod = False
+        oa = ostr.split(":")
+        na = oa[:2]
+        for opt in oa[2:]:
+            if re.match("c[^,]", opt):
+                mod = True
+                na.append("c," + opt[1:])
+            elif re.sub("^[rwmd]*", "", opt) and "," not in opt:
+                mod = True
+                perm = opt[0]
+                if perm == "a":
+                    perm = "rw"
+                na.append(perm + "," + opt[1:])
+            else:
+                na.append(opt)
+
+        nstr = ":".join(na)
+        nstrs.append(nstr if mod else ostr)
+        if mod:
+            msg = "\033[1;31mWARNING:\033[0;1m\n  -v {} \033[0;33mwas replaced with\033[0;1m\n  -v {} \n\033[0m"
+            lprint(msg.format(ostr, nstr))
+            anymod = True
+
+    if anymod:
+        al.v = nstrs
+        time.sleep(2)
+
     # propagate implications
     for k1, k2 in IMPLICATIONS:
         if getattr(al, k1):

copyparty/__version__.py

@@ -1,8 +1,8 @@
 # coding: utf-8
 
-VERSION = (0, 11, 45)
-CODENAME = "the grid"
-BUILD_DT = (2021, 7, 20)
+VERSION = (0, 12, 5)
+CODENAME = "fil\033[33med"
+BUILD_DT = (2021, 7, 30)
 
 S_VERSION = ".".join(map(str, VERSION))
 S_BUILD_DT = "{0:04d}-{1:02d}-{2:02d}".format(*BUILD_DT)

copyparty/authsrv.py

@@ -10,20 +10,35 @@ import hashlib
 import threading
 
 from .__init__ import WINDOWS
-from .util import IMPLICATIONS, uncyg, undot, Pebkac, fsdec, fsenc, statdir
+from .util import IMPLICATIONS, uncyg, undot, absreal, Pebkac, fsdec, fsenc, statdir
+from .bos import bos
+
+
+class AXS(object):
+    def __init__(self, uread=None, uwrite=None, umove=None, udel=None):
+        self.uread = {} if uread is None else {k: 1 for k in uread}
+        self.uwrite = {} if uwrite is None else {k: 1 for k in uwrite}
+        self.umove = {} if umove is None else {k: 1 for k in umove}
+        self.udel = {} if udel is None else {k: 1 for k in udel}
+
+    def __repr__(self):
+        return "AXS({})".format(
+            ", ".join(
+                "{}={!r}".format(k, self.__dict__[k])
+                for k in "uread uwrite umove udel".split()
+            )
+        )
 
 
 class VFS(object):
     """single level in the virtual fs"""
 
-    def __init__(self, log, realpath, vpath, uread, uwrite, uadm, flags):
+    def __init__(self, log, realpath, vpath, axs, flags):
         self.log = log
         self.realpath = realpath  # absolute path on host filesystem
         self.vpath = vpath  # absolute path in the virtual filesystem
-        self.uread = uread  # users who can read this
-        self.uwrite = uwrite  # users who can write this
-        self.uadm = uadm  # users who are regular admins
-        self.flags = flags  # config switches
+        self.axs = axs  # type: AXS
+        self.flags = flags  # config options
         self.nodes = {}  # child nodes
         self.histtab = None  # all realpath->histpath
         self.dbv = None  # closest full/non-jump parent
@@ -31,15 +46,23 @@ class VFS(object):
         if realpath:
             self.histpath = os.path.join(realpath, ".hist")  # db / thumbcache
             self.all_vols = {vpath: self}  # flattened recursive
+            self.aread = {}
+            self.awrite = {}
+            self.amove = {}
+            self.adel = {}
         else:
             self.histpath = None
             self.all_vols = None
+            self.aread = None
+            self.awrite = None
+            self.amove = None
+            self.adel = None
 
     def __repr__(self):
         return "VFS({})".format(
             ", ".join(
                 "{}={!r}".format(k, self.__dict__[k])
-                for k in "realpath vpath uread uwrite uadm flags".split()
+                for k in "realpath vpath axs flags".split()
             )
         )
 
@@ -66,9 +89,7 @@ class VFS(object):
                 self.log,
                 os.path.join(self.realpath, name) if self.realpath else None,
                 "{}/{}".format(self.vpath, name).lstrip("/"),
-                self.uread,
-                self.uwrite,
-                self.uadm,
+                self.axs,
                 self._copy_flags(name),
             )
             vn.dbv = self.dbv or self
@@ -81,7 +102,7 @@ class VFS(object):
 
         # leaf does not exist; create and keep permissions blank
         vp = "{}/{}".format(self.vpath, dst).lstrip("/")
-        vn = VFS(self.log, src, vp, [], [], [], {})
+        vn = VFS(self.log, src, vp, AXS(), {})
         vn.dbv = self.dbv or self
         self.nodes[dst] = vn
         return vn
@@ -121,23 +142,32 @@ class VFS(object):
         return [self, vpath]
 
     def can_access(self, vpath, uname):
-        """return [readable,writable]"""
+        # type: (str, str) -> tuple[bool, bool, bool, bool]
+        """can Read,Write,Move,Delete"""
         vn, _ = self._find(vpath)
+        c = vn.axs
         return [
-            uname in vn.uread or "*" in vn.uread,
-            uname in vn.uwrite or "*" in vn.uwrite,
+            uname in c.uread or "*" in c.uread,
+            uname in c.uwrite or "*" in c.uwrite,
+            uname in c.umove or "*" in c.umove,
+            uname in c.udel or "*" in c.udel,
         ]
 
-    def get(self, vpath, uname, will_read, will_write):
-        # type: (str, str, bool, bool) -> tuple[VFS, str]
+    def get(self, vpath, uname, will_read, will_write, will_move=False, will_del=False):
+        # type: (str, str, bool, bool, bool, bool) -> tuple[VFS, str]
         """returns [vfsnode,fs_remainder] if user has the requested permissions"""
         vn, rem = self._find(vpath)
+        c = vn.axs
 
-        if will_read and (uname not in vn.uread and "*" not in vn.uread):
-            raise Pebkac(403, "you don't have read-access for this location")
-
-        if will_write and (uname not in vn.uwrite and "*" not in vn.uwrite):
-            raise Pebkac(403, "you don't have write-access for this location")
+        for req, d, msg in [
+            [will_read, c.uread, "read"],
+            [will_write, c.uwrite, "write"],
+            [will_move, c.umove, "move"],
+            [will_del, c.udel, "delete"],
+        ]:
+            if req and (uname not in d and "*" not in d):
+                m = "you don't have {}-access for this location"
+                raise Pebkac(403, m.format(msg))
 
         return vn, rem
 
@@ -150,65 +180,50 @@ class VFS(object):
         vrem = "/".join([x for x in vrem if x])
         return dbv, vrem
 
-    def canonical(self, rem):
+    def canonical(self, rem, resolve=True):
         """returns the canonical path (fully-resolved absolute fs path)"""
         rp = self.realpath
         if rem:
             rp += "/" + rem
 
-        try:
-            return fsdec(os.path.realpath(fsenc(rp)))
-        except:
-            if not WINDOWS:
-                raise
+        return absreal(rp) if resolve else rp
 
-            # cpython bug introduced in 3.8, still exists in 3.9.1;
-            # some win7sp1 and win10:20H2 boxes cannot realpath a
-            # networked drive letter such as b"n:" or b"n:\\"
-            #
-            # requirements to trigger:
-            #  * bytestring (not unicode str)
-            #  * just the drive letter (subfolders are ok)
-            #  * networked drive (regular disks and vmhgfs are ok)
-            #  * on an enterprise network (idk, cannot repro with samba)
-            #
-            # hits the following exceptions in succession:
-            #  * access denied at L601: "path = _getfinalpathname(path)"
-            #  * "cant concat str to bytes" at L621: "return path + tail"
-            #
-            return os.path.realpath(rp)
-
-    def ls(self, rem, uname, scandir, incl_wo=False, lstat=False):
-        # type: (str, str, bool, bool, bool) -> tuple[str, str, dict[str, VFS]]
+    def ls(self, rem, uname, scandir, permsets, lstat=False):
+        # type: (str, str, bool, list[list[bool]], bool) -> tuple[str, str, dict[str, VFS]]
         """return user-readable [fsdir,real,virt] items at vpath"""
         virt_vis = {}  # nodes readable by user
         abspath = self.canonical(rem)
         real = list(statdir(self.log, scandir, lstat, abspath))
         real.sort()
         if not rem:
-            for name, vn2 in sorted(self.nodes.items()):
-                ok = uname in vn2.uread or "*" in vn2.uread
+            # no vfs nodes in the list of real inodes
+            real = [x for x in real if x[0] not in self.nodes]
 
-                if not ok and incl_wo:
-                    ok = uname in vn2.uwrite or "*" in vn2.uwrite
+            for name, vn2 in sorted(self.nodes.items()):
+                ok = False
+                axs = vn2.axs
+                axs = [axs.uread, axs.uwrite, axs.umove, axs.udel]
+                for pset in permsets:
+                    ok = True
+                    for req, lst in zip(pset, axs):
+                        if req and uname not in lst and "*" not in lst:
+                            ok = False
+                    if ok:
+                        break
 
                 if ok:
                     virt_vis[name] = vn2
 
-            # no vfs nodes in the list of real inodes
-            real = [x for x in real if x[0] not in self.nodes]
-
         return [abspath, real, virt_vis]
 
-    def walk(self, rel, rem, seen, uname, dots, scandir, lstat):
+    def walk(self, rel, rem, seen, uname, permsets, dots, scandir, lstat):
         """
         recursively yields from ./rem;
         rel is a unix-style user-defined vpath (not vfs-related)
         """
 
-        fsroot, vfs_ls, vfs_virt = self.ls(
-            rem, uname, scandir, incl_wo=False, lstat=lstat
-        )
+        fsroot, vfs_ls, vfs_virt = self.ls(rem, uname, scandir, permsets, lstat=lstat)
+        dbv, vrem = self.get_dbv(rem)
 
         if (
             seen
@@ -226,7 +241,7 @@ class VFS(object):
         rfiles.sort()
         rdirs.sort()
 
-        yield rel, fsroot, rfiles, rdirs, vfs_virt
+        yield dbv, vrem, rel, fsroot, rfiles, rdirs, vfs_virt
 
         for rdir, _ in rdirs:
             if not dots and rdir.startswith("."):
@@ -234,7 +249,7 @@ class VFS(object):
 
             wrel = (rel + "/" + rdir).lstrip("/")
             wrem = (rem + "/" + rdir).lstrip("/")
-            for x in self.walk(wrel, wrem, seen, uname, dots, scandir, lstat):
+            for x in self.walk(wrel, wrem, seen, uname, permsets, dots, scandir, lstat):
                 yield x
 
         for n, vfs in sorted(vfs_virt.items()):
@@ -242,7 +257,7 @@ class VFS(object):
                 continue
 
             wrel = (rel + "/" + n).lstrip("/")
-            for x in vfs.walk(wrel, "", seen, uname, dots, scandir, lstat):
+            for x in vfs.walk(wrel, "", seen, uname, permsets, dots, scandir, lstat):
                 yield x
 
     def zipgen(self, vrem, flt, uname, dots, scandir):
@@ -253,9 +268,8 @@ class VFS(object):
         f2a = os.sep + "dir.txt"
         f2b = "{0}.hist{0}".format(os.sep)
 
-        for vpath, apath, files, rd, vd in self.walk(
-            "", vrem, [], uname, dots, scandir, False
-        ):
+        g = self.walk("", vrem, [], uname, [[True]], dots, scandir, False)
+        for _, _, vpath, apath, files, rd, vd in g:
             if flt:
                 files = [x for x in files if x[0] in flt]
 
@@ -295,20 +309,6 @@ class VFS(object):
             for f in [{"vp": v, "ap": a, "st": n[1]} for v, a, n in files]:
                 yield f
 
-    def user_tree(self, uname, readable, writable, admin):
-        is_readable = False
-        if uname in self.uread or "*" in self.uread:
-            readable.append(self.vpath)
-            is_readable = True
-
-        if uname in self.uwrite or "*" in self.uwrite:
-            writable.append(self.vpath)
-            if is_readable:
-                admin.append(self.vpath)
-
-        for _, vn in sorted(self.nodes.items()):
-            vn.user_tree(uname, readable, writable, admin)
-
 
 class AuthSrv(object):
     """verifies users against given paths"""
@@ -341,7 +341,8 @@ class AuthSrv(object):
 
         yield prev, True
 
-    def _parse_config_file(self, fd, user, mread, mwrite, madm, mflags, mount):
+    def _parse_config_file(self, fd, acct, daxs, mflags, mount):
+        # type: (any, str, dict[str, AXS], any, str) -> None
         vol_src = None
         vol_dst = None
         self.line_ctr = 0
@@ -357,7 +358,7 @@ class AuthSrv(object):
             if vol_src is None:
                 if ln.startswith("u "):
                     u, p = ln[2:].split(":", 1)
-                    user[u] = p
+                    acct[u] = p
                 else:
                     vol_src = ln
                 continue
@@ -368,50 +369,50 @@ class AuthSrv(object):
                     raise Exception('invalid mountpoint "{}"'.format(vol_dst))
 
                 # cfg files override arguments and previous files
-                vol_src = fsdec(os.path.abspath(fsenc(vol_src)))
+                vol_src = bos.path.abspath(vol_src)
                 vol_dst = vol_dst.strip("/")
                 mount[vol_dst] = vol_src
-                mread[vol_dst] = []
-                mwrite[vol_dst] = []
-                madm[vol_dst] = []
+                daxs[vol_dst] = AXS()
                 mflags[vol_dst] = {}
                 continue
 
-            if len(ln) > 1:
-                lvl, uname = ln.split(" ")
-            else:
+            try:
+                lvl, uname = ln.split(" ", 1)
+            except:
                 lvl = ln
                 uname = "*"
 
-            self._read_vol_str(
-                lvl,
-                uname,
-                mread[vol_dst],
-                mwrite[vol_dst],
-                madm[vol_dst],
-                mflags[vol_dst],
-            )
+            if lvl == "a":
+                m = "WARNING (config-file): permission flag 'a' is deprecated; please use 'rw' instead"
+                self.log(m, 1)
 
-    def _read_vol_str(self, lvl, uname, mr, mw, ma, mf):
+            self._read_vol_str(lvl, uname, daxs[vol_dst], mflags[vol_dst])
+
+    def _read_vol_str(self, lvl, uname, axs, flags):
+        # type: (str, str, AXS, any) -> None
         if lvl == "c":
             cval = True
             if "=" in uname:
                 uname, cval = uname.split("=", 1)
 
-            self._read_volflag(mf, uname, cval, False)
+            self._read_volflag(flags, uname, cval, False)
             return
 
         if uname == "":
             uname = "*"
 
-        if lvl in "ra":
-            mr.append(uname)
+        for un in uname.split(","):
+            if "r" in lvl:
+                axs.uread[un] = 1
 
-        if lvl in "wa":
-            mw.append(uname)
+            if "w" in lvl:
+                axs.uwrite[un] = 1
 
-        if lvl == "a":
-            ma.append(uname)
+            if "m" in lvl:
+                axs.umove[un] = 1
+
+            if "d" in lvl:
+                axs.udel[un] = 1
 
     def _read_volflag(self, flags, name, value, is_list):
         if name not in ["mtp"]:
@@ -433,21 +434,24 @@ class AuthSrv(object):
         before finally building the VFS
         """
 
-        user = {}  # username:password
-        mread = {}  # mountpoint:[username]
-        mwrite = {}  # mountpoint:[username]
-        madm = {}  # mountpoint:[username]
+        acct = {}  # username:password
+        daxs = {}  # type: dict[str, AXS]
         mflags = {}  # mountpoint:[flag]
         mount = {}  # dst:src (mountpoint:realpath)
 
         if self.args.a:
             # list of username:password
-            for u, p in [x.split(":", 1) for x in self.args.a]:
-                user[u] = p
+            for x in self.args.a:
+                try:
+                    u, p = x.split(":", 1)
+                    acct[u] = p
+                except:
+                    m = '\n  invalid value "{}" for argument -a, must be username:password'
+                    raise Exception(m.format(x))
 
         if self.args.v:
             # list of src:dst:permset:permset:...
-            # permset is [rwa]username or [c]flag
+            # permset is <rwmd>[,username][,username] or <c>,<flag>[=args]
             for v_str in self.args.v:
                 m = self.re_vol.match(v_str)
                 if not m:
@@ -458,27 +462,21 @@ class AuthSrv(object):
                     src = uncyg(src)
 
                 # print("\n".join([src, dst, perms]))
-                src = fsdec(os.path.abspath(fsenc(src)))
+                src = bos.path.abspath(src)
                 dst = dst.strip("/")
                 mount[dst] = src
-                mread[dst] = []
-                mwrite[dst] = []
-                madm[dst] = []
+                daxs[dst] = AXS()
                 mflags[dst] = {}
 
-                perms = perms.split(":")
-                for (lvl, uname) in [[x[0], x[1:]] for x in perms]:
-                    self._read_vol_str(
-                        lvl, uname, mread[dst], mwrite[dst], madm[dst], mflags[dst]
-                    )
+                for x in perms.split(":"):
+                    lvl, uname = x.split(",", 1) if "," in x else [x, ""]
+                    self._read_vol_str(lvl, uname, daxs[dst], mflags[dst])
 
         if self.args.c:
             for cfg_fn in self.args.c:
                 with open(cfg_fn, "rb") as f:
                     try:
-                        self._parse_config_file(
-                            f, user, mread, mwrite, madm, mflags, mount
-                        )
+                        self._parse_config_file(f, acct, daxs, mflags, mount)
                     except:
                         m = "\n\033[1;31m\nerror in config file {} on line {}:\n\033[0m"
                         self.log(m.format(cfg_fn, self.line_ctr), 1)
@@ -488,19 +486,17 @@ class AuthSrv(object):
         if WINDOWS:
             cased = {}
             for k, v in mount.items():
-                try:
-                    cased[k] = fsdec(os.path.realpath(fsenc(v)))
-                except:
-                    cased[k] = v
+                cased[k] = absreal(v)
 
             mount = cased
 
         if not mount:
             # -h says our defaults are CWD at root and read/write for everyone
-            vfs = VFS(self.log_func, os.path.abspath("."), "", ["*"], ["*"], ["*"], {})
+            axs = AXS(["*"], ["*"], None, None)
+            vfs = VFS(self.log_func, bos.path.abspath("."), "", axs, {})
         elif "" not in mount:
             # there's volumes but no root; make root inaccessible
-            vfs = VFS(self.log_func, None, "", [], [], [], {})
+            vfs = VFS(self.log_func, None, "", AXS(), {})
             vfs.flags["d2d"] = True
 
         maxdepth = 0
@@ -511,32 +507,34 @@ class AuthSrv(object):
 
             if dst == "":
                 # rootfs was mapped; fully replaces the default CWD vfs
-                vfs = VFS(
-                    self.log_func,
-                    mount[dst],
-                    dst,
-                    mread[dst],
-                    mwrite[dst],
-                    madm[dst],
-                    mflags[dst],
-                )
+                vfs = VFS(self.log_func, mount[dst], dst, daxs[dst], mflags[dst])
                 continue
 
             v = vfs.add(mount[dst], dst)
-            v.uread = mread[dst]
-            v.uwrite = mwrite[dst]
-            v.uadm = madm[dst]
+            v.axs = daxs[dst]
             v.flags = mflags[dst]
             v.dbv = None
 
         vfs.all_vols = {}
         vfs.get_all_vols(vfs.all_vols)
 
+        for perm in "read write move del".split():
+            axs_key = "u" + perm
+            unames = ["*"] + list(acct.keys())
+            umap = {x: [] for x in unames}
+            for usr in unames:
+                for mp, vol in vfs.all_vols.items():
+                    if usr in getattr(vol.axs, axs_key):
+                        umap[usr].append(mp)
+            setattr(vfs, "a" + perm, umap)
+
+        all_users = {}
         missing_users = {}
-        for d in [mread, mwrite]:
-            for _, ul in d.items():
-                for usr in ul:
-                    if usr != "*" and usr not in user:
+        for axs in daxs.values():
+            for d in [axs.uread, axs.uwrite, axs.umove, axs.udel]:
+                for usr in d.keys():
+                    all_users[usr] = 1
+                    if usr != "*" and usr not in acct:
                         missing_users[usr] = 1
 
         if missing_users:
@@ -560,10 +558,7 @@ class AuthSrv(object):
             elif self.args.hist:
                 for nch in range(len(hid)):
                     hpath = os.path.join(self.args.hist, hid[: nch + 1])
-                    try:
-                        os.makedirs(hpath)
-                    except:
-                        pass
+                    bos.makedirs(hpath)
 
                     powner = os.path.join(hpath, "owner.txt")
                     try:
@@ -583,9 +578,9 @@ class AuthSrv(object):
                     vol.histpath = hpath
                     break
 
-            vol.histpath = os.path.realpath(vol.histpath)
+            vol.histpath = absreal(vol.histpath)
             if vol.dbv:
-                if os.path.exists(os.path.join(vol.histpath, "up2k.db")):
+                if bos.path.exists(os.path.join(vol.histpath, "up2k.db")):
                     promote.append(vol)
                     vol.dbv = None
                 else:
@@ -611,7 +606,7 @@ class AuthSrv(object):
         all_mte = {}
         errors = False
         for vol in vfs.all_vols.values():
-            if (self.args.e2ds and vol.uwrite) or self.args.e2dsa:
+            if (self.args.e2ds and vol.axs.uwrite) or self.args.e2dsa:
                 vol.flags["e2ds"] = True
 
             if self.args.e2d or "e2ds" in vol.flags:
@@ -700,6 +695,27 @@ class AuthSrv(object):
 
         vfs.bubble_flags()
 
+        m = "volumes and permissions:\n"
+        for v in vfs.all_vols.values():
+            if not self.warn_anonwrite:
+                break
+
+            m += '\n\033[36m"/{}"  \033[33m{}\033[0m'.format(v.vpath, v.realpath)
+            for txt, attr in [
+                ["  read", "uread"],
+                [" write", "uwrite"],
+                ["  move", "umove"],
+                ["delete", "udel"],
+            ]:
+                u = list(sorted(getattr(v.axs, attr).keys()))
+                u = ", ".join("\033[35meverybody\033[0m" if x == "*" else x for x in u)
+                u = u if u else "\033[36m--none--\033[0m"
+                m += "\n|  {}:  {}".format(txt, u)
+            m += "\n"
+
+        if self.warn_anonwrite and not self.args.no_voldump:
+            self.log(m)
+
         try:
             v, _ = vfs.get("/", "*", False, True)
             if self.warn_anonwrite and os.getcwd() == v.realpath:
@@ -711,17 +727,14 @@ class AuthSrv(object):
 
         with self.mutex:
             self.vfs = vfs
-            self.user = user
-            self.iuser = {v: k for k, v in user.items()}
+            self.acct = acct
+            self.iacct = {v: k for k, v in acct.items()}
 
             self.re_pwd = None
-            pwds = [re.escape(x) for x in self.iuser.keys()]
+            pwds = [re.escape(x) for x in self.iacct.keys()]
             if pwds:
                 self.re_pwd = re.compile("=(" + "|".join(pwds) + ")([]&; ]|$)")
 
-        # import pprint
-        # pprint.pprint({"usr": user, "rd": mread, "wr": mwrite, "mnt": mount})
-
     def dbg_ls(self):
         users = self.args.ls
         vols = "*"
@@ -739,12 +752,12 @@ class AuthSrv(object):
             pass
 
         if users == "**":
-            users = list(self.user.keys()) + ["*"]
+            users = list(self.acct.keys()) + ["*"]
         else:
             users = [users]
 
         for u in users:
-            if u not in self.user and u != "*":
+            if u not in self.acct and u != "*":
                 raise Exception("user not found: " + u)
 
         if vols == "*":
@@ -760,8 +773,10 @@ class AuthSrv(object):
                 raise Exception("volume not found: " + v)
 
         self.log({"users": users, "vols": vols, "flags": flags})
+        m = "/{}: read({}) write({}) move({}) del({})"
         for k, v in self.vfs.all_vols.items():
-            self.log("/{}: read({}) write({})".format(k, v.uread, v.uwrite))
+            vc = v.axs
+            self.log(m.format(k, vc.uread, vc.uwrite, vc.umove, vc.udel))
 
         flag_v = "v" in flags
         flag_ln = "ln" in flags
@@ -775,13 +790,15 @@ class AuthSrv(object):
             for u in users:
                 self.log("checking /{} as {}".format(v, u))
                 try:
-                    vn, _ = self.vfs.get(v, u, True, False)
+                    vn, _ = self.vfs.get(v, u, True, False, False, False)
                 except:
                     continue
 
                 atop = vn.realpath
-                g = vn.walk("", "", [], u, True, not self.args.no_scandir, False)
-                for vpath, apath, files, _, _ in g:
+                g = vn.walk(
+                    vn.vpath, "", [], u, [[True]], True, not self.args.no_scandir, False
+                )
+                for _, _, vpath, apath, files, _, _ in g:
                     fnames = [n[0] for n in files]
                     vpaths = [vpath + "/" + n for n in fnames] if vpath else fnames
                     vpaths = [vtop + x for x in vpaths]

copyparty/bos/bos.py

@@ -0,0 +1,59 @@
+# coding: utf-8
+from __future__ import print_function, unicode_literals
+
+import os
+from ..util import fsenc, fsdec
+from . import path
+
+
+# grep -hRiE '(^|[^a-zA-Z_\.-])os\.' . | gsed -r 's/ /\n/g;s/\(/(\n/g' | grep -hRiE '(^|[^a-zA-Z_\.-])os\.' | sort | uniq -c
+# printf 'os\.(%s)' "$(grep ^def bos/__init__.py | gsed -r 's/^def //;s/\(.*//' | tr '\n' '|' | gsed -r 's/.$//')"
+
+
+def chmod(p, mode):
+    return os.chmod(fsenc(p), mode)
+
+
+def listdir(p="."):
+    return [fsdec(x) for x in os.listdir(fsenc(p))]
+
+
+def lstat(p):
+    return os.lstat(fsenc(p))
+
+
+def makedirs(name, mode=0o755, exist_ok=True):
+    bname = fsenc(name)
+    try:
+        os.makedirs(bname, mode=mode)
+    except:
+        if not exist_ok or not os.path.isdir(bname):
+            raise
+
+
+def mkdir(p, mode=0o755):
+    return os.mkdir(fsenc(p), mode=mode)
+
+
+def rename(src, dst):
+    return os.rename(fsenc(src), fsenc(dst))
+
+
+def replace(src, dst):
+    return os.replace(fsenc(src), fsenc(dst))
+
+
+def rmdir(p):
+    return os.rmdir(fsenc(p))
+
+
+def stat(p):
+    return os.stat(fsenc(p))
+
+
+def unlink(p):
+    return os.unlink(fsenc(p))
+
+
+def utime(p, times=None):
+    return os.utime(fsenc(p), times)

copyparty/bos/path.py

@@ -0,0 +1,33 @@
+# coding: utf-8
+from __future__ import print_function, unicode_literals
+
+import os
+from ..util import fsenc, fsdec
+
+
+def abspath(p):
+    return fsdec(os.path.abspath(fsenc(p)))
+
+
+def exists(p):
+    return os.path.exists(fsenc(p))
+
+
+def getmtime(p):
+    return os.path.getmtime(fsenc(p))
+
+
+def getsize(p):
+    return os.path.getsize(fsenc(p))
+
+
+def isdir(p):
+    return os.path.isdir(fsenc(p))
+
+
+def islink(p):
+    return os.path.islink(fsenc(p))
+
+
+def realpath(p):
+    return fsdec(os.path.realpath(fsenc(p)))

copyparty/broker_mp.py

@@ -22,12 +22,9 @@ class BrokerMp(object):
         self.retpend_mutex = threading.Lock()
         self.mutex = threading.Lock()
 
-        cores = self.args.j
-        if not cores:
-            cores = mp.cpu_count()
-
-        self.log("broker", "booting {} subprocesses".format(cores))
-        for n in range(1, cores + 1):
+        self.num_workers = self.args.j or mp.cpu_count()
+        self.log("broker", "booting {} subprocesses".format(self.num_workers))
+        for n in range(1, self.num_workers + 1):
             q_pend = mp.Queue(1)
             q_yield = mp.Queue(64)
 
@@ -103,5 +100,8 @@ class BrokerMp(object):
             for p in self.procs:
                 p.q_pend.put([0, dest, [args[0], len(self.procs)]])
 
+        elif dest == "cb_httpsrv_up":
+            self.hub.cb_httpsrv_up()
+
         else:
             raise Exception("what is " + str(dest))

copyparty/broker_thr.py

@@ -17,6 +17,7 @@ class BrokerThr(object):
         self.asrv = hub.asrv
 
         self.mutex = threading.Lock()
+        self.num_workers = 1
 
         # instantiate all services here (TODO: inheritance?)
         self.httpsrv = HttpSrv(self, None)

copyparty/httpcli.py

@@ -15,6 +15,7 @@ import calendar
 
 from .__init__ import E, PY2, WINDOWS, ANYWIN, unicode
 from .util import *  # noqa  # pylint: disable=unused-wildcard-import
+from .bos import bos
 from .authsrv import AuthSrv
 from .szip import StreamZip
 from .star import StreamTar
@@ -58,9 +59,12 @@ class HttpCli(object):
 
     def unpwd(self, m):
         a, b = m.groups()
-        return "=\033[7m {} \033[27m{}".format(self.asrv.iuser[a], b)
+        return "=\033[7m {} \033[27m{}".format(self.asrv.iacct[a], b)
+
+    def _check_nonfatal(self, ex, post):
+        if post:
+            return ex.code < 300
 
-    def _check_nonfatal(self, ex):
         return ex.code < 400 or ex.code in [404, 429]
 
     def _assert_safe_rem(self, rem):
@@ -102,7 +106,7 @@ class HttpCli(object):
             self.req = "[junk]"
             self.http_ver = "HTTP/1.1"
             # self.log("pebkac at httpcli.run #1: " + repr(ex))
-            self.keepalive = self._check_nonfatal(ex)
+            self.keepalive = False
             self.loud_reply(unicode(ex), status=ex.code)
             return self.keepalive
 
@@ -178,12 +182,14 @@ class HttpCli(object):
 
         self.uparam = uparam
         self.cookies = cookies
-        self.vpath = unquotep(vpath)
+        self.vpath = unquotep(vpath)  # not query, so + means +
 
         pwd = uparam.get("pw")
-        self.uname = self.asrv.iuser.get(pwd, "*")
-        self.rvol, self.wvol, self.avol = [[], [], []]
-        self.asrv.vfs.user_tree(self.uname, self.rvol, self.wvol, self.avol)
+        self.uname = self.asrv.iacct.get(pwd, "*")
+        self.rvol = self.asrv.vfs.aread[self.uname]
+        self.wvol = self.asrv.vfs.awrite[self.uname]
+        self.mvol = self.asrv.vfs.amove[self.uname]
+        self.dvol = self.asrv.vfs.adel[self.uname]
 
         if pwd and "pw" in self.ouparam and pwd != cookies.get("cppwd"):
             self.out_headers["Set-Cookie"] = self.get_pwd_cookie(pwd)[0]
@@ -213,7 +219,8 @@ class HttpCli(object):
         except Pebkac as ex:
             try:
                 # self.log("pebkac at httpcli.run #2: " + repr(ex))
-                if not self._check_nonfatal(ex):
+                post = self.mode in ["POST", "PUT"] or "content-length" in self.headers
+                if not self._check_nonfatal(ex, post):
                     self.keepalive = False
 
                 self.log("{}\033[0m, {}".format(str(ex), self.vpath), 3)
@@ -339,11 +346,36 @@ class HttpCli(object):
             static_path = os.path.join(E.mod, "web/", self.vpath[5:])
             return self.tx_file(static_path)
 
+        x = self.asrv.vfs.can_access(self.vpath, self.uname)
+        self.can_read, self.can_write, self.can_move, self.can_delete = x
+        if not self.can_read and not self.can_write:
+            if self.vpath:
+                self.log("inaccessible: [{}]".format(self.vpath))
+                raise Pebkac(404)
+
+            self.uparam["h"] = False
+
         if "tree" in self.uparam:
             return self.tx_tree()
 
-        if "stack" in self.uparam:
-            return self.tx_stack()
+        if "delete" in self.uparam:
+            return self.handle_rm()
+
+        if "move" in self.uparam:
+            return self.handle_mv()
+
+        if "scan" in self.uparam:
+            return self.scanvol()
+
+        if not self.vpath:
+            if "stack" in self.uparam:
+                return self.tx_stack()
+
+            if "ups" in self.uparam:
+                return self.tx_ups()
+
+            if "h" in self.uparam:
+                return self.tx_mounts()
 
         # conditional redirect to single volumes
         if self.vpath == "" and not self.ouparam:
@@ -359,21 +391,6 @@ class HttpCli(object):
                     self.redirect(vpath, flavor="redirecting to", use302=True)
                     return True
 
-        self.readable, self.writable = self.asrv.vfs.can_access(self.vpath, self.uname)
-        if not self.readable and not self.writable:
-            if self.vpath:
-                self.log("inaccessible: [{}]".format(self.vpath))
-                raise Pebkac(404)
-
-            self.uparam = {"h": False}
-
-        if "h" in self.uparam:
-            self.vpath = None
-            return self.tx_mounts()
-
-        if "scan" in self.uparam:
-            return self.scanvol()
-
         return self.tx_browser()
 
     def handle_options(self):
@@ -488,7 +505,14 @@ class HttpCli(object):
         if not self.args.nw:
             vfs, vrem = vfs.get_dbv(rem)
             self.conn.hsrv.broker.put(
-                False, "up2k.hash_file", vfs.realpath, vfs.flags, vrem, fn
+                False,
+                "up2k.hash_file",
+                vfs.realpath,
+                vfs.flags,
+                vrem,
+                fn,
+                self.ip,
+                time.time(),
             )
 
         return post_sz, sha_b64, remains, path
@@ -582,6 +606,9 @@ class HttpCli(object):
         if "srch" in self.uparam or "srch" in body:
             return self.handle_search(body)
 
+        if "delete" in self.uparam:
+            return self.handle_rm(body)
+
         # up2k-php compat
         for k in "chunkpit.php", "handshake.php":
             if self.vpath.endswith(k):
@@ -606,11 +633,11 @@ class HttpCli(object):
         if sub:
             try:
                 dst = os.path.join(vfs.realpath, rem)
-                if not os.path.isdir(fsenc(dst)):
-                    os.makedirs(fsenc(dst))
+                if not bos.path.isdir(dst):
+                    bos.makedirs(dst)
             except OSError as ex:
                 self.log("makedirs failed [{}]".format(dst))
-                if not os.path.isdir(fsenc(dst)):
+                if not bos.path.isdir(dst):
                     if ex.errno == 13:
                         raise Pebkac(500, "the server OS denied write-access")
 
@@ -756,7 +783,7 @@ class HttpCli(object):
             times = (int(time.time()), int(lastmod))
             self.log("no more chunks, setting times {}".format(times))
             try:
-                os.utime(fsenc(path), times)
+                bos.utime(path, times)
             except:
                 self.log("failed to utime ({}, {})".format(path, times))
 
@@ -775,7 +802,7 @@ class HttpCli(object):
         return True
 
     def get_pwd_cookie(self, pwd):
-        if pwd in self.asrv.iuser:
+        if pwd in self.asrv.iacct:
             msg = "login ok"
             dt = datetime.utcfromtimestamp(time.time() + 60 * 60 * 24 * 365)
             exp = dt.strftime("%a, %d %b %Y %H:%M:%S GMT")
@@ -801,14 +828,14 @@ class HttpCli(object):
             fdir = os.path.join(vfs.realpath, rem)
             fn = os.path.join(fdir, sanitized)
 
-            if not os.path.isdir(fsenc(fdir)):
+            if not bos.path.isdir(fdir):
                 raise Pebkac(500, "parent folder does not exist")
 
-            if os.path.isdir(fsenc(fn)):
+            if bos.path.isdir(fn):
                 raise Pebkac(500, "that folder exists already")
 
             try:
-                os.mkdir(fsenc(fn))
+                bos.mkdir(fn)
             except OSError as ex:
                 if ex.errno == 13:
                     raise Pebkac(500, "the server OS denied write-access")
@@ -838,7 +865,7 @@ class HttpCli(object):
             fdir = os.path.join(vfs.realpath, rem)
             fn = os.path.join(fdir, sanitized)
 
-            if os.path.exists(fsenc(fn)):
+            if bos.path.exists(fn):
                 raise Pebkac(500, "that file exists already")
 
             with open(fsenc(fn), "wb") as f:
@@ -868,7 +895,7 @@ class HttpCli(object):
                         p_file, "", [".prologue.html", ".epilogue.html"]
                     )
 
-                    if not os.path.isdir(fsenc(fdir)):
+                    if not bos.path.isdir(fdir):
                         raise Pebkac(404, "that folder does not exist")
 
                     suffix = ".{:.6f}-{}".format(time.time(), self.ip)
@@ -895,6 +922,8 @@ class HttpCli(object):
                             dbv.flags,
                             vrem,
                             fname,
+                            self.ip,
+                            time.time(),
                         )
                         self.conn.nbyte += sz
 
@@ -907,10 +936,10 @@ class HttpCli(object):
 
                         suffix = ".PARTIAL"
                         try:
-                            os.rename(fsenc(fp), fsenc(fp2 + suffix))
+                            bos.rename(fp, fp2 + suffix)
                         except:
                             fp2 = fp2[: -len(suffix) - 1]
-                            os.rename(fsenc(fp), fsenc(fp2 + suffix))
+                            bos.rename(fp, fp2 + suffix)
 
                     raise
 
@@ -994,13 +1023,6 @@ class HttpCli(object):
         vfs, rem = self.asrv.vfs.get(self.vpath, self.uname, False, True)
         self._assert_safe_rem(rem)
 
-        # TODO:
-        #   the per-volume read/write permissions must be replaced with permission flags
-        #   which would decide how to handle uploads to filenames which are taken,
-        #   current behavior of creating a new name is a good default for binary files
-        #   but should also offer a flag to takeover the filename and rename the old one
-        #
-        # stopgap:
         if not rem.endswith(".md"):
             raise Pebkac(400, "only markdown pls")
 
@@ -1015,7 +1037,7 @@ class HttpCli(object):
         fp = os.path.join(vfs.realpath, rem)
         srv_lastmod = srv_lastmod3 = -1
         try:
-            st = os.stat(fsenc(fp))
+            st = bos.stat(fp)
             srv_lastmod = st.st_mtime
             srv_lastmod3 = int(srv_lastmod * 1000)
         except OSError as ex:
@@ -1051,14 +1073,13 @@ class HttpCli(object):
                 self.reply(response.encode("utf-8"))
                 return True
 
-            # TODO another hack re: pending permissions rework
             mdir, mfile = os.path.split(fp)
             mfile2 = "{}.{:.3f}.md".format(mfile[:-3], srv_lastmod)
             try:
-                os.mkdir(fsenc(os.path.join(mdir, ".hist")))
+                bos.mkdir(os.path.join(mdir, ".hist"))
             except:
                 pass
-            os.rename(fsenc(fp), fsenc(os.path.join(mdir, ".hist", mfile2)))
+            bos.rename(fp, os.path.join(mdir, ".hist", mfile2))
 
         p_field, _, p_data = next(self.parser.gen)
         if p_field != "body":
@@ -1067,7 +1088,7 @@ class HttpCli(object):
         with open(fsenc(fp), "wb", 512 * 1024) as f:
             sz, sha512, _ = hashcopy(p_data, f)
 
-        new_lastmod = os.stat(fsenc(fp)).st_mtime
+        new_lastmod = bos.stat(fp).st_mtime
         new_lastmod3 = int(new_lastmod * 1000)
         sha512 = sha512[:56]
 
@@ -1112,7 +1133,7 @@ class HttpCli(object):
         for ext in ["", ".gz", ".br"]:
             try:
                 fs_path = req_path + ext
-                st = os.stat(fsenc(fs_path))
+                st = bos.stat(fs_path)
                 file_ts = max(file_ts, st.st_mtime)
                 editions[ext or "plain"] = [fs_path, st.st_size]
             except:
@@ -1289,11 +1310,9 @@ class HttpCli(object):
         else:
             fn = self.headers.get("host", "hey")
 
-        afn = "".join(
-            [x if x in (string.ascii_letters + string.digits) else "_" for x in fn]
-        )
-
-        bascii = unicode(string.ascii_letters + string.digits).encode("utf-8")
+        safe = (string.ascii_letters + string.digits).replace("%", "")
+        afn = "".join([x if x in safe.replace('"', "") else "_" for x in fn])
+        bascii = unicode(safe).encode("utf-8")
         ufn = fn.encode("utf-8", "xmlcharrefreplace")
         if PY2:
             ufn = [unicode(x) if x in bascii else "%{:02x}".format(ord(x)) for x in ufn]
@@ -1308,6 +1327,7 @@ class HttpCli(object):
 
         cdis = "attachment; filename=\"{}.{}\"; filename*=UTF-8''{}.{}"
         cdis = cdis.format(afn, fmt, ufn, fmt)
+        self.log(cdis)
         self.send_headers(None, mime=mime, headers={"Content-Disposition": cdis})
 
         fgen = vn.zipgen(rem, items, self.uname, dots, not self.args.no_scandir)
@@ -1364,10 +1384,10 @@ class HttpCli(object):
         html_path = os.path.join(E.mod, "web", "{}.html".format(tpl))
         template = self.j2(tpl)
 
-        st = os.stat(fsenc(fs_path))
+        st = bos.stat(fs_path)
         ts_md = st.st_mtime
 
-        st = os.stat(fsenc(html_path))
+        st = bos.stat(html_path)
         ts_html = st.st_mtime
 
         sz_md = 0
@@ -1424,12 +1444,13 @@ class HttpCli(object):
 
     def tx_mounts(self):
         suf = self.urlq({}, ["h"])
+        avol = [x for x in self.wvol if x in self.rvol]
         rvol, wvol, avol = [
             [("/" + x).rstrip("/") + "/" for x in y]
-            for y in [self.rvol, self.wvol, self.avol]
+            for y in [self.rvol, self.wvol, avol]
         ]
 
-        if self.avol and not self.args.no_rescan:
+        if avol and not self.args.no_rescan:
             x = self.conn.hsrv.broker.put(True, "up2k.get_state")
             vs = json.loads(x.get())
             vstate = {("/" + k).rstrip("/") + "/": v for k, v in vs["volstate"].items()}
@@ -1454,8 +1475,8 @@ class HttpCli(object):
         return True
 
     def scanvol(self):
-        if not self.readable or not self.writable:
-            raise Pebkac(403, "not admin")
+        if not self.can_read or not self.can_write:
+            raise Pebkac(403, "not allowed for user " + self.uname)
 
         if self.args.no_rescan:
             raise Pebkac(403, "disabled by argv")
@@ -1473,8 +1494,8 @@ class HttpCli(object):
         raise Pebkac(500, x)
 
     def tx_stack(self):
-        if not self.avol:
-            raise Pebkac(403, "not admin")
+        if not [x for x in self.wvol if x in self.rvol]:
+            raise Pebkac(403, "not allowed for user " + self.uname)
 
         if self.args.no_stack:
             raise Pebkac(403, "disabled by argv")
@@ -1512,7 +1533,7 @@ class HttpCli(object):
         try:
             vn, rem = self.asrv.vfs.get(top, self.uname, True, False)
             fsroot, vfs_ls, vfs_virt = vn.ls(
-                rem, self.uname, not self.args.no_scandir, incl_wo=True
+                rem, self.uname, not self.args.no_scandir, [[True], [False, True]]
             )
         except:
             vfs_ls = []
@@ -1539,6 +1560,74 @@ class HttpCli(object):
         ret["a"] = dirs
         return ret
 
+    def tx_ups(self):
+        if not self.args.unpost:
+            raise Pebkac(400, "the unpost feature was disabled by server config")
+
+        filt = self.uparam.get("filter")
+        lm = "ups [{}]".format(filt)
+        self.log(lm)
+
+        ret = []
+        t0 = time.time()
+        idx = self.conn.get_u2idx()
+        lim = time.time() - self.args.unpost
+        for vol in self.asrv.vfs.all_vols.values():
+            cur = idx.get_cur(vol.realpath)
+            if not cur:
+                continue
+
+            q = "select sz, rd, fn, at from up where ip=? and at>?"
+            for sz, rd, fn, at in cur.execute(q, (self.ip, lim)):
+                vp = "/" + "/".join([rd, fn]).strip("/")
+                if filt and filt not in vp:
+                    continue
+
+                ret.append({"vp": vp, "sz": sz, "at": at})
+                if len(ret) > 3000:
+                    ret.sort(key=lambda x: x["at"], reverse=True)
+                    ret = ret[:2000]
+
+        ret.sort(key=lambda x: x["at"], reverse=True)
+        ret = ret[:2000]
+
+        jtxt = json.dumps(ret, indent=2, sort_keys=True).encode("utf-8", "replace")
+        self.log("{} #{} {:.2f}sec".format(lm, len(ret), time.time() - t0))
+        self.reply(jtxt, mime="application/json")
+
+    def handle_rm(self, req=None):
+        if not req and not self.can_delete:
+            raise Pebkac(403, "not allowed for user " + self.uname)
+
+        if self.args.no_del:
+            raise Pebkac(403, "disabled by argv")
+
+        if not req:
+            req = [self.vpath]
+
+        x = self.conn.hsrv.broker.put(True, "up2k.handle_rm", self.uname, self.ip, req)
+        self.loud_reply(x.get())
+
+    def handle_mv(self):
+        if not self.can_move:
+            raise Pebkac(403, "not allowed for user " + self.uname)
+
+        if self.args.no_mv:
+            raise Pebkac(403, "disabled by argv")
+
+        # full path of new loc (incl filename)
+        dst = self.uparam.get("move")
+        if not dst:
+            raise Pebkac(400, "need dst vpath")
+
+        # x-www-form-urlencoded (url query part) uses
+        # either + or %20 for 0x20 so handle both
+        dst = unquotep(dst.replace("+", " "))
+        x = self.conn.hsrv.broker.put(
+            True, "up2k.handle_mv", self.uname, self.vpath, dst
+        )
+        self.loud_reply(x.get())
+
     def tx_browser(self):
         vpath = ""
         vpnodes = [["", "/"]]
@@ -1551,18 +1640,16 @@ class HttpCli(object):
 
                 vpnodes.append([quotep(vpath) + "/", html_escape(node, crlf=True)])
 
-        vn, rem = self.asrv.vfs.get(
-            self.vpath, self.uname, self.readable, self.writable
-        )
+        vn, rem = self.asrv.vfs.get(self.vpath, self.uname, False, False)
         abspath = vn.canonical(rem)
         dbv, vrem = vn.get_dbv(rem)
 
         try:
-            st = os.stat(fsenc(abspath))
+            st = bos.stat(abspath)
         except:
             raise Pebkac(404)
 
-        if self.readable:
+        if self.can_read:
             if rem.startswith(".hist/up2k.") or (
                 rem.endswith("/dir.txt") and rem.startswith(".hist/th/")
             ):
@@ -1574,7 +1661,7 @@ class HttpCli(object):
                 if is_dir:
                     for fn in self.args.th_covers.split(","):
                         fp = os.path.join(abspath, fn)
-                        if os.path.exists(fp):
+                        if bos.path.exists(fp):
                             vrem = "{}/{}".format(vrem.rstrip("/"), fn)
                             is_dir = False
                             break
@@ -1629,10 +1716,14 @@ class HttpCli(object):
         srv_info = "</span> /// <span>".join(srv_info)
 
         perms = []
-        if self.readable:
+        if self.can_read:
             perms.append("read")
-        if self.writable:
+        if self.can_write:
             perms.append("write")
+        if self.can_move:
+            perms.append("move")
+        if self.can_delete:
+            perms.append("delete")
 
         url_suf = self.urlq({}, [])
         is_ls = "ls" in self.uparam
@@ -1644,7 +1735,7 @@ class HttpCli(object):
         logues = ["", ""]
         for n, fn in enumerate([".prologue.html", ".epilogue.html"]):
             fn = os.path.join(abspath, fn)
-            if os.path.exists(fsenc(fn)):
+            if bos.path.exists(fn):
                 with open(fsenc(fn), "rb") as f:
                     logues[n] = f.read().decode("utf-8")
 
@@ -1667,14 +1758,17 @@ class HttpCli(object):
             "tag_order": [],
             "have_up2k_idx": ("e2d" in vn.flags),
             "have_tags_idx": ("e2t" in vn.flags),
+            "have_mv": (not self.args.no_mv),
+            "have_del": (not self.args.no_del),
             "have_zip": (not self.args.no_zip),
-            "have_b_u": (self.writable and self.uparam.get("b") == "u"),
+            "have_unpost": (self.args.unpost > 0),
+            "have_b_u": (self.can_write and self.uparam.get("b") == "u"),
             "url_suf": url_suf,
             "logues": logues,
             "title": html_escape(self.vpath, crlf=True),
             "srv_info": srv_info,
         }
-        if not self.readable:
+        if not self.can_read:
             if is_ls:
                 ret = json.dumps(ls_ret)
                 self.reply(
@@ -1697,7 +1791,7 @@ class HttpCli(object):
                 return self.tx_zip(k, v, vn, rem, [], self.args.ed)
 
         fsroot, vfs_ls, vfs_virt = vn.ls(
-            rem, self.uname, not self.args.no_scandir, incl_wo=True
+            rem, self.uname, not self.args.no_scandir, [[True], [False, True]]
         )
         stats = {k: v for k, v in vfs_ls}
         vfs_ls = [x[0] for x in vfs_ls]
@@ -1708,7 +1802,7 @@ class HttpCli(object):
         histdir = os.path.join(fsroot, ".hist")
         ptn = re.compile(r"(.*)\.([0-9]+\.[0-9]{3})(\.[^\.]+)$")
         try:
-            for hfn in os.listdir(histdir):
+            for hfn in bos.listdir(histdir):
                 m = ptn.match(hfn)
                 if not m:
                     continue
@@ -1749,7 +1843,7 @@ class HttpCli(object):
                 fspath = fsroot + "/" + fn
 
             try:
-                inf = stats.get(fn) or os.stat(fsenc(fspath))
+                inf = stats.get(fn) or bos.stat(fspath)
             except:
                 self.log("broken symlink: {}".format(repr(fspath)))
                 continue

copyparty/httpsrv.py

@@ -28,6 +28,7 @@ except ImportError:
 
 from .__init__ import E, PY2, MACOS
 from .util import spack, min_ex, start_stackmon, start_log_thrs
+from .bos import bos
 from .httpconn import HttpConn
 
 if PY2:
@@ -73,7 +74,7 @@ class HttpSrv(object):
         }
 
         cert_path = os.path.join(E.cfg, "cert.pem")
-        if os.path.exists(cert_path):
+        if bos.path.exists(cert_path):
             self.cert_path = cert_path
         else:
             self.cert_path = None
@@ -140,6 +141,7 @@ class HttpSrv(object):
         fno = srv_sck.fileno()
         msg = "subscribed @ {}:{}  f{}".format(ip, port, fno)
         self.log(self.name, msg)
+        self.broker.put(False, "cb_httpsrv_up")
         while not self.stopping:
             if self.args.log_conn:
                 self.log(self.name, "|%sC-ncli" % ("-" * 1,), c="1;30")
@@ -307,7 +309,7 @@ class HttpSrv(object):
             try:
                 with os.scandir(os.path.join(E.mod, "web")) as dh:
                     for fh in dh:
-                        inf = fh.stat(follow_symlinks=False)
+                        inf = fh.stat()
                         v = max(v, inf.st_mtime)
             except:
                 pass

copyparty/mtag.py

@@ -9,6 +9,7 @@ import subprocess as sp
 
 from .__init__ import PY2, WINDOWS, unicode
 from .util import fsenc, fsdec, uncyg, REKOBO_LKEY
+from .bos import bos
 
 
 def have_ff(cmd):
@@ -44,7 +45,7 @@ class MParser(object):
                 if WINDOWS:
                     bp = uncyg(bp)
 
-                if os.path.exists(bp):
+                if bos.path.exists(bp):
                     self.bin = bp
                     return
             except:
@@ -420,7 +421,7 @@ class MTag(object):
         except Exception as ex:
             return self.get_ffprobe(abspath) if self.can_ffprobe else {}
 
-        sz = os.path.getsize(fsenc(abspath))
+        sz = bos.path.getsize(abspath)
         ret = {".q": [0, int((sz / md.info.length) / 128)]}
 
         for attr, k, norm in [

copyparty/star.py

@@ -1,12 +1,12 @@
 # coding: utf-8
 from __future__ import print_function, unicode_literals
 
-import os
 import tarfile
 import threading
 
 from .sutil import errdesc
 from .util import Queue, fsenc
+from .bos import bos
 
 
 class QFile(object):
@@ -61,7 +61,7 @@ class StreamTar(object):
 
         yield None
         if self.errf:
-            os.unlink(self.errf["ap"])
+            bos.unlink(self.errf["ap"])
 
     def ser(self, f):
         name = f["vp"]

copyparty/sutil.py

@@ -1,11 +1,12 @@
 # coding: utf-8
 from __future__ import print_function, unicode_literals
 
-import os
 import time
 import tempfile
 from datetime import datetime
 
+from .bos import bos
+
 
 def errdesc(errors):
     report = ["copyparty failed to add the following files to the archive:", ""]
@@ -20,9 +21,9 @@ def errdesc(errors):
     dt = datetime.utcfromtimestamp(time.time())
     dt = dt.strftime("%Y-%m%d-%H%M%S")
 
-    os.chmod(tf_path, 0o444)
+    bos.chmod(tf_path, 0o444)
     return {
         "vp": "archive-errors-{}.txt".format(dt),
         "ap": tf_path,
-        "st": os.stat(tf_path),
+        "st": bos.stat(tf_path),
     }, report

copyparty/svchub.py

@@ -13,7 +13,7 @@ import threading
 from datetime import datetime, timedelta
 import calendar
 
-from .__init__ import E, PY2, WINDOWS, MACOS, VT100, unicode
+from .__init__ import E, PY2, WINDOWS, ANYWIN, MACOS, VT100, unicode
 from .util import mp, start_log_thrs, start_stackmon, min_ex
 from .authsrv import AuthSrv
 from .tcpsrv import TcpSrv
@@ -39,6 +39,7 @@ class SvcHub(object):
         self.stop_req = False
         self.stopping = False
         self.stop_cond = threading.Condition()
+        self.httpsrv_up = 0
 
         self.ansi_re = re.compile("\033\\[[^m]*m")
         self.log_mutex = threading.Lock()
@@ -55,7 +56,7 @@ class SvcHub(object):
             start_log_thrs(self.log, args.log_thrs, 0)
 
         # initiate all services to manage
-        self.asrv = AuthSrv(self.args, self.log, False)
+        self.asrv = AuthSrv(self.args, self.log)
         if args.ls:
             self.asrv.dbg_ls()
 
@@ -86,6 +87,29 @@ class SvcHub(object):
 
         self.broker = Broker(self)
 
+    def thr_httpsrv_up(self):
+        time.sleep(5)
+        failed = self.broker.num_workers - self.httpsrv_up
+        if not failed:
+            return
+
+        m = "{}/{} workers failed to start"
+        m = m.format(failed, self.broker.num_workers)
+        self.log("root", m, 1)
+        os._exit(1)
+
+    def cb_httpsrv_up(self):
+        self.httpsrv_up += 1
+        if self.httpsrv_up != self.broker.num_workers:
+            return
+
+        self.log("root", "workers OK\n")
+        self.up2k.init_vols()
+
+        thr = threading.Thread(target=self.sd_notify, name="sd-notify")
+        thr.daemon = True
+        thr.start()
+
     def _logname(self):
         dt = datetime.utcfromtimestamp(time.time())
         fn = self.args.lo
@@ -135,24 +159,33 @@ class SvcHub(object):
     def run(self):
         self.tcpsrv.run()
 
-        thr = threading.Thread(target=self.sd_notify, name="sd-notify")
-        thr.daemon = True
-        thr.start()
-
-        thr = threading.Thread(target=self.stop_thr, name="svchub-sig")
+        thr = threading.Thread(target=self.thr_httpsrv_up)
         thr.daemon = True
         thr.start()
 
         for sig in [signal.SIGINT, signal.SIGTERM]:
             signal.signal(sig, self.signal_handler)
 
-        try:
-            while not self.stop_req:
-                time.sleep(9001)
-        except:
-            pass
+        # macos hangs after shutdown on sigterm with while-sleep,
+        # windows cannot ^c stop_cond (and win10 does the macos thing but winxp is fine??)
+        # linux is fine with both,
+        # never lucky
+        if ANYWIN:
+            # msys-python probably fine but >msys-python
+            thr = threading.Thread(target=self.stop_thr, name="svchub-sig")
+            thr.daemon = True
+            thr.start()
 
-        self.shutdown()
+            try:
+                while not self.stop_req:
+                    time.sleep(1)
+            except:
+                pass
+
+            self.shutdown()
+            thr.join()
+        else:
+            self.stop_thr()
 
     def stop_thr(self):
         while not self.stop_req:
@@ -161,7 +194,7 @@ class SvcHub(object):
 
         self.shutdown()
 
-    def signal_handler(self):
+    def signal_handler(self, sig, frame):
         if self.stopping:
             return
 
@@ -175,6 +208,10 @@ class SvcHub(object):
 
         self.stopping = True
         self.stop_req = True
+        with self.stop_cond:
+            self.stop_cond.notify_all()
+
+        ret = 1
         try:
             with self.log_mutex:
                 print("OPYTHAT")
@@ -194,11 +231,14 @@ class SvcHub(object):
                         print("waiting for thumbsrv (10sec)...")
 
             print("nailed it", end="")
+            ret = 0
         finally:
             print("\033[0m")
             if self.logf:
                 self.logf.close()
 
+            sys.exit(ret)
+
     def _log_disabled(self, src, msg, c=0):
         if not self.logf:
             return

copyparty/szip.py

@@ -8,6 +8,7 @@ from datetime import datetime
 
 from .sutil import errdesc
 from .util import yieldfile, sanitize_fn, spack, sunpack
+from .bos import bos
 
 
 def dostime2unix(buf):
@@ -271,4 +272,4 @@ class StreamZip(object):
         yield self._ct(ecdr)
 
         if errors:
-            os.unlink(errf["ap"])
+            bos.unlink(errf["ap"])

copyparty/tcpsrv.py

@@ -4,6 +4,7 @@ from __future__ import print_function, unicode_literals
 import re
 import socket
 
+from .__init__ import MACOS, ANYWIN
 from .util import chkcmd
 
 
@@ -29,14 +30,16 @@ class TcpSrv(object):
                 for x in nonlocals:
                     eps[x] = "external"
 
+        msgs = []
+        m = "available @ http://{}:{}/  (\033[33m{}\033[0m)"
         for ip, desc in sorted(eps.items(), key=lambda x: x[1]):
             for port in sorted(self.args.p):
-                self.log(
-                    "tcpsrv",
-                    "available @ http://{}:{}/  (\033[33m{}\033[0m)".format(
-                        ip, port, desc
-                    ),
-                )
+                msgs.append(m.format(ip, port, desc))
+
+        if msgs:
+            msgs[-1] += "\n"
+            for m in msgs:
+                self.log("tcpsrv", m)
 
         self.srv = []
         for ip in self.args.i:
@@ -81,25 +84,100 @@ class TcpSrv(object):
 
         self.log("tcpsrv", "ok bye")
 
-    def detect_interfaces(self, listen_ips):
+    def ips_linux(self):
         eps = {}
-
-        # get all ips and their interfaces
         try:
-            ip_addr, _ = chkcmd("ip", "addr")
+            txt, _ = chkcmd(["ip", "addr"])
         except:
-            ip_addr = None
+            return eps
 
-        if ip_addr:
-            r = re.compile(r"^\s+inet ([^ ]+)/.* (.*)")
-            for ln in ip_addr.split("\n"):
-                try:
-                    ip, dev = r.match(ln.rstrip()).groups()
-                    for lip in listen_ips:
-                        if lip in ["0.0.0.0", ip]:
-                            eps[ip] = dev
-                except:
-                    pass
+        r = re.compile(r"^\s+inet ([^ ]+)/.* (.*)")
+        for ln in txt.split("\n"):
+            try:
+                ip, dev = r.match(ln.rstrip()).groups()
+                eps[ip] = dev
+            except:
+                pass
+
+        return eps
+
+    def ips_macos(self):
+        eps = {}
+        try:
+            txt, _ = chkcmd(["ifconfig"])
+        except:
+            return eps
+
+        rdev = re.compile(r"^([^ ]+):")
+        rip = re.compile(r"^\tinet ([0-9\.]+) ")
+        dev = None
+        for ln in txt.split("\n"):
+            m = rdev.match(ln)
+            if m:
+                dev = m.group(1)
+
+            m = rip.match(ln)
+            if m:
+                eps[m.group(1)] = dev
+                dev = None
+
+        return eps
+
+    def ips_windows_ipconfig(self):
+        eps = {}
+        try:
+            txt, _ = chkcmd(["ipconfig"])
+        except:
+            return eps
+
+        rdev = re.compile(r"(^[^ ].*):$")
+        rip = re.compile(r"^ +IPv?4? [^:]+: *([0-9\.]{7,15})$")
+        dev = None
+        for ln in txt.replace("\r", "").split("\n"):
+            m = rdev.match(ln)
+            if m:
+                dev = m.group(1).split(" adapter ", 1)[-1]
+
+            m = rip.match(ln)
+            if m and dev:
+                eps[m.group(1)] = dev
+                dev = None
+
+        return eps
+
+    def ips_windows_netsh(self):
+        eps = {}
+        try:
+            txt, _ = chkcmd("netsh interface ip show address".split())
+        except:
+            return eps
+
+        rdev = re.compile(r'.* "([^"]+)"$')
+        rip = re.compile(r".* IP\b.*: +([0-9\.]{7,15})$")
+        dev = None
+        for ln in txt.replace("\r", "").split("\n"):
+            m = rdev.match(ln)
+            if m:
+                dev = m.group(1)
+
+            m = rip.match(ln)
+            if m and dev:
+                eps[m.group(1)] = dev
+                dev = None
+
+        return eps
+
+    def detect_interfaces(self, listen_ips):
+        if MACOS:
+            eps = self.ips_macos()
+        elif ANYWIN:
+            eps = self.ips_windows_ipconfig()  # sees more interfaces
+            eps.update(self.ips_windows_netsh())  # has better names
+        else:
+            eps = self.ips_linux()
+
+        if "0.0.0.0" not in listen_ips:
+            eps = {k: v for k, v in eps if k in listen_ips}
 
         default_route = None
         s = socket.socket(socket.AF_INET, socket.SOCK_DGRAM)

copyparty/th_cli.py

@@ -5,6 +5,7 @@ import os
 
 from .util import Cooldown
 from .th_srv import thumb_path, THUMBABLE, FMT_FF
+from .bos import bos
 
 
 class ThumbCli(object):
@@ -36,7 +37,7 @@ class ThumbCli(object):
         tpath = thumb_path(histpath, rem, mtime, fmt)
         ret = None
         try:
-            st = os.stat(tpath)
+            st = bos.stat(tpath)
             if st.st_size:
                 ret = tpath
             else:

copyparty/th_srv.py

@@ -10,7 +10,8 @@ import threading
 import subprocess as sp
 
 from .__init__ import PY2, unicode
-from .util import fsenc, runcmd, Queue, Cooldown, BytesIO, min_ex
+from .util import fsenc, vsplit, runcmd, Queue, Cooldown, BytesIO, min_ex
+from .bos import bos
 from .mtag import HAVE_FFMPEG, HAVE_FFPROBE, ffprobe
 
 
@@ -73,12 +74,7 @@ def thumb_path(histpath, rem, mtime, fmt):
     # base16 = 16 = 256
     # b64-lc = 38 = 1444
     # base64 = 64 = 4096
-    try:
-        rd, fn = rem.rsplit("/", 1)
-    except:
-        rd = ""
-        fn = rem
-
+    rd, fn = vsplit(rem)
     if rd:
         h = hashlib.sha512(fsenc(rd)).digest()
         b64 = base64.urlsafe_b64encode(h).decode("ascii")[:24]
@@ -159,13 +155,10 @@ class ThumbSrv(object):
                 self.log("wait {}".format(tpath))
             except:
                 thdir = os.path.dirname(tpath)
-                try:
-                    os.makedirs(thdir)
-                except:
-                    pass
+                bos.makedirs(thdir)
 
                 inf_path = os.path.join(thdir, "dir.txt")
-                if not os.path.exists(inf_path):
+                if not bos.path.exists(inf_path):
                     with open(inf_path, "wb") as f:
                         f.write(fsenc(os.path.dirname(abspath)))
 
@@ -185,7 +178,7 @@ class ThumbSrv(object):
                 cond.wait(3)
 
         try:
-            st = os.stat(tpath)
+            st = bos.stat(tpath)
             if st.st_size:
                 return tpath
         except:
@@ -202,7 +195,7 @@ class ThumbSrv(object):
             abspath, tpath = task
             ext = abspath.split(".")[-1].lower()
             fun = None
-            if not os.path.exists(tpath):
+            if not bos.path.exists(tpath):
                 if ext in FMT_PIL:
                     fun = self.conv_pil
                 elif ext in FMT_FF:
@@ -313,7 +306,7 @@ class ThumbSrv(object):
 
         cmd += [fsenc(tpath)]
 
-        ret, sout, serr = runcmd(*cmd)
+        ret, sout, serr = runcmd(cmd)
         if ret != 0:
             msg = ["ff: {}".format(x) for x in serr.split("\n")]
             self.log("FFmpeg failed:\n" + "\n".join(msg), c="1;30")
@@ -328,7 +321,7 @@ class ThumbSrv(object):
             p1 = os.path.dirname(tdir)
             p2 = os.path.dirname(p1)
             for dp in [tdir, p1, p2]:
-                os.utime(fsenc(dp), (ts, ts))
+                bos.utime(dp, (ts, ts))
         except:
             pass
 
@@ -355,7 +348,7 @@ class ThumbSrv(object):
         prev_b64 = None
         prev_fp = None
         try:
-            ents = os.listdir(thumbpath)
+            ents = bos.listdir(thumbpath)
         except:
             return 0
 
@@ -366,7 +359,7 @@ class ThumbSrv(object):
 
             # "top" or b64 prefix/full (a folder)
             if len(f) <= 3 or len(f) == 24:
-                age = now - os.path.getmtime(fp)
+                age = now - bos.path.getmtime(fp)
                 if age > maxage:
                     with self.mutex:
                         safe = True
@@ -398,7 +391,7 @@ class ThumbSrv(object):
 
             if b64 == prev_b64:
                 self.log("rm replaced [{}]".format(fp))
-                os.unlink(prev_fp)
+                bos.unlink(prev_fp)
 
             prev_b64 = b64
             prev_fp = fp

copyparty/u2idx.py

@@ -9,6 +9,7 @@ from datetime import datetime
 
 from .__init__ import unicode
 from .util import s3dec, Pebkac, min_ex
+from .bos import bos
 from .up2k import up2k_wark_from_hashlist
 
 
@@ -67,7 +68,7 @@ class U2idx(object):
 
         histpath = self.asrv.vfs.histtab[ptop]
         db_path = os.path.join(histpath, "up2k.db")
-        if not os.path.exists(db_path):
+        if not bos.path.exists(db_path):
             return None
 
         cur = sqlite3.connect(db_path, 2).cursor()
@@ -243,7 +244,7 @@ class U2idx(object):
             sret = []
             c = cur.execute(q, v)
             for hit in c:
-                w, ts, sz, rd, fn = hit
+                w, ts, sz, rd, fn, ip, at = hit
                 lim -= 1
                 if lim <= 0:
                     break

copyparty/up2k.py

@@ -23,15 +23,20 @@ from .util import (
     ProgressPrinter,
     fsdec,
     fsenc,
+    absreal,
     sanitize_fn,
     ren_open,
     atomic_move,
+    vsplit,
     s3enc,
     s3dec,
+    rmdirs,
     statdir,
     s2hms,
     min_ex,
 )
+from .bos import bos
+from .authsrv import AuthSrv
 from .mtag import MTag, MParser
 
 try:
@@ -40,20 +45,13 @@ try:
 except:
     HAVE_SQLITE3 = False
 
-DB_VER = 4
+DB_VER = 5
 
 
 class Up2k(object):
-    """
-    TODO:
-      * documentation
-      * registry persistence
-        * ~/.config flatfiles for active jobs
-    """
-
     def __init__(self, hub):
         self.hub = hub
-        self.asrv = hub.asrv
+        self.asrv = hub.asrv  # type: AuthSrv
         self.args = hub.args
         self.log_func = hub.log
 
@@ -67,6 +65,7 @@ class Up2k(object):
         self.n_hashq = 0
         self.n_tagq = 0
         self.volstate = {}
+        self.need_rescan = {}
         self.registry = {}
         self.entags = {}
         self.flags = {}
@@ -101,17 +100,16 @@ class Up2k(object):
 
         if self.args.no_fastboot:
             self.deferred_init()
-        else:
-            t = threading.Thread(
-                target=self.deferred_init, name="up2k-deferred-init", args=(0.5,)
-            )
-            t.daemon = True
-            t.start()
 
-    def deferred_init(self, wait=0):
-        if wait:
-            time.sleep(wait)
+    def init_vols(self):
+        if self.args.no_fastboot:
+            return
 
+        t = threading.Thread(target=self.deferred_init, name="up2k-deferred-init")
+        t.daemon = True
+        t.start()
+
+    def deferred_init(self):
         all_vols = self.asrv.vfs.all_vols
         have_e2d = self.init_indexes(all_vols)
 
@@ -124,6 +122,10 @@ class Up2k(object):
             thr.daemon = True
             thr.start()
 
+            thr = threading.Thread(target=self._sched_rescan, name="up2k-rescan")
+            thr.daemon = True
+            thr.start()
+
             if self.mtag:
                 thr = threading.Thread(target=self._tagger, name="up2k-tagger")
                 thr.daemon = True
@@ -173,6 +175,38 @@ class Up2k(object):
         t.start()
         return None
 
+    def _sched_rescan(self):
+        maxage = self.args.re_maxage
+        volage = {}
+        while True:
+            time.sleep(self.args.re_int)
+            now = time.time()
+            vpaths = list(sorted(self.asrv.vfs.all_vols.keys()))
+            with self.mutex:
+                if maxage:
+                    for vp in vpaths:
+                        if vp not in volage:
+                            volage[vp] = now
+
+                        if now - volage[vp] >= maxage:
+                            self.need_rescan[vp] = 1
+
+                if not self.need_rescan:
+                    continue
+
+                vols = list(sorted(self.need_rescan.keys()))
+                self.need_rescan = {}
+
+            err = self.rescan(self.asrv.vfs.all_vols, vols)
+            if err:
+                for v in vols:
+                    self.need_rescan[v] = True
+
+                continue
+
+            for v in vols:
+                volage[v] = now
+
     def _vis_job_progress(self, job):
         perc = 100 - (len(job["need"]) * 100.0 / len(job["hash"]))
         path = os.path.join(job["ptop"], job["prel"], job["name"])
@@ -218,7 +252,7 @@ class Up2k(object):
             # only need to protect register_vpath but all in one go feels right
             for vol in vols:
                 try:
-                    os.listdir(vol.realpath)
+                    bos.listdir(vol.realpath)
                 except:
                     self.volstate[vol.vpath] = "OFFLINE (cannot access folder)"
                     self.log("cannot access " + vol.realpath, c=1)
@@ -356,14 +390,14 @@ class Up2k(object):
 
         reg = {}
         path = os.path.join(histpath, "up2k.snap")
-        if "e2d" in flags and os.path.exists(path):
+        if "e2d" in flags and bos.path.exists(path):
             with gzip.GzipFile(path, "rb") as f:
                 j = f.read().decode("utf-8")
 
             reg2 = json.loads(j)
             for k, job in reg2.items():
                 path = os.path.join(job["ptop"], job["prel"], job["name"])
-                if os.path.exists(fsenc(path)):
+                if bos.path.exists(path):
                     reg[k] = job
                     job["poke"] = time.time()
                 else:
@@ -378,10 +412,7 @@ class Up2k(object):
         if not HAVE_SQLITE3 or "e2d" not in flags or "d2d" in flags:
             return None
 
-        try:
-            os.makedirs(histpath)
-        except:
-            pass
+        bos.makedirs(histpath)
 
         try:
             cur = self._open_db(db_path)
@@ -420,14 +451,7 @@ class Up2k(object):
             return True, n_add or n_rm or do_vac
 
     def _build_dir(self, dbw, top, excl, cdir, nohash, seen):
-        rcdir = cdir
-        if not ANYWIN:
-            try:
-                # a bit expensive but worth
-                rcdir = os.path.realpath(cdir)
-            except:
-                pass
-
+        rcdir = absreal(cdir)  # a bit expensive but worth
         if rcdir in seen:
             m = "bailing from symlink loop,\n  prev: {}\n  curr: {}\n  from: {}"
             self.log(m.format(seen[-1], rcdir, cdir), 3)
@@ -498,7 +522,7 @@ class Up2k(object):
 
                     wark = up2k_wark_from_hashlist(self.salt, sz, hashes)
 
-                self.db_add(dbw[0], wark, rd, fn, lmod, sz)
+                self.db_add(dbw[0], wark, rd, fn, lmod, sz, "", 0)
                 dbw[1] += 1
                 ret += 1
                 td = time.time() - dbw[2]
@@ -513,8 +537,8 @@ class Up2k(object):
         rm = []
         nchecked = 0
         nfiles = next(cur.execute("select count(w) from up"))[0]
-        c = cur.execute("select * from up")
-        for dwark, dts, dsz, drd, dfn in c:
+        c = cur.execute("select rd, fn from up")
+        for drd, dfn in c:
             nchecked += 1
             if drd.startswith("//") or dfn.startswith("//"):
                 drd, dfn = s3dec(drd, dfn)
@@ -523,7 +547,7 @@ class Up2k(object):
             # almost zero overhead dw
             self.pp.msg = "b{} {}".format(nfiles - nchecked, abspath)
             try:
-                if not os.path.exists(fsenc(abspath)):
+                if not bos.path.exists(abspath):
                     rm.append([drd, dfn])
             except Exception as ex:
                 self.log("stat-rm: {} @ [{}]".format(repr(ex), abspath))
@@ -911,12 +935,21 @@ class Up2k(object):
         # x.set_trace_callback(trace)
 
     def _open_db(self, db_path):
-        existed = os.path.exists(db_path)
+        existed = bos.path.exists(db_path)
         cur = self._orz(db_path)
         ver = self._read_ver(cur)
         if not existed and ver is None:
             return self._create_db(db_path, cur)
 
+        if ver == 4:
+            try:
+                m = "creating backup before upgrade: "
+                cur = self._backup_db(db_path, cur, ver, m)
+                self._upgrade_v4(cur)
+                ver = 5
+            except:
+                self.log("WARN: failed to upgrade from v4", 3)
+
         if ver == DB_VER:
             try:
                 nfiles = next(cur.execute("select count(w) from up"))[0]
@@ -929,19 +962,38 @@ class Up2k(object):
             m = "database is version {}, this copyparty only supports versions <= {}"
             raise Exception(m.format(ver, DB_VER))
 
-        bak = "{}.bak.{:x}.v{}".format(db_path, int(time.time()), ver)
-        db = cur.connection
-        cur.close()
-        db.close()
         msg = "creating new DB (old is bad); backup: {}"
         if ver:
             msg = "creating new DB (too old to upgrade); backup: {}"
 
-        self.log(msg.format(bak))
-        os.rename(fsenc(db_path), fsenc(bak))
-
+        cur = self._backup_db(db_path, cur, ver, msg)
+        db = cur.connection
+        cur.close()
+        db.close()
+        bos.unlink(db_path)
         return self._create_db(db_path, None)
 
+    def _backup_db(self, db_path, cur, ver, msg):
+        bak = "{}.bak.{:x}.v{}".format(db_path, int(time.time()), ver)
+        self.log(msg + bak)
+        try:
+            c2 = sqlite3.connect(bak)
+            with c2:
+                cur.connection.backup(c2)
+            return cur
+        except:
+            m = "native sqlite3 backup failed; using fallback method:\n"
+            self.log(m + min_ex())
+        finally:
+            c2.close()
+
+        db = cur.connection
+        cur.close()
+        db.close()
+
+        shutil.copy2(fsenc(db_path), fsenc(bak))
+        return self._orz(db_path)
+
     def _read_ver(self, cur):
         for tab in ["ki", "kv"]:
             try:
@@ -968,9 +1020,10 @@ class Up2k(object):
             idx = r"create index up_w on up(w)"
 
         for cmd in [
-            r"create table up (w text, mt int, sz int, rd text, fn text)",
+            r"create table up (w text, mt int, sz int, rd text, fn text, ip text, at int)",
             r"create index up_rd on up(rd)",
             r"create index up_fn on up(fn)",
+            r"create index up_ip on up(ip)",
             idx,
             r"create table mt (w text, k text, v int)",
             r"create index mt_w on mt(w)",
@@ -985,6 +1038,17 @@ class Up2k(object):
         self.log("created DB at {}".format(db_path))
         return cur
 
+    def _upgrade_v4(self, cur):
+        for cmd in [
+            r"alter table up add column ip text",
+            r"alter table up add column at int",
+            r"create index up_ip on up(ip)",
+            r"update kv set v=5 where k='sver'",
+        ]:
+            cur.execute(cmd)
+
+        cur.connection.commit()
+
     def handle_json(self, cj):
         with self.mutex:
             if not self.register_vpath(cj["ptop"], cj["vcfg"]):
@@ -1008,13 +1072,13 @@ class Up2k(object):
                     argv = (wark[:16], wark)
 
                 cur = cur.execute(q, argv)
-                for _, dtime, dsize, dp_dir, dp_fn in cur:
+                for _, dtime, dsize, dp_dir, dp_fn, ip, at in cur:
                     if dp_dir.startswith("//") or dp_fn.startswith("//"):
                         dp_dir, dp_fn = s3dec(dp_dir, dp_fn)
 
                     dp_abs = "/".join([cj["ptop"], dp_dir, dp_fn])
                     # relying on path.exists to return false on broken symlinks
-                    if os.path.exists(fsenc(dp_abs)):
+                    if bos.path.exists(dp_abs):
                         job = {
                             "name": dp_fn,
                             "prel": dp_dir,
@@ -1022,6 +1086,8 @@ class Up2k(object):
                             "ptop": cj["ptop"],
                             "size": dsize,
                             "lmod": dtime,
+                            "addr": ip,
+                            "at": at,
                             "hash": [],
                             "need": [],
                         }
@@ -1038,7 +1104,7 @@ class Up2k(object):
                     for fn in names:
                         path = os.path.join(job["ptop"], job["prel"], fn)
                         try:
-                            if os.path.getsize(fsenc(path)) > 0:
+                            if bos.path.getsize(path) > 0:
                                 # upload completed or both present
                                 break
                         except:
@@ -1072,9 +1138,15 @@ class Up2k(object):
                         job["name"] = self._untaken(pdir, cj["name"], now, cj["addr"])
                         dst = os.path.join(job["ptop"], job["prel"], job["name"])
                         if not self.args.nw:
-                            os.unlink(fsenc(dst))  # TODO ed pls
+                            bos.unlink(dst)  # TODO ed pls
                             self._symlink(src, dst)
 
+                        if cur:
+                            a = [cj[x] for x in "prel name lmod size addr".split()]
+                            a += [cj.get("at") or time.time()]
+                            self.db_add(cur, wark, *a)
+                            cur.connection.commit()
+
             if not job:
                 job = {
                     "wark": wark,
@@ -1124,17 +1196,18 @@ class Up2k(object):
         with ren_open(fname, "wb", fdir=fdir, suffix=suffix) as f:
             return f["orz"][1]
 
-    def _symlink(self, src, dst):
-        # TODO store this in linktab so we never delete src if there are links to it
-        self.log("linking dupe:\n  {0}\n  {1}".format(src, dst))
+    def _symlink(self, src, dst, verbose=True):
+        if verbose:
+            self.log("linking dupe:\n  {0}\n  {1}".format(src, dst))
+
         if self.args.nw:
             return
 
         try:
             lsrc = src
             ldst = dst
-            fs1 = os.stat(fsenc(os.path.split(src)[0])).st_dev
-            fs2 = os.stat(fsenc(os.path.split(dst)[0])).st_dev
+            fs1 = bos.stat(os.path.dirname(src)).st_dev
+            fs2 = bos.stat(os.path.dirname(dst)).st_dev
             if fs1 == 0:
                 # py2 on winxp or other unsupported combination
                 raise OSError()
@@ -1217,27 +1290,21 @@ class Up2k(object):
                 a = [dst, job["size"], (int(time.time()), int(job["lmod"]))]
                 self.lastmod_q.put(a)
 
-            # legit api sware 2 me mum
-            if self.idx_wark(
-                job["ptop"],
-                job["wark"],
-                job["prel"],
-                job["name"],
-                job["lmod"],
-                job["size"],
-            ):
+            a = [job[x] for x in "ptop wark prel name lmod size addr".split()]
+            a += [job.get("at") or time.time()]
+            if self.idx_wark(*a):
                 del self.registry[ptop][wark]
                 # in-memory registry is reserved for unfinished uploads
 
         return ret, dst
 
-    def idx_wark(self, ptop, wark, rd, fn, lmod, sz):
+    def idx_wark(self, ptop, wark, rd, fn, lmod, sz, ip, at):
         cur = self.cur.get(ptop)
         if not cur:
             return False
 
         self.db_rm(cur, rd, fn)
-        self.db_add(cur, wark, rd, fn, int(lmod), sz)
+        self.db_add(cur, wark, rd, fn, lmod, sz, ip, at)
         cur.connection.commit()
 
         if "e2t" in self.flags[ptop]:
@@ -1253,16 +1320,304 @@ class Up2k(object):
         except:
             db.execute(sql, s3enc(self.mem_cur, rd, fn))
 
-    def db_add(self, db, wark, rd, fn, ts, sz):
-        sql = "insert into up values (?,?,?,?,?)"
-        v = (wark, int(ts), sz, rd, fn)
+    def db_add(self, db, wark, rd, fn, ts, sz, ip, at):
+        sql = "insert into up values (?,?,?,?,?,?,?)"
+        v = (wark, int(ts), sz, rd, fn, ip or "", int(at or 0))
         try:
             db.execute(sql, v)
         except:
             rd, fn = s3enc(self.mem_cur, rd, fn)
-            v = (wark, ts, sz, rd, fn)
+            v = (wark, int(ts), sz, rd, fn, ip or "", int(at or 0))
             db.execute(sql, v)
 
+    def handle_rm(self, uname, ip, vpaths):
+        n_files = 0
+        ok = {}
+        ng = {}
+        for vp in vpaths:
+            a, b, c = self._handle_rm(uname, ip, vp)
+            n_files += a
+            for k in b:
+                ok[k] = 1
+            for k in c:
+                ng[k] = 1
+
+        ng = {k: 1 for k in ng if k not in ok}
+        ok = len(ok)
+        ng = len(ng)
+
+        return "deleted {} files (and {}/{} folders)".format(n_files, ok, ok + ng)
+
+    def _handle_rm(self, uname, ip, vpath):
+        try:
+            permsets = [[True, False, False, True]]
+            vn, rem = self.asrv.vfs.get(vpath, uname, *permsets[0])
+            unpost = False
+        except:
+            # unpost with missing permissions? try read+write and verify with db
+            if not self.args.unpost:
+                raise Pebkac(400, "the unpost feature was disabled by server config")
+
+            unpost = True
+            permsets = [[True, True]]
+            vn, rem = self.asrv.vfs.get(vpath, uname, *permsets[0])
+            _, _, _, _, dip, dat = self._find_from_vpath(vn.realpath, rem)
+
+            m = "you cannot delete this: "
+            if not dip:
+                m += "file not found"
+            elif dip != ip:
+                m += "not uploaded by (You)"
+            elif dat < time.time() - self.args.unpost:
+                m += "uploaded too long ago"
+            else:
+                m = None
+
+            if m:
+                raise Pebkac(400, m)
+
+        ptop = vn.realpath
+        atop = vn.canonical(rem, False)
+        adir, fn = os.path.split(atop)
+        st = bos.lstat(atop)
+        scandir = not self.args.no_scandir
+        if stat.S_ISLNK(st.st_mode) or stat.S_ISREG(st.st_mode):
+            dbv, vrem = self.asrv.vfs.get(vpath, uname, *permsets[0])
+            dbv, vrem = dbv.get_dbv(vrem)
+            voldir = vsplit(vrem)[0]
+            vpath_dir = vsplit(vpath)[0]
+            g = [[dbv, voldir, vpath_dir, adir, [[fn, 0]], [], []]]
+        else:
+            g = vn.walk("", rem, [], uname, permsets, True, scandir, True)
+            if unpost:
+                raise Pebkac(400, "cannot unpost folders")
+
+        n_files = 0
+        for dbv, vrem, _, adir, files, rd, vd in g:
+            for fn in [x[0] for x in files]:
+                n_files += 1
+                abspath = os.path.join(adir, fn)
+                volpath = "{}/{}".format(vrem, fn).strip("/")
+                vpath = "{}/{}".format(dbv.vpath, volpath).strip("/")
+                self.log("rm {}\n  {}".format(vpath, abspath))
+                _ = dbv.get(volpath, uname, *permsets[0])
+                with self.mutex:
+                    try:
+                        ptop = dbv.realpath
+                        cur, wark, _, _, _, _ = self._find_from_vpath(ptop, volpath)
+                        self._forget_file(ptop, volpath, cur, wark)
+                    finally:
+                        cur.connection.commit()
+
+                bos.unlink(abspath)
+
+        rm = rmdirs(self.log_func, scandir, True, atop)
+        return n_files, rm[0], rm[1]
+
+    def handle_mv(self, uname, svp, dvp):
+        svn, srem = self.asrv.vfs.get(svp, uname, True, False, True)
+        svn, srem = svn.get_dbv(srem)
+        sabs = svn.canonical(srem, False)
+
+        if not srem:
+            raise Pebkac(400, "mv: cannot move a mountpoint")
+
+        st = bos.stat(sabs)
+        if stat.S_ISREG(st.st_mode):
+            return self._mv_file(uname, svp, dvp)
+
+        jail = svn.get_dbv(srem)[0]
+        permsets = [[True, False, True]]
+        scandir = not self.args.no_scandir
+
+        # following symlinks is too scary
+        g = svn.walk("", srem, [], uname, permsets, True, scandir, True)
+        for dbv, vrem, _, atop, files, rd, vd in g:
+            if dbv != jail:
+                # fail early (prevent partial moves)
+                raise Pebkac(400, "mv: source folder contains other volumes")
+
+        g = svn.walk("", srem, [], uname, permsets, True, scandir, True)
+        for dbv, vrem, _, atop, files, rd, vd in g:
+            if dbv != jail:
+                # the actual check (avoid toctou)
+                raise Pebkac(400, "mv: source folder contains other volumes")
+
+            for fn in files:
+                svpf = "/".join(x for x in [dbv.vpath, vrem, fn[0]] if x)
+                if not svpf.startswith(svp + "/"):  # assert
+                    raise Pebkac(500, "mv: bug at {}, top {}".format(svpf, svp))
+
+                dvpf = dvp + svpf[len(svp) :]
+                self._mv_file(uname, svpf, dvpf)
+
+        rmdirs(self.log_func, scandir, True, sabs)
+        return "k"
+
+    def _mv_file(self, uname, svp, dvp):
+        svn, srem = self.asrv.vfs.get(svp, uname, True, False, True)
+        svn, srem = svn.get_dbv(srem)
+
+        dvn, drem = self.asrv.vfs.get(dvp, uname, False, True)
+        dvn, drem = dvn.get_dbv(drem)
+
+        sabs = svn.canonical(srem, False)
+        dabs = dvn.canonical(drem)
+        drd, dfn = vsplit(drem)
+
+        if bos.path.exists(dabs):
+            raise Pebkac(400, "mv2: target file exists")
+
+        bos.makedirs(os.path.dirname(dabs))
+
+        if bos.path.islink(sabs):
+            dlabs = absreal(sabs)
+            m = "moving symlink from [{}] to [{}], target [{}]"
+            self.log(m.format(sabs, dabs, dlabs))
+            os.unlink(sabs)
+            self._symlink(dlabs, dabs, False)
+
+            # folders are too scary, schedule rescan of both vols
+            self.need_rescan[svn.vpath] = 1
+            self.need_rescan[dvn.vpath] = 1
+            return "k"
+
+        c1, w, ftime, fsize, ip, at = self._find_from_vpath(svn.realpath, srem)
+        c2 = self.cur.get(dvn.realpath)
+
+        if ftime is None:
+            st = bos.stat(sabs)
+            ftime = st.st_mtime
+            fsize = st.st_size
+
+        if w:
+            if c2:
+                self._copy_tags(c1, c2, w)
+
+            self._forget_file(svn.realpath, srem, c1, w)
+            self._relink(w, svn.realpath, srem, dabs)
+            c1.connection.commit()
+
+            if c2:
+                self.db_add(c2, w, drd, dfn, ftime, fsize, ip, at)
+                c2.connection.commit()
+        else:
+            self.log("not found in src db: [{}]".format(svp))
+
+        bos.rename(sabs, dabs)
+        return "k"
+
+    def _copy_tags(self, csrc, cdst, wark):
+        """copy all tags for wark from src-db to dst-db"""
+        w = wark[:16]
+
+        if cdst.execute("select * from mt where w=? limit 1", (w,)).fetchone():
+            return  # existing tags in dest db
+
+        for _, k, v in csrc.execute("select * from mt where w=?", (w,)):
+            cdst.execute("insert into mt values(?,?,?)", (w, k, v))
+
+    def _find_from_vpath(self, ptop, vrem):
+        cur = self.cur.get(ptop)
+        if not cur:
+            return None, None
+
+        rd, fn = vsplit(vrem)
+        q = "select w, mt, sz, ip, at from up where rd=? and fn=? limit 1"
+        try:
+            c = cur.execute(q, (rd, fn))
+        except:
+            c = cur.execute(q, s3enc(self.mem_cur, rd, fn))
+
+        hit = c.fetchone()
+        if hit:
+            wark, ftime, fsize, ip, at = hit
+            return cur, wark, ftime, fsize, ip, at
+        return cur, None, None, None, None, None
+
+    def _forget_file(self, ptop, vrem, cur, wark):
+        """forgets file in db, fixes symlinks, does not delete"""
+        srd, sfn = vsplit(vrem)
+        self.log("forgetting {}".format(vrem))
+        if wark:
+            self.log("found {} in db".format(wark))
+            self._relink(wark, ptop, vrem, None)
+
+            q = "delete from mt where w=?"
+            cur.execute(q, (wark[:16],))
+            self.db_rm(cur, srd, sfn)
+
+        reg = self.registry.get(ptop)
+        if reg:
+            if not wark:
+                wark = [
+                    x
+                    for x, y in reg.items()
+                    if fn in [y["name"], y.get("tnam")] and y["prel"] == vrem
+                ]
+
+            if wark and wark in reg:
+                m = "forgetting partial upload {} ({})"
+                p = self._vis_job_progress(wark)
+                self.log(m.format(wark, p))
+                del reg[wark]
+
+    def _relink(self, wark, sptop, srem, dabs):
+        """
+        update symlinks from file at svn/srem to dabs (rename),
+        or to first remaining full if no dabs (delete)
+        """
+        dupes = []
+        sabs = os.path.join(sptop, srem)
+        q = "select rd, fn from up where substr(w,1,16)=? and w=?"
+        for ptop, cur in self.cur.items():
+            for rd, fn in cur.execute(q, (wark[:16], wark)):
+                if rd.startswith("//") or fn.startswith("//"):
+                    rd, fn = s3dec(rd, fn)
+
+                dvrem = "/".join([rd, fn]).strip("/")
+                if ptop != sptop or srem != dvrem:
+                    dupes.append([ptop, dvrem])
+                    self.log("found {} dupe: [{}] {}".format(wark, ptop, dvrem))
+
+        if not dupes:
+            return
+
+        full = {}
+        links = {}
+        for ptop, vp in dupes:
+            ap = os.path.join(ptop, vp)
+            try:
+                d = links if bos.path.islink(ap) else full
+                d[ap] = [ptop, vp]
+            except:
+                self.log("relink: not found: [{}]".format(ap))
+
+        if not dabs and not full and links:
+            # deleting final remaining full copy; swap it with a symlink
+            slabs = list(sorted(links.keys()))[0]
+            ptop, rem = links.pop(slabs)
+            self.log("linkswap [{}] and [{}]".format(sabs, dabs))
+            bos.unlink(slabs)
+            bos.rename(sabs, slabs)
+            self._symlink(slabs, sabs, False)
+            full[slabs] = [ptop, rem]
+
+        if not dabs:
+            dabs = list(sorted(full.keys()))[0]
+
+        for alink in links.keys():
+            try:
+                if alink != sabs and absreal(alink) != sabs:
+                    continue
+
+                self.log("relinking [{}] to [{}]".format(alink, dabs))
+                bos.unlink(alink)
+            except:
+                pass
+
+            self._symlink(dabs, alink, False)
+
     def _get_wark(self, cj):
         if len(cj["name"]) > 1024 or len(cj["hash"]) > 512 * 1024:  # 16TiB
             raise Pebkac(400, "name or numchunks not according to spec")
@@ -1284,7 +1639,7 @@ class Up2k(object):
 
     def _hashlist_from_file(self, path):
         pp = self.pp if hasattr(self, "pp") else None
-        fsz = os.path.getsize(fsenc(path))
+        fsz = bos.path.getsize(path)
         csz = up2k_chunksize(fsz)
         ret = []
         with open(fsenc(path), "rb", 512 * 1024) as f:
@@ -1352,7 +1707,7 @@ class Up2k(object):
             for path, sz, times in ready:
                 self.log("lmod: setting times {} on {}".format(times, path))
                 try:
-                    os.utime(fsenc(path), times)
+                    bos.utime(path, times)
                 except:
                     self.log("lmod: failed to utime ({}, {})".format(path, times))
 
@@ -1388,13 +1743,13 @@ class Up2k(object):
                 try:
                     # remove the filename reservation
                     path = os.path.join(job["ptop"], job["prel"], job["name"])
-                    if os.path.getsize(fsenc(path)) == 0:
-                        os.unlink(fsenc(path))
+                    if bos.path.getsize(path) == 0:
+                        bos.unlink(path)
 
                     if len(job["hash"]) == len(job["need"]):
                         # PARTIAL is empty, delete that too
                         path = os.path.join(job["ptop"], job["prel"], job["tnam"])
-                        os.unlink(fsenc(path))
+                        bos.unlink(path)
                 except:
                     pass
 
@@ -1402,8 +1757,8 @@ class Up2k(object):
         if not reg:
             if ptop not in self.snap_prev or self.snap_prev[ptop] is not None:
                 self.snap_prev[ptop] = None
-                if os.path.exists(fsenc(path)):
-                    os.unlink(fsenc(path))
+                if bos.path.exists(path):
+                    bos.unlink(path)
             return
 
         newest = max(x["poke"] for _, x in reg.items()) if reg else 0
@@ -1411,10 +1766,7 @@ class Up2k(object):
         if etag == self.snap_prev.get(ptop):
             return
 
-        try:
-            os.makedirs(histpath)
-        except:
-            pass
+        bos.makedirs(histpath)
 
         path2 = "{}.{}".format(path, os.getpid())
         j = json.dumps(reg, indent=2, sort_keys=True).encode("utf-8")
@@ -1472,23 +1824,23 @@ class Up2k(object):
                 self.n_hashq -= 1
             # self.log("hashq {}".format(self.n_hashq))
 
-            ptop, rd, fn = self.hashq.get()
+            ptop, rd, fn, ip, at = self.hashq.get()
             # self.log("hashq {} pop {}/{}/{}".format(self.n_hashq, ptop, rd, fn))
             if "e2d" not in self.flags[ptop]:
                 continue
 
             abspath = os.path.join(ptop, rd, fn)
             self.log("hashing " + abspath)
-            inf = os.stat(fsenc(abspath))
+            inf = bos.stat(abspath)
             hashes = self._hashlist_from_file(abspath)
             wark = up2k_wark_from_hashlist(self.salt, inf.st_size, hashes)
             with self.mutex:
-                self.idx_wark(ptop, wark, rd, fn, inf.st_mtime, inf.st_size)
+                self.idx_wark(ptop, wark, rd, fn, inf.st_mtime, inf.st_size, ip, at)
 
-    def hash_file(self, ptop, flags, rd, fn):
+    def hash_file(self, ptop, flags, rd, fn, ip, at):
         with self.mutex:
             self.register_vpath(ptop, flags)
-            self.hashq.put([ptop, rd, fn])
+            self.hashq.put([ptop, rd, fn, ip, at])
             self.n_hashq += 1
         # self.log("hashq {} push {}/{}/{}".format(self.n_hashq, ptop, rd, fn))
 

copyparty/util.py

@@ -4,6 +4,7 @@ from __future__ import print_function, unicode_literals
 import re
 import os
 import sys
+import stat
 import time
 import base64
 import select
@@ -758,6 +759,19 @@ def sanitize_fn(fn, ok, bad):
     return fn.strip()
 
 
+def absreal(fpath):
+    try:
+        return fsdec(os.path.abspath(os.path.realpath(fsenc(fpath))))
+    except:
+        if not WINDOWS:
+            raise
+
+        # cpython bug introduced in 3.8, still exists in 3.9.1,
+        # some win7sp1 and win10:20H2 boxes cannot realpath a
+        # networked drive letter such as b"n:" or b"n:\\"
+        return os.path.abspath(os.path.realpath(fpath))
+
+
 def u8safe(txt):
     try:
         return txt.encode("utf-8", "xmlcharrefreplace").decode("utf-8", "replace")
@@ -815,6 +829,13 @@ def unquotep(txt):
     return w8dec(unq2)
 
 
+def vsplit(vpath):
+    if "/" not in vpath:
+        return "", vpath
+
+    return vpath.rsplit("/", 1)
+
+
 def w8dec(txt):
     """decodes filesystem-bytes to wtf8"""
     if PY2:
@@ -1014,6 +1035,9 @@ def sendfile_kern(lower, upper, f, s):
 
 
 def statdir(logger, scandir, lstat, top):
+    if lstat and not os.supports_follow_symlinks:
+        scandir = False
+
     try:
         btop = fsenc(top)
         if scandir and hasattr(os, "scandir"):
@@ -1038,6 +1062,29 @@ def statdir(logger, scandir, lstat, top):
         logger(src, "{} @ {}".format(repr(ex), top), 1)
 
 
+def rmdirs(logger, scandir, lstat, top):
+    if not os.path.exists(fsenc(top)) or not os.path.isdir(fsenc(top)):
+        top = os.path.dirname(top)
+    
+    dirs = statdir(logger, scandir, lstat, top)
+    dirs = [x[0] for x in dirs if stat.S_ISDIR(x[1].st_mode)]
+    dirs = [os.path.join(top, x) for x in dirs]
+    ok = []
+    ng = []
+    for d in dirs[::-1]:
+        a, b = rmdirs(logger, scandir, lstat, d)
+        ok += a
+        ng += b
+
+    try:
+        os.rmdir(fsenc(top))
+        ok.append(top)
+    except:
+        ng.append(top)
+
+    return ok, ng
+
+
 def unescape_cookie(orig):
     # mw=idk; doot=qwe%2Crty%3Basd+fgh%2Bjkl%25zxc%26vbn  # qwe,rty;asd fgh+jkl%zxc&vbn
     ret = ""
@@ -1081,7 +1128,7 @@ def guess_mime(url, fallback="application/octet-stream"):
     return ret
 
 
-def runcmd(*argv):
+def runcmd(argv):
     p = sp.Popen(argv, stdout=sp.PIPE, stderr=sp.PIPE)
     stdout, stderr = p.communicate()
     stdout = stdout.decode("utf-8", "replace")
@@ -1089,8 +1136,8 @@ def runcmd(*argv):
     return [p.returncode, stdout, stderr]
 
 
-def chkcmd(*argv):
-    ok, sout, serr = runcmd(*argv)
+def chkcmd(argv):
+    ok, sout, serr = runcmd(argv)
     if ok != 0:
         raise Exception(serr)
 

copyparty/web/browser.css

@@ -25,20 +25,99 @@ html, body {
 body {
 	padding-bottom: 5em;
 }
-#tt {
+pre, code, tt {
+	font-family: monospace, monospace;
+}
+
+
+
+#tt, #toast {
 	position: fixed;
 	max-width: 34em;
 	background: #222;
 	border: 0 solid #777;
+	box-shadow: 0 .2em .5em #222;
+	border-radius: .4em;
+	z-index: 9001;
+}
+#tt {
 	overflow: hidden;
 	margin-top: 1em;
 	padding: 0 1.3em;
 	height: 0;
 	opacity: .1;
 	transition: opacity 0.14s, height 0.14s, padding 0.14s;
-	box-shadow: 0 .2em .5em #222;
-	border-radius: .4em;
-	z-index: 9001;
+}
+#toast {
+	top: 1.4em;
+	right: -1em;
+	line-height: 1.5em;
+	padding: 1em 1.3em;
+	border-width: .4em 0;
+	transform: translateX(100%);
+	transition:
+		transform .4s cubic-bezier(.2, 1.2, .5, 1),
+		right .4s cubic-bezier(.2, 1.2, .5, 1);
+	text-shadow: 1px 1px 0 #000;
+	color: #fff;
+}
+#toastc {
+	display: inline-block;
+	position: absolute;
+	overflow: hidden;
+	left: 0;
+	width: 0;
+	opacity: 0;
+	padding: .3em 0;
+	margin: -.3em 0 0 0;
+	line-height: 1.5em;
+	color: #000;
+	border: none;
+	outline: none;
+	text-shadow: none;
+	border-radius: .5em 0 0 .5em;
+	transition: left .3s, width .3s, padding .3s, opacity .3s;
+}
+#toast pre {
+	margin: 0;
+}
+#toast.vis {
+	right: 1.3em;
+	transform: unset;
+}
+#toast.vis #toastc {
+	left: -2em;
+	width: .4em;
+	padding: .3em .8em;
+	opacity: 1;
+}
+#toast.inf {
+	background: #07a;
+	border-color: #0be;
+}
+#toast.inf #toastc {
+	background: #0be;
+}
+#toast.ok {
+	background: #4a0;
+	border-color: #8e4;
+}
+#toast.ok #toastc {
+	background: #8e4;
+}
+#toast.warn {
+	background: #970;
+	border-color: #fc0;
+}
+#toast.warn #toastc {
+	background: #fc0;
+}
+#toast.err {
+	background: #900;
+	border-color: #d06;
+}
+#toast.err #toastc {
+	background: #d06;
 }
 #tt.b {
 	padding: 0 2em;
@@ -60,7 +139,6 @@ body {
 	padding: .1em .3em;
 	border-top: 1px solid #777;
 	border-radius: .3em;
-	font-family: monospace, monospace;
 	line-height: 1.7em;
 }
 #tt em {
@@ -96,6 +174,10 @@ body {
 	padding: .3em 0;
 	scroll-margin-top: 45vh;
 }
+#files tr {
+	scroll-margin-top: 25vh;
+	scroll-margin-bottom: 20vh;
+}
 #files tbody div a {
 	color: #f5a;
 }
@@ -150,8 +232,7 @@ a, #files tbody div a:last-child {
 	border-top: 1px solid #383838;
 }
 #files tbody td:nth-child(3) {
-	font-family: monospace;
-	font-size: 1.3em;
+	font-family: monospace, monospace;
 	text-align: right;
 	padding-right: 1em;
 	white-space: nowrap;
@@ -229,21 +310,13 @@ a, #files tbody div a:last-child {
 	right: 2em;
 	color: #999;
 }
-#acc_info span:before {
-	color: #f4c;
-	border-bottom: 1px solid rgba(255,68,204,0.6);
+#acc_info span {
+	color: #999;
 	margin-right: .6em;
 }
-html.read #acc_info span:before {
-	content: 'Read-Only access';
-}
-html.write #acc_info span:before {
-	content: 'Write-Only access';
-}
-html.read.write #acc_info span:before {
-	content: 'Read-Write access';
-	color: #999;
-	border: none;
+#acc_info span.warn {
+	color: #f4c;
+	border-bottom: 1px solid rgba(255,68,204,0.6);
 }
 #files tbody a.play {
 	color: #e70;
@@ -270,6 +343,7 @@ html.light #ggrid a.sel {
 	border-color: #c37;
 }
 #files tbody tr.sel:hover td,
+#files tbody tr.sel:focus td,
 #ggrid a.sel:hover,
 html.light #ggrid a.sel:hover {
 	color: #fff;
@@ -304,6 +378,21 @@ html.light #ggrid a.sel {
 	color: #fff;
 	text-shadow: 0 0 1px #fff;
 }
+#files tr:focus {
+	outline: none;
+	position: relative;
+}
+#files tr:focus td {
+	background: #111;
+	border-color: #fc0 #111 #fc0 #111;
+	box-shadow: 0 .2em 0 #fc0, 0 -.2em 0 #fc0;
+}
+#files tr:focus td:first-child {
+	box-shadow: -.2em .2em 0 #fc0, -.2em -.2em 0 #fc0;
+}
+#files tr:focus+tr td {
+	border-top: 1px solid transparent;
+}
 #blocked {
 	position: fixed;
 	top: 0;
@@ -391,7 +480,6 @@ html.light #ggrid a.sel {
 	white-space: nowrap;
 	top: -1.2em;
 	right: 0;
-	width: 2.5em;
 	height: 1em;
 	font-size: 2em;
 	line-height: 1em;
@@ -400,7 +488,7 @@ html.light #ggrid a.sel {
 	background: #3c3c3c;
 	box-shadow: 0 0 .5em #222;
 	border-radius: .3em 0 0 0;
-	padding: .2em 0 0 .07em;
+	padding: .2em .2em;
 	color: #fff;
 }
 #wzip, #wnp {
@@ -422,12 +510,6 @@ html.light #ggrid a.sel {
 #wtoggle * {
 	line-height: 1em;
 }
-#wtoggle.np {
-	width: 6.63em;
-}
-#wtoggle.sel {
-	width: 7.57em;
-}
 #wtoggle.sel #wzip,
 #wtoggle.np #wnp {
 	display: inline-block;
@@ -435,15 +517,42 @@ html.light #ggrid a.sel {
 #wtoggle.sel.np #wnp {
 	display: none;
 }
+#wfm a,
 #wzip a {
-	font-size: .4em;
+	font-size: .5em;
 	padding: 0 .3em;
 	margin: -.3em .2em;
 	position: relative;
 	display: inline-block;
 }
-#wzip a+a {
-	margin-left: .8em;
+#wfm span {
+	font-size: .6em;
+	display: block;
+}
+#wfm a:not(.en) {
+	opacity: .3;
+	color: #f6c;
+}
+html.light #wfm a:not(.en) {
+	color: #c4a;
+}
+#files tbody tr.c1 td {
+	animation: fcut1 .5s ease-out;
+}
+#files tbody tr.c2 td {
+	animation: fcut2 .5s ease-out;
+}
+@keyframes fcut1 {
+	0% {opacity:0}
+	100% {opacity:1}
+}
+@keyframes fcut2 {
+	0% {opacity:0}
+	100% {opacity:1}
+}
+#wzip a {
+	font-size: .4em;
+	margin: -.3em .3em;
 }
 #wtoggle.sel #wzip #selzip {
 	top: -.6em;
@@ -846,7 +955,8 @@ input.eq_gain {
 	color: #300;
 	background: #fea;
 }
-.opwide {
+.opwide,
+#op_unpost {
 	max-width: none;
 	margin-right: 1.5em;
 }
@@ -948,6 +1058,16 @@ html.light #ggrid a:hover {
 	color: #015;
 	box-shadow: 0 .1em .5em #aaa;
 }
+#op_unpost {
+	padding: 1em;
+}
+#op_unpost td {
+	padding: .2em .4em;
+}
+#op_unpost a {
+	margin: 0;
+	padding: 0;
+}
 #pvol,
 #barbuf,
 #barpos,
@@ -986,6 +1106,9 @@ html.light {
 html.light #tt {
 	background: #fff;
 	border-color: #888 #000 #777 #000;
+}
+html.light #tt,
+html.light #toast {
 	box-shadow: 0 .3em 1em rgba(0,0,0,0.4);
 }
 html.light #tt code {
@@ -1079,6 +1202,17 @@ html.light #files td {
 html.light #files tbody tr:last-child td {
 	border-bottom: .2em solid #ccc;
 }
+html.light #files tr:focus td {
+	background: #fff;
+	border-color: #c37;
+	box-shadow: 0 .2em 0 #e80 , 0 -.2em 0 #e80;
+}
+html.light #files tr:focus td:first-child {
+	box-shadow: -.2em .2em 0 #e80, -.2em -.2em 0 #e80;
+}
+html.light #files tr.sel td {
+	background: #925;
+}
 html.light #files td:nth-child(2n) {
 	color: #d38;
 }
@@ -1132,7 +1266,8 @@ html.light #wnp {
 html.light #barbuf {
 	background: none;
 }
-html.light #files tr.sel:hover td {
+html.light #files tr.sel:hover td,
+html.light #files tr.sel:focus td {
 	background: #c37;
 }
 html.light #files tr.sel td {

copyparty/web/browser.html

@@ -59,12 +59,14 @@
 		</form>
 	</div>
 
+	<div id="op_unpost" class="opview opbox"></div>
+
 	<div id="op_up2k" class="opview"></div>
 
 	<div id="op_cfg" class="opview opbox opwide"></div>
 	
 	<h1 id="path">
-		<a href="#" id="entree" tt="show directory tree$NHotkey: B">🌲</a>
+		<a href="#" id="entree" tt="show navpane (directory tree sidebar)$NHotkey: B">🌲</a>
 		{%- for n in vpnodes %}
 		<a href="/{{ n[0] }}">{{ n[1] }}</a>
 		{%- endfor %}
@@ -126,6 +128,9 @@
 			tag_order_cfg = {{ tag_order }},
 			have_up2k_idx = {{ have_up2k_idx|tojson }},
 			have_tags_idx = {{ have_tags_idx|tojson }},
+			have_mv = {{ have_mv|tojson }},
+			have_del = {{ have_del|tojson }},
+			have_unpost = {{ have_unpost|tojson }},
 			have_zip = {{ have_zip|tojson }};
 	</script>
 	<script src="/.cpr/util.js?_={{ ts }}"></script>

copyparty/web/browser.js

@@ -12,6 +12,7 @@ ebi('ops').innerHTML = (
 	'<a href="#" data-dest="" tt="close submenu">---</a>\n' +
 	(have_up2k_idx ? (
 		'<a href="#" data-perm="read" data-dest="search" tt="search for files by attributes, path/name, music tags, or any combination of those.$N$N&lt;code&gt;foo bar&lt;/code&gt; = must contain both foo and bar,$N&lt;code&gt;foo -bar&lt;/code&gt; = must contain foo but not bar,$N&lt;code&gt;^yana .opus$&lt;/code&gt; = must start with yana and have the opus extension">🔎</a>\n' +
+		(have_del && have_unpost ? '<a href="#" data-dest="unpost" tt="unpost: delete your recent uploads">🧯</a>\n' : '') +
 		'<a href="#" data-dest="up2k" tt="up2k: upload files (if you have write-access) or toggle into the search-mode and drag files onto the search button to see if they exist somewhere on the server">🚀</a>\n'
 	) : (
 		'<a href="#" data-perm="write" data-dest="up2k" tt="up2k: upload files with resume support (close your browser and drop the same files in later)">🚀</a>\n'
@@ -29,15 +30,20 @@ ebi('ops').innerHTML = (
 // media player
 ebi('widget').innerHTML = (
 	'<div id="wtoggle">' +
-	'<span id="wzip"><a' +
-	' href="#" id="selall" tt="select all files">sel.<br />all</a><a' +
+	'<span id="wfm"><a' +
+	' href="#" id="fren" tt="rename selected item$NHotkey: F2">✎<span>name</span></a><a' +
+	' href="#" id="fdel" tt="delete selected items$NHotkey: ctrl-K">⌫<span>delete</span></a><a' +
+	' href="#" id="fcut" tt="cut selected items &lt;small&gt;(then paste somewhere else)&lt;/small&gt;$NHotkey: ctrl-X">✂<span>cut</span></a><a' +
+	' href="#" id="fpst" tt="paste a previously cut/copied selection$NHotkey: ctrl-V">📋<span>paste</span></a>' +
+	'</span><span id="wzip"><a' +
+	' href="#" id="selall" tt="select all files$NHotkey: ctrl-A (when file focused)">sel.<br />all</a><a' +
 	' href="#" id="selinv" tt="invert selection">sel.<br />inv.</a><a' +
 	' href="#" id="selzip" tt="download selection as archive">zip</a>' +
 	'</span><span id="wnp"><a' +
 	' href="#" id="npirc" tt="copy irc-formatted track info">📋irc</a><a' +
 	' href="#" id="nptxt" tt="copy plaintext track info">📋txt</a>' +
 	'</span><a' +
-	'	href="#" id="wtgrid">田</a><a' +
+	'	href="#" id="wtgrid" tt="toggle grid/list view">田</a><a' +
 	'	href="#" id="wtico">♫</a>' +
 	'</div>' +
 	'<div id="widgeti">' +
@@ -62,7 +68,7 @@ ebi('op_up2k').innerHTML = (
 	'		</td>\n' +
 	'		<td rowspan="2">\n' +
 	'			<input type="checkbox" id="ask_up" />\n' +
-	'			<label for="ask_up" tt="ask for confirmation befofre upload starts">💭</label>\n' +
+	'			<label for="ask_up" tt="ask for confirmation before upload starts">💭</label>\n' +
 	'		</td>\n' +
 	'		<td rowspan="2">\n' +
 	'			<input type="checkbox" id="flag_en" />\n' +
@@ -146,7 +152,7 @@ ebi('op_cfg').innerHTML = (
 );
 
 
-// tree sidebar
+// navpane
 ebi('tree').innerHTML = (
 	'<div id="treeh">\n' +
 	'	<a href="#" id="detree" tt="show breadcrumbs$NHotkey: B">🍞...</a>\n' +
@@ -208,17 +214,6 @@ function goto(dest) {
 }
 
 
-(function () {
-	goto();
-	var op = sread('opmode');
-	if (op !== null && op !== '.')
-		try {
-			goto(op);
-		}
-		catch (ex) { }
-})();
-
-
 var have_webp = null;
 (function () {
 	var img = new Image();
@@ -280,7 +275,7 @@ var mpl = (function () {
 		r.os_ctl = !r.os_ctl && have_mctl;
 		bcfg_set('au_os_ctl', r.os_ctl);
 		if (!have_mctl)
-			alert('need firefox 82+ or chrome 73+');
+			toast.err(5, 'need firefox 82+ or chrome 73+');
 	};
 
 	ebi('au_osd_cv').onclick = function (e) {
@@ -1348,7 +1343,7 @@ function play(tid, is_ev, seek, call_depth) {
 		return true;
 	}
 	catch (ex) {
-		alert('playback failed: ' + ex);
+		toast.err(0, 'playback failed: ' + ex);
 	}
 	setclass(oid, 'play');
 	setTimeout(next_song, 500);
@@ -1459,6 +1454,242 @@ function play_linked() {
 })();
 
 
+var fileman = (function () {
+	var bren = ebi('fren'),
+		bdel = ebi('fdel'),
+		bcut = ebi('fcut'),
+		bpst = ebi('fpst'),
+		r = {};
+
+	r.clip = null;
+	r.bus = new BroadcastChannel("fileman_bus");
+
+	r.render = function () {
+		if (r.clip === null)
+			r.clip = jread('fman_clip', []);
+
+		var nsel = msel.getsel().length;
+		clmod(bren, 'en', nsel == 1);
+		clmod(bdel, 'en', nsel);
+		clmod(bcut, 'en', nsel);
+		clmod(bpst, 'en', r.clip && r.clip.length);
+		bren.style.display = have_mv && has(perms, 'write') && has(perms, 'move') ? '' : 'none';
+		bdel.style.display = have_del && has(perms, 'delete') ? '' : 'none';
+		bcut.style.display = have_mv && has(perms, 'move') ? '' : 'none';
+		bpst.style.display = have_mv && has(perms, 'write') ? '' : 'none';
+		bpst.setAttribute('tt', 'paste ' + r.clip.length + ' items$NHotkey: ctrl-V');
+		ebi('wfm').style.display = QS('#wfm a.en:not([display])') ? '' : 'none';
+	};
+
+	r.rename = function (e) {
+		ev(e);
+		if (bren.style.display)
+			return toast.err(3, 'cannot rename:\nyou do not have “move” permission in this folder');
+
+		var sel = msel.getsel();
+		if (sel.length !== 1)
+			return toast.err(3, 'select exactly 1 item to rename');
+
+		var src = sel[0].vp;
+		if (src.endsWith('/'))
+			src = src.slice(0, -1);
+
+		var vsp = vsplit(src),
+			base = vsp[0],
+			ofn = vsp[1];
+
+		var fn = prompt('new filename:', ofn);
+		if (!fn || fn == ofn)
+			return toast.warn(1, 'rename aborted');
+
+		var dst = base + fn;
+
+		function rename_cb() {
+			if (this.readyState != XMLHttpRequest.DONE)
+				return;
+
+			if (this.status !== 200) {
+				var msg = this.responseText;
+				toast.err(9, 'rename failed:\n' + msg);
+				return;
+			}
+			toast.ok(2, 'rename OK');
+			treectl.goto(get_evpath());
+		}
+		var xhr = new XMLHttpRequest();
+		xhr.open('GET', src + '?move=' + dst, true);
+		xhr.onreadystatechange = rename_cb;
+		xhr.send();
+	};
+
+	r.delete = function (e) {
+		ev(e);
+		if (bdel.style.display)
+			return toast.err(3, 'cannot delete:\nyou do not have “delete” permission in this folder');
+
+		var sel = msel.getsel(),
+			vps = [];
+
+		for (var a = 0; a < sel.length; a++)
+			vps.push(sel[a].vp);
+
+		if (!sel.length)
+			return toast.err(3, 'select at least 1 item to delete');
+
+		if (!confirm('===== DANGER =====\nDELETE these ' + vps.length + ' items?\n\n' + vps.join('\n')))
+			return;
+
+		if (!confirm('Last chance! Delete?'))
+			return;
+
+		function deleter() {
+			var xhr = new XMLHttpRequest(),
+				vp = vps.shift();
+
+			if (!vp) {
+				toast.ok(2, 'delete OK');
+				treectl.goto(get_evpath());
+				return;
+			}
+			toast.inf(0, 'deleting ' + (vps.length + 1) + ' items\n\n' + vp);
+
+			xhr.open('GET', vp + '?delete', true);
+			xhr.onreadystatechange = delete_cb;
+			xhr.send();
+		}
+		function delete_cb() {
+			if (this.readyState != XMLHttpRequest.DONE)
+				return;
+
+			if (this.status !== 200) {
+				var msg = this.responseText;
+				toast.err(9, 'delete failed:\n' + msg);
+				return;
+			}
+			deleter();
+		}
+		deleter();
+	};
+
+	r.cut = function (e) {
+		ev(e);
+		if (bcut.style.display)
+			return toast.err(3, 'cannot cut:\nyou do not have “move” permission in this folder');
+
+		var sel = msel.getsel(),
+			vps = [];
+
+		if (!sel.length)
+			return toast.err(3, 'select at least 1 item to cut');
+
+		for (var a = 0; a < sel.length; a++) {
+			vps.push(sel[a].vp);
+			var cl = ebi(sel[a].id).closest('tr').classList,
+				inv = cl.contains('c1');
+
+			cl.remove(inv ? 'c1' : 'c2');
+			cl.add(inv ? 'c2' : 'c1');
+		}
+
+		toast.inf(1, 'cut ' + sel.length + ' items');
+		jwrite('fman_clip', vps);
+		r.tx(1);
+	};
+
+	r.paste = function (e) {
+		ev(e);
+		if (bpst.style.display)
+			return toast.err(3, 'cannot paste:\nyou do not have “write” permission in this folder');
+
+		if (!r.clip.length)
+			return toast.err(5, 'first cut some files/folders to paste\n\nnote: you can cut/paste across different browser tabs');
+
+		var req = [],
+			exists = [],
+			indir = [],
+			srcdir = vsplit(r.clip[0])[0],
+			links = QSA('#files tbody td:nth-child(2) a');
+
+		for (var a = 0, aa = links.length; a < aa; a++)
+			indir.push(links[a].getAttribute('name'));
+
+		for (var a = 0; a < r.clip.length; a++) {
+			var found = false;
+			for (var b = 0; b < indir.length; b++) {
+				if (r.clip[a].endsWith('/' + indir[b])) {
+					exists.push(r.clip[a]);
+					found = true;
+				}
+			}
+			if (!found)
+				req.push(r.clip[a]);
+		}
+
+		if (exists.length)
+			alert('these ' + exists.length + ' items cannot be pasted here (names already exist):\n\n' + exists.join('\n'));
+
+		if (!req.length)
+			return;
+
+		if (!confirm('paste these ' + req.length + ' items here?\n\n' + req.join('\n')))
+			return;
+
+		function paster() {
+			var xhr = new XMLHttpRequest(),
+				vp = req.shift();
+
+			if (!vp) {
+				toast.ok(2, 'paste OK');
+				treectl.goto(get_evpath());
+				r.tx(srcdir);
+				return;
+			}
+			toast.inf(0, 'pasting ' + (req.length + 1) + ' items\n\n' + vp);
+
+			var dst = get_evpath() + vp.split('/').slice(-1)[0];
+
+			xhr.open('GET', vp + '?move=' + dst, true);
+			xhr.onreadystatechange = paste_cb;
+			xhr.send();
+		}
+		function paste_cb() {
+			if (this.readyState != XMLHttpRequest.DONE)
+				return;
+
+			if (this.status !== 200) {
+				var msg = this.responseText;
+				toast.err(9, 'paste failed:\n' + msg);
+				return;
+			}
+			paster();
+		}
+		paster();
+
+		jwrite('fman_clip', []);
+	};
+
+	r.bus.onmessage = function (e) {
+		r.clip = null;
+		r.render();
+		var me = get_evpath();
+		if (e && e.data == me)
+			treectl.goto(e.data);
+	};
+
+	r.tx = function (msg) {
+		r.bus.postMessage(msg);
+		r.bus.onmessage();
+	};
+
+	bren.onclick = r.rename;
+	bdel.onclick = r.delete;
+	bcut.onclick = r.cut;
+	bpst.onclick = r.paste;
+
+	return r;
+})();
+
+
 var thegrid = (function () {
 	var lfiles = ebi('files'),
 		gfiles = mknod('div');
@@ -1497,9 +1728,6 @@ var thegrid = (function () {
 
 	ebi('griden').onclick = ebi('wtgrid').onclick = function (e) {
 		ev(e);
-		if (!this.closest)
-			return;
-
 		r.en = !r.en;
 		bcfg_set('griden', r.en);
 		if (r.en) {
@@ -1785,10 +2013,11 @@ function tree_up() {
 
 
 document.onkeydown = function (e) {
-	if (!document.activeElement || document.activeElement != document.body && document.activeElement.nodeName.toLowerCase() != 'a')
-		return;
+	var ae = document.activeElement, aet = '';
+	if (ae && ae != document.body)
+		aet = ae.nodeName.toLowerCase();
 
-	if (e.ctrlKey || e.altKey || e.metaKey || e.isComposing)
+	if (e.altKey || e.isComposing)
 		return;
 
 	if (QS('#bbox-overlay.visible'))
@@ -1796,6 +2025,55 @@ document.onkeydown = function (e) {
 
 	var k = e.code + '', pos = -1, n;
 
+	if (aet == 'tr' && ae.closest('#files')) {
+		var d = '';
+		if (k == 'ArrowUp') d = 'previous';
+		if (k == 'ArrowDown') d = 'next';
+		if (d) {
+			var el = ae[d + 'ElementSibling'];
+			if (el) {
+				el.focus();
+				if (ctrl(e))
+					document.documentElement.scrollTop += (d == 'next' ? 1 : -1) * el.offsetHeight;
+
+				if (e.shiftKey) {
+					clmod(el, 'sel', 't');
+					msel.selui();
+				}
+
+				return ev(e);
+			}
+		}
+		if (k == 'Space') {
+			clmod(ae, 'sel', 't');
+			msel.selui();
+			return ev(e);
+		}
+		if (k == 'KeyA' && ctrl(e)) {
+			var sel = msel.getsel(),
+				all = msel.getall();
+
+			msel.evsel(e, sel.length < all.length);
+			return ev(e);
+		}
+	}
+
+	if (aet && aet != 'a' && aet != 'tr')
+		return;
+
+	if (ctrl(e)) {
+		if (k == 'KeyX')
+			return fileman.cut();
+
+		if (k == 'KeyV')
+			return fileman.paste();
+
+		if (k == 'KeyK')
+			return fileman.delete();
+
+		return;
+	}
+
 	if (e.shiftKey && k != 'KeyA' && k != 'KeyD')
 		return;
 
@@ -1834,6 +2112,9 @@ document.onkeydown = function (e) {
 	if (k == 'KeyT')
 		return ebi('thumbs').click();
 
+	if (k == 'F2')
+		return fileman.rename();
+
 	if (!treectl.hidden && (!e.shiftKey || !thegrid.en)) {
 		if (k == 'KeyA')
 			return QS('#twig').click();
@@ -2118,7 +2399,6 @@ document.onkeydown = function (e) {
 		ebi('files').innerHTML = orig_html;
 		ebi('files').removeAttribute('q_raw');
 		orig_html = null;
-		msel.render();
 		reload_browser();
 	}
 })();
@@ -2264,7 +2544,7 @@ var treectl = (function () {
 			return;
 
 		if (this.status !== 200) {
-			alert("http " + this.status + ": " + this.responseText);
+			toast.err(0, "recvtree, http " + this.status + ": " + this.responseText);
 			return;
 		}
 
@@ -2386,7 +2666,7 @@ var treectl = (function () {
 			return;
 
 		if (this.status !== 200) {
-			alert("http " + this.status + ": " + this.responseText);
+			toast.err(0, "recvls, http " + this.status + ": " + this.responseText);
 			return;
 		}
 
@@ -2453,7 +2733,6 @@ var treectl = (function () {
 
 		filecols.set_style();
 		mukey.render();
-		msel.render();
 		reload_tree();
 		reload_browser();
 
@@ -2558,9 +2837,22 @@ function despin(sel) {
 function apply_perms(newperms) {
 	perms = newperms || [];
 
-	ebi('acc_info').innerHTML = '<span>' + (acct != '*' ?
-		'<a href="?pw=x">Logout ' + acct + '</a>' :
-		'<a href="?h">Login</a>') + '</span>';
+	var axs = [],
+		aclass = '>',
+		chk = ['read', 'write', 'rename', 'delete'];
+
+	for (var a = 0; a < chk.length; a++)
+		if (has(perms, chk[a]))
+			axs.push(chk[a].slice(0, 1).toUpperCase() + chk[a].slice(1));
+
+	axs = axs.join('-');
+	if (perms.length == 1) {
+		aclass = ' class="warn">';
+		axs += '-Only';
+	}
+
+	ebi('acc_info').innerHTML = '<span' + aclass + axs + ' access</span>' + (acct != '*' ?
+		'<a href="/?pw=x">Logout ' + acct + '</a>' : '<a href="/?h">Login</a>');
 
 	var o = QSA('#ops>a[data-perm], #u2footfoot');
 	for (var a = 0; a < o.length; a++) {
@@ -3007,44 +3299,81 @@ var arcfmt = (function () {
 
 
 var msel = (function () {
-	function getsel() {
-		var names = [],
-			links = QSA('#files tbody tr.sel td:nth-child(2) a');
+	var r = {};
+	r.sel = null;
+	r.all = null;
 
-		for (var a = 0, aa = links.length; a < aa; a++)
-			names.push(links[a].getAttribute('href').replace(/\/$/, "").split('/').slice(-1));
+	r.load = function () {
+		if (r.sel)
+			return;
 
-		return names;
-	}
-	function selui() {
-		clmod(ebi('wtoggle'), 'sel', getsel().length);
+		r.sel = [];
+		r.all = [];
+		var links = QSA('#files tbody td:nth-child(2) a:last-child'),
+			vbase = get_evpath();
+
+		for (var a = 0, aa = links.length; a < aa; a++) {
+			var href = links[a].getAttribute('href').replace(/\/$/, ""),
+				item = {};
+
+			item.id = links[a].getAttribute('id');
+			item.sel = links[a].closest('tr').classList.contains('sel');
+			item.vp = href.indexOf('/') !== -1 ? href : vbase + href;
+			item.name = href.split('/').slice(-1);
+
+			r.all.push(item);
+			if (item.sel)
+				r.sel.push(item);
+
+			links[a].setAttribute('name', item.name);
+			links[a].closest('tr').setAttribute('tabindex', '0');
+		}
+	};
+
+	r.getsel = function () {
+		r.load();
+		return r.sel;
+	};
+	r.getall = function () {
+		r.load();
+		return r.all;
+	};
+	r.selui = function () {
+		r.sel = r.all = null;
+		clmod(ebi('wtoggle'), 'sel', r.getsel().length);
 		thegrid.loadsel();
+		fileman.render();
 	}
-	function seltgl(e) {
+	r.seltgl = function (e) {
 		ev(e);
 		var tr = this.parentNode;
 		clmod(tr, 'sel', 't');
-		selui();
+		r.selui();
 	}
-	function evsel(e, fun) {
+	r.evsel = function (e, fun) {
 		ev(e);
 		var trs = QSA('#files tbody tr');
 		for (var a = 0, aa = trs.length; a < aa; a++)
 			clmod(trs[a], 'sel', fun);
-		selui();
+		r.selui();
 	}
 	ebi('selall').onclick = function (e) {
-		evsel(e, "add");
+		r.evsel(e, "add");
 	};
 	ebi('selinv').onclick = function (e) {
-		evsel(e, "t");
+		r.evsel(e, "t");
 	};
 	ebi('selzip').onclick = function (e) {
 		ev(e);
-		var names = getsel(),
+		var names = r.getsel(),
 			arg = ebi('selzip').getAttribute('fmt'),
-			txt = names.join('\n'),
-			frm = mknod('form');
+			frm = mknod('form'),
+			txt = [];
+
+		for (var a = 0; a < names.length; a++)
+			txt.push(names[a].name);
+
+		txt = txt.join('\n');
 
 		frm.setAttribute('action', '?' + arg);
 		frm.setAttribute('method', 'post');
@@ -3064,16 +3393,17 @@ var msel = (function () {
 		console.log(txt);
 		frm.submit();
 	};
-	function render() {
+	r.render = function () {
 		var tds = QSA('#files tbody td+td+td');
 		for (var a = 0, aa = tds.length; a < aa; a++) {
-			tds[a].onclick = seltgl;
+			tds[a].onclick = r.seltgl;
 		}
+		r.selui();
 		arcfmt.render();
+		fileman.render();
+		ebi('selzip').style.display = ebi('unsearch') ? 'none' : '';
 	}
-	return {
-		"render": render
-	};
+	return r;
 })();
 
 
@@ -3100,6 +3430,160 @@ function ev_row_tgl(e) {
 }
 
 
+var unpost = (function () {
+	ebi('op_unpost').innerHTML = (
+		"you can delete your recent uploads below &ndash; click the fire-extinguisher icon to refresh" +
+		'<p>optional filter:&nbsp; URL must contain <input type="text" id="unpost_filt" size="20" /><a id="unpost_nofilt" href="#">clear filter</a></p>' +
+		'<div id="unpost"></div>'
+	);
+
+	var r = {},
+		ct = ebi('unpost'),
+		filt = ebi('unpost_filt');
+
+	r.files = [];
+	r.me = null;
+
+	r.load = function () {
+		var me = Date.now(),
+			html = [];
+
+		function unpost_load_cb() {
+			if (this.readyState != XMLHttpRequest.DONE)
+				return;
+
+			if (this.status !== 200) {
+				var msg = this.responseText;
+				toast.err(9, 'unpost-load failed:\n' + msg);
+				ebi('op_unpost').innerHTML = html.join('\n');
+				return;
+			}
+
+			var res = JSON.parse(this.responseText);
+			if (res.length) {
+				if (res.length == 2000)
+					html.push("<p>showing first 2000 files (use the filter)");
+				else
+					html.push("<p>" + res.length + " uploads can be deleted");
+
+				html.push(" &ndash; sorted by upload time &ndash; most recent first:</p>");
+				html.push("<table><thead><tr><td></td><td>time</td><td>size</td><td>file</td></tr></thead><tbody>");
+			}
+			else
+				html.push("<p>sike! no uploads " + (filt.value ? 'matching that filter' : '') + " are sufficiently recent</p>");
+
+			var mods = [1000, 100, 10];
+			for (var a = 0; a < res.length; a++) {
+				for (var b = 0; b < mods.length; b++)
+					if (a % mods[b] == 0 && res.length > a + mods[b] / 10)
+						html.push(
+							'<tr><td></td><td colspan="3" style="padding:.5em">' +
+							'<a me="' + me + '" class="n' + a + '" n2="' + (a + mods[b]) +
+							'" href="#">delete the next ' + Math.min(mods[b], res.length - a) + ' files below</a></td></tr>');
+				html.push(
+					'<tr><td><a me="' + me + '" class="n' + a + '" href="#">delete</a></td>' +
+					'<td>' + unix2iso(res[a].at) + '</td>' +
+					'<td>' + res[a].sz + '</td>' +
+					'<td>' + linksplit(res[a].vp).join(' ') + '</td></tr>');
+			}
+
+			html.push("</tbody></table>");
+			ct.innerHTML = html.join('\n');
+			r.files = res;
+			r.me = me;
+		}
+
+		var q = '/?ups';
+		if (filt.value)
+			q += '&filter=' + uricom_enc(filt.value, true);
+
+		var xhr = new XMLHttpRequest();
+		xhr.open('GET', q, true);
+		xhr.onreadystatechange = unpost_load_cb;
+		xhr.send();
+
+		ct.innerHTML = "<p><em>loading your recent uploads...</em></p>";
+	};
+
+	function unpost_delete_cb() {
+		if (this.readyState != XMLHttpRequest.DONE)
+			return;
+
+		if (this.status !== 200) {
+			var msg = this.responseText;
+			toast.err(9, 'unpost-delete failed:\n' + msg);
+			return;
+		}
+
+		for (var a = this.n; a < this.n2; a++) {
+			var o = QSA('#op_unpost a.n' + a);
+			for (var b = 0; b < o.length; b++) {
+				var o2 = o[b].closest('tr');
+				o2.parentNode.removeChild(o2);
+			}
+		}
+		toast.ok(5, this.responseText);
+
+		if (!QS('#op_unpost a[me]'))
+			ebi(goto_unpost());
+	}
+
+	ct.onclick = function (e) {
+		var tgt = e.target.closest('a[me]');
+		if (!tgt)
+			return;
+
+		if (!tgt.getAttribute('href'))
+			return;
+
+		var ame = tgt.getAttribute('me');
+		if (ame != r.me)
+			return toast.err(0, 'something broke, please try a refresh');
+
+		var n = parseInt(tgt.className.slice(1)),
+			n2 = parseInt(tgt.getAttribute('n2') || n + 1),
+			req = [];
+
+		for (var a = n; a < n2; a++)
+			if (QS('#op_unpost a.n' + a))
+				req.push(r.files[a].vp);
+
+		var links = QSA('#op_unpost a.n' + n);
+		for (var a = 0, aa = links.length; a < aa; a++) {
+			links[a].removeAttribute('href');
+			links[a].innerHTML = '[busy]';
+		}
+
+		toast.inf(0, "deleting " + req.length + " files...");
+
+		var xhr = new XMLHttpRequest();
+		xhr.n = n;
+		xhr.n2 = n2;
+		xhr.open('POST', '/?delete', true);
+		xhr.onreadystatechange = unpost_delete_cb;
+		xhr.send(JSON.stringify(req));
+	};
+
+	var tfilt = null;
+	filt.oninput = function () {
+		clearTimeout(tfilt);
+		tfilt = setTimeout(r.load, 250);
+	};
+
+	ebi('unpost_nofilt').onclick = function () {
+		filt.value = '';
+		r.load();
+	};
+
+	return r;
+})();
+
+
+function goto_unpost(e) {
+	unpost.load();
+}
+
+
 function reload_mp() {
 	if (mp && mp.au) {
 		mp.au.pause();
@@ -3148,8 +3632,8 @@ function reload_browser(not_mp) {
 		up2k.set_fsearch();
 
 	thegrid.setdirty();
+	msel.render();
 }
 reload_browser(true);
 mukey.render();
-msel.render();
 play_linked();

copyparty/web/md.css

@@ -8,20 +8,96 @@ html, body {
 	font-family: sans-serif;
 	line-height: 1.5em;
 }
-#tt {
+
+
+
+#tt, #toast {
 	position: fixed;
 	max-width: 34em;
 	background: #222;
 	border: 0 solid #777;
+	box-shadow: 0 .2em .5em #222;
+	border-radius: .4em;
+	z-index: 9001;
+}
+#tt {
 	overflow: hidden;
 	margin-top: 1em;
 	padding: 0 1.3em;
 	height: 0;
 	opacity: .1;
 	transition: opacity 0.14s, height 0.14s, padding 0.14s;
-	box-shadow: 0 .2em .5em #222;
-	border-radius: .4em;
-	z-index: 9001;
+}
+#toast {
+	top: 1.4em;
+	right: -1em;
+	line-height: 1.5em;
+	padding: 1em 1.3em;
+	border-width: .4em 0;
+	transform: translateX(100%);
+	transition:
+		transform .4s cubic-bezier(.2, 1.2, .5, 1),
+		right .4s cubic-bezier(.2, 1.2, .5, 1);
+	text-shadow: 1px 1px 0 #000;
+	color: #fff;
+}
+#toast pre {
+	margin: 0;
+}
+#toastc {
+	display: inline-block;
+	position: absolute;
+	overflow: hidden;
+	left: 0;
+	width: 0;
+	opacity: 0;
+	padding: .3em 0;
+	margin: -.3em 0 0 0;
+	line-height: 1.5em;
+	color: #000;
+	border: none;
+	outline: none;
+	text-shadow: none;
+	border-radius: .5em 0 0 .5em;
+	transition: left .3s, width .3s, padding .3s, opacity .3s;
+}
+#toast.vis {
+	right: 1.3em;
+	transform: unset;
+}
+#toast.vis #toastc {
+	left: -2em;
+	width: .4em;
+	padding: .3em .8em;
+	opacity: 1;
+}
+#toast.inf {
+	background: #07a;
+	border-color: #0be;
+}
+#toast.inf #toastc {
+	background: #0be;
+}
+#toast.ok {
+	background: #4a0;
+	border-color: #8e4;
+}
+#toast.ok #toastc {
+	background: #8e4;
+}
+#toast.warn {
+	background: #970;
+	border-color: #fc0;
+}
+#toast.warn #toastc {
+	background: #fc0;
+}
+#toast.err {
+	background: #900;
+	border-color: #d06;
+}
+#toast.err #toastc {
+	background: #d06;
 }
 #tt.b {
 	padding: 0 2em;
@@ -43,12 +119,29 @@ html, body {
 	padding: .1em .3em;
 	border-top: 1px solid #777;
 	border-radius: .3em;
-	font-family: monospace, monospace;
 	line-height: 1.7em;
 }
 #tt em {
 	color: #f6a;
 }
+html.light #tt {
+	background: #fff;
+	border-color: #888 #000 #777 #000;
+}
+html.light #tt,
+html.light #toast {
+	box-shadow: 0 .3em 1em rgba(0,0,0,0.4);
+}
+html.light #tt code {
+	background: #060;
+	color: #fff;
+}
+html.light #tt em {
+	color: #d38;
+}
+
+
+
 #mtw {
 	display: none;
 }
@@ -67,7 +160,7 @@ pre, code, a {
 code {
 	font-size: .96em;
 }
-pre, code {
+pre, code, tt {
 	font-family: 'scp', monospace, monospace;
 	white-space: pre-wrap;
 	word-break: break-all;
@@ -207,7 +300,7 @@ small {
 	z-index: 99;
 	position: relative;
 	display: inline-block;
-	font-family: monospace, monospace;
+	font-family: 'scp', monospace, monospace;
 	font-weight: bold;
 	font-size: 1.3em;
 	line-height: .1em;

copyparty/web/md.html

@@ -131,18 +131,18 @@ var md_opt = {
 };
 
 (function () {
-    var btn = document.getElementById("lightswitch");
-    var toggle = function (e) {
-		if (e) e.preventDefault();
-        var dark = !document.documentElement.getAttribute("class");
-        document.documentElement.setAttribute("class", dark ? "dark" : "");
-        btn.innerHTML = "go " + (dark ? "light" : "dark");
-        if (window.localStorage)
-            localStorage.setItem('lightmode', dark ? 0 : 1);
-    };
-    btn.onclick = toggle;
-    if (window.localStorage && localStorage.getItem('lightmode') != 1)
-		toggle();
+    var l = localStorage,
+		drk = l.getItem('lightmode') != 1,
+		btn = document.getElementById("lightswitch"),
+		f = function (e) {
+if (e) { e.preventDefault(); drk = !drk; }
+document.documentElement.setAttribute("class", drk? "dark":"light");
+btn.innerHTML = "go " + (drk ? "light":"dark");
+l.setItem('lightmode', drk? 0:1);
+    	};
+	
+	btn.onclick = f;
+	f();
 })();
 
 	</script>

copyparty/web/md2.css

@@ -84,13 +84,10 @@ html.dark #save.force-save {
 #save.disabled {
 	opacity: .4;
 }
-#helpbox,
-#toast {
+#helpbox {
 	background: #f7f7f7;
 	border-radius: .4em;
 	z-index: 9001;
-}
-#helpbox {
 	display: none;
 	position: fixed;
 	padding: 2em;
@@ -107,19 +104,7 @@ html.dark #save.force-save {
 }
 html.dark #helpbox {
 	box-shadow: 0 .5em 2em #444;
-}
-html.dark #helpbox,
-html.dark #toast {
 	background: #222;
 	border: 1px solid #079;
 	border-width: 1px 0;
 }
-#toast {
-	font-weight: bold;
-	text-align: center;
-	padding: .6em 0;
-	position: fixed;
-	top: 30%;
-	transition: opacity 0.2s ease-in-out;
-	opacity: 1;
-}

copyparty/web/md2.js

@@ -236,7 +236,7 @@ function Modpoll() {
 
         var skip = null;
 
-        if (ebi('toast'))
+        if (toast.visible)
             skip = 'toast';
 
         else if (this.skip_one)
@@ -285,16 +285,15 @@ function Modpoll() {
             console.log("modpoll diff |" + server_ref.length + "|, |" + server_now.length + "|");
             this.modpoll.disabled = true;
             var msg = [
-                "The document has changed on the server.<br />" +
+                "The document has changed on the server.",
                 "The changes will NOT be loaded into your editor automatically.",
-
-                "Press F5 or CTRL-R to refresh the page,<br />" +
+                "",
+                "Press F5 or CTRL-R to refresh the page,",
                 "replacing your document with the server copy.",
-
-                "You can click this message to ignore and contnue."
+                "",
+                "You can close this message to ignore and contnue."
             ];
-            return toast(false, "box-shadow:0 1em 2em rgba(64,64,64,0.8);font-weight:normal",
-                36, "<p>" + msg.join('</p>\n<p>') + '</p>');
+            return toast.warn(0, msg.join('\n'));
         }
 
         console.log('modpoll eq');
@@ -323,16 +322,12 @@ function save(e) {
     var save_btn = ebi("save"),
         save_cls = save_btn.getAttribute('class') + '';
 
-    if (save_cls.indexOf('disabled') >= 0) {
-        toast(true, ";font-size:2em;color:#c90", 9, "no changes");
-        return;
-    }
+    if (save_cls.indexOf('disabled') >= 0)
+        return toast.inf(2, "no changes");
 
     var force = (save_cls.indexOf('force-save') >= 0);
-    if (force && !confirm('confirm that you wish to lose the changes made on the server since you opened this document')) {
-        alert('ok, aborted');
-        return;
-    }
+    if (force && !confirm('confirm that you wish to lose the changes made on the server since you opened this document'))
+        return toast.inf(3, 'aborted');
 
     var txt = dom_src.value;
 
@@ -357,18 +352,15 @@ function save_cb() {
     if (this.readyState != XMLHttpRequest.DONE)
         return;
 
-    if (this.status !== 200) {
-        alert('Error!  The file was NOT saved.\n\n' + this.status + ": " + (this.responseText + '').replace(/^<pre>/, ""));
-        return;
-    }
+    if (this.status !== 200)
+        return alert('Error!  The file was NOT saved.\n\n' + this.status + ": " + (this.responseText + '').replace(/^<pre>/, ""));
 
     var r;
     try {
         r = JSON.parse(this.responseText);
     }
     catch (ex) {
-        alert('Failed to parse reply from server:\n\n' + this.responseText);
-        return;
+        return alert('Failed to parse reply from server:\n\n' + this.responseText);
     }
 
     if (!r.ok) {
@@ -443,46 +435,10 @@ function savechk_cb() {
     last_modified = this.lastmod;
     server_md = this.txt;
     draw_md();
-    toast(true, ";font-size:6em;font-family:serif;color:#9b4", 4,
-        'OK✔️<span style="font-size:.2em;color:#999;position:absolute">' + this.ntry + '</span>');
-
+    toast.ok(2, 'save OK' + (this.ntry ? '\nattempt ' + this.ntry : ''));
     modpoll.disabled = false;
 }
 
-function toast(autoclose, style, width, msg) {
-    var ok = ebi("toast");
-    if (ok)
-        ok.parentNode.removeChild(ok);
-
-    style = "width:" + width + "em;left:calc(50% - " + (width / 2) + "em);" + style;
-    ok = mknod('div');
-    ok.setAttribute('id', 'toast');
-    ok.setAttribute('style', style);
-    ok.innerHTML = msg;
-    var parent = ebi('m');
-    document.documentElement.appendChild(ok);
-
-    var hide = function (delay) {
-        delay = delay || 0;
-
-        setTimeout(function () {
-            ok.style.opacity = 0;
-        }, delay);
-
-        setTimeout(function () {
-            if (ok.parentNode)
-                ok.parentNode.removeChild(ok);
-        }, delay + 250);
-    }
-
-    ok.onclick = function () {
-        hide(0);
-    };
-
-    if (autoclose)
-        hide(500);
-}
-
 
 // firefox bug: initial selection offset isn't cleared properly through js
 var ff_clearsel = (function () {
@@ -761,7 +717,7 @@ function fmt_table(e) {
 
         var ind2 = tab[a].match(re_ind)[0];
         if (ind != ind2 && a != 1)  // the table can be a list entry or something, ignore [0]
-            return alert(err + 'indentation mismatch on row#2 and ' + row_name + ',\n' + tab[a]);
+            return toast.err(7, err + 'indentation mismatch on row#2 and ' + row_name + ',\n' + tab[a]);
 
         var t = tab[a].slice(ind.length);
         t = t.replace(re_lpipe, "");
@@ -771,7 +727,7 @@ function fmt_table(e) {
         if (a == 0)
             ncols = tab[a].length;
         else if (ncols < tab[a].length)
-            return alert(err + 'num.columns(' + row_name + ') exceeding row#2;  ' + ncols + ' < ' + tab[a].length);
+            return toast.err(7, err + 'num.columns(' + row_name + ') exceeding row#2;  ' + ncols + ' < ' + tab[a].length);
 
         // if row has less columns than row2, fill them in
         while (tab[a].length < ncols)
@@ -788,7 +744,7 @@ function fmt_table(e) {
     for (var col = 0; col < tab[1].length; col++) {
         var m = tab[1][col].match(re_align);
         if (!m)
-            return alert(err + 'invalid column specification, row#2, col ' + (col + 1) + ', [' + tab[1][col] + ']');
+            return toast.err(7, err + 'invalid column specification, row#2, col ' + (col + 1) + ', [' + tab[1][col] + ']');
 
         if (m[2]) {
             if (m[1])
@@ -876,10 +832,9 @@ function mark_uni(e) {
         ptn = new RegExp('([^' + js_uni_whitelist + ']+)', 'g'),
         mod = txt.replace(/\r/g, "").replace(ptn, "\u2588\u2770$1\u2771");
 
-    if (txt == mod) {
-        alert('no results;  no modifications were made');
-        return;
-    }
+    if (txt == mod)
+        return toast.inf(5, 'no results;  no modifications were made');
+
     dom_src.value = mod;
 }
 
@@ -893,10 +848,9 @@ function iter_uni(e) {
         re = new RegExp('([^' + js_uni_whitelist + ']+)'),
         m = re.exec(txt.slice(ofs));
 
-    if (!m) {
-        alert('no more hits from cursor onwards');
-        return;
-    }
+    if (!m)
+        return toast.inf(5, 'no more hits from cursor onwards');
+
     ofs += m.index;
 
     dom_src.setSelectionRange(ofs, ofs + m[0].length, "forward");

copyparty/web/mde.html

@@ -30,16 +30,15 @@ var md_opt = {
 };
 
 var lightswitch = (function () {
-	var fun = function () {
-		var dark = !document.documentElement.getAttribute("class");
-		document.documentElement.setAttribute("class", dark ? "dark" : "");
-		if (window.localStorage)
-			localStorage.setItem('lightmode', dark ? 0 : 1);
-	};
-	if (window.localStorage && localStorage.getItem('lightmode') != 1)
-		fun();
-	
-	return fun;
+	var l = localStorage,
+		drk = l.getItem('lightmode') != 1,
+		f = function (e) {
+if (e) drk = !drk;
+document.documentElement.setAttribute("class", drk? "dark":"light");
+l.setItem('lightmode', drk? 0:1);
+		};
+	f();
+	return f;
 })();
 
 	</script>

copyparty/web/mde.js

@@ -75,7 +75,7 @@ function set_jumpto() {
 }
 
 function jumpto(ev) {
-    var tgt = ev.target || ev.srcElement;
+    var tgt = ev.target;
     var ln = null;
     while (tgt && !ln) {
         ln = tgt.getAttribute('data-ln');
@@ -106,15 +106,12 @@ function md_changed(mde, on_srv) {
 
 function save(mde) {
     var save_btn = QS('.editor-toolbar button.save');
-    if (save_btn.classList.contains('disabled')) {
-        alert('there is nothing to save');
-        return;
-    }
+    if (save_btn.classList.contains('disabled'))
+        return toast.inf(2, 'no changes');
+
     var force = save_btn.classList.contains('force-save');
-    if (force && !confirm('confirm that you wish to lose the changes made on the server since you opened this document')) {
-        alert('ok, aborted');
-        return;
-    }
+    if (force && !confirm('confirm that you wish to lose the changes made on the server since you opened this document'))
+        return toast.inf(3, 'aborted');
 
     var txt = mde.value();
 
@@ -138,18 +135,15 @@ function save_cb() {
     if (this.readyState != XMLHttpRequest.DONE)
         return;
 
-    if (this.status !== 200) {
-        alert('Error!  The file was NOT saved.\n\n' + this.status + ": " + (this.responseText + '').replace(/^<pre>/, ""));
-        return;
-    }
+    if (this.status !== 200)
+        return alert('Error!  The file was NOT saved.\n\n' + this.status + ": " + (this.responseText + '').replace(/^<pre>/, ""));
 
     var r;
     try {
         r = JSON.parse(this.responseText);
     }
     catch (ex) {
-        alert('Failed to parse reply from server:\n\n' + this.responseText);
-        return;
+        return alert('Failed to parse reply from server:\n\n' + this.responseText);
     }
 
     if (!r.ok) {

copyparty/web/splash.html

@@ -68,7 +68,7 @@
     </div>
     <script>
 
-if (window.localStorage && localStorage.getItem('lightmode') != 1)
+if (localStorage.getItem('lightmode') != 1)
     document.documentElement.setAttribute("class", "dark");
 
 </script>

copyparty/web/up2k.js

@@ -926,8 +926,11 @@ function up2k_init(subtle) {
                 return;
 
             clearTimeout(tto);
+            if (crashed)
+                return defer();
+
             running = true;
-            while (window['vis_exh']) {
+            while (true) {
                 var now = Date.now(),
                     is_busy = 0 !=
                         st.todo.head.length +
@@ -1009,7 +1012,7 @@ function up2k_init(subtle) {
                     mou_ikkai = true;
                 }
 
-                if (!mou_ikkai)
+                if (!mou_ikkai || crashed)
                     return defer();
             }
         }
@@ -1173,10 +1176,6 @@ function up2k_init(subtle) {
                 }
 
                 t.t_hashed = Date.now();
-                if (t.n == 0 && window.location.hash == '#dbg') {
-                    var spd = (t.size / ((t.t_hashed - t.t_hashing) / 1000.)) / (1024 * 1024.);
-                    alert('{0} ms, {1} MB/s\n'.format(t.t_hashed - t.t_hashing, spd.toFixed(3)) + t.hash.join('\n'));
-                }
 
                 pvis.seth(t.n, 2, 'hashing done');
                 pvis.seth(t.n, 1, '📦 wait');
@@ -1519,7 +1518,7 @@ function up2k_init(subtle) {
                 try { orz(xhr); } catch (ex) { vis_exh(ex + '', '', '', '', ex); }
             };
             xhr.onerror = function (xev) {
-                if (!window['vis_exh'])
+                if (crashed)
                     return;
 
                 console.log('chunkpit onerror, retrying', t);
@@ -1774,3 +1773,14 @@ if (QS('#op_up2k.act'))
     goto_up2k();
 
 apply_perms(perms);
+
+
+(function () {
+    goto();
+    var op = sread('opmode');
+    if (op !== null && op !== '.')
+        try {
+            goto(op);
+        }
+        catch (ex) { }
+})();

copyparty/web/upload.css

@@ -87,8 +87,9 @@
 #u2tab td:nth-child(3) {
 	width: 40%;
 }
-#op_up2k.srch #u2tab td:nth-child(3) {
+#op_up2k.srch td.prog {
 	font-family: sans-serif;
+	font-size: 1em;
 	width: auto;
 }
 #u2tab tbody tr:hover td {
@@ -245,7 +246,7 @@ html.light #u2foot .warn span {
 	margin-bottom: -1em;
 }
 .prog {
-	font-family: monospace;
+	font-family: monospace, monospace;
 }
 #u2tab a>span {
 	font-weight: bold;

copyparty/web/util.js

@@ -11,6 +11,12 @@ var is_touch = 'ontouchstart' in window,
     ANDROID = /android/i.test(navigator.userAgent);
 
 
+var ebi = document.getElementById.bind(document),
+    QS = document.querySelector.bind(document),
+    QSA = document.querySelectorAll.bind(document),
+    mknod = document.createElement.bind(document);
+
+
 // error handler for mobile devices
 function esc(txt) {
     return txt.replace(/[&"<>]/g, function (c) {
@@ -22,13 +28,18 @@ function esc(txt) {
         }[c];
     });
 }
+var crashed = false, ignexd = {};
 function vis_exh(msg, url, lineNo, columnNo, error) {
-    if (!window.onerror)
+    if ((msg + '').indexOf('ResizeObserver') !== -1)
+        return;  // chrome issue 809574 (benign, from <video>)
+
+    var ekey = url + '\n' + lineNo + '\n' + msg;
+    if (ignexd[ekey] || crashed)
         return;
 
+    crashed = true;
     window.onerror = undefined;
-    window['vis_exh'] = null;
-    var html = ['<h1>you hit a bug!</h1><p style="font-size:1.3em;margin:0">try to <a href="#" onclick="localStorage.clear();location.reload();" style="text-decoration:underline;color:#fc0">reset copyparty settings</a> if you are stuck here</p><p>please send me a screenshot arigathanks gozaimuch: <code>ed/irc.rizon.net</code> or <code>ed#2644</code><br />&nbsp; (and if you can, press F12 and include the "Console" tab in the screenshot too)</p><p>',
+    var html = ['<h1>you hit a bug!</h1><p style="font-size:1.3em;margin:0">try to <a href="#" onclick="localStorage.clear();location.reload();">reset copyparty settings</a> if you are stuck here, or <a href="#" onclick="ignex();">ignore this</a> / <a href="#" onclick="ignex(true);">ignore all</a></p><p>please send me a screenshot arigathanks gozaimuch: <code>ed/irc.rizon.net</code> or <code>ed#2644</code><br />&nbsp; (and if you can, press F12 and include the "Console" tab in the screenshot too)</p><p>',
         esc(url + ' @' + lineNo + ':' + columnNo), '<br />' + esc(String(msg)) + '</p>'];
 
     try {
@@ -39,23 +50,38 @@ function vis_exh(msg, url, lineNo, columnNo, error) {
                     html.push('<h3>' + find[a] + '</h3>' +
                         esc(String(error[find[a]])).replace(/\n/g, '<br />\n'));
         }
+        ignexd[ekey] = true;
         html.push('<h3>localStore</h3>' + esc(JSON.stringify(localStorage)));
     }
     catch (e) { }
-    document.body.innerHTML = html.join('\n');
 
-    var s = mknod('style');
-    s.innerHTML = 'body{background:#333;color:#ddd;font-family:sans-serif;font-size:0.8em;padding:0 1em 1em 1em} h1{margin:.5em 1em 0 0;padding:0} h3{border-top:1px solid #999;margin:1em 0 0 0} code{color:#bf7;background:#222;padding:.1em;margin:.2em;font-size:1.1em;font-family:monospace,monospace} *{line-height:1.5em}';
-    document.head.appendChild(s);
+    try {
+        var exbox = ebi('exbox');
+        if (!exbox) {
+            exbox = mknod('div');
+            exbox.setAttribute('id', 'exbox');
+            document.body.appendChild(exbox);
 
+            var s = mknod('style');
+            s.innerHTML = '#exbox{background:#333;color:#ddd;font-family:sans-serif;font-size:0.8em;padding:0 1em 1em 1em;z-index:80386;position:fixed;top:0;left:0;right:0;bottom:0;width:100%;height:100%} #exbox h1{margin:.5em 1em 0 0;padding:0} #exbox h3{border-top:1px solid #999;margin:1em 0 0 0} #exbox a{text-decoration:underline;color:#fc0} #exbox code{color:#bf7;background:#222;padding:.1em;margin:.2em;font-size:1.1em;font-family:monospace,monospace} #exbox *{line-height:1.5em}';
+            document.head.appendChild(s);
+        }
+        exbox.innerHTML = html.join('\n');
+        exbox.style.display = 'block';
+    }
+    catch (e) {
+        document.body.innerHTML = html.join('\n');
+    }
     throw 'fatal_err';
 }
-
-
-var ebi = document.getElementById.bind(document),
-    QS = document.querySelector.bind(document),
-    QSA = document.querySelectorAll.bind(document),
-    mknod = document.createElement.bind(document);
+function ignex(all) {
+    var o = ebi('exbox');
+    o.style.display = 'none';
+    o.innerHTML = '';
+    crashed = false;
+    if (!all)
+        window.onerror = vis_exh;
+}
 
 
 function ctrl(e) {
@@ -97,6 +123,15 @@ if (!String.startsWith) {
         return this.substring(i, i + s.length) === s;
     };
 }
+if (!Element.prototype.closest) {
+    Element.prototype.closest = function (s) {
+        var el = this;
+        do {
+            if (el.msMatchesSelector(s)) return el;
+            el = el.parentElement || el.parentNode;
+        } while (el !== null && el.nodeType === 1);
+    }
+}
 
 
 // https://stackoverflow.com/a/950146
@@ -326,6 +361,18 @@ function linksplit(rp) {
 }
 
 
+function vsplit(vp) {
+    if (vp.endsWith('/'))
+        vp = vp.slice(0, -1);
+
+    var ofs = vp.lastIndexOf('/') + 1,
+        base = vp.slice(0, ofs),
+        fn = vp.slice(ofs);
+
+    return [base, fn];
+}
+
+
 function uricom_enc(txt, do_fb_enc) {
     try {
         return encodeURIComponent(txt);
@@ -412,19 +459,14 @@ function jcp(obj) {
 
 
 function sread(key) {
-    if (window.localStorage)
-        return localStorage.getItem(key);
-
-    return null;
+    return localStorage.getItem(key);
 }
 
 function swrite(key, val) {
-    if (window.localStorage) {
-        if (val === undefined || val === null)
-            localStorage.removeItem(key);
-        else
-            localStorage.setItem(key, val);
-    }
+    if (val === undefined || val === null)
+        localStorage.removeItem(key);
+    else
+        localStorage.setItem(key, val);
 }
 
 function jread(key, fb) {
@@ -603,3 +645,54 @@ var tt = (function () {
 
     return r;
 })();
+
+
+var toast = (function () {
+    var r = {},
+        te = null,
+        visible = false,
+        obj = mknod('div');
+
+    obj.setAttribute('id', 'toast');
+    document.body.appendChild(obj);;
+
+    r.hide = function (e) {
+        ev(e);
+        clearTimeout(te);
+        clmod(obj, 'vis');
+        r.visible = false;
+    };
+
+    r.show = function (cl, ms, txt) {
+        clearTimeout(te);
+        if (ms)
+            te = setTimeout(r.hide, ms * 1000);
+
+        var html = '', hp = txt.split(/(?=<.?pre>)/i);
+        for (var a = 0; a < hp.length; a++)
+            html += hp[a].startsWith('<pre>') ? hp[a] :
+                hp[a].replace(/<br ?.?>\n/g, '\n').replace(/\n<br ?.?>/g, '\n').replace(/\n/g, '<br />\n');
+
+        obj.innerHTML = '<a href="#" id="toastc">x</a>' + html;
+        obj.className = cl;
+        ms += obj.offsetWidth;
+        obj.className += ' vis';
+        ebi('toastc').onclick = r.hide;
+        r.visible = true;
+    };
+
+    r.ok = function (ms, txt) {
+        r.show('ok', ms, txt);
+    };
+    r.inf = function (ms, txt) {
+        r.show('inf', ms, txt);
+    };
+    r.warn = function (ms, txt) {
+        r.show('warn', ms, txt);
+    };
+    r.err = function (ms, txt) {
+        r.show('err', ms, txt);
+    };
+
+    return r;
+})();

docs/example.conf

@@ -10,19 +10,25 @@ u k:k
 # share "." (the current directory)
 # as "/" (the webroot) for the following users:
 # "r" grants read-access for anyone
-# "a ed" grants read-write to ed
+# "rw ed" grants read-write to ed
 .
 /
 r
-a ed
+rw ed
 
 # custom permissions for the "priv" folder:
-# user "k" can see/read the contents
-# and "ed" gets read-write access
+# user "k" can only see/read the contents
+# user "ed" gets read-write access
 ./priv
 /priv
 r k
-a ed
+rw ed
+
+# this does the same thing:
+./priv
+/priv
+r ed k
+w ed
 
 # share /home/ed/Music/ as /music and let anyone read it
 # (this will replace any folder called "music" in the webroot)
@@ -41,5 +47,5 @@ c e2d
 c nodupe
 
 # this entire config file can be replaced with these arguments:
-# -u ed:123 -u k:k -v .::r:aed -v priv:priv:rk:aed -v /home/ed/Music:music:r -v /home/ed/inc:dump:w
+# -u ed:123 -u k:k -v .::r:a,ed -v priv:priv:r,k:rw,ed -v /home/ed/Music:music:r -v /home/ed/inc:dump:w:c,e2d:c,nodupe
 # but note that the config file always wins in case of conflicts

docs/notes.sh

@@ -44,7 +44,7 @@ avg() { awk 'function pr(ncsz) {if (nsmp>0) {printf "%3s %s\n", csz, sum/nsmp} c
 dirs=("$HOME/vfs/ほげ" "$HOME/vfs/ほげ/ぴよ" "$HOME/vfs/$(printf \\xed\\x91)" "$HOME/vfs/$(printf \\xed\\x91/\\xed\\x92)")
 mkdir -p "${dirs[@]}"
 for dir in "${dirs[@]}"; do for fn in ふが "$(printf \\xed\\x93)" 'qwe,rty;asd fgh+jkl%zxc&vbn <qwe>"rty'"'"'uio&asd&nbsp;fgh'; do echo "$dir" > "$dir/$fn.html"; done; done
-
+# qw er+ty%20ui%%20op<as>df&gh&amp;jk#zx'cv"bn`m=qw*er^ty?ui@op,as.df-gh_jk
 
 ##
 ## upload mojibake

tests/test_httpcli.py

@@ -31,7 +31,11 @@ class Cfg(Namespace):
             rproxy=0,
             ed=False,
             nw=False,
+            unpost=600,
+            no_mv=False,
+            no_del=False,
             no_zip=False,
+            no_voldump=True,
             no_scandir=False,
             no_sendfile=True,
             no_rescan=True,
@@ -90,7 +94,7 @@ class TestHttpCli(unittest.TestCase):
                 if not vol.startswith(top):
                     continue
 
-                mode = vol[-2]
+                mode = vol[-2].replace("a", "rwmd")
                 usr = vol[-1]
                 if usr == "a":
                     usr = ""
@@ -99,7 +103,7 @@ class TestHttpCli(unittest.TestCase):
                     vol += "/"
 
                 top, sub = vol.split("/", 1)
-                vcfg.append("{0}/{1}:{1}:{2}{3}".format(top, sub, mode, usr))
+                vcfg.append("{0}/{1}:{1}:{2},{3}".format(top, sub, mode, usr))
 
             pprint.pprint(vcfg)
 

tests/test_vfs.py

@@ -24,6 +24,7 @@ class Cfg(Namespace):
             "hist": None,
             "no_hash": False,
             "css_browser": None,
+            "no_voldump": True,
             "rproxy": 0,
         }
         ex.update(ex2)
@@ -57,8 +58,8 @@ class TestVFS(unittest.TestCase):
         # type: (VFS, str, str) -> tuple[str, str, str]
         """helper for resolving and listing a folder"""
         vn, rem = vfs.get(vpath, uname, True, False)
-        r1 = vn.ls(rem, uname, False)
-        r2 = vn.ls(rem, uname, False)
+        r1 = vn.ls(rem, uname, False, [[True]])
+        r2 = vn.ls(rem, uname, False, [[True]])
         self.assertEqual(r1, r2)
 
         fsdir, real, virt = r1
@@ -68,6 +69,11 @@ class TestVFS(unittest.TestCase):
     def log(self, src, msg, c=0):
         pass
 
+    def assertAxs(self, dct, lst):
+        t1 = list(sorted(dct.keys()))
+        t2 = list(sorted(lst))
+        self.assertEqual(t1, t2)
+
     def test(self):
         td = os.path.join(self.td, "vfs")
         os.mkdir(td)
@@ -88,53 +94,53 @@ class TestVFS(unittest.TestCase):
         self.assertEqual(vfs.nodes, {})
         self.assertEqual(vfs.vpath, "")
         self.assertEqual(vfs.realpath, td)
-        self.assertEqual(vfs.uread, ["*"])
-        self.assertEqual(vfs.uwrite, ["*"])
+        self.assertAxs(vfs.axs.uread, ["*"])
+        self.assertAxs(vfs.axs.uwrite, ["*"])
 
         # single read-only rootfs (relative path)
         vfs = AuthSrv(Cfg(v=["a/ab/::r"]), self.log).vfs
         self.assertEqual(vfs.nodes, {})
         self.assertEqual(vfs.vpath, "")
         self.assertEqual(vfs.realpath, os.path.join(td, "a", "ab"))
-        self.assertEqual(vfs.uread, ["*"])
-        self.assertEqual(vfs.uwrite, [])
+        self.assertAxs(vfs.axs.uread, ["*"])
+        self.assertAxs(vfs.axs.uwrite, [])
 
         # single read-only rootfs (absolute path)
         vfs = AuthSrv(Cfg(v=[td + "//a/ac/../aa//::r"]), self.log).vfs
         self.assertEqual(vfs.nodes, {})
         self.assertEqual(vfs.vpath, "")
         self.assertEqual(vfs.realpath, os.path.join(td, "a", "aa"))
-        self.assertEqual(vfs.uread, ["*"])
-        self.assertEqual(vfs.uwrite, [])
+        self.assertAxs(vfs.axs.uread, ["*"])
+        self.assertAxs(vfs.axs.uwrite, [])
 
         # read-only rootfs with write-only subdirectory (read-write for k)
         vfs = AuthSrv(
-            Cfg(a=["k:k"], v=[".::r:ak", "a/ac/acb:a/ac/acb:w:ak"]),
+            Cfg(a=["k:k"], v=[".::r:rw,k", "a/ac/acb:a/ac/acb:w:rw,k"]),
             self.log,
         ).vfs
         self.assertEqual(len(vfs.nodes), 1)
         self.assertEqual(vfs.vpath, "")
         self.assertEqual(vfs.realpath, td)
-        self.assertEqual(vfs.uread, ["*", "k"])
-        self.assertEqual(vfs.uwrite, ["k"])
+        self.assertAxs(vfs.axs.uread, ["*", "k"])
+        self.assertAxs(vfs.axs.uwrite, ["k"])
         n = vfs.nodes["a"]
         self.assertEqual(len(vfs.nodes), 1)
         self.assertEqual(n.vpath, "a")
         self.assertEqual(n.realpath, os.path.join(td, "a"))
-        self.assertEqual(n.uread, ["*", "k"])
-        self.assertEqual(n.uwrite, ["k"])
+        self.assertAxs(n.axs.uread, ["*", "k"])
+        self.assertAxs(n.axs.uwrite, ["k"])
         n = n.nodes["ac"]
         self.assertEqual(len(vfs.nodes), 1)
         self.assertEqual(n.vpath, "a/ac")
         self.assertEqual(n.realpath, os.path.join(td, "a", "ac"))
-        self.assertEqual(n.uread, ["*", "k"])
-        self.assertEqual(n.uwrite, ["k"])
+        self.assertAxs(n.axs.uread, ["*", "k"])
+        self.assertAxs(n.axs.uwrite, ["k"])
         n = n.nodes["acb"]
         self.assertEqual(n.nodes, {})
         self.assertEqual(n.vpath, "a/ac/acb")
         self.assertEqual(n.realpath, os.path.join(td, "a", "ac", "acb"))
-        self.assertEqual(n.uread, ["k"])
-        self.assertEqual(n.uwrite, ["*", "k"])
+        self.assertAxs(n.axs.uread, ["k"])
+        self.assertAxs(n.axs.uwrite, ["*", "k"])
 
         # something funky about the windows path normalization,
         # doesn't really matter but makes the test messy, TODO?
@@ -173,24 +179,24 @@ class TestVFS(unittest.TestCase):
 
         # admin-only rootfs with all-read-only subfolder
         vfs = AuthSrv(
-            Cfg(a=["k:k"], v=[".::ak", "a:a:r"]),
+            Cfg(a=["k:k"], v=[".::rw,k", "a:a:r"]),
             self.log,
         ).vfs
         self.assertEqual(len(vfs.nodes), 1)
         self.assertEqual(vfs.vpath, "")
         self.assertEqual(vfs.realpath, td)
-        self.assertEqual(vfs.uread, ["k"])
-        self.assertEqual(vfs.uwrite, ["k"])
+        self.assertAxs(vfs.axs.uread, ["k"])
+        self.assertAxs(vfs.axs.uwrite, ["k"])
         n = vfs.nodes["a"]
         self.assertEqual(len(vfs.nodes), 1)
         self.assertEqual(n.vpath, "a")
         self.assertEqual(n.realpath, os.path.join(td, "a"))
-        self.assertEqual(n.uread, ["*"])
-        self.assertEqual(n.uwrite, [])
-        self.assertEqual(vfs.can_access("/", "*"), [False, False])
-        self.assertEqual(vfs.can_access("/", "k"), [True, True])
-        self.assertEqual(vfs.can_access("/a", "*"), [True, False])
-        self.assertEqual(vfs.can_access("/a", "k"), [True, False])
+        self.assertAxs(n.axs.uread, ["*"])
+        self.assertAxs(n.axs.uwrite, [])
+        self.assertEqual(vfs.can_access("/", "*"), [False, False, False, False])
+        self.assertEqual(vfs.can_access("/", "k"), [True, True, False, False])
+        self.assertEqual(vfs.can_access("/a", "*"), [True, False, False, False])
+        self.assertEqual(vfs.can_access("/a", "k"), [True, False, False, False])
 
         # breadth-first construction
         vfs = AuthSrv(
@@ -247,26 +253,26 @@ class TestVFS(unittest.TestCase):
                     ./src
                     /dst
                     r a
-                    a asd
+                    rw asd
                     """
                 ).encode("utf-8")
             )
 
         au = AuthSrv(Cfg(c=[cfg_path]), self.log)
-        self.assertEqual(au.user["a"], "123")
-        self.assertEqual(au.user["asd"], "fgh:jkl")
+        self.assertEqual(au.acct["a"], "123")
+        self.assertEqual(au.acct["asd"], "fgh:jkl")
         n = au.vfs
         # root was not defined, so PWD with no access to anyone
         self.assertEqual(n.vpath, "")
         self.assertEqual(n.realpath, None)
-        self.assertEqual(n.uread, [])
-        self.assertEqual(n.uwrite, [])
+        self.assertAxs(n.axs.uread, [])
+        self.assertAxs(n.axs.uwrite, [])
         self.assertEqual(len(n.nodes), 1)
         n = n.nodes["dst"]
         self.assertEqual(n.vpath, "dst")
         self.assertEqual(n.realpath, os.path.join(td, "src"))
-        self.assertEqual(n.uread, ["a", "asd"])
-        self.assertEqual(n.uwrite, ["asd"])
+        self.assertAxs(n.axs.uread, ["a", "asd"])
+        self.assertAxs(n.axs.uwrite, ["asd"])
         self.assertEqual(len(n.nodes), 0)
 
         os.unlink(cfg_path)

tests/util.py

@@ -31,7 +31,7 @@ if MACOS:
 from copyparty.util import Unrecv
 
 
-def runcmd(*argv):
+def runcmd(argv):
     p = sp.Popen(argv, stdout=sp.PIPE, stderr=sp.PIPE)
     stdout, stderr = p.communicate()
     stdout = stdout.decode("utf-8")
@@ -39,8 +39,8 @@ def runcmd(*argv):
     return [p.returncode, stdout, stderr]
 
 
-def chkcmd(*argv):
-    ok, sout, serr = runcmd(*argv)
+def chkcmd(argv):
+    ok, sout, serr = runcmd(argv)
     if ok != 0:
         raise Exception(serr)
 
@@ -60,12 +60,12 @@ def get_ramdisk():
 
     if os.path.exists("/Volumes"):
         # hdiutil eject /Volumes/cptd/
-        devname, _ = chkcmd("hdiutil", "attach", "-nomount", "ram://131072")
+        devname, _ = chkcmd("hdiutil attach -nomount ram://131072".split())
         devname = devname.strip()
         print("devname: [{}]".format(devname))
         for _ in range(10):
             try:
-                _, _ = chkcmd("diskutil", "eraseVolume", "HFS+", "cptd", devname)
+                _, _ = chkcmd(["diskutil", "eraseVolume", "HFS+", "cptd", devname])
                 with open("/Volumes/cptd/.metadata_never_index", "w") as f:
                     f.write("orz")