v0.12.8

* extract codec and format info
* add FFprobe as fallback when mutagen fails
* add option to blacklist FFprobe for tags

.vscode/tasks.json

@@ -9,7 +9,10 @@
         {
             "label": "no_dbg",
             "type": "shell",
-            "command": "${config:python.pythonPath} .vscode/launch.py"
+            "command": "${config:python.pythonPath}",
+            "args": [
+                ".vscode/launch.py"
+            ]
         }
     ]
 }

README.md

@@ -16,6 +16,11 @@ turn your phone or raspi into a portable file server with resumable uploads/down
 📷 **screenshots:** [browser](#the-browser) // [upload](#uploading) // [thumbnails](#thumbnails) // [md-viewer](#markdown-viewer) // [search](#searching) // [fsearch](#file-search) // [zip-DL](#zip-downloads) // [ie4](#browser-support)
 
 
+## breaking changes \o/
+
+this is the readme for v0.12 which has a different expression for volume permissions (`-v`); see [the v0.11.x readme](https://github.com/9001/copyparty/tree/15b59822112dda56cee576df30f331252fc62628#readme) for stuff regarding the [current stable release](https://github.com/9001/copyparty/releases/tag/v0.11.47)
+
+
 ## readme toc
 
 * top
@@ -30,11 +35,12 @@ turn your phone or raspi into a portable file server with resumable uploads/down
 * [the browser](#the-browser)
     * [tabs](#tabs)
     * [hotkeys](#hotkeys)
-    * [tree-mode](#tree-mode)
+    * [navpane](#navpane)
     * [thumbnails](#thumbnails)
     * [zip downloads](#zip-downloads)
     * [uploading](#uploading)
         * [file-search](#file-search)
+    * [file manager](#file-manager)
     * [markdown viewer](#markdown-viewer)
     * [other tricks](#other-tricks)
 * [searching](#searching)
@@ -59,21 +65,21 @@ turn your phone or raspi into a portable file server with resumable uploads/down
     * [just the sfx](#just-the-sfx)
     * [complete release](#complete-release)
 * [todo](#todo)
+    * [discarded ideas](#discarded-ideas)
 
 
 ## quickstart
 
 download [copyparty-sfx.py](https://github.com/9001/copyparty/releases/latest/download/copyparty-sfx.py) and you're all set!
 
-running the sfx without arguments (for example doubleclicking it on Windows) will give everyone full access to the current folder; see `-h` for help if you want accounts and volumes etc
+running the sfx without arguments (for example doubleclicking it on Windows) will give everyone full access to the current folder; see `-h` for help if you want [accounts and volumes](#accounts-and-volumes) etc
 
 some recommended options:
 * `-e2dsa` enables general file indexing, see [search configuration](#search-configuration)
-* `-e2ts` enables audio metadata indexing (needs either FFprobe or mutagen), see [optional dependencies](#optional-dependencies)
+* `-e2ts` enables audio metadata indexing (needs either FFprobe or Mutagen), see [optional dependencies](#optional-dependencies)
-* `-v /mnt/music:/music:r:afoo -a foo:bar` shares `/mnt/music` as `/music`, `r`eadable by anyone, with user `foo` as `a`dmin (read/write), password `bar`
+* `-v /mnt/music:/music:r:rw,foo -a foo:bar` shares `/mnt/music` as `/music`, `r`eadable by anyone, and read-write for user `foo`, password `bar`
-  * the syntax is `-v src:dst:perm:perm:...` so local-path, url-path, and one or more permissions to set
+  * replace `:r:rw,foo` with `:r,foo` to only make the folder readable by `foo` and nobody else
-  * replace `:r:afoo` with `:rfoo` to only make the folder readable by `foo` and nobody else
+  * see [accounts and volumes](#accounts-and-volumes) for the syntax and other access levels (`r`ead, `w`rite, `m`ove, `d`elete)
-  * in addition to `r`ead and `a`dmin, `w`rite makes a folder write-only, so cannot list/access files in it
 * `--ls '**,*,ln,p,r'` to crash on startup if any of the volumes contain a symlink which point outside the volume, as that could give users unintended access
 
 you may also want these, especially on servers:
@@ -114,28 +120,29 @@ summary: all planned features work! now please enjoy the bloatening
 
 * backend stuff
   * ☑ sanic multipart parser
-  * ☑ load balancer (multiprocessing)
+  * ☑ multiprocessing (actual multithreading)
   * ☑ volumes (mountpoints)
-  * ☑ accounts
+  * ☑ [accounts](#accounts-and-volumes)
 * upload
   * ☑ basic: plain multipart, ie6 support
   * ☑ up2k: js, resumable, multithreaded
   * ☑ stash: simple PUT filedropper
+  * ☑ unpost: undo/delete accidental uploads
   * ☑ symlink/discard existing files (content-matching)
 * download
   * ☑ single files in browser
   * ☑ folders as zip / tar files
   * ☑ FUSE client (read-only)
 * browser
-  * ☑ tree-view
+  * ☑ navpane (directory tree sidebar)
   * ☑ audio player (with OS media controls)
   * ☑ thumbnails
-    * ☑ images using Pillow
+    * ☑ ...of images using Pillow
-    * ☑ videos using FFmpeg
+    * ☑ ...of videos using FFmpeg
     * ☑ cache eviction (max-age; maybe max-size eventually)
-  * ☑ image gallery
+  * ☑ image gallery with webm player
   * ☑ SPA (browse while uploading)
-    * if you use the file-tree on the left only, not folders in the file list
+    * if you use the navpane to navigate, not folders in the file list
 * server indexing
   * ☑ locate files by contents
   * ☑ search by name/path/date/size
@@ -154,17 +161,15 @@ small collection of user feedback
 
 # bugs
 
-* Windows: python 3.7 and older cannot read tags with ffprobe, so use mutagen or upgrade
+* Windows: python 3.7 and older cannot read tags with FFprobe, so use Mutagen or upgrade
 * Windows: python 2.7 cannot index non-ascii filenames with `-e2d`
 * Windows: python 2.7 cannot handle filenames with mojibake
-* MacOS: `--th-ff-jpg` may fix thumbnails using macports-FFmpeg
+* `--th-ff-jpg` may fix video thumbnails on some FFmpeg versions
 
 ## general bugs
 
 * all volumes must exist / be available on startup; up2k (mtp especially) gets funky otherwise
 * cannot mount something at `/d1/d2/d3` unless `d2` exists inside `d1`
-* dupe files will not have metadata (audio tags etc) displayed in the file listing
-  * because they don't get `up` entries in the db (probably best fix) and `tx_browser` does not `lstat`
 * probably more, pls let me know
 
 ## not my bugs
@@ -172,9 +177,36 @@ small collection of user feedback
 * Windows: folders cannot be accessed if the name ends with `.`
   * python or windows bug
 
-* Windows: msys2-python 3.8.6 occasionally throws "RuntimeError: release unlocked lock" when leaving a scoped mutex in up2k
+* Windows: msys2-python 3.8.6 occasionally throws `RuntimeError: release unlocked lock` when leaving a scoped mutex in up2k
   * this is an msys2 bug, the regular windows edition of python is fine
 
+* VirtualBox: sqlite throws `Disk I/O Error` when running in a VM and the up2k database is in a vboxsf
+  * use `--hist` or the `hist` volflag (`-v [...]:chist=/tmp/foo`) to place the db inside the vm instead
+
+
+# accounts and volumes
+
+* `-a usr:pwd` adds account `usr` with password `pwd`
+* `-v .::r` adds current-folder `.` as the webroot, `r`eadable by anyone
+  * the syntax is `-v src:dst:perm:perm:...` so local-path, url-path, and one or more permissions to set
+  * when granting permissions to an account, the names are comma-separated: `-v .::r,usr1,usr2:rw,usr3,usr4`
+
+permissions:
+* `r` (read): browse folder contents, download files, download as zip/tar
+* `w` (write): upload files, move files *into* folder
+* `m` (move): move files/folders *from* folder
+* `d` (delete): delete files/folders
+
+example:
+* add accounts named u1, u2, u3 with passwords p1, p2, p3: `-a u1:p1 -a u2:p2 -a u3:p3`
+* make folder `/srv` the root of the filesystem, read-only by anyone: `-v /srv::r`
+* make folder `/mnt/music` available at `/music`, read-only for u1 and u2, read-write for u3: `-v /mnt/music:music:r,u1,u2:rw,u3`
+  * unauthorized users accessing the webroot can see that the `music` folder exists, but cannot open it
+* make folder `/mnt/incoming` available at `/inc`, write-only for u1, read-move for u2: `-v /mnt/incoming:inc:w,u1:rm,u2`
+  * unauthorized users accessing the webroot can see that the `inc` folder exists, but cannot open it
+  * `u1` can open the `inc` folder, but cannot see the contents, only upload new files to it
+  * `u2` can browse it and move files *from* `/inc` into any folder where `u2` has write-access
+
 
 # the browser
 
@@ -184,21 +216,32 @@ small collection of user feedback
 ## tabs
 
 * `[🔎]` search by size, date, path/name, mp3-tags ... see [searching](#searching)
+* `[🧯]` unpost: undo/delete accidental uploads
 * `[🚀]` and `[🎈]` are the uploaders, see [uploading](#uploading)
-* `[📂]` mkdir, create directories
+* `[📂]` mkdir: create directories
-* `[📝]` new-md, create a new markdown document
+* `[📝]` new-md: create a new markdown document
-* `[📟]` send-msg, either to server-log or into textfiles if `--urlform save`
+* `[📟]` send-msg: either to server-log or into textfiles if `--urlform save`
-* `[⚙️]` client configuration options
+* `[🎺]` audio-player config options
+* `[⚙️]` general client config options
 
 
 ## hotkeys
 
-the browser has the following hotkeys
+the browser has the following hotkeys (assumes qwerty, ignores actual layout)
-* `B` toggle breadcrumbs / directory tree
+* `B` toggle breadcrumbs / navpane
 * `I/K` prev/next folder
-* `M` parent folder
+* `M` parent folder (or unexpand current)
 * `G` toggle list / grid view
 * `T` toggle thumbnails / icons
+* `ctrl-X` cut selected files/folders
+* `ctrl-V` paste
+* `F2` rename selected file/folder
+* when a file/folder is selected (in not-grid-view):
+  * `Up/Down` move cursor
+  * shift+`Up/Down` select and move cursor
+  * ctrl+`Up/Down` move cursor and scroll viewport
+  * `Space` toggle file selection
+  * `Ctrl-A` toggle select all
 * when playing audio:
   * `J/L` prev/next song
   * `U/O` skip 10sec back/forward
@@ -215,16 +258,23 @@ the browser has the following hotkeys
     * `C` continue playing next video
     * `R` loop
     * `M` mute
-* when tree-sidebar is open:
+* when the navpane is open:
   * `A/D` adjust tree width
 * in the grid view:
   * `S` toggle multiselect
   * shift+`A/D` zoom
+* in the markdown editor:
+  * `^s` save
+  * `^h` header
+  * `^k` autoformat table
+  * `^u` jump to next unicode character
+  * `^e` toggle editor / preview
+  * `^up, ^down` jump paragraphs
 
 
-## tree-mode
+## navpane
 
-by default there's a breadcrumbs path; you can replace this with a tree-browser sidebar thing by clicking the `🌲` or pressing the `B` hotkey
+by default there's a breadcrumbs path; you can replace this with a navpane (tree-browser sidebar thing) by clicking the `🌲` or pressing the `B` hotkey
 
 click `[-]` and `[+]` (or hotkeys `A`/`D`) to adjust the size, and the `[a]` toggles if the tree should widen dynamically as you go deeper or stay fixed-size
 
@@ -233,7 +283,7 @@ click `[-]` and `[+]` (or hotkeys `A`/`D`) to adjust the size, and the `[a]` tog
 
 ![copyparty-thumbs-fs8](https://user-images.githubusercontent.com/241032/120070302-10836b00-c08a-11eb-8eb4-82004a34c342.png)
 
-it does static images with Pillow and uses FFmpeg for video files, so you may want to `--no-thumb` or maybe just `--no-vthumb` depending on how destructive your users are
+it does static images with Pillow and uses FFmpeg for video files, so you may want to `--no-thumb` or maybe just `--no-vthumb` depending on how dangerous your users are
 
 images with the following names (see `--th-covers`) become the thumbnail of the folder they're in: `folder.png`, `folder.jpg`, `cover.png`, `cover.jpg`
 
@@ -252,9 +302,10 @@ the `zip` link next to folders can produce various types of zip/tar files using
 | `zip_crc` | `?zip=crc` | cp437 with crc32 computed early for truly ancient software |
 
 * hidden files (dotfiles) are excluded unless `-ed`
-  * the up2k.db is always excluded
+  * `up2k.db` and `dir.txt` is always excluded
 * `zip_crc` will take longer to download since the server has to read each file twice
-  * please let me know if you find a program old enough to actually need this
+  * this is only to support MS-DOS PKZIP v2.04g (october 1993) and older
+    * how are you accessing copyparty actually
 
 you can also zip a selection of files or folders by clicking them in the browser, that brings up a selection editor and zip button in the bottom right
 
@@ -263,15 +314,19 @@ you can also zip a selection of files or folders by clicking them in the browser
 ## uploading
 
 two upload methods are available in the html client:
-* `🎈 bup`, the basic uploader, supports almost every browser since netscape 4.0
+* `[🎈] bup`, the basic uploader, supports almost every browser since netscape 4.0
-* `🚀 up2k`, the fancy one
+* `[🚀] up2k`, the fancy one
+
+you can undo/delete uploads using `[🧯] unpost` if the server is running with `-e2d`
 
 up2k has several advantages:
 * you can drop folders into the browser (files are added recursively)
 * files are processed in chunks, and each chunk is checksummed
-  * uploads resume if they are interrupted (for example by a reboot)
+  * uploads autoresume if they are interrupted by network issues
+  * uploads resume if you reboot your browser or pc, just upload the same files again
   * server detects any corruption; the client reuploads affected chunks
   * the client doesn't upload anything that already exists on the server
+* much higher speeds than ftp/scp/tarpipe on some internet connections (mainly american ones) thanks to parallel connections
 * the last-modified timestamp of the file is preserved
 
 see [up2k](#up2k) for details on how it works
@@ -304,11 +359,18 @@ in the `[🚀 up2k]` tab, after toggling the `[🔎]` switch green, any files/fo
 files go into `[ok]` if they exist (and you get a link to where it is), otherwise they land in `[ng]`
 * the main reason filesearch is combined with the uploader is cause the code was too spaghetti to separate it out somewhere else, this is no longer the case but now i've warmed up to the idea too much
 
-adding the same file multiple times is blocked, so if you first search for a file and then decide to upload it, you have to click the `[cleanup]` button to discard `[done]` files
+adding the same file multiple times is blocked, so if you first search for a file and then decide to upload it, you have to click the `[cleanup]` button to discard `[done]` files (or just refresh the page)
 
 note that since up2k has to read the file twice, `[🎈 bup]` can be up to 2x faster in extreme cases (if your internet connection is faster than the read-speed of your HDD)
 
-up2k has saved a few uploads from becoming corrupted in-transfer already; caught an android phone on wifi redhanded in wireshark with a bitflip, however bup with https would *probably* have noticed as well thanks to tls also functioning as an integrity check
+up2k has saved a few uploads from becoming corrupted in-transfer already; caught an android phone on wifi redhanded in wireshark with a bitflip, however bup with https would *probably* have noticed as well (thanks to tls also functioning as an integrity check)
+
+
+## file manager
+
+if you have the required permissions, you can cut/paste, rename, and delete files/folders
+
+you can move files across browser tabs (cut in one tab, paste in another)
 
 
 ## markdown viewer
@@ -346,11 +408,11 @@ searching relies on two databases, the up2k filetree (`-e2d`) and the metadata t
 
 through arguments:
 * `-e2d` enables file indexing on upload
-* `-e2ds` scans writable folders on startup
+* `-e2ds` scans writable folders for new files on startup
 * `-e2dsa` scans all mounted volumes (including readonly ones)
 * `-e2t` enables metadata indexing on upload
 * `-e2ts` scans for tags in all files that don't have tags yet
-* `-e2tsr` deletes all existing tags, so a full reindex
+* `-e2tsr` deletes all existing tags, does a full reindex
 
 the same arguments can be set as volume flags, in addition to `d2d` and `d2t` for disabling:
 * `-v ~/music::r:ce2dsa:ce2tsr` does a full reindex of everything on startup
@@ -358,11 +420,11 @@ the same arguments can be set as volume flags, in addition to `d2d` and `d2t` fo
 * `-v ~/music::r:cd2t` disables all `-e2t*` (tags), does not affect `-e2d*`
 
 note:
-* `e2tsr` is probably always overkill, since `e2ds`/`e2dsa` would pick up any file modifications and `e2ts` would then reindex those
+* `e2tsr` is probably always overkill, since `e2ds`/`e2dsa` would pick up any file modifications and `e2ts` would then reindex those, unless there is a new copyparty version with new parsers and the release note says otherwise
 * the rescan button in the admin panel has no effect unless the volume has `-e2ds` or higher
 
 you can choose to only index filename/path/size/last-modified (and not the hash of the file contents) by setting `--no-hash` or the volume-flag `cdhash`, this has the following consequences:
-* initial indexing is way faster, especially when the volume is on a networked disk
+* initial indexing is way faster, especially when the volume is on a network disk
 * makes it impossible to [file-search](#file-search)
 * if someone uploads the same file contents, the upload will not be detected as a dupe, so it will not get symlinked or rejected
 
@@ -395,17 +457,17 @@ tags that start with a `.` such as `.bpm` and `.dur`(ation) indicate numeric val
 
 see the beautiful mess of a dictionary in [mtag.py](https://github.com/9001/copyparty/blob/master/copyparty/mtag.py) for the default mappings (should cover mp3,opus,flac,m4a,wav,aif,)
 
-`--no-mutagen` disables mutagen and uses ffprobe instead, which...
+`--no-mutagen` disables Mutagen and uses FFprobe instead, which...
-* is about 20x slower than mutagen
+* is about 20x slower than Mutagen
-* catches a few tags that mutagen doesn't
+* catches a few tags that Mutagen doesn't
   * melodic key, video resolution, framerate, pixfmt
 * avoids pulling any GPL code into copyparty
-* more importantly runs ffprobe on incoming files which is bad if your ffmpeg has a cve
+* more importantly runs FFprobe on incoming files which is bad if your FFmpeg has a cve
 
 
 ## file parser plugins
 
-copyparty can invoke external programs to collect additional metadata for files using `mtp` (as argument or volume flag), there is a default timeout of 30sec
+copyparty can invoke external programs to collect additional metadata for files using `mtp` (either as argument or volume flag), there is a default timeout of 30sec
 
 * `-mtp .bpm=~/bin/audio-bpm.py` will execute `~/bin/audio-bpm.py` with the audio file as argument 1 to provide the `.bpm` tag, if that does not exist in the audio metadata
 * `-mtp key=f,t5,~/bin/audio-key.py` uses `~/bin/audio-key.py` to get the `key` tag, replacing any existing metadata tag (`f,`), aborting if it takes longer than 5sec (`t5,`)
@@ -437,13 +499,15 @@ copyparty can invoke external programs to collect additional metadata for files
 | send message    | yep | yep | yep  | yep  | yep   | yep  | yep | yep  |
 | set sort order  |  -  | yep | yep  | yep  | yep   | yep  | yep | yep  |
 | zip selection   |  -  | yep | yep  | yep  | yep   | yep  | yep | yep  |
-| directory tree  |  -  |  -  | `*1` | yep  | yep   | yep  | yep | yep  |
+| navpane         |  -  |  -  | `*1` | yep  | yep   | yep  | yep | yep  |
 | up2k            |  -  |  -  | yep  | yep  | yep   | yep  | yep | yep  |
-| icons work      |  -  |  -  | yep  | yep  | yep   | yep  | yep | yep  |
 | markdown editor |  -  |  -  | yep  | yep  | yep   | yep  | yep | yep  |
 | markdown viewer |  -  |  -  | yep  | yep  | yep   | yep  | yep | yep  |
 | play mp3/m4a    |  -  | yep | yep  | yep  | yep   | yep  | yep | yep  |
 | play ogg/opus   |  -  |  -  |  -   |  -   | yep   | yep  | `*2` | yep |
+| thumbnail view  |  -  |  -  | -    | -    | yep   | yep  | yep | yep  |
+| image viewer    |  -  |  -  | -    | -    | yep   | yep  | yep | yep  |
+| **= feature =** | ie6 | ie9 | ie10 | ie11 | ff 52 | c 49 | iOS | Andr |
 
 * internet explorer 6 to 8 behave the same
 * firefox 52 and chrome 49 are the last winxp versions
@@ -461,7 +525,7 @@ quick summary of more eccentric web-browsers trying to view a directory index:
 | **w3m** (0.5.3/macports)  | can browse, login, upload at 100kB/s, mkdir/msg |
 | **netsurf** (3.10/arch)   | is basically ie6 with much better css (javascript has almost no effect) | 
 | **ie4** and **netscape** 4.0  | can browse (text is yellow on white), upload with `?b=u` |
-| **SerenityOS** (22d13d8)  | hits a page fault, works with `?b=u`, file input not-impl, url params are multiplying |
+| **SerenityOS** (7e98457)  | hits a page fault, works with `?b=u`, file upload not-impl |
 
 
 # client examples
@@ -515,7 +579,7 @@ below are some tweaks roughly ordered by usefulness:
 * `-q` disables logging and can help a bunch, even when combined with `-lo` to redirect logs to file
 * `--http-only` or `--https-only` (unless you want to support both protocols) will reduce the delay before a new connection is established
 * `--hist` pointing to a fast location (ssd) will make directory listings and searches faster when `-e2d` or `-e2t` is set
-* `--no-hash` when indexing a networked disk if you don't care about the actual filehashes and only want the names/tags searchable
+* `--no-hash` when indexing a network-disk if you don't care about the actual filehashes and only want the names/tags searchable
 * `-j` enables multiprocessing (actual multithreading) and can make copyparty perform better in cpu-intensive workloads, for example:
   * huge amount of short-lived connections
   * really heavy traffic (downloads/uploads)
@@ -532,18 +596,18 @@ below are some tweaks roughly ordered by usefulness:
 
 enable music tags:
 * either `mutagen` (fast, pure-python, skips a few tags, makes copyparty GPL? idk)
-* or `FFprobe` (20x slower, more accurate, possibly dangerous depending on your distro and users)
+* or `ffprobe` (20x slower, more accurate, possibly dangerous depending on your distro and users)
 
-enable image thumbnails:
+enable thumbnails of images:
 * `Pillow` (requires py2.7 or py3.5+)
 
-enable video thumbnails:
+enable thumbnails of videos:
 * `ffmpeg` and `ffprobe` somewhere in `$PATH`
 
-enable reading HEIF pictures:
+enable thumbnails of HEIF pictures:
 * `pyheif-pillow-opener` (requires Linux or a C compiler)
 
-enable reading AVIF pictures:
+enable thumbnails of AVIF pictures:
 * `pillow-avif-plugin`
 
 
@@ -557,7 +621,7 @@ python -m pip install --user -U jinja2 mutagen Pillow
 
 some bundled tools have copyleft dependencies, see [./bin/#mtag](bin/#mtag)
 
-these are standalone programs and will never be imported / evaluated by copyparty
+these are standalone programs and will never be imported / evaluated by copyparty, and must be enabled through `-mtp` configs
 
 
 # sfx
@@ -573,10 +637,10 @@ pls note that `copyparty-sfx.sh` will fail if you rename `copyparty-sfx.py` to `
 
 ## sfx repack
 
-if you don't need all the features you can repack the sfx and save a bunch of space; all you need is an sfx and a copy of this repo (nothing else to download or build, except for either msys2 or WSL if you're on windows)
+if you don't need all the features, you can repack the sfx and save a bunch of space; all you need is an sfx and a copy of this repo (nothing else to download or build, except if you're on windows then you need msys2 or WSL)
-* `724K` original size as of v0.4.0
+* `525k` size of original sfx.py as of v0.11.30
-* `256K` after `./scripts/make-sfx.sh re no-ogv`
+* `315k` after `./scripts/make-sfx.sh re no-ogv`
-* `164K` after `./scripts/make-sfx.sh re no-ogv no-cm`
+* `223k` after `./scripts/make-sfx.sh re no-ogv no-cm`
 
 the features you can opt to drop are
 * `ogv`.js, the opus/vorbis decoder which is needed by apple devices to play foss audio files
@@ -623,7 +687,7 @@ rm -rf copyparty/web/deps
 curl -L https://github.com/9001/copyparty/releases/latest/download/copyparty-sfx.py >x.py
 python3 x.py -h
 rm x.py
-mv /tmp/pe-copyparty/copyparty/web/deps/ copyparty/web/
+mv /tmp/pe-copyparty/copyparty/web/deps/ copyparty/web/deps/
 ```
 
 then build the sfx using any of the following examples:
@@ -651,14 +715,16 @@ in the `scripts` folder:
 
 roughly sorted by priority
 
+* hls framework for Someone Else to drop code into :^)
 * readme.md as epilogue
+
+
+## discarded ideas
+
 * reduce up2k roundtrips
   * start from a chunk index and just go
   * terminate client on bad data
-* logging to file
+    * not worth the effort, just throw enough conncetions at it
-
-discarded ideas
-
 * single sha512 across all up2k chunks?
   * crypto.subtle cannot into streaming, would have to use hashwasm, expensive
 * separate sqlite table per tag

contrib/systemd/copyparty.service

@@ -7,11 +7,23 @@
 # you may want to:
 #   change '/usr/bin/python' to another interpreter
 #   change '/mnt::a' to another location or permission-set
+#
+# with `Type=notify`, copyparty will signal systemd when it is ready to
+#   accept connections; correctly delaying units depending on copyparty.
+#   But note that journalctl will get the timestamps wrong due to
+#   python disabling line-buffering, so messages are out-of-order:
+#   https://user-images.githubusercontent.com/241032/126040249-cb535cc7-c599-4931-a796-a5d9af691bad.png
+#
+# enable line-buffering for realtime logging (slight performance cost):
+#   modify ExecStart and prefix it with `/bin/stdbuf -oL` like so:
+#   ExecStart=/bin/stdbuf -oL /usr/bin/python3 [...]
 
 [Unit]
 Description=copyparty file server
 
 [Service]
+Type=notify
+SyslogIdentifier=copyparty
 ExecStart=/usr/bin/python3 /usr/local/bin/copyparty-sfx.py -q -v /mnt::a
 ExecStartPre=/bin/bash -c 'mkdir -p /run/tmpfiles.d/ && echo "x /tmp/pe-copyparty*" > /run/tmpfiles.d/copyparty.conf'
 

copyparty/__main__.py

@@ -199,24 +199,30 @@ def run_argparse(argv, formatter):
         epilog=dedent(
             """
             -a takes username:password,
-            -v takes src:dst:permset:permset:cflag:cflag:...
+            -v takes src:dst:perm1:perm2:permN:cflag1:cflag2:cflagN:...
-               where "permset" is accesslevel followed by username (no separator)
+               where "perm" is "accesslevels,username1,username2,..."
                and "cflag" is config flags to set on this volume
             
+            list of accesslevels:
+              "r" (read):   list folder contents, download files
+              "w" (write):  upload files; need "r" to see the uploads
+              "m" (move):   move files and folders; need "w" at destination
+              "d" (delete): permanently delete files and folders
+
             list of cflags:
-              "cnodupe" rejects existing files (instead of symlinking them)
+              "c,nodupe" rejects existing files (instead of symlinking them)
-              "ce2d" sets -e2d (all -e2* args can be set using ce2* cflags)
+              "c,e2d" sets -e2d (all -e2* args can be set using ce2* cflags)
-              "cd2t" disables metadata collection, overrides -e2t*
+              "c,d2t" disables metadata collection, overrides -e2t*
-              "cd2d" disables all database stuff, overrides -e2*
+              "c,d2d" disables all database stuff, overrides -e2*
 
             example:\033[35m
-              -a ed:hunter2 -v .::r:aed -v ../inc:dump:w:aed:cnodupe  \033[36m
+              -a ed:hunter2 -v .::r:rw,ed -v ../inc:dump:w:rw,ed:c,nodupe  \033[36m
               mount current directory at "/" with
                * r (read-only) for everyone
-               * a (read+write) for ed
+               * rw (read+write) for ed
               mount ../inc at "/dump" with
                * w (write-only) for everyone
-               * a (read+write) for ed
+               * rw (read+write) for ed
                * reject duplicate files  \033[0m
             
             if no accounts or volumes are configured,
@@ -258,9 +264,12 @@ def run_argparse(argv, formatter):
     ap2.add_argument("-ed", action="store_true", help="enable ?dots")
     ap2.add_argument("-emp", action="store_true", help="enable markdown plugins")
     ap2.add_argument("-mcr", metavar="SEC", type=int, default=60, help="md-editor mod-chk rate")
+    ap2.add_argument("--urlform", metavar="MODE", type=u, default="print,get", help="how to handle url-forms; examples: [stash], [save,get]")
+
+    ap2 = ap.add_argument_group('upload options')
     ap2.add_argument("--dotpart", action="store_true", help="dotfile incomplete uploads")
     ap2.add_argument("--sparse", metavar="MiB", type=int, default=4, help="up2k min.size threshold (mswin-only)")
-    ap2.add_argument("--urlform", metavar="MODE", type=u, default="print,get", help="how to handle url-forms; examples: [stash], [save,get]")
+    ap2.add_argument("--unpost", metavar="SEC", type=int, default=3600*12, help="grace period where uploads can be deleted by the uploader, even without delete permissions; 0=disabled")
 
     ap2 = ap.add_argument_group('network options')
     ap2.add_argument("-i", metavar="IP", type=u, default="0.0.0.0", help="ip to bind (comma-sep.)")
@@ -277,6 +286,8 @@ def run_argparse(argv, formatter):
 
     ap2 = ap.add_argument_group('opt-outs')
     ap2.add_argument("-nw", action="store_true", help="disable writes (benchmark)")
+    ap2.add_argument("--no-del", action="store_true", help="disable delete operations")
+    ap2.add_argument("--no-mv", action="store_true", help="disable move/rename operations")
     ap2.add_argument("-nih", action="store_true", help="no info hostname")
     ap2.add_argument("-nid", action="store_true", help="no info disk-usage")
     ap2.add_argument("--no-zip", action="store_true", help="disable download as zip/tar")
@@ -288,6 +299,7 @@ def run_argparse(argv, formatter):
     ap2 = ap.add_argument_group('logging options')
     ap2.add_argument("-q", action="store_true", help="quiet")
     ap2.add_argument("-lo", metavar="PATH", type=u, help="logfile, example: cpp-%%Y-%%m%%d-%%H%%M%%S.txt.xz")
+    ap2.add_argument("--no-voldump", action="store_true", help="do not list volumes and permissions on startup")
     ap2.add_argument("--log-conn", action="store_true", help="print tcp-server msgs")
     ap2.add_argument("--log-htp", action="store_true", help="print http-server threadpool scaling")
     ap2.add_argument("--ihead", metavar="HEADER", type=u, action='append', help="dump incoming header")
@@ -310,22 +322,27 @@ def run_argparse(argv, formatter):
     ap2.add_argument("--th-maxage", metavar="SEC", type=int, default=604800, help="max folder age")
     ap2.add_argument("--th-covers", metavar="N,N", type=u, default="folder.png,folder.jpg,cover.png,cover.jpg", help="folder thumbnails to stat for")
 
-    ap2 = ap.add_argument_group('database options')
+    ap2 = ap.add_argument_group('general db options')
     ap2.add_argument("-e2d", action="store_true", help="enable up2k database")
     ap2.add_argument("-e2ds", action="store_true", help="enable up2k db-scanner, sets -e2d")
     ap2.add_argument("-e2dsa", action="store_true", help="scan all folders (for search), sets -e2ds")
+    ap2.add_argument("--hist", metavar="PATH", type=u, help="where to store volume state")
+    ap2.add_argument("--no-hash", action="store_true", help="disable hashing during e2ds folder scans")
+    ap2.add_argument("--re-int", metavar="SEC", type=int, default=30, help="disk rescan check interval")
+    ap2.add_argument("--re-maxage", metavar="SEC", type=int, default=0, help="disk rescan volume interval (0=off)")
+    ap2.add_argument("--srch-time", metavar="SEC", type=int, default=30, help="search deadline")
+    
+    ap2 = ap.add_argument_group('metadata db options')
     ap2.add_argument("-e2t", action="store_true", help="enable metadata indexing")
     ap2.add_argument("-e2ts", action="store_true", help="enable metadata scanner, sets -e2t")
     ap2.add_argument("-e2tsr", action="store_true", help="rescan all metadata, sets -e2ts")
-    ap2.add_argument("--hist", metavar="PATH", type=u, help="where to store volume state")
+    ap2.add_argument("--no-mutagen", action="store_true", help="use FFprobe for tags instead")
-    ap2.add_argument("--no-hash", action="store_true", help="disable hashing during e2ds folder scans")
-    ap2.add_argument("--no-mutagen", action="store_true", help="use ffprobe for tags instead")
     ap2.add_argument("--no-mtag-mt", action="store_true", help="disable tag-read parallelism")
+    ap2.add_argument("--no-mtag-ff", action="store_true", help="never use FFprobe as tag reader")
     ap2.add_argument("-mtm", metavar="M=t,t,t", type=u, action="append", help="add/replace metadata mapping")
     ap2.add_argument("-mte", metavar="M,M,M", type=u, help="tags to index/display (comma-sep.)",
         default="circle,album,.tn,artist,title,.bpm,key,.dur,.q,.vq,.aq,ac,vc,res,.fps")
     ap2.add_argument("-mtp", metavar="M=[f,]bin", type=u, action="append", help="read tag M using bin")
-    ap2.add_argument("--srch-time", metavar="SEC", type=int, default=30, help="search deadline")
 
     ap2 = ap.add_argument_group('appearance options')
     ap2.add_argument("--css-browser", metavar="L", type=u, help="URL to additional CSS to include")
@@ -376,6 +393,36 @@ def main(argv=None):
     except AssertionError:
         al = run_argparse(argv, Dodge11874)
 
+    nstrs = []
+    anymod = False
+    for ostr in al.v or []:
+        mod = False
+        oa = ostr.split(":")
+        na = oa[:2]
+        for opt in oa[2:]:
+            if re.match("c[^,]", opt):
+                mod = True
+                na.append("c," + opt[1:])
+            elif re.sub("^[rwmd]*", "", opt) and "," not in opt:
+                mod = True
+                perm = opt[0]
+                if perm == "a":
+                    perm = "rw"
+                na.append(perm + "," + opt[1:])
+            else:
+                na.append(opt)
+
+        nstr = ":".join(na)
+        nstrs.append(nstr if mod else ostr)
+        if mod:
+            msg = "\033[1;31mWARNING:\033[0;1m\n  -v {} \033[0;33mwas replaced with\033[0;1m\n  -v {} \n\033[0m"
+            lprint(msg.format(ostr, nstr))
+            anymod = True
+
+    if anymod:
+        al.v = nstrs
+        time.sleep(2)
+
     # propagate implications
     for k1, k2 in IMPLICATIONS:
         if getattr(al, k1):

copyparty/__version__.py

@@ -1,8 +1,8 @@
 # coding: utf-8
 
-VERSION = (0, 11, 40)
+VERSION = (0, 12, 8)
-CODENAME = "the grid"
+CODENAME = "fil\033[33med"
-BUILD_DT = (2021, 7, 15)
+BUILD_DT = (2021, 8, 1)
 
 S_VERSION = ".".join(map(str, VERSION))
 S_BUILD_DT = "{0:04d}-{1:02d}-{2:02d}".format(*BUILD_DT)

copyparty/authsrv.py

@@ -10,20 +10,35 @@ import hashlib
 import threading
 
 from .__init__ import WINDOWS
-from .util import IMPLICATIONS, uncyg, undot, Pebkac, fsdec, fsenc, statdir
+from .util import IMPLICATIONS, uncyg, undot, absreal, Pebkac, fsdec, fsenc, statdir
+from .bos import bos
+
+
+class AXS(object):
+    def __init__(self, uread=None, uwrite=None, umove=None, udel=None):
+        self.uread = {} if uread is None else {k: 1 for k in uread}
+        self.uwrite = {} if uwrite is None else {k: 1 for k in uwrite}
+        self.umove = {} if umove is None else {k: 1 for k in umove}
+        self.udel = {} if udel is None else {k: 1 for k in udel}
+
+    def __repr__(self):
+        return "AXS({})".format(
+            ", ".join(
+                "{}={!r}".format(k, self.__dict__[k])
+                for k in "uread uwrite umove udel".split()
+            )
+        )
 
 
 class VFS(object):
     """single level in the virtual fs"""
 
-    def __init__(self, log, realpath, vpath, uread, uwrite, uadm, flags):
+    def __init__(self, log, realpath, vpath, axs, flags):
         self.log = log
         self.realpath = realpath  # absolute path on host filesystem
         self.vpath = vpath  # absolute path in the virtual filesystem
-        self.uread = uread  # users who can read this
+        self.axs = axs  # type: AXS
-        self.uwrite = uwrite  # users who can write this
+        self.flags = flags  # config options
-        self.uadm = uadm  # users who are regular admins
-        self.flags = flags  # config switches
         self.nodes = {}  # child nodes
         self.histtab = None  # all realpath->histpath
         self.dbv = None  # closest full/non-jump parent
@@ -31,15 +46,23 @@ class VFS(object):
         if realpath:
             self.histpath = os.path.join(realpath, ".hist")  # db / thumbcache
             self.all_vols = {vpath: self}  # flattened recursive
+            self.aread = {}
+            self.awrite = {}
+            self.amove = {}
+            self.adel = {}
         else:
             self.histpath = None
             self.all_vols = None
+            self.aread = None
+            self.awrite = None
+            self.amove = None
+            self.adel = None
 
     def __repr__(self):
         return "VFS({})".format(
             ", ".join(
                 "{}={!r}".format(k, self.__dict__[k])
-                for k in "realpath vpath uread uwrite uadm flags".split()
+                for k in "realpath vpath axs flags".split()
             )
         )
 
@@ -66,9 +89,7 @@ class VFS(object):
                 self.log,
                 os.path.join(self.realpath, name) if self.realpath else None,
                 "{}/{}".format(self.vpath, name).lstrip("/"),
-                self.uread,
+                self.axs,
-                self.uwrite,
-                self.uadm,
                 self._copy_flags(name),
             )
             vn.dbv = self.dbv or self
@@ -81,7 +102,7 @@ class VFS(object):
 
         # leaf does not exist; create and keep permissions blank
         vp = "{}/{}".format(self.vpath, dst).lstrip("/")
-        vn = VFS(self.log, src, vp, [], [], [], {})
+        vn = VFS(self.log, src, vp, AXS(), {})
         vn.dbv = self.dbv or self
         self.nodes[dst] = vn
         return vn
@@ -121,23 +142,32 @@ class VFS(object):
         return [self, vpath]
 
     def can_access(self, vpath, uname):
-        """return [readable,writable]"""
+        # type: (str, str) -> tuple[bool, bool, bool, bool]
+        """can Read,Write,Move,Delete"""
         vn, _ = self._find(vpath)
+        c = vn.axs
         return [
-            uname in vn.uread or "*" in vn.uread,
+            uname in c.uread or "*" in c.uread,
-            uname in vn.uwrite or "*" in vn.uwrite,
+            uname in c.uwrite or "*" in c.uwrite,
+            uname in c.umove or "*" in c.umove,
+            uname in c.udel or "*" in c.udel,
         ]
 
-    def get(self, vpath, uname, will_read, will_write):
+    def get(self, vpath, uname, will_read, will_write, will_move=False, will_del=False):
-        # type: (str, str, bool, bool) -> tuple[VFS, str]
+        # type: (str, str, bool, bool, bool, bool) -> tuple[VFS, str]
         """returns [vfsnode,fs_remainder] if user has the requested permissions"""
         vn, rem = self._find(vpath)
+        c = vn.axs
 
-        if will_read and (uname not in vn.uread and "*" not in vn.uread):
+        for req, d, msg in [
-            raise Pebkac(403, "you don't have read-access for this location")
+            [will_read, c.uread, "read"],
-
+            [will_write, c.uwrite, "write"],
-        if will_write and (uname not in vn.uwrite and "*" not in vn.uwrite):
+            [will_move, c.umove, "move"],
-            raise Pebkac(403, "you don't have write-access for this location")
+            [will_del, c.udel, "delete"],
+        ]:
+            if req and (uname not in d and "*" not in d):
+                m = "you don't have {}-access for this location"
+                raise Pebkac(403, m.format(msg))
 
         return vn, rem
 
@@ -150,65 +180,50 @@ class VFS(object):
         vrem = "/".join([x for x in vrem if x])
         return dbv, vrem
 
-    def canonical(self, rem):
+    def canonical(self, rem, resolve=True):
         """returns the canonical path (fully-resolved absolute fs path)"""
         rp = self.realpath
         if rem:
             rp += "/" + rem
 
-        try:
+        return absreal(rp) if resolve else rp
-            return fsdec(os.path.realpath(fsenc(rp)))
-        except:
-            if not WINDOWS:
-                raise
 
-            # cpython bug introduced in 3.8, still exists in 3.9.1;
+    def ls(self, rem, uname, scandir, permsets, lstat=False):
-            # some win7sp1 and win10:20H2 boxes cannot realpath a
+        # type: (str, str, bool, list[list[bool]], bool) -> tuple[str, str, dict[str, VFS]]
-            # networked drive letter such as b"n:" or b"n:\\"
-            #
-            # requirements to trigger:
-            #  * bytestring (not unicode str)
-            #  * just the drive letter (subfolders are ok)
-            #  * networked drive (regular disks and vmhgfs are ok)
-            #  * on an enterprise network (idk, cannot repro with samba)
-            #
-            # hits the following exceptions in succession:
-            #  * access denied at L601: "path = _getfinalpathname(path)"
-            #  * "cant concat str to bytes" at L621: "return path + tail"
-            #
-            return os.path.realpath(rp)
-
-    def ls(self, rem, uname, scandir, incl_wo=False, lstat=False):
-        # type: (str, str, bool, bool, bool) -> tuple[str, str, dict[str, VFS]]
         """return user-readable [fsdir,real,virt] items at vpath"""
         virt_vis = {}  # nodes readable by user
         abspath = self.canonical(rem)
         real = list(statdir(self.log, scandir, lstat, abspath))
         real.sort()
         if not rem:
-            for name, vn2 in sorted(self.nodes.items()):
+            # no vfs nodes in the list of real inodes
-                ok = uname in vn2.uread or "*" in vn2.uread
+            real = [x for x in real if x[0] not in self.nodes]
 
-                if not ok and incl_wo:
+            for name, vn2 in sorted(self.nodes.items()):
-                    ok = uname in vn2.uwrite or "*" in vn2.uwrite
+                ok = False
+                axs = vn2.axs
+                axs = [axs.uread, axs.uwrite, axs.umove, axs.udel]
+                for pset in permsets:
+                    ok = True
+                    for req, lst in zip(pset, axs):
+                        if req and uname not in lst and "*" not in lst:
+                            ok = False
+                    if ok:
+                        break
 
                 if ok:
                     virt_vis[name] = vn2
 
-            # no vfs nodes in the list of real inodes
-            real = [x for x in real if x[0] not in self.nodes]
-
         return [abspath, real, virt_vis]
 
-    def walk(self, rel, rem, seen, uname, dots, scandir, lstat):
+    def walk(self, rel, rem, seen, uname, permsets, dots, scandir, lstat):
         """
         recursively yields from ./rem;
         rel is a unix-style user-defined vpath (not vfs-related)
         """
 
-        fsroot, vfs_ls, vfs_virt = self.ls(
+        fsroot, vfs_ls, vfs_virt = self.ls(rem, uname, scandir, permsets, lstat=lstat)
-            rem, uname, scandir, incl_wo=False, lstat=lstat
+        dbv, vrem = self.get_dbv(rem)
-        )
 
         if (
             seen
@@ -226,7 +241,7 @@ class VFS(object):
         rfiles.sort()
         rdirs.sort()
 
-        yield rel, fsroot, rfiles, rdirs, vfs_virt
+        yield dbv, vrem, rel, fsroot, rfiles, rdirs, vfs_virt
 
         for rdir, _ in rdirs:
             if not dots and rdir.startswith("."):
@@ -234,7 +249,7 @@ class VFS(object):
 
             wrel = (rel + "/" + rdir).lstrip("/")
             wrem = (rem + "/" + rdir).lstrip("/")
-            for x in self.walk(wrel, wrem, seen, uname, dots, scandir, lstat):
+            for x in self.walk(wrel, wrem, seen, uname, permsets, dots, scandir, lstat):
                 yield x
 
         for n, vfs in sorted(vfs_virt.items()):
@@ -242,7 +257,7 @@ class VFS(object):
                 continue
 
             wrel = (rel + "/" + n).lstrip("/")
-            for x in vfs.walk(wrel, "", seen, uname, dots, scandir, lstat):
+            for x in vfs.walk(wrel, "", seen, uname, permsets, dots, scandir, lstat):
                 yield x
 
     def zipgen(self, vrem, flt, uname, dots, scandir):
@@ -253,9 +268,8 @@ class VFS(object):
         f2a = os.sep + "dir.txt"
         f2b = "{0}.hist{0}".format(os.sep)
 
-        for vpath, apath, files, rd, vd in self.walk(
+        g = self.walk("", vrem, [], uname, [[True]], dots, scandir, False)
-            "", vrem, [], uname, dots, scandir, False
+        for _, _, vpath, apath, files, rd, vd in g:
-        ):
             if flt:
                 files = [x for x in files if x[0] in flt]
 
@@ -295,20 +309,6 @@ class VFS(object):
             for f in [{"vp": v, "ap": a, "st": n[1]} for v, a, n in files]:
                 yield f
 
-    def user_tree(self, uname, readable, writable, admin):
-        is_readable = False
-        if uname in self.uread or "*" in self.uread:
-            readable.append(self.vpath)
-            is_readable = True
-
-        if uname in self.uwrite or "*" in self.uwrite:
-            writable.append(self.vpath)
-            if is_readable:
-                admin.append(self.vpath)
-
-        for _, vn in sorted(self.nodes.items()):
-            vn.user_tree(uname, readable, writable, admin)
-
 
 class AuthSrv(object):
     """verifies users against given paths"""
@@ -341,7 +341,8 @@ class AuthSrv(object):
 
         yield prev, True
 
-    def _parse_config_file(self, fd, user, mread, mwrite, madm, mflags, mount):
+    def _parse_config_file(self, fd, acct, daxs, mflags, mount):
+        # type: (any, str, dict[str, AXS], any, str) -> None
         vol_src = None
         vol_dst = None
         self.line_ctr = 0
@@ -357,7 +358,7 @@ class AuthSrv(object):
             if vol_src is None:
                 if ln.startswith("u "):
                     u, p = ln[2:].split(":", 1)
-                    user[u] = p
+                    acct[u] = p
                 else:
                     vol_src = ln
                 continue
@@ -368,50 +369,50 @@ class AuthSrv(object):
                     raise Exception('invalid mountpoint "{}"'.format(vol_dst))
 
                 # cfg files override arguments and previous files
-                vol_src = fsdec(os.path.abspath(fsenc(vol_src)))
+                vol_src = bos.path.abspath(vol_src)
                 vol_dst = vol_dst.strip("/")
                 mount[vol_dst] = vol_src
-                mread[vol_dst] = []
+                daxs[vol_dst] = AXS()
-                mwrite[vol_dst] = []
-                madm[vol_dst] = []
                 mflags[vol_dst] = {}
                 continue
 
-            if len(ln) > 1:
+            try:
-                lvl, uname = ln.split(" ")
+                lvl, uname = ln.split(" ", 1)
-            else:
+            except:
                 lvl = ln
                 uname = "*"
 
-            self._read_vol_str(
+            if lvl == "a":
-                lvl,
+                m = "WARNING (config-file): permission flag 'a' is deprecated; please use 'rw' instead"
-                uname,
+                self.log(m, 1)
-                mread[vol_dst],
-                mwrite[vol_dst],
-                madm[vol_dst],
-                mflags[vol_dst],
-            )
 
-    def _read_vol_str(self, lvl, uname, mr, mw, ma, mf):
+            self._read_vol_str(lvl, uname, daxs[vol_dst], mflags[vol_dst])
+
+    def _read_vol_str(self, lvl, uname, axs, flags):
+        # type: (str, str, AXS, any) -> None
         if lvl == "c":
             cval = True
             if "=" in uname:
                 uname, cval = uname.split("=", 1)
 
-            self._read_volflag(mf, uname, cval, False)
+            self._read_volflag(flags, uname, cval, False)
             return
 
         if uname == "":
             uname = "*"
 
-        if lvl in "ra":
+        for un in uname.split(","):
-            mr.append(uname)
+            if "r" in lvl:
+                axs.uread[un] = 1
 
-        if lvl in "wa":
+            if "w" in lvl:
-            mw.append(uname)
+                axs.uwrite[un] = 1
 
-        if lvl == "a":
+            if "m" in lvl:
-            ma.append(uname)
+                axs.umove[un] = 1
+
+            if "d" in lvl:
+                axs.udel[un] = 1
 
     def _read_volflag(self, flags, name, value, is_list):
         if name not in ["mtp"]:
@@ -433,21 +434,24 @@ class AuthSrv(object):
         before finally building the VFS
         """
 
-        user = {}  # username:password
+        acct = {}  # username:password
-        mread = {}  # mountpoint:[username]
+        daxs = {}  # type: dict[str, AXS]
-        mwrite = {}  # mountpoint:[username]
-        madm = {}  # mountpoint:[username]
         mflags = {}  # mountpoint:[flag]
         mount = {}  # dst:src (mountpoint:realpath)
 
         if self.args.a:
             # list of username:password
-            for u, p in [x.split(":", 1) for x in self.args.a]:
+            for x in self.args.a:
-                user[u] = p
+                try:
+                    u, p = x.split(":", 1)
+                    acct[u] = p
+                except:
+                    m = '\n  invalid value "{}" for argument -a, must be username:password'
+                    raise Exception(m.format(x))
 
         if self.args.v:
             # list of src:dst:permset:permset:...
-            # permset is [rwa]username or [c]flag
+            # permset is <rwmd>[,username][,username] or <c>,<flag>[=args]
             for v_str in self.args.v:
                 m = self.re_vol.match(v_str)
                 if not m:
@@ -458,27 +462,21 @@ class AuthSrv(object):
                     src = uncyg(src)
 
                 # print("\n".join([src, dst, perms]))
-                src = fsdec(os.path.abspath(fsenc(src)))
+                src = bos.path.abspath(src)
                 dst = dst.strip("/")
                 mount[dst] = src
-                mread[dst] = []
+                daxs[dst] = AXS()
-                mwrite[dst] = []
-                madm[dst] = []
                 mflags[dst] = {}
 
-                perms = perms.split(":")
+                for x in perms.split(":"):
-                for (lvl, uname) in [[x[0], x[1:]] for x in perms]:
+                    lvl, uname = x.split(",", 1) if "," in x else [x, ""]
-                    self._read_vol_str(
+                    self._read_vol_str(lvl, uname, daxs[dst], mflags[dst])
-                        lvl, uname, mread[dst], mwrite[dst], madm[dst], mflags[dst]
-                    )
 
         if self.args.c:
             for cfg_fn in self.args.c:
                 with open(cfg_fn, "rb") as f:
                     try:
-                        self._parse_config_file(
+                        self._parse_config_file(f, acct, daxs, mflags, mount)
-                            f, user, mread, mwrite, madm, mflags, mount
-                        )
                     except:
                         m = "\n\033[1;31m\nerror in config file {} on line {}:\n\033[0m"
                         self.log(m.format(cfg_fn, self.line_ctr), 1)
@@ -488,19 +486,17 @@ class AuthSrv(object):
         if WINDOWS:
             cased = {}
             for k, v in mount.items():
-                try:
+                cased[k] = absreal(v)
-                    cased[k] = fsdec(os.path.realpath(fsenc(v)))
-                except:
-                    cased[k] = v
 
             mount = cased
 
         if not mount:
             # -h says our defaults are CWD at root and read/write for everyone
-            vfs = VFS(self.log_func, os.path.abspath("."), "", ["*"], ["*"], ["*"], {})
+            axs = AXS(["*"], ["*"], None, None)
+            vfs = VFS(self.log_func, bos.path.abspath("."), "", axs, {})
         elif "" not in mount:
             # there's volumes but no root; make root inaccessible
-            vfs = VFS(self.log_func, None, "", [], [], [], {})
+            vfs = VFS(self.log_func, None, "", AXS(), {})
             vfs.flags["d2d"] = True
 
         maxdepth = 0
@@ -511,32 +507,34 @@ class AuthSrv(object):
 
             if dst == "":
                 # rootfs was mapped; fully replaces the default CWD vfs
-                vfs = VFS(
+                vfs = VFS(self.log_func, mount[dst], dst, daxs[dst], mflags[dst])
-                    self.log_func,
-                    mount[dst],
-                    dst,
-                    mread[dst],
-                    mwrite[dst],
-                    madm[dst],
-                    mflags[dst],
-                )
                 continue
 
             v = vfs.add(mount[dst], dst)
-            v.uread = mread[dst]
+            v.axs = daxs[dst]
-            v.uwrite = mwrite[dst]
-            v.uadm = madm[dst]
             v.flags = mflags[dst]
             v.dbv = None
 
         vfs.all_vols = {}
         vfs.get_all_vols(vfs.all_vols)
 
+        for perm in "read write move del".split():
+            axs_key = "u" + perm
+            unames = ["*"] + list(acct.keys())
+            umap = {x: [] for x in unames}
+            for usr in unames:
+                for mp, vol in vfs.all_vols.items():
+                    if usr in getattr(vol.axs, axs_key):
+                        umap[usr].append(mp)
+            setattr(vfs, "a" + perm, umap)
+
+        all_users = {}
         missing_users = {}
-        for d in [mread, mwrite]:
+        for axs in daxs.values():
-            for _, ul in d.items():
+            for d in [axs.uread, axs.uwrite, axs.umove, axs.udel]:
-                for usr in ul:
+                for usr in d.keys():
-                    if usr != "*" and usr not in user:
+                    all_users[usr] = 1
+                    if usr != "*" and usr not in acct:
                         missing_users[usr] = 1
 
         if missing_users:
@@ -560,10 +558,7 @@ class AuthSrv(object):
             elif self.args.hist:
                 for nch in range(len(hid)):
                     hpath = os.path.join(self.args.hist, hid[: nch + 1])
-                    try:
+                    bos.makedirs(hpath)
-                        os.makedirs(hpath)
-                    except:
-                        pass
 
                     powner = os.path.join(hpath, "owner.txt")
                     try:
@@ -583,9 +578,9 @@ class AuthSrv(object):
                     vol.histpath = hpath
                     break
 
-            vol.histpath = os.path.realpath(vol.histpath)
+            vol.histpath = absreal(vol.histpath)
             if vol.dbv:
-                if os.path.exists(os.path.join(vol.histpath, "up2k.db")):
+                if bos.path.exists(os.path.join(vol.histpath, "up2k.db")):
                     promote.append(vol)
                     vol.dbv = None
                 else:
@@ -611,7 +606,7 @@ class AuthSrv(object):
         all_mte = {}
         errors = False
         for vol in vfs.all_vols.values():
-            if (self.args.e2ds and vol.uwrite) or self.args.e2dsa:
+            if (self.args.e2ds and vol.axs.uwrite) or self.args.e2dsa:
                 vol.flags["e2ds"] = True
 
             if self.args.e2d or "e2ds" in vol.flags:
@@ -700,6 +695,27 @@ class AuthSrv(object):
 
         vfs.bubble_flags()
 
+        m = "volumes and permissions:\n"
+        for v in vfs.all_vols.values():
+            if not self.warn_anonwrite:
+                break
+
+            m += '\n\033[36m"/{}"  \033[33m{}\033[0m'.format(v.vpath, v.realpath)
+            for txt, attr in [
+                ["  read", "uread"],
+                [" write", "uwrite"],
+                ["  move", "umove"],
+                ["delete", "udel"],
+            ]:
+                u = list(sorted(getattr(v.axs, attr).keys()))
+                u = ", ".join("\033[35meverybody\033[0m" if x == "*" else x for x in u)
+                u = u if u else "\033[36m--none--\033[0m"
+                m += "\n|  {}:  {}".format(txt, u)
+            m += "\n"
+
+        if self.warn_anonwrite and not self.args.no_voldump:
+            self.log(m)
+
         try:
             v, _ = vfs.get("/", "*", False, True)
             if self.warn_anonwrite and os.getcwd() == v.realpath:
@@ -711,17 +727,14 @@ class AuthSrv(object):
 
         with self.mutex:
             self.vfs = vfs
-            self.user = user
+            self.acct = acct
-            self.iuser = {v: k for k, v in user.items()}
+            self.iacct = {v: k for k, v in acct.items()}
 
             self.re_pwd = None
-            pwds = [re.escape(x) for x in self.iuser.keys()]
+            pwds = [re.escape(x) for x in self.iacct.keys()]
             if pwds:
                 self.re_pwd = re.compile("=(" + "|".join(pwds) + ")([]&; ]|$)")
 
-        # import pprint
-        # pprint.pprint({"usr": user, "rd": mread, "wr": mwrite, "mnt": mount})
-
     def dbg_ls(self):
         users = self.args.ls
         vols = "*"
@@ -739,12 +752,12 @@ class AuthSrv(object):
             pass
 
         if users == "**":
-            users = list(self.user.keys()) + ["*"]
+            users = list(self.acct.keys()) + ["*"]
         else:
             users = [users]
 
         for u in users:
-            if u not in self.user and u != "*":
+            if u not in self.acct and u != "*":
                 raise Exception("user not found: " + u)
 
         if vols == "*":
@@ -760,8 +773,10 @@ class AuthSrv(object):
                 raise Exception("volume not found: " + v)
 
         self.log({"users": users, "vols": vols, "flags": flags})
+        m = "/{}: read({}) write({}) move({}) del({})"
         for k, v in self.vfs.all_vols.items():
-            self.log("/{}: read({}) write({})".format(k, v.uread, v.uwrite))
+            vc = v.axs
+            self.log(m.format(k, vc.uread, vc.uwrite, vc.umove, vc.udel))
 
         flag_v = "v" in flags
         flag_ln = "ln" in flags
@@ -775,13 +790,15 @@ class AuthSrv(object):
             for u in users:
                 self.log("checking /{} as {}".format(v, u))
                 try:
-                    vn, _ = self.vfs.get(v, u, True, False)
+                    vn, _ = self.vfs.get(v, u, True, False, False, False)
                 except:
                     continue
 
                 atop = vn.realpath
-                g = vn.walk("", "", [], u, True, not self.args.no_scandir, False)
+                g = vn.walk(
-                for vpath, apath, files, _, _ in g:
+                    vn.vpath, "", [], u, [[True]], True, not self.args.no_scandir, False
+                )
+                for _, _, vpath, apath, files, _, _ in g:
                     fnames = [n[0] for n in files]
                     vpaths = [vpath + "/" + n for n in fnames] if vpath else fnames
                     vpaths = [vtop + x for x in vpaths]

copyparty/bos/bos.py

@@ -0,0 +1,59 @@
+# coding: utf-8
+from __future__ import print_function, unicode_literals
+
+import os
+from ..util import fsenc, fsdec
+from . import path
+
+
+# grep -hRiE '(^|[^a-zA-Z_\.-])os\.' . | gsed -r 's/ /\n/g;s/\(/(\n/g' | grep -hRiE '(^|[^a-zA-Z_\.-])os\.' | sort | uniq -c
+# printf 'os\.(%s)' "$(grep ^def bos/__init__.py | gsed -r 's/^def //;s/\(.*//' | tr '\n' '|' | gsed -r 's/.$//')"
+
+
+def chmod(p, mode):
+    return os.chmod(fsenc(p), mode)
+
+
+def listdir(p="."):
+    return [fsdec(x) for x in os.listdir(fsenc(p))]
+
+
+def lstat(p):
+    return os.lstat(fsenc(p))
+
+
+def makedirs(name, mode=0o755, exist_ok=True):
+    bname = fsenc(name)
+    try:
+        os.makedirs(bname, mode=mode)
+    except:
+        if not exist_ok or not os.path.isdir(bname):
+            raise
+
+
+def mkdir(p, mode=0o755):
+    return os.mkdir(fsenc(p), mode=mode)
+
+
+def rename(src, dst):
+    return os.rename(fsenc(src), fsenc(dst))
+
+
+def replace(src, dst):
+    return os.replace(fsenc(src), fsenc(dst))
+
+
+def rmdir(p):
+    return os.rmdir(fsenc(p))
+
+
+def stat(p):
+    return os.stat(fsenc(p))
+
+
+def unlink(p):
+    return os.unlink(fsenc(p))
+
+
+def utime(p, times=None):
+    return os.utime(fsenc(p), times)

copyparty/bos/path.py

@@ -0,0 +1,33 @@
+# coding: utf-8
+from __future__ import print_function, unicode_literals
+
+import os
+from ..util import fsenc, fsdec
+
+
+def abspath(p):
+    return fsdec(os.path.abspath(fsenc(p)))
+
+
+def exists(p):
+    return os.path.exists(fsenc(p))
+
+
+def getmtime(p):
+    return os.path.getmtime(fsenc(p))
+
+
+def getsize(p):
+    return os.path.getsize(fsenc(p))
+
+
+def isdir(p):
+    return os.path.isdir(fsenc(p))
+
+
+def islink(p):
+    return os.path.islink(fsenc(p))
+
+
+def realpath(p):
+    return fsdec(os.path.realpath(fsenc(p)))

copyparty/broker_mp.py

@@ -22,12 +22,9 @@ class BrokerMp(object):
         self.retpend_mutex = threading.Lock()
         self.mutex = threading.Lock()
 
-        cores = self.args.j
+        self.num_workers = self.args.j or mp.cpu_count()
-        if not cores:
+        self.log("broker", "booting {} subprocesses".format(self.num_workers))
-            cores = mp.cpu_count()
+        for n in range(1, self.num_workers + 1):
-
-        self.log("broker", "booting {} subprocesses".format(cores))
-        for n in range(1, cores + 1):
             q_pend = mp.Queue(1)
             q_yield = mp.Queue(64)
 
@@ -103,5 +100,8 @@ class BrokerMp(object):
             for p in self.procs:
                 p.q_pend.put([0, dest, [args[0], len(self.procs)]])
 
+        elif dest == "cb_httpsrv_up":
+            self.hub.cb_httpsrv_up()
+
         else:
             raise Exception("what is " + str(dest))

copyparty/broker_mpw.py

@@ -29,7 +29,8 @@ class MpWorker(object):
         # we inherited signal_handler from parent,
         # replace it with something harmless
         if not FAKE_MP:
-            signal.signal(signal.SIGINT, self.signal_handler)
+            for sig in [signal.SIGINT, signal.SIGTERM]:
+                signal.signal(sig, self.signal_handler)
 
         # starting to look like a good idea
         self.asrv = AuthSrv(args, None, False)
@@ -44,7 +45,7 @@ class MpWorker(object):
         thr.start()
         thr.join()
 
-    def signal_handler(self, signal, frame):
+    def signal_handler(self, sig, frame):
         # print('k')
         pass
 

copyparty/broker_thr.py

@@ -17,6 +17,7 @@ class BrokerThr(object):
         self.asrv = hub.asrv
 
         self.mutex = threading.Lock()
+        self.num_workers = 1
 
         # instantiate all services here (TODO: inheritance?)
         self.httpsrv = HttpSrv(self, None)

copyparty/httpcli.py

@@ -15,6 +15,7 @@ import calendar
 
 from .__init__ import E, PY2, WINDOWS, ANYWIN, unicode
 from .util import *  # noqa  # pylint: disable=unused-wildcard-import
+from .bos import bos
 from .authsrv import AuthSrv
 from .szip import StreamZip
 from .star import StreamTar
@@ -58,9 +59,12 @@ class HttpCli(object):
 
     def unpwd(self, m):
         a, b = m.groups()
-        return "=\033[7m {} \033[27m{}".format(self.asrv.iuser[a], b)
+        return "=\033[7m {} \033[27m{}".format(self.asrv.iacct[a], b)
+
+    def _check_nonfatal(self, ex, post):
+        if post:
+            return ex.code < 300
 
-    def _check_nonfatal(self, ex):
         return ex.code < 400 or ex.code in [404, 429]
 
     def _assert_safe_rem(self, rem):
@@ -102,7 +106,7 @@ class HttpCli(object):
             self.req = "[junk]"
             self.http_ver = "HTTP/1.1"
             # self.log("pebkac at httpcli.run #1: " + repr(ex))
-            self.keepalive = self._check_nonfatal(ex)
+            self.keepalive = False
             self.loud_reply(unicode(ex), status=ex.code)
             return self.keepalive
 
@@ -178,12 +182,14 @@ class HttpCli(object):
 
         self.uparam = uparam
         self.cookies = cookies
-        self.vpath = unquotep(vpath)
+        self.vpath = unquotep(vpath)  # not query, so + means +
 
         pwd = uparam.get("pw")
-        self.uname = self.asrv.iuser.get(pwd, "*")
+        self.uname = self.asrv.iacct.get(pwd, "*")
-        self.rvol, self.wvol, self.avol = [[], [], []]
+        self.rvol = self.asrv.vfs.aread[self.uname]
-        self.asrv.vfs.user_tree(self.uname, self.rvol, self.wvol, self.avol)
+        self.wvol = self.asrv.vfs.awrite[self.uname]
+        self.mvol = self.asrv.vfs.amove[self.uname]
+        self.dvol = self.asrv.vfs.adel[self.uname]
 
         if pwd and "pw" in self.ouparam and pwd != cookies.get("cppwd"):
             self.out_headers["Set-Cookie"] = self.get_pwd_cookie(pwd)[0]
@@ -213,7 +219,8 @@ class HttpCli(object):
         except Pebkac as ex:
             try:
                 # self.log("pebkac at httpcli.run #2: " + repr(ex))
-                if not self._check_nonfatal(ex):
+                post = self.mode in ["POST", "PUT"] or "content-length" in self.headers
+                if not self._check_nonfatal(ex, post):
                     self.keepalive = False
 
                 self.log("{}\033[0m, {}".format(str(ex), self.vpath), 3)
@@ -339,11 +346,36 @@ class HttpCli(object):
             static_path = os.path.join(E.mod, "web/", self.vpath[5:])
             return self.tx_file(static_path)
 
+        x = self.asrv.vfs.can_access(self.vpath, self.uname)
+        self.can_read, self.can_write, self.can_move, self.can_delete = x
+        if not self.can_read and not self.can_write:
+            if self.vpath:
+                self.log("inaccessible: [{}]".format(self.vpath))
+                raise Pebkac(404)
+
+            self.uparam["h"] = False
+
         if "tree" in self.uparam:
             return self.tx_tree()
 
-        if "stack" in self.uparam:
+        if "delete" in self.uparam:
-            return self.tx_stack()
+            return self.handle_rm()
+
+        if "move" in self.uparam:
+            return self.handle_mv()
+
+        if "scan" in self.uparam:
+            return self.scanvol()
+
+        if not self.vpath:
+            if "stack" in self.uparam:
+                return self.tx_stack()
+
+            if "ups" in self.uparam:
+                return self.tx_ups()
+
+            if "h" in self.uparam:
+                return self.tx_mounts()
 
         # conditional redirect to single volumes
         if self.vpath == "" and not self.ouparam:
@@ -359,21 +391,6 @@ class HttpCli(object):
                     self.redirect(vpath, flavor="redirecting to", use302=True)
                     return True
 
-        self.readable, self.writable = self.asrv.vfs.can_access(self.vpath, self.uname)
-        if not self.readable and not self.writable:
-            if self.vpath:
-                self.log("inaccessible: [{}]".format(self.vpath))
-                raise Pebkac(404)
-
-            self.uparam = {"h": False}
-
-        if "h" in self.uparam:
-            self.vpath = None
-            return self.tx_mounts()
-
-        if "scan" in self.uparam:
-            return self.scanvol()
-
         return self.tx_browser()
 
     def handle_options(self):
@@ -488,7 +505,14 @@ class HttpCli(object):
         if not self.args.nw:
             vfs, vrem = vfs.get_dbv(rem)
             self.conn.hsrv.broker.put(
-                False, "up2k.hash_file", vfs.realpath, vfs.flags, vrem, fn
+                False,
+                "up2k.hash_file",
+                vfs.realpath,
+                vfs.flags,
+                vrem,
+                fn,
+                self.ip,
+                time.time(),
             )
 
         return post_sz, sha_b64, remains, path
@@ -582,6 +606,9 @@ class HttpCli(object):
         if "srch" in self.uparam or "srch" in body:
             return self.handle_search(body)
 
+        if "delete" in self.uparam:
+            return self.handle_rm(body)
+
         # up2k-php compat
         for k in "chunkpit.php", "handshake.php":
             if self.vpath.endswith(k):
@@ -606,11 +633,11 @@ class HttpCli(object):
         if sub:
             try:
                 dst = os.path.join(vfs.realpath, rem)
-                if not os.path.isdir(fsenc(dst)):
+                if not bos.path.isdir(dst):
-                    os.makedirs(fsenc(dst))
+                    bos.makedirs(dst)
             except OSError as ex:
                 self.log("makedirs failed [{}]".format(dst))
-                if not os.path.isdir(fsenc(dst)):
+                if not bos.path.isdir(dst):
                     if ex.errno == 13:
                         raise Pebkac(500, "the server OS denied write-access")
 
@@ -756,7 +783,7 @@ class HttpCli(object):
             times = (int(time.time()), int(lastmod))
             self.log("no more chunks, setting times {}".format(times))
             try:
-                os.utime(fsenc(path), times)
+                bos.utime(path, times)
             except:
                 self.log("failed to utime ({}, {})".format(path, times))
 
@@ -775,7 +802,7 @@ class HttpCli(object):
         return True
 
     def get_pwd_cookie(self, pwd):
-        if pwd in self.asrv.iuser:
+        if pwd in self.asrv.iacct:
             msg = "login ok"
             dt = datetime.utcfromtimestamp(time.time() + 60 * 60 * 24 * 365)
             exp = dt.strftime("%a, %d %b %Y %H:%M:%S GMT")
@@ -801,14 +828,14 @@ class HttpCli(object):
             fdir = os.path.join(vfs.realpath, rem)
             fn = os.path.join(fdir, sanitized)
 
-            if not os.path.isdir(fsenc(fdir)):
+            if not bos.path.isdir(fdir):
                 raise Pebkac(500, "parent folder does not exist")
 
-            if os.path.isdir(fsenc(fn)):
+            if bos.path.isdir(fn):
                 raise Pebkac(500, "that folder exists already")
 
             try:
-                os.mkdir(fsenc(fn))
+                bos.mkdir(fn)
             except OSError as ex:
                 if ex.errno == 13:
                     raise Pebkac(500, "the server OS denied write-access")
@@ -838,7 +865,7 @@ class HttpCli(object):
             fdir = os.path.join(vfs.realpath, rem)
             fn = os.path.join(fdir, sanitized)
 
-            if os.path.exists(fsenc(fn)):
+            if bos.path.exists(fn):
                 raise Pebkac(500, "that file exists already")
 
             with open(fsenc(fn), "wb") as f:
@@ -868,7 +895,7 @@ class HttpCli(object):
                         p_file, "", [".prologue.html", ".epilogue.html"]
                     )
 
-                    if not os.path.isdir(fsenc(fdir)):
+                    if not bos.path.isdir(fdir):
                         raise Pebkac(404, "that folder does not exist")
 
                     suffix = ".{:.6f}-{}".format(time.time(), self.ip)
@@ -895,6 +922,8 @@ class HttpCli(object):
                             dbv.flags,
                             vrem,
                             fname,
+                            self.ip,
+                            time.time(),
                         )
                         self.conn.nbyte += sz
 
@@ -907,10 +936,10 @@ class HttpCli(object):
 
                         suffix = ".PARTIAL"
                         try:
-                            os.rename(fsenc(fp), fsenc(fp2 + suffix))
+                            bos.rename(fp, fp2 + suffix)
                         except:
                             fp2 = fp2[: -len(suffix) - 1]
-                            os.rename(fsenc(fp), fsenc(fp2 + suffix))
+                            bos.rename(fp, fp2 + suffix)
 
                     raise
 
@@ -994,13 +1023,6 @@ class HttpCli(object):
         vfs, rem = self.asrv.vfs.get(self.vpath, self.uname, False, True)
         self._assert_safe_rem(rem)
 
-        # TODO:
-        #   the per-volume read/write permissions must be replaced with permission flags
-        #   which would decide how to handle uploads to filenames which are taken,
-        #   current behavior of creating a new name is a good default for binary files
-        #   but should also offer a flag to takeover the filename and rename the old one
-        #
-        # stopgap:
         if not rem.endswith(".md"):
             raise Pebkac(400, "only markdown pls")
 
@@ -1015,7 +1037,7 @@ class HttpCli(object):
         fp = os.path.join(vfs.realpath, rem)
         srv_lastmod = srv_lastmod3 = -1
         try:
-            st = os.stat(fsenc(fp))
+            st = bos.stat(fp)
             srv_lastmod = st.st_mtime
             srv_lastmod3 = int(srv_lastmod * 1000)
         except OSError as ex:
@@ -1051,14 +1073,13 @@ class HttpCli(object):
                 self.reply(response.encode("utf-8"))
                 return True
 
-            # TODO another hack re: pending permissions rework
             mdir, mfile = os.path.split(fp)
             mfile2 = "{}.{:.3f}.md".format(mfile[:-3], srv_lastmod)
             try:
-                os.mkdir(fsenc(os.path.join(mdir, ".hist")))
+                bos.mkdir(os.path.join(mdir, ".hist"))
             except:
                 pass
-            os.rename(fsenc(fp), fsenc(os.path.join(mdir, ".hist", mfile2)))
+            bos.rename(fp, os.path.join(mdir, ".hist", mfile2))
 
         p_field, _, p_data = next(self.parser.gen)
         if p_field != "body":
@@ -1067,7 +1088,7 @@ class HttpCli(object):
         with open(fsenc(fp), "wb", 512 * 1024) as f:
             sz, sha512, _ = hashcopy(p_data, f)
 
-        new_lastmod = os.stat(fsenc(fp)).st_mtime
+        new_lastmod = bos.stat(fp).st_mtime
         new_lastmod3 = int(new_lastmod * 1000)
         sha512 = sha512[:56]
 
@@ -1112,7 +1133,7 @@ class HttpCli(object):
         for ext in ["", ".gz", ".br"]:
             try:
                 fs_path = req_path + ext
-                st = os.stat(fsenc(fs_path))
+                st = bos.stat(fs_path)
                 file_ts = max(file_ts, st.st_mtime)
                 editions[ext or "plain"] = [fs_path, st.st_size]
             except:
@@ -1289,11 +1310,9 @@ class HttpCli(object):
         else:
             fn = self.headers.get("host", "hey")
 
-        afn = "".join(
+        safe = (string.ascii_letters + string.digits).replace("%", "")
-            [x if x in (string.ascii_letters + string.digits) else "_" for x in fn]
+        afn = "".join([x if x in safe.replace('"', "") else "_" for x in fn])
-        )
+        bascii = unicode(safe).encode("utf-8")
-
-        bascii = unicode(string.ascii_letters + string.digits).encode("utf-8")
         ufn = fn.encode("utf-8", "xmlcharrefreplace")
         if PY2:
             ufn = [unicode(x) if x in bascii else "%{:02x}".format(ord(x)) for x in ufn]
@@ -1308,6 +1327,7 @@ class HttpCli(object):
 
         cdis = "attachment; filename=\"{}.{}\"; filename*=UTF-8''{}.{}"
         cdis = cdis.format(afn, fmt, ufn, fmt)
+        self.log(cdis)
         self.send_headers(None, mime=mime, headers={"Content-Disposition": cdis})
 
         fgen = vn.zipgen(rem, items, self.uname, dots, not self.args.no_scandir)
@@ -1364,10 +1384,10 @@ class HttpCli(object):
         html_path = os.path.join(E.mod, "web", "{}.html".format(tpl))
         template = self.j2(tpl)
 
-        st = os.stat(fsenc(fs_path))
+        st = bos.stat(fs_path)
         ts_md = st.st_mtime
 
-        st = os.stat(fsenc(html_path))
+        st = bos.stat(html_path)
         ts_html = st.st_mtime
 
         sz_md = 0
@@ -1376,7 +1396,7 @@ class HttpCli(object):
             for c, v in [[b"&", 4], [b"<", 3], [b">", 3]]:
                 sz_md += (len(buf) - len(buf.replace(c, b""))) * v
 
-        file_ts = max(ts_md, ts_html)
+        file_ts = max(ts_md, ts_html, E.t0)
         file_lastmod, do_send = self._chk_lastmod(file_ts)
         self.out_headers["Last-Modified"] = file_lastmod
         self.out_headers.update(NO_CACHE)
@@ -1424,12 +1444,13 @@ class HttpCli(object):
 
     def tx_mounts(self):
         suf = self.urlq({}, ["h"])
+        avol = [x for x in self.wvol if x in self.rvol]
         rvol, wvol, avol = [
             [("/" + x).rstrip("/") + "/" for x in y]
-            for y in [self.rvol, self.wvol, self.avol]
+            for y in [self.rvol, self.wvol, avol]
         ]
 
-        if self.avol and not self.args.no_rescan:
+        if avol and not self.args.no_rescan:
             x = self.conn.hsrv.broker.put(True, "up2k.get_state")
             vs = json.loads(x.get())
             vstate = {("/" + k).rstrip("/") + "/": v for k, v in vs["volstate"].items()}
@@ -1454,8 +1475,8 @@ class HttpCli(object):
         return True
 
     def scanvol(self):
-        if not self.readable or not self.writable:
+        if not self.can_read or not self.can_write:
-            raise Pebkac(403, "not admin")
+            raise Pebkac(403, "not allowed for user " + self.uname)
 
         if self.args.no_rescan:
             raise Pebkac(403, "disabled by argv")
@@ -1473,8 +1494,8 @@ class HttpCli(object):
         raise Pebkac(500, x)
 
     def tx_stack(self):
-        if not self.avol:
+        if not [x for x in self.wvol if x in self.rvol]:
-            raise Pebkac(403, "not admin")
+            raise Pebkac(403, "not allowed for user " + self.uname)
 
         if self.args.no_stack:
             raise Pebkac(403, "disabled by argv")
@@ -1512,7 +1533,7 @@ class HttpCli(object):
         try:
             vn, rem = self.asrv.vfs.get(top, self.uname, True, False)
             fsroot, vfs_ls, vfs_virt = vn.ls(
-                rem, self.uname, not self.args.no_scandir, incl_wo=True
+                rem, self.uname, not self.args.no_scandir, [[True], [False, True]]
             )
         except:
             vfs_ls = []
@@ -1539,6 +1560,74 @@ class HttpCli(object):
         ret["a"] = dirs
         return ret
 
+    def tx_ups(self):
+        if not self.args.unpost:
+            raise Pebkac(400, "the unpost feature was disabled by server config")
+
+        filt = self.uparam.get("filter")
+        lm = "ups [{}]".format(filt)
+        self.log(lm)
+
+        ret = []
+        t0 = time.time()
+        idx = self.conn.get_u2idx()
+        lim = time.time() - self.args.unpost
+        for vol in self.asrv.vfs.all_vols.values():
+            cur = idx.get_cur(vol.realpath)
+            if not cur:
+                continue
+
+            q = "select sz, rd, fn, at from up where ip=? and at>?"
+            for sz, rd, fn, at in cur.execute(q, (self.ip, lim)):
+                vp = "/" + "/".join([rd, fn]).strip("/")
+                if filt and filt not in vp:
+                    continue
+
+                ret.append({"vp": vp, "sz": sz, "at": at})
+                if len(ret) > 3000:
+                    ret.sort(key=lambda x: x["at"], reverse=True)
+                    ret = ret[:2000]
+
+        ret.sort(key=lambda x: x["at"], reverse=True)
+        ret = ret[:2000]
+
+        jtxt = json.dumps(ret, indent=2, sort_keys=True).encode("utf-8", "replace")
+        self.log("{} #{} {:.2f}sec".format(lm, len(ret), time.time() - t0))
+        self.reply(jtxt, mime="application/json")
+
+    def handle_rm(self, req=None):
+        if not req and not self.can_delete:
+            raise Pebkac(403, "not allowed for user " + self.uname)
+
+        if self.args.no_del:
+            raise Pebkac(403, "disabled by argv")
+
+        if not req:
+            req = [self.vpath]
+
+        x = self.conn.hsrv.broker.put(True, "up2k.handle_rm", self.uname, self.ip, req)
+        self.loud_reply(x.get())
+
+    def handle_mv(self):
+        if not self.can_move:
+            raise Pebkac(403, "not allowed for user " + self.uname)
+
+        if self.args.no_mv:
+            raise Pebkac(403, "disabled by argv")
+
+        # full path of new loc (incl filename)
+        dst = self.uparam.get("move")
+        if not dst:
+            raise Pebkac(400, "need dst vpath")
+
+        # x-www-form-urlencoded (url query part) uses
+        # either + or %20 for 0x20 so handle both
+        dst = unquotep(dst.replace("+", " "))
+        x = self.conn.hsrv.broker.put(
+            True, "up2k.handle_mv", self.uname, self.vpath, dst
+        )
+        self.loud_reply(x.get())
+
     def tx_browser(self):
         vpath = ""
         vpnodes = [["", "/"]]
@@ -1551,18 +1640,16 @@ class HttpCli(object):
 
                 vpnodes.append([quotep(vpath) + "/", html_escape(node, crlf=True)])
 
-        vn, rem = self.asrv.vfs.get(
+        vn, rem = self.asrv.vfs.get(self.vpath, self.uname, False, False)
-            self.vpath, self.uname, self.readable, self.writable
-        )
         abspath = vn.canonical(rem)
         dbv, vrem = vn.get_dbv(rem)
 
         try:
-            st = os.stat(fsenc(abspath))
+            st = bos.stat(abspath)
         except:
             raise Pebkac(404)
 
-        if self.readable:
+        if self.can_read:
             if rem.startswith(".hist/up2k.") or (
                 rem.endswith("/dir.txt") and rem.startswith(".hist/th/")
             ):
@@ -1574,7 +1661,7 @@ class HttpCli(object):
                 if is_dir:
                     for fn in self.args.th_covers.split(","):
                         fp = os.path.join(abspath, fn)
-                        if os.path.exists(fp):
+                        if bos.path.exists(fp):
                             vrem = "{}/{}".format(vrem.rstrip("/"), fn)
                             is_dir = False
                             break
@@ -1629,10 +1716,14 @@ class HttpCli(object):
         srv_info = "</span> /// <span>".join(srv_info)
 
         perms = []
-        if self.readable:
+        if self.can_read:
             perms.append("read")
-        if self.writable:
+        if self.can_write:
             perms.append("write")
+        if self.can_move:
+            perms.append("move")
+        if self.can_delete:
+            perms.append("delete")
 
         url_suf = self.urlq({}, [])
         is_ls = "ls" in self.uparam
@@ -1644,7 +1735,7 @@ class HttpCli(object):
         logues = ["", ""]
         for n, fn in enumerate([".prologue.html", ".epilogue.html"]):
             fn = os.path.join(abspath, fn)
-            if os.path.exists(fsenc(fn)):
+            if bos.path.exists(fn):
                 with open(fsenc(fn), "rb") as f:
                     logues[n] = f.read().decode("utf-8")
 
@@ -1653,6 +1744,7 @@ class HttpCli(object):
             "files": [],
             "taglist": [],
             "srvinf": srv_info,
+            "acct": self.uname,
             "perms": perms,
             "logues": logues,
         }
@@ -1660,19 +1752,23 @@ class HttpCli(object):
             "vdir": quotep(self.vpath),
             "vpnodes": vpnodes,
             "files": [],
+            "acct": self.uname,
             "perms": json.dumps(perms),
             "taglist": [],
             "tag_order": [],
             "have_up2k_idx": ("e2d" in vn.flags),
             "have_tags_idx": ("e2t" in vn.flags),
+            "have_mv": (not self.args.no_mv),
+            "have_del": (not self.args.no_del),
             "have_zip": (not self.args.no_zip),
-            "have_b_u": (self.writable and self.uparam.get("b") == "u"),
+            "have_unpost": (self.args.unpost > 0),
+            "have_b_u": (self.can_write and self.uparam.get("b") == "u"),
             "url_suf": url_suf,
             "logues": logues,
             "title": html_escape(self.vpath, crlf=True),
             "srv_info": srv_info,
         }
-        if not self.readable:
+        if not self.can_read:
             if is_ls:
                 ret = json.dumps(ls_ret)
                 self.reply(
@@ -1695,7 +1791,7 @@ class HttpCli(object):
                 return self.tx_zip(k, v, vn, rem, [], self.args.ed)
 
         fsroot, vfs_ls, vfs_virt = vn.ls(
-            rem, self.uname, not self.args.no_scandir, incl_wo=True
+            rem, self.uname, not self.args.no_scandir, [[True], [False, True]]
         )
         stats = {k: v for k, v in vfs_ls}
         vfs_ls = [x[0] for x in vfs_ls]
@@ -1706,7 +1802,7 @@ class HttpCli(object):
         histdir = os.path.join(fsroot, ".hist")
         ptn = re.compile(r"(.*)\.([0-9]+\.[0-9]{3})(\.[^\.]+)$")
         try:
-            for hfn in os.listdir(histdir):
+            for hfn in bos.listdir(histdir):
                 m = ptn.match(hfn)
                 if not m:
                     continue
@@ -1747,7 +1843,7 @@ class HttpCli(object):
                 fspath = fsroot + "/" + fn
 
             try:
-                inf = stats.get(fn) or os.stat(fsenc(fspath))
+                inf = stats.get(fn) or bos.stat(fspath)
             except:
                 self.log("broken symlink: {}".format(repr(fspath)))
                 continue

copyparty/httpsrv.py

@@ -28,6 +28,7 @@ except ImportError:
 
 from .__init__ import E, PY2, MACOS
 from .util import spack, min_ex, start_stackmon, start_log_thrs
+from .bos import bos
 from .httpconn import HttpConn
 
 if PY2:
@@ -73,7 +74,7 @@ class HttpSrv(object):
         }
 
         cert_path = os.path.join(E.cfg, "cert.pem")
-        if os.path.exists(cert_path):
+        if bos.path.exists(cert_path):
             self.cert_path = cert_path
         else:
             self.cert_path = None
@@ -140,6 +141,7 @@ class HttpSrv(object):
         fno = srv_sck.fileno()
         msg = "subscribed @ {}:{}  f{}".format(ip, port, fno)
         self.log(self.name, msg)
+        self.broker.put(False, "cb_httpsrv_up")
         while not self.stopping:
             if self.args.log_conn:
                 self.log(self.name, "|%sC-ncli" % ("-" * 1,), c="1;30")
@@ -307,7 +309,7 @@ class HttpSrv(object):
             try:
                 with os.scandir(os.path.join(E.mod, "web")) as dh:
                     for fh in dh:
-                        inf = fh.stat(follow_symlinks=False)
+                        inf = fh.stat()
                         v = max(v, inf.st_mtime)
             except:
                 pass

copyparty/mtag.py

@@ -9,6 +9,7 @@ import subprocess as sp
 
 from .__init__ import PY2, WINDOWS, unicode
 from .util import fsenc, fsdec, uncyg, REKOBO_LKEY
+from .bos import bos
 
 
 def have_ff(cmd):
@@ -44,7 +45,7 @@ class MParser(object):
                 if WINDOWS:
                     bp = uncyg(bp)
 
-                if os.path.exists(bp):
+                if bos.path.exists(bp):
                     self.bin = bp
                     return
             except:
@@ -227,37 +228,47 @@ def parse_ffprobe(txt):
 class MTag(object):
     def __init__(self, log_func, args):
         self.log_func = log_func
+        self.args = args
         self.usable = True
-        self.prefer_mt = False
+        self.prefer_mt = not args.no_mtag_ff
-        mappings = args.mtm
         self.backend = "ffprobe" if args.no_mutagen else "mutagen"
-        or_ffprobe = " or ffprobe"
+        self.can_ffprobe = (
+            HAVE_FFPROBE
+            and not args.no_mtag_ff
+            and (not WINDOWS or sys.version_info >= (3, 8))
+        )
+        mappings = args.mtm
+        or_ffprobe = " or FFprobe"
 
         if self.backend == "mutagen":
             self.get = self.get_mutagen
             try:
                 import mutagen
             except:
-                self.log("could not load mutagen, trying ffprobe instead", c=3)
+                self.log("could not load Mutagen, trying FFprobe instead", c=3)
                 self.backend = "ffprobe"
 
         if self.backend == "ffprobe":
+            self.usable = self.can_ffprobe
             self.get = self.get_ffprobe
             self.prefer_mt = True
-            # about 20x slower
-            self.usable = HAVE_FFPROBE
 
-            if self.usable and WINDOWS and sys.version_info < (3, 8):
+            if not HAVE_FFPROBE:
-                self.usable = False
+                pass
+
+            elif args.no_mtag_ff:
+                msg = "found FFprobe but it was disabled by --no-mtag-ff"
+                self.log(msg, c=3)
+
+            elif WINDOWS and sys.version_info < (3, 8):
                 or_ffprobe = " or python >= 3.8"
-                msg = "found ffprobe but your python is too old; need 3.8 or newer"
+                msg = "found FFprobe but your python is too old; need 3.8 or newer"
                 self.log(msg, c=1)
 
         if not self.usable:
-            msg = "need mutagen{} to read media tags so please run this:\n{}{} -m pip install --user mutagen\n"
+            msg = "need Mutagen{} to read media tags so please run this:\n{}{} -m pip install --user mutagen\n"
-            self.log(
+            pybin = os.path.basename(sys.executable)
-                msg.format(or_ffprobe, " " * 37, os.path.basename(sys.executable)), c=1
+            self.log(msg.format(or_ffprobe, " " * 37, pybin), c=1)
-            )
             return
 
         # https://picard-docs.musicbrainz.org/downloads/MusicBrainz_Picard_Tag_Map.html
@@ -387,7 +398,7 @@ class MTag(object):
             v2 = r2.get(k)
             if v1 == v2:
                 print("  ", k, v1)
-            elif v1 != "0000":  # ffprobe date=0
+            elif v1 != "0000":  # FFprobe date=0
                 diffs.append(k)
                 print(" 1", k, v1)
                 print(" 2", k, v2)
@@ -408,20 +419,33 @@ class MTag(object):
             md = mutagen.File(fsenc(abspath), easy=True)
             x = md.info.length
         except Exception as ex:
-            return {}
+            return self.get_ffprobe(abspath) if self.can_ffprobe else {}
 
-        ret = {}
+        sz = bos.path.getsize(abspath)
-        try:
+        ret = {".q": [0, int((sz / md.info.length) / 128)]}
-            dur = int(md.info.length)
+
+        for attr, k, norm in [
+            ["codec", "ac", unicode],
+            ["channels", "chs", int],
+            ["sample_rate", ".hz", int],
+            ["bitrate", ".aq", int],
+            ["length", ".dur", int],
+        ]:
             try:
-                q = int(md.info.bitrate / 1024)
+                v = getattr(md.info, attr)
             except:
-                q = int((os.path.getsize(fsenc(abspath)) / dur) / 128)
+                continue
 
-            ret[".dur"] = [0, dur]
+            if not v:
-            ret[".q"] = [0, q]
+                continue
-        except:
+
-            pass
+            if k == ".aq":
+                v /= 1000
+
+            if k == "ac" and v.startswith("mp4a.40."):
+                v = "aac"
+
+            ret[k] = [0, norm(v)]
 
         return self.normalize_tags(ret, md)
 

copyparty/star.py

@@ -1,12 +1,12 @@
 # coding: utf-8
 from __future__ import print_function, unicode_literals
 
-import os
 import tarfile
 import threading
 
 from .sutil import errdesc
 from .util import Queue, fsenc
+from .bos import bos
 
 
 class QFile(object):
@@ -61,7 +61,7 @@ class StreamTar(object):
 
         yield None
         if self.errf:
-            os.unlink(self.errf["ap"])
+            bos.unlink(self.errf["ap"])
 
     def ser(self, f):
         name = f["vp"]

copyparty/sutil.py

@@ -1,11 +1,12 @@
 # coding: utf-8
 from __future__ import print_function, unicode_literals
 
-import os
 import time
 import tempfile
 from datetime import datetime
 
+from .bos import bos
+
 
 def errdesc(errors):
     report = ["copyparty failed to add the following files to the archive:", ""]
@@ -20,9 +21,9 @@ def errdesc(errors):
     dt = datetime.utcfromtimestamp(time.time())
     dt = dt.strftime("%Y-%m%d-%H%M%S")
 
-    os.chmod(tf_path, 0o444)
+    bos.chmod(tf_path, 0o444)
     return {
         "vp": "archive-errors-{}.txt".format(dt),
         "ap": tf_path,
-        "st": os.stat(tf_path),
+        "st": bos.stat(tf_path),
     }, report

copyparty/svchub.py

@@ -6,12 +6,15 @@ import os
 import sys
 import time
 import shlex
+import string
+import signal
+import socket
 import threading
 from datetime import datetime, timedelta
 import calendar
 
-from .__init__ import E, PY2, WINDOWS, MACOS, VT100
+from .__init__ import E, PY2, WINDOWS, ANYWIN, MACOS, VT100, unicode
-from .util import mp, start_log_thrs, start_stackmon
+from .util import mp, start_log_thrs, start_stackmon, min_ex
 from .authsrv import AuthSrv
 from .tcpsrv import TcpSrv
 from .up2k import Up2k
@@ -33,6 +36,10 @@ class SvcHub(object):
         self.args = args
         self.argv = argv
         self.logf = None
+        self.stop_req = False
+        self.stopping = False
+        self.stop_cond = threading.Condition()
+        self.httpsrv_up = 0
 
         self.ansi_re = re.compile("\033\\[[^m]*m")
         self.log_mutex = threading.Lock()
@@ -49,7 +56,7 @@ class SvcHub(object):
             start_log_thrs(self.log, args.log_thrs, 0)
 
         # initiate all services to manage
-        self.asrv = AuthSrv(self.args, self.log, False)
+        self.asrv = AuthSrv(self.args, self.log)
         if args.ls:
             self.asrv.dbg_ls()
 
@@ -80,6 +87,29 @@ class SvcHub(object):
 
         self.broker = Broker(self)
 
+    def thr_httpsrv_up(self):
+        time.sleep(5)
+        failed = self.broker.num_workers - self.httpsrv_up
+        if not failed:
+            return
+
+        m = "{}/{} workers failed to start"
+        m = m.format(failed, self.broker.num_workers)
+        self.log("root", m, 1)
+        os._exit(1)
+
+    def cb_httpsrv_up(self):
+        self.httpsrv_up += 1
+        if self.httpsrv_up != self.broker.num_workers:
+            return
+
+        self.log("root", "workers OK\n")
+        self.up2k.init_vols()
+
+        thr = threading.Thread(target=self.sd_notify, name="sd-notify")
+        thr.daemon = True
+        thr.start()
+
     def _logname(self):
         dt = datetime.utcfromtimestamp(time.time())
         fn = self.args.lo
@@ -127,21 +157,68 @@ class SvcHub(object):
         print(msg, end="")
 
     def run(self):
-        thr = threading.Thread(target=self.tcpsrv.run, name="svchub-main")
+        self.tcpsrv.run()
+
+        thr = threading.Thread(target=self.thr_httpsrv_up)
         thr.daemon = True
         thr.start()
 
-        # winxp/py2.7 support: thr.join() kills signals
+        for sig in [signal.SIGINT, signal.SIGTERM]:
-        try:
+            signal.signal(sig, self.signal_handler)
-            while True:
-                time.sleep(9001)
 
-        except KeyboardInterrupt:
+        # macos hangs after shutdown on sigterm with while-sleep,
+        # windows cannot ^c stop_cond (and win10 does the macos thing but winxp is fine??)
+        # linux is fine with both,
+        # never lucky
+        if ANYWIN:
+            # msys-python probably fine but >msys-python
+            thr = threading.Thread(target=self.stop_thr, name="svchub-sig")
+            thr.daemon = True
+            thr.start()
+
+            try:
+                while not self.stop_req:
+                    time.sleep(1)
+            except:
+                pass
+
+            self.shutdown()
+            thr.join()
+        else:
+            self.stop_thr()
+
+    def stop_thr(self):
+        while not self.stop_req:
+            with self.stop_cond:
+                self.stop_cond.wait(9001)
+
+        self.shutdown()
+
+    def signal_handler(self, sig, frame):
+        if self.stopping:
+            return
+
+        self.stop_req = True
+        with self.stop_cond:
+            self.stop_cond.notify_all()
+
+    def shutdown(self):
+        if self.stopping:
+            return
+
+        self.stopping = True
+        self.stop_req = True
+        with self.stop_cond:
+            self.stop_cond.notify_all()
+
+        ret = 1
+        try:
             with self.log_mutex:
                 print("OPYTHAT")
 
             self.tcpsrv.shutdown()
             self.broker.shutdown()
+            self.up2k.shutdown()
             if self.thumbsrv:
                 self.thumbsrv.shutdown()
 
@@ -154,11 +231,14 @@ class SvcHub(object):
                         print("waiting for thumbsrv (10sec)...")
 
             print("nailed it", end="")
+            ret = 0
         finally:
             print("\033[0m")
             if self.logf:
                 self.logf.close()
 
+            sys.exit(ret)
+
     def _log_disabled(self, src, msg, c=0):
         if not self.logf:
             return
@@ -268,3 +348,22 @@ class SvcHub(object):
         else:
             self.log("svchub", err)
             return False
+
+    def sd_notify(self):
+        try:
+            addr = os.getenv("NOTIFY_SOCKET")
+            if not addr:
+                return
+
+            addr = unicode(addr)
+            if addr.startswith("@"):
+                addr = "\0" + addr[1:]
+
+            m = "".join(x for x in addr if x in string.printable)
+            self.log("sd_notify", m)
+
+            sck = socket.socket(socket.AF_UNIX, socket.SOCK_DGRAM)
+            sck.connect(addr)
+            sck.sendall(b"READY=1")
+        except:
+            self.log("sd_notify", min_ex())

copyparty/szip.py

@@ -8,6 +8,7 @@ from datetime import datetime
 
 from .sutil import errdesc
 from .util import yieldfile, sanitize_fn, spack, sunpack
+from .bos import bos
 
 
 def dostime2unix(buf):
@@ -271,4 +272,4 @@ class StreamZip(object):
         yield self._ct(ecdr)
 
         if errors:
-            os.unlink(errf["ap"])
+            bos.unlink(errf["ap"])

copyparty/tcpsrv.py

@@ -4,6 +4,7 @@ from __future__ import print_function, unicode_literals
 import re
 import socket
 
+from .__init__ import MACOS, ANYWIN
 from .util import chkcmd
 
 
@@ -29,14 +30,16 @@ class TcpSrv(object):
                 for x in nonlocals:
                     eps[x] = "external"
 
+        msgs = []
+        m = "available @ http://{}:{}/  (\033[33m{}\033[0m)"
         for ip, desc in sorted(eps.items(), key=lambda x: x[1]):
             for port in sorted(self.args.p):
-                self.log(
+                msgs.append(m.format(ip, port, desc))
-                    "tcpsrv",
+
-                    "available @ http://{}:{}/  (\033[33m{}\033[0m)".format(
+        if msgs:
-                        ip, port, desc
+            msgs[-1] += "\n"
-                    ),
+            for m in msgs:
-                )
+                self.log("tcpsrv", m)
 
         self.srv = []
         for ip in self.args.i:
@@ -81,25 +84,100 @@ class TcpSrv(object):
 
         self.log("tcpsrv", "ok bye")
 
-    def detect_interfaces(self, listen_ips):
+    def ips_linux(self):
         eps = {}
-
-        # get all ips and their interfaces
         try:
-            ip_addr, _ = chkcmd("ip", "addr")
+            txt, _ = chkcmd(["ip", "addr"])
         except:
-            ip_addr = None
+            return eps
 
-        if ip_addr:
+        r = re.compile(r"^\s+inet ([^ ]+)/.* (.*)")
-            r = re.compile(r"^\s+inet ([^ ]+)/.* (.*)")
+        for ln in txt.split("\n"):
-            for ln in ip_addr.split("\n"):
+            try:
-                try:
+                ip, dev = r.match(ln.rstrip()).groups()
-                    ip, dev = r.match(ln.rstrip()).groups()
+                eps[ip] = dev
-                    for lip in listen_ips:
+            except:
-                        if lip in ["0.0.0.0", ip]:
+                pass
-                            eps[ip] = dev
+
-                except:
+        return eps
-                    pass
+
+    def ips_macos(self):
+        eps = {}
+        try:
+            txt, _ = chkcmd(["ifconfig"])
+        except:
+            return eps
+
+        rdev = re.compile(r"^([^ ]+):")
+        rip = re.compile(r"^\tinet ([0-9\.]+) ")
+        dev = None
+        for ln in txt.split("\n"):
+            m = rdev.match(ln)
+            if m:
+                dev = m.group(1)
+
+            m = rip.match(ln)
+            if m:
+                eps[m.group(1)] = dev
+                dev = None
+
+        return eps
+
+    def ips_windows_ipconfig(self):
+        eps = {}
+        try:
+            txt, _ = chkcmd(["ipconfig"])
+        except:
+            return eps
+
+        rdev = re.compile(r"(^[^ ].*):$")
+        rip = re.compile(r"^ +IPv?4? [^:]+: *([0-9\.]{7,15})$")
+        dev = None
+        for ln in txt.replace("\r", "").split("\n"):
+            m = rdev.match(ln)
+            if m:
+                dev = m.group(1).split(" adapter ", 1)[-1]
+
+            m = rip.match(ln)
+            if m and dev:
+                eps[m.group(1)] = dev
+                dev = None
+
+        return eps
+
+    def ips_windows_netsh(self):
+        eps = {}
+        try:
+            txt, _ = chkcmd("netsh interface ip show address".split())
+        except:
+            return eps
+
+        rdev = re.compile(r'.* "([^"]+)"$')
+        rip = re.compile(r".* IP\b.*: +([0-9\.]{7,15})$")
+        dev = None
+        for ln in txt.replace("\r", "").split("\n"):
+            m = rdev.match(ln)
+            if m:
+                dev = m.group(1)
+
+            m = rip.match(ln)
+            if m and dev:
+                eps[m.group(1)] = dev
+                dev = None
+
+        return eps
+
+    def detect_interfaces(self, listen_ips):
+        if MACOS:
+            eps = self.ips_macos()
+        elif ANYWIN:
+            eps = self.ips_windows_ipconfig()  # sees more interfaces
+            eps.update(self.ips_windows_netsh())  # has better names
+        else:
+            eps = self.ips_linux()
+
+        if "0.0.0.0" not in listen_ips:
+            eps = {k: v for k, v in eps if k in listen_ips}
 
         default_route = None
         s = socket.socket(socket.AF_INET, socket.SOCK_DGRAM)

copyparty/th_cli.py

@@ -5,6 +5,7 @@ import os
 
 from .util import Cooldown
 from .th_srv import thumb_path, THUMBABLE, FMT_FF
+from .bos import bos
 
 
 class ThumbCli(object):
@@ -25,6 +26,9 @@ class ThumbCli(object):
         if is_vid and self.args.no_vthumb:
             return None
 
+        if rem.startswith(".hist/th/") and rem.split(".")[-1] in ["webp", "jpg"]:
+            return os.path.join(ptop, rem)
+
         if fmt == "j" and self.args.th_no_jpg:
             fmt = "w"
 
@@ -36,7 +40,7 @@ class ThumbCli(object):
         tpath = thumb_path(histpath, rem, mtime, fmt)
         ret = None
         try:
-            st = os.stat(tpath)
+            st = bos.stat(tpath)
             if st.st_size:
                 ret = tpath
             else:

copyparty/th_srv.py

@@ -10,7 +10,8 @@ import threading
 import subprocess as sp
 
 from .__init__ import PY2, unicode
-from .util import fsenc, runcmd, Queue, Cooldown, BytesIO, min_ex
+from .util import fsenc, vsplit, runcmd, Queue, Cooldown, BytesIO, min_ex
+from .bos import bos
 from .mtag import HAVE_FFMPEG, HAVE_FFPROBE, ffprobe
 
 
@@ -73,12 +74,7 @@ def thumb_path(histpath, rem, mtime, fmt):
     # base16 = 16 = 256
     # b64-lc = 38 = 1444
     # base64 = 64 = 4096
-    try:
+    rd, fn = vsplit(rem)
-        rd, fn = rem.rsplit("/", 1)
-    except:
-        rd = ""
-        fn = rem
-
     if rd:
         h = hashlib.sha512(fsenc(rd)).digest()
         b64 = base64.urlsafe_b64encode(h).decode("ascii")[:24]
@@ -121,10 +117,10 @@ class ThumbSrv(object):
         if not self.args.no_vthumb and (not HAVE_FFMPEG or not HAVE_FFPROBE):
             missing = []
             if not HAVE_FFMPEG:
-                missing.append("ffmpeg")
+                missing.append("FFmpeg")
 
             if not HAVE_FFPROBE:
-                missing.append("ffprobe")
+                missing.append("FFprobe")
 
             msg = "cannot create video thumbnails because some of the required programs are not available: "
             msg += ", ".join(missing)
@@ -159,13 +155,10 @@ class ThumbSrv(object):
                 self.log("wait {}".format(tpath))
             except:
                 thdir = os.path.dirname(tpath)
-                try:
+                bos.makedirs(thdir)
-                    os.makedirs(thdir)
-                except:
-                    pass
 
                 inf_path = os.path.join(thdir, "dir.txt")
-                if not os.path.exists(inf_path):
+                if not bos.path.exists(inf_path):
                     with open(inf_path, "wb") as f:
                         f.write(fsenc(os.path.dirname(abspath)))
 
@@ -185,7 +178,7 @@ class ThumbSrv(object):
                 cond.wait(3)
 
         try:
-            st = os.stat(tpath)
+            st = bos.stat(tpath)
             if st.st_size:
                 return tpath
         except:
@@ -202,7 +195,7 @@ class ThumbSrv(object):
             abspath, tpath = task
             ext = abspath.split(".")[-1].lower()
             fun = None
-            if not os.path.exists(tpath):
+            if not bos.path.exists(tpath):
                 if ext in FMT_PIL:
                     fun = self.conv_pil
                 elif ext in FMT_FF:
@@ -313,7 +306,7 @@ class ThumbSrv(object):
 
         cmd += [fsenc(tpath)]
 
-        ret, sout, serr = runcmd(*cmd)
+        ret, sout, serr = runcmd(cmd)
         if ret != 0:
             msg = ["ff: {}".format(x) for x in serr.split("\n")]
             self.log("FFmpeg failed:\n" + "\n".join(msg), c="1;30")
@@ -328,7 +321,7 @@ class ThumbSrv(object):
             p1 = os.path.dirname(tdir)
             p2 = os.path.dirname(p1)
             for dp in [tdir, p1, p2]:
-                os.utime(fsenc(dp), (ts, ts))
+                bos.utime(dp, (ts, ts))
         except:
             pass
 
@@ -355,7 +348,7 @@ class ThumbSrv(object):
         prev_b64 = None
         prev_fp = None
         try:
-            ents = os.listdir(thumbpath)
+            ents = bos.listdir(thumbpath)
         except:
             return 0
 
@@ -366,7 +359,7 @@ class ThumbSrv(object):
 
             # "top" or b64 prefix/full (a folder)
             if len(f) <= 3 or len(f) == 24:
-                age = now - os.path.getmtime(fp)
+                age = now - bos.path.getmtime(fp)
                 if age > maxage:
                     with self.mutex:
                         safe = True
@@ -398,7 +391,7 @@ class ThumbSrv(object):
 
             if b64 == prev_b64:
                 self.log("rm replaced [{}]".format(fp))
-                os.unlink(prev_fp)
+                bos.unlink(prev_fp)
 
             prev_b64 = b64
             prev_fp = fp

copyparty/u2idx.py

@@ -7,7 +7,9 @@ import time
 import threading
 from datetime import datetime
 
+from .__init__ import unicode
 from .util import s3dec, Pebkac, min_ex
+from .bos import bos
 from .up2k import up2k_wark_from_hashlist
 
 
@@ -66,7 +68,7 @@ class U2idx(object):
 
         histpath = self.asrv.vfs.histtab[ptop]
         db_path = os.path.join(histpath, "up2k.db")
-        if not os.path.exists(db_path):
+        if not bos.path.exists(db_path):
             return None
 
         cur = sqlite3.connect(db_path, 2).cursor()
@@ -90,6 +92,8 @@ class U2idx(object):
         mt_ctr = 0
         mt_keycmp = "substr(up.w,1,16)"
         mt_keycmp2 = None
+        ptn_lc = re.compile(r" (mt[0-9]+\.v) ([=<!>]+) \? $")
+        ptn_lcv = re.compile(r"[a-zA-Z]")
 
         while True:
             uq = uq.strip()
@@ -182,6 +186,21 @@ class U2idx(object):
             va.append(v)
             is_key = True
 
+            # lowercase tag searches
+            m = ptn_lc.search(q)
+            if not m or not ptn_lcv.search(unicode(v)):
+                continue
+
+            va.pop()
+            va.append(v.lower())
+            q = q[: m.start()]
+
+            field, oper = m.groups()
+            if oper in ["=", "=="]:
+                q += " {} like ? ".format(field)
+            else:
+                q += " lower({}) {} ? ".format(field, oper)
+
         try:
             return self.run_query(vols, joins + "where " + q, va)
         except Exception as ex:
@@ -225,7 +244,7 @@ class U2idx(object):
             sret = []
             c = cur.execute(q, v)
             for hit in c:
-                w, ts, sz, rd, fn = hit
+                w, ts, sz, rd, fn, ip, at = hit
                 lim -= 1
                 if lim <= 0:
                     break

copyparty/up2k.py

@@ -23,15 +23,20 @@ from .util import (
     ProgressPrinter,
     fsdec,
     fsenc,
+    absreal,
     sanitize_fn,
     ren_open,
     atomic_move,
+    vsplit,
     s3enc,
     s3dec,
+    rmdirs,
     statdir,
     s2hms,
     min_ex,
 )
+from .bos import bos
+from .authsrv import AuthSrv
 from .mtag import MTag, MParser
 
 try:
@@ -40,20 +45,13 @@ try:
 except:
     HAVE_SQLITE3 = False
 
-DB_VER = 4
+DB_VER = 5
 
 
 class Up2k(object):
-    """
-    TODO:
-      * documentation
-      * registry persistence
-        * ~/.config flatfiles for active jobs
-    """
-
     def __init__(self, hub):
         self.hub = hub
-        self.asrv = hub.asrv
+        self.asrv = hub.asrv  # type: AuthSrv
         self.args = hub.args
         self.log_func = hub.log
 
@@ -67,6 +65,7 @@ class Up2k(object):
         self.n_hashq = 0
         self.n_tagq = 0
         self.volstate = {}
+        self.need_rescan = {}
         self.registry = {}
         self.entags = {}
         self.flags = {}
@@ -101,17 +100,16 @@ class Up2k(object):
 
         if self.args.no_fastboot:
             self.deferred_init()
-        else:
-            t = threading.Thread(
-                target=self.deferred_init, name="up2k-deferred-init", args=(0.5,)
-            )
-            t.daemon = True
-            t.start()
 
-    def deferred_init(self, wait=0):
+    def init_vols(self):
-        if wait:
+        if self.args.no_fastboot:
-            time.sleep(wait)
+            return
 
+        t = threading.Thread(target=self.deferred_init, name="up2k-deferred-init")
+        t.daemon = True
+        t.start()
+
+    def deferred_init(self):
         all_vols = self.asrv.vfs.all_vols
         have_e2d = self.init_indexes(all_vols)
 
@@ -124,6 +122,10 @@ class Up2k(object):
             thr.daemon = True
             thr.start()
 
+            thr = threading.Thread(target=self._sched_rescan, name="up2k-rescan")
+            thr.daemon = True
+            thr.start()
+
             if self.mtag:
                 thr = threading.Thread(target=self._tagger, name="up2k-tagger")
                 thr.daemon = True
@@ -173,6 +175,38 @@ class Up2k(object):
         t.start()
         return None
 
+    def _sched_rescan(self):
+        maxage = self.args.re_maxage
+        volage = {}
+        while True:
+            time.sleep(self.args.re_int)
+            now = time.time()
+            vpaths = list(sorted(self.asrv.vfs.all_vols.keys()))
+            with self.mutex:
+                if maxage:
+                    for vp in vpaths:
+                        if vp not in volage:
+                            volage[vp] = now
+
+                        if now - volage[vp] >= maxage:
+                            self.need_rescan[vp] = 1
+
+                if not self.need_rescan:
+                    continue
+
+                vols = list(sorted(self.need_rescan.keys()))
+                self.need_rescan = {}
+
+            err = self.rescan(self.asrv.vfs.all_vols, vols)
+            if err:
+                for v in vols:
+                    self.need_rescan[v] = True
+
+                continue
+
+            for v in vols:
+                volage[v] = now
+
     def _vis_job_progress(self, job):
         perc = 100 - (len(job["need"]) * 100.0 / len(job["hash"]))
         path = os.path.join(job["ptop"], job["prel"], job["name"])
@@ -218,7 +252,7 @@ class Up2k(object):
             # only need to protect register_vpath but all in one go feels right
             for vol in vols:
                 try:
-                    os.listdir(vol.realpath)
+                    bos.listdir(vol.realpath)
                 except:
                     self.volstate[vol.vpath] = "OFFLINE (cannot access folder)"
                     self.log("cannot access " + vol.realpath, c=1)
@@ -304,7 +338,7 @@ class Up2k(object):
         self.log(msg.format(len(vols), time.time() - t0))
 
         if needed_mutagen:
-            msg = "could not read tags because no backends are available (mutagen or ffprobe)"
+            msg = "could not read tags because no backends are available (Mutagen or FFprobe)"
             self.log(msg, c=1)
 
         thr = None
@@ -356,14 +390,14 @@ class Up2k(object):
 
         reg = {}
         path = os.path.join(histpath, "up2k.snap")
-        if "e2d" in flags and os.path.exists(path):
+        if "e2d" in flags and bos.path.exists(path):
             with gzip.GzipFile(path, "rb") as f:
                 j = f.read().decode("utf-8")
 
             reg2 = json.loads(j)
             for k, job in reg2.items():
                 path = os.path.join(job["ptop"], job["prel"], job["name"])
-                if os.path.exists(fsenc(path)):
+                if bos.path.exists(path):
                     reg[k] = job
                     job["poke"] = time.time()
                 else:
@@ -378,10 +412,7 @@ class Up2k(object):
         if not HAVE_SQLITE3 or "e2d" not in flags or "d2d" in flags:
             return None
 
-        try:
+        bos.makedirs(histpath)
-            os.makedirs(histpath)
-        except:
-            pass
 
         try:
             cur = self._open_db(db_path)
@@ -420,14 +451,7 @@ class Up2k(object):
             return True, n_add or n_rm or do_vac
 
     def _build_dir(self, dbw, top, excl, cdir, nohash, seen):
-        rcdir = cdir
+        rcdir = absreal(cdir)  # a bit expensive but worth
-        if not ANYWIN:
-            try:
-                # a bit expensive but worth
-                rcdir = os.path.realpath(cdir)
-            except:
-                pass
-
         if rcdir in seen:
             m = "bailing from symlink loop,\n  prev: {}\n  curr: {}\n  from: {}"
             self.log(m.format(seen[-1], rcdir, cdir), 3)
@@ -498,7 +522,7 @@ class Up2k(object):
 
                     wark = up2k_wark_from_hashlist(self.salt, sz, hashes)
 
-                self.db_add(dbw[0], wark, rd, fn, lmod, sz)
+                self.db_add(dbw[0], wark, rd, fn, lmod, sz, "", 0)
                 dbw[1] += 1
                 ret += 1
                 td = time.time() - dbw[2]
@@ -513,8 +537,8 @@ class Up2k(object):
         rm = []
         nchecked = 0
         nfiles = next(cur.execute("select count(w) from up"))[0]
-        c = cur.execute("select * from up")
+        c = cur.execute("select rd, fn from up")
-        for dwark, dts, dsz, drd, dfn in c:
+        for drd, dfn in c:
             nchecked += 1
             if drd.startswith("//") or dfn.startswith("//"):
                 drd, dfn = s3dec(drd, dfn)
@@ -523,7 +547,7 @@ class Up2k(object):
             # almost zero overhead dw
             self.pp.msg = "b{} {}".format(nfiles - nchecked, abspath)
             try:
-                if not os.path.exists(fsenc(abspath)):
+                if not bos.path.exists(abspath):
                     rm.append([drd, dfn])
             except Exception as ex:
                 self.log("stat-rm: {} @ [{}]".format(repr(ex), abspath))
@@ -596,7 +620,7 @@ class Up2k(object):
             c2 = conn.cursor()
             c3 = conn.cursor()
             n_left = cur.execute("select count(w) from up").fetchone()[0]
-            for w, rd, fn in cur.execute("select w, rd, fn from up"):
+            for w, rd, fn in cur.execute("select w, rd, fn from up order by rd, fn"):
                 n_left -= 1
                 q = "select w from mt where w = ?"
                 if c2.execute(q, (w[:16],)).fetchone():
@@ -911,12 +935,21 @@ class Up2k(object):
         # x.set_trace_callback(trace)
 
     def _open_db(self, db_path):
-        existed = os.path.exists(db_path)
+        existed = bos.path.exists(db_path)
         cur = self._orz(db_path)
         ver = self._read_ver(cur)
         if not existed and ver is None:
             return self._create_db(db_path, cur)
 
+        if ver == 4:
+            try:
+                m = "creating backup before upgrade: "
+                cur = self._backup_db(db_path, cur, ver, m)
+                self._upgrade_v4(cur)
+                ver = 5
+            except:
+                self.log("WARN: failed to upgrade from v4", 3)
+
         if ver == DB_VER:
             try:
                 nfiles = next(cur.execute("select count(w) from up"))[0]
@@ -929,19 +962,38 @@ class Up2k(object):
             m = "database is version {}, this copyparty only supports versions <= {}"
             raise Exception(m.format(ver, DB_VER))
 
-        bak = "{}.bak.{:x}.v{}".format(db_path, int(time.time()), ver)
-        db = cur.connection
-        cur.close()
-        db.close()
         msg = "creating new DB (old is bad); backup: {}"
         if ver:
             msg = "creating new DB (too old to upgrade); backup: {}"
 
-        self.log(msg.format(bak))
+        cur = self._backup_db(db_path, cur, ver, msg)
-        os.rename(fsenc(db_path), fsenc(bak))
+        db = cur.connection
-
+        cur.close()
+        db.close()
+        bos.unlink(db_path)
         return self._create_db(db_path, None)
 
+    def _backup_db(self, db_path, cur, ver, msg):
+        bak = "{}.bak.{:x}.v{}".format(db_path, int(time.time()), ver)
+        self.log(msg + bak)
+        try:
+            c2 = sqlite3.connect(bak)
+            with c2:
+                cur.connection.backup(c2)
+            return cur
+        except:
+            m = "native sqlite3 backup failed; using fallback method:\n"
+            self.log(m + min_ex())
+        finally:
+            c2.close()
+
+        db = cur.connection
+        cur.close()
+        db.close()
+
+        shutil.copy2(fsenc(db_path), fsenc(bak))
+        return self._orz(db_path)
+
     def _read_ver(self, cur):
         for tab in ["ki", "kv"]:
             try:
@@ -968,9 +1020,10 @@ class Up2k(object):
             idx = r"create index up_w on up(w)"
 
         for cmd in [
-            r"create table up (w text, mt int, sz int, rd text, fn text)",
+            r"create table up (w text, mt int, sz int, rd text, fn text, ip text, at int)",
             r"create index up_rd on up(rd)",
             r"create index up_fn on up(fn)",
+            r"create index up_ip on up(ip)",
             idx,
             r"create table mt (w text, k text, v int)",
             r"create index mt_w on mt(w)",
@@ -985,6 +1038,17 @@ class Up2k(object):
         self.log("created DB at {}".format(db_path))
         return cur
 
+    def _upgrade_v4(self, cur):
+        for cmd in [
+            r"alter table up add column ip text",
+            r"alter table up add column at int",
+            r"create index up_ip on up(ip)",
+            r"update kv set v=5 where k='sver'",
+        ]:
+            cur.execute(cmd)
+
+        cur.connection.commit()
+
     def handle_json(self, cj):
         with self.mutex:
             if not self.register_vpath(cj["ptop"], cj["vcfg"]):
@@ -1008,13 +1072,13 @@ class Up2k(object):
                     argv = (wark[:16], wark)
 
                 cur = cur.execute(q, argv)
-                for _, dtime, dsize, dp_dir, dp_fn in cur:
+                for _, dtime, dsize, dp_dir, dp_fn, ip, at in cur:
                     if dp_dir.startswith("//") or dp_fn.startswith("//"):
                         dp_dir, dp_fn = s3dec(dp_dir, dp_fn)
 
                     dp_abs = "/".join([cj["ptop"], dp_dir, dp_fn])
                     # relying on path.exists to return false on broken symlinks
-                    if os.path.exists(fsenc(dp_abs)):
+                    if bos.path.exists(dp_abs):
                         job = {
                             "name": dp_fn,
                             "prel": dp_dir,
@@ -1022,6 +1086,8 @@ class Up2k(object):
                             "ptop": cj["ptop"],
                             "size": dsize,
                             "lmod": dtime,
+                            "addr": ip,
+                            "at": at,
                             "hash": [],
                             "need": [],
                         }
@@ -1038,7 +1104,7 @@ class Up2k(object):
                     for fn in names:
                         path = os.path.join(job["ptop"], job["prel"], fn)
                         try:
-                            if os.path.getsize(fsenc(path)) > 0:
+                            if bos.path.getsize(path) > 0:
                                 # upload completed or both present
                                 break
                         except:
@@ -1072,9 +1138,15 @@ class Up2k(object):
                         job["name"] = self._untaken(pdir, cj["name"], now, cj["addr"])
                         dst = os.path.join(job["ptop"], job["prel"], job["name"])
                         if not self.args.nw:
-                            os.unlink(fsenc(dst))  # TODO ed pls
+                            bos.unlink(dst)  # TODO ed pls
                             self._symlink(src, dst)
 
+                        if cur:
+                            a = [cj[x] for x in "prel name lmod size addr".split()]
+                            a += [cj.get("at") or time.time()]
+                            self.db_add(cur, wark, *a)
+                            cur.connection.commit()
+
             if not job:
                 job = {
                     "wark": wark,
@@ -1124,17 +1196,18 @@ class Up2k(object):
         with ren_open(fname, "wb", fdir=fdir, suffix=suffix) as f:
             return f["orz"][1]
 
-    def _symlink(self, src, dst):
+    def _symlink(self, src, dst, verbose=True):
-        # TODO store this in linktab so we never delete src if there are links to it
+        if verbose:
-        self.log("linking dupe:\n  {0}\n  {1}".format(src, dst))
+            self.log("linking dupe:\n  {0}\n  {1}".format(src, dst))
+
         if self.args.nw:
             return
 
         try:
             lsrc = src
             ldst = dst
-            fs1 = os.stat(fsenc(os.path.split(src)[0])).st_dev
+            fs1 = bos.stat(os.path.dirname(src)).st_dev
-            fs2 = os.stat(fsenc(os.path.split(dst)[0])).st_dev
+            fs2 = bos.stat(os.path.dirname(dst)).st_dev
             if fs1 == 0:
                 # py2 on winxp or other unsupported combination
                 raise OSError()
@@ -1217,27 +1290,21 @@ class Up2k(object):
                 a = [dst, job["size"], (int(time.time()), int(job["lmod"]))]
                 self.lastmod_q.put(a)
 
-            # legit api sware 2 me mum
+            a = [job[x] for x in "ptop wark prel name lmod size addr".split()]
-            if self.idx_wark(
+            a += [job.get("at") or time.time()]
-                job["ptop"],
+            if self.idx_wark(*a):
-                job["wark"],
-                job["prel"],
-                job["name"],
-                job["lmod"],
-                job["size"],
-            ):
                 del self.registry[ptop][wark]
                 # in-memory registry is reserved for unfinished uploads
 
         return ret, dst
 
-    def idx_wark(self, ptop, wark, rd, fn, lmod, sz):
+    def idx_wark(self, ptop, wark, rd, fn, lmod, sz, ip, at):
         cur = self.cur.get(ptop)
         if not cur:
             return False
 
         self.db_rm(cur, rd, fn)
-        self.db_add(cur, wark, rd, fn, int(lmod), sz)
+        self.db_add(cur, wark, rd, fn, lmod, sz, ip, at)
         cur.connection.commit()
 
         if "e2t" in self.flags[ptop]:
@@ -1253,16 +1320,308 @@ class Up2k(object):
         except:
             db.execute(sql, s3enc(self.mem_cur, rd, fn))
 
-    def db_add(self, db, wark, rd, fn, ts, sz):
+    def db_add(self, db, wark, rd, fn, ts, sz, ip, at):
-        sql = "insert into up values (?,?,?,?,?)"
+        sql = "insert into up values (?,?,?,?,?,?,?)"
-        v = (wark, int(ts), sz, rd, fn)
+        v = (wark, int(ts), sz, rd, fn, ip or "", int(at or 0))
         try:
             db.execute(sql, v)
         except:
             rd, fn = s3enc(self.mem_cur, rd, fn)
-            v = (wark, ts, sz, rd, fn)
+            v = (wark, int(ts), sz, rd, fn, ip or "", int(at or 0))
             db.execute(sql, v)
 
+    def handle_rm(self, uname, ip, vpaths):
+        n_files = 0
+        ok = {}
+        ng = {}
+        for vp in vpaths:
+            a, b, c = self._handle_rm(uname, ip, vp)
+            n_files += a
+            for k in b:
+                ok[k] = 1
+            for k in c:
+                ng[k] = 1
+
+        ng = {k: 1 for k in ng if k not in ok}
+        ok = len(ok)
+        ng = len(ng)
+
+        return "deleted {} files (and {}/{} folders)".format(n_files, ok, ok + ng)
+
+    def _handle_rm(self, uname, ip, vpath):
+        try:
+            permsets = [[True, False, False, True]]
+            vn, rem = self.asrv.vfs.get(vpath, uname, *permsets[0])
+            unpost = False
+        except:
+            # unpost with missing permissions? try read+write and verify with db
+            if not self.args.unpost:
+                raise Pebkac(400, "the unpost feature was disabled by server config")
+
+            unpost = True
+            permsets = [[True, True]]
+            vn, rem = self.asrv.vfs.get(vpath, uname, *permsets[0])
+            _, _, _, _, dip, dat = self._find_from_vpath(vn.realpath, rem)
+
+            m = "you cannot delete this: "
+            if not dip:
+                m += "file not found"
+            elif dip != ip:
+                m += "not uploaded by (You)"
+            elif dat < time.time() - self.args.unpost:
+                m += "uploaded too long ago"
+            else:
+                m = None
+
+            if m:
+                raise Pebkac(400, m)
+
+        ptop = vn.realpath
+        atop = vn.canonical(rem, False)
+        adir, fn = os.path.split(atop)
+        st = bos.lstat(atop)
+        scandir = not self.args.no_scandir
+        if stat.S_ISLNK(st.st_mode) or stat.S_ISREG(st.st_mode):
+            dbv, vrem = self.asrv.vfs.get(vpath, uname, *permsets[0])
+            dbv, vrem = dbv.get_dbv(vrem)
+            voldir = vsplit(vrem)[0]
+            vpath_dir = vsplit(vpath)[0]
+            g = [[dbv, voldir, vpath_dir, adir, [[fn, 0]], [], []]]
+        else:
+            g = vn.walk("", rem, [], uname, permsets, True, scandir, True)
+            if unpost:
+                raise Pebkac(400, "cannot unpost folders")
+
+        n_files = 0
+        for dbv, vrem, _, adir, files, rd, vd in g:
+            for fn in [x[0] for x in files]:
+                n_files += 1
+                abspath = os.path.join(adir, fn)
+                volpath = "{}/{}".format(vrem, fn).strip("/")
+                vpath = "{}/{}".format(dbv.vpath, volpath).strip("/")
+                self.log("rm {}\n  {}".format(vpath, abspath))
+                _ = dbv.get(volpath, uname, *permsets[0])
+                with self.mutex:
+                    try:
+                        ptop = dbv.realpath
+                        cur, wark, _, _, _, _ = self._find_from_vpath(ptop, volpath)
+                        self._forget_file(ptop, volpath, cur, wark, True)
+                    finally:
+                        cur.connection.commit()
+
+                bos.unlink(abspath)
+
+        rm = rmdirs(self.log_func, scandir, True, atop)
+        return n_files, rm[0], rm[1]
+
+    def handle_mv(self, uname, svp, dvp):
+        svn, srem = self.asrv.vfs.get(svp, uname, True, False, True)
+        svn, srem = svn.get_dbv(srem)
+        sabs = svn.canonical(srem, False)
+
+        if not srem:
+            raise Pebkac(400, "mv: cannot move a mountpoint")
+
+        st = bos.stat(sabs)
+        if stat.S_ISREG(st.st_mode):
+            return self._mv_file(uname, svp, dvp)
+
+        jail = svn.get_dbv(srem)[0]
+        permsets = [[True, False, True]]
+        scandir = not self.args.no_scandir
+
+        # following symlinks is too scary
+        g = svn.walk("", srem, [], uname, permsets, True, scandir, True)
+        for dbv, vrem, _, atop, files, rd, vd in g:
+            if dbv != jail:
+                # fail early (prevent partial moves)
+                raise Pebkac(400, "mv: source folder contains other volumes")
+
+        g = svn.walk("", srem, [], uname, permsets, True, scandir, True)
+        for dbv, vrem, _, atop, files, rd, vd in g:
+            if dbv != jail:
+                # the actual check (avoid toctou)
+                raise Pebkac(400, "mv: source folder contains other volumes")
+
+            for fn in files:
+                svpf = "/".join(x for x in [dbv.vpath, vrem, fn[0]] if x)
+                if not svpf.startswith(svp + "/"):  # assert
+                    raise Pebkac(500, "mv: bug at {}, top {}".format(svpf, svp))
+
+                dvpf = dvp + svpf[len(svp) :]
+                self._mv_file(uname, svpf, dvpf)
+
+        rmdirs(self.log_func, scandir, True, sabs)
+        return "k"
+
+    def _mv_file(self, uname, svp, dvp):
+        svn, srem = self.asrv.vfs.get(svp, uname, True, False, True)
+        svn, srem = svn.get_dbv(srem)
+
+        dvn, drem = self.asrv.vfs.get(dvp, uname, False, True)
+        dvn, drem = dvn.get_dbv(drem)
+
+        sabs = svn.canonical(srem, False)
+        dabs = dvn.canonical(drem)
+        drd, dfn = vsplit(drem)
+
+        if bos.path.exists(dabs):
+            raise Pebkac(400, "mv2: target file exists")
+
+        bos.makedirs(os.path.dirname(dabs))
+
+        if bos.path.islink(sabs):
+            dlabs = absreal(sabs)
+            m = "moving symlink from [{}] to [{}], target [{}]"
+            self.log(m.format(sabs, dabs, dlabs))
+            os.unlink(sabs)
+            self._symlink(dlabs, dabs, False)
+
+            # folders are too scary, schedule rescan of both vols
+            self.need_rescan[svn.vpath] = 1
+            self.need_rescan[dvn.vpath] = 1
+            return "k"
+
+        c1, w, ftime, fsize, ip, at = self._find_from_vpath(svn.realpath, srem)
+        c2 = self.cur.get(dvn.realpath)
+
+        if ftime is None:
+            st = bos.stat(sabs)
+            ftime = st.st_mtime
+            fsize = st.st_size
+
+        if w:
+            if c2 and c2 != c1:
+                self._copy_tags(c1, c2, w)
+
+            self._forget_file(svn.realpath, srem, c1, w, c1 != c2)
+            self._relink(w, svn.realpath, srem, dabs)
+            c1.connection.commit()
+
+            if c2:
+                self.db_add(c2, w, drd, dfn, ftime, fsize, ip, at)
+                c2.connection.commit()
+        else:
+            self.log("not found in src db: [{}]".format(svp))
+
+        bos.rename(sabs, dabs)
+        return "k"
+
+    def _copy_tags(self, csrc, cdst, wark):
+        """copy all tags for wark from src-db to dst-db"""
+        w = wark[:16]
+
+        if cdst.execute("select * from mt where w=? limit 1", (w,)).fetchone():
+            return  # existing tags in dest db
+
+        for _, k, v in csrc.execute("select * from mt where w=?", (w,)):
+            cdst.execute("insert into mt values(?,?,?)", (w, k, v))
+
+    def _find_from_vpath(self, ptop, vrem):
+        cur = self.cur.get(ptop)
+        if not cur:
+            return None, None
+
+        rd, fn = vsplit(vrem)
+        q = "select w, mt, sz, ip, at from up where rd=? and fn=? limit 1"
+        try:
+            c = cur.execute(q, (rd, fn))
+        except:
+            c = cur.execute(q, s3enc(self.mem_cur, rd, fn))
+
+        hit = c.fetchone()
+        if hit:
+            wark, ftime, fsize, ip, at = hit
+            return cur, wark, ftime, fsize, ip, at
+        return cur, None, None, None, None, None
+
+    def _forget_file(self, ptop, vrem, cur, wark, drop_tags):
+        """forgets file in db, fixes symlinks, does not delete"""
+        srd, sfn = vsplit(vrem)
+        self.log("forgetting {}".format(vrem))
+        if wark:
+            self.log("found {} in db".format(wark))
+            if self._relink(wark, ptop, vrem, None):
+                drop_tags = False
+
+            if drop_tags:
+                q = "delete from mt where w=?"
+                cur.execute(q, (wark[:16],))
+                self.db_rm(cur, srd, sfn)
+
+        reg = self.registry.get(ptop)
+        if reg:
+            if not wark:
+                wark = [
+                    x
+                    for x, y in reg.items()
+                    if fn in [y["name"], y.get("tnam")] and y["prel"] == vrem
+                ]
+
+            if wark and wark in reg:
+                m = "forgetting partial upload {} ({})"
+                p = self._vis_job_progress(wark)
+                self.log(m.format(wark, p))
+                del reg[wark]
+
+    def _relink(self, wark, sptop, srem, dabs):
+        """
+        update symlinks from file at svn/srem to dabs (rename),
+        or to first remaining full if no dabs (delete)
+        """
+        dupes = []
+        sabs = os.path.join(sptop, srem)
+        q = "select rd, fn from up where substr(w,1,16)=? and w=?"
+        for ptop, cur in self.cur.items():
+            for rd, fn in cur.execute(q, (wark[:16], wark)):
+                if rd.startswith("//") or fn.startswith("//"):
+                    rd, fn = s3dec(rd, fn)
+
+                dvrem = "/".join([rd, fn]).strip("/")
+                if ptop != sptop or srem != dvrem:
+                    dupes.append([ptop, dvrem])
+                    self.log("found {} dupe: [{}] {}".format(wark, ptop, dvrem))
+
+        if not dupes:
+            return 0
+
+        full = {}
+        links = {}
+        for ptop, vp in dupes:
+            ap = os.path.join(ptop, vp)
+            try:
+                d = links if bos.path.islink(ap) else full
+                d[ap] = [ptop, vp]
+            except:
+                self.log("relink: not found: [{}]".format(ap))
+
+        if not dabs and not full and links:
+            # deleting final remaining full copy; swap it with a symlink
+            slabs = list(sorted(links.keys()))[0]
+            ptop, rem = links.pop(slabs)
+            self.log("linkswap [{}] and [{}]".format(sabs, dabs))
+            bos.unlink(slabs)
+            bos.rename(sabs, slabs)
+            self._symlink(slabs, sabs, False)
+            full[slabs] = [ptop, rem]
+
+        if not dabs:
+            dabs = list(sorted(full.keys()))[0]
+
+        for alink in links.keys():
+            try:
+                if alink != sabs and absreal(alink) != sabs:
+                    continue
+
+                self.log("relinking [{}] to [{}]".format(alink, dabs))
+                bos.unlink(alink)
+            except:
+                pass
+
+            self._symlink(dabs, alink, False)
+
+        return len(full) + len(links)
+
     def _get_wark(self, cj):
         if len(cj["name"]) > 1024 or len(cj["hash"]) > 512 * 1024:  # 16TiB
             raise Pebkac(400, "name or numchunks not according to spec")
@@ -1284,7 +1643,7 @@ class Up2k(object):
 
     def _hashlist_from_file(self, path):
         pp = self.pp if hasattr(self, "pp") else None
-        fsz = os.path.getsize(fsenc(path))
+        fsz = bos.path.getsize(path)
         csz = up2k_chunksize(fsz)
         ret = []
         with open(fsenc(path), "rb", 512 * 1024) as f:
@@ -1352,7 +1711,7 @@ class Up2k(object):
             for path, sz, times in ready:
                 self.log("lmod: setting times {} on {}".format(times, path))
                 try:
-                    os.utime(fsenc(path), times)
+                    bos.utime(path, times)
                 except:
                     self.log("lmod: failed to utime ({}, {})".format(path, times))
 
@@ -1363,19 +1722,22 @@ class Up2k(object):
                         self.log("could not unsparse [{}]".format(path), 3)
 
     def _snapshot(self):
-        persist_interval = 30  # persist unfinished uploads index every 30 sec
+        self.snap_persist_interval = 300  # persist unfinished index every 5 min
-        discard_interval = 21600  # drop unfinished uploads after 6 hours inactivity
+        self.snap_discard_interval = 21600  # drop unfinished after 6 hours inactivity
-        prev = {}
+        self.snap_prev = {}
         while True:
-            time.sleep(persist_interval)
+            time.sleep(self.snap_persist_interval)
-            with self.mutex:
+            self.do_snapshot()
-                for k, reg in self.registry.items():
-                    self._snap_reg(prev, k, reg, discard_interval)
 
-    def _snap_reg(self, prev, ptop, reg, discard_interval):
+    def do_snapshot(self):
+        with self.mutex:
+            for k, reg in self.registry.items():
+                self._snap_reg(k, reg)
+
+    def _snap_reg(self, ptop, reg):
         now = time.time()
         histpath = self.asrv.vfs.histtab[ptop]
-        rm = [x for x in reg.values() if now - x["poke"] > discard_interval]
+        rm = [x for x in reg.values() if now - x["poke"] > self.snap_discard_interval]
         if rm:
             m = "dropping {} abandoned uploads in {}".format(len(rm), ptop)
             vis = [self._vis_job_progress(x) for x in rm]
@@ -1385,33 +1747,30 @@ class Up2k(object):
                 try:
                     # remove the filename reservation
                     path = os.path.join(job["ptop"], job["prel"], job["name"])
-                    if os.path.getsize(fsenc(path)) == 0:
+                    if bos.path.getsize(path) == 0:
-                        os.unlink(fsenc(path))
+                        bos.unlink(path)
 
                     if len(job["hash"]) == len(job["need"]):
                         # PARTIAL is empty, delete that too
                         path = os.path.join(job["ptop"], job["prel"], job["tnam"])
-                        os.unlink(fsenc(path))
+                        bos.unlink(path)
                 except:
                     pass
 
         path = os.path.join(histpath, "up2k.snap")
         if not reg:
-            if ptop not in prev or prev[ptop] is not None:
+            if ptop not in self.snap_prev or self.snap_prev[ptop] is not None:
-                prev[ptop] = None
+                self.snap_prev[ptop] = None
-                if os.path.exists(fsenc(path)):
+                if bos.path.exists(path):
-                    os.unlink(fsenc(path))
+                    bos.unlink(path)
             return
 
         newest = max(x["poke"] for _, x in reg.items()) if reg else 0
         etag = [len(reg), newest]
-        if etag == prev.get(ptop):
+        if etag == self.snap_prev.get(ptop):
             return
 
-        try:
+        bos.makedirs(histpath)
-            os.makedirs(histpath)
-        except:
-            pass
 
         path2 = "{}.{}".format(path, os.getpid())
         j = json.dumps(reg, indent=2, sort_keys=True).encode("utf-8")
@@ -1421,7 +1780,7 @@ class Up2k(object):
         atomic_move(path2, path)
 
         self.log("snap: {} |{}|".format(path, len(reg.keys())))
-        prev[ptop] = etag
+        self.snap_prev[ptop] = etag
 
     def _tagger(self):
         with self.mutex:
@@ -1469,26 +1828,31 @@ class Up2k(object):
                 self.n_hashq -= 1
             # self.log("hashq {}".format(self.n_hashq))
 
-            ptop, rd, fn = self.hashq.get()
+            ptop, rd, fn, ip, at = self.hashq.get()
             # self.log("hashq {} pop {}/{}/{}".format(self.n_hashq, ptop, rd, fn))
             if "e2d" not in self.flags[ptop]:
                 continue
 
             abspath = os.path.join(ptop, rd, fn)
             self.log("hashing " + abspath)
-            inf = os.stat(fsenc(abspath))
+            inf = bos.stat(abspath)
             hashes = self._hashlist_from_file(abspath)
             wark = up2k_wark_from_hashlist(self.salt, inf.st_size, hashes)
             with self.mutex:
-                self.idx_wark(ptop, wark, rd, fn, inf.st_mtime, inf.st_size)
+                self.idx_wark(ptop, wark, rd, fn, inf.st_mtime, inf.st_size, ip, at)
 
-    def hash_file(self, ptop, flags, rd, fn):
+    def hash_file(self, ptop, flags, rd, fn, ip, at):
         with self.mutex:
             self.register_vpath(ptop, flags)
-            self.hashq.put([ptop, rd, fn])
+            self.hashq.put([ptop, rd, fn, ip, at])
             self.n_hashq += 1
         # self.log("hashq {} push {}/{}/{}".format(self.n_hashq, ptop, rd, fn))
 
+    def shutdown(self):
+        if hasattr(self, "snap_prev"):
+            self.log("writing snapshot")
+            self.do_snapshot()
+
 
 def up2k_chunksize(filesize):
     chunksize = 1024 * 1024
@@ -1504,7 +1868,7 @@ def up2k_chunksize(filesize):
 
 
 def up2k_wark_from_hashlist(salt, filesize, hashes):
-    """ server-reproducible file identifier, independent of name or location """
+    """server-reproducible file identifier, independent of name or location"""
     ident = [salt, str(filesize)]
     ident.extend(hashes)
     ident = "\n".join(ident)

copyparty/util.py

@@ -4,6 +4,7 @@ from __future__ import print_function, unicode_literals
 import re
 import os
 import sys
+import stat
 import time
 import base64
 import select
@@ -758,6 +759,19 @@ def sanitize_fn(fn, ok, bad):
     return fn.strip()
 
 
+def absreal(fpath):
+    try:
+        return fsdec(os.path.abspath(os.path.realpath(fsenc(fpath))))
+    except:
+        if not WINDOWS:
+            raise
+
+        # cpython bug introduced in 3.8, still exists in 3.9.1,
+        # some win7sp1 and win10:20H2 boxes cannot realpath a
+        # networked drive letter such as b"n:" or b"n:\\"
+        return os.path.abspath(os.path.realpath(fpath))
+
+
 def u8safe(txt):
     try:
         return txt.encode("utf-8", "xmlcharrefreplace").decode("utf-8", "replace")
@@ -815,6 +829,13 @@ def unquotep(txt):
     return w8dec(unq2)
 
 
+def vsplit(vpath):
+    if "/" not in vpath:
+        return "", vpath
+
+    return vpath.rsplit("/", 1)
+
+
 def w8dec(txt):
     """decodes filesystem-bytes to wtf8"""
     if PY2:
@@ -1014,6 +1035,9 @@ def sendfile_kern(lower, upper, f, s):
 
 
 def statdir(logger, scandir, lstat, top):
+    if lstat and not os.supports_follow_symlinks:
+        scandir = False
+
     try:
         btop = fsenc(top)
         if scandir and hasattr(os, "scandir"):
@@ -1038,6 +1062,29 @@ def statdir(logger, scandir, lstat, top):
         logger(src, "{} @ {}".format(repr(ex), top), 1)
 
 
+def rmdirs(logger, scandir, lstat, top):
+    if not os.path.exists(fsenc(top)) or not os.path.isdir(fsenc(top)):
+        top = os.path.dirname(top)
+    
+    dirs = statdir(logger, scandir, lstat, top)
+    dirs = [x[0] for x in dirs if stat.S_ISDIR(x[1].st_mode)]
+    dirs = [os.path.join(top, x) for x in dirs]
+    ok = []
+    ng = []
+    for d in dirs[::-1]:
+        a, b = rmdirs(logger, scandir, lstat, d)
+        ok += a
+        ng += b
+
+    try:
+        os.rmdir(fsenc(top))
+        ok.append(top)
+    except:
+        ng.append(top)
+
+    return ok, ng
+
+
 def unescape_cookie(orig):
     # mw=idk; doot=qwe%2Crty%3Basd+fgh%2Bjkl%25zxc%26vbn  # qwe,rty;asd fgh+jkl%zxc&vbn
     ret = ""
@@ -1081,7 +1128,7 @@ def guess_mime(url, fallback="application/octet-stream"):
     return ret
 
 
-def runcmd(*argv):
+def runcmd(argv):
     p = sp.Popen(argv, stdout=sp.PIPE, stderr=sp.PIPE)
     stdout, stderr = p.communicate()
     stdout = stdout.decode("utf-8", "replace")
@@ -1089,8 +1136,8 @@ def runcmd(*argv):
     return [p.returncode, stdout, stderr]
 
 
-def chkcmd(*argv):
+def chkcmd(argv):
-    ok, sout, serr = runcmd(*argv)
+    ok, sout, serr = runcmd(argv)
     if ok != 0:
         raise Exception(serr)
 

copyparty/web/baguettebox.js

@@ -22,7 +22,7 @@ window.baguetteBox = (function () {
             afterHide: null,
             onChange: null,
         },
-        overlay, slider, btnPrev, btnNext, btnVmode, btnClose,
+        overlay, slider, btnPrev, btnNext, btnHelp, btnVmode, btnClose,
         currentGallery = [],
         currentIndex = 0,
         isOverlayVisible = false,
@@ -97,7 +97,7 @@ window.baguetteBox = (function () {
     }
 
     function bindImageClickListeners(selector, userOptions) {
-        var galleryNodeList = document.querySelectorAll(selector);
+        var galleryNodeList = QSA(selector);
         var selectorData = {
             galleries: [],
             nodeList: galleryNodeList
@@ -174,22 +174,57 @@ window.baguetteBox = (function () {
                 '<button id="bbox-prev" class="bbox-btn" type="button" aria-label="Previous">&lt;</button>' +
                 '<button id="bbox-next" class="bbox-btn" type="button" aria-label="Next">&gt;</button>' +
                 '<div id="bbox-btns">' +
+                '<button id="bbox-help" type="button">?</button>' +
                 '<button id="bbox-vmode" type="button" tt="a"></button>' +
-                '<button id="bbox-close" type="button" aria-label="Close">&times;</button>' +
+                '<button id="bbox-close" type="button" aria-label="Close">X</button>' +
                 '</div></div>'
             );
             overlay = ctr.firstChild;
             QS('body').appendChild(overlay);
-            tt.init();
+            tt.att(overlay);
         }
         slider = ebi('bbox-slider');
         btnPrev = ebi('bbox-prev');
         btnNext = ebi('bbox-next');
+        btnHelp = ebi('bbox-help');
         btnVmode = ebi('bbox-vmode');
         btnClose = ebi('bbox-close');
         bindEvents();
     }
 
+    function halp() {
+        if (ebi('bbox-halp'))
+            return;
+
+        var list = [
+            ['<b># hotkey</b>', '<b># operation</b>'],
+            ['escape', 'close'],
+            ['left, J', 'previous file'],
+            ['right, L', 'next file'],
+            ['home', 'first file'],
+            ['end', 'last file'],
+            ['space, P, K', 'video: play / pause'],
+            ['U', 'video: seek 10sec back'],
+            ['P', 'video: seek 10sec ahead'],
+            ['M', 'video: toggle mute'],
+            ['R', 'video: toggle loop'],
+            ['C', 'video: toggle auto-next'],
+            ['F', 'video: toggle fullscreen'],
+        ],
+            d = mknod('table'),
+            html = ['<tbody>'];
+
+        for (var a = 0; a < list.length; a++)
+            html.push('<tr><td>' + list[a][0] + '</td><td>' + list[a][1] + '</td></tr>');
+
+        d.innerHTML = html.join('\n') + '</tbody>';
+        d.setAttribute('id', 'bbox-halp');
+        d.onclick = function () {
+            overlay.removeChild(d);
+        };
+        overlay.appendChild(d);
+    }
+
     function keyDownHandler(e) {
         if (e.ctrlKey || e.altKey || e.metaKey || e.isComposing)
             return;
@@ -240,21 +275,23 @@ window.baguetteBox = (function () {
         if (!v)
             return;
 
-        var msg = 'When video ends, ', lbl;
+        var msg = 'When video ends, ', tts = '', lbl;
         if (vloop) {
             lbl = 'Loop';
             msg += 'repeat it';
+            tts = '$NHotkey: R';
         }
         else if (vnext) {
             lbl = 'Cont';
             msg += 'continue to next';
+            tts = '$NHotkey: C';
         }
         else {
             lbl = 'Stop';
             msg += 'just stop'
         }
         btnVmode.setAttribute('aria-label', msg);
-        btnVmode.setAttribute('tt', msg);
+        btnVmode.setAttribute('tt', msg + tts);
         btnVmode.textContent = lbl;
 
         v.loop = vloop
@@ -310,6 +347,7 @@ window.baguetteBox = (function () {
         bind(btnNext, 'click', showNextImage);
         bind(btnClose, 'click', hideOverlay);
         bind(btnVmode, 'click', tglVmode);
+        bind(btnHelp, 'click', halp);
         bind(slider, 'contextmenu', contextmenuHandler);
         bind(overlay, 'touchstart', touchstartHandler, nonPassiveEvent);
         bind(overlay, 'touchmove', touchmoveHandler, passiveEvent);
@@ -323,6 +361,7 @@ window.baguetteBox = (function () {
         unbind(btnNext, 'click', showNextImage);
         unbind(btnClose, 'click', hideOverlay);
         unbind(btnVmode, 'click', tglVmode);
+        unbind(btnHelp, 'click', halp);
         unbind(slider, 'contextmenu', contextmenuHandler);
         unbind(overlay, 'touchstart', touchstartHandler, nonPassiveEvent);
         unbind(overlay, 'touchmove', touchmoveHandler, passiveEvent);
@@ -435,6 +474,10 @@ window.baguetteBox = (function () {
             if (options.bodyClass && document.body.classList)
                 document.body.classList.remove(options.bodyClass);
 
+            var h = ebi('bbox-halp');
+            if (h)
+                h.parentNode.removeChild(h);
+
             if (options.afterHide)
                 options.afterHide();
 
@@ -491,7 +534,7 @@ window.baguetteBox = (function () {
 
         image.addEventListener(is_vid ? 'loadedmetadata' : 'load', function () {
             // Remove loader element
-            var spinner = document.querySelector('#baguette-img-' + index + ' .bbox-spinner');
+            var spinner = QS('#baguette-img-' + index + ' .bbox-spinner');
             figure.removeChild(spinner);
             if (!options.async && callback)
                 callback();

copyparty/web/browser.css

@@ -25,20 +25,99 @@ html, body {
 body {
 	padding-bottom: 5em;
 }
-#tt {
+pre, code, tt {
+	font-family: monospace, monospace;
+}
+
+
+
+#tt, #toast {
 	position: fixed;
 	max-width: 34em;
 	background: #222;
 	border: 0 solid #777;
+	box-shadow: 0 .2em .5em #222;
+	border-radius: .4em;
+	z-index: 9001;
+}
+#tt {
 	overflow: hidden;
 	margin-top: 1em;
 	padding: 0 1.3em;
 	height: 0;
 	opacity: .1;
 	transition: opacity 0.14s, height 0.14s, padding 0.14s;
-	box-shadow: 0 .2em .5em #222;
+}
-	border-radius: .4em;
+#toast {
-	z-index: 9001;
+	bottom: 5em;
+	right: -1em;
+	line-height: 1.5em;
+	padding: 1em 1.3em;
+	border-width: .4em 0;
+	transform: translateX(100%);
+	transition:
+		transform .4s cubic-bezier(.2, 1.2, .5, 1),
+		right .4s cubic-bezier(.2, 1.2, .5, 1);
+	text-shadow: 1px 1px 0 #000;
+	color: #fff;
+}
+#toastc {
+	display: inline-block;
+	position: absolute;
+	overflow: hidden;
+	left: 0;
+	width: 0;
+	opacity: 0;
+	padding: .3em 0;
+	margin: -.3em 0 0 0;
+	line-height: 1.5em;
+	color: #000;
+	border: none;
+	outline: none;
+	text-shadow: none;
+	border-radius: .5em 0 0 .5em;
+	transition: left .3s, width .3s, padding .3s, opacity .3s;
+}
+#toast pre {
+	margin: 0;
+}
+#toast.vis {
+	right: 1.3em;
+	transform: unset;
+}
+#toast.vis #toastc {
+	left: -2em;
+	width: .4em;
+	padding: .3em .8em;
+	opacity: 1;
+}
+#toast.inf {
+	background: #07a;
+	border-color: #0be;
+}
+#toast.inf #toastc {
+	background: #0be;
+}
+#toast.ok {
+	background: #4a0;
+	border-color: #8e4;
+}
+#toast.ok #toastc {
+	background: #8e4;
+}
+#toast.warn {
+	background: #970;
+	border-color: #fc0;
+}
+#toast.warn #toastc {
+	background: #fc0;
+}
+#toast.err {
+	background: #900;
+	border-color: #d06;
+}
+#toast.err #toastc {
+	background: #d06;
 }
 #tt.b {
 	padding: 0 2em;
@@ -60,7 +139,6 @@ body {
 	padding: .1em .3em;
 	border-top: 1px solid #777;
 	border-radius: .3em;
-	font-family: monospace, monospace;
 	line-height: 1.7em;
 }
 #tt em {
@@ -96,6 +174,10 @@ body {
 	padding: .3em 0;
 	scroll-margin-top: 45vh;
 }
+#files tr {
+	scroll-margin-top: 25vh;
+	scroll-margin-bottom: 20vh;
+}
 #files tbody div a {
 	color: #f5a;
 }
@@ -150,8 +232,7 @@ a, #files tbody div a:last-child {
 	border-top: 1px solid #383838;
 }
 #files tbody td:nth-child(3) {
-	font-family: monospace;
+	font-family: monospace, monospace;
-	font-size: 1.3em;
 	text-align: right;
 	padding-right: 1em;
 	white-space: nowrap;
@@ -211,15 +292,31 @@ a, #files tbody div a:last-child {
 	margin: .8em 0;
 }
 #srv_info {
-	opacity: .5;
+	color: #a73;
-	font-size: .8em;
+	background: #333;
-	color: #fc5;
 	position: absolute;
-	top: .5em;
+	font-size: .8em;
+ 	top: .5em;
 	left: 2em;
+	padding-right: .5em;
 }
 #srv_info span {
-	color: #fff;
+	color: #aaa;
+}
+#acc_info {
+	position: absolute;
+	font-size: .81em;
+	top: .5em;
+	right: 2em;
+	color: #999;
+}
+#acc_info span {
+	color: #999;
+	margin-right: .6em;
+}
+#acc_info span.warn {
+	color: #f4c;
+	border-bottom: 1px solid rgba(255,68,204,0.6);
 }
 #files tbody a.play {
 	color: #e70;
@@ -246,6 +343,7 @@ html.light #ggrid a.sel {
 	border-color: #c37;
 }
 #files tbody tr.sel:hover td,
+#files tbody tr.sel:focus td,
 #ggrid a.sel:hover,
 html.light #ggrid a.sel:hover {
 	color: #fff;
@@ -280,6 +378,21 @@ html.light #ggrid a.sel {
 	color: #fff;
 	text-shadow: 0 0 1px #fff;
 }
+#files tr:focus {
+	outline: none;
+	position: relative;
+}
+#files tr:focus td {
+	background: #111;
+	border-color: #fc0 #111 #fc0 #111;
+	box-shadow: 0 .2em 0 #fc0, 0 -.2em 0 #fc0;
+}
+#files tr:focus td:first-child {
+	box-shadow: -.2em .2em 0 #fc0, -.2em -.2em 0 #fc0;
+}
+#files tr:focus+tr td {
+	border-top: 1px solid transparent;
+}
 #blocked {
 	position: fixed;
 	top: 0;
@@ -337,10 +450,18 @@ html.light #ggrid a.sel {
 	height: 100%;
 	background: #3c3c3c;
 }
+#wtgrid,
 #wtico {
 	cursor: url(/.cpr/dd/4.png), pointer;
 	animation: cursor 500ms;
+	position: relative;
+	top: -.06em;
 }
+#wtgrid {
+	font-size: .8em;
+	top: -.12em;
+}
+#wtgrid:hover,
 #wtico:hover {
 	animation: cursor 500ms infinite;
 }
@@ -356,9 +477,9 @@ html.light #ggrid a.sel {
 }
 #wtoggle {
 	position: absolute;
+	white-space: nowrap;
 	top: -1.2em;
 	right: 0;
-	width: 1.2em;
 	height: 1em;
 	font-size: 2em;
 	line-height: 1em;
@@ -367,7 +488,7 @@ html.light #ggrid a.sel {
 	background: #3c3c3c;
 	box-shadow: 0 0 .5em #222;
 	border-radius: .3em 0 0 0;
-	padding: .2em 0 0 .07em;
+	padding: .2em .2em;
 	color: #fff;
 }
 #wzip, #wnp {
@@ -389,12 +510,6 @@ html.light #ggrid a.sel {
 #wtoggle * {
 	line-height: 1em;
 }
-#wtoggle.np {
-	width: 5.5em;
-}
-#wtoggle.sel {
-	width: 6.4em;
-}
 #wtoggle.sel #wzip,
 #wtoggle.np #wnp {
 	display: inline-block;
@@ -402,15 +517,42 @@ html.light #ggrid a.sel {
 #wtoggle.sel.np #wnp {
 	display: none;
 }
+#wfm a,
 #wzip a {
-	font-size: .4em;
+	font-size: .5em;
 	padding: 0 .3em;
 	margin: -.3em .2em;
 	position: relative;
 	display: inline-block;
 }
-#wzip a+a {
+#wfm span {
-	margin-left: .8em;
+	font-size: .6em;
+	display: block;
+}
+#wfm a:not(.en) {
+	opacity: .3;
+	color: #f6c;
+}
+html.light #wfm a:not(.en) {
+	color: #c4a;
+}
+#files tbody tr.c1 td {
+	animation: fcut1 .5s ease-out;
+}
+#files tbody tr.c2 td {
+	animation: fcut2 .5s ease-out;
+}
+@keyframes fcut1 {
+	0% {opacity:0}
+	100% {opacity:1}
+}
+@keyframes fcut2 {
+	0% {opacity:0}
+	100% {opacity:1}
+}
+#wzip a {
+	font-size: .4em;
+	margin: -.3em .3em;
 }
 #wtoggle.sel #wzip #selzip {
 	top: -.6em;
@@ -476,6 +618,17 @@ html.light #ggrid a.sel {
 		max-width: 9em;
 	}
 }
+@media (max-width: 35em) {
+	#ops>a[data-dest="new_md"],
+	#ops>a[data-dest="msg"] {
+		display: none;
+	}
+	#op_mkdir.act+div,
+	#op_mkdir.act+div+div {
+		display: block;
+		margin-top: 1em;
+	}
+}
 
 
 
@@ -773,9 +926,14 @@ input.eq_gain {
 	display: block;
 	width: 1em;
 	border-radius: .2em;
-	margin: -1.3em auto 0 auto;
+	margin: -1.2em auto 0 auto;
+	top: 2em;
+	position: relative;
 	background: #444;
 }
+#files th span {
+	position: relative;
+}
 #files>thead>tr>th.min,
 #files td.min {
 		display: none;
@@ -797,7 +955,8 @@ input.eq_gain {
 	color: #300;
 	background: #fea;
 }
-.opwide {
+.opwide,
+#op_unpost {
 	max-width: none;
 	margin-right: 1.5em;
 }
@@ -899,6 +1058,53 @@ html.light #ggrid a:hover {
 	color: #015;
 	box-shadow: 0 .1em .5em #aaa;
 }
+#op_unpost {
+	padding: 1em;
+}
+#op_unpost td {
+	padding: .2em .4em;
+}
+#op_unpost a {
+	margin: 0;
+	padding: 0;
+}
+#rui {
+	position: fixed;
+	top: 0;
+	left: 0;
+	width: calc(100% - 2em);
+	height: auto;
+	overflow: auto;
+	max-height: calc(100% - 2em);
+	border-bottom: .5em solid #999;
+	background: #333;
+	padding: 1em;
+	z-index: 765;
+}
+#rui div+div {
+	margin-top: 1em;
+}
+#rui table {
+	width: 100%;
+}
+#rui td {
+	padding: .2em .5em;
+}
+#rui td+td,
+#rui td input {
+	width: 100%;
+}
+#rui input[readonly] {
+	color: #fff;
+	background: #444;
+	border: 1px solid #777;
+	padding: .2em .25em;
+}
+#rui h1 {
+	margin: 0 0 .3em 0;
+	padding: 0;
+	font-size: 1.5em;
+}
 #pvol,
 #barbuf,
 #barpos,
@@ -937,6 +1143,9 @@ html.light {
 html.light #tt {
 	background: #fff;
 	border-color: #888 #000 #777 #000;
+}
+html.light #tt,
+html.light #toast {
 	box-shadow: 0 .3em 1em rgba(0,0,0,0.4);
 }
 html.light #tt code {
@@ -976,10 +1185,14 @@ html.light .tgl.btn.on {
 }
 html.light #srv_info {
 	color: #c83;
+	background: #eee;
+}
+html.light #srv_info,
+html.light #acc_info {
 	text-shadow: 1px 1px 0 #fff;
 }
 html.light #srv_info span {
-	color: #000;
+	color: #777;
 }
 html.light #treeul a+a {
 	background: inherit;
@@ -1026,6 +1239,17 @@ html.light #files td {
 html.light #files tbody tr:last-child td {
 	border-bottom: .2em solid #ccc;
 }
+html.light #files tr:focus td {
+	background: #fff;
+	border-color: #c37;
+	box-shadow: 0 .2em 0 #e80 , 0 -.2em 0 #e80;
+}
+html.light #files tr:focus td:first-child {
+	box-shadow: -.2em .2em 0 #e80, -.2em -.2em 0 #e80;
+}
+html.light #files tr.sel td {
+	background: #925;
+}
 html.light #files td:nth-child(2n) {
 	color: #d38;
 }
@@ -1079,7 +1303,8 @@ html.light #wnp {
 html.light #barbuf {
 	background: none;
 }
-html.light #files tr.sel:hover td {
+html.light #files tr.sel:hover td,
+html.light #files tr.sel:focus td {
 	background: #c37;
 }
 html.light #files tr.sel td {
@@ -1249,8 +1474,7 @@ html.light #bbox-overlay figcaption a {
 .bbox-btn {
 	position: fixed;
 }
-.bbox-btn,
+#bbox-overlay button {
-#bbox-btns>button {
 	cursor: pointer;
 	outline: none;
 	padding: 0 .3em;
@@ -1258,20 +1482,18 @@ html.light #bbox-overlay figcaption a {
 	border: 0;
 	border-radius: 15%;
 	background: rgba(50, 50, 50, 0.5);
-	color: #ddd;
+	color: rgba(255,255,255,0.7);
-	font: 1.6em sans-serif;
 	transition: background-color .3s ease;
-	line-height: 1em;
+	transition: color .3s ease;
+	font-size: 1.4em;
+	line-height: 1.4em;
 	vertical-align: top;
 }
-.bbox-btn:focus,
+#bbox-overlay button:focus,
-.bbox-btn:hover {
+#bbox-overlay button:hover {
+	color: rgba(255,255,255,0.9);
 	background: rgba(50, 50, 50, 0.9);
 }
-button#bbox-vmode {
-	font-size: 1em;
-	line-height: 1.6em;
-}
 #bbox-next {
 	right: 1%;
 }
@@ -1283,6 +1505,21 @@ button#bbox-vmode {
 	right: 2%;
 	position: fixed;
 }
+#bbox-halp {
+	color: #fff;
+	background: #333;
+	position: absolute;
+	top: 0;
+	left: 0;
+	z-index: 20;
+	padding: .4em;
+}
+#bbox-halp td {
+	padding: .2em .5em;
+}
+#bbox-halp td:first-child {
+	text-align: right;
+}
 .bbox-spinner {
 	width: 40px;
 	height: 40px;

copyparty/web/browser.html

@@ -39,32 +39,34 @@
 	<div id="op_mkdir" class="opview opbox act">
 		<form method="post" enctype="multipart/form-data" accept-charset="utf-8" action="{{ url_suf }}">
 			<input type="hidden" name="act" value="mkdir" />
-			<input type="text" name="name" size="30">
+			📂<input type="text" name="name" size="30">
-			<input type="submit" value="mkdir">
+			<input type="submit" value="make directory">
 		</form>
 	</div>
 
 	<div id="op_new_md" class="opview opbox">
 		<form method="post" enctype="multipart/form-data" accept-charset="utf-8" action="{{ url_suf }}">
 			<input type="hidden" name="act" value="new_md" />
-			<input type="text" name="name" size="30">
+			📝<input type="text" name="name" size="30">
-			<input type="submit" value="create doc">
+			<input type="submit" value="new markdown doc">
 		</form>
 	</div>
 
 	<div id="op_msg" class="opview opbox act">
 		<form method="post" enctype="application/x-www-form-urlencoded" accept-charset="utf-8" action="{{ url_suf }}">
-			<input type="text" name="msg" size="30">
+			📟<input type="text" name="msg" size="30">
-			<input type="submit" value="send msg">
+			<input type="submit" value="send msg to server log">
 		</form>
 	</div>
 
+	<div id="op_unpost" class="opview opbox"></div>
+
 	<div id="op_up2k" class="opview"></div>
 
 	<div id="op_cfg" class="opview opbox opwide"></div>
 	
 	<h1 id="path">
-		<a href="#" id="entree" tt="show directory tree$NHotkey: B">🌲</a>
+		<a href="#" id="entree" tt="show navpane (directory tree sidebar)$NHotkey: B">🌲</a>
 		{%- for n in vpnodes %}
 		<a href="/{{ n[0] }}">{{ n[1] }}</a>
 		{%- endfor %}
@@ -121,10 +123,14 @@
 	<div id="widget"></div>
 
 	<script>
-		var perms = {{ perms }},
+		var acct = "{{ acct }}",
+			perms = {{ perms }},
 			tag_order_cfg = {{ tag_order }},
 			have_up2k_idx = {{ have_up2k_idx|tojson }},
 			have_tags_idx = {{ have_tags_idx|tojson }},
+			have_mv = {{ have_mv|tojson }},
+			have_del = {{ have_del|tojson }},
+			have_unpost = {{ have_unpost|tojson }},
 			have_zip = {{ have_zip|tojson }};
 	</script>
 	<script src="/.cpr/util.js?_={{ ts }}"></script>

copyparty/web/md.css

@@ -8,6 +8,140 @@ html, body {
 	font-family: sans-serif;
 	line-height: 1.5em;
 }
+
+
+
+#tt, #toast {
+	position: fixed;
+	max-width: 34em;
+	background: #222;
+	border: 0 solid #777;
+	box-shadow: 0 .2em .5em #222;
+	border-radius: .4em;
+	z-index: 9001;
+}
+#tt {
+	overflow: hidden;
+	margin-top: 1em;
+	padding: 0 1.3em;
+	height: 0;
+	opacity: .1;
+	transition: opacity 0.14s, height 0.14s, padding 0.14s;
+}
+#toast {
+	top: 1.4em;
+	right: -1em;
+	line-height: 1.5em;
+	padding: 1em 1.3em;
+	border-width: .4em 0;
+	transform: translateX(100%);
+	transition:
+		transform .4s cubic-bezier(.2, 1.2, .5, 1),
+		right .4s cubic-bezier(.2, 1.2, .5, 1);
+	text-shadow: 1px 1px 0 #000;
+	color: #fff;
+}
+#toast pre {
+	margin: 0;
+}
+#toastc {
+	display: inline-block;
+	position: absolute;
+	overflow: hidden;
+	left: 0;
+	width: 0;
+	opacity: 0;
+	padding: .3em 0;
+	margin: -.3em 0 0 0;
+	line-height: 1.5em;
+	color: #000;
+	border: none;
+	outline: none;
+	text-shadow: none;
+	border-radius: .5em 0 0 .5em;
+	transition: left .3s, width .3s, padding .3s, opacity .3s;
+}
+#toast.vis {
+	right: 1.3em;
+	transform: unset;
+}
+#toast.vis #toastc {
+	left: -2em;
+	width: .4em;
+	padding: .3em .8em;
+	opacity: 1;
+}
+#toast.inf {
+	background: #07a;
+	border-color: #0be;
+}
+#toast.inf #toastc {
+	background: #0be;
+}
+#toast.ok {
+	background: #4a0;
+	border-color: #8e4;
+}
+#toast.ok #toastc {
+	background: #8e4;
+}
+#toast.warn {
+	background: #970;
+	border-color: #fc0;
+}
+#toast.warn #toastc {
+	background: #fc0;
+}
+#toast.err {
+	background: #900;
+	border-color: #d06;
+}
+#toast.err #toastc {
+	background: #d06;
+}
+#tt.b {
+	padding: 0 2em;
+	border-radius: .5em;
+	box-shadow: 0 .2em 1em #000;
+}
+#tt.show {
+	padding: 1em 1.3em;
+	border-width: .4em 0;
+	height: auto;
+	opacity: 1;
+}
+#tt.show.b {
+	padding: 1.5em 2em;
+	border-width: .5em 0;
+}
+#tt code {
+	background: #3c3c3c;
+	padding: .1em .3em;
+	border-top: 1px solid #777;
+	border-radius: .3em;
+	line-height: 1.7em;
+}
+#tt em {
+	color: #f6a;
+}
+html.light #tt {
+	background: #fff;
+	border-color: #888 #000 #777 #000;
+}
+html.light #tt,
+html.light #toast {
+	box-shadow: 0 .3em 1em rgba(0,0,0,0.4);
+}
+html.light #tt code {
+	background: #060;
+	color: #fff;
+}
+html.light #tt em {
+	color: #d38;
+}
+
+
+
 #mtw {
 	display: none;
 }
@@ -26,7 +160,7 @@ pre, code, a {
 code {
 	font-size: .96em;
 }
-pre, code {
+pre, code, tt {
 	font-family: 'scp', monospace, monospace;
 	white-space: pre-wrap;
 	word-break: break-all;
@@ -166,7 +300,7 @@ small {
 	z-index: 99;
 	position: relative;
 	display: inline-block;
-	font-family: monospace, monospace;
+	font-family: 'scp', monospace, monospace;
 	font-weight: bold;
 	font-size: 1.3em;
 	line-height: .1em;

copyparty/web/md.html

@@ -14,9 +14,9 @@
 		<a id="lightswitch" href="#">go dark</a>
 		<a id="navtoggle" href="#">hide nav</a>
 		{%- if edit %}
-			<a id="save" href="?edit">save</a>
+			<a id="save" href="?edit" tt="Hotkey: ctrl-s">save</a>
-			<a id="sbs" href="#">sbs</a>
+			<a id="sbs" href="#" tt="editor and preview side by side">sbs</a>
-			<a id="nsbs" href="#">editor</a>
+			<a id="nsbs" href="#" tt="switch between editor and preview$NHotkey: ctrl-e">editor</a>
 			<div id="toolsbox">
 				<a id="tools" href="#">tools</a>
 				<a id="fmt_table" href="#">prettify table (ctrl-k)</a>
@@ -26,8 +26,8 @@
 				<a id="help" href="#">help</a>
 			</div>
 		{%- else %}
-			<a href="?edit">edit (basic)</a>
+			<a href="?edit" tt="good: higher performance$Ngood: same document width as viewer$Nbad: assumes you know markdown">edit (basic)</a>
-			<a href="?edit2">edit (fancy)</a>
+			<a href="?edit2" tt="not in-house so probably less buggy">edit (fancy)</a>
 			<a href="?raw">view raw</a>
 		{%- endif %}
 	</div>
@@ -131,18 +131,18 @@ var md_opt = {
 };
 
 (function () {
-    var btn = document.getElementById("lightswitch");
+    var l = localStorage,
-    var toggle = function (e) {
+		drk = l.getItem('lightmode') != 1,
-		if (e) e.preventDefault();
+		btn = document.getElementById("lightswitch"),
-        var dark = !document.documentElement.getAttribute("class");
+		f = function (e) {
-        document.documentElement.setAttribute("class", dark ? "dark" : "");
+if (e) { e.preventDefault(); drk = !drk; }
-        btn.innerHTML = "go " + (dark ? "light" : "dark");
+document.documentElement.setAttribute("class", drk? "dark":"light");
-        if (window.localStorage)
+btn.innerHTML = "go " + (drk ? "light":"dark");
-            localStorage.setItem('lightmode', dark ? 0 : 1);
+l.setItem('lightmode', drk? 0:1);
-    };
+    	};
-    btn.onclick = toggle;
+	
-    if (window.localStorage && localStorage.getItem('lightmode') != 1)
+	btn.onclick = f;
-		toggle();
+	f();
 })();
 
 	</script>

copyparty/web/md.js

@@ -176,7 +176,7 @@ function md_plug_err(ex, js) {
         var lns = js.split('\n');
         if (ln < lns.length) {
             o = mknod('span');
-            o.style.cssText = 'color:#ac2;font-size:.9em;font-family:scp;display:block';
+            o.style.cssText = "color:#ac2;font-size:.9em;font-family:'scp',monospace,monospace;display:block";
             o.textContent = lns[ln - 1];
         }
     }
@@ -530,3 +530,6 @@ dom_navtgl.onclick = function () {
 
 if (sread('hidenav') == 1)
     dom_navtgl.onclick();
+
+if (window['tt'])
+    tt.init();

copyparty/web/md2.css

@@ -84,13 +84,10 @@ html.dark #save.force-save {
 #save.disabled {
 	opacity: .4;
 }
-#helpbox,
+#helpbox {
-#toast {
 	background: #f7f7f7;
 	border-radius: .4em;
 	z-index: 9001;
-}
-#helpbox {
 	display: none;
 	position: fixed;
 	padding: 2em;
@@ -107,19 +104,7 @@ html.dark #save.force-save {
 }
 html.dark #helpbox {
 	box-shadow: 0 .5em 2em #444;
-}
-html.dark #helpbox,
-html.dark #toast {
 	background: #222;
 	border: 1px solid #079;
 	border-width: 1px 0;
 }
-#toast {
-	font-weight: bold;
-	text-align: center;
-	padding: .6em 0;
-	position: fixed;
-	top: 30%;
-	transition: opacity 0.2s ease-in-out;
-	opacity: 1;
-}

copyparty/web/md2.js

@@ -236,7 +236,7 @@ function Modpoll() {
 
         var skip = null;
 
-        if (ebi('toast'))
+        if (toast.visible)
             skip = 'toast';
 
         else if (this.skip_one)
@@ -285,16 +285,15 @@ function Modpoll() {
             console.log("modpoll diff |" + server_ref.length + "|, |" + server_now.length + "|");
             this.modpoll.disabled = true;
             var msg = [
-                "The document has changed on the server.<br />" +
+                "The document has changed on the server.",
                 "The changes will NOT be loaded into your editor automatically.",
-
+                "",
-                "Press F5 or CTRL-R to refresh the page,<br />" +
+                "Press F5 or CTRL-R to refresh the page,",
                 "replacing your document with the server copy.",
-
+                "",
-                "You can click this message to ignore and contnue."
+                "You can close this message to ignore and contnue."
             ];
-            return toast(false, "box-shadow:0 1em 2em rgba(64,64,64,0.8);font-weight:normal",
+            return toast.warn(0, msg.join('\n'));
-                36, "<p>" + msg.join('</p>\n<p>') + '</p>');
         }
 
         console.log('modpoll eq');
@@ -323,16 +322,12 @@ function save(e) {
     var save_btn = ebi("save"),
         save_cls = save_btn.getAttribute('class') + '';
 
-    if (save_cls.indexOf('disabled') >= 0) {
+    if (save_cls.indexOf('disabled') >= 0)
-        toast(true, ";font-size:2em;color:#c90", 9, "no changes");
+        return toast.inf(2, "no changes");
-        return;
-    }
 
     var force = (save_cls.indexOf('force-save') >= 0);
-    if (force && !confirm('confirm that you wish to lose the changes made on the server since you opened this document')) {
+    if (force && !confirm('confirm that you wish to lose the changes made on the server since you opened this document'))
-        alert('ok, aborted');
+        return toast.inf(3, 'aborted');
-        return;
-    }
 
     var txt = dom_src.value;
 
@@ -357,18 +352,15 @@ function save_cb() {
     if (this.readyState != XMLHttpRequest.DONE)
         return;
 
-    if (this.status !== 200) {
+    if (this.status !== 200)
-        alert('Error!  The file was NOT saved.\n\n' + this.status + ": " + (this.responseText + '').replace(/^<pre>/, ""));
+        return alert('Error!  The file was NOT saved.\n\n' + this.status + ": " + (this.responseText + '').replace(/^<pre>/, ""));
-        return;
-    }
 
     var r;
     try {
         r = JSON.parse(this.responseText);
     }
     catch (ex) {
-        alert('Failed to parse reply from server:\n\n' + this.responseText);
+        return alert('Failed to parse reply from server:\n\n' + this.responseText);
-        return;
     }
 
     if (!r.ok) {
@@ -443,46 +435,10 @@ function savechk_cb() {
     last_modified = this.lastmod;
     server_md = this.txt;
     draw_md();
-    toast(true, ";font-size:6em;font-family:serif;color:#9b4", 4,
+    toast.ok(2, 'save OK' + (this.ntry ? '\nattempt ' + this.ntry : ''));
-        'OK✔️<span style="font-size:.2em;color:#999;position:absolute">' + this.ntry + '</span>');
-
     modpoll.disabled = false;
 }
 
-function toast(autoclose, style, width, msg) {
-    var ok = ebi("toast");
-    if (ok)
-        ok.parentNode.removeChild(ok);
-
-    style = "width:" + width + "em;left:calc(50% - " + (width / 2) + "em);" + style;
-    ok = mknod('div');
-    ok.setAttribute('id', 'toast');
-    ok.setAttribute('style', style);
-    ok.innerHTML = msg;
-    var parent = ebi('m');
-    document.documentElement.appendChild(ok);
-
-    var hide = function (delay) {
-        delay = delay || 0;
-
-        setTimeout(function () {
-            ok.style.opacity = 0;
-        }, delay);
-
-        setTimeout(function () {
-            if (ok.parentNode)
-                ok.parentNode.removeChild(ok);
-        }, delay + 250);
-    }
-
-    ok.onclick = function () {
-        hide(0);
-    };
-
-    if (autoclose)
-        hide(500);
-}
-
 
 // firefox bug: initial selection offset isn't cleared properly through js
 var ff_clearsel = (function () {
@@ -761,7 +717,7 @@ function fmt_table(e) {
 
         var ind2 = tab[a].match(re_ind)[0];
         if (ind != ind2 && a != 1)  // the table can be a list entry or something, ignore [0]
-            return alert(err + 'indentation mismatch on row#2 and ' + row_name + ',\n' + tab[a]);
+            return toast.err(7, err + 'indentation mismatch on row#2 and ' + row_name + ',\n' + tab[a]);
 
         var t = tab[a].slice(ind.length);
         t = t.replace(re_lpipe, "");
@@ -771,7 +727,7 @@ function fmt_table(e) {
         if (a == 0)
             ncols = tab[a].length;
         else if (ncols < tab[a].length)
-            return alert(err + 'num.columns(' + row_name + ') exceeding row#2;  ' + ncols + ' < ' + tab[a].length);
+            return toast.err(7, err + 'num.columns(' + row_name + ') exceeding row#2;  ' + ncols + ' < ' + tab[a].length);
 
         // if row has less columns than row2, fill them in
         while (tab[a].length < ncols)
@@ -788,7 +744,7 @@ function fmt_table(e) {
     for (var col = 0; col < tab[1].length; col++) {
         var m = tab[1][col].match(re_align);
         if (!m)
-            return alert(err + 'invalid column specification, row#2, col ' + (col + 1) + ', [' + tab[1][col] + ']');
+            return toast.err(7, err + 'invalid column specification, row#2, col ' + (col + 1) + ', [' + tab[1][col] + ']');
 
         if (m[2]) {
             if (m[1])
@@ -876,10 +832,9 @@ function mark_uni(e) {
         ptn = new RegExp('([^' + js_uni_whitelist + ']+)', 'g'),
         mod = txt.replace(/\r/g, "").replace(ptn, "\u2588\u2770$1\u2771");
 
-    if (txt == mod) {
+    if (txt == mod)
-        alert('no results;  no modifications were made');
+        return toast.inf(5, 'no results;  no modifications were made');
-        return;
+
-    }
     dom_src.value = mod;
 }
 
@@ -893,10 +848,9 @@ function iter_uni(e) {
         re = new RegExp('([^' + js_uni_whitelist + ']+)'),
         m = re.exec(txt.slice(ofs));
 
-    if (!m) {
+    if (!m)
-        alert('no more hits from cursor onwards');
+        return toast.inf(5, 'no more hits from cursor onwards');
-        return;
+
-    }
     ofs += m.index;
 
     dom_src.setSelectionRange(ofs, ofs + m[0].length, "forward");
@@ -924,10 +878,9 @@ function cfg_uni(e) {
 (function () {
     function keydown(ev) {
         ev = ev || window.event;
-        var kc = ev.keyCode || ev.which;
+        var kc = ev.code || ev.keyCode || ev.which;
-        var ctrl = ev.ctrlKey || ev.metaKey;
+        //console.log(ev.key, ev.code, ev.keyCode, ev.which);
-        //console.log(ev.code, kc);
+        if (ctrl(ev) && (ev.code == "KeyS" || kc == 83)) {
-        if (ctrl && (ev.code == "KeyS" || kc == 83)) {
             save();
             return false;
         }
@@ -936,23 +889,15 @@ function cfg_uni(e) {
             if (d)
                 d.click();
         }
-        if (document.activeElement == dom_src) {
+        if (document.activeElement != dom_src)
-            if (ev.code == "Tab" || kc == 9) {
+            return true;
-                md_indent(ev.shiftKey);
+
-                return false;
+        if (ctrl(ev)) {
-            }
+            if (ev.code == "KeyH" || kc == 72) {
-            if (ctrl && (ev.code == "KeyH" || kc == 72)) {
                 md_header(ev.shiftKey);
                 return false;
             }
-            if (!ctrl && (ev.code == "Home" || kc == 36)) {
+            if (ev.code == "KeyZ" || kc == 90) {
-                md_home(ev.shiftKey);
-                return false;
-            }
-            if (!ctrl && !ev.shiftKey && (ev.code == "Enter" || kc == 13)) {
-                return md_newline();
-            }
-            if (ctrl && (ev.code == "KeyZ" || kc == 90)) {
                 if (ev.shiftKey)
                     action_stack.redo();
                 else
@@ -960,33 +905,45 @@ function cfg_uni(e) {
 
                 return false;
             }
-            if (ctrl && (ev.code == "KeyY" || kc == 89)) {
+            if (ev.code == "KeyY" || kc == 89) {
                 action_stack.redo();
                 return false;
             }
-            if (!ctrl && !ev.shiftKey && kc == 8) {
+            if (ev.code == "KeyK") {
-                return md_backspace();
-            }
-            if (ctrl && (ev.code == "KeyK")) {
                 fmt_table();
                 return false;
             }
-            if (ctrl && (ev.code == "KeyU")) {
+            if (ev.code == "KeyU") {
                 iter_uni();
                 return false;
             }
-            if (ctrl && (ev.code == "KeyE")) {
+            if (ev.code == "KeyE") {
                 dom_nsbs.click();
-                //fmt_table();
                 return false;
             }
             var up = ev.code == "ArrowUp" || kc == 38;
             var dn = ev.code == "ArrowDown" || kc == 40;
-            if (ctrl && (up || dn)) {
+            if (up || dn) {
                 md_p_jump(dn);
                 return false;
             }
         }
+        else {
+            if (ev.code == "Tab" || kc == 9) {
+                md_indent(ev.shiftKey);
+                return false;
+            }
+            if (ev.code == "Home" || kc == 36) {
+                md_home(ev.shiftKey);
+                return false;
+            }
+            if (!ev.shiftKey && (ev.code == "Enter" || kc == 13)) {
+                return md_newline();
+            }
+            if (!ev.shiftKey && kc == 8) {
+                return md_backspace();
+            }
+        }
     }
     document.onkeydown = keydown;
     ebi('save').onclick = save;

copyparty/web/mde.html

@@ -30,16 +30,15 @@ var md_opt = {
 };
 
 var lightswitch = (function () {
-	var fun = function () {
+	var l = localStorage,
-		var dark = !document.documentElement.getAttribute("class");
+		drk = l.getItem('lightmode') != 1,
-		document.documentElement.setAttribute("class", dark ? "dark" : "");
+		f = function (e) {
-		if (window.localStorage)
+if (e) drk = !drk;
-			localStorage.setItem('lightmode', dark ? 0 : 1);
+document.documentElement.setAttribute("class", drk? "dark":"light");
-	};
+l.setItem('lightmode', drk? 0:1);
-	if (window.localStorage && localStorage.getItem('lightmode') != 1)
+		};
-		fun();
+	f();
-	
+	return f;
-	return fun;
 })();
 
 	</script>

copyparty/web/mde.js

@@ -75,7 +75,7 @@ function set_jumpto() {
 }
 
 function jumpto(ev) {
-    var tgt = ev.target || ev.srcElement;
+    var tgt = ev.target;
     var ln = null;
     while (tgt && !ln) {
         ln = tgt.getAttribute('data-ln');
@@ -106,15 +106,12 @@ function md_changed(mde, on_srv) {
 
 function save(mde) {
     var save_btn = QS('.editor-toolbar button.save');
-    if (save_btn.classList.contains('disabled')) {
+    if (save_btn.classList.contains('disabled'))
-        alert('there is nothing to save');
+        return toast.inf(2, 'no changes');
-        return;
+
-    }
     var force = save_btn.classList.contains('force-save');
-    if (force && !confirm('confirm that you wish to lose the changes made on the server since you opened this document')) {
+    if (force && !confirm('confirm that you wish to lose the changes made on the server since you opened this document'))
-        alert('ok, aborted');
+        return toast.inf(3, 'aborted');
-        return;
-    }
 
     var txt = mde.value();
 
@@ -138,18 +135,15 @@ function save_cb() {
     if (this.readyState != XMLHttpRequest.DONE)
         return;
 
-    if (this.status !== 200) {
+    if (this.status !== 200)
-        alert('Error!  The file was NOT saved.\n\n' + this.status + ": " + (this.responseText + '').replace(/^<pre>/, ""));
+        return alert('Error!  The file was NOT saved.\n\n' + this.status + ": " + (this.responseText + '').replace(/^<pre>/, ""));
-        return;
-    }
 
     var r;
     try {
         r = JSON.parse(this.responseText);
     }
     catch (ex) {
-        alert('Failed to parse reply from server:\n\n' + this.responseText);
+        return alert('Failed to parse reply from server:\n\n' + this.responseText);
-        return;
     }
 
     if (!r.ok) {

copyparty/web/splash.html

@@ -68,7 +68,7 @@
     </div>
     <script>
 
-if (window.localStorage && localStorage.getItem('lightmode') != 1)
+if (localStorage.getItem('lightmode') != 1)
     document.documentElement.setAttribute("class", "dark");
 
 </script>

copyparty/web/up2k.js

@@ -290,8 +290,7 @@ function U2pvis(act, btns) {
                 if (this.is_act(this.tab[a].in))
                     console.log("tab %d/%d = sz %s", a, aa, this.tab[a].bt);
 
-            console.log("a");
+            throw new Error('see console');
-            throw 42;
         }
 
         obj.innerHTML = fo.hp;
@@ -304,15 +303,8 @@ function U2pvis(act, btns) {
             oldcat = fo.in,
             bz_act = this.act == "bz";
 
-        if (oldcat == newcat) {
+        if (oldcat == newcat)
-            throw 42;
+            return;
-        }
-
-        //console.log("oldcat %s %d, newcat %s %d, head=%d, tail=%d, file=%d, act.old=%s, act.new=%s, bz_act=%s",
-        //    oldcat, this.ctr[oldcat],
-        //    newcat, this.ctr[newcat],
-        //    this.head, this.tail, nfile,
-        //    this.is_act(oldcat), this.is_act(newcat), bz_act);
 
         fo.in = newcat;
         this.ctr[oldcat]--;
@@ -468,6 +460,15 @@ function U2pvis(act, btns) {
 }
 
 
+function fsearch_explain(e) {
+    ev(e);
+    if (!has(perms, 'write'))
+        return alert('your access to this folder is Read-Only\n\n' + (acct == '*' ? 'you are currently not logged in' : 'you are currently logged in as ' + acct));
+
+    alert('you are currently in file-search mode\n\nswitch to upload-mode by clicking the green magnifying glass (next to the big yellow search button), and then refresh\n\nsorry');
+}
+
+
 function up2k_init(subtle) {
     // show modal message
     function showmodal(msg) {
@@ -495,8 +496,9 @@ function up2k_init(subtle) {
         shame = 'your browser is impressively ancient';
 
     // upload ui hidden by default, clicking the header shows it
+    var got_deps = false;
     function init_deps() {
-        if (!subtle && !window.asmCrypto) {
+        if (!got_deps && !subtle && !window.asmCrypto) {
             var fn = 'sha512.' + sha_js + '.js';
             showmodal('<h1>loading ' + fn + '</h1><h2>since ' + shame + '</h2><h4>thanks chrome</h4>');
             import_js('/.cpr/deps/' + fn, unmodal);
@@ -507,6 +509,7 @@ function up2k_init(subtle) {
                 ebi('u2foot').innerHTML = 'seems like ' + shame + ' so do that if you want more performance <span style="color:#' +
                     (sha_js == 'ac' ? 'c84">(expecting 20' : '8a5">(but dont worry too much, expect 100') + ' MiB/s)</span>';
         }
+        got_deps = true;
     }
 
     // show uploader if the user only has write-access
@@ -923,8 +926,11 @@ function up2k_init(subtle) {
                 return;
 
             clearTimeout(tto);
+            if (crashed)
+                return defer();
+
             running = true;
-            while (window['vis_exh']) {
+            while (true) {
                 var now = Date.now(),
                     is_busy = 0 !=
                         st.todo.head.length +
@@ -1006,7 +1012,7 @@ function up2k_init(subtle) {
                     mou_ikkai = true;
                 }
 
-                if (!mou_ikkai)
+                if (!mou_ikkai || crashed)
                     return defer();
             }
         }
@@ -1170,10 +1176,6 @@ function up2k_init(subtle) {
                 }
 
                 t.t_hashed = Date.now();
-                if (t.n == 0 && window.location.hash == '#dbg') {
-                    var spd = (t.size / ((t.t_hashed - t.t_hashing) / 1000.)) / (1024 * 1024.);
-                    alert('{0} ms, {1} MB/s\n'.format(t.t_hashed - t.t_hashing, spd.toFixed(3)) + t.hash.join('\n'));
-                }
 
                 pvis.seth(t.n, 2, 'hashing done');
                 pvis.seth(t.n, 1, '📦 wait');
@@ -1298,8 +1300,8 @@ function up2k_init(subtle) {
                         smsg = '';
 
                     if (!response || !response.hits || !response.hits.length) {
-                        msg = 'not found on server';
                         smsg = '404';
+                        msg = 'not found on server <a href="#" onclick="fsearch_explain()" class="fsearch_explain">(explain)</a>';
                     }
                     else {
                         smsg = 'found';
@@ -1516,7 +1518,7 @@ function up2k_init(subtle) {
                 try { orz(xhr); } catch (ex) { vis_exh(ex + '', '', '', '', ex); }
             };
             xhr.onerror = function (xev) {
-                if (!window['vis_exh'])
+                if (crashed)
                     return;
 
                 console.log('chunkpit onerror, retrying', t);
@@ -1570,7 +1572,7 @@ function up2k_init(subtle) {
     for (var a = o.length - 1; a >= 0; a--) {
         o[a].parentNode.getElementsByTagName('input')[0].setAttribute('tt', o[a].getAttribute('tt'));
     }
-    tt.init();
+    tt.att(QS('#u2conf'));
 
     function bumpthread2(e) {
         if (e.ctrlKey || e.altKey || e.metaKey || e.isComposing)
@@ -1771,3 +1773,14 @@ if (QS('#op_up2k.act'))
     goto_up2k();
 
 apply_perms(perms);
+
+
+(function () {
+    goto();
+    var op = sread('opmode');
+    if (op !== null && op !== '.')
+        try {
+            goto(op);
+        }
+        catch (ex) { }
+})();

copyparty/web/upload.css

@@ -87,8 +87,9 @@
 #u2tab td:nth-child(3) {
 	width: 40%;
 }
-#op_up2k.srch #u2tab td:nth-child(3) {
+#op_up2k.srch td.prog {
 	font-family: sans-serif;
+	font-size: 1em;
 	width: auto;
 }
 #u2tab tbody tr:hover td {
@@ -245,7 +246,7 @@ html.light #u2foot .warn span {
 	margin-bottom: -1em;
 }
 .prog {
-	font-family: monospace;
+	font-family: monospace, monospace;
 }
 #u2tab a>span {
 	font-weight: bold;
@@ -257,6 +258,11 @@ html.light #u2foot .warn span {
 	float: right;
 	margin-bottom: -.3em;
 }
+.fsearch_explain {
+	padding-left: .7em;
+	font-size: 1.1em;
+	line-height: 0;
+}
 
 
 

copyparty/web/util.js

@@ -7,7 +7,14 @@ if (!window['console'])
 
 
 var is_touch = 'ontouchstart' in window,
-    ANDROID = /(android)/i.test(navigator.userAgent);
+    IPHONE = /iPhone|iPad|iPod/i.test(navigator.userAgent),
+    ANDROID = /android/i.test(navigator.userAgent);
+
+
+var ebi = document.getElementById.bind(document),
+    QS = document.querySelector.bind(document),
+    QSA = document.querySelectorAll.bind(document),
+    mknod = document.createElement.bind(document);
 
 
 // error handler for mobile devices
@@ -21,36 +28,60 @@ function esc(txt) {
         }[c];
     });
 }
+var crashed = false, ignexd = {};
 function vis_exh(msg, url, lineNo, columnNo, error) {
-    if (!window.onerror)
+    if ((msg + '').indexOf('ResizeObserver') !== -1)
+        return;  // chrome issue 809574 (benign, from <video>)
+
+    var ekey = url + '\n' + lineNo + '\n' + msg;
+    if (ignexd[ekey] || crashed)
         return;
 
+    crashed = true;
     window.onerror = undefined;
-    window['vis_exh'] = null;
+    var html = ['<h1>you hit a bug!</h1><p style="font-size:1.3em;margin:0">try to <a href="#" onclick="localStorage.clear();location.reload();">reset copyparty settings</a> if you are stuck here, or <a href="#" onclick="ignex();">ignore this</a> / <a href="#" onclick="ignex(true);">ignore all</a></p><p>please send me a screenshot arigathanks gozaimuch: <code>ed/irc.rizon.net</code> or <code>ed#2644</code><br />&nbsp; (and if you can, press F12 and include the "Console" tab in the screenshot too)</p><p>',
-    var html = ['<h1>you hit a bug!</h1><p style="font-size:1.3em;margin:0">try to <a href="#" onclick="localStorage.clear();location.reload();" style="text-decoration:underline;color:#fc0">reset copyparty settings</a> if you are stuck here</p><p>please send me a screenshot arigathanks gozaimuch: <code>ed/irc.rizon.net</code> or <code>ed#2644</code><br />&nbsp; (and if you can, press F12 and include the "Console" tab in the screenshot too)</p><p>',
         esc(url + ' @' + lineNo + ':' + columnNo), '<br />' + esc(String(msg)) + '</p>'];
 
-    if (error) {
+    try {
-        var find = ['desc', 'stack', 'trace'];
+        if (error) {
-        for (var a = 0; a < find.length; a++)
+            var find = ['desc', 'stack', 'trace'];
-            if (String(error[find[a]]) !== 'undefined')
+            for (var a = 0; a < find.length; a++)
-                html.push('<h3>' + find[a] + '</h3>' +
+                if (String(error[find[a]]) !== 'undefined')
-                    esc(String(error[find[a]])).replace(/\n/g, '<br />\n'));
+                    html.push('<h3>' + find[a] + '</h3>' +
+                        esc(String(error[find[a]])).replace(/\n/g, '<br />\n'));
+        }
+        ignexd[ekey] = true;
+        html.push('<h3>localStore</h3>' + esc(JSON.stringify(localStorage)));
     }
-    document.body.innerHTML = html.join('\n');
+    catch (e) { }
 
-    var s = mknod('style');
+    try {
-    s.innerHTML = 'body{background:#333;color:#ddd;font-family:sans-serif;font-size:0.8em;padding:0 1em 1em 1em} h1{margin:.5em 1em 0 0;padding:0} h3{border-top:1px solid #999;margin:0} code{color:#bf7;background:#222;padding:.1em;margin:.2em;font-size:1.1em;font-family:monospace,monospace} *{line-height:1.5em}';
+        var exbox = ebi('exbox');
-    document.head.appendChild(s);
+        if (!exbox) {
+            exbox = mknod('div');
+            exbox.setAttribute('id', 'exbox');
+            document.body.appendChild(exbox);
 
+            var s = mknod('style');
+            s.innerHTML = '#exbox{background:#333;color:#ddd;font-family:sans-serif;font-size:0.8em;padding:0 1em 1em 1em;z-index:80386;position:fixed;top:0;left:0;right:0;bottom:0;width:100%;height:100%} #exbox h1{margin:.5em 1em 0 0;padding:0} #exbox h3{border-top:1px solid #999;margin:1em 0 0 0} #exbox a{text-decoration:underline;color:#fc0} #exbox code{color:#bf7;background:#222;padding:.1em;margin:.2em;font-size:1.1em;font-family:monospace,monospace} #exbox *{line-height:1.5em}';
+            document.head.appendChild(s);
+        }
+        exbox.innerHTML = html.join('\n');
+        exbox.style.display = 'block';
+    }
+    catch (e) {
+        document.body.innerHTML = html.join('\n');
+    }
     throw 'fatal_err';
 }
-
+function ignex(all) {
-
+    var o = ebi('exbox');
-var ebi = document.getElementById.bind(document),
+    o.style.display = 'none';
-    QS = document.querySelector.bind(document),
+    o.innerHTML = '';
-    QSA = document.querySelectorAll.bind(document),
+    crashed = false;
-    mknod = document.createElement.bind(document);
+    if (!all)
+        window.onerror = vis_exh;
+}
 
 
 function ctrl(e) {
@@ -92,6 +123,15 @@ if (!String.startsWith) {
         return this.substring(i, i + s.length) === s;
     };
 }
+if (!Element.prototype.closest) {
+    Element.prototype.closest = function (s) {
+        var el = this;
+        do {
+            if (el.msMatchesSelector(s)) return el;
+            el = el.parentElement || el.parentNode;
+        } while (el !== null && el.nodeType === 1);
+    }
+}
 
 
 // https://stackoverflow.com/a/950146
@@ -321,6 +361,18 @@ function linksplit(rp) {
 }
 
 
+function vsplit(vp) {
+    if (vp.endsWith('/'))
+        vp = vp.slice(0, -1);
+
+    var ofs = vp.lastIndexOf('/') + 1,
+        base = vp.slice(0, ofs),
+        fn = vp.slice(ofs);
+
+    return [base, fn];
+}
+
+
 function uricom_enc(txt, do_fb_enc) {
     try {
         return encodeURIComponent(txt);
@@ -346,6 +398,15 @@ function uricom_dec(txt) {
 }
 
 
+function uricom_adec(arr) {
+    var ret = [];
+    for (var a = 0; a < arr.length; a++)
+        ret.push(uricom_dec(arr[a])[0]);
+
+    return ret;
+}
+
+
 function get_evpath() {
     var ret = document.location.pathname;
 
@@ -407,19 +468,14 @@ function jcp(obj) {
 
 
 function sread(key) {
-    if (window.localStorage)
+    return localStorage.getItem(key);
-        return localStorage.getItem(key);
-
-    return null;
 }
 
 function swrite(key, val) {
-    if (window.localStorage) {
+    if (val === undefined || val === null)
-        if (val === undefined || val === null)
+        localStorage.removeItem(key);
-            localStorage.removeItem(key);
+    else
-        else
+        localStorage.setItem(key, val);
-            localStorage.setItem(key, val);
-    }
 }
 
 function jread(key, fb) {
@@ -502,13 +558,20 @@ function hist_replace(url) {
 var tt = (function () {
     var r = {
         "tt": mknod("div"),
-        "en": true
+        "en": true,
+        "el": null,
+        "skip": false
     };
 
     r.tt.setAttribute('id', 'tt');
     document.body.appendChild(r.tt);
 
     r.show = function () {
+        if (r.skip) {
+            r.skip = false;
+            return;
+        }
+
         var cfg = sread('tooltips');
         if (cfg !== null && cfg != '1')
             return;
@@ -517,11 +580,18 @@ var tt = (function () {
         if (!msg)
             return;
 
+        r.el = this;
         var pos = this.getBoundingClientRect(),
+            dir = this.getAttribute('ttd') || '',
             left = pos.left < window.innerWidth / 2,
             top = pos.top < window.innerHeight / 2,
             big = this.className.indexOf(' ttb') !== -1;
 
+        if (dir.indexOf('u') + 1) top = false;
+        if (dir.indexOf('d') + 1) top = true;
+        if (dir.indexOf('l') + 1) left = false;
+        if (dir.indexOf('r') + 1) left = true;
+
         clmod(r.tt, 'b', big);
         r.tt.style.top = top ? pos.bottom + 'px' : 'auto';
         r.tt.style.bottom = top ? 'auto' : (window.innerHeight - pos.top) + 'px';
@@ -529,13 +599,45 @@ var tt = (function () {
         r.tt.style.right = left ? 'auto' : (window.innerWidth - pos.right) + 'px';
 
         r.tt.innerHTML = msg.replace(/\$N/g, "<br />");
+        r.el.addEventListener('mouseleave', r.hide);
         clmod(r.tt, 'show', 1);
     };
 
-    r.hide = function () {
+    r.hide = function (e) {
+        ev(e);
         clmod(r.tt, 'show');
+        if (r.el)
+            r.el.removeEventListener('mouseleave', r.hide);
     };
 
+    if (is_touch && IPHONE) {
+        var f1 = r.show,
+            f2 = r.hide;
+
+        r.show = function () {
+            setTimeout(f1.bind(this), 301);
+        };
+        r.hide = function () {
+            setTimeout(f2.bind(this), 301);
+        };
+    }
+
+    r.tt.onclick = r.hide;
+
+    r.att = function (ctr) {
+        var _show = r.en ? r.show : null,
+            _hide = r.en ? r.hide : null,
+            o = ctr.querySelectorAll('*[tt]');
+
+        for (var a = o.length - 1; a >= 0; a--) {
+            o[a].onfocus = _show;
+            o[a].onblur = _hide;
+            o[a].onmouseenter = _show;
+            o[a].onmouseleave = _hide;
+        }
+        r.hide();
+    }
+
     r.init = function () {
         var ttb = ebi('tooltips');
         if (ttb) {
@@ -547,18 +649,58 @@ var tt = (function () {
             };
             r.en = bcfg_get('tooltips', true)
         }
-
+        r.att(document);
-        var _show = r.en ? r.show : null,
+    };
-            _hide = r.en ? r.hide : null;
+
-
+    return r;
-        var o = QSA('*[tt]');
+})();
-        for (var a = o.length - 1; a >= 0; a--) {
+
-            o[a].onfocus = _show;
+
-            o[a].onblur = _hide;
+var toast = (function () {
-            o[a].onmouseenter = _show;
+    var r = {},
-            o[a].onmouseleave = _hide;
+        te = null,
-        }
+        visible = false,
-        r.hide();
+        obj = mknod('div');
+
+    obj.setAttribute('id', 'toast');
+    document.body.appendChild(obj);;
+
+    r.hide = function (e) {
+        ev(e);
+        clearTimeout(te);
+        clmod(obj, 'vis');
+        r.visible = false;
+    };
+
+    r.show = function (cl, ms, txt) {
+        clearTimeout(te);
+        if (ms)
+            te = setTimeout(r.hide, ms * 1000);
+
+        var html = '', hp = txt.split(/(?=<.?pre>)/i);
+        for (var a = 0; a < hp.length; a++)
+            html += hp[a].startsWith('<pre>') ? hp[a] :
+                hp[a].replace(/<br ?.?>\n/g, '\n').replace(/\n<br ?.?>/g, '\n').replace(/\n/g, '<br />\n');
+
+        obj.innerHTML = '<a href="#" id="toastc">x</a>' + html;
+        obj.className = cl;
+        ms += obj.offsetWidth;
+        obj.className += ' vis';
+        ebi('toastc').onclick = r.hide;
+        r.visible = true;
+    };
+
+    r.ok = function (ms, txt) {
+        r.show('ok', ms, txt);
+    };
+    r.inf = function (ms, txt) {
+        r.show('inf', ms, txt);
+    };
+    r.warn = function (ms, txt) {
+        r.show('warn', ms, txt);
+    };
+    r.err = function (ms, txt) {
+        r.show('err', ms, txt);
     };
 
     return r;

docs/example.conf

@@ -10,19 +10,25 @@ u k:k
 # share "." (the current directory)
 # as "/" (the webroot) for the following users:
 # "r" grants read-access for anyone
-# "a ed" grants read-write to ed
+# "rw ed" grants read-write to ed
 .
 /
 r
-a ed
+rw ed
 
 # custom permissions for the "priv" folder:
-# user "k" can see/read the contents
+# user "k" can only see/read the contents
-# and "ed" gets read-write access
+# user "ed" gets read-write access
 ./priv
 /priv
 r k
-a ed
+rw ed
+
+# this does the same thing:
+./priv
+/priv
+r ed k
+w ed
 
 # share /home/ed/Music/ as /music and let anyone read it
 # (this will replace any folder called "music" in the webroot)
@@ -41,5 +47,5 @@ c e2d
 c nodupe
 
 # this entire config file can be replaced with these arguments:
-# -u ed:123 -u k:k -v .::r:aed -v priv:priv:rk:aed -v /home/ed/Music:music:r -v /home/ed/inc:dump:w
+# -u ed:123 -u k:k -v .::r:a,ed -v priv:priv:r,k:rw,ed -v /home/ed/Music:music:r -v /home/ed/inc:dump:w:c,e2d:c,nodupe
 # but note that the config file always wins in case of conflicts

docs/notes.sh

@@ -44,7 +44,7 @@ avg() { awk 'function pr(ncsz) {if (nsmp>0) {printf "%3s %s\n", csz, sum/nsmp} c
 dirs=("$HOME/vfs/ほげ" "$HOME/vfs/ほげ/ぴよ" "$HOME/vfs/$(printf \\xed\\x91)" "$HOME/vfs/$(printf \\xed\\x91/\\xed\\x92)")
 mkdir -p "${dirs[@]}"
 for dir in "${dirs[@]}"; do for fn in ふが "$(printf \\xed\\x93)" 'qwe,rty;asd fgh+jkl%zxc&vbn <qwe>"rty'"'"'uio&asd&nbsp;fgh'; do echo "$dir" > "$dir/$fn.html"; done; done
-
+# qw er+ty%20ui%%20op<as>df&gh&amp;jk#zx'cv"bn`m=qw*er^ty?ui@op,as.df-gh_jk
 
 ##
 ## upload mojibake
@@ -79,6 +79,10 @@ command -v gdate && date() { gdate "$@"; }; while true; do t=$(date +%s.%N); (ti
 # get all up2k search result URLs
 var t=[]; var b=document.location.href.split('#')[0].slice(0, -1); document.querySelectorAll('#u2tab .prog a').forEach((x) => {t.push(b+encodeURI(x.getAttribute("href")))}); console.log(t.join("\n"));
 
+# rename all selected songs to <leading-track-number> + <Title> + <extension>
+var sel=msel.getsel(), ci=find_file_col('Title')[0], re=[]; for (var a=0; a<sel.length; a++) { var url=sel[a].vp, tag=ebi(sel[a].id).closest('tr').querySelectorAll('td')[ci].textContent, name=uricom_dec(vsplit(url)[1])[0], m=/^([0-9]+[\. -]+)?.*(\.[^\.]+$)/.exec(name), name2=(m[1]||'')+tag+m[2], url2=vsplit(url)[0]+uricom_enc(name2,false); if (url!=url2) re.push([url, url2]); }
+console.log(JSON.stringify(re, null, '  '));
+function f() { if (!re.length) return treectl.goto(get_evpath()); var [u1,u2] = re.shift(); fetch(u1+'?move='+u2).then((rsp) => {if (rsp.ok) f(); }); }; f();
 
 ##
 ## bash oneliners
@@ -166,7 +170,10 @@ dbg.asyncStore.pendingBreakpoints = {}
 about:config >> devtools.debugger.prefs-schema-version = -1
 
 # determine server version
-git pull; git reset --hard origin/HEAD && git log --format=format:"%H %ai %d" --decorate=full > ../revs && cat ../{util,browser}.js >../vr && cat ../revs | while read -r rev extra; do (git reset --hard $rev >/dev/null 2>/dev/null && dsz=$(cat copyparty/web/{util,browser}.js >../vg 2>/dev/null && diff -wNarU0 ../{vg,vr} | wc -c) && printf '%s %6s %s\n' "$rev" $dsz "$extra") </dev/null; done                
+git pull; git reset --hard origin/HEAD && git log --format=format:"%H %ai %d" --decorate=full > ../revs && cat ../{util,browser,up2k}.js >../vr && cat ../revs | while read -r rev extra; do (git reset --hard $rev >/dev/null 2>/dev/null && dsz=$(cat copyparty/web/{util,browser,up2k}.js >../vg 2>/dev/null && diff -wNarU0 ../{vg,vr} | wc -c) && printf '%s %6s %s\n' "$rev" $dsz "$extra") </dev/null; done                
+
+# download all sfx versions
+curl https://api.github.com/repos/9001/copyparty/releases?per_page=100 | jq -r '.[] | .tag_name + " " + .name' | while read v t; do fn="copyparty $v $t.py"; [ -e $fn ] || curl https://github.com/9001/copyparty/releases/download/$v/copyparty-sfx.py -Lo "$fn"; done
 
 
 ##

scripts/make-sfx.sh

@@ -20,6 +20,11 @@ echo
 #
 # `no-cm` saves ~90k by removing easymde/codemirror
 #   (the fancy markdown editor)
+#
+# `no-fnt` saves ~9k by removing the source-code-pro font
+#   (mainly used my the markdown viewer/editor)
+#
+# `no-dd` saves ~2k by removing the mouse cursor
 
 
 # port install gnutar findutils gsed coreutils
@@ -29,6 +34,7 @@ gtar=$(command -v gtar || command -v gnutar) || true
 	sed()  { gsed  "$@"; }
 	find() { gfind "$@"; }
 	sort() { gsort "$@"; }
+	sha1sum() { shasum "$@"; }
 	unexpand() { gunexpand "$@"; }
 	command -v grealpath >/dev/null &&
 		realpath() { grealpath "$@"; }
@@ -57,14 +63,18 @@ use_gz=
 do_sh=1
 do_py=1
 while [ ! -z "$1" ]; do
-	[ "$1" = clean  ] && clean=1  && shift && continue
+	case $1 in
-	[ "$1" = re     ] && repack=1 && shift && continue
+		clean)  clean=1  ; ;;
-	[ "$1" = gz     ] && use_gz=1 && shift && continue
+		re)     repack=1 ; ;;
-	[ "$1" = no-ogv ] && no_ogv=1 && shift && continue
+		gz)     use_gz=1 ; ;;
-	[ "$1" = no-cm  ] && no_cm=1  && shift && continue
+		no-ogv) no_ogv=1 ; ;;
-	[ "$1" = no-sh  ] && do_sh=   && shift && continue
+		no-fnt) no_fnt=1 ; ;;
-	[ "$1" = no-py  ] && do_py=   && shift && continue
+		no-dd)  no_dd=1  ; ;;
-	break
+		no-cm)  no_cm=1  ; ;;
+		no-sh)  do_sh=   ; ;;
+		no-py)  do_py=   ; ;;
+	esac
+	shift
 done
 
 tmv() {
@@ -72,16 +82,23 @@ tmv() {
 	mv t "$1"
 }
 
+stamp=$(
+	for d in copyparty scripts; do
+		find $d -type f -printf '%TY-%Tm-%Td %TH:%TM:%TS %p\n'
+	done | sort | tail -n 1 | sha1sum | cut -c-16
+)
+
 rm -rf sfx/*
 mkdir -p sfx build
 cd sfx
 
-[ $repack ] && {
+tmpdir="$(
-	old="$(
+	printf '%s\n' "$TMPDIR" /tmp |
-		printf '%s\n' "$TMPDIR" /tmp |
+	awk '/./ {print; exit}'
-		awk '/./ {print; exit}'
+)"
-	)/pe-copyparty"
 
+[ $repack ] && {
+	old="$tmpdir/pe-copyparty"
 	echo "repack of files in $old"
 	cp -pR "$old/"*{dep-j2,copyparty} .
 }
@@ -163,12 +180,12 @@ mkdir -p ../dist
 sfx_out=../dist/copyparty-sfx
 
 echo cleanup
-find .. -name '*.pyc' -delete
+find -name '*.pyc' -delete
-find .. -name __pycache__ -delete
+find -name __pycache__ -delete
 
 # especially prevent osx from leaking your lan ip (wtf apple)
-find .. -type f \( -name .DS_Store -or -name ._.DS_Store \) -delete
+find -type f \( -name .DS_Store -or -name ._.DS_Store \) -delete
-find .. -type f -name ._\* | while IFS= read -r f; do cmp <(printf '\x00\x05\x16') <(head -c 3 -- "$f") && rm -f -- "$f"; done
+find -type f -name ._\* | while IFS= read -r f; do cmp <(printf '\x00\x05\x16') <(head -c 3 -- "$f") && rm -f -- "$f"; done
 
 echo use smol web deps
 rm -f copyparty/web/deps/*.full.* copyparty/web/dbg-* copyparty/web/Makefile
@@ -190,6 +207,18 @@ done
 	sed -r '/edit2">edit \(fancy/d' <$f >t && tmv "$f"
 }
 
+[ $no_fnt ] && {
+	rm -f copyparty/web/deps/scp.woff2
+	f=copyparty/web/md.css
+	sed -r '/scp\.woff2/d' <$f >t && tmv "$f"
+}
+
+[ $no_dd ] && {
+	rm -rf copyparty/web/dd
+	f=copyparty/web/browser.css
+	sed -r 's/(cursor: )url\([^)]+\), (pointer)/\1\2/; /[0-9]+% \{cursor:/d; /animation: cursor/d' <$f >t && tmv "$f"
+}
+
 [ $repack ] ||
 find | grep -E '\.py$' |
   grep -vE '__version__' |
@@ -220,20 +249,42 @@ find | grep -E '\.(js|html)$' | while IFS= read -r f; do
 	tmv "$f"
 done
 
-
 gzres() {
-command -v pigz &&
+	command -v pigz &&
-	pk='pigz -11 -J 34 -I 100' ||
+		pk='pigz -11 -J 34 -I 256' ||
-	pk='gzip'
+		pk='gzip'
 
-echo "$pk"
+	echo "$pk"
-find | grep -E '\.(js|css)$' | grep -vF /deps/ | while IFS= read -r f; do
+	find | grep -E '\.(js|css)$' | grep -vF /deps/ | while IFS= read -r f; do
-	echo -n .
+		echo -n .
-	$pk "$f"
+		$pk "$f"
-done
+	done
-echo
+	echo
+}
+
+
+zdir="$tmpdir/cpp-mksfx"
+[ -e "$zdir/$stamp" ] || rm -rf "$zdir"
+mkdir -p "$zdir"
+echo a > "$zdir/$stamp"
+nf=$(ls -1 "$zdir"/arc.* | wc -l)
+[ $nf -ge 2 ] && [ ! $repack ] && use_zdir=1 || use_zdir=
+
+[ $use_zdir ] || {
+	echo "$nf alts += 1"
+	gzres
+	[ $repack ] ||
+		tar -cf "$zdir/arc.$(date +%s)" copyparty/web/*.gz
+}
+[ $use_zdir ] && {
+	arcs=("$zdir"/arc.*)
+	arc="${arcs[$RANDOM % ${#arcs[@]} ] }"
+	echo "using $arc"
+	tar -xf "$arc"
+	for f in copyparty/web/*.gz; do
+		rm "${f%.*}"
+	done
 }
-gzres
 
 
 echo gen tarlist

scripts/sfx.py

@@ -380,7 +380,7 @@ def run(tmp, j2):
         fcntl.flock(fd, fcntl.LOCK_EX | fcntl.LOCK_NB)
     except Exception as ex:
         if not WINDOWS:
-            msg("\033[31mflock:", repr(ex))
+            msg("\033[31mflock:{!r}\033[0m".format(ex))
 
     t = threading.Thread(target=utime, args=(tmp,))
     t.daemon = True

scripts/uncomment.py

@@ -65,9 +65,9 @@ def uncomment(fpath):
 
 
 def main():
-    print("uncommenting", end="")
+    print("uncommenting", end="", flush=True)
     for f in sys.argv[1:]:
-        print(".", end="")
+        print(".", end="", flush=True)
         uncomment(f)
 
     print("k")

tests/test_httpcli.py

@@ -31,7 +31,11 @@ class Cfg(Namespace):
             rproxy=0,
             ed=False,
             nw=False,
+            unpost=600,
+            no_mv=False,
+            no_del=False,
             no_zip=False,
+            no_voldump=True,
             no_scandir=False,
             no_sendfile=True,
             no_rescan=True,
@@ -90,7 +94,7 @@ class TestHttpCli(unittest.TestCase):
                 if not vol.startswith(top):
                     continue
 
-                mode = vol[-2]
+                mode = vol[-2].replace("a", "rwmd")
                 usr = vol[-1]
                 if usr == "a":
                     usr = ""
@@ -99,7 +103,7 @@ class TestHttpCli(unittest.TestCase):
                     vol += "/"
 
                 top, sub = vol.split("/", 1)
-                vcfg.append("{0}/{1}:{1}:{2}{3}".format(top, sub, mode, usr))
+                vcfg.append("{0}/{1}:{1}:{2},{3}".format(top, sub, mode, usr))
 
             pprint.pprint(vcfg)
 

tests/test_vfs.py

@@ -24,6 +24,7 @@ class Cfg(Namespace):
             "hist": None,
             "no_hash": False,
             "css_browser": None,
+            "no_voldump": True,
             "rproxy": 0,
         }
         ex.update(ex2)
@@ -57,8 +58,8 @@ class TestVFS(unittest.TestCase):
         # type: (VFS, str, str) -> tuple[str, str, str]
         """helper for resolving and listing a folder"""
         vn, rem = vfs.get(vpath, uname, True, False)
-        r1 = vn.ls(rem, uname, False)
+        r1 = vn.ls(rem, uname, False, [[True]])
-        r2 = vn.ls(rem, uname, False)
+        r2 = vn.ls(rem, uname, False, [[True]])
         self.assertEqual(r1, r2)
 
         fsdir, real, virt = r1
@@ -68,6 +69,11 @@ class TestVFS(unittest.TestCase):
     def log(self, src, msg, c=0):
         pass
 
+    def assertAxs(self, dct, lst):
+        t1 = list(sorted(dct.keys()))
+        t2 = list(sorted(lst))
+        self.assertEqual(t1, t2)
+
     def test(self):
         td = os.path.join(self.td, "vfs")
         os.mkdir(td)
@@ -88,53 +94,53 @@ class TestVFS(unittest.TestCase):
         self.assertEqual(vfs.nodes, {})
         self.assertEqual(vfs.vpath, "")
         self.assertEqual(vfs.realpath, td)
-        self.assertEqual(vfs.uread, ["*"])
+        self.assertAxs(vfs.axs.uread, ["*"])
-        self.assertEqual(vfs.uwrite, ["*"])
+        self.assertAxs(vfs.axs.uwrite, ["*"])
 
         # single read-only rootfs (relative path)
         vfs = AuthSrv(Cfg(v=["a/ab/::r"]), self.log).vfs
         self.assertEqual(vfs.nodes, {})
         self.assertEqual(vfs.vpath, "")
         self.assertEqual(vfs.realpath, os.path.join(td, "a", "ab"))
-        self.assertEqual(vfs.uread, ["*"])
+        self.assertAxs(vfs.axs.uread, ["*"])
-        self.assertEqual(vfs.uwrite, [])
+        self.assertAxs(vfs.axs.uwrite, [])
 
         # single read-only rootfs (absolute path)
         vfs = AuthSrv(Cfg(v=[td + "//a/ac/../aa//::r"]), self.log).vfs
         self.assertEqual(vfs.nodes, {})
         self.assertEqual(vfs.vpath, "")
         self.assertEqual(vfs.realpath, os.path.join(td, "a", "aa"))
-        self.assertEqual(vfs.uread, ["*"])
+        self.assertAxs(vfs.axs.uread, ["*"])
-        self.assertEqual(vfs.uwrite, [])
+        self.assertAxs(vfs.axs.uwrite, [])
 
         # read-only rootfs with write-only subdirectory (read-write for k)
         vfs = AuthSrv(
-            Cfg(a=["k:k"], v=[".::r:ak", "a/ac/acb:a/ac/acb:w:ak"]),
+            Cfg(a=["k:k"], v=[".::r:rw,k", "a/ac/acb:a/ac/acb:w:rw,k"]),
             self.log,
         ).vfs
         self.assertEqual(len(vfs.nodes), 1)
         self.assertEqual(vfs.vpath, "")
         self.assertEqual(vfs.realpath, td)
-        self.assertEqual(vfs.uread, ["*", "k"])
+        self.assertAxs(vfs.axs.uread, ["*", "k"])
-        self.assertEqual(vfs.uwrite, ["k"])
+        self.assertAxs(vfs.axs.uwrite, ["k"])
         n = vfs.nodes["a"]
         self.assertEqual(len(vfs.nodes), 1)
         self.assertEqual(n.vpath, "a")
         self.assertEqual(n.realpath, os.path.join(td, "a"))
-        self.assertEqual(n.uread, ["*", "k"])
+        self.assertAxs(n.axs.uread, ["*", "k"])
-        self.assertEqual(n.uwrite, ["k"])
+        self.assertAxs(n.axs.uwrite, ["k"])
         n = n.nodes["ac"]
         self.assertEqual(len(vfs.nodes), 1)
         self.assertEqual(n.vpath, "a/ac")
         self.assertEqual(n.realpath, os.path.join(td, "a", "ac"))
-        self.assertEqual(n.uread, ["*", "k"])
+        self.assertAxs(n.axs.uread, ["*", "k"])
-        self.assertEqual(n.uwrite, ["k"])
+        self.assertAxs(n.axs.uwrite, ["k"])
         n = n.nodes["acb"]
         self.assertEqual(n.nodes, {})
         self.assertEqual(n.vpath, "a/ac/acb")
         self.assertEqual(n.realpath, os.path.join(td, "a", "ac", "acb"))
-        self.assertEqual(n.uread, ["k"])
+        self.assertAxs(n.axs.uread, ["k"])
-        self.assertEqual(n.uwrite, ["*", "k"])
+        self.assertAxs(n.axs.uwrite, ["*", "k"])
 
         # something funky about the windows path normalization,
         # doesn't really matter but makes the test messy, TODO?
@@ -173,24 +179,24 @@ class TestVFS(unittest.TestCase):
 
         # admin-only rootfs with all-read-only subfolder
         vfs = AuthSrv(
-            Cfg(a=["k:k"], v=[".::ak", "a:a:r"]),
+            Cfg(a=["k:k"], v=[".::rw,k", "a:a:r"]),
             self.log,
         ).vfs
         self.assertEqual(len(vfs.nodes), 1)
         self.assertEqual(vfs.vpath, "")
         self.assertEqual(vfs.realpath, td)
-        self.assertEqual(vfs.uread, ["k"])
+        self.assertAxs(vfs.axs.uread, ["k"])
-        self.assertEqual(vfs.uwrite, ["k"])
+        self.assertAxs(vfs.axs.uwrite, ["k"])
         n = vfs.nodes["a"]
         self.assertEqual(len(vfs.nodes), 1)
         self.assertEqual(n.vpath, "a")
         self.assertEqual(n.realpath, os.path.join(td, "a"))
-        self.assertEqual(n.uread, ["*"])
+        self.assertAxs(n.axs.uread, ["*"])
-        self.assertEqual(n.uwrite, [])
+        self.assertAxs(n.axs.uwrite, [])
-        self.assertEqual(vfs.can_access("/", "*"), [False, False])
+        self.assertEqual(vfs.can_access("/", "*"), [False, False, False, False])
-        self.assertEqual(vfs.can_access("/", "k"), [True, True])
+        self.assertEqual(vfs.can_access("/", "k"), [True, True, False, False])
-        self.assertEqual(vfs.can_access("/a", "*"), [True, False])
+        self.assertEqual(vfs.can_access("/a", "*"), [True, False, False, False])
-        self.assertEqual(vfs.can_access("/a", "k"), [True, False])
+        self.assertEqual(vfs.can_access("/a", "k"), [True, False, False, False])
 
         # breadth-first construction
         vfs = AuthSrv(
@@ -247,26 +253,26 @@ class TestVFS(unittest.TestCase):
                     ./src
                     /dst
                     r a
-                    a asd
+                    rw asd
                     """
                 ).encode("utf-8")
             )
 
         au = AuthSrv(Cfg(c=[cfg_path]), self.log)
-        self.assertEqual(au.user["a"], "123")
+        self.assertEqual(au.acct["a"], "123")
-        self.assertEqual(au.user["asd"], "fgh:jkl")
+        self.assertEqual(au.acct["asd"], "fgh:jkl")
         n = au.vfs
         # root was not defined, so PWD with no access to anyone
         self.assertEqual(n.vpath, "")
         self.assertEqual(n.realpath, None)
-        self.assertEqual(n.uread, [])
+        self.assertAxs(n.axs.uread, [])
-        self.assertEqual(n.uwrite, [])
+        self.assertAxs(n.axs.uwrite, [])
         self.assertEqual(len(n.nodes), 1)
         n = n.nodes["dst"]
         self.assertEqual(n.vpath, "dst")
         self.assertEqual(n.realpath, os.path.join(td, "src"))
-        self.assertEqual(n.uread, ["a", "asd"])
+        self.assertAxs(n.axs.uread, ["a", "asd"])
-        self.assertEqual(n.uwrite, ["asd"])
+        self.assertAxs(n.axs.uwrite, ["asd"])
         self.assertEqual(len(n.nodes), 0)
 
         os.unlink(cfg_path)

tests/util.py

@@ -31,7 +31,7 @@ if MACOS:
 from copyparty.util import Unrecv
 
 
-def runcmd(*argv):
+def runcmd(argv):
     p = sp.Popen(argv, stdout=sp.PIPE, stderr=sp.PIPE)
     stdout, stderr = p.communicate()
     stdout = stdout.decode("utf-8")
@@ -39,8 +39,8 @@ def runcmd(*argv):
     return [p.returncode, stdout, stderr]
 
 
-def chkcmd(*argv):
+def chkcmd(argv):
-    ok, sout, serr = runcmd(*argv)
+    ok, sout, serr = runcmd(argv)
     if ok != 0:
         raise Exception(serr)
 
@@ -60,12 +60,20 @@ def get_ramdisk():
 
     if os.path.exists("/Volumes"):
         # hdiutil eject /Volumes/cptd/
-        devname, _ = chkcmd("hdiutil", "attach", "-nomount", "ram://131072")
+        devname, _ = chkcmd("hdiutil attach -nomount ram://131072".split())
         devname = devname.strip()
         print("devname: [{}]".format(devname))
         for _ in range(10):
             try:
-                _, _ = chkcmd("diskutil", "eraseVolume", "HFS+", "cptd", devname)
+                _, _ = chkcmd(["diskutil", "eraseVolume", "HFS+", "cptd", devname])
+                with open("/Volumes/cptd/.metadata_never_index", "w") as f:
+                    f.write("orz")
+
+                try:
+                    shutil.rmtree("/Volumes/cptd/.fseventsd")
+                except:
+                    pass
+
                 return subdir("/Volumes/cptd")
             except Exception as ex:
                 print(repr(ex))
@@ -119,7 +127,7 @@ class VHttpConn(object):
         self.addr = ("127.0.0.1", "42069")
         self.args = args
         self.asrv = asrv
-        self.is_mp = False
+        self.nid = None
         self.log_func = log
         self.log_src = "a"
         self.lf_url = None