mirror of
https://github.com/kyantech/Palmr.git
synced 2025-10-22 22:02:00 +00:00
fix: issue allowing multiple files with the same name - auto-rename on upload and rename operations (#309)
This commit is contained in:
374
apps/docs/content/docs/3.2-beta/cleanup-orphan-files.mdx
Normal file
374
apps/docs/content/docs/3.2-beta/cleanup-orphan-files.mdx
Normal file
@@ -0,0 +1,374 @@
|
||||
---
|
||||
title: Cleanup Orphan Files
|
||||
icon: Trash2
|
||||
---
|
||||
|
||||
This guide provides detailed instructions on how to identify and remove orphan file records from your Palmr database. Orphan files are database entries that reference files that no longer exist in the storage system, typically resulting from failed uploads or interrupted transfers.
|
||||
|
||||
## When and why to use this tool
|
||||
|
||||
The orphan file cleanup script is designed to maintain database integrity by removing stale file records. Consider using this tool if:
|
||||
|
||||
- Users are experiencing "File not found" errors when attempting to download files that appear in the UI
|
||||
- You've identified failed uploads that left incomplete database records
|
||||
- You're performing routine database maintenance
|
||||
- You've migrated storage systems and need to verify file consistency
|
||||
- You need to free up quota space occupied by phantom file records
|
||||
|
||||
> **Note:** This script only removes **database records** for files that don't exist in storage. It does not delete physical files. Files that exist in storage will remain untouched.
|
||||
|
||||
## How the cleanup works
|
||||
|
||||
Palmr provides a maintenance script that scans all file records in the database and verifies their existence in the storage system (either filesystem or S3). The script operates in two modes:
|
||||
|
||||
- **Dry-run mode (default):** Identifies orphan files and displays what would be deleted without making any changes
|
||||
- **Confirmation mode:** Actually removes the orphan database records after explicit confirmation
|
||||
|
||||
The script maintains safety by:
|
||||
- Checking file existence before marking as orphan
|
||||
- Providing detailed statistics and file listings
|
||||
- Requiring explicit `--confirm` flag to delete records
|
||||
- Working with both filesystem and S3 storage providers
|
||||
- Preserving all files that exist in storage
|
||||
|
||||
## Understanding orphan files
|
||||
|
||||
### What are orphan files?
|
||||
|
||||
Orphan files occur when:
|
||||
|
||||
1. **Failed chunked uploads:** A large file upload starts, creates a database record, but the upload fails before completion
|
||||
2. **Interrupted transfers:** Network issues or server restarts interrupt file transfers mid-process
|
||||
3. **Manual deletions:** Files are manually deleted from storage without removing the database record
|
||||
4. **Storage migrations:** Files are moved or lost during storage system changes
|
||||
|
||||
### Why they cause problems
|
||||
|
||||
When orphan records exist in the database:
|
||||
- Users see files in the UI that cannot be downloaded
|
||||
- Download attempts result in "ENOENT: no such file or directory" errors
|
||||
- Storage quota calculations become inaccurate
|
||||
- The system returns 500 errors instead of proper 404 responses (in older versions)
|
||||
|
||||
### Renamed files with suffixes
|
||||
|
||||
Files with duplicate names are automatically renamed with suffixes (e.g., `file (1).png`, `file (2).png`). Sometimes the upload fails after the database record is created but before the physical file is saved, creating an orphan record with a suffix.
|
||||
|
||||
**Example:**
|
||||
```
|
||||
Database record: photo (1).png → objectName: user123/1758805195682-Rjn9at692HdR.png
|
||||
Physical file: Does not exist ❌
|
||||
```
|
||||
|
||||
## Step-by-step instructions
|
||||
|
||||
### 1. Access the server environment
|
||||
|
||||
**For Docker installations:**
|
||||
|
||||
```bash
|
||||
docker exec -it <container_name> /bin/sh
|
||||
cd /app/palmr-app
|
||||
```
|
||||
|
||||
**For bare-metal installations:**
|
||||
|
||||
```bash
|
||||
cd /path/to/palmr/apps/server
|
||||
```
|
||||
|
||||
### 2. Run the cleanup script in dry-run mode
|
||||
|
||||
First, run the script without the `--confirm` flag to see what would be deleted:
|
||||
|
||||
```bash
|
||||
pnpm cleanup:orphan-files
|
||||
```
|
||||
|
||||
This will:
|
||||
- Scan all file records in the database
|
||||
- Check if each file exists in storage
|
||||
- Display a summary of orphan files
|
||||
- Show what would be deleted (without actually deleting)
|
||||
|
||||
### 3. Review the output
|
||||
|
||||
The script will provide detailed information about orphan files:
|
||||
|
||||
```text
|
||||
Starting orphan file cleanup...
|
||||
Storage mode: Filesystem
|
||||
Found 7 files in database
|
||||
❌ Orphan: photo(1).png (cmddjchw80000gmiimqnxga2g/1758805195682-Rjn9at692HdR.png)
|
||||
❌ Orphan: document.pdf (cmddjchw80000gmiimqnxga2g/1758803757558-JQxlvF816UVo.pdf)
|
||||
|
||||
📊 Summary:
|
||||
Total files in DB: 7
|
||||
✅ Files with storage: 5
|
||||
❌ Orphan files: 2
|
||||
|
||||
🗑️ Orphan files to be deleted:
|
||||
- photo(1).png (0.76 MB) - cmddjchw80000gmiimqnxga2g/1758805195682-Rjn9at692HdR.png
|
||||
- document.pdf (2.45 MB) - cmddjchw80000gmiimqnxga2g/1758803757558-JQxlvF816UVo.pdf
|
||||
|
||||
⚠️ Dry run mode. To actually delete orphan records, run with --confirm flag:
|
||||
pnpm cleanup:orphan-files:confirm
|
||||
```
|
||||
|
||||
### 4. Confirm and execute the cleanup
|
||||
|
||||
If you're satisfied with the results and want to proceed with the deletion:
|
||||
|
||||
```bash
|
||||
pnpm cleanup:orphan-files:confirm
|
||||
```
|
||||
|
||||
This will remove the orphan database records and display a confirmation:
|
||||
|
||||
```text
|
||||
🗑️ Deleting orphan file records...
|
||||
✓ Deleted: photo(1).png
|
||||
✓ Deleted: document.pdf
|
||||
|
||||
✅ Cleanup complete!
|
||||
Deleted 2 orphan file records
|
||||
```
|
||||
|
||||
## Example session
|
||||
|
||||
Below is a complete example of running the cleanup script:
|
||||
|
||||
```bash
|
||||
$ pnpm cleanup:orphan-files
|
||||
|
||||
> palmr-api@3.2.3-beta cleanup:orphan-files
|
||||
> tsx src/scripts/cleanup-orphan-files.ts
|
||||
|
||||
Starting orphan file cleanup...
|
||||
Storage mode: Filesystem
|
||||
Found 15 files in database
|
||||
❌ Orphan: video.mp4 (user123/1758803869037-1WhtnrQioeFQ.mp4)
|
||||
❌ Orphan: image(1).png (user123/1758805195682-Rjn9at692HdR.png)
|
||||
❌ Orphan: image(2).png (user123/1758803757558-JQxlvF816UVo.png)
|
||||
|
||||
📊 Summary:
|
||||
Total files in DB: 15
|
||||
✅ Files with storage: 12
|
||||
❌ Orphan files: 3
|
||||
|
||||
🗑️ Orphan files to be deleted:
|
||||
- video.mp4 (97.09 MB) - user123/1758803869037-1WhtnrQioeFQ.mp4
|
||||
- image(1).png (0.01 MB) - user123/1758805195682-Rjn9at692HdR.png
|
||||
- image(2).png (0.76 MB) - user123/1758803757558-JQxlvF816UVo.png
|
||||
|
||||
⚠️ Dry run mode. To actually delete orphan records, run with --confirm flag:
|
||||
pnpm cleanup:orphan-files:confirm
|
||||
|
||||
$ pnpm cleanup:orphan-files:confirm
|
||||
|
||||
> palmr-api@3.2.3-beta cleanup:orphan-files:confirm
|
||||
> tsx src/scripts/cleanup-orphan-files.ts --confirm
|
||||
|
||||
Starting orphan file cleanup...
|
||||
Storage mode: Filesystem
|
||||
Found 15 files in database
|
||||
❌ Orphan: video.mp4 (user123/1758803869037-1WhtnrQioeFQ.mp4)
|
||||
❌ Orphan: image(1).png (user123/1758805195682-Rjn9at692HdR.png)
|
||||
❌ Orphan: image(2).png (user123/1758803757558-JQxlvF816UVo.png)
|
||||
|
||||
📊 Summary:
|
||||
Total files in DB: 15
|
||||
✅ Files with storage: 12
|
||||
❌ Orphan files: 3
|
||||
|
||||
🗑️ Orphan files to be deleted:
|
||||
- video.mp4 (97.09 MB) - user123/1758803869037-1WhtnrQioeFQ.mp4
|
||||
- image(1).png (0.01 MB) - user123/1758805195682-Rjn9at692HdR.png
|
||||
- image(2).png (0.76 MB) - user123/1758803757558-JQxlvF816UVo.png
|
||||
|
||||
🗑️ Deleting orphan file records...
|
||||
✓ Deleted: video.mp4
|
||||
✓ Deleted: image(1).png
|
||||
✓ Deleted: image(2).png
|
||||
|
||||
✅ Cleanup complete!
|
||||
Deleted 3 orphan file records
|
||||
|
||||
Script completed successfully
|
||||
```
|
||||
|
||||
## Troubleshooting common issues
|
||||
|
||||
### No orphan files found
|
||||
|
||||
```text
|
||||
📊 Summary:
|
||||
Total files in DB: 10
|
||||
✅ Files with storage: 10
|
||||
❌ Orphan files: 0
|
||||
|
||||
✨ No orphan files found!
|
||||
```
|
||||
|
||||
**This is good!** It means your database is in sync with your storage system.
|
||||
|
||||
### Script cannot connect to database
|
||||
|
||||
If you see database connection errors:
|
||||
|
||||
1. Verify the database file exists:
|
||||
```bash
|
||||
ls -la prisma/palmr.db
|
||||
```
|
||||
|
||||
2. Check database permissions:
|
||||
```bash
|
||||
chmod 644 prisma/palmr.db
|
||||
```
|
||||
|
||||
3. Ensure you're in the correct directory:
|
||||
```bash
|
||||
pwd # Should show .../palmr/apps/server
|
||||
```
|
||||
|
||||
### Storage provider errors
|
||||
|
||||
For **S3 storage:**
|
||||
- Verify your S3 credentials are configured correctly
|
||||
- Check that the bucket is accessible
|
||||
- Ensure network connectivity to S3
|
||||
|
||||
For **Filesystem storage:**
|
||||
- Verify the uploads directory exists and is readable
|
||||
- Check file system permissions
|
||||
- Ensure sufficient disk space
|
||||
|
||||
### Script fails to delete records
|
||||
|
||||
If deletion fails for specific files:
|
||||
- Check database locks (close other connections)
|
||||
- Verify you have write permissions to the database
|
||||
- Review the error message for specific details
|
||||
|
||||
## Understanding the output
|
||||
|
||||
### File statistics
|
||||
|
||||
The script provides several key metrics:
|
||||
|
||||
- **Total files in DB:** All file records in your database
|
||||
- **Files with storage:** Records where the physical file exists
|
||||
- **Orphan files:** Records where the physical file is missing
|
||||
|
||||
### File information
|
||||
|
||||
For each orphan file, you'll see:
|
||||
|
||||
- **Name:** Display name in the UI
|
||||
- **Size:** File size as recorded in the database
|
||||
- **Object name:** Internal storage path
|
||||
|
||||
Example: `photo(1).png (0.76 MB) - user123/1758805195682-Rjn9at692HdR.png`
|
||||
|
||||
## Prevention and best practices
|
||||
|
||||
### Prevent orphan files from occurring
|
||||
|
||||
1. **Monitor upload failures:** Check server logs for upload errors
|
||||
2. **Stable network:** Ensure reliable network connectivity for large uploads
|
||||
3. **Adequate resources:** Provide sufficient disk space and memory
|
||||
4. **Regular maintenance:** Run this script periodically as part of maintenance
|
||||
|
||||
### When to run cleanup
|
||||
|
||||
Consider running the cleanup script:
|
||||
|
||||
- **Monthly:** As part of routine database maintenance
|
||||
- **After incidents:** Following server crashes or storage issues
|
||||
- **Before migrations:** Before moving to new storage systems
|
||||
- **When users report errors:** If download failures are reported
|
||||
|
||||
### Safe cleanup practices
|
||||
|
||||
1. **Always run dry-run first:** Review what will be deleted before confirming
|
||||
2. **Backup your database:** Create a backup before running with `--confirm`
|
||||
3. **Check during low usage:** Run during off-peak hours to minimize disruption
|
||||
4. **Document the cleanup:** Keep records of when and why cleanup was performed
|
||||
5. **Verify after cleanup:** Check that file counts match expectations
|
||||
|
||||
## Technical details
|
||||
|
||||
### How files are stored
|
||||
|
||||
When files are uploaded to Palmr:
|
||||
|
||||
1. Frontend generates a safe object name using random identifiers
|
||||
2. Backend creates the final `objectName` as: `${userId}/${timestamp}-${randomId}.${extension}`
|
||||
3. If a duplicate name exists, the **display name** gets a suffix, but `objectName` remains unique
|
||||
4. Physical file is stored using `objectName`, display name is stored separately in database
|
||||
|
||||
### Storage providers
|
||||
|
||||
The script works with both storage providers:
|
||||
|
||||
- **FilesystemStorageProvider:** Uses `fs.promises.access()` to check file existence
|
||||
- **S3StorageProvider:** Uses `HeadObjectCommand` to verify objects in S3 bucket
|
||||
|
||||
### Database schema
|
||||
|
||||
Files table structure:
|
||||
```typescript
|
||||
{
|
||||
name: string // Display name (can have suffixes like "file (1).png")
|
||||
objectName: string // Physical storage path (always unique)
|
||||
size: bigint // File size in bytes
|
||||
extension: string // File extension
|
||||
userId: string // Owner of the file
|
||||
folderId: string? // Parent folder (null for root)
|
||||
}
|
||||
```
|
||||
|
||||
## Related improvements
|
||||
|
||||
### Download validation (v3.2.3-beta+)
|
||||
|
||||
Starting from version 3.2.3-beta, Palmr includes enhanced download validation:
|
||||
|
||||
- Files are checked for existence **before** attempting download
|
||||
- Returns proper 404 error if file is missing (instead of 500)
|
||||
- Provides helpful error message to users
|
||||
|
||||
This prevents errors when trying to download orphan files that haven't been cleaned up yet.
|
||||
|
||||
## Security considerations
|
||||
|
||||
- **Read-only by default:** Dry-run mode is safe and doesn't modify data
|
||||
- **Explicit confirmation:** Requires `--confirm` flag to delete records
|
||||
- **No file deletion:** Only removes database records, never deletes physical files
|
||||
- **Audit trail:** All actions are logged to console
|
||||
- **Permission-based:** Only users with server access can run the script
|
||||
|
||||
> **Important:** This script does not delete physical files from storage. It only removes database records for files that don't exist. This is intentional to prevent accidental data loss.
|
||||
|
||||
## FAQ
|
||||
|
||||
**Q: Will this delete my files?**
|
||||
A: No. The script only removes database records for files that are already missing from storage. Physical files are never deleted.
|
||||
|
||||
**Q: Can I undo the cleanup?**
|
||||
A: No. Once orphan records are deleted, they cannot be recovered. Always run dry-run mode first and backup your database.
|
||||
|
||||
**Q: Why do orphan files have suffixes like (1), (2)?**
|
||||
A: When duplicate files are uploaded, Palmr renames them with suffixes. If the upload fails after creating the database record, an orphan with a suffix remains.
|
||||
|
||||
**Q: How often should I run this script?**
|
||||
A: Monthly maintenance is usually sufficient. Run more frequently if you experience many upload failures.
|
||||
|
||||
**Q: Does this work with S3 storage?**
|
||||
A: Yes! The script automatically detects your storage provider (filesystem or S3) and works with both.
|
||||
|
||||
**Q: What if I have thousands of orphan files?**
|
||||
A: The script handles large numbers efficiently. Consider running during off-peak hours for very large cleanups.
|
||||
|
||||
**Q: Can this fix "File not found" errors?**
|
||||
A: Yes, if the errors are caused by orphan database records. The script removes those records, preventing future errors.
|
@@ -16,6 +16,7 @@
|
||||
"reverse-proxy-configuration",
|
||||
"download-memory-management",
|
||||
"password-reset-without-smtp",
|
||||
"cleanup-orphan-files",
|
||||
"oidc-authentication",
|
||||
"troubleshooting",
|
||||
"---Developers---",
|
||||
|
1
apps/server/.gitignore
vendored
1
apps/server/.gitignore
vendored
@@ -4,3 +4,4 @@ dist/*
|
||||
uploads/*
|
||||
temp-uploads/*
|
||||
prisma/*.db
|
||||
tsconfig.tsbuildinfo
|
||||
|
@@ -25,7 +25,9 @@
|
||||
"format:check": "prettier . --check",
|
||||
"type-check": "npx tsc --noEmit",
|
||||
"validate": "pnpm lint && pnpm type-check",
|
||||
"db:seed": "ts-node prisma/seed.js"
|
||||
"db:seed": "ts-node prisma/seed.js",
|
||||
"cleanup:orphan-files": "tsx src/scripts/cleanup-orphan-files.ts",
|
||||
"cleanup:orphan-files:confirm": "tsx src/scripts/cleanup-orphan-files.ts --confirm"
|
||||
},
|
||||
"prisma": {
|
||||
"seed": "node prisma/seed.js"
|
||||
@@ -77,4 +79,4 @@
|
||||
"tsx": "^4.19.2",
|
||||
"typescript": "^5.7.3"
|
||||
}
|
||||
}
|
||||
}
|
||||
|
4498
apps/server/pnpm-lock.yaml
generated
4498
apps/server/pnpm-lock.yaml
generated
File diff suppressed because it is too large
Load Diff
@@ -59,7 +59,6 @@ model File {
|
||||
shares Share[] @relation("ShareFiles")
|
||||
|
||||
@@index([folderId])
|
||||
|
||||
@@map("files")
|
||||
}
|
||||
|
||||
@@ -278,40 +277,40 @@ enum PageLayout {
|
||||
}
|
||||
|
||||
model TrustedDevice {
|
||||
id String @id @default(cuid())
|
||||
userId String
|
||||
user User @relation(fields: [userId], references: [id], onDelete: Cascade)
|
||||
deviceHash String @unique
|
||||
deviceName String?
|
||||
userAgent String?
|
||||
ipAddress String?
|
||||
lastUsedAt DateTime @default(now())
|
||||
expiresAt DateTime
|
||||
createdAt DateTime @default(now())
|
||||
updatedAt DateTime @updatedAt
|
||||
id String @id @default(cuid())
|
||||
userId String
|
||||
user User @relation(fields: [userId], references: [id], onDelete: Cascade)
|
||||
deviceHash String @unique
|
||||
deviceName String?
|
||||
userAgent String?
|
||||
ipAddress String?
|
||||
lastUsedAt DateTime @default(now())
|
||||
expiresAt DateTime
|
||||
createdAt DateTime @default(now())
|
||||
updatedAt DateTime @updatedAt
|
||||
|
||||
@@map("trusted_devices")
|
||||
}
|
||||
|
||||
model Folder {
|
||||
id String @id @default(cuid())
|
||||
id String @id @default(cuid())
|
||||
name String
|
||||
description String?
|
||||
objectName String
|
||||
|
||||
parentId String?
|
||||
parent Folder? @relation("FolderHierarchy", fields: [parentId], references: [id], onDelete: Cascade)
|
||||
children Folder[] @relation("FolderHierarchy")
|
||||
parentId String?
|
||||
parent Folder? @relation("FolderHierarchy", fields: [parentId], references: [id], onDelete: Cascade)
|
||||
children Folder[] @relation("FolderHierarchy")
|
||||
|
||||
userId String
|
||||
user User @relation(fields: [userId], references: [id], onDelete: Cascade)
|
||||
userId String
|
||||
user User @relation(fields: [userId], references: [id], onDelete: Cascade)
|
||||
|
||||
files File[]
|
||||
files File[]
|
||||
|
||||
shares Share[] @relation("ShareFolders")
|
||||
shares Share[] @relation("ShareFolders")
|
||||
|
||||
createdAt DateTime @default(now())
|
||||
updatedAt DateTime @updatedAt
|
||||
createdAt DateTime @default(now())
|
||||
updatedAt DateTime @updatedAt
|
||||
|
||||
@@index([userId])
|
||||
@@index([parentId])
|
||||
|
@@ -3,6 +3,11 @@ import { FastifyReply, FastifyRequest } from "fastify";
|
||||
|
||||
import { env } from "../../env";
|
||||
import { prisma } from "../../shared/prisma";
|
||||
import {
|
||||
generateUniqueFileName,
|
||||
generateUniqueFileNameForRename,
|
||||
parseFileName,
|
||||
} from "../../utils/file-name-generator";
|
||||
import { ConfigService } from "../config/service";
|
||||
import {
|
||||
CheckFileInput,
|
||||
@@ -93,9 +98,13 @@ export class FileController {
|
||||
}
|
||||
}
|
||||
|
||||
// Parse the filename and generate a unique name if there's a duplicate
|
||||
const { baseName, extension } = parseFileName(input.name);
|
||||
const uniqueName = await generateUniqueFileName(baseName, extension, userId, input.folderId);
|
||||
|
||||
const fileRecord = await prisma.file.create({
|
||||
data: {
|
||||
name: input.name,
|
||||
name: uniqueName,
|
||||
description: input.description,
|
||||
extension: input.extension,
|
||||
size: BigInt(input.size),
|
||||
@@ -169,9 +178,20 @@ export class FileController {
|
||||
});
|
||||
}
|
||||
|
||||
return reply.status(201).send({
|
||||
// Check for duplicate filename and provide the suggested unique name
|
||||
const { baseName, extension } = parseFileName(input.name);
|
||||
const uniqueName = await generateUniqueFileName(baseName, extension, userId, input.folderId);
|
||||
|
||||
// Include suggestedName in response if the name was changed
|
||||
const response: any = {
|
||||
message: "File checks succeeded.",
|
||||
});
|
||||
};
|
||||
|
||||
if (uniqueName !== input.name) {
|
||||
response.suggestedName = uniqueName;
|
||||
}
|
||||
|
||||
return reply.status(201).send(response);
|
||||
} catch (error: any) {
|
||||
console.error("Error in checkFile:", error);
|
||||
return reply.status(400).send({ error: error.message });
|
||||
@@ -359,6 +379,13 @@ export class FileController {
|
||||
return reply.status(403).send({ error: "Access denied." });
|
||||
}
|
||||
|
||||
// If renaming the file, check for duplicates and auto-rename if necessary
|
||||
if (updateData.name && updateData.name !== fileRecord.name) {
|
||||
const { baseName, extension } = parseFileName(updateData.name);
|
||||
const uniqueName = await generateUniqueFileNameForRename(baseName, extension, userId, fileRecord.folderId, id);
|
||||
updateData.name = uniqueName;
|
||||
}
|
||||
|
||||
const updatedFile = await prisma.file.update({
|
||||
where: { id },
|
||||
data: updateData,
|
||||
|
@@ -203,6 +203,17 @@ export class FilesystemController {
|
||||
}
|
||||
|
||||
const filePath = provider.getFilePath(tokenData.objectName);
|
||||
|
||||
const fileExists = await provider.fileExists(tokenData.objectName);
|
||||
if (!fileExists) {
|
||||
console.error(`[DOWNLOAD] File not found: ${tokenData.objectName}`);
|
||||
return reply.status(404).send({
|
||||
error: "File not found",
|
||||
message:
|
||||
"The requested file does not exist on the server. It may have been deleted or the upload was incomplete.",
|
||||
});
|
||||
}
|
||||
|
||||
const stats = await fs.promises.stat(filePath);
|
||||
const fileSize = stats.size;
|
||||
const fileName = tokenData.fileName || "download";
|
||||
|
@@ -35,21 +35,13 @@ export class FolderController {
|
||||
}
|
||||
}
|
||||
|
||||
const existingFolder = await prisma.folder.findFirst({
|
||||
where: {
|
||||
name: input.name,
|
||||
parentId: input.parentId || null,
|
||||
userId,
|
||||
},
|
||||
});
|
||||
|
||||
if (existingFolder) {
|
||||
return reply.status(400).send({ error: "A folder with this name already exists in this location" });
|
||||
}
|
||||
// Check for duplicates and auto-rename if necessary
|
||||
const { generateUniqueFolderName } = await import("../../utils/file-name-generator.js");
|
||||
const uniqueName = await generateUniqueFolderName(input.name, userId, input.parentId);
|
||||
|
||||
const folderRecord = await prisma.folder.create({
|
||||
data: {
|
||||
name: input.name,
|
||||
name: uniqueName,
|
||||
description: input.description,
|
||||
objectName: input.objectName,
|
||||
parentId: input.parentId,
|
||||
@@ -231,19 +223,11 @@ export class FolderController {
|
||||
return reply.status(403).send({ error: "Access denied." });
|
||||
}
|
||||
|
||||
// If renaming the folder, check for duplicates and auto-rename if necessary
|
||||
if (updateData.name && updateData.name !== folderRecord.name) {
|
||||
const duplicateFolder = await prisma.folder.findFirst({
|
||||
where: {
|
||||
name: updateData.name,
|
||||
parentId: folderRecord.parentId,
|
||||
userId,
|
||||
id: { not: id },
|
||||
},
|
||||
});
|
||||
|
||||
if (duplicateFolder) {
|
||||
return reply.status(400).send({ error: "A folder with this name already exists in this location" });
|
||||
}
|
||||
const { generateUniqueFolderName } = await import("../../utils/file-name-generator.js");
|
||||
const uniqueName = await generateUniqueFolderName(updateData.name, userId, folderRecord.parentId, id);
|
||||
updateData.name = uniqueName;
|
||||
}
|
||||
|
||||
const updatedFolder = await prisma.folder.update({
|
||||
|
@@ -1,4 +1,4 @@
|
||||
import { DeleteObjectCommand, GetObjectCommand, PutObjectCommand } from "@aws-sdk/client-s3";
|
||||
import { DeleteObjectCommand, GetObjectCommand, HeadObjectCommand, PutObjectCommand } from "@aws-sdk/client-s3";
|
||||
import { getSignedUrl } from "@aws-sdk/s3-request-presigner";
|
||||
|
||||
import { bucketName, s3Client } from "../config/storage.config";
|
||||
@@ -122,4 +122,25 @@ export class S3StorageProvider implements StorageProvider {
|
||||
|
||||
await s3Client.send(command);
|
||||
}
|
||||
|
||||
async fileExists(objectName: string): Promise<boolean> {
|
||||
if (!s3Client) {
|
||||
throw new Error("S3 client is not available");
|
||||
}
|
||||
|
||||
try {
|
||||
const command = new HeadObjectCommand({
|
||||
Bucket: bucketName,
|
||||
Key: objectName,
|
||||
});
|
||||
|
||||
await s3Client.send(command);
|
||||
return true;
|
||||
} catch (error: any) {
|
||||
if (error.name === "NotFound" || error.$metadata?.httpStatusCode === 404) {
|
||||
return false;
|
||||
}
|
||||
throw error;
|
||||
}
|
||||
}
|
||||
}
|
||||
|
102
apps/server/src/scripts/cleanup-orphan-files.ts
Normal file
102
apps/server/src/scripts/cleanup-orphan-files.ts
Normal file
@@ -0,0 +1,102 @@
|
||||
import { isS3Enabled } from "../config/storage.config";
|
||||
import { FilesystemStorageProvider } from "../providers/filesystem-storage.provider";
|
||||
import { S3StorageProvider } from "../providers/s3-storage.provider";
|
||||
import { prisma } from "../shared/prisma";
|
||||
import { StorageProvider } from "../types/storage";
|
||||
|
||||
/**
|
||||
* Script to clean up orphan file records in the database
|
||||
* (files that are registered in DB but don't exist in storage)
|
||||
*/
|
||||
async function cleanupOrphanFiles() {
|
||||
console.log("Starting orphan file cleanup...");
|
||||
console.log(`Storage mode: ${isS3Enabled ? "S3" : "Filesystem"}`);
|
||||
|
||||
let storageProvider: StorageProvider;
|
||||
if (isS3Enabled) {
|
||||
storageProvider = new S3StorageProvider();
|
||||
} else {
|
||||
storageProvider = FilesystemStorageProvider.getInstance();
|
||||
}
|
||||
|
||||
// Get all files from database
|
||||
const allFiles = await prisma.file.findMany({
|
||||
select: {
|
||||
id: true,
|
||||
name: true,
|
||||
objectName: true,
|
||||
userId: true,
|
||||
size: true,
|
||||
},
|
||||
});
|
||||
|
||||
console.log(`Found ${allFiles.length} files in database`);
|
||||
|
||||
const orphanFiles: typeof allFiles = [];
|
||||
const existingFiles: typeof allFiles = [];
|
||||
|
||||
// Check each file
|
||||
for (const file of allFiles) {
|
||||
const exists = await storageProvider.fileExists(file.objectName);
|
||||
if (!exists) {
|
||||
orphanFiles.push(file);
|
||||
console.log(`❌ Orphan: ${file.name} (${file.objectName})`);
|
||||
} else {
|
||||
existingFiles.push(file);
|
||||
}
|
||||
}
|
||||
|
||||
console.log(`\n📊 Summary:`);
|
||||
console.log(` Total files in DB: ${allFiles.length}`);
|
||||
console.log(` ✅ Files with storage: ${existingFiles.length}`);
|
||||
console.log(` ❌ Orphan files: ${orphanFiles.length}`);
|
||||
|
||||
if (orphanFiles.length === 0) {
|
||||
console.log("\n✨ No orphan files found!");
|
||||
return;
|
||||
}
|
||||
|
||||
console.log(`\n🗑️ Orphan files to be deleted:`);
|
||||
orphanFiles.forEach((file) => {
|
||||
const sizeMB = Number(file.size) / (1024 * 1024);
|
||||
console.log(` - ${file.name} (${sizeMB.toFixed(2)} MB) - ${file.objectName}`);
|
||||
});
|
||||
|
||||
// Ask for confirmation (if running interactively)
|
||||
const shouldDelete = process.argv.includes("--confirm");
|
||||
|
||||
if (!shouldDelete) {
|
||||
console.log(`\n⚠️ Dry run mode. To actually delete orphan records, run with --confirm flag:`);
|
||||
console.log(` node dist/scripts/cleanup-orphan-files.js --confirm`);
|
||||
return;
|
||||
}
|
||||
|
||||
console.log(`\n🗑️ Deleting orphan file records...`);
|
||||
|
||||
let deletedCount = 0;
|
||||
for (const file of orphanFiles) {
|
||||
try {
|
||||
await prisma.file.delete({
|
||||
where: { id: file.id },
|
||||
});
|
||||
deletedCount++;
|
||||
console.log(` ✓ Deleted: ${file.name}`);
|
||||
} catch (error) {
|
||||
console.error(` ✗ Failed to delete ${file.name}:`, error);
|
||||
}
|
||||
}
|
||||
|
||||
console.log(`\n✅ Cleanup complete!`);
|
||||
console.log(` Deleted ${deletedCount} orphan file records`);
|
||||
}
|
||||
|
||||
// Run the cleanup
|
||||
cleanupOrphanFiles()
|
||||
.then(() => {
|
||||
console.log("\nScript completed successfully");
|
||||
process.exit(0);
|
||||
})
|
||||
.catch((error) => {
|
||||
console.error("\n❌ Script failed:", error);
|
||||
process.exit(1);
|
||||
});
|
@@ -2,6 +2,7 @@ export interface StorageProvider {
|
||||
getPresignedPutUrl(objectName: string, expires: number): Promise<string>;
|
||||
getPresignedGetUrl(objectName: string, expires: number, fileName?: string): Promise<string>;
|
||||
deleteObject(objectName: string): Promise<void>;
|
||||
fileExists(objectName: string): Promise<boolean>;
|
||||
}
|
||||
|
||||
export interface StorageConfig {
|
||||
|
206
apps/server/src/utils/file-name-generator.ts
Normal file
206
apps/server/src/utils/file-name-generator.ts
Normal file
@@ -0,0 +1,206 @@
|
||||
import { prisma } from "../shared/prisma";
|
||||
|
||||
/**
|
||||
* Generates a unique filename by checking for duplicates in the database
|
||||
* and appending a numeric suffix if necessary (e.g., file (1).txt, file (2).txt)
|
||||
*
|
||||
* @param baseName - The original filename without extension
|
||||
* @param extension - The file extension
|
||||
* @param userId - The user ID who owns the file
|
||||
* @param folderId - The folder ID where the file will be stored (null for root)
|
||||
* @returns A unique filename with extension
|
||||
*/
|
||||
export async function generateUniqueFileName(
|
||||
baseName: string,
|
||||
extension: string,
|
||||
userId: string,
|
||||
folderId: string | null | undefined
|
||||
): Promise<string> {
|
||||
const fullName = `${baseName}.${extension}`;
|
||||
const targetFolderId = folderId || null;
|
||||
|
||||
// Check if the original filename exists in the target folder
|
||||
const existingFile = await prisma.file.findFirst({
|
||||
where: {
|
||||
name: fullName,
|
||||
userId,
|
||||
folderId: targetFolderId,
|
||||
},
|
||||
});
|
||||
|
||||
// If no duplicate, return the original name
|
||||
if (!existingFile) {
|
||||
return fullName;
|
||||
}
|
||||
|
||||
// Find the next available suffix number
|
||||
let suffix = 1;
|
||||
let uniqueName = `${baseName} (${suffix}).${extension}`;
|
||||
|
||||
while (true) {
|
||||
const duplicateFile = await prisma.file.findFirst({
|
||||
where: {
|
||||
name: uniqueName,
|
||||
userId,
|
||||
folderId: targetFolderId,
|
||||
},
|
||||
});
|
||||
|
||||
if (!duplicateFile) {
|
||||
return uniqueName;
|
||||
}
|
||||
|
||||
suffix++;
|
||||
uniqueName = `${baseName} (${suffix}).${extension}`;
|
||||
}
|
||||
}
|
||||
|
||||
/**
|
||||
* Generates a unique filename for rename operations by checking for duplicates
|
||||
* and appending a numeric suffix if necessary (e.g., file (1).txt, file (2).txt)
|
||||
*
|
||||
* @param baseName - The original filename without extension
|
||||
* @param extension - The file extension
|
||||
* @param userId - The user ID who owns the file
|
||||
* @param folderId - The folder ID where the file will be stored (null for root)
|
||||
* @param excludeFileId - The ID of the file being renamed (to exclude from duplicate check)
|
||||
* @returns A unique filename with extension
|
||||
*/
|
||||
export async function generateUniqueFileNameForRename(
|
||||
baseName: string,
|
||||
extension: string,
|
||||
userId: string,
|
||||
folderId: string | null | undefined,
|
||||
excludeFileId: string
|
||||
): Promise<string> {
|
||||
const fullName = `${baseName}.${extension}`;
|
||||
const targetFolderId = folderId || null;
|
||||
|
||||
// Check if the original filename exists in the target folder (excluding current file)
|
||||
const existingFile = await prisma.file.findFirst({
|
||||
where: {
|
||||
name: fullName,
|
||||
userId,
|
||||
folderId: targetFolderId,
|
||||
id: { not: excludeFileId },
|
||||
},
|
||||
});
|
||||
|
||||
// If no duplicate, return the original name
|
||||
if (!existingFile) {
|
||||
return fullName;
|
||||
}
|
||||
|
||||
// Find the next available suffix number
|
||||
let suffix = 1;
|
||||
let uniqueName = `${baseName} (${suffix}).${extension}`;
|
||||
|
||||
while (true) {
|
||||
const duplicateFile = await prisma.file.findFirst({
|
||||
where: {
|
||||
name: uniqueName,
|
||||
userId,
|
||||
folderId: targetFolderId,
|
||||
id: { not: excludeFileId },
|
||||
},
|
||||
});
|
||||
|
||||
if (!duplicateFile) {
|
||||
return uniqueName;
|
||||
}
|
||||
|
||||
suffix++;
|
||||
uniqueName = `${baseName} (${suffix}).${extension}`;
|
||||
}
|
||||
}
|
||||
|
||||
/**
|
||||
* Generates a unique folder name by checking for duplicates in the database
|
||||
* and appending a numeric suffix if necessary (e.g., folder (1), folder (2))
|
||||
*
|
||||
* @param name - The original folder name
|
||||
* @param userId - The user ID who owns the folder
|
||||
* @param parentId - The parent folder ID (null for root)
|
||||
* @param excludeFolderId - The ID of the folder being renamed (to exclude from duplicate check)
|
||||
* @returns A unique folder name
|
||||
*/
|
||||
export async function generateUniqueFolderName(
|
||||
name: string,
|
||||
userId: string,
|
||||
parentId: string | null | undefined,
|
||||
excludeFolderId?: string
|
||||
): Promise<string> {
|
||||
const targetParentId = parentId || null;
|
||||
|
||||
// Build the where clause
|
||||
const whereClause: any = {
|
||||
name,
|
||||
userId,
|
||||
parentId: targetParentId,
|
||||
};
|
||||
|
||||
// Exclude the current folder if this is a rename operation
|
||||
if (excludeFolderId) {
|
||||
whereClause.id = { not: excludeFolderId };
|
||||
}
|
||||
|
||||
// Check if the original folder name exists in the target location
|
||||
const existingFolder = await prisma.folder.findFirst({
|
||||
where: whereClause,
|
||||
});
|
||||
|
||||
// If no duplicate, return the original name
|
||||
if (!existingFolder) {
|
||||
return name;
|
||||
}
|
||||
|
||||
// Find the next available suffix number
|
||||
let suffix = 1;
|
||||
let uniqueName = `${name} (${suffix})`;
|
||||
|
||||
while (true) {
|
||||
const whereClauseForSuffix: any = {
|
||||
name: uniqueName,
|
||||
userId,
|
||||
parentId: targetParentId,
|
||||
};
|
||||
|
||||
if (excludeFolderId) {
|
||||
whereClauseForSuffix.id = { not: excludeFolderId };
|
||||
}
|
||||
|
||||
const duplicateFolder = await prisma.folder.findFirst({
|
||||
where: whereClauseForSuffix,
|
||||
});
|
||||
|
||||
if (!duplicateFolder) {
|
||||
return uniqueName;
|
||||
}
|
||||
|
||||
suffix++;
|
||||
uniqueName = `${name} (${suffix})`;
|
||||
}
|
||||
}
|
||||
|
||||
/**
|
||||
* Parses a filename into base name and extension
|
||||
*
|
||||
* @param filename - The full filename with extension
|
||||
* @returns Object with baseName and extension
|
||||
*/
|
||||
export function parseFileName(filename: string): { baseName: string; extension: string } {
|
||||
const lastDotIndex = filename.lastIndexOf(".");
|
||||
|
||||
if (lastDotIndex === -1 || lastDotIndex === 0) {
|
||||
// No extension or hidden file with no name before dot
|
||||
return {
|
||||
baseName: filename,
|
||||
extension: "",
|
||||
};
|
||||
}
|
||||
|
||||
return {
|
||||
baseName: filename.substring(0, lastDotIndex),
|
||||
extension: filename.substring(lastDotIndex + 1),
|
||||
};
|
||||
}
|
@@ -38,8 +38,8 @@ async function getAppInfo() {
|
||||
}
|
||||
}
|
||||
|
||||
function getBaseUrl(): string {
|
||||
const headersList = headers();
|
||||
async function getBaseUrl(): Promise<string> {
|
||||
const headersList = await headers();
|
||||
const protocol = headersList.get("x-forwarded-proto") || "http";
|
||||
const host = headersList.get("x-forwarded-host") || headersList.get("host") || "localhost:3000";
|
||||
return `${protocol}://${host}`;
|
||||
@@ -57,7 +57,7 @@ export async function generateMetadata({ params }: { params: { alias: string } }
|
||||
? t("reverseShares.upload.metadata.descriptionWithLimit", { limit: metadata.maxFiles })
|
||||
: t("reverseShares.upload.metadata.description"));
|
||||
|
||||
const baseUrl = getBaseUrl();
|
||||
const baseUrl = await getBaseUrl();
|
||||
const shareUrl = `${baseUrl}/r/${params.alias}`;
|
||||
|
||||
return {
|
||||
|
@@ -43,8 +43,8 @@ async function getAppInfo() {
|
||||
}
|
||||
}
|
||||
|
||||
function getBaseUrl(): string {
|
||||
const headersList = headers();
|
||||
async function getBaseUrl(): Promise<string> {
|
||||
const headersList = await headers();
|
||||
const protocol = headersList.get("x-forwarded-proto") || "http";
|
||||
const host = headersList.get("x-forwarded-host") || headersList.get("host") || "localhost:3000";
|
||||
return `${protocol}://${host}`;
|
||||
@@ -62,7 +62,7 @@ export async function generateMetadata({ params }: { params: { alias: string } }
|
||||
? t("share.metadata.filesShared", { count: metadata.totalFiles + (metadata.totalFolders || 0) })
|
||||
: appInfo.appDescription || t("share.metadata.defaultDescription"));
|
||||
|
||||
const baseUrl = getBaseUrl();
|
||||
const baseUrl = await getBaseUrl();
|
||||
const shareUrl = `${baseUrl}/s/${params.alias}`;
|
||||
|
||||
return {
|
||||
|
Reference in New Issue
Block a user