Storage API File Manager

File manager API with SQLite indexing in Go.

Source code

Overview

This project is a high-performance, self-hosted file manager API written in Go using the Fiber framework. It is designed to manage files across multiple storage mounts, such as SSDs and HDDs, while providing instant search capabilities. The system uses an embedded SQLite database to index file metadata, eliminating the need to traverse slow mechanical filesystems during search operations.

Features

  • Multiple Storage Mounts: Unifies access to various drives (e.g., SSD, HDD) under a single API.
  • SQLite Indexing: Instant search results and file statistics without filesystem traversal.
  • Background Auto-Indexing: Automatically scans and updates the file index every 30 minutes.
  • Real-Time Cache Invalidation: Write operations instantly invalidate cache and trigger re-indexing.
  • JWT Authentication: Secures protected endpoints using Bearer tokens.
  • Comprehensive File Operations: Supports upload, folder creation, rename, copy, duplicate, and deletion.
  • Video Thumbnails: Automatically generates video previews on the fly using ffmpeg.

Technical Implementation

The application operates as a REST API, providing endpoints for file manipulation and advanced querying.

Requirements

  • Go 1.25+ (for manual execution)
  • Docker and Docker Compose (recommended for deployment)
  • ffmpeg (required for generating video thumbnails)

Installation

For Linux servers, use the provided bash script to configure permanent storage mounts automatically:

bash 
chmod +x setup_ssd.sh
./setup_ssd.sh

Alternatively, deploy using Docker Compose:

bash 
docker-compose up -d --build

Configuration Files

  • .env: Primary configuration file for managing secrets and storage paths.
  • docker-compose.yml: Defines the containerized deployment setup.

Environment Variables

Configuration is handled via the .env file. Example:

env 
APP_PORT=3003
PASSWORD=your_secure_password
JWT_SECRET=your_secret_key
HOST_PATH_SSD=/mnt/ssd
HOST_PATH_HDD=/home/roniserv
STORAGE_MOUNTS=ssd:/mnt/ssd,hdd:/home/roniserv

Database Setup

The database is an embedded SQLite file named storage_index.db. It is automatically created and initialized by the application on startup. No external database server setup is required.

How to run the project

To run the project manually outside of Docker:

bash 
go run cmd/api/main.go

Important Commands & API Endpoints

The system provides the following endpoints (all /api/* routes except /api/login require a JWT Bearer token):

Public Endpoints:

  • GET /ping: Public health check, storage mount list, and latency test.
  • POST /api/login: Authenticate and receive a JWT token.

Storage & File Operations (Protected):

  • GET /api/: List all available storage drives.
  • GET /api/files: List files and directories within a specific path.
  • GET /api/preview: Preview a file or stream video (supports thumbnail generation).
  • GET /api/download: Force a file download.
  • POST /api/upload: Upload a file to a specific storage path.
  • POST /api/folder: Create a new folder.
  • PUT /api/rename: Rename or move a file/folder.
  • POST /api/copy: Copy a file/folder.
  • POST /api/duplicate: Duplicate a file.
  • DELETE /api/delete: Delete a file/folder.

Indexing & Search (Protected):

  • GET /api/search: Perform a fast SQLite-indexed search (e.g., ?storage=ssd&ext=jpg&days=7).
  • GET /api/recent: Retrieve the most recently modified files.
  • POST /api/stats: Get file counts categorized by type (images, videos, etc.).
  • GET /api/reindex: Force a manual re-index of all storage mounts.

Main Workflow

The application starts by loading configurations and initializing the local filesystem driver. It spawns a background goroutine to execute ReindexAll(), which recursively scans mounted drives and populates the SQLite index. Incoming HTTP requests are intercepted by Fiber middleware for JWT validation and logging before being routed to the appropriate file handler. When a file is modified (e.g., via upload or rename), the service invalidates the cache and triggers a targeted background re-index to keep the SQLite database synchronized.

Important Usage Notes

The API enforces a strict 100MB maximum upload limit by default. Additionally, for permanent Linux deployments, it is highly recommended to mount drives via /etc/fstab rather than relying on temporary paths, preventing synchronization issues after reboots.

Development

The architecture follows a modular approach, separating transport, application logic, and infrastructure layers.

  • Go (Golang): Chosen for its high performance, concurrency model, and easy deployment via static binaries.
  • Fiber: A web framework inspired by Express.js, used for routing, middleware, and handling multipart file uploads efficiently.
  • SQLite (mattn/go-sqlite3): Acts as the high-speed indexer. It runs in WAL (Write-Ahead Logging) mode to handle concurrent read/write operations smoothly.
  • JWT (golang-jwt/jwt): Provides stateless authentication for the admin user.
  • FFmpeg: Invoked as a sub-process via os/exec to extract single frames from video files for thumbnail generation.

Highlights

The most notable technical challenge resolved in this project is the speed of file discovery on large, slow mechanical hard drives.

Traditional file managers traverse the directory tree for every search, which is extremely slow on HDDs. This project solves that by implementing an embedded SQLite indexing system. By running a persistent background indexer and utilizing SQL queries for searching (e.g., filtering by extension, modification date, and storage volume), the application delivers instant search results and statistics without touching the actual filesystem during read queries.

Furthermore, the system is designed to be self-healing. Write operations inherently clear the in-memory cache and asynchronously update the SQLite index, ensuring the database remains a highly accurate reflection of the filesystem state.

Back to Projects