As developers, we’ve all been there: you’re building a hobby project, a side hustle, or a microservice, and you need object storage. You look at AWS S3 or dedicated cloud providers, and the costs start adding up. Meanwhile, most of us have hundreds of gigabytes of idle, wasted space sitting in our personal Google Drive, Dropbox, or Mega accounts.
I wanted to find a way to use that personal cloud storage programmatically—specifically as an S3-compatible bucket—without paying premium storage fees.
But I had one strict rule for myself: The solution had to be 100% stateless. No caching files on my servers, no data retention, and zero storage costs on my end. Security and privacy meant that files had to exist on my infrastructure only in-flight as streaming data packets.
Here is a deep technical breakdown of the architecture I built to make this happen using NestJS, Fastify, OpenDAL, and BullMQ in a monorepo structure.
1. The Core Architecture: A Monorepo Approach
To keep performance blazing fast and maintain a clean separation of concerns, I broke the system down into three distinct, decoupled services inside a monorepo, sharing underlying core modules.
Because raw Node.js HTTP overhead can become a bottleneck when proxying heavy streams, I swapped out Express for Fastify as the underlying HTTP provider for NestJS.
┌────────────────────────────────────────┐
│ Main API Server │
│ (Handles Auth, Web Dashboard, OAuth) │
└───────────────────┬────────────────────┘
│
│ (Pushes Sync/Heavy Tasks)
▼
┌───────────┐
│ BullMQ │
└─────┬─────┘
│
▼
┌────────────────────────────────────────┐
│ Worker Server │
│ (Processes Heavy Background Jobs) │
└────────────────────────────────────────┘
─────────────────────────────────────────────────────────────────────────
┌────────────────────────────────────────┐
│ S3-Compatibility Server │
│ (Streams Data / Translates S3 XML) │
└────────────────────────────────────────┘
- The Main API Server: Handles user authentication, the web dashboard management, and OAuth flows. To ensure high availability, this server does not handle raw file processing or synchronization jobs.
- The Worker Server: When a heavy asynchronous task or sync job is triggered, the Main API pushes it to a BullMQ queue. The Worker server listens and processes these background jobs safely without blocking incoming API traffic.
- The S3-Compatibility Server: A dedicated standalone server focused entirely on parsing standard AWS S3 API signatures, handling S3-specific XML payloads, and streaming object data directly.
2. Decoupling Providers with the Adapter Pattern & OpenDAL
Every cloud storage provider has a completely different API ecosystem. To prevent my codebase from turning into spaghetti code, I heavily relied on the Adapter Pattern.
I defined a unified StorageAdapter interface. Whether a user connects Google Drive or Mega, the core engine interacts with them identically. To abstract away the low-level file system operations, I utilized OpenDAL (Open Data Access Layer), which provides a brilliant data access abstraction layer.
Here is a simplified look at how the adapters are structured:
export interface StorageAdapter {
uploadFile(path: string, stream: ReadableStream): Promise<UploadResult>;
downloadFile(path: string): Promise<ReadableStream>;
deleteFile(path: string): Promise<void>;
listFiles(path: string): Promise<FileObject[]>;
}
The Authentication Split: OAuth vs. Credentials
Implementing this pattern threw a major curveball when dealing with provider authentication. The providers essentially split into two categories:
- The OAuth Providers (Google Drive, Dropbox): These use standard OAuth 2.0 flows. The Main API handles the redirect, grabs the refresh token, and securely manages access tokens.
- The Credential Providers (Mega): Mega doesn't have an OAuth layer; it strictly requires a username and password.
Because we are 100% stateless, we must store these credentials to authenticate requests on the fly. To do this safely, the credentials undergo strong encryption before hitting our database and are decrypted strictly in-flight within the isolated adapter instance during runtime execution.
3. The Google Drive Blockblock: Fighting Permissions
If you ever try to build a platform that hooks into Google Drive for programmatic developer access, you will run into a massive security UX roadblock.
Initially, I requested the standard drive access scope. However, Google treats full drive access as a highly restricted permission. If you use it, Google aggressively flags your application with a terrifying "This app is not secure / Not authorized" warning screen for any external user attempting to connect their account. Bypassing this requires a lengthy, expensive independent security verification process.
The Fix: I downgraded the requested scope to strictly drive.file.
This scope only grants our application access to files and folders that the app itself creates. It completely resolved the security verification warning, making the onboarding flow seamless while executing perfect least-privilege security. The user gets a secure sandbox within their own Google Drive, and our app gets immediate authorization.
4. Keeping it 100% Stateless via Streaming
The magic of this gateway is that it doesn't scale with disk space; it scales with network throughput. When an S3 request comes into the S3-compatible server, we don't download the file to our disk and then upload it to the provider.
Instead, we use Node.js and Fastify's native streaming capabilities. We pipe the incoming S3 request stream directly into the OpenDAL abstraction layer for the respective adapter:
// A high-level conceptual example of pass-through streaming
async handleS3Upload(req: FastifyRequest, reply: FastifyReply) {
const targetAdapter = this.adapterFactory.get(req.user.provider);
// Pipe the raw incoming request payload directly to the cloud provider
const result = await targetAdapter.uploadFile(req.params.filepath, req.raw);
// Reply with standard S3 XML format
return reply.type('application/xml').send(this.xmlBuilder.build(result));
}
What I Learned & What’s Next
Building an infrastructure tool like this taught me that the hardest part isn't writing the code; it's mapping completely mismatched paradigms (like translating S3 XML APIs into standard REST JSON responses used by consumer cloud drives).
I've packaged this entire architecture into a managed platform called Uploom (uploom.io). It lets you spin up an S3-compatible gateway on top of your personal cloud storage in less than 2 minutes without hosting anything yourself.
I’d love to hear your thoughts on this setup!
- How would you handle credential encryption for non-OAuth providers differently?
- Have you run into similar streaming bottlenecks with Node.js/Fastify under heavy loads?
Let's discuss in the comments below!
