Soshi

Soshi

Nest js
Flutter
Firebase
⭐ Featured Project

🚀 Overview

Soshi is a cutting-edge social media ecosystem that integrates Affective Computing, Embedded Systems (IoT), and Deep Learning into a seamless communication platform. Beyond typical social interactions, Soshi monitors user stress through physiological data and optimizes content delivery to prioritize mental well-being, creating a truly empathetic digital environment.

✨ Key Features

📡 Diversified Content & Communication

  • Multi-Format Content: Support for Posts, Videos, and Shorts (TikTok-style) for a dynamic content experience.
  • Real-Time Communication: Seamless Video and Audio Calls powered by WebRTC, and real-time Chat using WebSockets.
  • Instant Notifications: Critical updates and event alerts delivered via Firebase (FCM).

🧠 Affective Intelligence & Stress Detection

  • Stress Monitoring: A custom Bi-LSTM model analyzes real-time physiological data to detect stress levels.
  • Mental Well-being Alerts: If high stress is detected, the system proactively sends notifications suggesting users "Take a Break" or stop watching to prioritize health.
  • IoT Integration: Data is acquired through an ESP32 wearable and synchronized via a custom Flutter Mobile App.

🔍 Advanced Recommendation Engine

  • Fine-Tuned Embeddings: Utilizing a fine-tuned Sentence-Transformer (Nomic) to generate high-dimensional embeddings for precise video similarity search.
  • Wearable Augmented Recommendations: Injects physiological state data into the recommendation loop for truly reactive content suggestions.
  • Admin Optimization Dashboard: A dedicated interface for real-time parameter tuning and weight optimization of the recommendation engine.

fine-tuning-flow.png

accuracy.jpg

🛠️ Technical Stack

Communication & Frameworks

  • Web: Next.js 15 (App Router), Tailwind CSS
  • Mobile: Flutter (Dart) - Acts as a gateway between ESP32 and Server / Dummy Wearable.
  • Real-Time: WebRTC (A/V Calls), WebSockets (Chat), Pusher.
  • Push Notifications: Firebase Cloud Messaging (FCM).

Machine Learning & Affect Analysis

  • Affect Model: Bi-LSTM (trained on WESAD dataset for stress classification).
  • Search Engine: Fine-tuned Sentence-Transformers (Nomic-Embed) for semantic video search.
  • Backend API: FastAPI (Python), PyTorch.

IoT & Embedded Systems

  • Hardware: ESP32 Microcontroller.
  • Architecture: Circuit design with MAX30105 (Pulse) and MPU6050 (Motion).
  • Communication: Seamless ESP32-to-Flutter data synchronization.

🔬 Technical Deep Dive

The Flutter Gateway & Dummy Wearable

The custom-built Flutter application serves as the vital link between the ESP32 hardware and the cloud infrastructure. It handles raw sensor data pre-processing and enables a "Dummy Wearable" mode for users without physical hardware, ensuring accessibility across the platform.

Bi-LSTM for User Well-being

Our stress detection engine doesn't just harvest data; it acts. The Bi-LSTM model identifies high-arousal negative states with high accuracy, triggering a "Stop Watching" notification in the UI to prevent content-induced fatigue.

app.jpeg

app3.jpeg

esp.jpeg

🏁 Conclusion

Soshi is a vision of a healthier social future—where complex engineering in WebRTC, WebSockets, and Affective Deep Learning work together to create a platform that understands and cares for its users.