🌿 Welcome to Rubus-Cron

Rubus-Cron is a Node.js microservice built with NestJS that periodically fetches economic and IT-related data from multiple sources (EU Parliament, Eurostat, Federal Reserve, US-BEA) and stores them in a PostgreSQL database.

The project is designed to:

• 🔄 Collect data from APIs and RSS feeds.
• 🏗 Normalize and transform the data into a unified format.
• 🧠 Provide AI agents with structured historical data for context-based analysis.
• 🕰 Run on a cron job, ensuring the database is always up-to-date.

🚀 Features

• Automated Data Collection 📡 (via API & RSS fetchers)
• Scheduled Execution ⏳ (via cron jobs)
• Data Normalization & Storage 🗃️ (via PostgreSQL)
• Modular & Scalable Architecture 🏗️ (NestJS & Docker)

📦 Tech Stack

• Backend: NestJS (TypeScript)
• Database: PostgreSQL
• Task Scheduling: @nestjs/schedule
• Containerization: Docker & Docker Compose
• ORM: TypeORM

---

⚙️ Installation & Setup

1️⃣ Clone the repository

git clone https://github.com/YOUR_GITHUB_USERNAME/rubus-cron.git
cd rubus-cron

2️⃣ Install dependencies

npm install

3️⃣ Configure environment variables

Create a .env file at the project root:

POSTGRES_HOST=db
POSTGRES_PORT=5432
POSTGRES_USER=postgres
POSTGRES_PASSWORD=password
POSTGRES_DB=rubus

4️⃣ Run with Docker

docker-compose up --build

5️⃣ Access the application

• API runs on http://localhost:3000
• PostgreSQL database is available on port 5432

📡 Data Fetchers

Rubus-Cron integrates multiple data sources:

European Union:
    EU Parliament RSS Feeds
    Eurostat API
United States:
    Federal Reserve RSS Feeds
    US Bureau of Economic Analysis (BEA) API

Each data fetcher is modular and located in src/data-fetchers/.

🛠 Project Structure

rubus-cron/
 ├── src/
 │   ├── app.module.ts          # Main NestJS module
 │   ├── main.ts                # App entry point
 │   ├── config/                # Environment configuration
 │   ├── cron/                  # Scheduled jobs (cron)
 │   ├── database/              # PostgreSQL integration
 │   ├── data-fetchers/         # API & RSS fetchers
 ├── .env                       # Environment variables (not committed)
 ├── .gitignore                 # Ignore unnecessary files
 ├── Dockerfile                 # Docker container setup
 ├── docker-compose.yml         # Multi-container setup (app + PostgreSQL)
 ├── package.json               # Project dependencies
 ├── README.md                  # You are here

🚀 Running Tests

🔹 Run unit tests (fetchers & transformers)

npm run test:unit

🔹 Run functional tests (full ingestion flow)

npm npm run test:functional

🔹 Run integration tests (database persistence)

npm run test:integration

🔹 Run all tests

npm run test:all

🛠 Contributing

• Fork the repo
• Clone your fork
• Create a new feature branch
• Commit & push your changes
• Create a Pull Request

📝 License

This project is licensed under the MIT License.

✨ Future Enhancements

• 🧠 AI Integration: Feed the collected data into LLMs (ChatGPT, LLaMA).
• 🔍 Memory Vectorization: Store key insights for long-term AI context.
• 📊 Visualization: Create dashboards for economic & IT trends.

📝 Author

• 👤 0xRubusData
• 📧 Contact: [email protected]
• 🌍 GitHub: https://github.com/0xrubusdata/rubus-cron

🌐 Connect with Us

• Twitter (X): 0xRubusData
• Website: RubusLab

🎯 Happy Coding! 🚀