| --- |
| tags: |
| - fred |
| - economic |
| - ML |
| --- |
| # FRED ML - Enterprise Economic Analytics Platform |
|
|
| A comprehensive, enterprise-grade Machine Learning system for analyzing Federal Reserve Economic Data (FRED) with automated data processing, advanced analytics, and interactive visualizations. |
|
|
| ## 🏢 Enterprise Features |
|
|
| ### 🚀 Core Capabilities |
| - **📊 Real-time Data Processing**: Automated FRED API integration with enhanced client |
| - **🔍 Data Quality Assessment**: Comprehensive data validation and quality metrics |
| - **🔄 Automated Workflows**: CI/CD pipeline with quality gates |
| - **☁️ Cloud-Native**: AWS Lambda and S3 integration |
| - **🧪 Comprehensive Testing**: Unit, integration, and E2E tests |
| - **🔒 Security**: Enterprise-grade security with audit logging |
| - **📈 Performance**: Optimized for high-throughput data processing |
| - **🛡️ Reliability**: Robust error handling and recovery mechanisms |
|
|
| ### 🤖 Advanced Analytics |
| - **📊 Statistical Modeling**: |
| - Linear regression with lagged variables |
| - Correlation analysis (Pearson, Spearman, Kendall) |
| - Granger causality testing |
| - Comprehensive diagnostic testing (normality, homoscedasticity, autocorrelation, multicollinearity) |
| - Principal Component Analysis (PCA) |
|
|
| - **🔮 Time Series Forecasting**: |
| - ARIMA models with automatic order selection |
| - Exponential Smoothing (ETS) models |
| - Stationarity testing (ADF, KPSS) |
| - Time series decomposition (trend, seasonal, residual) |
| - Backtesting with performance metrics (MAE, RMSE, MAPE) |
| - Confidence intervals and uncertainty quantification |
|
|
| - **🎯 Economic Segmentation**: |
| - Time period clustering (economic regimes) |
| - Series clustering (behavioral patterns) |
| - K-means and hierarchical clustering |
| - Optimal cluster detection (elbow method, silhouette analysis) |
| - Dimensionality reduction (PCA, t-SNE) |
|
|
| - **📈 Interactive Visualizations**: Dynamic charts and dashboards |
| - **💡 Comprehensive Insights**: Automated insights extraction and key findings identification |
|
|
| ## 📁 Enterprise Project Structure |
|
|
| ``` |
| FRED_ML/ |
| ├── 📁 src/ # Core application code |
| │ ├── 📁 core/ # Core pipeline components |
| │ ├── 📁 analysis/ # Economic analysis modules |
| │ ├── 📁 visualization/ # Data visualization components |
| │ └── 📁 lambda/ # AWS Lambda functions |
| ├── 📁 tests/ # Enterprise test suite |
| │ ├── 📁 unit/ # Unit tests |
| │ ├── 📁 integration/ # Integration tests |
| │ ├── 📁 e2e/ # End-to-end tests |
| │ └── 📄 run_tests.py # Comprehensive test runner |
| ├── 📁 scripts/ # Enterprise automation scripts |
| │ ├── 📄 cleanup_redundant_files.py # Project cleanup |
| │ ├── 📄 deploy_complete.py # Complete deployment |
| │ └── 📄 health_check.py # System health monitoring |
| ├── 📁 config/ # Enterprise configuration |
| │ └── 📄 settings.py # Centralized configuration management |
| ├── 📁 docs/ # Comprehensive documentation |
| │ ├── 📁 api/ # API documentation |
| │ ├── 📁 architecture/ # System architecture docs |
| │ └── 📄 CONVERSATION_SUMMARY.md |
| ├── 📁 data/ # Data storage |
| │ ├── 📁 raw/ # Raw data files |
| │ ├── 📁 processed/ # Processed data |
| │ └── 📁 exports/ # Generated exports |
| ├── 📁 deploy/ # Deployment configurations |
| │ ├── 📁 docker/ # Docker configurations |
| │ ├── 📁 kubernetes/ # Kubernetes manifests |
| │ └── 📁 helm/ # Helm charts |
| ├── 📁 infrastructure/ # Infrastructure as code |
| │ ├── 📁 ci-cd/ # CI/CD configurations |
| │ ├── 📁 monitoring/ # Monitoring setup |
| │ └── 📁 alerts/ # Alert configurations |
| ├── 📁 .github/workflows/ # GitHub Actions workflows |
| ├── 📄 requirements.txt # Python dependencies |
| ├── 📄 pyproject.toml # Project configuration |
| ├── 📄 Dockerfile # Container configuration |
| ├── 📄 Makefile # Enterprise build automation |
| └── 📄 README.md # This file |
| ``` |
|
|
| ## 🛠️ Enterprise Quick Start |
|
|
| ### Prerequisites |
|
|
| - Python 3.9+ |
| - AWS Account (for cloud features) |
| - FRED API Key |
| - Docker (optional, for containerized deployment) |
|
|
| ### Installation |
|
|
| 1. **Clone the repository** |
| ```bash |
| git clone https://github.com/your-org/FRED_ML.git |
| cd FRED_ML |
| ``` |
|
|
| 2. **Set up development environment** |
| ```bash |
| # Complete setup with all dependencies |
| make setup |
| |
| # Or manual setup |
| python -m venv .venv |
| source .venv/bin/activate # On Windows: .venv\Scripts\activate |
| pip install -r requirements.txt |
| pip install -e . |
| ``` |
|
|
| 3. **Configure environment variables** |
| ```bash |
| export FRED_API_KEY="your_fred_api_key" |
| export AWS_ACCESS_KEY_ID="your_aws_access_key" |
| export AWS_SECRET_ACCESS_KEY="your_aws_secret_key" |
| export AWS_DEFAULT_REGION="us-east-1" |
| export ENVIRONMENT="development" # or "production" |
| ``` |
|
|
| 4. **Validate configuration** |
| ```bash |
| make config-validate |
| ``` |
|
|
| 5. **Run comprehensive tests** |
| ```bash |
| make test |
| ``` |
|
|
| ## 🧪 Enterprise Testing |
|
|
| ### Run all tests |
| ```bash |
| make test |
| ``` |
|
|
| ### Run specific test types |
| ```bash |
| # Unit tests only |
| make test-unit |
| |
| # Integration tests only |
| make test-integration |
| |
| # End-to-end tests only |
| make test-e2e |
| |
| # Tests with coverage |
| make test-coverage |
| ``` |
|
|
| ### Quality Assurance |
| ```bash |
| # Full QA suite (linting, formatting, type checking, tests) |
| make qa |
| |
| # Pre-commit checks |
| make pre-commit |
| ``` |
|
|
| ## 🚀 Enterprise Deployment |
|
|
| ### Local Development |
| ```bash |
| # Start development environment |
| make dev |
| |
| # Start local development server |
| make dev-local |
| ``` |
|
|
| ### Production Deployment |
| ```bash |
| # Production environment |
| make prod |
| |
| # Deploy to AWS |
| make deploy-aws |
| |
| # Deploy to Streamlit Cloud |
| make deploy-streamlit |
| ``` |
|
|
| ### Docker Deployment |
| ```bash |
| # Build Docker image |
| make build-docker |
| |
| # Run with Docker |
| docker run -p 8501:8501 fred-ml:latest |
| ``` |
|
|
| ## 📊 Enterprise Monitoring |
|
|
| ### Health Checks |
| ```bash |
| # System health check |
| make health |
| |
| # View application logs |
| make logs |
| |
| # Clear application logs |
| make logs-clear |
| ``` |
|
|
| ### Performance Monitoring |
| ```bash |
| # Performance tests |
| make performance-test |
| |
| # Performance profiling |
| make performance-profile |
| ``` |
|
|
| ### Security Audits |
| ```bash |
| # Security scan |
| make security-scan |
| |
| # Security audit |
| make security-audit |
| ``` |
|
|
| ## 🔧 Enterprise Configuration |
|
|
| ### Configuration Management |
| The project uses a centralized configuration system in `config/settings.py`: |
|
|
| ```python |
| from config.settings import get_config |
| |
| config = get_config() |
| fred_api_key = config.get_fred_api_key() |
| aws_credentials = config.get_aws_credentials() |
| ``` |
|
|
| ### Environment Variables |
| - `FRED_API_KEY`: Your FRED API key |
| - `AWS_ACCESS_KEY_ID`: AWS access key for cloud features |
| - `AWS_SECRET_ACCESS_KEY`: AWS secret key |
| - `ENVIRONMENT`: Set to 'production' for production mode |
| - `LOG_LEVEL`: Logging level (DEBUG, INFO, WARNING, ERROR) |
| - `DB_HOST`, `DB_PORT`, `DB_NAME`, `DB_USER`, `DB_PASSWORD`: Database configuration |
|
|
| ## 📈 Enterprise Analytics |
|
|
| ### Running Analytics Pipeline |
| ```bash |
| # Run complete analytics pipeline |
| make analytics-run |
| |
| # Clear analytics cache |
| make analytics-cache-clear |
| ``` |
|
|
| ### Custom Analytics |
| ```python |
| from src.analysis.comprehensive_analytics import ComprehensiveAnalytics |
| |
| analytics = ComprehensiveAnalytics(api_key="your_key") |
| results = analytics.run_complete_analysis() |
| ``` |
|
|
| ## 🛡️ Enterprise Security |
|
|
| ### Security Features |
| - **API Rate Limiting**: Configurable rate limits for API calls |
| - **Audit Logging**: Comprehensive audit trail for all operations |
| - **SSL/TLS**: Secure communication protocols |
| - **Input Validation**: Robust input validation and sanitization |
| - **Error Handling**: Secure error handling without information leakage |
|
|
| ### Security Best Practices |
| - All API keys stored as environment variables |
| - No hardcoded credentials in source code |
| - Regular security audits and dependency updates |
| - Comprehensive logging for security monitoring |
|
|
| ## 📊 Enterprise Performance |
|
|
| ### Performance Optimizations |
| - **Caching**: Intelligent caching of frequently accessed data |
| - **Parallel Processing**: Multi-threaded data processing |
| - **Memory Management**: Efficient memory usage and garbage collection |
| - **Database Optimization**: Optimized database queries and connections |
| - **CDN Integration**: Content delivery network for static assets |
|
|
| ### Performance Monitoring |
| - Real-time performance metrics |
| - Automated performance testing |
| - Resource usage monitoring |
| - Scalability testing |
|
|
| ## 🔄 Enterprise CI/CD |
|
|
| ### Automated Workflows |
| - **Quality Gates**: Automated quality checks before deployment |
| - **Testing**: Comprehensive test suite execution |
| - **Security Scanning**: Automated security vulnerability scanning |
| - **Performance Testing**: Automated performance regression testing |
| - **Deployment**: Automated deployment to multiple environments |
|
|
| ### GitHub Actions |
| The project includes comprehensive GitHub Actions workflows: |
| - Automated testing on pull requests |
| - Security scanning and vulnerability assessment |
| - Performance testing and monitoring |
| - Automated deployment to staging and production |
|
|
| ## 📚 Enterprise Documentation |
|
|
| ### Documentation Structure |
| - **API Documentation**: Comprehensive API reference |
| - **Architecture Documentation**: System design and architecture |
| - **Deployment Guides**: Step-by-step deployment instructions |
| - **Troubleshooting**: Common issues and solutions |
| - **Performance Tuning**: Optimization guidelines |
|
|
| ### Generating Documentation |
| ```bash |
| # Generate documentation |
| make docs |
| |
| # Serve documentation locally |
| make docs-serve |
| ``` |
|
|
| ## 🤝 Enterprise Support |
|
|
| ### Getting Help |
| - **Documentation**: Comprehensive documentation in `/docs` |
| - **Issues**: Report bugs and feature requests via GitHub Issues |
| - **Discussions**: Community discussions via GitHub Discussions |
| - **Security**: Report security vulnerabilities via GitHub Security |
|
|
| ### Contributing |
| 1. Fork the repository |
| 2. Create a feature branch |
| 3. Make your changes |
| 4. Run the full test suite: `make test` |
| 5. Submit a pull request |
|
|
| ### Code Quality Standards |
| - **Linting**: Automated code linting with flake8 |
| - **Formatting**: Consistent code formatting with black and isort |
| - **Type Checking**: Static type checking with mypy |
| - **Testing**: Comprehensive test coverage requirements |
| - **Documentation**: Inline documentation and docstrings |
|
|
| ## 📄 License |
|
|
| This project is licensed under the Apache License 2.0 - see the [LICENSE](LICENSE) file for details. |
|
|
| ## 🙏 Acknowledgments |
|
|
| - Federal Reserve Economic Data (FRED) for providing the economic data API |
| - Streamlit for the interactive web framework |
| - The open-source community for various libraries and tools |
|
|
| ## 📞 Contact |
|
|
| For enterprise support and inquiries: |
| - **Email**: edwinsalguero@parallelllc.com |
| - **Documentation**: https://docs.your-org.com/fred-ml |
| - **Issues**: https://github.com/your-org/FRED_ML/issues |
| |
| --- |
| |
| **FRED ML** - Enterprise Economic Analytics Platform |
| *Version 2.0.1 - Enterprise Grade* |
