A Python-based service that processes transactions using Google Cloud Vertex AI. This service handles data processing with configurable logging and supports multiple input datasets.
- Transaction processing with Vertex AI integration
- Configurable logging system
- YAML-based configuration management
- Support for multiple input datasets
- Automated result generation
- Python 3.8 or higher
- Google Cloud Platform account
- Vertex AI API enabled
- Valid GCP credentials
- Clone the repository:
git clone
- Install the required using poetry or pip:
poetry install
pip install -r requirements.txt
-
Configure your credentials:
- Place your GCP service account key in
credentials/credential.json
- Update
config.yaml
with your project settings
- Place your GCP service account key in
-
Example
config.yaml
structure:
project_id: "your-project-id"
location: "us-central1"
model_id: "your-model-id"
input_datasets:
- name: "dataset1"
logging:
level: "INFO"
format: "%(asctime)s - %(name)s - %(levelname)s - %(message)s"
file: "logs.log"
Run the service using the following command:
python main.py
The service will:
- Process input files from the
dataset
directory - Generate logs in the
logs
directory - Output results to the
results
directory
.
├── config.py # Configuration management
├── config.yaml # Configuration settings
├── credentials/credential.json # GCP credentials
├── dataset/ # Input data files
├── logs/ # Application logs
├── main.py # Application entry point
├── services/ # Core service modules
└── utils/ # Utility functions
- Logs are automatically generated in the
logs
directory - Format:
process_YYYYMMDD_HHMMSS.log
- Contains detailed processing information and error tracking
-
Credential Error
Solution: Ensure credentials/credential.json is properly configured and has the required permissions
-
Configuration Error
Solution: Verify config.yaml format and required fields
-
Processing Error
Solution: Check input data format in dataset directory and logs for specific error messages