Skip to main content

​
Prerequisites

Before installing Spark, make sure you have: 
# Check your Python version
python --version

# Should output Python 3.10.0 or higher
If you need to install or upgrade Python:
  • macOS: brew install python@3.11
  • Ubuntu/Debian: sudo apt install python3.11 python3-pip
  • Windows: Download from python.org
# Check if Git is installed
git --version
Spark can automate Git operations, so Git is recommended.
Spark runs scheduled tasks, so you may need:
  • Linux/macOS: Access to cron or systemd
  • Windows: Task Scheduler access
  • Docker: For containerized deployments

​
Installation Methods

​
Configuration

After installation, initialize Spark: 
# Initialize Spark workspace
spark init

# This creates:
# - .spark/ directory
# - workflows/ directory for your workflow definitions
# - config.json (Spark configuration)
Configure Spark in config.json: 
{
  "scheduler": {
    "timezone": "America/New_York",
    "maxConcurrent": 5,
    "retryAttempts": 3
  },
  "monitoring": {
    "enabled": true,
    "logLevel": "info",
    "retentionDays": 90
  },
  "notifications": {
    "enabled": true,
    "onFailure": true,
    "onSuccess": false,
    "channels": ["email", "slack"]
  }
}

​
Create Your First Workflow

Create a workflow file in workflows/daily-report.yaml: 
name: daily-report
description: Generate and send daily project report
schedule: "0 9 * * *"  # Every day at 9 AM
timezone: America/New_York

tasks:
  - name: analyze-commits
    type: git
    action: log
    args:
      since: "24 hours ago"

  - name: generate-report
    type: script
    script: |
      python scripts/generate_report.py

  - name: send-email
    type: email
    to: "team@example.com"
    subject: "Daily Project Report"
    body: "{{ tasks.generate-report.output }}"

retryOn:
  - network-error
  - timeout

notifications:
  onSuccess: true
  onFailure: true

​
Start Spark

Start the Spark scheduler: 
# Start Spark daemon
spark start

# Or run in foreground
spark start --foreground

# With specific workflows directory
spark start --workflows ./my-workflows
You should see output like: 
⚡ Spark v1.0.0
✓ Configuration loaded
✓ Workflows loaded: 3
✓ Scheduler started (timezone: America/New_York)
✓ Next execution: daily-report at 2025-10-31 09:00:00

Spark is running! Press Ctrl+C to stop.

​
Verify Installation

Test that everything is working: 
# Check version
spark --version

# List workflows
spark workflow list

# Test a workflow
spark workflow run daily-report --dry-run

# Check scheduler status
spark status

# View execution history
spark history --limit 10

​
Troubleshooting

Check if another instance is running:
# Check running processes
ps aux | grep spark

# Stop existing instance
spark stop

# Restart
spark start
Make sure timezone is correctly configured:
# Check available timezones
spark config timezones

# Set timezone in config.json or:
spark config set scheduler.timezone "America/Los_Angeles"

# Restart Spark
spark restart
Check logs for details:
# View recent logs
spark logs --tail 50

# View logs for specific workflow
spark logs --workflow daily-report

# Debug mode
spark start --debug
Verify notification configuration:
# Test notification system
spark notify test

# Check notification config
spark config get notifications

# Test email (if using email)
spark notify test-email --to your@email.com

​
System Service Setup (Optional)

For production deployments, run Spark as a system service:
  • systemd (Linux)
  • Docker Compose
Create /etc/systemd/system/spark.service:
[Unit]
Description=Automagik Spark Scheduler
After=network.target

[Service]
Type=simple
User=your-user
WorkingDirectory=/path/to/your/project
ExecStart=/usr/local/bin/spark start --foreground
Restart=on-failure

[Install]
WantedBy=multi-user.target
Enable and start:
sudo systemctl enable spark
sudo systemctl start spark
sudo systemctl status spark

​
Next Steps

Quick Start

Create your first automated workflow

Introduction

Learn more about Spark