📊 Dashboard

—
Connected
—
Stale
—
Error
—
Not Active
—
Missing

📈 System Overview

Data Status
—
CSV Data Date ?
—
DB Date ?
—
Last Activity
—
Active Profile
—

📅 Business Date

— (Auto)

⚡ Quick Actions

📐 Data Model
🏭 Sources
📦 Subscriptions

🏭 Sources

📋 CCP Source Configuration

Target tables and their data source assignments per CCP. Each tab represents a clearing house with its specific data requirements.

Target Table ↕ Source Type ↕ Status ↕ Resolved From ↕ Actions
Loading source data...
🔀 LCH Data Flow ▸ expand

How source reports flow through transforms into COP target tables and the optimiser.

👥 Member Report Overrides

Members with individual reports in dataimports/members/ can override the LCH pipeline. Toggle a member on to use their individual reports instead of the centrally-transformed data.

Loading...
Last Generated
—
Total Rows
—
Data Status
—
Day Period Types
—
Config
—
⏳
No generation run yet
Generating... 0%

🏭 Load CCP Pipeline

Reads source files configured in the Sources tab, transforms LCH reports into 19 COP tables ready for database loading.

LCH LCH Feed Pipeline

Transform actual LCH report files from report-samples/dataimports/ into COP-format CSV files.
Source: dataimports/YYYYMMDD-lch/ + dataimports/static-lch/ + dataimports/members/  ·  Output: 19 COP CSV files → /app/data/

Eurex Coming soon
CME Coming soon
ICE Coming soon
JSCC Coming soon

⏰ Scheduler

▸ expand

Quick presets or custom cron schedules for automatic data generation.

📦 Load Snapshot

▸ expand

🗄️ Stale Verified Data

Load cop.sql — a verified snapshot of real production data captured on 2025-05-13. This replaces all data in the cop schema with a known-good baseline. Useful for regression testing and UI validation.

Source: cop.sql (mounted from project root)  ·  Size: ~150 MB  ·  Schema: cop (PostgreSQL 17.5 dump)

0%
—
Not yet run

🏭 Full Production Dump

Load the full production database dump captured on 2025-02-19. The dump is stored as split zip files in db_dump/. When you click the button, the system will:

  1. Reassemble all split parts into a single zip
  2. Unzip to a temporary SQL file
  3. Restore into PostgreSQL (DROP SCHEMA cop CASCADE + psql)
  4. Delete the temporary SQL file (split zips are kept)
Source: db_dump/cop_prod_dump.sql.zip.split_* (28 parts)  ·  Compressed: ~1.2 GB  ·  Uncompressed: ~3.1 GB  ·  Restore time: ~10–15 min

⚠ This will replace ALL data in the cop schema.

0%
—
Not yet run

🔧 Generate Synthetic Data

▸ expand
Leave as today for current business date
±% from baseline values
Data Status
—
Business Date
—
Last Consumed
—
Total Rows
—
Database
Checking...
⏳
No consumption run yet
Consuming... 0%

📥 Load to DB Stopped

Load CSV files from the data directory into PostgreSQL (truncate + insert for all tables).

Scope: All CCPs, All Members — 19 tables will be loaded

⏰ Consumer Scheduler

Automatically poll for new data (checks _control.json status). When new data is detected, it is consumed automatically.

📋 Active Scheduler Jobs

Loading...

📋 Activity Log

TimeServiceUserActionStatusDetail
Loading…

📊 Active Data Profile

—
Loading...

🌐 Infrastructure

💡 Mount Points:
… — generated/transformed CSVs written here
…/dataimports → LCH source reports
… — stale database dump
… — production database splits

📊 Generate Page

Choose which sections are visible on the Generate page.

📂 LCH Report Paths

Paths to LCH central clearing data used by the feed pipeline. Per-table source configuration is managed in Sources.

Dataimports Directory
Data Date Folder
💡 These paths are inside the container. Dataimports is mounted from the host at report-samples/dataimports/ (Docker) or via PVC (Kubernetes).

📊 LCH Source Reports

All internal LCH reports consumed by the pipeline, grouped by importance tier. The pipeline adapts gracefully — more reports available → better data quality.

CRITICAL — core pipeline dependency IMPORTANT — significant quality improvement OPTIONAL — nice-to-have enrichment
Source Tier Directory Files

👥 Member Report Sources

Configure per-member report directories for the LCH feed pipeline. When enabled, member-specific reports replace/supplement LCH central data for that member's rows during the transform. Members with directories in dataimports/members/ are auto-detected. Add any member below — directory can be created later.

📤 Report File Transfer

Upload or download report archives to sync between local and remote environments.

📥 Download

🧮 Reconciliation

⚖️ Internal vs Member Reports

Run both source paths independently, compare the reconciled COP tables, and review matches, gaps, and discrepancies.

Supports either the dataimports root or a specific dated LCH folder.
Defaults to the configured members folder for the selected mnemonic.
Optional. Leave blank to let the engine detect dates from the inputs.

🔁 Reconcile After LCH Feed

Once the LCH feed completes, automatically run reconciliation against every enabled Member Report Source configured in Settings. Requires the new member_vs_internal framework.

No run yet.

📌 Summary

Status
Awaiting run
Start a reconciliation to populate summary metrics.

📋 Table Results

No report yet.

📦 Data Source Subscriptions

Enable or disable data source adapters. Enabled sources appear in the Sources and Configure tabs. ℹ️

📐 Data Model View

Business-oriented view of all COP data requirements. Shows what data is needed, where it comes from, and what's missing. ℹ️

Loading data model…