Skip to content

Migration to HoreKa 2

HoreKa 2 replaces the Tier 2 High Performance Computing system "Hochleistungsrechner Karlsruhe" (HoreKa) at KIT, introducing a new infrastructure with advanced CPU and GPU partitions. A primary focus during this transition is to conduct the migration in careful phases, ensuring minimal restrictions and disruptions for our users.

This guide helps existing HoreKa users transition to the new HoreKa 2 system.

Timeline

%%{init: { 
  'theme': 'basic', 
  'themeVariables': {
    'taskBkgColor': '#009682',
    'taskBorderColor': '#005b50',
    'activeTaskBkgColor': '#7d7d7d',
    'activeTaskBorderColor': '#505050',
    'critBkgColor': '#f47c00',
    'critBorderColor': '#b35b00',
    'milestoneBkgColor': '#7d7d7d',
    'milestoneBorderColor': '#505050',
    'gridColor': '#000000'
  },
  'gantt': {  
    'fontSize': 16,
    'sectionFontSize': 18,
    'barHeight': 26,
    'useMaxWidth': true,
    'useWidth': 1200
  }
}}%%
gantt
    dateFormat  YYYY-MM-DD
    axisFormat  %b %Y

    section HoreKa (Legacy)
    Operation (Blue, Green, Teal, Ruby) :active, h1_full, 2026-01-01, 2026-06-01
    Reduced Operation (only Blue, Green) :active, h1_parallel, 2026-06-01, 2026-09-01
    Reduced Operation (Data Migration) :active, h1_parallel, 2026-09-01, 2026-12-01
    Decommissioning                :milestone, active, 2026-12-01, 0d

    section HoreKa 2
    Setup Phase 1 (Onyx, Teal, Ruby, new file system): p1, 2026-01-01, 2026-06-15
    Availability (Onyx, Teal, Ruby)   :2026-06-15, 2026-09-01
    Setup Phase 2 (Jade)     :p2, 2026-08-01, 2026-09-01
    Full Operation (Onyx, Jade, Teal, Ruby)           :after p2, 2026-12-01

    section User Migration
    User Migration I :crit, m1, 2026-06-15, 2026-09-01
    User Migration II (Final Transfer):crit, m2, 2026-09-01, 2026-12-01

Migration Phases

The migration will be conducted in two phases, which are tied to the operational start dates of the CPU and GPU partitions respectively.

Phase 1: Optional (Mid-June – September 2026)

Start: As soon as the new CPU partition HoreKa Onyx is operational.

During this initial phase, migration to HoreKa 2 is optional. Early adopters can begin transitioning to the new system while the legacy HoreKa system continues to operate in reduced mode.

In this period, legacy HoreKa consists of the CPU nodes HoreKa Blue, and the GPU nodes HoreKa Green.
HoreKa 2 consists of the CPU nodes HoreKa Onyx and Blue (extra large memory), the GPU nodes HoreKa Teal, Ruby, and the new file system.

Phase 2: Mandatory (September – December 2026)

Start: As soon as the new GPU partition HoreKa Jade is operational.

In this second phase, all remaining HoreKa users must migrate to the new system. This is the final migration phase before the legacy system is decommissioned.

In this period, legacy HoreKa only allows login and access to its file system. In December, legacy HoreKa will be decommissioned.
HoreKa 2 is in full operation and consists of the CPU nodes HoreKa Onyx and Blue (extra large memory), the GPU nodes HoreKa Jade, Teal, Ruby, and the new file system.

Project and Data Migration

Please note that users are responsible for migrating their data to the new system, although we provide helper scripts to assist with the process.

When to migrate

The ideal time to migrate depends on your resource requirements:

  • Migrate during Phase 1: If your workloads have a high demand for CPU resources and only a medium need for GPU resources, we recommend migrating as soon as the CPU partition (HoreKa Onyx) becomes available.
  • Wait until Phase 2: If your projects require large amounts of GPU resources, it is better to stay on the legacy HoreKa system until Phase 2 begins and the new GPU partition (HoreKa Jade) is fully operational.

What to migrate

A complete migration to HoreKa 2 consists of two main components: 1. Compute time projects: The allocation of computational resources. 2. User data: Personal files, code, and project data stored in your HOME directory as well as any active workspaces.

Migration of existing compute time projects

The migration of existing compute time projects must be initiated by the respective project administrators (PIs).

When a project is migrated, the remaining Slurm resource allocations (compute time quotas) are transferred from the legacy system to HoreKa 2. Consequently, the compute resources for that project will be zeroed out on the old system to prevent parallel accounting.

Please note that your existing project names and identifiers will remain exactly the same on the new system.

Migration of data

Users are responsible for migrating their own data from the legacy systems to HoreKa 2. This includes data in your HOME directory and in your Workspaces.

We recommend using rsync for transferring files efficiently. To assist you with this process, we will provide dedicated helper scripts. Detailed instructions, exact paths, and the helper scripts themselves will be made available in due time before the migration starts.

Key Changes from HoreKa to HoreKa 2

Hardware Architecture

For the first time, the main cluster consists of two fundamentally different compute architectures: x86 and ARM.

  • The new CPU partitions, as well as the GPU partitions that have been carried over from the legacy system, feature x86 host processors.
  • The new GPU partition (HoreKa Jade) features ARM host processors.

Because of this architectural split, HoreKa 2 provides two distinct types of login nodes: one dedicated to the x86 architecture and one dedicated to the ARM architecture. Users must select the appropriate login node depending on the architecture they intend to compile for or use.

Software Environment

With HoreKa 2, we are completely transitioning our software provisioning framework to EasyBuild, a well-established standard in the HPC community.

  • Unified Build System: Moving to EasyBuild is essential for efficiently maintaining the software stack across our new dual-architecture (x86 and ARM) environment.
  • Naming Conventions: Please be aware that module naming conventions in EasyBuild differ significantly from our legacy system.
  • Early Testing: If you would like to familiarize yourself with the new environment, EasyBuild is already available as an optional module system on the legacy HoreKa cluster.

HOME directories

A major structural change has been made regarding personal directories. HOME directories are now associated directly with individual users, rather than being nested inside project directories.

This means you will have a single, unified HOME directory that remains exactly the same, irrespective of which compute time project you are currently working on.

HAICORE

HAICORE resources are no longer treated as a separate cluster. The resources funded within the HAICORE 3 framework are now an integral part of the combined HoreKa 2 + HAICORE 3 system.

  • Expanded Access: HAICORE users now get access to the entire system according to the funding split. This means CPU resources will also be available for HAICORE users, not just GPUs.
  • Application Process: Access to HAICORE is no longer a simple sign-in process to start computing. Users must now submit small project proposals, comparable to the "AI Fast Track" projects on the legacy HoreKa system.