Our all-day class at SC14 on Sunday November 16, “From ‘Hello World’ to Exascale Using x86, GPUs and Intel Xeon Phi Coprocessors” (tut106s1) received more than double our expected enrollment! Students will be able to run on both Intel Xeon Phi and GPU supercomputers at TACC via an Xsede allocation (thank you very much) and on a CreativeC supercomputer and visualization cluster shown below.
CreativeC Stella compute cluster
Amazingly, a CreativeC supercomputer cluster containing both Intel Xeon Phi coprocessors and GPUs is running live while being transported to New Orleans. This system has been setup to allow 100 students to concurrently build and test CUDA, OpenACC plus Intel Xeon Phi offload and native mode example codes using a head-node and two compute nodes. It is remarkable that amount of compute power can be powered up and connected to the Internet via a 4G modem while traveling down a highway in a van. Kudos to the hard working CreativeC team members of Greg Scantlen and Tim Thomas!
The current design has three nodes, a single socket E5-1620V3 (SPINE) between two dual socket E5-2650V3 motherboards (WINGS or BACKPACKS) or a total of 5 CPUs. The center spine “Admin” Node has a ZFS RAID1, OS vol / scratch vol comprised of 2x 240GB SSDs, plus a ZFS RAID5 home / export /data vol made of 3x SSHD 2TB with a 128GB cache. There are 6x FDR cables. The SC14 system uses a 12 port switch, but Greg Scantlen expects to go switchless using multiple OpenSM’s (one per link) CreativeC has been building switch-less small clusters this way since 2010.
In order to use MV2-GDR, and MV2-MIC in RDMA mode there must be a IB HCA on the same CPU as the accelerator. So 4 cables are RDMA links, the remaining two cables are Admin Node to Compute_1 and Admin to Compute_2 for Data. Customers can configure two additional more links so that data can move from disk directly to any CPU without use of interCPU QPI links. This can also be RDMA over IB using TCPoIB and NFSv4)
This same configuration is also used to demonstrate visualization walls and yurts. Her e it is being used by Tim Thomas as an uber-laptop while on the road.
The CreativeC system is (an apparently mobile) visualization platform
Rob Farber logged in (bottom line on screen) from Gig Harbor, WA to the CreativeC compute cluster outside of Dallas, TX.
CreativeC offers two recommended configurations
The Startup
This system is a Standard Compute node with a narrow (1U) spine for holding disk drives (same as described above) configured as a workstation, but it can scale to 3 node cluster easily. Pricing for this system with 2x E5-2650V3 CPUs, 128G 2133 RAM, 1x GTX980, is ~$12K. Customers can reserve one online for a $1K deposit (fully refundable). Customers can also add more GPUs, or Phi’s up to 4x accelerators.
Full System
This configuration is a three node cluster with IB for MV2-GDR, MV2-MIC as described above. Users can distribute work over 8x GPUs, or 8x MIC’s, or 4x GPUs and 4x MICs.
It is possible to scale from a Startup to a Full System, One would remove hard drives from narrow spine, insert into 3u wide spine (with Admin CPU), place startup Node onto 3U spine, and connect power/IB to convert startup to 2nd compute node. Seems like a drawing is necessary here, like LEGO instructions?
Oh if the traffic only cooperated!
(See Greg Scantlen attempting to move traffic with hand signals.)
Leave a Reply