Hybridizer supports generics and virtual functions. These concepts allow writing flexible code with type parameters, defering actual behavior resolution to type instanciation by client code. These are fondamental features of modern languages, easing encapsulation, code factorization and concepts expression.

However in C#, type parameters are resolved at runtime, which comes with a significant performance penalty. Hybridizer maps them to C++ templates, which are resolved at compile time. As such, templates allow inlining and interprocedural optimization as in plain C code. Performance penalty is therefore inexisting.

As an example, we will demonstrate usage of generics on a fun mathematical example : the resolution of heat equation with random walks.

Mathematic background

Given a connected bounded 2D domain Ω in ℝ2, its border ∂Ω and a function 𝑓∈:L2(∂Ω), we search 𝑢 such as:

Δ𝑢=0 on Ω

𝑢=𝑓 on ∂Ω

Classic ways to numerically solve this problem involve finite elements or similar discretization methods which come with different regularity constraits on ∂Ω.

It happens we can solve it using montecarlo methods by launching brownian motions. For each point (𝑥,𝑦) in Ω we launch 𝑁 random walks. For each random walk, we wait until it reaches ∂Ω and sample the temperature at exit point. We then sum all those exit temperatures and divide by 𝑁 to get the numerical solution :

This method is quite slow (compared to finite elements or similar). But it has some advantages:

• It can easily be distributed among many threads (all points are independants)
• It’s possible to compute the solution at a specific location
• Ω has almost no regularity constraint (external cone)
• Absolutely no memory footprint (except for the solution)
• It works similarly in higher dimensions

Full explanations can be found on this old research report (in french).

Code

We structured our code as would be a generic mathematical solver. The main class is MonteCarloHeatSolver, which takes a I2DProblem to solve it.
This code is generic and solves a problem described by an interface:

 public class MonteCarloHeatSolver
 {
  I2DProblem _problem;
  
 public MonteCarloHeatSolver(I2DProblem problem)
 {
  _problem = problem;
 }
  
 [EntryPoint]
 public void Solve()
 {
 int stopX = _problem.MaxX();
 int stopY = _problem.MaxY();
 for (int j = 1 + threadIdx.y + blockIdx.y * blockDim.y; j < stopY; j += blockDim.y * gridDim.y) {
 for (int i = 1 + threadIdx.x + blockIdx.x * blockDim.x; i < stopX; i += blockDim.x * gridDim.x) {
  float2 position;
  position.x = i;
  position.y = j;
  _problem.Solve(position);
 }
 }
 }
 } 

where actual resolution (geometry related) is deferred to a I2DProblem. The call to Solve (virtual) won’t be matched to templates. But the Hybridizer will handle this and dispatch this call correctly at runtime. This will cost a vtable lookup, but only once per thread. Performance critical code is in the random walk and the boundary conditions, which are generic parameters of the 2D problem.

An example of C# code instanciation can be:

 var problem = new SquareProblem<SimpleWalker, SimpleBoundaryCondition>(N, iterCount);
 var solver = new MonteCarloHeatSolver(problem);
 solver.Solve(); 

We then have two interfaces describing random walks and boundary conditions :

 [HybridTemplateConcept]
 public interface IRandomWalker
 {
 [Kernel]
 void Init();
 [Kernel]
 void Walk(float2 f, out float2 t);
 }
 [HybridTemplateConcept]
 public interface IBoundaryCondition
 {
 [Kernel]
 float Temperature(float x, float y);
 } 

These interfaces are decorated with [HybridTemplateConcept] which tells the Hybridizer that these types will be used as type parameters. They can be extended by actual classes such as:

 public struct SimpleBoundaryCondition : IBoundaryCondition
 {
 [Kernel]
 public float Temperature(float x, float y)
 {
 if ((x >= 1.0F && y >= 0.5F) || (x <= 0.0F && y <= 0.5F))
 return 1.0F;
 return 0.0F;
 }
 } 

Generic types using these interfaces have to tell the hybridizer how they want it to generate template code from generics. This is again done by using attributes:

For example:

 [HybridRegisterTemplate(Specialize = typeof(SquareProblem<SimpleWalker, SimpleBoundaryCondition>))]
 public class SquareProblem<TRandomWalker, TBoundaryCondition>: I2DProblem 
  where TRandomWalker : struct, IRandomWalker 
  where TBoundaryCondition: struct, IBoundaryCondition
 {
  // other interface methods implementations
  // ...
  
 [Kernel]
 public void Solve(float2 position)
 {
  TRandomWalker walker = default(TRandomWalker);
  TBoundaryCondition boundaryCondition = default(TBoundaryCondition);
  walker.Init(); // generic parameter method call -- will be inlined
 float temperature = 0.0F;
 float size = (float)_N;
 for (int iter = 0; iter < _iter; ++iter)
 {
  float2 f = position;
  
 while (true)
 {
  float2 t;
  walker.Walk(f, out t); // generic parameter method call -- will be inlined
  
  // when on border, break
 if(t.x <= 0.0F || t.y >= size || t.x >= size || t.y <= 0.0F)
 {
  // generic parameter method call -- will be inlined
  temperature += boundaryCondition.Temperature((float)t.x * _h, (float)t.y * _h);
 break;
 }
  
  // otherwise continue walk
  f = t;
 }
 }
  
  _inner[((int)(position.y - 1)) * (_N - 1) + (int)(position.x - 1)] = temperature * _invIter;
 }
  
 } 

 [HybridRegisterTemplate(Specialize = typeof(TetrisProblem<SimpleWalker, TetrisBoundaryCondition>))]
 public class TetrisProblem<TRandomWalker, TBoundaryCondition> : I2DProblem
  where TRandomWalker : struct, IRandomWalker
  where TBoundaryCondition : struct, IBoundaryCondition
 {
  // actual interface implementation
 } 

Results

Dispatchant calls

Virtual functions trigger a callvirt in the MSIL.

IL_005e: callvirt instance void MonteCarloHeatEquation.I2DProblem::Solve

Profiling the generated code with nvvp shows us that a vtable is generated by the Hybridizer (ensuring the right method is called):

Generics and templates

On the other hand, IRandomWalker and IBoundaryCondition type parameters are mapped to templates. Their methods are therefore inlined, as shown in this nvvp profiling:

By the way: the images above show that your C# code is linked to the sass in the profiler. See our post about debugging and profiling

Conclusion

With few restrictions, you can safely use generics type parameters and dispatchant calls in your C# code. Hybridizer will map that to the correct concept (vtable or template) as your required for.
Dispatchant calls give you full flexibility of an inheritance hierarchy, but come at a performance cost. On the other hand, generics deliver full performance (inlined calls), as long as the right metadata has been provided.

The post Generics and virtual functions first appeared on In Fine - Le Blog.

System.Numerics.Vector is a library provided by .Net (as a nuget package), which tries to leverage SIMD instruction on target hardware. It exposes a few value types, such as Vector<T>, which are recognized by RyuJIT as intrinsics.
Supported intrinsics are listed in the core-clr github repository.
This allows C# SIMD acceleration, as long as code is modified to use these intrinsic types, instead of scalar floating point elements.

On the other hand, Hybridizer aims to provide those benefits without being intrusive in the code (only metadata is required).

We naturally wanted to test if System.Numerics.Vector delivers good performance, compared to Hybridizer.

We wrote and ran two benchmarks, and for each of them we have four versions:

• Simple C# scalar code
• Numerics.Vector
• Simple C# scalar code, hybridized
• Numerics.Vector, hybridized

Processor is a core i7-4770S @ 3.1GHz (max measured turbo in AVX mode being 3.5GHz). Peak flops is 224 GFlop/s, or 112 GCFlop/s, if we count FMA as one (since our processor supports it).

Compute bound benchmark

This is a compute-intensive benchmark. For each element of a large double precision array (8 millions elements: 67MBytes), we iterate twelve times the computation of an exponential’s Taylor expansion (expm1). This is largely enough to enter the compute-bound world, by hiding memory operations latency behind a full bunch of floatin point operations.
Scalar code is simply:

 [MethodImpl(MethodImplOptions.AggressiveInlining)]
 public static double expm1(double x)
 {
   return ((((((((((((((15.0 + x)
     * x + 210.0)
     * x + 2730.0)
     * x + 32760.0)
     * x + 360360.0)
     * x + 3603600.0)
     * x + 32432400.0)
     * x + 259459200.0)
     * x + 1816214400.0)
     * x + 10897286400.0)
     * x + 54486432000.0)
     * x + 217945728000.0)
     * x + 653837184000.0)
     * x + 1307674368000.0)
     * x * 7.6471637318198164759011319857881e-13;
 }
 [MethodImpl(MethodImplOptions.AggressiveInlining)]
 public static double twelve(double x)
 {
   return expm1(expm1(expm1(expm1(expm1(expm1(expm1(expm1(expm1(expm1(expm1(x)))))))))));
 } 

on which we added the AggressiveInlining attribute to help RyuJit to merge operations at JIT time.

The Numerics.Vector version of the code is quite the same:

 [MethodImpl(MethodImplOptions.AggressiveInlining)]
 public static Vector<double> expm1(Vector<double> x)
 {
   return ((((((((((((((new Vector<double>(15.0) + x)
     * x + new Vector<double>(210.0))
     * x + new Vector<double>(2730.0))
     * x + new Vector<double>(32760.0))
     * x + new Vector<double>(360360.0))
     * x + new Vector<double>(3603600.0))
     * x + new Vector<double>(32432400.0))
     * x + new Vector<double>(259459200.0))
     * x + new Vector<double>(1816214400.0))
     * x + new Vector<double>(10897286400.0))
     * x + new Vector<double>(54486432000.0))
     * x + new Vector<double>(217945728000.0))
     * x + new Vector<double>(653837184000.0))
     * x + new Vector<double>(1307674368000.0))
     * x * new Vector<double>(7.6471637318198164759011319857881e-13);
} 

The four versions of this code give the following performance results:

As stated, Numerics.Vector delivers a close to 4x speedup from scalar. However, performance is far from what we reach with the Hybridizer. If we look at generated assembly, it’s quite clear why:

 vbroadcastsd ymm0,mmword ptr [7FF7C2255B48h]
 vbroadcastsd ymm1,mmword ptr [7FF7C2255B50h]
 vbroadcastsd ymm2,mmword ptr [7FF7C2255B58h]
 vbroadcastsd ymm3,mmword ptr [7FF7C2255B60h]
 vbroadcastsd ymm4,mmword ptr [7FF7C2255B68h]
 vbroadcastsd ymm5,mmword ptr [7FF7C2255B70h]
 vbroadcastsd ymm7,mmword ptr [7FF7C2255B78h]
 vbroadcastsd ymm8,mmword ptr [7FF7C2255B80h]
 vaddpd ymm0,ymm0,ymm6 
 vmulpd ymm0,ymm0,ymm6
 vaddpd ymm0,ymm0,ymm1
 vmulpd ymm0,ymm0,ymm6
 vaddpd ymm0,ymm0,ymm2
 vmulpd ymm0,ymm0,ymm6
 vaddpd ymm0,ymm0,ymm3
 vmulpd ymm0,ymm0,ymm6
 vaddpd ymm0,ymm0,ymm4
 vmulpd ymm0,ymm0,ymm6
 vaddpd ymm0,ymm0,ymm5
 vmulpd ymm0,ymm0,ymm6
 vaddpd ymm0,ymm0,ymm7
 vmulpd ymm0,ymm0,ymm6
 vaddpd ymm0,ymm0,ymm8
 vmulpd ymm0,ymm0,ymm6
 ; repeated 

Fused multiply add are not reconstructed, and constant operands are reloaded from constant pool at each expm1 invokation. This leads to high registry pressure (for constants), where memory operands could save some.

Here is what the Hybridizer generates from scalar code:

 vaddpd ymm1,ymm0,ymmword ptr []
 vfmadd213pd ymm1,ymm0,ymmword ptr []
 vfmadd213pd ymm1,ymm0,ymmword ptr []
 vfmadd213pd ymm1,ymm0,ymmword ptr []
 vfmadd213pd ymm1,ymm0,ymmword ptr []
 vfmadd213pd ymm1,ymm0,ymmword ptr []
 vfmadd213pd ymm1,ymm0,ymmword ptr []
 vfmadd213pd ymm1,ymm0,ymmword ptr []
 vfmadd213pd ymm1,ymm0,ymmword ptr []
 vfmadd213pd ymm1,ymm0,ymmword ptr []
 vfmadd213pd ymm1,ymm0,ymmword ptr []
 vfmadd213pd ymm1,ymm0,ymmword ptr []
 vfmadd213pd ymm1,ymm0,ymmword ptr []
 vfmadd213pd ymm1,ymm0,ymmword ptr []
 vmulpd ymm0,ymm0,ymm1<br />
 vmulpd ymm0,ymm0,ymmword ptr []
 vmovapd ymmword ptr [rsp+0A20h],ymm0
 ; repeated 

This reconstructs fused multiply-add, and leverages memory operands to save registers.

Why are we not to peak performance (112GCFlops)? That is because Haswell has two pipelines for FMA, and a latency of 5 (see intel intrinsic guide. To reach peak performance, we would need to interleave 2 independant FMA instruction at each cycle. This could be done by reordering instructions, since reorder buffer is not long enough to execute instructions too far in the pipeline. LLVM, our backend compiler, is not capable of such reordering. To get better performance, we unfortunately have to write assembly by hand (which is not exactly what a C# programmer expects to do in the morning).

Invoke transcendentals

In this second benchmark, we need to compute the exponential of all the components of a vector. To do that, we invoke Math.Exp.
Scalar code is:

 [EntryPoint]
 public static void Apply_scal(double[] d, double[] a, double[] b, double[] c, int start, int stop)
 {
   int sstart = start + threadIdx.x + blockDim.x * blockIdx.x;
   int step = blockDim.x * gridDim.x;
   for (int i = sstart; i < stop; i += step)
   {
     d[i] = a[i] * Math.Exp(b[i]) * Math.Exp(c[i]);
   }
 } 

This function is later called in a Parallel.For construct.

However, Numerics.Vector does not provide a vectorized exponential function. Therefore, we have to write our own:

 [IntrinsicFunction("hybridizer::exp")]
 [MethodImpl(MethodImplOptions.AggressiveInlining)]
 public static Vector<double> Exp(Vector<double> x)
 {
   double[] tmp = new double[Vector<double>.Count];
   for(int k = 0; k < Vector<double>.Count; ++k)
   {
     tmp[k] = Math.Exp(x[k]);
   }
   return new Vector<double>(tmp);
 } 

As a glance, we can see the problems: each exponential will first break the AVX context (which cost tens of cycles), and trigger 4 function calls instead of one.

With no surprise, this code performs really badly:

If we look at the generated assembly, it confirms what we suspected (context switched, and ymm register splitting):

 vextractf128 xmm9,ymm6,1
 vextractf128 xmm10,ymm7,1
 vextractf128 xmm11,ymm8,1
 call 00007FF8127C6B80 // exp
 vinsertf128 ymm8,ymm8,xmm11,1
 vinsertf128 ymm7,ymm7,xmm10,1
 vinsertf128 ymm6,ymm6,xmm9,1 

Branching

Branch are expressed using if or ternary operators in scalar code. However, those are not available in Numerics.Vector, since the code is manually vectorized.
Branches must be expressed using ConditionalSelect, which leads to code:

 public static Vector<double> func(Vector<double> x)
 {
   Vector<long> mask = Vector.GreaterThan(x, one);
   Vector<double> result = Vector.ConditionalSelect(mask, x, one);
   return result;
 } 

As we can see, expressing conditions with Numerics.Vector is not intuitive, intrusive, and bug prone. It’s actually the same as writing AVX compiler intrinsics in C++. On the other hand, Hybridizer supports conditions, which allow you to write the above code this way:

 [Kernel]
 public static double func(double x)
 {
   if (x > 1.0)
     return x;
   return 1.0;
 } 

Conclusion

Numerics.Vector gives easily reasonable performances on simple code (no branches, no function calls). Speed-up is what we expect (vector unit width) on simple code. However, it’s time-consuming and error-prone to express conditions, and performance is completely broken as soon as some Jitter Intrinsic is missing (such as exponential).

The post From C# to SIMD : Numerics.Vector and Hybridizer first appeared on In Fine - Le Blog.

We describe here the implementation of Mandelbrot fractal image generation using Hybridizer. The language of choice is C#, and implementation is done using 32-bits precision arithmetic.

Mandelbrot set is the set of values c for which the sequence:

remains bounded in the complex plane.
It happens an equivalent definition is:

That means that while calculating 𝑧𝑛 values exceeds 2 at any iteration, the point c is not in the set.

C# implementation

In other words, this can be simulated via this code:

public static int IterCount(float cx, float cy)
{
    int result = 0;
    float x = 0.0f;
    float y = 0.0f;
    float xx = 0.0f, yy = 0.0f;
    while (xx + yy <= 4.0f && result < maxiter) // are we out of control disk?
    {
        xx = x * x;
        yy = y * y;
        float xtmp = xx - yy + cx;
        y = 2.0f * x * y + cy; // computes z^2 + c
        x = xtmp;
        result++;
    }

    return result;
}

which has to be run for every point (cx, cy) in the complex plane.
To produce an output image, we therefore compute IterCount for every pixel in the square [−2,2]×[−2,2], discretized as a 𝑁×𝑁 square grid:

public static void Run(int[] light)
{
    for (int i = 0; i < N; i += 1)
    {
        for (int j = 0; j < N; j += 1)
        {
            float x = fromX + i * h;
            float y = fromY + j * h;
            light[i* N + j] = IterCount(x, y);
        }
    }
}

where N, h, fromX and fromY are application parameters.
We here compute a 2048×2048 image using C# on a core i7-4770S @ 3.10GHz.
This unoptimized versions runs in 420 milliseconds, yielding

Its crystal clear that this code is embarrassingly parallel, since all pixels are independant from each other.

A first trivial optimization would therefore to make the first loop parallel:

public static void Run(int[] light)
{
    Parallel.For(0, N, (i) => {
        for (int j = 0; j < N; j += 1)
        {
            float x = fromX + i * h;
            float y = fromY + j * h;
            light[i * N + j] = IterCount(x, y);
        }
    });
}

This second version runs in 67 milliseconds, giving:

Run on the GPU

In order to run that on a GPU, we just need to decorate the Run method with EntryPointAttribute:

[EntryPoint("run")]
public static void Run(int[] light)
{
    Parallel.For(0, N, (i) => {
        for (int j = 0; j < N; j += 1)
        {
            float x = fromX + i * h;
            float y = fromY + j * h;
            light[i * N + j] = IterCount(x, y);
        }
    });
}

and some boilerplate code to invoke the generated method:

HybRunner runner = HybRunner.Cuda("Mandelbrot_CUDA.dll").SetDistrib(N, 128);
wrapper = runner.Wrap(new Program());
wrapper.Run(light_cuda);

This modified code runs on the GPU (a 1080 Ti) in 10.6 milliseconds (32 when counting the memory copies), giving:

Further optimization

Launching a block per image line is highly suboptimal, due to the unevenly distribute nature of the computations. For example, threads at the square’s border will immediately converge, while those on the set will take the longest runtime.
This can be seen by profiling the above code using Nsight:

As we can see, half of multiprocessors are idle.

We can instead distribute the work more evenly by using a 2D grid of relatively small blocks.

Fortunately, Hybridizer supports CUDA-like parallelism, so we can modify our entrypoint this way:

[EntryPoint("run")]
public static void Run(int[] light)
{
    for (int i = threadIdx.y + blockDim.y * blockIdx.y; i < N; i += blockDim.x * gridDim.x)
    {
        for (int j = threadIdx.x + blockDim.x * blockIdx.x; j < N; j += blockDim.y * gridDim.y)
        {
            float x = fromX + i * h;
            float y = fromY + j * h;
            light[i * N + j] = IterCount(x, y);
        }
    }
}

and run it with a 2D grid:

HybRunner runner = HybRunner.Cuda("Mandelbrot_CUDA.dll").SetDistrib(32, 32, 16, 16, 1, 0);

The modified code runs now in 920 microseconds on the GPU, meaning:

If we profile the newly generated kernel, we get:

main stall reason being “Other” with 69% followed by instruction fetch for only 5% of not eligible warps.
It’s reasonable to not try further optimization.

Conclusion

In this post we presented a basic usage of hybridizer, from unoptimal c# to highly efficient generated cuda code. We started by optimize our c# code and the migrated it seamlessly to CUDA.
We then optimized our c# code for CUDA GPUs, until we reached a close to peak performance level.

The post Mandelbrot with Hybridizer first appeared on In Fine - Le Blog.