This example runs OpenMDAO in parallel which requires petsc4py and mpi4py. You must have these packages installed in order to proceed. To get these packages set up on Linux, see MPI on Linux. To get these packages set up on Windows, see MPI on Windows.

Distributed ComponentsΒΆ

OpenMDAO can work with components that are actually distributed themselves. This is useful for dealing with complex tools, like PDE solver (CFD or FEA). But it can also be used to speed up any calculations you’re implementing yourself directly in OpenMDAO using our MPI-based parallel data passing.

Why should you use OpenMDAO to build your own distributed components? Because OpenMDAO lets you build distributed components without writing any significant MPI code yourself. Here is a simple example where we break up the job of adding a value to a large float array (1,000,000 elements).

from __future__ import print_function
import numpy as np
from six.moves import range

from openmdao.api import Component
from openmdao.util.array_util import evenly_distrib_idxs

class DistributedAdder(Component):
    Distributes the work of adding 10 to every item in the param vector

    def __init__(self, size=100):
        super(DistributedAdder, self).__init__()

        self.local_size = self.size = int(size)

        #NOTE: we declare the variables at full size so that the component will work in serial too
        self.add_param('x', shape=size)
        self.add_output('y', shape=size)

    def get_req_procs(self):
        min/max number of procs that this component can use
        return (1,self.size)

    def setup_distrib(self):
        specify the local sizes of the variables and which specific indices this specific
        distributed component will handle. Indices do NOT need to be sequential or

        comm = self.comm
        rank = comm.rank

        # NOTE: evenly_distrib_idxs is a helper function to split the array
        #       up as evenly as possible
        sizes, offsets = evenly_distrib_idxs(comm.size, self.size)
        local_size, local_offset = sizes[rank], offsets[rank]
        self.local_size = int(local_size)

        start = local_offset
        end = local_offset + local_size

        self.set_var_indices('x', val=np.zeros(local_size, float),
            src_indices=np.arange(start, end, dtype=int))
        self.set_var_indices('y', val=np.zeros(local_size, float),
            src_indices=np.arange(start, end, dtype=int))

    def solve_nonlinear(self, params, unknowns, resids):

        #NOTE: Each process will get just its local part of the vector
        print('process {0:d}: {1}'.format(self.comm.rank, params['x'].shape))

        unknowns['y'] = params['x'] + 10

class Summer(Component):
    Agreggation component that collects all the values from the distributed
    vector addition and computes a total

    def __init__(self, size=100):
        super(Summer, self).__init__()

        #NOTE: this component depends on the full y array, so OpenMDAO
        #      will automatically gather all the values for it
        self.add_param('y', val=np.zeros(size))
        self.add_output('sum', shape=1)

    def solve_nonlinear(self, params, unknowns, resids):

        unknowns['sum'] = np.sum(params['y'])

The distributed component magic happens in the setup_distrib method of the DistributedAdder class. This is where we tell the framework how to split up the the big array into smaller chunks handled separately by each distributed process. In this case, we just split the array up one chuck at a time in order as we go from process to process. But OpenMDAO does not require that the src_indices be ordered or sequential!


Only the DistributedAdder class is a distributed component. The Summer is class is a normal component that aggregates the whole array to sum it up.

Next we’ll use these components to build an actual distributed model:

import time

from openmdao.api import Problem, Group, IndepVarComp

from openmdao.core.mpi_wrap import MPI

if MPI:
    # if you called this script with 'mpirun', then use the petsc data passing
    from openmdao.core.petsc_impl import PetscImpl as impl
    # if you didn't use `mpirun`, then use the numpy data passing
    from openmdao.api import BasicImpl as impl

#how many items in the array
size = 1000000

prob = Problem(impl=impl)
prob.root = Group()

prob.root.add('des_vars', IndepVarComp('x', np.ones(size)), promotes=['x'])
prob.root.add('plus', DistributedAdder(size), promotes=['x', 'y'])
prob.root.add('summer', Summer(size), promotes=['y', 'sum'])


prob['x'] = np.ones(size)

st = time.time()

#only print from the rank 0 process
if prob.root.comm.rank == 0:
    print("run time:", time.time() - st)
    #expected answer is 11
    print("answer: ", prob['sum']/size)

You can run this model in either serial or parallel, depending on how you call the script. Lets say you put the above code into a python script called Then to run it in serial you would call it just like any other python script:


In that case, you’ll expect to see some output that looks like this:

process 0: (30000000,)
run time: 1.76785802841
answer:  11.0

To run the model in parallel you need to have an MPI library (e.g. OpenMPI), mpi4py, PETSc, and petsc4py installed. Then you can call the script like this:

mpirun -n 2 python

And you can expect to see some output as follows:

process 0: (15000000,)
process 1: (15000000,)
run time: 1.00080680847
answer:  11.0

With two processes running, you get a decent speed up. You can see that each process took half the array. Why don’t we get a full 2x speedup? Two reasons. The first, and more significant factor is that we don’t have a fully parallel model. The DistributedAdder component is distributed, but the Summer component is not. This introduces a bottleneck because we have to wait for the serial operation to complete.