Python: optimising loops -

i wish optimise python code consisting of 2 nested loops. not familar numpy, understand should enable me improve efficiency such task. below test code wrote reflects happens in actual code. using numpy range , iterator slower usual python one. doing wrong? best solution problem?

thanks help!

import numpy import time  # setup problem analagous in real code npoints_per_plane = 1000 nplanes = 64 naxis = 1000 npoints3d = naxis + npoints_per_plane * nplanes npoints = naxis + npoints_per_plane specres = 1000  # data being mapped sol = dict() sol["ems"] = numpy.zeros(npoints3d) sol["abs"] = numpy.zeros(npoints3d)  # non-random input data data = dict() data["ems"] = numpy.zeros((npoints,specres)) data["abs"] = numpy.zeros((npoints,specres)) ip in range(npoints):     data["ems"][ip,:] = numpy.random.random(specres)[:]     data["abs"][ip,:] = numpy.random.random(specres)[:] ems_mod = numpy.random.random(1)[0] abs_mod = numpy.random.random(1)[0] ispec = numpy.random.randint(specres)  # code want optimize  t0 = time.time()  # usual python range , iterator ip in range(npoints_per_plane):     jp = naxis + ip     ipl in range(nplanes):         ip3d = jp + npoints_per_plane * ipl         sol["ems"][ip3d] = data["ems"][jp,ispec] * ems_mod         sol["abs"][ip3d] = data["abs"][jp,ispec] * abs_mod  t1 = time.time()  # numpy ranges , iterator ip_vals = numpy.arange(npoints_per_plane) ipl_vals = numpy.arange(nplanes) ip in numpy.nditer(ip_vals):     jp = naxis + ip     ipl in numpy.nditer(ipl_vals):         ip3d = jp + npoints_per_plane * ipl         sol["ems"][ip3d] = data["ems"][jp,ispec] * ems_mod         sol["abs"][ip3d] = data["abs"][jp,ispec] * abs_mod   t2 = time.time()  print "plain python: %0.3f seconds" % ( t1 - t0 ) print "numpy: %0.3f seconds" % ( t2 - t1 ) 

edit: put "jp = naxis + ip" in first loop only

additional note:

i worked out how numpy inner loop, not outer loop:

# numpy vectorization ip in xrange(npoints_per_plane):     jp = naxis + ip     sol["ems"][jp:jp+npoints_per_plane*nplanes:npoints_per_plane] = data["ems"][jp,ispec] * ems_mod     sol["abs"][jp:jp+npoints_per_plane*nplanes:npoints_per_plane] = data["abs"][jp,ispec] * abs_mod 

joe's solution below shows how both together, thanks!

the best way of writing loops in numpy not writing loops , instead using vectorized operations. example:

c = 0 in range(len(a)):     c += a[i] + b[i] 

becomes

c = np.sum(a + b, axis=0) 

for a , b shape of (100000, 100) takes 0.344 seconds in first variant, , 0.062 seconds in second.

in case presented in question following want:

sol['ems'][naxis:] = numpy.ravel(     numpy.repeat(         data['ems'][naxis:,ispec,numpy.newaxis] * ems_mod,         nplanes,         axis=1     ),     order='f' ) 

this further optimized some tricks, reduce clarity , premature optimization because:

plain python: 0.064 seconds

numpy: 0.002 seconds

the solution works follows:

your original version contains jp = naxis + ip merely skips first naxis elements [naxis:] selects first naxis elements. inner loop repeats value of data[jp,ispec] nplanes times , writes multiple locations ip3d = jp + npoints_per_plane * ipl equivalent flattened 2d array offset naxis. therefore second dimension added via numpy.newaxis (previously 1d) data['ems'][naxis:, ispec], values repeated nplanes times along new dimension via numpy.repeat. resulting 2d array flattened again via numpy.ravel (in fortran order, i.e., lowest axis having smallest stride) , written appropriate subarray of sol['ems']. if target array 2d, repeat skipped using automatic array broadcasting.

if run situation cannot avoid using loops, use cython (which supports efficient buffer views on numpy arrays).