MParallel: batch processor with multi-threading support

Martin Brinkmann
Nov 9, 2016
Software
|
9

MParallel is a free command line program for Microsoft Windows devices that enables you to run multiple tasks you specify in parallel.

It is probably best to start with a few examples to showcase how powerful MParallel actually is. At a basic level, you could run multiple commands that you can execute from the command line in parallel.

Think of five ping commands to different destinations for example that can run in parallel.

From there, you may explore advanced options like finding all jpg images in a directory and converting them to png format using ImageMagick, or doing the same for video files.

MParallel

The program ships as a portable program. Simply download the latest archive for your operating system -- there is a 32-bit and 64-bit version, and the source code -- and extract the archive when the download completes.

Once done, you may want to run the command mparallel.exe --help to display the list of supported options.

You will notice that there are quite a few that it supports. Generally speaking, it is necessary to begin the command with the program's parameters before you add the programs that you want executed.

The command MParallel.exe --count=3 ping.exe -n 16 fsf.org : ping.exe -n 16 gnu.org : ping.exe -n 16 w3c.org begins with --count=3 which tells MParallel that there will be three commands that you want run in parallel.

This is a basic example, and there is a lot more that you can do. For instance, instead of specifying commands directly, you could tell the program to look for them in a text file instead. This can be useful if you want lots of commands to be executed, or better editing and management options for those commands.

You'd achieve that by using --input=commands.txt, and adding each command to a line in the text file.

A slightly more complicated command is the following one: dir /b *.jpg | MParallel.exe --shell --stdin ---pattern="copy {{0}} {{0:N}}.png"

This uses the dir command to find all .jpg images, to run MParallel and rename the files to .png. Not something that you want to do, but it showcases several new options such as running shell commands, and using pattern.

The package ships with two example command files that you may look at for additional examples. The first uses MParallel in conjunction with ImageMagick to convert jpg images to png files, the other FFmpeg to convert avi files to mp4 files.

Closing Words

MParallel is a mighty program that may take some getting used to time but if you worked with the command line or batch files before, you should feel at home right away. Probably the biggest advantage of it is that it lets you run commands in parallel which should speed up execution significantly on modern systems.

Now You: Do you use the command line at all?

 

 

Summary
software image
Author Rating
1star1star1star1star1star
no rating based on 0 votes
Software Name
MParallel
Operating System
Windows
Software Category
Administration
Landing Page
Advertisement

Tutorials & Tips


Previous Post: «
Next Post: «

Comments

  1. Greg Enlow said on December 5, 2018 at 4:54 pm
    Reply

    Thanks for your tool. I was excited to see it and it looks very promising. However, I seem to have run into a problem with it when it is used in a cmd prompt started per runas as another user (admin account) and with elevated privileges (per nircmdc). For example:

    runas /savecred /user:myAdmin “c:\nircmd\nircmdc.exe elevate cmd /k”

    The tool fails with access denied, though that same command prompt is used extensively without issues:

    dir /a /b Z* | mparallel –count=1 –no-split-lines –stdin –auto-wrap –trace –pattern=”icacls.exe {{0}}”
    Process token: –pattern=icacls.exe {{0}}
    Read line: ZeitBeleg
    Separator: “´´
    Pattern: “icacls.exe {{0}}´´
    Process token: ZeitBeleg
    Enqueue: “icacls.exe ZeitBeleg´´
    Read line: zip2tgz
    Separator: “´´
    Pattern: “icacls.exe {{0}}´´
    Process token: zip2tgz
    Enqueue: “icacls.exe zip2tgz´´

    MParallel v1.0.4 [Jun 8 2016]

    Tasks in queue: 2
    Maximum parallel instances: 1
    icacls.exe ZeitBeleg

    CreateProcessW() failed with Win32 error code: 0x5.

    Process creation has failed: Access is denied.

    ERROR: Process “icacls.exe ZeitBeleg´´could not be created!

    It works fine in a normally started cmd.

    I have high hopes for using this to mitigate extreme run times when processing a large number (>1m) files.
    I will continue testing to see if a work around is possible, but hope you might take a look to see what might be going wrong.
    My thanks in advance,
    Greg

  2. NIYONGABO Olivier said on May 7, 2018 at 1:51 pm
    Reply

    Nice one!!!

  3. Shawn said on November 10, 2016 at 7:04 pm
    Reply

    This is a perfect example of a tool that works great with something like imagemgick.

    Take for example:

    @echo off
    For %%F in (0 1 2 3 4 5 6 7 8 9 A B C D E F) do For %%E in (0 1 2 3 4 5 6 7 8 9 A B C D E F) do For %%D in (0 1 2 3 4 5 6 7 8 9 A B C D E F) do For %%C in (0 1 2 3 4 5 6 7 8 9 A B C D E F) do For %%B in (0 1 2 3 4 5 6 7 8 9 A B C D E F) do For %%A in (0 1 2 3 4 5 6 7 8 9 A B C D E F) do convert -size 1920×1080 xc:#%%F%%E%%D%%C%%B%%A #\%%F%%E\%%D%%C\#%%F%%E%%D%%C%%B%%A.png

    Well instead of waiting for all 16 million to get done in sequence I can split the task in 16 pieces and use all my process power to get the job done faster as the convert command opens and closes itself every single time.

  4. max said on November 10, 2016 at 12:30 pm
    Reply

    Why would you use MParallel in that PNG-copy-example?
    You could do that with a for-loop.
    http://ss64.com/nt/for.html

    1. Matthew Borcherding said on November 10, 2016 at 7:59 pm
      Reply

      For loops do one-by-one-by-one. Serial, not parallel.

      Though I guess you might be able to use a START command without a /WAIT in a FOR loop. Spawn multiple windows that way that run in parallel.

      http://ss64.com/nt/start.html

    2. Martin Brinkmann said on November 10, 2016 at 12:44 pm
      Reply

      Those commands are not run using multi-threading then, or am I mistaken?

  5. trek100 said on November 10, 2016 at 1:14 am
    Reply

    Anything similar for Linux?

    1. Greg said on November 10, 2016 at 9:08 am
      Reply
      1. Ole Tange said on November 10, 2016 at 5:33 pm
        Reply

        GNU Parallel also works on Cygwin.

Leave a Reply

Check the box to consent to your data being stored in line with the guidelines set out in our privacy policy

We love comments and welcome thoughtful and civilized discussion. Rudeness and personal attacks will not be tolerated. Please stay on-topic.
Please note that your comment may not appear immediately after you post it.