MParallel: batch processor with multi-threading support
MParallel is a free command line program for Microsoft Windows devices that enables you to run multiple tasks you specify in parallel.
It is probably best to start with a few examples to showcase how powerful MParallel actually is. At a basic level, you could run multiple commands that you can execute from the command line in parallel.
Think of five ping commands to different destinations for example that can run in parallel.
From there, you may explore advanced options like finding all jpg images in a directory and converting them to png format using ImageMagick, or doing the same for video files.
MParallel
The program ships as a portable program. Simply download the latest archive for your operating system -- there is a 32-bit and 64-bit version, and the source code -- and extract the archive when the download completes.
Once done, you may want to run the command mparallel.exe --help to display the list of supported options.
You will notice that there are quite a few that it supports. Generally speaking, it is necessary to begin the command with the program's parameters before you add the programs that you want executed.
The command MParallel.exe --count=3 ping.exe -n 16 fsf.org : ping.exe -n 16 gnu.org : ping.exe -n 16 w3c.org begins with --count=3 which tells MParallel that there will be three commands that you want run in parallel.
This is a basic example, and there is a lot more that you can do. For instance, instead of specifying commands directly, you could tell the program to look for them in a text file instead. This can be useful if you want lots of commands to be executed, or better editing and management options for those commands.
You'd achieve that by using --input=commands.txt, and adding each command to a line in the text file.
A slightly more complicated command is the following one: dir /b *.jpg | MParallel.exe --shell --stdin ---pattern="copy {{0}} {{0:N}}.png"
This uses the dir command to find all .jpg images, to run MParallel and rename the files to .png. Not something that you want to do, but it showcases several new options such as running shell commands, and using pattern.
The package ships with two example command files that you may look at for additional examples. The first uses MParallel in conjunction with ImageMagick to convert jpg images to png files, the other FFmpeg to convert avi files to mp4 files.
Closing Words
MParallel is a mighty program that may take some getting used to time but if you worked with the command line or batch files before, you should feel at home right away. Probably the biggest advantage of it is that it lets you run commands in parallel which should speed up execution significantly on modern systems.
Now You: Do you use the command line at all?
Thanks for your tool. I was excited to see it and it looks very promising. However, I seem to have run into a problem with it when it is used in a cmd prompt started per runas as another user (admin account) and with elevated privileges (per nircmdc). For example:
runas /savecred /user:myAdmin “c:\nircmd\nircmdc.exe elevate cmd /k”
The tool fails with access denied, though that same command prompt is used extensively without issues:
dir /a /b Z* | mparallel –count=1 –no-split-lines –stdin –auto-wrap –trace –pattern=”icacls.exe {{0}}”
Process token: –pattern=icacls.exe {{0}}
Read line: ZeitBeleg
Separator: “´´
Pattern: “icacls.exe {{0}}´´
Process token: ZeitBeleg
Enqueue: “icacls.exe ZeitBeleg´´
Read line: zip2tgz
Separator: “´´
Pattern: “icacls.exe {{0}}´´
Process token: zip2tgz
Enqueue: “icacls.exe zip2tgz´´
MParallel v1.0.4 [Jun 8 2016]
Tasks in queue: 2
Maximum parallel instances: 1
icacls.exe ZeitBeleg
CreateProcessW() failed with Win32 error code: 0x5.
Process creation has failed: Access is denied.
ERROR: Process “icacls.exe ZeitBeleg´´could not be created!
…
It works fine in a normally started cmd.
I have high hopes for using this to mitigate extreme run times when processing a large number (>1m) files.
I will continue testing to see if a work around is possible, but hope you might take a look to see what might be going wrong.
My thanks in advance,
Greg
Nice one!!!
This is a perfect example of a tool that works great with something like imagemgick.
Take for example:
@echo off
For %%F in (0 1 2 3 4 5 6 7 8 9 A B C D E F) do For %%E in (0 1 2 3 4 5 6 7 8 9 A B C D E F) do For %%D in (0 1 2 3 4 5 6 7 8 9 A B C D E F) do For %%C in (0 1 2 3 4 5 6 7 8 9 A B C D E F) do For %%B in (0 1 2 3 4 5 6 7 8 9 A B C D E F) do For %%A in (0 1 2 3 4 5 6 7 8 9 A B C D E F) do convert -size 1920×1080 xc:#%%F%%E%%D%%C%%B%%A #\%%F%%E\%%D%%C\#%%F%%E%%D%%C%%B%%A.png
Well instead of waiting for all 16 million to get done in sequence I can split the task in 16 pieces and use all my process power to get the job done faster as the convert command opens and closes itself every single time.
Why would you use MParallel in that PNG-copy-example?
You could do that with a for-loop.
http://ss64.com/nt/for.html
For loops do one-by-one-by-one. Serial, not parallel.
Though I guess you might be able to use a START command without a /WAIT in a FOR loop. Spawn multiple windows that way that run in parallel.
http://ss64.com/nt/start.html
Those commands are not run using multi-threading then, or am I mistaken?
Anything similar for Linux?
GNU parallel:
https://www.gnu.org/software/parallel/
GNU Parallel also works on Cygwin.