GNU Parallel Command Execution

Install GNU Parallel

# Ubuntu/Debian
sudo apt install parallel

# macOS
brew install parallel

Basic parallel execution

parallel echo ::: arg1 arg2 arg3

Read from stdin

cat urls.txt | parallel curl -O {}

Multiple jobs

parallel -j 4 command ::: arg1 arg2 arg3 arg4

All CPU cores

parallel -j0 command ::: args

Half of CPU cores

parallel -j50% command ::: args

Process files

parallel gzip ::: *.txt

With arguments

parallel convert {} -resize 50% resized/{} ::: *.jpg

Multiple input sources

parallel echo {1} {2} ::: A B C ::: 1 2 3

Dry run

parallel --dry-run command ::: args

Show progress

parallel --progress command ::: args

ETA display

parallel --eta command ::: args

Verbose output

parallel -v command ::: args

Keep order

parallel -k command ::: args

Timeout per job

parallel --timeout 10 command ::: args

Retry failed jobs

parallel --retries 3 command ::: args

Download multiple files

cat urls.txt | parallel -j 5 wget {}

Compress files in parallel

parallel gzip ::: *.log

Convert images

parallel convert {} -resize 800x600 resized/{} ::: *.jpg

Process directories

find . -type d | parallel 'cd {} && npm install'

Run multiple commands

parallel ::: "make clean" "make test" "make build"

SSH to multiple servers

parallel -S server1,server2,server3 'hostname' ::: 1

Complex example with function

export -f process_file
process_file() {
    echo "Processing $1"
    # Your processing logic
}
parallel process_file ::: *.txt

Progress bar with number of jobs

parallel --bar command ::: args

Log output

parallel --joblog jobs.log command ::: args

Resume from log

parallel --resume --joblog jobs.log command ::: args

Results directory

parallel --results results/ command ::: args

Compress output

parallel --compress command ::: args

Pipe mode

cat large_file.txt | parallel --pipe grep "pattern"

Block size for pipe

cat file.txt | parallel --pipe --block 10M grep "pattern"

Combining with find

find . -name "*.log" | parallel gzip {}

XArgs replacement

# Instead of:
find . -name "*.txt" | xargs rm

# Use:
find . -name "*.txt" | parallel rm {}

Load regulation

parallel --load 80% command ::: args

Limit memory usage

parallel --memfree 1G command ::: args

Round robin distribution

parallel --round-robin command ::: args

Quote special characters

parallel -q command "arg with spaces" ::: more args

Null separator (for filenames with spaces)

find . -name "*.txt" -print0 | parallel -0 echo {}

Template replacement

parallel mv {} backup/{.}.bak ::: *.txt

Replacement strings:

• {} - Full input
• {.} - Input without extension
• {/} - Basename
• {//} - Directory
• {/.} - Basename without extension