Linux Ask!

Linux Ask! is a Q & A web site specific for Linux related questions. Questions are collected, answered and audited by experienced Linux users.

Dec 282013
 

Parallel the execution of a job that read from stdin

Answer:

Assume you have a huge list of file, e.g.

line1
line2
line3
..

You want to "feed every line" of this file to an external program that read from stdin, to parallel it, you can use the gnu parallel, e.g.

cat input.txt | parallel -j16 -N1 --pipe myprogram

 Leave a Reply

(required)

(required)

You may use these HTML tags and attributes: <a href="" title=""> <abbr title=""> <acronym title=""> <b> <blockquote cite=""> <cite> <code> <del datetime=""> <em> <i> <q cite=""> <strike> <strong>