.\" $Id: xapply.man,v 3.76 2013/08/14 19:14:51 ksb Exp $ .\" by Kevin Braunsdorf .\" $Compile: Display%h .\" $Display: ${groff:-groff} -Tascii -man %f | ${PAGER:-less} .\" $Display(*): ${groff:-groff} -T%s -man %f .\" $Install: %b -mDeinstall %o %f && cp %f $DESTDIR${TOP:-/usr/local}/man/man1/xapply.1 .\" $Deinstall: ${rm-rm} -f $DESTDIR${TOP:-/usr/local}/man/[cm]a[nt]1/xapply.1* .TH XAPPLY 1 LOCAL .SH NAME xapply - extended apply and xargs replacement .SH SYNOPSIS .ds PN "xapply \fI\*(PN\fP \fB\-f\fP [\fB\-Adgmnsuvxz\fP] [\fB\-\fP\fIcount\fP] [\fB\-a\fP\~\fIc\fP] [\fB\-e\fP\~\fIvar=dicer\fP] [\fB\-F\fP\~\fIscript\fP] [\fB\-i\fP\~\fIinput\fP] [\fB\-J\fP\~\fItasks\fP] [\fB\-N\fP\~\fIelse\fP] [\fB\-p\fP\~\fIpad\fP] [\fB\-P\fP\fIjobs\fP] [\fB\-R\fP\~\fIreq\fP] [\fB\-S\fP\~\fIshell\fP] [\fB\-t\fP\~\fItags\fP] [\fIcmd\fP] [\fIfiles\fP] .br \fI\*(PN\fP [\fB\-Adgmnsuvx\fP] [\fB\-\fP\fIcount\fP] [\fB\-a\fP\~\fIc\fP] [\fB\-e\fP\~\fIvar=dicer\fP] [\fB\-F\fP\~\fIscript\fP] [\fB\-i\fP\~\fIinput\fP] [\fB\-J\fP\~\fItasks\fP] [\fB\-N\fP\~\fIelse\fP] [\fB\-P\fP\fIjobs\fP] [\fB\-R\fP\~\fIreq\fP] [\fB\-S\fP\~\fIshell\fP] [\fB\-t\fP\~\fItags\fP] [\fIcmd\fP] [\fIargs\fP] .br \fI\*(PN\fP \fB\-h\fP .br \fI\*(PN\fP \fB\-H\fP .br \fI\*(PN\fP \fB\-V\fP .SH DESCRIPTION .PP Extended apply is intended to be a complete replacement for the \fBapply\fP(1) and \fBxargs\fP(1) commands. In the normal mode \fI\*(PN\fP emulates the best features of \fBapply\fP (substitutes limited percent escapes to form and execute shell commands). Under \fB\-f\fP it emulates a \fBxargs\fP-like processor. .PP Unlike \fBxargs\fP this program deals in fixed argument lists. Each positional parameter (or input line) is treated as a single argument for the expanded command (embedded blanks, tab, quotes and the like are \fB\¬\fP special). If a variable argument list is required it may be built with \fBfmt\fP or \fBadjust\fP (or any other text processor) and passed in pre-formatted (see BUGS). .PP The fixed argument list is built from a hunk of \fIcount\fP parameters from the command line. These parameters are usually interpreted as fixed strings, but under \fB\-f\fP they are taken as \fIfiles\fP whose lines are each taken sequentially as arguments. .PP The positional parameter \fIcmd\fP is the template command to control each task. In this template each occurrence of the escape character \fIc\fP, percent (\*(lq%\*(rq) by default, followed by a digit ([1-9]) is replaced by the current argument for that slot. When no such expansions occur in a template the whole argument list is catenated on the end of \fIcmd\fP (prefixed by a blank when needed). .PP The \fIcmd\fP is replaced by a \fIscript\fP under \fB\-F\fP. The contents of the file \fIscript\fP replaces the literal specification of a \fIcmd\fP template. Under this option, \fI\*(PN\fP may be a command interpreter for the specified shell (see \fB\-S\fP and an example below). The leading loader line is removed from the script, if it looks like a call to \fI\*(PN\fP. .PP To extend this a bit we also allow curly braces to enclose larger integer value (e.g. %{15}) as \fBmake\fP(1) does for macro names. In addition the value zero (in either case) is taken to mean no expansion is expected; this suppresses the default catenation of the whole argument list as well. The catenation of the whole list may be explicitly specified via the index star (\*(lq*\*(rq). The last parameter may be specified by the index dollar-sign (\*(lq$\*(rq). .PP When the \fB\-f\fP option is applied the name of the file that supplied a parameter is available with \fB%f\fP followed by a specification. .PP To keep up with more complex structures the dicer form is allowed to extract subfields from input parameters: .sp \fB%[\fP\fIparam\fP \fIseparator\fP \fIfield\fP \fI...\fP\fB]\fP .sp The positional parameter \fIparam\fP is broken into fields at the character \fIseparator\fP. The resultant \fIfield\fP is then substituted, or diced again. A negative \fIfield\fP inverts this meaning by removing the resultant field from the original list. The special field dollar-sign (\*(lq$\*(rq) is taken as the last field. .PP When a literal blank is used as the \fIseparator\fP, it matches any number of white space characters. The universal escape character backslash (\e) may be used to quote any of blank, backslash, close bracket, or any digit to suppress its special meaning. Additional \fIseparator\fP and \fIfield\fP pairs may be used to further limit the expansion. .PP The additional form \fB%t\fP selects a group of \fIreq\fP `resources', which are bound to a running task for the life of the process. The list of resources must be indexed by slot (as \fIparam\fPs are), and may be the data source for a dicer expression (e.g. \fB%t[1/$]\fP). A as bonus \fB%ft\fP expands to the filename presented for \fItags\fP, or the string \*(lqiota\*(rq when the original \fItags\fP name cannot be recovered. .PP The special form \fB%u\fP provides a sequential integer which is unique for each iteration of \fI\*(PN\fP's command processor. This is useful to form unique temporary files, to read the output of the last iteration as input, or as a "loop counter". It starts at zero. In addition \fB%u\fP may be the data source for a dicer expression (e.g. %[u\e0$]), but that's not really very useful. .PP The special field selection star (\*(lq*\*(rq) selects the catenation of the selected arguments, be they tokens, filenames, or parameters. This may also be used as the source for a dicer expression. .PP For the unusual case where all that is not enough power the form \fB%+\fP shifts the values of %1, %2, ... to the left once to form a recursive expression treating the old %1 as a \fIcmd\fP. After that expression expands processing continues with the rest of the present template. See \fBhxmd\fP(8L). .PP As a concession to shell programmers an additional flag "q" is allowed after the leading precent (as \*(lq%q\*(rq) to quote the result of any expansion such that the shell quoting rules for double-quotes produces the expanded text exactly. Each of double-quote (\*(lq"\*(rq), backslash (\*(lq\e\*(rq), grave accent (\*(lq`\*(rq), and dollar-sign (\*(lq$\*(rq) are protected with a single backslash. .PP Two stronger versions of \fB%q\fP quote more of the shell meta character: \fB%Q\fP also quotes \fB\&~\fP, \fB*\fP, \fB?\fP, \fB[\fP, \fB|\fP, \fB\fP, \fB&\fP, \fB;\fP, \fB(\fP, \fB)\fP, \fB#\fP, \fB=\fP, \fB'\fP, \fB<\fP, and \fB>\fP. While \fB%W\fP also quotes the default IFS charters (space, tab and newline). These expanders protect the shell from meta characters in unsecure filenames. .PP Enclose any of the above expressions in parenthesis to gain access to the mixer. .sp \fB%(\fP\fIdicer\fP \fB,\fP \fImixer\fP \fI...\fP\fB)\fP .sp This is a character-based expander, the characters in the expanded string are numbered from 1 to $. Any range may be selected for output. For example %([1 2],3) is replaced by the third character of the second word in the first parameter. To make %u more useful: the expansion of %(u,$) is the last digit of the counter, which is the counter modulo 10. More complete documentation on the mixer is available in the explode repository, or the \fBmk\fP(1) documentation. .SH TAGS Under any combination of \fB\-t\fP, \fB\-R\fP, or \fB\-J\fP \fI\*(PN\fP wraps itself in an instance of \fPptbw\fP to manage resource allocation, unless \fItags\fP indicates a reference to an enclosing manager. For details on resource processing see \fBptbw\fP(1L). Note that \fI\*(PN\fP uses \fBptbw\fP's itoa default (the integers from 0 to \fIjobs\fP-1) when none of \fBptbw\fP options are presented (making it transparent for other uses of \fBptbw\fP). .\" It is too late to save you, if you know why this is a Good Thing. -- ksb .SH OPTIONS Note that the option \fB\-F\fP suppresses the positional parameter \fIcmd\fP by providing its value from the specified \fIscript\fP file. .TP .nf \fB\-\fP\fIcount\fP .fi Each expanded command requires \fIcount\fP of arguments. .\" Given %* and such we could loosen the fixed argument count rule: .\" a count of count/[max] could mean pack the items until you have at .\" least count, then add more up to max that fit in the command buffer. .\" The implied max would be computed from the template command. --ksb .TP .nf \fB\-A\fP .fi Emulate \fBptbw\fP's \fB\-A\fP option by appending the \fB%t\fP values on the shell argument vector, which allows access to them as \fI$1\fP, \fI$2\fP, an so on. On systems where \fBcsh\fP(1) needs a pad argument, one (\*(lq_\*(rq) is inserted before the token list, this padding is always provided for \fBsh\fP(1) and compatible shells (viz. \fBksh\fP(1)). If the \fIshell\fP selected is \fBperl\fP(1) the normal \fB-c\fP is replaced with \fB\-e\fP. Note in \fBperl\fP the values are presented as \fB$ARGV[0]\fP, \fB$ARGV[1]\fP\fI...\fP. .TP .nf \fB\-a\fP \fIc\fP .fi Change the escape character from percent (%) to \fIc\fP. .TP .nf \fB\-d\fP .fi Passed on to \fBxclate\fP(1l) to inhibit the publication of our master socket into the \fBxclate\fP environment chain. To be sure this works the command must be called with \fB$xcl_d\fP unset. This might be used to isolate the use of \fB\-m\fP, in some rare cases. .TP .nf \fB\-e\fP \fIvar\fP=\fIdicer\fP .fi .TP .nf \fB\-e\fP \fIvar\fP .fi Set the environment variable \fIvar\fP to the expansion of \fIdicer\fP (with all the percent escapes supported for \fIcmd\fP) for each process launched. The default \fIdicer\fP expression (when only \fIvar\fP is presented) is an expression to extract the next fixed parameter, in turn. .TP .nf \fB\-f\fP .fi Arguments are read one per line from the input \fIfiles\fP. Each file is opened and scanned for arguments (which are only separated by newlines). Note that all references to the \fIstdin\fP ("-") stream share the stream: that is, arguments are taken round-robin from each active reference. Files are hunked together by the \fB\-\fP\fIcount\fP option as fixed parameters would be. .TP .nf \fB\-F\fP \fIscript\fP .fi The specified \fIscript\fP is a file, replacing any literal \fIcmd\fP template. Read \fIscript\fP then remove any leading loader line that matches a leading \fB#!\fP followed by our program name. This allows \fI\*(PN\fP to be used as an interpreter. The markup \fB%c\fP expands to the command read from the file, \fB%fc\fP the name of the \fIscript\fP file, and \fB%tc\fP the loader line from that file. .TP .nf \fB\-g\fP .fi Use \fBgtfw\fP(1l) to get a global unique value for \fB%u\fP, rather than the internal local counter. This only works when \fI\*(PN\fP is wrapped in a \fBgtfw\fP. This also changes \fB\-N\fP's \fIxid\fP from "00" to a unique number, prefixed with \*(lq0\(rq, as multiple parallel instances may have zero iterations. Even without \fB\-f\fP the specification \fB%fu\fP represents the source of the unique stream (\fIhostname\fP\fB:\fPdiversion\fP). Without \fB\-g\fP that source is spelled \*(lqiota\*(rq (the integer sequence function). .TP .nf \fB\-h\fP .fi Print a help message. .TP .nf \fB\-H\fP .fi Print a quick reminder of the available expander markup. .TP .nf \fB\-i\fP \fIinput\fP .fi Under \fB\-f\fP sub-tasks competing for input with \fI\*(PN\fP lead to madness. \fIXapply\fP opens all sub-tasks \fIstdin\fP to a shared stream from \fIinput\fP. Under \fB\-f\fP this defaults to \f(CR/dev/null\fP. .TP .nf \fB\-n\fP .fi Do not execute commands, trace only, which implies \fB\-v\fP. When \fB%t\fP is presented in \fIcmd\fP \fI\*(PN\fP might complain .nf %t used with \-n, possible locking issues .fi Because the output, when run in parallel, will not honor the semantics of \fB\-t\fP (below). .TP .nf \fB\-m\fP .fi Untangle the output of parallel commands with the \fBxclate\fP collation filter. The \fIxid\fP passed to the \fBxclate\fP filter is the expansion of \fI%1\fP, or if that is the empty string the expansion of \fI%u\fP. When not already in a collated environment, \fI\*(PN\fP wraps itself in one. .TP .nf \fB\-J\fP \fItasks\fP .fi Specify the number of tasks that \fBptbw\fP should accommodate under this \fB\&entire\fP process tree. Unless \fB\-t\fP prevents it, this switch forces an instance of \fBptbw\fP(1L) to be wrapped around \fI\*(PN\fP. When the number of \fItasks\fP is specified as zero the \fBptbw\fP system default may be employed, making \fIjobs\fP the upper bound. .TP .nf \fB\-N\fP \fIelse\fP .fi Launch the shell command \fIelse\fP when no \fIcmd\fP would be launched, either because \fIargs\fP was empty or all of \fIfiles\fP were zero length. This is a trap for zero iterations of the loop. The normal percent expansions for positional parameters are bound to the whole argument list, use an explicit "%0" to suppress the catenation of these parameters to \fIelse\fP. Also \fB%t\fP and \fI%u\fP work. The xid passed to \fBxclate\fP is double-zero (\*(lq00\*(rq), so that the exit code from the else may be processed out-of-band, if required. .TP .nf \fB\-p\fP \fIpad\fP .fi Files with inadequate length are padded with this token, by default the empty string. .TP .nf \fB\-P\fP\fIjobs\fP .fi Number of tasks to run in parallel. \fIXapply\fP tries to keep at most \fIjobs\fP processes running in parallel. If the environment variable \fB\&PARALLEL\fP is set then it is read for a default value. \fI\*(PN\fP waits for all tasks to finish before it exits. .TP .nf \fB\-R\fP \fIreq\fP .fi The number of tokens allocated to each task, either from \fItags\fP or from the default integer tableau. The also forces a \fBptbw\fP around the \fI\*(PN\fP process, unless \fB\-t\fP prevents it. There is a special case when \fIreq\fP is specified as zero: the \fB\-R\fP option is not passed on to \fBptbw\fP when started, and no tokens are allocated from it (in fact no tokens are available at all). .TP .nf \fB\-s\fP .fi Under this option when a task produces no output the \fBxclate\fP filter squeezes out the existence of the command (no header, output, or horizontal footer). This speeds the task stream substantially under \fB\-m\fP, but only when a large fraction of the tasks produce no output and are evenly distributed between the tasks that produce output. .TP .nf \fB\-S\fP \fIshell\fP .fi Specify shell to run tasks under. \fIXapply\fP looks for the environment variable \fB\&SHELL\fP or uses the built-in default \f(CR/bin/sh\fP. In any case \fIshell\fP must take \fB\-c\fP as \fBsh\fP(1) does, unless its basename contains the string \fBperl\fP, in which case it should take \fB\-e\fP. .TP .nf \fB\-t\fP \fItags\fP .fi The resources (tokens, targets, terms) read from \fItags\fP create the tableau used by \fB%t\fP, and are controlled by an instance of \fBptbw\fP. .sp The special \fItags\fP name dash (\*(lq-\*(rq) is taken to imply that an \fB\&existing\fP instance of \fBptbw\fP must enclose \fI\*(PN\fP. An alternate form is to give the path to a \fBptbw\fP (or \fBptbw\fP-like) socket as \fItags\fP. .\" See the client interface code in ptbw's module ptbc.m. .TP .nf \fB\-u\fP .fi Use the value of \fB%u\fP as the \fIxid\fP sent to any \fBxclate\fP output filters under \fB\-m\fP. .\" Used largely by \fBhxmd\fP. .TP .nf \fB\-v\fP .fi Be verbose. Echo shell commands to \fIstdout\fP. This option is \fB\&deprecated\fP, in light of \fB\-t\fP. .TP .nf \fB\-V\fP .fi Show version information. .TP .nf \fB\-x\fP .fi Echo processed commands to \fIstderr\fP as they are launched. .TP .nf \fB\-z\fP .fi Under \fB\-f\fP read input files as if they were output from \fBfind\fP's \-print0 option. .SH ENVIRONMENT Always export any environment variable need by any descendant commands. For example the \fBksh\fP script fragment: .RS TEMP1=/tmp/mtf$$.$((RANDOM%100)) .br mkdir $TEMP1 .br \*(PN 'cp %1 $TEMP1/%1' $LIST .RE may expand \fB$TEMP1\fP to the empty string, as the expanding shell is not the script, but a new one started by \fI\*(PN\fP. To make the script function as (I) intended, either export $\fB\&TEMP1\fP before the \fI\*(PN\fP with .RS export TEMP1 .RE or use double, rather than single, quotes around the \fIcmd\fP. .PP The \fB-e\fP option allows descendant \fBxclate\fP output management processes to see more data than they otherwise could. Using the dicer to produce a more informative $\fB\&XCLATE\fP itself, or, by reference, in some variable via \fBxclate\fP's %{\fIenv\fP} expansion might expand under the \fB\-T\fP and \fB\-H\fP options. .TP .nf $\fB\&XAPPLY_WRAP\fP .fi Used by \fI\*(PN\fP in the wrapping of itself in a \fBptbw\fP. This variable should be avoided. .TP .nf $\fB\&SHELL\fP .fi Setting this to an uncommon value may cause poorly formed \fI\*(PN\fP's in descendant processes to fail. Always use \fB\-S\fP explicitly to set the shell to something other than a Bourne compatible shell. .\" Your millage may vary .TP .nf $\fB\&PARALLEL\fP .fi The default value for \fB\-P\fP, when specified without a number. .SH EXAMPLES .TP .nf \fI\*(PN\fP 'cd %1 && make all' */ .fi Change context to each directory in the current directory in turn and run \*(lqmake all\*(rq. .TP .nf \fI\*(PN\fP '[ \-f %1/Makefile ] && cd %1 && make all' */ .fi As above, skipping any directory without a \*(lqMakefile\*(rq. .TP .nf \fI\*(PN\fP 'grep \-l \fItarget\fP %1 >/dev/null || echo %1' *.html .fi List all the HTML files which do not contain the pattern \fItarget\fP. This emulates \fBgrep\fP(1)'s \fI\-L\fP option, when it is not supported. .TP .nf \fI\*(PN\fP \-f 'diff %1 ../version5/%1' manifest | more .fi Compare the files listed in \f(CRmanifest\fP from the current directory to the versions with the same name in \f(CR../version5\fP. .TP .nf \fI\*(PN\fP \-p/dev/null \-f \-2 'diff %1 %2' manifest1 checklist1 .fi Compare (with \fBdiff\fP(1)) each file in \f(CRmanifest1\fP to the same file in \f(CRchecklist1\fP. Substitute \f(CR/dev/null\fP for missing files on the end. {This is only useful if the input files are known to have the arguments in a stable order.} .TP .nf \fI\*(PN\fP 'indent' *.c .fi Run \fBindent\fP(1) for each C source file in the current directory. .TP .nf yes | \fI\*(PN\fP \-f 'echo "%q1"; sleep 1' \- .fi Put a 1 second delay between \fByes\fP(1)'s affirmations. .TP .nf find ~ksb/bin \-type f ! \-perm \-111 \-print | \fI\*(PN\fP \-fx 'chmod a+x' \- .fi Find plain files in my bin which are not executable by all, let \fI\*(PN\fP try to fix each (and show me the progress). .TP .nf find */ \-... | fmt 960 1024 | \fI\*(PN\fP \-f \-i /dev/tty 'vi' \- .fi Run \fBvi\fP(1) on blocks of files from the \fBfind\fP command. This is \fB\&so\fP clever. .TP .nf find ... | \fI\*(PN\fP \-f \-5 \-i /dev/tty 'vi' \- \- \- \- \- .fi Edit five files at a time (until the last edit). .TP .nf \fI\*(PN\fP \-fn "" /etc/passwd .fi A slower version of \fBcat\fP(1). .TP .nf tr ':' '\e012' < /etc/passwd | \fI\*(PN\fP \-7 \-nf 'chown %1 %6' \- \- \- \- \- \- \- .fi Shows one way to get the fields of the password file into \fI\*(PN\fP escapes. .TP .nf \fI\*(PN\fP \-nf 'chown %[1:1] %[1:6]' /etc/passwd .fi A simpler way to get the same password information. .TP .nf \fI\*(PN\fP \-V .fi Output \fI\*(PN\fP's version information. .TP .nf \fI\*(PN\fP '[ \-d %1/RCS ] || echo %1' */ .fi Output each subdirectory that does not have an RCS cache. .TP .nf \fI\*(PN\fP \-f '[ \-f %1 ] && echo %1' List | ... .fi Output the pathnames from \fIList\fP that presently exist (note the inverse could be accomplished as well with \fB||\fP rather than \fB&&\fP). .TP .nf find . \-name \e*.core \-print0 |\fI\*(PN\fP \-fzx 'cd %[1/$] && mv core /crash/core.%u' \- .fi Locate core files, chdir into the target directory, move the core to a (fictional) /crash directory with a uniq extender. .TP .nf \fI\*(PN\fP \-n \-f \-2 '%2' \- \- .fi Output only the even lines from \fIstdin\fP (use '%1' rather than '%2' for the odd lines, use '' to join pairs of lines.) Often used in combination with \fBsed\fP(1) to join stanza files, like AIX uses, into something we can process. .TP .nf jot 999 2 |\fI\*(PN\fP \-f 'set @ `factor %1` && [ $# \-eq 3 ] && echo %1' \- .fi Output all 168 prime numbers from 2 to 999 (be sure /usr/games is in \fB$PATH\fP). .TP .nf \fI\*(PN\fP \-n \-4 '(%+)' "one.%2" two three four .fi Outputs "(one.three)". The \fB%+\fP expander is magic, when you need it. Use with care, as the old %1 may be tainted. .TP .nf \fI\*(PN\fP \-x \-P4 'ppp \-unit%t1 %1' $LIST .fi Run to start \fBppp\fP(1) up to 4 sessions explicitly on \fBtun\fP(4) units (0, 1, 2, and 3), as each finishes another from the list will be started on the newly available \fBtun\fP device. .TP .nf \fI\*(PN\fP \-f \-P8 \-t openif.ct 'iftester %t[1] %[1]' nets.cl .fi Given \fIopenif.ct\fP has a list of open interfaces (e.g. fxp1, bge0, xl0) and \fInets.cl\fP has a list of networks to test, the fictional \fIiftested\fP program runs once for each network, each time with an available interface. .TP .nf \fI\*(PN\fP \-f \-P8 'iftester $IF%t1 %1' nets.cl .fi Use the environment variables \fB$IF0\fP, \fB$IF1\fP... to replace the file \fIopenif.ct\fP above. This assumes there are at least 8 such variables exported to the subshells, so the use of ${IF%{t}:?...} might be wise here. .TP .nf \fI\*(PN\fP \-e C1=%1 \-e C2=%2 \-f \-2 '%0diff $C1 $C2' list1 list2 .fi Avoid using the dicer, in favor of using the shell. The \fB%0\fP expansion turns off the auto-append of \fB%*\fP. .TP .nf \fI\*(PN\fP \-e C1 \-e C2 \-f \-2 '%0diff $C1 $C2' list1 list2 .fi Shorthand for the command above. .TP .nf \fI\*(PN\fP \-fA \-R2 \-P4 \-t $HOME/lib/service.cl 'mystery $1 %1 $2' \- .fi Launch at most 4 copies of the \fImystery\fP program over the elements from \fIstdin\fP with 2 resources selected from \fB$HOME/lib/service.cl\fP This might be constrained by the count of the lines in the services.cl file, or the limit of 4 given to \fB\-P\fP or the number of input line from \fIstdin\fP, which ever is smaller. .TP .nf \fI\*(PN\fP \-fR2 \-P4 \-t $HOME/lib/service.cl 'mystery %t1 %1 %t2' \- .fi The same command, without the \fB-A\fP indirection. .TP .nf ptbw \-m \-t /etc/motd \fI\*(PN\fP \-t \- 'echo %ft' "" .fi Notice that the \fI\*(PN\fP instance is \fB\¬\fP given the name of the token file (\f(CR/etc/motd\fP). It is a little surprising that it outputs the correct path, but it does. .TP .nf \fI\*(PN\fP \fB\-N\fP '%0echo Nothing to compress.' 'gzip' $FILES .fi Compress all of \fB$\fPFILES, or complain there were none. The \fB%0\fP suppresses the catenation of any positional parameters, and is only really needed under \fB\-f\fP, but is always good style to include. .TP .nf \fBfind\fP ... \-print0 |\fI\*(PN\fP \-fz 'echo "gzip %q1" >/var/run/fifo' \- .fi Protect the command stream on input with the zero separator, then on output with the quote expansion. Thus files with a newline, dollar, or double-quote in their name will be passed literally to the fifo consumer process. .TP .nf \fBfind\fP ... \-ls | sed \-e 's/^ *//' | \fI\*(PN\fP \-f '...' \- .fi The output of \fBfind\fP right justifies the block count, which adds leading spaces to small files. We strip off that white-space to allow the dicer to access the "ls \-dgils" fields. Note that device files will require more filtering. .TP .nf \fI\*(PN\fP \-f 'echo "%q(1,$-1)"' \- .fi Use the mixer to reverse the character in each line of \fIstdin\fP to \fIstdout\fP. .TP .nf \fI\*(PN\fP \-nf '%(1,$-1)' \- .fi A much faster version of the previous example, which doesn't spawn at least one process per line, just to run \fBecho\fP(1). .\" actually the one built-in to the shell more than likely .TP .nf find ... \-print0 | binpack \-zN \fIlong-process\fP | xapply \-P10 \-f '' \- .fi Use the local \fBbinpack\fP utility to pack files into \fBvery long\fP argument vectors (close to KERN_ARGMAX), then run 10 of those in parallel. This is a great way to process very large searches, rather than creating a \fBgrep\fP process per file. Note that \fBbinpack\fP does permute the order of the files quite a bit. The \fIlong-process\fP might be something as simple as a \fBgrep\fP command. .TP .nf \fI\*(PN\fP \-xfP 'ssh \-n %1 \e~/bin/poll-host >/tmp/$USER.1/%1 2>&1' host.cl .fi Use $\fB\&PARALLEL\fP as the parallel factor to visit all the hosts in \fUhosts.cl\fP. Run the remote \*(lqpoll-host\*(rq command with \fIstdout\fP and \fIstderr\fP both redirected to a local files named for each host. Post-processing of these output files reveals the status of each host (via \fBgrep\fP filters, size filters, or even the execution of the resultant file). This is the on-ramp to \fBhxmd\fP. .TP .nf id | \fI\*(PN\fP \-f \-n '%[1 $=2,@(1]' \- .fi Output the list of numeric groups for the current session. This uses the dicer's list markup (via \fB@\fP), which is only explained in the dicer documentation. See the names with '%[1 $=2,@(2)1]'. .sp Read the example dicers as \*(lqfirst parameter broken on spaces, take the last field, broken on '=' take the second field, split on comma apply a split on open paren to merge the first fields into the result\*(rq. In the alternate \*(lq..., split on comma apply a split on open paren take the second field split on close paren taking the first field to merge into the result\*(rq. .TP .nf #!/usr/bin/env \-S xapply \-P \-F .fi An example interpreter line to process a script through \fI\*(PN\fP. The script uses markup to expand the parameters, as any \fIcmd\fP would, and options to \fI\*(PN\fP may be presented as part of the command. Note that the name of the script is appended by \fBexecve\fP(2). Also note that this \fB\&usually\fP puts the contents of the file in the process table, so don't include any private data in such a file. (Sadly, only BSD versions of \fBenv\fP(1) have the \fB\-S\fP option.) .TP .nf xapply 'echo cmd=\e"%Qc\e" loader=\e"%Qtc\e" script=\e"%Qfc\e"%0' 1 .fi An example of the values bound for script execution: output \fIcmd\fP, the current default shell as an interpreter line, and the token \fBcommand-line\fP. .SH "SHORT CIRCUIT" The \fI\*(PN\fP loop may be short circuited by sending a \fB\&USR1\fP signal (see \fBkill\fP(1)) to the \fI\*(PN\fP process itself. To that end the expander \fB%p\fP is replaced by the appropriate process-id. Any task that is already running is allowed to complete, but no new tasks will be launched. This is almost always used in processes that are matching sets of elements against some automated search criteria. .P \fIXapply\fP does consume the rest of the input data to prevent a broken pipe. It also opens a channel to \fIxclate\fP if the output is managed (under \fB\-m\fP) to add a notification of \*(lqcompletion\*(rq for each elided task. Such a notification never includes the exit code (which didn't happen), just the \fIxid\fP. Under \fB\-x\fP a trace of the elided process's \fIxid\fP is displayed on \fIstderr\fP. .P In the example below no even numbers are displayed: .RS .nf \fI\*(PN\fP '[ %1 \-eq 2 ] && kill \-USR1 %p || echo %1' 3 1 2 4 .fi .RE The 2 is removed by the execution of the \fBkill\fP command, while the 4 is removed by having never been launched. Adding \fB\-P5\fP \fImay\fP allow the 4th iteration to launch, because the \fBkill\fP process is racing the \fI\*(PN\fP loop. .P To see the exit code list replace \*(lq\*(PN\*(rq above with a new prefix: .RS .nf xclate \-m \-rN'>res' xapply \-m .fi .RE This creates a file named \*(lqres\*(rq with the notification stream in it. Display that with: .RS .nf tr '\e000' '\en' job%u' 1 3 5 7 9 3 4 .fi The shell is parsing \fB1>\fP (\fB3>\fP, \fB5>\fP...) as requests to redirect the specified descriptor. Take extra care with spaces and quotes to avoid such issues. .SH AUTHOR K S Braunsdorf, from the Non-Player Character Guild .br xapply at-not-a-spammer ksb dot npcguild.org .SH "SEE ALSO" .hlm 0 sh(1), cat(1), xclate(1l), ptbw(1l), apply(1), xargs(1), hxmd(8l), mk(1l) fmt(1) or adjust(1), make(1), find(1), sed(1), ppp(8), perl(1), execve(2), env(1), binpack(1l), wc(1)