XProc 2.0: An XML Pipeline Language

1 Introduction

An XProc Pipeline specifies a set of operations to be performed on a collection of input documents. Pipelines take documents as their input and produce documents as their output.

This document introduces an entirely new syntax for XProc:

XProc remains a data flow language. There are steps, indivisible black boxes of computation, connected together into a graph. The connections are the bindings between the output(s) of one step and the input(s) of another.

Step declarations (see Section 7, “Step Declarations and Invocations”), resemble function declarations. The use of a step resembles a function invocation. This has an important consequence: the inputs and outputs are both named and ordered. If the declaration for the xslt step has two inputs, source and stylesheet, declared in that order, then it is coherent to speak of them both by name and by ordinal position: the source input is first, the stylesheet input is second.

2 Terminology

All of the terminology in this document is subject to revision. Some of the terms are better and more clearly defined than others. We present a list of terms here so that you will recognize them when they first appear. Most are explained in more detail, but in this draft, not always at the point of first use.

Some terms, such as port, denote the same concept in XProc 2.0 that they did in XProc 1.0. For conciseness, we use those terms in this draft without further explanation.

3 Understanding XProc 2.0

A flow is a description of the data flow graph. On the left and right sides of a chain of steps, we use structuring and de-structuring to assign ports to variables. The result is a variable that (logically) denotes a sequence of items.

Steps have declarations but are not described otherwise within the flow. An implementation can associate a step signature with an implementation in their domain language by matching function signatures.

Step invocations can be chained together by ordering them in a sequence connected by the arrow operator (i.e., -> or → U+2192). A step chain must fully specify the input port bindings along with any required options.

Let’s begin with an example pipeline. This is the “example 3” pipeline from the XProc 1.0 specification.

xproc version = "2.0"; ❶

(: This example is from the XProc 1.0 specification (example 3). :)

 inputs $source as document-node(); ❷
outputs $result as document-node(); ❸

$source → { if (xs:decimal($1/*/@version) < 2.0) ❹
            then [$1,"v1schema.xsd"] → validate-with-xml-schema() ≫ @1 ❺
            else [$1,"v2schema.xsd"] → validate-with-xml-schema() ≫ @1}
        → [$1,"stylesheet.xsl"] → xslt() ❻
≫ $result ❼

4 Step Chains

A step chain is a sequence of step invocations separated by the chain operator (i.e., -> or → U+2192). On the left of the chain operator is always a preceding step or input bindings. On the right must be a step invocation, a block expression, or an optional output binding.

The simplest input binding is a single expression that evaluates to a sequence of one or more items. For example, the document(s) bound to $in can be an input binding for the XInclude step:

$in → xinclude()

If a step takes multiple inputs, the individual bindings must be surrounded by square brackets:

["document.xml", "style.xsl"] → xslt()

In a binding with multiple inputs, the first input is bound to the first input port (in declaration order), the second input to the second port, etc. If necessary, or for clarity, a binding may be preceded by a name assignment that explicitly names a port:

[source="document.xml", stylesheet="style.xs"] → xslt()

If positional and name references are mixed, all positional references must precede the first named reference.

Steps produce some number of outputs on named ports. The outputs of a step invocation immediately preceding the chain operator are available as numbered inputs $1, $2, etc. whose order is the order of the output declarations on the step. For example, the xslt step has two output ports, result and secondary, declared in that order. Following an xslt step, $1 refers to the result port and $2 refers to the secondary port.

$in → xinclude() → [$1,"stylesheet.xsl"] → xslt()

A reference to an ordinal port that does not exist produces an empty sequence of documents.

If two steps are connected together without an intervening input binding, the implicit input binding is that the ports are connected ordinally:

→ [$1,$2,$3,…$n] →

So this flow:

$in → xinclude() → store("included.xml")

is equivalent to this one:

$in → xinclude() → [$1] → store("included.xml")

5 Inputs

Inputs…

"doc.xml" → xinclude()

("d1.xml","d2.xml","d3.xml") → xinclude()

($in,"doc.xml") → xinclude()

[collection=($main,$secondary), query="query.xq"] → xquery()

[$in,"stylesheet.xsl"] → xslt() → [($1,$2),"query.xq"] → xquery()

data "application/json" {
   {
      name: "Alex",
      favoriteColor: "orange"
   }
}

data "application/json" { [ 1,2,3,4] }

data "application/xml" { <doc><title>A test</title></doc> }

data "text/html" {
    <!DOCTYPE html>
    <html>
    <head><title>Template</title>
    <link type="text/css" href="style.css">
    </head>
    <body>…</body>
    </html>
}

data "text/plain" { "Now is the time for all good XProc …" }

6 Output bindings

The output binding operator (i.e., '>>' or ≫ U+226B) takes a step chain or port variable reference on the left hand side and binds the output to the right hand side (i.e., a port variable reference, a URI reference, or an ordered port ordinal.). The output binding operator is used to construct more complex chains of data flows, store results, or write to output ports for returning results.

The symbol “≫” is evocative of the “append” operation familiar from many command-line systems. An output binding appends data to its right hand side in the sense that it causes data to be sent there and if several chains cause data to be sent to the same place, the effect will be logical appending.

The identity assignment is performed by simply binding the input to the output:

$in ≫ $out

The result is all the input on $in is sent to the output port $out as it flows through the graph.

A literal URI reference implies a document store:

"doc.xml" → xinclude() ≫ "included.xml"

In the case of implicit store, if the same output URI is used more than once, the result of sending a sequence there is implementation defined (e.g., the last document written).

If the outputs need to be referenced as inputs elsewhere, they can be assigned to variables:

$in → xinclude() ≫ $included
[$included,"schema.xsd"] → validate-with-xml-schema()
[$included,"stylesheet.xsl"] → xslt() ≫ [result=$out,secondary=$chunks]

Variables assigned in this way can be used like any other variable in expressions, but the implementation must enforce the following semantics:

For example, the following has two documents flowing through $included:

"doc1.xml" → xinclude() ≫ $included
"doc2.xml" → xinclude() ≫ $included
$included → validate-with-xml-schema()

The first two step chains are independent and the processor is free to run them in either order, or in parallel. However, what is passed to validate-with-xml-schema() when the $included variable is referenced must be all of the documents written by the first chain followed by all of the documents written by the second, or vice versa.

The names of output ports can be omitted in which case the assignments are taken in declaration order. For example, the XSLT step declares the result port first and the secondary port second. An explicit set of bindings:

[source=$in,stylesheet="stylesheet.xsl"] → xslt() ≫ [result=$out,secondary=$chunks]

can be shortened to:

[$in,"stylesheet.xsl"] → xslt() ≫ [$out,$chunks]

Within any context, every declared output port has an unnamed ordinal. Some expressions (e.g. block expressions) have implicitly declared output ports.

The ordinals can be referenced by name as @1, @2, etc.

$in → { if ($1/doc/cheese='cheddar')
        then consume() ≫ @1
        else reject() ≫ @1 }
     ≫ $out

7 Step Declarations and Invocations

All steps are declared as external procedures with any number of named inputs and outputs.

step my:computation()
 inputs $source as document-node(),
outputs $result as xs:int*;

Steps are always declared with qualified name. When they are are invoked, a default namespace may be assumed by the processor.

Steps may have any number of options that can be optional and defaulted:

step p:xslt(
  $initial-mode as xs:string ?,
  $template-name as xs:string?,
  $output-base-uri as xs:string?,
  $parameters as map()? = (),
  $version as xs:string = "2.0"
)
   inputs $source as document-node()+,
          $stylesheet as document-node()
   outputs $result as document-node()?,
           $secondary as document-node()*;

All required options must be listed first in the declaration.

Options values are specified on invocation. Any unnamed option values are matched in declaration order. Afterwards, all parameters must be specified with a name.

For example:

xslt("toc",$output-base-uri=base-uri($source))

invokes the xslt step with the option value "toc" for $initial-mode and explicitly named value for $output-base-uri but does not specify a value for $template-name. The value of $version is defaulted to "2.0".

8 Block Expressions

A step chain may contain a block expression. A block expression always has a ordinal set of inputs and outputs. The inputs are assigned from the context of the expression in the chain. The outputs are assigned based on the flow contained within the expression.

A block expression is enclosed within a set of curly brackets and contains any number of step chains or other statements.

9 Conditionals

A conditional may be placed within a step chain when surrounded by curly brackets:

$in → { if ($1/*/@version eq "v1.0")
        then [$1,"crummy.xsl"] → xslt() ≫ @1
        else [$1,"better.xsl"] → xslt() ≫ @1 }
    ≫ $out

When the if/then expression is invoked, it acts as a guard on the flows contained within the clause. Only one of the flows will execute.

The outputs of the block are completely determined by the flows executed. If they do not append any output to the ordinal outputs of the block expression, the expression will not have any output. That is, there is no implicit chaining of outputs.

10 Variables

Within curly bracketed expressions, a let clause may be use to assign variables to values:

$in → {
   let $version := xs:int($1/*/@version) {
      if ($version < 2)
      then [$1,"schema1.xsd"] → validate-with-xml-schema() ≫ @1
      else if ($version < 3)
      then [$1,"schema2.xsd"] → validate-with-xml-schema() ≫ @1
      else fail("No schema available")
   }
} ≫ $out

The variables share the same scope as port variable references but cannot be used within append operators on the right side.

For example, this is not legal:

$in → {
   let $dates := xs:dateTime($1/*/updated) {
     [$1,"schema1.xsd"] → validate-with-xml-schema() ≫ [@1,$dates]
   }
}

but you can do this:

$in → {
   let $dates := xs:dateTime($1/*/updated) {
     $dates >> @2
     [$1,"schema1.xsd"] → validate-with-xml-schema() ≫ @1
   }
}

11 Projections

A source can be turned into a sequence by an expression. The result is a port that contains a sequence of items.

For example:

$in//section → count() ≫ $out

assigns the count of section element subtrees.

12 Iteration

Iteration is a core operation and can be embedded within a step chain with the ! operator. For example:

("d1.xml","d2.xml","d3.xml") ! { [$1,"schema.xsd"] → validate-with-xml-schema() ≫ @1}

validates the three documents contained in the sequence.

The result of an iteration operation is a set of output bindings where the first binding contains all of the documents written to @1, the second all of the documents written to @2, etc.

13 Replacement

A portion of a document can be iterated over and replaced by an embedded step chain. The replace operator requires a single input, an expression, and a step chain body.

For each subtree matched, the block expression is run with the subtree on the positional input port $1. The item on the positional output port @1 will be its replacement.

It is an error if the block expression does not produce a replacement.

For example:

$in → replace (/doc/section) { [$1,"style.xsl"] → xslt() ≫ @1 } ≫ $out

applies XSLT over a subtree.

14 Tee

A chain can have an alternate flow embedded within the chain using the tee operator (tee or ⊤). The flow must be a block expression. The outputs following tee expression are exactly the same as if the tee operator had been omitted.

In the following example, the result of the xinclude step is stored via an tee operator and that result is also transformed by the xslt step.

$in → xinclude() ⊤ { $1 ≫ "included.xml" }
    → [$1,"stylesheet.xsl"] → xslt()
    ≫ $out

15 Flow Declarations

A flow can named and reused:

flow my:process
   inputs $source as document-node(),
  outputs $result as document-node() {
    $source → xinclude() → [$1,"stylesheet.xsl"] → xslt() ≫ $result
 };

"doc.xml" → my:process() ≫ "doc.html"

16 XProc Modules

XProc modules are top-level containers for reuse. Every XProc module must start with:

xproc version = "2.0";

A module consists of a version declaration (above), a set of declarations, and a single optional unnamed flow description.

A module may end with a flow description. The inputs and outputs of that port must be provided by the implementation when the module is invoked.

A module may import other declarations via the import statement:

import "library.xpl";

A module may import declarations in the expression language:

import "functions.xq";

A module may also declare options as parameters to the module.

option $user as xs:string;
option $passwd as xs:string;

A must provide declarations for any undefined inputs and outputs to the flow:

 inputs $source as document-node();
outputs $result as document-node();