PipelineOptions are used to configure Pipelines. You can extend
PipelineOptions to create
custom configuration options specific to your
Pipeline, for both local execution and
execution via a
PipelineRunner.
PipelineOptions and their subinterfaces represent a collection of properties which can
be manipulated in a type safe manner.
PipelineOptions is backed by a dynamic
Proxy which allows for type safe manipulation of properties in an extensible fashion through
plain old Java interfaces.
PipelineOptions can be created with
PipelineOptionsFactory#create() and
PipelineOptionsFactory#as(Class). They can be created from command-line arguments with
PipelineOptionsFactory#fromArgs(String[]). They can be converted to another type by invoking
PipelineOptions#as(Class) and can be accessed from within a
DoFn by invoking
getPipelineOptions() on the input
DoFn.ProcessContext object.
For example:
// The most common way to construct PipelineOptions is via command-line argument parsing:// To create options for the DirectRunner:
DirectOptions directRunnerOptions =
PipelineOptionsFactory.as(DirectOptions.class);
// To cast from one type to another using the as(Class) method:
ApplicationNameOptions applicationNameOptions =
directPipelineOptions.as(ApplicationNameOptions.class);
// Options for the same property are shared between types
// The statement below will print out the name of the enclosing class by default
System.out.println(applicationNameOptions.getApplicationName());
// Prints out registered options.
PipelineOptionsFactory.printHelp(System.out);
// Prints out options which are available to be set on ApplicationNameOptions
PipelineOptionsFactory.printHelp(System.out, ApplicationNameOptions.class);
}
Defining Your Own PipelineOptions
Defining your own
PipelineOptions is the way for you to make configuration options
available for both local execution and execution via a
PipelineRunner. By having
PipelineOptionsFactory as your command-line interpreter, you will provide a standardized way for
users to interact with your application via the command-line.
To define your own
PipelineOptions, you create a public interface which extends
PipelineOptions and define getter/setter pairs. These getter/setter pairs define a collection of
JavaBean
properties.
For example:
// Creates a user defined property called "myProperty"}
Note: Please see the section on Registration below when using custom property types.
Restrictions
Since PipelineOptions can be "cast" to multiple types dynamically using
PipelineOptions#as(Class), a property must conform to the following set of restrictions:
- Any property with the same name must have the same return type for all derived interfaces
of
PipelineOptions.
- Every bean property of any interface derived from
PipelineOptions must have a
getter and setter method.
- Every method must conform to being a getter or setter for a JavaBean.
- The derived interface of
PipelineOptions must be composable with every interface
part registered with the PipelineOptionsFactory.
- Only getters may be annotated with
JsonIgnore.
- If any getter is annotated with
JsonIgnore, then all getters for this
property must be annotated with
JsonIgnore.
Annotations For PipelineOptions
Description can be used to annotate an interface or a getter with useful
information which is output when
--help is invoked via
PipelineOptionsFactory#fromArgs(String[]).
Default represents a set of annotations that can be used to annotate getter
properties on
PipelineOptions with information representing the default value to be
returned if no value is specified. Any default implementation (using the
default keyword)
is ignored.
Hidden hides an option from being listed when
--help is invoked via
PipelineOptionsFactory#fromArgs(String[]).
Validation represents a set of annotations that can be used to annotate
getter properties on
PipelineOptions with information representing the validation
criteria to be used when validating with the
PipelineOptionsValidator. Validation will be
performed if during construction of the
PipelineOptions,
PipelineOptionsFactory#withValidation() is invoked.
JsonIgnore is used to prevent a property from being serialized and
available during execution of
DoFn. See the Serialization section below for more details.
Registration Of PipelineOptions
Registration of
PipelineOptions by an application guarantees that the
PipelineOptions is composable during execution of their
Pipeline and meets the
restrictions listed above or will fail during registration. Registration also lists the
registered
PipelineOptions when
--help is invoked via
PipelineOptionsFactory#fromArgs(String[]).
Registration can be performed by invoking
PipelineOptionsFactory#register within a
users application or via automatic registration by creating a
ServiceLoader entry and a
concrete implementation of the
PipelineOptionsRegistrar interface.
It is optional but recommended to use one of the many build time tools such as
AutoService to generate the necessary META-INF files automatically.
A list of registered options can be fetched from
PipelineOptionsFactory#getRegisteredOptions().
Serialization Of PipelineOptions
PipelineOptions is intentionally
not marked
java.io.Serializable, in order
to discourage pipeline authors from capturing
PipelineOptions at pipeline construction
time, because a pipeline may be saved as a template and run with a different set of options than
the ones it was constructed with. See
Pipeline#run(PipelineOptions).
However,
PipelineRunners require support for options to be serialized. Each property
within
PipelineOptions must be able to be serialized using Jackson's
ObjectMapperor the getter method for the property annotated with
JsonIgnore.
Jackson supports serialization of many types and supports a useful set of annotations to aid in serialization
of custom types. We point you to the public Jackson documentation when attempting to add
serialization support for your custom types. Note that
PipelineOptions relies on
Jackson's ability to automatically configure the
ObjectMapper with additional modules via
ObjectMapper#findModules().
Note: It is an error to have the same property available in multiple interfaces with only some
of them being annotated with
JsonIgnore. It is also an error to mark a setter
for a property with
JsonIgnore.