-
Notifications
You must be signed in to change notification settings - Fork 356
Creating custom input and output bindings
This describes writing binding extensions for WebJobs SDK. This focuses on non-triggering attributes.
See https://github.com/Azure/azure-webjobs-sdk-extensions/wiki for details on the binding process.
[for reference, here is older documentation about the rule system which needs some updates: https://github.com/Azure/azure-webjobs-sdk-extensions/wiki/Rule-based-Binding-Providers ]
The pattern for writing extension is:
- declare an attribute.
- pick which "binding rules" are supported
- add some converters to make the rules more expressive
The extensibility story takes a heavily declarative approach.
For each external service (like Azure Blobs, Event Hub, Dropbox, etc), the 'native sdk' refers to that resources specific SDK. This may then expose 'native types'. For example, Azure Queues has a native sdk in the WindowsAzure.Storage nuget package, and exposes native types like CloudQueueMessage
A WebJobs binding should both:
- expose bindings to the native types to give the consumer full control
- expose bindings to BCL types (like System, byte[], stream) or POCOs so that the user can access the service without requiring the service's native sdk.
- expose bindings to JObject and JArray so the binding is more consumable from non .NET languages that can't consume the .NET type system.
See https://github.com/Azure/azure-webjobs-sdk-extensions/wiki/Binding-Attributes for details.
Define an attribute. See an example of [BlobAttribute] at https://github.com/Azure/azure-webjobs-sdk/blob/dev/src/Microsoft.Azure.WebJobs/BlobAttribute.cs
Your attribute must have a [Binding] attribute on it.
If the attribute refers to an Appsetting name and not the actual value, then you can place an [AppSetting] attribute on it. This is a common pattern for connection strings and secrets.
public sealed class EventHubAttribute : Attribute, IConnectionProvider
{
// Other properties ...
[AppSetting]
public string Connection { get; set; }
}
You can put a [AutoResolve] attribute on a property to make it automatically support:
-
%appsettting%-these are resolved by an INameResolver at startup. This provides configuration options (like switching between dev and test environments) and also a means to get secrets like connection strings. -
{key}values - these are resolved per-trigger instance with trigger data and thus can be runtime data.
For example, Blob path supports autoresolve semantics, so you could use it like:
class Payload { public string name {get;set; } }
void Foo([QueueTrigger] Payload msg, [Blob("%container%/{name}")] TextReader reader) {}
%container% is resolved at startup based on the appsettings.
Foo is triggered on a queue message, and the trigger provides a runtime value for name based on the queue payload.
The SDK will do the substitutions for your extension. If the container appsetting was storagetest and the queue received a message with name = 'bob', then the blob path would be invoked with 'storagetest/bob'
In the examples below, we'll use an "bind to excel table" example with this attribute:
public class ExcelTableAttribute : Attribute
{
public ExcelTableAttribute(string fileName, string tableName)
{
this.Filename = fileName;
this.TableName = tableName;
}
public string Filename { get; private set; }
[AutoResolve]
public string TableName { get; private set; }
}
You can use the standard System.ComponentModel.DataAnnotations attributes on your attribute properties, such as RegularExpressionAttribute, to apply validation rules to your new attribute. The validation is run as early as possible. If there are no { } tokens, then validation is run at index time. If there are { } tokens, then validation is done at runtime after the [AutoResolve] substitution.
An extension is an instance of IExtensionConfigProvider. See https://github.com/Azure/azure-webjobs-sdk-extensions/wiki/Extension-Registration for details
The key part of this is implementing the public void Initialize(ExtensionConfigContext context) method.
Binding rules provide a few common patterns with strong semantics. An extension describes which rules it supports, and then the SDK picks the appropriate rule based on the target functions' signature.
The common rules are:
- BindToInput - bind to a specific object.
- BindToCollector - bind to an IAsyncCollector, useful for outputting discrete messages like Queues and EventHub,
- BindToStream - bind to stream based systems. This is useful for blob, file, dropbox, ftp, etc. (Not yet implemented see https://github.com/Azure/azure-webjobs-sdk/issues/1001 )
Binding rules are exposed via helper methods on JobHostConfiguration.BindingFactory
The rules commonly refer to a builder object that implements an IConverter<TAttribute, TObject> interface that can be used to instantiate a core object based on an attribute.
Use this rule to bind to a single input type like CloudTable, CloudQueue, etc.
IConverter<TAttribute, TObject> builder = ...; // builder object to create a TObject instance
bf.BindToInput<TAttribute, TObject>(builder);
Where TAttribute is the attribute type this rule is for, and TType is the target parameter type in the user's function signature we're enabling.
Here is an example of registering a basic input rule.
public class ExcelBindingProvider : IExtensionConfigProvider,
IConverter<ExcelTableAttribute, DataRectangle>
{
// converter object to instantiate the table
public DataRectangle Convert(ExcelTableAttribute input)
{
// Call into Excel SDK to read the table.
var dt = Class1.GetTableContents(input.Filename, input.TableName);
return dt;
}
// callback invoked by SDK to initialize the extension
public void Initialize(ExtensionConfigContext context)
{
var rule = context.AddBindingRule<ExcelTableAttribute>();
IConverter<ExcelTableAttribute, DataRectangle> builder = this;
rule.BindToInput<DataRectangle>(builder);
}
}
Where DataRectangle is a native type representing the raw excel table contents (think String[][]).
And that can enable binding to a user function like this:
public static void Direct(
[QueueTrigger] Payload msg,
[ExcelTable(@"c:\temp\test.xlsx", "{tableName}")] DataRectangle contents)
{
}
Use this rule for binding to things that emit discrete output messages, like queues.
For example, the [Table] attribute supports binding to IAsyncCollector via:
var rule = context.AddBindingRule<TableAttribute>();
rule.BindToCollector<ITableEntity>(builder);
A single BindToCollector lights up multiple patterns:
| User Parameter Type | Becomes |
|---|---|
| IAsyncCollector x | identity |
| ICollector x | sync wrapper around IAsyncCollector |
| out T item | ICollector collector; collector.Add(item) ; |
| out T[] array | ICollector collector; foreach(var item in array) collector.Add(item); |
This will also automatically apply converters registered with IConverterManager from the user's parameter type.
Todo: https://github.com/Azure/azure-webjobs-sdk/issues/1001
Suppose you use BindToCollector to support IAsyncCollector. You could then add a AlphaType-->BetaType converter, and the SDK can now also bind to an IAsyncCollector.
Here's an example of registering a DataRectangle-->JArray converter for the ExcelTableAttribute
public void Initialize(ExtensionConfigContext context)
{
var rule = context.AddBindingRule<ExcelTableAttribute>();
rule.AddConverter<DataRectangle, JArray>(new JArrayConverter());
...
}
class JArrayConverter : IConverter<DataRectangle, JArray>
{
public JArray Convert(DataRectangle input)
{
return input.ToJarray();
}
}
This converter would now enable user functions to have a JArray type, like this:
[ExcelTable(@"c:\temp\test.xlsx", "{x}")] JArray contents)
Sometimes your binding doesn't know the parameter type in the user's function.
For example, [Table] supports binding to IQueryable . This is accomplished via adding a CloudTable --> IQueryable<T> converter.
The SDK has a sentinel type, called OpenType, which can be a placeholder for a generic T. This is because the Extenion's Initialize method is not generic and so can't directly refer to a T in its method body.
For example, you could register a generic DataRectangle-->T[] converter via:
cm.AddConverter<DataRectangle, OpenType[], ExcelTableAttribute>(typeof(PocoArrayBuilder<>));
Where the builder is generic. The SDK will pattern match and figure out T and instantiate the builder.
// Convert from a DataTable to a T[]
class PocoArrayBuilder<T> : IConverter<DataRectangle, T[]>
where T :new()
{
public T[] Convert(DataRectangle input)
{
var array = input.ToJarray();
var final = array.ToObject<T[]>(); // uses T!
return final;
}
}
This would now enable a user function signature like this:
// Arbitrary Poco, no special base class
public class MyPoco
{
public string Id { get; set; }
public string Name { get; set; }
public string Score { get; set; }
}
public static void Test(
[ExcelTable(@"c:\temp\test.xlsx", "Scores")] MyPoco[] contents)
{ ... }
OpenType is a base class with an IsMatch(Type t) method. In general, the SDK does pattern matching for to enable things like OpenType[] and ISomeInterface<OpenType>.
If you need constraints (like only match to types that have an "Id" property), you could derive from OpenType and override IsMatch to implement your constraint.
See the tests for more examples of conversions: https://github.com/Azure/azure-webjobs-sdk/blob/dev/test/Microsoft.Azure.WebJobs.Host.UnitTests/ConverterManagerTests.cs
The converter manager is centralized and shared across extensions. That means that you can add extensions to an existing binding. For example, you could extend the existing [Table] binding by adding a CloudTable --> MyCustomObject converter.
(Blob still needs to hook into this. See https://github.com/Azure/azure-webjobs-sdk/issues/995 )
The converter manager allows some implicit conversions. For example, if TDest is assignable from TSrc, it will provide an implicit conversion.
You can also use OpenType with binding rules. For example, if you want to support direct binding to generic parameters with an intermediate converter.
To use WebJobs extensions in script, they should support bindings to JObject and JArray. The Script runtime can then marshal those types to other languages like Node and Python.