I've been back at work on StreamInsight lately, and looking at using the Windows service-based host rather than run a self-contained .exe. Since I'd also been using the console for output, I was looking for a way to run the service as a console app. Turns out this is easy. StreamInsightHost.exe takes two optional command line parameters: -console and -config. You can guess what -console is for; -config specifies an alternate location for the config file. This parameter can use a relative filesystem path, but needs to specify the entire name of the config file (ie you can't just point it at the subdirectory that corresponds to your instance name).

So, if I'm trying to start the "MyInstance1" service equivalent, using the config file that the Windows service uses, these two lines will do it:

cd "C:\Program Files\Microsoft StreamInsight 1.0\Host"
.\StreamInsightHost -console -config MSSI.MyInstance1\StreamInsightHost.exe.config

This posting is a journey through StreamInsight 1.0 APIs, undertaking by porting the CTP3 samples. Part 3.

In the StreamInsight CTPs, there existed three methods to code applications. My original post describes the current state of Observable. That leaves us with "Explicit Server" and "Implicit Server". Coding using the "Implicit Server" method,  you'd make (basically) two main calls. You'd create a CepStream with input provider information. Then use stream.ToQuery(...) specifying output provider information. This call would not only bind the query, but create a Server and Application object under the covers. Since (I'm not sure this is the reasoning) your Server object must specify the correct StreamInsight instance you named at install time, this would be more difficult to abstract.

So....even using the CepStream.Create() and ToQuery method for binding a query to a query template, you still must instanciate your own Server and Application instance. That's what the error message was telling me, unless I missed something obvious. So,

In ImplicitServer.cs:

1. Added these lines at the beginning. Remember "default" is my StreamInsight instance name from installation. Use your own instance name.

Server server = Server.Create("default");
Application application = server.CreateApplication("ImplicitServer");

2. Pass the application instance into the ConsumeQuery method.

3. In ConsumeQuery, use the overload of ToQuery(...) that specifies the application as the first parameter. Just like the error message said to.

And voila, ImplicitServer works too! To conclude this series, I need to also address the SQLApp example, the one that uses the StreamInsight sample input and output providers over SQL Server. This example (in SQLApp.cs) uses the "implicit server" method of coding query binding, so you'll need to make analogous changes like you did to ImplicitServer.cs (above).

And you have working example code with StreamInsight 1.0 RTM. Now I'm "caught up", time to look at the new stuff. And (maybe) determine how to use Reactive Framework for Observable providers and how this all fits together. Maybe starting with my good friend Bart De Smet's blog for that.

Cheers.

This posting is a journey through StreamInsight 1.0 APIs, undertaking by porting the CTP3 samples. Part 2.

So let's update the SimulatedDataInputFactory provider, now. The problem seems to be the replacement of the interface that allows you to configure the CTI frequency. Sure enough, ITypedDeclareAdvanceTimePolicy has been replaced by ITypedDeclareAdvanceTimeProperties.

In SimulatedDataInputFactory.cs

1. Change the interface declaration to the new name (above) ITypedDeclareAdvanceTimeProperties.

2. This interface has one method that now returns a AdapterAdvanceTimeSettings instead of an AdvanceTimeSetting. So change the code declaration at the end of the module to AdapterAdvanceTimeSettings. Now, to make an AdapterAdvanceTimeSettings....

3. Change the code in the DeclareAdvanceTimePolicy to the following:

return new AdapterAdvanceTimeSettings(
           new AdvanceTimeGenerationSettings(configInfo.CtiFrequency, TimeSpan.FromSeconds(0)),
           AdvanceTimePolicy.Drop);

It's the same information we provided before, just refactored a bit. So far, so good. if you've made all the changes so far, including the multiple Snapshot() -> SnapshotWindow(...) instances. The code should compile. Let me know of there's any changes I've missed writing about; my code compiles at this point.

First, to test the ExplicitServer sample. It runs fine....OK! Now the ImplicitServer sample. That's a different story. Although it compiled, running it produces the following error on the Stream.ToQuery(...) statement.

System.InvalidOperationException: The query application must be specified explicitly because none of query inputs are bound to queries. Use an overload that explicitly provides the target application for the query.

That's the subject for the next blog post...

This posting is a journey through StreamInsight 1.0 APIs, undertaking by porting the CTP3 samples.

The first look at the errors produced by recompiling the CTP3 examples is the realization that the IObservable model for coding adapters doesn't exist in the RTM version. This is quickly confirmed with a look at a StreamInsight forum thread; it may return in future. Meanwhile, use the reactive framework. So I removed the Observable and PatternDetector projects for the samples solution.

Concentrated on making the projects compile, first off. So, here's a list of the code that had to be changed for that purpose.

In ExplicitServer.cs

1. Server.Create() now requires an instance name. That's the instance you named when you installed the project. In my case, that's "default". Note that this doesn't mean you're using the service, you'd still need Server.Connect for that.

2. CreateQueryTemplate needs two extra arguments, template name and template description. Call them whatever you want and describe them as you'd like.

3. application.CreateQuery has the arguments in a different order (at least in the 3-arg version the samples use). It's now, 'name, description, and query binder instance. There's a few of these in other samples.

4. The Snapshot() method in LINQ queries is replaced by SnapshotWindow(SnapshotWinderOutputPolicy.Clip). Clip is the only policy supported, so there's no real choice here. There's a few uses the Snapshot() in the samples.

5. While we're on Windowing code in LINQ queries, the TumblingWindow constructor has one new required parameter (in UserAggregateQuery.cs), that's HoppingWindowOutputPolicy.ClipToWindowEnd. Again, ClipToWindowEnd is the only policy currently supported.

Now, although most of the adapters seem to build just fine, as is, there's a little problem with the SimulatedDataInputFactory for that set of providers. That's covered in the next blog entry.

StreamInsight V1.0 RTM'd a few weeks ago. Although I worked with it, looking especially at the infrastructure and programming models, I just installed the RTM version a few days ago.

One thing I noticed immediately upon install is that it requires a license key. This is because there are different versions of StreamInsight based upon which SQL Server SKU you'd installed. See the licensing explanation, versioning, and SKUs information here. Because I hadn't yet installed SQL Server 2008 R2, I chose to enter no license key, which meant that I installed the 180-day trial version. Now that SQL Server 2008 R2 is available via MSDN, I'll need to revisit this.

Next, I was prompted for a StreamInsight instance name. Being the SQL Server and PowerShell user I am, and having no imagination, I chose the name "default" for my default instance. After installation, I noticed a shiny new StreamInsight (default) Service listed under Windows Services. This service is meant to run the standalone StreamInsight host, if you chose to run a separate host rather than imbed StreamInsight in your application. Much easier than starting up the service from the command line.

There's some neat new features since CTP3, namely support for composing queries at runtime, count windows, and left-anti-semi joins in the LINQ provider. Check the updated documentation. Those will be interesting to look at, but first I wanted to get my bearings by looking at the samples. Interestingly, no samples come with the RTM. Although there will be a codeplex project devoted to them, it's not active yet. Time to recompile the CTP3 samples.

Although there's a few changes noted to the APIs in the documentation, there are many more than documented. A naive 'recompile against the 1.0 libraries' netted 30 or so compile errors. Time to "up-port" the existing samples. This turns out to be a process that yields a lot of insight into the programming model of the 1.0 product, and I'll discuss this in the next few blog posts. I have the CTP3 samples working, but can't post the code, as it's not my code to post (ie, I don't "own" the code). So I'll describe the process.

In the last installment about the object model, let's the at the third development style, using .NET's IObservable/IObserver. IObservable/IObserver is a new interface in .NET 4.0, the "mirror image" (to use a less controversial term) of IEnumerable/IEnumerator. A really simplistic way of thinking about it is the Enumerable/Enumarator is allowing you to pull, IObservable/IObserver is watching someone push. So supporting IObservable/IObserver makes it possible to use StreamInsight with .NET classes.

What they've done in this case is write a generic StreamInsight adapter set over the interfaces. It's called ObservableXXXInputAdapter and ObservableXXXOutputAdapter, where XXX stands for the three StreamInsight event shapes (Point, Edge, and Interval). So there's six classes plus a Factories and Config class for each adapter. Similar in concept to some of the generic text providers they use in the samples, but shipped in a supported library.

To encapsulate the rest of the model there are some ExtensionMethods for the .NET classes that implement IObservables (and a hook for IEnumerables). The main ones are ToObservable and ToCepStream.  They work with other extension methods and helper classes to provide for a generic transformation of .NET classes that implement the right interfaces to StreamInsight objects from the familiar Object Model. These extension methods create the (same) StreamInsight object model along the way. Although there is one method currently that may allow you to inject/use your own Server instance, therefore, being able to navigate the whole Object Model.

So at the end of the day, you work with IObservables/IObservers (and maybe IEnumerables/IEnumerators). And the same object model. For a code-assisted review of the main components of the object model, you may now return the the Explicit Server sample. Or for a data-assisted review, a CE database that's been "used" to register CEP objects. ;-)

Now that you know the StreamInsight metadata (see inside the metadata), its fairly easy to take apart the implicit model because you already know which objects are being created for you. In the implicit development model, you:

1. Create a CepStream for input - specifying
   EventType as a template parameter
   Event Shape (Point, Edge, or Interval)
   Factory class of the input adapter
   Config info for the adapter
2. Build your LINQ query over the stream
3. Call Stream.ToQuery on input stream - specifying
   Factory class of output adapter
   Output configuration
   Output event shape
   A stream event order

When calling Stream.ToQuery in step 3, the internals have all of the information to build the objects you need. That one call:
   Implicitly creates Server and Application
   Registers and configures Adapters, QueryTemplate (from step 2), EventType info.
   Binds the adapters to the QueryTemplate producing the Query.

All you need to do is start the query.  Just to prove that the entire object model is in place, you can use the references to navigate from the Query instance back to the Server, and retrieve the diagnostic information, just like in the Explicit Model. Using the ImplicitServer sample...

Query query = ConsumeQuery(top);  // step 3 - see above, now add the following code.

//Now, get the Server
Server server = query.Application.Server;

At the end of the run, you can use your server instance to get the diagnostic views. Cribbing a little code from the ExplicitServer...

RetrieveDiagnostics(server.GetDiagnosticView(new Uri("cep:/Server/EventManager")), tracer);
RetrieveDiagnostics(server.GetDiagnosticView(new Uri("cep:/Server/PlanManager")), tracer);

// View for your query from Stream.ToQuery
RetrieveDiagnostics(server.GetDiagnosticView(query.Name), tracer);

The main differences between this model and the explicit server model are:
  Because you're not instanciating the server, you don't specify server options or use Server.Connect
  Because you didn't name some of the objects (because you didn't register them yourself), it would be more difficult to reuse them in a robust manner. If you're going to go the trouble of trundling through collections or depending on registration order (if this would work), you might as well use the explicit model. If you look at the query.Name, for example, you can see that the implicit model has made the application name "default" and the query name "DefaultQuery".

But its still the same object model.

The StreamInsight object model seems fairly complex at first. There's a series of choices as you progress down the development path which make things appear more complex than they are. One thing to keep in mind is, no matter how you populate it, there is only one object model that encapsulates the metadata.

One way to get a handle of the model is to look into the metadata. The easiest way to do this is to use SQL Server CE to register your metadata. If you don't specify SQL Server CE at runtime, the default behavior is to store the information in memory, making it more difficult to see.

Note that the included standalone server host application, StreamInsight.exe, can use either SQL Server CE or inprocess metadata based on the presence of a configuration file setting.

To use SQL Server CE, use the Explicit Server Development Model (StreamInsight Books Online has nice information comparing and contrasting the development models), but instead of using:

Server s = Server.Create();

these additional lines will use SQL Server CE.

SqlCeMetadataProviderConfiguration config = new SqlCeMetadataProviderConfiguration();
config.DataSource = "MyMetabase.sdf";  // these should be in a config file
config.CreateDataSourceIfMissing = true;
Server s = Server.Create(config);

When you register the rest of the StreamInsight objects (Application, EventTypes, Adapters, etc), the metadata will be written into the CE database. Because the database information does not disappear when the application completes (unless you specifically delete it), you can browse the StreamInsight metadata afterwards.

You can also go a step further than that. Because the metadata is stored in the database, its possible to pre-provision the database with the registration information by running the program once. With a pre-provisioned database your program would consist of creating the Server (as in the code above) and pulling out the configuration like this:

Application a = s.Applications["TheApplication"]; // the Application name from original run
if (a == null)
  throw new Exception("Application Metadata Not Found");

It's possible to run your entire application this way, because all of the metadata you provided has been stored with full fidelity in the CE database. If you run, say, the sample ExplicitServer app this way, you need only fetch the Application and Query objects and start your Query. The other objects (Adapters, EventTypes, Bindings, Streams, and QueryTemplate) are already in place from the database where you populated the metadata originally.