Welcome to VSTSC's Community Server Sign in | Join | Help

Report Delivery Database (SSIS and Reporting Services Web Service part 1a)

Ok time to get side tracked sort of.  In my last post I talked about inspecting the Reporting Services web service to gather information about reports and possibly even building that out to execute reports.  In order to drive the automatic execution of reports we need a manner in which to hold report names and locations, along with parameters for those reports.  In this post I will go over a database I designed to handle feeding this data to an SSIS package to execute reports.  It’s worth noting that this structure can be used to feed the data driven subscription engine in SQL Server Reporting Services Enterprise Edition.  (You will still be lacking the event driven ability).  I'm sure I'll also manage to sneak in a little of my database development and design philosophy and possible a PowerShell script or two I use to automated the build process. Ok let’s start out at the end:

 Here is the DB that we will end up with.

To do this we need to create 4 tables.  One to hold the report name and its location, one to hold Parameter Names but not values, One to hold the linkage between what reports get what parameters, and finally one to hold the actual data for an given scheduled run of a report ting this data all the way back to an individual report and parameter information.

Here are the scripts, I'll talk a little bit about each one.

Script 0000

use master

go

if @@SERVERNAME = 'YOURDEVELOPMENTDBINSTANCE!'

if exists (select * from sys.databases where name ='steves')

begin

print 'killing connections!'

declare @sqlString varchar(100)

while (exists (select spid from sys.sysprocesses s inner join sys.databases db on db.database_id=s.dbid where db.name='steves'))

begin

select @sqlString='kill '+cast(max(spid) as varchar(10)) from sys.sysprocesses s inner join sys.databases db on db.database_id=s.dbid where db.name='steves'

print @sqlString

execute(@sqlString)

end

begin

print 'I hope you know what you are doing!'

drop database steves

print 'steves database dropped!'

end

end

Ok first off you may wonder why I'm dropping the DB in this first script it’s because this is my development methodology.  I guess it’s almost test driven database development but with a few differences. I like to start from scratch testing all of my build scripts that get me to the current production version, then I like to run any of the scripts that add metadata to the system, next I run any data loading scripts or SSIS packages necessary to get the system in a active state similar to production, finally I like to add any of the new development scripts to the system. To accomplish this I normally use a combination of SubVersion, PowerShell, and SQL Server Agent.  (Maybe in a future post I'll go into my PowerShell,SubVersion, SQL Server interactions, to be honest I would like to move this in Visual Studio and Source Safe but haven't had the time to do a lot of detail looking around)  Ok long story longer, If the I'm in the right environment I kill the connections and drop the DB, then create a new DB and schema.  Otherwise this script does nothing.

Script 0001

use master

if not exists (select * from sys.databases where name ='Steves')

begin

create database steves

print 'database "steves" created!'

end

else

begin

print 'database "steves" already exists'

end

go

use steves

go

if not exists(select * from sys.schemas where name = 'RD')

begin

execute ('create schema rd authorization dbo')

print 'schema "rd" created!'

end

else

begin

print 'schema "rd" already exits!'

end

Simple enough create a new database and new schema if they don't exist. 

Script 0002

use steves

go

if not exists (select * from sys.tables where name = 'reports' and SCHEMA_NAME(schema_id)='rd')

begin

create table rd.reports

(

report_id int identity(1,1) not null

,report_name varchar(150) not null

,report_location varchar(255) not null

,constraint pk_reports_01 primary key (report_id)

)

print 'table rd.reports created!'

end

else

begin

print 'table rd.reports already exists!'

end

Ok, this script just creates a table nothing special.  It does keep up the theme of providing information back based on success of the script.  (I get enough information back from a failure, and if we have a need to handle them in a special manner we can introduce try/catch blocks)   This is the report table, the report name is a user defined name for a report it can match the actual report name but does not need to.  The report location field with hold the location off of the ReportServer/ URL to the name including the name actual rdl file.

Script 0003

use steves

go

if not exists (select * from sys.tables where name = 'parameters' and SCHEMA_NAME(schema_id)='rd')

begin

create table rd.parameters

(

param_id int identity(1,1) not null

,param_name varchar(20) not null

,param_data_type tinyint not null

,constraint pk_parameters_01 primary key (param_id)

)

print 'table rd.parameters created!'

end

else

begin

print 'table rd.parameters already exists!'

end

Once again pretty straight forward table create with a few print statements. This table holds a name for each parameter; this name should match what the name of the parameter in the reports will be.  This requires a semi high level of standardization and correlation between report writing and parameter entry into this system.   We are not locked into this and if we need to specify a different name for the same parameter we can do so at the report/parameter cross reference level but this is the going to the default name used.  You also see param_data_type which holds the data type that this parameter represents,  because all parameter data is going to be held in a varchar field it is useful to know what the data type should be.  The last thing I want to point out it deals with database standards is the primary key constraint it is named with pk to signifying the constraint is a primary key.  Followed by the table name and then the ordinal position of the column(s) that are in the primary key.  The reason I am pointing this out is that I do not believe good database design happens by chance, it takes in my opinion forethought and planning to come with a good design.

Script 0004

use steves

go

if not exists (select * from sys.tables where name = 'reportsParametersXrefs' and SCHEMA_NAME(schema_id)='rd')

begin

create table rd.reportsParametersXrefs

(

xref_id int identity(1,1) not null

,report_id int not null

,param_id int not null

,alternate_param_name varchar(40) null

,constraint pk_reportsParametersXrefs_01 primary key (xref_id)

,constraint un_reportsParametersXrefs_02_03 unique (report_id,param_id)

,constraint fk_reportsParametersXrefs_02 foreign key (report_id) references rd.reports(report_id)

,constraint fk_reportsParametersXrefs_03 foreign key (param_id) references rd.parameters(param_id)

)

print 'table rd.reportsParametersXrefs created!'

end

else

begin

print 'table rd.reportsParametersXrefs already exists!'

end

Alright now we are getting somewhere.  First I want to point out the continuation of the constraint naming, once again I am not saying this is the only way to do it, I'm just saying think about everything in your database in the planning stages it will make its use and maintenance much easier.  Ok this table basically just links reports to parameters, it also gives you a chance to rename the parameter name to one that matches the value in the report, this will be key in the overall Report Delivery Engine we are building.

Script 0005

use steves

go

if not exists (select * from sys.tables where name = 'schedules' and SCHEMA_NAME(schema_id)='rd')

begin

create table rd.schedules

(

schedule_num int not null

,xref_id int not null

,param_value varchar(255) not null

,constraint pk_schedules_01 primary key (schedule_num,xref_id)

,constraint fk_schedules_02 foreign key (xref_id) references rd.reportsParametersXrefs(xref_id)

)

print 'table rd.schedules created!'

end

else

begin

print 'table rd.schedules already exists!'

end

Finally we are ready to pair the data with the report/parameter information.  Notice how the parameter value is held in a varchar field.  This is done to make this system as flexible as possible.  I toyed with the idea of creating a parameter value column for each data type and making those fields nullable and then enforcing the business logic that requires one of the parameter values to be filled with data via a trigger but this seemed overly complex and didn't truly buy me any additional data integrity.  I can still enforce domain integrity via a trigger that looks back to the parameter table but I'm not too sure its necessary but I will keep that idea ready just in case.

Script 0006

use steves

go

create view rd.reportSchedules

as

select

r.report_name

,r.report_location

,p.param_name

,t.name

,rpx.alternate_param_name

,s.param_value

,s.schedule_num

,r.report_id

,p.param_id

,rpx.xref_id

,t.system_type_id

from

rd.reports r

inner join

rd.reportsParametersXrefs rpx

on

r.report_id=rpx.report_id

inner join

rd.parameters p

on

p.param_id=rpx.param_id

inner join

rd.schedules s

on

s.xref_id=rpx.xref_id

inner join

sys.types t

on

t.system_type_id =p.param_data_type

go

if (@@ERROR=2714)

begin

print 'view rd.reportSchedules already exists!'

end

else if exists (select * from sys.views where name = 'reportSchedules' and SCHEMA_NAME(schema_id)='rd')

print 'view rd.reportSchedules created!'

else

print 'error look above'

Finally we have a view to make all of these identity filled normalized tables provide some useful data to the end user.  This view just shows all of the value fields along with only one copy of each of the idents across all four tables.  This script is also in a different format where I create the object and wait to see what the error value is if any.  The reason for this is that the create view statement must be the only statement in its batch.  I'm not exactly sure why this is but it is what it is.

What does all this get us? Well nothing, but wasn't that a fun 30 minutes, Seriously this gives a building block, and it gives us a system to record and retrieve parameters for an multiple reports and multiple schedules for any given report.  There are few more scripts that can make this an solution for providing data to the Data Driven subscription area on Enterprise Edition of reporting services but I'll save those for another post. (Hint: 1 proc for retrieving data, 1 proc for updating key parameters like begin date and end date, 1 proc for dynamically feeding data to this set of tables prior to a report running).

 

Ok as a little bonus here is the PowerShell script I use to automate my build process.  Basically you pass it a directory and a SQL Server instance and it spins through the directory executing any script that starts with four numeric values in order from lowest to highest.  I normally schedule this to run from SQL Agent at a set time every week.

#-----------------------------------------------------------------------------
#
#SFIBICH
#7/2/08
#Version 1.0
#versionControlSQLScript.ps1
#This script looks for files in the directory and executs them in order
#based on the script having an .SQL file extension and the first for characters of the
#name being numeric.  It will execute the scripts in ascending order.
#
#-----------------------------------------------------------------------------
#
#$ars[0] - relative path to the folder
#args[1] - the name of the server that the code will be executed on.
#$a = path to this file when executed
#serverName - the name of the server this code will be executed on defaults to DEV
#
#-----------------------------------------------------------------------------
$a=$MyInvocation.InvocationName
$serverName ="localhost"
if ($args[1].length -gt 0) {
 $serverName=$args[1]
}
"ServerName:"+$serverName
if ($args[0].length -gt 0) {
 $fileLocation=$args[0]
 if ($fileLocation -eq '?') {
  get-content $a | ? {$_.length -gt 0} | ? {$_.substring(0,1) -eq '#' }
  exit
 }
 else {
  [reflection.assembly]::LoadWithPartialName("'Microsoft.VisualBasic")
  gci $fileLocation | ? {
   $_.name -match "sql" -and [Microsoft.VisualBasic.Information]::isNumeric($_.name.trim().substring(0,4)) -eq "TRUE"} |
   sort-object @{Expression={[Int](([String]$_).trim().substring(0,4))};Ascending=$true} |
   % {$_.FullName; sqlcmd -S $serverName -E -e -i $_.FullName}
 }
}else{
 write-warning('Default Directory required - no work executed')
}

 

 

SSIS and Reporting Services Web Services

This post covers accessing web services in an SQL Server 2008 SSIS package, in particular the web service provided by Reporting Services 2005 in both MSRS 2005 and MSRS 2008.  I have a need to generate reports on a schedule and/or on an event driven basis from Reporting Services based on a number of outside conditions.  I could use Reporting Services 2008 Enterprise Edition which has data driven subscriptions but this falls short in the event driven category and will run the report regardless based on a schedule only.  This post is more or less a walkthrough of the ReportService web service based on MSDN, applied to SQL Server 2008 SSIS Script task.  The information below works in both Data Flow and Control Flow Script tasks.  (Read: there is nothing earths shaking going on here but you need to start somewhere and being familiar with the basics is going to help us in the long run.)

Reporting Services exposes a number of web services classes in SQL Server 2005 and SQL Server 2008 Reporting Service.   For this post we will be looking into the ReportService2005 web service namespace.  For future reference there is a ReportExecution, ReportService2006 and ReportServiceAuthentication web service name spaces that are available for use in both 2005 and 2008 of Reporting services.

Before we can get into building our report delivery engine we need to get a feel for how to interact with the Reporting Services programmatically.  We will do this by setting up a simple SSIS package that outputs basic properties from a given report on a Reporting Services environment.  This is not as simple as dragging the web service component from the tool bar on to the control flow tab and typing in the URL of the web server.  Unfortunately the SSIS web service task is incompatible with the level of control we need to have to successfully execute our goal.

Step 1 – Drag the Script Task onto the control flow.  Note most of what I am going to show you can be done exactly the same inside a data flow task as a data source, transformation, or destination.

Step 2 – Ok, not really a step but I want to make that point that the script language drop now shows and supports C#.

 

Step 3 – Once the script is open left click on the reference section and choose Add Web Reference.  Note the namespace of this particular script task.  You will need to reference this later in the Using Statement to properly reference the web service.

Step 4 – Put in the URL of your reporting services server followed with reportservice2005.asmx?wsdl  to get the WSDL for the ReportService2005 web service class.  Then rename the web reference name to something meaningful to you, ReportService2005 is a good choice.  (and follows along with the MSDN documentation example if you need a really good reason)

WSDL URL Example: http://FQDN:8080/ReportServer_SQL2008/reportservice2005.asmx?wsdl

Note: I did have issues referencing the https version of the WSDL from my local machine inside visual studio, so I went with the http reference. 

 

Step 5 – Reference the name space by adding your script tasks name space along with name of the web service.

Step 6 – Code Explanation:

Ok here is a little detail on what we are doing first we declare a Boolean fireAgianFlag for use later in a fire Information event.  Next we declare a new instance of the ReportingService2005 class.  Once we have created an instance we set its credentials and URL properties.  The URL is set to the instance of Reporting Services running on my local machine, pointing to the reportservice2005.asmx page.  Next we create two new instances of the property class set the name properties to Description and Name and add them to an array of property objects.  The get properties method will fill any property array sent to it filling the value property of the property array with information that corresponds to the value in the name of that particular property object.  (So basically we have an array that has two objects each object is a property object.  The property object has two properties of its own name and value.  I only mentioned this because the word property is use a whole lot in the above few sentences and reads a little funny to me so I expect it may be a little confusing for everyone else.)  After the property array is created and set with the appropriate values it is used in the call to the ReportService classes’ getproperties method.  This method fills the property array with information for a given report, in this example the ConnectionInfo in the StevesReportingExamples folder of the root of the ReportServer directory.  We then loop through the property array and show the name and value of each property we have retrieved data for.

 

Step 7 – Running the package you see that I have a number of property names and there values on the right hand side outputted by the fire information event.  I have a few more than those shown in the code screen shot above because I added a number of additional property objects to the property array.  The code for the additional properties is below, its worth to note there are a few more properties specific to reports, folders, and objects that are available but not shown.

Additional valid property objects names and values:

        //Item(Generic) Properties

        Property[] properties = new Property[12];

        Property name = new Property();

        Property description = new Property();

        Property CreatedBy = new Property();

        Property CreationDate = new Property();

        Property Hidden = new Property();

        Property ID = new Property();

        Property ModifiedBy = new Property();

        Property ModifiedDate = new Property();

        Property Path = new Property();

        Property Size = new Property();

        Property Type = new Property();

        Property VirtualPath = new Property();

        properties[0] = name;

        properties[1] = description;

        properties[2] = CreatedBy;

        properties[3] = CreationDate;

        properties[4] = Hidden;

        properties[5] = ID;

        propertiesDevil = ModifiedBy;

        properties[7] = ModifiedDate;

        propertiesMusic = Path;

        properties[9] = Size;

        properties[10] = Type;

        properties[11] = VirtualPath;

 

        //Report Specific Properties

        Property[] Reportproperties = new PropertyMusic;

        Property Language = new Property();

        Property ReportProcessingTimeout = new Property();

        Property ExecutionDate = new Property();

        Property CanRunUnattended = new Property();

        Property HasParameterDefaultValues = new Property();

        Property HasDataSourceCredentials = new Property();

        Property IsSnapshotExecution = new Property();

        Property HasScheduleReadyDataSources = new Property();

        Reportproperties[0] = Language;

        Reportproperties[1] = ReportProcessingTimeout;

        Reportproperties[2] = ExecutionDate;

        Reportproperties[3] = CanRunUnattended;

        Reportproperties[4] = HasParameterDefaultValues;

        Reportproperties[5] = HasDataSourceCredentials;

        ReportpropertiesDevil = IsSnapshotExecution;

        Reportproperties[7] = HasScheduleReadyDataSources;

 

        //Folder Specific Properties

        Property[] Folderproperties = new Property[1];

        Property Reserved = new Property();

        Folderproperties[0] = Reserved;

 

        //Resource Specific Properties

        Property[] Resourceproperties = new Property[1];

        Property MimeType = new Property();

        Resourceproperties[0] = MimeType;

 

        //Item(Generic) Properties

        name.Name = "name";

        description.Name = "description";

        CreatedBy.Name = "CreatedBy";

        CreationDate.Name = "CreationDate";

        Hidden.Name = "Hidden";

        ID.Name = "ID";

        ModifiedBy.Name = "ModifiedBy";

        ModifiedDate.Name = "ModifiedDate";

        Path.Name = "Path";

        Size.Name = "Size";

        Type.Name = "Type";

        VirtualPath.Name = "VirtualPath";

 

        //Report Specific Properties

        Language.Name = "Language";

        ReportProcessingTimeout.Name = "ReportProcessingTimeOut";

        ExecutionDate.Name = "ExecutionDate";

        CanRunUnattended.Name = "CanRunUnattended";

        HasParameterDefaultValues.Name = "HasParameterDefaultValues";

        HasDataSourceCredentials.Name = "HasDataSourceCredentials";

        IsSnapshotExecution.Name = "IsSnapshotExecution";

        HasScheduleReadyDataSources.Name = "HasScheduleReadyDataSources";

 

        //Folder Specific Properties

        Reserved.Name = "Reserved";

 

        //Resource Specific Properties

        MimeType.Name = "MimeType ";

 

I hope that’s helpful and I’m pretty sure I’ll post a few more entries around this topic.

PowerShell Certification, why I think we need one.

I normally do not write opinion pieces, I feel that if I can contribute anything worthwhile to the community my efforts are best served explaining or demonstrating a technical area where I have some level of expertise.  I have however one subject that I have been mulling over for the past few days that is an exception to this rule, that is certification.  Not any certification but a PowerShell certification, this however does not exist.  I think we (the community) need a PowerShell certification for a number of reasons (the following are my personal reasons, which I hope some of you share). 

·         First, trying to obtain a certification drives the individual to a certain competency in a given product or area.  

·         Second, it gives the individual an idea of what given competency level in a particular product or area is and what is required to have that competency. 

·         Third and simplest reason,  it’s a goal that one can drive towards. 

I have taken a number of different certifications in my career and I find the process of preparing for the certification the most valuable part of the certification process.  This is not to say that certification doesn’t help getting you a job, certification at the least has gotten my foot in the door at one employer.  Employer’s look to certification as a sign post on an employee’s career as saying, yes I have interest in this subject, I have dedicate some of my time outside of my job to show you what I can do.  This does not mean if you take a certification you are necessary an expert, or people who  have certifications are better than those IT professional that do not, but it is a sign of sorts of dedication or interest to an area going beyond the standard 9-5 (or 7:45-5, plus off hours support and project go-live/implementation weekends J  ).  It is does offer a potential employer something to differentiate two otherwise equally qualified individuals for the same position.

 So back to the original point, we (or at least think the community and if nothing else I would like) need a PowerShell certification.    Let me give you a little history as to why I am asking for this certification to exist.   I am not a complete newbie to PowerShell, I’ve been using it for about a year or so in production environments, and I’ve read a few books one in particular Professional Windows PowerShell (a Wrox book by Andrew Watt)  which was the first book I read and  was great and helped me come up to speed.  Recently I started to read what I am beginning to consider the bible for PowerShell, Windows PowerShell in Action by  Bruce Payette.  This is a fantastic book, but it has also made me feel like I know little to nothing about PowerShell.  That is why I am asking for a PowerShell certification; until I started reading Bruce Payette’s book I thought that I had a pretty good handle on PowerShell.  Yes, there was a few things I didn’t’ think I knew, but over all I thought I had this PowerShell thing down.  Now I feel like a newbie in this area and I have to go back and re-examine everything I’ve written, thinking how poor the implementation is.  This is a good thing, though it is a little humbling of an experience.  (I have some examples of scripts that should be re-evaluated on this site, but I’m going to leave them up as hopefully I’ll be able to look back and see my skills improve over time, and if nothing else they work)  For a while now I have known of the 3 different PowerShell Version 2 CTPs and its upcoming release in Windows7/Windows 2k8 R2.  Last night however I found a post (from May 1st 2008) from the PowerShell Team detailing some of the new upcoming features in PowerShell V2.  That got me thinking, I just barely have a handle on V1, V2 already has three CTP’s out and I have no idea what is going on in the newest release, I need something to help guide me or at least something I can use as a sign post to make sure I’m on the right track with PowerShell V1 so I have a chance of hanging on to V2.

If you think about it a PowerShell certification is not straying too far outside certifications that are already offered by Microsoft.  Currently Microsoft offers certifications in Operating Systems, Server Application, and different parts of the .Net FrameWork.  So in closing if anyone who reads this blog has any interest in a PowerShell certification show your support by putting in a comment or sending me an email.  I would like to know if I am a lone wolf on this one.    If anyone one who reads this blog has any influence over certifications tracks at Microsoft please do what you can to make this certification a reality.

Posted by steve | 1 Comments

PrimeFinder

Alright everyone has written the obligatory "Hello World!" program when they start to learn a new language.  I have my own variety of that practice and its PrimeFinder.  Whenever I start to learn a new language I write a program that finds primes.  I know this may sound a little strange but this normally allows me to practice a number of new skills and techniques as well as see if there is a way I can improve both my prime number finding logic and find out if the new language has anything to offer that is unique for this task.   In this post I'll examine PrimeFinder written in PowerShell, C#, and T-SQL.  I will discuss each and the surprises I found out about each.   Feel free to take this as a challenge to find more efficient ways in these languages or any others; I would like to see any different solutions that are more efficient.

There are some rules though, here they are:

1.     The solution must be robust enough to find all primes ranging from 1 to 2,147,483,647 (signed 4 byte int limit)

2.     The solution must output the prime numbers in some manner which is recordable. (printing to standard out is fine since that can be pushed into a text file)

3.     The solution must restartable, or easily changed to a state that allows it to be restartable.  In that it can produce a certain number of primes, stopped and pick up where it left off.  (this is the reason for the requirement #2, very flexible on this requirment)

4.     The solution must be timed

5.     The solution should find the first 10000 primes by default when executed

6.     The solution is allowed three primary constants of 1, 2, and 3.  These are allowed to remove any single use code that would be in place to find these 3 primes.

The first solution was written in C# and is the reason that I was looking into the problem to begin with.