Quantcast
Channel: SSIS Team Blog
Viewing all 132 articles
Browse latest View live

Changes to the Execute Package Task

0
0

Todays post is from Terri Chen, a developer on the SQL Server Integration Services team.

--------------------------

SSIS has introduced new concepts in SQL Server “Denali”, such as the project deployment mode, projects and parameters. We have extended Execute Package Task accordingly to best use parameters in child packages.

To pass configurations for child packages used to be quite some work and our customers used to complain a lot about it. With the new functionalities of Execute package Task, the experience with passing parameters for child packages is much simplified.

On opening the editor of Execute Package Task, you will find a new property, named ReferenceType on the “Package” page. If you choose “External Reference”, it will show the familiar SQL Server 2008 options of SQL Server or File System. All the scenarios are back compatible. The “Project Reference” is the new reference type.

clip_image002

After choosing, “Project Reference” as the reference type, you will see PackageNameFromProjectReference property with options of all package names in the project. Choose the targeted child package name. In the example, the child package is “ChildPacakge.dtsx”.

clip_image004

The child package in the example is a simple package with a single Execute Process Task. The value of RerquiredFullPathName property of the Execute Process Task is false. The executable expression uses a string type parameter called “ExcutableName”. The value of the parameter is “notepad”. By executing the child package itself, it invokes process “notepad”.

clip_image006

The “Parameter bindings” page is a new page introduced in In SQL Server “Denali”. It is used to bind a variable or a parameter of the parent package or a project level parameter with a parameter of the child package. During execution, the binding parameter or variable value is assigned to the parameter of the child package. In the example, the parent package has a string type parameter “ParentParameter” whose value is “calculator”. By clicking on “Add” button, a binding can be created as following. By executing the parent package, the value “calculator” is passed to the parameter “ExecutableName” of the child package and the child package will invoke process “calculator” instead of “notepad”.

clip_image008

ProjectReference is design for the new project deployment mode projects and only supports child packages in the same project as the parent package. If you want to reference a package in another project, you have to use External Reference and reference the package by its path on the file system or on the SQL Server. With external reference, you can only use configurations for child packages. Controls on “Parameter bindings” page is disabled when using external reference.


Project Connection Managers

0
0

Today’s post is by Sergio Clemente Filho – a developer on the SQL Server Integration Services team.

--------------------------------------

 

Simple Project Connection example in BIDS

One of the first new things you will notice in the solution explorer when you create a new SSIS project (opening existing SQL Server 2008 R2 or previous versions will not show this node, unless you convert the project to “Project Deployment Model”) is the “Connection Managers” node (See Figure 1). This is a new feature in Denali that allows sharing connection managers across multiple packages.

clip_image001

Figure 1 - Solution explorer

To create a project connection manager, right click on the “Connection Managers” node and click on the “New Connection Manager” option (as seen in Figure 2).

clip_image002

Figure 2 - Creating new project connection manager

This will prompt an existing familiar dialog to choose the connection manager type, then the connection manager information as it can be shown in figures Figure 3 and Figure 4 respectively.

clip_image003

Figure 3 - Select connection manager type

clip_image005

Figure 4 - Configuring connection manager

After the project connection manager is created, it will automatically appear in both solution explorer and connection manager list view as it can be shown on Figure 5. Currently project connection managers are being shown in bold but this might change before RTM.

clip_image007

Figure 5 - After creation

Once the project connection manager is created, it becomes available for being used similar to how package connection managers are used. An example is given below with an Execute SQL Task in Figure 6:

clip_image009

Figure 6 - Using project connection managers in SQL Task

The package should successfully run as shown in Figure 7 .

clip_image010

Figure 7 - Running in BIDS

 

Promoting and Demoting project connection managers

Project connection managers can me demoted to package connection managers as can be shown below in Figure 8. Once a project connection manager gets demoted all other packages that use this project connection will have their reference broken.

clip_image011

Figure 8 - Demoting a project connection

You can also promote a package connection back to a project connection manager by right clicking on the package connection and choosing the option “Convert to Project Connection”

 

Note: Is worth noting that all operations on project connection managers do not participate in the undo transaction. This is true for creation, deletion, editing, promotion and demotion of project connection managers. This is unfortunately a by design behavior because undo cannot span across different documents.

 

Creating project connection managers programmatically

Let’s now see how to use project connection managers programmatically. Table 1 shows the code to create a project connection manager and access the newly created connection from the package Connections collection.

· Line 8: Creates a project

· Line 9: Creates an OLEDB project connection with the stream name “Connection.conmgr”. The two arguments of the ConnectionManagerItems.Add are explained below:

o Creation name: The connection type of the connection manager, examples are: ADO, ADO.NET, FILE, FLATFILE, HTTP, etc. This is the identical creation name used in Connections.Add (http://msdn.microsoft.com/en-us/library/microsoft.sqlserver.dts.runtime.connections.add.aspx)

o Stream name: An unique file name that ends with the suffix “.conmgr”. The name cannot have more than 128 characters.

· Line 10: Sets the name of the underlying runtime object. cmi.ConnectionManager is a reference to a ConnectionManager object (http://msdn.microsoft.com/en-us/library/microsoft.sqlserver.dts.runtime.connectionmanager.aspx)

· Line 12-14: Creates a package and adds to the project

· Line 16: Accesses the project connection manager from the package connections. One thing worth noticing is that the project connection managers will automatically appear in the Package.Connections connections. This is why it automatically appeared in the existing UIs without any effort.

Table 1 - Creating SCM programmatically

  1. using System;
  2. using Microsoft.SqlServer.Dts.Runtime;
  3. namespace SCMSample {
  4. class Program {
  5. static void Main() {
  6. using (Project proj = Project.CreateProject()) {
  7.             ConnectionManagerItem cmi = proj.ConnectionManagerItems.Add("OLEDB", "Connection.conmgr");
  8.             cmi.ConnectionManager.Name = "HelloWorldSCM";
  9.             Package package = new Package();
  10.             proj.PackageItems.Add(package, "Package.dtsx");
  11.             Console.WriteLine(package.Connections[0].Name);
  12. }
  13. }
  14. }
  15. }

Note: There is one caveat though, if a package contains the same name as the project connection, the package connection will take precedence over the project connection (Similar to in a programming language when a local variable contains the same name as an attribute). You can make the project connection visible by either renaming either connection, or by deleting the package connection.

 

More advanced example of project connection managers

One important concept of project connection managers is that the same object is shared across all packages. This allows caching the information and reuse in multiple packages which will improve performance. For the next example I will quickly show how a cache connection manager can be used to share information across two child packages.

Imagine I have the following parent package as it can be seen in Figure 9:

- Contains a data flow that populates a cache connection manager that is at project scope.

- Executes two child packages (Child1, Child2)

clip_image013

Figure 9 - Parent Package

The data flow of the parent it’s pretty straightforward and it’s shown in Figure 10. The OLE DB Source retrieves all columns from the table Person from AdventureWorks database and the cache connection managers will contain all columns and will index FirstName and LastName with indexes 1 and 2 respectively.

clip_image014

Figure 10 - Data flow that populates the cache

clip_image016

Figure 11 - Cache connection manager

Once this is done, the child packages can reference the project connection manager named “Shared CCM” and use them.

clip_image017

Figure 12 - Child package 1

In the lookup transform, make sure to select “Cache connection manager” as the connection type as it can be seen in Figure 13 and select the connection “Shared CCM” manager in the “Connections” tab as it can be seen in Figure 14.

clip_image019

Figure 13 - Cache connection manager UI 1

clip_image021

Figure 14 - Cache connection manager UI 2

 

Summary

Hope that was a useful overview of project connection managers, we saw how to create the project connection manager from BIDS and from API. We saw that project connection managers will show automatically in existing UIs (Unless the name collide with a package connection) so you can use them as it was a normal connection manager. We also saw a more advanced example where the project connection manager was used in order to fetch the information only once through the cache connection manager.

 

Known issues

· Expressions are not supported on project connection managers. BIDS will hide the expressions option in the property grid.

· Logging might not always work. There might be scenarios where if you log on a project connection manager the logging won’t appear in the package logging

· If you try to click on “Parse Query” on SQLTask you will get a “Specified cast is not valid”

These existing issues should be addressed before RTM

Offline Connection Managers

0
0

Today’s post is from Carla Sabotta, lead writer for the SQL Server Integration Services team.

-------------------------------------------------------------------

The SSIS Designer in BIDS tries to connect to each data source that is used by your package to validate the metadata associated with sources and destinations. There may be times when you can’t connect to your data sources or the connection is slow. This slows validation and prolongs a package startup or opening a package data flow.

In previous versions of SSIS, you worked around this issue by setting the entire package to work offline. In SQL Server “Denali”, you can turn off the slow and unavailable connections by setting the connection managers to work offline. You configure a connection manager to work offline, by right-clicking it in the Connection Managers area of SSIS Designer and then clicking Work Offline.

clip_image002

When you set a connection manager to work offline, it will remain offline while the project is open. When you close the project, the connection manager is reset to work online.

For more information about troubleshooting package validation, see Troubleshooting Tools for Package Development in Books Online.

Managing SSIS Projects through SSMS

0
0

Before you can deploy your Projects to the SSIS Server, you’ll need to create a new SSIS Catalog through SSMS.

Connect to your SQL Server instance (local or remote), right click on the Integration Services node, and select Create Catalog. This will launch the catalog creation dialog.

image

A lot of the SSIS server functionality is based on SQL CLR, so enabling CLR integration is required to create the SSIS Catalog. You are also prompted for a password which is used during the creation of the database master key (the catalog uses standard encryption to protect sensitive values in parameters and packages – please see the Encryption Hierarchy post in BOL for more information).

When the second checkbox – “Enable automatic execution of Integration Services stored procedures at SQL Server startup” – is checked, an SSIS cleanup job will be executed each time the server instance is restarted. This cleanup job will fix the status of any packages that were running at the time the instance went down.

Once the catalog is created, you will see a new user database (SSISDB), and an SSISDB node under the Integration Services node in the SSMS Object Explorer. From there, you can create Folders while you public your SSIS project to.

image

Right clicking on a Project gives you a number of management options:

image

Configure

This UI allows you to set default values for project and package parameters, and connection managers. The scope drop down lets you filter the list of parameters down to Entry Point packages (which is a flag that can be set on the package when it is created in BIDS), or down to individual packages in the project. The references tab allows you to associate this project with server environments defined on this server.

image

This option is also available when you right click on a single package – the only difference is the list of parameters will automatically filtered to the current package.

Validate

Allows you to validate packages on the server. The option appears both at the Project level, and from the right-click menu of individual packages. When launched from the Project level, all packages in the project will be validated.

Move

This UI allows you to move the Project to a new folder in the SSIS Catalog. Note, you can use the SSIS Deployment Wizard to move projects between servers.

Versions

The SSIS Catalog allows you to roll back to previous versions of the project.

image

The server will maintain 10 versions of your project by default, but you can configure this setting on the Catalog Properties page.

image

Running Packages

There are three main ways to run packages stored in the SSIS Catalog through SSMS.

  1. Interactive Execution
  2. SQL Agent
  3. T-SQL

Interactive Execution

Through SSMS, you can right click a package and select Run. This will launch a package execution on the server, using your user credentials. The run package UI lets you set parameter values (overriding whatever defaults were set in the package at design time, or set on the server), and change values for connection managers.

image

SQL Agent

The Integration Services job step for SQL Agent has been updated in CTP3 to support execution of packages from an SSIS Catalog. The UI is very similar to the one you get for interactive package execution. Just like in previous versions, you can choose to run the package as the SQL Agent service account, or a proxy account.

Note: we found a last minute bug in the SQL Agent UI that prevents it from showing any package parameters (or connection managers). It is outlined in the release notes.

Using T-SQL to Run Packages

All of the SSIS functionality exposed in SSMS is (and more!) is also exposed through a T-SQL API. We expose a set of public views and stored procedures (in the catalog schema), as well as some private tables and stored procedures meant for internal use (in the internal schema).

image

You need to call two stored procedures to launch a package via T-SQL: catalog.create_exectution, which gives you an @execution_id, and catalog.start_execution, which starts that execution. The execution is created in two steps, because you might want to set parameter values (catalog.set_execution_parameter_value), or set up one or more data taps (catalog.add_data_tap, which will be covered in a later blog post).

Conclusion

This post touches on some of the new SSIS management functionality in SQL Server Denali. We’ll continue to post more about these new features (such as the reporting, troubleshooting and logging, and using “environments”) as the 30 Days of SSIS blog series continues.

Jumping-start SSIS Catalog procedures from SSMS Dialogs

0
0

Today’s post is from Ke Yang – a developer on the SSIS team based in Shanghai.

-----------------------------------

At times, we need T-SQL scripts to do batch jobs or to communicate with others. Since IS Catalog provides rich functionalities with over 40 stored procedures and functions, it can take you a while to learn how to use them and write your scripts. Luckily, you can easily see examples of how the T-SQL API functions by using the “Script” button on the SSMS dialogs.

Below screenshot shows the example of getting scripts from Execute Package dialog. After you finish the settings through the UI, you click “Script” and obtain the whole batch of runnable scripts including [create_execution], [set_execution_parameter_value], [start_execution]. Quite a jump start, isn’t it?

clip_image002

Below lists the operations that scripts are provided from dialogs for each type of SSIS object. I personally find the ones with “*” marks particularly helpful, because their scripts can be long and complex.

 

SSIS Object Type

Operations that scripts are provided from dialogs

Catalog

Set properties

Folder

Create, Delete, Set description, Set permissions

Project

Configure parameters and references*, Validate*, Delete, Set permissions

Package

Execute*, Validate*

Environment

Create, Delete, Set description, Set variables*, Set permissions

Parameterized Connection Managers

0
0

Today’s post is from Bob Bojanic – a developer on the SSIS Team.

---------------------------

There is often a need to update connection manager properties for subsequent package runs. Whether you need to point to a different server or change credentials, file locations or some other connectivity detail, all those properties will be exposed on connection managers. People often assign expressions to these properties, and that way add flexibility of parameterizing values associated with connection managers. This is a good practice and we encourage people to continue doing it, as it forces SSIS developers to think of exposing and documenting necessary input sets for their packages.

However, we recognize people will sometimes forget or miss to parameterize these properties. Whenever that happens, it is certain that a need will arise, in the production, to quickly change one of connection manager properties (i.e. point to a different server, share, etc.).

A new feature we are adding to SQL Server “Denali” version of SSIS will keep stress of admins and op engineers under control. We are making sure to automatically parameterize all properties (except read-only ones and those already parameterized) of connection managers when a project gets deployed to the SSIS server. We will do this for both, shared project connection managers and those that belong to packages.

Let us show how it works on a simple example. We are going to use a small data flow to export a table from a database to a flat file.

clip_image002

The OLE DB Source component will get data from the Product table in AdventureWorks as can be seen below.

clip_image004

The Flat File Destination component will send data to the “c:\test\product.txt” file.

clip_image006

There is no need for additional settings in order to parameterize properties of these two connection managers. We can simply deploy the project with our simple package to the local SSIS server.

clip_image008

The connection manager parameters will behave the same way the regular parameters do on the server. They can be set in the same dialogs and values from environments can be referenced in the same fashion they are used for other parameters.

Once the package is deployed we can see all the parameters generated from connection manager properties in Configure or Execute dialogs. We can also change values of those parameters in those two dialogs. If connection manager parameters are changes in the Configure dialog those values will be applied in all subsequent package executions and validations. On the other hand, if connection manager parameters are set in the Execute dialog, they will be applied only for a current package execution.

clip_image010

Here is the example of changes we did for the simple package we have just deployed. The connection manager pointing to the AdventureWorks database is changed to point to another server (sqlcldb2) and the initial catalog on that server has a different name (SSIS_AdventureWorks) as well.

clip_image012

The flat file connection manager is changed as well, so the new connection string places our file (Product.txt) to a location on the drive d.

clip_image014

After we execute the package, the execution overview report will have the following section that lists our changed properties and their assigned values.

clip_image016

As you could see, the automatic connection manager property parameterization feature in SQL Server “Denali” is going to make updates, of connection parameters in production, an extremely simple task. That should increase flexibility in how packages are linked with external dependencies. An important class of configurations/updates will be allowed without a need to redeploy packages.

Let us know how you like this feature.

Using DTEXEC with Packages on the IS Server

0
0

Today’s post is by Terri Chen – a developer on the SQL Server Integration Services team.

--------------

In Denali, the project is a unit of packages that are deployed to the server. DTExec has been enhanced to support executing packages stored in the SSIS Catalog. The path of the package on the server can be specified, as well as the values of project parameters, package parameters and connection manager properties. Server execution options can be configured as well. All existing options are supported for back compatibility.

The main new options for executing project packages are listed in the following table.

 

Option

Description

/ISSERVER package_path

(Optional). Loads a package that is saved on SSIS server. The package_path argument specifies the path and file name of the package. If the path or file name specified in the package_path argument contains a space, you must put quotation marks around the package_path argument.

You use /Server option together with the /ISSERVER option. Only Windows Authentication can execute a package on the SSIS Server. The current Windows user is used to access the package.

If the /Server option is omitted, the default local instance of SQL Server is assumed.

The /ISSERVER option cannot be used together with the /DTS or /SQL or /File option. If multiple options are specified, dtexec fails.

/Par[ameter] parameter_name(type);value

(Optional). Sets the parameter with a specified value. parameter_name can be a project ($Project::) or package level common parameter, connection manager parameter starting with “CM” or a server option parameter name starting with “$ServerOption::” . If the parameter type is not string, the type is specified in brackets after the parameter name. The value is separated by “;”.

/Envreference reference_id

(Optional). Sets the id of the referenced environment used in the execution. The parameters configured to bind to variables will use the values of the variables in the environment.

An example of executing a package from SSIS server:

DTExec /ISSERVER "\SSISDB\folderB\Integration Services Project17\Package.dtsx" /SERVER "." /Envreference 2 /Par "$Project::ProjectParameter(Int32)";1 /Par "Parameter(Int32)";21 /Par "CM.sqlcldb2.SSIS_repro.InitialCatalog";ssisdb /Par "$ServerOption::SYNCHRONIZED(Boolean)";True

The execution of SSIS server packages occurs on the server. Similar as to execute a package on the IS Server, DTExec calls catalog.create_execution, catalog.set_execution_parameter_value and catalog.start_execution to create an execution, set parameter values and start the execution. All execution logs can be seen from the server in the related views or by SSIS reports. So in addition to error messages from DTExec, the user can leverage all the utilities and tools on the server to trouble shooting the execution.

DTExec also supports executing packages from .ispac project. The related options are: /Proj[ect] and /Pack[age] to specify the project path and package stream name. Parameter values can be set by /SET and /CONF. To set a parameter value, reference it using the $Project or $Package namespace. Here are examples for a project level parameter and a package level parameter.

\Package.Variables[$Project::Parameter];1

\Package.Variables[$Package::Parameter];1

In SSIS SQL agent job step, dtexec is used to schedule job steps for packages from the IS server. Customers can also use T-SQL SQL Agent job step to run a package from the IS server. To generate the script easily, you can click on the Script button of the Execute Package Dialog from SSMS.

Note that the tool DTExecUI is not updated with the new features in dtexec.

All About Server Environments

0
0

Today’s post is from Renhe Li – a developer on the SSIS team based in Shanghai.

-----------------

Server Environments are a new concept introduced in SQL Server Denali for the Integration Services Catalog. It is a container for “server variables”. Each folder in the IS Server contains a number of environments.

When you add an environment to a folder, you are basically exposing a set of values and allowing them to be used when the package is executed or validated in IS Catalog. You can create an environment on server under the environment node as shown in Figure 1.

clip_image001

Figure 1: Creating a server environment

We could also use T-SQL stored procedure to do this directly:

create_environment [ @folder_name = ] folder_name
                 , [ @environment_name = ] environment_name
               [ , [ @environment_description = ] environment_description ]

A Server Environment name should be unique under the same folder. The access of the environment is controlled by SSIS_Admin, so you could only view the environments you have permission to.

Server Variables

After the environment is created, you could add environment variables into that container. We list the basic information of an environment variable in the below table:

Environment Variable Properties 

Property name

Description

Data Type

Note

Environment_id

Unique identifier for the environment variable to indicate which environment it belongs to

BigInt

 

Name

Name of the environment variable

nvarchar (255)

The environment variable name should be unique under the same environment

Type

Type of environment variable

nvarchar(128)

Type supported:

Boolean, byte, datetime, decimal, double, int16, int32, int64, sbyte, single, string, uint32, uint64

Sensitive

Whether the environment variable contains a sensitive value

Bit

Sensitive values are encrypted in IS catalog

Value

The value of the environment variable

Sql_variant

Only non-sensitive variable are shown here.

You could add, remove or modify an environment variable by invoking server APIs: catalog.create_environment_variable, catalog.delete_environment_variable etc. or you could do this directly in SSMS as shown in Figure 2:

clip_image002

Figure 2: Create server variables

Project Environment Reference

The project would not use any of the environments by default, and in order to use the value of the environment variable, you need to specify the reference between project and environment.

You could create a project environment reference by using SSMS UI:

image

Figure 3: Create the project environment reference

A project-environment reference can be marked as relative or absolute when it is created: The relative reference means the project will use the specified environment under the same folder and absolute reference means the project will use environment pointed by {environment folder name, environment name}.

Figure 4 shows the difference between relative and absolute reference when we move the project from Test folder to Production folder:

Relative Reference: The reference is changed to use the environment with same name under production folder.

clip_image011

clip_image013

Before the project is moved

After the project is moved

Absolute Reference: the reference remains unchanged

clip_image014

clip_image016

Before the project is moved

After the project is moved

Figure 4: Difference between absolute and relative reference when moving a project

After the reference is created, you could invoke the following API, with the value_type set to ‘R’ to specify a referenced value for a package parameter:

set_object_parameter_value [ @object_type = ] object_type
                         , [ @folder_name = ] folder_name
                         , [ @project_name = ] project_name
                         , [ @parameter_name = ] parameter _name
                         , [ @parameter_value = ] parameter_value
                       [ , [ @object_name = ] object_name ]
                       [ , [ @value_type = ] value_type ]

Example
A project Project1 contains a package - P1. The project is stored in a folder “Finance”.
P1 has two parameters: start_date.
If you want to set the value of P1.start_date to contain a referenced value, you can invoke the set_object_parameter_value API as follows directly:

exec set_object_parameter_value 30, 'Finance', 'project1', 'start_date', 'env_variable1', 'P1', 'R'

Then the value of the environment variable could be used in package validation and execution as shown in figure 5.

clip_image018

Figure 5: Use environment variable in package execution


Overview of the SSIS MOM

0
0

Today’s post is from Ke Yang – a developer on the SSIS Team based in Shanghai.

---------------------------------------

The namespace Microsoft.SqlServer.Management.IntegrationServices, contains a rich set of managed APIs that encapsulate most IS Server T-SQL APIs via ADO.NET commands. It’s not a mere T-SQL wrapper; it supports advanced features such as script generation, query via iterators, SQL Powershell, dependency discovery etc. We made this namespace so that developers can build their applications upon it, without having to build a similar infrastructure from T-SQL.

Now, let’s take a glimpse at some code pieces that use this namespace. You will soon find how handy it is!

Create the Catalog

// Create an IntegrationServices instance using an SMO server connection.
// Here SMO = Microsoft.SqlServer.Management.Smo; connection is a ServerConnection.
SMO.Server server = new SMO.Server(connection);
IntegrationServices isserver = new IntegrationServices(server);
//Create a catalog under isserver, specify the password.
//In current release, we allow only one catalog and fix the name to be "SSISDB"; other names would get an error
Catalog catalog = new Catalog(isserver, "SSISDB", "password");
catalog.Create();

Create a Folder

//Create a folder under catalog, with folder description
CatalogFolder f = new CatalogFolder(catalog, "folder1", "Description of folder1.");
f.Create();
//Create an environment under folder1
EnvironmentInfo e = new EnvironmentInfo(f, "env1", "Description of env1.");
e.Create();
//Add variables into e1. We declare v2 as sensitive.
e.Variables.Add("var1", TypeCode.Int32, 1, false, "Description of var1.");
e.Variables.Add("var2", TypeCode.String, "sensitive value", true, "");
e.Alter();

Deploy a Project

//stream is a byte[] containing the project binary obtained from Project runtime object model.
//You can use CreateProject() to create a project, PackageItems.Add() and Parameters.Add() to add packages and parameters,
//SaveAs() to save the project binary, and finally use System.IO.File.OpenRead() to read it into byte[].
//Here we omit these steps and assume we've already created packages and parameters within the project.

folder.DeployProject("p1", stream);
folder.Alter();
//Add "folder\env1" as a reference.
folder.Projects["p1"].References.Add("env1", "folder1");
//We're able to also add e1 as a "relative" reference, since it's under the same folder with the project.
folder.Projects["p1"].References.Add("env1");
//Set parameter1 to be referencing an environment variable under e1.
folder.Projects["p1"].Parameters["param1"].Set(ParameterInfo.ParameterValueType.Referenced, "var1");
folder.Projects["p1"].Alter();

Execute and Validate

ProjectInfo p = folder.Projects["p1"];
foreach (var pkg in p.Packages)
{
    // We can specify whether to use 32 bit runtime for execution on a 64-bit server
    // (here we specify "false"), and specify the how to use references (here we 
    // validate the package against all its references; 
    // if it has no references, we use parameter default values). Since we don’t specify any 
    // specific reference, the 3rd argument is left null.
    pkg.Validate(false, PackageInfo.ReferenceUsage.UseAllReferences, null);

    //Execute the package. The meanings of the arguments are similar to those in PackageInfo.Validate.
    pkg.Execute(false, null);

}
//Validate the project. The meanings of the arguments are similar to those in PackageInfo.Validate.
p.Validate(false, ProjectInfo.ReferenceUsage.UseAllReferences, null);

Get Messages

//Print all operation messages, including the execution and validation messages.
//If we want only execution or validation messages, we can replace the 
// "Operations" into "ExecutionOperation" or "ValidationOperation" in below code piece.
catalog.Operations.Refresh();
foreach (Operation op in catalog.Operations)
{
    op.Refresh();
    foreach (OperationMessage msg in op.Messages)
    {
        Console.WriteLine(msg.Message);
    }
}

Report Authoring on the SSIS Catalog

0
0

After a short hiatus, the 30 Days of SSIS series is back! Today’s special guest post is from Jamie Thomson.

--------------------------------------------

SSIS Catalog Log Tables

As you may already have gleaned from earlier posts in the 30daysofSSIS series SSIS in Denali includes the new SSIS Catalog that is used for execution and administration of SSIS projects and their associated paraphernalia. An important new feature provided by the SSIS Catalog is the automatic logging of all package execution activity along with some built-in reports that provide a view over that log activity. One good example of such a report is the “All Executions” report that gives an overview of all executions of packages in a given folder:

image

image

All of the underlying data for this and other reports exists in tables in a database called [SSISDB] and this of course means that you also have the ability to build your own reports atop the same data. The volume of data that is available for reporting is determined by the LOGGING_LEVEL at which your packages are executed; the available LOGGING_LEVELs are:

  • None
  • Basic
  • Performance
  • Verbose

One notable improvement from earlier versions of SSIS regarding LOGGING_LEVEL is that it is configured externally from a package at execution-time. This means that if you need to increase the LOGGING_LEVEL (perhaps for problem investigation) you can do so without having to make changes to your packages.

If you want to build your own reports then you will need to familiarize yourself with the following views, all of which exist in a schema called [catalog]:

View Description
executions Contains a record for every instance of a package being executed on the SSIS Catalog
event_messages All events that are captured from executing packages. [event_messages] is most analogous to the [sysssislog] table that was provided in earlier versions of SSIS.
event_message_context Provides contextual information about a subset of records in [event_messages]
executable_statistics Every package, container or task that executes during a package execution has a record here providing duration and execution result (success or otherwise)
execution_parameter_values The inputs provided to each execution. If you want to know what LOGGING_LEVEL was used for an execution you will find that in [execution_parameter_values].
execution_component_phases Provides information that enables determination of bottlenecks in dataflows which is invaluable for performance tuning. Specifically it tells us how long each component in the dataflow spends in its various phases of execution; those phases are {Validate, PrepareForExecute, PreExecute, AcquireConnections*, PrimeOutput, ProcessInput, PostExecute, ReleaseConnections*, Cleanup}
execution_data_statistics Provides information about the number of rows processed by each component in a dataflow. If you find the rowcounts depicted in a dataflow when it executes in the development environment useful then [execution_data_statistics] is for you.

*AcquireConnections and ReleaseConnections phases are not logged in Denali CTP3 however they will be added later.

 

Integrating Your Custom Logging Infrastructure

Many of you will have existing logging frameworks that you have built for previous versions of SSIS and you will be happy to know that you are allowed to integrate those frameworks into [SSISDB] by creating your own tables without invalidating your SQL Server license. Moreover, in a later release of SQL Server Denali (its not available in CTP3 unfortunately) the [execution_id] that uniquely identifies each execution in [catalog].[executions] will be available in a system variable called @[System::ServerExecutionID].

image

You will be able to use this value in your own tables to link your custom logging framework to the built-in log tables in the SSIS Catalog.

Reporting Services Project Template

You can build reports atop the SSIS Catalog using any reporting tool you wish though of course many will choose to use SQL Server Reporting Services (SSRS). In order to kickstart your report authoring I have provided a SSRS project template that provides some shared datasets that may prove useful:

image

You can download this template as a zip file from here, note that it will currently only work with Denali CTP3. The Shared Data Source “SSISDB.rds” currently points to (localhost) so simply change that to point to your [SSISDB] of choice and away you go.

Jamie Thomson
http://sqlblog.com/blogs/jamie_thomson
http://twitter.com/jamiet

Undo, Redo, and new SSIS Toolbox Features

0
0

Today’s post is from Carla Sabotta - lead writer for the SQL Server Integration Services team.

------

In SQL Server “Denali”, you can undo and redo up to 20 actions in the SSIS Designer.

For a package, undo /redo is available in the Control Flow, Data Flow, Event Handlers, and Parameters tabs, and in the Variables window. For a project, undo/redo is available for the project parameters.

You can’t undo/redo changes to the new SSIS Toolbox, and you can’t undo/redo shared connections.

To undo an action, click the undo button or press CTRL+Z. To redo an action, click the redo button or press CTRL + Y.

clip_image001

You can undo and redo multiple actions, by clicking the arrow next to the button, highlighting multiple actions in the drop-down list, and then clicking in the list.

clip_image002

When you make changes to a component using the component editor, you undo and redo the changes as a set rather than undoing and redoing individual changes. The set of changes appears as a single action in the undo and redo drop-down list.

clip_image003

SSIS Toolbox

All components that are installed on the local machine, including third-party components built for SQL Server 2008 and 2008 R2, now automatically appear in the new SSIS Toolbox. When you install additional components, right-click inside the toolbox and then click Refresh Toolbox to add the components.

You can easily access more information about a component from the toolbox, by clicking the component to view a description at the bottom of the toolbox. Click the Help button next to the description to display the Books Online topic.

clip_image005

In the SSIS Toolbox, control flow and data flow components are organized into categories. You can expand and collapse categories for viewing and you can change the organization of components according to your preferences. You can restore the default organization, by right-clicking inside the toolbox and then click Restore Toolbox Defaults.

The Favorites and Common categories appear in the toolbox when you select the Control Flow, Data Flow, and Event Handlers tabs.

clip_image006

The Other Tasks category appears in the toolbox when you select the Control Flow tab or the Event Handlers tab.

clip_image008

The Other Transforms, Other Sources, and Other Destinations categories appear in the toolbox when you select the Data Flow tab.

clip_image010

To move an item to a category

· Right-click an item in the toolbox, and then click one of the following:

  • Move to Favorites
  • Move to Common
  • Move to Other Sources
  • Move to Other Transforms
  • Move to Other Destinations
  • Move to Other Tasks

When you create a new SSIS project or open an existing project, the SSIS Toolbox is automatically displayed. You can also open the toolbox by doing one of the following:

· Click the toolbox button that is located in the top-right corner of the package design surface

clip_image011

· On the View menu, click Other Windows, and then click SSIS Toolbox.

You can dock the toolbox, set it to collapse when docked, and you can close the toolbox.

clip_image012

SQLBits 9 conference schedule posted

0
0

The (preliminary) SQLBits 9 conference agenda has been posted!

I’ll be presenting an SSIS session entitled “What’s New in SQL Server Denali” which covers most of what we showed at TechEd, as well as some of the integration with the other EIM products in Denali, Data Quality Services and Master Data Services.

I’ll also be presenting a full day deep dive session for the Training Day - SSIS Performance Design Patterns. Here’s the abstract once again:

Need a fast data integration solution, but don't have the time or budget for heavy performance tuning? By selecting the right design, 90% of SSIS customers will achieve their ETL performance goals with little to no performance tuning. This session will teach you how to pick that design! Targeted at intermediate to advanced SSIS users, Matt Masson - a developer on the SSIS team - will guide you through a series of proven SSIS design patterns, and teach you how to apply them to solve many common ETL problems. We’ll start off with an overview of the Data Flow internals, and review the SQLCAT team’s Top Ten Best Practices for achieving optimal performance. We’ll then dive into a series of design patterns, including Advanced Lookup Patterns, Parallel Processing, Avoiding Transactions, handling Late Arriving Facts and multiple ways to process Slowly Changing Dimensions. If you’re looking for ways to maximize your SSIS ROI, then you won’t want to miss this!

If you can’t make it to SQL Bits this time around, I’ll also be doing a full day of training at the PASS Summit (along with Andy Leonard and Tim Mitchell).

The Balanced Data Distributor

0
0

Today’s post isn’t specifically about SQL Server Denali, but still a worthy addition to the 30 Days of SSIS series. The Balanced Data Distributor is a new data flow component created by the SQLCAT team, and available for download from Microsoft.com. Carla Sabotta and Debarchan Sarkar have put together a training video for it, which is now available on technet:

In this video about the new Balanced Data Distributor (BDD) transformation, you’ll learn how to increase the throughput of the package Data Flow transformations by paralleling the Data Flow buffers to multiple outputs. The BDD transformation is designed to take advantage of the concurrent processing capability of modern CPUs.

clip_image001

Links

The Project and Parameter Object Model

0
0

Back to the Denali theme - today’s post is from Xiaochen Wu, a tester on the SSIS team based in Shanghai.

-------------------

In SSIS, we can manipulate packages programmatically with the SSIS object model. In Denali, we extended the object model to support the new concepts we introduced including project and parameter.

Project Object Model

In Denali SSIS, we can create, load, modify and save projects dynamically with object model. Following is the code sample of how to manipulate projects programmatically:

//Update it to a valid file path. The “.ispac” is the extension of project file
string projectFileName = @"d:\temp\test.ispac";

//Create a new project and specify the project file name as the project storage
using (Project project = Project.CreateProject(projectFileName))
{
    //Set the project property
    project.Description = "This is a new project";

    //Add a package to the project
    project.PackageItems.Add(new Package(), "package.dtsx");

    //Get the package and modify its property
    project.PackageItems[0].Package.Description = "This is a new package";

    //Save the project
    project.Save();
}

//Load the project from an existing project file
using (Project project = Project.OpenProject(projectFileName))
{
    project.Description = "Loaded from exsting file";
}
Create and Load Projects

When we create a new project, we can specify where we want to save this project as a parameter. The project storage can be a file (the full file name) or stream (System.IO.Stream). The storage will take effort when you save the project. We will talk about the details in next sections.

We can also load the project from a project file (.ispac) or stream. Please notice that if we load project from stream, we should first set the position within the stream to the beginning. If our project file or stream is protected or partially protected by password, we can also specify the password when load the project.

Save Projects

In Denali SSIS, there’re 3 ways to save the projects: Save(), SaveAs() and SaveTo().

If the project storage is already specified, we can use the Save function to save the project to the default storage. If the file or stream is not empty, the existing content will be overwritten. If we call the Save function before we specify the project storage, we will get exception.

The SaveAs function can save project to the specified storage and change to default project storage to the specified one.

The SaveTo function also can save project to the specified storage. But it will not change the current default project storage.

Following is the code sample of saving projects.

using (MemoryStream ms = new MemoryStream())
{
    //Create a new project and specify the project file name as the project storage
    using (Project project = Project.CreateProject(ms))
    {
        //Save the project to the default storage
        project.Save();

        string projectFileName = @"d:\temp\test1.ispac";
        string anotherFileName = @"d:\temp\test2.ispac";

        //Save the project to the file storage and update the default storage
        project.SaveAs(projectFileName);

        //Save the project to the file storage but don't update the default storage
        project.SaveTo(anotherFileName);
    }
}
Manipulate Packages in Project

In Denali SSIS, we can add, get or remove packages in an existing project:

When we add packages to the project, we need to specify the package stream name which is a string ended with “.dtsx”. The stream name can identify a package within a project and it is not necessary to be the same with the package name in package properties.

Parameter Object Model

Parameter is another new concept we introduced in Denali SSIS. We can use the object model to add, modify and remove the parameter for packages or projects.

When we add a parameter, we need to specify the parameter name and data type. The parameter name should be identical for a project or a package. But we can have a project parameter and a package parameter with the same name, because they can be distinguished by the namespace. The namespace of project parameter is “$Project” and namespace of package parameter is “$Package”.

In Denali SSIS, the following data types are not supported for parameters: Empty, Object, DBNull, Char and UInt16. And when we specify values of parameters, it should have the same data type with the parameter. For example, the following code will throw exception:

using (Project project = Project.CreateProject())
{
    //Create a project parameter with type of Int64
    Parameter param = project.Parameters.Add("Param", TypeCode.Int64);

    //Assign a value with Int32 to the parameter. Exception thrown!
    param.Value = (Int32)100;
}

In the project or package, we can get the parameter by name or index and use it in the package as a variable. Following is the sample code of manipulating parameters:

using (Project project = Project.CreateProject())
{
    //Add a package to the project
    project.PackageItems.Add(new Package(), "Dataflow.dtsx");
    Package package = project.PackageItems[0].Package;

    //Add a package parameter with name "PkgParam" and type string
    package.Parameters.Add("PkgParam", TypeCode.String);

    //Add a project parameter with name "PrjParam" and type datetime
    project.Parameters.Add("PrjParam", TypeCode.DateTime);

    //Get parameter by name
    package.Parameters["PkgParam"].Value = "I'm a package parameter";

    //Get parameter by index
    project.Parameters[0].Value = "I'm a project parameter";

    //Use parameters in expression
    package.Properties["Description"].SetExpression(package, "@[$Project::PrjParam]");

    //Remove parameters
    project.Parameters.Remove("PrjParam");
    package.Parameters.RemoveAt(0);
}

 

Project Protection Level

If we want to protect our projects with password or user key, we need to specify the protection level and password of the project. When we add a package to project, the protection level and password of the package will be automatically updated to the same value with the corresponding project properties. And if we want to change it, we will get the exception.

In this article, we discussed how to use SSIS object model to manipulate projects and parameters. For more details, please refer to MSDN or SQL Server Book Online.

Debugging Configuration Issues on the Server Using Reports and T-SQL views

0
0

Today’s post is from Da Lin – a tester on the SQL Server Integration Services team.

-------------------------------------

With the built-in reports and TSQL views, Integration Services in SQL Server “Denali” allows you easily identify the root cause of the execution failure. In the following example, the execution failed because the server name was incorrect. This post will guide you through how to use these built-in features to find the problem.

Once you find out the execution failed, you go to the failed package and launch the Package Execution Report which will show you execution history of this package.

Then you click the “Overview” of the most recent failed execution and see a report showing an overview of this execution. The values of parameters used in the execution will show in the report so that you can examine if the values are set correctly. In this case because the property of the connection manager is parameterized, you can see the server name used in connection manager in the “Parameter Used” section and it was set to an invalid server name in the red box.

clip_image002

If you want to see the detailed messages of this execution, you can click the “View Messages” link. You can check out all types of messages recorded in this transaction.

clip_image004

For the message whose message type is “OnError” you can even drill down to see the context when the error occurred by clicking the “View Context” link.

clip_image006

In this report, you can see the values of the properties when error occurred.

The values of the parameters of the execution you saw in the overview page of the report can also be found in the view [Catalog].[ execution_parameter_values] in SSISDB. By providing the execution id you can query the view to retrieve the values of parameters used during the execution.

clip_image010


The Import Project Wizard

0
0

Today’s post is from Ranjeeta Nanda, a Software Development Engineer in Test on the SSIS team. This post describes the new “Integration Services Import Project Wizard” project type in BIDS in Denali.

-----------------

By now, you know that SSIS now has IS Projects that are saved on the disc in the form of .ispac files. These are zipped files that contain all the packages in the project, and shared connection managers, project parameters etc. together. You can very easily pass these .ispac files to another member in the team, or can open them in another machine. To help you with that, we have an Integration Services Import Project Wizard.

When you choose a New Project from BIDS, under Integration Services you will see a new Project type called Integration Services Import Project Wizard.

clip_image002

When you choose this option, on the next step, you will see the new Wizard called the “Integration Services Import Project Wizard”.

clip_image004

The wizard lets you choose the location of the project file (.ispac) from the file system.

clip_image006

You could also choose to import from a project that is already deployed to a server

clip_image008

On the next step, the wizard will load and validate the project file

clip_image010

If the project is validated successfully, it will show a summary of what it is going to import next.

clip_image012

On clicking the Import button at this stage will import the project in BIDS

clip_image014

You can then view all the contents of the project in BIDS. This will include all packages, connection managers, project connection managers, Project parameters etc.

This is an easy way for developers to share their projects with each other by just sending out a single file in the form of .ispac. It also enables developers to import back projects from server back to BIDS.

clip_image015

Troubleshooting SSIS Package Performance Issues

0
0

Today’s post is from Wee Hyong Tok, a Program Manager with the SSIS team based in Shanghai. It provides some tips on using T-SQL to analyze performance issues for packages running on the new SSIS Catalog. For more information about the new logging functionality in Denali, see Jamie Thomson’s post about report authoring.


When an SSIS package is running slower than usual, an administrator or SSIS developer might want to figure out the part of the package that is causing it to run slowly. An administrator might also want to build a monitoring mechanism (e.g. a SQL Agent job) that alerts him/her whenever packages that have been deployed to the SSIS catalog are running slower than usual.

In order to obtain performance-related information for packages that have been deployed to the SSIS catalog, you will need to make use of the Performance or Verbose Logging level.

Note: In the Verbose level, besides capturing performance related information, it might also log other messages which might not be useful in performance troubleshooting. Consider using the Performance Logging Level if you want error, warning, and performance information to be captured for packages that have been deployed to the SSIS catalog. The Performance Logging level provides a good balance between having sufficient information for troubleshooting a wide set of SSIS package issues (e.g. package failures, performance issues) and having a performance impact to the package (due to the information that gets logged).
We show how you can make use of the following public views to identify the component (and phase) for a SSIS package performance issue.

catalog.executions
catalog.executable_statistics
catalog.executables
catalog.execution_component_phases
catalog.execution_component_phases

The following example shows how you can make use of T-SQL to identify the package executions that might have performance issues. Given a package, we first identify all the package executions that are successful (i.e. status = 7).

use SSISDB

declare @foldername nvarchar(260)
declare @projectname nvarchar(260)
declare @packagename nvarchar(260)

set @foldername = 'Folder1'
set @projectname = 'Project1'
set @packagename = 'Dim_DCVendor.dtsx'

DECLARE @ExecIds table(execution_id bigint);
insert into @ExecIds

SELECT execution_id
FROM catalog.executions
WHERE folder_name = @foldername 
     AND project_name = @projectname 
     AND package_name = @packagename 
     AND status = 7

From these successful executions, we identify the tasks (and their corresponding execution ID). We order the results, in descending order), by the time spent in the execution.

SELECT es.execution_id, e.executable_name, ES.execution_duration
FROM catalog.executable_statistics es, catalog.executables e
WHERE 
es.executable_id = e.executable_id AND
es.execution_id = e.execution_id AND
es.execution_id in (select * from @ExecIds)
ORDER BY e.executable_name,es.execution_duration DESC;

In order to identify the tasks that are taking longer than usual, we first compute the average and standard deviation for the duration spent by each task. In the query, we define a “slower than usual” task as one whose duration is greater than average + standard deviation (i.e. es.execution_duration > (AvgDuration.avg_duration + AvgDuration.stddev))

With AverageExecDudration As (
    select executable_name, avg(es.execution_duration) as avg_duration,STDEV(es.execution_duration) as stddev
    from catalog.executable_statistics es, catalog.executables e
    where 
    es.executable_id = e.executable_id AND
    es.execution_id = e.execution_id AND
    es.execution_id in (select * from @ExecIds)
    group by e.executable_name
)
select es.execution_id, e.executable_name, ES.execution_duration, AvgDuration.avg_duration, AvgDuration.stddev
from catalog.executable_statistics es, catalog.executables e, 
    AverageExecDudration AvgDuration
where 
es.executable_id = e.executable_id AND
es.execution_id = e.execution_id AND
es.execution_id in (select * from @ExecIds) AND
e.executable_name = AvgDuration.executable_name AND
es.execution_duration > (AvgDuration.avg_duration + AvgDuration.stddev)
order by es.execution_duration desc

From the results of the query, we can identify all the “slower than usual” tasks. Suppose the name of the task is [DFT Load DC Vendor], and its corresponding ID is 188. We zoom into this specific execution, and identify the time spent in each phase of the data flow task.

declare @probExec bigint
set @probExec = 188

-- Identify the component’s total and active time
select package_name, task_name, subcomponent_name, execution_path,
    SUM(DATEDIFF(ms,start_time,end_time)) as active_time,
    DATEDIFF(ms,min(start_time), max(end_time)) as  total_time
from catalog.execution_component_phases
where execution_id = @probExec
group by package_name, task_name, subcomponent_name, execution_path
order by active_time desc

declare @component_name nvarchar(1024)
set @component_name = 'DFT Load DC Vendor'

-- See the breakdown of the component by phases
select package_name, task_name, subcomponent_name, execution_path,phase,
    SUM(DATEDIFF(ms,start_time,end_time)) as active_time,
    DATEDIFF(ms,min(start_time), max(end_time)) as  total_time
from catalog.execution_component_phases
where execution_id = @probExec AND subcomponent_name = @component_name
group by package_name, task_name, subcomponent_name, execution_path, phase
order by active_time desc

From the breakdown of the phases, we figure out that a particular phase is experiencing performance issue. Consequently, we can look into the design of the package for the specific data flow component to resolve the performance issue.

We will love to hear stories on how you are using these new views and capabilities introduced in SQL Server “Denali” to troubleshoot performance issues in SSIS package. Add your story as a comment to this post!

Column Mapping Improvements in Denali

0
0

We made a number of changes in Denali around the use of lineage IDs, and auto-mapping of metadata. Collectively, these improvements were internally called “Flexible Order of Authoring” (because the idea was that you could build a package backwards – from destination to source).

Take the following example (which I walked through in my Denali presentation at TechEd if you’d prefer this in video format…):

image

This package has a Raw File Source reading from file containing customer data. The metadata looks like this:

image

Notice the FirstName and LastName columns.

Metadata mapping errors vs. component errors

If I delete the path connecting the source to the Copy Column transform, my package is put into an error state. Notice where the error icons appear:

image

In Denali, we distinguish between errors caused by column mappings, and errors on the component itself. When an error is due to a mapping problem (i.e. missing columns), the error icon will be display on the path, or above the transform if no path is connected. Another change is that the component itself can still be edited - the component actually caches the metadata from it’s last working state. If I double click on it, it’s UI appears, and I am able to make changes.

image

Smarter Lineage IDs

The Data Flow in Denali still makes use of Lineage IDs (unique integer values assigned to each column in the data flow), but the designer is now a lot smarter about remapping them when a new data source in connected.

In my package, I have a second Raw File Source (called Alternative Source). It is reading a file that is similar to the other source component, but in the column mappings, I’ve renamed two of the columns.

image

Noticed that FirstName and LastName are now output as GivenName and FamilyName.

In previous versions of SQL Server I would get a number of mapping errors if I connect a new source component to existing downstream components because of the change in Lineage IDs. While the remapping process was improved in 2008, you still needed to fixup the columns coming in from the new source. In Denali we’ve improved the remapping functionality by automatically remapping columns based on their name (and data type), instead of by lineage ID.

After connecting the Alternative Source to the Copy Column transform, we see the error icon is moved to the path.

image

Double clicking the error brings up the new Resolve References UI.

Resolve References

This new UI was originally developed for the Parallel Data Warehouse destination (where it is common to have hundreds of columns per table). In Denali it’s been integrated into the main product to become our column reference resolving UI. It has a number of features, including:

  • Auto-map based on name/type
  • Filtering columns by name
  • Import column mappings from Excel
  • Export column mappings to Excel

image

In this case, all columns that had the same names were automatically remapped by the designer. The names of the columns I had renames were not mapped, but I can easily fixed up the mappings myself.

Another thing to note is that when the Resolve References UI is launched, you’re not just fixing a single component. The operation will attempt to resolve the mappings for the entire execution tree. This execution tree will be highlighted in the designer while the Resolve References UI is open.

image

After completing the UI, everything is reconnected, and the errors go away.

image

And of course, the great thing about this demo is that I can Undo things back to the way they were when I started, and the package is ready to demo once again!

That’s it for the column mapping improvements – we hope you’ll enjoy these and the other new features and functionality in Denali.

Making your Existing Custom SSIS Extensions and Applications Work in Denali

0
0

Today’s post is from Rujin Cao – a developer on the SSIS team based in Shanghai. This article talks about the .NET assembly binding we include in our .exes by default, which makes migrating custom SSIS code to Denali a lot easier. Note that while this is a good work around, recompiling your code to target the Denali versions of the SSIS assemblies is still recommended.


Upgrading custom extensions and applications which used the SSIS object model from 2005 to 2008 required some code changes. One of the goals in SQL Server “Denali” was to make this process easier. In general, most .NET custom extensions (i.e. ones that use the ManagedDTS API) will not need to be recompiled or require code changes, and most custom applications will just need to update their .exe.config file.

Custom SSIS Extensions

For custom SSIS extensions, we added four binding redirection rules in the *.exe.config of DTExec.exe, DTExecUI, dtshost.exe, DTSWizard.exe and DTUtil.exe to help redirect the runtime assemblies from version 10.0.0.0(in SQL Server 2008(R2)) to version 11.0.0.0(SQL Server “Denali”). Here it is (you can also see it in“%ProgramFiles%\Microsoft SQL Server\110\DTS\Binn\DTExec.exe.config” after installing Denali):

  1. <assemblyBinding xmlns="urn:schemas-microsoft-com:asm.v1">
  2.   <dependentAssembly>
  3.     <assemblyIdentity name="Microsoft.SqlServer.ManagedDTS" publicKeyToken="89845dcd8080cc91" culture="neutral" />
  4.     <bindingRedirect oldVersion="10.0.0.0" newVersion="11.0.0.0"/>
  5.   </dependentAssembly>
  6. </assemblyBinding>
  7. <assemblyBinding xmlns="urn:schemas-microsoft-com:asm.v1">
  8.   <dependentAssembly>
  9.     <assemblyIdentity name="Microsoft.SqlServer.DTSRuntimeWrap" publicKeyToken="89845dcd8080cc91" culture="neutral" />
  10.     <bindingRedirect oldVersion="10.0.0.0" newVersion="11.0.0.0"/>
  11.   </dependentAssembly>
  12. </assemblyBinding>
  13. <assemblyBinding xmlns="urn:schemas-microsoft-com:asm.v1">
  14.   <dependentAssembly>
  15.     <assemblyIdentity name="Microsoft.SqlServer.DTSPipelineWrap" publicKeyToken="89845dcd8080cc91" culture="neutral" />
  16.     <bindingRedirect oldVersion="10.0.0.0" newVersion="11.0.0.0"/>
  17.   </dependentAssembly>
  18. </assemblyBinding>
  19. <assemblyBinding xmlns="urn:schemas-microsoft-com:asm.v1">
  20.   <dependentAssembly>
  21.     <assemblyIdentity name="Microsoft.SqlServer.PipelineHost" publicKeyToken="89845dcd8080cc91" culture="neutral" />
  22.     <bindingRedirect oldVersion="10.0.0.0" newVersion="11.0.0.0"/>
  23.   </dependentAssembly>
  24. </assemblyBinding>

The four redirected assemblies are:

  1. Microsoft.SqlServer.ManagedDTS
  2. Microsoft.SqlServer.DTSRuntimeWrap
  3. Microsoft.SqlServer.DTSPipelineWrap
  4. Microsoft.SqlServer.PipelineHost

If you are using some other version 10.0.0.0 assemblies (for example, the 9.0.242.0 assemblies from SQL Server 2005), you might have to add more rules in the *.exe.config files to redirect them to the Denali version 11.0.0.0.

 

Custom SSIS Applications

For the custom SSIS application, the process is much similar. You can create one configuration file (i.e. *.exe.config) for the executable (if it doesn’t have one), and put the above redirect rules into configuration section, more details can be found here.

Package Format Changes in SQL Server Denali

0
0

We made some significant changes to the .dtsx file package format in SQL Server Denali. The goal of these changes was to make the SSIS package format easier to read, and easier to diff when working with source control systems. Some of the changes around data flows (specifically, the way we persist lineage IDs) also make it somewhat easier to merge changes between packages as well. This post will explore more of these changes, providing XML samples so you can see exactly what was changed.

Side note: Did you know that the current DTSX package file format is available online? If you like reading about XML schemas, it’s a highly entertaining read!

It’s Pretty (Printed)

If you open up a .dtsx file in an XML/Text editor, one of the first things you’ll notice is that the package XML is now pretty-printed. It’s also more concise – instead of having separate elements for each property, these fields are now persisted as attributes (except for the PackageFormatVersion, which was left in the same format so 2005/2008 versions of DTEXEC could identify that the package was for a new version of SQL Server). Another feature that makes packages more concise is that we no longer persist properties that have default values (ex. VersionMajor = 1), and attributes are always listed alphabetically. The code snippets below provide an example of these changes.

2008 Format
  1. <?xml version="1.0"?><DTS:Executable xmlns:DTS="www.microsoft.com/SqlServer/Dts" DTS:ExecutableType="SSIS.Package.2">
  2. <DTS:Property DTS:Name="PackageFormatVersion">3</DTS:Property>
  3. <DTS:Property DTS:Name="VersionComments"></DTS:Property>
  4. <DTS:Property DTS:Name="CreationDate" DTS:DataType="7">1/6/2011 3:58:59 PM</DTS:Property>
  5. <DTS:Property DTS:Name="PackageType">5</DTS:Property>
  6. <DTS:Property DTS:Name="ProtectionLevel">1</DTS:Property>
  7. <DTS:Property DTS:Name="MaxConcurrentExecutables">-1</DTS:Property>
  8. <DTS:Property DTS:Name="PackagePriorityClass">0</DTS:Property>
  9. <DTS:Property DTS:Name="VersionMajor">1</DTS:Property>
  10. <DTS:Property DTS:Name="VersionMinor">0</DTS:Property>
  11. <DTS:Property DTS:Name="VersionBuild">505</DTS:Property>
  12. <DTS:Property DTS:Name="VersionGUID">{D1F82D76-CC8D-400C-A9F8-706DF686C4B7}</DTS:Property>
  13. <DTS:Property DTS:Name="EnableConfig">0</DTS:Property>
  14. <DTS:Property DTS:Name="CheckpointFileName"></DTS:Property>
  15. <DTS:Property DTS:Name="SaveCheckpoints">0</DTS:Property>
  16. <DTS:Property DTS:Name="CheckpointUsage">0</DTS:Property>
  17. <DTS:Property DTS:Name="SuppressConfigurationWarnings">0</DTS:Property>
  18. <DTS:ConnectionManager>
  19. <DTS:Property DTS:Name="DelayValidation">0</DTS:Property>
  20. <DTS:Property DTS:Name="ObjectName">(local).DI-DW</DTS:Property>
  21. <DTS:Property DTS:Name="DTSID">{1B70C7B8-BFB6-4EBD-855B-0BAF7C417ADD}</DTS:Property>
  22. <DTS:Property DTS:Name="Description"></DTS:Property>
  23. <DTS:Property DTS:Name="CreationName">OLEDB</DTS:Property><DTS:ObjectData><DTS:ConnectionManager>
  24. <DTS:Property DTS:Name="Retain">0</DTS:Property>
  25. <DTS:Property DTS:Name="ConnectionString">Data Source=(local);Initial Catalog=DI-DW;Provider=SQLNCLI11;</DTS:Property></DTS:ConnectionManager></DTS:ObjectData></DTS:ConnectionManager>

Denali Format
  1. <?xml version="1.0"?>
  2. <DTS:Executable xmlns:DTS="www.microsoft.com/SqlServer/Dts"
  3.   DTS:refId="Package"
  4.   DTS:CreationDate="1/6/2011 3:58:59 PM"
  5.   DTS:CreationName="SSIS.Package.3"
  6.   DTS:DTSID="{FC5F3400-D2B4-40E4-9B51-ED252DC63634}"
  7.   DTS:ExecutableType="SSIS.Package.3"
  8.   DTS:LocaleID="1033"
  9.   DTS:ObjectName="Version4"
  10.   DTS:PackageType="5"
  11.   DTS:VersionBuild="506"
  12.   DTS:VersionGUID="{00D49CE3-FD6D-4CE8-8793-4F79F1F6B19C}">
  13.   <DTS:Property
  14.     DTS:Name="PackageFormatVersion">6</DTS:Property>
  15.   <DTS:ConnectionManagers>
  16.     <DTS:ConnectionManager
  17.       DTS:refId="Package.ConnectionManagers[(local).DI-DW]"
  18.       DTS:CreationName="OLEDB"
  19.       DTS:DTSID="{1B70C7B8-BFB6-4EBD-855B-0BAF7C417ADD}"
  20.       DTS:ObjectName="(local).DI-DW">
  21.       <DTS:ObjectData>
  22.         <DTS:ConnectionManager
  23.           DTS:ConnectionString="Data Source=(local);Initial Catalog=DI-DW;Provider=SQLNCLI11;" />
  24.       </DTS:ObjectData>
  25.     </DTS:ConnectionManager>

One final thing to note in the samples above is that elements that can appear multiple times (such as <DTS:ConnectionManager>, <DTS:EventHandler>) are now contained within a parent element (<DTS:ConnectionManagers>, <DTS:EventHandlers>).

Goodbye Lineage IDs

Most objects within a package that can be referred to by other objects now have a refId attribute defined on in the package XML. The refId value is a unique, human readable (and understandable) string that other elements can refer to. It’s a bit easier than using GUIDs or integer values (i.e. lineage IDs), since the text itself has actual meaning (the path to the object in the package, its type, and name).

Let’s examine the following example which provides the definition for an output column in a Flat File Source:

Output Column Format
  1. <?xml version="1.0" encoding="utf-8"?>
  2. <outputColumn
  3.   refId="Package\Historical Loads\Load DailyMarket\DailyMarket.Outputs[Flat File Source Output].Columns[DM_DATE]"
  4.   dataType="dbDate"
  5.   errorOrTruncationOperation="Conversion"
  6.   errorRowDisposition="FailComponent"
  7.   externalMetadataColumnId="Package\Historical Loads\Load DailyMarket\DailyMarket.Outputs[Flat File Source Output].ExternalColumns[DM_DATE]"
  8.   lineageId="Package\Historical Loads\Load DailyMarket\DailyMarket.Outputs[Flat File Source Output].Columns[DM_DATE]"
  9.   name="DM_DATE"
  10.   truncationRowDisposition="FailComponent">
  11.   <properties>
  12.     <property
  13.       dataType="System.Boolean"
  14.       description="Indicates whether the column uses the faster, locale-neutral parsing routines."
  15.       name="FastParse">false</property>
  16.     <property
  17.       dataType="System.Boolean"
  18.       description="Indicates whether the data is in binary format."
  19.       name="UseBinaryFormat">false</property>
  20.   </properties>
  21. </outputColumn>

This snippet defines the XML for the DM_DATE column on a Flat File Source named DailyMarket. Note the lack of lineage IDs, which are integer values. Instead of persisting an integer value, we persist the refId string. While lineage IDs are still used within the runtime, they are no longer persisted, and are actually regenerated when the package is loaded.

If we break down the pieces of the refId, we see that it looks a lot like path values used for 2005/2008 package configurations (although a little shorter, and supports data flow objects as well).

image

In the previous example, we can see that this output column is linked to an ExternalMetadataColumn by refId as well.

externalMetadataColumnId="Package\Historical Loads\Load DailyMarket\DailyMarket.Outputs[Flat File Source Output].ExternalColumns[DM_DATE]"

If you try to merge changes between two versions of a package, the refId can be used in find/replace operations to ensure that all references to that object have been correctly updated.

For comparison, here is the 2008 package format of that output column (pretty-printed to make it more readable):

Output Column (2008)
  1. <outputColumn id="10"
  2.               name="DM_DATE"
  3.               description=""
  4.               lineageId="10"
  5.               precision="0"
  6.               scale="0"
  7.               length="0"
  8.               dataType="dbDate"
  9.               codePage="0"
  10.               sortKeyPosition="0"
  11.               comparisonFlags="0"
  12.               specialFlags="0"
  13.               errorOrTruncationOperation="Conversion"
  14.               errorRowDisposition="FailComponent"
  15.               truncationRowDisposition="FailComponent"
  16.               externalMetadataColumnId="9"
  17.               mappedColumnId="0">
  18.     <properties>
  19.         <property id="11"
  20.                   name="FastParse"
  21.                   dataType="System.Boolean"
  22.                   state="default"
  23.                   isArray="false"
  24.                   description="Indicates whether the column uses the faster, locale-neutral parsing routines."
  25.                   typeConverter=""
  26.                   UITypeEditor=""
  27.                   containsID="false"
  28.                   expressionType="None">false</property>
  29.         <property id="12"
  30.                   name="UseBinaryFormat"
  31.                   dataType="System.Boolean"
  32.                   state="default"
  33.                   isArray="false"
  34.                   description="Indicates whether the data is in binary format."
  35.                   typeConverter=""
  36.                   UITypeEditor=""
  37.                   containsID="false"
  38.                   expressionType="None">false</property>
  39.     </properties>
  40. </outputColumn>

Show me the Annotations!

The DTSX file format stores both the package logic, and its layout information. One of the first improvements we did in Denali was move to a new WPF-based design surface, which allowed us to change the way we persisted all of our layout information. In 2005/2008, the layout information was stored as encoded XML within a PackageVariableValue node. Annotations were stored as binary streams that were very hard to extra and understand. In Denali, layout information is now contained with a CData section, without any additional encoding. Annotations are persisted in clear text, so they can easily be extracted for automated documentation generation.

Summary

Many changes have been made to the Denali package format to make it easier to diff and merge. Below are a summary of the changes that were covered in this article:

  • Pretty-printed XML
  • Property elements now persisted as attributes
  • Default values are not persisted
  • Attributes are sorted alphabetically
  • All elements that appear more than once are contained within parent elements
  • References identified by refId attribute
  • Tasks and Components are appear alphabetically in the XML
  • Readable layout nodes
  • Annotations persisted in clear text
Viewing all 132 articles
Browse latest View live




Latest Images