Showing posts with label SSIS. Show all posts
Showing posts with label SSIS. Show all posts

Friday, October 22, 2010

SSIS DB2 and Stored Procedures

The Scenario

Extracting data from a DB2 file using SSIS when the data is accessed through a stored procedure.

With DTS packages this could be done by using a ODBC driver to connect to the DB2.

The Problems

With DTS packages there was an option to choose Other Driver. From this I could set up an ODBC connection to a DB2 database and execute a stored procedure – this acted as my data source. This ODBC connection used a system DSN which pointed at eh db2 database on an AS400 server.


SSIS does not give you the option of using other driver. What they have instead are OLE DB drivers or .Net drivers.


The OLE DB drivers will not allow the execution of a stored proc on the DB2 to retrieve data.

The IBM OLE DB DB2 provider and Microsoft OLE DB DB2 provider do not support using an SQL command that calls a stored procedure. When this kind of command is used, the OLE DB source cannot create the column metadata and, as a result, the data flow components that follow the OLE DB source in the data flow have no column data available and the execution of the data flow fails..

See: http://msdn.microsoft.com/en-us/library/ms141696.aspx

The Dot.Net connection, in SSIS 2008, will allow you to execute the procedure in an Execute Script Task. It will not allow execution in a data flow task – although it will allow you to preview the data in data flow.


After setting up a Linked Server to the DB2 I encountered the same problem – although the error message was slightly misleading and I spent a long time investigating permissions.


So the only options now are to carry out a row by row import by using an Execute TSQL Task to populate a record set object and then rolling through this to import the data or asking the DB2 DBA to use the stored procedures to populate file which I can then import from.


My forum question on MSDN relating to the issue:
http://social.msdn.microsoft.com/Forums/en-US/sqlintegrationservices/thread/7ea40a98-c930-48fc-a961-2fdb977de0d3/#e2edc306-f0f4-45bb-9844-53cf83fb845a


Connect issue raised with Microsoft:
https://connect.microsoft.com/SQLServer/feedback/details/615901/ssis-db2-and-stored-procedures
Please follow the link and vote if the issue effects you and maybe Microsoft will bring out a new driver.

Thursday, October 21, 2010

Importing Large Amounts of Data

The Problem


I have a system which imports large amounts of data over night. This will normally be in the low millions. If it fails for a few nights it can then be tens of millions. The problems is space – I don't have enough drive space to allow the transaction log to grow large enough to accommodate more than two million rows of data.


The Solution


Create a variable called @loopRowCounter.

Create Another called @loopCount

Use an execute TSQL task to populate that variable with the number of rows in the data source.

You can do this by:

Under General

SQL command should start like this - SELECT count(*) as loopRowCount FROM …

Set the ResultSet to single row

Under Result Set

Set result name to loopRowCount and set it to match your variable - User::loopRowCounter.

Now you have a variable with the number of rows in.

Move on to a For Loop.

In InitExpressions enter this @loopCount = 2000002 – this is to set a default value

In EvalExpression enter this @loopCount > 0

In asignExpression enter @loopCount = @loopRowCounter – inside the container @loopRowCounter will decremented.

Now use a data flow and in the Data source use Select Top (500000) …

You will need to do a check that the rows in the source are not in the destination – I do a join on the table and compare ids

This will limit the flow of data to 500000 rows.

Next got to an Execute SQL Task

Under Parameter Mapping

Select the variable Name User::loopRowCounter give the parameter a name and set it to type Long with a size of 4

Under General

SQL command should start like this:

checkpoint;

SELECT ? - 500000 as loopRowCount

Set the ResultSet to single row

Under Result Set

Set result name to loopRowCount and set it to match your variable - User::loopRowCounter.

We know how may rows were in the table and we assigned that value to User::loopRowCounter. We know we have just move 500000 out so we subtract 500000 from the parameter which represents User::loopRowCounter and the result is then assigned back to User::loopRowCounter.

Now, no matter how many millions of records go through my log file only has to cope with 500000 at a time.

Saturday, May 8, 2010

SSIS: Configuration

Nearly everything in the package can be configured.


The main items you will want to configure are the passwords in server connections along with the name of the server and possible the initial catalog (database).
To set up a configuration option right click the package main window and select package configurations.
The options available are to use an XML file, Registry Entry, Parent Package Variable or a SQL database.
In this example I have a database called dataimports which I am going to use. In dataimports I have created a schema for each batch of work I use. This means I can separate out all the SSIS logging and configuration and I can give users access through the schemas.




Once you select the SQL server option you will need to select a database connection to use and then a table. If you already have config tables the drop down box will list them – else you can create a new one. You will need to type in a configuration filter. This is a key word that will match up the configuration settings in the table with this package. The name should be easily identifiably like >> DataDownload.
When you click next you will see a list of configurable items. If this is a new package there will not be very many.

You should try to restrict the configured values to the ones really needed else you will make reading the configuration table difficult which can lead to errors.


A config file should be used to store secure data – like passwords. It should also be used to store values which will or could change. For example a package which emails a user upon completion could have the To field of the email task configured. This way if you need to change the recipient you can change the config table. If a server name is going to change as the package is deployed then store the server name in a config file. You shouldn’t store static or trivial data like the protection level of a database connection – this will probably never change; or the connection string to a database and then all the individual elements of the connection. If you are using Windows Authentication you may not need to store any connection data.

What it looks like in the table:

I have created a primary key on ConfigurationFeature and PackagePath – the table is created without any primary key.

Note for passwords the value stored in the config file will be *****. This is a place holder. You will need to change this value to the actual password and set security on the table to stop people reading it.



Wednesday, February 3, 2010

Error when using dtutil to deploy SSIS packages

BIDS Helper is deploying packages...

Deploying to SQL Server MSDB on server: NAMEOFSERVER

Error : BIDS Helper encountered an error when deploying package PACKAGENAME.dtsx!

"c:\Program Files\Microsoft SQL Server\90\DTS\binn\dtutil.exe" /FILE "PATHTOPACKAGE.dtsx" /DestServer NAMEOFSERVER /COPY SQL;"Package" /Q

exit code = 6

Microsoft (R) SQL Server SSIS Package Utilities
Version 9.00.4035.00 for 32-bit
Copyright (C) Microsoft Corp 1984-2004. All rights reserved.

Could not save package "PACKAGE" because of error 0x80004005.
Description: Cannot open database "msdb" requested by the login. The login failed.
Source: Microsoft SQL Native Client



Had to give the user's account access to msdb and the dts_admin and dts_operator roles.


Tuesday, October 6, 2009

Error connecting to SSIS remotely

Nearly all of this post comes from the blog listeed in the reference below. I have duplicated most of it here as I found it hard to find and I know I am going to come accross this error again.

As well as the notes below the user will also need to be added to the DCOM users group on the server.

When a user tries to connect to SSIS remotely they get an access is denied error. They can connect locally with no problems. Administrators on the remote machine can connect remotely and locally.











For Windows 2003 Server or Windows XP

1. If the user running under non-admin account it needs to be added to Distributed COM Users group
2. Go to Start - Run and type %windir%\system32\Com\comexp.msc to launch Component Services
3. Expend Component Services\Computers\My Computer\DCOM Config
4. Right click on MsDtsServer node and choose properties
5. In MsDtsServer Properties dialog go to Security page
6. Configure your settings as described below step 7
7. Restart SSIS Service

In the Security page we are interested in “Launch and Activation Permissions” section. Click Edit button to see “Launch Permissions” dialog.

“Launch Permissions” dialog allows you to configure SSIS server access per user/group. In the bottom of the dialog you can select:

• Local / Remote Launch permissions if you allow a user/group to start service locally or remotely
• Local / Remote Activation permissions if you allow to a user/group to connect to SSIS server locally or remotely.

Remote Access:
By default low privileged users can only connect to SSIS Server on the local machine when the service already started. It is shown by the fact that only Local Activation checked for Machine\Users group. To grant the user permission connect to the running server remotely you need to check remote activation.

Reference:
http://deepakrangarajan.blogspot.com/2008/03/connecting-to-integration-service-on.html

Monday, September 14, 2009

Space related commands

Database Files
This script runs against the database you are connected to. This will show the total size of a database file along with the available space left in the file. This is for all the files that make up the database including the log file.

SELECT name AS 'File Name' , physical_name AS 'Physical Name', size/128 AS 'Total Size in MB',
size/128.0 - CAST(FILEPROPERTY(name, 'SpaceUsed') AS int)/128.0 AS 'Available Space In MB'
FROM sys.database_files;

Log Files
This runs against every log file in the database giving the database name, Log Size in MB, Log Space Used (%), and Status.

DBCC SQLPERF (LOGSPACE);

DBCC SQLPERF: http://msdn.microsoft.com/en-us/library/ms189768.aspx

Free Drive Space
Shows the name of the drives and how many MB of free space it has. It does not show how large the drive is – or if there is any compression on it etc. Good to use if you a quick check to see if there is space to run a backup up.

EXEC master..xp_fixeddrives

Database Space Used
Replace tempDB with the name of the database you want to querey. Shows the database name (although you know that already), the size of the database in MB, percentage of unallocated space. It also show: total amount of space allocated by objects in the database, total amount of space used by data, total amount of space used by indexes, Total amount of space reserved for objects in the database, but not yet used.

EXEC tempDB.dbo.sp_spaceused;

You can also use this to find the size of an object:
EXEC sp_spaceused N'NameofObject';

sp_spaceUsed: http://msdn.microsoft.com/en-us/library/ms188776.aspx

SSIS Script to show disk data on all servers populated from a table list of servers:
I have a management server with a list of servers – for this I use columns isAccessible and isSSPI. I do the isSSPI bit because this normally means I can run the WMI query due to the way the servers have been set up.

Imports System
Imports System.Data
Imports System.Data.SqlClient
Imports Microsoft.SqlServer.Dts.Runtime
Imports System.Collections
Imports System.Management
Imports System.IO

Public Class ScriptMain
    Dim sConn As String = "server=.;database=ServerAudit;Integrated Security=SSPI;max pool size=100;min pool size=1;"
    Dim serverNameString As String = String.Empty
    Dim serverIPString As String = String.Empty
    Public Sub Main()
        Call getServerRecordSet()
        Dim serverNameArray As String() = Split(serverNameString, ",")
        Dim serverIPArray As String() = Split(serverIPString, ",")
        Dim serverHDD As String = String.Empty
        Dim x As Integer = 0
        For Each Server As String In serverNameArray
            serverHDD = getDataDiskDrives(Server, serverIPArray(x))
            x += 1
            Dim aResultSet As String() = Split(serverHDD, "/*/")
            For Each sDisk As String In aResultSet
                Try
                    If Len(sDisk) > 1 Then
                       ' MsgBox("sending to db: " & sDisk)
                        sendDatatoDB(sDisk)
                    End If
                Catch ex As Exception
                    ' MsgBox("DB Failed: " & ex.Message)
                End Try
            Next
        Next
        Dts.TaskResult = Dts.Results.Success
    End Sub

    Public Function getDataDiskDrives(ByVal server As String, ByVal ipAddress As String) As String
        Dim query1 As ManagementObjectSearcher
        Dim queryCollection1 As ManagementObjectCollection
        Dim HDDSize As String = String.Empty
        Dim thisDrive As String = String.Empty
        Try
            query1 = New ManagementObjectSearcher("\\" + ipAddress + "\root\cimv2", "SELECT * FROM Win32_DiskDrive")
            queryCollection1 = query1.Get()
            For Each mo As ManagementObject In queryCollection1
                Dim HddKB As Long = 0
                Dim Hddletter As String = String.Empty
                HddKB = Convert.ToInt64(mo("size"))
                For Each b As ManagementObject In mo.GetRelated("Win32_DiskPartition")
                    For Each c As ManagementBaseObject In b.GetRelated("Win32_LogicalDisk")
                        Hddletter = Left(Convert.ToString(c("Name")), 1) & "," & (Convert.ToInt64(c("Size")) / 1073741824).ToString & "," & (Convert.ToInt64(c("FreeSpace")) / 1073741824).ToString
                    Next
                Next
                If Len(Hddletter) > 1 Then
                    thisDrive = server & "," & Hddletter & "/*/"
                    HDDSize += thisDrive
                End If
            Next
        Catch ex As Exception
            'MsgBox(ex.Message)
            HDDSize = String.Empty
        End Try
        getDataDiskDrives = HDDSize
    End Function

    Sub sendDatatoDB(ByVal dataArray As String)
        Dim SqlCommand As SqlCommand = New SqlCommand
        SqlCommand.CommandText = "[uspAddHDDItem]"
        SqlCommand.CommandType = CommandType.StoredProcedure
        SqlCommand.Connection = New SqlConnection(sConn)
        Dim DriveLetter As String
        Dim DriveSize As String
        Dim DriveSpace As String
        Dim serverName As String
        Dim resultSet As String() = Split(dataArray, ",")
        serverName = Trim(resultSet(0))
        DriveLetter = Trim(resultSet(1))
        DriveSize = Left(Trim(resultSet(2)), 7)
        DriveSpace = Left(Trim(resultSet(3)), 7)
        If InStr(DriveSize, ".") > 1 Then
            DriveSize = Left(DriveSize, InStr(DriveSize, ".") + 2)
        End If
        If InStr(DriveSpace, ".") > 1 Then
            DriveSpace = Left(DriveSpace, InStr(DriveSpace, ".") + 2)
        End If
        Dim server As SqlParameter = New SqlParameter("@server ", SqlDbType.VarChar, 25)
        Dim letter As SqlParameter = New SqlParameter("@letter", SqlDbType.Char, 1)
        Dim size As SqlParameter = New SqlParameter("@size", SqlDbType.VarChar, 7)
        Dim space As SqlParameter = New SqlParameter("@free", SqlDbType.VarChar, 7)
        server.Value = serverName
        letter.Value = DriveLetter
        size.Value = DriveSize
        space.Value = DriveSpace
        server.Direction = ParameterDirection.Input
        letter.Direction = ParameterDirection.Input
        size.Direction = ParameterDirection.Input
        space.Direction = ParameterDirection.Input
        SqlCommand.Parameters.Add(server)
        SqlCommand.Parameters.Add(letter)
        SqlCommand.Parameters.Add(size)
        SqlCommand.Parameters.Add(space)
        SqlCommand.Connection.Open()
        Try
            SqlCommand.ExecuteNonQuery()
        Catch ex As Exception
            MsgBox("DriveSpace: " & DriveSpace & " DriveSize " & DriveSize)
        End Try
        SqlCommand.Connection.Close()
        SqlCommand.Connection.Dispose()
    End Sub

    Private Sub getServerRecordSet()
        Dim sql As String = "SELECT DISTINCT ipAddress, serverName FROM servers WHERE (isAccessible = 1) AND ipAddress is not null and domainID = 1"
        Dim conn As SqlConnection = New SqlConnection(sConn)
        Dim cmd As SqlCommand = New SqlCommand(sql, conn)
        cmd.Connection.Open()
        Dim r As SqlDataReader = cmd.ExecuteReader(CommandBehavior.CloseConnection)
        While r.Read()
            serverNameString += r("serverName").ToString() & ","
            serverIPString += r("ipAddress").ToString() & ","
        End While
        r.Close()
        conn.Close()
        conn.Dispose()
        serverNameString = Left(serverNameString, Len(serverNameString) - 1)
        serverIPString = Left(serverIPString, Len(serverIPString) - 1)
    End Sub
End Class

This is the Stored Proc for Importing the data. From this you can work out the table schema – I delete data in the table first and use server as a primary key. You could keep it and add an update data column – and change the Primary Key.
CREATE PROCEDURE [dbo].[uspAddHDDItem]
    @server as Varchar(25),
    @letter as char(1),
    @size as Varchar(7),
    @free as Varchar(7)
AS
BEGIN
SET NOCOUNT ON;
    INSERT INTO HDDAudit(diskLetter,DiskSize,FreeSpace,ServerName)
    VALUES(@letter,Cast(@size as Decimal(7,2)),Cast(@free as Decimal(7,2)),@server);
END;