Friday, August 7, 2015

Dtabase Overview

Database overview

A database is an organized collection of data.

DBMS:

DBMS refers to the Database Management System. It is a set of programs that enables you to store, modify, and retrieve the data from a database.

RDBMS:

RDBMS refers to the Relational Database Management System. It is a database management system that is based on the relational model as introduced by E. F. Codd. All modern database systems like MS SQL Server, Oracle, IBM DB2, MySQL, and Microsoft Access are based on RDBMS.

Difference between DBMS and RDBMS:

                              DBMS                           RDBMS
  1. 1. DBMS does not define any constraints or security to ensure the ACID PROPERTY.
  2. 2. Normalization concept is not present.
  3. 3. In DBMS data is treated as files internally.
  4. 4. DBMS does not support distributed databases.
  5. 5. DBMS supports single user.
  1. 1. RDBMS define the integrity constraint to ensure the ACID PROPERTY.
  2. 2. Normalization concept is present.
  3. 3. In RDBMS data is treated as tables internally.
  4. 4. RDBMS support distributed databases.
  5. 5. RDBMS supports multiple users.

- See more at: http://www.beginnerstutorialexamples.com/database-overview/#sthash.MQpn8kHd.dpuf

Thursday, August 6, 2015

SQL Server: Useful Metadata queries

Metadata queries are really helpful in discovering information for a given database schema. Database information including the tables, views, columns names, data types, indexes, and table constraints are all available using queries such as these.
During this tutorial, I want to explore some useful metadata queries.
Let us start by finding the list of tables created in the given database.
select    *
from      information_schema.tables
where     table_type='base table';

Now let us list the views created in the given database.
select    *
from      information_schema.tables
where     table_type='view';

Let us create a query that lists the column names, data types, whether the column allows null or not, and the maximum allowed characters in the row.
select    column_name, data_type, is_nullable, 
          character_maximum_length
from      information_schema.columns
where     table_name='emp';

This query shows the table name, object id, table creation date, and the last table modified time.
select    name, object_id, create_date, modify_date
from      sys.tables;

Listing the created indexes for a table with the column names is frequently required. In this query a.name is the table name for which you are listing the indexes. By removing the a.name condition, you can see all the created indexes in your database.
SELECT    a.name table_name,
          b.name index_name,
          d.name column_name
FROM      sys.tables a,
          sys.indexes b,
          sys.index_columns c,
          sys.columns d
WHERE     a.object_id = b.object_id
AND       b.object_id = c.object_id
AND       b.index_id = c.index_id
AND       c.object_id = d.object_id
AND       c.column_id = d.column_id
AND       a.name = 'emp';

This query will list the defined constraints on tables with the column names. In thie example, we can see the emp table’s unique, primary or foreign key constraints.
SELECT    a.table_name,
          a.constraint_name,
          b.column_name,
          a.constraint_type
FROM      information_schema.table_constraints a,
          information_schema.key_column_usage b
WHERE     a.table_name = 'EMP'
AND       a.table_name = b.table_name
AND       a.table_schema = b.table_schema
AND       a.constraint_name = b.constraint_name;

Suppose you want to write a ‘select count(1) from table_name’ query for each table in your database, but you have more than 100 tables in your database. Instead of writing a separate query for each table, you can generate those queries using SQL. Therefore, you can write SQL code to generate SQL.
SELECT   'select count(1) from [' + table_name + '];'
FROM     information_schema.tables;

Tuesday, August 4, 2015

SSIS Concepts

What is SQL Server Integration Services (SSIS) ?
SSIS Project Architecture ?
What are the main components of SSIS Project Architecture ?
Control Flow:
         Connection managers         Variables
          Parameters
          Package Configurations
                 XML Configuration
                 Environment Variable Configuration
                 Parent Package Configuration
                 Registry Entries
                 SQL Server Configurations
          Annotations
          Package Configurations
          Work Offline
          Logging
          Log Events
         Check Points
         Break Points
         Transactions
         SSIS Toolbox:
                      Favorites:
                             Dataflow Task
                             Execute SQL Task
                     Containers:
                             For Loop Container
                             Foreach Loop Container
                             Sequence Container
                             Task Host Container
                     Common:
                              Analysis Services Processing Task
                              Bulk Insert Task
                              Data Profiling Task
                              Execute Process Task
                              Execute Package Task
                              Expression Task
                              File System Task
                              FTP Task
                              Script Task
                              Send Mail Task
                              Web Service Task
                              XML Task
                   Other Tasks:
                              Analysis Services Execute DDL Task
                              Backup Database Task
                              CDC Control Task
                              Check Database Integrity Task
                              Data Mining Query Task
                              Execute SQL Server Agent Job Task
                              Execute T-SQL Statement Task
                              History Cleanup Task
                              Maintenance Cleanup Task
                              Message Queue Task
                              Notify Operator Task
                              Rebuild Index Task
                              Reorganize Index Task
                              Shrink Database Task
                              Transfer Database Task
                              Transfer Error Message Task
                              Transfer Jobs Task
                              Transfer Logins Task
                              Transfer Master Stored Procedure Task
                              Transfer SQL Server Objects Task
                              Update Statistics Task
                              WMI Data Reader Task
                              WMI Event Watcher Task
                 Control flow Properties:
                               Package Level Properties:
               
                                              Delay Validation:
                                              Propogate Property:
                                              RetainsameConnection Property:
                                              DisableEventHandler Property:
                                              Checkpoint FileName
                                              CheckPoint Usage:
                                              Configurations
                                               FailParentOnFailure
                                               IsolationLevel:
                                               LoogingMode:
                                               MaxConcurrentExecutables
                                               MaximumErrorCount
                                               PackagePassword
                                               ProtectionLevel
                                               SaveCheckPoints
                                               TransactionOption
                                Task level Properties:
                                         Data Flow Task Properties:
                                                     DefaultBufferMaxRows
                                                     DefaultBufferSize
                                                     DelayValidation
                                                     DisableEventHandlers
                                                     EngineThreads
                                                     FailPackageOnFailure
                                                     FailParentOnFailure
                                                     LoogingMode:
                                                     MaximumErrorCount
                                                     TransactionOption
                              Solution Explorere Properties:
                                        Solution File Properties:
                                                     Deploy:
                                                     Build:
                                                     Rebuild:
                                                     Convert to Package Deployment Model
                                               Configuration Properties:
                                                      Debugging:
                                                               Data Flow Optimizations:
                                                                          RunInOptimizedMode:
                                                               Debug Options:
                                                                          InteractiveMode
                                                                          Run64bitRuntime
                            Project Parameters
                            Connection Managers (Project Level)
                             SSIS Package Folder Properties:
                                             New SSIS Package
                                             SSIS Import and Export Wizard
                                             Convert Deployment Model
                                             Upgrade All Packages
                                             Add Existing Package
                                             Sort by Name
                            Miscellaneous
Data Flow:
            Favorites:
                  Source Assistant:
                 
                  Destination Assistant:
Parameters:
Event Handlers:
        Executables:
        Event Handler:
              OnError
              OnExecStatusChanged
              OnInformation
              OnPostExecute
              OnPostValidate
              OnPreExecute
              OnPreValidate
              OnProgress
              OnQueryCancel
              OnTaskFailed
              OnVariableValueChanged
              OnWarning
Package Explorer:
Execution Results:
Deployment:
Common Errors:
                                                         
                                                   


Saturday, August 1, 2015

What is a Package in SSIS?

A Package is a core object with in Sql Server Integration Services (SSIS). It contains

• Business logic
• Work flow elements
• Connections

          Business logic to handle the data extraction, manipulation, and transformation tasks needed to move data from one location to another location depends on requirement.
          Workflow elements involve running a stored procedure, moving a file from an FTP server to a destination folder on your server, or sending an email message when an error occurs. The work flow elements are in control flow.
          Connections to connect to different external systems such as databases, files, File Transfer Protocol (FTP) Servers, Simple Mail Transfer Protocol (SMTP) Servers. Connections are used for this SSIS data processing engine called as dataflow.

Finally the package is best parallels an executable program that maintains workflow and business logic. Simply say that a package is a collection of tasks snapped together to execute in an orderly fashion.






SSIS package


In my previous article I am trying to explain related to What is data warehousing. If you don’t read it please follow this link before going to this…


In this article I am trying to explain related to SSIS package.


A Package is the core object within SQL server Integration Services (SSIS) that contains the business logic to handle workflow and data processing. SSIS package can be used to move data from source to destinations and also handle the timing precedence of when thing process.

**BIDS [ Microsoft SQL Server Business Intelligence Development Studio ]

SSIS package can be accomplished by two ways.


Built-in wizard
By using the Built-in wizard in SQL Server 2005 that asks you to move the data from source to destination and automatically generate the SSIS package.


SSIS BIDS
By explicitly create a project in SSIS BIDS. We need to create projects the new package is automatically created and developed.
So we now trying to discuss about our first option and that is

By Built-in Wizard

In SQL Server 2005 we can use the Import and the Export Wizard to Import and Export the data. For Import Wizard the source is the SQL Server 2005 table and destination should be SQL Server database, ORACLE database, Flat file, Microsoft Excel spread sheet, Microsoft Access database.

Exporting data with the wizard lets us send the data from SQL Server 2005 tables, Views or custom query to flat file or database connection.

Initialize the Import Export Wizard

To initialize, please follow this steps mentioned bellow.

What we want to do

We want to import a flat file to our existing database.

1.    Through the SSMS connects to the installed database engine. That should be your source or destination.

2.    Click on view menu select Object Explorer (or press F8). From the database folder select the desired database. Then right click of the desired database and select Tasks. From Tasks we can select Import or Export wizard.




3.    Select the Tasks. If the database is source of data that needed to send out to the different system, select the “Export Data” and if the database is destination for the file currently exists outside the system, than select “Import Data”. Here is this example we are choosing “Import data”.

Database is source of data  
à Export Data

Database is destination for the file 
àImport Data
4.     If we choose any one the “Welcome to SQL Server Import Export Wizard” appears. Then click the next button on the wizard. “Choose the data source” allow you to specify from the data is coming from. Here in this example I am choosing Flat file source and brows the flat file. Please specify others options if needed.

“Choose a Destination” allow us to specify the destination where the data will be sending. We can choose the destination if needed. The server name and the security settings must be specified. If we select a relational database source that allow customer queries.


5.    For now in “Save and Execute” page of wizard we choose the options Execute Immediate for now. In the complete the wizard gives us all the information that we selected. If needed we can go back and modified it. Now use the SQL query to see the result output.
SELECT * FROM <table name>

In my next session we are discussing about saving and Editing Package created by wizard.

Hope you like it.

Tuesday, July 28, 2015

Installing SQL Server 2012 Analysis Services Tabular Mode


In this article, I will show you how to Install SQL Server 2012 Analysis Services(Tabular Mode).
1. First of all Run SQL Server 2012 Setup and go to Installation. Then click onNew SQL Server stand-alone installation or add features to an existing installation.
1-Installing SQL Server 2012 Analysis Services Tabular Mode
2. It will run Setup Support Rules.
2-Installing SQL Server 2012 Analysis Services Tabular Mode
Then click on OK button.
3. In next window click on Install button. It will again run Setup Support Rules.
3-Installing SQL Server 2012 Analysis Services Tabular Mode
If all rules run successfully, click on Next Button.
4. In next step, select Perform a new installation of SQL Server 2012 if you want to install new installation otherwise select Add features to and existing instance of SQL Server 2012.
Here we select 2nd option. Then click on Next button.
4-Installing SQL Server 2012 Analysis Services Tabular Mode
5. In next step, Select Analysis Services from Feature Selection. Then click on Next Button.
5-Installing SQL Server 2012 Analysis Services Tabular Mode
6. In next step, it will show you Disk usage Summary. Now click on Next Button.
6-Installing SQL Server 2012 Analysis Services Tabular Mode
7. In next step, select startup mode of SQL Server Analysis Services. Here we select Automatic mode. Then click on Next Button.
7-Installing SQL Server 2012 Analysis Services Tabular Mode
8. In next step, select Tabular Mode. Also specify users which have administrative permissions to access analysis services. Then click on OK button.
8-Installing SQL Server 2012 Analysis Services Tabular Mode
9. Now it will run Installation Configuration Rules.
9-Installing SQL Server 2012 Analysis Services Tabular Mode
Then click on Next Button.
10. Now all ready to Install Analysis Services .Click on Install Button.
10-Installing SQL Server 2012 Analysis Services Tabular Mode
11. After completion of installation, you will see screen like below.
11-Installing SQL Server 2012 Analysis Services Tabular Mode
Congratulations! We successfully completed installation of Analysis Services.

Parallel execution in SSIS



Parallel execution in SSIS improves performance on computers that have multiple physical or logical processors. To support parallel execution of different tasks in a package, SSIS uses two properties: MaxConcurrentExecutables and EngineThreads.

If SSIS runs on a dedicated server and you have a lot of operations that run in parallel, you will likely want to increase this setting if some of the operations do a lot of waiting for external systems to reply. On the other hand, if you do not have a dedicated SSIS machine and your data integration application runs alongside several other applications, you may need to reduce this setting to avoid resource conflicts.

The MaxConcurrentExecutables property is a property of the package. This property defines how many tasks can run simultaneously by specifying the maximum number of executables that can execute in parallel per package. The default value is -1, which equates to the number of physical or logical processors plus 2.

Please note that if your box has hyper threading turned on, it is the logical processor rather than the physically present processor that is counted.

The EngineThreads property is a property of each Data Flow task. This property defines how many threads the data flow engine can create and run in parallel. The EngineThreads property applies equally to both the source threads that the data flow engine creates for sources and the worker threads that the engine creates for transformations and destinations. Therefore, setting EngineThreads to 10 means that the engine can create up to ten source threads and up to ten worker threads. The default is 5 in SQL Server 2005 and 10 in SQL Server 2008, with a minimum value of 2.

One thing we want to be clear about EngineThreads is that it governs both source threads (for source components) and work threads (for transformation and destination components). Source threads and work threads are both engine threads created by the Data Flow’s scheduler.

One other thing to consider: If you are using the Execute Package Task, the child package to be executed can be run in-process or out-of-process by use of the ExecuteOutOfProcess property. If a child package is executed out-of-process, you will see another dtshost.exe process start. These processes will remain “live”, using up resources, for quite a while after execution is complete.

If executing in-process, a bug in a task of the child package will cause the master package to fail. Not so if executing out-of-process. On 32-bit systems a process is able to consume up to 2GB of virtual memory. Executing out-of-process means each process can claim its own 2GB portion of virtual memory. Therefore if you are simply using many packages to structure your solution in a more modular fashion, executing in-process is probably the way to go because you don’t have the overhead of launching more processes.

A thread will process one buffer at a time, executing it against all transforms in the execution tree before working on the next buffer in the flow, at which point it would pass the current buffer to another thread executing another execution tree and it would pull a new data buffer from its buffer list which was queued from an upstream component (either a data source or the last asynchronous transform before this execution tree started).

However, the general rule is to not run more threads in parallel than the number of available processors. Running more threads than the number of available processors can hinder performance because of the frequent context-switching between threads.

MaxConcurrent

·         This is a property on the ForEachLoop which says how many instances of the loop contents can be run in parallel.

Example:

Suppose we have a package with 3 Data Flow Tasks. Each task has 10 flows in the form of “OLE DB Source -> SQL Server Destination”.

Set MaxConcurrentExecutables to 3, then all 3 Data Flow Tasks will run simultaneously.



Now whether all 10 flows in each individual Data Flow Task get started concurrently is a different story. This is controlled by the second property: EngineThreads.



The EngineThreads is a property of the Data Flow Task that defines how many work threads the scheduler will create and run in parallel. Its default value is 5.  

If we set EngineThreads to 10 on all 3 Data Flow Tasks, then all the 30 flows will start off at once.

Logging details from sysssislog


select Package,
source,StepId GroupId,
 STUFF(CONVERT(CHAR(8), DATEADD(SECOND, ABS(TimeInMinutes), '19000101'), 8), 1, 2, CAST(TimeInMinutes / 3600 AS VARCHAR(12))) Duration,
 TimeInMinutes TimeInSeconds,
 [Status]from (
 selectA.starttime, 
A.id, 
c.source Package,
 A.[source],
 DENSE_RANK() OVER (order by A.executionID ) StepID,
 CONVERT(VARCHAR(100),DATEDIFF(SECOND,A.starttime,ISNULL(B.endtime,GETDATE()))) TimeInMinutes,
 CASE WHEN B.endtime IS NULL THEN ' (running)' ELSE '' END AS [Status]
from 
(select * from sysssislog where event = 'OnPreExecute') As A LEFT JOIN 
(select * from sysssislog where event = 'OnPostExecute') As B ON A.sourceid = B.sourceid and A.executionid = B.executionid
LEFT JOIN (select * from 
(select source,executionid,Row_Number() over (PARTITION by executionid order by executionid) RowId from sysssislog where event = 'OnPreExecute' )
 where RowId=1) C on C.source=A.source and C.executionid=a.executionid
) t 
order by StepID,id