12/24/13

SQL Bulk copy method to insert large amount of data to the sql database - CodeProject
10,284,208 members (59,050 online) Sign in

home

articles

quick answers

discussions

features

community

help

Search for articles, questions, tips

Articles Database Database General

Next

Tip Browse Code Stats Revisions (2) Alternatives (1) Comments & Discussions (11)

SQL Bulk copy method to insert large amount of data to the sql database
By bluesathish, 3 Jan 2012
Sign Up to vote 5.00 (5 votes) Tweet
5

About Article
Fastest and most efficient method to insert large amount of records to a SQL Server database from our system generated datatable. Type Licence First Posted Views Bookmarked Tip/Trick CPOL 3 Jan 2012 39,485 20 times

I was recently tasked with a project at a company to update an SQL Server 2008 database with large amounts of data each day. The task at first seemed daunting due to the files exceeding well over 400,000 records and there were several that needed processing daily. I first tried LINQ to SQL, but with the amount of data, the inserts were slow performing to say the least. Then I remembered the S q l B u l k C o p y class. S q l B u l k C o p y lets you efficiently bulk load a SQL Server table with data from another source. The S q l B u l k C o p yclass can be used to write data only to SQL Server tables. However, the data source is not limited to SQL Server; any data source can be used, as long as the data can be loaded to a D a t a T a b l e instance or read with a I D a t a R e a d e r instance. For this example the file will contain roughly 1000 records, but this code can handle large amounts of data. To begin with lets create a table in SQL Server that will hold the data. Copy the following T-SQL into SQL Server to create your table:
Collapse | Copy Code

SQL VB.NET insert

Top News
Microsoft bets on Windows XP disaster
Get the Insider News free each morning.

C R E A T ET A B L E[ d b o ] . [ C e n s i s ] ( [ S u b u r b ][ v a r c h a r ] ( 2 0 0 )N U L L , [ N o t S t a t e d ][ i n t ]N U L L , [ N o t A p p l i c a b l e ][ i n t ]N U L L , [ F i s h i n g ][ i n t ]N U L L , [ M i n i n g ][ i n t ]N U L L , [ M a n u f a c t u r i n g ][ i n t ]N U L L , [ E l e c t r i c i t y ][ i n t ]N U L L , [ C o n s t r u c t i o n ][ i n t ]N U L L )O N[ P R I M A R Y ] G O

Related Videos

The table above will hold Census data that is freely available to download in Australia. The next item to do is create a console application that will bulk load the data. Open Visual Studio 2008 and choose File > New > Windows > Console Application. Before moving on, to explain the code I have to work backwards and explain the final method that bulk loads data. S q l B u l k C o p yhas a method called W r i t e T o S e r v e r . One of the overloads of this method takes a D a t a T a b l e as the parameter. Because a D a t a T a b l econtains rows and columns, this seemed like a logical choice for the task I was facing. Jumping back to the example we now know we need to create a D a t a T a b l e that contains the information from the text file. The code below demonstrates how to do this:
Collapse | Copy Code

D a t a T a b l ed t=n e wD a t a T a b l e ( ) ; s t r i n gl i n e=n u l l ; i n ti=0 ; u s i n g( S t r e a m R e a d e rs r=F i l e . O p e n T e x t ( @ " c : \ t e m p \ t a b l e 1 . c s v " ) ) { w h i l e( ( l i n e=s r . R e a d L i n e ( ) )! =n u l l ) { s t r i n g [ ]d a t a=l i n e . S p l i t ( ' , ' ) ; i f( d a t a . L e n g t h>0 ) { i f( i= =0 ) { f o r e a c h( v a ri t e mi nd a t a )

Related Articles
SQL Bulk copy method to insert large amount of data to the sql database Handling BULK Data insert from CSV to SQL Server SQL Bulk Copy with C#.Net Oracle to SQL Server VB.NET Utility application

www.codeproject.com/Tips/309564/SQL-Bulk-copy-method-to-insert-large-amount-of-dat

1/4

12/24/13

SQL Bulk copy method to insert large amount of data to the sql database - CodeProject
{ } i + + ; d t . C o l u m n s . A d d ( n e wD a t a C o l u m n ( ) ) ; How To Store Any File into SQL Database Using SQL bulk copy with your LINQ-to-SQL datacontext Retrieving failed records after an SqlBulkCopy exception SqlBulkCopy in ADO.NET 2.0, SqlBulkCopy Class in C#, SqlBulkCopy help, SqlBulkCopy guide, SqlBulkCopy class, SqlBulkCopy tutorial
Collapse | Copy Code

} VB.NET

} D a t a R o wr o w=d t . N e w R o w ( ) ; r o w . I t e m A r r a y=d a t a ; d t . R o w s . A d d ( r o w ) ; }

D i md tA sN e wD a t a T a b l e ( ) D i ml i n eA sS t r i n g=N o t h i n g D i miA sI n t e g e r=0 U s i n gs rA sS t r e a m R e a d e r=F i l e . O p e n T e x t ( " c : \ t e m p \ t a b l e 1 . c s v " ) l i n e=s r . R e a d L i n e ( ) D oW h i l el i n eI s N o tN o t h i n g D i md a t a ( )A sS t r i n g=l i n e . S p l i t ( " , " c ) I fd a t a . L e n g t h>0T h e n I fi=0T h e n F o rE a c hi t e mI nd a t a d t . C o l u m n s . A d d ( N e wD a t a C o l u m n ( ) ) N e x ti t e m i+ =1 E n dI f D i mr o wA sD a t a R o w=d t . N e w R o w ( ) r o w . I t e m A r r a y=d a t a d t . R o w s . A d d ( r o w ) E n dI f l i n e=s r . R e a d L i n e ( ) L o o p E n dU s i n g

Bulk CRUD Operation using XQuery and Reflection in ASP.NET Using SqlBulkCopy with ASP.NET 2.0 Transfer data or script between two SQL Server databases Very fast test data generation using exponential INSERT Bulk INSERT / UPDATE / DELETE in LINQ to SQL Using SQL Bulk Copy with your LINQ-to-Entities datacontext Part 1 Generate SQL Insert statements for your SQL Server 2000 Database Database performance optimization part 2 (Index maintenance) Generating INSERT statements in SQL Server SQL Server Interview Questions and Answers Complete List Download SQL Server 2000 - Merge Replication Step by Step Procedure Fast Exporting from DataSet to Excel

In the code above, I created a D a t a T a b l ethat will store all the information from the csv file. The CSV file resides in the C:\Temp directory. I am using a S t r e a m R e a d e robject to open the file and read each line in the file. Each line is then split up into a string array. That string array will be assigned to each D a t a R o was the I t e m A r r a y value. This sets the values for the row through the array. When the file has been read, the next thing to do is use the S q l B u l k C o p yclass to insert the data into SQL Server. The following code demonstrates how to do this:
Collapse | Copy Code

u s i n g( S q l C o n n e c t i o nc n=n e wS q l C o n n e c t i o n ( C o n f i g u r a t i o n M a n a g e r . C o n n e c t i o n S t r i n g s [ " C o n s o l e A p p l i c a t i o n 3 . P r o p e r t i e s . S e t t i n g s . d a a s C o n n e c t i o n S t r i n g " ] . C o n n e c t i o n S t r i n g ) ) { c n . O p e n ( ) ; u s i n g( S q l B u l k C o p yc o p y=n e wS q l B u l k C o p y ( c n ) ) { c o p y . C o l u m n M a p p i n g s . A d d ( 0 ,0 ) ; c o p y . C o l u m n M a p p i n g s . A d d ( 1 ,1 ) ; c o p y . C o l u m n M a p p i n g s . A d d ( 2 ,2 ) ; c o p y . C o l u m n M a p p i n g s . A d d ( 3 ,3 ) ; c o p y . C o l u m n M a p p i n g s . A d d ( 4 ,4 ) ; c o p y . D e s t i n a t i o n T a b l e N a m e=" C e n s i s " ; c o p y . W r i t e T o S e r v e r ( d t ) ; } }

VB.NET

Collapse | Copy Code

U s i n gc nA sN e wS q l C o n n e c t i o n ( C o n f i g u r a t i o n M a n a g e r . C o n n e c t i o n S t r i n g s ( _ " C o n s o l e A p p l i c a t i o n 3 . P r o p e r t i e s . S e t t i n g s . d a a s C o n n e c t i o n S t r i n g " ) . C o n n e c t i o n S t r i n g ) c n . O p e n ( ) U s i n gc o p yA sN e wS q l B u l k C o p y ( c n ) c o p y . C o l u m n M a p p i n g s . A d d ( 0 ,0 ) c o p y . C o l u m n M a p p i n g s . A d d ( 1 ,1 ) c o p y . C o l u m n M a p p i n g s . A d d ( 2 ,2 ) c o p y . C o l u m n M a p p i n g s . A d d ( 3 ,3 ) c o p y . C o l u m n M a p p i n g s . A d d ( 4 ,4 ) c o p y . D e s t i n a t i o n T a b l e N a m e=" C e n s i s " c o p y . W r i t e T o S e r v e r ( d t ) E n dU s i n g E n dU s i n g

S q l B u l k C o p yuses ADO.NET to connect to a database to bulk load the data. I have created an S q l C o n n e c t i o n object, and that object reference is used to create the S q l B u l k C o p yobject. The D e s t i n a t i o n T a b l e N a m e property references a table in the database where the data is to be loaded. A handy feature of S q l B u l k C o p yis
www.codeproject.com/Tips/309564/SQL-Bulk-copy-method-to-insert-large-amount-of-dat 2/4

12/24/13

SQL Bulk copy method to insert large amount of data to the sql database - CodeProject
the S q l B u l k C o p y C o l u m n M a p p i n g C o l l e c t i o n . Column mappings define the relationships between columns in the data source and columns in the destination. This is handy if the data source file has columns that dont need to be inserted into the database. Column mappings can be set by an index, such as the example above, or they can be set by the name of the column. Using the index is handy when youre working with files that contain no column names. Make sure both of your datatable and sqltable columns are in the same order. Finally the data is sent to the database by running the W r i t e T o S e r v e rmethod. Hence using s q i b u l k c o p y ( )method is a very fastest than any other insertion method.

License
This article, along with any associated source code and files, is licensed under The Code Project Open License (CPOL)

About the Author

bluesathish
Software Developer India

No Biography provided Article Top

Comments and Discussions

You must Sign In to use this message board. Search this forum Profile popups Spacing Relaxed Noise Very High Layout Normal Per page 10 Go Update

First Prev Next

how about data type difference between data table and destination table What if data is large and getting 'System.OutOfMemoryException' ERROR Re: What if data is large and getting 'System.OutOfMemoryException' ERROR Re: What if data is large and getting 'System.OutOfMemoryException' ERROR SqlBulkCopy from CSV to SQLServer with uniqueidentifier datatype for each item in data Re: for each item in data Re: for each item in data

Member 8331618

23-Jul-13 0:58

vishal_h

13-May-13 19:24

bluesathish

13-May-13 19:46

vishal_h

13-May-13 21:10

franco.fral

31-Jan-13 6:59

Sefavolon bluesathish Sefavolon

26-Sep-12 3:43 26-Sep-12 3:56 26-Sep-12 4:52

www.codeproject.com/Tips/309564/SQL-Bulk-copy-method-to-insert-large-amount-of-dat

3/4