More on high performance data loading

Today I stumbled across what is possibly the most comprehensive, useful page on the net about bulk loading data into SQL Server 2008. While it is packed with useful information, one of the pearls is about trace flag 610.

In a nutshell, trace flag 610 instructs the server to minimally log all operations that can be minimally logged. In practical terms, this means that MERGE and INSERT operations will minimally log new page allocations. Without trace flag 610, these operations are fully logged. The link (above) goes into great detail about the usage of trace flag 610, so I'll simply summarize as follows:

DBCC TRACEON (610);               -- enables trace flag 610
MERGE INTO xxx USING yyy ON       -- do minimally logged upsert
DBCC TRACEOFF(610);               -- restore normal mode.

Note that setting trace flag 610 will affect all connections to the server until it is reset, so avoid using it on main OLTP servers. For a usable T-SQL MERGE example, see Sunil's blog posting.

Comments

Reading Zip files in PowerQuery / M

Being a fan of PowerBI, I recently looked for a way to read zip files directly into the Data Model, and found this blog which showed a usable technique. Inspired by the possibilities revealed in Ken's solution, but frustrated by slow performance, I set out to learn the M language and write a faster alternative. UnzipContents The result of these efforts is an M function - UnzipContents - that you can paste into any PowerBI / PowerQuery report. It takes the contents of a ZIP file, and returns a list of files contained therein, along with their decompressed data: If you're not sure how to make this function available in your document, simply: Open up PowerQuery (either in Excel or in PowerBI) Create a new Blank Query. Open up the Advanced Editor (found on the View tab in PowerBI). Copy-Paste the above code into the editor, then close the editor. In the properties window, rename the the function to UnzipContents Usage Using the function is fairly straight for...

Power Query: Transforming YYYYMM dates (the quick way)

Accountants. Their unit of work seems to be the month, as if individual days don't exists, or don't count somehow. Nowhere is this better seen than in the notion of the accounting period , which all too often follows the form YYYYMM. Try converting this directly into a date and Power Query starts making excuses faster than a kid with his hand caught in the cookie jar. The quick solution to this is to understand what Power Query's Table.TransformColumns does, and then leverage this knowledge to transform your YYYYMM values into proper date type columns. Table.TransformColumns As it's name suggests, this handy function allows you to convert the contents of a column from one type to another. The basic syntax is: = Table.TransformColumns( #"Your Source Table", { A list of tuples, specifying the columns and what functions to apply to each value} ) Lists {denoted by curly braces} are something you need to get comfortable with if you...

Loading data from an Oracle source

Building Business Intelligence using the Microsoft BI stack in an organization that has an Oracle based transactional system is not uncommon, and here I'll outline a couple of tips and tricks that should ease the building of SSIS packages in that type of environment. Attunity Components Attunity offer several nice solutions for getitng data out of Oracle and into SQL - one of which is their CDC (Change Data Capture) offering . If you don't have the budget or stomach for setting up CDC in an oracle environment, then your next best bet is to use the free Attunity oracle connectors for SSIS , which demonstrate a measurable performance boost. The Oracle Index and Order By Gotcha In previous posts I mention the performance benefits of loading sorted data into target tables. I'm currently loading data from Oracle 10, which exhibits a rather strange characteristic... Oracle does not use make us of an index for ordering data . In other words, Oracle only uses ...

Mark White's BI Blog

Search This Blog