A method to compare blobs in different Azure blob storage accounts

I was recently asked to run a comparison of the blobs in two Azure Storage Accounts, where they should contain the same blob entries. However, some of the entries had been deleted from one account. Since it requires scanning the whole account it is expensive in cost and time. The fastest mechanism I found (and I’m sure there are better ways, please comment) was;

var blobsInBackupContainer = backupContainer.ListBlobs(useFlatBlobListing: true).OfType<CloudBlockBlob>().Select(b => b.Name);
var blobsInPrimaryContainer = primaryContainer.ListBlobs(useFlatBlobListing: true).OfType<CloudBlockBlob>().Select(b => b.Name);                

var missing = blobsInBackupContainer.Except(blobsInPrimaryContainer);
Console.WriteLine($"Missing count {missing.Count()}");
foreach (var item in missing)
{
    Console.WriteLine($"Missing item = {item}");
}
Advertisements
Posted in Uncategorized | Leave a comment

How to encode a date condition in Azure Data Factory

I keep forgetting the correct format; here is an example;

where created_date >= \\'2017-05-11\\'

Posted in Azure, Uncategorized | Leave a comment

How to remove a search term/url from Edge

The other day I wanted to lookup the price of a PlayStation in France. I entered amazon.fr into Edge and found the details – all was good. However, now whenever I start to type ‘am..’ into Edge it keeps trying to ‘help’ me auto-filling in amazon.fr again. Grrr. The way I managed to stop Edge doing this is to hack the values out of the Registry. As with any Registry changes you must be careful and do this at your own risk;

    Close Edge
    Open Registry Editor: Windows->Run. Regedit
    Select the first node
    Edit->Find…
    Type in the core part of the url you want to remove, e.g. amazon.fr
    Find
    Look at the path (in the status bar at the bottom). Delete each instance your find that has any path related to Edge
    Repeat by pressing F3 once after each delete
Posted in Uncategorized | Leave a comment

Monitoring the progress of CreateIndex

One of the frustrating aspects of SQL Server is the lack of progress indicators. One such command falling into this pit is CreateIndex. However, this is a great link for a solution; http://sqlblog.com/blogs/michael_zilberstein/archive/2013/10/21/51415.aspx

Step 1 – Before you run the following query make sure you have ‘include actual execution plan’ option set in query analyser. If you don’t have this, or similar, then the DMV won’t produce any results.

Step 2 – Select @@SPID

Step 3 – run your Create Index in the same query window as step 2

Step 4 – Create a new query with the following SQL, replace the SPID in the first line with that from step (2)

DECLARE @SPID INT = 51;
;WITH agg AS
(
SELECT SUM(qp.[row_count]) AS [RowsProcessed],
SUM(qp.[estimate_row_count]) AS [TotalRows],
MAX(qp.last_active_time) – MIN(qp.first_active_time) AS [ElapsedMS],
MAX(IIF(qp.[close_time] = 0 AND qp.[first_row_time] > 0,
[physical_operator_name],
N'<Transition>’)) AS [CurrentStep]
FROM sys.dm_exec_query_profiles qp
WHERE qp.[physical_operator_name] IN (N’Table Scan’, N’Clustered Index Scan’, N’Sort’)
AND   qp.[session_id] = @SPID
), comp AS
(
SELECT *,
([TotalRows] – [RowsProcessed]) AS [RowsLeft],
([ElapsedMS] / 1000.0) AS [ElapsedSeconds]
FROM   agg
)
SELECT [CurrentStep],
[TotalRows],
[RowsProcessed],
[RowsLeft],
CONVERT(DECIMAL(5, 2),
(([RowsProcessed] * 1.0) / [TotalRows]) * 100) AS [PercentComplete],
[ElapsedSeconds],
(([ElapsedSeconds] / [RowsProcessed]) * [RowsLeft]) AS [EstimatedSecondsLeft],
DATEADD(SECOND,
(([ElapsedSeconds] / [RowsProcessed]) * [RowsLeft]),
GETDATE()) AS [EstimatedCompletionTime]
FROM   comp;
Posted in SQL Server, Uncategorized | Leave a comment

Problem shrinking tempdb – could not be moved because it is a work table page

SQL holds a lot of cached information in tempdb. Until it’s cleared it won’t allow tempdb to be shrunk. So here’s a quick script you might have some joy with (obviously be aware that you will suffer the cost of losing this cached information) (thanks to MSDN);

USE [tempdb]

GO

DBCC DROPCLEANBUFFERS

GO

DBCC FREEPROCCACHE

GO

DBCC FREESESSIONCACHE

GO

DBCC FREESYSTEMCACHE ( ‘ALL’)

GO

DBCC SHRINKDATABASE (tempdb,10)

GO

 

Posted in SQL Server, Uncategorized | Leave a comment

How to check the progress of SQL Server Shrink Database

Periodically run;

SELECT GetDate(),percent_complete, start_time, status, command, estimated_completion_time, cpu_time, total_elapsed_time
FROM sys.dm_exec_requests where command = ‘DbccFilesCompact’

Posted in SQL Server, Uncategorized | Leave a comment

How to Backup SQL to Azure Storage

SQL Server 2016 allows you to backup a database to Azure Storage. I found the information about this to be disjoint at best. So, if you want to do an old style (non-snapshot) backup of a database this is how I achieved it;

Background

My Server is running on a Virtual Machine in on the same Azure Resource Group as the storage account, I’m not too concerned about SQL knowing the storage account keys, so I’m going to show the shortest path to getting this working

Steps

Assume we’re backing up a Database called Products.

  1. Create Storage Account – Create a ‘standard’ non premium storage account. Don’t be tempted to use premium, it will not work. Let’s call it sqlnoprembackup
  2. Create a container in the blob storage, perhaps ‘backupsql1’. Grab the full path to the container, if you’re using the portal select the container and view the properties. In this example it is; ‘https://sqlnoprembackup.blob.core.windows.net/backupsql1
  3. Grab the account name and key for the storage account (Access Keys if you’re using the portal). Here it is sqlnoprembackup and the first long key.
  4. Create the certificate on SQL Server, let’s call it backupstorage;
  5. IF NOT EXISTS 
    (SELECT * FROM sys.credentials  
    WHERE name = ‘backupstorage’) 
    CREATE CREDENTIAL [backupstorage] WITH IDENTITY = ‘sqlnoprembackup’ 
    ,SECRET = ‘<insert secret storage key from (3) here>’; 

  6. Backup the database, use the full path to the container from (2). Note stats=5 will display the % progress at 5% internals in the messages tab;
  7. BACKUP DATABASE Products
    TO URL = ‘https://sqlnoprembackup.blob.core.windows.net/backupsql1/products.bak’  
    WITH CREDENTIAL = ‘backupstorage’  
         ,COMPRESSION 
         ,STATS = 5; 

  8. Run it. Yay.

    Restoring the database

    To do a full restore of the database;

  1. Take the database offline;
    ALTER DATABASE Products SET OFFLINE WITH NO_WAIT
  2. To restore the database;
  3. RESTORE DATABASE Products FROM URL = ‘https://sqlnoprembackup.blob.core.windows.net/backupsql1/products.bak’  
    WITH CREDENTIAL = ‘backupstorage’, REPLACE, STATS=5 

    NB, take a look at Snapshot backups, it’s a much better solution. Sometimes, as in my case, it’s not available and this hybrid approach works quite nicely.
    Posted in Azure, SQL Server | Tagged | Leave a comment