Jakew
Consulting, hacking, and motorcycles

Prototyping some mobile goodness

Friday, 2 December 2011 18:04 by jakew

Been an insane year – might be getting back to blogging. Ha…as if.

So I have an idea for a mobile application that uses GPS. Before I invest time in building the real thing I’m going to prototype it on the web using html and javascript. Who knows…maybe that will be good enough to go all the way.

First thing to learn is how to get the phone’s location. Initially I thought this would require real programming (everybody knows real programming is done with a real language like C,C++,C# or Java not one of those baby languages like VB, Javascript or *gag* Python). However, it turns out I was wrong. Those nice people at W3C have already specified that the browser should have this nice geolocation object that offers methods for either getting the location (call once) or watching the position (keeps calling back with the location). Sweet!

Anyway this nice person wrote a decent sample you can look at here. You’ll need to provide two div (output & location) to show information. I killed the geturl stuff and just threw the lat & long on my page. My initial application just needs to know where the phone is, there isn’t any mapping.

The next piece I’m working on, just for fun, uses Google gears to go ahead and show the map. I’m still working on that step.

You can see my progress here.

Once I’m done I’ll provide my prototyping code. Should have what I’m after by the end of the weekend.

Categories:  
Actions:   E-mail | del.icio.us | Permalink | Comments (0) | Comment RSSRSS comment feed

Broadening horizons: Objectified

Monday, 30 May 2011 11:32 by jakew

Right now I’m working on a project that has provided me another opportunity to work with a Usability Engineer. He’s pretty damn good at and it’s a lot of fun watching him work. Although his documentation makes mine look like crap in comparison. Jerk.

Anyway, I forget where I got the reference, but I just watch this documentary called Objectified. Three things I’d like to say about it. First, the documentary itself is well worth watching. It is an hour or so long, covers a lot and provides a bounty in food for thought. Second, Amazon’s Instant Video rocks. If you happen to subscribe to prime you can watch the movie for free right now. So subscribe to prime. Finally, it works great on the XOOM! Only complaint about the XOOM is that the speakers could be a lot better. Easily solved with a pair of headphones.

Anyway, if you’re a software developer and have to create UI’s give Objectified a watch.

The next thing I want is more material on Personas and Scenarios.

Categories:  
Actions:   E-mail | del.icio.us | Permalink | Comments (0) | Comment RSSRSS comment feed

Switching to Android

Friday, 27 May 2011 19:28 by jakew

I really like my Zune, cracked screen and all. 120Gb of storage and the Zune subscription is pretty darn good. However, new toys mean it is time to move on. Move on to Android.

Yesterday, I bought a Motorola XOOM (Wi-Fi only) to go with my DROID X. The DROID has a 32Gb card which as it turns out is plenty of room for the music I listen to and a month’s worth of RMS podcasts. Only problem: playlists. The playlist feature on the droid sorta stinks. Not that the Zune player does all that well.

So at this point it looks like I’ll be leaving my Zune at home. The phone will take over it’s job.

Another win for me is that having bought the XOOM I broke down and rooted the droid so that I can run Wireless Tether. Previously I was using PDANet via a USB cable. Now – just start Wireless Tether and my laptop and tablet have internet access. Oorah!

Next little win: Evernote. I’m still learning my way around it, but it does a great job letting me take notes on my phone, table, laptop or workstation and then keeping them all in sync. That’s pretty sweet in my opinion. I still have a huge amount of information stored in OneNote and don’t really expect that Evernote will replace OneNote, but it will do nicely for letting me take notes in a meeting and them be able to review them later.

And still more cool stuff – I have a NookColor and a few Kindles….Well, both are available for Android so my XOOM tablet can download both libraries. The NookColor is going to get rooted and have HoneyComb installed as soon as it is available. I want the regular Android marketplace on my Nook, not the lame B&N marketplace. Sorry guys….but you’re still kicking M$FT’s ass – you at least have a marketplace.

Finally, a little project for this weekend is going to be seeing if I can get a simple WPF application running on my XOOM. Should be an interesting project.

Categories:  
Actions:   E-mail | del.icio.us | Permalink | Comments (0) | Comment RSSRSS comment feed

A Dream come true: A windows App Marketplace

Friday, 27 May 2011 10:00 by jakew

Why? Why? Why doesn’t Microsoft have a marketplace for Windows? How friggin hard could it be? Those pointy toed shoe wearing dorks at Apple seems to have figured it out for their crummy phone. The morlocks at Google seem to be doing it pretty well for Android. Even that redheaded step child Windows Phone7 has a freaking marketplace.

Part of the success of Windows has always been developers and the huge number of applications available. What has really made the iPhone and Android phones successful is their marketplaces. The phones themselves are nice, but not really all that great. Until you can load apps, smart phones are pretty dumb.

For an entrepreneurial developer like myself I’m really attracted to the iPhone and Android platform because of their marketplaces. Those marketplaces represent an easily accessible distribution channel for my products. It means marketing is easier, meaning that the distance between my hard work and getting paid for it is that much shorter. I can tell prospective customers: go to the marketplace and buy my application.

However, I have mad skills on Windows. I can’t just tell the guy I meet at Starbucks to download my application from the Windows marketplace. I have to tell him to go to my web-site and get it there. You think he is going to remember my URL for more than 15 seconds? He’ll remember the Windows Marketplace. Not my URL. I don’t care if my URL was yomomma.com. He isn’t going to remember. He’ll remember Windows Marketplace or app store. End of story.

So what am I to do? Wish the Ballmer would just friggin retire, get BillG to quit screwing around and comeback to save his company. Or Microsoft needs to hire a technical person to run the show. Not another friggin MBA type. Somebody who has a vision that goes beyond the tip of his nose. And while they are at it: fire all those friggin directions and VPs from IBM that they have been hiring. They are killing it!

While I wait for that to happen: I’m going to start learning how to get .NET code to run on Android.

Categories:  
Actions:   E-mail | del.icio.us | Permalink | Comments (0) | Comment RSSRSS comment feed

It just keeps going

Sunday, 31 October 2010 00:22 by jakew

Really glad I’m not a professional blogger.

Last time I posted was back in July.  In the past 3 months I got myself a concussion on the race track, developed a huge amount of CRM integration code taken up shooting (both hand guns and rifles) and I’ve been helping a startup get their web-site going.

The little stunt on the race track has me off the bike until next year (I’m counting the days).  The concussion was pretty serious and having your wife have to meet the ambulance at the hospital is never good.  So for now I’m doing something much safer: working on my marksmanship.

If things will settle down I have a huge amount of stuff that needs to be written that I’d like to put here.  For the past 2 months I have been working 50-60 hour weeks.  Occasionally more.  But it looks like it will all slow down over the holidays so I’ll get a nice break.

Hope everybody else is having as much fun as I am.

Categories:  
Actions:   E-mail | del.icio.us | Permalink | Comments (0) | Comment RSSRSS comment feed

I give up – comments

Wednesday, 21 July 2010 21:28 by jakew

I thought I had the comment spam problem cured.  I thought wrong.

So no more comments.  Wanna comment: get your own blog, put your comment there and link to my post.  It will create a trackback that will show up over here.

Or hit me on my twitter feed – jakew there as well.

Categories:  
Actions:   E-mail | del.icio.us | Permalink | Comments (0) | Comment RSSRSS comment feed

Does the insanity ever stop?

Wednesday, 21 July 2010 21:19 by jakew

The past 3 months have been a freaking grinder.  I’m finishing up another CRM integration project right now and am just now getting to a point where I can see some day light again.

The really cool part: I know CRM’s API’s really well.  I’ve been writing plugins, workflows, workflow actions, WCF web-services.  I’ve been doing jQuery and getting fairly good with the whole AJAX thing.  Heck, I even get CSS sort of. 

Probably to coolest thing though is that I have a new code smith template nearly done.  This template when I finish it (there isn’t a lot left) basically creates a software factory that produces the integration service I built.  All you have to do is point it at your CRM server’s metadata service, fill in a few options, pick the entities you need to integrate with and the template generates a visual studio project with the framework for you to work with.  The CRM Dynamic Entity gets wrapped inside an entity that maps everything out.  Lookups are either the instance of the entity or collections of entities depending upon the lookup type.  Entity factories that load your entities based on whatever criteria you specify.  With this framework in place writing plug-ins or WF activities is a snap.

I’m almost done and should have time to get it finished starting next week.

Categories:  
Actions:   E-mail | del.icio.us | Permalink | Comments (0) | Comment RSSRSS comment feed

ADO.NET SqlBulkCopy

Tuesday, 30 March 2010 09:04 by jakew

I recently started hanging out on StackOverflow where I found a great question: Dealing with large number of text strings. My answer to the question is here. I’ve dealt with this type of problem in the past: A program that collects a huge pile of data, does some processing to the data and then stuffs the results in a database. In some ways you could solve the stuffing into the database using an ETL tool like SSIS. However, that would require dumping out a file and then firing up the SSIS package. Not hard, but I don’t like working that way.

The solution in my opinion is to use ADO.NET’s SQLBulkCopy. The only problem is that SQLBulkCopy wants you to setup a data source that closely matches the destination for the data. Not really a big deal but it does take a little work. A further issue is dealing with referential integrity. If the destination table has foreign keys that are requires you’ll need to fill in values for them.

To demonstrate this I used the Adventure Works database and created 2 programs to load up the products table. The first program creates as many products you want to test with and saves them to a CSV file. The second program reads the CSV file, sets up a DataTable and then loads it using SqlBulkCopy. To see how fast it is, the time is taken before and after the bulk copy.

You can grab the code here

Creating the data

Creating the data file isn’t a big deal. The only problem was picking out what fields I’d provide. The rest is just making all of the products.

The file layout

To create the file layout I first looked at the definition for the Product table in Adventure works to see what fields absolutely had to be provided. A few of the required fields could be calculated on the file (date modified for instance), others would just be hard coded to whatever value I chose. This is just a demo after all.

The layout I arrived at is:

  1. Name
  2. ProductNumber
  3. Color
  4. Category
  5. Cost
  6. Listprice
  7. Size
  8. Sellstartdate

As a matter of convention I went ahead and added the column names to the file header so somebody that picks up the file might have a chance of figuring out what I’ve done.

Creating the file

With the file layout decided on all I have to do is dump everything to a file:

       private void button1_Click(object sender, RoutedEventArgs e)
        {
            int numberOfItems = 0;
            string fileName = "";
            int.TryParse(txtProductsCount.Text, out numberOfItems);
            if(0 == numberOfItems)
            {
                return;
            }
            SaveFileDialog sfd = new SaveFileDialog();
            if(true == sfd.ShowDialog())
            {
                fileName = sfd.FileName;                
            }
            else
            {
                return;
            }

            labelStatus.Content = "Status:processing";
            this.Cursor = Cursors.Wait;
            if (true == File.Exists(sfd.FileName))
            {
                File.Delete(sfd.FileName);
            }
            StreamWriter writer = new StreamWriter(sfd.FileName);

            writer.WriteLine("Name,ProductNumber,Color,category,cost,listprice,size,sellstartdate");
            for(int counter = 0; counter < numberOfItems; counter++)
            {
                
                writer.WriteLine(string.Format("Name{0},Tst{1},Red,Brakes,1,2,sm,1.0,{2}"
                    ,counter, DateTime.Now.ToString("hhmmss")
                    ,DateTime.Now.ToString("yyyy-MM-dd")));
            }
            writer.Close();
            labelStatus.Content = "Status:idle";
            this.Cursor = Cursors.Arrow;
        }

Nothing terribly exciting here, but it does generate the test data I want.

Loading the data

The loader itself reads the CSV file and uses the data to create rows in a DataTable. This program is a little more interesting. The first problem is to setup a DataTable that matches the destination. Then you have to load the data and do any adjustments (transforms) to make sure it will load in to the table. Finally you use SQLBulkCopy to move the data in the table to the server.

Setting up the DataTable

SqlBulkCopy can use other data sources as long as they provide an IDataReader interface, but why go through the trouble when DataTable already does what you need? All you have to do is setup your DataTable to match the destination and then fill it up. To setup my DataTable I created a method to create columns for me:

        private static DataColumn GetColumn (string columnType, string columnName)
        {
            DataColumn newColumn = new DataColumn();
            newColumn.DataType = System.Type.GetType(columnType);
            newColumn.ColumnName = columnName;

            return newColumn;
        }

This function just creates a DataColumn using the provided type string and column name. This column is then added to the DataTable:

        private static DataTable CreateTable()
        {
            DataTable newProducts = new DataTable("Products");

            // Add the column objects to the table.                                              col   file  
            newProducts.Columns.Add(GetColumn("System.Int32", "ProductID"));                    //0     
            newProducts.Columns.Add(GetColumn("System.String", "Name"));                        //1     0
            newProducts.Columns.Add(GetColumn("System.String","ProductNumber"));                //2     1
            newProducts.Columns.Add(GetColumn("System.String","Color"));                        //3     2
            newProducts.Columns.Add(GetColumn("System.Decimal","StandardCost"));                //4     4
            newProducts.Columns.Add(GetColumn("System.Decimal","ListPrice"));                   //5     5
            newProducts.Columns.Add(GetColumn("System.String","Size"));                         //6     6
            newProducts.Columns.Add(GetColumn("System.Decimal","Weight"));                      //7
            newProducts.Columns.Add(GetColumn("System.Int32", "ProductCategoryID"));            //8     3
            newProducts.Columns.Add(GetColumn("System.Int32", "ProductModelID"));               //9
            newProducts.Columns.Add(GetColumn("System.DateTime", "SellStartDate"));             //10    7
            newProducts.Columns.Add(GetColumn("System.DateTime", "SellEndDate"));               //11
            newProducts.Columns.Add(GetColumn("System.DateTime", "DiscontinuedDate"));          //12
            newProducts.Columns.Add(GetColumn("System.Byte[]", "ThumbNailPhoto"));              //13
            newProducts.Columns.Add(GetColumn("System.String", "ThumbnailPhotoFileName"));      //14
            newProducts.Columns.Add(GetColumn("System.Guid", "rowguid"));                       //15
            newProducts.Columns.Add(GetColumn("System.DateTime", "ModifiedDate"));              //16

            return newProducts;
        }

The DataTable setup is really pretty easy. This matches up the columns and I added columns to help make it easier for me to remember which column in the file goes where in the DataTable.

Transforming the Data

Before actually loading the file I need to have a way to look up the product categories. For this I made a quick EntityFramework model of the AdventureWorks database and then wrote a method to lookup ProductCategories by name:

        private int GetCateogryByName(string categoryName)
        {
            ProductCategory category = (from cat in Categories
                                        where cat.Name == categoryName
                                        select cat).FirstOrDefault<ProductCategory>();
            if(null == category)
            {
                return -1;
            }

            return category.ProductCategoryID;
        }

This method just uses a simple LINQ statement to find a category whose name matches the one provided and then returns the ProductCateogryID. With all of this in place we can now load the file in to the DataTable:

        private int LoadFile(string fileName, DataTable productTable)
        {
            StreamReader reader = new StreamReader(fileName);

            try
            {
                bool done = false;
                int count = 0;
                reader.ReadLine(); //get rid of the header
                while (!done)
                {
                    string row = reader.ReadLine();
                    if (true == string.IsNullOrEmpty(row))
                    {
                        done = true;
                        break;
                    }
                    DataRow product = productTable.NewRow();
                    GetDataRow(row, product);
                    productTable.Rows.Add(product);
                    count++;
                }
                reader.Close();
                return count;
            }
            catch (Exception exc)
            {
                reader.Close();
                throw;
            }
        }

This has been a million times so it isn’t really worth talking about. Now that we have the data loaded we just call SqlBulkCopy:

        private void BulkLoadProducts()
        {
            SqlConnection connection = GetConnection();
            connection.Open();
            
            SqlCommand countCommand = new SqlCommand("SELECT COUNT(*) FROM SalesLT.Product", connection);
            long count = (int) countCommand.ExecuteScalar();
            listBox1.Items.Add("product count before:" + count.ToString());
            SqlBulkCopy bulkCopy = new SqlBulkCopy(connection);
            bulkCopy.DestinationTableName = "SalesLT.Product";
            DateTime start = DateTime.Now;
            bulkCopy.WriteToServer(ProductTable);
            DateTime end = DateTime.Now;
            TimeSpan diff = end.Subtract(start);
            connection.Close();
            listBox1.Items.Add(string.Format("took {0}ms", diff.Milliseconds));
        }

The magic really takes two three lines: first, create the SqlBulkCopy instance. Second tell it what table it will be loading. Third, call WriteToServer passing in the data. The other stuff in the method is just there to provide us some timing information. On my laptop running everything I’m able to load 1200 rows in about a third of a second. Running on a real server I’d imagine you’d be able to load data even faster.

SqlBulkCopy

For most of us we won’t be using SqlBulkCopy everyday. But on the rare occasions where you need to blast a ton of data in to a database this is a handy object to have around. You can read the MSDN docs here (http://msdn.microsoft.com/en-us/library/system.data.sqlclient.sqlbulkcopy.aspx). And David Hayden also has a good write up on SqlBulkCopy here (http://davidhayden.com/blog/dave/archive/2006/03/08/2877.aspx).

I hope you this helps you. Please let me know what you think or if you know of any better alternatives for this situation.

Categories:   Tech
Actions:   E-mail | del.icio.us | Permalink | Comments (1) | Comment RSSRSS comment feed

Azure Boot Camp Day 2 Notes

Wednesday, 24 March 2010 20:50 by jakew

bit slow getting these out.  Day 2 was great.  Service bus rocks and the opportunities that the cloud & Azure offer are really exciting.  Unfortunately by the end of the day I had 0 energy left and today was insanely busy so this took longer to get out than I expected.  Hope you enjoy it.  By the way: all of the materials for the boot camp are available hereSlide decks, Lab materials.

Thanks to Tracy Bazzell for leading the class.  Great job!

Building a business on Azure

The path:

· Train your people (duh)

· Learn the sales strategies

· Determine your offerings

· Understand how Azure bills

Know Azure (duh) –

· Azure bootcamp

· Development time – MSDN

Where can Azure overcome project challenges? (IT as an adoption blocker) Overcome the challenges: prototypes, startups, leverage no startup costs, cost is below managers spending limit

Nice azure features:

· Reduce project costs

· Improve time to delivery

· New capabilities

Where do the reduced costs come from? Cutting out the need to deploy infrastructure. This can be as much as 25% of a project’s total budget and time.

Determine your offering

Not all things to all people

Understand Billing

· Learn to use the TCO calculator

· Focus on managing services, not servers

· Customers want to shift CapEx -> OpEx

· Low risk, can’t guess wrong. Just delete the stuff and move on.

· Partners receive special pricing

Partner Opportunity

· You get a percentage of the customer’s ongoing monthly charges (18% for year 1, 6% thereafter)

· You aren’t in middle of the billing

· You add value on top of BPOS (MOSS customization, bespoke application developer, etc)

More benefits

· Internal use: 250 seats of BPOS for the first year free of charge, sell 2 deals w/ at least 25 seats

· Get up to 20 demo accounts

· CRM - $19/mo per seat up to 100 users

Basic Worker Roles

Worker role – focuses on backend process. No inbound processes. Basically a windows 2008 hyper V instance to do work. Normal use:

· Batch processing

· Queue processing

· Hosting non HTTP WCF services

· Number crunching

Local storage

· volatile storage. Gets wiped when the instance restarts or moves between servers.

· Can allocate up to 20Gb but cannot be resized.

Local resources are not the same as the XDrive thing you can mount to an instance. Local storage basically give you the ability to work with temp files. First you configure the storage item:

<LocalResources>

<LocalStorage name="Storage1" cleanOnRoleRecycle="false" sizeInMB="5" />

</LocalResources>

Keep in mind that once you’ve hit 5mb in Storage1 you are done. It will not grow. However, don’t go off setting it to 20Gb because you’re going to pay for it. Figure out how much room you really need and go with that. To work with the file you’ll do the following:

LocalResource storage = RoleEnvironment.GetLocalResource("Storage1");

string filePath = Path.Combine(storage.RootPath, "tempfile.txt");

StreamWriter writer = new StreamWriter(filePath);

writer.Write("test data");

writer.Close();

Pretty easy stuff. Just keep in mind that when your process recycles the file will be gone, if the app fabric decides to move your service the file will be gone. Simply put: do not expect that file to be there. Although you might want to check first. :*)

Endpoints for Worker processes

Exposes WCF services to the internet. These end points get enrolled with the Azure load balancer. You can use HTTP, HTTPS and TCP and open as many as you like. You can also open internal WCF endpoints using other protocols. The internal endpoints do not get enrolled with the load balancer.

You must configure the end points in your service’s configuration file. You cannot dynamically open up new end points.

The WCF end points are just WCF so you need to know how to program against WCF’s API.

The sample application includes a chat application that runs in a worker role. The azure specific stuff is pretty easy:

Threads in Workers

You can create threads inside your worker just like normal, however: be careful. AppFabric will allow many instances of your worker to be up and running so you need to write good threadsafe code and think about parallelism. Consider using the parallel extension for .NET.

Given the right problem though, breaking crypto keys for instance, Azure could kick some serious ass.

Other things workers can do

· PHP on Azure

· External Processes (run EXEs)

· Call Native Libraries (call unsafe code)

Diagnostics and Service Management

· Azure will log all the usual stuff (trace, iis logs, perf counters, event log, crash dump, files), but you have to configure it.

· The default vs project already has the basic diagnostic stuff (trace) configured.

· You need a storage account to put the data in. trace goes to a table, iis logs to a blob, events to a table)

· Have to start it up (DiagnosticMonitor.Start("DiagnosticsConnectionString");)

· You can change your configuration for diagnostic on the fly. The change only affects the instance and will go away when the instance gets restarted.

· To trace stuff just call Trace.Information like you would Trace.WriteLine

Service management API

Can do almost everything the Azure portal does

Excludes:

· No access to billing or utilization data

· Azure subscriptions

· Create Storage or compute service accounts

· Deploy management certificates

API Auth uses an x509 certificate

API is built on rest

Can use PowerShell to talk to the API

Csmanage is a command line tool to push packages to Azure

Can swap IP addresses to swap staging and production very quickly

In place upgrades

You can create upgrade domains and azure will manage the process of shutting down the services in each domain, upgrading them and then restarting them.

Autoscaling

Azure does not autoscale

Azure does give you tools to help determine how to scale. Recommend against completely automating the scaling of your service. You don’t want to spin up tons of instances just because the GoogleBot comes a calling.

SQL Azure

It’s mostly just SQL server in the cloud. You talk to it via ADO.NET (actually anything that speaks TDS can use it). So your EntityFramework and NHibernate stuff will still work.

· Database can go up to 10Gb (50Gb in June). You pay for the storage. Clarification you can have 1Gb or 10Gb (apparently they don’t grow the db

· Can hook up with Enterprise Manager 2008R2

· Use SQL Auth so protect that password!

· Setup via the azure portal

· There is a migration wizard that will allow you to move around.

Just use it

App Fabric

Authentication and authorization is hard. The cloud makes it harder

Tokens & Claims

· Token contains a group of claims

· Should be signed to protect against changes

Claims/tokens are better b/c they provide more information than applications typically get (username). This usually requires calls to AD or something else. This way the app tells the client what it wants and the client provides it.

OAuth

· Open standard

· Microsoft, Google, Yahoo

· REST oriented

SWT

· OAuth service authenticates the user

· Creates a SWT Token

· No xml

· Put in request body or on query string

WCF has an ACSTokenValidator

The client just authenticates against an OAuth service that the service trusts. Anybody can create this service. Then the client puts its token in the request headers and calls the service. The service will have to trust the issuer of the token.

Your service will use the ACSTokenValidator to make sure the caller has the token. If they do then your service will use the claims in the token to provide authentication & authorization. This is really awesome.

Service Bus

Basically it is WS-Routing. The services do not have to run in Azure, they just have to register with the Azure service bus. So Service A could be running at Microsoft and Service B could be running at Google and they could communicate with each other by using Azure’s service bus.

Very cool.

It’s just more WCF stuff.

Cloud Computing Scenarios

Only 2 reason for going to the cloud: Improve bottom line or improve strategy

Determine ROI

· Current hosting costs

· Current staff costs

· Cooling/power/space

· Migration costs

· Licensing

AzureROI.CloudApp.net

Common scenarios:

· Dynamic scale/reduce costs – load shunting

· Startups & POCs

· Data in motion

· Mass storage

· Multitenant

Common road blocks

· Security concerns

· Legacy systems

· Regulations & certifications

· Local hardware integration

Available Resources

· BizSpark – for startups (less than $1MM revenue)

· WebSiteSpark – for web design firms

· DreamSpark – for students

· WABCD – boot camp

· Pinpoint – place to register your product or service

· FrontRunner – get access to MSFT products early

· MTC Visit – for partners

· TAP – for customers wanting to use new products

Categories:   Tech
Actions:   E-mail | del.icio.us | Permalink | Comments (3) | Comment RSSRSS comment feed

Expiring Azure Queue Message

Tuesday, 23 March 2010 10:13 by jakew

In my notes from Day 1 of Azure Boot Camp I briefly mentioned that my class got hung up on the 30 second expiration of reservations on queue messages. What this means is that your code has a 30 second reservation on a queue message after it gets it. In those 30 seconds you need to complete the processing and then delete the message from the queue. If your code fails to do this the message will be made available again and the next call to Get will be given a reservation to the message. This will prevent the first thread from deleting the message.

For a vast majority of situation this should even be a consideration. The usual work we do in IT land doesn’t require 30 seconds of processing. That’s like an ice age worth of time to the computer. But being programmers we must obsess about edge cases, so lets say we’ve been hired by NSA to break a bunch of 768bit RSA keys. Obviously this is going to take a little more than 30 seconds. Your ‘normal’ queue processing patter would look like:

CloudStorageAccount storageAccount = CloudStorageAccount.FromConfigurationSetting("DataConnectionString");
CloudQueueClient queueClient = storageAccount.CreateCloudQueueClient();
CloudQueue queue = queueClient.GetQueueReference(_MessageQueueName);

if (null == queue)
{
    return;
}

bool done = false;
while (!done)
{
    IEnumerable<CloudQueueMessage> messages = queue.GetMessages(_BlockSize);
    if (_BlockSize > messages.Count())
    {
        done = true;
    }
    foreach (CloudQueueMessage message in messages)
    {
        BreakKey(message);
        queue.DeleteMessage(message);
    }
}

 

Obviously I don’t know how to break RSA keys, but BreakKey is where you would do that work. Once you are finished you would then delete the message from the queue. This is where the small issue pops up.

If the process takes more than the default 30 seconds the lease on the message will have expired so the delete will fail. In addition to not being able to delete the message the message will be available to be picked up by another process. What to do?

1) Change the default setting.

This is a horrible idea. Whatever amount you come up with won’t be enough. It is sort of like the “640Kb will be plenty of memory” idea. Just leave the defaults alone.

2) Use a tracking log

Create a queue manager that first gets the message, put its id and message body in a tracking table and then delete the message from the queue and then return the message to the caller. When the client is done the manager deletes the message from the queue and the table.

If the processor fails the message will still be in the table so it can be recovered. This is similar to BizTalk’s suspended message queue.

3) Just delete the message

A lot of people don’t like this, but frankly it isn’t as risky as they make it sound. Your code is running in the cloud. The servers are guaranteed to have 99.9% so it’s more likely your code is going to fail than the server. So debug your damn code! Get the queue item out of the queue, delete it, then process it and get on with your life.

All the above is pretty irrelevant, but my class still spent a good 15 minutes talk about it. In most cases the processing will be done in less than a seconds so the ‘usual’ pattern of get, process, delete will work just fine.

Categories:   Tech
Actions:   E-mail | del.icio.us | Permalink | Comments (0) | Comment RSSRSS comment feed