Friday, August 24, 2012

Generics and Implicit Conversion Operators Create an Identity Crisis


I keep running into problems with my Unit Tests for a project I'm working on at work, in particular, when I execute Assert.AreEqual(expected, actual).

Some Background Information

Currently, I'm working on some software that will integrate an accounting system with some online banking web services. This is so that instead of manually entering transactions in both the accounting system and then again up at the bank's website, firms can enter them once into the accounting system and the integration software I'm writing will handle making the appropriate web service calls to initiate the actual transactions at the bank.

There is a consortium out there known as the Interactive Financial eXchange Forum, usually known simply as IFX. They're an industry group that writes specifications for the transfer of financial data using XML and web services. Many financial instutions and software packages use specifications from the IFX, such as Intuit, the makers of Quicken software.

Some of these specifications are available for download for free. The most recent versions (and the most useful, e.g. XSD schemas) are often available to members only (and that pricetag is quite steep). I'm using version 1.7. Within the version 1.7 schemas, there is a type known as NC, which stands for "narrow-character" string. A narrow-character string is basically all 7-bit ASCII characters below 0x7F.

My first thought was to write an extension method to perform the conversion:

public static string UTF8ToLatin1(this string value);

This worked well until I discovered that I had to constantly remember that if a string variable needed to be a narrow-character string, I needed to call that extension method anytime I changed its value. Needless to say, I found occassions where I forgot to call the method. So while cool, this was not a good long-term solution.

A New Class is Born

So then I decided to get "cute"; but hopefully not too cute. It turns out that string is a sealed class. This is unfortunate, but there's probably a good reason for that. So I decided to write my own wrapper class that wraps a string value that provides all the necessary validation while delegating much of its behavior to string and all the while making it inheritable (there are several specializations of a narrow-character string used throughout the IFX specification schemas). I called it NCString.

Everything was going pretty well until I realized that I couldn't just simply assign a string to a NCString object and vice versa without using some sort of property or method accessor. Well I don't know about you, but that just smelled to me.

I then went researching to see if there was any way to create some sort of overloaded casting operator (sort of like C++'s static_cast and dynamic_cast operators, and what-not). It turns out, .NET has what they call conversion operators. I had forgotten all about these, mainly because they look a lot different since CLR 2.0 than from CLR 1.0, and honestly, I never had the need to implement one before (except for the occasional ToXXXX/FromXXXX methods).

Conversion Operators

In CLR 1.0, conversion operators were merely method calls in the form of ToXXX(...)/FromXXX(...) for languages that don't support operator overloading, and named operators called op_Explicit and op_Implicit for those languages that do support operator overloading.

In CLR 2.0, conversion operators got some syntactic sugar (at least, in C#). Explicit conversion operators, when used, look like regular casts from one type to another (much like how you can create a constructor in C++ that takes the type to convert from, which then allows you to use the C++ cast convention instead of calling the constructor directly). Implicit operators are even cooler in that the syntactic sugar allows you to directly assign the object to convert from directly to the object being converted to, as if both objects were of the same type. Obviously, there's much room for abuse here, so there are some compiler-enforced rules as well as general guidelines on their (as well as explicit conversion operator's) use, which can be found here and here.

So in my NCString class, I created the required constructors, implemented the requisite object.Equals methods, overloaded the equality operators, and finally created implicit conversion operators, both to and from string. Now, if you took the time to read the guidelines I linked to above, you will notice that I may have violated one of the guidelines:

Do not provide an implicit conversion operator if the conversion is potentially lossy.

This is sort of a judgement call. On the one hand, you could consider the conversion from string to NCString as "lossy", because potentially, a string can hold a wider range of characters than what is allowed in a NCString. However, if you try to assign a string with characters that fall outside the allowable range for a NCString, a System.ArgumentException is thrown—from the constructor of NCString, which curiously, is shown as an example in the Framework Design Guidelines documentation, and yet, which also seems to violate the following guideline:

Do not throw exceptions from implicit casts.

(If anyone has anything to say about the examples shown in the Framework Design Guidelines versus the guidelines themselves, I'd be interested to hear; leave a comment.)

As I was saying, on the one hand, the implicit conversion from string to NCString could be considered lossy. On the other hand, the two are strings, it's just that one has a more restrictive character set (e.g. it's not like I'm trying to convert from a double to an int, the conversion of which could result in actual data loss). So, in light of that, it would make it much simpler to work with the NCString class if I could convert between instances of NCString and string without needing to explicitly write a cast all the time.

Unit Testing Problems Begin

About this time is when some unit testing problems began to rear their ugly head when it came to calls to Assert.AreEqual(expected, actual). I initially began writing this blog post thinking that the generic version of this method was not properly inferring the parameter types or that the non-generic version was being called instead of the generic version. Well, it turns out to be the latter rather than the former.

When I began writing unit tests, I would have code similar to the following:

// Assert
NCString nc = "Hello World";

// Act
target = new SomeObject(nc);

// Assert
Assert.AreEqual(nc, target.NarrowString);

/* Results of assertion:
 *
 * Assert.AreEqual failed: Expected<MyNamespace.NCString>  Actual<System.String>
 *
 */

Identity Crisis

I expected that, 1) with the method having a generic method signature overload, 2) .NET's ability to infer generic method type parameters, and 3) since the parameters I passed were two different types with conversions from one to another, that actual would attempt to be casted into the type of expected, after which, they would then be tested for equality.

As I just mentioned above, in .NET, generic methods can most often infer their generic parameters, so you don't need to explicitly tell .NET what the type of the generic parameter is. This is precisely what's causing me issues. I'm used to letting .NET infer my generic parameter types (especially when it comes to LINQ statements). But in this case, it's not being inferred. Why not?

The answer is rather simple; but throw in some implicit conversion operators and the answer becomes more complex. So here's the not-so-simple answer. What was happening was that the compiler did (probably) attempt to use the generic method. But, because I didn't specify the type of the generic parameter T, instead, letting .NET (try to) infer the type, .NET was unable to infer the type. Why? Because of the implicit conversion operators. What type should .NET infer for the parameters? string? NCString? Both parameters are implicitly convertible to one another. Furthermore, even if .NET did choose one type over another, there really isn't a way to dynamically cast one type to the other through reflection. Indeed, in the end, the compiler had to choose to use the overload that takes two object parameters. And, of course, object has an Equals method that does little more than test for type equality and referential equality.

Finding Myself

There are two solutions to this problem. One easy, the other somewhat easy.

First, the somewhat easy one (my first approach, mostly because I didn't look to see what else was available in the Assert class). Explicitly tell the compiler what the generic type parameter should be so that the generic method signature is used: Assert.AreEqual<NCString>(expected, actual);. Because the two types are implicitly convertible to/from one another, and I've explicitly specified the generic type parameter to be a NCString, the actual parameter will be converted to a NCString and NCString's implementation of Equals will be used.

The really easy solution (for my case) is to use the overload that takes two string parameters and a bool (indicating whether or not the assertion should be handled case-insensitively). Using this overload, I'd have to use the third boolean parameter (in my case, always set to false) and no explicit type casting/conversion would need to be performed. (Note however, that an implicit type conversion would still be performed from NCString to string.)

Which is better? Hard to say and it mostly depends on what you're trying to test for equality. In my case, both types were strings (one more specialized than another), so there's not much of a diffirence one way or another and not much more in the way of typing for one way or the other.

Some Final Questions and Concluding Remarks

So, I wonder, does the .NET Unit Test Framework always use the Assert.AreEquals(expected, actual) overload that takes two object parameters whenever the two parameters are of different types, or just in the case when the two parameters are not the primitives for which an overload exists and a generic type parameter was not specified?

Anyway, I just wanted to post this, because this has been bothering me for a while. I didn't expect this behavior, but now that I've worked through it, I understand it. I hope that I can save you many hours of troubleshooting your unit tests.

.NET Regular Expressions and XML NAME tokens


Hello everyone. I'm back again. Before we get started, I just kind of wanted to put out a notice. I'm working on my blog template. So if you come here and see that stuff doesn't look exactly the way it should, well, you've been warned. This includes the mobile template (so far, I'm not impressed with blogger's mobile templates).

Wow, so many articles in such a short amount of time! Yes, I'm being facetious ;). Let's hope I can continue this pattern. Afterall, I started this blog 3 years ago to start writing down things I learn and which I'd like not to forget.

Well, this blog post certainly fits the bill. I know I've done some research on this once before, but I failed to write it down. So I wasted some more time re-researching this problem. What a waste. So, to help you not waste your time, I hope that you find this blog post helpful. Enough already, let's dive in!

Extensible Markup Language NAME Tokens

I first learned eXtensible Markup Language (XML) back in 2000-something-or-other and haven't used it much. A lot of people use XML everywhere for anything. I'm not a big believer in that, actually. It's a great tool and it makes sense to use it where and when it needs to be used.

Having said that, there have been 3 occassions over the last eight months where I had to brush up on my XML skills and really know it well. I'll put a shameless plug in for a book that, while old, has helped me tremendously (I haven't seen its equal): XML Primer Plus by Nicholas Chase.

Like I said, the book is a bit dated now, but it still has very valuable information in it and not much has changed. Having said that, one thing that has changed are the allowable characters in NMTOKENs. I recently needed to validate values I read from user input to ensure that they were in fact valid an id attribute values. I naïvely assumed the following regular expression:

if ( ... && Regex.IsMatch(id, @"^\\d.*|\\w*[\\p{P}-[_]]+.*$") { ... }

So, basically, a XML NMTOKEN can have anything in it except punctuation characters (except for the underscore). I just realized as I'm typing this that I completely forgot to check if the NMTOKEN started with a number, which also isn't valid.

This might have been fine back in 1999, but the XML standard has changed since then. It's still version 1.0 (well, there is a version 1.1, but let's not go there), but it's the 5th edition.

The current XML specification defines a NAME token (to which ID tokens must adhere and which is a specialized form of a NMTOKEN as follows:

[Definition: A Name is an Nmtoken with a restricted set of initial characters.] Disallowed initial characters for Names include digits, diacritics, the full stop and the hyphen.

Names beginning with the string "xml", or with any string which would match (('X'|'x') ('M'|'m') ('L'|'l')), are reserved for standardization in this or future versions of this specification.

Note:

The Namespaces in XML Recommendation [XML Names] assigns a meaning to names containing colon characters. Therefore, authors should not use the colon in XML names except for namespace purposes, but XML processors must accept the colon as a name character.

The first character of a Name MUST be a NameStartChar, and any other characters MUST be NameChars; this mechanism is used to prevent names from beginning with European (ASCII) digits or with basic combining characters. Almost all characters are permitted in names, except those which either are or reasonably could be used as delimiters. The intention is to be inclusive rather than exclusive, so that writing systems not yet encoded in Unicode can be used in XML names. See J Suggestions for XML Names for suggestions on the creation of names.

Document authors are encouraged to use names which are meaningful words or combinations of words in natural languages, and to avoid symbolic or white space characters in names. Note that COLON, HYPHEN-MINUS, FULL STOP (period), LOW LINE (underscore), and MIDDLE DOT are explicitly permitted.

The ASCII symbols and punctuation marks, along with a fairly large group of Unicode symbol characters, are excluded from names because they are more useful as delimiters in contexts where XML names are used outside XML documents; providing this group gives those contexts hard guarantees about what cannot be part of an XML name. The character #x037E, GREEK QUESTION MARK, is excluded because when normalized it becomes a semicolon, which could change the meaning of entity references.

Names and Tokens

RuleProduction
[4]
NameStartChar
::=
":" | [A-Z] | "_" | [a-z] | [#xC0-#xD6] | [#xD8-#xF6] | [#xF8-#x2FF] | [#x370-#x37D] | [#x37F-#x1FFF] | [#x200C-#x200D] | [#x2070-#x218F] | [#x2C00-#x2FEF] | [#x3001-#xD7FF] | [#xF900-#xFDCF] | [#xFDF0-#xFFFD] | [#x10000-#xEFFFF]
[4a]
NameChar
::=
NameStartChar | "-" | "." | [0-9] | #xB7 | [#x0300-#x036F] | [#x203F-#x2040]
[5]
Name
::=
NameStartChar (NameChar)*

At this point, I thought, "No problem," I'll just use those productions in a Regular Expression while making sure the token doesn't start with xml or some form of that thereof and that'll be that. During unit testing, I discovered that out of 29 invalid sequences that I tried (which were all in the ASCII portion, so this unit test was not comprehensive in any way for the time being), only 7 were flagged. What's going on here?

Well, I read the MSDN documentation for .NET Regular Expressions. Here was the expression I used that was failing the unit test:

@"^(?i:(?!xml))(?inx:[a-z]|_|:|[\u00C0-\u00D6]|[\u00D8-\u00F6]|[\u00F8-\u02FF]|[\u0370-\u037D]|[\u037F-\u1FFF]|[\u200C-\u200D]|[\u2070-\u218F]|[\u2C00-\u2FEF]|[\u3001-\uD7FF]|[\uF900-\uFDCF]|[\uFDF0-\uFFFD]|[\u10000-\uEFFFF])(?inx:[a-z]|_|:|[\u00C0-\u00D6]|[\u00D8-\u00F6]|[\u00F8-\u02FF]|[\u0370-\u037D]|[\u037F-\u1FFF]|[\u200C-\u200D]|[\u2070-\u218F]|[\u2C00-\u2FEF]|[\u3001-\uD7FF]|[\uF900-\uFDCF]|[\uFDF0-\uFFFD]|[\u10000-\uEFFFF]|-|\.|\d|\u00B7|[\u0300-\u036F]|[\u203F-\u2040])*$"

Now, I must admit, .NET's Regular Expression syntax is a bit funky. Anyway, the (?i:(?!xml)) says if the string case-insensitively starts with XML, then it doesn't match. Otherwise, use the production rules as shown above from the W3C.

The catch is, .NET strings are encoded in UTF-16 (and therefore, the escape sequence \u can only be followed by 4 hexadecimal digits), but according to the production for a NameStartChar I need to detect some characters outside of this range (the [\u10000-\uEFFFF] character class inside of the Regular Expression). Now, I had done some research on internationalization and Unicode, but I never really had to pay much attention to it. I'm the one who's mostly using my software. However, this is not the case this time. This software may be used by firms doing banking all over the country and aronud the world. So I definitely need to check that the ID string I'm getting is valid. During my research, I came across Joel Spolsky's blog on Unicode and character encoding. This was a good start (and you should read this if you haven't already), but I needed to know more. Specifically, how can I encode a Unicode character that falls outside of the range 0 - 65535 into a 16-bit value?

UTF-16 Surrogate Pairs

The answer is UTF-16 surrogate pairs. I had heard of these, but I didn't know how to generate them. This article helped me out. In the UTF-16 character encoding, there are no UTF character code points defined in the range 0xD800 - 0xDFFF. This range is used in an algorithm to generate the UTF-16 surrogate pairs that represent Unicode character code points above 65535. The algorithm is really simple and is outlined below..

  1. Take the hex value of the Unicode character to encode as UTF-16 and subtract 0x10000 from it.
  2. Take the result from step 1 above and shift it right 10 bits (0xA).
  3. Take the result from step 2 and add 0xD800. This gives you the first surrogate of the surrogate pair.
  4. Again, taking the result from step 1, AND the value with 0x3FF to mask off the upper 10 bits.
  5. Add 0xDC00 to the result from step 4 above. This represents the second surrogate of the UTF-16 surrogate pair.

The resulting 16-bit surrogates from above properly encode Unicode character code points above 65535 in UTF-16. Let's run through a quick examlpe.

Enocding a Unicode Character Code Point Above 65535 into UTF-16

Let's start with the Unicode character code point U+18657. (I don't know what this character is, I just chose something at random.) Following our algorithm above:

  1. 0x18657 - 0x10000 = 0x8657
  2. 0x8657 >> 0xA = 0x21
  3. 0x21 + 0xD800 = 0xD821 (This is the first surrogate of the surrogate pair.)
  4. 0x8657 & 0x3FF = 0x257
  5. 0x257 + 0xDC00 = 0xDE57 (And this is the second surrogate of the surrogate pair.)

So, from the algorithm above, the Unicode character code point U+18657 can be encoded into UTF-16 using the surrogate pair U+D821 U+DE57.

Putting it All Together

Finally, we come to the end. I needed to replace the character class [\u10000-\uEFFFF] with a valid \u escape sequence construct. That involves calculating the range of the surrogate pairs for the character class. I used the algorithm above to calculate the range of surrogate pairs which results in the following pattern that should be used to replace the invalid character class: ([\uD800-\uDB7F][\uDC00-\uDFFF]). This pattern will match all UTF-16 encoded Unicode character code points between the range of U+10000 - U+EFFFF, which is exactly what we want. Here's the final Regular Expression that will validate an XML ID token:

@"^(?i:(?!xml))(?inx:[a-z]|_|:|[\u00C0-\u00D6]|[\u00D8-\u00F6]|[\u00F8-\u02FF]|[\u0370-\u037D]|[\u037F-\u1FFF]|[\u200C-\u200D]|[\u2070-\u218F]|[\u2C00-\u2FEF]|[\u3001-\uD7FF]|[\uF900-\uFDCF]|[\uFDF0-\uFFFD]|([\uD800-\uDB7F][\uDC00-\uDFFF]))(?inx:[a-z]|_|:|[\u00C0-\u00D6]|[\u00D8-\u00F6]|[\u00F8-\u02FF]|[\u0370-\u037D]|[\u037F-\u1FFF]|[\u200C-\u200D]|[\u2070-\u218F]|[\u2C00-\u2FEF]|[\u3001-\uD7FF]|[\uF900-\uFDCF]|[\uFDF0-\uFFFD]|([\uD800-\uDB7F][\UDC00-\uDFFF])|-|\.|\d|\u00B7|[\u0300-\u036F]|[\u203F-\u2040])*$"

A Few Closing Remarks

Firstly, the second and third Regular Expressions shown in this article could be simplified (e.g. getting rid of all the alternation (|) constructs between character classes and creating one big character class). Second, and most important, this probably isn't exactly what you should normally do. The productions given by the W3C are meant to be inclusive, as they noted for their justification of the productions. But it would probably be more efficient to write a Regular Expression that matches on what should not be present in a XML ID token (though, the Regular Expression would be almost as long and complex).

I did test my final Regular Expression as shown above, and it did pass (again, my unit test was not comprehensive with regards to characters outside the 7-bit ASCII character set). However, since I used the production shown in the XML specification, I have no reason to think that this Regular Expression would not let an invalid ID token "slip" through.

Wednesday, August 22, 2012

Updating a Visual Studio 2010 Extension for Visual Studio 2012


UPDATE - 16 January 2013

The contents of this article have been updated to correct a problem that existed at the time of publication. In addition, I have uploaded this updated Visual Studio extension to the Visual Studio Gallery.

Hello everyone. It's been way too long. It seems I'm always doing this and that and never have time to write down anything, anywhere—whether that be on this blog, or on Facebook. But today, I have something special for those of you who have that favorite Visual Studio 2010 Extension that they just can't live without, but it hasn't been updated for Visual Studio 2012.

In order to update the extension, you will need access to the extension's source code; so these instructions will have limited impact for most of you out there unless you can get your hands on the source code for that favorite extension.

Fortunately for me, the author of my favorite extension, AllMargins, did post his source code, especially since this extension is no longer available from the Visual Studio Gallery.

Install the Visual Studio 2012 and 2012 SDKs

Get Visual Studio Professional 2012 with MSDN from the Microsoft Store

In order to author extensions for Visual Studio, one must have the Visual Studio SDK installed. The same is true for modifying them. If you don't already have it installed, go out and download it and install it.

Prepare the Source Code for Visual Studio 2012

If you want to maintain backward compatibility for the extension, it would be best not to open the original Visual Studio 2010 solution in Visual Studio 2012. That's because Visual Studio Extension projects for Visual Studio 2010 are not compatible with Visual Studio 2012. Well, that's not exactly true. The real truth is that the VSIX manifest XML schema has changed between Visual Studio 2010 and Visual Studio 2012. The old schema still works for Visual Studio 2012, but Microsoft would have you use the new schema going forward. The problem is, if you use the new schema, you won't be able to use the extension in Visual Studio 2010. Don't worry though, Visual Studio 2012 will not upgrade the schema, just the project file(s); but in doing so, the projects will no longer open in Visual Studio 2010.

Why should you care? Well, for one, testing. You can't simply instruct VsixInstaller to install the extension to an experimental instance of Visual Studio, unfortunately. That really makes it hard to test the extension for more than one version of Visual Studio. Even so, you really should be testing your extension on a virtual machine with the installed version of Visual Studio to test it with and which does not have the Visual Studio SDK installed. If the Visual Studio SDK is installed on your test machine, your extension will always work but, you may find that it doesn't work users who don't have the Visual Studio SDK installed. Your extension should not explicitly require the Visual Studio SDK be installed (unless you're writing a Visual Studio Extension project template! Lol...).

If you want to be able to develop and test in both Visual Studio 2010 and Visual Studio 2012 on the same machine, here's my suggestion.

  1. Create a new empty solution called <Your Extension>.2012 in the same place as the original solution.
  2. Open Windows Explorer to the location of one of the projects that are in the Visual Studio 2010 solution.
    1. Select the project file and copy and paste it into the same folder. The file will be renamed 'Copy of <Your Project File>.csproj' (or .vbproj, if you use Visual Basic). Rename the file to <Your Project File>.2012.csproj.
    2. Back in Visual Studio 2012, add <Your Project File>.csproj to your solution.
  3. Add all the files that are part of the Visual Studio 2010 project to the newly created Visual Studio 2012 project.
  4. Repeat these steps for each of the other projects that were part of the original Visual Studio 2010 solution.

Now you can work on the source code in both Visual Studio 2010 and Visual Studio 2012, opening the respective solution for the version of Visual Studio you're using to make changes. When you build the Visual Studio 2012 solution, it will be deployed to the Visual Studio 2012 experimental instance. The same is true for 2010, the extension will be deployed to the Visual Studio 2010 experimental instance. Also, it allows you to work on a single code-base that should work in both environments so long as you don't use features exclusive to Visual Studio 2012.

While upgrading this extension to work with Visual Studio 2012, I discovered that there is an enumeration value in the VSIX XML schema for specifying the Express editions of Visual Studio. However, the only "extensions" that Express SKUs support are project/item template extensions.

We're now ready to start modifying the various files to make this extension compatible with Visual Studio 2012!

Update the *.vsixmanifest file

The first order of business in updating the extension is to modify the *.vsixmanifest file. Here's an example of one of the files from the AllMargins.vsix extension:

<?xml version="1.0" encoding="utf-8"?>
<Vsix xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xmlns:xsd="http://www.w3.org/2001/XMLSchema" Version="1.0.0" xmlns="http://schemas.microsoft.com/developer/vsx-schema/2010">
    <Identifier Id="SettingsStore.dpugh.ce3ce74a-4966-4a66-b9ad-a3adf4f9e940">
        <Name>SettingsStore</Name>
        <Author>dpugh</Author>
        <Version>1.2</Version>
        <Description xml:space="preserve">This defines an API for persisting IEditorOptions across sessions.</Description>
        <Locale>1033</Locale>
        <MoreInfoUrl>http://code.msdn.microsoft.com/OverviewMargin/Release/ProjectReleases.aspx?ReleaseId=3957</MoreInfoUrl>
        <SupportedProducts>
            <VisualStudio Version="10.0">
                <Edition>Ultimate</Edition>
                <Edition>Premium</Edition>
                <Edition>Pro</Edition>
            </VisualStudio>
        </SupportedProducts>
        <SupportedFrameworkRuntimeEdition MinVersion="4.0" MaxVersion="4.0" />
    </Identifier>
    <References />
    <Content>
        <MefComponent>|%CurrentProject%|

    </Content>
</Vsix>

You need to update the SupportedProducts and SupportedFrameworkRuntimeEdition elements as follows:

<SupportedProducts>
  <VisualStudio Version="10.0">
    <Edition>Utimate</Edition>
    <Edition>Premium</Edition>
    <Edition>Pro</Edition>
  </VisualStudio>
  <VisualStudio Version="11.0">
    <Edition>Utimate</Edition>
    <Edition>Premium</Edition>
    <Edition>Pro</Edition>
  </VisualStudio>
</SupportedProducts>
<SupportedFrameworkRuntimeEdition MinVersion="4.0" />

By removing the MaxVersion attribute for the SupportedFrameworkRuntimeEdition element, we're telling Visual Studio that so long as the machine supports the .NET Framework 4.0, the extension can be used.

Also for good measure, I updated the Version value (e.g. 1.3) and updated the Description element noting that I changed the VSIX package and it is not the original. As an aside, every time you make a change to the extension that you wish to distribute, you must increment the value for the Version element.

Now I ran across a tricky problem with updating the AllMargins extension. This extension has several "sub-extensions" that are contained within the AllMargins extension itself. And some of these "sub-extensions" have yet other "sub-extensions". The long and short is, during testing, Visual Studio had a problem locating some of the required DLLs.

What I found out is that some of the sub-extension projects contained an additional file—a *.pkgdef file. This is an older file that was used with VSPackages back in the VSI days of Visual Studio 2005/2008. This file is not much unlike a *.reg file, though with a slightly different syntax.

This file for the SettingsStore sub-extension (for which the VSIX manifest file was shown above) looks like this:

[$RootKey$\BindingPaths\{8ADD56CC-E512-4CB0-AF58-33ECCC0B4912}]
"$PackageFolder$"=""

This creates the GUID under HKCU\Software\Microsoft\VisualStudio\<Maj. Version Num>_Config\BindingPaths key. $PackageFolder$ expands to the folder where the extension was installed by VSIXInstaller.

Apparently, in Visual Studio 2010, just placing this file within your VSIX package was enough for Visual Studio to go ahead and create the requisite registry keys and values. However, this does not work with Visual Studio 2012 and attempts to use extensions relying on this extension result in a System.IO.FileNotFoundException. In order to resolve this issue, you must place some additional content in your VSIX manifest file as shown below.

<References>
    <Reference Id="Microsoft.VisualStudio.MPF" MinVersion="10.0" />
        <Name>Visual Studio MPF</Name>
    </Reference>
    <!-- Any other references you may have -->
</References>
<Content>
    <MefComponent>|%CurrentProject%|</MefComponent><!-- Expands to the assembly name -->
    <VsPackage>SettingsStore.pkgdef</VsPackage><!-- Not required by VS2010, but is required by VS2012 in order for the file to be processed -->
</Content>

With all of that out of the way, save the file and we're done with the *.vsixmanifest file. If there are several VSIX extensions that make up the extension you're updating (as is the case for AllMargins.vsix), you will need to do this for each *.vsixmanifest file in the solution.

At this point, at the original date of publication (22 August 2012), I had you retarget the projects in the solution and replace some DLL references. If you want to maintain compatibility with Visual Studio 2010, this is not necessary. So I have removed that section of the original blog entry.

Recompile a Debug Build

You should now recompile the solution using a Debug build. This will automatically install the extension(s) into the Visual Studio experimental instance(i.e. it won't muck up your Visual Studio that you use for everyday coding). Make sure that everything works correctly. If all is well, you're ready for the final step.

Recompile in Release Mode

If your testing above went OK, then change the build mode from Debug to Release and recompile. Then simply navigate to the bin\Release directory under the project folder for the extension and double-click the VSIX file to install it.

Now, in this case, I was not required to change a single line of code. This may not be the case for every extension you wist to upgrade. YMMV.

Enjoy using Visual Studio 2012

That's it! Enjoy using Visual Studio 2012 with your beloved extension.

Tuesday, July 20, 2010

Sample Title Marked up Using x-hTitle Microformat Proposal

This blog entry will demonstrate using the proposed x-hTitle microformat to mark up a title. Since this is a sample for a microformat proposal, what better title to use than the one that I first read regarding the subject. So, without further adieu, here's a sample. One other quick note: I may edit this blog entry from time-to-time, adding more samples. Eventually, they will all be moved to the microformats.org site if this proposal becomes a draft specification.

Sample x-hTitle Formatted Title

Microformats: Empowering Your Markup for Web 2.0
Publisher: friends of ED
Edition: First Edition
Abstract

Microformats burst onto the scene a couple of years ago and are fast becoming an essential tool for all professional web designers and developers. Imagine being able to integrate all of yourr web-based contact details, tagged articles, and geographical information seamlessly in web and desktop applications, without having to add anything extra to your websites except a little specialized HTML markup.

Microformats provide a more formalized technology for adding commonly used semantics (such as contact details, locations, and reviews) to today's Web. Unlike XML or the Semantic Web, microformats use ubiquitous technologies like HTML and XHTML, existing developer skills and current web tools, and, perhaps most important, they all work in today's web browsers.

This book is a comprehensive guide to microformats. It explores why, in Bill Gates' words, We need microformats; how microformats work; and the kinds of problems microformats help solve. The book covers every current microformat, with complete details of the syntax, semantics, and uses of each, along with real-world examples and a comprehensive survey of the tools available for working with them. The book also features case studies detailing how major web content publishers such as Yahoo! put microformats to work in their web applications.

Written by one of the Web's best-known educators, John Allsopp, Microformats: Empowering Your Markup for Web 2.0 will help you painlessly get up to speed with this exciting technology.

In this Book You'll Learn...
  • How to take best advantage of the built-in semantics of XHTML and HTML.
  • How to extend the semantics of HTML using microformats and open up a world of new possibilities with web applications.
  • Every aspect of all the common microformats currently in use.
  • How microformats help your websites and applications easily integrate with web applications like Google Maps, as well as desktop applications like iCal, Outlook, and Entourage.
  • What innovative publishers and services, big and small, are doing right now with microformats.
Summary of Contents
  • Part One: Introducing Microformats
    • Chapter 1: What are Microformats?
    • Chapter 2: The State of the Art in Microformats
  • Part Two: Using Microformats
    • Chapter 3: Structural and Semantic HTML
    • Chapter 4: Link-Based Microformats: rel-license, rel-tag, rel-nofollow, and VoteLinks.
    • Chapter 5: Microformat to Describe Relationships Between People: XFN
    • Chapter 6: Location Microformats: geo and adr
    • Chapter 7: Contact Information Microformat: hCard
    • Chapter 8: Event Microformat: hCalendar
    • Chapter 9: Review and Resume Microformats: hReview and hResume
    • Chapter 10: Syndicated Content Microformat: hAtom.
  • Part Three: Case Studies
    • Chapter 11: Case Study: Cork'd
    • Chapter 12: Case Study: Yahoo!
  • Part Four: Developing Microformats
    • Chapter 13: The Process of Developing Microformats
  • Part Five: Appendixes
    • Appendix A: Microformat Specification Reference
    • Appendix B: Microformat Design Patterns
    • Appendix C: People, Tools, Services, and Publishers

A New Microformat Propoal: x-hTitle

Comments Regarding the Proposed x-hTitle Pre-Draft Specification

This blog post is meant to document my vision of a microformat for semantically marking up information about titles. On the microformats.org process page, it is recommended that you not name your new proposed format hsomething. But, since this is a proposed pre-draft specification, I need to be able to reference something throughout this specification. Therefore, I am naming this proposed format x-hTitle, where x stands for experimental. If there is interest in developing this microformat further, the x in the name could be removed. In addition, the name of the proposed microformat could change according to community consensus during the specification process.

The x-hTitle Proposed Pre-Draft Specification

Editor/Author
  • Craig S
Contributors

None yet. If you would like to contribute, please register on the microformats.org's microformats-new mailing list and read posts regarding this proposal and contribute your thoughts and suggestions. If this proposal gets the green light to become a draft specification, you can contribute to the specification at the microformats.org's website and your name will be listed in this section of the specification.

Introduction

It is difficult for a browser to extract semantic information about a book title described on a web page. Metadata such as author, cover image, and book genres/categories provide relevant content for titles.

In order to keep this microformat as generic as possible, I want to make a distinction between titles and books. A title is something than an author writes. A book is a published representation of a title. There may be more than one book for a given title, for example, a hardcover book, a paperback book, a large-print or illustrated book, etc. Each of these books will have a different ISBN even though they are the same title.

This microformat aims to codify the semantic structured markup of titles. Since a title can describe (possibly) many books, semantically marking up title information has one very important implication: a title which contains reviews (using the hReview or the hReview-aggregate microformat) may have all reviews for it aggregated in one place. This aggregation may then be used to display this information for several books (e.g. the same reviews would be shown for a hardcover, softcover, or other formats of a title) because they are all for the same title.

Having such information marked up can provide many benefits to the viewer of such a page. If a web browser understands that a particular web page contains a title, it can provide rich interactions for the user—for example, specific searches for titles via search services such as Google, Amazon, Yahoo!, Bing, etc. While Amazon already does a good job of returning search results for titles (since that is its primary business) and searches for titles on major search engines usually yield a link to Amazon, marking up titles using a microformat can make indexing title information easier while making this information more accessible to users. In addition, classification of titles by search engines (and crawlers) can be made more accurate. Also, if a page contains many titles by the same author(s), then the web page is not only a list of titles, but also about a particular author(s).

In order to enable and encourage the sharing, distribution, syndication, and aggregation of title content and information, the author proposes a pre-draft specification of a x-hTitle microformat, an open standard for distributed metadata about titles. The author has researched numerous sites which contain titles for sale or advertisement and has designed the x-hTitle microformat around a simple, minimal schema for information regarding a particular title. Feedback is encouraged on this blog entry. If there is enough interest in this pre-draft specification to warrant making a sanctioned draft specification of this microformat, future feedback will be moved to the official microformats.org website.

Inspiration and Acknowledgements

This proposal came about because of reading the book Microformats: Empowering Your Markup for web 2.0 by John Allsopp (note: the book's title has been marked up using this very proposed microformat) and my need to create an inventory system for all media that I own, including CDs, DVDs, books (again, a specific representation of a title), magazines, and anything else media related. As there already exists a hAudio and hMedia microformat, titles are notably left out and most likely relegated to another microformat such as hProduct or hListing which are not the most suitable microformats for this information given the distinction between a title and a particular publication of that title—a book (which is a product and which can be listed, whereas a title in and of itself, is not and can not).

Scope

Information about titles consistently share several common fields. Where possible, this proposed microformat has been based on this minimum common subset.

Out of Scope

This microformat does not aim to be a means of publishing a title online (such as the proposed book microformat boom). Instead, it is a means of marking up information about a title suitable for searching, viewing, and aggregating on sites such as Amazon and Google Books. This microformat, therefore, is not a competitor to boom, rather, it serves a much different purpose.

Format
Abstract

The proposed x-hTitle microformat is based on a set of fields common to many Internet web sites and formats in use today that provide information about titles (in the form of books), whether for sale, advertisement, or promotional or informational purposes. Where possible, field names have been chosen based on those defined by many related microformat standards.

Schema Key
Table 1. Schema Key
Field Format Description
bold{1}must be present exactly once.
italic* optional, and may occur more than once
+ must be present, and may occur more than once.
? optional, but must not occur more than once.
[square brackets]A list of common values
(paretheses)Data format
#A comment.
!Awaiting documentation
Schema

The proposed x-hTitle microformat schema consists of the following:

  • x-hTitle{1}.
    • version ?. Using text.
    • fn{1}. Reused from hCard
    • edition ?. Using text.
    • volume ?. Using text.
    • contributor*. Reused from hMedia.
    • description ?. Using any valid XHTML markup.
    • review*. Using hReview or hReview-aggregate .
    • category* Using text.
Property Details
version
This optional property permits title publishers to specify a particular version of the x-hTitle microformat that their content uses. By omitting this property, the publisher is stating that implementations may interpret titles marked up using this microformat according to the schema version <to be decided> or later. In practice, the author of this specification is committed to maintaining backward compatability with content produced using earlier versions of the specification. This property is syntax compatible with, and thus reuses semantics of VERSION as defined in the vCard RFC2426 section 3.6.9 VERSION Type Definition. The value of this property for this specification is <to be decided>.
  • The element is identified by the class name version
  • x-hTitle may include only one version property.
fn
The full name of the title, including any subtitle.
  • x-hTitle must include one fn element.
  • The element is identified by the class name fn.
  • If the title being marked up with this microformat has its own website (for example, for forum discussions related to the title, source code, or other content), then hCard may be used to provide that information with the title.
  • This element is reused from hCard.
edition
This property specifies the specific edition of a title.
  • The element is identified by the class name edition.
  • x-hTitle may include one instance of this property.
  • This property may use the abbr-design-pattern .
  • Since the content of a title from one edition to a next can differ significantly, this information should be included if applicable and available.
volume
This property specifies that the title is a particular volume of a larger collection of works which share a main title.
  • This element is identified by the class name volume.
  • x-hTitle may include one volume element.
  • This information should be included if applicable and available.
  • This property may use the abbr-design-pattern .
contributor
A contributor is any entity that takes part in the creation and distribution of a title. Examples include: author, illustrator, publisher, editor, technical reviewer, etc.
  • This element is identified by the class name contributor.
  • x-hTitle may include one or more contributors.
  • The contents of the element should include a valid hCard.
    • The role element of hCard should be used to specify the contributor's responsibility related to the title.
description
An abstract or explanation of the contents of the title being marked up. for example, this field could be used to contain the text on the back of a technical book that summarizes the content of the title. Or, such as with fiction hardcover books, the content that appears on the inside flaps of the dust jacket.
  • This element is identified by the class name description.
  • The element's contents may include any valid XHTML.
  • x-hTitle may include more than one description.
    • If more than one description is defined, the full description is the concatenation, in order, of these elements.
review
The existence of this property identifies review(s) for the title.
  • The element is identified by the class name review.
  • Reviews should use the hReview microformat.
  • If many reviews are to be listed for a title, consider using the hReview-aggregate microformat, instead.
category
The category property can be used to classify the genre, style, or keywords and subject matter associated with the title. Examples could include: fiction, horror, mystery, computer, technical, programming, instructional, biography, etc.
  • The element is identified by the class name category.
  • x-hTitle may include one or more category elements.
  • This element may be expressed using the rel-tag elemental microformat. When a category is expressed using rel-tag, the content of this element is used as the text for the category. For example: <a class="category" rel="tag" href="/tags/romance">Romance</a> would have "Romance" as the text for the category.

Sunday, July 11, 2010

Wow, it's been a long while!

We Thought You Fell Off the Face of the Earth!

Hello everyone. I'm still around. Well, I warned you, I may not write often. I hope, though, that there are still some readers out there (or new ones will find this blog).

First, a very quick update in fast-forward. I moved because my wife got laid off about a year and a quarter ago. She got a new job fairly quickly given the economy, so that's that. In addition, I've completed two more semesters at Kutztown University and I will be entering my last semester this Fall. I've also been taking two Summer courses.

The main reason for not writing in so long has been three-fold:

  1. The last two semesters have been extremely difficult—on the flip side, I managed to pull off very good grades; so I'm very pleased about that.
  2. I've been trying to work whenever I can to get all those computers installed at Thermocouple Technology, Inc., but it's been really slow going.
  3. Making things even more slow going at TTEC is the fact that I decided to get a different Summer job that would be involved with coding. It is to a certain extent (if you call writing SQL programming), plus, the job is about an hour away from where I live.

My coursework at school over the last two semesters included taking Compiler Design I and II using the famous Dragon Book. I tried to implement a language that could replace SQL. Well, being I wrote the language in academia for academia, it didn't quite turn out the way I would've liked. I spent every waking minute last semester writing code for that class. Anyway, the idea is that the language could be used not only for relational databases, but also XML data, structured textual data (e.g. CSV files, tab-delimited files, etc.), or any other structured data (possibly using some kind of schema definition). As the semester wore on, I whittled the language down to just working with relational databases for the start. However, I didn't have time to learn another database technology with an open API where I could directly access the query engine, so I ended up translating my language to SQL anyway, which didn't work out quite so well. Having said that, my ideas are completely valid and workable, given enough time and resources to develop it properly, and it very well could replace SQL using a much more human-readable/writable syntax. In fact, part of my coursework this Summer is writing a paper on this language and its goals.

Anyway, enough of that. So in a nutshell (which is a first for me, believe me!), that's what's happened over the last year and a half or so.

Why write now?

Well, I'm glad you asked. As mentioned, it's my last semester coming up this Fall at Kutztown University. As such, I'm required to take a course colloquially called Senior Sem, the cap-stone course that's supposed to tie together everything we learned into a final project.

Being who I am, I plan on making the best of it. My idea: an inventory system for all the different types of media I own—books, CDs, DVDs, magazines, sheet music, etc. It will be web-based using the new ASP.net MVC 2.0 framework. I will also be investigating technologies such as jQuery, and AJAX. In addition, I plan on proposing two new microformats to microformats.org to use with my project. I will be writing a few more blog posts over the next couple of days/weeks that contain pre-pre-draft specifications of the proposed microformats, as well as samples of how they could be used alone or along with other existing microformats.

I'm very excited about my project and hope to learn a lot of real-world skills and create a usable application. Whether anyone else would find it useful, I don't particularly care, I've been wanting a comprehensive application like this for years.

More on Microformats

If you don't know what microformats are, I'd encourage you to visit the website I linked to above. In addition, there's also RDFa. The two technologies are complementary. RDFa, as I understand it, is for machines first, whereas microformats are for humans first. Both can be used together to make the web a much more meaningful place for humans and machines.

To make the most of this embedded information, you can use various browser plug-ins to access the information. For example, there is Oomph for Internet Explorer, and Operator for Firefox. When microformats are available on the page, these tools will let you export contact information, calendar events, product information, RSS/Atom feeds, etc. There's a lot of information that can be encoded in semantic XHTML. If you haven't noticed, this blog post has tons of microformats embedded within it.

Conclusion

So, stay tuned, as I begin "the process" of proposing new microformats. I hope to write again in a few days.

Saturday, May 16, 2009

The Good Old Days

Just read an entry on Scott Hanselman's blog called 10 Awesome Things I Remember About Computers. The comments on the entry closed May 1, so I thought I'd write my own entry and sort of chain off his. Feel free to leave your own memories!

10 Awesome Things I Remember About Computers (give or take...)

  1. Using my S100 CP/M based computer with CP/M DOS
  2. Using the same computer above, typing my school reports in WordStar (how I miss WordStar)
  3. Typing up BASIC (in that day, I used A-Basic, M-Basic, GW-Basic, and in the '90s, QBasic) game listings
  4. Upgraded to an IBM XT with a 40MB HDD (that was HUGE, and the largest available consumer HDD)
  5. DOS 3.x was out, and sub-directories were new. I didn't know about them/understand them (I was 11!). Anyway, I watched my dad type a command: del *.* and it deleted some files. Pretty cool. I accidentally wrote a file in the wrong spot. I couldn't get it deleted, so I tried this (not understanding directories, they didn't exist in DOS 2.0!): C:\> del *.*. Rebooted, nothing happened. Hmmm... "DAD! My computer won't boot." He said, "What did you do to it?" To which I said, "I don't know, it just wouldn't start." He wasn't too happy. ;) Luckily, it was only my computer, so none of his stuff was on it. Whew!
  6. Playing Tie Fighter on my 486 (this is now about 5 years later, about 1992 I'd say).
  7. Using DriveSpace on Windows95 to compress my 810MB HDD to get 2.4GB space. Boy was that a bad idea!
  8. Buying my first computer from scratch and putting it together myself (1997) for $2400: PII 350MHz w/ ABIT BX6 mobo, 256MB RAM, 12GB HDD (yeah baby!) and a LS-120 FDD (that was a total waste).
  9. One year later, bought another computer for $1100: PII 400MHz w/ 512MB RAM, ABIT BX6 Rev 2 mobo, Acer 40x CD-ROM, Yamaha 24/8/4 CD-RW drive and another 12GB HDD
  10. Windows 2000. Finally, a true 32-bit OS that worked well.
  11. Windows XP—so much better!
  12. One more for good kicks (though this is recent history)—Microsoft releases their second-most horrible OS (after ME), Vista. Can't wait for Windows 7!

Though I have some recent stuff on my list, I've been using computers since I was 8 (so just shy 25 years). I'd like to hear from you, too. What are some of your memories?

Thursday, May 14, 2009

Book Review: Microformats: Empowering Your Markup for Web 2.0

Microformats: giving birth to the semantic web

I just finished reading Microformats: Empowering Your Markup for Web 2.0. I've heard of microformats here and there over the last 3 or 4 years, but until now, I haven't really looked into them. I saw this book for sale on Amazon.com at a price I couldn't resist. It was high about time that I read about what this semantic web thing is all about.

I thought the semantic web was simply about using the appropriate XHTML when creating your web pages and being sure that you were meeting various other standards along the way, such as WCAG. The semantic web is that and much more.

Microformats give web content designers and developers a way to use existing XHTML to markup their content so that it can be easily read by humans but also easily read by machines. This has the advantage that, obviously, humans can read it, but that the content can be aggregated by various web services.

This book is one of the best organized books I have ever read and I highly recommend reading this book for a detailed, but quick introduction to microformats. In fact, microformats are so easy to use, an introduction is all you really need to begin using them. If you need more information, just check out microformats.org.

This review is written using the hReview format as discussed in the book. Since this is my first attempt at using this microformat, if there's anything wrong with it, please leave a comment. In any event, I give this book 5 stars out of 5.

FireFox 3.0.10 and IE8

Time for another, though much shorter, rant.

My Headaches with Firefox 3

I've had it with Firefox. I was late to jump on the Firefox bandwagon. I jumped on around 2.08 or so and upgraded all the way up to 3.0.10 as recently as two weeks ago. I upgrade Firefox as soon as I can, though I admit, I waited a bit to upgrade to Firefox 3; and it seems for good reason. At first, I was very impressed with Firefox 3. It provided better tab scrolling and seemed much more stable than Firefox 2. That is, until I "upgraded" to Firefox 3.0.9. Then everything came to a screeching halt.

You have to understand that I'm a power browser user. Some would say I'm just crazy. But I have so many things going on that I have a ton of tabs open (at current count, about 344) so that I can quickly get back to whatever else I was doing/researching within the past few weeks (ok, months). As Firefox has continued to release updates, its memory footprint continues to grow, rivaling Internet Explorer with it's famed JavaScript memory leak. At first, Firefox was crashing because I actually had 360+ tabs open and during loading all of the tabs, the process size reached 1.5GB, which is apparently the maximum process size on Windows even though each process has 3GB of memory space (go figure?). Ok, so that's my fault. But since then, I've closed unneeded tabs and had gotten as low as 320. My Firefox process size was around 600MB - 700MB. However, I experienced browser crashes 2 or 3 times a day.

Yay, Firefox came out with an update, 3.0.10. So I upgraded to that seeing that they fixed (yet another) memory corruption issue (how many of those things do they have??). I hoped that it would solve my problem. Nope. Firefox routinely continues to crash at least once a day, if not twice. I tried things like turning off Google Desktop Search (which is a huge memory waster anyway and should be uninstalled from every computer on earth because it puts its hooks into every single program that's opened on your computer). But to no avail. I've done some searching on this issue and have found that many are experiencing the same problem (I hear it can be much worse on Linux). This leads me into my next segment.

Enter Internet Explorer 8

OK, so Microsoft copies folks, but so far as I can tell, IE8 was in the works before Google Chrome hit (though, I can't be absolutely sure about this). Whatever the case may be, Chrome and IE8 use the same "memory model". Each browser tab is a separate process. This has advantages, and disadvantages. However, today, the disadvantages are smaller than they used to be.

When I first heard that Chrome used separate processes for tabs, I was not impressed. Sure, you have tab isolation, but at what price? Processes are heavy in terms of memory and creation time. However, what I didn't realize (and I was very surprised I had missed this fact!) was that in Windows 2000, a new type of process was created—called a job process. A job process is a new process, but it is related to some parent process. It's almost like a Unix fork, but not quite. In any event, these processes, while providing isolation, don't have as steep memory requirements or creation time. IE8 makes heavy use of the job process (as I'm sure Chrome does).

So I started playing around with IE8 last week. I must say I'm pretty impressed. It's stable, and if a tab crashes, it will attempt to reopen the tab. With IE8, you don't have to explicitly save your browsing session, just close the browser and all your tabs are remembered. Personally, I don't care for that. In fact, that's one thing I'm going to miss from Firefox, the Session Manager plug-in. I think one could be written for IE8, but I haven't seen one yet and I don't have time to write one. Anyway, IE8 only lets you save the last browsing session (since it's explicitly saved). You can't save multiple browsing sessions to open one up again at some later point. Too bad. That's a real short-sight. Another plus about IE8 is that it renders much better than it ever used to. I still have to put it through its paces though (and check out how much of ACID2 it passes). Overall, I'm very pleased with it and can't wait to ditch Firefox.

An Open Call (and Mind) to Mozilla

If you're going to release patches for a browser, especially a build update (e.g. 3.0.xxxx), you better do thorough regression testing to make sure you don't break the browser. That's what happened between 3.0.8 and 3.0.9. It just completely broke. 3.0.10 is better, but it's still broken. I think Firefox needs to get back to its roots as a browser: a low memory footprint, extensibility, and stability, and frankly, more cool features that just kill the browser market (tabbing was definitely a killer feature). I'm going to keep my eye on Firefox, but for now, I need stability, speed, and something that just plain works.

P.S.

To the Microsoft haters and open source lovers and the passionate (or even fanatic) people out there: this post is meant as an open honest discussion of my findings in regards to the latest Firefox 3.0.10 browser, and not about the quality of FOSS software, in general. This is particular to Firefox. Also, note that I did not say that I hate Firefox, only that it has become unusable to me. I'm disappointed with it. I hope that the next major (or minor, 3.5, is it?) release will resolve many of the issues I and others have been having. I'm willing to give Firefox another shot, but until I can actually use it to work (without it freezing up every 30 seconds for 10 seconds at a time), I will have to stick with IE8.

Wednesday, May 13, 2009

ATI Catalyst 9.4 Is No Good for Business

Today's post is a bit of a rant. Also, it took me so long to write, that I didn't want to spend much more time formatting it. So, if it looks a bit messy, I'll try to get back and clean it up a bit later. If you've been reading my blog, you know that I'm in the process of a network deployment. Recently, we qualified for a (basically, free) computer form UPS. Unfortunately, all they had to offer were machines with AMD processors and chipsets. If you haven't guessed it by now, I'm an Intel guy. And now, I know why.

The Problem with AMD

Apparently, AMD doesn't want to make it easy for (particularly small) business to install their software. This is evidenced by the fact that they have completely non-standard Windows installers. They probably use InstallShield for some of their software (which, arguably, makes horrible installers, in my opinion, after having looked at just a few InstallShield MSIs with ORCA from Microsoft). However, their Catalyst Display Driver package uses something called Monet, which until last week, I've never heard of before. And let me say that I hope I never hear of it again, as it caused me nothing but trouble this week.

The idea is that I'm using the Microsoft Deployment Toolkit 2008 to deploy my clients. This driver comes with software that I would like to have installed on the machine, so I decided to go ahead and install the whole package (minus a few items that I really don't need). But it turned out to be a real chore. To start, there is the lack of documentation. Here's the help documentation from Setup.exe and InstallManagerApp.exe, which are responsible for the GUI installation:

Figure 1. Setup.exe/InstalManagerApp.exe command-line switches

Well, I'm confused already. But here's what happens. If you use the slash (/) switches, you can get an unattended GUI install. What do I mean? What I mean is, if you use the right combination of switches, you can achieve an unattended install that still shows a GUI. Ok, that's not so bad. Let's step through these switches a bit. By specifying /UNATTENDED_INSTALL, you would think that would be all I have to do to achieve my goal. Well, you would be wrong. Even when using this switch, the installer stops on the very first wizard page to ask what language you want. Why do I need to choose a language if I'm doing an unattended install? (More precisely, if no language is specified on the command line, English ought to be the default and installation should proceed unattended.) So, in order to get around this, we need to pass the /FORCE_HIDE_FIRST_RUN switch, which does just that—it skips the language selection step. So you run the new command and now the wizard appears to stop in its tracks on the last page and just sits there. That's because even during an "unattended" install, the setup program expects you to click the Finish button. Stupid. So we need to pass the /FORCE_CLOSE_WHEN_DONE switch. Just to be extra cautious, I pass the /AUTOACCEPT_ALL switch, which automatically accepts all EULAs (which according to AMD is supposed to be unpublished, hehe). I also pass the /ON_REBOOT_MESSAGE:NO switch to make sure the installer doesn't tell the operating system to reboot, a big no-no when using the Microsoft Deployment Toolkit. So our final command looks like this: setup.exe /UNATTENDED_INSTALL /FORCE_HIDE_FIRST_RUN /FORCE_CLOSE_WHEN_DONE /AUTOACCEPT_ALL /ON_REBOOT_MESSAGE:NO.

Well now there are three problems. In older releases of the Catalyst Display Driver package, HydraVision was installed with everything else and the Windows Vista Experience Index score was also updated so that the Aero user interface would be enabled. This no longer happens in the 9.4 release of the package. It turns out they have a few errors in their new install routine. The lack of installation of HydraVision (a multi-desktop or even desktop spitting application) can be gotten around by replacing /UNATTENDED_INSTALL with /UNATTENDED_INSTALL:"\Packaes\Apps\CCC\HydraVision-Full\ccc-hv-full.msi;<driver_folder>\Packgaes".

Enabling the Aero user interface (in an unattended manner) is also similarly easy. In previous releases, there was an MSI called EnableAero.msi. At first, I was including this in the installation, but I soon realized that all the MSI was doing is running the following command: C:\Windows\System32\WinSat features C:\Windows\System32\WinSAT formal. So, number one, the EnableAero.msi installer is completely unnecessary. Secondly, the way they are running the command is also unnecessary. Instead, there is a command line switch that enables you to only update the graphics and restart the Display Window Manager: C:\Windows\System32\WinSAT dwm.

Now we have achieved the following: an unattended installation (though, still showing a GUI) that installs HydraVision and enables the Aero user interface in Vista. But what if we want a truly silent install? Is this possible? Yes, it is. There are two ways of doing this.

First, there is a file called ATISetup.exe in the Bin{64} folder of the Catalyst Display Driver package. If you run this file without any command line options, you will see the following command line options (there are more, but for brevity, which it's already too late for, I'll omit the unimportant ones):

-install
Performs a silent install
-uninstall [all]
Performs a silent uninstall ([all] uninstalls all hidden packages)
-output screen [detail]
Screen outputs basic information to the console while including detail will output more verbose information. Useful for debugging.
-h #### -v #### -b ## -j ###[ddc]
Sets the horizontal resolution, vertical resolution, color depth (or bits per pixel (a/k/a bits per pel)), and vertical refresh rate. Using DDC for the vertical refresh rate tells the driver to get the information from the monitor via DDC or EDID, if the monitor supports it. Most LCDs in use do, as do most CRTs that are left in use. Also, be aware that the documentation for these switches is incorrect: -b only takes 2 digits, and -j can take 3. They have this reversed on their display of the help options (and thus, you can specify 120Hz refresh rate for your CRT if it supports it).

Here's the kicker: you try to run the following command: ATISetup -install -h 1280 -v 1024 -b 32 -j ddc, and it doesn't work. The install occurs, but again, HydraVision is not installed, and when you reboot, the display defaults to a resolution of 800 x 600 @ 32-bits. Again, the problem with HydraVision can be fixed and the installer used, but not the screen resolution. For me, this is back to square one. Why did I go through all of this? To save you the trouble—and to pick on AMD.

So how do I get the driver to install, with all the companion applications, and have the screen resolution set to the desired size and enable the Aero user interface? It comes down to two lowly batch files (or in Vista world, a .cmd file).

First, we need to write the batch (or command) file that will install the driver and companion applications. Apparently, what really happens when you run the Setup.exe file in the root driver package with the -install switch, it just passes it onto the .\Bin{64}\ATISetup.exe installer. If you use the slash (/) switches, then the GUI installer is used (e.g. the GUI installer is shown, but is unattended if you use the switches I outlined above).

Here's the install batch file (I named it install.cmd):

NOTE: You may freely copy and use the batch file source code contained in Figures 2 and 3 of this blog entry without needing to give attribution. I hope someone may find it useful.

Figure 2. Install.cmd source code.

This will (more or less) silently install the driver and all accompanying software that you chose. (NOTE: I removed the following folders from the \Packages\Apps directory: CatalystRegistration (a registration nag-ware) and Blizzard (an Internet shortcut to download a trial version of the game World of Warcraft.) Once this batch file has been executed, you need to restart the computer. (NOTE: if you are using the Microsoft Deployment Toolkit to deploy your operating system, you must tell MDT that the application requires a reboot and let MDT reboot the computer or you will mess up the automated deployment.)

Once the computer has rebooted, your video resolution will be 800x600 @ 32-bit. Unfortunately, Windows still does not have a way to "script" the video resolution through an API. No Windows Script interface, no WMI, no PowerShell. So we are left to "hacking" the Windows registry. Please note that the batch file below has only been tested on Windows Vista. For any other operating system, verify the keys that need to be set (perhaps the resolution switches only work for Windows XP and below?).

Figure 3. SetVideoRes.cmd source code.

The > nul makes the batch script not output anything to the screen since REG.exe and WinSAT probably use stdout or stderr bypassing the @ECHO OFF command in the batch file. I called this batch file SetVideoRes.cmd. It first uses REG QUERY to determine the GUID of the video card and stores it in a variable. Then it creates several registry entries in the registry pertaining to the display resolution. Finally, it runs the Windows System Assessment Tool to enable the Aero user interface. Once this script has run, the computer needs to be rebooted again so that Windows picks up the new display resolution settings. And finally, there you have it, a completely unattended installation of the Catalyst Display Driver package. Too much work. For Intel, this would have been as easy as calling setup /s /v/qn if it's an InstallShield installer or <driver_installer>.msi /qn /norestart if it were an actual Windows MSI.

I hope this helps someone out there. It took me a whole week, installing the operating system about 25 times to figure this all out. What a pain! I did contact AMD Customer Care, but they basically told me that they provided limited unattended install support and that they only supported unattended installs on Windows Vista for particular ATi products. Poor business practice, in my opinion. Not everyone is a Hewlett-Packard or a Microsoft.

One final note, I just finished a book on microformats on the web. This blog entry is long enough, so suffice it to say, I've tried my hand at a few hCard entries in this post. Not every company mentioned has an hCard entry. I may go back and update this entry later. I mention this only for those who may be looking. I'm going to write more about that in another post.

That's it for now. Sorry it was so long. Take care.

Updates

  • 2009-05-14 - Fixed code in SetVideoRes.cmd in figure 3: Added quotes around the registry key because the keyname has a space. Also removed the warning about testing on Vista. I have tested the code and it does work. However, it should go without saying that anything you take off the Internet should be tested in a secure, isolated environment before being put into production. Also fixed my SetVideoRes.cmd batch file to call C:\Windows\System32\WinSAT formal to make sure that the experience score actually gets updated in the System display window.