Australian Weather Data (RSS…)

Despite my best efforts, I've been disappointed in my (repeated) attempts over the last twelve months to locate a free Australian weather feed I can consume via RSS and present as I wish. The Bureau of Meteorology is operating in the dark ages when it comes to the distribution of its authoritative weather data, Yahoo! weather—although available in a nicely structured XML document—is not available for commercial use, and all other feeds available at the time of writing send down HTML-formatted data with very limited rights for modifying the HTML itself.

Naturally, this is all very frustrating, especially when your client's intranet site needs weather for three specific towns in the middle of the outback or you're trying to build a redistributable weather widget (SharePoint web part anyone?).

Having given up on a free RSS feed, I returned to the BoM site as they do provide weather data via FTP (they also provide registered users access to additional services for a subscription fee). The terms are a bit unclear about distribution and attribution so I'll leave them up to you to interpret. This post presents my approach to accessing and parsing the basic (free) weather data file. A bit more work is involved than simply pointing a nice XDocument at an RSS feed URL and LINQ-ing away but hey, lower-level stuff is always fun and educational ;-)

By the way, this code will eventually make its way into SharePoint web part; if you're interesting in licensing this web part for use on one of your own sites, please drop me a line – info@mediawhole.com.

Access

The BoM provides weather data in the form of .dat, .txt, .html, and other file types via anonymous FTP. The .dat file is what we're after and for this example, I'll be using the IDA00006.dat file (Western Australia).

.dat files contain forecast information by town with details separated by a hash (#) character. The first line in the file describes the file format so if you get lost is a sea of hashes, start there. The Precis Forecast Products page provides formal documentation, of a sort.

Since the .dat file is just a text file we'll be using the WebClient class to manage access to the FTP location and open a readable stream over the .dat file URL:

WebClient client = new WebClient();

using (Stream data = client.OpenRead("ftp://ftp2.bom.gov.au/anon/gen/fwo/IDA00006.dat";))
{
}

Parsing

With access to the file contents, we can now parse it for easier access. To do so, we'll use a StreamReader to enumerate the file, tokenize each line, and store the results in a generic list:

using (StreamReader reader = new StreamReader(data))
{
    List<List<string>> weatherData = new List<List<string>>();
    string currentLine;

    while ((currentLine = reader.ReadLine()) != null)
    {
        string[] tokens = currentLine.Split('#');
        weatherData.Add(tokens.ToList<string>());
    }
}

It's worth pointing out the nested list structure I'm using here in place of a dedicated collection type. Each line in the file represents data for a single town; individual data elements within each line are separated by a hash character. To accommodate this, I tokenise each data element using the String.Split() method after reading the next line and store those elements in a nested list; each outer list item therefore contains data for a single town.

If that sounds at all complicated, here's a picture of the end result:

The parsed data is finally cached (not shown) for subsequent access. Notably, the BoM doesn't specify any limits on access attempts per day or an effective TTL value as do many RSS feeds. There's nothing stopping you from retrieving this data with every page load but your site will likely slow down and you'll also be guilty of gumming up the intertubes. You may also run the risk of being blocked from accessing the site.

Data Extraction

The final step in this process is locate and extract the weather data you're after (most likely for a specific town). I use the List<T>.Find() method, supplying a predicate that matches on town name, to accomplish this:

List<string> town = weatherData.Find(currentTown => currentTown[WeatherIndex.Location].ToLowerInvariant() == "perth");

Final Touches

The free data doesn't provide any form of condition code so associating a forecast (e.g. "Mostly sunny") with an icon is going to be tricky. I'm hopeful the BoM uses standard phrases that can be matched (or at least partially matched as a fallback (e.g. contains "sun") against known terms for hookup to generic icons but this could become a maintenance problem.

But that's it, weather!