MavEtJu's Distorted View of the World

FreeBSD mirrors Spring cleanup!

Posted on 2008-09-02 15:00:00
Tags: FreeBSD, Networking

It's September, which means it is Spring in Australia! The semi-cold weather is gone, the birds are shrieking (still) and the trees still have leaves. And what better can you do in Spring than having a Spring Clean?

This years Spring Clean will be about the mirrors of the FreeBSD project. Over the years a huge amount of mirrors have been gathered, and most of are pretty good. Unfortunately, one or two are broken, not up-to-date, out-of-service, have disappeared or can't be found.

Earlier this month, I did the Spring for the Australian mirrors. DNS is now clean, FTP mirrors are all fine again except for ftp4.au.freebsd.org, CVSup now points to a working server, rsync is still not available and the ISO distribution looks fine. It wasn't even Spring yet, imagine that!

Without statistics you don't know anything, so I made a tool which checks for...

The availability and age of the file TIMESTAMP in the FTP mirrors. This file is updated once per hour on the FTP master site.
The availability of the ISO images on the FTP mirrors. These files are only created at release time, but at least it shows where you can find your ISO images!
The availability and age of the file CVSROOT-ports/modules on the CVSup mirrors. This file is updated once per day.
The availability and age of the file TIMESTAMP in the RSync mirrors. This file is updated once per hour on the FTP master site.
The availability of all dates on the FreeBSD.org main website on the WWW mirrors. This is not often updated (couple of times per month), but it is the only way I could find to overcome timestamp issues with locally generated HTML files.
The consitency of the SOA serial numbers in DNS, and to make sure that all CNAMEs are pointing to defined DNS entries.

So, where to find these goodies? At http://www.mavetju.org/unix/freebsd-mirrors/ is the main overview. The score is calculated on the number of (correct) elements found in each test. It is possible to browse through the history of the statistics (as far as collected that is) and to see which items are changed between two dates.

I have applied for a hat to contact the mirror maintainers and ask them to fix the issues, but haven't heard anything yet.

| Share on Facebook | Share on Twitter

Comments:

From:	Tim
URL:
Posted on:	2008-09-03 07:58:15
Comment	Maybe this information could be analysed long term to reorder mirrors? (and chuck away dead or poorly available ones, of course) For example, if ftp.cc.freebsd.org is regularly a couple of days out of date, and ftp2.cc.freebsd.org is bang up to date, they could be swapped around. Good effort in sorting all this data out btw!
Reply	It is indeed information which can be used to re-order, but at this moment the first goal is to collect the data and weed out the bad stuff. Once it is looking okay again, we can do more sophisticed and long-term analyzing. But be careful, because people often use a specific ftpX.cc.FreeBSD.org because it is close to them (networkwise).