You have mail! (posted: 13:25:13 29/01/2008)
As we're moving along with the project, the time has come to talk about emails, and there's a lot to talk about.
Some people have wondered why the project hasn't sprung forth with email functionality already built in. Consider:
There are 4,200 groups within TFN, with 4.3 million users. Let's be pessimistic and say that over half of those members are duplicates - let's say the real figure is 2 million individuals. (A nice round figure)
Now assume that the average group generates 20 emails per day (Birmingham, UK generates 150!) and that only one in three is on 'Individual Emails' - that's (2,000,000 / 3) x 20 = 13,333,333 emails per day, or 154 emails per second! (on average)
And that doesn't even count the people on the digests - the time (or post ID) of the last email they received needs to be recorded, and checked each time to see whether this person needs another digest sent out. If we assume (once again) that only one in three is on the digest option (666,666 individuals,) that all groups make 20 posts per day and that on average everyone is on two groups (see the adjusted total, above) then that means the number of checks for new digests each day would be 666,666 x 20 x 2 = 26,666,640 - that is, on average, 308 checks per second, which is another impressive figure. Even if half of those people actually get an email sent each day, that's still another 333,333 emails, or 3 per second.
Looking at my mailbox for Birmingham Freecycle, most emails seems to be about 5Kb in size. With 13,333,333 individual emails and 333,333 digests (of 25 emails) that's a daily data transfer figure of (13,333,333 + (333,333 x 25)) x 5 = 108,333,290Kb or 103Gb!
And we haven't even considered incoming emails...
So, the short answer is: it's a big thing. Really big. We think we know how it can be done, but we're being cautious so we get it right.
(note: those were some pretty big numbers I was juggling - if you spot an error in my math, let me know :) )
People of Earth, your attention please. This is Prostetnic Vogon Jeltz of the Galactic Hyperspace Planning Council.
(posted: 09:33:35 26/09/2007)As you will no doubt be aware, the plans for development of the outlying regions of the Galaxy require the building of a hyperspatial express route through your star system, and regrettably your planet is one of those scheduled for demolition. The process will take slightly less than two of your Earth minutes. Thank you.
(posted: 16:51:01 02/02/2007)Wow! Has it really been three weeks since I last did this? Obviously so...
Well, in my defence I'll say it's been an interesting three weeks. Things are really starting to shape up, and now the
groundwork has been done things should start to pick up pace.
A quick overview of what's working right now:
* Joining the Network
* Browsing to find Groups
* Joining Groups (automatic - pending queue is still to be done)
* Viewing Posts on Groups (basic, but still being worked on)
* Making Posts to Groups (automatic - pending queue still to be done)
I've also noticed that our original spec document completely neglects any mention of Members leaving Groups - oops! That'll
obviously be available (and if I forget, it'll be here to remind me)
So, plans for the next week:
* Finish the viewing of Posts
* Responding to Posts
(Searching will be coming Real Soon Now™)
Don't worry, the toys for Moderators will be coming soon. I realise that this is going to get interesting (we all mod our
groups in different ways) but we need to at least start somewhere, and then see how we can work better.
So, back to the grindstone...
(posted: 13:48:36 15/01/2007)Okay, scratch what I've said about this page being ugly. Erik is obviously unhappy about being associated with something so ugly, so he's done some styling for me. :)
(posted: 16:57:04 12/01/2007)Once again, I have that Friday feeling. Despite having spent most of the week suffering from what feels like Captain Trips, things have still progressed (albeit a bit slower than I'd have liked). I've even managed to write an email on OIDG that has been described as 'enjoyable' and 'informative', but don't worry folks - I'll be back to my usual terse and surly self soon enough... :D
So, what's been happening? Well, the signup form is pretty much complete now, so that's nice. Still working on the group pages, but have had to do some admin and config work
The most exciting thing this week has been getting Erik Calop up and running as the frontend design lead. He's taken over the HTML side of things, which has the main advantages that:
1. I should have more time to concentrate on the techie bits,
2. The project isn't so reliant on a single person now, and
3. The new site shouldn't look as boring as this page. :)
I'll probably be mobilising more of the volunteer army once I'm feeling a little better, but this is an important first step. As we get more people into the teams, the rate of progress should pick up (and shouldn't suffer so much when people are ill...)
So, more (slightly) techie stuff...
The new site is being built with a clear split between "how it works" and "how it looks" - developers vs. designers.
The 'designer' team will be doing all of the HTML and text stuff (the "how it looks") but won't be dealing with any of the actual code. This leaves the creative people to be creative, without any major worry that they'll break something.
The 'developer' team will be doing all of the major coding, but pretty much all of their work will be behind-the-scenes. This means the techies can work on the best, fastest and most efficient ways of getting things done without having to worry about making it look pretty as well.
Good thing is that both teams can work pretty much independently of each other, which should make things even faster. :)
Sorry, what with the cold and sleep deprivation due to Mr "Kilroy was here" I've got the attention span of a kitten in a Christmas tree. I'm going to post this now before something shiny distracts me, or I fall asleep, or something...
(posted: 15:21:20 05/01/2007)Looks like it's that time of the week again... (sigh)
First of all, the eagle-eyed amongst you might have noticed there's now an RSS feed available. It's probably not my greatest
bit of work, but it was the best I could fit into the 15 minutes I have for a cup of tea before leaving for work in the
morning, and it seems to work with Thunderbird and passes the
W3C Feed validator, so I'm happy with it for the moment.
So, what has the last week brought us?
First of all, it's been a bit of a slow week, what with all of the festivities, but progress has still been made. Thank
you to all the people who have volunteered to help - if you haven't received an email from me, there's probably one in my
'Drafts' folder. Give me a nudge in a couple of days to remind me to finish it and send it. :)
Speaking of volunteers, I'm getting the first batch ready to start work. The pace of work should pick up once people start
getting comfortable, but my main aim is to get a team in place that'll be able to continue with any development work
that's needed, both in terms of the initial version of the new system and any new things that people ask for. Like any
real website, the new system will probably never be finished - there'll always be some new ideas to try out.
Finishing - I remember seeing on one of the groups (can't remember which one - sorry!) about how difficult it is to gauge
how close the project is to completion without knowing the full spec. This is a fairly tricky one - there are certain
third-parties that would be extremely interested in this information, and the spec only covers a basic system (which might
spook people into thinking "Is that all we're going to get?", so at this moment in time it's probably best if we don't
reveal too much.
Briefly, the system is split into four groups of people - Visitors, Members, Mods and Admins. There's
some parts there for Visitors, work is in progress to make Visitors into Members, not much available for Mods right now,
and there's a couple of things available for Admins. I might be persuaded to guesstimate percentages, but don't hold
your breath...
Breaking this week's work down:
Forgotten passwords - users can request a new password for their account if they forget it. They're given a randomly
generated password, and will need to change it when they next log in. Always a useful feature to have.
Facility to add notes to users accounts in the master admin section - the first step towards letting mods add notes to users
accounts. These notes will be available to all mods, so if someone's playing up on another group it might give advance
warning, and if someone's really making a pain of themselves it'll be easier to spot.
Group navigation pages - visitors will be able to see how many members and the number of posts in the last 7 days on groups
before they join them. There's also a basic search on region/group name, just to make things easier for users to find
groups.
Joining - still working on this bit, but the new member's country, region and "home" group will be stored (if available)
with their new account. I'm sure the statistic fans out there will enjoy the numbers...
This week? Getting the volunteers working, finishing off the signup stuff, start working on the group stuff. Nothing
particularly exciting, I'm afraid.
If you're not interesting in the geeky stuff, you can stop reading now :)
Still here? Then let us begin!
There has been some call for some design decisions to be explained, so while I've still got some time/energy/patience left
let's start with one of the most important ones - the choice of database system. And let's make it clear right now that
I'm not interested in getting into a RDBMS-war...
The contenders were: Microsoft SQL Server, MySQL, PostgreSQL and
Oracle.
Oracle, while having some really nice features, immediately fell out of the running due to the extremely high cost of
licences. There's just not that kind of money available, and even if there was, it could probably be put to better use.
Next casualty was Microsoft SQL Server. Once again, fairly high licence costs (but a lot lower than Oracle!) but the
additional overhead of having the entire Windows operating system running doesn't make it particularly appealing (why
have an entire GUI for a server that'll only run one service?). Coupled with some high-profile virus vulnerabilities in the
past, Microsoft SQL Server was rejected.
So, we're left with the two Open Source servers - MySQL and PostgreSQL. And this is where the fighting usually breaks
out...
Cost-wise, there's nothing in it - both are free. MySQL does offer a paid-for 'Enterprise' edition, but that's still
moderately pricy and didn't really offer any really good return on investment.
Feature-wise, they've both got the basics (which helps!)
MySQL has support for replication, and newer versions have support for stored procedures and triggers, but these could cause
issues with replication. Things like foreign keys and ACID compliance are only available by using InnoDB tables, and InnoDB
was recently eaten by Oracle, so I'm not entirely convinced about the long-term future of InnoDB in MySQL.
PostgreSQL has full support for transactions, foreign keys, stored procedures, triggers, and has ACID compliance, but lacks
replication support out of the box (there are extensions to do it, but I'd be much happier if they were part of the core
server).
In the end, replication was the key issue that won the day for MySQL. The other things would have been nice (and I'm sure
the MySQL guys will get there eventually) but scalability is a lot easier to acheive with replicated database, and that
(scalability) is a key concern.
And to answer anyone's thoughts about bias, I personally prefer PostgreSQL (all of my own stuff uses it), but I'm also
a certified MySQL 4.0 Professional (and
I should get round to becoming a MySQL 5.0 DBA by the end of the month), so it's really 50/50 with me.
Phew! I think that should be enough for now... :)
(posted: 11:52:12 29/12/2006)Okay, so how far have we got...
Coding has started - there's a skeleton site already, although there's a lot missing from it, and even more TODO notes. We've got an exciting mockup for the front page, which we'll be implementing once the holidays are out of the way.
Members: A basic joining page is ready (still needs some additional work to capture a 'home' group/region). Still to do: the entire member's "home" pages.
Groups: browsing countries/regions to find groups works. Still to do: searching for groups, group main page, and all the other goodies that go with groups.
Moderators/Owners tools: still waiting for the groups pages to be completed. These features are in addition to the features that 'normal' members have, so it makes sense to complete that work first.
Languages: English, mainly, but we've got some French-speaking people available to do that language, and there's some German volunteers in the pipeline as well. We'll probably just stick to these three languages for the moment - it should be enough to highlight any problems adding other languages might pose.
Volunteers: could still do with some more techie Perl help (PHP is close enough) but we should have the HTML side of things covered. Email address at the top. :)
Uhh...brain dump running out of steam, so I'll end it here. If you'd like to know about something, email address is at the top. I can't guarantee I'll include it, but if you don't ask you've got no chance of getting. :)
New blog created
(posted: 10:55:59 29/12/2006)New blog created.
First of all, let me point out that I hate 'blogs' and 'blogging' - I'm far too busy living my life to write about it. Don't expect Shakespeare here...
This is intended to give people a general overview of what's happening within the project. I'm not going to go into great details or long discussions about where the project is headed, or how much progress has been made, or how decisions were made - I really can't spare the time on these diversions. Suffice to say, all of these things have been carefully considered.
So, what is the point of all this? Well, I know there are plenty of people who are interested in the project and about moving away from Yahoo!Groups, and as the actual development group is a closed mailing list the information on it isn't publicly available. It can also get a bit technical on there, so filtering out the good stuff can be difficult.
And so this page is born. I'll try to update it every week with the current progress, but don't shout at me if I'm late or miss an entry (I'm probably too busy actually doing the work to write about it!) If you're all really, really good, I might even answer any questions sent to me, but I guarantee nothing. :)