In reply to Josef "Jeff" Sipeks reply to my post entitle SMTP -- time to chuck it from a couple of years ago.
This is a (long overdue) reply to Ilya's post: SMPT -- Time to chuck it.
[...]
There are two apparent problems at the root of the SMTP protocol which allow for easy manipulation: lack of authentication and sender validation, and lack of user interaction. It would not be difficult to design a more flexible protocol which would allow for us to enjoy the functionality that we are familiar with all the while address some, if not all of the problems within SMTP.
To allow for greater flexibility in the protocol, it would first be broken from a server-server model into a client-server model.
This is first point I 100% disagree with...
That is, traditionally when one would send mail, it would be sent to a local SMTP server which would then relay the message onto the next server until the email reached its destination. This approach allowed for email caching and delayed-send (when a (receiving) mail server was off-line for hours (or even days) on end, messages could still trickle through as the sending server would try to periodically resend the messages.) Todays mail servers have very high up times and many are redundant so caching email for delayed delivery is not very important.
"Delayed delivery is not very important"?! What? What happened to the whole "better late than never" idiom?
It is not just about uptime of the server. There are other variables one must consider when thinking about the whole system of delivering email. Here's a short list; I'm sure I'm forgetting something:
- server uptime
- server reliability
- network connection (all the routers between the server and the "source") uptime
- network connection reliability
It does little to no good if the network connection is flakey. Ilya is arguing that that's rarely the case, and while I must agree that it isn't as bad as it used to be back in the 80's, I also know from experience that networks are very fragile and it doesn't take much to break them.
A couple of times over the past few years, I noticed that my ISP's routing tables got screwed up. Within two hours of such a screwup, things returned to normal, but that's 2 hours of "downtime."
Another instance of a network going haywire: one day, at Stony Brook University, the internet connection stopped working. Apparently, a compromised machine on the university campus caused a campus edge device to become overwhelmed. This eventually lead to a complete failure of the device. It took almost a day until the compromised machine got disconnected, the failed device reset, and the backlog of all the traffic on both sides of the router settled down.
Failures happen. Network failures happen frequently. More frequently that I would like them to, more frequently than the network admins would like them to. Failures happen near the user, far away from the user. One can hope that dynamic routing tables keep the internet as a whole functioning, but even those can fail. Want an example? Sure. Not that long ago, the well know video repository YouTube disappeared off the face of the Earth...well, to some degree. As this RIPE NCC RIS case study shows, on February 24, 2008, Pakistan Telecom decided to announce BGP routes for YouTube's IP range. The result was, that if you tried to access any of YouTube's servers on the 208.65.152.0/22 subnet, your packets were directed to Pakistan. For about an hour and twenty minutes that was the case. Then YouTube started announcing more granular subnets, diverting some of the traffic back to itself. Eleven minutes later, YouTube announced even more granular subnets, diverting large bulk of the traffic back to itself. Few dozen minutes later, PCCW Global (Pakistan Telecom's provider responsible for forwarding the "offending" BGP announcements to the rest of the world) stopped forwarding the incorrect routing information.
So, networks are fragile, which is why having an email transfer protocol that allows for retransmission a good idea.
Pas touche! I have not conducted extensive surveys of mail server configurations, but, from personal experience; most mail server give up on sending email a lot sooner than recommended. RFC 2821 calls for a 4-5 day period. This is a reflection of the times, email is expected to deliver messages almost instantaneously (Just ask Ted Stevens!).
As you are well aware I am not implying that networks are anywhere near perfect, it just does not matter. If you send a message and it does not get delivered immediately your mail client would be able to tell you so. This allows you to reacts, had the message been urgent you can use other forms of communication to try to get it through (phone </gasp>). The client can also queue the message (assuming no CAPTCHA system, more on that later) and try to deliver it later. Granted machines which run clients have significantly shorter uptimes than servers but is it really that big of a deal, especially now that servers give up on delivery just a few hours after first attempt?
I, for one, am looking forward to the day when I no longer have to ask my potential recipient whether or not they have received my message.
Instead, having direct communication between the sender-client and the receiver-server has many advantages: opens up the possibility for CAPTCHA systems, makes the send-portion of the protocol easier to upgrade, and allows for new functionality in the protocol.
Wow. So much to disagree with!
- CAPTCHA doesn't work
- What about mailing lists? How does the mailing list server answer the CAPTCHAs?
- How does eliminating server-to-server communication make the protocol easier to upgrade?
- New functionality is a nice thing in theory, but what do you want from your mail transfer protocol? I, personally, want it to transfer my email between where I send it from and where it is supposed to be delivered to.
- If anything eliminating the server-to-server communication would cause the MUAs to be "in charge" of the protocols. This means that at first there would be many competing protocols, until one takes over - not necessarily the better one (Betamax vs. VHS comes to mind).
- What happens in the case of overzealous firewall admins? What if I really want to send email to bob@example.com, but the firewall (for whatever reason) is blocking all traffic to example.com?
[...]
And so this brings us to the next point, authentication, how do you know that the email actually did, originate from the sender. This is one of the largest problems with SMTP as it is so easy to fake ones outgoing email address. The white list has to rely on a verifiable and consistent flag in the email. A sample implementation of such a control could work similar to the current hack to the email system, SPF, in which a special entry is made in the DNS entry which says where the mail can originate from. While this approach is quite effective in a sever-server architecture it would not work in a client-server architecture. Part of the protocol could require the sending client to send a cryptographic-hash of the email to his own receiving mail server, so that the receiving party's mail server could verify the authenticity of the source of the email. In essence this creates a 3 way handshake between the senders client, the senders (receiving) mail server and the receiver's mail server.
I tend to stay away from making custom authentication protocols.
In this scheme, what guarantees you that the client and his "home server" aren't both trying to convince the receiving server that the email is really from whom they say it is? In kerberos, you have a key for each system, and a password for each user. The kerberos server knows it all, and this central authority is why things work. With SSL certificates, you rely on the strength of the crypto used, as well as blind faith in the certificate authority.
They might, the point is not so much to authenticate the user but to link him to a server. If the server he is linked to is dirty, well you can blacklist it. Much of the spam today is sent from bot-nets, in this schema all the individual botnet senders would have to link themselves to a server. Obviously, a clever spammer would run a server on each of the zombie machines to auth for itself. The catch is that he would have to ensure that the Firewalls/NATs are open and that there is a (sub-) domain pointing back at the server. This is all costly for the spammer and for the good guy it'll be easy to trace down the dirty domains.
At first it might seem that this process uses up more bandwidth and increases the delay of sending mail but one has to remember that in usual configuration of sending email using IMAP or POP for mail storage one undergoes a similar process,
Umm...while possible, I believe that very very large majority of email is sent via SMTP (and I'm not even counting all the spam).
Carton jaune, I addressed that issue in my original posting which is just 2 sentences below this one. Excessive lobotomy is not appreciated.
first email is sent for storage (over IMAP or POP) to the senders mail server and then it is sent over SMTP to the senders email for redirection to the receivers mail server. It is even feasible to implement hooks in the IMAP and POP stacks to talk to the mail sending daemon directly eliminating an additional socket connection by the client.
Why would you want to stick with IMAP and POP? They do share certain ideas with SMTP.
Carton rouge, I said nothing about sticking to IMAP/POP. The point is that the system can be streamlined somewhat.
For legitimate mass mail this process would not encumber the sending procedure as for this case the sending server would be located on the same machine as the senders receiving mail server (which would store the hash for authentication), and they could even be streamlined into one monolithic process.
Not necessarily. There are entire businesses that specialize in mailing list maintenance. You pay them, and they give you an account with software that maintains your mailing list. Actually, it's amusing how similar it is to what spammers do. The major difference is that in the legitimate case, the customer supplies their own list of email address to mail. Anyway, my point is, in these cases (and they are more common than you think) the mailing sender is on a different computer than the "from" domain's MX record.
I do not think that increasing the burden on mass mailers even good ones is such a bad thing.
[...]I really can't help but read that as "If we use this magical protocol that will make things better, things will get better!" Sorry, but unless I see some protocol which would be a good candidate, I will remain sceptical.
And I can not help but read this as "We should not think about improving protocols because it impossible to do better." In any case I appreciate your mal-parè. The discussion is important as letting protocols rot is not a good idea.
[...]2009-04-22 10:47:24
Recently my local LUG (LILUG) invited Eric S. Raymond (ESR) to come and speak. For those of you who are not familiar with ESR, he is one of the three largest icons of the Open Source/Free Software movement. Needless to say, it was an honor so see him speak. For the most part, his talk was quite tame but one of the points he raised seemed quite controversial. According to him the GPL and other viral licenses are no longer needed as they do more harm than good to the community. I don't want to put words into his mouth so I've transcribed what he said during the talk. You can view the ESR Q/A talk in its entirety, this specific excerpt is about 45 minutes into the video.
What is the point of being famous and respected if you can't speak heresy about your own movement. What is the point?
One of my heretical opinions is that we worry way too much about licensing. And in particular; I don't think we really need reciprocal licensing. I don't think we need licenses like the GPL, that punish people for taking code closed-source. Let me explain what I think. And then I'll explain [why] the fact we don't actually need those [licenses] matters.
I don't think we need them because. There has been a fair amount of economic analysis done in the last 10 years, significant amount of it has been done by, well, me. Which seems to demonstrate that open source is what the economist call a more efficient mode of production use, superior mode of production. You get better investment, better return out of the resources you invested by doing open source development than closed source development. In particular, there have been a number of occasions on which people have taken open source products that were reasonable successful, and just taken them closed. Effectively putting them under proprietary control, proprietary licensing and then tried to make a business model out of that. They generally fail. And the reason they fail is pretty simple. That is because when you take a product closed, you are now limited to what ever small number of developers that your corporation can afford to hire. The open source community that you just turned your back on does not, they have more people than you. They can put out releases more frequently, getting more user feedback. So the suggestion is, simply because of the numerical dynamics of the process: taking open software closed is something that the market is going to punish. You are going to lose. The inefficiencies inherent in closed source development are eventually going to ambush you, going to [inaudible] you, and your are not going to have a business model or product anymore. We've seen this happened number of times.
But now, lets look at the implications of taking this seriously. The question I found myself asking is: if the market punished people for taking open source closed, then why do our licenses need to punish people for taking open source closed? That is why I don't think you really need GPL or a reciprocal licenses anymore. It is attempting to prevent the behavior that the market punishes anyway. That attempt has a downside, the downside is that people, especially lawyers, especially corporate bosses look at the GPL and experience fear. Fear that all of their corporate secrets, business knowledge, and special sauce will suddenly be everted to the outside world by some inadvertent slip by some internal code. I think that fear is now costing us more than the threat of [inaudible]. And that is why I don't we need the GPL anymore.
-- Eric S. Raymond
Eric then went on to say that the BSD license is a good alternative to the GPL. This has sparked a heated discussion on the Free Software Round Table (FSRT) radio shows mailing list. While one can admire of the simplicity and clarity of the license it seems far fetched to say that it should be replacing the GPL. While yes there are economical incentive for corporations to keep code Open Source but the relative cost of closing the source depends largely on the size of company. Yes some small companies will not be able to afford to keep a code base alive with internal/contracted developers for larger companies the costs are a lot easier to digest.
Prime example of such a large company is Apple. In 2001 Apple came out with a completely new version of its operating system, MAC OS X. Although a successor to MAC OS 9, it was very different. OS X borrowed a very large code base from the BSDs, and some (pretty much everything but Darwin) of the code was effectively closed. This has not prevented Apple or OS X from thriving.
From the other end of the spectrum, are the companies such as MySQL AB which produce Free Software but also sell closed source licenses of the same code for a 'living.' There is a market for this, it exists because of those scared lawyers and corporate bosses. Killing the GPL would effectively kill this market, as a result development on some of these projects would slow down significantly.
The Open Source/Free Software movement is thriving, it does not mean its a good time to kill the GPL. In fact I don't think there will ever be a time when killing the GPL will do more good than harm.
2009-03-23 11:30:14I've finally caught up on rendering all the LILUG presentations. A total of 5 were posted within last 24 hours, among them the much requested Eric S. Raymond Q/A session. If your clock reads around 1236983978 UET then some of the talks have not been derived yet. That means you can't stream them or watch the thumbnails, but you can download them in the glorious OGG Theora format. Enough talking, find links below.
Enjoy
2009-03-13 18:48:11I've gotten into a photo mood lately. Both shooting editing and organizing. With it I've discovered some new useful tools as well gotten to know the ones I've used before better.
The first and foremost is digiKam its a photo manager. Its primary job is maintain a database of all your photos. Photo-managers are not something most people have used so it might need some getting used to. The interface for digiKam is quite intuitive and easy too pick up. And for the average photo junkie it will have everything they need. But it certainly lacks some features which I think a self respecting photo managers must have. Here are some things I wish it had:
Most of these issues are not major, especially since some of these will be solved with the multi-root support of the KDE4 release. I started with the negatives but it has a lot of cool features also. One of my favorite is calendar view. Regardless of how your galleries are organized it will use the EXIF date tag to arrange all your photos by date. It really helps when organizing photos. Tagging is also very useful, you can tag any photo and then you can view all photos by particular tag really make it easy to organize data. DigiKam also has a slew of semi-functional export features such as gallery2, flickr, and picasa. These are provided through the kipi framework, they are nice but most require some more work to become completely feature-full and userfriendly.
Almost forgot, digiKam is also an excellent tool for downloading photos from cameras. Most cameras are not plain UMS devices so they need special software to fetch the pictures out of them. If you are on windows you can usually use the manufacturer software to do this, but on Linux this is a tad complicated. Unless of course you use digiKam -- which turns the process into a magic "detect [the camera type] and download" single click operation.
To share my photos with the world I use a web based photo-manager as a front-end to my local database. Its called gallery. I have tried this tool in the past and it was just too cumbersome to use (I ended up writing my own PHP gallery system). But with the kipi export plug-in to digiKam and the remote plug in to gallery life just become easy.
The last few tools are only important for someone who is seriously into photography. The first is a gimp plug-in called ufraw, its basically a frontend to dcraw. It allows you to preform advanced raw editing before you import your photo to gimp -- you can adjust almost any aspect of your raw file conversion (lightness, white balance, hue, saturation..). UFRaw is a bit daunting but you don't always have to use all the features it provides, lightness is probably the only one you'll have to adjust on a regular basis. Another tool is called exiftool its used to read and manipulate EXIF information in pictures. There are times where you can loose the EXIF data while editing a photo (IE when saving to png in gimp) and using this tool you can quickly clone the EXIF info of one file onto another using the -TagsFromFile option. It even supports batch mode, for example "exiftool -TagsFromFile IMG_%4.4f.CR2 *.png" will copy the EXIF information to all PNGs from its parent file using the file name as mapping (sample file names: IMG_2565.png IMG_2573_1.png IMG_2565.CR2 IMG_2573.CR2)
So that's it for now, shoot away. And if you like, you can check out my public gallery.
2008-06-17 23:19:55My instant message client of choice is centerim (a fork of centericq). It does everything I need, send and receive messages in a very simple interface. Now this might sound like any ordinary client out there. But its special in that it runs completely in the terminal (ncurses based) -- and its good at it. I've tried some other terminal based clients and they all feel very cumbersome.
One major inconvenience with ncurses applications is the lack of clearly defined text areas. So copying text out is not trivial in fact its nearly impossible. So usually if I need to get text out of the application I just look in its log files. Unfortunately centerim has not-so-convenient history log format. It looks something like this:
IN MSG 1212455295 1212455295 pong OUT MSG 1212455668 1212455668 pong
(each message entry is separated by "\f\n" not just "\n")
So using a little awk magic I wrote a simple converter which parses history file into something more readable and something you can paste as a quote.
gawk -vto=anon -vfrom=me 'BEGIN {FS="\n";RS="\f\n";}{if (match($1,"IN")) a=to; else a=from; printf("%s %s:\t %s\n", strftime("%H:%M:%S", $4), a, $5);for (i=6; i<=NF;i++) printf("\t\t%s\n", $i);}' /PATH/TO/HISTORY/FILE
You need to modify the -vto and -vfrom values to correspond to your name and the name of the person you're talking to. You obviously need to also specify the path to the file. If you don't like the time stamp you can alter the string passed to strftime (man 3 strftime for format options).
Sample output of the above sample looks like this.
2008-06-04 14:25:1221:08:15 anon: ping 21:14:28 me: pong