OfflineIMAP community's website

v7.0 series is ready, deprecating the v6.7 series

Sat, 11 Mar 2017 00:00:00 +0000

Near 8 months after the first v7.0.0 release the latest v7.0.14 is quite reliable.

The v7.0 series
Time to deprecate the v6.7 series
Welcoming the official testers

The v7.0 series

The v7.0 series was born due to breaking changes. The main change was the move to the SQLite backend driver by default.

A good work have been done to improve this series in different areas:

the SQLite backend;
concurrency issues;
the structure algo comparison;
IMAP/IMAP syncs;
XOAUTH2;
mbnames;
some fixes for some other bugs.

Actually, all the releases since v7.0.0 are small bugfix releases.

With this fresh new release of v7.0.14, we get a reliable version of offlineimap.

Time to deprecate the v6.7 series

At the time of the release of v7.0.0 I’ve officially provided support for the v6.7 series to make the transition as smooth as possible. There is not anymore reasons to keep it and remain with the legacy text backend.

I plan to remove the DEPRECATED legacy text backend in the near future. This may happen for the next release.

Welcoming the official testers

We improved the release cycle by adding a new step. Before new releases, I freeze the code and make a call for testing. The official testers are expected to try and test the tip of the git branch called next.

This is usually done after I’ve tried it myself. Actually, I’m always running the tip of the next branch for all my offlineimap accounts.

The distribution maintainers are already very involved for a long time. I’m happy some users decided to join our efforts. Here are the current official testers:

Abdó Roig-Maranges
Ben Boeckel
Chris Coleman
Darshit Shah
Eygene Ryabinkin
Igor Almeida
Ilias Tsitsimpis
“J”
Łukasz Żarnowiecki
Rainer M Krug

Obviously, newcomers are welcome at any time. We provide better support to the official testers. To become a member of this team you just have to declare you intend to join. Let us know! ,-)

We’ve written documentation about testing offlineimap.

tracking regressions

Fri, 29 Jul 2016 00:00:00 +0000

Four days after v7.0.0 and v6.7.0.2, I’ve had to release v7.0.1 and v6.7.0.3. One day later it was for v7.0.2. Again, I intend a v7.0.3 soon. Something gone wrong, obviously.

The release cycle
Oops, this failed
Imagine yourself the maintainer
Responsability is shared, somewhat
A call for enthusiasts

The release cycle

Our release cycle is efficient and quite standard. Once a stable is out, we expect the release candidates (-rcN). The more we move ahead, the more the maintainers apply bug fixes only. This helps fixing most of the bugs in the new features and fixing the regressions. Once the code stabilize, the stable is tagged.

Oops, this failed

For the v7.0.0, I did not think I’ll have to tag little bugfix releases soon after. And for sure not more than once.

So, what happened?

It appeared users did not help to test out the late candidates. Years ago, we had offlineimap enthusiasts running the latest releases, including the intermediate candidates. Some even used to work on the next development branch. They were doing a fucking good job at either fixing or reporting issues.

Lately, it looks like we’ve “lost” most of them. At least, this “team” of testers stopped to track on new patches and latest release to try and see. As a result, the v7.0.0 was not stable enough while I’ve strictly applied the same release cycle policy.

Imagine yourself the maintainer

How you work is simple. Code changes as we move forward by applying patches. You (the maintainer) feel it’s time to stabilize the code. So, you release a candidate and let the time go. At the rhythm of feedbacks you fix some issues and postpone the stable until all are fixed.

Side note: as a maintainer, you’re a tester too. You’re fixing the issues you find.

But this time, you get very FEW to NO feedback at all.

This might mean too things:

The users did not find issues. That’s good. It means you should release the stable.
No users are testing your job. You’re out of luck. If you release a stable, you’ll likely release a crappy version.

This is what happened for this v7.0.0.

IOW, the implicit contract broke. We make our best to write code and test but we can’t do all the job. We just can’t try our own code in the wild and see if this work for all the possible options and servers out there.

We used to have enthusiasts. For whatever reasons, most are gone. I suspect some left because they felt not usefull anymore at some point in time.

Responsability is shared, somewhat

From my point of view, nobody can expect offlineimap to “magically” work without any issue for each and new release if few contributors help. We are a small community. We do rely on you.

The implicit contract is that if nobody test the code, a crappy stable will get out despite our good will. By installing and using it, you’re finally doing the job that was not done in late. No one is comfortable when this happen. Users tend to not trust much anymore. Maintainers work in a hurry to fix issues. Reputation decrease.

A call for enthusiasts

Join us!

We have a “testers” team in Github. By joining, your profile can be publicly linked to the OfflineIMAP organization (at your own preference).
I can add you in the copy list of the emailed announces when requested.
We try to take care to credit you when you report issues with your real name.
You’re wining close proximity to the hard core team. Your own issues become priority because we know we can trust you: you’re learning or you already know how to make proper bug reports. You’ll be one of the rare people we are willing to provide real private support to fix or enhance your environment.

That’s teamwork we enjoy.

You’re reading this blog post. You might be subscribed to the RSS feed or you’re subscribed to the mailing list in order to be aware of what’s happening. I know you can be more a bit more involved. ,-)

Drop me a note that you’re willing to test releases. The more I know about your environemnt and configuration, the more you’ll help. Let me know if I can request you on demand. Provide both negative and positive feedbacks, especially for late candidates.

Contrary to wrong belief, it’s not very harmful to your precious emails. There are ways to prevent from hard issues:

We make backups. E.g., I use a script that takes care to snapshot the (dedicated) filesystem storing both my mails and the local cache, prior to any sync.
When we try a new version, we first try on non-critical accounts.

As one of the hard core testers, I’ve never had to restore a backup because of breakage. The internal design of offlineimap is robust. I don’t pretend this will never happen. I observe this didn’t happen for now. Also, since I’m used to test, I’ve setup a reliable environment like above for all my accounts.

The side effect is that I’m prepared in case of troubles on my most precious accounts, too. Probably more than the lambda offlineimap user. Not because I’m a developer or a maintainer. Only because all the environment is up and ready in case of strong issue. ,-)

how to ignore UIDs

Mon, 18 Jul 2016 00:00:00 +0000

When offlineimap is on trouble, it’s now possible to ignore UIDs.

The copy_ignore_eval feature
Output sample

The copy_ignore_eval feature

If UIDs can’t be correctly synced for whatever reason, it might be usefull to ignore them. This allows to workaround issues or postpone a bug report.

This feature will be included in v7.0.0.

Here is the configuration file extract:

# This option stands in the [Repository RemoteExample] section.
#
# If offlineiamp is having troubles to download some UIDS, it's possible to get
# them ignored in a list. This only ignore the download.
#
# The function must return the list of UIDs (integers), None otherwise. It is
# passed the folder name (using the remote name with the remote separator,
# likely '/').
#
#copy_ignore_eval = lambda foldername: {'INBOX': [2, 3, 4]}.get(foldername)

The UID numbers to ignore are in a list which is mapped to the folder name.

Output sample

Folder INBOX/sp&AOk-cial [acc: Test]:
 IGNORED: Copy message UID 2428 RemoteTest:INBOX/sp&AOk-cial -> LocalTest

improving the policy about the imaplib2 library

Tue, 14 Jun 2016 00:00:00 +0000

Since John left the project, many users and contributors asked why imaplib2 (the IMAP library) is packaged within offlineimap.

Code duplication?
Updating imaplib2 only?
The new policy
Official support

In an ideal world, any Python dependency not in Python core is declared as dependency and that’s all. Pypi helps handling the dependencies.

In offlineimap, the main dependency is imaplib2. Since the early days, it is packaged with offlineimap and I usually hear that users would benefit from having imaplib2 packaged outside for mainly two reasons:

imaplib2 could be updated wihout updating offlineimap.
This would avoid code duplication.

While this makes sense at the first glance, I don’t think this would benefit most users.

Code duplication?

Well, if you actually install any other software depending on imaplib2 and that you care about 92Ko of disk space, then you might wonder about this code duplication.

In real life, few other projects are using imaplib2. Most distributions don’t even package imaplib2. In distributions packaging imaplib2, I’m not aware of any other package depending on this library.

Code duplication is not a real issue. The distributions having strong policies in this area are already doing the job of removing imaplib2 in our sources and packaging it outside.

If imaplib2 were removed from our sources, I wonder that package maintainers would not maintain it much wisely. They would not pay the attention it requires because it’s a small file and it’s not widely used. They would likely spend their time on other issues and more important packages.

Updating imaplib2 only?

Hmm, let’s get the full picture.

Offlineimap is built as a big wrapper on top of imaplib2. It is definetly very tied to imaplib2. Offlineimap has always been shipped with imaplib2. Actually, imaplib2 is a one file library to allow bundling it like we do. We don’t diverge from upstream and we closely work with upstream.

imaplib2 is moving slowly. At least, not as fast as offlineimap. Each time we see a new version of imaplib2, we first check that the changes are fine and if they are, we update it in our sources. When not sure, we integrate imaplib2 at correct times in the release cycle so it can be tested enough and validate the update. It already happened that imaplib2 was not updated or downgraded in our sources. Some releases have been deliberately ignored for good reasons. Most users (and even distribution maintainers) don’t know when there are issues in imaplib2. Usually, they just pick a recent enough version and use it.

Lot of changes in imaplib2 have been reported, initiated or even made by the offlineimap contributors. I think we are in a good position to know which is the best version of imaplib2 that offlineimap should work with. This policy is working well.

Removing imaplib2 would break historical assumptions that imaplib2 is bundled in offlineimap. This would not be an issue if all users knew how to handle Python libraries correctly. I don’t think this is the case. We could learn them how to do this but I’m pretty sure most just don’t care and don’t want to care about that.

Also, I know for sure that our users are used to download the tarball or simply git clone and run ./offlineimap.py directly. This is what we have everywhere in our docs and what we request to do while helping and digging into issues. As a maintainer, I like the idea of just downloading offlineimap and run it directly.

Since v7.0.0-rc2, this statement is not true anymore. With the support for Python 3, we made the choice to use the six dependency. We don’t package it. However, contrary to imaplib2, six is widely used and I expect that all the distributions are packaging this library.

Finally, relying on an external imaplib2 would mean we would have to handle different versions of imaplib2 in the code and the bug reports. This would be much more additional work for our small team.

The new policy

There’s no way in my mind we would benefice from removing the bundled imaplib2.

However, we have ways to make the life easier for both the distribution maintainers and users/developers wanting to play with other versions of imaplib2.

This is why “superseding” the bundled imaplib2 version is now allowed.

Since v7.0.0-rc2, offlineimap imports a virtual imaplib2. This virtual library is in charge of importing the real imaplib2. The policy is the following:

Try to load imaplib2 from the system (PYTHONPATH). If found, check that the version is allowed and recent enough and use it.
If not found or if the version is not allowed, load the bundled imaplib2.

Currently, the code allows any newer release than the bundled imaplib2 version. I expect we will stick to this policy.

Official support

While loading and running a newer version is allowed, this does not mean this release is officialy supported for offlineimap.

Actually, I decided we will only support offlineimap when it is using the exact same version as the bundled version of imaplib2.

This is not an offense. As said previously, offlineimap is a big wrapper of imaplib2 and there are assumptions in the code regarding imaplib2. This is historical heritage and it’s hard to get rid of this.

With this new policy, I think we are reaching enough flexibility for anyone wanting/needing tuning while still providing the best by default.

offlineimap learns Python 3 support

Sat, 04 Jun 2016 00:00:00 +0000

What to expect
What NOT to expect
Known issues

With the great work done by Łukasz Żarnowiecki, offlineimap starting from v7.0.0-rc2 can run with Python 3.

This feature is marked EXPERIMENTAL for now because it’s still very new.

You’re very welcome to test this feature and report bugs in our issues tracker and the mailing list.

Thank you Łukasz!

What to expect

With Python 3, some bugs gets fixed. The ones that come to my mind are:

What NOT to expect

Improved encodings support. It is true that Python 3 has much better support for Unicode than Python 2. However (you likely already know it), supporting encodings other than UTF-7 with offlineimap is hard, very hard. So, I don’t expect this feature to come while Python 2 is supported.

Known issues

If you have enabled the unsupported decodefoldernames configuration option or if your are using your own tricks to have other encodings than (IMAP) UTF-7 for your Maildirs, be aware that the FMD5 won’t match. This will cause issues while syncing.

Users following the expected usage of offlineimap are not concerned.

sqlite becomes the default

Thu, 19 May 2016 00:00:00 +0000

The change
Prevent from errors
Other considerations

The change

After years of development and improvements, the sqlite database becomes the default with v7.0.0-rc1. The database is used for offlineimap to track what was already synced in the cache.

Prevent from errors

This change means that downgrading from v7.* to any previous release might be prone to errors. While the migration from the (legacy) plain text database to sqlite is seemingly handled, it is not the case the other way around. In other words, this might lead to emails re-downloads and duplications.

Before downgrading from v7.* to any version 6.*, users must make sure to have the sqlite backend enabled in the configuration file.

Other considerations

While sticking to the plain text database is still possible as for now, users should expect this backend to become deprecated and removed in the future.
Once switched of backend from plain to sqlite, you may want to manually delete the old cache directory in <metadata directory>/Account-<account>/LocalStatus . The sqlite cache stands in the LocalStatus-sqlite folder.

Our GSoC 2016 with Gentoo

Sat, 23 Apr 2016 00:00:00 +0000

A sad story.

The GSoC
Why Gentoo?
Time for Google to select the orgs
The ranking
The slots
The accepted projects
After the facts
Our experience
Why this blog post?

The GSoC

Basically, the process is quite simple. Everybody follows the official timeline provided by Google.

Students submit proposals based on their own ideas or the ideas provided in the public ideas page of the organization. Proposals describe their expected contributions for the project during summer.
Once the deadline for submitting proposals is reached, the mentors in the organization rank the proposals and the organization admins request slots to Google.
Google allocates a number of slots for the projects to the organizations. Next, the admin allocates the slots to the projects for their org.

Why Gentoo?

That’s easy. They have been my first contact. Also, this was my first experience in the GSoC program and since I’ve had a warm welcome from the admins, I decided to rely on experienced people.

Relying on an organization applying to the GSoC for several years would avoid us some administrative tasks and allow us to learn how it’s done.

I was aware that there are other organizations. After all, if joining Gentoo was incongruous they would have warned me about that.

Time for Google to select the orgs

Later, I discovered that it’s not so obvious. For example, admins at Gentoo are having a hard time at motivating mentors to apply to the GSoC.

Having a good ideas page is definetly relevant for the org to be accepted by Google.

I’ve been surprised by the number of mentors/projects waiting for the organization to be accepted by Google before they decide to join. Not all came lately because of good reasons. There was 6 late projects for a total of 23. This is more than 1/4! The main org admin said most came lately by “pure laziness”. I tend to agree with him on that.

The ranking

Each mentor has been requested to rank all the proposals before April 3rd to the dashboard provided by Google:

0: Invalid proposal.
1: Bad, I absolutely do not want this student with us this year.
2: Not good enough, we should not accept this student.
3: OK
4: Very good, we should accept this student.
5: Excellent, we absolutely have to have this student this year.

The deadline for requesting slots to Google was April 11th. During this reviewing period, we are requested to communicate with the students and the other mentors.

Ranking is not so easy. This is mainly about the proposal but there are other criteria to consider like the student interactions, the feedbacks from the other mentors, the effective contributions, etc.

This is a real job and I think all the mentors did their best to rank the proposals.

So far so good. We got the final results before the deadline requested by Google. Here are the notations for the 7 best proposals, ranked in order:

Notation		Mentors
4.14	imapfw	2 mentors	(Abdó and me)
4.125	gentoo	1 mentor
4	imapfw	2 mentors	(Abdó and me)
3.89	gentoo	1 mentor
3.59	gentoo	4 mentors
3.56	gentoo	2 mentors
3.5	gentoo	1 mentor

This table is mine. We were asked to rank on the dashboard so it’s easy to calculate the averages.

Notice that since we are new to the GSoC, the org admin decided to allocate 1 slot at most to imapfw. This was clear since the beginning. Later, it appeared that all the proposals for imapfw were too similar to get more than one student. Similar proposals are hard to handle because student contributions would clash. This would not fit well for a GSoC.

imapfw had 4 proposals for a total of 17 (actually 20 but I’m excluding the most obvious spams). This is not bad at all compared to the other projects of the org having at most 2 proposals. Also, our applicants were all aware of the other applicants before applying. I’m happy imapfw has good traction!

The slots

On April 13th, slots allocation was announced. Gentoo had 5 slots. At the next step, the org admin has to allocate the slots for the accepted projects on the official dashboard. The deadline was April 20th.

I consider this job was far from easy. Especially to select the 4th and 5th students.

Because this was not so obvious for those, the org admin decided to poll the mentors.

Extract of the poll request:

I’m having second thoughts about picking among the 6 projects we have for our 5 slots. So, I would like to ask each of you to send me an email with your list of 5 preferred projects from the following students: […]. No need to rank them, just the 5 you prefer.

Notice that there are 6 projects considered by the org admin for this poll, one of the imapfw project was already excluded while highly ranked. Also, I’m not sure this poll made sense to decide at the bottom of the scale because listing the 5 “preferred“ projects is obviously not objective…

In the selection criteria, the org admin emphasized more than once that “planing to be involved with gentoo in the future” was a contributing factor in the decision. We didn’t became aware of it until we got irreversibly involved. However, I thought that it was clear for everybody that imapfw students would be interested in becoming imapfw contributors and not much for Gentoo. During the selection process, I was trusting the org admin to get this in mind and balance the results.

Now that they know they are not accepted to the GSoC, two of our students expressed they intend to contribute to imapfw once they are done with their exams! We hope to see you later!

The accepted projects

On April 20th, one day before the deadline, the results of the poll were announced: the imapfw project in the course was the last for the 6 projects considered in this poll.

While I was a bit worried, this was not much an issue for me. It was not clear he would take those result as the only relevant criteria to select the projects. I was trusting the org admin to put those late and surprising results in perspective. And the results we got are quite surprising. Even the org admin said he was surprised.

I was wrong to trust the org admin. He decided to accept the projects by only considering those late results. Hence, none of the imapfw projects got accepted.

This was obviously unfair:

1 imapfw project of the top 3 was ignored in this poll.
The poll was requesting a matter of preference which is a quite subjective notion.
There were grades of technical review already available to decide.

No constructive exchanges for allocating the slots. No team work. No open process. Only a late opaque poll, discarding the most objective ranking and all of the work done before. Erf…

After the facts

As mentors for imapfw we were not expecting that students interested in imapfw had to be interested into becoming a Gentoo contributor, too. That handicaped us.

Here is an extract of the last response I got from one of the org admin after the deadline:

And last, but not least, if you want to run GSoC your own way, apply as an organization yourself (google accepts plenty of small projects), or at least choose an organization that fits better with your project goals (PSF maybe?). You can’t blame gentoo mentors for not selecting a project that has nothing to do with gentoo, instead of another related project with qualified student/proposal.

So,

This late poll was uncalled for.
The results of this poll could be put in perspective to get the full picture but this was not done.
The accepted projects are explicitly neither the best quality according to the internal evluation of the projects nor the pairs of mentors/students with the best chances to succeed.

I don’t blame the mentors. I think anyone of us realized what was going to happen while responding this late poll.

Thanks to this org admin for the honest response.

Our experience

First, I’d like to thank all of our applicants for this GSoC. You did a good job and we had real fun time with you. Please, accept the sincere apologies from Abdó and I. We both would certainly not have make this GSoC with Gentoo if we could imagine how it turned out to be. We are sad this GSoC experience has been biased like that.

I’m not crying because we didn’t get a slot. I regret we didn’t get a slot while we had 2 of the 3 best projects. As mentors, we obeyed all the requests from the org admins in due time. We definetely applied all of them. Never heard complaints either. I regret the mentors were finally not considered to evaluate the success of the projects.

Actually, I’m mainly blaming myself. I guess that what happened is part of the game, at least at Gentoo.

Perhaps I should have asked what are the exact criteria to select the projects. I never asked if we could be handicaped by joining Gentoo as a sub-org, either. I probably should have started discussions when I’ve had doubts on the selection process. Instead, I was trusting the org admins to draw the full picture by themselves and encourage discussions to decide what projects to accept. I was wrong.

My apologies to the students.

Why this blog post?

I’m having a hard time to explain our students what they did wrong and why they are not accepted. I’d rather be honest about what happened and why they are disqualified. It’s not easy task to do right in my position.

Also, I want the org admins to get a chance to catch what I’m saying to the students so they can provide their own feedbacks. It’s well possible we did things wrong as mentors while we didn’t got hints about that. Maybe this feedback can help them to improve their internal process while accepting sub-orgs, too.

I believe that sharing our experience might be beneficial to other umbrella and sub-orgs in the future. This experience was not a pleasure.

Finally, most of us didn’t started contributing in open source via the GSoC program. The main reason for me to be involved was to help students being involved. I’ll continue outside of the GSoC.

Thanks a lot to Abdó who spent quite some time for this GSoC!

Why Github pull requests suck so hard

Tue, 08 Mar 2016 00:00:00 +0000

One thing that really irritates me at Github is the pull request feature. Almost all about this feature is done wrong.

Signing the patches
Managing pull requests
Notifications
Tracking changes
Inter-diffs
History deletions
Addressing PR to reviewers
Granting reviewers and testers
PR by email
Continous Integration
Encouraging good practices
Conlusion

Signing the patches

Github offers no way to add your Signed-off-by line to the patches you merge from the webapp. “Signing off” the patches at merge time is relevant information and it means something. This is a rather common practice in communities.

Not being able to signing off the patches on a merge is a pity.

Managing pull requests

When a contributor sends a pull request it is stacked to the list of PR.

Tracking changes in this list is a pain. New pull requests since your previous visit aren’t highlighted. The list of pull request is lacking lot of features like personal prioritization, bookmarking, etc.

Expect to consume a lot of time only for managing your pull requests at Github.

@Github: shared labels can’t address everything.

Notifications

The most important task before merging any change is the peer review. Skipping this process is very very bad. Hence, proper notifications are critical. However, the way Github has PR implemented is the best way for skipping it.

Email notifications are randomly served. Sometimes you’ll get notifications, sometimes not.

Online notifications are a bit better while the notifications page happily mix everything.

The best workaround would be to associate an issue to each PR. Not very glorious when you know it doesn’t address all the points above and that it can’t be created automatically by enabling a configuration setting.

Github notifications for PR are a joke.

@Github, this issue is know for YEARS.

Tracking changes

Contributors learn this soon. Getting changes merged upstream might require far more than one shot. Hence, the first simplest feature one needs is to go back and forth accross the different versions of the submissions. Comparing revisions of the submissions is required to make a proper review.

Forget about that. Github is in no help. It lacks the notion of “topics”. What a pain, again.

@Github: implementing topics (almost tags, BTW) for PR is so hard to implement?

Inter-diffs

On top of this, the reviewing process is made much more easy when inter-diffs are available. When working on a new submission, you usually only have the previous submission in mind. This is because that’s where the most relevant information stand. Also, it’s a brain saver to work incrementally…

Here again, Github won’t help you. All provided diffs are against the base of the changes. You have no way to get inter-diff generated for you easily. While critical, it is missing.

@Github: of course, when the basic pre-requisites to implement this feature are missing (the notion of topics), no one will expect inter-diffs to become reality one day or another.

History deletions

This one is big. You can actually lose all your reviewing job. All comments are deleted and lost if the sender erase the PR with another one having the same branch name. 👏

I wonder if Github offers other ways to lose your job…

Big thanks to Github for this one! Losing work is always a pleasure.

Addressing PR to reviewers

When changing code, it’s welcome to address the changes you wrote to the original authors and the others contributors who made changes around yours. It’s easy to retrieve their name and emails with git blame or tig blame and cc’ them by mail. However, that’s unhelpful with Github.

The best workaround would be to @mention the other contributors but this would require you to know the nick names for all of them. This is not something you’re going to remember nor easily retrieve, really. Finally, the contributors won’t call for reviews.

Also, not everyone agree to use your service, Github. Thanks for considering them, too.

Hence, adressing a PR to the best reviewers is yet another true pain.

@Github: why don’t you provide a form when the user make a PR to send email notifications, including those not subscribed?

Granting reviewers and testers

When working in teams, it’s important to grant the contributors for their work. When it comes to reviewing, the standard way of doing so is by adding Reviewed-by, Acked-by and the likes to the Signed-off-by lines.

Github lacks a way to grant reviewers and testers.

PR by email

What about hooks? I know so many organizations and maintainers out-there would love to get the patches from pull requests automatically sent to their mailing list. This would FIX almost all of the above issues in ONE SHOT!

Github was requested for this feature FOR YEARS and never made the first step to support it. Thank you.

@Github: how long are you going to ignore the feedbacks of you users?

Continous Integration

One thing Github has usefull is integration of third services. For example, Travis CI greatly helps testing PR. What is missing is a way to append the commit message with the success/failure information.

Also, people tend to rely on the tests blindly. However, having the tests pass is not enough and won’t replace peer review.

Encouraging good practices

Contributors are willing to send pull requests with Github. I’d even say some users don’t understand anymore why I’m in favor of sending patches by emails. The service supports the feature… Why not use it?

Sending patches to the mailing list is still a best because Github support of PR is wrong by many ways.

To go a bit further, I’d even say the current implementation is a team killer. If most users don’t get the patches in their INBOX, they just won’t take the pain to provide reviews.

Yes, I know. Github displays the commands to fetch the patches on the PR page. Do you really expect all the reviewers to connect, checkout each PR one by one, execute those commands, checkout the changes locally, return online to the pull request page and find all the relevant excepts to add comments?

As a side effect, some long-term contributors get discouraged to provide feedbacks and stop to provide reviews when they don’t just leave the project. While the interactions between members are critical, Github’s way of handling pull requests will perversely turn down all the emulation of the community members quickly. Fun is lost. Motivation gets lost. A lot of contributors (not comfortable with the provided process for reviews) will stop contributing soon.

I am CERTAIN this is HOW we lost MOST of our OCCASIONAL but LONG-TERM contributors. Still, NOBODY but the maintainer does code review in our projects.

@Github: Thank you Github. Their feedbacks were very usefull and greatly appreciated. While peer review was the norm is our community, it is completely lost today.

I’m aware of the integration services. No one can revamp the peer review process correctly.

Conlusion

Working with PR and encouraging code reviews in teams is a nightmare with Github©.

Not using this feature at all is the best. You know why.

Finally, removing the support for pull requests is not available. THAT SUCKS HARD.

It’s easy to understand why Github don’t enable this feature. If you make your repository public and people want to contribute, they will likely use the integrated PR feature. Hence, you’re almost forced to use Github if you don’t want to discard contributions for your project. IOW, publicing your project on Github is far more than just open your work to the public: you have to connect to the webapp or use whatever client at some point in time. In pratice, rather often than occasionaly. The more a maintainer will use a platform, the more contributors feel the need of using it, too. The result is that this strongly increases the number of users for Github. This is the real reason why we can’t disable PR. You are used as a marketing promoter. Not by what you could say, but by what you actually DO.

@Github: is there no other way to convince new users to sign up to your website than almost forcing maintainers to accept PR in their public repositories?

One day or another, we will move away from github. This will be for good reasons.

A meaningful blog post on this topic: Github doesn’t support pull-request notifications to mailing lists, by Cole Robinson

Feel free to join the chat and provide your point of you!

GSoC 2016, we are accepting students to work on imapfw!

Mon, 29 Feb 2016 00:00:00 +0000

This year, we applied to the Google Summer of Code 2016 via the Gentoo organization, our umbrella organization.

Since Gentoo is accepted, students interested to work on imapfw should contact us!

Know more:

A tale of FMD5 (or upgrading offlineimap from 6.3.4 to 6.6.1)

Fri, 12 Feb 2016 00:00:00 +0000

While upgrading my Debian machine from jessie to stretch, offlineimap was upgraded from 6.3.4 to 6.6.1 and it had a unexpected effect in my case.

Since the time of this writing, OfflineIMAP v6.7.0 learned the CLI option --migrate-fmd5-using-nametrans. You should use this feature for your updates!

For versions from 6.3.5 up to 6.7.0, check out if your distribution maintainer has backported this feature for you: offlineimap -h.

Initial configuration and decision to upgrade

I saw the big warning in the Debian NEWS file about the “nametrans” change:

  Reverse nametrans
    +++++++++++++++++

    Starting with v6.4.0, OfflineImap supports the creation of folders
        on the remote repostory. This change means that people that
        only had a nametrans option on the remote repository
        (everyone) will need to have a nametrans setting on the local
        repository too that will reverse the name transformation.

    For further information, see documentation about this feature at:

      file:///usr/share/doc/offlineimap/html/nametrans.html#reverse-nametrans

But after reading carefully the linked documentation my configuration seemed to be Ok. To be clear my configuration includes the following snippets.

For Local repositories:

    folderfilter = lambda foldername: foldername.startswith("localprefix.")
    nametrans = lambda foldername: trans_local(foldername, "localprefix")

For Remote repositories:

    nametrans = lambda foldername: trans_remote(foldername, "localprefix")

which use those functions

    def trans_local(foldername, prefix):
        if foldername.startswith(prefix):
            return foldername.replace(prefix+".","",1)
        else:
            #raise AssertionError("config of nametrans with prefix '"+prefix+"' is not good")
            return "notfound_"+prefix+"_"+foldername

    def trans_remote(foldername, prefix):
           return prefix+"."+foldername

The unexpected effect

So far so good.

I upgrade, modify the completely unrelated ssl parameters, and just wait for my automatic mail process (I have offlineimap in a crontab) to continue business as usual.

And I wait, and I wait, and after some time I just wonder what the hell offlineimap is doing. And I found out that it was completely re-uploading all my mails. I have some accounts with more than 5G of mails and with the crappy connexion I had, it took forever.

I stopped offlineimap, deactivated all my crontabs and investigated the problem.

I think that the problem comes from the use of nametrans in visiblename for Maildir folder (commit Apply nametrans to all Foldertypes) because it leads to change the FMD5 part of the filename calculated by offlineimap.

Offlineimap thus believed that I removed all my mails and added it back. With no way for it to detect it was the same mails. It then reuploaded all the “new” mail before deleting all the “old” mail which is really inefficient.

My problem was exactly the inverse of the one in this blog post.

How I avoided to do some reupload

Once the problem was understood I came up with a workaround which seems to work. The principle is to rename all filenames with the new expected name (with the new calculated md5sum) by offlineimap in concerned Maildirs.

With my setup I used this dirty hack in shell:

    MYPREFIX=localprefix
    for box in $HOME/Maildir/${MYPREFIX}.* ; do
        cd $box
        oldmd5=$(pwd | sed 's!.*/!!' | tr -d '\n' | md5sum | sed 's/ *-$//')
        newmd5=$(pwd | sed "s!.*/!!;s!^${MYPREFIX}\.!!" | tr -d '\n' | md5sum | sed 's/ *-$//')
        for dir in cur new ; do
            rename -v "s/FMD5=${oldmd5}/FMD5=${newmd5}/" ${dir}/* ;
        done
    done

Note that this only works if done after the last use of offlineimap 6.3.4 and before the first use of offlineimap 6.6.1. If done at other moments I think we risk loss of mails and duplicates of mails.

I opened one Debian bug on this matter and wait for any idea to work up this upgrade cleanly.

No, I'm not using maxconnections

Fri, 29 Jan 2016 00:00:00 +0000

I’m not using the maxconnections feature in the configuration file. Here is why.

Why it hurts

In the good old days of OfflineIMAP, you were probably told to enable maxsyncaccounts and maxconnections to either synchronize multiple accounts or improve performance. I would not ask you to use this feature anymore.

This is something hard to say but it’s true that nowadays nobody has good/full understanding on multithreading in the code. Multithreading was introduced early but it appears that it badly supported all the later changes. IMAP connections internally requires locks that have to be correctly handled everywhere. While it’s commonly known to be a hard task by itself, things are currently worse because the same chunks of code can be used in different contexts (e.g.: with/without IDLE).

A workaround

Those configuration options were the only way to synchronize multiple accounts because of a global lock. OfflineIMAP used to allow only one instance of the program. For some years, things have changed: locking is done at the account level.

See this commit and this one for details.

Thanks to Sebastian Spaeth, running OfflineIMAP more than once is fine as long as it’s not with the same account(s).

That’s what greatly helps: running multiple instances is a good workaround of the messing code. This way, you only rely on code about locking at the account level which required only 23 lines of change when introduced. Much better than having to rely on locks spread all over the code about IMAP sessions, each handled by one Python thread.

Because OfflineIMAP proved to have some glitches with multithreading, my advice is you should make your best to avoid weird errors and tracebacks: simply avoid maxsyncaccounts and maxconnections if you can.

That’s why I’m much more confident with multiple instances running at the same time.

If you want to enable IDLE with more than one folder, you can’t apply. Each IDLE folder requires a dedicated IMAP connection.

Running more than one OfflineIMAP instance

In order to synchronize all my accounts in one command, I wrote a shell script with this kind of tricks:

offlineimap -a account_one & pid1=$!
offlineimap -a account_two & pid2=$!

wait $pid1
wait $pid2
echo "Last execution: $(date)"

Ok, I agree with you this full thing is not glorious. But it’s effective. And if you hit weird errors, you might like to be aware it’s possible to improve stability. ,-)

Does not have a message with UID 15541

Wed, 27 Jan 2016 00:00:00 +0000

A message refuse to be downloaded despite reported as a mail with an UID. Server or OfflineIMAP then say that the account does not have that UID, and fetch of the message fails.

Background
Installed OfflineIMAP and got trouble
Is this a problem of OfflineIMAP?
Manual imap session
Conclusion

Background

My private mail accounts are served by remote servers. One of them (one of two) is a remain from my studies at the university KTH in Stockholm Sweden. They kindly let you keep the mail address and also they are not tight-fisted on storage space.

I’m a Mutt user. Starting out with Mutt, I was using it’s built-in functionality to connect to the server and read my mail. Being curious on other mail user agents, (to make fast mail body searches), I started to understand that a maildir is a good thing to have, because a number of agents work with maildirs out of the box.

Through the documentation of Notmuch, OfflineIMAP was suggested as something descent for pulling down and synchronizing your mail in and out of a maildir. So let’s try with OfflineIMAP.

Installed OfflineIMAP and got trouble

Or, I got half way. I got an error traceback such as

Thread 'Copy message 15541 from INBOX' terminated with exception:
Traceback (most recent call last):
  File "/usr/lib/pymodules/python2.7/offlineimap/threadutil.py", line 140, in run
    Thread.run(self)
  File "/usr/lib/python2.7/threading.py", line 763, in run
    self.__target(*self.__args, **self.__kwargs)
  File "/usr/lib/pymodules/python2.7/offlineimap/folder/Base.py", line 250, in copymessageto
    message = self.getmessage(uid)
  File "/usr/lib/pymodules/python2.7/offlineimap/folder/IMAP.py", line 215, in getmessage
    raise OfflineImapError(reason, severity)
OfflineImapError: IMAP server 'remote_kth' does not have a message with UID '15541'

and from there, OfflineIMAP gave up and only some mails where pulled. It turned out my OfflineIMAP version was not really the shiny new thing (a vanilla apt-get install), so I installed version 6.6.1.

With that newer version, a similar traceback was given, but now OfflineIMAP persisted, so everything but some culprit mail was downloaded.

Is this a problem of OfflineIMAP?

Mailing list to the rescue. On my first post to the mailing list, Nicolas responded:

If the issue is not solve, I would ask to the Microsoft support or a forum. If OfflineIMAP wants to download this UID, it’s because the server pretend to have this UID. IOW, I’d say it really looks like a server issue.

That gave me comfort. I was easily convinced it was an issue with the server. It was like, I sort of wanted it to be an issue with the server. I mean, rather than it would be a problem with a tool made by developers who maintain a dialog with it’s users. On the other hand, in that case the problem could have been addressed properly of course, but anyway…

The mailing list dialog continued:

On Thu, Jan 21, 2016 at 08:26:15PM +0100, Tomas Nordin wrote:

I am in contact with the IT people there. They are not aware of UID:s. Do you have a suggestion on how I can guide them in helping me trouble-shoot this.

I mean, is there a suggestion of a way they can query the Imap server so as to see the UID being reported?

Manual IMAP sessions are well described in the net. ,-)

I’m pretty sure the issue comes from the exchange server. We had another report of this in github:

https://github.com/OfflineIMAP/offlineimap/issues/296

– Nicolas Sebrecht

Well thank you Nicolas, “Manual IMAP sessions are well described in the net”. Is that so? IMAP sessions huh? Well, OK.

That other reported issue by the way, seem to share my problem also in that it was with the same kind of server.

Manual imap session

So it seems I was up to the task to somehow debug the mail server myself. Because, it seemed to me I’m not going to be able to get the people over there to do it. Even if they know how to, I guess they have to say they cannot because my credentials are needed.

Well described in the net

I did not search and research that very much. This is interesting indeed, but I feel that I have other things to do than to spend days on learning much and plenty about IMAP servers and mail. Those are the links that I found and read from to get somewhere:

The first one was actually found by quack’ing the term “manual imap session”. The second one is from the people trying to maintain some interoperability in the digital world, by maintaining specifications on how stuff should work.

openssl

On my system there is a tool preinstalled I think, at least I don’t remember installing it explicitly, called openssl. From the top of the manual:

OpenSSL is a cryptography toolkit implementing the Secure Sockets Layer (SSL v2/v3) and Transport Layer Security (TLS v1) network protocols and related cryptography standards required by them.

The openssl program is a command line tool for using the various cryptography functions of OpenSSL’s crypto library from the shell.

It comes with a sub command s_client. The top of it’s manual:

The s_client command implements a generic SSL/TLS client which connects to a remote host using SSL/TLS. It is a very useful diagnostic tool for SSL servers.

openssl mail server session

I used the program script to log my sessions with openssl, they tend to be lengthy. One need to be aware that the script program logs each and every key-press. So when viewing the log in a text editor, it might look more or less weird depending on how the editor render non-visible things, and how many back-spaces and none-charecter key-strokes was done.

The connecting command of openssl was this:

openssl s_client -connect webmail.kth.se:993 -crlf

The -crlf option translates a line feed from the terminal into CR+LF. Totally required in my case, the server I am talking to require it, and I think it is required by the RFC too.

Now a lot of output is given, ending with:

* OK The Microsoft Exchange IMAP4 service is ready.

To talk to the server, one has to adhere to a communication protocol. It is required to make up a (unique?) command identifier for each command. I went with a001, a002, and so on. I don’t know if it is strictly required to iterate the identifier, but I did that for each command or request. The session started with a log in:

a001 login <user> <passwd>
a001 OK LOGIN completed.
a002 select INBOX
* 1233 EXISTS
* 0 RECENT
* FLAGS (\Seen \Answered \Flagged \Deleted \Draft $MDNSent)
* OK [PERMANENTFLAGS (\Seen \Answered \Flagged \Deleted \Draft $MDNSent)] Permanent flags
* OK [UNSEEN 1045] Is the first unseen message
* OK [UIDVALIDITY 14] UIDVALIDITY value
* OK [UIDNEXT 16124] The next unique identifier value
a002 OK [READ-WRITE] SELECT completed.

As a client I say a001 followed by some command. The server finally responds with something, prefixed with the client command identifier. Like a001 OK LOGIN completed. I can reveal already that the information given on the first unseen message, OK [UNSEEN 1045], is the culprit message.

The first step was to figure out if the culprit message exist, since it was not certain given the error traceback:

a004 fetch 1040:1050 (uid)
* 1040 FETCH (UID 15500)
* 1041 FETCH (UID 15507)
* 1042 FETCH (UID 15514)
* 1043 FETCH (UID 15516)
* 1044 FETCH (UID 15523)
* 1045 FETCH (UID 15541)
* 1046 FETCH (UID 15544)
* 1047 FETCH (UID 15545)
* 1048 FETCH (UID 15553)
* 1049 FETCH (UID 15561)
* 1050 FETCH (UID 15562)
a004 OK FETCH completed.
a005 logout
* BYE Microsoft Exchange Server 2013 IMAP4 server signing off.
a005 OK LOGOUT completed.
read:errno=0

When I do the fetch command, I give as an argument not the UID number but something I think is referred to as the index number or message number. This number is consecutive and is not the same for one message always. It will be different when the folder is rearranged by deleting messages.

The command fetch (commands are specified in the RFC spec as linked to above) take a set as an argument followed with what to fetch, (UID) in this case. Now it was clear that the message with UID 15541 is reported. Gossip on the mailing list:

On Sun, Jan 24, 2016 at 05:13:20PM +0100, Tomas Nordin wrote:

So now I am hacking around with openssl s_client. Here is some output of my session:

…

It seems to me the UID 15541 is reported. Does that prove something. What should I do now?

It seems there’s really a mail with UID 15541. Try fetching this one alone. OfflineIMAP is doing something like

tagx FETCH 15541 (‘BODY.PEEK[]’)

– Nicolas Sebrecht

I made some trials to fetch the mail “manually”. Another session looked like this:

a00 OK LOGIN completed.
a01 select INBOX
* 1234 EXISTS
* 0 RECENT
* FLAGS (\Seen \Answered \Flagged \Deleted \Draft $MDNSent)
* OK [PERMANENTFLAGS (\Seen \Answered \Flagged \Deleted \Draft $MDNSent)] Permanent flags
* OK [UNSEEN 1045] Is the first unseen message
* OK [UIDVALIDITY 14] UIDVALIDITY value
* OK [UIDNEXT 16130] The next unique identifier value
a01 OK [READ-WRITE] SELECT completed.
a02 fetch 1045 (uid rfc822.size flags body.peek[header.fields (date to cc from subject x-priority content-type)])
a02 OK FETCH completed.
a03 fetch 1045 (body)
* 1045 FETCH (BODY ("text" "plain" ("charset" "iso-8859-1") NIL NIL "quoted-printable" 1 1) FLAGS ())
a03 OK FETCH completed.
a04 fetch 1045 (body.peek[header.fields (from subject)])
* 1045 FETCH (BODY[HEADER.FIELDS (from subject)] {127}
From: Microsoft Exchange Server 2010
Subject: Retrieval using the IMAP4 protocol failed for the following message:
15541

There is a lot to learn in communicating with a mail server. For example, I thought that the command above a03 fetch 1045 (body), would spit out the body of the message, but not so. (My idea of the body of the message is the part containing the human readable message). He just say some details of the message like the charset and so on and then completed.

Then I tried to get some information out from the message like from: and subject: (a04). It seems the message was from the server. Concerning the subject my conclusion is that the query fails. If the subject would really be “Retrieval using the IMAP4 protocol failed for the following message: 15541”, this is a recursive mind-blower for me.

At debug-your-imap-server-with-telnet, a problem was described about some over-sized mail that crashed the mail box. The manual IMAP session was used to delete that one mail. This occurred very tempting to me.

Nicolas said on the mailing list that it would be very interesting to fetch that whole message in raw to see what is going on. Of course I agree, but with my limited knowledge about this, and the output of my sessions, did not give me an idea how to do that. On the contrary, it seemed not possible to me. It failed even to serve the subject.

Kill the culprit message

What to do when there is a problem with something? Kill it. And here goes the killing session:

a02 select INBOX
* 1234 EXISTS
* 0 RECENT
* FLAGS (\Seen \Answered \Flagged \Deleted \Draft $MDNSent)
* OK [PERMANENTFLAGS (\Seen \Answered \Flagged \Deleted \Draft $MDNSent)] Permanent flags
* OK [UNSEEN 1045] Is the first unseen message
* OK [UIDVALIDITY 14] UIDVALIDITY value
* OK [UIDNEXT 16133] The next unique identifier value
a02 OK [READ-WRITE] SELECT completed.
a03 fetch 1045 (uid)
* 1045 FETCH (UID 15541)
a03 OK FETCH completed.
a04 uid fetch 15541 (uid flags)
* 1045 FETCH (UID 15541 FLAGS ())
a04 OK FETCH completed.
a05 store 1045 +flags (\Deleted)
* 1045 FETCH (FLAGS (\Deleted))
a05 OK STORE completed.
a06 fetch 1045 (uid)
* 1045 FETCH (UID 15541)
a06 OK FETCH completed.
a07 expunge
* 1045 EXPUNGE
* 1233 EXISTS
a07 OK EXPUNGE completed.
a08 fetch 1040:1050 (uid)
* 1040 FETCH (UID 15500)
* 1041 FETCH (UID 15507)
* 1042 FETCH (UID 15514)
* 1043 FETCH (UID 15516)
* 1044 FETCH (UID 15523)
* 1045 FETCH (UID 15544)
* 1046 FETCH (UID 15545)
* 1047 FETCH (UID 15553)
* 1048 FETCH (UID 15561)
* 1049 FETCH (UID 15562)
* 1050 FETCH (UID 15585)
a08 OK FETCH completed.
a09 logout
* BYE Microsoft Exchange Server 2013 IMAP4 server signing off.
a09 OK LOGOUT completed.

The killing was done by first tagging the message for deletion, (a05 store 1045 +flags (\Deleted)). Then by expunging it: a07 expunge. This was the most scary part, expunge take no arguments, just expunge. It kills the messages flagged for deletion, similar to the behavior in popular user agents. The RFC said that this is how it works, so I trusted them.

Conclusion

Since the culprit message has been killed, there is no more tracebacks raised on me. That is a relief bigger than the disappointment by not understanding what was going on with that message.

I can have a theory on what this mail is, and maybe I would guess that it is a sort of auto-reply that happens when a mail cannot be sent. What do I know?

However, next time I am better armed to discuss with the IT department. I will have a manual IMAP session and give them the output. Maybe then they can extract that message and see what it is. We will see.

Happy OfflineIMAPing!

Update: since this post was published, the issue got further details about what’s going on.

users of Gmail, my mails might go to your SPAM folder

Mon, 25 Jan 2016 00:00:00 +0000

Becare Gmail users, my emails might go to your SPAM folder! This includes those sent to the mailing list.

It appears that Google is getting things wrong to the war against spam. While I never made spam and never got my address spoofed, I’ve had reports from users of Gmail that they are seing my mails going to the spam box.

I’m not going to detail the reasons here. I am well aware about what’s going on. That’s not the point of this post. As a mail user complying to the netiquette, I expect my mails to reach the INBOX of the recipients. Not having them marked as spam.

Don’t get me wrong, war against spam is hard. Still, false positive in spam filtering is wrong, bad and hurting for the users. This might even cause serious damages into your economy.

Google appears to have very strong policies. As a consequence, many respectable mail users trying to reach you at Gmail might be discarded from your eyes.

It looks like the big company decided to rely on automatic rules only. At least, Google makes no efforts to fix their spam filtering issues since it is well known for years. Their policy massively affects millions of users.

In my opinion Google Intelligence Agency makes no point into making mails right. They are too much obsessed to build and own the Algorithm That Will Change The World.

Also, it’s clearly not the first time that Google breaks things. Even the IMAP protocol is not correctly implemented/extended.

The sad news: in this case there’s not much I can do but let you know the best way I can.

So, if you use Gmail, you might need to add a filtering rule.

imapfw is now open to contributions

Fri, 04 Dec 2015 00:00:00 +0000

imapfw is reaching an interesting stage for the developers. Also, I made a screencast for starting with imapfw.

While imapfw won’t still synchronize your accounts, the framework is reaching an insteresting stage for developers.

To know more about imapfw usage and internals, screencasts are available in this channel.

Current state

Since the publication (2 months ago), the main internals and features have either been started or done: imapfw earned 8677 physical lines through 173 commits.

The features

enable the rascal, allows key frame redefinitions.
support both multiprocessing and threading via unified internal interfaces.
asynchronous and synchronous communication between workers.
started the syncing engines, repositories, drivers and controllers.

Actions

Actions are real softwares packaged within the framework, actually.

unitTests.
syncAccounts.
shell: run code with predefined contexts, optionally in interactive mode.

Internals

wrapper on top of the IMAP library to work with high-level objects.
supporting python 3 hints/annotations is partially done. Will be expanded everywhere with code stabilization.
architect modules to set up the running code with workers.

What next?

I expect to write code more slowly. This should benefit both myself to get more free time and contributors to not be disturbed by code changing too fast.

I’m planning to start writing a offlineimap action to allow imapfw to become an alternative to the current offlineimap software in a legacy compatible way.

This suppose supporting both the configuration file and the cache. Full compatibility is not a goal. For example, using the sqlite cache format will be a pre-requisite.

A website, like the one used for OfflineIMAP (from where you’re reading this page). :-)

What you can do

Developers

There are things that are more time consuming than hard to achieve, actually. Involve yourself would greatly help to get the first stable release out. E.g.:

Complete the internal APIs.
Write drivers, controllers, engines.
Write tests.

For more details, the TODO list is online.

Users

Try features out.
Run the tests from time to time.

The sceencasts

First, screencasts won’t ever replace documentation. However, there are some cases where they can be usefull like introducing features or concepts. That’s why I’ve started to make some. I intend to make more with time and code advance.

Dailymotion vs Youtube

There might be other media out there but I’ve only looked at the most well-known (to me, at least). I’ve started by creating an acount for Youtube and tried to publish my first video on it. Up to my surprise, the video has been refused.

In order to post videos longer than 15 minutes, I have to give my phone number to Google. The reason given is that it effectively avoids spams and abuses. It’s obviously well-known that robots and spammers are only uploading videos longer than 15 minutes…

I consider that my phone number is private data and I don’t want to offer it to Google, the NSA or whatever. They might already have it via other ways but that’s not the point. I’m strongly against such practices.

Hence, I’ve decided to use Dailymotion only. The latter doesn’t requires me to expose them anything other than a mail address while they have much less ressources than Google to make war against spammers.

Monetization

Be aware that I’ve activated monetization on some videos. I hate having to watch ads myself and I intend to limit such practice to some of the available contents only. I don’t want advertising to become rule.

imapfw is made public

Thu, 08 Oct 2015 00:00:00 +0000

I’m publishing imapfw, a new IMAP tool which I intend to replace OfflineIMAP. ,-)

Some months passed since my previous announce to make a deep refactoring of OfflineIMAP. While doing this job, I realized it was something very hard and it would be better to write a new software. That’s what I’m currently doing and it’s called imapfw.

This tool won’t do much for now. This is still in very early development stage.

Because it’s WORK IN PROGRESS and EXPERIMENTAL stuff, it won’t make anything good. Expect it to have strong failures.

Introduction

Here are the main goals I have in mind for imapfw:

use the last Python 3 technologies;
clean and beautifull code;
easily extensible;
fully (aggressively?) concurrent;
fit the needs to both novice and experts;
support a lot of use cases;
safe and with embedded unit tests;
fucking good documentation;
IDLE mode? I’m still not sure about this one.

Believe me, those goals altogether are very ambitious.

That’s why I decided to make imapfw a framework (hence the name imapfw) which I want both simple and powerfull.

Beginners shouldn’t be scared, though. All the framework architecture only requires ONE file which is called the rascal. It won’t require the user to write one line of Python code.

A call for feedbacks

I’m publishing this tool today in order to get your feedbacks. Any constructive comment is always welcome but keep it in mind this is WIP, please!

Also, feel free to give your opinion about anything.

Users of OfflineIMAP

Since I intend imapfw to replace OfflineIMAP in the long term, you might want to follow the development and give your feedbacks. My best guess is that you will mostly be interested by the rascal: how it looks, is it hard, etc.

Developers

This call for feedbacks is mostly for you! Please, do fork the project and read a bit of code. Early feedbacks about code are very important.

I don’t expect patches, for now. However, let me know if you’d like to contribute soon. I’ll share more about what I have in mind.

How it looks

This section is partially obsolete.

A good start is to fork the project and look at the CLI options:

$ ./imapfw.py -h

The main argument is the ACTION. It tells what imapfw must do. I’ve started the implementation of the most expected feature: syncAccounts. For now, it won’t sync anything and you can run it safely.

The rascal

Here’s an extract of the rascal:

__VERSION__ = 0.1


##########
# GLOBAL #
##########

#
# The main configuration options are set in this dict.
#
MainConf = {
    'concurrency_backend': 'multiprocessing',
    'max_sync_accounts': 2,
}


from imapfw.api import actions, engines, controllers, types, drivers
# Support for contributor's backends. ,-)
#from imapfw.api import contrib


################
# REPOSITORIES #
################

ImapExampleConf = {
    'dns':      'imap.gmail.com',
    'port':     '143',
    'username': 'myname',
    'max_connections': 2,
}

MaildirExampleConf = {
    'path': '~/Maildir',
    'max_connections': 2,
}

class MaildirRepositoryExample(types.Maildir):

    conf = MaildirExampleConf
    driver = drivers.Maildir # Default: drivers.Maildir.


class ImapRepositoryExample(types.Imap):

    conf = ImapExampleConf
    driver = ImapDriverExample # Default: drivers.Imap.

    def _getPassword(self):
        return self.conf.password

    def credentials(self):
        return self.conf.username, self._getPassword()

    def search(self, server, uid_list, conditions):
        title = mail.getTitle()

        if title.startswith('spam'):
            mail.deleteRemote('deleting spam')

        if title.startswith('offlineimap'):
            mail.setTitle("filtered: %s"% title)
            mail.moveTo('INBOX.offlineimap') # IMAP MOVE.

        return # Continue proceeding to the folder.

    def getFolders(self):
        return ['sample', 'of', 'static', 'remote', 'folder', 'list']



############
# ACCOUNTS #
############

class AccountExample(types.Account):
    left = MaildirRepositoryExample
    right = ImapRepositoryExample

    # Optional. The folders considered for a sync.
    # Arguments:
    # - folder_list: list of folder names to sync.
    # Returned values: the new list of folders to sync.
    def syncFolders(self, foldersList):
        return ['folderA', 'folderB', 'folderC']

The first thing to notice is that configuration options are dictionnaries. Forget the INI style. I aim simple users to not have to write ONE line of Python code.
OfflineIMAP users should feel comfortable reading this file. It follows almost the same semantic and the same logic around accounts.
Most experienced users will notice the classes. Here is the hierarchy and how they are logically linked:
- an account is defined (derivates from types.Account) with both left and right “repositories”.
- a “repository” (I’m about to fully remove this term) maps to a configuration dictionary and a driver to actually access it. It derivates from the associated type (types.Maildir, types.Imap, etc) and the drivers are similar (drivers.Maildir, drivers.Imap, etc).

On top of this, each type acts as a controller for the underlying driver. I expect experimented users to make most of their “crappy” things^W^W tuning here. ,-) Think about filtering folders, nametrans, etc.

How everything is linked is important: it’s possible to decide what type uses which driver for what account.

Now, look at the imports:

from imapfw.api import actions, engines, types, drivers

actions are the actions available from the command line.
engines allows to define the default engine to use for the action (if this action supports more than one engine). It can be redefined at the CLI!
types are the little sisters of the OfflineIMAP’s repositories: Maildir, Imap, Gmail, etc, in the sense they hold a driver and some configuration options (like the credentials) to access it.
drivers are the low-level drivers to use for the types (didn’t exist in OfflineIMAP).

The imports are what makes the framework so powerfull. They allow users to redefine the default behaviour, or even write their own new full backends.

Because actions, engines, types and drivers are well orthogonal and distinct concepts, extending imapfw should be much easier than it was with OfflineIMAP.

A brief introduction about imapfw’s internals

Internally, there are other implementation concepts that some of you might like to be aware of. The most important are the concurrency, the workers and the managers.

The concurrency is a module to use a single and very simple API whatever the Python backend in use (multiprocessing or threading). Yes, I’m aware they are already very similar but they still have some subtil differences it’s better to avoid introducing in the long run. We force using a subset of them.

Because the conccurency backend is easily switchable, worker is a simple term to refer to either a process (multiprocessing) or a thread (threading).

The managers are objects to nicely handle concurrency. It takes inspiration from the managers of the multiprocessing library. They allow to have an intuitive usage for the “passing by message” design.

Final note !

It’s time for me to write the last words. I’d rather let you to discover imapfw more in details by yourself.

This post is long enough for today! Also, I’m getting very tired (I ddin’t sleep this night to polish the code, publissh it and write this bloog post). I hope nobody have noticcce…. :-D