Monthly Archives: 2011-3

rsync: remote copy incrementally and on-line compressed

Posted on by
4

rsync is a UNIX command to synchronize files and folders between two computers. It is very useful to download files to and from the server. The key difference to more commonly used scp (SSH copy) is that rsync can do incremental copies: it only copies the modified files. This is especially useful if you sync a large source code tree between different computers. rsync should be in any developer’s toolbox who are working with UNIX based software and deployments.

rsync works over SSH by default (I think it was over telnet by default in long past, but nowadays it’s SSH based, someone please refresh my memory). rsync also supports SSH:s on-line compression, which greatly reduces bytes needed to transfer when copying textual content (source code, logs).

To copy a remote folder to your local computer:

rsync –compress-level=9 -av username@server:/home/user/yourlogfiles

Explanation

  • –compression-level: Use maximum on-line compression
  • -a: Archive. Keep file timestamps in intact and copy only modified files.
  • -v: Verbose. Show what files we are copying.

If the copy process is interrupted rsync will resume the copy from the last file it didn’t manage to copy. In the end rsync tells the copy speed and (incremental) speed-up stats:

sent 376 bytes  received 80696946 bytes  658753.65 bytes/sec
total size is 2604253766  speedup is 32.2

Note: incremental copy might be confused by continuously changing files (logs, databases).

See also a prior blog post about scp copy and compression level.

\"\" Subscribe to RSS feed Follow me on Twitter Follow me on Facebook Follow me Google+

Android, Webkit, XHR status code 0 and Expires headers

Posted on by
2

I have narrowed down a very nasty Android bug.

If I turn on Expires headers on our content delivery network Android AJAX stops working (may affect CSS well). Android XMLHttpRequest responds with a status code zero:

03-20 12:55:33.564: WARN/browser(238): Console: AJAX loading:http://cdn.mobilizejs.com/releases/trunk/templates/wordpress.html http://cdn.mobilizejs.com/releases/trunk/js/mobilize.wordpress.min.js?ver=3.1:1
03-20 12:55:33.663: WARN/browser(238): Console: done http://cdn.mobilizejs.com/releases/trunk/js/mobilize.wordpress.min.js?ver=3.1:1
03-20 12:55:33.743: WARN/browser(238): Console: Could not AJAX url:http://cdn.mobilizejs.com/releases/trunk/templates/wordpress.html got status:0 http://cdn.mobilizejs.com/releases/trunk/js/mobilize.wordpress.min.js?ver=3.1:1
03-20 12:55:33.853: WARN/browser(238): Console: Status text: http://cdn.mobilizejs.com/releases/trunk/js/mobilize.wordpress.min.js?ver=3.1:1
03-20 12:55:33.864: WARN/browser(238): Console: Payload: http://cdn.mobilizejs.com/releases/trunk/js/mobilize.wordpress.min.js?ver=3.1:
  1. It is a definitely some sort of error code (without information, not even internal Androidd info in LogCat).
  2. It only occures with Expires header

So I suspect something is wrong in Expires headers… Headers serving attempt #1:

Expires: 21 Mar 2011 11:37:52 GMT
Date: Sun, 20 Mar 2011 11:37:52 GMT
Cache-Control: no-cache

Note that Cache-control: no-cache is set automatically by App Engine which we use to host file. This works, though from App Engine logs it is obvious the browser is re-fetching all resources on every request.

Adding weekday to Expires header doesn’t seem to work. Google App Engine writes it.

If you add only Expires and no Cache-Control, Google App Engine defaults to Cache-Control: no-cache.

If you add only Cache-Control header, no Expires, it seems that the client caches the result. However, in this case, Android AJAX calls stop working.

These seem to trigger the bug in Android emulator (XHR status code 0 for AJAX call):

Content-type: text/html
Access-Control-Allow-Origin: *
Cache-Control: max-age=86400
Date: Sun, 20 Mar 2011 11:56:39 GMT
Server: Google Frontend

As a temporary workaround, I just make this particular problematic resource re-fetched on every HTTP request on Android phones by inserting a random token into URL:

var url = mobilize.toFullCDNURL(mobilize.cdnOptions.template);

 if(navigator.userAgent.toLowerCase().indexOf("android") >= 0) {
 mobilize.log("Avoiding Android cache control problem for AJAX");
 // Android AJAX + cache is busted
 // http://mobilizejs.com/2011/03/20/android-webkit-xhr-status-code-0-and-expires-headers/
 url += "?android-buster=" + Math.random();
 }

 console.log("Loading mobile template from URL:" + url);

Logs tell this seem to be effective and other resources besides this AJAX call are no longer loaded:

2011-03-20 05:22:36.966 /releases/trunk/templates/wordpress.html?android-buster=0.7062135764234672 200 181ms 298cpu_ms 251api_cpu_ms 0kb Mozilla/5.0 2011-03-20 05:21:09.978 /releases/trunk/templates/wordpress.html?android-buster=0.42276109455817257 200 77ms 298cpu_ms 251api_cpu_ms 0kb Mozilla/5.0
2011-03-20 05:20:42.687 /releases/trunk/templates/wordpress.html?android-buster=0.8590675375212578 200 85ms 298cpu_ms 251api_cpu_ms 0kb Mozilla/5.0

Note that this does not solve the problem How to cache AJAX responses on Android at all, it justs works around this particular case.

More info

 

Making an offline file system copy of a Plone site using WebDAV

Posted on by
2

You might want to create an offline copy of a Plone site because

  • You are traveling and you want to have all files on the site (PDFs for reading)
  • You are taking a site down and making the final back-up
  • You just want to feel how cool Plone is

Plone supports WebDAV.

Creating a file system viewable offline copy of a Plone site is a task of

  • Enabling WebDAV
  • Login to site via WebDAV. On OSX use Cyberduck, Finder (the file browser of OSX itself) may have issues, though works. You might need Zope admin priviledges for certain operations.
  • Drag and drop Plone site to your hard disk

WebDAV copy process works smoothly

  • Folder and page structure is intact
  • Files are copied as is (think PDFs, Docs)
  • Images are copied as is
  • Pages (HTML) are converted to special files, which are still readable in plain-text editor

Note that some special folders (acl_users, reference_catalog, etc.) might be exposed through WebDAV, but they are not really copyable. Just ignore these during the copy process.

You can also use WebDAV to mass upload files and images for your image bank instead of manually uploading them through web interface.

\"\" Subscribe to RSS feed Follow me on Twitter Follow me on Facebook Follow me Google+

RFC: Simple Internet Question Asking Protocol (for human beings)

Posted on by
2

This is my  attempt version 0.1 to teach the world how one should ask questions in the simplest possible way in Internet discussion. To make it simple, I try to keep this short. This post sprouts from my frustration from the lack of people’s ability to form questions one could easily answer.

1. Assumptions

If you want to ask a question in forum, IRC (chat) or mailing list

  1. Assume people are busy
  2. Assume that people want to help you, even though they are busy, since they volunteer to participate the community discussion and thus they must care about the community

To make it win-win situation, you as the question maker, are responsible of making the process of asking the question and answering the question as easy as possible. Form your question in such a way that it is as easy as possible for the readers to place themselves into your situation and think how they would themselves solve the situation (Mikko’s rule of empathy).

The less time it takes to undestand your situation the more likely people are willing to contribute their time.

2. Question process

Thus, I propose that you always follow the simple three steps when asking a question

  1. Before asking the question tell what you already know
  2. Describe the problem
  3. Ask what you do not know yet

Then wait patiently for the answer (the busy part).

3. Pitfalls

These issues often stem from the fact that the person asking the question is not familiar with text-based communication where people’s time (bandwidth) is limited and the lack of body gestures often leads to misinterpretations.

  1. Do not ask yes / no questions. You are skipping steps #1 and #3.
  2. Do not saturate the bandwidth: do not repeat yourself or otherwise flood the medium. If people are busy it it does not make them un-busy by repeating yourself. You are breaking the assumption #1.
  3. Do not try to pull excessive attention on you – do not try to highlight your question like “PLEASE HELP !!!!” Even if it is a matter of life and dead for you it is not for the other people who are dealing with their own matters of life and dead. You are breaking the assumption #2.

4. Example

Q: Is it possible to fly me to the Moon? A: Yes

Q: I am an evil super-villain whose plan overtake the world failed.  Now I must escape. I am looking for methods to take me to the Moon or the orbit where national laws to do not apply. I am not sure should I use a shuttle or a rocket. Where could I obtain such a vehicle?

A: US of A just retired one reliable space shuttle what you could use. But if I were you I’d consider underwater base instead, as they will become cheaper in long run, since you can more easily produce breathable oxygen.

5. More info

\"\" Subscribe to RSS feed Follow me on Twitter Follow me on Facebook Follow me Google+