[Developers] metaweb.py- when readservice returns a gzip'd response

Tue Jul 29 00:47:32 UTC 2008

Hi Ben and David,

Yes, I think the issue is on my end.  I loaded metaweb.py up on a different machine with clean python install and
the response there is not gzipped.     

I must have a config setting some where that is forcing the Accept-Encoding: gzip, compress
headers on all requests issued from urllib2.   Time to cleanup my lib\site-packages dir.

thanks for your help,
Bob

unencoded f.info().headers from the clean python install
['Date: Mon, 28 Jul 2008 23:26:07 GMT\r\n', 'Server: Apache\r\n', 'X-metaweb-success: 1/1\r\n', 'Content-length: 1209\r\
n', 'X-metaweb-cost: dt=0.004, cc=0.008, utime=0.008, stime=0.0, maxrss=0, ixrss=0, idrss=0, isrss=0, minflt=0, majflt=0
, nswap=0, inblock=0, oublock=0, msgsnd=0, msgrcv=0, nsignals=0, nvcsw=1, nivcsw=0, nreqs=0, cr=1, cm=0, ch=1, rt=0.0330
469608307\r\n', 'Last-modified: Mon, 28-Jul-2008 23:19:12 GMT\r\n', 'Etag: 9866c73d4d1090baa8d14910c9708d0d\r\n', 'Expir
es: Mon, 28-Jul-2008 23:19:13 GMT\r\n', 'Cache-control: public, max-age=1, s-maxage=1, maxage-vary-cookie="3600|mwLastWr
iteTime"\r\n', 'Vary: Accept-Encoding\r\n', 'Content-type: text/plain; charset="utf-8"\r\n', 'Age: 2242\r\n', 'X-cache:
HIT from cache01.p01.sjc1.metaweb.com\r\n', 'X-metaweb-tid: cache;cache01.p01.sjc1:8101;2008-07-28T23:56:34Z;0001\r\n',
'Via: 1.1 S1PS\r\n', 'Connection: close\r\n']

 response headers from the "dirty" python with the gzipped return
['Date: Mon, 28 Jul 2008 23:19:12 GMT\r\n', 'Server: Apache\r\n', 'X-metaweb-success: 1/1\r\n', 'X-metaweb-cost: dt=0.02
6, cc=0.02, utime=0.02, stime=0.0, maxrss=0, ixrss=0, idrss=0, isrss=0, minflt=0, majflt=0, nswap=0, inblock=0, oublock=
0, msgsnd=0, msgrcv=0, nsignals=0, nvcsw=7, nivcsw=0, nreqs=2, tr=0.007, tu=0.004, ts=0.0, pr=0, pf=0, dw=0, dr=1675, in
=528, ir=0, iw=0, va=3274, te=0.007, tg=0.00698900222778, tf=0.00916886329651, tm=0.0206868648529, cr=1, cm=1, ch=0, gqr
=0, rt=2.36789035797\r\n', 'Last-modified: Mon, 28-Jul-2008 23:19:12 GMT\r\n', 'Etag: 9866c73d4d1090baa8d14910c9708d0d\r
\n', 'Expires: Mon, 28-Jul-2008 23:19:13 GMT\r\n', 'Cache-control: public, max-age=1, s-maxage=1, maxage-vary-cookie="36
00|mwLastWriteTime"\r\n', 'Vary: Accept-Encoding\r\n', 'Content-encoding: gzip\r\n', 'Content-length: 488\r\n', 'Content
-type: text/plain; charset="utf-8"\r\n', 'Age: 1630\r\n', 'X-cache: HIT from cache01.p01.sjc1.metaweb.com\r\n', 'X-metaw
eb-tid: cache;cache01.p01.sjc1:8101;2008-07-28T23:46:22Z;0001\r\n', 'Via: 1.1 S1PS\r\n', 'Connection: close\r\n']

----- Original Message ----
From: Ben Drees <ben at metaweb.com>
To: "For discussions about MQL, Freebase API and apps built on Freebase" <developers at freebase.com>
Sent: Monday, July 28, 2008 3:04:52 PM
Subject: Re: [Developers] metaweb.py- when readservice returns a gzip'd response

Hi Bob,

It looks like urllib2 includes "Accept-Encoding: identity" in requests 
by default under Python 2.4 and 2.5.  A random sample of the responses 
generated by Freebase for these requests indicates that they are 
appropriately encoded and marked with "Vary: Accept-Encoding".  There 
might be a problem on the Freebase side for the specific URLs you are 
using, however.

If you could provide either 1) the transaction IDs of some of the 
problematic responses (from the "X-Metaweb-TID" response headers), or 2) 
some other distinguishing details of the transactions, such as the exact 
query text, the time of day and username under which they were executed, 
etc., we might have an easier time tracking this down.

Thanks,
Ben

David Flanagan wrote:
> Bob,
>
> I'm the author of metaweb.py, but I'm stumped by this problem.  I've 
> never seen it in my own tests of the module.  Perhaps one of the Metaweb 
> engineers can tell us when and under what circumstances Metaweb 
> compresses its response. (I would have liked to believe that urllib2 
> would handle content-encoding for us transparently, but I guess it doesn't.)
>
> Are you using urllib2 elsewhere in your application?  Could you somehow 
> have set something to tell the metaweb services that you'd like gzipped 
> repsonses?  (I'm not much of a pythonista, and I don't know anything 
> about HTTP and compression, so I'm just guessing here...)
>
> Are you using a complete standard Python 2.4 with its standard urllib2?
>
>     David Flanagan
>
> I have a new (and hopefully improved) version of metaweb.py developed 
> for a new version of the Metaweb documentation.  Its completely 
> incompatible with the old version, and only the read methods work right 
> now.  But let me know if you'd like to try it out!
>
>
> Bob Skinner wrote:
>  
>> Hi,
>>
>> I'm just getting started with the python interface using the metaweb.py 
>> example from Appendix B. (Using python 2.4)
>>
>> Using the .login or .read methods right "out of the box" always returned a
>> "ValueError: No JSON object could be decoded" error in the 
>> simplejson.load call.
>>
>> The reason was the response was gzip encoded, I had to change the read 
>> method from:
>>
>>     f = urllib2.urlopen(req)        # Open the URL
>>     response = simplejson.load(f)
>>
>> to:
>>
>>     f = urllib2.urlopen(req)        # Open the URL
>>     if 'Content-Encoding' in f.info() and f.info()['Content-Encoding'] 
>> == 'gzip':
>>         buf = cStringIO.StringIO(f.read())
>>         gbuf = gzip.GzipFile(mode='rb', fileobj=buf)
>>         response = simplejson.load(gbuf)   # Parse JSON response to an 
>> object
>>     else:
>>         response = simplejson.load(f)
>>
>>
>> I couldn't find anything on the documentation or in the developer 
>> mailing list archives to this end.    Is there some better way of 
>> handling this that I should be using?
>>
>> best regards,
>> Bob Skinner
>>
>> p.s.   I think you guys are on to something HUGE with Freebase.  Your 
>> implementation/api looks stellar.  I'm a big fan.
>>
>>
>> ------------------------------------------------------------------------
>>
>> _______________________________________________
>> Developers mailing list
>> Developers at freebase.com
>> http://lists.freebase.com/mailman/listinfo/developers
>>    
>
> _______________________________________________
> Developers mailing list
> Developers at freebase.com
> http://lists.freebase.com/mailman/listinfo/developers
>
>  

_______________________________________________
Developers mailing list
Developers at freebase.com
http://lists.freebase.com/mailman/listinfo/developers
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://lists.freebase.com/pipermail/developers/attachments/20080728/aa74ba9c/attachment-0001.htm 