[Developers] metaweb.py- when readservice returns a gzip'd response

[Ben Drees]

Hi Bob,

It looks like urllib2 includes "Accept-Encoding: identity" in requests 
by default under Python 2.4 and 2.5.  A random sample of the responses 
generated by Freebase for these requests indicates that they are 
appropriately encoded and marked with "Vary: Accept-Encoding".  There 
might be a problem on the Freebase side for the specific URLs you are 
using, however.

If you could provide either 1) the transaction IDs of some of the 
problematic responses (from the "X-Metaweb-TID" response headers), or 2) 
some other distinguishing details of the transactions, such as the exact 
query text, the time of day and username under which they were executed, 
etc., we might have an easier time tracking this down.

Thanks,
Ben

David Flanagan wrote:
> Bob,
>
> I'm the author of metaweb.py, but I'm stumped by this problem.  I've 
> never seen it in my own tests of the module.  Perhaps one of the Metaweb 
> engineers can tell us when and under what circumstances Metaweb 
> compresses its response. (I would have liked to believe that urllib2 
> would handle content-encoding for us transparently, but I guess it doesn't.)
>
> Are you using urllib2 elsewhere in your application?  Could you somehow 
> have set something to tell the metaweb services that you'd like gzipped 
> repsonses?  (I'm not much of a pythonista, and I don't know anything 
> about HTTP and compression, so I'm just guessing here...)
>
> Are you using a complete standard Python 2.4 with its standard urllib2?
>
> 	David Flanagan
>
> I have a new (and hopefully improved) version of metaweb.py developed 
> for a new version of the Metaweb documentation.  Its completely 
> incompatible with the old version, and only the read methods work right 
> now.  But let me know if you'd like to try it out!
>
>
> Bob Skinner wrote:
>   
>> Hi,
>>
>> I'm just getting started with the python interface using the metaweb.py 
>> example from Appendix B. (Using python 2.4)
>>
>> Using the .login or .read methods right "out of the box" always returned a
>> "ValueError: No JSON object could be decoded" error in the 
>> simplejson.load call.
>>
>> The reason was the response was gzip encoded, I had to change the read 
>> method from:
>>
>>     f = urllib2.urlopen(req)        # Open the URL
>>     response = simplejson.load(f)
>>
>> to:
>>
>>     f = urllib2.urlopen(req)        # Open the URL
>>     if 'Content-Encoding' in f.info() and f.info()['Content-Encoding'] 
>> == 'gzip':
>>         buf = cStringIO.StringIO(f.read())
>>         gbuf = gzip.GzipFile(mode='rb', fileobj=buf)
>>         response = simplejson.load(gbuf)   # Parse JSON response to an 
>> object
>>     else:
>>         response = simplejson.load(f)
>>
>>
>> I couldn't find anything on the documentation or in the developer 
>> mailing list archives to this end.    Is there some better way of 
>> handling this that I should be using?
>>
>> best regards,
>> Bob Skinner
>>
>> p.s.   I think you guys are on to something HUGE with Freebase.  Your 
>> implementation/api looks stellar.  I'm a big fan.
>>
>>
>> ------------------------------------------------------------------------
>>
>> _______________________________________________
>> Developers mailing list
>> Developers at freebase.com
>> http://lists.freebase.com/mailman/listinfo/developers
>>     
>
> _______________________________________________
> Developers mailing list
> Developers at freebase.com
> http://lists.freebase.com/mailman/listinfo/developers
>
>