[Developers] 503 error when trying to do a large write to Sandbox

[Jason Douglas]

I'm out of my depth here, but could your issue be related to this  
thread?

	Comparison operation in write?
	http://lists.freebase.com/pipermail/developers/2008-January/001198.html

-jason


On Feb 20, 2008, at 4:31 PM, Alexander Botero-Lowry wrote:

> Hi,
>
> Monday night I tried to import 2561 records as a single query on  
> Sandbox.
> After fixing some bugs in how I was formatting my date, I finally  
> got the
> query to execute but I received a 503. I tried the query a few more  
> times
> and then checked sandbox to make sure nothing had happened and low and
> behold some of the entries showed up! At that point I realized that I
> wasn't sure all of them were there so I rewrote my importer script to
> do it in 100 block increments, and I got an error that there were 2
> unique entries, which I looked up and determined was the result of
> create=unless_exists not being able to disambiguate! So it seems like
> somehow, even though i was using create=unless_exists the entry got
> added twice. I'm not exactly sure how transactions work internally
> so I can't really speculate further on how that happened.
>
> Follows is my importer:
>
> #!/usr/bin/env python
>
> import metaweb
>
> USERNAME=''
> PASSWORD=''
>
> MONTH_MAP = {'Jan':1, 'Feb':2, 'Mar':3, 'Apr':4, 'May':5, 'Jun':6,  
> 'Jul':7, 'Aug':8, 'Sep':9, 'Oct':10, 'Nov':11,
> 'Dec':12}
> TYPEID = '/user/alexbl/default_domain/exchange_rate'
> SOURCE_CURR = {'name':'US $', 'type':'/finance/currency'}
> TARGET_CURR = {'name':'Australian dollar', 'type':'/finance/currency'}
>
> def generate_query(a):
>        data = a.split()
>        # FIXME: find a better way to do this
>        rate_date = data[0].split('-')
>        rate_date[2] = '19'+rate_date[2]
>        rate_date[1] = "%02d" % (MONTH_MAP[rate_date[1]])
>        rate_date[0] = "%02d" % int(rate_date[0])
>        data[0] = '-'.join(reversed(rate_date))
>
>        q = {'create':'unless_exists',
>                'id':None,
>                'type':[TYPEID],
>                'source_of_exchange':SOURCE_CURR,
>                'target_of_exchange':TARGET_CURR,
>                'amount':float(data[1]),
>                'date_of_rate':data[0]
>        }
>        return q
>
> if __name__ == '__main__':
>        query = [ generate_query(a) for a in file('USD- 
> AUD-90-99.txt') ]
>        credentials = metaweb.login(USERNAME, PASSWORD)
>        for a in range(100, len(query), 100):
>                result = metaweb.write(query[a:a+100], credentials)
>                for r in result:
>                        print r['create'], r['id']
>
>
> Before I added the stepper, it was simply directly doing what's in  
> side the
> for loop with the query list.
>
> The data format is like:
> 2-Jan-90                                                        0.7855
> 3-Jan-90                                                        0.7818
> ...
>
> I will most likely write a script to query the ids and then detype  
> them and then
> do an import again with the 100 block steps version to see if that's  
> the only problem.
>
> Luckily this was all on sandbox :)
>
> Thanks,
> Alex
>
> _______________________________________________
> Developers mailing list
> Developers at freebase.com
> http://lists.freebase.com/mailman/listinfo/developers