[Developers] 503 error when trying to do a large write to Sandbox

[Alexander Botero-Lowry]

On Thu, Feb 21, 2008 at 11:02:56AM -0800, Jason Douglas wrote:
> I'm out of my depth here, but could your issue be related to this  
> thread?
> 
> 	Comparison operation in write?
> 	http://lists.freebase.com/pipermail/developers/2008-January/001198.html
> 
I don't think so. I was doing a [{...},{...},{...}] style query. 

Alex
> -jason
> 
> 
> On Feb 20, 2008, at 4:31 PM, Alexander Botero-Lowry wrote:
> 
> > Hi,
> >
> > Monday night I tried to import 2561 records as a single query on  
> > Sandbox.
> > After fixing some bugs in how I was formatting my date, I finally  
> > got the
> > query to execute but I received a 503. I tried the query a few more  
> > times
> > and then checked sandbox to make sure nothing had happened and low and
> > behold some of the entries showed up! At that point I realized that I
> > wasn't sure all of them were there so I rewrote my importer script to
> > do it in 100 block increments, and I got an error that there were 2
> > unique entries, which I looked up and determined was the result of
> > create=unless_exists not being able to disambiguate! So it seems like
> > somehow, even though i was using create=unless_exists the entry got
> > added twice. I'm not exactly sure how transactions work internally
> > so I can't really speculate further on how that happened.
> >
> > Follows is my importer:
> >
> > #!/usr/bin/env python
> >
> > import metaweb
> >
> > USERNAME=''
> > PASSWORD=''
> >
> > MONTH_MAP = {'Jan':1, 'Feb':2, 'Mar':3, 'Apr':4, 'May':5, 'Jun':6,  
> > 'Jul':7, 'Aug':8, 'Sep':9, 'Oct':10, 'Nov':11,
> > 'Dec':12}
> > TYPEID = '/user/alexbl/default_domain/exchange_rate'
> > SOURCE_CURR = {'name':'US $', 'type':'/finance/currency'}
> > TARGET_CURR = {'name':'Australian dollar', 'type':'/finance/currency'}
> >
> > def generate_query(a):
> >        data = a.split()
> >        # FIXME: find a better way to do this
> >        rate_date = data[0].split('-')
> >        rate_date[2] = '19'+rate_date[2]
> >        rate_date[1] = "%02d" % (MONTH_MAP[rate_date[1]])
> >        rate_date[0] = "%02d" % int(rate_date[0])
> >        data[0] = '-'.join(reversed(rate_date))
> >
> >        q = {'create':'unless_exists',
> >                'id':None,
> >                'type':[TYPEID],
> >                'source_of_exchange':SOURCE_CURR,
> >                'target_of_exchange':TARGET_CURR,
> >                'amount':float(data[1]),
> >                'date_of_rate':data[0]
> >        }
> >        return q
> >
> > if __name__ == '__main__':
> >        query = [ generate_query(a) for a in file('USD- 
> > AUD-90-99.txt') ]
> >        credentials = metaweb.login(USERNAME, PASSWORD)
> >        for a in range(100, len(query), 100):
> >                result = metaweb.write(query[a:a+100], credentials)
> >                for r in result:
> >                        print r['create'], r['id']
> >
> >
> > Before I added the stepper, it was simply directly doing what's in  
> > side the
> > for loop with the query list.
> >
> > The data format is like:
> > 2-Jan-90                                                        0.7855
> > 3-Jan-90                                                        0.7818
> > ...
> >
> > I will most likely write a script to query the ids and then detype  
> > them and then
> > do an import again with the 100 block steps version to see if that's  
> > the only problem.
> >
> > Luckily this was all on sandbox :)
> >
> > Thanks,
> > Alex
> >
> > _______________________________________________
> > Developers mailing list
> > Developers at freebase.com
> > http://lists.freebase.com/mailman/listinfo/developers
> 
> _______________________________________________
> Developers mailing list
> Developers at freebase.com
> http://lists.freebase.com/mailman/listinfo/developers
>