frequent Gateway Time-out for requests

[t-silvers]

I've been a user of xenaPython for almost 2 years, and HTTP errors when communicating with Xena seem to have gotten worse… i.e., fewer and less frequent requests throw a gateway time-out. Ideally, these time-outs would never happen, because xena allows users to query large data sets frequently in a way that's portable and removes the need to store data somewhere.

Here’s most of a traceback, for example, while pulling this whole data set: https://xenabrowser.net/datapages/?dataset=TCGA-BRCA%2FXena_Matrices%2FTCGA-BRCA.htseq_fpkm-uq.tsv&host=https%3A%2F%2Fgdc.xenahubs.net&removeHub=https%3A%2F%2Fxena.treehouse.gi.ucsc.edu%3A443:

  File "~/site-packages/xenaPython/xenaAPI.py", line 16, in Probes_values
    values = xena.dataset_probe_values (hub, dataset, samples, probes)
  File "~/site-packages/xenaPython/xenaQuery.py", line 242, in dataset_probe_values
    return json.loads(post(host, dataset_probe_str % (quote(dataset), array_fmt(probes), array_fmt(samples))))
  File "~/site-packages/xenaPython/xenaQuery.py", line 201, in post
    response = urlopen(req)
  File "~/urllib/request.py", line 223, in urlopen
    return opener.open(url, data, timeout)
  File "~/urllib/request.py", line 532, in open
    response = meth(req, response)
  File "~/urllib/request.py", line 642, in http_response
    'http', request, response, code, msg, hdrs)
  File "~/urllib/request.py", line 570, in error
    return self._call_chain(*args)
  File "~/urllib/request.py", line 504, in _call_chain
    result = func(*args)
  File "~/urllib/request.py", line 650, in http_error_default
    raise HTTPError(req.full_url, code, msg, hdrs, fp)
urllib.error.HTTPError: HTTP Error 504: Gateway Time-out

[acthp]

Thanks for the report! We'll look into it.

For pulling entire datasets, have you considered using the download service? It should be much faster. The url is the host name + '/download/' + the dataset name + '.gz', as you can see on the page you posted.

The API provides a lot more functionality, primarily matrix slicing, which just slows things down if you want everything.

[acthp]

Maybe we can expose this as a xenaPython call, a "get full dataset" function that takes advantage of the download service.

[t-silvers]

Thanks for the report! We'll look into it.

For pulling entire datasets, have you considered using the download service? It should be much faster. The url is the host name + '/download/' + the dataset name + '.gz', as you can see on the page you posted.

The API provides a lot more functionality, primarily matrix slicing, which just slows things down if you want everything.

Thanks for these quick replies. You're right that download service is faster, but in practice I am actually using xenaPython's matrix slicing functionality to subset by probes and samples. I just pulled the data set above to reproduce the error.

[t-silvers]

quick follow-up to clarify that parallelizing smaller requests is, in practice, where I usually encounter time-outs

[hello1910]

Sorry, has there been any update on this? I'm experiencing a urllib.error.HTTPError: HTTP Error 500: Server Error. My code was initially working and then I've been persistently getting the 500 error for the past day now. I've tried different computers, but it hasn't helped.

[acthp]

The 504 gateway timeouts usually occur when trying to make parallel requests faster than the hub can service them. In that case, the later connections sit idle waiting for the initial connections to finish, and the gateway eventually times them out due to lack of activity. The solution is to use less parallelization.

A 500 is some sort of error at the application layer, which you generally shouldn't see. Can you send us the failing code, at genome-cancer@soe.ucsc.edu?

[hello1910]

It works now, thank you so much for resolving the issue!

[vappiah]

I have the same 504 gateway error.

I was trying to subset gene count values (all genes ) 50 samples