Guidance to get JSONSource throught https redirect

Lumen
#1

I need to get json data by Https but all the services are behind a SSO (called JOSSO).

So when I use json type, Lumen can’t create the dataframe because it get a 302 redirect temp
code.

I am not sure how to resolve this.

  • should I create a intake plugin which could follow redirection ?
  • might it be include in Lumen ?
  • do I need to create a middleware that do the redirection ? (I use lumen behind a Django app)

Do I missed up something ?
What is the best option ?

#2

sorry, I was wrong, the pandas JSONReader folows the redirection.
but I should give josso_sessionid cookie on the request … don’t know if it’s possible.

#3

Honestly not sure how to pass cookies through read_json we may have to add an alternative reader using requests (and eventually aiohttp).

#4

you mean, we may add aiohttp reader in load_fn as an alternative to the pandas.read_json ?
which it will be a more advanced loader (set cookies, headers, timeout values, etc).
For that we will need to add a parameter in the JSONSource to perform the switch between panda/aiohttp, and kwargs to parametrize the request.

OR we should use another type of source and should duplicate some part of FileSource code to keep resolve_template_vars behaviour and other things ?

I thought the first is better but not sur if it’s what you thought