从 Tableau 地图中抓取数据

编辑我制作了一个tableau scraper 库来将工作表数据提取到 pandas 数据框中。代码更简单，但在您的情况下，您仍然需要使用 xsrf 令牌构建 URL：from tableauscraper import TableauScraper as TSimport requestsfrom bs4 import BeautifulSoupinit_url = "https://idph.illinois.gov/OpioidDataDashboard/"r = requests.get(init_url)soup = BeautifulSoup(r.text, "html.parser")paramTags = dict([    (t["name"], t["value"])    for t in soup.find("div", {"class": "tableauPlaceholder"}).findAll("param")])url = f'{paramTags["host_url"]}trusted/{paramTags["ticket"]}{paramTags["site_root"]}/views/{paramTags["name"]}'ts = TS()ts.loads(url)dashboard = ts.getWorkbook()for t in dashboard.worksheets:    # show worksheet name    print(f"WORKSHEET NAME : {t.name}")    # show dataframe for this worksheet    print(t.data)在 repl.it 上试试这个原帖这有点复杂，因为有以下组合：有 tsconfig textarea 的画面“配置页面”不是原始页面的一部分。url 是从一些paramhtml 标签动态构建的它在 cookie 中使用交叉伪造令牌，但为了获取该 cookie，您需要调用特定的 api，其 url 是从某些paramhtml 标记动态构建的从 tsconfig 参数，我们可以构建数据 url，正如您在其他 stackoverflow 帖子中发现的那样，例如this，this和this流程如下：call GET https://idph.illinois.gov/OpioidDataDashboard/,param用class抓取div下的标签tableauPlaceholder从那里主机是： https: //interactive.data.illinois.gov从前面的 param标签中，构建如下所示的“会话 URL”：GET /trusted/{ticket}/t/DPH/views/opioidTDWEB_prod/MortalityandMorbidity上面的url将只用于存储cookies（包括cookies中的xsrf token）从前面的param标签中，构建如下所示的“配置 URL”：GET /t/DPH/views/opioidTDWEB_prod/MortalityandMorbidity使用 id 提取 textareatsConfigContainer并从中解析 json从上面提取的 json 构建“数据 url”，url 如下所示：POST /vizql/t/DPH/w/opioidTDWEB_prod/v/MortalityandMorbidity/bootstrapSession/sessions/{session_id}然后你有一个 json 响应，它前面有一些字符串以防止json 劫持。你需要正则表达式来提取它然后解析巨大的json数据所有需要的 url 都是这样的：GET https://idph.illinois.gov/OpioidDataDashboard/GET https://interactive.data.illinois.gov/trusted/yIm7jkXyRQuH9Ff1oPvz_w==:790xMcZuwmnvijXHg6ymRTrU/t/DPH/views/opioidTDWEB_prod/MortalityandMorbidityGET https://interactive.data.illinois.gov/t/DPH/views/opioidTDWEB_prod/MortalityandMorbidityPOST https://interactive.data.illinois.gov/vizql/t/DPH/w/opioidTDWEB_prod/v/MortalityandMorbidity/bootstrapSession/sessions/2A3E3BA96A6C4E65B36AEDB4A536D09F-1:0完整代码：import requestsfrom bs4 import BeautifulSoupimport jsonimport res = requests.Session()init_url = "https://idph.illinois.gov/OpioidDataDashboard/"print(f"GET {init_url}")r = s.get(init_url)soup = BeautifulSoup(r.text, "html.parser")paramTags = dict([    (t["name"], t["value"])     for t in soup.find("div", {"class":"tableauPlaceholder"}).findAll("param")])# get xsrf cookiesession_url = f'{paramTags["host_url"]}trusted/{paramTags["ticket"]}{paramTags["site_root"]}/views/{paramTags["name"]}'print(f"GET {session_url}")r = s.get(session_url)config_url = f'{paramTags["host_url"][:-1]}{paramTags["site_root"]}/views/{paramTags["name"]}'print(f"GET {config_url}")r = s.get(config_url,    params = {        ":embed": "y",        ":showVizHome": "no",        ":host_url": "https://interactive.data.illinois.gov/",        ":embed_code_version": 2,        ":tabs": "yes",        ":toolbar": "no",        ":showShareOptions": "false",        ":display_spinner": "no",        ":loadOrderID": 0,})soup = BeautifulSoup(r.text, "html.parser")tableauData = json.loads(soup.find("textarea",{"id": "tsConfigContainer"}).text)dataUrl = f'{paramTags["host_url"][:-1]}{tableauData["vizql_root"]}/bootstrapSession/sessions/{tableauData["sessionid"]}'print(f"POST {dataUrl}")r = s.post(dataUrl, data= {    "sheet_id": tableauData["sheetId"],})dataReg = re.search('\d+;({.*})\d+;({.*})', r.text, re.MULTILINE)info = json.loads(dataReg.group(1))data = json.loads(dataReg.group(2))print(data["secondaryInfo"]["presModelMap"]["dataDictionary"]["presModelHolder"]["genDataDictionaryPresModel"]["dataSegments"]["0"]["dataColumns"])在 repl.it 上试试这个

从 Tableau 地图中抓取数据

1回答