Colly Max Depth and encoding/json - null

Question

我已经完成了围棋之旅，现在正在学习一些Colly教程。我了解最大深度，并一直试图在这样的围棋程序中实现它：

package main

import (

    "encoding/json"

    "log"

    "net/http"

    "github.com/gocolly/colly"

)

func ping(w http.ResponseWriter, r *http.Request) {

    log.Println("Ping")

    w.Write([]byte("ping"))

}

func getData(w http.ResponseWriter, r *http.Request) {

    //Verify the param "URL" exists

    URL := r.URL.Query().Get("url")

    if URL == "" {

        log.Println("missing URL argument")

        return

    }

    log.Println("visiting", URL)

    //Create a new collector which will be in charge of collect the data from HTML

    c := colly.NewCollector(

        // MaxDepth is 2, so only the links on the scraped page

        // and links on those pages are visited

        colly.MaxDepth(2),

        colly.Async(true),

    )

    // Limit the maximum parallelism to 2

    // This is necessary if the goroutines are dynamically

    // created to control the limit of simultaneous requests.

    //

    // Parallelism can be controlled also by spawning fixed

    // number of go routines.

    c.Limit(&colly.LimitRule{DomainGlob: "*", Parallelism: 2})

    //Slices to store the data

    var response []string

    //onHTML function allows the collector to use a callback function when the specific HTML tag is reached

    //in this case whenever our collector finds an

    //anchor tag with href it will call the anonymous function

    // specified below which will get the info from the href and append it to our slice

    c.OnHTML("a[href]", func(e *colly.HTMLElement) {

        link := e.Request.AbsoluteURL(e.Attr("href"))

        if link != "" {

            response = append(response, link)

        }

    })

执行此操作时，响应为 null。取出 MaxDepth 和 Async 行会导致预期的响应（仅使用顶级链接）。

任何帮助是值得赞赏的！

Answer 1

在异步模式下运行时，将在实际发出请求之前返回（请参阅此处）;并行演示中演示了正确的过程。在您的情况下，这意味着：c.Visitc.Visit(URL) c.Wait()仅使用 async 在仅发出一个请求时不是很有用。查看reddit示例，了解如何在一次操作中访问多个URL。注意：您确实应该检查这些函数返回的错误值，添加错误处理程序也是很好的做法。