Compare commits

..
1 Commits
Author SHA1 Message Date
Martin Angers 96ed0f357a add PathForNode and NodeAtPath functions 2017-08-12 13:26:49 -04:00
33 changed files with 383 additions and 1460 deletions
-2
View File
@@ -1,2 +0,0 @@
github: [mna]
custom: ["https://www.buymeacoffee.com/mna"]
-12
View File
@@ -1,12 +0,0 @@
version: 2
updates:
# Maintain dependencies for Go
- package-ecosystem: "gomod"
directory: "/"
schedule:
interval: "daily"
# Enable version updates for GitHub action workflows
- package-ecosystem: "github-actions"
directory: "/"
schedule:
interval: "daily"
-25
View File
@@ -1,25 +0,0 @@
name: test
on: [push, pull_request]
env:
GOPROXY: https://proxy.golang.org,direct
jobs:
test:
strategy:
matrix:
go-version: [1.26.x, 1.27.x]
os: [ubuntu-latest, macos-latest, windows-latest]
runs-on: ${{ matrix.os }}
steps:
- name: Checkout code
uses: actions/checkout@v7
- name: Install Go
uses: actions/setup-go@v7
with:
go-version: ${{ matrix.go-version }}
- name: Test
run: go test ./... -v -cover
+12
View File
@@ -0,0 +1,12 @@
language: go
go:
- 1.1.x
- 1.2.x
- 1.3.x
- 1.4.x
- 1.5.x
- 1.6.x
- 1.7.x
- 1.8.x
- tip
+1 -1
View File
@@ -1,4 +1,4 @@
Copyright (c) 2012-2021, Martin Angers & Contributors
Copyright (c) 2012-2016, Martin Angers & Contributors
All rights reserved.
Redistribution and use in source and binary forms, with or without modification, are permitted provided that the following conditions are met:
+19 -97
View File
@@ -1,8 +1,6 @@
# goquery - a little like that j-thing, only in Go
[![build status](https://secure.travis-ci.org/PuerkitoBio/goquery.png)](http://travis-ci.org/PuerkitoBio/goquery) [![GoDoc](https://godoc.org/github.com/PuerkitoBio/goquery?status.png)](http://godoc.org/github.com/PuerkitoBio/goquery) [![Sourcegraph Badge](https://sourcegraph.com/github.com/PuerkitoBio/goquery/-/badge.svg)](https://sourcegraph.com/github.com/PuerkitoBio/goquery?badge)
[![Build Status](https://github.com/PuerkitoBio/goquery/actions/workflows/test.yml/badge.svg?branch=master)](https://github.com/PuerkitoBio/goquery/actions)
[![Go Reference](https://pkg.go.dev/badge/github.com/PuerkitoBio/goquery.svg)](https://pkg.go.dev/github.com/PuerkitoBio/goquery)
[![Sourcegraph Badge](https://sourcegraph.com/github.com/PuerkitoBio/goquery/-/badge.svg)](https://sourcegraph.com/github.com/PuerkitoBio/goquery?badge)
goquery brings a syntax and a set of features similar to [jQuery][] to the [Go language][go]. It is based on Go's [net/html package][html] and the CSS Selector library [cascadia][]. Since the net/html parser returns nodes, and not a full-featured DOM tree, jQuery's stateful manipulation functions (like height(), css(), detach()) have been left off.
@@ -17,20 +15,11 @@ Syntax-wise, it is as close as possible to jQuery, with the same function names
* [API](#api)
* [Examples](#examples)
* [Related Projects](#related-projects)
* [Support](#support)
* [License](#license)
## Installation
Required Go version:
* Starting with version `v1.12.0` of goquery, Go 1.25+ is required due to its dependencies.
* Starting with version `v1.11.0` of goquery, Go 1.24+ is required due to its dependencies.
* Starting with version `v1.10.0` of goquery, Go 1.23+ is required due to the use of function-based iterators.
* For `v1.9.0` of goquery, Go 1.18+ is required due to the use of generics.
* For previous goquery versions, a Go version of 1.1+ was required because of the `net/html` dependency.
Ongoing goquery development is tested on the latest 2 versions of Go.
Please note that because of the net/html dependency, goquery requires Go1.1+.
$ go get github.com/PuerkitoBio/goquery
@@ -48,35 +37,12 @@ Ongoing goquery development is tested on the latest 2 versions of Go.
**Note that goquery's API is now stable, and will not break.**
* **2026-08-27 (v1.13.0)** : Performance improvements (thanks [@jvoisin][jvs]), add top-level `Text` function with options similar to BeautifulSoup's `get_text` (thanks [@ChrisJr404][chrisjr]), update `go.mod` dependencies, add go1.27 to the test matrix.
* **2026-03-15 (v1.12.0)** : Update `go.mod` dependencies, add go1.26 to the test matrix, **goquery now requires Go version 1.25+**.
* **2025-11-16 (v1.11.0)** : Update `go.mod` dependencies, add go1.25 to the test matrix, **goquery now requires Go version 1.24+**.
* **2025-04-11 (v1.10.3)** : Update `go.mod` dependencies, small optimization (thanks [@myxzlpltk](https://github.com/myxzlpltk)).
* **2025-02-13 (v1.10.2)** : Update `go.mod` dependencies, add go1.24 to the test matrix.
* **2024-12-26 (v1.10.1)** : Update `go.mod` dependencies.
* **2024-09-06 (v1.10.0)** : Add `EachIter` which provides an iterator that can be used in `for..range` loops on the `*Selection` object. **goquery now requires Go version 1.23+** (thanks [@amikai](https://github.com/amikai)).
* **2024-09-06 (v1.9.3)** : Update `go.mod` dependencies.
* **2024-04-29 (v1.9.2)** : Update `go.mod` dependencies.
* **2024-02-29 (v1.9.1)** : Improve allocation and performance of the `Map` function and `Selection.Map` method, better document the cascadia differences (thanks [@jwilsson](https://github.com/jwilsson)).
* **2024-02-22 (v1.9.0)** : Add a generic `Map` function, **goquery now requires Go version 1.18+** (thanks [@Fesaa](https://github.com/Fesaa)).
* **2023-02-18 (v1.8.1)** : Update `go.mod` dependencies, update CI workflow.
* **2021-10-25 (v1.8.0)** : Add `Render` function to render a `Selection` to an `io.Writer` (thanks [@anthonygedeon](https://github.com/anthonygedeon)).
* **2021-07-11 (v1.7.1)** : Update go.mod dependencies and add dependabot config (thanks [@jauderho](https://github.com/jauderho)).
* **2021-06-14 (v1.7.0)** : Add `Single` and `SingleMatcher` functions to optimize first-match selection (thanks [@gdollardollar](https://github.com/gdollardollar)).
* **2021-01-11 (v1.6.1)** : Fix panic when calling `{Prepend,Append,Set}Html` on a `Selection` that contains non-Element nodes.
* **2020-10-08 (v1.6.0)** : Parse html in context of the container node for all functions that deal with html strings (`AfterHtml`, `AppendHtml`, etc.). Thanks to [@thiemok][thiemok] and [@davidjwilkins][djw] for their work on this.
* **2020-02-04 (v1.5.1)** : Update module dependencies.
* **2018-11-15 (v1.5.0)** : Go module support (thanks @Zaba505).
* **2018-06-07 (v1.4.1)** : Add `NewDocumentFromReader` examples.
* **2018-03-24 (v1.4.0)** : Deprecate `NewDocument(url)` and `NewDocumentFromResponse(response)`.
* **2018-01-28 (v1.3.0)** : Add `ToEnd` constant to `Slice` until the end of the selection (thanks to @davidjwilkins for raising the issue).
* **2018-01-11 (v1.2.0)** : Add `AddBack*` and deprecate `AndSelf` (thanks to @davidjwilkins).
* **2017-02-12 (v1.1.0)** : Add `SetHtml` and `SetText` (thanks to @glebtv).
* **2016-12-29 (v1.0.2)** : Optimize allocations for `Selection.Text` (thanks to @radovskyb).
* **2016-08-28 (v1.0.1)** : Optimize performance for large documents.
* **2016-07-27 (v1.0.0)** : Tag version 1.0.0.
* **2016-06-15** : Invalid selector strings internally compile to a `Matcher` implementation that never matches any node (instead of a panic). So for example, `doc.Find("~")` returns an empty `*Selection` object.
* **2016-02-02** : Add `NodeName` utility function similar to the DOM's `nodeName` property. It returns the tag name of the first element in a selection, and other relevant values of non-element nodes (see [doc][] for details). Add `OuterHtml` utility function similar to the DOM's `outerHTML` property (named `OuterHtml` in small caps for consistency with the existing `Html` method on the `Selection`).
* **2016-02-02** : Add `NodeName` utility function similar to the DOM's `nodeName` property. It returns the tag name of the first element in a selection, and other relevant values of non-element nodes (see godoc for details). Add `OuterHtml` utility function similar to the DOM's `outerHTML` property (named `OuterHtml` in small caps for consistency with the existing `Html` method on the `Selection`).
* **2015-04-20** : Add `AttrOr` helper method to return the attribute's value or a default value if absent. Thanks to [piotrkowalczuk][piotr].
* **2015-02-04** : Add more manipulation functions - Prepend* - thanks again to [Andrew Stone][thatguystone].
* **2014-11-28** : Add more manipulation functions - ReplaceWith*, Wrap* and Unwrap - thanks again to [Andrew Stone][thatguystone].
@@ -105,9 +71,9 @@ jQuery often has many variants for the same function (no argument, a selector st
Utility functions that are not in jQuery but are useful in Go are implemented as functions (that take a `*Selection` as parameter), to avoid a potential naming clash on the `*Selection`'s methods (reserved for jQuery-equivalent behaviour).
The complete [package reference documentation can be found here][doc].
The complete [godoc reference documentation can be found here][doc].
Please note that Cascadia's selectors do not necessarily match all supported selectors of jQuery (Sizzle). See the [cascadia project][cascadia] for details. Also, the selectors work more like the DOM's `querySelectorAll`, than jQuery's matchers - they have no concept of contextual matching (for some concrete examples of what that means, see [this ticket](https://github.com/andybalholm/cascadia/issues/61)). In practice, it doesn't matter very often but it's something worth mentioning. Invalid selector strings compile to a `Matcher` that fails to match any node. Behaviour of the various functions that take a selector string as argument follows from that fact, e.g. (where `~` is an invalid selector string):
Please note that Cascadia's selectors do not necessarily match all supported selectors of jQuery (Sizzle). See the [cascadia project][cascadia] for details. Invalid selector strings compile to a `Matcher` that fails to match any node. Behaviour of the various functions that take a selector string as argument follows from that fact, e.g. (where `~` is an invalid selector string):
* `Find("~")` returns an empty selection because the selector string doesn't match anything.
* `Add("~")` returns a new selection that holds the same nodes as the original selection, because it didn't add any node (selector string didn't match anything).
@@ -126,34 +92,23 @@ package main
import (
"fmt"
"log"
"net/http"
"github.com/PuerkitoBio/goquery"
)
func ExampleScrape() {
// Request the HTML page.
res, err := http.Get("http://metalsucks.net")
if err != nil {
log.Fatal(err)
}
defer res.Body.Close()
if res.StatusCode != 200 {
log.Fatalf("status code error: %d %s", res.StatusCode, res.Status)
}
// Load the HTML document
doc, err := goquery.NewDocumentFromReader(res.Body)
doc, err := goquery.NewDocument("http://metalsucks.net")
if err != nil {
log.Fatal(err)
}
// Find the review items
doc.Find(".left-content article .post-title").Each(func(i int, s *goquery.Selection) {
// For each item found, get the title
title := s.Find("a").Text()
fmt.Printf("Review %d: %s\n", i, title)
})
doc.Find(".sidebar-reviews article .content-block").Each(func(i int, s *goquery.Selection) {
// For each item found, get the band and title
band := s.Find("a").Text()
title := s.Find("i").Text()
fmt.Printf("Review %d: %s - %s\n", i, band, title)
})
}
func main() {
@@ -166,56 +121,23 @@ func main() {
- [Goq][goq], an HTML deserialization and scraping library based on goquery and struct tags.
- [andybalholm/cascadia][cascadia], the CSS selector library used by goquery.
- [suntong/cascadia][cascadiacli], a command-line interface to the cascadia CSS selector library, useful to test selectors.
- [gocolly/colly](https://github.com/gocolly/colly), a lightning fast and elegant Scraping Framework
- [gnulnx/goperf](https://github.com/gnulnx/goperf), a website performance test tool that also fetches static assets.
- [MontFerret/ferret](https://github.com/MontFerret/ferret), declarative web scraping.
- [tacusci/berrycms](https://github.com/tacusci/berrycms), a modern simple to use CMS with easy to write plugins
- [Dataflow kit](https://github.com/slotix/dataflowkit), Web Scraping framework for Gophers.
- [Geziyor](https://github.com/geziyor/geziyor), a fast web crawling & scraping framework for Go. Supports JS rendering.
- [Pagser](https://github.com/foolin/pagser), a simple, easy, extensible, configurable HTML parser to struct based on goquery and struct tags.
- [stitcherd](https://github.com/vhodges/stitcherd), A server for doing server side includes using css selectors and DOM updates.
- [goskyr](https://github.com/jakopako/goskyr), an easily configurable command-line scraper written in Go.
- [goGetJS](https://github.com/davemolk/goGetJS), a tool for extracting, searching, and saving JavaScript files (with optional headless browser).
- [fitter](https://github.com/PxyUp/fitter), a tool for selecting values from JSON, XML, HTML and XPath formatted pages.
- [seltabl](github.com/conneroisu/seltabl), an orm-like package and supporting language server for extracting values from HTML
## Support
There are a number of ways you can support the project:
* Use it, star it, build something with it, spread the word!
- If you do build something open-source or otherwise publicly-visible, let me know so I can add it to the [Related Projects](#related-projects) section!
* Raise issues to improve the project (note: doc typos and clarifications are issues too!)
- Please search existing issues before opening a new one - it may have already been addressed.
* Pull requests: please discuss new code in an issue first, unless the fix is really trivial.
- Make sure new code is tested.
- Be mindful of existing code - PRs that break existing code have a high probability of being declined, unless it fixes a serious issue.
* Sponsor the developer
- See the Github Sponsor button at the top of the repo on github
- or via BuyMeACoffee.com, below
<a href="https://www.buymeacoffee.com/mna" target="_blank"><img src="https://www.buymeacoffee.com/assets/img/custom_images/orange_img.png" alt="Buy Me A Coffee" style="height: 41px !important;width: 174px !important;box-shadow: 0px 3px 2px 0px rgba(190, 190, 190, 0.5) !important;-webkit-box-shadow: 0px 3px 2px 0px rgba(190, 190, 190, 0.5) !important;" ></a>
## License
The [BSD 3-Clause license][bsd], the same as the [Go language][golic]. Cascadia's license is [here][caslic].
[jquery]: https://jquery.com/
[go]: https://go.dev/
[jquery]: http://jquery.com/
[go]: http://golang.org/
[cascadia]: https://github.com/andybalholm/cascadia
[cascadiacli]: https://github.com/suntong/cascadia
[bsd]: https://opensource.org/licenses/BSD-3-Clause
[golic]: https://go.dev/LICENSE
[bsd]: http://opensource.org/licenses/BSD-3-Clause
[golic]: http://golang.org/LICENSE
[caslic]: https://github.com/andybalholm/cascadia/blob/master/LICENSE
[doc]: https://pkg.go.dev/github.com/PuerkitoBio/goquery
[index]: https://api.jquery.com/index/
[doc]: http://godoc.org/github.com/PuerkitoBio/goquery
[index]: http://api.jquery.com/index/
[gonet]: https://github.com/golang/net/
[html]: https://pkg.go.dev/golang.org/x/net/html
[html]: http://godoc.org/golang.org/x/net/html
[wiki]: https://github.com/PuerkitoBio/goquery/wiki/Tips-and-tricks
[thatguystone]: https://github.com/thatguystone
[piotr]: https://github.com/piotrkowalczuk
[goq]: https://github.com/andrewstuart/goq
[thiemok]: https://github.com/thiemok
[djw]: https://github.com/davidjwilkins
[jvs]: https://github.com/jvoisin
[chrisjr]: https://github.com/ChrisJr404
+3 -30
View File
@@ -4,16 +4,6 @@ import (
"golang.org/x/net/html"
)
const (
maxUint = ^uint(0)
maxInt = int(maxUint >> 1)
// ToEnd is a special index value that can be used as end index in a call
// to Slice so that all elements are selected until the end of the Selection.
// It is equivalent to passing (*Selection).Length().
ToEnd = maxInt
)
// First reduces the set of matched elements to the first in the set.
// It returns a new Selection object, and an empty Selection object if the
// the selection is empty.
@@ -45,23 +35,12 @@ func (s *Selection) Eq(index int) *Selection {
}
// Slice reduces the set of matched elements to a subset specified by a range
// of indices. The start index is 0-based and indicates the index of the first
// element to select. The end index is 0-based and indicates the index at which
// the elements stop being selected (the end index is not selected).
//
// The indices may be negative, in which case they represent an offset from the
// end of the selection.
//
// The special value ToEnd may be specified as end index, in which case all elements
// until the end are selected. This works both for a positive and negative start
// index.
// of indices.
func (s *Selection) Slice(start, end int) *Selection {
if start < 0 {
start += len(s.Nodes)
}
if end == ToEnd {
end = len(s.Nodes)
} else if end < 0 {
if end < 0 {
end += len(s.Nodes)
}
return pushStack(s, s.Nodes[start:end])
@@ -81,13 +60,7 @@ func (s *Selection) Get(index int) *html.Node {
// relative to its sibling elements.
func (s *Selection) Index() int {
if len(s.Nodes) > 0 {
i := 0
for n := s.Nodes[0].PrevSibling; n != nil; n = n.PrevSibling {
if n.Type == html.ElementNode {
i++
}
}
return i
return newSingleSelection(s.Nodes[0], s.document).PrevAll().Length()
}
return -1
}
-22
View File
@@ -98,17 +98,6 @@ func TestSlice(t *testing.T) {
sel := Doc().Find(".pvk-content").Slice(0, 2)
assertLength(t, sel.Nodes, 2)
assertSelectionIs(t, sel, "#pc1", "#pc2")
}
func TestSliceToEnd(t *testing.T) {
sel := Doc().Find(".pvk-content").Slice(1, ToEnd)
assertLength(t, sel.Nodes, 2)
assertSelectionIs(t, sel.Eq(0), "#pc2")
if _, ok := sel.Eq(1).Attr("id"); ok {
t.Error("Want no attribute ID, got one")
}
}
func TestSliceEmpty(t *testing.T) {
@@ -121,11 +110,6 @@ func TestSliceInvalid(t *testing.T) {
Doc().Find("").Slice(0, 2)
}
func TestSliceInvalidToEnd(t *testing.T) {
defer assertPanic(t)
Doc().Find("").Slice(2, ToEnd)
}
func TestSliceOutOfBounds(t *testing.T) {
defer assertPanic(t)
Doc().Find(".pvk-content").Slice(2, 12)
@@ -151,12 +135,6 @@ func TestNegativeSliceBoth(t *testing.T) {
assertSelectionIs(t, sel.Eq(1), "#cf3")
}
func TestNegativeSliceToEnd(t *testing.T) {
sel := Doc().Find(".container-fluid").Slice(-3, ToEnd)
assertLength(t, sel.Nodes, 3)
assertSelectionIs(t, sel, "#cf2", "#cf3", "#cf4")
}
func TestNegativeSliceOutOfBounds(t *testing.T) {
defer assertPanic(t)
Doc().Find(".container-fluid").Slice(-12, -7)
+4 -4
View File
@@ -1,15 +1,15 @@
package goquery
import (
"bytes"
"fmt"
"strconv"
"strings"
"testing"
)
func BenchmarkMetalReviewExample(b *testing.B) {
var n int
var builder strings.Builder
var buf bytes.Buffer
b.StopTimer()
doc := loadDoc("metalreview.html")
@@ -27,12 +27,12 @@ func BenchmarkMetalReviewExample(b *testing.B) {
if score, e = strconv.ParseFloat(s.Find(".score").Text(), 64); e != nil {
// Not a valid float, ignore score
if n <= 4 {
builder.WriteString(fmt.Sprintf("Review %d: %s - %s.\n", i, band, title))
buf.WriteString(fmt.Sprintf("Review %d: %s - %s.\n", i, band, title))
}
} else {
// Print all, including score
if n <= 4 {
builder.WriteString(fmt.Sprintf("Review %d: %s - %s (%2.1f).\n", i, band, title, score))
buf.WriteString(fmt.Sprintf("Review %d: %s - %s (%2.1f).\n", i, band, title, score))
}
}
})
+1 -22
View File
@@ -2,8 +2,6 @@ package goquery
import (
"testing"
"github.com/andybalholm/cascadia"
)
func BenchmarkFilter(b *testing.B) {
@@ -42,25 +40,6 @@ func BenchmarkNot(b *testing.B) {
}
}
func BenchmarkNotMatcher(b *testing.B) {
var n int
b.StopTimer()
sel := DocW().Find("li")
m := cascadia.MustCompile(".toclevel-2")
b.StartTimer()
for i := 0; i < b.N; i++ {
if n == 0 {
n = sel.NotMatcher(m).Length()
} else {
sel.NotMatcher(m)
}
}
if n != 371 {
b.Fatalf("want 371, got %d", n)
}
}
func BenchmarkFilterFunction(b *testing.B) {
var n int
@@ -252,6 +231,6 @@ func BenchmarkEnd(b *testing.B) {
}
}
if n != 373 {
b.Fatalf("want 373, got %d", n)
b.Fatalf("wnat 373, got %d", n)
}
}
+1 -44
View File
@@ -1,7 +1,6 @@
package goquery
import (
"strconv"
"testing"
)
@@ -25,48 +24,6 @@ func BenchmarkEach(b *testing.B) {
}
}
func BenchmarkEachIter(b *testing.B) {
var tmp, n int
b.StopTimer()
sel := DocW().Find("td")
b.StartTimer()
for i := 0; i < b.N; i++ {
for range sel.EachIter() {
tmp++
}
if n == 0 {
n = tmp
}
}
if n != 59 {
b.Fatalf("want 59, got %d", n)
}
}
func BenchmarkEachIterWithBreak(b *testing.B) {
var tmp, n int
b.StopTimer()
sel := DocW().Find("td")
b.StartTimer()
for i := 0; i < b.N; i++ {
tmp = 0
for range sel.EachIter() {
tmp++
if tmp >= 10 {
break
}
}
if n == 0 {
n = tmp
}
}
if n != 10 {
b.Fatalf("want 10, got %d", n)
}
}
func BenchmarkMap(b *testing.B) {
var tmp, n int
@@ -74,7 +31,7 @@ func BenchmarkMap(b *testing.B) {
sel := DocW().Find("td")
f := func(i int, s *Selection) string {
tmp++
return strconv.Itoa(tmp)
return string(tmp)
}
b.StartTimer()
for i := 0; i < b.N; i++ {
+1 -20
View File
@@ -46,25 +46,6 @@ func BenchmarkHtml(b *testing.B) {
sel := DocW().Find("h2")
b.StartTimer()
for i := 0; i < b.N; i++ {
_, _ = sel.Html()
}
}
func BenchmarkAddClass(b *testing.B) {
b.StopTimer()
sel := DocW().Find("li")
b.StartTimer()
for i := 0; i < b.N; i++ {
sel.AddClass("foo bar baz")
}
}
func BenchmarkRemoveClass(b *testing.B) {
b.StopTimer()
sel := DocW().Find("li")
sel.AddClass("foo bar baz")
b.StartTimer()
for i := 0; i < b.N; i++ {
sel.RemoveClass("foo bar baz")
sel.Html()
}
}
-20
View File
@@ -2,8 +2,6 @@ package goquery
import (
"testing"
"github.com/andybalholm/cascadia"
)
func BenchmarkFind(b *testing.B) {
@@ -802,21 +800,3 @@ func BenchmarkClosestNodes(b *testing.B) {
b.Fatalf("want 2, got %d", n)
}
}
func BenchmarkSingleMatcher(b *testing.B) {
doc := Doc()
multi := cascadia.MustCompile(`div`)
single := SingleMatcher(multi)
b.ResetTimer()
b.Run("multi", func(b *testing.B) {
for i := 0; i < b.N; i++ {
_ = doc.FindMatcher(multi)
}
})
b.Run("single", func(b *testing.B) {
for i := 0; i < b.N; i++ {
_ = doc.FindMatcher(single)
}
})
}
+3 -3
View File
@@ -4,13 +4,13 @@
The `go.net/html` package used by `goquery` requires that the html document is UTF-8 encoded. When you know the encoding of the html page is not UTF-8, you can use the `iconv` package to convert it to UTF-8 (there are various implementation of the `iconv` API, see [godoc.org][iconv] for other options):
```bash
```
$ go get -u github.com/djimenez/iconv-go
```
and then:
```golang
```
// Load the URL
res, err := http.Get(url)
if err != nil {
@@ -51,7 +51,7 @@ You can find a code example using `otto` [in this gist][exotto]. Thanks to githu
If all you need is a normal `for` loop over all nodes in the current selection, where `Map/Each`-style iteration is not necessary, you can use the following:
```golang
```
sel := Doc().Find(".selector")
for i := range sel.Nodes {
single := sel.Eq(i)
+1 -119
View File
@@ -3,28 +3,14 @@ package goquery_test
import (
"fmt"
"log"
"net/http"
"os"
"strings"
"github.com/PuerkitoBio/goquery"
"golang.org/x/net/html"
)
// This example scrapes the reviews shown on the home page of metalsucks.net.
func Example() {
// Request the HTML page.
res, err := http.Get("http://metalsucks.net")
if err != nil {
log.Fatal(err)
}
defer res.Body.Close()
if res.StatusCode != 200 {
log.Fatalf("status code error: %d %s", res.StatusCode, res.Status)
}
// Load the HTML document
doc, err := goquery.NewDocumentFromReader(res.Body)
doc, err := goquery.NewDocument("http://metalsucks.net")
if err != nil {
log.Fatal(err)
}
@@ -42,107 +28,3 @@ func Example() {
// xOutput: voluntarily fail the Example output.
}
// This example shows how to use NewDocumentFromReader from a file.
func ExampleNewDocumentFromReader_file() {
// create from a file
f, err := os.Open("some/file.html")
if err != nil {
log.Fatal(err)
}
defer f.Close()
doc, err := goquery.NewDocumentFromReader(f)
if err != nil {
log.Fatal(err)
}
// use the goquery document...
_ = doc.Find("h1")
}
// This example shows how to use NewDocumentFromReader from a string.
func ExampleNewDocumentFromReader_string() {
// create from a string
data := `
<html>
<head>
<title>My document</title>
</head>
<body>
<h1>Header</h1>
</body>
</html>`
doc, err := goquery.NewDocumentFromReader(strings.NewReader(data))
if err != nil {
log.Fatal(err)
}
header := doc.Find("h1").Text()
fmt.Println(header)
// Output: Header
}
func ExampleSingle() {
html := `
<html>
<body>
<div>1</div>
<div>2</div>
<div>3</div>
</body>
</html>
`
doc, err := goquery.NewDocumentFromReader(strings.NewReader(html))
if err != nil {
log.Fatal(err)
}
// By default, the selector string selects all matching nodes
multiSel := doc.Find("div")
fmt.Println(multiSel.Text())
// Using goquery.Single, only the first match is selected
singleSel := doc.FindMatcher(goquery.Single("div"))
fmt.Println(singleSel.Text())
// Output:
// 123
// 1
}
// This example shows how to use the goquery.Text function to extract clean,
// human-readable text from a selection, similar to BeautifulSoup's get_text.
func ExampleText() {
page := `
<html>
<body>
<div id="content">
<h1> Hello </h1>
<p>world</p>
<script>var ignored = 1;</script>
</div>
</body>
</html>
`
doc, err := goquery.NewDocumentFromReader(strings.NewReader(page))
if err != nil {
log.Fatal(err)
}
// Trim each text node, join the remaining ones with a space, and skip the
// text of <script> and <style> elements.
text := goquery.Text(doc.Find("#content"), &goquery.TextOptions{
Separator: " ",
Trim: true,
Keep: func(n *html.Node) bool {
if p := n.Parent; p != nil && p.Type == html.ElementNode {
return p.Data != "script" && p.Data != "style"
}
return true
},
})
fmt.Println(text)
// Output:
// Hello world
}
-24
View File
@@ -41,30 +41,6 @@ func (s *Selection) AddNodes(nodes ...*html.Node) *Selection {
// AndSelf adds the previous set of elements on the stack to the current set.
// It returns a new Selection object containing the current Selection combined
// with the previous one.
// Deprecated: This function has been deprecated and is now an alias for AddBack().
func (s *Selection) AndSelf() *Selection {
return s.AddBack()
}
// AddBack adds the previous set of elements on the stack to the current set.
// It returns a new Selection object containing the current Selection combined
// with the previous one.
func (s *Selection) AddBack() *Selection {
return s.AddSelection(s.prevSel)
}
// AddBackFiltered reduces the previous set of elements on the stack to those that
// match the selector string, and adds them to the current set.
// It returns a new Selection object containing the current Selection combined
// with the filtered previous one
func (s *Selection) AddBackFiltered(selector string) *Selection {
return s.AddSelection(s.prevSel.Filter(selector))
}
// AddBackMatcher reduces the previous set of elements on the stack to those that match
// the matcher, and adds them to the current set.
// It returns a new Selection object containing the current Selection combined
// with the filtered previous one
func (s *Selection) AddBackMatcher(m Matcher) *Selection {
return s.AddSelection(s.prevSel.FilterMatcher(m))
}
-22
View File
@@ -94,25 +94,3 @@ func TestAndSelfRollback(t *testing.T) {
sel2 := sel.Find("a").AndSelf().End().End()
assertEqual(t, sel, sel2)
}
func TestAddBack(t *testing.T) {
sel := Doc().Find(".span12").Last().AddBack()
assertLength(t, sel.Nodes, 2)
}
func TestAddBackRollback(t *testing.T) {
sel := Doc().Find(".pvk-content")
sel2 := sel.Find("a").AddBack().End().End()
assertEqual(t, sel, sel2)
}
func TestAddBackFiltered(t *testing.T) {
sel := Doc().Find(".span12, .footer").Find("h1").AddBackFiltered(".footer")
assertLength(t, sel.Nodes, 2)
}
func TestAddBackFilteredRollback(t *testing.T) {
sel := Doc().Find(".span12, .footer")
sel2 := sel.Find("h1").AddBackFiltered(".footer").End().End()
assertEqual(t, sel, sel2)
}
+24 -53
View File
@@ -12,7 +12,7 @@ func (s *Selection) Filter(selector string) *Selection {
// the given matcher. It returns a new Selection object for this subset
// of matching elements.
func (s *Selection) FilterMatcher(m Matcher) *Selection {
return pushStack(s, winnow(s.Nodes, m, true))
return pushStack(s, winnow(s, m, true))
}
// Not removes elements from the Selection that match the selector string.
@@ -24,7 +24,7 @@ func (s *Selection) Not(selector string) *Selection {
// NotMatcher removes elements from the Selection that match the given matcher.
// It returns a new Selection object with the matching elements removed.
func (s *Selection) NotMatcher(m Matcher) *Selection {
return pushStack(s, winnow(s.Nodes, m, false))
return pushStack(s, winnow(s, m, false))
}
// FilterFunction reduces the set of matched elements to those that pass the function's test.
@@ -79,46 +79,29 @@ func (s *Selection) Intersection(sel *Selection) *Selection {
// that matches the selector.
// It returns a new Selection object with the matching elements.
func (s *Selection) Has(selector string) *Selection {
return s.HasMatcher(compileMatcher(selector))
return s.HasSelection(s.document.Find(selector))
}
// HasMatcher reduces the set of matched elements to those that have a descendant
// that matches the matcher.
// It returns a new Selection object with the matching elements.
func (s *Selection) HasMatcher(m Matcher) *Selection {
result := make([]*html.Node, 0, len(s.Nodes))
// Manually create a singleMatcher (rather than using SingleMatcher, which
// returns a Matcher interface) so we can call its MatchFirst method
// directly. MatchFirst probes for the first match in a subtree without
// building a result slice, avoiding a throwaway one-element slice
// allocation for every matching child subtree.
sm := singleMatcher{m}
for _, n := range s.Nodes {
for c := n.FirstChild; c != nil; c = c.NextSibling {
if c.Type == html.ElementNode && sm.MatchFirst(c) != nil {
result = append(result, n)
break
}
}
}
return pushStack(s, result)
return s.HasSelection(s.document.FindMatcher(m))
}
// HasNodes reduces the set of matched elements to those that have a
// descendant that matches one of the nodes.
// It returns a new Selection object with the matching elements.
func (s *Selection) HasNodes(nodes ...*html.Node) *Selection {
var result []*html.Node
for _, n := range s.Nodes {
for _, candidate := range nodes {
if nodeContains(n, candidate) {
result = append(result, n)
break
return s.FilterFunction(func(_ int, sel *Selection) bool {
// Add all nodes that contain one of the specified nodes
for _, n := range nodes {
if sel.Contains(n) {
return true
}
}
}
return pushStack(s, result)
return false
})
}
// HasSelection reduces the set of matched elements to those that have a
@@ -142,45 +125,33 @@ func (s *Selection) End() *Selection {
// Filter based on the matcher, and the indicator to keep (Filter) or
// to get rid of (Not) the matching elements.
func winnow(nodes []*html.Node, m Matcher, keep bool) []*html.Node {
func winnow(sel *Selection, m Matcher, keep bool) []*html.Node {
// Optimize if keep is requested
if keep {
return m.Filter(nodes)
return m.Filter(sel.Nodes)
}
// Not path: call Match directly on each node, no Selection wrapper needed
result := make([]*html.Node, 0, len(nodes))
for _, n := range nodes {
if !m.Match(n) {
result = append(result, n)
}
}
return result
// Use grep
return grep(sel, func(i int, s *Selection) bool {
return !m.Match(s.Get(0))
})
}
// Filter based on an array of nodes, and the indicator to keep (Filter) or
// to get rid of (Not) the matching elements.
func winnowNodes(sel *Selection, nodes []*html.Node, keep bool) []*html.Node {
if len(nodes)+len(sel.Nodes) < minNodesForSet {
var result []*html.Node
for _, n := range sel.Nodes {
if isInSlice(nodes, n) == keep {
result = append(result, n)
}
}
return result
return grep(sel, func(i int, s *Selection) bool {
return isInSlice(nodes, s.Get(0)) == keep
})
}
set := make(map[*html.Node]bool, len(nodes))
set := make(map[*html.Node]bool)
for _, n := range nodes {
set[n] = true
}
var result []*html.Node
for _, n := range sel.Nodes {
if set[n] == keep {
result = append(result, n)
}
}
return result
return grep(sel, func(i int, s *Selection) bool {
return set[s.Get(0)] == keep
})
}
// Filter based on a function test, and the indicator to keep (Filter) or
-8
View File
@@ -1,8 +0,0 @@
module github.com/PuerkitoBio/goquery
require (
github.com/andybalholm/cascadia v1.3.4
golang.org/x/net v0.58.0
)
go 1.25.0
-4
View File
@@ -1,4 +0,0 @@
github.com/andybalholm/cascadia v1.3.4 h1:vM2lgh0Vru9Vwyfm4cQqWP2HHMW0u0+2PAW7Q38Qufg=
github.com/andybalholm/cascadia v1.3.4/go.mod h1:BLRmbRjpEtNKieZOCCvYj4RqN+KRA41GBe/5O+G93kM=
golang.org/x/net v0.58.0 h1:ynWG7rqYi4ccpTEuPZ2QGWHktVEM9DMCj9yzDE0Q7To=
golang.org/x/net v0.58.0/go.mod h1:YwCddHnFlT7eLQqVprV19OnhLGtc5xOKgE0RyqgfWAU=
+1 -23
View File
@@ -1,7 +1,5 @@
package goquery
import "iter"
// Each iterates over a Selection object, executing a function for each
// matched element. It returns the current Selection object. The function
// f is called for each element in the selection with the index of the
@@ -14,18 +12,6 @@ func (s *Selection) Each(f func(int, *Selection)) *Selection {
return s
}
// EachIter returns an iterator that yields the Selection object in order.
// The implementation is similar to Each, but it returns an iterator instead.
func (s *Selection) EachIter() iter.Seq2[int, *Selection] {
return func(yield func(int, *Selection) bool) {
for i, n := range s.Nodes {
if !yield(i, newSingleSelection(n, s.document)) {
return
}
}
}
}
// EachWithBreak iterates over a Selection object, executing a function for each
// matched element. It is identical to Each except that it is possible to break
// out of the loop by returning false in the callback function. It returns the
@@ -45,16 +31,8 @@ func (s *Selection) EachWithBreak(f func(int, *Selection) bool) *Selection {
// element in that selection starting at 0, and a *Selection that contains
// only that element.
func (s *Selection) Map(f func(int, *Selection) string) (result []string) {
return Map(s, f)
}
// Map is the generic version of Selection.Map, allowing any type to be
// returned.
func Map[E any](s *Selection, f func(int, *Selection) E) (result []E) {
result = make([]E, len(s.Nodes))
for i, n := range s.Nodes {
result[i] = f(i, newSingleSelection(n, s.document))
result = append(result, f(i, newSingleSelection(n, s.document)))
}
return result
-55
View File
@@ -86,58 +86,3 @@ func TestForRange(t *testing.T) {
t.Errorf("expected initial selection to still have length %d, got %d", initLen, sel.Length())
}
}
func TestGenericMap(t *testing.T) {
sel := Doc().Find(".pvk-content")
vals := Map(sel, func(i int, s *Selection) *html.NodeType {
n := s.Get(0)
if n.Type == html.ElementNode {
return &n.Type
}
return nil
})
for _, v := range vals {
if v == nil || *v != html.ElementNode {
t.Error("Expected Map array result to be all 'div's.")
}
}
if len(vals) != 3 {
t.Errorf("Expected Map array result to have a length of 3, found %v.", len(vals))
}
}
func TestEachIter(t *testing.T) {
var cnt int
sel := Doc().Find(".hero-unit .row-fluid")
for i, s := range sel.EachIter() {
cnt++
t.Logf("At index %v, node %v", i, s.Nodes[0].Data)
}
sel = sel.Find("a")
if cnt != 4 {
t.Errorf("Expected Each() to call function 4 times, got %v times.", cnt)
}
assertLength(t, sel.Nodes, 6)
}
func TestEachIterWithBreak(t *testing.T) {
var cnt int
sel := Doc().Find(".hero-unit .row-fluid")
for i, s := range sel.EachIter() {
cnt++
t.Logf("At index %v, node %v", i, s.Nodes[0].Data)
break
}
sel = sel.Find("a")
if cnt != 1 {
t.Errorf("Expected Each() to call function 1 time, got %v times.", cnt)
}
assertLength(t, sel.Nodes, 6)
}
+39 -146
View File
@@ -39,15 +39,8 @@ func (s *Selection) AfterSelection(sel *Selection) *Selection {
// AfterHtml parses the html and inserts it after the set of matched elements.
//
// This follows the same rules as Selection.Append.
func (s *Selection) AfterHtml(htmlStr string) *Selection {
return s.eachNodeHtml(htmlStr, true, func(node *html.Node, nodes []*html.Node) {
nextSibling := node.NextSibling
for _, n := range nodes {
if node.Parent != nil {
node.Parent.InsertBefore(n, nextSibling)
}
}
})
func (s *Selection) AfterHtml(html string) *Selection {
return s.AfterNodes(parseHtml(html)...)
}
// AfterNodes inserts the nodes after each element in the set of matched elements.
@@ -92,12 +85,8 @@ func (s *Selection) AppendSelection(sel *Selection) *Selection {
}
// AppendHtml parses the html and appends it to the set of matched elements.
func (s *Selection) AppendHtml(htmlStr string) *Selection {
return s.eachNodeHtml(htmlStr, false, func(node *html.Node, nodes []*html.Node) {
for _, n := range nodes {
node.AppendChild(n)
}
})
func (s *Selection) AppendHtml(html string) *Selection {
return s.AppendNodes(parseHtml(html)...)
}
// AppendNodes appends the specified nodes to each node in the set of matched elements.
@@ -134,14 +123,8 @@ func (s *Selection) BeforeSelection(sel *Selection) *Selection {
// BeforeHtml parses the html and inserts it before the set of matched elements.
//
// This follows the same rules as Selection.Append.
func (s *Selection) BeforeHtml(htmlStr string) *Selection {
return s.eachNodeHtml(htmlStr, true, func(node *html.Node, nodes []*html.Node) {
for _, n := range nodes {
if node.Parent != nil {
node.Parent.InsertBefore(n, node)
}
}
})
func (s *Selection) BeforeHtml(html string) *Selection {
return s.BeforeNodes(parseHtml(html)...)
}
// BeforeNodes inserts the nodes before each element in the set of matched elements.
@@ -166,16 +149,8 @@ func (s *Selection) Clone() *Selection {
// Empty removes all children nodes from the set of matched elements.
// It returns the children nodes in a new Selection.
func (s *Selection) Empty() *Selection {
// Count the children first so nodes can be presized, avoiding append's
// repeated reallocations.
count := 0
for _, n := range s.Nodes {
for c := n.FirstChild; c != nil; c = c.NextSibling {
count++
}
}
var nodes []*html.Node
nodes := make([]*html.Node, 0, count)
for _, n := range s.Nodes {
for c := n.FirstChild; c != nil; c = n.FirstChild {
n.RemoveChild(c)
@@ -209,13 +184,8 @@ func (s *Selection) PrependSelection(sel *Selection) *Selection {
}
// PrependHtml parses the html and prepends it to the set of matched elements.
func (s *Selection) PrependHtml(htmlStr string) *Selection {
return s.eachNodeHtml(htmlStr, false, func(node *html.Node, nodes []*html.Node) {
firstChild := node.FirstChild
for _, n := range nodes {
node.InsertBefore(n, firstChild)
}
})
func (s *Selection) PrependHtml(html string) *Selection {
return s.PrependNodes(parseHtml(html)...)
}
// PrependNodes prepends the specified nodes to each node in the set of
@@ -242,19 +212,14 @@ func (s *Selection) Remove() *Selection {
return s
}
// RemoveFiltered removes from the current set of matched elements those that
// match the selector filter. It returns the Selection of removed nodes.
//
// For example if the selection s contains "<h1>", "<h2>" and "<h3>"
// and s.RemoveFiltered("h2") is called, only the "<h2>" node is removed
// (and returned), while "<h1>" and "<h3>" are kept in the document.
// RemoveFiltered removes the set of matched elements by selector.
// It returns the Selection of removed nodes.
func (s *Selection) RemoveFiltered(selector string) *Selection {
return s.RemoveMatcher(compileMatcher(selector))
}
// RemoveMatcher removes from the current set of matched elements those that
// match the Matcher filter. It returns the Selection of removed nodes.
// See RemoveFiltered for additional information.
// RemoveMatcher removes the set of matched elements.
// It returns the Selection of removed nodes.
func (s *Selection) RemoveMatcher(m Matcher) *Selection {
return s.FilterMatcher(m).Remove()
}
@@ -291,16 +256,8 @@ func (s *Selection) ReplaceWithSelection(sel *Selection) *Selection {
// It returns the removed elements.
//
// This follows the same rules as Selection.Append.
func (s *Selection) ReplaceWithHtml(htmlStr string) *Selection {
s.eachNodeHtml(htmlStr, true, func(node *html.Node, nodes []*html.Node) {
nextSibling := node.NextSibling
for _, n := range nodes {
if node.Parent != nil {
node.Parent.InsertBefore(n, nextSibling)
}
}
})
return s.Remove()
func (s *Selection) ReplaceWithHtml(html string) *Selection {
return s.ReplaceWithNodes(parseHtml(html)...)
}
// ReplaceWithNodes replaces each element in the set of matched elements with
@@ -313,23 +270,13 @@ func (s *Selection) ReplaceWithNodes(ns ...*html.Node) *Selection {
return s.Remove()
}
// SetHtml sets the html content of each element in the selection to
// specified html string.
func (s *Selection) SetHtml(htmlStr string) *Selection {
for _, context := range s.Nodes {
for c := context.FirstChild; c != nil; c = context.FirstChild {
context.RemoveChild(c)
}
}
return s.eachNodeHtml(htmlStr, false, func(node *html.Node, nodes []*html.Node) {
for _, n := range nodes {
node.AppendChild(n)
}
})
// Set the html content of each element in the selection to specified html string.
func (s *Selection) SetHtml(html string) *Selection {
return setHtmlNodes(s, parseHtml(html)...)
}
// SetText sets the content of each element in the selection to specified content.
// The provided text string is escaped.
// Set the content of each element in the selection to specified content. The
// provided text string is escaped.
func (s *Selection) SetText(text string) *Selection {
return s.SetHtml(html.EscapeString(text))
}
@@ -381,17 +328,8 @@ func (s *Selection) WrapSelection(sel *Selection) *Selection {
// most child of the given HTML.
//
// It returns the original set of elements.
func (s *Selection) WrapHtml(htmlStr string) *Selection {
cache := make(map[string][]*html.Node)
for _, n := range s.Nodes {
parent := n.Parent
if parent == nil {
parent = &html.Node{Type: html.ElementNode}
}
nodes := cachedParseHtmlWithContext(cache, htmlStr, parent)
newSingleSelection(n, s.document).wrapAllNodes(cloneNodes(nodes)...)
}
return s
func (s *Selection) WrapHtml(html string) *Selection {
return s.wrapNodes(parseHtml(html)...)
}
// WrapNode wraps each element in the set of matched elements inside the inner-
@@ -443,18 +381,8 @@ func (s *Selection) WrapAllSelection(sel *Selection) *Selection {
// document.
//
// It returns the original set of elements.
func (s *Selection) WrapAllHtml(htmlStr string) *Selection {
var context *html.Node
var nodes []*html.Node
if len(s.Nodes) > 0 {
context = s.Nodes[0]
if context.Parent != nil {
nodes = parseHtmlWithContext(htmlStr, context)
} else {
nodes = parseHtml(htmlStr)
}
}
return s.wrapAllNodes(nodes...)
func (s *Selection) WrapAllHtml(html string) *Selection {
return s.wrapAllNodes(parseHtml(html)...)
}
func (s *Selection) wrapAllNodes(ns ...*html.Node) *Selection {
@@ -523,13 +451,8 @@ func (s *Selection) WrapInnerSelection(sel *Selection) *Selection {
// cloned before being inserted into the document.
//
// It returns the original set of elements.
func (s *Selection) WrapInnerHtml(htmlStr string) *Selection {
cache := make(map[string][]*html.Node)
for _, n := range s.Nodes {
nodes := cachedParseHtmlWithContext(cache, htmlStr, n)
newSingleSelection(n, s.document).wrapInnerNodes(cloneNodes(nodes)...)
}
return s
func (s *Selection) WrapInnerHtml(html string) *Selection {
return s.wrapInnerNodes(parseHtml(html)...)
}
// WrapInnerNode wraps an HTML structure, matched by the given selector, around
@@ -560,19 +483,27 @@ func (s *Selection) wrapInnerNodes(ns ...*html.Node) *Selection {
}
func parseHtml(h string) []*html.Node {
return parseHtmlWithContext(h, &html.Node{Type: html.ElementNode})
}
func parseHtmlWithContext(h string, context *html.Node) []*html.Node {
// Errors are only returned when the io.Reader returns any error besides
// EOF, but strings.Reader never will
nodes, err := html.ParseFragment(strings.NewReader(h), context)
nodes, err := html.ParseFragment(strings.NewReader(h), &html.Node{Type: html.ElementNode})
if err != nil {
panic("goquery: failed to parse HTML: " + err.Error())
}
return nodes
}
func setHtmlNodes(s *Selection, ns ...*html.Node) *Selection {
for _, n := range s.Nodes {
for c := n.FirstChild; c != nil; c = n.FirstChild {
n.RemoveChild(c)
}
for _, c := range ns {
n.AppendChild(cloneNode(c))
}
}
return s
}
// Get the first child that is an ElementNode
func getFirstChildEl(n *html.Node) *html.Node {
c := n.FirstChild
@@ -640,41 +571,3 @@ func (s *Selection) manipulateNodes(ns []*html.Node, reverse bool,
return s
}
// eachNodeHtml parses the given html string and inserts the resulting nodes in the dom with the mergeFn.
// The parsed nodes are inserted for each element of the selection.
// isParent can be used to indicate that the elements of the selection should be treated as the parent for the parsed html.
// A cache is used to avoid parsing the html multiple times should the elements of the selection result in the same context.
func (s *Selection) eachNodeHtml(htmlStr string, isParent bool, mergeFn func(n *html.Node, nodes []*html.Node)) *Selection {
cache := make(map[string][]*html.Node)
for _, n := range s.Nodes {
var context *html.Node
if isParent {
context = n.Parent
} else {
if n.Type != html.ElementNode {
continue
}
context = n
}
if context == nil {
continue
}
nodes := cachedParseHtmlWithContext(cache, htmlStr, context)
mergeFn(n, cloneNodes(nodes))
}
return s
}
// cachedParseHtmlWithContext returns parseHtmlWithContext(htmlStr, context), reusing a prior
// result when context's nodeName has already been seen. Callers pass their own
// cache map so the cache lifetime matches the caller's loop scope.
func cachedParseHtmlWithContext(cache map[string][]*html.Node, htmlStr string, context *html.Node) []*html.Node {
key := nodeName(context)
if nodes, ok := cache[key]; ok {
return nodes
}
nodes := parseHtmlWithContext(htmlStr, context)
cache[key] = nodes
return nodes
}
-226
View File
@@ -1,7 +1,6 @@
package goquery
import (
"log"
"testing"
)
@@ -57,26 +56,6 @@ func TestAfterHtml(t *testing.T) {
printSel(t, doc.Selection)
}
func TestAfterHtmlContext(t *testing.T) {
doc := loadString(t, `
<html>
<body>
<table>
<tr>
<td>Before1</td>
</tr>
<tr>
<td>Before2</td>
</tr>
</table>
</body>
</html>`)
doc.Find("table tr td").AfterHtml("<td class='c1'>Test</td><td class='c2'>Again</td>")
assertLength(t, doc.Find("table tr td").Nodes, 6)
assertClass(t, doc.Find("table tr td").Last(), "c2")
printSel(t, doc.Selection)
}
func TestAppend(t *testing.T) {
doc := Doc2Clone()
doc.Find("#main").Append("#nf6")
@@ -134,27 +113,6 @@ func TestAppendHtml(t *testing.T) {
printSel(t, doc.Selection)
}
func TestAppendHtmlContext(t *testing.T) {
doc := loadString(t, `
<html>
<body>
<table>
<tr>
<td>Before1</td>
</tr>
<tr>
<td>Before2</td>
</tr>
</table>
</body>
</html>`)
doc.Find("table tr").AppendHtml("<td class='c1'>new1</td><td class='c2'>new2</td>")
assertLength(t, doc.Find("table td").Nodes, 6)
assertClass(t, doc.Find("table td").Last(), "c2")
printSel(t, doc.Selection)
}
func TestBefore(t *testing.T) {
doc := Doc2Clone()
doc.Find("#main").Before("#nf6")
@@ -193,27 +151,6 @@ func TestBeforeHtml(t *testing.T) {
printSel(t, doc.Selection)
}
func TestBeforeHtmlContext(t *testing.T) {
doc := loadString(t, `
<html>
<body>
<table>
<tr>
<td>Before1</td>
</tr>
<tr>
<td>Before2</td>
</tr>
</table>
</body>
</html>`)
doc.Find("table tr td:first-child").BeforeHtml("<td class='c1'>new1</td><td class='c2'>new2</td>")
assertLength(t, doc.Find("table td").Nodes, 6)
assertClass(t, doc.Find("table td").First(), "c1")
printSel(t, doc.Selection)
}
func TestEmpty(t *testing.T) {
doc := Doc2Clone()
s := doc.Find("#main").Empty()
@@ -281,27 +218,6 @@ func TestPrependHtml(t *testing.T) {
printSel(t, doc.Selection)
}
func TestPrependHtmlContext(t *testing.T) {
doc := loadString(t, `
<html>
<body>
<table>
<tr>
<td>Before1</td>
</tr>
<tr>
<td>Before2</td>
</tr>
</table>
</body>
</html>`)
doc.Find("table tr").PrependHtml("<td class='c1'>new node</td><td class='c2'>other new node</td>")
assertLength(t, doc.Find("table td").Nodes, 6)
assertClass(t, doc.Find("table tr td").First(), "c1")
printSel(t, doc.Selection)
}
func TestRemove(t *testing.T) {
doc := Doc2Clone()
doc.Find("#nf1").Remove()
@@ -362,28 +278,6 @@ func TestReplaceWithHtml(t *testing.T) {
printSel(t, doc.Selection)
}
func TestReplaceWithHtmlContext(t *testing.T) {
doc := loadString(t, `
<html>
<body>
<table>
<tr>
<th>Before1</th>
</tr>
<tr>
<th>Before2</th>
</tr>
</table>
</body>
</html>`)
doc.Find("table th").ReplaceWithHtml("<td class='c1'>Test</td><td class='c2'>Replace</td>")
assertLength(t, doc.Find("table th").Nodes, 0)
assertLength(t, doc.Find("table tr td").Nodes, 4)
assertClass(t, doc.Find("table tr td").First(), "c1")
printSel(t, doc.Selection)
}
func TestSetHtml(t *testing.T) {
doc := Doc2Clone()
q := doc.Find("#main, #foot")
@@ -419,28 +313,6 @@ func TestSetHtmlEmpty(t *testing.T) {
printSel(t, doc.Selection)
}
func TestSetHtmlContext(t *testing.T) {
doc := loadString(t, `
<html>
<body>
<table>
<tr>
<th>Before1</th>
</tr>
<tr>
<th>Before2</th>
</tr>
</table>
</body>
</html>`)
doc.Find("table tr").SetHtml("<td class='c1'>Test</td><td class='c2'>Again</td>")
assertLength(t, doc.Find("table th").Nodes, 0)
assertLength(t, doc.Find("table td").Nodes, 4)
assertLength(t, doc.Find("table tr").Nodes, 2)
printSel(t, doc.Selection)
}
func TestSetText(t *testing.T) {
doc := Doc2Clone()
q := doc.Find("#main, #foot")
@@ -639,101 +511,3 @@ func TestWrapInnerHtml(t *testing.T) {
printSel(t, doc.Selection)
}
func TestParsingRespectsVaryingContext(t *testing.T) {
docA := loadString(t, `
<html>
<body>
<a class="x"></a>
</body>
</html>`)
docTable := loadString(t, `
<html>
<body>
<table class="x"></table>
</body>
</html>`)
docBoth := loadString(t, `
<html>
<body>
<table class="x"></table>
<a class="x"></a>
</body>
</html>`)
sA := docA.Find(".x").AppendHtml("<tr><td>Hello</td></tr>")
sTable := docTable.Find(".x").AppendHtml("<tr><td>Hello</td></tr>")
sBoth := docBoth.Find(".x").AppendHtml("<tr><td>Hello</td></tr>")
printSel(t, docA.Selection)
printSel(t, docTable.Selection)
printSel(t, docBoth.Selection)
oA, _ := sA.Html()
oTable, _ := sTable.Html()
if oA == oTable {
t.Errorf("Expected inner html of <a> and <table> to not be equal, but got %s and %s", oA, oTable)
}
oBothTable, _ := sBoth.First().Html()
if oBothTable != oTable {
t.Errorf("Expected inner html of <table> and <table> in doc containing both tags to be equal, but got %s and %s",
oTable,
oBothTable)
}
oBothA, _ := sBoth.Last().Html()
if oBothA != oA {
t.Errorf("Expected inner html of <a> and <a> in doc containing both tags to be equal, but got %s and %s",
oA,
oBothA)
}
}
func TestHtmlWithNonElementNode(t *testing.T) {
const data = `
<html>
<head>
</head>
<body>
<p>
This is <span>some</span><b>text</b>.
</p>
</body>
</html>
`
cases := map[string]func(*Selection, string) *Selection{
"AfterHtml": (*Selection).AfterHtml,
"AppendHtml": (*Selection).AppendHtml,
"BeforeHtml": (*Selection).BeforeHtml,
"PrependHtml": (*Selection).PrependHtml,
"ReplaceWithHtml": (*Selection).ReplaceWithHtml,
"SetHtml": (*Selection).SetHtml,
}
for nm, fn := range cases {
// this test is only to make sure that the HTML parsing/manipulation
// methods do not raise panics when executed over Selections that contain
// non-Element nodes.
t.Run(nm, func(t *testing.T) {
doc := loadString(t, data)
sel := doc.Find("p").Contents()
func() {
defer func() {
if err := recover(); err != nil {
t.Fatal(err)
}
}()
fn(sel, "<div></div>")
}()
// print the resulting document in verbose mode
h, err := OuterHtml(doc.Selection)
if err != nil {
log.Fatal(err)
}
t.Log(h)
})
}
}
+15
View File
@@ -18,6 +18,21 @@ gosimple $(go list ./... | grep -v /vendor/)
echo "<<< gosimple"
echo
echo ">>> staticcheck"
staticcheck $(go list ./... | grep -v /vendor/)
echo "<<< staticcheck"
echo
echo ">>> unused"
unused $(go list ./... | grep -v /vendor/)
echo "<<< unused"
echo
echo ">>> gas"
gas $(find . -name "*.go" | grep -v /vendor/ | grep -v '_test.go$')
echo "<<< gas"
echo
# Check for gofmt problems and report if any.
gofiles=$(git diff --cached --name-only --diff-filter=ACM | grep '.go$' | grep -v /vendor/)
[ -z "$gofiles" ] && echo "EXIT $vetres" && exit $vetres
+58 -49
View File
@@ -1,31 +1,37 @@
package goquery
import (
"bytes"
"regexp"
"strings"
"golang.org/x/net/html"
)
var classTrimReplacer = strings.NewReplacer("\t", " ", "\r", " ", "\n", " ")
var rxClassTrim = regexp.MustCompile("[\t\r\n]")
// Attr gets the specified attribute's value for the first element in the
// Selection. To get the value for each element individually, use a looping
// construct such as Each or Map method.
func (s *Selection) Attr(attrName string) (string, bool) {
if len(s.Nodes) != 0 {
if attr := getAttributePtr(attrName, s.Nodes[0]); attr != nil {
return attr.Val, true
}
func (s *Selection) Attr(attrName string) (val string, exists bool) {
if len(s.Nodes) == 0 {
return
}
return "", false
return getAttributeValue(attrName, s.Nodes[0])
}
// AttrOr works like Attr but returns default value if attribute is not present.
func (s *Selection) AttrOr(attrName, defaultValue string) string {
if val, exists := s.Attr(attrName); exists {
return val
if len(s.Nodes) == 0 {
return defaultValue
}
return defaultValue
val, exists := getAttributeValue(attrName, s.Nodes[0])
if !exists {
return defaultValue
}
return val
}
// RemoveAttr removes the named attribute from each element in the set of matched elements.
@@ -54,22 +60,26 @@ func (s *Selection) SetAttr(attrName, val string) *Selection {
// Text gets the combined text contents of each element in the set of matched
// elements, including their descendants.
func (s *Selection) Text() string {
var builder strings.Builder
var buf bytes.Buffer
// Slightly optimized vs calling Each: no single selection object created
var f func(*html.Node)
f = func(n *html.Node) {
if n.Type == html.TextNode {
// Keep newlines and spaces, like jQuery
buf.WriteString(n.Data)
}
if n.FirstChild != nil {
for c := n.FirstChild; c != nil; c = c.NextSibling {
f(c)
}
}
}
for _, n := range s.Nodes {
s.textHelper(n, &builder)
f(n)
}
return builder.String()
}
func (s *Selection) textHelper(n *html.Node, builder *strings.Builder) {
if n.Type == html.TextNode {
// Keep newlines and spaces, like jQuery
builder.WriteString(n.Data)
}
for c := n.FirstChild; c != nil; c = c.NextSibling {
s.textHelper(c, builder)
}
return buf.String()
}
// Size is an alias for Length.
@@ -87,16 +97,16 @@ func (s *Selection) Length() int {
func (s *Selection) Html() (ret string, e error) {
// Since there is no .innerHtml, the HTML content must be re-created from
// the nodes using html.Render.
var builder strings.Builder
var buf bytes.Buffer
if len(s.Nodes) > 0 {
for c := s.Nodes[0].FirstChild; c != nil; c = c.NextSibling {
e = html.Render(&builder, c)
e = html.Render(&buf, c)
if e != nil {
return
}
}
ret = builder.String()
ret = buf.String()
}
return
@@ -113,7 +123,7 @@ func (s *Selection) AddClass(class ...string) *Selection {
tcls := getClassesSlice(classStr)
for _, n := range s.Nodes {
curClasses, attr := getClassesAndAttr(n)
curClasses, attr := getClassesAndAttr(n, true)
for _, newClass := range tcls {
if !strings.Contains(curClasses, " "+newClass+" ") {
curClasses += newClass + " "
@@ -129,20 +139,11 @@ func (s *Selection) AddClass(class ...string) *Selection {
// HasClass determines whether any of the matched elements are assigned the
// given class.
func (s *Selection) HasClass(class string) bool {
rawClass := class
class = " " + class + " "
for _, n := range s.Nodes {
if n.Type != html.ElementNode {
continue
}
if attr := getAttributePtr("class", n); attr != nil {
if !strings.Contains(attr.Val, rawClass) {
continue
}
val := classTrimReplacer.Replace(attr.Val)
if strings.Contains(" "+val+" ", class) {
return true
}
classes, _ := getClassesAndAttr(n, false)
if strings.Contains(classes, class) {
return true
}
}
return false
@@ -165,9 +166,9 @@ func (s *Selection) RemoveClass(class ...string) *Selection {
if remove {
removeAttr(n, "class")
} else {
classes, attr := getClassesAndAttr(n)
classes, attr := getClassesAndAttr(n, true)
for _, rcl := range rclasses {
classes = strings.ReplaceAll(classes, " "+rcl+" ", " ")
classes = strings.Replace(classes, " "+rcl+" ", " ", -1)
}
setClasses(n, attr, classes)
@@ -189,11 +190,10 @@ func (s *Selection) ToggleClass(class ...string) *Selection {
tcls := getClassesSlice(classStr)
for _, n := range s.Nodes {
classes, attr := getClassesAndAttr(n)
classes, attr := getClassesAndAttr(n, true)
for _, tcl := range tcls {
spaceAroundTcl := " " + tcl + " "
if strings.Contains(classes, spaceAroundTcl) {
classes = strings.ReplaceAll(classes, spaceAroundTcl, " ")
if strings.Contains(classes, " "+tcl+" ") {
classes = strings.Replace(classes, " "+tcl+" ", " ", -1)
} else {
classes += tcl + " "
}
@@ -218,12 +218,21 @@ func getAttributePtr(attrName string, n *html.Node) *html.Attribute {
return nil
}
// Private function to get the specified attribute's value from a node.
func getAttributeValue(attrName string, n *html.Node) (val string, exists bool) {
if a := getAttributePtr(attrName, n); a != nil {
val = a.Val
exists = true
}
return
}
// Get and normalize the "class" attribute from the node.
func getClassesAndAttr(n *html.Node) (classes string, attr *html.Attribute) {
func getClassesAndAttr(n *html.Node, create bool) (classes string, attr *html.Attribute) {
// Applies only to element nodes
if n.Type == html.ElementNode {
attr = getAttributePtr("class", n)
if attr == nil {
if attr == nil && create {
n.Attr = append(n.Attr, html.Attribute{
Key: "class",
Val: "",
@@ -235,14 +244,14 @@ func getClassesAndAttr(n *html.Node) (classes string, attr *html.Attribute) {
if attr == nil {
classes = " "
} else {
classes = classTrimReplacer.Replace(" " + attr.Val + " ")
classes = rxClassTrim.ReplaceAllString(" "+attr.Val+" ", " ")
}
return
}
func getClassesSlice(classes string) []string {
return strings.Fields(classes)
return strings.Split(rxClassTrim.ReplaceAllString(" "+classes+" ", " "), " ")
}
func removeAttr(n *html.Node, attrName string) {
+13 -20
View File
@@ -5,49 +5,42 @@ import "golang.org/x/net/html"
// Is checks the current matched set of elements against a selector and
// returns true if at least one of these elements matches.
func (s *Selection) Is(selector string) bool {
return s.IsMatcher(compileMatcher(selector))
if len(s.Nodes) > 0 {
return s.IsMatcher(compileMatcher(selector))
}
return false
}
// IsMatcher checks the current matched set of elements against a matcher and
// returns true if at least one of these elements matches.
func (s *Selection) IsMatcher(m Matcher) bool {
for _, n := range s.Nodes {
if m.Match(n) {
return true
if len(s.Nodes) > 0 {
if len(s.Nodes) == 1 {
return m.Match(s.Nodes[0])
}
return len(m.Filter(s.Nodes)) > 0
}
return false
}
// IsFunction checks the current matched set of elements against a predicate and
// returns true if at least one of these elements matches.
func (s *Selection) IsFunction(f func(int, *Selection) bool) bool {
for i, n := range s.Nodes {
if f(i, newSingleSelection(n, s.document)) {
return true
}
}
return false
return s.FilterFunction(f).Length() > 0
}
// IsSelection checks the current matched set of elements against a Selection object
// and returns true if at least one of these elements matches.
func (s *Selection) IsSelection(sel *Selection) bool {
if sel == nil {
return false
}
return s.IsNodes(sel.Nodes...)
return s.FilterSelection(sel).Length() > 0
}
// IsNodes checks the current matched set of elements against the specified nodes
// and returns true if at least one of these elements matches.
func (s *Selection) IsNodes(nodes ...*html.Node) bool {
for _, n := range s.Nodes {
if isInSlice(nodes, n) {
return true
}
}
return false
return s.FilterNodes(nodes...).Length() > 0
}
// Contains returns true if the specified Node is within,
+20 -48
View File
@@ -20,12 +20,6 @@ const (
// Find gets the descendants of each element in the current set of matched
// elements, filtered by a selector. It returns a new Selection object
// containing these matched elements.
//
// Note that as for all methods accepting a selector string, the selector is
// compiled and applied by the cascadia package and inherits its behavior and
// constraints regarding supported selectors. See the note on cascadia in
// the goquery documentation here:
// https://github.com/PuerkitoBio/goquery?tab=readme-ov-file#api
func (s *Selection) Find(selector string) *Selection {
return pushStack(s, findWithMatcher(s.Nodes, compileMatcher(selector)))
}
@@ -149,7 +143,7 @@ func (s *Selection) ClosestMatcher(m Matcher) *Selection {
// ClosestNodes gets the first element that matches one of the nodes by testing the
// element itself and traversing up through its ancestors in the DOM tree.
func (s *Selection) ClosestNodes(nodes ...*html.Node) *Selection {
set := make(map[*html.Node]bool, len(nodes))
set := make(map[*html.Node]bool)
for _, n := range nodes {
set[n] = true
}
@@ -537,7 +531,10 @@ func (s *Selection) PrevMatcherUntilNodes(filter Matcher, nodes ...*html.Node) *
// Filter and push filters the nodes based on a matcher, and pushes the results
// on the stack, with the srcSel as previous selection.
func filterAndPush(srcSel *Selection, nodes []*html.Node, m Matcher) *Selection {
return pushStack(srcSel, winnow(nodes, m, true))
// Create a temporary Selection with the specified nodes to filter using winnow
sel := &Selection{nodes, srcSel.document, nil}
// Filter based on matcher and push on stack
return pushStack(srcSel, winnow(sel, m, true))
}
// Internal implementation of Find that return raw nodes.
@@ -559,12 +556,13 @@ func findWithMatcher(nodes []*html.Node, m Matcher) []*html.Node {
func getParentsNodes(nodes []*html.Node, stopm Matcher, stopNodes []*html.Node) []*html.Node {
return mapNodes(nodes, func(i int, n *html.Node) (result []*html.Node) {
for p := n.Parent; p != nil; p = p.Parent {
sel := newSingleSelection(p, nil)
if stopm != nil {
if stopm.Match(p) {
if sel.IsMatcher(stopm) {
break
}
} else if len(stopNodes) > 0 {
if isInSlice(stopNodes, p) {
if sel.IsNodes(stopNodes...) {
break
}
}
@@ -585,9 +583,13 @@ func getSiblingNodes(nodes []*html.Node, st siblingType, untilm Matcher, untilNo
if st == siblingNextUntil || st == siblingPrevUntil {
f = func(n *html.Node) bool {
if untilm != nil {
return untilm.Match(n)
// Matcher-based condition
sel := newSingleSelection(n, nil)
return sel.IsMatcher(untilm)
} else if len(untilNodes) > 0 {
return isInSlice(untilNodes, n)
// Nodes-based condition
sel := newSingleSelection(n, nil)
return sel.IsNodes(untilNodes...)
}
return false
}
@@ -654,20 +656,6 @@ func getChildrenWithSiblingType(parent *html.Node, st siblingType, skipNode *htm
}
}
// For the cases that collect every matching sibling, count them in a
// cheap pointer walk first so the result slice can be sized exactly,
// avoiding repeated slice growth. The Until cases are skipped (counting
// would require running the predicate twice) and so are the single-result
// Next/Prev cases.
switch st {
case siblingAll, siblingAllIncludingNonElements, siblingPrevAll, siblingNextAll:
n := 0
for c := iter(nil); c != nil; c = iter(c) {
n++
}
result = make([]*html.Node, 0, n)
}
for c := iter(nil); c != nil; c = iter(c) {
// If this is an ...Until case, test before append (returns true
// if the until condition is reached)
@@ -687,21 +675,12 @@ func getChildrenWithSiblingType(parent *html.Node, st siblingType, skipNode *htm
// Internal implementation of parent nodes that return a raw slice of Nodes.
func getParentNodes(nodes []*html.Node) []*html.Node {
// Collect parents inline rather than going through mapNodes, which would
// allocate a throwaway one-element slice per source node. Many source
// nodes (e.g. siblings) share the same parent, so deduplicate as we go.
var result []*html.Node
set := make(map[*html.Node]bool, len(nodes))
for _, n := range nodes {
p := n.Parent
if p == nil || p.Type != html.ElementNode || set[p] {
continue
return mapNodes(nodes, func(i int, n *html.Node) []*html.Node {
if n.Parent != nil && n.Parent.Type == html.ElementNode {
return []*html.Node{n.Parent}
}
set[p] = true
result = append(result, p)
}
return result
return nil
})
}
// Internal map function used by many traversing methods. Takes the source nodes
@@ -709,14 +688,7 @@ func getParentNodes(nodes []*html.Node) []*html.Node {
// Returns an array of nodes mapped by calling the callback function once for
// each node in the source nodes.
func mapNodes(nodes []*html.Node, f func(int, *html.Node) []*html.Node) (result []*html.Node) {
switch len(nodes) {
case 0:
return nil
case 1:
return f(0, nodes[0])
}
set := make(map[*html.Node]bool, len(nodes))
set := make(map[*html.Node]bool)
for i, n := range nodes {
if vals := f(i, n); len(vals) > 0 {
result = appendWithoutDuplicates(result, vals, set)
-17
View File
@@ -3,8 +3,6 @@ package goquery
import (
"strings"
"testing"
"golang.org/x/net/html"
)
func TestFind(t *testing.T) {
@@ -53,21 +51,6 @@ func TestChildren(t *testing.T) {
assertLength(t, sel.Nodes, 5)
}
func TestChildrenDuplicateSourceNodes(t *testing.T) {
// A Selection's Nodes field is exported, so callers can construct a
// selection that contains the same node more than once. Children (and the
// other children-based traversals) must still deduplicate so that the
// shared child set is not returned multiple times.
base := Doc().Find(".pvk-content")
node := base.Nodes[0]
expected := base.Eq(0).Children().Length()
dup := &Selection{Nodes: []*html.Node{node, node}, document: base.document}
assertLength(t, dup.Children().Nodes, expected)
assertLength(t, dup.ChildrenFiltered("*").Nodes, expected)
assertLength(t, dup.Contents().Nodes, base.Eq(0).Contents().Length())
}
func TestChildrenRollback(t *testing.T) {
sel := Doc().Find(".pvk-content")
sel2 := sel.Children().End()
+4 -81
View File
@@ -7,6 +7,7 @@ import (
"net/url"
"github.com/andybalholm/cascadia"
"golang.org/x/net/html"
)
@@ -30,10 +31,6 @@ func NewDocumentFromNode(root *html.Node) *Document {
// NewDocument is a Document constructor that takes a string URL as argument.
// It loads the specified document, parses it, and stores the root Document
// node, ready to be manipulated.
//
// Deprecated: Use the net/http standard library package to make the request
// and validate the response before calling goquery.NewDocumentFromReader
// with the response's body.
func NewDocument(url string) (*Document, error) {
// Load the URL
res, e := http.Get(url)
@@ -43,10 +40,10 @@ func NewDocument(url string) (*Document, error) {
return NewDocumentFromResponse(res)
}
// NewDocumentFromReader returns a Document from an io.Reader.
// NewDocumentFromReader returns a Document from a generic reader.
// It returns an error as second value if the reader's data cannot be parsed
// as html. It does not check if the reader is also an io.Closer, the
// provided reader is never closed by this call. It is the responsibility
// as html. It does *not* check if the reader is also an io.Closer, so the
// provided reader is never closed by this call, it is the responsibility
// of the caller to close it if required.
func NewDocumentFromReader(r io.Reader) (*Document, error) {
root, e := html.Parse(r)
@@ -59,8 +56,6 @@ func NewDocumentFromReader(r io.Reader) (*Document, error) {
// NewDocumentFromResponse is another Document constructor that takes an http response as argument.
// It loads the specified response's document, parses it, and stores the root Document
// node, ready to be manipulated. The response's body is closed on return.
//
// Deprecated: Use goquery.NewDocumentFromReader with the response's body.
func NewDocumentFromResponse(res *http.Response) (*Document, error) {
if res == nil {
return nil, errors.New("Response is nil")
@@ -121,45 +116,6 @@ type Matcher interface {
Filter([]*html.Node) []*html.Node
}
// Single compiles a selector string to a Matcher that stops after the first
// match is found.
//
// By default, Selection.Find and other functions that accept a selector string
// to select nodes will use all matches corresponding to that selector. By
// using the Matcher returned by Single, at most the first match will be
// selected.
//
// For example, those two statements are semantically equivalent:
//
// sel1 := doc.Find("a").First()
// sel2 := doc.FindMatcher(goquery.Single("a"))
//
// The one using Single is optimized to be potentially much faster on large
// documents.
//
// Only the behaviour of the MatchAll method of the Matcher interface is
// altered compared to standard Matchers. This means that the single-selection
// property of the Matcher only applies for Selection methods where the Matcher
// is used to select nodes, not to filter or check if a node matches the
// Matcher - in those cases, the behaviour of the Matcher is unchanged (e.g.
// FilterMatcher(Single("div")) will still result in a Selection with multiple
// "div"s if there were many "div"s in the Selection to begin with).
func Single(selector string) Matcher {
return singleMatcher{compileMatcher(selector)}
}
// SingleMatcher returns a Matcher matches the same nodes as m, but that stops
// after the first match is found.
//
// See the documentation of function Single for more details.
func SingleMatcher(m Matcher) Matcher {
if _, ok := m.(singleMatcher); ok {
// m is already a singleMatcher
return m
}
return singleMatcher{m}
}
// compileMatcher compiles the selector string s and returns
// the corresponding Matcher. If s is an invalid selector string,
// it returns a Matcher that fails all matches.
@@ -171,39 +127,6 @@ func compileMatcher(s string) Matcher {
return cs
}
type singleMatcher struct {
Matcher
}
// MatchFirst returns the first node that matches, or nil if none does. It uses
// the underlying Matcher's MatchFirst method if it provides one (cascadia-
// compiled matchers all do), avoiding building a result slice. Otherwise it
// falls back to MatchAll and returns its first element.
func (m singleMatcher) MatchFirst(n *html.Node) *html.Node {
// Optimized version - stops finding at the first match (cascadia-compiled
// matchers all use this code path).
if mm, ok := m.Matcher.(interface{ MatchFirst(*html.Node) *html.Node }); ok {
return mm.MatchFirst(n)
}
// Fallback version, for e.g. test mocks that don't provide the MatchFirst
// method.
nodes := m.Matcher.MatchAll(n)
if len(nodes) > 0 {
return nodes[0]
}
return nil
}
func (m singleMatcher) MatchAll(n *html.Node) []*html.Node {
// Wrap MatchFirst's single result in a slice to respect the Matcher
// interface's MatchAll signature.
if node := m.MatchFirst(n); node != nil {
return []*html.Node{node}
}
return nil
}
// invalidMatcher is a Matcher that always fails to match.
type invalidMatcher struct{}
-53
View File
@@ -7,7 +7,6 @@ import (
"strings"
"testing"
"github.com/andybalholm/cascadia"
"golang.org/x/net/html"
)
@@ -122,14 +121,6 @@ func loadDoc(page string) *Document {
return NewDocumentFromNode(node)
}
func loadString(t *testing.T, doc string) *Document {
d, err := NewDocumentFromReader(strings.NewReader(doc))
if err != nil {
t.Error("Failed to parse test document")
}
return d
}
func TestNewDocument(t *testing.T) {
if f, e := os.Open("./testdata/page.html"); e != nil {
t.Error(e.Error())
@@ -209,47 +200,3 @@ func TestIssue103(t *testing.T) {
}
t.Log(text)
}
func TestSingle(t *testing.T) {
data := `
<html>
<body>
<div class="b">1</div>
<div class="a">2</div>
<div class="a">3</div>
<p class="b">4</p>
</body>
</html>
`
doc, err := NewDocumentFromReader(strings.NewReader(data))
if err != nil {
t.Fatal(err)
}
text := doc.FindMatcher(Single("div")).Text()
if text != "1" {
t.Fatalf("want %q, got %q", "1", text)
}
// Verify semantic equivalence
sel1 := doc.Find("div").First()
sel2 := doc.FindMatcher(Single("div"))
if sel1.Text() != sel2.Text() {
t.Fatalf("want sel1 to equal sel2")
}
// Here, the Single has no effect as the selector is used to filter
// from the existing selection, not to find nodes in the document.
divs := doc.Find("div")
text = divs.FilterMatcher(Single(".a")).Text()
if text != "23" {
t.Fatalf("want %q, got %q", "23", text)
}
classA := cascadia.MustCompile(".a")
classB := cascadia.MustCompile(".b")
text = doc.FindMatcher(classB).AddMatcher(SingleMatcher(classA)).Text()
if text != "142" {
t.Fatalf("want %q, got %q", "142", text)
}
}
+78 -104
View File
@@ -1,8 +1,7 @@
package goquery
import (
"io"
"strings"
"bytes"
"golang.org/x/net/html"
)
@@ -19,6 +18,62 @@ var nodeNames = []string{
html.CommentNode: "#comment",
}
// PathForNode returns a unique path to retrieve the specified node
// from its document tree. The path is a slice of int indices, starting
// at the root of the tree.
func PathForNode(n *html.Node) []int {
var indices []int
for n := n; n != nil; n = n.Parent {
ix := 0
for prev := n.PrevSibling; prev != nil; prev = prev.PrevSibling {
ix++
}
indices = append(indices, ix)
}
// reverse the slice of indices
for l, r := 0, len(indices)-1; l < r; l, r = l+1, r-1 {
indices[l], indices[r] = indices[r], indices[l]
}
return indices
}
// NodeAtPath returns the HTML node at the specified path in the
// document tree of the specified n node. The path is followed from
// the root of the tree. If no node is found by following the path,
// nil is returned.
func NodeAtPath(path []int, n *html.Node) *html.Node {
if n == nil {
return n
}
// start at root
for n.Parent != nil {
n = n.Parent
}
for n.PrevSibling != nil {
n = n.PrevSibling
}
for i, ix := range path {
if i > 0 {
n = n.FirstChild
if n == nil {
return n
}
}
for j := 0; j < ix; j++ {
n = n.NextSibling
if n == nil {
return n
}
}
}
return n
}
// NodeName returns the node name of the first element in the selection.
// It tries to behave in a similar way as the DOM's nodeName property
// (https://developer.mozilla.org/en-US/docs/Web/API/Node/nodeName).
@@ -26,127 +81,46 @@ var nodeNames = []string{
// Go's net/html package defines the following node types, listed with
// the corresponding returned value from this function:
//
// ErrorNode : #error
// TextNode : #text
// DocumentNode : #document
// ElementNode : the element's tag name
// CommentNode : #comment
// DoctypeNode : the name of the document type
// ErrorNode : #error
// TextNode : #text
// DocumentNode : #document
// ElementNode : the element's tag name
// CommentNode : #comment
// DoctypeNode : the name of the document type
//
func NodeName(s *Selection) string {
if s.Length() == 0 {
return ""
}
return nodeName(s.Get(0))
}
// nodeName returns the node name of the given html node.
// See NodeName for additional details on behaviour.
func nodeName(node *html.Node) string {
if node == nil {
return ""
}
switch node.Type {
switch n := s.Get(0); n.Type {
case html.ElementNode, html.DoctypeNode:
return node.Data
return n.Data
default:
if int(node.Type) < len(nodeNames) {
return nodeNames[node.Type]
if n.Type >= 0 && int(n.Type) < len(nodeNames) {
return nodeNames[n.Type]
}
return ""
}
}
// TextOptions controls the behaviour of the Text function.
type TextOptions struct {
// Separator is inserted between the contents of the text nodes that are
// included in the result. It defaults to the empty string, which
// concatenates the text nodes without any separator.
Separator string
// Trim, when true, removes leading and trailing whitespace from the
// content of each text node and omits text nodes that are empty once
// trimmed. This is useful to discard the insignificant whitespace that
// comes from the indentation of the source HTML.
Trim bool
// Keep, when non-nil, is called for each text node encountered while
// traversing the selection. The content of the text node is included in
// the result only if Keep returns true. It can be used, for example, to
// drop the text of <script> and <style> elements by inspecting the node's
// parent. When Keep is nil, every text node is included (subject to Trim).
Keep func(node *html.Node) bool
}
// Text returns the combined text contents of the nodes in the selection,
// including their descendants, in document order. It is a package-level
// function - and not a method on the Selection, because it is not part of the
// jQuery API - that offers control over how the text of distinct text nodes is
// joined and which text nodes are included, in the same spirit as Python's
// BeautifulSoup get_text.
//
// With a nil opts (or a zero-value TextOptions), Text behaves like calling the
// Selection.Text method on the selection. Setting TextOptions.Separator inserts
// a separator between the text nodes, Trim strips the surrounding whitespace of
// each text node (dropping the ones that become empty), and Keep filters which
// text nodes contribute to the result.
func Text(s *Selection, opts *TextOptions) string {
if opts == nil {
opts = &TextOptions{}
}
var parts []string
var collect func(*html.Node)
collect = func(n *html.Node) {
if n.Type == html.TextNode {
if opts.Keep != nil && !opts.Keep(n) {
return
}
text := n.Data
if opts.Trim {
text = strings.TrimSpace(text)
if text == "" {
return
}
}
parts = append(parts, text)
return
}
for c := n.FirstChild; c != nil; c = c.NextSibling {
collect(c)
}
}
for _, n := range s.Nodes {
collect(n)
}
return strings.Join(parts, opts.Separator)
}
// Render renders the HTML of the first item in the selection and writes it to
// the writer. It behaves the same as OuterHtml but writes to w instead of
// returning the string.
func Render(w io.Writer, s *Selection) error {
if s.Length() == 0 {
return nil
}
n := s.Get(0)
return html.Render(w, n)
}
// OuterHtml returns the outer HTML rendering of the first item in
// the selection - that is, the HTML including the first element's
// tag and attributes.
//
// Unlike Html, this is a function and not a method on the Selection,
// Unlike InnerHtml, this is a function and not a method on the Selection,
// because this is not a jQuery method (in javascript-land, this is
// a property provided by the DOM).
func OuterHtml(s *Selection) (string, error) {
var builder strings.Builder
if err := Render(&builder, s); err != nil {
var buf bytes.Buffer
if s.Length() == 0 {
return "", nil
}
n := s.Get(0)
if err := html.Render(&buf, n); err != nil {
return "", err
}
return builder.String(), nil
return buf.String(), nil
}
// Loop through all container nodes to search for the target node.
@@ -209,7 +183,7 @@ func appendWithoutDuplicates(target []*html.Node, nodes []*html.Node, targetSet
// if a targetSet is passed, then assume it is reliable, otherwise create one
// and initialize it with the current target contents.
if targetSet == nil {
targetSet = make(map[*html.Node]bool, len(target)+len(nodes))
targetSet = make(map[*html.Node]bool, len(target))
for _, n := range target {
targetSet[n] = true
}
+85 -106
View File
@@ -9,6 +9,91 @@ import (
"golang.org/x/net/html"
)
var invalidPathNodes = []struct {
in string
path []int
}{
{"<a>", []int{0, 1, 2}},
{"<html><head><meta><title></title></head><body><div><p></p><a></a><span></span></div></body></html>", []int{0, 0, 1, 2, 0}},
{"<html><head><meta><title></title></head><body><div><p></p><a></a><span></span></div></body></html>", []int{1}},
{"<html><head><meta><title></title></head><body><div><p></p><a></a><span></span></div></body></html>", []int{1, 2}},
{"<html><head><meta><title></title></head><body><div><p></p><a></a><span></span></div></body></html>", []int{1, 2, 10}},
}
var validPathNodes = []struct {
in string
el string
path []int
}{
{"<a>", "a", []int{0, 0, 1, 0}}, // root html body(1) a
{"<html><head><meta></head><body></body></html>", "meta", []int{0, 0, 0, 0}}, // root html head meta
{"<html><head><meta><title></title></head><body></body></html>", "title", []int{0, 0, 0, 1}}, // root html head title
{"<html><head><meta><title></title></head><body><div><p></p></div></body></html>", "div", []int{0, 0, 1, 0}}, // root html body(1) div
{"<html><head><meta><title></title></head><body><div><p></p></div></body></html>", "p", []int{0, 0, 1, 0, 0}}, // root html body(1) div p
{"<html><head><meta><title></title></head><body><div><p></p><a></a><span></span></div></body></html>", "a", []int{0, 0, 1, 0, 1}}, // root html body(1) div a(1)
{"<html><head><meta><title></title></head><body><div><p></p><a></a><span></span></div></body></html>", "span", []int{0, 0, 1, 0, 2}}, // root html body(1) div span(2)
}
func TestPathForNode(t *testing.T) {
for i, c := range validPathNodes {
doc, err := NewDocumentFromReader(strings.NewReader(c.in))
if err != nil {
t.Errorf("%d: failed to parse: %v", i, err)
continue
}
var n *html.Node
if sel := doc.Find(c.el); sel.Length() > 0 {
n = sel.Get(0)
}
got := PathForNode(n)
if !reflect.DeepEqual(c.path, got) {
h, _ := OuterHtml(doc.Selection)
t.Errorf("%d: want %v, got %v (html: %s)", i, c.path, got, h)
}
}
// test a nil node
if got := PathForNode(nil); got != nil {
t.Errorf("want nil for nil node, got %v", got)
}
}
func TestNodeAtPath(t *testing.T) {
// valid cases
for i, c := range validPathNodes {
n, err := html.Parse(strings.NewReader(c.in))
if err != nil {
t.Errorf("%d: failed to parse: %v", i, err)
continue
}
nn := NodeAtPath(c.path, n)
if nn.Data != c.el {
t.Errorf("%d: want element %s, got %s (%v)", i, c.el, nn.Data, nn)
}
}
// invalid cases
for i, c := range invalidPathNodes {
n, err := html.Parse(strings.NewReader(c.in))
if err != nil {
t.Errorf("%d: failed to parse: %v", i, err)
continue
}
if got := NodeAtPath(c.path, n); got != nil {
t.Errorf("%d: want nil, got %v", i, got)
}
}
// test a nil node
if got := NodeAtPath([]int{1, 2, 3}, nil); got != nil {
t.Errorf("want nil for nil node, got %v", got)
}
}
var allNodes = `<!doctype html>
<html>
<head>
@@ -81,112 +166,6 @@ func TestNodeNameMultiSel(t *testing.T) {
}
}
var textNodes = `<!doctype html>
<html>
<body>
<div id="content">
<h1> Hello </h1>
<p>world</p>
<!-- a comment -->
<script>var ignored = 1;</script>
</div>
</body>
</html>`
func TestText_NilOptions(t *testing.T) {
doc, err := NewDocumentFromReader(strings.NewReader(textNodes))
if err != nil {
t.Fatal(err)
}
sel := doc.Find("#content")
// A nil options value must behave exactly like the Text method.
if got, want := Text(sel, nil), sel.Text(); got != want {
t.Errorf("nil options: want %q, got %q", want, got)
}
}
func TestText_SeparatorAndTrim(t *testing.T) {
doc, err := NewDocumentFromReader(strings.NewReader(textNodes))
if err != nil {
t.Fatal(err)
}
// Trim drops the whitespace-only text nodes coming from indentation, and
// Separator joins the remaining fragments. The <script> text is included
// because no Keep filter is provided.
got := Text(doc.Find("#content"), &TextOptions{Separator: "|", Trim: true})
want := "Hello|world|var ignored = 1;"
if got != want {
t.Errorf("want %q, got %q", want, got)
}
}
func TestText_Keep(t *testing.T) {
doc, err := NewDocumentFromReader(strings.NewReader(textNodes))
if err != nil {
t.Fatal(err)
}
// Keep skips the text of <script> (and <style>) elements.
keep := func(n *html.Node) bool {
if n.Parent != nil && n.Parent.Type == html.ElementNode {
switch n.Parent.Data {
case "script", "style":
return false
}
}
return true
}
got := Text(doc.Find("#content"), &TextOptions{Separator: " ", Trim: true, Keep: keep})
want := "Hello world"
if got != want {
t.Errorf("want %q, got %q", want, got)
}
}
func TestText_MultiSelection(t *testing.T) {
doc, err := NewDocumentFromReader(strings.NewReader(textNodes))
if err != nil {
t.Fatal(err)
}
got := Text(doc.Find("h1, p"), &TextOptions{Separator: ",", Trim: true})
want := "Hello,world"
if got != want {
t.Errorf("want %q, got %q", want, got)
}
}
func TestText_EmptySelection(t *testing.T) {
doc, err := NewDocumentFromReader(strings.NewReader(textNodes))
if err != nil {
t.Fatal(err)
}
// An empty selection has no text nodes to walk, so the result is empty
// and the call must not panic.
got := Text(doc.Find(".no-such-thing"), &TextOptions{Separator: " ", Trim: true})
if got != "" {
t.Errorf("want empty string, got %q", got)
}
}
func TestText_KeepFiltersAll(t *testing.T) {
doc, err := NewDocumentFromReader(strings.NewReader(textNodes))
if err != nil {
t.Fatal(err)
}
// When Keep rejects every text node, nothing is joined and the result is
// empty rather than a string of separators.
keep := func(n *html.Node) bool { return false }
got := Text(doc.Find("#content"), &TextOptions{Separator: " ", Trim: true, Keep: keep})
if got != "" {
t.Errorf("want empty string, got %q", got)
}
}
func TestOuterHtml(t *testing.T) {
doc, err := NewDocumentFromReader(strings.NewReader(allNodes))
if err != nil {