Adds goquery.Text(s, *TextOptions), a package-level counterpart to the
Selection.Text method that gives control over how the text of distinct
text nodes is joined (Separator), whether each node is trimmed of
surrounding whitespace (Trim), and which text nodes are included (Keep).
This covers the common need to extract clean, readable text from a
document - for example joining fragments with a space and dropping the
text of script/style elements - without hand-rolling a node walk.
Passing a nil TextOptions keeps the behaviour identical to Selection.Text.
Includes tests and a runnable example.
Attr was a 4-line wrapper around getAttributeValue, which itself was
a 6-line wrapper around getAttributePtr.
This commit inlines getAttributeValue into Attr and
have AttrOr delegate to Attr. Drops a layer of indirection and one
helper function; both methods get shorter and the empty-selection
check lives in one place.
WrapHtml, WrapInnerHtml, and eachNodeHtml each open-coded the same
lookup-or-parse-and-store sequence against a map[string][]*html.Node
keyed by the context node's name. Extract that 4-line sequence into
a tiny cachedParseHtmlWithContext(cache, htmlStr, context) helper; each call
site becomes a linear, closure-free loop and the only duplicated bit (the cache
lookup) lives in one place.
winnow only ever reads sel.Nodes; the *Selection parameter was
incidental. Change its signature to winnow(nodes []*html.Node, ...)
and update its three callers (FilterMatcher, NotMatcher,
filterAndPush). filterAndPush no longer needs to fabricate a
throwaway Selection to satisfy the call.
While this change has o measurable runtime effect, it simplifies the code a bit,
by making winnow's signature clearer, and removes a now-useless line and its
associated comment in filterAndPush
parseHtml and parseHtmlWithContext duplicated the ParseFragment call,
the error check, and the panic message. parseHtml only differed by
passing a synthetic ElementNode as the context.
This commit simply delegates parseHtml to parseHtmlWithContext.
Move the "stop at first match" optimization out of HasMatcher and into
singleMatcher, where it can be reused elsewhere.
singleMatcher now implements MatchFirst, which calls the underlying
Matcher's MatchFirst if it provides one (cascadia-compiled matchers all
do), otherwise falls back to MatchAll and returns its first element.
singleMatcher.MatchAll wraps MatchFirst's result in a single-item slice
to respect the Matcher interface, so external SingleMatcher() users get
the slice signature while internal callers can avoid that allocation.
HasMatcher now constructs a singleMatcher directly to call MatchFirst
without a type assertion, removing the duplicated probe branch.
A Selection's Nodes field is exported, so callers can build a selection that
holds the same node more than once. Add a regression test asserting that
Children, ChildrenFiltered and Contents deduplicate the shared child set
rather than returning it once per duplicate source node.
Empty() collected the removed children into a nil slice grown via append,
forcing a series of grow-and-double reallocations proportional to the number
of children.
This commit walks the children once up front to count them, then allocate the
result slice at its final capacity before the removal loop.
The count pass is a cheap, branch-predicted pointer walk that replaces several
slice reallocations with a single allocation.
benchstat (-benchtime=2000x, count=10):
| old | new |
| allocs | allocs vs base |
Empty10-8 | 3.000 | 2.000 -33.33% |
Empty100-8 | 6.000 | 2.000 -66.67% |
Empty1000-8 | 9.000 | 2.000 -77.78% |
B/op: -47% / -56% / -53%
sec/op: Empty10 -53%, Empty100 -58%, Empty1000 ~ (noise)
HasMatcher only needs to know whether a child subtree contains a match,
but it tested len(m.MatchAll(c)) > 0 after wrapping the matcher with
SingleMatcher. SingleMatcher.MatchAll computes MatchFirst internally and
then wraps the result in a throwaway []*html.Node{node}, allocating a
one-element slice per matching child subtree just to check a boolean.
When the matcher exposes MatchFirst, probe it directly and test the returned
node against nil, avoiding the allocation entirely. This is the same trick
already used in SingleMatcher.MatchAll, just applied at the HasMatcher call
site so the intermediate slice is never built in the first place. The
SingleMatcher path is kept as a fallback for matchers that don't implement
MatchFirst.
benchstat (count=10):
| before | after |
| sec/op | sec/op vs base|
Has-8 | 2.675µ ± 17% | 2.208µ ± 17% -17.47%|
| before | after |
| B/op | B/op vs base |
Has-8 | 360.0 ± 0% | 240.0 ± 0% -33.33% |
| before | after |
|allocs/op | allocs/op vs base |
Has-8 | 20.00 ± 0% | 6.000 ± 0% -70.00% |
getParentNodes ran through mapNodes with a callback that allocated a
throwaway one-element []*html.Node{n.Parent} for every source node. Since
sibling nodes share the same parent, almost all of those slices were
immediately discarded as duplicates.
This commit collects the parents directly in a single loop, deduplicating via a
set as we go, so no per-node slice is allocated.
benchstat (count=10, all p=0.000):
| before | after |
| sec/op | sec/op vs base |
Parent-8 | 38.54µ ± 21%| 17.49µ ± 14% -54.63% |
ParentFiltered-8 | 46.57µ ± 13%| 20.91µ ± 24% -55.09% |
| before | after |
| B/op | B/op vs base |
Parent-8 | 13.24Ki ± 0%| 10.28Ki ± 0% -22.36% |
ParentFiltered-8 | 13.49Ki ± 0%| 10.58Ki ± 0% -21.60% |
| before | after |
| allocs/op | allocs/op vs base |
Parent-8 | 384.0 ± 0% | 10.00 ± 0% -97.40% |
ParentFiltered-8 | 392.0 ± 0% | 19.00 ± 0% -95.15% |
For the sibling types that collect every match (siblingAll,
siblingAllIncludingNonElements, siblingPrevAll, siblingNextAll), do a
cheap first-pass pointer walk to count the matches, then make the result
slice with that exact capacity. This removes the repeated append-driven
slice growth that dominated allocations on these hot traversal paths.
The Until cases are left untouched because counting would require running
the user predicate twice, and the single-result Next/Prev cases have
nothing to presize.
This drives Children/Contents/Siblings/Next*/Prev* through fewer
allocations:
allocs/op vs base B/op vs base
Siblings -57.23% -9.23%
SiblingsFiltered -54.49% -8.94%
NextAll -59.82% -10.04%
NextAllFiltered -55.83% -9.65%
PrevAll -55.32% -10.66%
PrevAllFiltered -52.53% -10.40%
ChildrenFiltered -15.38% -6.45%
Contents -7.69% -1.73%
geomean -25.07% -3.88%
The Until and single Next/Prev benchmarks are unchanged (bit-identical
allocs/op), confirming the count pre-pass only touches the collect-all
paths.
The quite verbose comment on top of the change is there so that future
generations won't waste time wondering why this weird loop is here.
BenchmarkAddNodesBig (DocW().Find("li") duplicated to ~1500 nodes,
then AddNodes onto an empty selection), -count=20:
sec/op B/op allocs/op
before 214.3µs ±12% 27.58Ki ±0% 24
after 104.8µs ±24% 45.24Ki ±0% 16
-51% +64% -33%
The B/op increase is the cost of presizing the map to its final
capacity in one shot rather than letting the runtime grow it
through smaller bucket arrays (which are freed but counted in
total bytes allocated). Peak resident memory is comparable.
Has(selector) and HasMatcher(m) used to call
s.document.Find(selector)/FindMatcher(m), materializing every matching
descendant in the entire document, then iterate s.Nodes x matchedNodes x
treeDepth via nodeContains to keep the elements that contain one.
This commit probes each selection node's subtree directly with SingleMatcher, which
short-circuits on the first descendant match.
BenchmarkHas (DocW().Find("h2").Has(".editsection")), -count=10:
sec/op B/op
before 61.97µs 744 B
after 3.33µs 360 B
-94.6% -51.6%
Three internal helpers build a map[*html.Node]bool from a known set of
nodes (or use one as a dedup accumulator over a known number of input
nodes) but allocate the map without a size hint. They then incur
grow-and-double rehashing as entries are inserted.
Pass len(nodes) as the size hint at make time:
- traversal.go mapNodes: dedup map shared by appendWithoutDuplicates
across every per-node result. mapNodes is the engine behind most
traversal helpers (Parents, Children, Next/Prev, Siblings, Find*,
Contents, ...), so the impact is broad.
- traversal.go ClosestNodes: target-node set built from nodes...
- filter.go winnowNodes: large-N path's lookup set built from nodes...
Double-digit performance gains across ~all benchmarks.
mapNodes always allocated a map[*html.Node]bool for deduplication, even
when called with a single source node where duplicates cannot arise.
Short-circuit to return the callback result directly when len(nodes)==1.
Also add a special case for when there are zero nodes.
This benefits all traversal methods (Find, Children, Parent, Next, Prev,
etc.) when operating on a single-node selection, which is the common
case after First(), Eq(), or directly on a Document.
name old ns/op new ns/op delta
Find-8 23200 14200 -38.8%
name old B/op new B/op delta
Find-8 4632 1496 -67.7%
name old allocs new allocs delta
Find-8 27 13 -51.9%
Replace the recursive closure in Selection.Text with a helper method call.
This keeps behavior unchanged while reducing runtime overhead in the hot path.
Benchmark (BenchmarkText, benchstat, n=12):
- sec/op: 2.133us -> 1.692us (-20.70%, p=0.000)
- B/op: 504 -> 504 (no change)
- allocs/op: 6 -> 6 (no change)
HasClass normalized every matching element's class attribute with
classTrimReplacer before checking for the target token, even when the
raw attribute value could not possibly contain that class at all.
Add a cheap strings.Contains(attr.Val, rawClass) pre-check and only run
the replacer on plausible matches. This preserves behavior for class
attributes containing tabs/newlines while skipping most of the work on
misses.
BenchmarkHasClass on arm64 over 10 runs:
name old time/op new time/op delta
HasClass-8 9.809µs 2.660µs -72.89% (p=0.000)
IsMatcher handled multi-node selections by calling m.Filter(s.Nodes) and
checking whether the returned slice was non-empty, forcing the matcher to walk
the whole selection and materialize all matches even though the API only needs
an existence check.
This commit replaces this with a simple loop over s.Nodes that calls m.Match(n)
and returns true on the first match. This also makes the single-node special
case unnecessary.
Benchmark results on arm64 over 10 runs:
name old time/op new time/op delta
Is-8 4.67µs 1.32µs -71.7%
IsPositional-8 21.44µs 0.82µs -96.2%
name old B/op new B/op delta
Is-8 96 80 -16.7%
IsPositional-8 1144 184 -83.9%
name old allocs new allocs delta
Is-8 5 4 -20.0%
IsPositional-8 11 7 -36.4%
HasNodes routed through FilterFunction -> winnowFunction -> grep, which
allocated a single-node Selection and a closure capture per source element,
just to call sel.Contains(n) which is itself a one-iteration sliceContains over
the wrapped node. This commit inlines a direct double loop over s.Nodes x nodes
calling nodeContains, and push the result via pushStack.
name old allocs/op new allocs/op delta
Has-8 55.00 ± 0% 24.00 ± 0% -56.36% (p=0.000)
HasNodes-8 752.00 ± 0% 4.00 ± 0% -99.47% (p=0.000)
HasSelection-8 752.00 ± 0% 3.00 ± 0% -99.60% (p=0.000)
In the same spirit as #545, replaces costly grep() with a loop.
While escape analysis managed to stack-allocate the per-node
Selection through grep's inlining, the closure capture itself (nodes and keep)
was still heap-allocated. Local benchmarks are showing a 30% allocs/op
reduction, and a ~10-20% reduction in sec/op.
getClassesSlice used rxClassTrim.ReplaceAllString followed by
strings.Split, which allocated twice and produced empty strings from
leading/trailing spaces. strings.Fields handles all Unicode whitespace
in a single pass with no empty entries.
name old ns/op new ns/op delta
AddClass-8 176700 107500 -39.2%
RemoveClass-8 142900 64600 -54.8%
name old B/op new B/op delta
AddClass-8 12544 7024 -44.0%
RemoveClass-8 6503 1777 -72.7%
name old allocs new allocs delta
AddClass-8 697 374 -46.3%
RemoveClass-8 697 374 -46.3%
The regex [\t\r\n] only does single-byte substitution, which is exactly
what strings.Replacer's byteReplacer is optimized for. This removes the
regexp dependency from property.go and drops the redundant ContainsAny
fast-path guard in HasClass, since byteReplacer already returns the
original string unchanged (no allocation) when nothing matches.
Benchmark shows ~18% improvement on HasClass with 0 allocs either way.
getClassesAndAttr unconditionally ran rxClassTrim.ReplaceAllString to
normalize tabs, carriage returns, and newlines in class attribute values.
The vast majority of HTML class attributes contain none of these, so add
a strings.ContainsAny fast path that falls back to simple concatenation.
This benefits all callers: HasClass, AddClass, RemoveClass, ToggleClass. Here's
a benchmark of HasClass to illustrate the gains:
name old ns/op new ns/op delta
HasClass-8 222000 45300 -79.6%
name old B/op new B/op delta
HasClass-8 19152 4280 -77.6%
name old allocs new allocs delta
HasClass-8 1300 325 -75.0%
These methods were building a full filtered Selection (via
FilterFunction/FilterSelection/FilterNodes -> winnow -> grep) only to
check Length() > 0. This commit replaces with early-return loops that stop at
the first match, significantly speeding them up:
name old ns/op new ns/op delta
IsFunction-8 1530 3.9 -99.7%
IsSelection-8 1890 19.9 -98.9%
IsNodes-8 2016 19.9 -99.0%
name old B/op new B/op delta
IsFunction-8 784 0 -100.0%
IsSelection-8 72 0 -100.0%
IsNodes-8 72 0 -100.0%
name old allocs new allocs delta
IsFunction-8 28 0 -100.0%
IsSelection-8 3 0 -100.0%
IsNodes-8 3 0 -100.0%
Index() was creating a Selection, calling PrevAll() which traversed
through getSiblingNodes -> mapNodes (allocating a dedup map and
intermediate slices), then calling Length() on the result.
This commit replaces this with a direct PrevSibling pointer walk that counts
element nodes, bringing the number of allocations down to zero instead of a
couple of hundreds on local benchmarks
The winnow() "Not" path routed through grep(), which called
newSingleSelection() for every node, allocating a Selection and
a 1-element []*html.Node slice on the heap per node, only to
immediately unwrap it via s.Get(0) for m.Match().
This commit call m.Match(n) directly and pre-allocate the result slice.
It also adds BenchmarkNotMatcher to isolate the winnow path from
compileMatcher overhead.
Here are my local benchmark results, for BenchmarkNotMatcher with 373 <li>
nodes, and a pre-compiled matcher:
old ns/op new ns/op delta
22750 19350 ~-15%
old B/op new B/op delta
9384 3120 -66.7%
old allocs new allocs delta
11 2 -81.8%
- We don't need to use the iter.Seq2 type; instead, we can use func(yield func(int, *Selection) bool). In fact, the underlying type of iter.Seq2 is func(yield func(int, *Selection) bool), so we don't need to upgrade the go.mod version.
- For for-range testing in version 1.23, we can use build tags to compile only for versions above 1.23.
@@ -22,7 +22,15 @@ Syntax-wise, it is as close as possible to jQuery, with the same function names
## Installation
Please note that starting with version `v1.9.0` of goquery, Go 1.18+ is required due to the use of generics. For previous goquery versions, a Go version of 1.1+ was required because of the `net/html` dependency. Ongoing goquery development is tested on the latest 2 versions of Go.
Required Go version:
* Starting with version `v1.12.0` of goquery, Go 1.25+ is required due to its dependencies.
* Starting with version `v1.11.0` of goquery, Go 1.24+ is required due to its dependencies.
* Starting with version `v1.10.0` of goquery, Go 1.23+ is required due to the use of function-based iterators.
* For `v1.9.0` of goquery, Go 1.18+ is required due to the use of generics.
* For previous goquery versions, a Go version of 1.1+ was required because of the `net/html` dependency.
Ongoing goquery development is tested on the latest 2 versions of Go.
$ go get github.com/PuerkitoBio/goquery
@@ -40,6 +48,13 @@ Please note that starting with version `v1.9.0` of goquery, Go 1.18+ is required
**Note that goquery's API is now stable, and will not break.**
***2026-08-27 (v1.13.0)** : Performance improvements (thanks [@jvoisin][jvs]), add top-level `Text` function with options similar to BeautifulSoup's `get_text` (thanks [@ChrisJr404][chrisjr]), update `go.mod` dependencies, add go1.27 to the test matrix.
***2026-03-15 (v1.12.0)** : Update `go.mod` dependencies, add go1.26 to the test matrix, **goquery now requires Go version 1.25+**.
***2025-11-16 (v1.11.0)** : Update `go.mod` dependencies, add go1.25 to the test matrix, **goquery now requires Go version 1.24+**.
***2025-04-11 (v1.10.3)** : Update `go.mod` dependencies, small optimization (thanks [@myxzlpltk](https://github.com/myxzlpltk)).
***2025-02-13 (v1.10.2)** : Update `go.mod` dependencies, add go1.24 to the test matrix.
***2024-09-06 (v1.10.0)** : Add `EachIter` which provides an iterator that can be used in `for..range` loops on the `*Selection` object. **goquery now requires Go version 1.23+** (thanks [@amikai](https://github.com/amikai)).
***2024-02-29 (v1.9.1)** : Improve allocation and performance of the `Map` function and `Selection.Map` method, better document the cascadia differences (thanks [@jwilsson](https://github.com/jwilsson)).
@@ -202,3 +217,5 @@ The [BSD 3-Clause license][bsd], the same as the [Go language][golic]. Cascadia'
Blocking a user prevents them from interacting with repositories, such as opening or commenting on pull requests or issues. Learn more about blocking a user.
Martin Angers