Golang caches downloaded modules including unpacked source code of versioned dependencies in $GOPATH/pkg/mod.
Naturally this cache grows over time. It can be cleaned up with the following command:

go clean -modcache

(go Documentation – go help clean)