Description
DHT fully implements BitTorrent DHT Protocol. It is also a fast spider that crawling the BitTorrent DHT network. You can use it to crawl torrent files info.
DHT alternatives and similar packages
Based on the "Distributed Systems" category.
Alternatively, view DHT alternatives based on common mentions on social networks and blogs.
-
-
-
Nomad
Nomad is an easy-to-use, flexible, and performant workload orchestrator that can deploy a mix of microservice, batch, containerized, and non-containerized applications. Nomad is easy to operate and scale and has native Consul and Vault integrations. -
go-zero
DISCONTINUED. go-zero is a web and rpc framework written in Go. It's born to ensure the stability of the busy sites with resilient design. Builtin goctl greatly improves the development productivity. [Moved to: https://github.com/zeromicro/go-zero] -
-
-
rpcx
Best microservices framework in Go, like alibaba Dubbo, but with more features, Scale easily. Try it. Test it. If you feel it's better, use it! ๐๐๐ฏ๐ๆ๐๐ฎ๐๐๐จ, ๐๐จ๐ฅ๐๐ง๐ ๆ๐ซ๐ฉ๐๐ฑ! build for cloud! -
Encore
Open Source Development Platform for building robust type-safe distributed systems with declarative infrastructure -
-
-
-
-
-
-
-
-
-
-
gleam
Fast, efficient, and scalable distributed map/reduce system, DAG execution, in memory or on disk, written in pure Go, runs standalone or distributedly. -
glow
Glow is an easy-to-use distributed computation system written in Go, similar to Hadoop Map Reduce, Spark, Flink, Storm, etc. I am also working on another similar pure Go system, https://github.com/chrislusf/gleam , which is more flexible and more performant. -
Olric
Distributed, in-memory key/value store and cache. It can be used as an embedded Go library and a language-independent service. -
-
Dragonfly
DISCONTINUED. Dragonfly is an open source P2P-based file distribution and image acceleration system. It is hosted by the Cloud Native Computing Foundation (CNCF) as an Incubating Level Project. [Moved to: https://github.com/dragonflyoss/dragonfly] -
-
-
go-doudou
go-doudou๏ผdoudou pronounce /dษudษu/๏ผis OpenAPI 3.0 (for REST) spec and Protobuf v3 (for grpc) based lightweight microservice framework. It supports monolith service application as well. -
-
-
-
-
-
-
-
-
resgate
A Realtime API Gateway used with NATS to build REST, real time, and RPC APIs, where all your clients are synchronized seamlessly. -
-
-
go-sundheit
A library built to provide support for defining service health for golang services. It allows you to register async health checks for your dependencies and the service itself, provides a health endpoint that exposes their status, and health metrics. -
-
-
-
-
-
Maestro
Take control of your data, connect with anything, and expose it anywhere through protocols such as HTTP, GraphQL, and gRPC. -
-
-
celeriac
Golang client library for adding support for interacting and monitoring Celery workers, tasks and events. -
drmaa
Compute cluster (HPC) job submission library for Go (#golang) based on the open DRMAA standard. -
-
InfluxDB high-performance time series database
Do you think we are missing an alternative of DHT or a related project?
README
See the video on the Youtube.
Introduction
DHT implements the bittorrent DHT protocol in Go. Now it includes:
It contains two modes, the standard mode and the crawling mode. The standard mode follows the BEPs, and you can use it as a standard dht server. The crawling mode aims to crawl as more metadata info as possiple. It doesn't follow the standard BEPs protocol. With the crawling mode, you can build another BTDigg.
bthub.io is a BT search engine based on the crawling mode.
Installation
go get github.com/shiyanhui/dht
Example
Below is a simple spider. You can move here to see more samples.
import (
"fmt"
"github.com/shiyanhui/dht"
)
func main() {
downloader := dht.NewWire(65535)
go func() {
// once we got the request result
for resp := range downloader.Response() {
fmt.Println(resp.InfoHash, resp.MetadataInfo)
}
}()
go downloader.Run()
config := dht.NewCrawlConfig()
config.OnAnnouncePeer = func(infoHash, ip string, port int) {
// request to download the metadata info
downloader.Request([]byte(infoHash), ip, port)
}
d := dht.New(config)
d.Run()
}
Download
You can download the demo compiled binary file here.
Note
- The default crawl mode configure costs about 300M RAM. Set MaxNodes and BlackListMaxSize to fit yourself.
- Now it cant't run in LAN because of NAT.
TODO
- [ ] NAT Traversal.
- [ ] Implements the full BEP-3.
- [ ] Optimization.
FAQ
Why it is slow compared to other spiders ?
Well, maybe there are several reasons.
- DHT aims to implements the standard BitTorrent DHT protocol, not born for crawling the DHT network.
- NAT Traversal issue. You run the crawler in a local network.
- It will block ip which looks like bad and a good ip may be mis-judged.
License
MIT, read more here
*Note that all licence references and agreements mentioned in the DHT README section above
are relevant to that project's source code only.