UbiCrawler

UbiCrawler: a scalable fully distributed web crawler. We report our experience in implementing UbiCrawler, a scalable distributed Web crawler, using the Java programming language. The main features of UbiCrawler are platform independence, linear scalability, graceful degradation in the presence of faults, a very effective assignment function (based on consistent hashing) for partitioning the domain to crawl, and more in general the complete decentralization of every task. The necessity of handling very large sets of data has highlighted some limitations of the Java APIs, which prompted the authors to partially reimplement them.

This software is also peer reviewed by journal TOMS.


References in zbMATH (referenced in 33 articles )

Showing results 1 to 20 of 33.
Sorted by year (citations)

1 2 next

  1. Shen, Zhao-Li; Huang, Ting-Zhu; Carpentieri, Bruno; Wen, Chun; Gu, Xian-Ming; Tan, Xue-Yuan: Off-diagonal low-rank preconditioner for difficult PageRank problems (2019)
  2. Arz, Julian; Fischer, Johannes: Lempel-Ziv-78 compressed string dictionaries (2018)
  3. Broß, Jan; Gog, Simon; Hauck, Matthias; Paradies, Marcus: Fast construction of compressed web graphs (2017)
  4. Kanda, Shunsuke; Morita, Kazuhiro; Fuketa, Masao: Practical implementation of space-efficient dynamic keyword dictionaries (2017)
  5. Mania, Horia; Pan, Xinghao; Papailiopoulos, Dimitris; Recht, Benjamin; Ramchandran, Kannan; Jordan, Michael I.: Perturbed iterate analysis for asynchronous stochastic optimization (2017)
  6. Salim Khalil, Mohamed Fakir: RCrawler: An R package for parallel web crawling and scraping (2017) not zbMATH
  7. Shen, Zhao-Li; Huang, Ting-Zhu; Carpentieri, Bruno; Gu, Xian-Ming; Wen, Chun: An efficient elimination strategy for solving PageRank problems (2017)
  8. Fischer, Johannes; Peters, Daniel: GLOUDS: representing tree-like graphs (2016)
  9. Huynh, The Dang; Mathieu, Fabien; Viennot, Laurent: LiveRank: how to refresh old datasets (2016)
  10. Slota, George M.; Madduri, Kamesh; Rajamanickam, Sivasankaran: Complex network partitioning using label propagation (2016)
  11. Brandes, Ulrik; Hamann, Michael; Strasser, Ben; Wagner, Dorothea: Fast quasi-threshold editing (2015)
  12. Rossi, Ryan A.; Gleich, David F.; Gebremedhin, Assefaw H.: Parallel maximum clique algorithms with applications to network analysis (2015)
  13. Grabowski, Szymon; Bieniecki, Wojciech: Tight and simple web graph compression for forward and reverse neighbor queries (2014)
  14. Grossi, Roberto; Ottaviano, Giuseppe: Fast compressed tries through path decompositions (2014)
  15. Kolda, Tamara G.; Pinar, Ali; Plantenga, Todd; Seshadhri, C.; Task, Christine: Counting triangles in massive graphs with MapReduce (2014)
  16. Çatalyürek, Ümit V.; Kaya, Kamer; Langguth, Johannes; Uçar, Bora: A partitioning-based divisive clustering technique for maximizing the modularity (2013)
  17. Crescenzi, Pilu; Grossi, Roberto; Habib, Michel; Lanzi, Leonardo; Marino, Andrea: On computing the diameter of real-world undirected graphs (2013)
  18. Ferragina, Paolo; Nitto, Igor; Venturini, Rossano: On the bit-complexity of Lempel-Ziv compression (2013)
  19. Ferragina, Paolo; Gagie, Travis; Manzini, Giovanni: Lightweight data indexing and compression in external memory (2012)
  20. He, Jing; Zhang, Yanchun; Huang, Guangyan; Shi, Yong; Cao, Jie: Distributed data possession checking for securing multiple replicas in geographically-dispersed clouds (2012)

1 2 next