UbiCrawler

UbiCrawler: a scalable fully distributed web crawler. We report our experience in implementing UbiCrawler, a scalable distributed Web crawler, using the Java programming language. The main features of UbiCrawler are platform independence, linear scalability, graceful degradation in the presence of faults, a very effective assignment function (based on consistent hashing) for partitioning the domain to crawl, and more in general the complete decentralization of every task. The necessity of handling very large sets of data has highlighted some limitations of the Java APIs, which prompted the authors to partially reimplement them.

This software is also peer reviewed by journal TOMS.


References in zbMATH (referenced in 21 articles )

Showing results 1 to 20 of 21.
Sorted by year (citations)

1 2 next

  1. Fischer, Johannes; Peters, Daniel: GLOUDS: representing tree-like graphs (2016)
  2. Slota, George M.; Madduri, Kamesh; Rajamanickam, Sivasankaran: Complex network partitioning using label propagation (2016)
  3. Rossi, Ryan A.; Gleich, David F.; Gebremedhin, Assefaw H.: Parallel maximum clique algorithms with applications to network analysis (2015)
  4. Grabowski, Szymon; Bieniecki, Wojciech: Tight and simple web graph compression for forward and reverse neighbor queries (2014)
  5. Kolda, Tamara G.; Pinar, Ali; Plantenga, Todd; Seshadhri, C.; Task, Christine: Counting triangles in massive graphs with MapReduce (2014)
  6. Çatalyürek, Ümit V.; Kaya, Kamer; Langguth, Johannes; Uçar, Bora: A partitioning-based divisive clustering technique for maximizing the modularity (2013)
  7. Crescenzi, Pilu; Grossi, Roberto; Habib, Michel; Lanzi, Leonardo; Marino, Andrea: On computing the diameter of real-world undirected graphs (2013)
  8. Ferragina, Paolo; Nitto, Igor; Venturini, Rossano: On the bit-complexity of Lempel-Ziv compression (2013)
  9. Ferragina, Paolo; Gagie, Travis; Manzini, Giovanni: Lightweight data indexing and compression in external memory (2012)
  10. He, Jing; Zhang, Yanchun; Huang, Guangyan; Shi, Yong; Cao, Jie: Distributed data possession checking for securing multiple replicas in geographically-dispersed clouds (2012)
  11. Zhou, Yunkai: Practical acceleration for computing the HITS expertrank vectors (2012)
  12. Belazzougui, Djamal; Boldi, Paolo; Pagh, Rasmus; Vigna, Sebastiano: Theory and practice of monotone minimal perfect hashing (2011)
  13. Dayar, Tugrul; Noyan, Gökçe N.: Steady-state analysis of google-like stochastic matrices with block iterative methods (2011)
  14. Laura, Luigi; Santaroni, Federico: Computing strongly connected components in the streaming model (2011)
  15. Karande, Chinmay; Chellapilla, Kumar; Andersen, Reid: Speeding up algorithms on compressed web graphs (2009)
  16. Boldi, Paolo; Posenato, Roberto; Santini, Massimo; Vigna, Sebastiano: Traps and pitfalls of topic-biased PageRank (2008)
  17. Latapy, Matthieu: Main-memory triangle computations for very large (sparse (power-law)) graphs (2008)
  18. Nasri, Mitra; Shariati, Saeed; Azgomi, Mohammad Abdollahi: Performance modeling of a distributed web crawler using stochastic activity networks (2008)
  19. Constantine, Paul G.; Gleich, David F.: Using polynomial chaos to compute the influence of multiple random surfers in the PageRank model (2007)
  20. Liu, Zhen; Nain, Philippe: Optimization issues in web search engines (2006)

1 2 next