UbiCrawler

UbiCrawler: a scalable fully distributed web crawler. We report our experience in implementing UbiCrawler, a scalable distributed Web crawler, using the Java programming language. The main features of UbiCrawler are platform independence, linear scalability, graceful degradation in the presence of faults, a very effective assignment function (based on consistent hashing) for partitioning the domain to crawl, and more in general the complete decentralization of every task. The necessity of handling very large sets of data has highlighted some limitations of the Java APIs, which prompted the authors to partially reimplement them.

This software is also peer reviewed by journal TOMS.


References in zbMATH (referenced in 27 articles )

Showing results 1 to 20 of 27.
Sorted by year (citations)

1 2 next

  1. Arz, Julian; Fischer, Johannes: Lempel-Ziv-78 compressed string dictionaries (2018)
  2. Mania, Horia; Pan, Xinghao; Papailiopoulos, Dimitris; Recht, Benjamin; Ramchandran, Kannan; Jordan, Michael I.: Perturbed iterate analysis for asynchronous stochastic optimization (2017)
  3. Salim Khalil, Mohamed Fakir: RCrawler: An R package for parallel web crawling and scraping (2017)
  4. Fischer, Johannes; Peters, Daniel: GLOUDS: representing tree-like graphs (2016)
  5. Slota, George M.; Madduri, Kamesh; Rajamanickam, Sivasankaran: Complex network partitioning using label propagation (2016)
  6. Brandes, Ulrik; Hamann, Michael; Strasser, Ben; Wagner, Dorothea: Fast quasi-threshold editing (2015)
  7. Rossi, Ryan A.; Gleich, David F.; Gebremedhin, Assefaw H.: Parallel maximum clique algorithms with applications to network analysis (2015)
  8. Grabowski, Szymon; Bieniecki, Wojciech: Tight and simple web graph compression for forward and reverse neighbor queries (2014)
  9. Grossi, Roberto; Ottaviano, Giuseppe: Fast compressed tries through path decompositions (2014)
  10. Kolda, Tamara G.; Pinar, Ali; Plantenga, Todd; Seshadhri, C.; Task, Christine: Counting triangles in massive graphs with MapReduce (2014)
  11. Çatalyürek, Ümit V.; Kaya, Kamer; Langguth, Johannes; Uçar, Bora: A partitioning-based divisive clustering technique for maximizing the modularity (2013)
  12. Crescenzi, Pilu; Grossi, Roberto; Habib, Michel; Lanzi, Leonardo; Marino, Andrea: On computing the diameter of real-world undirected graphs (2013)
  13. Ferragina, Paolo; Nitto, Igor; Venturini, Rossano: On the bit-complexity of Lempel-Ziv compression (2013)
  14. Ferragina, Paolo; Gagie, Travis; Manzini, Giovanni: Lightweight data indexing and compression in external memory (2012)
  15. He, Jing; Zhang, Yanchun; Huang, Guangyan; Shi, Yong; Cao, Jie: Distributed data possession checking for securing multiple replicas in geographically-dispersed clouds (2012)
  16. Zhou, Yunkai: Practical acceleration for computing the HITS expertrank vectors (2012)
  17. Belazzougui, Djamal; Boldi, Paolo; Pagh, Rasmus; Vigna, Sebastiano: Theory and practice of monotone minimal perfect hashing (2011)
  18. Dayar, Tugrul; Noyan, Gökçe N.: Steady-state analysis of google-like stochastic matrices with block iterative methods (2011)
  19. Laura, Luigi; Santaroni, Federico: Computing strongly connected components in the streaming model (2011)
  20. Karande, Chinmay; Chellapilla, Kumar; Andersen, Reid: Speeding up algorithms on compressed web graphs (2009)

1 2 next