What design pattern should I use for my (small) web crawler project?

I am a hobbyist programmer and don't really know much about software architecture or design patterns. The only pattern I am familiar with is the MVC. After having to create more than 2 classes, the code got messy and I am starting to appreciate the value of clean code practices, debugging, testing, handling exceptions properly, etc. I am now looking for a suitable design pattern for my project.

The main components of project are

1- Utility class for connecting to URLS, getting their HTML content, and parsing the content.

2- A runnable class that does the crawling part (polls an unvisited URL from the tasks queue, visits it and collects all hyperlinks in it and adds them to the tasks queue).

3- A GUI class where the user can specify the number of crawler threads, the maximum depth (number of pages to be crawled), the time limit, etc.

What design pattern is best suited for such a project? Should I add more classes?

submitted by /u/bovine_scatology
[link] [comments]

from Software Development – methodologies, techniques, and tools. Covering Agile, RUP, Waterfall + more! https://ift.tt/3mtjX1J

Leave a comment

Design a site like this with WordPress.com
Get started
search previous next tag category expand menu location phone mail time cart zoom edit close