« first day (21 days earlier)      last day (16 days later) » 

12:23 AM
1) I have to do a daily Web Scraping to a certain web page to get some specific data from it.
2) This will be done through a daemon or "JOB" scheduled to run everyday at 2am
3) This job will do the "Web Scraping" and will save the obtained data to our MySql Database (will save the data along with some custom calculations of this data, into a let's say table called "National_Taxes")
4) I have to create a Web Page that will have just twoo controls: a "day-picker" Calendar and the "Search" button
5) Basically, the web page will query the table "National_Taxes" returning the "Scraped" data along with the calculated data stored on that table, to the user interface.
I want to write the WEBAPP and JOB in just one php project.
Obviously: both will use the same Model layer, same entity object (e.g. they both use the same mapper to the "National_Taxes" table).

Refering to the "Web Scraping" demon (or job), I will use 4 classes:
1) Class D: The class that will do all the hard and dirty job for the "Web Scraping" using the library "Goutte".
2) Class C: This class calls Class D (hiding the dirty job done by "D") and that prepare an specialized objetc (just a simple array by now) of the retrieved data to be returned (to Class B), previously doing a validation of the retrieved data among other things.
3) Class B: The class that calls Class B to obtain the "validated" and "ready to store" data, doing some previous logic and doing some post logic (e.g. doing the custom calculations of the retrieved data), and that finally store the data to the table (calling the corresponding Mapper).
4) Class A: The class that represents the actual JOB.
So, How am I saving these classes in your MVC structure folders?
1) domain/Infrastructure/Scraper/Class_D
2) domain/Service/Class_C (it's an intermediate service, as it does the validation logic of the retrieved data, but dont use any mapper)
Inside the constructor the Class D will be created. Class D not need to be passed through the constructor as it will be the only implementation
3) domain/Service/Class_B (it's the actual service, as it does the business logic and use the mapper)
Inside the constructor the Class C will be created, and trough the constructor an instance of the Mapper class will be passed.
The mapper class is passed through the constructor as I can change the implementation to an Oracle class later.
4) domain/Infrastructure/Job/Class_A
He creates the db connection, create the mapper (passing through its constructor the db objetc), creates the service Class B (passing through its constructor the mapper object), and call the corresponding method of Class B to to the actual routine (passing it some parameters as the current day or the running job, etc).

Class C could be contained inside Class B, but I prefered to separate them for a better understanding.
Please I'd be glad to hear your commments about how I structured my project refering your Structure in your SO comment.
I think that Class A should be moved outside the domain folder, perhaps that's my only doubt. I'd like you can give some feedback, thanks.
 
 
14 hours later…
2:23 PM
Hi.
To 1)
a)
// Location of interfaces to external services or own developed libraries.
namespace MyApp\Domain\Model\Service;

interface Scraper {

	public function scrap(string $httpMethod, string $url);

}
b)
namespace MyApp\Domain\Infrastructure\Scraper;

use Goutte\Client as GoutteClient;
use MyApp\Domain\Model\Service\Scraper;
use Symfony\Component\DomCrawler\Crawler as SymfonyCrawler;

// This is "class D".
class Goutte implements Scraper  {

	private $client;

	public function __construct(GoutteClient $client) {
		$this->client = $client;
	}

	public function scrap(string $httpMethod, string $url) {
		$crawler = $client->request($httpMethod, $url);

		return $this->process($crawler);
	}

	// Process the scraped data.
The Scraper interface should be used in order to be able to use other libraries than Goutte too.
To 2) and 3):
 
3:08 PM
Sorry, must go. I'll be back later.
 
Ok, by the way, I've already coded all the mentioned classes and all are working good, my problem is if the mentioned classes are ok, or should I create another or merge some of them into one, AND mainly the structure of folders that I've made.
 
 
5 hours later…
8:11 PM
I think you should split the whole functionality into two independent parts:
A) SCRAPING data - Here I can't really know what you mean by the validation of the scraped data. Though, here are some thoughts:
a) You could create a service which would use a validator and a scraping mapper. The mapper would be responsible with scraping the data (by using the Goutte service) and returning it as an array of NationalTax objects (for example). The validator would afterwords validate - whatever this means - each NationalTax instance.
b) You could create a service which would use a scraping mapper. The validation - whatever this means - would be "included" in the mapper. The mapper should be responsible with scraping the data (by using the Goutte service) and returning it all as an array of "valid" NationalTax objects (for example). In other words, when each NationalTax object is created, a validation step of the corresponding "input data", e.g. of the corresponding scraped data, would take place.
Even if the validation seems "too much" for the mapper, IMO, in this case at least, it isn't. Imagine that a data mapper to a db decides as well, in which form the db values are to be stored as properties in each of the returned objects. For example, let's say that a UserMapper would fetch all users from a table users. And let's suppose, the table has two columns among others: first_name and last_name.
The UserMapper would fetch all records and, for each of them, would create a User object. At that moment, the data mapper could map the two column values to two corresponding properties of the User instance: User->firstName and User->lastName. Or it could just concatenate the two column values to a single User property: User->name. So, why would this step not contain a validation as well?
c) You could create a service as above (in (b)), though, the mapper itself would not be responsible with the validation. The validation would be part of each NationalTax entity of the array returned by the mapper. Other said, you would make use of the so-called value objects, which will have the proper validation encapsulated in them.
This video + corresponding slides explain this type of validation in detail. I think, I would choose this alternative for your task.
B) STORING data (resulted from scraping, or not) - It would be quite straightforward: You would create a service, passing it an array of ǸationalTax objects and a data mapper. The mapper would not be the same as any of the above mentioned. This mapper would be responsible with the transfer of the ǸationalTax entities to and from the database.
As for the JOB class, it should have, in principle, the role of a controller action: to defer the request to the service layer. It would call the service for the scraping, followed by the service for storing the data into national_taxes table.
The mappers should be placed into MyApp\Domain\Infrastructure\Mapper namespace. The services into MyApp\Domain\Service. The entities and the value objects into MyApp\Domain\Model\Entity.
To your comment: Try to avoid tight coupling as much as possible!
I hope I could help. Have a nice evening.
 

« first day (21 days earlier)      last day (16 days later) »