Making Asynchronous HTTP Requests With ReactPHP
The Problem
We need to perform batches of HTTP requests. For example, we need to download several video files. We can start downloading them one by one, but it will take a lot of time since we need to wait for each request to be finished before we can start a new one. The larger the number of requests we are dealing with, the more this latency grows. We also cannot perform any other operations until all files will be downloaded.
In an asynchronous way, there is no need to wait until the last request is being finished. We can start processing the results immediately when any of the requests are being finished. ReactPHP has HttpClient component which allows you to send HTTP requests asynchronously.
ReactPHP HttpClient
The component itself is very simple. To start sending requests we need an instance of the React\HttpClient\Client class:
<?php
$loop = React\EventLoop\Factory::create();
$client = new React\HttpClient\Client($loop);Client class is very simple and its interface consists of the one request() method. It accepts a request method, URL, optional additional headers and returns an instance of the React\HttpClient\Request class. Let’s create a GET request to https://php.net:
<?php
$loop = React\EventLoop\Factory::create();
$client = new React\HttpClient\Client($loop);
$request = $client->request('GET', 'http://php.net/');
$loop->run();By this moment we are not executing any real requests, we have only prepared a request. Next step is to set up it.
React\HttpClient\Request class implements WritableStreamInterface, so we can attach handlers for some events of this stream. Now the most interesting for us is the response event. This event is emitted when the response headers were received from the server and successfully parsed:
<?php
$loop = React\EventLoop\Factory::create();
$client = new React\HttpClient\Client($loop);
$request = $client->request('GET', 'http://php.net/');
$request->on('response', function (\React\HttpClient\Response $response) {
// ...
});
$loop->run();The callback for this event accepts an instance of the \React\HttpClient\Response as an argument. This class implements ReadableStreamInterface which means that we can also consider it as a stream and read data from it. To receive the response body we can listen to the data event. The handler for this event receives a chunk of the response body as an argument.
<?php
$loop = React\EventLoop\Factory::create();
$client = new React\HttpClient\Client($loop);
$request = $client->request('GET', 'http://php.net/');
$request->on('response', function (\React\HttpClient\Response $response) {
$response->on('data', function ($chunk) {
echo $chunk;
});
});
$loop->run();But if we run this code we still get nothing. This is because there will be no execution till we call end() method on the request object:
<?php
$loop = React\EventLoop\Factory::create();
$client = new React\HttpClient\Client($loop);
$request = $client->request('GET', 'http://php.net/');
$request->on('response', function (\React\HttpClient\Response $response) {
$response->on('data', function ($chunk) {
echo $chunk;
});
});
$request->end();
$loop->run();This method indicates that we have finished sending the request.
Downloading File
We can use an instance of the \React\HttpClient\Response class as a readable stream and then pipe it to a writable stream as a source. As a result, we can read from the response and write it to a file. Create a filesystem object and then open a file in a write mode:
<?php
$filesystem = \React\Filesystem\Filesystem::create($loop);
$file = \React\Promise\Stream\unwrapWritable($filesystem->file('sample.mp4')->open('cw'));We unwrap the stream with \React\Promise\Stream\unwrapWritable() function. This function can be used to unwrap a Promise which resolves with a WritableStreamInterface. It returns a writable stream which acts as a proxy for the future promise resolution.
<?php
$loop = React\EventLoop\Factory::create();
$client = new React\HttpClient\Client($loop);
$filesystem = \React\Filesystem\Filesystem::create($loop);
$file = \React\Promise\Stream\unwrapWritable($filesystem->file('sample.mp4')->open('cw'));
$request = $client->request('GET', 'http://www.sample-videos.com/video/mp4/720/big_buck_bunny_720p_1mb.mp4');
$request->on('response', function (\React\HttpClient\Response $response) use ($file) {
$response->pipe($file);
});
$request->end();
$loop->run();If you are new to streams check this article about ReactPHP streams.
In this example, we download a small video sample using a GET request and stream it to a local file. As soon as the request starts returning chunks of the downloading video it will pipe that data to the sample.mp4 file.
Note that this example uses
fopen()for simplicity and demo purposes only! This should not be used in a truly asynchronous application because the filesystem is inherently blocking and each call could potentially take several seconds. Read this tutorial in case you need to work asynchronously with the filesystem in ReactPHP ecosystem.
As a next step, we can add a progress for our download. To track the progress we need to know the total size of the downloading file and the current downloaded size. We can use getHeaders() method of the response object to retrieve server headers. We need a Content-Length header, which contains the full size of the file:
<?php
$request->on('response', function (\React\HttpClient\Response $response) use ($file) {
$size = $response->getHeaders()['Content-Length'];
$currentSize = 0;
$response->pipe($file);
});To get the current downloaded size we can use the length of the received data. We start with zero and then every time we receive a new chunk of data we increase this value by the length of this data:
<?php
$response->on('data', function($data) use ($size, &$currentSize){
$currentSize += strlen($data);
echo "Downloading: ", number_format($currentSize / $size * 100), "%\n";
});Now we need to merge streaming to the local file and tracking the progress. We can wrap the outputting the download progress into an instance of the \React\Stream\ThroughStream(). This sort of streams can be used to process data through the pipes, exactly what we need. We write data to this stream, the data is being processed and then we can read the processed data from it.
<?php
$loop = React\EventLoop\Factory::create();
$client = new React\HttpClient\Client($loop);
$filesystem = \React\Filesystem\Filesystem::create($loop);
$file = unwrapWritable($filesystem->file('sample.mp4')->open('cw'));
$request = $client->request('GET', 'http://www.sample-videos.com/video/mp4/720/big_buck_bunny_720p_1mb.mp4');
$request->on('response', function (\React\HttpClient\Response $response) use ($file) {
$size = $response->getHeaders()['Content-Length'];
$currentSize = 0;
$progress = new \React\Stream\ThroughStream();
$progress->on('data', function($data) use ($size, &$currentSize){
$currentSize += strlen($data);
echo "Downloading: ", number_format($currentSize / $size * 100), "%\n";
});
$response->pipe($progress)->pipe($file);
});
$request->end();
$loop->run();In this snippet, we read data from the response. Then pipe it to track the download progress and then pipe this data to the local file on disk. The progress is showing but the output doesn’t look great.
To fix this issue we can use cursor movement character. ANSI escape sequences allow moving the cursor around the screen. We can use this sequence to move the cursor N lines up \033[<N>A. In our case, we need to move the cursor one line up (\033[1A). And because we are moving a cursor one line up, we should add one line break before we start showing the progress:
<?php
$loop = React\EventLoop\Factory::create();
$client = new React\HttpClient\Client($loop);
$filesystem = \React\Filesystem\Filesystem::create($loop);
$file = unwrapWritable($filesystem->file('sample.mp4')->open('cw'));
$request = $client->request('GET', 'http://www.sample-videos.com/video/mp4/720/big_buck_bunny_720p_1mb.mp4');
$request->on('response', function (\React\HttpClient\Response $response) use ($file) {
$size = $response->getHeaders()['Content-Length'];
$currentSize = 0;
$progress = new \React\Stream\ThroughStream();
$progress->on('data', function($data) use ($size, &$currentSize){
$currentSize += strlen($data);
echo "\033[1A", "Downloading: ", number_format($currentSize / $size * 100), "%\n";
});
$response->pipe($progress)->pipe($file);
});
$request->end();
$loop->run();
echo "\n";With this simple changes, the output looks pretty nice!
Parallel Downloading
When our main logic for downloading file is ready we can extract it to a class and improve it for handling multiple parallel downloads. This class will have a single download() method which will accept an array of links. When we call download() it starts downloading specified files in parallel, like this:
<?php
// ...
$loop = React\EventLoop\Factory::create();
$client = React\HttpClient\Client($loop);
$filesystem = \React\Filesystem\Filesystem::create($loop);
$files = [
'http://www.sample-videos.com/video/mp4/720/big_buck_bunny_720p_1mb.mp4',
'http://www.sample-videos.com/video/mp4/720/big_buck_bunny_720p_2mb.mp4',
'http://www.sample-videos.com/video/mp4/720/big_buck_bunny_720p_5mb.mp4',
];
$downloader = new Downloader($client, $filesystem);
$downloader->download($files);This class will be a wrapper over the HTTP client. We also need an instance of the event loop to perform some async operations. So, we require them in the constructor:
<?php
final class Downloader
{
private $client;
private $filesystem;
public function __construct(Client $client, FilesystemInterface $filesystem)
{
$this->client = $client;
$this->filesystem = $filesystem;
}
/**
* @param array $files
*/
public function download(array $files)
{
// ...
}
}The main process of handling multiple downloads will be the following. For every specified link we:
- Instantiate a request.
- Setup all handlers for this request.
- Store this request in a property.
When all requests are instantiated and configured we walk through them and for each request call end() method to start sending data. The last step is to run() the event loop:
<?php
final class Downloader
{
// ...
/**
* @param array $files
*/
public function download(array $files)
{
foreach ($files as $index => $file) {
$this->initRequest($file, $index + 1);
}
echo str_repeat("\n", count($this->requests));
$this->runRequests();
}
}Our initRequest() method will be very similar to the code from the previous section where we download a single file:
<?php
final class Downloader
{
// ...
/**
* @param string $url
* @param int $position
*/
private function initRequest($url, $position)
{
$fileName = basename($url);
$file = \React\Promise\Stream\unwrapWritable($this->filesystem->file($fileName)->open('cw'));
$request = $this->client->request('GET', $url);
$request->on('response', function (\React\HttpClient\Response $response) use ($file, $fileName, $position) {
$size = $response->getHeaders()['Content-Length'];
$currentSize = 0;
$progress = new \React\Stream\ThroughStream();
$progress->on('data', function($data) use ($size, &$currentSize, $fileName, $position){
$currentSize += strlen($data);
echo str_repeat("\033[1A", $position),
"$fileName: ", number_format($currentSize / $size * 100), "%",
str_repeat("\n", $position);
});
$response->pipe($progress)->pipe($file);
});
$this->requests[] = $request;
}
// ...
}The only difference is that now we need to show several lines of output. That’s why we need a $position variable which is used to format the output properly according to the specified number of links. All the rest code is the same: we create an instance of the request and setup the handlers. Then we store this request in the $requests property.
We can refactor this method and extract configuring an instance of the ThroughStream into its own method:
<?php
final class Downloader
{
// ...
/**
* @param string $url
* @param int $position
*/
private function initRequest($url, $position)
{
$fileName = basename($url);
$file = \React\Promise\Stream\unwrapWritable($this->filesystem->file($fileName)->open('cw'));
$request = $this->client->request('GET', $url);
$request->on('response', function (\React\HttpClient\Response $response) use ($file, $fileName, $position) {
$size = $response->getHeaders()['Content-Length'];
$progress = $this->makeProgressStream($size, $fileName, $position);
$response->pipe($progress)->pipe($file);
});
$this->requests[] = $request;
}
/**
* @param int $size
* @param string $fileName
* @param int $position
* @return \React\Stream\ThroughStream
*/
private function makeProgressStream($size, $fileName, $position)
{
$currentSize = 0;
$progress = new \React\Stream\ThroughStream();
$progress->on('data', function($data) use ($size, &$currentSize, $fileName, $position){
$currentSize += strlen($data);
echo str_repeat("\033[1A", $position),
"$fileName: ", number_format($currentSize / $size * 100), "%",
str_repeat("\n", $position);
});
return $progress;
}
// ...
}Now the last step is to run the requests. We need to call end() method on each request stored in the $requests property and then call run() method on the event loop. After that we clear the $requests property. This is the final version of Downloader class:
<?php
final class Downloader
{
private $requests = [];
private $client;
private $filesystem;
public function __construct(Client $client, FilesystemInterface $filesystem)
{
$this->client = $client;
$this->filesystem = $filesystem;
}
/**
* @param string|array $files
*/
public function download(array $files)
{
foreach ($files as $index => $file) {
$this->initRequest($file, $index + 1);
}
echo str_repeat("\n", count($this->requests));
$this->runRequests();
}
/**
* @param string $url
* @param int $position
*/
private function initRequest($url, $position)
{
$fileName = basename($url);
$file = \React\Promise\Stream\unwrapWritable($this->filesystem->file($fileName)->open('cw'));
$request = $this->client->request('GET', $url);
$request->on('response', function (\React\HttpClient\Response $response) use ($file, $fileName, $position) {
$size = $response->getHeaders()['Content-Length'];
$progress = $this->makeProgressStream($size, $fileName, $position);
$response->pipe($progress)->pipe($file);
});
$this->requests[] = $request;
}
/**
* @param int $size
* @param string $fileName
* @param int $position
* @return \React\Stream\ThroughStream
*/
private function makeProgressStream($size, $fileName, $position)
{
$currentSize = 0;
$progress = new \React\Stream\ThroughStream();
$progress->on('data', function($data) use ($size, &$currentSize, $fileName, $position){
$currentSize += strlen($data);
echo str_repeat("\033[1A", $position),
"$fileName: ", number_format($currentSize / $size * 100), "%",
str_repeat("\n", $position);
});
return $progress;
}
private function runRequests()
{
foreach ($this->requests as $request) {
$request->end();
}
$this->requests = [];
}
}Then we init an event loop and a client, pass them into the constructor and call download() method with a list of links:
<?php
$loop = React\EventLoop\Factory::create();
$client = new React\HttpClient\Client($loop);
$files = [
'http://www.sample-videos.com/video/mp4/720/big_buck_bunny_720p_1mb.mp4',
'http://www.sample-videos.com/video/mp4/720/big_buck_bunny_720p_2mb.mp4',
'http://www.sample-videos.com/video/mp4/720/big_buck_bunny_720p_5mb.mp4',
];
$downloader = new Downloader($client, Filesystem::create($loop));
$downloader->download($files);
$loop->run();The files are downloaded in parallel:
Conclusion
When it comes to sending HTTP requests in an asynchronous way ReactPHP HttpClient can be a very useful component. It becomes really powerful when using it with streams and piping the data.
You can find examples from this article on GitHub.
This article is a part of the ReactPHP Series.
Learning Event-Driven PHP With ReactPHP
The book about asynchronous PHP that you NEED!
A complete guide to writing asynchronous applications with ReactPHP. Discover event-driven architecture and non-blocking I/O with PHP!
Minimum price: 5.99$