Qt Network Access: Flexible and Powerful Access to Data on the Internet

Thiago Macieira Qt Developer Days 2008

Who am I?
Senior Software Engineer at Qt Software / Nokia Degrees in Engineering and an MBA Almost 2 years with the company Part of the Core Team
Work mostly with Networking, I/O, IPC, Threading, Tool Classes

Other official duties:

Liaison to the KDE Community Release Manager for Qt

Agenda
Background and rationale The Qt Network Access API The API in use Whats coming in Qt 4.5

Agenda
Background and rationale The Qt Network Access API The API in use Whats coming in Qt 4.5

Data on the Internet

The Internet is full of data
Some good, some bad, but that's not the point

Intranets and Extranets too

Not just the World Wide Web

Applications often need to access data

Data on the Internet

The Internet is full of data
Some good, some bad, but that's not the point

Intranets and Extranets too

Not just the World Wide Web

Applications often need to access data

Download updates or optional features Upload results Obtain reports, weather forecast

Before Qt Network Access...

There was QHttp and QFtp Low-level access to HTTP and FTP communication
Ability to send any command Full access to the response (non-standard replies)

In Qt 3.x, there was QUrlOperator

But QHttp and QFtp could be better

Operations are too low-level
Developer must know how HTTP and FTP work Easy to make mistakes

Fragile
Easy to break

Almost no parsing of replies

You must implement the rest

Sample QHttp usage

Things to remember:
Read everything at the requestFinished signal Match the contents with the id you saved

QHttp http; http.setHost(trolltech.com); int id1 = http.get(/); int id2 = http.get(/qtdevdays2008); connect(&http, SIGNAL(requestFinished(int,bool)), ...);

And QHttp could support more of HTTP

The HTTP protocol is full of features QHttp supports a small subset only Improving support was on the plans
It would require a full refactoring

10

Then QtWebKit came along...

WebKit required an abstraction layer
Early code worked with QHttp No support for FTP Separate code for dealing with files

WebKit comes from KDEs KHTML

Originally, KHTML used KIO

11

We had no choice, did we?

Requirements were:
Easy to use, but possible to extend HTTP/1.1, FTP, local files, data: Abstract the differences away

WebKit also required some more features

Cookies Proxy handling Authentication cache Simultaneous requests

12

Agenda
Background and rationale The Qt Network Access API The API in use Whats coming in Qt 4.5

13

Enter Qt Network Access

Three main classes

QNetworkRequest

QNetworkReply

QNetworkAccessManager

14

QNetworkRequest
Represents one request to be sent Main component: one URL Meta-data to the request:
HTTP headers SSL options Settings controlling behaviour

15

QNetworkReply
Read-only sequential-access QIODevice Data of the reply Meta-data:
URL Request associated HTTP headers and status Encryption status Error conditions

16

QNetworkReply (cont.)
Already open, but not finished yet Emits readyRead() when data arrives Emits finished() when done Emits error() if problem occurs Progress signals: downloadProgress() and uploadProgress()

17

QNetworkAccessManager
Operations supported:
HEAD obtain status GET download PUT upload POST HTTP-only

For each operation, theres one method

QNetworkRequest as input Data to be uploaded (if appropriate) QNetworkReply as output

18

QNetworkAccessManager (cont.)
Manages request queue Centralises settings
Cookie Jar Proxy More in 4.5

Caches previous authentication successes Keeps connections open for some time

19

Cookies
Documentation defines as:
Cookies are small bits of information that stateless protocols like HTTP use to maintain some persistent information across requests.

One class: QNetworkCookie Does the parsing for you Recreates the raw form

20

Getting and setting cookies

Set cookies in a QNetworkRequest Retrieve them from a QNetworkReply
QNetworkCookie cookie(key, value 42); qDebug() << cookie; // prints: key=value 42 QNetworkRequest request; request.setHeader(QNetworkRequest::SetCookieHeader, cookie); QList<QNetworkCookie> cookies = reply->header(QNetworkRequest::CookieHeader) .value<QList<QNetworkCookie> >();

21

Where do you keep your cookies?

In the cookie jar! QNetworkCookieJar Basic security only Two virtual functions:
cookiesForUrl setCookiesFromUrl

22

Extending Qt Network Access Why?

Handling special requests Using a different backend Applying policies/filtering to requests allowed Rewriting URLs Changing default behaviour

23

How do I extend it?

QNetworkAccessManager has one virtual function:
createRequest() All requests are queued and sent by this function

QNetworkReply is abstract
Implement pure virtuals Emit readyRead() and other signals

24

Agenda
Background and rationale The Qt Network Access API The API in use Whats coming in Qt 4.5

25

How easy is it?

Simplest ways:
QNetworkAccessManager manager; QNetworkReply *reply = manager.get(QUrl(http://trolltech.com)); connect(reply, SIGNAL(finished()), ...);

QNetworkAccessManager manager; connect(&manager, SIGNAL(finished(QNetworkReply*)), ...); manager.get(QUrl(http://trolltech.com)); manager.get(QUrl(http://trolltech.com/qtdevdays2008));

26

Old versus new

Code does the same operation
QHttp http; http.setHost(trolltech.com); int id1 = http.get(/); int id2 = http.get(/qtdevdays2008); connect(&http, SIGNAL(requestFinished(int,bool)), ...); QNetworkAccessManager manager; connect(&manager, SIGNAL(finished(QNetworkReply*)), ...); manager.get(QUrl(http://trolltech.com)); manager.get(QUrl(http://trolltech.com/qtdevdays2008));

27

Full example Downloader class

class Download: public QObject { Q_OBJECT QNetworkAccessManager manager; public: Download() { connect(&manager, SIGNAL(finished(QNetworkReply*)), SLOT(finished(QNetworkReply*))); } void download(const QUrl &url) { manager.get(url); } private slots: void finished(QNetworkReply *reply) { reply->deleteLater(); emit done(reply->url(), reply->readAll()); } signals: void done(const QUrl &url, const QByteArray &data); };
28

Full example Downloader class

class Download: public QObject { Q_OBJECT QNetworkAccessManager manager; public: Download() { connect(&manager, SIGNAL(finished(QNetworkReply*)), SLOT(finished(QNetworkReply*))); } QNetworkAccessManager manager;

Download() private slots:{ void finished(QNetworkReply *reply) connect(&manager, SIGNAL(finished(QNetworkReply*)), { reply->deleteLater(); SLOT(finished(QNetworkReply*))); emit done(reply->url(), reply->readAll()); } }
signals: void done(const QUrl &url, const QByteArray &data); };
29

public: void download(const QUrl &url) { manager.get(url); }

Full example Downloader class

class Download: public QObject { Q_OBJECT QNetworkAccessManager manager; public: Download() { void download(const QUrl &url) connect(&manager, SIGNAL(finished(QNetworkReply*)), { manager.get(url); } SLOT(finished(QNetworkReply*))); } void download(const QUrl &url) { manager.get(url); } private slots: void finished(QNetworkReply *reply) signals: { reply->deleteLater(); void done(const QUrl &url, const emit done(reply->url(), reply->readAll()); }

QByteArray &data);

signals: void done(const QUrl &url, const QByteArray &data); };

30

Full example Downloader class

class Download: public QObject { Q_OBJECT QNetworkAccessManager manager; public: private slots: Download() void finished(QNetworkReply *reply) { connect(&manager, SIGNAL(finished(QNetworkReply*)), { SLOT(finished(QNetworkReply*))); reply->deleteLater(); } void download(const QUrl &url) } { manager.get(url); }

emit done(reply->url(), reply->readAll());

private slots: void finished(QNetworkReply *reply) { reply->deleteLater(); emit done(reply->url(), reply->readAll()); } signals: void done(const QUrl &url, const QByteArray &data); };
31

Full example Downloader class

QNetworkAccessManager manager; public: Download() { connect(&manager, SIGNAL(finished(QNetworkReply*)), SLOT(finished(QNetworkReply*))); } void download(const QUrl &url) { manager.get(url); } private slots: void finished(QNetworkReply *reply) { reply->deleteLater(); emit done(reply->url(), reply->readAll()); } signals: void done(const QUrl &url, const QByteArray &data);

32

Downloader class improving it

Handle authentication challenges Handle error conditions Handle SSL issues Allow access to the meta-data Use a shared QNetworkAccessManager

33

Downloader class authentication challenges

Just connect to the extra signals

QNetworkAccessManager manager; public: Download() { connect(&manager, SIGNAL(finished(QNetworkReply*)), SLOT(finished(QNetworkReply*))); }

34

Downloader class authentication challenges

Just connect to the extra signals
QNetworkAccessManager manager; public: Download() { connect(&manager, SIGNAL(finished(QNetworkReply*)), SLOT(finished(QNetworkReply*))); connect(&manager, SIGNAL(authenticationRequired(QNetworkReply*,QAuthenticator*)), ...); connect(&manager, SIGNAL(proxyAuthenticationRequired(QNetworkProxy,QAuthenticator*)), ...); }

Downloader class error handling (alternative 1)

The finished signal is still emitted The error code is saved in the reply
private slots: void finished(QNetworkReply *reply) { reply->deleteLater(); if (reply->error() != QNetworkReply::NoError) { // do something about this error } else emit done(reply->url(), reply->readAll()); }

Downloader class error handling (alternative 2)

Or you can use the error signal That one is per reply
void download(const QUrl &url) { QNetworkReply *reply = manager.get(url); connect(reply, SIGNAL(error(QNetworkReply::NetworkError)), ...); }

37

Synchronous downloading
No waitFor functions you must use the event loop
QByteArray synchronousDownload(const QNetworkRequest &request) { QNetworkReply *reply = manager.get(request); QEventLoop loop; connect(reply, SIGNAL(finished()), &loop, SLOT(quit())); loop.exec(); return reply->readAll(); }

38

Agenda
Background and rationale The Qt Network Access API The API in use Whats coming in Qt 4.5

39

Whats coming in Qt 4.5?

Page caching Proxy factory

40

Data caching
Problematic:
Network transfers are costly Therefore, we should avoid re-transferring

New abstract base class: QAbstractNetworkCache Integrated with QNetworkAccessManager

setCache(QAbstractNetworkCache *) Automatically uses the cache

41

Cache access policies

Four policies to choose from:
AlwaysNetwork PreferNetwork PreferCache AlwaysCache

In a web browser
Default: PreferNetwork Reload button: AlwaysNetwork Offline browsing: AlwaysCache

42

Caching on disk
Public class: QNetworkDiskCache Sample implementation of data caching Saves all data to one directory on disk Controls cache size Should work from multiple processes...

43

Proxy factory
Problematic:
Different destinations accessed via different proxies Support for Proxy Auto Configuration (PAC) scripts

One single proxy for all requests doesnt cut it... New abstract class: QNetworkProxyFactory Integrated with QNetworkAccessManager:
setProxyFactory(QNetworkProxyFactory *)

44

How does it work?

For each request, obtain a list of proxies to try Currently we use the suitable one

Request

Proxy factory

QNetworkReply QNetworkReply QNetworkReply QNetworkProxy

45

Proxy Auto Configuration scripts

Basically JavaScript code to choose a proxy Maps to the QNetworkProxyFactory model Interpreter provided on Labs
Hostname, URL QNetworkReply QNetworkReply QNetworkReply QNetworkProxy

JavaScript

46

Qt 4.5 is just around the corner

Technical Preview just released Updates and fixes available via Snapshots

47

Agenda

Summary

48

Summary
Qt Network Access is very powerful Easily extensible

49

Appendix

Extra slides

Uniform Resource Locator (URL)

Data is identified by a URL The Qt class for this is QUrl Examples:
http://www.example.net/faq.html#question13 ftp://ftp.example.com/pub/something/ ftp://tray:5uQQo_f@ftp.example.com:2021 mailto:postmaster@example.com

51