This article describes how to extract a list of all web resources (URL's) from the web servers like http://www.clevercomponents.com, http://www.borland.com using the Clever Internet Suite components.

There is big demand for applications that allow look up through the web server (URL) and collect list of all web resources available from that web server. Most famous programs with such functionality probably would be Teleport Pro and Flash Get.

The general idea however is very simple - download a web page, parse it and extract all links and urls.

In this article we will discuss very simple and generic algorithm based on recursive downloading and parsing web pages in asynchronous mode.

As we mentioned above the main steps of algorithm are:
1. Download web page / URL
2. Parse downloaded page and extract all URL's (you can use any method you like to parse pages)
3. Save extracted URLs into the URL list
4. Take next URL from the URL list
5. Go to the first step until end of the URL list

The first step can be implemented with Clever Downloader component. Clever Downloader component in addition to base functionality provided by another popular libraries (such as Indy, IPWorks and so on) allows you to download web page / URL in asynchronous mode without interfering with main application process. After downloading process completed the OnProcessCompleted component event occurs. In order to implement recursive downloading we also need to use OnIsBusyChanged component event. OnIsBusyChanged event will protect us from stack overflow during crawling through server URL's.

Here is the code for the first step of our algorithm:


clDownLoader: TclDownLoader;
memURLList: TMemo;  

...

procedure TForm1.btnStartClick(Sender: TObject);
begin
   if clDownLoader.IsBusy then Exit;
   memURLList.Lines.Clear();
   memURLList.Lines.Add(edtRootURL.Text);
   FCurrentURLIndex := -1;
   ProcessNextURL();
end;  

procedure TForm1.ProcessNextURL();
begin
   repeat
      Inc(FCurrentURLIndex);
   until (FCurrentURLIndex >= memURLList.Lines.Count) or (Pos('.asp', memURLList.Lines[FCurrentURLIndex]) > 0)
      or (Pos('.htm', memURLList.Lines[FCurrentURLIndex]) > 0);
   if (FCurrentURLIndex < memURLList.Lines.Count) then
   begin
      clDownLoader.URL := memURLList.Lines[FCurrentURLIndex];
      clDownLoader.Start(); end else
   begin
      ShowMessage('Process Completed');
   end;
end;


Main loop within the ProcessNextURL method is searching for next URL in URL list and the important thing is that URL should looks like html page (in this example we just check for page extension). We simplified this method but if you need more advanced analysis you can easily modify it according your needs.

After downloading completed the OnIsBusyChanged event occurs. Here is a code for this event:


procedure TForm1.clDownLoaderIsBusyChanged(Sender: TObject);
var
   List: TStrings;
begin
   if clDownLoader.IsBusy then Exit;
   if FileExists(clDownLoader.LocalFile) then
   begin
      List := TStringList.Create();
      try
         List.LoadFromFile(clDownLoader.LocalFile);
         ExtractURLS(List, memURLList.Lines);
      finally
         List.Free();
      end;
      DeleteFile(clDownLoader.LocalFile);
   end;
   ProcessNextURL();
end;  


You can extract URLs from downloaded web page with any method you like the most. In our example we used simple page parsing. Full source code can be downloaded at urlextractor.zip.


procedure TForm1.ExtractURLS(APage, AURLList: TStrings);
var
   i: Integer;
   List: TStrings;
begin
   List := TStringList.Create();
   try
      ParsePage(APage, List);
      for i := 0 to List.Count - 1 do
      begin
         if (AURLList.IndexOf(List[i]) < 0) then
         begin
            AURLList.Add(List[i]);
         end;
      end;
   finally
      List.Free();
   end;
end;


When parsing URL content please pay attention to the following issues:

1. Check for duplicate URL entries before adding a URL to URL list. Most web pages have cross linked references within web site.
2. Check for links to foreign web sites. This will prevents from jumping to another web server while crawling through the specified web site.

In order to check if link belongs to the web site on which this web page is hosted you can simply compare host part of both URL's (for example http://www.site.com and http://www.site.com/index.asp). Clever Internet Suite has TclURLParser class specially designed for this purpose.


function TForm1.IsURLNative(const AURL, ABaseURL: string): Boolean;
var
   URLParser, BaseURLParser: TclURLParser;
begin
   Result := False;
   URLParser := TclURLParser.Create();
   BaseURLParser := TclURLParser.Create();
   try
      BaseURLParser.Parse(ABaseURL);
      if (URLParser.Parse(AURL) <> '') then
      begin
         Result := (URLParser.Host = BaseURLParser.Host);
      end;
   finally
      BaseURLParser.Free();
      URLParser.Free();
   end;
end;  


Given example is far from perfect and might reqire additional alterations. You can always enhance this code to desired functionality.

We provided two examples for your convenience:
1. Delphi urlextractor.zip
2. C++ Builder urlextractor_bcb.zip

Enjoy,

Sergey S
Clever Components team.
Please write to info@clevercomponents.com