(Print this page)

Using The WebRequest object to retrieve Yahoo stock quotes
Published date: Monday, March 1, 2004
On: Moer and Éric Moreau's web site

The WebRequest class is a request/response model for accessing data from the Internet. This class can be used to query any uniform resource identifier (URI). A Web page on a server is probably the most used example.

The WebResponse class is similar to the WebRequest class. This class is used to receive data from the Internet resource.

These 2 classes are located into the System.Net namespace. Using these 2 classes, we have everything we need to download a complete Web page into a stream. But what I really want to show you is how to retrieve data (stock quotes and currency exchange rate) from the Yahoo web sites.

Note: For those of you who read the February 2004 edition of the UTMag, Tom Bellmer wrote an article that uses XMLHTTP with Visual FoxPro. The approach is different but the results are much the same.

Figure 1: The demo application in action

The WebRequest class

This class is used to establish a communication with a URI. It initiates a communication. We use the Create method that receives a URL as parameter to initialize a WebRequest object. This new object can then be used with the GetResponse method to synchronously download the content and store it into a WebResponse object instance. Asynchronous download is also possible if you use BeginGetResponse and EndGetResponse methods but I won’t use them here.

The WebResponse class

Once that the WebResponse instance has been filled using the GetResponse method (see previous paragraph), you can get its content and fill a stream with it. The easiest way is to fill a StreamReader object using the GetResponseStream method.

Our first method

This first method will be used by most examples of this article to read character contents.

Private Function RequestWebData(ByVal pstrURL As String) As String
    Dim objWReq As WebRequest
    Dim objWResp As WebResponse
    Dim strBuffer As String

    'Contact the website
    objWReq = HttpWebRequest.Create(pstrURL)
    objWResp = objWReq.GetResponse()

    'Read the answer from the Web site and store it into a stream
    Dim objSR As StreamReader
    objSR = New StreamReader(objWResp.GetResponseStream)
    strBuffer = objSR.ReadToEnd


    Return strBuffer
End Function
As you can see, this method uses the WebRequest and the WebResponse as described above to read the response of a URL passed as argument and returns the response into a string (named strBuffer).

Reading a complete Web Page

NNow, using our RequestWebData method, we can read a whole page from a URL and display it with into a message box simply by doing:

If you have a text file somewhere on the Internet, the previous call would only need to be changed to:
Now that you have the complete page (or a file) in a string, you can do whatever you want with it. But, the best is yet to come. Continue to read!

Retrieving data from Yahoo finance web site

Some web sites are built to also return precise data (using a somewhat different URL) instead of a long page that you would need to parse. Don’t be fooled, we are not talking of WebServices here.

For example, open Internet Explorer and type http://quote.yahoo.com/q?s=msft to see the latest Microsoft quote. Notice the last trade price. Now type http://quote.yahoo.com/d/quotes.csv?s=msft&d=t&f=sl1d1t1c1ohgvj1pp2wern, and you will find much the same information but in a coma-separated string. This comes handy when you want to retrieve data.

As you know, the page layout of the first URL you typed may change from time to time which means that if you built something to parse the page source to find the data in it, your application will probably fail. Since the second URL returns only the data, it is not related to the page layout and your application will continue to run without any problem.

We can use our RequestWebData method to retrieve this data as well as the page source or the text file of the preceding example. It would then be easy to find the second element and return it as the price of the stock we are searching. But what if we passed 2 or more symbols in the URL? I built a simple wrapper around this to transform the coma-separated value string received into a XML string that is much easier to work it. Here is this method:

Public Function GetQuoteLatest(ByVal pstrSymbol As String) As String
    Dim strURL As String
    Dim strBuffer As String

    'Creates the request URL for Yahoo
    strURL = "http://quote.yahoo.com/d/quotes.csv?" & _
             "s=" & pstrSymbol & _
             "&d=t" & _

    strBuffer = RequestWebData(strURL)

    'Loop through the lines returned and transform it to a XML string
    Dim strReturn As New System.Text.StringBuilder
    strReturn.Append("" & Environment.NewLine)
    For Each strLine As String In strBuffer.Split(ControlChars.Lf)
        If strLine.Length > 0 Then
            strReturn.Append(TransformLatestLine(strLine) & Environment.NewLine)
        End If
    strReturn.Append("" & Environment.NewLine)

    Return strReturn.ToString
End Function
This method receives a symbol and uses the RequestWebData method to get the data of Yahoo. It then loops through all the lines (in case we passed more then one symbol) calling the TransformLatestLineMethod that creates XML data from the string. The TransformLatestLineMethod is simply this:
Private Function TransformLatestLine(ByVal pstrLine As String) As String
    Dim arrLine() As String
    Dim strXML As New System.Text.StringBuilder

    arrLine = pstrLine.Split(","c)

    strXML.Append("" & arrLine(0).Replace(Chr(34), "") & "")
    strXML.Append("" & arrLine(1) & "")
    strXML.Append("" & arrLine(2).Replace(Chr(34), "") & "")
    strXML.Append("" & arrLine(4) & "")
    strXML.Append("" & arrLine(5) & "")
    strXML.Append("" & arrLine(6) & "")
    strXML.Append("" & arrLine(7) & "")
    strXML.Append("" & arrLine(8) & "")
    strXML.Append("" & arrLine(9) & "")
    strXML.Append("" & arrLine(10) & "")
    strXML.Append("" & arrLine(11).Replace(Chr(34), "") & "")
    strXML.Append("" & arrLine(12).Replace(Chr(34), "") & "")
    strXML.Append("" & arrLine(13) & "")
    strXML.Append("" & arrLine(14) & "")
    strXML.Append("" & arrLine(15).Replace(Chr(34), "") & "")
    Return strXML.ToString
End Function

If you know the URL to pass, you can retrieve any values from Yahoo Web site. The thing is that I didn’t find any clear documentation on that. If you find some, please let me know! The easiest way to find the URL is to copy the URL from the “Download Data” button (see the bottom-right of figure 2) and removing the “&e=.csv” parameter.

Using this very same technique the downloadable demo also includes the retrieval of historical price and also the currency exchange rate.

Figure 2: Download data

Retrieving images

When using the WebRequest and the WebResponse classes, we are not limited to text data. Images are also available using this technique.

This is the main method that connects to a URL, retrieves the image and save it:

Private Function RequestWebImage(ByVal pstrURL As String) As String
    Dim objWReq As WebRequest
    Dim objWResp As WebResponse
    Dim strBuffer As String

    'Contact the website
    objWReq = HttpWebRequest.Create(pstrURL)
    objWResp = objWReq.GetResponse()

    'Read the answer from the website and store it into a stream
    Dim objStream As Stream
    objStream = objWResp.GetResponseStream
    Dim inBuf(100000) As Byte
    Dim bytesToRead As Integer = CInt(inBuf.Length)
    Dim bytesRead As Integer = 0
    While bytesToRead > 0
        Dim n As Integer = objStream.Read(inBuf, bytesRead, bytesToRead)
        If n = 0 Then
            Exit While
        End If
        bytesRead += n
        bytesToRead -= n
    End While
    Dim fstr As New FileStream("weather.jpg", FileMode.OpenOrCreate, FileAccess.Write)
    fstr.Write(inBuf, 0, bytesRead)


    Return "weather.jpg"
End Function
As you can see this new method is not very different from the first. Only the processing of the returned stream changed. This can be used to display the saved image to a picture box like this:
Dim strFile As String
strFile = RequestWebImage("http://image.weather.com/images/sat/canadasat_720x486.jpg")
PictureBox1.Image = Image.FromFile(strFile)


As you can see, retrieving data from a Web site can be easy. Other sites that don’t give you that kind of returned string are much more difficult to handle. In a perfect world, Yahoo would offer a WebService but none is available for now!

I hope you appreciated the topic and see you next month.

(Print this page)