The WebRequest class is a request/response model for accessing data from the Internet. This class can be used to query any uniform resource identifier (URI). A Web page on a server is probably the most used example.
The WebResponse class is similar to the WebRequest class. This class is used to receive data from the Internet resource.
These 2 classes are located into the System.Net namespace. Using these 2 classes, we have everything we need to download a complete Web page into a stream. But what I really want to show you is how to retrieve data (stock quotes and currency exchange rate) from the Yahoo web sites.
Note: For those of you who read the February 2004 edition of the UTMag, Tom Bellmer wrote an article that uses XMLHTTP with Visual FoxPro. The approach is different but the results are much the same.
Figure 1: The demo application in action
The WebRequest class
This class is used to establish a communication with a URI. It initiates a communication. We use the Create method that receives a URL as parameter to initialize a WebRequest object. This new object can then be used with the GetResponse method to synchronously download the content and store it into a WebResponse object instance. Asynchronous download is also possible if you use BeginGetResponse and EndGetResponse methods but I won’t use them here.
The WebResponse class
Once that the WebResponse instance has been filled using the GetResponse method (see previous paragraph), you can get its content and fill a stream with it. The easiest way is to fill a StreamReader object using the GetResponseStream method.
Our first method
This first method will be used by most examples of this article to read character contents.
Private Function RequestWebData(ByVal pstrURL As String) As String
Dim objWReq As WebRequest
Dim objWResp As WebResponse
Dim strBuffer As String
'Contact the website
objWReq = HttpWebRequest.Create(pstrURL)
objWResp = objWReq.GetResponse()
'Read the answer from the Web site and store it into a stream
Dim objSR As StreamReader
objSR = New StreamReader(objWResp.GetResponseStream)
strBuffer = objSR.ReadToEnd
objSR.Close()
objWResp.Close()
Return strBuffer
End Function
As you can see, this method uses the WebRequest and the WebResponse as described above to read the response of a URL passed as argument and returns the response into a string (named strBuffer).
Reading a complete Web Page
NNow, using our RequestWebData method, we can read a whole page from a URL and display it with into a message box simply by doing:
MessageBox.Show(RequestWebData("http://www.utmag.com"))
If you have a text file somewhere on the Internet, the previous call would only need to be changed to: MessageBox.Show(RequestWebData("http://www.YourServer.com/YourFile.txt"))
Now that you have the complete page (or a file) in a string, you can do whatever you want with it. But, the best is yet to come. Continue to read!
Retrieving data from Yahoo finance web site
Some web sites are built to also return precise data (using a somewhat different URL) instead of a long page that you would need to parse. Don’t be fooled, we are not talking of WebServices here.
For example, open Internet Explorer and type http://quote.yahoo.com/q?s=msft to see the latest Microsoft quote. Notice the last trade price. Now type http://quote.yahoo.com/d/quotes.csv?s=msft&d=t&f=sl1d1t1c1ohgvj1pp2wern, and you will find much the same information but in a coma-separated string. This comes handy when you want to retrieve data.
As you know, the page layout of the first URL you typed may change from time to time which means that if you built something to parse the page source to find the data in it, your application will probably fail. Since the second URL returns only the data, it is not related to the page layout and your application will continue to run without any problem.
We can use our RequestWebData method to retrieve this data as well as the page source or the text file of the preceding example. It would then be easy to find the second element and return it as the price of the stock we are searching. But what if we passed 2 or more symbols in the URL? I built a simple wrapper around this to transform the coma-separated value string received into a XML string that is much easier to work it. Here is this method:
Public Function GetQuoteLatest(ByVal pstrSymbol As String) As String
Dim strURL As String
Dim strBuffer As String
'Creates the request URL for Yahoo
strURL = "http://quote.yahoo.com/d/quotes.csv?" & _
"s=" & pstrSymbol & _
"&d=t" & _
"&f=sl1d1t1c1ohgvj1pp2wern"
strBuffer = RequestWebData(strURL)
'Loop through the lines returned and transform it to a XML string
Dim strReturn As New System.Text.StringBuilder
strReturn.Append("" & Environment.NewLine)
For Each strLine As String In strBuffer.Split(ControlChars.Lf)
If strLine.Length > 0 Then
strReturn.Append(TransformLatestLine(strLine) & Environment.NewLine)
End If
Next
strReturn.Append("" & Environment.NewLine)
Return strReturn.ToString
End Function
This method receives a symbol and uses the RequestWebData method to get the data of Yahoo. It then loops through all the lines (in case we passed more then one symbol) calling the TransformLatestLineMethod that creates XML data from the string. The TransformLatestLineMethod is simply this: Private Function TransformLatestLine(ByVal pstrLine As String) As String
Dim arrLine() As String
Dim strXML As New System.Text.StringBuilder
arrLine = pstrLine.Split(","c)
strXML.Append("")
strXML.Append("" & arrLine(0).Replace(Chr(34), "") & "")
strXML.Append("" & arrLine(1) & "")
strXML.Append("" & arrLine(2).Replace(Chr(34), "") & "")
strXML.Append("")
strXML.Append("" & arrLine(4) & "")
strXML.Append("" & arrLine(5) & "")
strXML.Append("" & arrLine(6) & "")
strXML.Append("" & arrLine(7) & "")
strXML.Append("" & arrLine(8) & "")
strXML.Append("" & arrLine(9) & "")
strXML.Append("" & arrLine(10) & "")
strXML.Append("" & arrLine(11).Replace(Chr(34), "") & "")
strXML.Append("" & arrLine(12).Replace(Chr(34), "") & "")
strXML.Append("" & arrLine(13) & "")
strXML.Append("" & arrLine(14) & "")
strXML.Append("" & arrLine(15).Replace(Chr(34), "") & "")
strXML.Append("")
Return strXML.ToString
End Function
If you know the URL to pass, you can retrieve any values from Yahoo Web site. The thing is that I didn’t find any clear documentation on that. If you find some, please let me know! The easiest way to find the URL is to copy the URL from the “Download Data” button (see the bottom-right of figure 2) and removing the “&e=.csv” parameter.
Using this very same technique the downloadable demo also includes the retrieval of historical price and also the currency exchange rate.
Figure 2: Download data
Retrieving images
When using the WebRequest and the WebResponse classes, we are not limited to text data. Images are also available using this technique.
This is the main method that connects to a URL, retrieves the image and save it:
Private Function RequestWebImage(ByVal pstrURL As String) As String
Dim objWReq As WebRequest
Dim objWResp As WebResponse
Dim strBuffer As String
'Contact the website
objWReq = HttpWebRequest.Create(pstrURL)
objWResp = objWReq.GetResponse()
'Read the answer from the website and store it into a stream
Dim objStream As Stream
objStream = objWResp.GetResponseStream
Dim inBuf(100000) As Byte
Dim bytesToRead As Integer = CInt(inBuf.Length)
Dim bytesRead As Integer = 0
While bytesToRead > 0
Dim n As Integer = objStream.Read(inBuf, bytesRead, bytesToRead)
If n = 0 Then
Exit While
End If
bytesRead += n
bytesToRead -= n
End While
Dim fstr As New FileStream("weather.jpg", FileMode.OpenOrCreate, FileAccess.Write)
fstr.Write(inBuf, 0, bytesRead)
objStream.Close()
fstr.Close()
objWResp.Close()
Return "weather.jpg"
End Function
As you can see this new method is not very different from the first. Only the processing of the returned stream changed. This can be used to display the saved image to a picture box like this: Dim strFile As String
strFile = RequestWebImage("http://image.weather.com/images/sat/canadasat_720x486.jpg")
PictureBox1.Image = Image.FromFile(strFile)
Conclusion
As you can see, retrieving data from a Web site can be easy. Other sites that don’t give you that kind of returned string are much more difficult to handle. In a perfect world, Yahoo would offer a WebService but none is available for now!
I hope you appreciated the topic and see you next month.