TheDeveloperBlog.com
C-Sharp | Java | Python | Swift | GO | WPF | Ruby | Scala | F# | JavaScript | SQL | PHP | Angular | HTML
Golang http.Get Examples: Download Web Pages
Use the net http package to download web pages. Specify http.Get, Body and Status.Http.Get. Golang can be used to download web pages. This helps with validating a site works correctly (or for trying to get important data from the web page).
With the net http package, we have many helpful funcs for accessing files through HTTP. We do not need to write any low-level code to do this.
Http.Get example. Let us begin with this example program that uses http.Get. Please note we must import the "net/http" package at the top.
Get: We use the Get() func to download the home page of Wikipedia—the HTTPS protocol is used here.
Defer: We require that the Body (a stream) is entirely closed before continuing. The defer keyword waits for the Body to close.
ReadAll: We use the ioutil package to access the ReadAll method—this func is useful in many Go programs.
FileGolang program that uses http.Get, Body
package main
import (
"fmt"
"io/ioutil"
"net/http"
)
func main() {
// Get Wikipedia home page.
resp, _ := http.Get("https://en.wikipedia.org/")
// Defer close the response Body.
defer resp.Body.Close()
// Read everything from Body.
body, _ := ioutil.ReadAll(resp.Body)
// Convert body bytes to string.
bodyText := string(body)
fmt.Println("DONE")
fmt.Println("Response length: ", len(bodyText))
// Convert to rune slice to take substring.
runeSlice := []rune(bodyText)
fmt.Println("First chars: ", string(runeSlice[0:10]))
}
Output
DONE
Response length: 73727
First chars: <!DOCTYPE
Status. When a HTTP request is made, the response includes a status code. We can access the string containing the status with "Status."
And: We can access the int code itself with StatusCode. This is probably easier to use when testing for a 404 error.
Tip: When using http.Get(), a 404 error is not returned as an error in Go. It is returned as a successful download with a 404 code.
Golang program that uses Status, StatusCode
package main
import (
"fmt"
"net/http"
"strings"
)
func main() {
// This file does not exist.
resp, _ := http.Get("https://www.dotnetCodex.com/robots.txt")
defer resp.Body.Close()
// Test status string.
fmt.Println("Status:", resp.Status)
if strings.Contains(resp.Status, "404") {
fmt.Println("Is not found (string tested)")
}
// Test status code of the response.
fmt.Println("Status code:", resp.StatusCode)
if resp.StatusCode == 404 {
fmt.Println("Is not found (int tested)")
}
}
Output
Status: 404 Not Found
Is not found (string tested)
Status code: 404
Is not found (int tested)
Serve. With the ListenAndServe func, also part of the http module, we can build a simple web server in Golang. Each response can have a special function with HandleFunc.ListenAndServe
A review. Making sure your website is working correctly is an important job. And Golang is well-suited to download pages and validate them.
© TheDeveloperBlog.com
The Dev Codes
Related Links:
- Golang strconv, Convert Int to String
- Golang Odd and Even Numbers
- Golang Recover Built In: Handle Errors, Panics
- Learn Go Language Tutorial
- Golang html template Example
- Golang http.Get Examples: Download Web Pages
- Golang container list Example (Linked List)
- Golang base64 Encoding Example: EncodeToString
- Golang os exec Examples: Command Start and Run
- Golang String Between, Before and After
- Golang os.Remove: Delete All Files in Directory
- Golang First Words in String
- Golang flag Examples
- Golang Regexp Find Examples: FindAllString
- Golang Regexp Examples: MatchString, MustCompile
- Golang Index, LastIndex: strings Funcs
- Golang Compress GZIP Examples
- Golang Interface Example
- Golang 2D Slices and Arrays
- Golang Sscan, Sscanf Examples (fmt)
- Top 41 Go Programming (Golang) Interview Questions (2021)
- Golang Padding String Example (Right or Left Align)
- Golang Equal String, EqualFold (If Strings Are the Same)
- Golang map Examples
- Golang Map With String Slice Values
- Golang Array Examples
- Golang Remove Duplicates From Slice
- Golang If, Else Statements
- Golang ParseInt Examples: Convert String to Int
- Golang Strings
- Golang strings.Map func
- Golang bufio.ScanBytes, NewScanner (Read Bytes in File)
- Golang Built In Functions
- Golang bytes.Buffer Examples (WriteString, Fprintf)
- Golang Bytes: Slices and Methods
- Golang Caesar Cipher Method
- Golang Chan: Channels, Make Examples
- Golang Math Module: math.Abs, Pow
- Golang Reverse String
- Golang Struct Examples: Types and Pointers
- Golang path and filepath Examples (Base, Dir)
- Golang Substring Examples (Rune Slices)
- Golang Suffixarray Examples: New, Lookup Benchmark
- Golang switch Examples
- Golang Convert Map to Slice
- Golang Convert Slice to String: int, string Slices
- Golang Const, Var Examples: Iota
- Golang ROT13 Method
- Golang strings.Contains and ContainsAny
- Golang rand, crypto: Random Number Generators
- Golang String Literal Examples (Repeat Method)
- Golang ToLower, ToUpper String Examples
- Golang Trim, TrimSpace and TrimFunc Examples
- Golang Join Examples (strings.Join)
- Golang Len (String Length)
- Golang Convert String to Rune Slice (append)
- Golang JSON Example: Marshal, Unmarshal
- Golang Replace String Examples: Replacer, NewReplacer
- Golang nil (Cannot Use nil as Type)
- Golang Slice Examples
- Golang ListenAndServe Examples (HandleFunc)
- Golang Fibonacci Sequence Example
- Golang Time: Now, Parse and Duration
- Golang bits, OnesCount (Get Bitcount From Int)
- Golang Fprint, Fprintf and Fprintln Examples (fmt)
- Golang Func Examples
- Golang csv Examples
- Golang Fields and FieldsFunc
- Golang unicode.IsSpace (If Char Is Whitespace)
- Golang fmt.Println Examples
- Golang for Loop Examples: Foreach and While
- Golang ioutil.WriteFile, os.Create (Write File to Disk)
- Golang File Handling
- Golang range: Slice, String and Map
- Golang Readdir Example (Get All Files in Directory)
- Golang Sort Slice: Len, Less, Swap in Interface
- Golang Get Lines in File (String Slice)
- Golang Split Examples (SplitAfter, SplitN)