Wget ошибка 404 not found

I tried to download an image using wget but got an error like the following.

--2011-10-01 16:45:42--  http://www.icerts.com/images/logo.jpg
Resolving www.icerts.com... 97.74.86.3
Connecting to www.icerts.com|97.74.86.3|:80... connected.
HTTP request sent, awaiting response... 404 Not Found
2011-10-01 16:45:43 ERROR 404: Not Found.

My browser has no problem loading the image.
What’s the problem?
curl can’t download either.

Thanks.

Sam

asked Oct 1, 2011 at 23:47

I had same problem.
Solved using single quotes like this:

$ wget 'http://www.icerts.com/images/logo.jpg'

wget version in use:

$ wget --version
GNU Wget 1.11.4 Red Hat modified

answered Nov 9, 2017 at 13:57

Actually I don’t know what is the reason exactly, I have faced this like of problem.
if you have the domain’s IP address (ex 208.113.139.4), please use the IP address instead of domain (in this case www.icerts.com)

wget 192.243.111.11/images/logo.jpg

Go to find the IP from URL https://ipinfo.info/html/ip_checker.php

answered Aug 27, 2020 at 11:23

I want to add something to @blotus’s answer,

In case adding the referrer header does not solve the issue, May be you are using the wrong referrer (Sometimes the referrer is different from the URL’s domain name).

Paste the URL on a web browser and find the referrer from developer tools (Network -> Request Headers).

answered Jun 25, 2021 at 11:26

I met exactly the same problem while setting up GitHub actions with Cygwin. Only after I used wget --debug <url>, I realized that URL is appended with 0xd symbol which is r (carriage return).

For this kind of problem there is the solution described in docs:

you can also use igncr in the SHELLOPTS environment variable

So I added the following lines to my YAML script to make wget work properly, as well as other shell commands in my GHA workflow:

env:
  SHELLOPTS: igncr 

answered Nov 24, 2022 at 12:51

When trying to use the command line on Windows to wget a file, I’m getting a 404 error

C:Usersxxxx>wget + http://www.restaurantanzu.com/PDFs/Dinner908.pdf;
SYSTEM_WGETRC = c:/progra~1/wget/etc/wgetrc
syswgetrc = C:Program FilesGnuWin32/etc/wgetrc
--2011-06-07 15:59:19--  http://+/

Resolving . 67.199.65.121
Connecting to |67.199.65.121|:80... connected.
HTTP request sent, awaiting response... 404 Not Found
2011-06-07 15:59:21 ERROR 404: Not Found.

Drop the + (as mentioned by @baraboom) and also drop the semi-colon at the end of the line.

If you use brace expansion with wget, you can fetch sequentially-numbered images with ease:

$ wget 'http://www.iqandreas.com/sample-images/100-100-color/'{90..110}'.jpg'

It fetches the first 10 files numbered 90.jpg to 99.jpg just fine, but 100.jpg and onward return a 404: File not found error (I only have 100 images stored on the server). These non-existent files become more of «a problem» if you use a larger range, such as {00..200}, with 100 non-existent files, it increases the script’s execution time, and might even become a slight burden (or at least annoyance) on the server.

Is there any way for wget to stop after it has received its first 404 error? (or even better, two in a row, in case there was a missing file in the range for another reason) The answer does not need to use brace expansion; loops are fine too.

IMO, focusing in on wget‘s exit code/status may be too naive for some use-cases, so here is one that considers the HTTP Status Code as well for some granular decision making.

wget provides a -S/--server-response flag to print out the HTTP Response Headers on STDERR of the command — which we can extract and act upon.

#!/bin/bash

set -eu

error_max=2
error_count=0

urls=( 'http://www.iqandreas.com/sample-images/100-100-color/'{90..110}'.jpg' )

for url in "${urls[@]}"; do
  set +e
  http_status=$( wget --server-response -c "$url" 2>&1 )
  exit_status=$?
  http_status=$( awk '/HTTP//{ print $2 }' <<<"$http_status" | tail -n 1 )

  if (( http_status >= 400 )); then
    # Considering only HTTP Status errors
    case "$http_status" in
      # Define your actions for each 4XX Status Code below
      410) : Gone
        ;;
      416) : Requested Range Not Satisfiable
        error_count=0  # Reset error_count in case of `wget -c`
        ;;
      403) : Forbidden
        ;&
      404) : Not Found
        ;&
      *)     (( error_count++ ))
        ;;
    esac
  elif (( http_status >= 300 )); then
     # We're unlikely to reach here in case of 1XX, 3XX in $http_status
     # but ..
     exit_status=0
  elif (( http_status >= 200 )); then
     # 2XX in $http_status considered successful
     exit_status=0
  elif (( exit_status > 0 )); then

    # Where wget's exit status is one of
    # 1   Generic error code.
    # 2   Parse error 
    #     - when parsing command-line options, the .wgetrc or .netrc...
    # 3   File I/O error.
    # 4   Network failure.
    # 5   SSL verification failure.
    # 6   Username/password authentication failure.
    # 7   Protocol errors.

    (( error_count++ ))
  fi

  echo "$url -> http_status: $http_status, exit_status=$exit_status, error_count=$error_count" >&2

  if (( error_count >= error_max )); then
    echo "error_count $error_count >= $error_max, bailing out .." >&2
    exit "$exit_status"
  fi

done

answered Jul 2, 2017 at 14:55

With GNU Parallel this ought to work:

parallel --halt 1 wget ::: 'http://www.iqandreas.com/sample-images/100-100-color/'{90..110}'.jpg'

From version 20140722 you can almost have your «two in a row»-failure: —halt 2% will allow for 2% of the jobs to fail:

parallel --halt 2% wget ::: 'http://www.iqandreas.com/sample-images/100-100-color/'{90..110}'.jpg'

answered Jul 22, 2014 at 23:12

What I’ve used successfully is

wget 'http://www.iqandreas.com/sample-images/100-100-color/'{90..110}'.jpg' 2>&1 | grep -q 'ERROR 404: Not Found'

grep -q looks for the 404 error message pattern in its input and dies as soon as it sees it. wget receives a SIGPIPE signal as soon as it tries to write to the pipe from which grep is no longer reading. In practice wget dies pretty quickly after getting that first 404.

answered Aug 14, 2021 at 0:58