Python 3 Programming Tutorial - urllib module

Share
Embed
  • Published on Jul 19, 2014
  • The urllib module in Python 3 allows you access websites via your program. This opens up as many doors for your programs as the internet opens up for you. urllib in Python 3 is slightly different than urllib2 in Python 2, but they are mostly the same. Through urllib, you can access websites, download data, parse data, modify your headers, and do any GET and POST requests you might need to do.
    Sample code for this basics series: pythonprogramming.net/beginner-python-programming-tutorials/
    Python 3 Programming tutorial Playlist: ru-clip.net/video/oVp1vrfL_w4/video.html&feature=share&list=PLQVvvaa0QuDe8XSftW-RAxdo6OmaeL85M
    seaofbtc.com
    sentdex.com
    hkinsley.com
    twitter.com/sentdex
    Bitcoin donations: 1GV7srgR4NJx4vrk7avCmmVQQrqmv87ty6
  • Howto & StyleHowto & Style

Comments • 383

  • NIC FEL
    NIC FEL Month ago

    Hearing my glass sliding door open .. It's the Google COPS!

  • Troll
    Troll Month ago

    *Your dog should of used headers, than he wouldn’t of been caught*

  • Aline Guimarães
    Aline Guimarães Month ago

    I used this code from the video today and it didn't work, I think google has gotten smarter or something hahah

  • Tay
    Tay Month ago

    Hello sir thank you for this video! Can you please provide the user-agent header string? Thank you! subscribed/bell on!

  • Ajay Kumar
    Ajay Kumar 2 months ago

    What mistake I am doing? Can anyone figure out
    import urllib.request
    import urllib.parse
    url='pythonprogramming.net'
    values={'s':'basic','submit':'search'}
    data=urllib.phase.urlencode(values)
    data=data.encode('utf-8')
    req=urllib.request.Request(url,data)
    resp=urllib.request.urlopen(req)
    respData=resp.read()
    print(respData)

    error:
    Traceback (most recent call last):
    File "D:/PYTHON CODES/36.2.py", line 7, in
    data=urllib.phase.urlencode(values)
    AttributeError: module 'urllib' has no attribute 'phase'

  • A Jim Fan
    A Jim Fan 2 months ago

    Must be odd to discover that your dog has velociraptor powers

  • Manish Gupta
    Manish Gupta 2 months ago

    Sir, I am try to download json file (Amazon Transcribe json file) using url in python but it is given to me SSL: CERTIFICATE_VERIFY_FAILED. Please suggest me , how can do it.

  • Minespidur
    Minespidur 3 months ago

    I'm getting an error that there's no module named urllib.request and that 'urllib' is not a package? Has anyone else figured this out?

  • Damián Hruban
    Damián Hruban 3 months ago

    Cheers mate! You saved my project!

  • Ivan Gradjansky
    Ivan Gradjansky 3 months ago +2

    Hi all, I'm having difficulty with getting url.request to work on sublimetext, has anyone run into an error saying:
    import urllib.request
    ImportError: No module named request?
    If so how did you fix the error? I've also tried running it as his code suggested using "import urllib"
    req = urllib.urlretrieve.urlopen(url,data=None)
    and it printed out: AttributeError: 'function' object has no attribute 'urlopen'
    Thanks

    • Bin Liu
      Bin Liu 2 months ago

      +Minespidur same

    • Minespidur
      Minespidur 3 months ago

      I too am having this issue and curious

  • Julius Waweru
    Julius Waweru 3 months ago

    Your video was awesome

  • prashant pal
    prashant pal 4 months ago

    Any recent success using urllib.request.urlopen(url, headers=headers)?
    I get the following error:
    urlopen() got an unexpected keyword argument 'headers'
    I am using python 3.7 and the urlopen here:
    urllib.request.urlopen(url, data=None, [timeout, ]*, cafile=None, capath=None, cadefault=False, context=None)
    Ultimately does the site know you are using a python script with the latest lib?

    • prashant pal
      prashant pal 4 months ago

      Found a solution:
      from urllib.request import Request, urlopen
      url = 'pythonprogramming.net/'
      headers = {'User-Agent': 'Mozilla/5.0'}
      request = Request('url', headers=headers)
      print(urlopen(request).read())

  • Ethan Lee
    Ethan Lee 4 months ago

    what about verifying SSL certs?

  • Luka Biskupić
    Luka Biskupić 4 months ago

    Can someone explain urllib.request.Reguest(url,data)
    Thanks

  • Remil Eigis
    Remil Eigis 4 months ago

    "Turns out my dog can open sliding doors". Has ML gone too far??

  • Everette Shaw
    Everette Shaw 4 months ago

    I know what u did with Harris's site I am now exposing you

  • Everette Shaw
    Everette Shaw 4 months ago

    HumanRightsDay_2018

  • Everette Shaw
    Everette Shaw 4 months ago

    GOT_LATAM_2018

  • Anirvan Sen
    Anirvan Sen 5 months ago

    Working example:
    import urllib.request
    import urllib.parse

    url = 'www.google.com/search?q=test'

    try:
    x = urllib.request.urlopen(url)
    print(x.read())
    except Exception as e:
    print(str(e))

    try:
    headers = {}
    headers['User-Agent'] ="Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/70.0.3538.77 Safari/537.36"
    req = urllib.request.Request(url,headers=headers)
    res = urllib.request.urlopen(req)
    resData = res.read()
    with open('withHeaders.txt','w') as wf:
    wf.write(str(resData))

    except Exception as e:
    print(str(e))

  • q zorn
    q zorn 6 months ago

    5StarLike. Do you have information on how to urllib in opencv to a IP camera video, audio, PTZ controls? Thanks.

  • Raptr3x
    Raptr3x 6 months ago

    Why not import urllib.request as rq or whatever so you don't have to write every single time urllb.request? I mean you probably forgot or something but it's a nice touch I'd add to this video :) On the other side I love what you are doing and your tutorials got me far, so thank you for all of this :)

  • TipsyPistole
    TipsyPistole 6 months ago +2

    just a heads up if you name your file urllib.py your gonna have a bad day.... don't do it...

  • Ray Lee
    Ray Lee 7 months ago

    one issue,
    I got the following: after input. how to deal with?
    "import urllib.request
    x=urllib.request.urlopen('www.google.com')
    print(x.read())
    "
    Unhandled server exception!
    Thread: SockThread
    Client Address: ('127.0.0.1', 53046)
    Request:
    Traceback (most recent call last):
    File "C:\Users\lizrx\AppData\Local\Programs\Python\Python37\lib\socketserver.py", line 313, in _handle_request_noblock
    self.process_request(request, client_address)
    File "C:\Users\lizrx\AppData\Local\Programs\Python\Python37\lib\socketserver.py", line 344, in process_request
    self.finish_request(request, client_address)
    File "C:\Users\lizrx\AppData\Local\Programs\Python\Python37\lib\socketserver.py", line 357, in finish_request
    self.RequestHandlerClass(request, client_address, self)
    File "C:\Users\lizrx\AppData\Local\Programs\Python\Python37\lib\idlelib\rpc.py", line 511, in __init__
    socketserver.BaseRequestHandler.__init__(self, sock, addr, svr)
    File "C:\Users\lizrx\AppData\Local\Programs\Python\Python37\lib\socketserver.py", line 717, in __init__
    self.handle()
    File "C:\Users\lizrx\AppData\Local\Programs\Python\Python37\lib\idlelib\run.py", line 433, in handle
    import pydoc # import must be done here to capture i/o binding
    File "C:\Users\lizrx\AppData\Local\Programs\Python\Python37\lib\pydoc.py", line 71, in
    import urllib.parse
    File "C:\Users\lizrx\AppData\Local\Programs\Python\Python37\urllib.py", line 1, in
    import urllib.request
    ModuleNotFoundError: No module named 'urllib.request'; 'urllib' is not a package
    *** Unrecoverable, server exiting!

  • Saad Ilyas
    Saad Ilyas 7 months ago

    Sentdex, amazing video Sir. Instead of using urllib to get content from a website why don't we use requests with bs4?

  • Fayozbek Rustamov
    Fayozbek Rustamov 7 months ago

    hello Ser! thank you for your great tutorials. I have tried to use your code for google.com and others it works but for this url (www.cvedetails.com) it does not work? I will be glad if help someone to solve this problem! thank you!

  • fr06t
    fr06t 7 months ago

    "Mozilla/5.0 (X11; Linux i686) AppleWebKit/537.17 (KHTML, like Gecko) Chrome/24.0.1312.27 Safari/537.17"
    for those who don't want to bother going to the website

  • can you see me
    can you see me 7 months ago

    No where does it state "data" needs to be a dictionary,how did you know all that?

  • vardhan negi
    vardhan negi 7 months ago +1

    what i'm getting when i run the code
    "ModuleNotFoundError: No module named 'urllib.request'; 'urllib' is not a package"
    i'm using python3.7 and having urllib3 in my site-packages but still getting this error

  • webie seo
    webie seo 8 months ago

    Can you cover requests.get(image_url) I have made a script which is my first whats happening is when it hits 404 script just stops it don't crash it waits for ever how can i set if image URL don't exist 404 error download alternative image url then continue ? hope you can help my friend

  • Honest Landscapes
    Honest Landscapes 8 months ago

    3:21 in case of google i got the source code but in other website i got urllib.error.HTTPError: HTTP Error 405: Not Allowed, why so????

  • Venom bb
    Venom bb 8 months ago +7

    Make a series on Advance Python 3. Cover all the hacky things in that series.

  • WOYAODIAOREN
    WOYAODIAOREN 8 months ago

    I tried to follow everything along, and got error "", not able to create the text file.

  • Alina Stechyshyn
    Alina Stechyshyn 8 months ago

    What app do you run to display the python output? It opens a new window every time you execute a script.

  • Bas Stock
    Bas Stock 8 months ago +2

    17:16 FBI got a tipoff that Snowden was in the US all along

  • enyao93
    enyao93 8 months ago

    why did you not print(respData) and instead saveFile.write(str(respData))?

  • Leigh Thorpe
    Leigh Thorpe 9 months ago

    Hi sir, thanks for doing these videos, they're really good. Whilst going through this lesson, I got an error . Could you help me understand why that happened?

  • Daniel Keehn
    Daniel Keehn 9 months ago

    This helped a lot, thanks!

  • Adon Lando
    Adon Lando 9 months ago

    Great information. Great delivery! Subscribed.

  • Amogh Dadhich
    Amogh Dadhich 10 months ago

    Hi! For some reason I can't seem to make requests to google while passing data(it throws the 405 exception). However, I'm able to make the same request if I 'hard-code' the parameters into the url. Any idea where I might be going wrong?

  • Hariharasudan Sudhan
    Hariharasudan Sudhan 10 months ago

    i tried this code and then on my python shell not showing any other python script results even print('hello world') too and as well it's launching but can't able to close shell window.. what might be the problem..? i have tried to reinstall and the problem is continuing. give me the solution

  • mangesh rane
    mangesh rane 10 months ago

    url = "reqres.in/api/users"
    data = { "page": 2}

    urlencoded = urllib.parse.urlencode(data)
    urlencoded = urlencoded.encode('utf-8')
    request = urllib.request.Request(url, data)
    response = urllib.request.urlopen(request)
    print(response.read())
    This is giving me error
    TypeError: can't concat str to bytes

  • Sarthak Garg
    Sarthak Garg 10 months ago +1

    in the last example, if instead of barcoding, I want to use urllib.parse.encode function, how will my req variable look like?
    like this?
    req=urllib.request.Request(url, data,headers=headers)

  • Daniel
    Daniel 10 months ago

    15:23 your address, your gender, your political views, what are your goals and aspirations, your job, if you pay your taxes or not mr neo, what you ate this morning

  • shikhar gairola
    shikhar gairola 11 months ago

    Hi, when I am trying to open a url it gives me a nonnumeric port: 'XXXX':DIRECT exception.
    could you please help me solving this issue.

  • Tirthraj Mahajan
    Tirthraj Mahajan 11 months ago +1

    Here you go - Only thing i changed is the headers['User-Agent'].. thats it and it should work just fine!
    import urllib.request
    import urllib.parse
    try:
    url = 'www.google.com/search?q=test'
    headers = {}
    headers['User-Agent'] = 'Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/51.0.2704.103 Safari/537.36'
    req = urllib.request.Request(url, headers=headers)
    resp = urllib.request.urlopen(req)
    respData = resp.read()
    saveFile = open('withHeaders.txt','w')
    saveFile.write(str(respData))
    saveFile.close()
    except Exception as e:
    print(str(e))

  • Zachary Kolansky
    Zachary Kolansky 11 months ago +1

    Great tutorial still works perfectly except that you need to run "Install Certificates.command" (at least for MacOS user like me) before using urllib. If you don't, then you get the world's scariest error.

  • Harith Perera
    Harith Perera 11 months ago

    How to bypass google.com/finance ?

  • Abhishek Gautam
    Abhishek Gautam Year ago

    u r awesome bro

  • Stephen Mandelbaum

    Could you just do: from urllib import *

  • Puli Poorna Shekar Reddy

    can u help me in scarping this site
    vtopbeta.vit.ac.in/vtop/

  • Sanasi Sitlani
    Sanasi Sitlani Year ago

    Is this data only from first search page or all of it from all pages?

  • Sanasi Sitlani
    Sanasi Sitlani Year ago

    How to get Image load time through this?

  • Lol Lol
    Lol Lol Year ago

    Love and respect you

  • Karol F
    Karol F Year ago

    You pretty cool guy showing some pretty cool stuff. Its a bit funny when you speak about that kind of stuff with this hat on your head. Youre doing some great job. Thanks! :)

  • Jean DAVID
    Jean DAVID Year ago

    Unfortunately, the video is out of date !
    the search parameters have changed on your website : search now begins with 'url/search/?q=basic' instead of 'url/?s=basic & submit='Search'

  • Explosive
    Explosive Year ago

    Are you Bucky Roberts?

  • Raghav Gupta
    Raghav Gupta Year ago

    are urllib and requests modules same??

    • sentdex
      sentdex  Year ago +1

      They can do the same things, but they are not the same.

  • Guilherme
    Guilherme Year ago

    Thanks for your videos, they are very helpful! I've disabled my adblock to watch your videos, hope the other viewers would do the same! :)

  • bell canada
    bell canada Year ago

    hello where is the file saved??..i dont find it..it just prints the source code in the cmd prompt..thNK YOU

    • bell canada
      bell canada Year ago

      well its ok..in notepad regualr its all vertical..but not notepad++..cheers

    • bell canada
      bell canada Year ago

      but i noticed it prints the contents in the new file only on one line..whcuh is not readable really..do you know how i can modify this so its like in you vid ..verticall?/..thnaxz

    • bell canada
      bell canada Year ago

      ok ..i found it ..its in the same folder ..which is the /python/python.exe folder..where i have to be when using the python at the cmd prompt..cheers

  • UU
    UU Year ago

    thanks for this but the code in the beginning of the tutorial gives error.
    import urllib.request
    x = urllib.request.urlopen('www.google.com')
    print(x.read())
    URLError:

  • Fabio Marcos
    Fabio Marcos Year ago

    Work!!!
    try:
    url = 'www.google.com/search?q=python'
    # now, with the below headers, we defined ourselves as a simpleton who is
    # still using internet explorer.
    headers = {}
    headers['User-Agent'] = "Mozilla/5.0 (X11; Linux i686) AppleWebKit/537.17 (KHTML, like Gecko) Chrome/24.0.1312.27 Safari/537.17"
    req = urllib.request.Request(url, headers = headers)
    resp = urllib.request.urlopen(req)
    respData = resp.read()
    saveFile = open('withHeaders.txt','w')
    saveFile.write(str(respData))
    saveFile.close()
    except Exception as e:
    print(str(e))

  • Anjaly Abraham
    Anjaly Abraham Year ago

    when i used sys and run through the command prompt trying to add with a value in that file it says str and int objects cannot concannete.is it cz it takes all input as strings?if yes why?u tried with numbers and it worked.

  • Boris Schnider
    Boris Schnider Year ago

    Hi, I tried many different User-Agents, even the one given directly by google and I always get:
    HTTP Error 403: Forbidden
    Would anyone have any idea whether it is due to google filter or a mistake of mine? (see code below)
    try:
    url = urllib.request.urlopen("www.google.com/search?q=test")
    headers = {'User-Agent': 'Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:58.0) Gecko/20100101 Firefox/58.0'}
    req = urllib.request.Request(url, headers=headers)#make headers the default headers
    resp = urllib.request.urlopen(req)
    respData = resp.read()

    saveFile = open('withHeaders.txt', 'w')
    saveFile.write(str(respData))
    saveFile.close()
    except Exception as e:
    print(str(e))

  • Precious Chima
    Precious Chima Year ago

    thats my erro how do i fix it ???
    Traceback (most recent call last):
    File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/urllib/request.py", line 1318, in do_open
    encode_chunked=req.has_header('Transfer-encoding'))
    File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/http/client.py", line 1239, in request
    self._send_request(method, url, body, headers, encode_chunked)
    File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/http/client.py", line 1285, in _send_request
    self.endheaders(body, encode_chunked=encode_chunked)
    File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/http/client.py", line 1234, in endheaders
    self._send_output(message_body, encode_chunked=encode_chunked)
    File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/http/client.py", line 1026, in _send_output
    self.send(msg)
    File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/http/client.py", line 964, in send
    self.connect()
    File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/http/client.py", line 1400, in connect
    server_hostname=server_hostname)
    File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/ssl.py", line 407, in wrap_socket
    _context=self, _session=session)
    File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/ssl.py", line 814, in __init__
    self.do_handshake()
    File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/ssl.py", line 1068, in do_handshake
    self._sslobj.do_handshake()
    File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/ssl.py", line 689, in do_handshake
    self._sslobj.do_handshake()
    ssl.SSLError: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:777)
    During handling of the above exception, another exception occurred:
    Traceback (most recent call last):
    File "/Users/ikennapreciouschima/Documents/Practice.py", line 3, in
    x = urllib.request.urlopen('www.google.com')
    File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/urllib/request.py", line 223, in urlopen
    return opener.open(url, data, timeout)
    File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/urllib/request.py", line 526, in open
    response = self._open(req, data)
    File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/urllib/request.py", line 544, in _open
    '_open', req)
    File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/urllib/request.py", line 504, in _call_chain
    result = func(*args)
    File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/urllib/request.py", line 1361, in https_open
    context=self._context, check_hostname=self._check_hostname)
    File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/urllib/request.py", line 1320, in do_open
    raise URLError(err)
    urllib.error.URLError:
    >>>
    ========= RESTART: /Users/ikennapreciouschima/Documents/Practice.py =========
    Traceback (most recent call last):
    File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/urllib/request.py", line 1318, in do_open
    encode_chunked=req.has_header('Transfer-encoding'))
    File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/http/client.py", line 1239, in request
    self._send_request(method, url, body, headers, encode_chunked)
    File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/http/client.py", line 1285, in _send_request
    self.endheaders(body, encode_chunked=encode_chunked)
    File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/http/client.py", line 1234, in endheaders
    self._send_output(message_body, encode_chunked=encode_chunked)
    File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/http/client.py", line 1026, in _send_output
    self.send(msg)
    File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/http/client.py", line 964, in send
    self.connect()
    File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/http/client.py", line 1400, in connect
    server_hostname=server_hostname)
    File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/ssl.py", line 407, in wrap_socket
    _context=self, _session=session)
    File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/ssl.py", line 814, in __init__
    self.do_handshake()
    File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/ssl.py", line 1068, in do_handshake
    self._sslobj.do_handshake()
    File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/ssl.py", line 689, in do_handshake
    self._sslobj.do_handshake()
    ssl.SSLError: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:777)
    During handling of the above exception, another exception occurred:
    Traceback (most recent call last):
    File "/Users/ikennapreciouschima/Documents/Practice.py", line 16, in
    resp = urllib.request.urlopen(req)
    File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/urllib/request.py", line 223, in urlopen
    return opener.open(url, data, timeout)
    File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/urllib/request.py", line 532, in open
    response = meth(req, response)
    File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/urllib/request.py", line 642, in http_response
    'http', request, response, code, msg, hdrs)
    File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/urllib/request.py", line 564, in error
    result = self._call_chain(*args)
    File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/urllib/request.py", line 504, in _call_chain
    result = func(*args)
    File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/urllib/request.py", line 756, in http_error_302
    return self.parent.open(new, timeout=req.timeout)
    File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/urllib/request.py", line 526, in open
    response = self._open(req, data)
    File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/urllib/request.py", line 544, in _open
    '_open', req)
    File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/urllib/request.py", line 504, in _call_chain
    result = func(*args)
    File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/urllib/request.py", line 1361, in https_open
    context=self._context, check_hostname=self._check_hostname)
    File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/urllib/request.py", line 1320, in do_open
    raise URLError(err)
    urllib.error.URLError:
    >>>

  • Abhishek Mann
    Abhishek Mann Year ago +2

    This User-Agent string worked for me:
    Mozilla/5.0 (Windows NT 6.1; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/62.0.3202.62 Safari/537.36
    You can basically find a plethora of user-agent strings to play around with on here:
    udger.com/resources/ua-list

  • Chris Razon
    Chris Razon Year ago

    mr. sentdex you should do a dictionary for python, i mean some keywords are hard to understand

  • Chris Razon
    Chris Razon Year ago

    can you do software development

  • Chris Razon
    Chris Razon Year ago

    what is parse?

  • Chris Razon
    Chris Razon Year ago

    he gave meaning to my life by using python

  • ok ko
    ok ko Year ago

    anyone know what the python script would be to open chrome and search a question which is in the form of .txt from an ocr?

  • IGR Gaming
    IGR Gaming Year ago

    I have got one question, When we can open the page in our browser and do view source, why would we want to take the same data in programming way where we have to bypass some things ?

  • Nikolakakis Konstantinos

    Hi, great video! Where can I find the information for the required values for different websites?

  • ENGINEER R.J.
    ENGINEER R.J. Year ago

    exceptions are showing ''
    Unhandled server exception!'

  • Jermain Singleton

    Was nice to see this but I could not follow at all.

  • Hans Monmister
    Hans Monmister Year ago

    how to deal with "HTTP Error 405: Method Not Allowed"??

  • impyrobot
    impyrobot Year ago +49

    when the NSA are coming for you 17:16

  • Benjamin Shtark
    Benjamin Shtark Year ago

    a question.
    how would u do authentication if a website requires username + password?

  • znd mhyn
    znd mhyn Year ago

    /usr/local/lib/python2.7/dist-packages/urllib3/util/ssl_.py:339: SNIMissingWarning: An HTTPS request has been made, but the SNI (Subject Name Indication) extension to TLS is not available on this platform. This may cause the server to present an incorrect TLS certificate, which can cause validation failures. You can upgrade to a newer version of Python to solve this. For more information, see urllib3.readthedocs.io/en/latest/advanced-usage.html#ssl-warnings
    SNIMissingWarning
    /usr/local/lib/python2.7/dist-packages/urllib3/util/ssl_.py:137: InsecurePlatformWarning: A true SSLContext object is not available. This prevents urllib3 from configuring SSL appropriately and may cause certain SSL connections to fail. You can upgrade to a newer version of Python to solve this. For more information, see urllib3.readthedocs.io/en/latest/advanced-usage.html#ssl-warnings
    InsecurePlatformWarning

    ^ how to fix this ?

  • Jatin jain
    Jatin jain Year ago

    it's showing exception:

  • Damien Cupif
    Damien Cupif Year ago

    Is this video outdated, and should I be learning about the requests library? Seems much easier to handle than urllib. Thanks for your great videos sentdex.

  • Martin Kaspar
    Martin Kaspar Year ago

    Hello great tutorial

    btw : when i run this on my machine (python 3.4xy ) i get errors
    import urllib.request
    import urllib.parse
    url = 'pythonprogramming.net'
    values = ('s':'basic',
    'submit':'search')

    data = urllib.parse.urlencode(values)
    data = data.encode ('utf-8')
    req = urllib.request.Request(url, data)
    res = urllib.request.urlopen(req)
    resp.Data = resp.read ()
    print (resp.Data)

  • Martin Kaspar
    Martin Kaspar Year ago

    Hello great tutorial

    btw : when i run this on my machine (python 3.4xy ) i get errors
    import urllib.request
    import urllib.parse
    url = 'pythonprogramming.net'
    values = ('s':'basic',
    'submit':'search')

    data = urllib.parse.urlencode(values)
    data = data.encode ('utf-8')
    req = urllib.request.Request(url, data)
    res = urllib.request.urlopen(req)
    resp.Data = resp.read ()
    print (resp.Data)

  • Lopoga X
    Lopoga X Year ago +1

    LOL: i.imgur.com/VOG2Qd9.png

  • MakCorps API
    MakCorps API Year ago

    great buddy

  • Dominic Brown
    Dominic Brown Year ago

    I cant even get past the second line [x = urllib.request.urlopen('www.google.com')]
    all i get is a load of error messages and a big one at the end saying ssl.SSLError: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:777) ...
    I would love some help

  • Moulik Aggarwal
    Moulik Aggarwal Year ago

    Can you give a video of module urllib3 as urllib and urllib2 module is now not in python3..

  • Jatin jain
    Jatin jain Year ago

    It's showing error in pycharm
    Showing module urllib.request module doesnt exist

  • sanj
    sanj Year ago +5

    Just found out about this channel today - and i've already bookmarked the channel as "Snowden the second"

  • Mohamed Ayman
    Mohamed Ayman Year ago

    Why did you assign - values - with a dictionary?
    6:08

  • avi redmi
    avi redmi Year ago

    Change headers: read documents? You mean py documentation?

  • Alaka Premanand
    Alaka Premanand Year ago

    i get an error like "HTTP error 405 , method not allowed' i use python 3.6.2

  • Chad Williams
    Chad Williams Year ago

    What do I do if I am getting a 403 error and I need to get at a javascript data? Doesn't the urllib.request.urlopen only get the html?

  • sairam reddy
    sairam reddy Year ago

    hiii everyone ,
    i need small regarding how to we use urllib and why we use it in the python,
    so any one help to clear understand to me .

  • Bread Eater
    Bread Eater Year ago

    Ah este vato c mamo

  • aditya verma
    aditya verma Year ago +83

    That moment when you think to fool Google and you doors start opening ... XD

  • Jake Ambrose
    Jake Ambrose Year ago

    u scared the shit out of me with the door opening

  • Jake Ambrose
    Jake Ambrose Year ago

    so it could be used say if i wanted to find a specific section or terms in the code of a page i could use this to find it amongst all the code?

  • Sungchul Lee
    Sungchul Lee Year ago

    Many thanks for the great presentation. Here is a working code that I modify a little bit from your last code in this class:
    import urllib.request
    url_plus_data = 'www.google.com/search?q=python'
    headers = {}
    headers['User-Agent'] = "Mozilla/5.0 (X11; Linux i686) AppleWebKit/537.17 (KHTML, like Gecko) Chrome/24.0.1312.27 Safari/537.17"
    url_plus_data_with_header = urllib.request.Request(url_plus_data, headers=headers)
    x = urllib.request.urlopen(url_plus_data_with_header)
    # print(x.read()) # have to kill this line to save as a txt file
    x_data = x.read()
    with open('with_headers.txt','w') as f:
    f.write(str(x_data))
    When I dis-activate the line "# print(x.read()) # have to kill this line to save as a txt file", I get a correct txt file. However, when I activate the line "# print(x.read()) # have to kill this line to save as a txt file", I get an almost empty txt file. I don't know why? Would you explain this?

  • Mahmutaj Dren
    Mahmutaj Dren Year ago

    Fantastic Tutorial man ! I've been struggeling on that issue for 3 days, now thanks to you I can carry on my web scrapping !!

  • Deco Padaratz
    Deco Padaratz Year ago

    Hi Dude.
    I have a very large .csv file that is on the internet
    I would like to bring a sample of this csv using the urlopen. I have been looking for some solution and I have not found it.
    Is there any way to do this?
    This is the dataset:
    from urllib.request import urlopen
    response = urlopen("s3.amazonaws.com/carto-1000x/data/yellow_tripdata_2016-01.csv").read().decode('utf8')