Codegena
extract-urls-from-website

How to extract urls from webpage for free?

Extracting links from a webpage has been a difficult task. You have to copy the links one by one. But this is tiresome and monotonous process. So we have come up with a simple and smart solution that works for all. All you need is a modern browser such as Chrome, IE, Safari, Firefox, etc.

1) Extract URLs Using Browser Console

The browser console is a great feature for developers and designers to try out their codes instantly. Until now the console area was loved only by them but today you will also start loving it. All you have to do to extract links from any website is to copy the code given below, then paste it into your browser console and hit enter. All of the hyperlinks will now to extracted from the webpage and displayed on the console. This method is the proven to be the fastest for extracting urls. The different variations of the javascript code are given below.

>> Extract URLs & Corresponding Anchor Text

The following is a cross browser supported code for extracting urls along with their anchor text.

var urls=$$('a');
for(url in urls){
 console.log("#"+url+" > "+urls[url].innerHTML +" >> "+urls[url].href)
}

>> Extract Links with their anchor text (For Chrome & Firefox)- Styled version

If you are using Chrome or Firefox use the following code for a styled version of the same.

extract-links-from-webpage-using-dev-console
Demo of extracting links from wikipedia page using dev console

 

var urls=$$('a');
for(url in urls){
 console.log("%c#"+url+" > %c"+urls[url].innerHTML +" >> %c"+urls[url].href,"color:red;","color:green;","color:blue;");
}

>> Extract URLs Only 

And if you want to extract just the links without the anchor text, then use the following code.

var urls=$$('a');
for(url in urls)
 console.log(urls[url].href);

 

>> Extract External URLs Only

External Links are the ones that point outside the current domain. If you want to extract the external URLs only, then this is the code you need to use.

var links = $$('a');
for (var i = links.length - 1; i > 0; i--) {
    if (links[i].host !== location.host) {
       console.log(links[i].href);
    }
}

>> Extract URLs with a specific extension

If you would like to extract links having a particular extension then paste the following code into the console. Pass the extension wrapped in quotes to the getLinksWithExtension() function. Please note that the following code extracts links from HTML link tag only (<a></a>)  and not from other tags such as script or img.

function getLinksWithExtension(extension) {
    var links = document.querySelectorAll('a[href$="' + extension + '"]'),
        i;

    for (i=0; i<links.length; i++){
        console.log(links[i]);
    }
}
getLinksWithExtension('mp3') //change mp3 to any extension

 

2) Online URL Extractor Website

There are situations when you cannot follow the above method such as when you are using a mobile. In situations like that you can follow this trick. iwebtool is a great site that offers URL extraction along with other features such as selective extraction of inbound or outbound links, anchor text extraction, etc. You can make 10 requests per hour with the free version of the tool. Visit iwebtool link extractor to get started. Enter the url in the text box and wait for the site to extract the links.

Weblogcode URL Extractor is another useful tool for extracting links from website.

3) Extract URLs from block of text

open-multi-urls-with-multiple-url-openerLet’s say you got a text file with a bunch of links in it and you are opening the links one by one. How boring it is? The answer to this tiring and monotonous work is multiple url opener tool by Codegena. To extract multiple urls from a text file:

  1. Go to Codegena URL Extractor and bulk url opener
  2. Paste the text into the text area and hit Linkify to generate clickable links.
  3. Click on “Open all url” button to automatically open all of the links in new tabs (allow popups to enable this feature)

You can also paste the source code of a website into this tool to extract the urls from it. However in such cases this tool will fail to extract relative links.

A similar tool is BuzzStream URL Extractor which also offers CSV export.

If you need any assistance, leave a comment below and I will get back to you.

Also Read

  • Siddharth Saini

    this is result of your code on this site:https://www.dudemobile.net/cl.php?id=8ab158761ba9eb7fcd38d98a4294a534

    “undefined”

    “Uncaught (in promise) DOMException: Failed to register a ServiceWorker: No URL is associated with the caller’s document”

    “Uncaught SyntaxError: Unexpected identifie”

    please explain properly n stepwise…….

    • Shan Eapen Koshy

      There are no URLs to extract from that site. That’s why it returned an error.

  • mcleuven

    Hi

    I know it´s an old threat but how can it be that the excact url extract doesnt show the same amount as an site:xxx?

    And thanks for great tips 🙂

    • mcleuven

      it´s regarding tilbuds-portalen.dk, this extract shows about 66 and a site:tilbuds-portalen.dk shows 177

  • ???????????????

    all medthod cannot extract url type “onclick” in sourcecode i am super surprise !!!!!!!!!!!!!!

    you try go to any web which contain url of id line which it is this format

    onclick=”location.href=’http://line.me/ti/p/~petermarrylove789′”

    your method can not extract this url !!!

    pls help me too. thank you very much

Follow Us

Don't be shy, get in touch. We love meeting interesting people and making new friends.